datalad.cmd

Wrapper for command and function calls, allowing for dry runs and output handling

class datalad.cmd.BatchedCommand(cmd, path=None, output_proc=None)[source]

Bases: datalad.cmd.SafeDelCloseMixin

Container for a process which would allow for persistent communication

close(return_stderr=False)[source]

Close communication and wait for process to terminate

Returns:stderr output if return_stderr and stderr file was there. None otherwise
Return type:str
proc1(arg)[source]

Same as __call__, but only takes a single command argument

and returns a single result.

yield_(cmds)[source]

Same as __call__, but requires cmds to be an iterable

and yields results for each item.

class datalad.cmd.GitRunnerBase[source]

Bases: object

Mix-in class for Runners to be used to run git and git annex commands

Overloads the runner class to check & update GIT_DIR and GIT_WORK_TREE environment variables set to the absolute path if is defined and is relative path

static get_git_environ_adjusted(env=None)[source]

Replaces GIT_DIR and GIT_WORK_TREE with absolute paths if relative path and defined

class datalad.cmd.GitWitlessRunner(*args, **kwargs)[source]

Bases: datalad.cmd.WitlessRunner, datalad.cmd.GitRunnerBase

A WitlessRunner for git and git-annex commands.

See GitRunnerBase it mixes in for more details

run_on_filelist_chunks(cmd, files, protocol=None, cwd=None, env=None, **kwargs)[source]

Run a git-style command multiple times if files is too long

Parameters:
  • cmd (list) – Sequence of program arguments.
  • files (list) – List of files.
  • protocol (WitlessProtocol, optional) – Protocol class handling interaction with the running process (e.g. output capture). A number of pre-crafted classes are provided (e.g KillOutput, NoCapture, GitProgress).
  • cwd (path-like, optional) – If given, commands are executed with this path as PWD, the PWD of the parent process is used otherwise. Overrides any cwd given to the constructor.
  • env (dict, optional) – Environment to be used for command execution. If cwd was given, ‘PWD’ in the environment is set to its value. This must be a complete environment definition, no values from the current environment will be inherited. Overrides any env given to the constructor.
  • kwargs – Passed to the Protocol class constructor.
Returns:

At minimum there will be keys ‘stdout’, ‘stderr’ with unicode strings of the cumulative standard output and error of the process as values.

Return type:

dict

Raises:
  • CommandError – On execution failure (non-zero exit code) this exception is raised which provides the command (cmd), stdout, stderr, exit code (status), and a message identifying the failed command, as properties.
  • FileNotFoundError – When a given executable does not exist.
class datalad.cmd.KillOutput(done_future, encoding=None)[source]

Bases: datalad.cmd.WitlessProtocol

WitlessProtocol that swallows stdout/stderr of a subprocess

pipe_data_received(fd, data)[source]

Called when the subprocess writes data into stdout/stderr pipe.

fd is int file descriptor. data is bytes object.

proc_err = True
proc_out = True
class datalad.cmd.NoCapture(done_future, encoding=None)[source]

Bases: datalad.cmd.WitlessProtocol

WitlessProtocol that captures no subprocess output

As this is identical with the behavior of the WitlessProtocol base class, this class is merely a more readable convenience alias.

class datalad.cmd.SafeDelCloseMixin[source]

Bases: object

A helper class to use where __del__ would call .close() which might fail if “too late in GC game”

class datalad.cmd.StdErrCapture(done_future, encoding=None)[source]

Bases: datalad.cmd.WitlessProtocol

WitlessProtocol that only captures and returns stderr of a subprocess

proc_err = True
class datalad.cmd.StdOutCapture(done_future, encoding=None)[source]

Bases: datalad.cmd.WitlessProtocol

WitlessProtocol that only captures and returns stdout of a subprocess

proc_out = True
class datalad.cmd.StdOutErrCapture(done_future, encoding=None)[source]

Bases: datalad.cmd.WitlessProtocol

WitlessProtocol that captures and returns stdout/stderr of a subprocess

proc_err = True
proc_out = True
class datalad.cmd.WitlessProtocol(done_future, encoding=None)[source]

Bases: asyncio.protocols.SubprocessProtocol

Subprocess communication protocol base class for run_async_cmd

This class implements basic subprocess output handling. Derived classes like StdOutCapture should be used for subprocess communication that need to capture and return output. In particular, the pipe_data_received() method can be overwritten to implement “online” processing of process output.

This class defines a default return value setup that causes run_async_cmd() to return a 2-tuple with the subprocess’s exit code and a list with bytestrings of all captured output streams.

FD_NAMES = ['stdin', 'stdout', 'stderr']
connection_made(transport)[source]

Called when a connection is made.

The argument is the transport representing the pipe connection. To receive data, wait for data_received() calls. When the connection is closed, connection_lost() is called.

pipe_data_received(fd, data)[source]

Called when the subprocess writes data into stdout/stderr pipe.

fd is int file descriptor. data is bytes object.

proc_err = None
proc_out = None
process_exited()[source]

Called when subprocess has exited.

class datalad.cmd.WitlessRunner(cwd=None, env=None)[source]

Bases: object

Minimal Runner with support for online command output processing

It aims to be as simple as possible, providing only essential functionality.

cwd
env
run(cmd, protocol=None, stdin=None, cwd=None, env=None, **kwargs)[source]

Execute a command and communicate with it.

Parameters:
  • cmd (list or str) – Sequence of program arguments. Passing a single string causes execution via the platform shell.
  • protocol (WitlessProtocol, optional) – Protocol class handling interaction with the running process (e.g. output capture). A number of pre-crafted classes are provided (e.g KillOutput, NoCapture, GitProgress).
  • stdin (byte stream, optional) – File descriptor like, used as stdin for the process. Passed verbatim to subprocess.Popen().
  • cwd (path-like, optional) – If given, commands are executed with this path as PWD, the PWD of the parent process is used otherwise. Overrides any cwd given to the constructor.
  • env (dict, optional) – Environment to be used for command execution. If cwd was given, ‘PWD’ in the environment is set to its value. This must be a complete environment definition, no values from the current environment will be inherited. Overrides any env given to the constructor.
  • kwargs – Passed to the Protocol class constructor.
Returns:

At minimum there will be keys ‘stdout’, ‘stderr’ with unicode strings of the cumulative standard output and error of the process as values.

Return type:

dict

Raises:
  • CommandError – On execution failure (non-zero exit code) this exception is raised which provides the command (cmd), stdout, stderr, exit code (status), and a message identifying the failed command, as properties.
  • FileNotFoundError – When a given executable does not exist.
datalad.cmd.readline_rstripped(stdout)[source]
datalad.cmd.run_async_cmd(loop, cmd, protocol, stdin, protocol_kwargs=None, **kwargs)[source]

Run a command in a subprocess managed by asyncio

This implementation has been inspired by https://pymotw.com/3/asyncio/subprocesses.html

Parameters:
  • loop (asyncio.AbstractEventLoop) – asyncio event loop instance. Must support subprocesses on the target platform.
  • cmd (list or str) – Command to be executed, passed to subprocess_exec (list), or subprocess_shell (str).
  • protocol (WitlessProtocol) – Protocol class to be instantiated for managing communication with the subprocess.
  • stdin (file-like or None) – Passed to the subprocess as its standard input.
  • protocol_kwargs (dict, optional) – Passed to the Protocol class constructor.
  • kwargs (Pass to subprocess_exec, will typically be parameters) – supported by subprocess.Popen.
Returns:

The nature of the return value is determined by the given protocol class.

Return type:

undefined