datalad_next.shell

A persistent shell connection

This module provides a context manager that establishes a connection to a shell and can be used to execute multiple commands in that shell. Shells are usually remote shells, e.g. connected via an ssh-client, but local shells like zsh, bash or PowerShell can also be used.

The context manager returns an instance of ShellCommandExecutor that can be used to execute commands in the shell via the method ShellCommandExecutor.__call__(). The method will return an instance of a subclass of ShellCommandResponseGenerator that can be used to retrieve the output of the command, the result code of the command, and the stderr-output of the command.

Every response generator expects a certain output structure. It is responsible for ensuring that the output structure is generated. To this end every response generator provides a method ShellCommandResponseGenerator.get_command_list(). The method ShellCommandExecutor.__call__ will pass the user-provided command to ShellCommandResponseGenerator.get_command_list() and receive a list of final commands that should be executed in the connected shell and that will generate the expected output structure. Instances of ShellCommandResponseGenerator have therefore four tasks:

1. Create a final command list that is used to execute the user provided command. This could, for example, execute the command, print an end marker, and print the return code of the command.

2. Parse the output of the command, yield it to the user.

3. Read the return code and provide it to the user.

4. Provide stderr-output to the user.

A very versatile example of a response generator is the class VariableLengthResponseGenerator. It can be used to execute a command that will result in an output of unknown length, e.g. ls, and will yield the output of the command to the user. It does that by using a random end marker to detect the end of the output and read the trailing return code. This is suitable for almost all commands.

If VariableLengthResponseGenerator is so versatile, why not just implement its functionality in ShellCommandExecutor? There are two major reasons for that:

1. Although the VariableLengthResponseGenerator is very versatile, it is not the most efficient implementation for commands that produce large amounts of output. In addition, there is also a minimal risk that the end marker is part of the output of the command, which would trip up the response generator. Putting response generation into a separate class allows to implement specific operations more efficiently and more safely. For example, DownloadResponseGenerator implements the download of files. It takes a remote file name as user "command" and creates a final command list that emits the length of the file, a newline, the file content, a return code, and a newline. This allows DownloadResponseGenerator to parse the output without relying on an end marker, thus increasing efficiency and safety

2. Factoring out the response generation creates an interface that can be used to support the syntax of different shells and the difference in command names and options in different operating systems. For example, the response generator class VariableLengthResponseGeneratorPowerShell supports the invocation of commands with variable length output in a PowerShell.

In short, parser generator classes encapsulate details of shell-syntax and operation implementation. That allows support of different shell syntax, and the efficient implementation of specific higher level operations, e.g. download. It also allows users to extend the functionality of ShellCommandExecutor by providing their own response generator classes.

The module datalad_next.shell.response_generators provides two generally applicable abstract response generator classes:

• VariableLengthResponseGenerator

• FixedLengthResponseGenerator

The functionality of the former is described above. The latter can be used to execute a command that will result in output of known length, e.g. echo -n 012345. It reads the specified number of bytes and a trailing return code. This is more performant than the variable length response generator (because it does not have to search for the end marker). In addition, it does not rely on the uniqueness of the end marker. It is most useful for operation like download, where the length of the output can be known in advance.

As mentioned above, the classes VariableLengthResponseGenerator and FixedLengthResponseGenerator are abstract. The module datalad_next.shell.response_generators provides the following concrete implementations for them:

• VariableLengthResponseGeneratorPosix

• VariableLengthResponseGeneratorPowerShell

• FixedLengthResponseGeneratorPosix

• FixedLengthResponseGeneratorPowerShell

When datalad_next.shell.shell() is executed it will use a VariableLengthResponseClass to skip the login message of the shell. This is done by executing a zero command (a command that will possibly generate some output, and successfully return) in the shell. The zero command is provided by the concrete implementation of class VariableLengthResponseGenerator. For example, the zero command for POSIX shells is test 0 -eq 0, for PowerShell it is Write-Host hello.

Because there is no way for func:shell to determine the kind of shell it connects to, the user can provide an alternative response generator class, in the zero_command_rg_class-parameter. Instance of that class will then be used to execute the zero command. Currently, the following two response generator classes are available:

• VariableLengthResponseGeneratorPosix: works with POSIX-compliant shells, e.g. sh or bash. This is the default.

• VariableLengthResponseGeneratorPowerShell: works with PowerShell.

Whenever a command is executed via ShellCommandExecutor.__call__(), the class identified by zero_command_rg_class will be used by default to create the final command list and to parse the result. Users can override this on a per-call basis by providing a different response generator class in the response_generator-parameter of ShellCommandExecutor.__call__().

Examples

See the documentation of datalad_next.shell.shell() for examples of how to use the shell-function and different response generator classes.

API overview

ShellCommandExecutor(process_inputs, stdout, ...)	Execute a command in a shell and return a generator that yields output
ShellCommandResponseGenerator(stdout_gen, ...)	An abstract class the specifies the minimal functionality of a response generator
VariableLengthResponseGenerator(stdout)	Response generator that handles outputs of unknown length
VariableLengthResponseGeneratorPosix(stdout)	A variable length response generator for POSIX shells
VariableLengthResponseGeneratorPowerShell(stdout)	A variable length response generator for PowerShell shells
FixedLengthResponseGenerator(stdout, length)	Response generator for efficient handling of outputs of known length
FixedLengthResponseGeneratorPosix(stdout, length)
FixedLengthResponseGeneratorPowerShell(...)
DownloadResponseGenerator(stdout)	Response generator interface for efficient download
DownloadResponseGeneratorPosix(stdout)	A response generator for efficient download commands from Linux systems
operations.posix.upload(shell, local_path, ...)	Upload a local file to a named file in the connected shell
operations.posix.download(shell, ...[, ...])	Download a file from the connected shell
operations.posix.delete(shell, files, *[, ...])	Delete files on the connected shell