datalad_next.annexbackends.base

Interface and essential utilities to implement external git-annex backends

exception datalad_next.annexbackends.base.AnnexError

Bases: Exception

Common base class for all annexbackend exceptions.

class datalad_next.annexbackends.base.Backend(annex)

Bases: object

Metaclass for backends.

It implements the communication with git-annex via the external backend protocol. More information on the protocol is available at https://git-annex.branchable.com/design/external_backend_protocol/

External backends can be built by implementing the abstract methods defined in this class.

annex

The Master object to which this backend is linked. Master acts as an abstraction layer for git-annex.

Type:

Master

abstract can_verify()

Returns whether the backend can verify the content of files match a key it generated. The verification does not need to be cryptographically secure, but should catch data corruption.

Return type:

bool

error(error_msg)

Communicate a generic error.

Can be sent at any time if things get too messed up to continue. If the program receives an error() from git-annex, it can exit with its own error(). Eg.: self.annex.error("Error received. Exiting.") raise SystemExit

Parameters:

error_msg (str) -- The error message received from git-annex

abstract gen_key(local_file)

Examine the content of local_file and from it generate a key.

While it is doing this, it can send any number of PROGRESS messages indication the position in the file that it's gotten to.

Parameters:

local_file (str) -- Path for which to generate a key. Note that in some cases, local_file may contain whitespace.

Returns:

The generated key.

Return type:

str

Raises:

BackendError -- If the file could not be received from the backend.

abstract is_cryptographically_secure()

Returns whether keys it generates are verified using a cryptographically secure hash.

Note that sha1 is not a cryptographically secure hash any longer. A program can change its answer to this question as the state of the art advances, and should aim to stay ahead of the state of the art by a reasonable amount of time.

Return type:

bool

abstract is_stable()

Returns whether a key it has generated will always have the same content. The answer to this is almost always yes; URL keys are an example of a type of key that may have different content at different times.

Return type:

bool

abstract verify_content(key, content_file)

Examine a file and verify it has the content expected given a key

While it is doing this, it can send any number of PROGRESS messages indicating the position in the file that it's gotten to.

If can_verify() == False, git-annex not ask to do this.

Return type:

bool

exception datalad_next.annexbackends.base.BackendError

Bases: AnnexError

Must be raised by the backend when a request did not succeed.

class datalad_next.annexbackends.base.Master(output=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Bases: object

Metaclass for backends.

input

Where to listen for git-annex request messages. Default: sys.stdin

Type:

io.TextIOBase

output

Where to send replies and backend messages Default: sys.stdout

Type:

io.TextIOBase

backend

A class implementing the Backend interface to which this master is linked.

Type:

Backend

LinkBackend(backend)

Link the Master to a backend. This must be done before calling Listen()

Parameters:

backend (Backend) -- A class implementing Backend interface to which this master will be linked.

Listen(input=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>)

Listen on input for messages from git annex.

Parameters:

input (io.TextIOBase) -- Where to listen for git-annex request messages. Default: sys.stdin

Raises:

NotLinkedError -- If there is no backend linked to this master.

debug(*args)

Tells git-annex to display the message if --debug is enabled.

Parameters:

message (str) -- The message to be displayed to the user

error(*args)

Generic error. Can be sent at any time if things get too messed up to continue. When possible, raise a BackendError inside the respective functions. The backend program should exit after sending this, as git-annex will not talk to it any further.

Parameters:

error_msg (str) -- The error message to be sent to git-annex

progress(progress)

Indicates the current progress of the transfer (in bytes). May be repeated any number of times during the transfer process, but it's wasteful to update the progress until at least another 1% of the file has been sent. This is highly recommended for *_store(). (It is optional but good for *_retrieve().)

Parameters:

progress (int) -- The current progress of the transfer in bytes.

exception datalad_next.annexbackends.base.NotLinkedError

Bases: AnnexError

Will be raised when a Master instance is accessed without being linked to a Backend instance

class datalad_next.annexbackends.base.Protocol(backend)

Bases: object

Helper class handling the receiving part of the protocol (git-annex to backend) It parses the requests coming from git-annex and calls the respective method of the backend object.

command(line)

do_CANVERIFY()

do_ERROR(message)

do_GENKEY(*arg)

do_GETVERSION()

do_ISCRYPTOGRAPHICALLYSECURE()

do_ISSTABLE()

do_VERIFYKEYCONTENT(*arg)

lookupMethod(command)

exception datalad_next.annexbackends.base.ProtocolError

Bases: AnnexError

Base class for protocol errors

exception datalad_next.annexbackends.base.UnexpectedMessage

Bases: ProtocolError

Raised when git-annex sends a message which is not expected at the moment

exception datalad_next.annexbackends.base.UnsupportedRequest

Bases: ProtocolError

Must be raised when an optional request is not supported by the backend.