datalad.customremotes.base

Base classes to custom git-annex remotes (e.g. extraction from archives)

class datalad.customremotes.base.AnnexCustomRemote(path=None, cost=None, fin=None, fout=None)

Bases: object

Base class to provide custom special remotes for git-annex

Implements git-annex special custom remotes protocol described at http://git-annex.branchable.com/design/external_special_remote_protocol/

AVAILABILITY = 'LOCAL'

COST = 100

CUSTOM_REMOTE_NAME = None

SUPPORTED_SCHEMES = ()

debug(msg)

error(msg, annex_err='ERROR')

get_DIRHASH(key, full=False)

Gets a two level hash associated with a Key.

Parameters:

• full (bool, optional) – If True, would spit out full DIRHASH path, i.e. with a KEY/ directory
• like "abc/def". This is always the same for any given Key, so (Something) –
• be used for eg, creating hash directory structures to store Keys in. (can) –

get_URLS(key)

Gets URL(s) associated with a Key.

get_contentlocation(key, absolute=False, verify_exists=True)

Return (relative to top or absolute) path to the file containing the key

This is a wrapper around AnnexRepo.get_contentlocation which provides caching of the result (we are asking the location for the same archive key often)

heavydebug(msg, *args, **kwargs)

info(msg)

main()

Interface to the command line tool

progress(bytes)

read(req=None, n=1)

Read a message from git-annex

Parameters:

• req (string, optional) – Expected request - first msg of the response
• n (int) – Number of response elements after first msg

req_CHECKPRESENT(key)

CHECKPRESENT-SUCCESS Key

Indicates that a key has been positively verified to be present in the remote.

CHECKPRESENT-FAILURE Key

Indicates that a key has been positively verified to not be present in the remote.

CHECKPRESENT-UNKNOWN Key ErrorMsg

Indicates that it is not currently possible to verify if the key is present in the remote. (Perhaps the remote cannot be contacted.)

req_CHECKURL(url)

The remote replies with one of CHECKURL-FAILURE, CHECKURL-CONTENTS, or CHECKURL-MULTI.

CHECKURL-CONTENTS Size|UNKNOWN Filename

Indicates that the requested url has been verified to exist. The Size is the size in bytes, or use “UNKNOWN” if the size could not be determined. The Filename can be empty (in which case a default is used), or can specify a filename that is suggested to be used for this url.

CHECKURL-MULTI Url Size|UNKNOWN Filename …

Indicates that the requested url has been verified to exist, and contains multiple files, which can each be accessed using their own url. Note that since a list is returned, neither the Url nor the Filename can contain spaces.

CHECKURL-FAILURE

Indicates that the requested url could not be accessed.

req_CLAIMURL(url)

req_EXPORTSUPPORTED()

req_GETAVAILABILITY()

req_GETCOST()

req_INITREMOTE(*args)

Initialize this remote. Provides high level abstraction.

Specific implementation should go to _initialize

req_PREPARE(*args)

Prepare “to deliver”. Provides high level abstraction

Specific implementation should go to _prepare

req_REMOVE(key)

REMOVE-SUCCESS Key

Indicates the key has been removed from the remote. May be returned if the remote didn’t have the key at the point removal was requested.

REMOVE-FAILURE Key ErrorMsg

Indicates that the key was unable to be removed from the remote.

req_TRANSFER(cmd, key, file)

req_WHEREIS(key)

Added in 5.20150812-17-g6bc46e3

provide any information about ways to access the content of a key stored in it, such as eg, public urls. This will be displayed to the user by eg, git annex whereis. The remote replies with WHEREIS-SUCCESS or WHEREIS-FAILURE. Note that users expect git annex whereis to run fast, without eg, network access. This is not needed when SETURIPRESENT is used, since such uris are automatically displayed by git annex whereis.

WHEREIS-SUCCESS String

Indicates a location of a key. Typically an url, the string can be anything that it makes sense to display to the user about content stored in the special remote.

WHEREIS-FAILURE

Indicates that no location is known for a key.

send(*args)

Send a message to git-annex

Parameters:*args (list of strings) – arguments to be joined by a space and passed to git-annex

send_unsupported(msg=None)

Send UNSUPPORTED-REQUEST to annex and log optional message in our log

stop(msg=None)

class datalad.customremotes.base.AnnexExchangeProtocol(repopath, custom_remote_name=None)

Bases: datalad.support.protocol.ProtocolInterface

A little helper to protocol interactions of custom remote with annex

HEADER = '#!/bin/bash\n\nset -e\n\n# Gets a VALUE response and stores it in $RET\nreport () {\n echo "$@" >&2\n}\n\nrecv () {\n read resp\n #resp=${resp%\\n}\n target="$@"\n if [ "$resp" != "$target" ]; then\n report "! exp $target"\n report " got $resp"\n else\n report "+ got $resp"\n fi\n}\n\nsend () {\n echo "$@"\n report "sent $@"\n}\n\n'

add_section(cmd, exception)

Adds a section to the protocol.

This is an alternative to the use of start_section() and end_section(). In opposition to start_section, this one can be called anytime.

Parameters:

• cmd (list) – The actual command and its options/arguments as a list
• exception (Exception) – The exception raised by the command if any or None otherwise.

do_execute_callables

do_execute_ext_commands

end_section(id_, exception)

Ends the section id.

To call after the command call to be recorded. This ends the section defined by id as returned by start_section().

Parameters:

• id (int) –
• exception (Exception) – The exception raised by the command if any or None otherwise.
• Raises –
• ------- –
• IndexError – in case id is invalid.

initiate()

records_callables

records_ext_commands

start_section(cmd)

Starts a new section of the protocol.

To call before the command call to be recorded. To be used with a corresponding call of end_section().

Parameters:cmd (list) – The actual command and its options/arguments as a list Returns:An id of the started section to be used as argument of the corresponding call of end_section(). Return type:int

write_entries(entries)

write_section(cmd)

exception datalad.customremotes.base.AnnexRemoteQuit

Bases: exceptions.Exception

datalad.customremotes.base.generate_uuids()

Generate UUIDs for our remotes. Even though quick, for consistency pre-generated and recorded in consts.py

datalad.customremotes.base.get_function_nargs(f)

datalad.customremotes.base.init_datalad_remote(repo, remote, encryption=None, autoenable=False, opts=[])

Initialize datalad special remote