datalad.api.add_archive_content

datalad.api.add_archive_content(archive, *, dataset=None, annex=None, add_archive_leading_dir=False, strip_leading_dirs=False, leading_dirs_depth=None, leading_dirs_consider=None, use_current_dir=False, delete=False, key=False, exclude=None, rename=None, existing='fail', overwrite_prior_check='error', annex_options=None, copy=False, commit=True, allow_dirty=False, stats=None, drop_after=False, delete_after=False)

Add content of an archive under git annex control.

Given an already annex’ed archive, extract and add its files to the dataset, and reference the original archive as a custom special remote.

Multiple archives can be given in a single invocation. They are processed in the given order, and their content is added within a single commit, which is substantially faster than invoking the command once per archive.

All given archives are checked before any of them is acted on: if any one of them cannot be used (it is untracked, absent, not under annex control, or its content is not available locally), nothing is added at all. Should adding the content of an archive fail nevertheless, the remaining archives are not processed, and nothing is committed: the content added up to that point is left staged for inspection, and a subsequent invocation will refuse to operate on such a dirty dataset until those changes are dealt with. Original archives are removed (delete=True) only once all of them were added successfully.

A file which one archive adds and a later one would overwrite is a likely mistake, since the content of the earlier one would be lost. Such an overwrite is only possible with existing=’overwrite’, and leads to an error unless overwrite_prior_check says otherwise.

Changed in version 1.7: More than one archive can be given. A result record is now yielded for each given archive, in addition to the one for the dataset. overwrite_prior_check was added, to guard the content of an archive against the archives which follow it.

Examples

Add files from the archive ‘big_tarball.tar.gz’, but keep big_tarball.tar.gz in the index:

> add_archive_content(archive='big_tarball.tar.gz')

Add files from the archive ‘big_tarball.tar.gz’, and remove big_tarball.tar.gz from the index:

> add_archive_content(archive='big_tarball.tar.gz', delete=True)

Add files from the archive ‘s3.zip’ but remove the leading directory:

> add_archive_content(archive='s3.zip', strip_leading_dirs=True)

Add files from multiple archives, resulting in a single commit:

> add_archive_content(archive=['1.zip', '2.zip'])
Parameters:

  • archive (non-empty sequence of str) – archive file or a key (if key=True specified). More than one can be given; the content of all of them is added within a single commit.

  • dataset (Dataset or None, optional) – specify the dataset to operate on. [Default: None]

  • annex – DEPRECATED. Use the ‘dataset’ parameter instead. [Default: None]

  • add_archive_leading_dir (bool, optional) – place extracted content under a directory which would correspond to the archive name with all suffixes stripped. E.g. the content of archive.tar.gz will be extracted under archive/. [Default: False]

  • strip_leading_dirs (bool, optional) – remove one or more leading directories from the archive layout on extraction. [Default: False]

  • leading_dirs_depth – maximum depth of leading directories to strip. If not specified (None), no limit. [Default: None]

  • leading_dirs_consider (list of str or None, optional) – regular expression(s) for directories to consider to strip away. [Default: None]

  • use_current_dir (bool, optional) – extract the archive under the current directory, not the directory where the archive is located. This parameter is applied automatically if key=True was used. [Default: False]

  • delete (bool, optional) – delete original archives from the filesystem/Git in current tree, once the content of all of them was added. Note that it will be of no effect if key=True is given. [Default: False]

  • key (bool, optional) – signal that the provided archives are not filenames on their own but annex keys. Such archives are extracted in the current directory. [Default: False]

  • exclude (list of str or None, optional) – regular expressions for filenames which to exclude from being added to annex. Applied after –rename if that one is specified. For exact matching, use anchoring. [Default: None]

  • rename (list of str or None, optional) – regular expressions to rename files before added them under to Git. The first defines how to split provided string into two parts: Python regular expression (with groups), and replacement string. [Default: None]

  • existing – what operation to perform if a file from an archive tries to overwrite an existing file with the same name. ‘fail’ (default) leads to an error result, ‘overwrite’ silently replaces existing file, ‘archive-suffix’ instructs to add a suffix (prefixed with a ‘-’) matching archive name from which file gets extracted, and if that one is present as well, ‘numeric-suffix’ is in effect in addition, when incremental numeric suffix (prefixed with a ‘.’) is added until no name collision is longer detected. [Default: ‘fail’]

  • overwrite_prior_check ({'error', 'stats', 'ignore'}, optional) – how to react when a file extracted from an archive would overwrite a file which one of the preceding archives of the same invocation had just added. Such an overwrite discards the content contributed by that earlier archive, so by default (‘error’) it leads to an error result, and nothing is committed. ‘stats’ permits the overwrite and reports the number of files it affected as ‘overwritten prior’ in the statistics. ‘ignore’ permits it without keeping track of the paths added, at a marginal saving of memory. This option is orthogonal to existing, which governs collisions with content which was already in the dataset: only existing=’overwrite’ can discard the contribution of a preceding archive, while the suffix modes rename the incoming file instead and are thus never reported here. Files with identical content are not overwrites, and are never reported either. Has no effect if delete_after=True is given, where each archive is extracted into its own temporary directory and no two of them can collide. [Default: ‘error’]

  • annex_options (str or None, optional) – additional options to pass to git-annex. [Default: None]

  • copy (bool, optional) – copy the content of the archive instead of moving. [Default: False]

  • commit (bool, optional) – don’t commit upon completion. [Default: True]

  • allow_dirty (bool, optional) – flag that operating on a dirty repository (uncommitted or untracked content) is ok. [Default: False]

  • stats – ActivityStats instance for global tracking. [Default: None]

  • drop_after (bool, optional) – drop extracted files after adding to annex. [Default: False]

  • delete_after (bool, optional) – extract under a temporary directory, git-annex add, and delete afterwards. To be used to “index” files within annex without actually creating corresponding files under git. Note that annex dropunused would later remove that load. [Default: False]

  • on_failure ({'ignore', 'continue', 'stop'}, optional) – behavior to perform on failure: ‘ignore’ any failure is reported, but does not cause an exception; ‘continue’ if any failure occurs an exception will be raised at the end, but processing other actions will continue for as long as possible; ‘stop’: processing will stop on first failure and an exception is raised. A failure is any result with status ‘impossible’ or ‘error’. Raised exception is an IncompleteResultsError that carries the result dictionaries of the failures in its failed attribute. [Default: ‘continue’]

  • result_filter (callable or None, optional) – if given, each to-be-returned status dictionary is passed to this callable, and is only returned if the callable’s return value does not evaluate to False or a ValueError exception is raised. If the given callable supports **kwargs it will additionally be passed the keyword arguments of the original API call. [Default: None]

  • result_renderer – select rendering mode command results. ‘tailored’ enables a command- specific rendering style that is typically tailored to human consumption, if there is one for a specific command, or otherwise falls back on the the ‘generic’ result renderer; ‘generic’ renders each result in one line with key info like action, status, path, and an optional message); ‘json’ a complete JSON line serialization of the full result record; ‘json_pp’ like ‘json’, but pretty-printed spanning multiple lines; ‘disabled’ turns off result rendering entirely; ‘<template>’ reports any value(s) of any result properties in any format indicated by the template (e.g. ‘{path}’, compare with JSON output for all key-value choices). The template syntax follows the Python “format() language”. It is possible to report individual dictionary values, e.g. ‘{metadata[name]}’. If a 2nd-level key contains a colon, e.g. ‘music:Genre’, ‘:’ must be substituted by ‘#’ in the template, like so: ‘{metadata[music#Genre]}’. [Default: ‘tailored’]

  • result_xfm ({'datasets', 'successdatasets-or-none', 'paths', 'relpaths', 'metadata'} or callable or None, optional) – if given, each to-be-returned result status dictionary is passed to this callable, and its return value becomes the result instead. This is different from result_filter, as it can perform arbitrary transformation of the result value. This is mostly useful for top- level command invocations that need to provide the results in a particular format. Instead of a callable, a label for a pre-crafted result transformation can be given. [Default: None]

  • return_type ({'generator', 'list', 'item-or-list'}, optional) – return value behavior switch. If ‘item-or-list’ a single value is returned instead of a one-item return value list, or a list in case of multiple return values. None is return in case of an empty list. [Default: ‘list’]