datalad add-archive-content
Synopsis
datalad add-archive-content [-h] [-d DATASET] [--annex ANNEX] [--add-archive-leading-dir]
[--strip-leading-dirs] [--leading-dirs-depth LEADING_DIRS_DEPTH]
[--leading-dirs-consider LEADING_DIRS_CONSIDER]
[--use-current-dir] [-D] [--key] [-e EXCLUDE] [-r RENAME]
[--existing {fail,overwrite,archive-suffix,numeric-suffix}]
[--overwrite-prior-check {error|stats|ignore}] [-o
ANNEX_OPTIONS] [--copy] [--no-commit] [--allow-dirty] [--stats
STATS] [--drop-after] [--delete-after] [--version] ARCHIVE
[ARCHIVE ...]
Description
Add content of an archive under git annex control.
Given an already annex’ed archive, extract and add its files to the dataset, and reference the original archive as a custom special remote.
Multiple archives can be given in a single invocation. They are processed in the given order, and their content is added within a single commit, which is substantially faster than invoking the command once per archive.
All given archives are checked before any of them is acted on: if any one of them cannot be used (it is untracked, absent, not under annex control, or its content is not available locally), nothing is added at all. Should adding the content of an archive fail nevertheless, the remaining archives are not processed, and nothing is committed: the content added up to that point is left staged for inspection, and a subsequent invocation will refuse to operate on such a dirty dataset until those changes are dealt with. Original archives are removed (–delete) only once all of them were added successfully.
A file which one archive adds and a later one would overwrite is a likely mistake, since the content of the earlier one would be lost. Such an overwrite is only possible with –existing=overwrite, and leads to an error unless –overwrite-prior-check says otherwise.
- Changed in version 1.7
More than one archive can be given. A result record is now yielded for each given archive, in addition to the one for the dataset. –overwrite-prior-check was added, to guard the content of an archive against the archives which follow it.
Examples
Add files from the archive ‘big_tarball.tar.gz’, but keep big_tarball.tar.gz in the index:
% datalad add-archive-content big_tarball.tar.gz
Add files from the archive ‘big_tarball.tar.gz’, and remove big_tarball.tar.gz from the index:
% datalad add-archive-content big_tarball.tar.gz --delete
Add files from the archive ‘s3.zip’ but remove the leading directory:
% datalad add-archive-content s3.zip --strip-leading-dirs
Add files from multiple archives, resulting in a single commit:
% datalad add-archive-content 1.zip 2.zip
Options
ARCHIVE
archive file or a key (if –key specified). More than one can be given; the content of all of them is added within a single commit. Constraints: value must be a string
-h, --help, --help-np
show this help message. –help-np forcefully disables the use of a pager for displaying the help message
-d DATASET, --dataset DATASET
specify the dataset to operate on. Constraints: Value must be a Dataset or a valid identifier of a Dataset (e.g. a path) or value must be NONE
--annex ANNEX
DEPRECATED. Use the ‘dataset’ parameter instead.
--add-archive-leading-dir
place extracted content under a directory which would correspond to the archive name with all suffixes stripped. E.g. the content of archive.tar.gz will be extracted under archive/.
--strip-leading-dirs
remove one or more leading directories from the archive layout on extraction.
--leading-dirs-depth LEADING_DIRS_DEPTH
maximum depth of leading directories to strip. If not specified (None), no limit.
--leading-dirs-consider LEADING_DIRS_CONSIDER
regular expression(s) for directories to consider to strip away. Constraints: value must be a string or value must be NONE
--use-current-dir
extract the archive under the current directory, not the directory where the archive is located. This parameter is applied automatically if –key was used.
-D, --delete
delete original archives from the filesystem/Git in current tree, once the content of all of them was added. Note that it will be of no effect if –key is given.
--key
signal that the provided archives are not filenames on their own but annex keys. Such archives are extracted in the current directory.
-e EXCLUDE, --exclude EXCLUDE
regular expressions for filenames which to exclude from being added to annex. Applied after –rename if that one is specified. For exact matching, use anchoring. Constraints: value must be a string or value must be NONE
-r RENAME, --rename RENAME
regular expressions to rename files before added them under to Git. The first defines how to split provided string into two parts: Python regular expression (with groups), and replacement string. Constraints: value must be a string or value must be NONE
--existing {fail,overwrite,archive-suffix,numeric-suffix}
what operation to perform if a file from an archive tries to overwrite an existing file with the same name. ‘fail’ (default) leads to an error result, ‘overwrite’ silently replaces existing file, ‘archive-suffix’ instructs to add a suffix (prefixed with a ‘-’) matching archive name from which file gets extracted, and if that one is present as well, ‘numeric-suffix’ is in effect in addition, when incremental numeric suffix (prefixed with a ‘.’) is added until no name collision is longer detected. [Default: ‘fail’]
--overwrite-prior-check {error|stats|ignore}
how to react when a file extracted from an archive would overwrite a file which one of the preceding archives of the same invocation had just added. Such an overwrite discards the content contributed by that earlier archive, so by default (‘error’) it leads to an error result, and nothing is committed. ‘stats’ permits the overwrite and reports the number of files it affected as ‘overwritten prior’ in the statistics. ‘ignore’ permits it without keeping track of the paths added, at a marginal saving of memory. This option is orthogonal to –existing, which governs collisions with content which was already in the dataset: only –existing=overwrite can discard the contribution of a preceding archive, while the suffix modes rename the incoming file instead and are thus never reported here. Files with identical content are not overwrites, and are never reported either. Has no effect if –delete-after is given, where each archive is extracted into its own temporary directory and no two of them can collide. Constraints: value must be one of (‘error’, ‘stats’, ‘ignore’) [Default: ‘error’]
-o ANNEX_OPTIONS, --annex-options ANNEX_OPTIONS
additional options to pass to git-annex. Constraints: value must be a string or value must be NONE
--copy
copy the content of the archive instead of moving.
--no-commit
don’t commit upon completion.
--allow-dirty
flag that operating on a dirty repository (uncommitted or untracked content) is ok.
--stats STATS
ActivityStats instance for global tracking.
--drop-after
drop extracted files after adding to annex.
--delete-after
extract under a temporary directory, git-annex add, and delete afterwards. To be used to “index” files within annex without actually creating corresponding files under git. Note that annex dropunused would later remove that load.
--version
show the module and its version which provides the command