datalad ls-file-collection

Synopsis

datalad ls-file-collection [-h] [--hash ALGORITHM] [--version] {directory,tarfile,zipfile,gittree,gitworktree,annexworktree} ID/LOCATION

Description

Report information on files in a collection

This is a utility that can be used to query information on files in different file collections. The type of information reported varies across collection types. However, each result at minimum contains some kind of identifier for the collection ('collection' property), and an identifier for the respective collection item ('item' property). Each result also contains a type property that indicates particular type of file that is being reported on. In most cases this will be file, but other categories like symlink or directory are recognized too.

If a collection type provides file-access, this command can compute one or more hashes (checksums) for any file in a collection.

Supported file collection types are:

directory: Reports on the content of a given directory (non-recursively). The collection identifier is the path of the directory. Item identifiers are the names of items within that directory. Standard properties like size, mtime, or link_target are included in the report.
gittree: Reports on the content of a Git "tree-ish". The collection identifier is that tree-ish. The command must be executed inside a Git repository. If the working directory for the command is not the repository root (in case of a non-bare repository), the report is constrained to items underneath the working directory. Item identifiers are the relative paths of items within that working directory. Reported properties include gitsha and gittype; note that the gitsha is not equivalent to a SHA1 hash of a file's content, but is the SHA-type blob identifier as reported and used by Git. Reporting of content hashes beyond the gitsha is presently not supported.
gitworktree: Reports on all tracked and untracked content of a Git repository's work tree. The collection identifier is a path of a directory in a Git repository (which can, but needs not be, its root). Item identifiers are the relative paths of items within that directory. Reported properties include gitsha and gittype; note that the gitsha is not equivalent to a SHA1 hash of a file's content, but is the SHA-type blob identifier as reported and used by Git.
tarfile: Reports on members of a TAR archive. The collection identifier is the path of the TAR file. Item identifiers are the relative paths of archive members within the archive. Reported properties are similar to the directory collection type.

Examples

Report on the content of a directory:

% datalad -f json ls-file-collection directory /tmp

Report on the content of a TAR archive with MD5 and SHA1 file hashes:

% datalad -f json ls-file-collection --hash md5 --hash sha1 tarfile myarchive.tar.gz

Register URLs for files in a directory that is also reachable via HTTP. This uses ls-file-collection for listing files and computing MD5 hashes, then using jq to filter and transform the output (just file records, and in a JSON array), and passes them to addurls, which generates annex keys/files and assigns URLs. When the command finishes, the dataset contains no data, but can retrieve the files after confirming their availability (i.e., via git annex fsck):

% datalad -f json ls-file-collection directory wwwdir --hash md5 \
  | jq '. | select(.type == "file")' \
  | jq --slurp . \
  | datalad addurls --key 'et:MD5-s{size}--{hash-md5}' - 'https://example.com/{item}'

Options

{directory,tarfile,zipfile,gittree,gitworktree,annexworktree}

Name of the type of file collection to report on.

ID/LOCATION

identifier or location of the file collection to report on. Depending on the type of collection to process, the specific nature of this parameter can be different. A common identifier for a file collection is a path (to a directory, to an archive), but might also be a URL. See the documentation for details on supported collection types.

-h, --help, --help-np

show this help message. --help-np forcefully disables the use of a pager for displaying the help message

--hash ALGORITHM

One or more names of algorithms to be used for reporting file hashes. They must be supported by the Python 'hashlib' module, e.g. 'md5' or 'sha256'. Reporting file hashes typically implies retrieving/reading file content. This processing may also enable reporting of additional properties that may otherwise not be readily available. This option can be given more than once.

--version

show the module and its version which provides the command

Authors

datalad is developed by The DataLad Team and Contributors <team@datalad.org>.