datalad ls-file-collection
Synopsis
datalad ls-file-collection [-h] [--hash ALGORITHM] [--version] {directory,tarfile,zipfile,gittree,gitworktree,annexworktree} ID/LOCATION
Description
Report information on files in a collection
This is a utility that can be used to query information on files in
different file collections. The type of information reported varies across
collection types. However, each result at minimum contains some kind of
identifier for the collection ('collection' property), and an identifier
for the respective collection item ('item' property). Each result
also contains a type
property that indicates particular type of file
that is being reported on. In most cases this will be file
, but
other categories like symlink
or directory
are recognized too.
If a collection type provides file-access, this command can compute one or more hashes (checksums) for any file in a collection.
Supported file collection types are:
directory
Reports on the content of a given directory (non-recursively). The collection identifier is the path of the directory. Item identifiers are the names of items within that directory. Standard properties like
size
,mtime
, orlink_target
are included in the report.gittree
Reports on the content of a Git "tree-ish". The collection identifier is that tree-ish. The command must be executed inside a Git repository. If the working directory for the command is not the repository root (in case of a non-bare repository), the report is constrained to items underneath the working directory. Item identifiers are the relative paths of items within that working directory. Reported properties include
gitsha
andgittype
; note that thegitsha
is not equivalent to a SHA1 hash of a file's content, but is the SHA-type blob identifier as reported and used by Git. Reporting of content hashes beyond thegitsha
is presently not supported.gitworktree
Reports on all tracked and untracked content of a Git repository's work tree. The collection identifier is a path of a directory in a Git repository (which can, but needs not be, its root). Item identifiers are the relative paths of items within that directory. Reported properties include
gitsha
andgittype
; note that thegitsha
is not equivalent to a SHA1 hash of a file's content, but is the SHA-type blob identifier as reported and used by Git.tarfile
Reports on members of a TAR archive. The collection identifier is the path of the TAR file. Item identifiers are the relative paths of archive members within the archive. Reported properties are similar to the
directory
collection type.
Examples
Report on the content of a directory:
% datalad -f json ls-file-collection directory /tmp
Report on the content of a TAR archive with MD5 and SHA1 file hashes:
% datalad -f json ls-file-collection --hash md5 --hash sha1 tarfile myarchive.tar.gz
Register URLs for files in a directory that is also reachable via
HTTP. This uses ls-file-collection
for listing files and computing
MD5 hashes, then using jq
to filter and transform the output (just
file records, and in a JSON array), and passes them to addurls
,
which generates annex keys/files and assigns URLs. When the command
finishes, the dataset contains no data, but can retrieve the files
after confirming their availability (i.e., via git annex fsck):
% datalad -f json ls-file-collection directory wwwdir --hash md5 \
| jq '. | select(.type == "file")' \
| jq --slurp . \
| datalad addurls --key 'et:MD5-s{size}--{hash-md5}' - 'https://example.com/{item}'
Options
{directory,tarfile,zipfile,gittree,gitworktree,annexworktree}
Name of the type of file collection to report on.
ID/LOCATION
identifier or location of the file collection to report on. Depending on the type of collection to process, the specific nature of this parameter can be different. A common identifier for a file collection is a path (to a directory, to an archive), but might also be a URL. See the documentation for details on supported collection types.
-h, --help, --help-np
show this help message. --help-np forcefully disables the use of a pager for displaying the help message
--hash ALGORITHM
One or more names of algorithms to be used for reporting file hashes. They must be supported by the Python 'hashlib' module, e.g. 'md5' or 'sha256'. Reporting file hashes typically implies retrieving/reading file content. This processing may also enable reporting of additional properties that may otherwise not be readily available. This option can be given more than once.
--version
show the module and its version which provides the command