datalad.api.meta_extract

datalad.api.meta_extract(dataset=None, path=None, sources=None, process_type=None, format='native')

Run one or more metadata extractors on a dataset or file.

This command does not modify a dataset, but may initiate required data transfers to perform metadata extraction that requires local file content availability. This command does not support recursion into subdataset.

The result(s) are structured like the metadata DataLad would extract during metadata aggregation (in fact, this command is employed during aggregation). There is one result per dataset/file.

Examples

Extract metadata with two extractors from a dataset in the current directory and also from all its files:

$ datalad extract-metadata -d . --source xmp --source metalad_core

Extract XMP metadata from a single PDF that is not part of any dataset:

$ datalad extract-metadata --source xmp Downloads/freshfromtheweb.pdf

Customization of extraction:

The following configuration settings can be used to customize extractor behavior

datalad.metadata.extract-from-<extractorname> = {all|dataset|content}
which type of information an enabled extractor will be operating on (see –process-type argument for details)
datalad.metadata.exclude-path = <path>
ignore all content underneath the given path for metadata extraction, must be relative to the root of the dataset and in POSIX convention, and can be given multiple times
Parameters:
  • dataset (Dataset or None, optional) – “Dataset to extract metadata from. If no further constraining path is given, metadata is extracted from all files of the dataset. [Default: None]
  • path (sequence of str or None, optional) – Path of a file to extract metadata from. [Default: None]
  • sources – Name of a metadata extractor to be executed. If none is given, a set of default configured extractors, plus any extractors enabled in a dataset’s configuration and invoked. Multiple extractors can be given as a list. [Default: None]
  • process_type ({None, 'all', 'dataset', 'content', 'extractors'}, optional) – type of information to process. If ‘all’, metadata will be extracted for the entire dataset and its content. If not specified, the dataset’s configuration will determine the selection, and will default to ‘all’. Note that not processing content can influence the dataset metadata composition (e.g. report of total size). There is an auxiliary category ‘extractors’ that will cause all enabled extractors to be loaded, and reports on their status and configuration. [Default: None]
  • format ({'native', 'jsonld'}, optional) – format to use for the ‘metadata’ result property. ‘native’ will report the output of extractors as separate metadata properties that are stored under the name of the associated extractor; ‘jsonld’ composes a JSON-LD graph document, while stripping any information that does not appear to be properly typed linked data (extractor reports no @context’ field). [Default: ‘native’]
  • on_failure ({'ignore', 'continue', 'stop'}, optional) – behavior to perform on failure: ‘ignore’ any failure is reported, but does not cause an exception; ‘continue’ if any failure occurs an exception will be raised at the end, but processing other actions will continue for as long as possible; ‘stop’: processing will stop on first failure and an exception is raised. A failure is any result with status ‘impossible’ or ‘error’. Raised exception is an IncompleteResultsError that carries the result dictionaries of the failures in its failed attribute. [Default: ‘continue’]
  • proc_post – Like proc_pre, but procedures are executed after the main command has finished. [Default: None]
  • proc_pre – DataLad procedure to run prior to the main command. The argument a list of lists with procedure names and optional arguments. Procedures are called in the order their are given in this list. It is important to provide the respective target dataset to run a procedure on as the dataset argument of the main command. [Default: None]
  • result_filter (callable or None, optional) – if given, each to-be-returned status dictionary is passed to this callable, and is only returned if the callable’s return value does not evaluate to False or a ValueError exception is raised. If the given callable supports **kwargs it will additionally be passed the keyword arguments of the original API call. [Default: None]
  • result_renderer ({'default', 'json', 'json_pp', 'tailored'} or None, optional) – format of return value rendering on stdout. [Default: None]
  • result_xfm ({'datasets', 'successdatasets-or-none', 'paths', 'relpaths', 'metadata'} or callable or None, optional) – if given, each to-be-returned result status dictionary is passed to this callable, and its return value becomes the result instead. This is different from result_filter, as it can perform arbitrary transformation of the result value. This is mostly useful for top- level command invocations that need to provide the results in a particular format. Instead of a callable, a label for a pre-crafted result transformation can be given. [Default: None]
  • return_type ({'generator', 'list', 'item-or-list'}, optional) – return value behavior switch. If ‘item-or-list’ a single value is returned instead of a one-item return value list, or a list in case of multiple return values. None is return in case of an empty list. [Default: ‘list’]