datalad.api.catalog_workflow

datalad.api.catalog_workflow(catalog: Path | WebCatalog, mode: str, dataset: Dataset, subdataset: Dataset = None, recursive=False, recursion_limit=None, extractor=['metalad_core'], config_file=None, force: bool = False)

Run a workflow to create or update a catalog

This functionality requires the installation of datalad-metalad as well as any datalad extensions providing relevant translators for the extracted metadata items.

It will run a workflow of metadata extraction, translation, and catalog (entry) generation, given a DataLad dataset hierarchy and a specified workflow type: new/update.

Examples

Run a workflow for recursive metadata extraction (using the 'metalad_core' extractor), translating metadata to the latestcatalog schema, and adding the translated metadata to a new catalog:

> catalog_workflow(mode='new', catalog='/tmp/my-cat/', dataset='path/to/superdataset', extractor='metalad_core')

Run a workflow for updating a catalog after registering a subdataset to the superdataset which the catalog represents.:

> catalog_workflow(mode='update', catalog='/tmp/my-cat/', dataset='path/to/superdataset', subdataset='path/to/subdataset', extractor='metalad_core')
Parameters:

  • catalog -- Location of the existing catalog.

  • mode -- Which type of workflow to run: one of ['new', 'update'].

  • dataset -- The datalad dataset on which to run the workflow.

  • subdataset -- The datalad subdataset on which to run the update workflow. [Default: None]

  • recursive (bool, optional) -- Specifies whether to recurse into subdatasets or not during workflow execution. [Default: False]

  • recursion_limit -- Specifies how many levels to recurse down into the hierarchy when recursing into subdatasets. [Default: None]

  • extractor -- Which extractors to use during metadata extraction of a workflow. Multiple can be provided. If none are provided, the default extractor 'metalad_core' is used. Any extractor name passed as an argument should first be known to the current installation via datalad's entrypoint mechanism. [Default: ['metalad_core']]

  • config_file -- Path to config file in YAML or JSON format. Default config is read from datalad_catalog/config/config.json. [Default: None]

  • force (bool, optional) -- If content for the user interface already exists in the catalog directory, force this content to be overwritten. Content overwritten with this flag include the 'artwork' and 'assets' directories and the 'index.html' and 'config.json' files. Content in the 'metadata' directory remain untouched. [Default: False]

  • on_failure ({'ignore', 'continue', 'stop'}, optional) -- behavior to perform on failure: 'ignore' any failure is reported, but does not cause an exception; 'continue' if any failure occurs an exception will be raised at the end, but processing other actions will continue for as long as possible; 'stop': processing will stop on first failure and an exception is raised. A failure is any result with status 'impossible' or 'error'. Raised exception is an IncompleteResultsError that carries the result dictionaries of the failures in its failed attribute. [Default: 'continue']

  • result_filter (callable or None, optional) -- if given, each to-be-returned status dictionary is passed to this callable, and is only returned if the callable's return value does not evaluate to False or a ValueError exception is raised. If the given callable supports **kwargs it will additionally be passed the keyword arguments of the original API call. [Default: None]

  • result_renderer -- select rendering mode command results. 'tailored' enables a command- specific rendering style that is typically tailored to human consumption, if there is one for a specific command, or otherwise falls back on the the 'generic' result renderer; 'generic' renders each result in one line with key info like action, status, path, and an optional message); 'json' a complete JSON line serialization of the full result record; 'json_pp' like 'json', but pretty-printed spanning multiple lines; 'disabled' turns off result rendering entirely; '<template>' reports any value(s) of any result properties in any format indicated by the template (e.g. '{path}', compare with JSON output for all key-value choices). The template syntax follows the Python "format() language". It is possible to report individual dictionary values, e.g. '{metadata[name]}'. If a 2nd-level key contains a colon, e.g. 'music:Genre', ':' must be substituted by '#' in the template, like so: '{metadata[music#Genre]}'. [Default: 'tailored']

  • result_xfm ({'datasets', 'successdatasets-or-none', 'paths', 'relpaths', 'metadata'} or callable or None, optional) -- if given, each to-be-returned result status dictionary is passed to this callable, and its return value becomes the result instead. This is different from result_filter, as it can perform arbitrary transformation of the result value. This is mostly useful for top- level command invocations that need to provide the results in a particular format. Instead of a callable, a label for a pre-crafted result transformation can be given. [Default: None]

  • return_type ({'generator', 'list', 'item-or-list'}, optional) -- return value behavior switch. If 'item-or-list' a single value is returned instead of a one-item return value list, or a list in case of multiple return values. None is return in case of an empty list. [Default: 'list']