DataLad — data management and publication multitool

Welcome to DataLad’s technical documentation. Information here is targeting software developers and is focused on the Python API and CLI, as well as software design, employed technologies, and key features. Comprehensive user documentation with information on installation, basic operation, support, and (advanced) use case descriptions is available in the DataLad handbook.

Content

• Change log
• Acknowledgments
• Publications

Concepts and technologies

• Background and motivation
  • Vision
  • Technological foundation: git-annex
  • Objective
• Delineation from related solutions
  • Data catalogs
  • Data delivery/management middleware
  • Git/Git-annex/DataLad
• Basic principles
  • Datasets
  • Basic command line usage
  • API principles
• Credentials
  • Integration with Git
  • Let Git query DataLad
  • Let DataLad query Git
• Customization and extension of functionality
  • Using an extension
  • Writing your own extensions
• Design
  • Command line interface
  • Provenance capture
  • Application-type vs. library-type usage
  • File URL handling
  • Result records
  • dataset argument
  • Log levels
  • Drop dataset components
  • Python import statements
  • Miscellaneous patterns
  • Exception handling
  • Credential management
  • URL substitution
  • Threaded runner
  • BatchedCommand and BatchedAnnex
  • Standard parameters
  • Positional vs Keyword parameters
  • Docstrings
  • Progress reporting
  • GitHub Action
  • Continuous integration and testing
  • User messaging: result records vs exceptions vs logging
• Glossary

Commands and API

• Command line reference
  • Main command
  • Core commands
  • Extended set of functionality
• Python module reference
  • High-level user interface
  • Support functionality
  • Configuration management
  • Test infrastructure
  • Command interface
  • Command line interface infrastructure
• Configuration
  • Global user configuration
  • Local repository configuration
  • Sticky dataset configuration
  • Miscellaneous configuration

Extension packages

DataLad can be customized and additional functionality can be integrated via extensions. Each extension provides its own documentation:

• Crawling web resources and automated data distributions

• Neuroimaging data and workflows

• Containerized computational environments

• Advanced metadata tooling with JSON-LD reporting and additional metadata extractors

• Staged additions, performance and user experience improvements for DataLad

• Resources for working with the UKBiobank as a DataLad dataset

• Deposit and retrieve DataLad datasets via the Open Science Framework

• Functionality that has been phased out of the core package

• Special interest functionality or drafts of future additions to DataLad proper

Indices and tables

• Index

• Module Index

• Search Page