Continuous integration and testing
Specification scope and status
This specification describes the current implementation.
DataLad is tested using a pytest-based testsuite that is run locally and via continuous integrations setups. Code development should ensure that old and new functionality is appropriately tested. The project aims for good unittest coverage (at least 80%).
Starting at the top level with
datalad/tests, every module in the package comes with a subdirectory
tests/, containing the tests for that portion of the codebase. This structure is meant to simplify (re-)running the tests for a particular module.
The test suite is run using
pip install -e .[tests] python -m pytest -c tox.ini datalad # or, with coverage reports python -m pytest -c tox.ini --cov=datalad datalad
Individual tests can be run using a path to the test file, followed by two colons and the test name:
python -m pytest datalad/core/local/tests/test_save.py::test_save_message_file
The set of to-be-run tests can be further sub-selected with environment variable based configurations that enable tests based on their Test annotations, or pytest-specific parameters.
Invoking a test run using
DATALAD_TESTS_KNOWNFAILURES_PROBE=True pytest datalad, for example, will run tests marked as known failures whether or not they still fail.
See section Configuration for all available configurations.
Invoking a test run using
DATALAD_TESTS_SSH=1 pytest -m xfail -c tox.ini datalad will run only those tests marked as xfail.
At the moment, Travis-CI, Appveyor, and GitHub Workflows exercise the tests battery for every PR and on the default branch, covering different operating systems, Python versions, and file systems. Tests should be ran on the oldest, latest, and current stable Python release. The projects uses https://codecov.io for an overview of code coverage.
Additional functionality is tested by extending existing similar tests with new test cases, or adding new tests to the respective test script of the module. Generally, every file example.py `with datalad code comes with a corresponding `tests/test_example.py.
Test helper functions assisting various general and DataLad specific assertions as well the construction of test directories and files can be found in
datalad/tests/utils_pytest.py also defines test decorators.
Some of those are used to annotate tests for various aspects to allow for easy sub-selection via environment variables.
Speed: Please annotate tests that take a while to complete with following decorators
@slowif test runs over 10 seconds
@turtleif test runs over 120 seconds (those would not typically be ran on CIs)
Purpose: Please further annotate tests with a special purpose specifically. As those tests also usually tend to be slower, use in conjunction with
@turtle when slow.
@integration- tests verifying correct operation with external tools/services beyond git/git-annex
@usecase- represents some (user) use-case, and not necessarily a “unit-test” of functionality
Dysfunction: If tests are not meant to be run on certain platforms or under certain conditions,
@skip annotations can be used. Examples include:
Migrating tests from nose to pytest
For the time being,
datalad.tests.utils keeps providing
nose-based utils, and
datalad.__init__ keeps providing nose-based fixtures to not break extensions that still use nose for testing.
A migration to
pytest is recommended, though.
To perform a typical migration of a DataLad extension to use pytest instead of nose, go through the following list:
keep all the
ok_helpers, but import them from
@with_*and other decorators populating positional arguments, convert corresponding posarg to kwarg by adding
convert all generator-based parametric tests into direct invocations or, preferably,
DeprecationWarningsin the code. Only where desired to test deprecation, add
@pytest.mark.filterwarnings("ignore: BEGINNING OF WARNING")decorator to the test.
For an example, see a “migrate to pytest” PR against
datalad-deprecated: datalad/datalad-deprecated#51 .