datalad_next.patches.customremotes_main

Connect log_progress-style progress reporting to git-annex

This patch introduces a dedicated progress log handler as a proxy between standard datalad progress logging and a git-annex special remote as an approach to report (data transfer) progress to a git-annex parent process.

This functionality is only (to be) used in dedicated special remote processes.

class datalad_next.patches.customremotes_main.AnnexProgressLogHandler(annexremote: SpecialRemote)[source]

Bases: Handler

Log handler to funnel progress logs to git-annex

For this purpose the handler wraps datalad_next.annexremotes.SpecialRemote instance. When it receives progress log messages, it converts any increment reports to absolute values, and then calls the special remote's send_progress() method, which will cause the respective progress update protocol message to be issued.

Note

Git-annex only supports "context-free" progress reporting. When a progress report is send, it is assumed to be on a currently running transfer. Only a single integer value can be reported, and it corresponds to the number of bytes transferred.

This approach implemented here cannot distinguish progress reports that corresponding to git-annex triggered data transfers and other (potentially co-occurring) operations. The likelihood of unrelated operations reporting progress is relatively low, because this handler is only supposed to be used in dedicated special remote processes, but remains possible.

This implementation is set up to support tracking multiple processes, and could report one of them selectively. However, at present any progress update is relayed to git-annex directly. This could lead to confusing and non-linear progress reporting.

emit(record: LogRecord)[source]

Process a log record

Any incoming log record, compliant with http://docs.datalad.org/design/progress_reporting.html is processed. Increment reports are converted to absolute values, and each update is eventually passed on to special remote, which issues a progress report to git-annex.

datalad_next.patches.customremotes_main.only_progress_logrecords(record: LogRecord) → bool[source]: Log filter to ignore any non-progress log message

datalad_next.patches.customremotes_main.patched_underscore_main(args: list, cls: Type[SpecialRemote])[source]

Full replacement for datalad.customremotes.main._main()

Its only purpose is to create a running instance of a SpecialRemote. The only difference to the original in datalad-core is that once this instance exists, it is linked to a log handler that converts incoming progress log messages to the equivalent annex protocol progress reports.

This additional log handler is a strict addition to the log handling setup established at this point. There should be no interference with any other log message processing.