datalad.log

Logging setup and utilities, including progress reporting

class datalad.log.ColorFormatter(use_color=None, log_name=False, log_pid=False)[source]

Bases: Formatter

format(record)[source]

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class datalad.log.LoggerHelper(name='datalad', logtarget=None)[source]

Bases: object

Helper to establish and control a Logger

get_initialized_logger(logtarget=None)[source]

Initialize and return the logger

Parameters:

logtarget ({'stderr', str }, optional) – Where to direct the logs. ‘stderr’ stands for the standard stream. Any other string is considered a filename. Multiple entries could be specified comma-separated

Return type:

logging.Logger

set_level(level=None, default='INFO')[source]

Helper to set loglevel for an arbitrary logger

By default operates for ‘datalad’. TODO: deduce name from upper module name so it could be reused without changes

datalad.log.filter_noninteractive_progress(logger, record)[source]

Companion of log_progress() to suppress undesired progress logging

This filter is to be used with a log handler’s addFilter() method for the case of a non-interactive session (e.g., pipe to log file).

It inspects the log record for dlm_progress_noninteractive_level keys that can be injected via log_progress(noninteractive_level=).

If a log-level was declared in this fashion, it will be evaluated against the logger’s effective level, and records are discarded if their level is too low. If no log-level was declared, a log record passes this filter unconditionally.

Parameters:

  • logger (logging.Logger) – The logger instance whose effective level to check against.

  • record – The log record to inspect.

Return type:

bool

datalad.log.log_progress(lgrcall, pid, *args, **kwargs)[source]

Emit progress log messages

This helper can be used to handle progress reporting without having to maintain display mode specific code.

Typical progress reporting via this function involves three types of calls:

  1. Start reporting progress about a process

  2. Update progress information about a process

  3. Report completion of a process

In order to be able to associate all three steps with a particular process, the pid identifier is used. This is an arbitrary string that must be chosen to be unique across all different, but simultaneously running progress reporting activities within a Python session. For many practical purposes this can be achieved by, for example, including path information in the identifier.

To initialize a progress report this function is called without an update parameter. To report a progress update, this function is called with an update parameter. To finish a reporting on a particular activity a final call without an update parameter is required.

Parameters:

  • lgrcall (callable) – Something like lgr.debug or lgr.info

  • pid (str) – Some kind of ID for the process the progress is reported on.

  • *args (str) – Log message, and potential arguments

  • total (int) – Max progress quantity of the process.

  • label (str) – Process description. Should be very brief, goes in front of progress bar on the same line.

  • unit (str) – Progress report unit. Should be very brief, goes after the progress bar on the same line.

  • update (int) – To (or by) which quantity to advance the progress. Also see increment.

  • increment (bool) – If set, update is interpreted as an incremental value, not absolute.

  • initial (int) – If set, start value for progress bar

  • noninteractive_level (int, optional) – In a non-interactive session where progress bars are not displayed, only log a progress report, if a logger’s effective level includes the specified level. This can be useful logging all progress is inappropriate or too noisy for a log.

  • maint ({'clear', 'refresh'}) – This is a special attribute that can be used by callers that are not actually reporting progress, but need to ensure that their (console) output does not interfere with any possibly ongoing progress reporting. Setting this attribute to ‘clear’ will cause the central ProgressHandler to temporarily stop the display of any active progress bars. With ‘refresh’, all active progress bars will be redisplayed. After a ‘clear’ individual progress bars would be reactivated upon the next update log message, even without an explicit ‘refresh’.

datalad.log.with_progress(items, lgrcall=None, label='Total', unit=' Files')[source]

Wrap a progress bar, with status counts, around an iterable.

Parameters:

  • items (some iterable) –

  • lgrcall (callable) – Callable for logging. If not specified - lgr.info is used

  • label (str) – Passed to log.log_progress.

  • unit (str) – Passed to log.log_progress.

Yields:

Items of it while displaying the progress

datalad.log.with_result_progress(fn, label='Total', unit=' Files', log_filter=None)[source]

Wrap a progress bar, with status counts, around a function.

Parameters:

  • fn (generator function) – This function should accept a collection of items as a positional argument and any number of keyword arguments. After processing each item in the collection, it should yield a status dict.

  • log_filter (callable, optional) – If defined, only result records for which callable evaluates to True will be passed to log_progress

  • label (str) – Passed to log.log_progress.

  • unit (str) – Passed to log.log_progress.

Returns:

  • A variant of fn that shows a progress bar. Note that the wrapped

  • function is not a generator function; the status dicts will be

  • returned as a list.