"""
*uncurl* git-annex external special remote
==========================================
This implementation is a git-annex accessible interface to datalad-next's URL
operations framework. It serves two main purposes:
1. Combine git-annex's capabilities of registering and accessing file content
via URLs with DataLad's access credential management and (additional or
alternative) transport protocol implementations.
2. Minimize the maintenance effort for datasets (primarily) composed from
content that is remotely accessible via URLs from systems other than
Datalad or git-annex in the event of an infrastructure transition
(e.g. moving to a different technical system or a different data
organization on a storage system).
Requirements
------------
This special remote implementation requires git-annex version 8.20210127 (or
later) to be available.
Download helper
---------------
The simplest way to use this remote is to initialize it without any particular
configuration::
$ git annex initremote uncurl type=external externaltype=uncurl encryption=none
initremote uncurl ok
(recording state in git...)
Once initialized, or later enabled in a clone, ``git-annex addurl`` will check
with the *uncurl* remote whether it can handle a particular URL, and will let
the remote perform the download in case of positive response. By default, the
remote will claim any URLs with a scheme that the local datalad-next
installation supports. This always includes ``file://``, ``http://``, and
``https://``, but is extensible, and a particular installation may also support
``ssh://`` (by default when openssh is installed), or other schemes.
This additional URL support is also available for other commands. Here is an
example how ``datalad addurls`` can be given any uncurl-supported URLs
(here an SSH-URL) directly, provided that the ``uncurl`` remote was initialized
for a dataset (as shown above)::
$ echo '[{"url":"ssh://my.server.org/home/me/file", "file":"dummy"}]' \\
| datalad addurls - '{url}' '{file}'
This makes legacy commands (e.g., ``datalad download-url``), unnecessary, and
facilitates the use of more advanced ``datalad addurls`` features (e.g.,
automatic creation of subdatasets) that are not provided by lower-level
commands like ``git annex addurl``.
Download helper with credential management support
--------------------------------------------------
With this setup, download requests now also use DataLad's credential system for
authentication. DataLad will automatically lookup matching credentials, prompt
for manual entry if none are found, and offer to store them securely for later
use after having used them successfully::
$ git annex addurl http://httpbin.org/basic-auth/myuser/mypassword
Credential needed for access to http://httpbin.org/basic-auth/myuser/mypassword
user: myuser
password:
password (repeat):
Enter a name to save the credential
(for accessing http://httpbin.org/basic-auth/myuser/mypassword) securely for future
reuse, or 'skip' to not save the credential
name: httpbin-dummy
addurl http://httpbin.org/basic-auth/myuser/mypassword (from uncurl) (to ...)
ok
(recording state in git...)
By adding files via downloads from URLs in this fashion, datasets can be built
that track information across a range of locations/services, using a possibly
heterogeneous set of access methods.
This feature is very similar to the ``datalad`` special remote implementation
included in the core DataLad package. The difference here is that alternative
implementations of downloaders are employed and the ``datalad-next`` credential
system is used instead of the "providers" mechanism from DataLad's core
package.
Transforming recorded URLs
--------------------------
The main benefit of using *uncurl* is, however, only revealed when the original
snapshot of where data used to be accessible becomes invalid, maybe because
data were moved to a different storage system, or simply a different host.
This would typically require an update of each, now broken, access URL. For
datasets with thousands or even millions of files this can be an expensive
operation. For data portal operators providing a large number of datasets it is
even more tedious.
*uncurl* enables programmatic, on-access URL rewriting. This is similar, in
spirit, to Git's ``url.<base>.insteadOf`` URL modification feature. However,
modification possibilities reach substantially beyond replacing a base URL.
This feature is based on two customizable settings: 1) a *URL template*; and
2) a *set of match expressions* that extract additional identifiers
from any recorded access URL for an annex key.
Here is an example: Let's say a file in a dataset has a recorded access URL
of::
https://data.example.org/c542/s7612_figure1.pdf
We can let *uncurl* know that ``c542`` is actually an identifier for a
particular collection of items in this data store. Likewise ``s7612`` is an
identifier of a particular item in that collection, and ``figure1.pdf`` is the
name of a component in that collection item. The following Python regular
expression can be used to "decompose" the above URL into these semantic
components::
(?P<site>https://[^/]+)/(?P<collection>c[^/]+)/(?P<item>s[^/]+)_(?P<component>.*)$
This expression is not the most readable, but it basically chunks the URL
into segments of ``(?P<name>...)``, so-called named groups (see a
`live demo of this expression <https://www.debuggex.com/r/Aa1yua-awXBuqZ39>`__).
This expression, and additional ones like it, can set as a configuration
parameter of an *uncurl* remote setup. Extending the configuration established
by the ``initremote`` call above::
$ git annex enableremote uncurl \\
'match=(?P<site>https://[^/]+)/(?P<collection>c[^/]+)/(?P<item>s[^/]+)_(?P<component>.*)$'
The last argument is quoted to prevent it from being processed by the shell.
With the match expression configured, URL rewriting can be enabled by declaring
a URL template as another configuration item. The URL template uses the `Python
Format String Syntax
<https://docs.python.org/3/library/string.html#format-string-syntax>`__. If the
new URL for the file above is now
``http://newsite.net/ex-archive/c542_s7612_figure1.pdf``, we can declare
the following URL template to have *uncurl* go to the new site::
http://newsite.net/ex-archive/{collection}_{item}_{component}
This template references the identifiers of the named groups we defined in the
match expression. Again, the URL template can be set via ``git annex
enableremote``::
$ git annex enableremote uncurl \\
'url=http://newsite.net/ex-archive/{collection}_{item}_{component}'
There is no need to separate the ``enableremote`` calls. Both configuration can
be given at the same time. In fact, they can also be given to ``initremote``
immediately.
The three identifiers ``site``, ``collection``, ``item``, and ``component`` are
actually a custom addition to a standard set of identifiers that are available
for composing URLs via a template.
- ``datalad_dsid`` - the DataLad dataset ID (UUID)
- ``annex_dirhash`` - "mixed" variant of the two level hash for a particular key
(uses POSIX directory separators, and included a trailing separator)
- ``annex_dirhash_lower`` - "lower case" variant of the two level hash for a
particular key (uses POSIX directory separators, and included a trailing
separator)
- ``annex_key`` - git-annex key name for a request
- ``annex_remoteuuid`` - UUID of the special remote (location) used by git-annex
- ``git_remotename`` - Name of the Git remote for the uncurl special remote
.. note::
The URL template must "resolve" to a complete and valid URL. This cannot
be verified at configuration time, because even the URL scheme could be a
dynamic setting.
Uploading content
-----------------
The *uncurl* special remote can upload file content or store annex keys
via supported URL schemes whenever a URL template is defined. At minimum,
storing at ``file://`` and ``ssh://`` URLs are supported. But other URL
scheme handlers with upload support may be available at a local DataLad
installation.
Deleting content
----------------
As for uploading, deleting content is only permitted with a configured
URL template. Moreover, it also depends on the delete operation being
supported for a particular URL scheme.
Configuration overrides
-----------------------
Both match expressions and the URL template can also be configured in a dataset's
configuration (committed branch configuration, or any Git configuration scope
(local, global, system) using the following configuration item names:
- ``remote.<remotename>.uncurl-url``
- ``remote.<remotename>.uncurl-match``
where ``<remotename>`` is the name of the special remote in the dataset.
A URL template provided via configuration *overrides* one defined in the special
remote setup via ``init/enableremote``.
Match expressions defined as configuration items *extend* the set of match
expressions that may be included in the special remote setup via
``init/enableremote``. The ``remote.<remotename>.uncurl-match`` configuration
item can be set as often as necessary (which one match expression each).
Tips
----
When multiple match expressions are defined, it is recommended to use unique
names for each match-group to avoid collisions.
"""
from __future__ import annotations
from annexremote import Master
from functools import partial
from pathlib import Path
import re
from typing import (
Callable,
Pattern,
)
from datalad_next.exceptions import (
CapturedException,
UrlOperationsRemoteError,
UrlOperationsResourceUnknown,
)
from datalad_next.url_operations import AnyUrlOperations
from datalad_next.utils import ensure_list
from . import (
RemoteError,
SpecialRemote,
super_main
)
[docs]
class UncurlRemote(SpecialRemote):
""" """
def __init__(self, annex: Master):
super().__init__(annex)
self.configs.update(
url='Python format language template composing an access URL',
match='(whitespace-separated list of) regular expression(s) to match particular components in supported URL via named groups',
)
self.url_tmpl: str | None = None
self.match: list[Pattern[str]] | None = None
self.url_handler: AnyUrlOperations | None = None
# cache of properties that do not vary within a session
# or across annex keys
self.persistent_tmpl_props: dict[str, str] = {}
[docs]
def initremote(self) -> None:
# at present there is nothing that needs to be done on init/enable.
# the remote is designed to work without any specific setup too
pass
[docs]
def prepare(self) -> None:
# we need the git remote name to be able to look up config about
# that remote
# check the config for a URL template setting
self.url_tmpl = self.get_remote_gitcfg('uncurl', 'url', '')
# only if we have no local, overriding, configuration ask git-annex
# for the committed special remote config on the URL template
if not self.url_tmpl:
# ask for the commit config, could still be empty
self.url_tmpl = self.annex.getconfig('url')
# TODO test the case of a fully absent URL template
# that would be fine and only verbatim recorded URLs could
# be sent to a downloader
# unconditionally ask git-annex for a match-url setting, any local
# config amends, and does not override
match_spec: str = self.annex.getconfig('match')
if match_spec:
# TODO implement sanity checks, but running it through re.compile()
# might just be enough
matches = [re.compile(m) for m in match_spec.split()]
else:
matches = []
# extend with additional matchers from local config
self.match = matches + [
re.compile(m)
for m in ensure_list(self.get_remote_gitcfg(
'uncurl', 'match', default=[], get_all=True))
]
self.message(
f'URL rewriting template: {self.url_tmpl!r}', type='debug')
self.message(
f'Active URL match expressions: {[e.pattern for e in self.match]!r}',
type='debug')
# let the URL handler use the repo's config
self.url_handler = AnyUrlOperations(cfg=self.repo.cfg)
# cache template properties
# using function arg name syntax, we need the identifiers to be valid
# Python symbols to work in `format()`
self.persistent_tmpl_props.update(
datalad_dsid=self.repo.cfg.get('datalad.dataset.id', ''),
git_remotename=self.remotename,
annex_remoteuuid=self.annex.getuuid(),
)
[docs]
def claimurl(self, url: str) -> bool:
"""Needs to check if want to handle a given URL
If match expressions are configured, matches the URL against all known
URL expressions, and returns `True` if there is any match, or
`False` otherwise.
If no match expressions are configured, return `True` of the URL
scheme is supported, or `False` otherwise.
"""
assert self.url_handler
if self.match:
return self.is_recognized_url(url)
else:
return self.url_handler.is_supported_url(url)
[docs]
def checkurl(self, url: str) -> bool:
"""
When running `git-annex addurl`, this is called after CLAIMURL
indicated that we could handle a URL. It can return information
on the URL target (e.g., size of the download, a target filename,
or a sequence thereof with additional URLs pointing to individual
components that would jointly make up the full download from the
given URL. However, all of that is optional, and a simple `True`
returned is sufficient to make git-annex call `TRANSFER RETRIEVE`.
"""
assert self.url_handler
url_tmpl = self.url_tmpl or ''
# try-except, because a URL template might need something
# that cannot be extracted from this very URL.
# we cannot consult a key that may already point to the same
# content as the URL, and may have other information --
# we simply dont have it at this point
try:
mangled_url = self.get_mangled_url(
url,
url_tmpl,
self.extract_tmpl_props(
tmpl=url_tmpl,
urls=[url],
),
)
except KeyError as e:
self.message(
'URL rewriting template requires unavailable component '
f'{e}, continuing with original URL',
type='debug',
)
# otherwise go ahead with the original URL. the template might
# just be here to aid structured uploads
mangled_url = url
try:
self.url_handler.stat(mangled_url)
return True
except UrlOperationsRemoteError as e:
# leave a trace in the logs
CapturedException(e)
return False
# we could return a URL/size/filename triple instead of a bool.
# this would make git annex download from a URL different from the input,
# and to the given filename.
# it could be nice to report the (mangled) url, even if the handler reports
# a potentially deviating URL (redirects, etc.). Keeping external
# resolvers in the loop can be intentional, and users could provide
# resolved URL if they consider that desirable.
# however, going with the original URL kinda does that already, rewriting
# is happening anyways. And not reporting triplets avoids the issue
# of git-annex insisting to write into a dedicated directory for this
# download.
[docs]
def transfer_retrieve(self, key: str, filename: str) -> None:
assert self.url_handler
self._check_retrieve(
key,
partial(self.url_handler.download, to_path=Path(filename)),
('download', 'from'),
)
[docs]
def checkpresent(self, key: str) -> bool:
assert self.url_handler
return self._check_retrieve(
key,
self.url_handler.stat,
('find', 'at'),
)
[docs]
def transfer_store(self, key: str, filename: str) -> None:
assert self.url_handler
self._store_delete(
key,
partial(self.url_handler.upload, from_path=Path(filename)),
'cannot store',
)
[docs]
def remove(self, key: str) -> None:
def _delete(to_url: str) -> None:
# we have to map parameter names to be able to use a common
# helper with transfer_store(), because UrlOperations.upload()
# needs to get the URL as a second argument, hence we need
# to pass parameters as keyword-args
assert self.url_handler
self.url_handler.delete(url=to_url)
return
try:
self._store_delete(
key,
_delete,
'refuses to delete',
)
except UrlOperationsResourceUnknown:
self.message(
f'{key!r} not found at the remote, skipping', type='debug')
#
# helpers
#
[docs]
def is_recognized_url(self, url: str) -> bool:
return any(m.match(url) for m in self.match or [])
[docs]
def get_key_urls(self, key: str) -> list[str]:
# ask git-annex for the URLs it has on record for the key.
# this will also work within checkurl() for a temporary key
# generated by git-annex after claimurl()
urls = self.annex.geturls(key, prefix='')
self.message(f'Known urls for {key!r}: {urls}', type='debug')
if self.url_tmpl:
# we have a rewriting template. extract all properties
# from all known URLs and instantiate the template
# to get the ONE desired URL
props = self.extract_tmpl_props(
tmpl=self.url_tmpl,
urls=urls,
key=key,
)
url = self.get_mangled_url(
fallback_url=None,
tmpl=self.url_tmpl,
tmpl_props=props,
)
return [url] if url else []
# we have no rewriting template, and must return all URLs we know
# to let the caller sort it out
return urls
[docs]
def get_mangled_url(
self,
fallback_url: str | None,
tmpl: str,
tmpl_props: dict[str, str],
) -> str | None:
if not tmpl:
# the best thing we can do without a URL template is to
# return the URL itself
return fallback_url
url = tmpl.format(**tmpl_props)
return url
def _check_retrieve(
self,
key: str,
handler: Callable,
action: tuple[str, str],
) -> bool:
urls = self.get_key_urls(key)
# depending on the configuration (rewriting template or not)
# we could have one or more URLs to try
for url in urls:
try:
handler(url)
# we succeeded, no need to try again
return True
except UrlOperationsResourceUnknown:
# general system access worked, but at the key location is nothing
# to be found
return False
except UrlOperationsRemoteError as e:
# return False only if we could be sure that the remote
# system works properly and just the key is not around
CapturedException(e)
self.message(
f'Failed to {action[0]} key {key!r} {action[1]} {url!r}',
type='debug')
raise RemoteError(
f'Failed to {action[0]} {key!r} {action[1]} any of {urls!r}')
def _store_delete(
self,
key: str,
handler: Callable,
action: str,
) -> None:
if not self.url_tmpl:
raise RemoteError(
f'Remote {action} content without a configured URL template')
urls = self.get_key_urls(key)
# we have a rewriting template, so we expect exactly one URL
assert len(urls) == 1
url = urls[0]
try:
handler(to_url=url)
except UrlOperationsResourceUnknown:
# pass-through, would happen when removing a non-existing key,
# which git-annex wants to be a OK thing to happen.
# handler in callers
raise
except Exception as e:
# we need to raise RemoteError whenever we could not perform
raise RemoteError from e
_stat2checkurl_map = {
'content-length': 'size',
}
"""Translate property names returned by AnyUrlOperations.stat()
to those expected from checkurl()"""
[docs]
def main():
"""cmdline entry point"""
super_main(
cls=UncurlRemote,
remote_name='uncurl',
description=\
"flexible access data (in archive systems) "
"via a variety of identification schemes",
)