|
def | getDatasetType (cls) |
|
def | __init__ (self, config=None, butler, kwds) |
|
def | run (self, files) |
|
def | readHeaders (self, file) |
|
def | ensureDataUnits (self, file) |
|
def | ingestFile (self, file, headers, dataId, run=None) |
|
def | processFile (self, file) |
|
def | extractDataId (self, file, headers) |
|
def | extractVisitEntry (self, file, headers, dataId, associated) |
|
def | extractExposureEntry (self, file, headers, dataId, associated) |
|
def | getFormatter (self, file, headers, dataId) |
|
Driver Task for ingesting raw data into Gen3 Butler repositories.
This Task is intended to be runnable from the command-line, but it doesn't
meet the other requirements of CmdLineTask or PipelineTask, and wouldn't
gain much from being one. It also wouldn't really be appropriate as a
subtask of a CmdLineTask or PipelineTask; it's a Task essentially just to
leverage the logging and configurability functionality that provides.
Each instance of `RawIngestTask` writes to the same Butler and maintains a
cache of DataUnit entries that have already been added to or extracted
from its Registry. Each invocation of `RawIngestTask.run` ingests a list
of files (possibly semi-atomically; see `RawIngestConfig.onError`).
RawIngestTask should be subclassed to specialize ingest for the actual
structure of raw data files produced by a particular camera. Subclasses
must either provide populated `MetadataReader` instances in the
`dataIdReader`, `visitReader`, and `exposureReader` class attributes, or
alternate implementations of the `extractDataId`, `extractVisit`, and
`extractExposure` methods that do not use those attributes (each
attribute-method pair may be handled differently). Subclasses may also
wish to override `getFormatter` and/or (rarely) `getDatasetType`. We do
not anticipate overriding `run`, `ensureDataUnits`, `ingestFile`, or
`processFile` to ever be necessary.
Parameters
----------
config : `RawIngestConfig`
Configuration for whether/how to transfer files and how to handle
conflicts and errors.
butler : `~lsst.daf.butler.Butler`
Butler instance. Ingested Datasets will be created as part of
``butler.run`` and associated with its Collection.
Other keyword arguments are forwarded to the Task base class constructor.
Definition at line 82 of file ingest.py.
def lsst.obs.base.gen3.ingest.RawIngestTask.extractDataId |
( |
|
self, |
|
|
|
file, |
|
|
|
headers |
|
) |
| |
Return the Data ID dictionary that should be used to label a file.
Parameters
----------
file : `str` or path-like object
Absolute path to the file being ingested (prior to any transfers).
headers : `list` of `~lsst.daf.base.PropertyList`
All headers returned by `readHeaders()`.
Returns
-------
dataId : `dict`
Must include "camera", "sensor", and "exposure" keys. If the
Exposure is associated with a PhysicalFilter and/or Visit,
"physical_filter" and "visit" keys should be provided as well
(respectively).
Definition at line 343 of file ingest.py.
def lsst.obs.base.gen3.ingest.RawIngestTask.extractExposureEntry |
( |
|
self, |
|
|
|
file, |
|
|
|
headers, |
|
|
|
dataId, |
|
|
|
associated |
|
) |
| |
Create an Exposure DataUnit entry from raw file metadata.
Parameters
----------
file : `str` or path-like object
Absolute path to the file being ingested (prior to any transfers).
headers : `list` of `~lsst.daf.base.PropertyList`
All headers returned by `readHeaders()`.
dataId : `dict`
The data ID for this file. Implementations are permitted to
modify this dictionary (generally by stripping off "sensor" and
adding new metadata key-value pairs) and return it.
associated : `dict`
A dictionary containing other associated DataUnit entries.
Guaranteed to have "Camera", "Sensor", "PhysicalFilter", and
"Visit" keys, but the latter two may map to ``None`` if
`extractDataId` did not contain keys for these or mapped them to
``None``. May also contain additional keys added by
`extractVisitEntry`.
Returns
-------
entry : `dict`
Dictionary corresponding to an Exposure database table row.
Must have all non-null columns in the Exposure table as keys.
Definition at line 403 of file ingest.py.
def lsst.obs.base.gen3.ingest.RawIngestTask.extractVisitEntry |
( |
|
self, |
|
|
|
file, |
|
|
|
headers, |
|
|
|
dataId, |
|
|
|
associated |
|
) |
| |
Create a Visit DataUnit entry from raw file metadata.
Parameters
----------
file : `str` or path-like object
Absolute path to the file being ingested (prior to any transfers).
headers : `list` of `~lsst.daf.base.PropertyList`
All headers returned by `readHeaders()`.
dataId : `dict`
The data ID for this file. Implementations are permitted to
modify this dictionary (generally by stripping off "sensor" and
"exposure" and adding new metadata key-value pairs) and return it.
associated : `dict`
A dictionary containing other associated DataUnit entries.
Guaranteed to have "Camera", "Sensor", and "PhysicalFilter" keys,
but the last may map to ``None`` if `extractDataId` either did not
contain a "physical_filter" key or mapped it to ``None``.
Also adds a "VisitInfo" key containing an `afw.image.VisitInfo`
object for use by `extractExposureEntry`.
Returns
-------
entry : `dict`
Dictionary corresponding to an Visit database table row.
Must have all non-null columns in the Visit table as keys.
Definition at line 370 of file ingest.py.
def lsst.obs.base.gen3.ingest.RawIngestTask.run |
( |
|
self, |
|
|
|
files |
|
) |
| |
Ingest files into a Butler data repository.
This creates any new Exposure or Visit DataUnit entries needed to
identify the ingested files, creates new Dataset entries in the
Registry and finally ingests the files themselves into the Datastore.
Any needed Camera, Sensor, and PhysicalFilter DataUnit entries must
exist in the Registry before `run` is called.
Parameters
----------
files : iterable over `str` or path-like objects
Paths to the files to be ingested. Will be made absolute
if they are not already.
Definition at line 149 of file ingest.py.