lsst.obs.base
17.0.1-11-g20c7f65+1
|
Public Member Functions | |
def | getDatasetType (cls) |
def | __init__ (self, config=None, butler, kwds) |
def | run (self, files) |
def | readHeaders (self, file) |
def | buildRegion (self, headers) |
def | ensureDimensions (self, file) |
def | ingestFile (self, file, headers, dataId, run=None) |
def | processFile (self, file) |
def | extractDataId (self, file, headers, obsInfo) |
def | getFormatter (self, file, headers, dataId) |
Public Attributes | |
butler | |
datasetType | |
dimensions | |
dimensionEntriesDone | |
instrumentCache | |
stashRun | |
visitRegions | |
Static Public Attributes | |
ConfigClass = RawIngestConfig | |
Driver Task for ingesting raw data into Gen3 Butler repositories. This Task is intended to be runnable from the command-line, but it doesn't meet the other requirements of CmdLineTask or PipelineTask, and wouldn't gain much from being one. It also wouldn't really be appropriate as a subtask of a CmdLineTask or PipelineTask; it's a Task essentially just to leverage the logging and configurability functionality that provides. Each instance of `RawIngestTask` writes to the same Butler and maintains a cache of Dimension entries that have already been added to or extracted from its Registry. Each invocation of `RawIngestTask.run` ingests a list of files (possibly semi-atomically; see `RawIngestConfig.onError`). RawIngestTask should be subclassed to specialize ingest for the actual structure of raw data files produced by a particular instrument. Subclasses must either provide populated `MetadataReader` instances in the `dataIdReader`, `visitReader`, and `exposureReader` class attributes, or alternate implementations of the `extractDataId`, `extractVisit`, and `extractExposure` methods that do not use those attributes (each attribute-method pair may be handled differently). Subclasses may also wish to override `getFormatter` and/or (rarely) `getDatasetType`. We do not anticipate overriding `run`, `ensureDimensions`, `ingestFile`, or `processFile` to ever be necessary. Parameters ---------- config : `RawIngestConfig` Configuration for whether/how to transfer files and how to handle conflicts and errors. butler : `~lsst.daf.butler.Butler` Butler instance. Ingested Datasets will be created as part of ``butler.run`` and associated with its Collection. Other keyword arguments are forwarded to the Task base class constructor.
def lsst.obs.base.gen3.ingest.RawIngestTask.__init__ | ( | self, | |
config = None , |
|||
butler, | |||
kwds | |||
) |
def lsst.obs.base.gen3.ingest.RawIngestTask.buildRegion | ( | self, | |
headers | |||
) |
Builds a region from information contained in a header Parameters ---------- headers : `lsst.daf.base.PropertyList` Property list containing the information from the header of one file. Returns ------- region : `lsst.sphgeom.ConvexPolygon` Raises ------ ValueError : If required header keys can not be found to construct region
def lsst.obs.base.gen3.ingest.RawIngestTask.ensureDimensions | ( | self, | |
file | |||
) |
Extract metadata from a raw file and add exposure and visit Dimension entries. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before `run` is called. Parameters ---------- file : `str` or path-like object Absolute path to the file to be ingested. Returns ------- headers : `list` of `~lsst.daf.base.PropertyList` Result of calling `readHeaders`. dataId : `DataId` Data ID dictionary, as returned by `extractDataId`.
def lsst.obs.base.gen3.ingest.RawIngestTask.extractDataId | ( | self, | |
file, | |||
headers, | |||
obsInfo | |||
) |
Return the Data ID dictionary that should be used to label a file. Parameters ---------- file : `str` or path-like object Absolute path to the file being ingested (prior to any transfers). headers : `list` of `~lsst.daf.base.PropertyList` All headers returned by `readHeaders()`. obsInfo : `astro_metadata_translator.ObservationInfo` Observational metadata extracted from the headers. Returns ------- dataId : `DataId` A mapping whose key-value pairs uniquely identify raw datasets. Must have ``dataId.dimensions() <= self.dimensions``, with at least instrument, exposure, and detector present.
def lsst.obs.base.gen3.ingest.RawIngestTask.getDatasetType | ( | cls | ) |
def lsst.obs.base.gen3.ingest.RawIngestTask.getFormatter | ( | self, | |
file, | |||
headers, | |||
dataId | |||
) |
def lsst.obs.base.gen3.ingest.RawIngestTask.ingestFile | ( | self, | |
file, | |||
headers, | |||
dataId, | |||
run = None |
|||
) |
Ingest a single raw file into the repository. All necessary Dimension entres must already be present. This method is not transactional; it must be wrapped in a ``with self.butler.transaction` block to make per-file ingest atomic. Parameters ---------- file : `str` or path-like object Absolute path to the file to be ingested. headers : `list` of `~lsst.daf.base.PropertyList` Result of calling `readHeaders`. dataId : `dict` Data ID dictionary, as returned by `extractDataId`. run : `~lsst.daf.butler.Run`, optional Run to add the Dataset to; defaults to ``self.butler.run``.
def lsst.obs.base.gen3.ingest.RawIngestTask.processFile | ( | self, | |
file | |||
) |
Ingest a single raw data file after extacting metadata. This creates any new exposure or visit Dimension entries needed to identify the ingest file, creates a new Dataset entry in the Registry and finally ingests the file itself into the Datastore. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before `run` is called. Parameters ---------- file : `str` or path-like object Absolute path to the file to be ingested.
def lsst.obs.base.gen3.ingest.RawIngestTask.readHeaders | ( | self, | |
file | |||
) |
Read and return any relevant headers from the given file. The default implementation simply reads the header of the first non-empty HDU, so it always returns a single-element list. Parameters ---------- file : `str` or path-like object Absolute path to the file to be ingested. Returns ------- headers : `list` of `~lsst.daf.base.PropertyList` Single-element list containing the header of the first non-empty HDU.
def lsst.obs.base.gen3.ingest.RawIngestTask.run | ( | self, | |
files | |||
) |
Ingest files into a Butler data repository. This creates any new exposure or visit Dimension entries needed to identify the ingested files, creates new Dataset entries in the Registry and finally ingests the files themselves into the Datastore. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before `run` is called. Parameters ---------- files : iterable over `str` or path-like objects Paths to the files to be ingested. Will be made absolute if they are not already.
|
static |
lsst.obs.base.gen3.ingest.RawIngestTask.dimensionEntriesDone |