lsst.obs.base
18.0.0-7-g898c2a1
|
Public Member Functions | |
def | getDatasetType (self) |
def | __init__ (self, config=None, butler, kwds) |
def | run (self, files) |
def | readHeaders (self, file) |
def | buildRegion (self, headers) |
def | ensureDimensions (self, file) |
def | ingestFile (self, file, headers, dataId, run=None) |
def | processFile (self, file) |
def | extractDataId (self, file, headers, obsInfo) |
def | getFormatter (self, file, headers, dataId) |
Public Attributes | |
butler | |
datasetType | |
dimensions | |
dimensionEntriesDone | |
instrumentCache | |
stashRun | |
visitRegions | |
Static Public Attributes | |
ConfigClass = RawIngestConfig | |
Driver Task for ingesting raw data into Gen3 Butler repositories. This Task is intended to be runnable from the command-line, but it doesn't meet the other requirements of CmdLineTask or PipelineTask, and wouldn't gain much from being one. It also wouldn't really be appropriate as a subtask of a CmdLineTask or PipelineTask; it's a Task essentially just to leverage the logging and configurability functionality that provides. Each instance of `RawIngestTask` writes to the same Butler and maintains a cache of Dimension entries that have already been added to or extracted from its Registry. Each invocation of `RawIngestTask.run` ingests a list of files (possibly semi-atomically; see `RawIngestConfig.onError`). RawIngestTask may be subclassed to specialize ingest for the actual structure of raw data files produced by a particular instrument, but this is usually unnecessary because the instrument-specific header-extraction provided by the ``astro_metadata_translator`` is usually enough. Parameters ---------- config : `RawIngestConfig` Configuration for whether/how to transfer files and how to handle conflicts and errors. butler : `~lsst.daf.butler.Butler` Butler instance. Ingested Datasets will be created as part of ``butler.run`` and associated with its Collection. Other keyword arguments are forwarded to the Task base class constructor.
def lsst.obs.base.gen3.ingest.RawIngestTask.__init__ | ( | self, | |
config = None , |
|||
butler, | |||
kwds | |||
) |
def lsst.obs.base.gen3.ingest.RawIngestTask.buildRegion | ( | self, | |
headers | |||
) |
Builds a region from information contained in a header Parameters ---------- headers : `lsst.daf.base.PropertyList` Property list containing the information from the header of one file. Returns ------- region : `lsst.sphgeom.ConvexPolygon` Raises ------ ValueError : If required header keys can not be found to construct region
def lsst.obs.base.gen3.ingest.RawIngestTask.ensureDimensions | ( | self, | |
file | |||
) |
Extract metadata from a raw file and add exposure and visit Dimension entries. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before `run` is called. Parameters ---------- file : `str` or path-like object Absolute path to the file to be ingested. Returns ------- headers : `list` of `~lsst.daf.base.PropertyList` Result of calling `readHeaders`. dataId : `DataId` Data ID dictionary, as returned by `extractDataId`.
def lsst.obs.base.gen3.ingest.RawIngestTask.extractDataId | ( | self, | |
file, | |||
headers, | |||
obsInfo | |||
) |
Return the Data ID dictionary that should be used to label a file. Parameters ---------- file : `str` or path-like object Absolute path to the file being ingested (prior to any transfers). headers : `list` of `~lsst.daf.base.PropertyList` All headers returned by `readHeaders()`. obsInfo : `astro_metadata_translator.ObservationInfo` Observational metadata extracted from the headers. Returns ------- dataId : `DataId` A mapping whose key-value pairs uniquely identify raw datasets. Must have ``dataId.dimensions() <= self.dimensions``, with at least instrument, exposure, and detector present.
def lsst.obs.base.gen3.ingest.RawIngestTask.getDatasetType | ( | self | ) |
def lsst.obs.base.gen3.ingest.RawIngestTask.getFormatter | ( | self, | |
file, | |||
headers, | |||
dataId | |||
) |
def lsst.obs.base.gen3.ingest.RawIngestTask.ingestFile | ( | self, | |
file, | |||
headers, | |||
dataId, | |||
run = None |
|||
) |
Ingest a single raw file into the repository. All necessary Dimension entres must already be present. Parameters ---------- file : `str` or path-like object Absolute path to the file to be ingested. headers : `list` of `~lsst.daf.base.PropertyList` Result of calling `readHeaders`. dataId : `dict` Data ID dictionary, as returned by `extractDataId`. run : `~lsst.daf.butler.Run`, optional Run to add the Dataset to; defaults to ``self.butler.run``. Returns ------- ref : `DatasetRef` Reference to the ingested dataset. Raises ------ ConflictingDefinitionError Raised if the dataset already exists in the registry.
def lsst.obs.base.gen3.ingest.RawIngestTask.processFile | ( | self, | |
file | |||
) |
Ingest a single raw data file after extacting metadata. This creates any new exposure or visit Dimension entries needed to identify the ingest file, creates a new Dataset entry in the Registry and finally ingests the file itself into the Datastore. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before `run` is called. Parameters ---------- file : `str` or path-like object Absolute path to the file to be ingested.
def lsst.obs.base.gen3.ingest.RawIngestTask.readHeaders | ( | self, | |
file | |||
) |
Read and return any relevant headers from the given file. The default implementation simply reads the header of the first non-empty HDU, so it always returns a single-element list. Parameters ---------- file : `str` or path-like object Absolute path to the file to be ingested. Returns ------- headers : `list` of `~lsst.daf.base.PropertyList` Single-element list containing the header of the first non-empty HDU.
def lsst.obs.base.gen3.ingest.RawIngestTask.run | ( | self, | |
files | |||
) |
Ingest files into a Butler data repository. This creates any new exposure or visit Dimension entries needed to identify the ingested files, creates new Dataset entries in the Registry and finally ingests the files themselves into the Datastore. Any needed instrument, detector, and physical_filter Dimension entries must exist in the Registry before `run` is called. Parameters ---------- files : iterable over `str` or path-like objects Paths to the files to be ingested. Will be made absolute if they are not already.
|
static |
lsst.obs.base.gen3.ingest.RawIngestTask.dimensionEntriesDone |