lsst.obs.base  18.0.0+3
Public Member Functions | Public Attributes | Static Public Attributes | List of all members
lsst.obs.base.gen3.ingest.RawIngestTask Class Reference
Inheritance diagram for lsst.obs.base.gen3.ingest.RawIngestTask:

Public Member Functions

def getDatasetType (cls)
 
def __init__ (self, config=None, butler, kwds)
 
def run (self, files)
 
def readHeaders (self, file)
 
def buildRegion (self, headers)
 
def ensureDimensions (self, file)
 
def ingestFile (self, file, headers, dataId, run=None)
 
def processFile (self, file)
 
def extractDataId (self, file, headers, obsInfo)
 
def getFormatter (self, file, headers, dataId)
 

Public Attributes

 butler
 
 datasetType
 
 dimensions
 
 dimensionEntriesDone
 
 instrumentCache
 
 stashRun
 
 visitRegions
 

Static Public Attributes

 ConfigClass = RawIngestConfig
 

Detailed Description

Driver Task for ingesting raw data into Gen3 Butler repositories.

This Task is intended to be runnable from the command-line, but it doesn't
meet the other requirements of CmdLineTask or PipelineTask, and wouldn't
gain much from being one.  It also wouldn't really be appropriate as a
subtask of a CmdLineTask or PipelineTask; it's a Task essentially just to
leverage the logging and configurability functionality that provides.

Each instance of `RawIngestTask` writes to the same Butler and maintains a
cache of Dimension entries that have already been added to or extracted
from its Registry.  Each invocation of `RawIngestTask.run` ingests a list
of files (possibly semi-atomically; see `RawIngestConfig.onError`).

RawIngestTask may be subclassed to specialize ingest for the actual
structure of raw data files produced by a particular instrument, but this
is usually unnecessary because the instrument-specific header-extraction
provided by the ``astro_metadata_translator`` is usually enough.

Parameters
----------
config : `RawIngestConfig`
    Configuration for whether/how to transfer files and how to handle
    conflicts and errors.
butler : `~lsst.daf.butler.Butler`
    Butler instance.  Ingested Datasets will be created as part of
    ``butler.run`` and associated with its Collection.

Other keyword arguments are forwarded to the Task base class constructor.

Definition at line 106 of file ingest.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.obs.base.gen3.ingest.RawIngestTask.__init__ (   self,
  config = None,
  butler,
  kwds 
)

Definition at line 148 of file ingest.py.

Member Function Documentation

◆ buildRegion()

def lsst.obs.base.gen3.ingest.RawIngestTask.buildRegion (   self,
  headers 
)
Builds a region from information contained in a header

Parameters
----------
headers : `lsst.daf.base.PropertyList`
    Property list containing the information from the header of
    one file.

Returns
-------
region : `lsst.sphgeom.ConvexPolygon`

Raises
------
ValueError :
    If required header keys can not be found to construct region

Definition at line 239 of file ingest.py.

◆ ensureDimensions()

def lsst.obs.base.gen3.ingest.RawIngestTask.ensureDimensions (   self,
  file 
)
Extract metadata from a raw file and add exposure and visit
Dimension entries.

Any needed instrument, detector, and physical_filter Dimension entries must
exist in the Registry before `run` is called.

Parameters
----------
file : `str` or path-like object
    Absolute path to the file to be ingested.

Returns
-------
headers : `list` of `~lsst.daf.base.PropertyList`
    Result of calling `readHeaders`.
dataId : `DataId`
    Data ID dictionary, as returned by `extractDataId`.

Definition at line 267 of file ingest.py.

◆ extractDataId()

def lsst.obs.base.gen3.ingest.RawIngestTask.extractDataId (   self,
  file,
  headers,
  obsInfo 
)
Return the Data ID dictionary that should be used to label a file.

Parameters
----------
file : `str` or path-like object
    Absolute path to the file being ingested (prior to any transfers).
headers : `list` of `~lsst.daf.base.PropertyList`
    All headers returned by `readHeaders()`.
obsInfo : `astro_metadata_translator.ObservationInfo`
    Observational metadata extracted from the headers.

Returns
-------
dataId : `DataId`
    A mapping whose key-value pairs uniquely identify raw datasets.
    Must have ``dataId.dimensions() <= self.dimensions``, with at least
    instrument, exposure, and detector present.

Definition at line 401 of file ingest.py.

◆ getDatasetType()

def lsst.obs.base.gen3.ingest.RawIngestTask.getDatasetType (   cls)
Return the DatasetType of the Datasets ingested by this Task.

Definition at line 142 of file ingest.py.

◆ getFormatter()

def lsst.obs.base.gen3.ingest.RawIngestTask.getFormatter (   self,
  file,
  headers,
  dataId 
)
Return the Formatter that should be used to read this file after
ingestion.

The default implementation obtains the formatter from the Instrument
class for the given data ID.

Definition at line 442 of file ingest.py.

◆ ingestFile()

def lsst.obs.base.gen3.ingest.RawIngestTask.ingestFile (   self,
  file,
  headers,
  dataId,
  run = None 
)
Ingest a single raw file into the repository.

All necessary Dimension entres must already be present.

Parameters
----------
file : `str` or path-like object
    Absolute path to the file to be ingested.
headers : `list` of `~lsst.daf.base.PropertyList`
    Result of calling `readHeaders`.
dataId : `dict`
    Data ID dictionary, as returned by `extractDataId`.
run : `~lsst.daf.butler.Run`, optional
    Run to add the Dataset to; defaults to ``self.butler.run``.

Returns
-------
ref : `DatasetRef`
    Reference to the ingested dataset.

Raises
------
ConflictingDefinitionError
    Raised if the dataset already exists in the registry.

Definition at line 327 of file ingest.py.

◆ processFile()

def lsst.obs.base.gen3.ingest.RawIngestTask.processFile (   self,
  file 
)
Ingest a single raw data file after extacting metadata.

This creates any new exposure or visit Dimension entries needed to
identify the ingest file, creates a new Dataset entry in the
Registry and finally ingests the file itself into the Datastore.
Any needed instrument, detector, and physical_filter Dimension entries must
exist in the Registry before `run` is called.

Parameters
----------
file : `str` or path-like object
    Absolute path to the file to be ingested.

Definition at line 363 of file ingest.py.

◆ readHeaders()

def lsst.obs.base.gen3.ingest.RawIngestTask.readHeaders (   self,
  file 
)
Read and return any relevant headers from the given file.

The default implementation simply reads the header of the first
non-empty HDU, so it always returns a single-element list.

Parameters
----------
file : `str` or path-like object
    Absolute path to the file to be ingested.

Returns
-------
headers : `list` of `~lsst.daf.base.PropertyList`
    Single-element list containing the header of the first
    non-empty HDU.

Definition at line 220 of file ingest.py.

◆ run()

def lsst.obs.base.gen3.ingest.RawIngestTask.run (   self,
  files 
)
Ingest files into a Butler data repository.

This creates any new exposure or visit Dimension entries needed to
identify the ingested files, creates new Dataset entries in the
Registry and finally ingests the files themselves into the Datastore.
Any needed instrument, detector, and physical_filter Dimension entries
must exist in the Registry before `run` is called.

Parameters
----------
files : iterable over `str` or path-like objects
    Paths to the files to be ingested.  Will be made absolute
    if they are not already.

Definition at line 184 of file ingest.py.

Member Data Documentation

◆ butler

lsst.obs.base.gen3.ingest.RawIngestTask.butler

Definition at line 150 of file ingest.py.

◆ ConfigClass

lsst.obs.base.gen3.ingest.RawIngestTask.ConfigClass = RawIngestConfig
static

Definition at line 137 of file ingest.py.

◆ datasetType

lsst.obs.base.gen3.ingest.RawIngestTask.datasetType

Definition at line 151 of file ingest.py.

◆ dimensionEntriesDone

lsst.obs.base.gen3.ingest.RawIngestTask.dimensionEntriesDone

Definition at line 156 of file ingest.py.

◆ dimensions

lsst.obs.base.gen3.ingest.RawIngestTask.dimensions

Definition at line 152 of file ingest.py.

◆ instrumentCache

lsst.obs.base.gen3.ingest.RawIngestTask.instrumentCache

Definition at line 159 of file ingest.py.

◆ stashRun

lsst.obs.base.gen3.ingest.RawIngestTask.stashRun

Definition at line 163 of file ingest.py.

◆ visitRegions

lsst.obs.base.gen3.ingest.RawIngestTask.visitRegions

Definition at line 164 of file ingest.py.


The documentation for this class was generated from the following file: