lsst.meas.algorithms g04242d3e92+8eaa23c173
Loading...
Searching...
No Matches
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Protected Member Functions | Static Protected Attributes | List of all members
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager Class Reference
Inheritance diagram for lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager:
lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager lsst.meas.algorithms.convertRefcatManager.ConvertGaiaXpManager

Public Member Functions

 __init__ (self, filenames, config, file_reader, indexer, schema, key_map, htmRange, addRefCatMetadata, log)
 
 run (self, inputFiles)
 
 getCatalog (self, pixelId, schema, nNewElements)
 

Static Public Member Functions

 computeCoord (row, ra_name, dec_name)
 

Public Attributes

 filenames
 
 config
 
 file_reader
 
 indexer
 
 schema
 
 key_map
 
 htmRange
 
 addRefCatMetadata
 
 log
 
 coord_err_unit
 
 nInputFiles
 

Static Public Attributes

 ConfigClass = ConvertRefcatManagerConfig
 

Protected Member Functions

 _convertOneFile (self, filename, fileLocks)
 
 _doOnePixel (self, inputData, matchedPixels, pixelId, fluxes, coordErr)
 
 _setIds (self, inputData, catalog)
 
 _getCoordErr (self, inputData)
 
 _setFlags (self, record, row)
 
 _getFluxes (self, inputData)
 
 _setProperMotion (self, record, row)
 
 _setParallax (self, record, row)
 
 _epochToMjdTai (self, nativeEpoch)
 
 _setCoordinateCovariance (self, record, row)
 
 _setExtra (self, record, row)
 
 _fillRecord (self, record, row)
 

Static Protected Attributes

list _flags = ['photometric', 'resolved', 'variable']
 
str _DefaultName = 'convertRefcatManager'
 

Detailed Description

Convert a reference catalog from external files into the LSST HTM sharded
format, using a multiprocessing Pool to speed up the work.

Parameters
----------
filenames : `dict` [`int`, `str`]
    The HTM pixel id and filenames to convert the catalog into.
config : `lsst.meas.algorithms.ConvertReferenceCatalogConfig`
    The Task configuration holding the field names.
file_reader : `lsst.pipe.base.Task`
    The file reader to use to load the files.
indexer : `lsst.meas.algorithms.HtmIndexer`
    The class used to compute the HTM pixel per coordinate.
schema : `lsst.afw.table.Schema`
    The schema of the output catalog.
key_map : `dict` [`str`, `lsst.afw.table.Key`]
    The mapping from output field names to keys in the Schema.
htmRange : `tuple` [`int`]
    The start and end HTM pixel ids.
addRefCatMetadata : callable
    A function called to add extra metadata to each output Catalog.
log : `lsst.log.Log` or `logging.Logger`
    The log to send messages to.

Definition at line 54 of file convertRefcatManager.py.

Constructor & Destructor Documentation

◆ __init__()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.__init__ ( self,
filenames,
config,
file_reader,
indexer,
schema,
key_map,
htmRange,
addRefCatMetadata,
log )

Member Function Documentation

◆ _convertOneFile()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._convertOneFile ( self,
filename,
fileLocks )
protected
Read and process one file, and write its records to the correct
indexed files, while handling exceptions in a useful way so that they
don't get swallowed by the multiprocess pool.

Parameters
----------
filename : `str`
    The file to process.
fileLocks : `dict` [`int`, `multiprocessing.Lock`]
    A Lock for each HTM pixel; each pixel gets one file written, and
    we need to block when one process is accessing that file.

Returns
-------
pixels, files : `list` [`int`]
    The pixel ids that were written to.

Definition at line 134 of file convertRefcatManager.py.

◆ _doOnePixel()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._doOnePixel ( self,
inputData,
matchedPixels,
pixelId,
fluxes,
coordErr )
protected
Process one HTM pixel, appending to an existing catalog or creating
a new catalog, as needed.

Parameters
----------
inputData : `numpy.ndarray`
    The data from one input file.
matchedPixels : `numpy.ndarray`
    The row-matched pixel indexes corresponding to ``inputData``.
pixelId : `int`
    The pixel index we are currently processing.
fluxes : `dict` [`str`, `numpy.ndarray`]
    The values that will go into the flux and fluxErr fields in the
    output catalog.
coordErr : `dict` [`str`, `numpy.ndarray`]
    The values that will go into the coord_raErr, coord_decErr, and
    coord_ra_dec_Cov fields in the output catalog (in radians).

Definition at line 185 of file convertRefcatManager.py.

◆ _epochToMjdTai()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._epochToMjdTai ( self,
nativeEpoch )
protected
Convert an epoch in native format to TAI MJD (a float).

Definition at line 398 of file convertRefcatManager.py.

◆ _fillRecord()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._fillRecord ( self,
record,
row )
protected
Fill a record in an indexed catalog to be persisted.

Parameters
----------
record : `lsst.afw.table.SimpleRecord`
    Row from indexed catalog to modify.
row : structured `numpy.array`
    Row from catalog being converted.

Definition at line 444 of file convertRefcatManager.py.

◆ _getCoordErr()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._getCoordErr ( self,
inputData )
protected
Compute the ra/dec error fields that will go into the output catalog.

Parameters
----------
inputData : `numpy.ndarray`
    The input data to compute fluxes for.

Returns
-------
coordErr : `dict` [`str`, `numpy.ndarray`]
    The values that will go into the coord_raErr, coord_decErr, fields
    in the output catalog (in radians).

Notes
-----
This does not handle the ra/dec covariance field,
``coord_ra_coord_dec_Cov``. That field is handled in
`_setCoordinateCovariance`.

Definition at line 294 of file convertRefcatManager.py.

◆ _getFluxes()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._getFluxes ( self,
inputData )
protected
Compute the flux fields that will go into the output catalog.

Parameters
----------
inputData : `numpy.ndarray`
    The input data to compute fluxes for.

Returns
-------
fluxes : `dict` [`str`, `numpy.ndarray`]
    The values that will go into the flux and fluxErr fields in the
    output catalog.

Reimplemented in lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager, and lsst.meas.algorithms.convertRefcatManager.ConvertGaiaXpManager.

Definition at line 338 of file convertRefcatManager.py.

◆ _setCoordinateCovariance()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._setCoordinateCovariance ( self,
record,
row )
protected
Set the off-diagonal position covariance in a record of an indexed
catalog.

There is no generic way to determine covariance. Override this method
in a subclass specialized for your dataset.

Parameters
----------
record : `lsst.afw.table.SimpleRecord`
    Row from indexed catalog to modify.
row : structured `numpy.array`
    Row from catalog being converted.

Reimplemented in lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager.

Definition at line 404 of file convertRefcatManager.py.

◆ _setExtra()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._setExtra ( self,
record,
row )
protected
Set extra data fields in a record of an indexed catalog.

Parameters
----------
record : `lsst.afw.table.SimpleRecord`
    Row from indexed catalog to modify.
row : structured `numpy.array`
    Row from catalog being converted.

Definition at line 421 of file convertRefcatManager.py.

◆ _setFlags()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._setFlags ( self,
record,
row )
protected
Set flags in an output record.

Parameters
----------
record : `lsst.afw.table.SimpleRecord`
    Row from indexed catalog to modify.
row : `numpy.ndarray`
    Row from catalog being converted.

Definition at line 322 of file convertRefcatManager.py.

◆ _setIds()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._setIds ( self,
inputData,
catalog )
protected
Fill the `id` field of catalog with a running index, filling the
last values up to the length of ``inputData``.

Fill with `self.config.id_name` if specified, otherwise use the
global running counter value.

Parameters
----------
inputData : `numpy.ndarray`
    The input data that is being processed.
catalog : `lsst.afw.table.SimpleCatalog`
    The output catalog to fill the ids.

Definition at line 223 of file convertRefcatManager.py.

◆ _setParallax()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._setParallax ( self,
record,
row )
protected
Set the parallax fields in a record of a refcat.

Definition at line 389 of file convertRefcatManager.py.

◆ _setProperMotion()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._setProperMotion ( self,
record,
row )
protected
Set proper motion fields in a record of an indexed catalog.

The proper motions are read from the specified columns,
scaled appropriately, and installed in the appropriate
columns of the output.

Parameters
----------
record : `lsst.afw.table.SimpleRecord`
    Row from indexed catalog to modify.
row : structured `numpy.array`
    Row from catalog being converted.

Definition at line 365 of file convertRefcatManager.py.

◆ computeCoord()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.computeCoord ( row,
ra_name,
dec_name )
static
Create an ICRS coord. from a row of a catalog being converted.

Parameters
----------
row : `numpy.ndarray`
    Row from catalog being converted.
ra_name : `str`
    Name of RA key in catalog being converted.
dec_name : `str`
    Name of Dec key in catalog being converted.

Returns
-------
coord : `lsst.geom.SpherePoint`
    ICRS coordinate.

Definition at line 275 of file convertRefcatManager.py.

◆ getCatalog()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.getCatalog ( self,
pixelId,
schema,
nNewElements )
Get a catalog from disk or create it if it doesn't exist.

Parameters
----------
pixelId : `dict`
    Identifier for catalog to retrieve
schema : `lsst.afw.table.Schema`
    Schema to use in catalog creation it does not exist.
nNewElements : `int`
    The number of new elements that will be added to the catalog,
    so space can be preallocated.

Returns
-------
catalog : `lsst.afw.table.SimpleCatalog`
    The new or read-and-resized catalog specified by `dataId`.

Definition at line 246 of file convertRefcatManager.py.

◆ run()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.run ( self,
inputFiles )
Index a set of input files from a reference catalog, and write the
output to the appropriate filenames, in parallel.

Parameters
----------
inputFiles : `list`
    A list of file paths to read data from.

Returns
-------
output : `dict` [`int`, `str`]
    The htm ids and the filenames that were written to.

Definition at line 100 of file convertRefcatManager.py.

Member Data Documentation

◆ _DefaultName

str lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._DefaultName = 'convertRefcatManager'
staticprotected

Definition at line 81 of file convertRefcatManager.py.

◆ _flags

list lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._flags = ['photometric', 'resolved', 'variable']
staticprotected

Definition at line 80 of file convertRefcatManager.py.

◆ addRefCatMetadata

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.addRefCatMetadata

Definition at line 93 of file convertRefcatManager.py.

◆ config

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.config

Definition at line 87 of file convertRefcatManager.py.

◆ ConfigClass

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.ConfigClass = ConvertRefcatManagerConfig
static

Definition at line 82 of file convertRefcatManager.py.

◆ coord_err_unit

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.coord_err_unit

Definition at line 98 of file convertRefcatManager.py.

◆ file_reader

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.file_reader

Definition at line 88 of file convertRefcatManager.py.

◆ filenames

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.filenames

Definition at line 86 of file convertRefcatManager.py.

◆ htmRange

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.htmRange

Definition at line 92 of file convertRefcatManager.py.

◆ indexer

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.indexer

Definition at line 89 of file convertRefcatManager.py.

◆ key_map

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.key_map

Definition at line 91 of file convertRefcatManager.py.

◆ log

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.log

Definition at line 94 of file convertRefcatManager.py.

◆ nInputFiles

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.nInputFiles

Definition at line 115 of file convertRefcatManager.py.

◆ schema

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.schema

Definition at line 90 of file convertRefcatManager.py.


The documentation for this class was generated from the following file: