|
lsst.meas.algorithms g33a09d4741+106f42d13d
|
Public Member Functions | |
| __init__ (self, filenames, config, file_reader, indexer, schema, key_map, htmRange, addRefCatMetadata, log) | |
| run (self, inputFiles) | |
| getCatalog (self, pixelId, schema, nNewElements) | |
Static Public Member Functions | |
| computeCoord (row, ra_name, dec_name) | |
Public Attributes | |
| filenames | |
| config | |
| file_reader | |
| indexer | |
| schema | |
| key_map | |
| htmRange | |
| addRefCatMetadata | |
| log | |
| coord_err_unit | |
| nInputFiles | |
Static Public Attributes | |
| ConfigClass = ConvertRefcatManagerConfig | |
Protected Member Functions | |
| _convertOneFile (self, filename, fileLocks) | |
| _doOnePixel (self, inputData, matchedPixels, pixelId, fluxes, coordErr) | |
| _setIds (self, inputData, catalog) | |
| _getCoordErr (self, inputData) | |
| _setFlags (self, record, row) | |
| _getFluxes (self, inputData) | |
| _setProperMotion (self, record, row) | |
| _setParallax (self, record, row) | |
| _epochToMjdTai (self, nativeEpoch) | |
| _setCoordinateCovariance (self, record, row) | |
| _setExtra (self, record, row) | |
| _fillRecord (self, record, row) | |
Static Protected Attributes | |
| list | _flags = ['photometric', 'resolved', 'variable'] |
| str | _DefaultName = 'convertRefcatManager' |
Convert a reference catalog from external files into the LSST HTM sharded
format, using a multiprocessing Pool to speed up the work.
Parameters
----------
filenames : `dict` [`int`, `str`]
The HTM pixel id and filenames to convert the catalog into.
config : `lsst.meas.algorithms.ConvertReferenceCatalogConfig`
The Task configuration holding the field names.
file_reader : `lsst.pipe.base.Task`
The file reader to use to load the files.
indexer : `lsst.meas.algorithms.HtmIndexer`
The class used to compute the HTM pixel per coordinate.
schema : `lsst.afw.table.Schema`
The schema of the output catalog.
key_map : `dict` [`str`, `lsst.afw.table.Key`]
The mapping from output field names to keys in the Schema.
htmRange : `tuple` [`int`]
The start and end HTM pixel ids.
addRefCatMetadata : callable
A function called to add extra metadata to each output Catalog.
log : `lsst.log.Log` or `logging.Logger`
The log to send messages to.
Definition at line 54 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.__init__ | ( | self, | |
| filenames, | |||
| config, | |||
| file_reader, | |||
| indexer, | |||
| schema, | |||
| key_map, | |||
| htmRange, | |||
| addRefCatMetadata, | |||
| log | |||
| ) |
Reimplemented in lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager.
Definition at line 84 of file convertRefcatManager.py.
|
protected |
Read and process one file, and write its records to the correct
indexed files, while handling exceptions in a useful way so that they
don't get swallowed by the multiprocess pool.
Parameters
----------
filename : `str`
The file to process.
fileLocks : `dict` [`int`, `multiprocessing.Lock`]
A Lock for each HTM pixel; each pixel gets one file written, and
we need to block when one process is accessing that file.
Returns
-------
pixels, files : `list` [`int`]
The pixel ids that were written to.
Definition at line 134 of file convertRefcatManager.py.
|
protected |
Process one HTM pixel, appending to an existing catalog or creating
a new catalog, as needed.
Parameters
----------
inputData : `numpy.ndarray`
The data from one input file.
matchedPixels : `numpy.ndarray`
The row-matched pixel indexes corresponding to ``inputData``.
pixelId : `int`
The pixel index we are currently processing.
fluxes : `dict` [`str`, `numpy.ndarray`]
The values that will go into the flux and fluxErr fields in the
output catalog.
coordErr : `dict` [`str`, `numpy.ndarray`]
The values that will go into the coord_raErr, coord_decErr, and
coord_ra_dec_Cov fields in the output catalog (in radians).
Definition at line 185 of file convertRefcatManager.py.
|
protected |
Convert an epoch in native format to TAI MJD (a float).
Definition at line 398 of file convertRefcatManager.py.
|
protected |
Fill a record in an indexed catalog to be persisted.
Parameters
----------
record : `lsst.afw.table.SimpleRecord`
Row from indexed catalog to modify.
row : structured `numpy.array`
Row from catalog being converted.
Definition at line 444 of file convertRefcatManager.py.
|
protected |
Compute the ra/dec error fields that will go into the output catalog.
Parameters
----------
inputData : `numpy.ndarray`
The input data to compute fluxes for.
Returns
-------
coordErr : `dict` [`str`, `numpy.ndarray`]
The values that will go into the coord_raErr, coord_decErr, fields
in the output catalog (in radians).
Notes
-----
This does not handle the ra/dec covariance field,
``coord_ra_coord_dec_Cov``. That field is handled in
`_setCoordinateCovariance`.
Definition at line 294 of file convertRefcatManager.py.
|
protected |
Compute the flux fields that will go into the output catalog.
Parameters
----------
inputData : `numpy.ndarray`
The input data to compute fluxes for.
Returns
-------
fluxes : `dict` [`str`, `numpy.ndarray`]
The values that will go into the flux and fluxErr fields in the
output catalog.
Reimplemented in lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager, and lsst.meas.algorithms.convertRefcatManager.ConvertGaiaXpManager.
Definition at line 338 of file convertRefcatManager.py.
|
protected |
Set the off-diagonal position covariance in a record of an indexed
catalog.
There is no generic way to determine covariance. Override this method
in a subclass specialized for your dataset.
Parameters
----------
record : `lsst.afw.table.SimpleRecord`
Row from indexed catalog to modify.
row : structured `numpy.array`
Row from catalog being converted.
Reimplemented in lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager.
Definition at line 404 of file convertRefcatManager.py.
|
protected |
Set extra data fields in a record of an indexed catalog.
Parameters
----------
record : `lsst.afw.table.SimpleRecord`
Row from indexed catalog to modify.
row : structured `numpy.array`
Row from catalog being converted.
Definition at line 421 of file convertRefcatManager.py.
|
protected |
Set flags in an output record.
Parameters
----------
record : `lsst.afw.table.SimpleRecord`
Row from indexed catalog to modify.
row : `numpy.ndarray`
Row from catalog being converted.
Definition at line 322 of file convertRefcatManager.py.
|
protected |
Fill the `id` field of catalog with a running index, filling the
last values up to the length of ``inputData``.
Fill with `self.config.id_name` if specified, otherwise use the
global running counter value.
Parameters
----------
inputData : `numpy.ndarray`
The input data that is being processed.
catalog : `lsst.afw.table.SimpleCatalog`
The output catalog to fill the ids.
Definition at line 223 of file convertRefcatManager.py.
|
protected |
Set the parallax fields in a record of a refcat.
Definition at line 389 of file convertRefcatManager.py.
|
protected |
Set proper motion fields in a record of an indexed catalog.
The proper motions are read from the specified columns,
scaled appropriately, and installed in the appropriate
columns of the output.
Parameters
----------
record : `lsst.afw.table.SimpleRecord`
Row from indexed catalog to modify.
row : structured `numpy.array`
Row from catalog being converted.
Definition at line 365 of file convertRefcatManager.py.
|
static |
Create an ICRS coord. from a row of a catalog being converted.
Parameters
----------
row : `numpy.ndarray`
Row from catalog being converted.
ra_name : `str`
Name of RA key in catalog being converted.
dec_name : `str`
Name of Dec key in catalog being converted.
Returns
-------
coord : `lsst.geom.SpherePoint`
ICRS coordinate.
Definition at line 275 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.getCatalog | ( | self, | |
| pixelId, | |||
| schema, | |||
| nNewElements | |||
| ) |
Get a catalog from disk or create it if it doesn't exist.
Parameters
----------
pixelId : `dict`
Identifier for catalog to retrieve
schema : `lsst.afw.table.Schema`
Schema to use in catalog creation it does not exist.
nNewElements : `int`
The number of new elements that will be added to the catalog,
so space can be preallocated.
Returns
-------
catalog : `lsst.afw.table.SimpleCatalog`
The new or read-and-resized catalog specified by `dataId`.
Definition at line 246 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.run | ( | self, | |
| inputFiles | |||
| ) |
Index a set of input files from a reference catalog, and write the
output to the appropriate filenames, in parallel.
Parameters
----------
inputFiles : `list`
A list of file paths to read data from.
Returns
-------
output : `dict` [`int`, `str`]
The htm ids and the filenames that were written to.
Definition at line 100 of file convertRefcatManager.py.
|
staticprotected |
Definition at line 81 of file convertRefcatManager.py.
|
staticprotected |
Definition at line 80 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.addRefCatMetadata |
Definition at line 93 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.config |
Definition at line 87 of file convertRefcatManager.py.
|
static |
Definition at line 82 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.coord_err_unit |
Definition at line 98 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.file_reader |
Definition at line 88 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.filenames |
Definition at line 86 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.htmRange |
Definition at line 92 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.indexer |
Definition at line 89 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.key_map |
Definition at line 91 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.log |
Definition at line 94 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.nInputFiles |
Definition at line 115 of file convertRefcatManager.py.
| lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.schema |
Definition at line 90 of file convertRefcatManager.py.