lsst.dax.apdb g011c388f00+c533e7e796
Loading...
Searching...
No Matches
Public Member Functions | Public Attributes | Static Public Attributes | Protected Member Functions | Protected Attributes | List of all members
lsst.dax.apdb.apdbSql.ApdbSql Class Reference
Inheritance diagram for lsst.dax.apdb.apdbSql.ApdbSql:
lsst.dax.apdb.apdb.Apdb

Public Member Functions

 __init__ (self, ApdbSqlConfig config)
 
VersionTuple apdbImplementationVersion (cls)
 
VersionTuple apdbSchemaVersion (self)
 
dict[str, int] tableRowCount (self)
 
Table|None tableDef (self, ApdbTables table)
 
None makeSchema (self, bool drop=False)
 
pandas.DataFrame getDiaObjects (self, Region region)
 
pandas.DataFrame|None getDiaSources (self, Region region, Iterable[int]|None object_ids, dafBase.DateTime visit_time)
 
pandas.DataFrame|None getDiaForcedSources (self, Region region, Iterable[int]|None object_ids, dafBase.DateTime visit_time)
 
bool containsVisitDetector (self, int visit, int detector)
 
bool containsCcdVisit (self, int ccdVisitId)
 
list[ApdbInsertId]|None getInsertIds (self)
 
None deleteInsertIds (self, Iterable[ApdbInsertId] ids)
 
ApdbTableData getDiaObjectsHistory (self, Iterable[ApdbInsertId] ids)
 
ApdbTableData getDiaSourcesHistory (self, Iterable[ApdbInsertId] ids)
 
ApdbTableData getDiaForcedSourcesHistory (self, Iterable[ApdbInsertId] ids)
 
pandas.DataFrame getSSObjects (self)
 
None store (self, dafBase.DateTime visit_time, pandas.DataFrame objects, pandas.DataFrame|None sources=None, pandas.DataFrame|None forced_sources=None)
 
None storeSSObjects (self, pandas.DataFrame objects)
 
None reassignDiaSources (self, Mapping[int, int] idMap)
 
None dailyJob (self)
 
int countUnassociatedObjects (self)
 
ApdbMetadata metadata (self)
 
- Public Member Functions inherited from lsst.dax.apdb.apdb.Apdb
ConfigurableField makeField (cls, str doc)
 

Public Attributes

 config
 
 pixelator
 
 use_insert_id
 
 metadataSchemaVersionKey
 
 metadataCodeVersionKey
 

Static Public Attributes

 ConfigClass = ApdbSqlConfig
 
str metadataSchemaVersionKey = "version:schema"
 
str metadataCodeVersionKey = "version:ApdbSql"
 
- Static Public Attributes inherited from lsst.dax.apdb.apdb.Apdb
 ConfigClass = ApdbConfig
 

Protected Member Functions

None _versionCheck (self, ApdbMetadataSql metadata)
 
ApdbTableData _get_history (self, Iterable[ApdbInsertId] ids, ApdbTables table_enum, ExtraTables history_table_enum)
 
pandas.DataFrame _getDiaSourcesInRegion (self, Region region, dafBase.DateTime visit_time)
 
pandas.DataFrame _getDiaSourcesByIDs (self, list[int] object_ids, dafBase.DateTime visit_time)
 
pandas.DataFrame _getSourcesByIDs (self, ApdbTables table_enum, list[int] object_ids, float midpointMjdTai_start)
 
None _storeInsertId (self, ApdbInsertId insert_id, dafBase.DateTime visit_time, sqlalchemy.engine.Connection connection)
 
None _storeDiaObjects (self, pandas.DataFrame objs, dafBase.DateTime visit_time, ApdbInsertId|None insert_id, sqlalchemy.engine.Connection connection)
 
None _storeDiaSources (self, pandas.DataFrame sources, ApdbInsertId|None insert_id, sqlalchemy.engine.Connection connection)
 
None _storeDiaForcedSources (self, pandas.DataFrame sources, ApdbInsertId|None insert_id, sqlalchemy.engine.Connection connection)
 
list[tuple[int, int]] _htm_indices (self, Region region)
 
sql.ColumnElement _filterRegion (self, sqlalchemy.schema.Table table, Region region)
 
pandas.DataFrame _add_obj_htm_index (self, pandas.DataFrame df)
 
pandas.DataFrame _add_src_htm_index (self, pandas.DataFrame sources, pandas.DataFrame objs)
 

Protected Attributes

 _engine
 
 _schema
 
 _metadata
 

Detailed Description

Implementation of APDB interface based on SQL database.

The implementation is configured via standard ``pex_config`` mechanism
using `ApdbSqlConfig` configuration class. For an example of different
configurations check ``config/`` folder.

Parameters
----------
config : `ApdbSqlConfig`
    Configuration object.

Member Function Documentation

◆ _add_obj_htm_index()

pandas.DataFrame lsst.dax.apdb.apdbSql.ApdbSql._add_obj_htm_index ( self,
pandas.DataFrame df )
protected
Calculate HTM index for each record and add it to a DataFrame.

Notes
-----
This overrides any existing column in a DataFrame with the same name
(pixelId). Original DataFrame is not changed, copy of a DataFrame is
returned.

◆ _add_src_htm_index()

pandas.DataFrame lsst.dax.apdb.apdbSql.ApdbSql._add_src_htm_index ( self,
pandas.DataFrame sources,
pandas.DataFrame objs )
protected
Add pixelId column to DiaSource catalog.

Notes
-----
This method copies pixelId value from a matching DiaObject record.
DiaObject catalog needs to have a pixelId column filled by
``_add_obj_htm_index`` method and DiaSource records need to be
associated to DiaObjects via ``diaObjectId`` column.

This overrides any existing column in a DataFrame with the same name
(pixelId). Original DataFrame is not changed, copy of a DataFrame is
returned.

◆ _filterRegion()

sql.ColumnElement lsst.dax.apdb.apdbSql.ApdbSql._filterRegion ( self,
sqlalchemy.schema.Table table,
Region region )
protected
Make SQLAlchemy expression for selecting records in a region.

◆ _get_history()

ApdbTableData lsst.dax.apdb.apdbSql.ApdbSql._get_history ( self,
Iterable[ApdbInsertId] ids,
ApdbTables table_enum,
ExtraTables history_table_enum )
protected
Return catalog of records for given insert identifiers, common
implementation for all DIA tables.

◆ _getDiaSourcesByIDs()

pandas.DataFrame lsst.dax.apdb.apdbSql.ApdbSql._getDiaSourcesByIDs ( self,
list[int] object_ids,
dafBase.DateTime visit_time )
protected
Return catalog of DiaSource instances given set of DiaObject IDs.

Parameters
----------
object_ids :
    Collection of DiaObject IDs
visit_time : `lsst.daf.base.DateTime`
    Time of the current visit.

Returns
-------
catalog : `pandas.DataFrame`
    Catalog contaning DiaSource records.

◆ _getDiaSourcesInRegion()

pandas.DataFrame lsst.dax.apdb.apdbSql.ApdbSql._getDiaSourcesInRegion ( self,
Region region,
dafBase.DateTime visit_time )
protected
Return catalog of DiaSource instances from given region.

Parameters
----------
region : `lsst.sphgeom.Region`
    Region to search for DIASources.
visit_time : `lsst.daf.base.DateTime`
    Time of the current visit.

Returns
-------
catalog : `pandas.DataFrame`
    Catalog containing DiaSource records.

◆ _getSourcesByIDs()

pandas.DataFrame lsst.dax.apdb.apdbSql.ApdbSql._getSourcesByIDs ( self,
ApdbTables table_enum,
list[int] object_ids,
float midpointMjdTai_start )
protected
Return catalog of DiaSource or DiaForcedSource instances given set
of DiaObject IDs.

Parameters
----------
table : `sqlalchemy.schema.Table`
    Database table.
object_ids :
    Collection of DiaObject IDs
midpointMjdTai_start : `float`
    Earliest midpointMjdTai to retrieve.

Returns
-------
catalog : `pandas.DataFrame`
    Catalog contaning DiaSource records. `None` is returned if
    ``read_sources_months`` configuration parameter is set to 0 or
    when ``object_ids`` is empty.

◆ _htm_indices()

list[tuple[int, int]] lsst.dax.apdb.apdbSql.ApdbSql._htm_indices ( self,
Region region )
protected
Generate a set of HTM indices covering specified region.

Parameters
----------
region: `sphgeom.Region`
    Region that needs to be indexed.

Returns
-------
Sequence of ranges, range is a tuple (minHtmID, maxHtmID).

◆ _storeDiaForcedSources()

None lsst.dax.apdb.apdbSql.ApdbSql._storeDiaForcedSources ( self,
pandas.DataFrame sources,
ApdbInsertId | None insert_id,
sqlalchemy.engine.Connection connection )
protected
Store a set of DiaForcedSources from current visit.

Parameters
----------
sources : `pandas.DataFrame`
    Catalog containing DiaForcedSource records

◆ _storeDiaObjects()

None lsst.dax.apdb.apdbSql.ApdbSql._storeDiaObjects ( self,
pandas.DataFrame objs,
dafBase.DateTime visit_time,
ApdbInsertId | None insert_id,
sqlalchemy.engine.Connection connection )
protected
Store catalog of DiaObjects from current visit.

Parameters
----------
objs : `pandas.DataFrame`
    Catalog with DiaObject records.
visit_time : `lsst.daf.base.DateTime`
    Time of the visit.
insert_id : `ApdbInsertId`
    Insert identifier.

◆ _storeDiaSources()

None lsst.dax.apdb.apdbSql.ApdbSql._storeDiaSources ( self,
pandas.DataFrame sources,
ApdbInsertId | None insert_id,
sqlalchemy.engine.Connection connection )
protected
Store catalog of DiaSources from current visit.

Parameters
----------
sources : `pandas.DataFrame`
    Catalog containing DiaSource records

◆ _versionCheck()

None lsst.dax.apdb.apdbSql.ApdbSql._versionCheck ( self,
ApdbMetadataSql metadata )
protected
Check schema version compatibility.

◆ apdbImplementationVersion()

VersionTuple lsst.dax.apdb.apdbSql.ApdbSql.apdbImplementationVersion ( cls)
Return version number for current APDB implementation.

Returns
-------
version : `VersionTuple`
    Version of the code defined in implementation class.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ apdbSchemaVersion()

VersionTuple lsst.dax.apdb.apdbSql.ApdbSql.apdbSchemaVersion ( self)
Return schema version number as defined in config file.

Returns
-------
version : `VersionTuple`
    Version of the schema defined in schema config file.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ containsCcdVisit()

bool lsst.dax.apdb.apdbSql.ApdbSql.containsCcdVisit ( self,
int ccdVisitId )
Test whether data for a given visit-detector is present in the APDB.

This method is a placeholder until `Apdb.containsVisitDetector` can
be implemented.

Parameters
----------
ccdVisitId : `int`
    The packed ID of the visit-detector to search for.

Returns
-------
present : `bool`
    `True` if some DiaSource records exist for the specified
    observation, `False` otherwise.

◆ containsVisitDetector()

bool lsst.dax.apdb.apdbSql.ApdbSql.containsVisitDetector ( self,
int visit,
int detector )
Test whether data for a given visit-detector is present in the APDB.

Parameters
----------
visit, detector : `int`
    The ID of the visit-detector to search for.

Returns
-------
present : `bool`
    `True` if some DiaObject, DiaSource, or DiaForcedSource records
    exist for the specified observation, `False` otherwise.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ countUnassociatedObjects()

int lsst.dax.apdb.apdbSql.ApdbSql.countUnassociatedObjects ( self)
Return the number of DiaObjects that have only one DiaSource
associated with them.

Used as part of ap_verify metrics.

Returns
-------
count : `int`
    Number of DiaObjects with exactly one associated DiaSource.

Notes
-----
This method can be very inefficient or slow in some implementations.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ dailyJob()

None lsst.dax.apdb.apdbSql.ApdbSql.dailyJob ( self)
Implement daily activities like cleanup/vacuum.

What should be done during daily activities is determined by
specific implementation.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ deleteInsertIds()

None lsst.dax.apdb.apdbSql.ApdbSql.deleteInsertIds ( self,
Iterable[ApdbInsertId] ids )
Remove insert identifiers from the database.

Parameters
----------
ids : `iterable` [`ApdbInsertId`]
    Insert identifiers, can include items returned from `getInsertIds`.

Notes
-----
This method causes Apdb to forget about specified identifiers. If there
are any auxiliary data associated with the identifiers, it is also
removed from database (but data in regular tables is not removed).
This method should be called after successful transfer of data from
APDB to PPDB to free space used by history.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ getDiaForcedSources()

pandas.DataFrame | None lsst.dax.apdb.apdbSql.ApdbSql.getDiaForcedSources ( self,
Region region,
Iterable[int] | None object_ids,
dafBase.DateTime visit_time )
Return catalog of DiaForcedSource instances from a given region.

Parameters
----------
region : `lsst.sphgeom.Region`
    Region to search for DIASources.
object_ids : iterable [ `int` ], optional
    List of DiaObject IDs to further constrain the set of returned
    sources. If list is empty then empty catalog is returned with a
    correct schema. If `None` then returned sources are not
    constrained. Some implementations may not support latter case.
visit_time : `lsst.daf.base.DateTime`
    Time of the current visit.

Returns
-------
catalog : `pandas.DataFrame`, or `None`
    Catalog containing DiaSource records. `None` is returned if
    ``read_forced_sources_months`` configuration parameter is set to 0.

Raises
------
NotImplementedError
    May be raised by some implementations if ``object_ids`` is `None`.

Notes
-----
This method returns DiaForcedSource catalog for a region with
additional filtering based on DiaObject IDs. Only a subset of DiaSource
history is returned limited by ``read_forced_sources_months`` config
parameter, w.r.t. ``visit_time``. If ``object_ids`` is empty then an
empty catalog is always returned with the correct schema
(columns/types). If ``object_ids`` is `None` then no filtering is
performed and some of the returned records may be outside the specified
region.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ getDiaForcedSourcesHistory()

ApdbTableData lsst.dax.apdb.apdbSql.ApdbSql.getDiaForcedSourcesHistory ( self,
Iterable[ApdbInsertId] ids )
Return catalog of DiaForcedSource instances from a given time
period.

Parameters
----------
ids : `iterable` [`ApdbInsertId`]
    Insert identifiers, can include items returned from `getInsertIds`.

Returns
-------
data : `ApdbTableData`
    Catalog containing DiaForcedSource records. In addition to all
    regular columns it will contain ``insert_id`` column.

Notes
-----
This part of API may not be very stable and can change before the
implementation finalizes.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ getDiaObjects()

pandas.DataFrame lsst.dax.apdb.apdbSql.ApdbSql.getDiaObjects ( self,
Region region )
Return catalog of DiaObject instances from a given region.

This method returns only the last version of each DiaObject. Some
records in a returned catalog may be outside the specified region, it
is up to a client to ignore those records or cleanup the catalog before
futher use.

Parameters
----------
region : `lsst.sphgeom.Region`
    Region to search for DIAObjects.

Returns
-------
catalog : `pandas.DataFrame`
    Catalog containing DiaObject records for a region that may be a
    superset of the specified region.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ getDiaObjectsHistory()

ApdbTableData lsst.dax.apdb.apdbSql.ApdbSql.getDiaObjectsHistory ( self,
Iterable[ApdbInsertId] ids )
Return catalog of DiaObject instances from a given time period
including the history of each DiaObject.

Parameters
----------
ids : `iterable` [`ApdbInsertId`]
    Insert identifiers, can include items returned from `getInsertIds`.

Returns
-------
data : `ApdbTableData`
    Catalog containing DiaObject records. In addition to all regular
    columns it will contain ``insert_id`` column.

Notes
-----
This part of API may not be very stable and can change before the
implementation finalizes.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ getDiaSources()

pandas.DataFrame | None lsst.dax.apdb.apdbSql.ApdbSql.getDiaSources ( self,
Region region,
Iterable[int] | None object_ids,
dafBase.DateTime visit_time )
Return catalog of DiaSource instances from a given region.

Parameters
----------
region : `lsst.sphgeom.Region`
    Region to search for DIASources.
object_ids : iterable [ `int` ], optional
    List of DiaObject IDs to further constrain the set of returned
    sources. If `None` then returned sources are not constrained. If
    list is empty then empty catalog is returned with a correct
    schema.
visit_time : `lsst.daf.base.DateTime`
    Time of the current visit.

Returns
-------
catalog : `pandas.DataFrame`, or `None`
    Catalog containing DiaSource records. `None` is returned if
    ``read_sources_months`` configuration parameter is set to 0.

Notes
-----
This method returns DiaSource catalog for a region with additional
filtering based on DiaObject IDs. Only a subset of DiaSource history
is returned limited by ``read_sources_months`` config parameter, w.r.t.
``visit_time``. If ``object_ids`` is empty then an empty catalog is
always returned with the correct schema (columns/types). If
``object_ids`` is `None` then no filtering is performed and some of the
returned records may be outside the specified region.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ getDiaSourcesHistory()

ApdbTableData lsst.dax.apdb.apdbSql.ApdbSql.getDiaSourcesHistory ( self,
Iterable[ApdbInsertId] ids )
Return catalog of DiaSource instances from a given time period.

Parameters
----------
ids : `iterable` [`ApdbInsertId`]
    Insert identifiers, can include items returned from `getInsertIds`.

Returns
-------
data : `ApdbTableData`
    Catalog containing DiaSource records. In addition to all regular
    columns it will contain ``insert_id`` column.

Notes
-----
This part of API may not be very stable and can change before the
implementation finalizes.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ getInsertIds()

list[ApdbInsertId] | None lsst.dax.apdb.apdbSql.ApdbSql.getInsertIds ( self)
Return collection of insert identifiers known to the database.

Returns
-------
ids : `list` [`ApdbInsertId`] or `None`
    List of identifiers, they may be time-ordered if database supports
    ordering. `None` is returned if database is not configured to store
    insert identifiers.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ getSSObjects()

pandas.DataFrame lsst.dax.apdb.apdbSql.ApdbSql.getSSObjects ( self)
Return catalog of SSObject instances.

Returns
-------
catalog : `pandas.DataFrame`
    Catalog containing SSObject records, all existing records are
    returned.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ makeSchema()

None lsst.dax.apdb.apdbSql.ApdbSql.makeSchema ( self,
bool drop = False )
Create or re-create whole database schema.

Parameters
----------
drop : `bool`
    If True then drop all tables before creating new ones.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ metadata()

ApdbMetadata lsst.dax.apdb.apdbSql.ApdbSql.metadata ( self)
Object controlling access to APDB metadata (`ApdbMetadata`).

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ reassignDiaSources()

None lsst.dax.apdb.apdbSql.ApdbSql.reassignDiaSources ( self,
Mapping[int, int] idMap )
Associate DiaSources with SSObjects, dis-associating them
from DiaObjects.

Parameters
----------
idMap : `Mapping`
    Maps DiaSource IDs to their new SSObject IDs.

Raises
------
ValueError
    Raised if DiaSource ID does not exist in the database.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ store()

None lsst.dax.apdb.apdbSql.ApdbSql.store ( self,
dafBase.DateTime visit_time,
pandas.DataFrame objects,
pandas.DataFrame | None sources = None,
pandas.DataFrame | None forced_sources = None )
Store all three types of catalogs in the database.

Parameters
----------
visit_time : `lsst.daf.base.DateTime`
    Time of the visit.
objects : `pandas.DataFrame`
    Catalog with DiaObject records.
sources : `pandas.DataFrame`, optional
    Catalog with DiaSource records.
forced_sources : `pandas.DataFrame`, optional
    Catalog with DiaForcedSource records.

Notes
-----
This methods takes DataFrame catalogs, their schema must be
compatible with the schema of APDB table:

  - column names must correspond to database table columns
  - types and units of the columns must match database definitions,
    no unit conversion is performed presently
  - columns that have default values in database schema can be
    omitted from catalog
  - this method knows how to fill interval-related columns of DiaObject
    (validityStart, validityEnd) they do not need to appear in a
    catalog
  - source catalogs have ``diaObjectId`` column associating sources
    with objects

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ storeSSObjects()

None lsst.dax.apdb.apdbSql.ApdbSql.storeSSObjects ( self,
pandas.DataFrame objects )
Store or update SSObject catalog.

Parameters
----------
objects : `pandas.DataFrame`
    Catalog with SSObject records.

Notes
-----
If SSObjects with matching IDs already exist in the database, their
records will be updated with the information from provided records.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ tableDef()

Table | None lsst.dax.apdb.apdbSql.ApdbSql.tableDef ( self,
ApdbTables table )
Return table schema definition for a given table.

Parameters
----------
table : `ApdbTables`
    One of the known APDB tables.

Returns
-------
tableSchema : `felis.simple.Table` or `None`
    Table schema description, `None` is returned if table is not
    defined by this implementation.

Reimplemented from lsst.dax.apdb.apdb.Apdb.

◆ tableRowCount()

dict[str, int] lsst.dax.apdb.apdbSql.ApdbSql.tableRowCount ( self)
Return dictionary with the table names and row counts.

Used by ``ap_proto`` to keep track of the size of the database tables.
Depending on database technology this could be expensive operation.

Returns
-------
row_counts : `dict`
    Dict where key is a table name and value is a row count.

The documentation for this class was generated from the following file: