Coverage for python/lsst/daf/butler/registries/sqlRegistry.py : 12%

Hot-keys on this page
r m x p toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
# This file is part of daf_butler. # # Developed for the LSST Data Management System. # This product includes software developed by the LSST Project # (http://www.lsst.org). # See the COPYRIGHT file at the top-level directory of this distribution # for details of code ownership. # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>.
AmbiguousDatasetError, OrphanedRecordError) DataId, DimensionRecord, Dimension) DatasetTypeExpression, CollectionsExpression)
"""Registry backed by a SQL database.
Parameters ---------- registryConfig : `SqlRegistryConfig` or `str` Load configuration schemaConfig : `SchemaConfig` or `str` Definition of the schema to use. dimensionConfig : `DimensionConfig` or `Config` or `DimensionGraph` configuration. create : `bool` Assume registry is empty and create a new one. """
"""Path to configuration defaults. Relative to $DAF_BUTLER_DIR/config or absolute path. Can be None if no defaults specified. """
registryConfig = SqlRegistryConfig(registryConfig) super().__init__(registryConfig, dimensionConfig=dimensionConfig) self.storageClasses = StorageClassFactory() # Build schema for dimensions. schemaSpec = self.dimensions.makeSchemaSpec() # Update with schema directly loaded from config. schemaSpec.update(schemaConfig.toSpec()) # Add dimension columns and foreign keys to the dataset table. datasetTableSpec = schemaSpec["dataset"] for dimension in self.dimensions.dimensions: addDimensionForeignKey(datasetTableSpec, dimension, primaryKey=False, nullable=True) # Translate the schema specification to SQLALchemy, allowing subclasses # to specialize. self._schema = self._createSchema(schemaSpec) self._datasetTypes = {} self._engine = self._createEngine() self._connection = self._createConnection(self._engine) self._runIdsByName = {} # key = name, value = id self._runNamesById = {} # key = id, value = name self._dimensionStorage = setupDimensionStorage(connection=self._connection, universe=self.dimensions, tables=self._schema.tables) self._datasetStorage = DatasetRegistryStorage(connection=self._connection, universe=self.dimensions, tables=self._schema.tables) if create: # In our tables we have columns that make use of sqlalchemy # Sequence objects. There is currently a bug in sqlalchmey # that causes a deprecation warning to be thrown on a # property of the Sequence object when the repr for the # sequence is created. Here a filter is used to catch these # deprecation warnings when tables are created. with warnings.catch_warnings(): warnings.simplefilter("ignore", category=SADeprecationWarning) self._createTables(self._schema, self._connection)
return self.config["db"]
def transaction(self): """Context manager that implements SQL transactions.
Will roll back any changes to the `SqlRegistry` database in case an exception is raised in the enclosed block.
This context manager may be nested. """ trans = self._connection.begin_nested() try: yield trans.commit() except BaseException: trans.rollback() for storage in self._dimensionStorage.values(): storage.clearCaches() raise
"""Create and return an `lsst.daf.butler.Schema` object containing SQLAlchemy table definitions.
This is a hook provided for customization by subclasses, but it is known to be insufficient for that purpose and is expected to change in the future.
Note that this method should not actually create any tables or views in the database - it is called even when an existing database is used in order to construct the SQLAlchemy representation of the expected schema.
Parameters ---------- spec : `dict` mapping `str` to `TableSpec` Specification of the logical tables to be created.
Returns ------- schema : `Schema` Structure containing SQLAlchemy objects representing the tables and views in the registry schema. """ return Schema(spec=spec)
"""Create and return a `sqlalchemy.Engine` for this `Registry`.
This is a hook provided for customization by subclasses.
SQLAlchemy generally expects engines to be created at module scope, with a pool of connections used by different parts of an application. Because our `Registry` instances don't know what database they'll connect to until they are constructed, that is impossible for us, so the engine is connected with the `Registry` instance. In addition, we do not expect concurrent usage of the same `Registry`, and hence don't gain anything from connection pooling. As a result, the default implementation of this function uses `sqlalchemy.pool.NullPool` to associate just a single connection with the engine. Unless they have a very good reason not to, subclasses that override this method should do the same. """ return create_engine(self.config.connectionString, poolclass=NullPool)
"""Create and return a `sqlalchemy.Connection` for this `Registry`.
This is a hook provided for customization by subclasses. """ return engine.connect()
"""Create all tables in the given schema, using the given connection.
This is a hook provided for customization by subclasses. """ schema.metadata.create_all(connection)
"""Check if given `DatasetType` instance is valid for this `Registry`.
.. todo::
Insert checks for `storageClass`, `dimensions` and `template`. """ return isinstance(datasetType, DatasetType)
# Docstring inherited from Registry.registerOpaqueTable. table = spec.toSqlAlchemy(name, self._schema) table.create(self._connection, checkfirst=True)
# Docstring inherited from Registry.insertOpaqueData. table = self._schema.tables[name] self._connection.execute(table.insert(), *data)
# Docstring inherited from Registry.insertExernalData. table = self._schema.tables[name] query = table.select() whereTerms = [table.columns[k] == v for k, v in where.items()] if whereTerms: query = query.where(and_(*whereTerms)) yield from self._connection.execute(query)
# Docstring inherited from Registry.deleteOpaqueData. table = self._schema.tables[name] query = table.delete() whereTerms = [table.columns[k] == v for k, v in where.items()] if whereTerms: query = query.where(and_(*whereTerms)) self._connection.execute(query)
"""Return the name of the run associated with the given integer ID. """ assert isinstance(id, int) name = self._runNamesById.get(id) if name is None: runTable = self._schema.tables["run"] name = self._connection.execute( select([runTable.columns.name]).select_from(runTable).where(runTable.columns.id == id) ).scalar() self._runNamesById[id] = name self._runIdsByName[name] = id return name
"""Return the integer ID of the run associated with the given name. """ assert isinstance(name, str) id = self._runIdsByName.get(name) if id is None: runTable = self._schema.tables["run"] id = self._connection.execute( select([runTable.columns.id]).select_from(runTable).where(runTable.columns.name == name) ).scalar() self._runNamesById[id] = name self._runIdsByName[name] = id return id
"""Construct a DatasetRef from the result of a query on the Dataset table.
Parameters ---------- row : `sqlalchemy.engine.RowProxy`. Row of a query that contains all columns from the `Dataset` table. May include additional fields (which will be ignored). datasetType : `DatasetType`, optional `DatasetType` associated with this dataset. Will be retrieved if not provided. If provided, the caller guarantees that it is already consistent with what would have been retrieved from the database. dataId : `DataCoordinate`, optional Dimensions associated with this dataset. Will be retrieved if not provided. If provided, the caller guarantees that it is already consistent with what would have been retrieved from the database.
Returns ------- ref : `DatasetRef`. A new `DatasetRef` instance. """ if datasetType is None: datasetType = self.getDatasetType(row["dataset_type_name"]) run = self._getRunNameFromId(row["run_id"]) datasetRefHash = row["dataset_ref_hash"] if dataId is None: # TODO: should we expand here? dataId = DataCoordinate.standardize( row, graph=datasetType.dimensions, universe=self.dimensions ) # Get components (if present) components = {} if datasetType.storageClass.isComposite(): datasetCompositionTable = self._schema.tables["dataset_composition"] datasetTable = self._schema.tables["dataset"] columns = list(datasetTable.c) columns.append(datasetCompositionTable.c.component_name) results = self._connection.execute( select( columns ).select_from( datasetTable.join( datasetCompositionTable, datasetTable.c.dataset_id == datasetCompositionTable.c.component_dataset_id ) ).where( datasetCompositionTable.c.parent_dataset_id == row["dataset_id"] ) ).fetchall() for result in results: componentName = result["component_name"] componentDatasetType = DatasetType( DatasetType.nameWithComponent(datasetType.name, componentName), dimensions=datasetType.dimensions, storageClass=datasetType.storageClass.components[componentName] ) components[componentName] = self._makeDatasetRefFromRow(result, dataId=dataId, datasetType=componentDatasetType) if not components.keys() <= datasetType.storageClass.components.keys(): raise RuntimeError( f"Inconsistency detected between dataset and storage class definitions: " f"{datasetType.storageClass.name} has components " f"{set(datasetType.storageClass.components.keys())}, " f"but dataset has components {set(components.keys())}" ) return DatasetRef(datasetType=datasetType, dataId=dataId, id=row["dataset_id"], run=run, hash=datasetRefHash, components=components)
# Docstring inherited from Registry.getAllCollections datasetCollectionTable = self._schema.tables["dataset_collection"] result = self._connection.execute(select([datasetCollectionTable.c.collection]).distinct()).fetchall() if result is None: return set() return {r[0] for r in result}
# Docstring inherited from Registry.find if not isinstance(datasetType, DatasetType): datasetType = self.getDatasetType(datasetType) dataId = DataCoordinate.standardize(dataId, graph=datasetType.dimensions, universe=self.dimensions, **kwds) datasetTable = self._schema.tables["dataset"] datasetCollectionTable = self._schema.tables["dataset_collection"] dataIdExpression = and_(self._schema.tables["dataset"].c[name] == dataId[name] for name in dataId.keys()) result = self._connection.execute( datasetTable.select().select_from( datasetTable.join(datasetCollectionTable) ).where( and_( datasetTable.c.dataset_type_name == datasetType.name, datasetCollectionTable.c.collection == collection, dataIdExpression ) ) ).fetchone() if result is None: return None return self._makeDatasetRefFromRow(result, datasetType=datasetType, dataId=dataId)
"""Execute a SQL SELECT statement directly.
Named parameters are specified in the SQL query string by preceeding them with a colon. Parameter values are provided as additional keyword arguments. For example:
registry.query("SELECT * FROM instrument WHERE instrument=:name", name="HSC")
Parameters ---------- sql : `str` SQL query string. Must be a SELECT statement. **params Parameter name-value pairs to insert into the query.
Yields ------- row : `dict` The next row result from executing the query.
""" # TODO: make this guard against non-SELECT queries. t = text(sql) for row in self._connection.execute(t, **params): yield dict(row)
def registerDatasetType(self, datasetType): # Docstring inherited from Registry.getDatasetType. # If the DatasetType is already in the cache, we assume it's already in # the DB (note that we don't actually provide a way to remove them from # the DB). existingDatasetType = self._datasetTypes.get(datasetType.name, None) # If it's not in the cache, try to insert it. if existingDatasetType is None: try: with self.transaction(): self._connection.execute( self._schema.tables["dataset_type"].insert().values( dataset_type_name=datasetType.name, storage_class=datasetType.storageClass.name ) ) except IntegrityError: # Insert failed on the only unique constraint on this table: # dataset_type_name. So now the question is whether the one in # there is the same as the one we tried to insert. existingDatasetType = self.getDatasetType(datasetType.name) else: # If adding the DatasetType record itself succeeded, add its # dimensions (if any). We don't guard this in a try block # because a problem with this insert means the database # content must be corrupted. if datasetType.dimensions: self._connection.execute( self._schema.tables["dataset_type_dimensions"].insert(), [{"dataset_type_name": datasetType.name, "dimension_name": dimensionName} for dimensionName in datasetType.dimensions.names] ) # Also register component DatasetTypes (if any). for compName, compStorageClass in datasetType.storageClass.components.items(): compType = DatasetType(datasetType.componentTypeName(compName), dimensions=datasetType.dimensions, storageClass=compStorageClass) self.registerDatasetType(compType) # Inserts succeeded, nothing left to do here. return True # A DatasetType with this name exists, check if is equal if datasetType == existingDatasetType: return False else: raise ConflictingDefinitionError(f"DatasetType: {datasetType} != existing {existingDatasetType}")
# Docstring inherited from Registry.getAllDatasetTypes. datasetTypeTable = self._schema.tables["dataset_type"]
# Get all the registered names result = self._connection.execute(select([datasetTypeTable.c.dataset_type_name])).fetchall() if result is None: return frozenset()
datasetTypeNames = [r[0] for r in result] return frozenset(self.getDatasetType(name) for name in datasetTypeNames)
# Docstring inherited from Registry.getDatasetType. datasetTypeTable = self._schema.tables["dataset_type"] datasetTypeDimensionsTable = self._schema.tables["dataset_type_dimensions"] # Get StorageClass from DatasetType table result = self._connection.execute(select([datasetTypeTable.c.storage_class]).where( datasetTypeTable.c.dataset_type_name == name)).fetchone()
if result is None: raise KeyError("Could not find entry for datasetType {}".format(name))
storageClass = self.storageClasses.getStorageClass(result["storage_class"]) # Get Dimensions (if any) from DatasetTypeDimensions table result = self._connection.execute(select([datasetTypeDimensionsTable.c.dimension_name]).where( datasetTypeDimensionsTable.c.dataset_type_name == name)).fetchall() dimensions = DimensionGraph(self.dimensions, names=(r[0] for r in result) if result else ()) datasetType = DatasetType(name=name, storageClass=storageClass, dimensions=dimensions) return datasetType
# Docstring inherited from Registry.addDataset
if not isinstance(datasetType, DatasetType): datasetType = self.getDatasetType(datasetType)
# Make an expanded, standardized data ID up front, so we don't do that # multiple times in calls below. Note that calling expandDataId with a # full ExpandedDataCoordinate is basically a no-op. dataId = self.expandDataId(dataId, graph=datasetType.dimensions, **kwds)
runId = self._getRunIdFromName(run)
# Add the Dataset table entry itself. Note that this will get rolled # back if the subsequent call to associate raises, which is what we # want. datasetTable = self._schema.tables["dataset"] datasetRef = DatasetRef(datasetType=datasetType, dataId=dataId) # TODO add producer row = {k.name: v for k, v in dataId.full.items()} row.update( dataset_type_name=datasetType.name, run_id=runId, dataset_ref_hash=datasetRef.hash, quantum_id=None ) result = self._connection.execute(datasetTable.insert(), row) datasetId = result.inserted_primary_key[0] # If the result is reported as a list of a number, unpack the list if isinstance(datasetId, list): datasetId = datasetId[0] datasetRef = datasetRef.resolved(id=datasetId, run=run)
# A dataset is always initially associated with its run as a # collection. self.associate(run, [datasetRef, ])
if recursive: for component in datasetType.storageClass.components: compTypeName = datasetType.componentTypeName(component) compDatasetType = self.getDatasetType(compTypeName) compRef = self.addDataset(compDatasetType, dataId, run=run, producer=producer, recursive=True) self.attachComponent(component, datasetRef, compRef) return datasetRef
# Docstring inherited from Registry.getDataset datasetTable = self._schema.tables["dataset"] result = self._connection.execute( select([datasetTable]).where(datasetTable.c.dataset_id == id)).fetchone() if result is None: return None return self._makeDatasetRefFromRow(result, datasetType=datasetType, dataId=dataId)
def removeDataset(self, ref): # Docstring inherited from Registry.removeDataset. if not ref.id: raise AmbiguousDatasetError(f"Cannot remove dataset {ref} without ID.")
# Remove component datasets. We assume ``ref.components`` is already # correctly populated, and rely on ON DELETE CASCADE to remove entries # from DatasetComposition. for componentRef in ref.components.values(): self.removeDataset(componentRef)
datasetTable = self._schema.tables["dataset"]
# Remove related quanta. We rely on ON DELETE CASCADE to remove any # related records in DatasetConsumers. Note that we permit a Quantum # to be deleted without removing the Datasets it refers to, but do not # allow a Dataset to be deleted without removing the Quanta that refer # to them. A Dataset is still quite usable without provenance, but # provenance is worthless if it's inaccurate. quantumTable = self._schema.tables["quantum"] datasetConsumersTable = self._schema.tables["dataset_consumers"] selectProducer = select( [datasetTable.c.quantum_id] ).where( datasetTable.c.dataset_id == ref.id ) selectConsumers = select( [datasetConsumersTable.c.quantum_id] ).where( datasetConsumersTable.c.dataset_id == ref.id ) self._connection.execute( quantumTable.delete().where( quantumTable.c.id.in_(union(selectProducer, selectConsumers)) ) )
# Remove the Dataset record itself. We rely on ON DELETE CASCADE to # remove from DatasetCollection, and assume foreign key violations # come from DatasetLocation (everything else should have an ON DELETE). try: self._connection.execute( datasetTable.delete().where(datasetTable.c.dataset_id == ref.id) ) except IntegrityError as err: raise OrphanedRecordError(f"Dataset {ref} is still present in one or more Datastores.") from err
def attachComponent(self, name, parent, component): # Docstring inherited from Registry.attachComponent. # TODO Insert check for component name and type against # parent.storageClass specified components if parent.id is None: raise AmbiguousDatasetError(f"Cannot attach component to dataset {parent} without ID.") if component.id is None: raise AmbiguousDatasetError(f"Cannot attach component {component} without ID.") datasetCompositionTable = self._schema.tables["dataset_composition"] values = dict(component_name=name, parent_dataset_id=parent.id, component_dataset_id=component.id) self._connection.execute(datasetCompositionTable.insert().values(**values)) parent._components[name] = component
# Docstring inherited from Registry.associate.
def records(refs): """Generate records to insert into database.
Parameters ---------- refs : iterable of `DatasetRef` An iterable of `DatasetRef` instances. """ for ref in refs: if ref.id is None: raise AmbiguousDatasetError(f"Cannot associate dataset {ref} without ID.") yield {"dataset_id": ref.id, "dataset_ref_hash": ref.hash, "collection": collection} yield from records(ref.components.values())
datasetCollectionTable = self._schema.tables["dataset_collection"] values = list(records(refs)) try: self._insert(datasetCollectionTable, values, onConflict="ignore") except IntegrityError as exc: raise ConflictingDefinitionError(f"A dataset already exists in collection {collection}") from exc
def disassociate(self, collection, refs): # Docstring inherited from Registry.disassociate. datasetCollectionTable = self._schema.tables["dataset_collection"] for ref in refs: if ref.id is None: raise AmbiguousDatasetError(f"Cannot disassociate dataset {ref} without ID.") self.disassociate(collection, ref.components.values()) self._connection.execute(datasetCollectionTable.delete().where( and_(datasetCollectionTable.c.dataset_id == ref.id, datasetCollectionTable.c.collection == collection)))
def addDatasetLocation(self, ref, datastoreName): # Docstring inherited from Registry.addDatasetLocation. if ref.id is None: raise AmbiguousDatasetError(f"Cannot add location for dataset {ref} without ID.") datasetStorageTable = self._schema.tables["dataset_storage"] values = dict(dataset_id=ref.id, datastore_name=datastoreName) self._connection.execute(datasetStorageTable.insert().values(**values))
# Docstring inherited from Registry.getDatasetLocation. if ref.id is None: raise AmbiguousDatasetError(f"Cannot add location for dataset {ref} without ID.") datasetStorageTable = self._schema.tables["dataset_storage"] result = self._connection.execute( select([datasetStorageTable.c.datastore_name]).where( and_(datasetStorageTable.c.dataset_id == ref.id))).fetchall()
return {r["datastore_name"] for r in result}
def removeDatasetLocation(self, datastoreName, ref): # Docstring inherited from Registry.getDatasetLocation. datasetStorageTable = self._schema.tables["dataset_storage"] self._connection.execute(datasetStorageTable.delete().where( and_(datasetStorageTable.c.dataset_id == ref.id, datasetStorageTable.c.datastore_name == datastoreName)))
# Docstring inherited from Registry.registerRun. runTable = self._schema.tables["run"] try: with self.transaction(): id = self._connection.execute(runTable.insert(), {"name": name}).inserted_primary_key[0] # No exception means we inserted a new run. Remember its ID. self._runIdsByName[name] = id self._runNamesById[id] = name return except IntegrityError: # Assume this means the run already existed pass id = self._connection.execute( select([runTable.columns.id]).select_from(runTable).where(runTable.columns.name == name) ).scalar() self._runIdsByName[name] = id self._runNamesById[id] = name
records: Optional[Mapping[DimensionElement, DimensionRecord]] = None, **kwds): # Docstring inherited from Registry.expandDataId. standardized = DataCoordinate.standardize(dataId, graph=graph, universe=self.dimensions, **kwds) if isinstance(standardized, ExpandedDataCoordinate): return standardized elif isinstance(dataId, ExpandedDataCoordinate): records = dict(records) if records is not None else {} records.update(dataId.records) else: records = dict(records) if records is not None else {} keys = dict(standardized) for element in standardized.graph._primaryKeyTraversalOrder: record = records.get(element.name, ...) # Use ... to mean not found; None might mean NULL if record is ...: storage = self._dimensionStorage[element] record = storage.fetch(keys) records[element] = record if record is not None: keys.update((d, getattr(record, d.name)) for d in element.implied) else: if element in standardized.graph.required: raise LookupError( f"Could not fetch record for required dimension {element.name} via keys {keys}." ) records.update((d, None) for d in element.implied) return ExpandedDataCoordinate(standardized.graph, standardized.values(), records=records)
*data: Union[dict, DimensionRecord], conform: bool = True): # Docstring inherited from Registry.insertDimensionData. if conform: element = self.dimensions[element] # if this is a name, convert it to a true DimensionElement. records = [element.RecordClass.fromDict(row) if not type(row) is element.RecordClass else row for row in data] else: records = data storage = self._dimensionStorage[element] storage.insert(*records)
"""Return a `QueryBuilder` instance capable of constructing and managing more complex queries than those obtainable via `Registry` interfaces.
This is an advanced `SqlRegistry`-only interface; downstream code should prefer `Registry.queryDimensions` and `Registry.queryDatasets` whenever those are sufficient.
Parameters ---------- summary: `QuerySummary` Object describing and categorizing the full set of dimensions that will be included in the query.
Returns ------- builder : `QueryBuilder` Object that can be used to construct and perform advanced queries. """ return QueryBuilder(connection=self._connection, summary=summary, dimensionStorage=self._dimensionStorage, datasetStorage=self._datasetStorage)
dataId: Optional[DataId] = None, datasets: Optional[Mapping[DatasetTypeExpression, CollectionsExpression]] = None, where: Optional[str] = None, expand: bool = True, **kwds) -> Iterator[DataCoordinate]: # Docstring inherited from Registry.queryDimensions dimensions = iterable(dimensions) standardizedDataId = self.expandDataId(dataId, **kwds) standardizedDatasets = NamedKeyDict() requestedDimensionNames = set(self.dimensions.extract(dimensions).names) if datasets is not None: for datasetTypeExpr, collectionsExpr in datasets.items(): for trueDatasetType in self._datasetStorage.fetchDatasetTypes(datasetTypeExpr, collections=collectionsExpr, dataId=standardizedDataId): requestedDimensionNames.update(trueDatasetType.dimensions.names) standardizedDatasets[trueDatasetType] = collectionsExpr summary = QuerySummary( requested=DimensionGraph(self.dimensions, names=requestedDimensionNames), dataId=standardizedDataId, expression=where, ) builder = self.makeQueryBuilder(summary) for datasetType, collections in standardizedDatasets.items(): builder.joinDataset(datasetType, collections, isResult=False) query = builder.finish() predicate = query.predicate() for row in query.execute(): if predicate(row): result = query.extractDataId(row) if expand: yield self.expandDataId(result, records=standardizedDataId.records) else: yield result
collections: CollectionsExpression, dimensions: Optional[Iterable[Union[Dimension, str]]] = None, dataId: Optional[DataId] = None, where: Optional[str] = None, deduplicate: bool = False, expand: bool = True, **kwds) -> Iterator[DatasetRef]: # Docstring inherited from Registry.queryDatasets. # Standardize and expand the data ID provided as a constraint. standardizedDataId = self.expandDataId(dataId, **kwds) # If the datasetType passed isn't actually a DatasetType, expand it # (it could be an expression that yields multiple DatasetTypes) and # recurse. if not isinstance(datasetType, DatasetType): for trueDatasetType in self._datasetStorage.fetchDatasetTypes(datasetType, collections=collections, dataId=standardizedDataId): yield from self.queryDatasets(trueDatasetType, collections=collections, dimensions=dimensions, dataId=standardizedDataId, where=where, deduplicate=deduplicate) return # The full set of dimensions in the query is the combination of those # needed for the DatasetType and those explicitly requested, if any. requestedDimensionNames = set(datasetType.dimensions.names) if dimensions is not None: requestedDimensionNames.update(self.dimensions.extract(dimensions).names) # Construct the summary structure needed to construct a QueryBuilder. summary = QuerySummary( requested=DimensionGraph(self.dimensions, names=requestedDimensionNames), dataId=standardizedDataId, expression=where, ) builder = self.makeQueryBuilder(summary) # Add the dataset subquery to the query, telling the QueryBuilder to # include the rank of the selected collection in the results only if we # need to deduplicate. Note that if any of the collections are # actually wildcard expressions, and we've asked for deduplication, # this will raise TypeError for us. builder.joinDataset(datasetType, collections, isResult=True, addRank=deduplicate) query = builder.finish() predicate = query.predicate() if not deduplicate or len(collections) == 1: # No need to de-duplicate across collections. for row in query.execute(): if predicate(row): dataId = query.extractDataId(row, graph=datasetType.dimensions) if expand: dataId = self.expandDataId(dataId, records=standardizedDataId.records) yield query.extractDatasetRef(row, datasetType, dataId)[0] else: # For each data ID, yield only the DatasetRef with the lowest # collection rank. bestRefs = {} bestRanks = {} for row in query.execute(): if predicate(row): ref, rank = query.extractDatasetRef(row, datasetType) bestRank = bestRanks.get(ref.dataId, sys.maxsize) if rank < bestRank: bestRefs[ref.dataId] = ref bestRanks[ref.dataId] = rank # If caller requested expanded data IDs, we defer that until here # so we do as little expansion as possible. if expand: for ref in bestRefs.values(): dataId = self.expandDataId(ref.dataId, records=standardizedDataId.records) yield ref.expanded(dataId) else: yield from bestRefs.values()
"""Insert new records into a table, with conflict resolution options.
Parameters ---------- table : `sqlalchemy.Table` Table to insert new records into. values : `list` [`dict`] Sequence of dictionaries with values for new records. onConflict: `str`, optional Option for conflict resolution, can be one of "ignore" or "replace". By default no conflict resolition is performed and conflicts will cause immediate exceptions. retryLimit : `int`, optional Number of retries for insertion.
Note ---- Conflict resolution is based on table primary key only, if there are other unique constraints defined for a table they are not checked and can result in `IntegrityError` exceptions.
Even with conflict resolution options it is possible that inserts will generate conflicts due to concurrency and implementation details of transaction isolation. When it happens the only reasonable course of action is to restart transaction and repeat the whole operation. Conflicts can also appear due to violation of other non-PK constraints and it is not possible to distinguish those. To avoid infinite looping on non-PK constraint violations this method only performs few retries.
This method needs to handle transactions itself, do not call it if you are already in a transaction.
Raises ------ IntegrityError Raised for all unique constaraint violations. """
# With abort on conflict we don't need anything special, if it fails # then it fails. if onConflict is None: with self._connection.begin(): query = table.insert() self._connection.execute(query, values) return
# When doing non-aborting conflict resolution the COMMIT could # potentially fail, in that case we want to restart transaction and # re-run the whole thing again, but not forever. retries = 0 query = self._makeInsertWithConflict(table, onConflict=onConflict) while True: try: with self._connection.begin(): self._connection.execute(query, values) # stop on success break except IntegrityError: # There error could be due to PK conflict or other unique key # conflict, there is no way to identify exact reason, so we # re-try several times. if retries > retryLimit: # stop trying, looks like we can't win raise retries += 1
def _makeInsertWithConflict(self, table, onConflict): """Build an query which inserts/replaces record in a table.
Parameters ---------- table : `sqlalchemy.Table` Table to insert into. onConflict: `str` Option for conflict resolution, can be one of "ignore" or "replace". """ raise NotImplementedError() |