lsst.pipe.base
21.0.0-8-gfbe0b4b+411cd868f8
|
Public Member Functions | |
def | __init__ (self, Mapping[TaskDef, Set[Quantum]] quanta) |
nx.DiGraph | taskGraph (self) |
nx.DiGraph | graph (self) |
Iterable[QuantumNode] | inputQuanta (self) |
Iterable[QuantumNode] | outputQuanta (self) |
Tuple[DatasetTypeName,...] | allDatasetTypes (self) |
bool | isConnected (self) |
QuantumNode | getQuantumNodeByNodeId (self, NodeId nodeId) |
FrozenSet[Quantum] | getQuantaForTask (self, TaskDef taskDef) |
Iterable[TaskDef] | findTasksWithInput (self, DatasetTypeName datasetTypeName) |
Optional[TaskDef] | findTaskWithOutput (self, DatasetTypeName datasetTypeName) |
Iterable[TaskDef] | tasksWithDSType (self, DatasetTypeName datasetTypeName) |
List[TaskDef] | findTaskDefByName (self, str taskName) |
Optional[TaskDef] | findTaskDefByLabel (self, str label) |
Set[Quantum] | findQuantaWithDSType (self, DatasetTypeName datasetTypeName) |
bool | checkQuantumInGraph (self, Quantum quantum) |
def | writeDotGraph (self, Union[str, io.BufferedIOBase] output) |
_T | subset (_T self, Union[QuantumNode, Iterable[QuantumNode]] nodes) |
Tuple[_T,...] | subsetToConnected (_T self) |
Set[QuantumNode] | determineInputsToQuantumNode (self, QuantumNode node) |
Set[QuantumNode] | determineOutputsOfQuantumNode (self, QuantumNode node) |
_T | determineConnectionsOfQuantumNode (_T self, QuantumNode node) |
_T | determineAncestorsOfQuantumNode (_T self, QuantumNode node) |
List[Tuple[QuantumNode, QuantumNode]] | findCycle (self) |
def | saveUri (self, uri) |
QuantumGraph | loadUri (cls, Union[ButlerURI, str] uri, DimensionUniverse universe, Optional[Iterable[int]] nodes=None, Optional[BuildId] graphID=None) |
def | save (self, io.IO[bytes] file) |
QuantumGraph | load (cls, io.IO[bytes] file, DimensionUniverse universe, Optional[Iterable[int]] nodes=None, Optional[BuildId] graphID=None) |
Generator[TaskDef, None, None] | iterTaskGraph (self) |
def | graphID (self) |
Generator[QuantumNode, None, None] | __iter__ (self) |
int | __len__ (self) |
bool | __contains__ (self, QuantumNode node) |
dict | __getstate__ (self) |
def | __setstate__ (self, dict state) |
bool | __eq__ (self, object other) |
QuantumGraph is a directed acyclic graph of `QuantumNode` objects This data structure represents a concrete workflow generated from a `Pipeline`. Parameters ---------- quanta : Mapping of `TaskDef` to sets of `Quantum` This maps tasks (and their configs) to the sets of data they are to process.
def lsst.pipe.base.graph.graph.QuantumGraph.__init__ | ( | self, | |
Mapping[TaskDef, Set[Quantum]] | quanta | ||
) |
bool lsst.pipe.base.graph.graph.QuantumGraph.__contains__ | ( | self, | |
QuantumNode | node | ||
) |
bool lsst.pipe.base.graph.graph.QuantumGraph.__eq__ | ( | self, | |
object | other | ||
) |
dict lsst.pipe.base.graph.graph.QuantumGraph.__getstate__ | ( | self | ) |
Generator[QuantumNode, None, None] lsst.pipe.base.graph.graph.QuantumGraph.__iter__ | ( | self | ) |
int lsst.pipe.base.graph.graph.QuantumGraph.__len__ | ( | self | ) |
def lsst.pipe.base.graph.graph.QuantumGraph.__setstate__ | ( | self, | |
dict | state | ||
) |
Tuple[DatasetTypeName, ...] lsst.pipe.base.graph.graph.QuantumGraph.allDatasetTypes | ( | self | ) |
bool lsst.pipe.base.graph.graph.QuantumGraph.checkQuantumInGraph | ( | self, | |
Quantum | quantum | ||
) |
_T lsst.pipe.base.graph.graph.QuantumGraph.determineAncestorsOfQuantumNode | ( | _T | self, |
QuantumNode | node | ||
) |
Return a graph of the specified node and all the ancestor nodes directly reachable by walking edges. Parameters ---------- node : `QuantumNode` The node for which all ansestors are to be determined Returns ------- graph of `QuantumNode` Graph of node and all of its ansestors
_T lsst.pipe.base.graph.graph.QuantumGraph.determineConnectionsOfQuantumNode | ( | _T | self, |
QuantumNode | node | ||
) |
Return a graph of `QuantumNode` that are direct inputs and outputs of a specified node. Parameters ---------- node : `QuantumNode` The node of the graph for which connected nodes are to be determined. Returns ------- graph : graph of `QuantumNode` All the nodes that are directly connected to specified node
Set[QuantumNode] lsst.pipe.base.graph.graph.QuantumGraph.determineInputsToQuantumNode | ( | self, | |
QuantumNode | node | ||
) |
Return a set of `QuantumNode` that are direct inputs to a specified node. Parameters ---------- node : `QuantumNode` The node of the graph for which inputs are to be determined Returns ------- set of `QuantumNode` All the nodes that are direct inputs to specified node
Set[QuantumNode] lsst.pipe.base.graph.graph.QuantumGraph.determineOutputsOfQuantumNode | ( | self, | |
QuantumNode | node | ||
) |
Return a set of `QuantumNode` that are direct outputs of a specified node. Parameters ---------- node : `QuantumNode` The node of the graph for which outputs are to be determined Returns ------- set of `QuantumNode` All the nodes that are direct outputs to specified node
List[Tuple[QuantumNode, QuantumNode]] lsst.pipe.base.graph.graph.QuantumGraph.findCycle | ( | self | ) |
Check a graph for the presense of cycles and returns the edges of any cycles found, or an empty list if there is no cycle. Returns ------- result : list of tuple of `QuantumNode`, `QuantumNode` A list of any graph edges that form a cycle, or an empty list if there is no cycle. Empty list to so support if graph.find_cycle() syntax as an empty list is falsy.
Set[Quantum] lsst.pipe.base.graph.graph.QuantumGraph.findQuantaWithDSType | ( | self, | |
DatasetTypeName | datasetTypeName | ||
) |
Return all the `Quantum` that contain a specified `DatasetTypeName`. Parameters ---------- datasetTypeName : `str` The name of the dataset type to search for as a string, can also accept a `DatasetTypeName` which is a `NewType` of str for type safety in static type checking. Returns ------- result : `set` of `QuantumNode` objects A `set` of `QuantumNode`s that contain specified `DatasetTypeName` Raises ------ KeyError Raised if the `DatasetTypeName` is not part of the `QuantumGraph`
Optional[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.findTaskDefByLabel | ( | self, | |
str | label | ||
) |
List[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.findTaskDefByName | ( | self, | |
str | taskName | ||
) |
Determine which `TaskDef` objects in this graph are associated with a `str` representing a task name (looks at the taskName property of `TaskDef` objects). Returns a list of `TaskDef` objects as a `PipelineTask` may appear multiple times in a graph with different labels. Parameters ---------- taskName : str Name of a task to search for Returns ------- result : list of `TaskDef` List of the `TaskDef` objects that have the name specified. Multiple values are returned in the case that a task is used multiple times with different labels.
Iterable[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.findTasksWithInput | ( | self, | |
DatasetTypeName | datasetTypeName | ||
) |
Find all tasks that have the specified dataset type name as an input. Parameters ---------- datasetTypeName : `str` A string representing the name of a dataset type to be queried, can also accept a `DatasetTypeName` which is a `NewType` of str for type safety in static type checking. Returns ------- tasks : iterable of `TaskDef` `TaskDef` objects that have the specified `DatasetTypeName` as an input, list will be empty if no tasks use specified `DatasetTypeName` as an input. Raises ------ KeyError Raised if the `DatasetTypeName` is not part of the `QuantumGraph`
Optional[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.findTaskWithOutput | ( | self, | |
DatasetTypeName | datasetTypeName | ||
) |
Find all tasks that have the specified dataset type name as an output. Parameters ---------- datasetTypeName : `str` A string representing the name of a dataset type to be queried, can also accept a `DatasetTypeName` which is a `NewType` of str for type safety in static type checking. Returns ------- `TaskDef` or `None` `TaskDef` that outputs `DatasetTypeName` as an output or None if none of the tasks produce this `DatasetTypeName`. Raises ------ KeyError Raised if the `DatasetTypeName` is not part of the `QuantumGraph`
FrozenSet[Quantum] lsst.pipe.base.graph.graph.QuantumGraph.getQuantaForTask | ( | self, | |
TaskDef | taskDef | ||
) |
QuantumNode lsst.pipe.base.graph.graph.QuantumGraph.getQuantumNodeByNodeId | ( | self, | |
NodeId | nodeId | ||
) |
Lookup a `QuantumNode` from an id associated with the node. Parameters ---------- nodeId : `NodeId` The number associated with a node Returns ------- node : `QuantumNode` The node corresponding with input number Raises ------ IndexError Raised if the requested nodeId is not in the graph. IncompatibleGraphError Raised if the nodeId was built with a different graph than is not this instance (or a graph instance that produced this instance through and operation such as subset)
nx.DiGraph lsst.pipe.base.graph.graph.QuantumGraph.graph | ( | self | ) |
Return a graph representing the relations between all the `QuantumNode` objects. Largely it should be preferred to iterate over, and use methods of this class, but sometimes direct access to the networkx object may be helpful Returns ------- graph : `networkx.Digraph` Internal datastructure that holds relations of `QuantumNode` objects
def lsst.pipe.base.graph.graph.QuantumGraph.graphID | ( | self | ) |
Iterable[QuantumNode] lsst.pipe.base.graph.graph.QuantumGraph.inputQuanta | ( | self | ) |
bool lsst.pipe.base.graph.graph.QuantumGraph.isConnected | ( | self | ) |
Generator[TaskDef, None, None] lsst.pipe.base.graph.graph.QuantumGraph.iterTaskGraph | ( | self | ) |
QuantumGraph lsst.pipe.base.graph.graph.QuantumGraph.load | ( | cls, | |
io.IO[bytes] | file, | ||
DimensionUniverse | universe, | ||
Optional[Iterable[int]] | nodes = None , |
||
Optional[BuildId] | graphID = None |
||
) |
Read QuantumGraph from a file that was made by `save`. Parameters ---------- file : `io.IO` of bytes File with pickle data open in binary mode. universe: `~lsst.daf.butler.DimensionUniverse` DimensionUniverse instance, not used by the method itself but needed to ensure that registry data structures are initialized. nodes: iterable of `int` or None Numbers that correspond to nodes in the graph. If specified, only these nodes will be loaded. Defaults to None, in which case all nodes will be loaded. graphID : `str` or `None` If specified this ID is verified against the loaded graph prior to loading any Nodes. This defaults to None in which case no validation is done. Returns ------- graph : `QuantumGraph` Resulting QuantumGraph instance. Raises ------ TypeError Raised if pickle contains instance of a type other than QuantumGraph. ValueError Raised if one or more of the nodes requested is not in the `QuantumGraph` or if graphID parameter does not match the graph being loaded or if the supplied uri does not point at a valid `QuantumGraph` save file. Notes ----- Reading Quanta from pickle requires existence of singleton DimensionUniverse which is usually instantiated during Registry initialization. To make sure that DimensionUniverse exists this method accepts dummy DimensionUniverse argument.
QuantumGraph lsst.pipe.base.graph.graph.QuantumGraph.loadUri | ( | cls, | |
Union[ButlerURI, str] | uri, | ||
DimensionUniverse | universe, | ||
Optional[Iterable[int]] | nodes = None , |
||
Optional[BuildId] | graphID = None |
||
) |
Read `QuantumGraph` from a URI. Parameters ---------- uri : `ButlerURI` or `str` URI from where to load the graph. universe: `~lsst.daf.butler.DimensionUniverse` DimensionUniverse instance, not used by the method itself but needed to ensure that registry data structures are initialized. nodes: iterable of `int` or None Numbers that correspond to nodes in the graph. If specified, only these nodes will be loaded. Defaults to None, in which case all nodes will be loaded. graphID : `str` or `None` If specified this ID is verified against the loaded graph prior to loading any Nodes. This defaults to None in which case no validation is done. Returns ------- graph : `QuantumGraph` Resulting QuantumGraph instance. Raises ------ TypeError Raised if pickle contains instance of a type other than QuantumGraph. ValueError Raised if one or more of the nodes requested is not in the `QuantumGraph` or if graphID parameter does not match the graph being loaded or if the supplied uri does not point at a valid `QuantumGraph` save file. Notes ----- Reading Quanta from pickle requires existence of singleton DimensionUniverse which is usually instantiated during Registry initialization. To make sure that DimensionUniverse exists this method accepts dummy DimensionUniverse argument.
Iterable[QuantumNode] lsst.pipe.base.graph.graph.QuantumGraph.outputQuanta | ( | self | ) |
def lsst.pipe.base.graph.graph.QuantumGraph.save | ( | self, | |
io.IO[bytes] | file | ||
) |
def lsst.pipe.base.graph.graph.QuantumGraph.saveUri | ( | self, | |
uri | |||
) |
_T lsst.pipe.base.graph.graph.QuantumGraph.subset | ( | _T | self, |
Union[QuantumNode, Iterable[QuantumNode]] | nodes | ||
) |
Create a new graph object that contains the subset of the nodes specified as input. Node number is preserved. Parameters ---------- nodes : `QuantumNode` or iterable of `QuantumNode` Returns ------- graph : instance of graph type An instance of the type from which the subset was created
Tuple[_T, ...] lsst.pipe.base.graph.graph.QuantumGraph.subsetToConnected | ( | _T | self | ) |
nx.DiGraph lsst.pipe.base.graph.graph.QuantumGraph.taskGraph | ( | self | ) |
Iterable[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.tasksWithDSType | ( | self, | |
DatasetTypeName | datasetTypeName | ||
) |
Find all tasks that are associated with the specified dataset type name. Parameters ---------- datasetTypeName : `str` A string representing the name of a dataset type to be queried, can also accept a `DatasetTypeName` which is a `NewType` of str for type safety in static type checking. Returns ------- result : iterable of `TaskDef` `TaskDef` objects that are associated with the specified `DatasetTypeName` Raises ------ KeyError Raised if the `DatasetTypeName` is not part of the `QuantumGraph`
def lsst.pipe.base.graph.graph.QuantumGraph.writeDotGraph | ( | self, | |
Union[str, io.BufferedIOBase] | output | ||
) |