lsst.pipe.base  21.0.0-8-gfbe0b4b+96330b2d83
Public Member Functions | List of all members
lsst.pipe.base.graph.graph.QuantumGraph Class Reference

Public Member Functions

def __init__ (self, Mapping[TaskDef, Set[Quantum]] quanta)
 
nx.DiGraph taskGraph (self)
 
nx.DiGraph graph (self)
 
Iterable[QuantumNodeinputQuanta (self)
 
Iterable[QuantumNodeoutputQuanta (self)
 
Tuple[DatasetTypeName,...] allDatasetTypes (self)
 
bool isConnected (self)
 
QuantumNode getQuantumNodeByNodeId (self, NodeId nodeId)
 
FrozenSet[Quantum] getQuantaForTask (self, TaskDef taskDef)
 
Iterable[TaskDeffindTasksWithInput (self, DatasetTypeName datasetTypeName)
 
Optional[TaskDeffindTaskWithOutput (self, DatasetTypeName datasetTypeName)
 
Iterable[TaskDeftasksWithDSType (self, DatasetTypeName datasetTypeName)
 
List[TaskDeffindTaskDefByName (self, str taskName)
 
Optional[TaskDeffindTaskDefByLabel (self, str label)
 
Set[Quantum] findQuantaWithDSType (self, DatasetTypeName datasetTypeName)
 
bool checkQuantumInGraph (self, Quantum quantum)
 
def writeDotGraph (self, Union[str, io.BufferedIOBase] output)
 
_T subset (_T self, Union[QuantumNode, Iterable[QuantumNode]] nodes)
 
Tuple[_T,...] subsetToConnected (_T self)
 
Set[QuantumNodedetermineInputsToQuantumNode (self, QuantumNode node)
 
Set[QuantumNodedetermineOutputsOfQuantumNode (self, QuantumNode node)
 
_T determineConnectionsOfQuantumNode (_T self, QuantumNode node)
 
_T determineAncestorsOfQuantumNode (_T self, QuantumNode node)
 
List[Tuple[QuantumNode, QuantumNode]] findCycle (self)
 
def saveUri (self, uri)
 
QuantumGraph loadUri (cls, Union[ButlerURI, str] uri, DimensionUniverse universe, Optional[Iterable[int]] nodes=None, Optional[BuildId] graphID=None)
 
def save (self, io.IO[bytes] file)
 
QuantumGraph load (cls, io.IO[bytes] file, DimensionUniverse universe, Optional[Iterable[int]] nodes=None, Optional[BuildId] graphID=None)
 
Generator[TaskDef, None, None] iterTaskGraph (self)
 
def graphID (self)
 
Generator[QuantumNode, None, None] __iter__ (self)
 
int __len__ (self)
 
bool __contains__ (self, QuantumNode node)
 
dict __getstate__ (self)
 
def __setstate__ (self, dict state)
 
bool __eq__ (self, object other)
 

Detailed Description

QuantumGraph is a directed acyclic graph of `QuantumNode` objects

This data structure represents a concrete workflow generated from a
`Pipeline`.

Parameters
----------
quanta : Mapping of `TaskDef` to sets of `Quantum`
    This maps tasks (and their configs) to the sets of data they are to
    process.

Definition at line 73 of file graph.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.pipe.base.graph.graph.QuantumGraph.__init__ (   self,
Mapping[TaskDef, Set[Quantum]]  quanta 
)

Definition at line 85 of file graph.py.

Member Function Documentation

◆ __contains__()

bool lsst.pipe.base.graph.graph.QuantumGraph.__contains__ (   self,
QuantumNode  node 
)

Definition at line 825 of file graph.py.

◆ __eq__()

bool lsst.pipe.base.graph.graph.QuantumGraph.__eq__ (   self,
object  other 
)

Definition at line 849 of file graph.py.

◆ __getstate__()

dict lsst.pipe.base.graph.graph.QuantumGraph.__getstate__ (   self)
Stores a compact form of the graph as a list of graph nodes, and a
tuple of task labels and task configs. The full graph can be
reconstructed with this information, and it preseves the ordering of
the graph ndoes.

Definition at line 828 of file graph.py.

◆ __iter__()

Generator[QuantumNode, None, None] lsst.pipe.base.graph.graph.QuantumGraph.__iter__ (   self)

Definition at line 819 of file graph.py.

◆ __len__()

int lsst.pipe.base.graph.graph.QuantumGraph.__len__ (   self)

Definition at line 822 of file graph.py.

◆ __setstate__()

def lsst.pipe.base.graph.graph.QuantumGraph.__setstate__ (   self,
dict  state 
)
Reconstructs the state of the graph from the information persisted
in getstate.

Definition at line 836 of file graph.py.

◆ allDatasetTypes()

Tuple[DatasetTypeName, ...] lsst.pipe.base.graph.graph.QuantumGraph.allDatasetTypes (   self)
Return all the `DatasetTypeName` objects that are contained inside
the graph.

Returns
-------
tuple of `DatasetTypeName`
    All the data set type names that are present in the graph

Definition at line 211 of file graph.py.

◆ checkQuantumInGraph()

bool lsst.pipe.base.graph.graph.QuantumGraph.checkQuantumInGraph (   self,
Quantum  quantum 
)
Check if specified quantum appears in the graph as part of a node.

Parameters
----------
quantum : `Quantum`
    The quantum to search for

Returns
-------
`bool`
    The result of searching for the quantum

Definition at line 420 of file graph.py.

◆ determineAncestorsOfQuantumNode()

_T lsst.pipe.base.graph.graph.QuantumGraph.determineAncestorsOfQuantumNode ( _T  self,
QuantumNode  node 
)
Return a graph of the specified node and all the ancestor nodes
directly reachable by walking edges.

Parameters
----------
node : `QuantumNode`
    The node for which all ansestors are to be determined

Returns
-------
graph of `QuantumNode`
    Graph of node and all of its ansestors

Definition at line 537 of file graph.py.

◆ determineConnectionsOfQuantumNode()

_T lsst.pipe.base.graph.graph.QuantumGraph.determineConnectionsOfQuantumNode ( _T  self,
QuantumNode  node 
)
Return a graph of `QuantumNode` that are direct inputs and outputs
of a specified node.

Parameters
----------
node : `QuantumNode`
    The node of the graph for which connected nodes are to be
    determined.

Returns
-------
graph : graph of `QuantumNode`
    All the nodes that are directly connected to specified node

Definition at line 518 of file graph.py.

◆ determineInputsToQuantumNode()

Set[QuantumNode] lsst.pipe.base.graph.graph.QuantumGraph.determineInputsToQuantumNode (   self,
QuantumNode  node 
)
Return a set of `QuantumNode` that are direct inputs to a specified
node.

Parameters
----------
node : `QuantumNode`
    The node of the graph for which inputs are to be determined

Returns
-------
set of `QuantumNode`
    All the nodes that are direct inputs to specified node

Definition at line 486 of file graph.py.

◆ determineOutputsOfQuantumNode()

Set[QuantumNode] lsst.pipe.base.graph.graph.QuantumGraph.determineOutputsOfQuantumNode (   self,
QuantumNode  node 
)
Return a set of `QuantumNode` that are direct outputs of a specified
node.

Parameters
----------
node : `QuantumNode`
    The node of the graph for which outputs are to be determined

Returns
-------
set of `QuantumNode`
    All the nodes that are direct outputs to specified node

Definition at line 502 of file graph.py.

◆ findCycle()

List[Tuple[QuantumNode, QuantumNode]] lsst.pipe.base.graph.graph.QuantumGraph.findCycle (   self)
Check a graph for the presense of cycles and returns the edges of
any cycles found, or an empty list if there is no cycle.

Returns
-------
result : list of tuple of `QuantumNode`, `QuantumNode`
    A list of any graph edges that form a cycle, or an empty list if
    there is no cycle. Empty list to so support if graph.find_cycle()
    syntax as an empty list is falsy.

Definition at line 555 of file graph.py.

◆ findQuantaWithDSType()

Set[Quantum] lsst.pipe.base.graph.graph.QuantumGraph.findQuantaWithDSType (   self,
DatasetTypeName  datasetTypeName 
)
Return all the `Quantum` that contain a specified `DatasetTypeName`.

Parameters
----------
datasetTypeName : `str`
    The name of the dataset type to search for as a string,
    can also accept a `DatasetTypeName` which is a `NewType` of str for
    type safety in static type checking.

Returns
-------
result : `set` of `QuantumNode` objects
    A `set` of `QuantumNode`s that contain specified `DatasetTypeName`

Raises
------
KeyError
    Raised if the `DatasetTypeName` is not part of the `QuantumGraph`

Definition at line 394 of file graph.py.

◆ findTaskDefByLabel()

Optional[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.findTaskDefByLabel (   self,
str  label 
)
Determine which `TaskDef` objects in this graph are associated
with a `str` representing a tasks label.

Parameters
----------
taskName : str
    Name of a task to search for

Returns
-------
result : `TaskDef`
    `TaskDef` objects that has the specified label.

Definition at line 375 of file graph.py.

◆ findTaskDefByName()

List[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.findTaskDefByName (   self,
str  taskName 
)
Determine which `TaskDef` objects in this graph are associated
with a `str` representing a task name (looks at the taskName property
of `TaskDef` objects).

Returns a list of `TaskDef` objects as a `PipelineTask` may appear
multiple times in a graph with different labels.

Parameters
----------
taskName : str
    Name of a task to search for

Returns
-------
result : list of `TaskDef`
    List of the `TaskDef` objects that have the name specified.
    Multiple values are returned in the case that a task is used
    multiple times with different labels.

Definition at line 348 of file graph.py.

◆ findTasksWithInput()

Iterable[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.findTasksWithInput (   self,
DatasetTypeName  datasetTypeName 
)
Find all tasks that have the specified dataset type name as an
input.

Parameters
----------
datasetTypeName : `str`
    A string representing the name of a dataset type to be queried,
    can also accept a `DatasetTypeName` which is a `NewType` of str for
    type safety in static type checking.

Returns
-------
tasks : iterable of `TaskDef`
    `TaskDef` objects that have the specified `DatasetTypeName` as an
    input, list will be empty if no tasks use specified
    `DatasetTypeName` as an input.

Raises
------
KeyError
    Raised if the `DatasetTypeName` is not part of the `QuantumGraph`

Definition at line 271 of file graph.py.

◆ findTaskWithOutput()

Optional[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.findTaskWithOutput (   self,
DatasetTypeName  datasetTypeName 
)
Find all tasks that have the specified dataset type name as an
output.

Parameters
----------
datasetTypeName : `str`
    A string representing the name of a dataset type to be queried,
    can also accept a `DatasetTypeName` which is a `NewType` of str for
    type safety in static type checking.

Returns
-------
`TaskDef` or `None`
    `TaskDef` that outputs `DatasetTypeName` as an output or None if
    none of the tasks produce this `DatasetTypeName`.

Raises
------
KeyError
    Raised if the `DatasetTypeName` is not part of the `QuantumGraph`

Definition at line 296 of file graph.py.

◆ getQuantaForTask()

FrozenSet[Quantum] lsst.pipe.base.graph.graph.QuantumGraph.getQuantaForTask (   self,
TaskDef  taskDef 
)
Return all the `Quantum` associated with a `TaskDef`.

Parameters
----------
taskDef : `TaskDef`
    The `TaskDef` for which `Quantum` are to be queried

Returns
-------
frozenset of `Quantum`
    The `set` of `Quantum` that is associated with the specified
    `TaskDef`.

Definition at line 255 of file graph.py.

◆ getQuantumNodeByNodeId()

QuantumNode lsst.pipe.base.graph.graph.QuantumGraph.getQuantumNodeByNodeId (   self,
NodeId  nodeId 
)
Lookup a `QuantumNode` from an id associated with the node.

Parameters
----------
nodeId : `NodeId`
    The number associated with a node

Returns
-------
node : `QuantumNode`
    The node corresponding with input number

Raises
------
IndexError
    Raised if the requested nodeId is not in the graph.
IncompatibleGraphError
    Raised if the nodeId was built with a different graph than is not
    this instance (or a graph instance that produced this instance
    through and operation such as subset)

Definition at line 229 of file graph.py.

◆ graph()

nx.DiGraph lsst.pipe.base.graph.graph.QuantumGraph.graph (   self)
Return a graph representing the relations between all the
`QuantumNode` objects. Largely it should be preferred to iterate
over, and use methods of this class, but sometimes direct access to
the networkx object may be helpful

Returns
-------
graph : `networkx.Digraph`
    Internal datastructure that holds relations of `QuantumNode`
    objects

Definition at line 170 of file graph.py.

◆ graphID()

def lsst.pipe.base.graph.graph.QuantumGraph.graphID (   self)
Returns the ID generated by the graph at construction time

Definition at line 814 of file graph.py.

◆ inputQuanta()

Iterable[QuantumNode] lsst.pipe.base.graph.graph.QuantumGraph.inputQuanta (   self)
Make a `list` of all `QuantumNode` objects that are 'input' nodes
to the graph, meaning those nodes to not depend on any other nodes in
the graph.

Returns
-------
inputNodes : iterable of `QuantumNode`
    A list of nodes that are inputs to the graph

Definition at line 185 of file graph.py.

◆ isConnected()

bool lsst.pipe.base.graph.graph.QuantumGraph.isConnected (   self)
Return True if all of the nodes in the graph are connected, ignores
directionality of connections.

Definition at line 223 of file graph.py.

◆ iterTaskGraph()

Generator[TaskDef, None, None] lsst.pipe.base.graph.graph.QuantumGraph.iterTaskGraph (   self)
Iterate over the `taskGraph` attribute in topological order

Yields
------
taskDef : `TaskDef`
    `TaskDef` objects in topological order

Definition at line 803 of file graph.py.

◆ load()

QuantumGraph lsst.pipe.base.graph.graph.QuantumGraph.load (   cls,
io.IO[bytes]  file,
DimensionUniverse  universe,
Optional[Iterable[int]]   nodes = None,
Optional[BuildId]   graphID = None 
)
Read QuantumGraph from a file that was made by `save`.

Parameters
----------
file : `io.IO` of bytes
    File with pickle data open in binary mode.
universe: `~lsst.daf.butler.DimensionUniverse`
    DimensionUniverse instance, not used by the method itself but
    needed to ensure that registry data structures are initialized.
nodes: iterable of `int` or None
    Numbers that correspond to nodes in the graph. If specified, only
    these nodes will be loaded. Defaults to None, in which case all
    nodes will be loaded.
graphID : `str` or `None`
    If specified this ID is verified against the loaded graph prior to
    loading any Nodes. This defaults to None in which case no
    validation is done.

Returns
-------
graph : `QuantumGraph`
    Resulting QuantumGraph instance.

Raises
------
TypeError
    Raised if pickle contains instance of a type other than
    QuantumGraph.
ValueError
    Raised if one or more of the nodes requested is not in the
    `QuantumGraph` or if graphID parameter does not match the graph
    being loaded or if the supplied uri does not point at a valid
    `QuantumGraph` save file.

Notes
-----
Reading Quanta from pickle requires existence of singleton
DimensionUniverse which is usually instantiated during Registry
initialization. To make sure that DimensionUniverse exists this method
accepts dummy DimensionUniverse argument.

Definition at line 746 of file graph.py.

◆ loadUri()

QuantumGraph lsst.pipe.base.graph.graph.QuantumGraph.loadUri (   cls,
Union[ButlerURI, str]  uri,
DimensionUniverse  universe,
Optional[Iterable[int]]   nodes = None,
Optional[BuildId]   graphID = None 
)
Read `QuantumGraph` from a URI.

Parameters
----------
uri : `ButlerURI` or `str`
    URI from where to load the graph.
universe: `~lsst.daf.butler.DimensionUniverse`
    DimensionUniverse instance, not used by the method itself but
    needed to ensure that registry data structures are initialized.
nodes: iterable of `int` or None
    Numbers that correspond to nodes in the graph. If specified, only
    these nodes will be loaded. Defaults to None, in which case all
    nodes will be loaded.
graphID : `str` or `None`
    If specified this ID is verified against the loaded graph prior to
    loading any Nodes. This defaults to None in which case no
    validation is done.

Returns
-------
graph : `QuantumGraph`
    Resulting QuantumGraph instance.

Raises
------
TypeError
    Raised if pickle contains instance of a type other than
    QuantumGraph.
ValueError
    Raised if one or more of the nodes requested is not in the
    `QuantumGraph` or if graphID parameter does not match the graph
    being loaded or if the supplied uri does not point at a valid
    `QuantumGraph` save file.


Notes
-----
Reading Quanta from pickle requires existence of singleton
DimensionUniverse which is usually instantiated during Registry
initialization. To make sure that DimensionUniverse exists this method
accepts dummy DimensionUniverse argument.

Definition at line 586 of file graph.py.

◆ outputQuanta()

Iterable[QuantumNode] lsst.pipe.base.graph.graph.QuantumGraph.outputQuanta (   self)
Make a `list` of all `QuantumNode` objects that are 'output' nodes
to the graph, meaning those nodes have no nodes that depend them in
the graph.

Returns
-------
outputNodes : iterable of `QuantumNode`
    A list of nodes that are outputs of the graph

Definition at line 198 of file graph.py.

◆ save()

def lsst.pipe.base.graph.graph.QuantumGraph.save (   self,
io.IO[bytes]  file 
)
Save QuantumGraph to a file.

Presently we store QuantumGraph in pickle format, this could
potentially change in the future if better format is found.

Parameters
----------
file : `io.BufferedIOBase`
    File to write pickle data open in binary mode.

Definition at line 652 of file graph.py.

◆ saveUri()

def lsst.pipe.base.graph.graph.QuantumGraph.saveUri (   self,
  uri 
)
Save `QuantumGraph` to the specified URI.

Parameters
----------
uri : `ButlerURI` or `str`
    URI to where the graph should be saved.

Definition at line 571 of file graph.py.

◆ subset()

_T lsst.pipe.base.graph.graph.QuantumGraph.subset ( _T  self,
Union[QuantumNode, Iterable[QuantumNode]]  nodes 
)
Create a new graph object that contains the subset of the nodes
specified as input. Node number is preserved.

Parameters
----------
nodes : `QuantumNode` or iterable of `QuantumNode`

Returns
-------
graph : instance of graph type
    An instance of the type from which the subset was created

Definition at line 448 of file graph.py.

◆ subsetToConnected()

Tuple[_T, ...] lsst.pipe.base.graph.graph.QuantumGraph.subsetToConnected ( _T  self)
Generate a list of subgraphs where each is connected.

Returns
-------
result : list of `QuantumGraph`
    A list of graphs that are each connected

Definition at line 475 of file graph.py.

◆ taskGraph()

nx.DiGraph lsst.pipe.base.graph.graph.QuantumGraph.taskGraph (   self)
Return a graph representing the relations between the tasks inside
the quantum graph.

Returns
-------
taskGraph : `networkx.Digraph`
    Internal datastructure that holds relations of `TaskDef` objects

Definition at line 158 of file graph.py.

◆ tasksWithDSType()

Iterable[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.tasksWithDSType (   self,
DatasetTypeName  datasetTypeName 
)
Find all tasks that are associated with the specified dataset type
name.

Parameters
----------
datasetTypeName : `str`
    A string representing the name of a dataset type to be queried,
    can also accept a `DatasetTypeName` which is a `NewType` of str for
    type safety in static type checking.

Returns
-------
result : iterable of `TaskDef`
    `TaskDef` objects that are associated with the specified
    `DatasetTypeName`

Raises
------
KeyError
    Raised if the `DatasetTypeName` is not part of the `QuantumGraph`

Definition at line 320 of file graph.py.

◆ writeDotGraph()

def lsst.pipe.base.graph.graph.QuantumGraph.writeDotGraph (   self,
Union[str, io.BufferedIOBase]  output 
)
Write out the graph as a dot graph.

Parameters
----------
output : str or `io.BufferedIOBase`
    Either a filesystem path to write to, or a file handle object

Definition at line 438 of file graph.py.


The documentation for this class was generated from the following file: