lsst.pipe.base  20.0.0-20-g67d4e31+605b5e4d13
Public Member Functions | List of all members
lsst.pipe.base.graph.graph.QuantumGraph Class Reference

Public Member Functions

def __init__ (self, Mapping[TaskDef, Set[Quantum]] quanta)
 
nx.DiGraph taskGraph (self)
 
nx.DiGraph graph (self)
 
Iterable[QuantumNodeinputQuanta (self)
 
Iterable[QuantumNodeoutputQuanta (self)
 
Tuple[DatasetTypeName,...] allDatasetTypes (self)
 
bool isConnected (self)
 
QuantumNode getQuantumNodeByNodeId (self, NodeId nodeId)
 
FrozenSet[Quantum] getQuantaForTask (self, TaskDef taskDef)
 
Iterable[TaskDeffindTasksWithInput (self, DatasetTypeName datasetTypeName)
 
Optional[TaskDeffindTaskWithOutput (self, DatasetTypeName datasetTypeName)
 
Iterable[TaskDeftasksWithDSType (self, DatasetTypeName datasetTypeName)
 
List[TaskDeffindTaskDefByName (self, str taskName)
 
Optional[TaskDeffindTaskDefByLabel (self, str label)
 
Set[Quantum] findQuantaWithDSType (self, DatasetTypeName datasetTypeName)
 
bool checkQuantumInGraph (self, Quantum quantum)
 
def writeDotGraph (self, Union[str, io.BufferedIOBase] output)
 
_T subset (_T self, Union[QuantumNode, Iterable[QuantumNode]] nodes)
 
Tuple[_T,...] subsetToConnected (_T self)
 
Set[QuantumNodedetermineInputsToQuantumNode (self, QuantumNode node)
 
Set[QuantumNodedetermineOutputsOfQuantumNode (self, QuantumNode node)
 
_T determineConnectionsOfQuantumNode (_T self, QuantumNode node)
 
_T determineAncestorsOfQuantumNode (_T self, QuantumNode node)
 
List[Tuple[QuantumNode, QuantumNode]] findCycle (self)
 
def save (self, file)
 
def load (cls, file, universe)
 
Generator[TaskDef, None, None] iterTaskGraph (self)
 
Generator[QuantumNode, None, None] __iter__ (self)
 
int __len__ (self)
 
bool __contains__ (self, QuantumNode node)
 
dict __getstate__ (self)
 
def __setstate__ (self, dict state)
 
bool __eq__ (self, object other)
 

Detailed Description

QuantumGraph is a directed acyclic graph of `QuantumNode`s

This data structure represents a concrete workflow generated from a
`Pipeline`.

Parameters
----------
quanta : Mapping of `TaskDef` to sets of `Quantum`
    This maps tasks (and their configs) to the sets of data they are to
    process.

Definition at line 54 of file graph.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.pipe.base.graph.graph.QuantumGraph.__init__ (   self,
Mapping[TaskDef, Set[Quantum]]  quanta 
)

Definition at line 66 of file graph.py.

Member Function Documentation

◆ __contains__()

bool lsst.pipe.base.graph.graph.QuantumGraph.__contains__ (   self,
QuantumNode  node 
)

Definition at line 611 of file graph.py.

◆ __eq__()

bool lsst.pipe.base.graph.graph.QuantumGraph.__eq__ (   self,
object  other 
)

Definition at line 635 of file graph.py.

◆ __getstate__()

dict lsst.pipe.base.graph.graph.QuantumGraph.__getstate__ (   self)
Stores a compact form of the graph as a list of graph nodes, and a
tuple of task labels and task configs. The full graph can be
reconstructed with this information, and it preseves the ordering of
the graph ndoes.

Definition at line 614 of file graph.py.

◆ __iter__()

Generator[QuantumNode, None, None] lsst.pipe.base.graph.graph.QuantumGraph.__iter__ (   self)

Definition at line 605 of file graph.py.

◆ __len__()

int lsst.pipe.base.graph.graph.QuantumGraph.__len__ (   self)

Definition at line 608 of file graph.py.

◆ __setstate__()

def lsst.pipe.base.graph.graph.QuantumGraph.__setstate__ (   self,
dict  state 
)
Reconstructs the state of the graph from the information persisted
in getstate.

Definition at line 622 of file graph.py.

◆ allDatasetTypes()

Tuple[DatasetTypeName, ...] lsst.pipe.base.graph.graph.QuantumGraph.allDatasetTypes (   self)
Return all the `DatasetTypeNames` that are contained inside the
graph.

Returns
-------
tuple of `DatasetTypeName`
    All the data set type names that are present in the graph

Definition at line 190 of file graph.py.

◆ checkQuantumInGraph()

bool lsst.pipe.base.graph.graph.QuantumGraph.checkQuantumInGraph (   self,
Quantum  quantum 
)
Check if specified quantum appears in the graph as part of a node.

Parameters
----------
quantum : `Quantum`
    The quantum to search for

Returns
-------
`bool`
    The result of searching for the quantum

Definition at line 398 of file graph.py.

◆ determineAncestorsOfQuantumNode()

_T lsst.pipe.base.graph.graph.QuantumGraph.determineAncestorsOfQuantumNode ( _T  self,
QuantumNode  node 
)
Return a graph of the specified node and all the ancestor nodes
directly reachable by walking edges.

Parameters
----------
node : `QuantumNode`
    The node for which all ansestors are to be determined

Returns
-------
graph of `QuantumNode`
    Graph of node and all of its ansestors

Definition at line 515 of file graph.py.

◆ determineConnectionsOfQuantumNode()

_T lsst.pipe.base.graph.graph.QuantumGraph.determineConnectionsOfQuantumNode ( _T  self,
QuantumNode  node 
)
Return a graph of `QuantumNode` that are direct inputs and outputs
of a specified node.

Parameters
----------
node : `QuantumNode`
    The node of the graph for which connected nodes are to be
    determined.

Returns
-------
graph : graph of `QuantumNode`
    All the nodes that are directly connected to specified node

Definition at line 496 of file graph.py.

◆ determineInputsToQuantumNode()

Set[QuantumNode] lsst.pipe.base.graph.graph.QuantumGraph.determineInputsToQuantumNode (   self,
QuantumNode  node 
)
Return a set of `QuantumNode` that are direct inputs to a specified
node.

Parameters
----------
node : `QuantumNode`
    The node of the graph for which inputs are to be determined

Returns
-------
set of `QuantumNode`
    All the nodes that are direct inputs to specified node

Definition at line 464 of file graph.py.

◆ determineOutputsOfQuantumNode()

Set[QuantumNode] lsst.pipe.base.graph.graph.QuantumGraph.determineOutputsOfQuantumNode (   self,
QuantumNode  node 
)
Return a set of `QuantumNode` that are direct outputs of a specified
node.

Parameters
----------
node : `QuantumNode`
    The node of the graph for which outputs are to be determined

Returns
-------
set of `QuantumNode`
    All the nodes that are direct outputs to specified node

Definition at line 480 of file graph.py.

◆ findCycle()

List[Tuple[QuantumNode, QuantumNode]] lsst.pipe.base.graph.graph.QuantumGraph.findCycle (   self)
Check a graph for the presense of cycles and returns the edges of
any cycles found, or an empty list if there is no cycle.

Returns
-------
result : list of tuple of `QuantumNode`, `QuantumNode`
    A list of any graph edges that form a cycle, or an empty list if
    there is no cycle. Empty list to so support if graph.find_cycle()
    syntax as an empty list is falsy.

Definition at line 533 of file graph.py.

◆ findQuantaWithDSType()

Set[Quantum] lsst.pipe.base.graph.graph.QuantumGraph.findQuantaWithDSType (   self,
DatasetTypeName  datasetTypeName 
)
Return all the `Quantum` that contain a specified `DatasetTypeName`.

Parameters
----------
datasetTypeName : `str`
    The name of the dataset type to search for as a string,
    can also accept a `DatasetTypeName` which is a `NewType` of str for
    type safety in static type checking.

Returns
-------
result : `set` of `QuantumNode`s
    A `set` of `QuantumNode`s that contain specified `DatasetTypeName`

Raises
------
KeyError
    Raised if the `DatasetTypeName` is not part of the `QuantumGraph`

Definition at line 372 of file graph.py.

◆ findTaskDefByLabel()

Optional[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.findTaskDefByLabel (   self,
str  label 
)
Determine which `TaskDef`s in this graph are associated with a `str`
representing a tasks label.

Parameters
----------
taskName : str
    Name of a task to search for

Returns
-------
result : `TaskDef`
    `TaskDef`s that has the specified label.

Definition at line 353 of file graph.py.

◆ findTaskDefByName()

List[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.findTaskDefByName (   self,
str  taskName 
)
Determine which `TaskDef`s in this graph are associated with a `str`
representing a task name (looks at the taskName property of
`TaskDef`s).

Returns a list of `TaskDef`s as a `PipelineTask` may appear multiple
times in a graph with different labels.

Parameters
----------
taskName : str
    Name of a task to search for

Returns
-------
result : list of `TaskDef`
    List of the `TaskDef`s that have the name specified. Multiple
    values are returned in the case that a task is used multiple times
    with different labels.

Definition at line 326 of file graph.py.

◆ findTasksWithInput()

Iterable[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.findTasksWithInput (   self,
DatasetTypeName  datasetTypeName 
)
Find all tasks that have the specified dataset type name as an
input.

Parameters
----------
datasetTypeName : `str`
    A string representing the name of a dataset type to be queried,
    can also accept a `DatasetTypeName` which is a `NewType` of str for
    type safety in static type checking.

Returns
-------
tasks : iterable of `TaskDef`
    `TaskDef`s that have the specified `DatasetTypeName` as an input,
    list will be empty if no tasks use specified `DatasetTypeName` as
    an input.

Raises
------
KeyError
    Raised if the `DatasetTypeName` is not part of the `QuantumGraph`

Definition at line 250 of file graph.py.

◆ findTaskWithOutput()

Optional[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.findTaskWithOutput (   self,
DatasetTypeName  datasetTypeName 
)
Find all tasks that have the specified dataset type name as an
output.

Parameters
----------
datasetTypeName : `str`
    A string representing the name of a dataset type to be queried,
    can also accept a `DatasetTypeName` which is a `NewType` of str for
    type safety in static type checking.

Returns
-------
`TaskDef` or `None`
    `TaskDef` that outputs `DatasetTypeName` as an output or None if
    none of the tasks produce this `DatasetTypeName`.

Raises
------
KeyError
    Raised if the `DatasetTypeName` is not part of the `QuantumGraph`

Definition at line 275 of file graph.py.

◆ getQuantaForTask()

FrozenSet[Quantum] lsst.pipe.base.graph.graph.QuantumGraph.getQuantaForTask (   self,
TaskDef  taskDef 
)
Return all the `Quantum` associated with a `TaskDef`.

Parameters
----------
taskDef : `TaskDef`
    The `TaskDef` for which `Quantum` are to be queried

Returns
-------
frozenset of `Quantum`
    The `set` of `Quantum` that is associated with the specified
    `TaskDef`.

Definition at line 234 of file graph.py.

◆ getQuantumNodeByNodeId()

QuantumNode lsst.pipe.base.graph.graph.QuantumGraph.getQuantumNodeByNodeId (   self,
NodeId  nodeId 
)
Lookup a `QuantumNode` from an id associated with the node.

Parameters
----------
nodeId : `NodeId`
    The number associated with a node

Returns
-------
node : `QuantumNode`
    The node corresponding with input number

Raises
------
IndexError
    Raised if the requested nodeId is not in the graph.
IncompatibleGraphError
    Raised if the nodeId was built with a different graph than is not
    this instance (or a graph instance that produced this instance
    through and operation such as subset)

Definition at line 208 of file graph.py.

◆ graph()

nx.DiGraph lsst.pipe.base.graph.graph.QuantumGraph.graph (   self)
Return a graph representing the relations between all the
`QuantumNode`s. Largely it should be preferred to iterate over, and use
methods of this class, but sometimes direct access to the networkx
object may be helpful

Returns
-------
graph : `networkx.Digraph`
    Internal datastructure that holds relations of `QuantumNode`s

Definition at line 151 of file graph.py.

◆ inputQuanta()

Iterable[QuantumNode] lsst.pipe.base.graph.graph.QuantumGraph.inputQuanta (   self)
Make a `list` of all `QuantumNode`s that are 'input' nodes to the
graph, meaning those nodes to not depend on any other nodes in the
graph.

Returns
-------
inputNodes : iterable of `QuantumNode`
    A list of nodes that are inputs to the graph

Definition at line 165 of file graph.py.

◆ isConnected()

bool lsst.pipe.base.graph.graph.QuantumGraph.isConnected (   self)
Return True if all of the nodes in the graph are connected, ignores
directionality of connections.

Definition at line 202 of file graph.py.

◆ iterTaskGraph()

Generator[TaskDef, None, None] lsst.pipe.base.graph.graph.QuantumGraph.iterTaskGraph (   self)
Iterate over the `taskGraph` attribute in topological order

Yields
------
`TaskDef`
    `TaskDef` objects in topological order

Definition at line 595 of file graph.py.

◆ load()

def lsst.pipe.base.graph.graph.QuantumGraph.load (   cls,
  file,
  universe 
)
Read QuantumGraph from a file that was made by `save`.

Parameters
----------
file : `io.BufferedIOBase`
    File with pickle data open in binary mode.
universe: `~lsst.daf.butler.DimensionUniverse`
    DimensionUniverse instance, not used by the method itself but
    needed to ensure that registry data structures are initialized.

Returns
-------
graph : `QuantumGraph`
    Resulting QuantumGraph instance.

Raises
------
TypeError
    Raised if pickle contains instance of a type other than
    QuantumGraph.
Notes
-----
Reading Quanta from pickle requires existence of singleton
DimensionUniverse which is usually instantiated during Registry
initialization. To make sure that DimensionUniverse exists this method
accepts dummy DimensionUniverse argument.

Definition at line 562 of file graph.py.

◆ outputQuanta()

Iterable[QuantumNode] lsst.pipe.base.graph.graph.QuantumGraph.outputQuanta (   self)
Make a `list` of all `QuantumNode`s that are 'output' nodes to the
graph, meaning those nodes have no nodes that depend them in the graph.

Returns
-------
outputNodes : iterable of `QuantumNode`
    A list of nodes that are outputs of the graph

Definition at line 178 of file graph.py.

◆ save()

def lsst.pipe.base.graph.graph.QuantumGraph.save (   self,
  file 
)
Save QuantumGraph to a file.
Presently we store QuantumGraph in pickle format, this could
potentially change in the future if better format is found.

Parameters
----------
file : `io.BufferedIOBase`
    File to write pickle data open in binary mode.

Definition at line 549 of file graph.py.

◆ subset()

_T lsst.pipe.base.graph.graph.QuantumGraph.subset ( _T  self,
Union[QuantumNode, Iterable[QuantumNode]]  nodes 
)
Create a new graph object that contains the subset of the nodes
specified as input. Node number is preserved.

Parameters
----------
nodes : `QuantumNode` or iterable of `QuantumNode`

Returns
-------
graph : instance of graph type
    An instance of the type from which the subset was created

Definition at line 426 of file graph.py.

◆ subsetToConnected()

Tuple[_T, ...] lsst.pipe.base.graph.graph.QuantumGraph.subsetToConnected ( _T  self)
Generate a list of subgraphs where each is connected.

Returns
-------
result : list of `QuantumGraph`
    A list of graphs that are each connected

Definition at line 453 of file graph.py.

◆ taskGraph()

nx.DiGraph lsst.pipe.base.graph.graph.QuantumGraph.taskGraph (   self)
Return a graph representing the relations between the tasks inside
the quantum graph.

Returns
-------
taskGraph : `networkx.Digraph`
    Internal datastructure that holds relations of `TaskDef`s

Definition at line 139 of file graph.py.

◆ tasksWithDSType()

Iterable[TaskDef] lsst.pipe.base.graph.graph.QuantumGraph.tasksWithDSType (   self,
DatasetTypeName  datasetTypeName 
)
Find all tasks that are associated with the specified dataset type
name.

Parameters
----------
datasetTypeName : `str`
    A string representing the name of a dataset type to be queried,
    can also accept a `DatasetTypeName` which is a `NewType` of str for
    type safety in static type checking.

Returns
-------
result : iterable of `TaskDef`
    `TaskDef`s that are associated with the specified `DatasetTypeName`

Raises
------
KeyError
    Raised if the `DatasetTypeName` is not part of the `QuantumGraph`

Definition at line 299 of file graph.py.

◆ writeDotGraph()

def lsst.pipe.base.graph.graph.QuantumGraph.writeDotGraph (   self,
Union[str, io.BufferedIOBase]  output 
)
Write out the graph as a dot graph.

Parameters
----------
output : str or `io.BufferedIOBase`
    Either a filesystem path to write to, or a file handle object

Definition at line 416 of file graph.py.


The documentation for this class was generated from the following file: