lsst.ctrl.pool
20.0.0+0f5db2c0a9
|
Public Member Functions | |
def | __init__ (self, *args, **kwargs) |
def | __del__ (self) |
def | log (self, msg, *args) |
def | command (self, cmd) |
def | map (self, context, func, dataList, *args, **kwargs) |
Scatter work to slaves and gather the results. More... | |
def | reduce (self, context, reducer, func, dataList, *args, **kwargs) |
Scatter work to slaves and reduce the results. More... | |
def | mapNoBalance (self, context, func, dataList, *args, **kwargs) |
Scatter work to slaves and gather the results. More... | |
def | reduceNoBalance (self, context, reducer, func, dataList, *args, **kwargs) |
Scatter work to slaves and reduce the results. More... | |
def | mapToPrevious (self, context, func, dataList, *args, **kwargs) |
Scatter work to the same target as before. More... | |
def | reduceToPrevious (self, context, reducer, func, dataList, *args, **kwargs) |
Reduction where work goes to the same target as before. More... | |
def | storeSet (self, context, **kwargs) |
Store data on slave for a particular context. More... | |
def | storeDel (self, context, *nameList) |
def | storeClear (self, context) |
def | cacheClear (self, context) |
def | cacheList (self, context) |
def | storeList (self, context) |
def | exit (self) |
def | isMaster (self) |
def | __call__ (cls, *args, **kwargs) |
Public Attributes | |
root | |
size | |
comm | |
rank | |
debugger | |
node | |
Master node instance of MPI process pool Only the master node should instantiate this. WARNING: You should not let a pool instance hang around at program termination, as the garbage collection behaves differently, and may cause a segmentation fault (signal 11).
def lsst.ctrl.pool.pool.PoolMaster.__init__ | ( | self, | |
* | args, | ||
** | kwargs | ||
) |
def lsst.ctrl.pool.pool.PoolMaster.__del__ | ( | self | ) |
|
inherited |
def lsst.ctrl.pool.pool.PoolMaster.cacheClear | ( | self, | |
context | |||
) |
Reset cache for a particular context on master and slaves
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
def lsst.ctrl.pool.pool.PoolMaster.cacheList | ( | self, | |
context | |||
) |
List cache contents for a particular context on master and slaves
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
def lsst.ctrl.pool.pool.PoolMaster.command | ( | self, | |
cmd | |||
) |
def lsst.ctrl.pool.pool.PoolMaster.exit | ( | self | ) |
def lsst.ctrl.pool.pool.PoolMaster.log | ( | self, | |
msg, | |||
* | args | ||
) |
Log a debugging message
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
def lsst.ctrl.pool.pool.PoolMaster.map | ( | self, | |
context, | |||
func, | |||
dataList, | |||
* | args, | ||
** | kwargs | ||
) |
Scatter work to slaves and gather the results.
Work is distributed dynamically, so that slaves that finish quickly will receive more work.
Each slave applies the function to the data they're provided. The slaves may optionally be passed a cache instance, which they can use to store data for subsequent executions (to ensure subsequent data is distributed in the same pattern as before, use the 'mapToPrevious' method). The cache also contains data that has been stored on the slaves.
The 'func' signature should be func(cache, data, *args, **kwargs) if 'context' is non-None; otherwise func(data, *args, **kwargs).
context | Namespace for cache |
func | function for slaves to run; must be picklable |
dataList | List of data to distribute to slaves; must be picklable |
args | List of constant arguments |
kwargs | Dict of constant arguments |
def lsst.ctrl.pool.pool.PoolMaster.mapNoBalance | ( | self, | |
context, | |||
func, | |||
dataList, | |||
* | args, | ||
** | kwargs | ||
) |
Scatter work to slaves and gather the results.
Work is distributed statically, so there is no load balancing.
Each slave applies the function to the data they're provided. The slaves may optionally be passed a cache instance, which they can store data in for subsequent executions (to ensure subsequent data is distributed in the same pattern as before, use the 'mapToPrevious' method). The cache also contains data that has been stored on the slaves.
The 'func' signature should be func(cache, data, *args, **kwargs) if 'context' is true; otherwise func(data, *args, **kwargs).
context | Namespace for cache |
func | function for slaves to run; must be picklable |
dataList | List of data to distribute to slaves; must be picklable |
args | List of constant arguments |
kwargs | Dict of constant arguments |
def lsst.ctrl.pool.pool.PoolMaster.mapToPrevious | ( | self, | |
context, | |||
func, | |||
dataList, | |||
* | args, | ||
** | kwargs | ||
) |
Scatter work to the same target as before.
Work is distributed so that each slave handles the same indices in the dataList as when 'map' was called. This allows the right data to go to the right cache.
It is assumed that the dataList is the same length as when it was passed to 'map'.
The 'func' signature should be func(cache, data, *args, **kwargs).
context | Namespace for cache |
func | function for slaves to run; must be picklable |
dataList | List of data to distribute to slaves; must be picklable |
args | List of constant arguments |
kwargs | Dict of constant arguments |
def lsst.ctrl.pool.pool.PoolMaster.reduce | ( | self, | |
context, | |||
reducer, | |||
func, | |||
dataList, | |||
* | args, | ||
** | kwargs | ||
) |
Scatter work to slaves and reduce the results.
Work is distributed dynamically, so that slaves that finish quickly will receive more work.
Each slave applies the function to the data they're provided. The slaves may optionally be passed a cache instance, which they can use to store data for subsequent executions (to ensure subsequent data is distributed in the same pattern as before, use the 'mapToPrevious' method). The cache also contains data that has been stored on the slaves.
The 'func' signature should be func(cache, data, *args, **kwargs) if 'context' is non-None; otherwise func(data, *args, **kwargs).
The 'reducer' signature should be reducer(old, new). If the 'reducer' is None, then we will return the full list of results
context | Namespace for cache |
reducer | function for master to run to reduce slave results; or None |
func | function for slaves to run; must be picklable |
dataList | List of data to distribute to slaves; must be picklable |
args | List of constant arguments |
kwargs | Dict of constant arguments |
def lsst.ctrl.pool.pool.PoolMaster.reduceNoBalance | ( | self, | |
context, | |||
reducer, | |||
func, | |||
dataList, | |||
* | args, | ||
** | kwargs | ||
) |
Scatter work to slaves and reduce the results.
Work is distributed statically, so there is no load balancing.
Each slave applies the function to the data they're provided. The slaves may optionally be passed a cache instance, which they can store data in for subsequent executions (to ensure subsequent data is distributed in the same pattern as before, use the 'mapToPrevious' method). The cache also contains data that has been stored on the slaves.
The 'func' signature should be func(cache, data, *args, **kwargs) if 'context' is true; otherwise func(data, *args, **kwargs).
The 'reducer' signature should be reducer(old, new). If the 'reducer' is None, then we will return the full list of results
context | Namespace for cache |
reducer | function for master to run to reduce slave results; or None |
func | function for slaves to run; must be picklable |
dataList | List of data to distribute to slaves; must be picklable |
args | List of constant arguments |
kwargs | Dict of constant arguments |
def lsst.ctrl.pool.pool.PoolMaster.reduceToPrevious | ( | self, | |
context, | |||
reducer, | |||
func, | |||
dataList, | |||
* | args, | ||
** | kwargs | ||
) |
Reduction where work goes to the same target as before.
Work is distributed so that each slave handles the same indices in the dataList as when 'map' was called. This allows the right data to go to the right cache.
It is assumed that the dataList is the same length as when it was passed to 'map'.
The 'func' signature should be func(cache, data, *args, **kwargs).
The 'reducer' signature should be reducer(old, new). If the 'reducer' is None, then we will return the full list of results
context | Namespace for cache |
reducer | function for master to run to reduce slave results; or None |
func | function for slaves to run; must be picklable |
dataList | List of data to distribute to slaves; must be picklable |
args | List of constant arguments |
kwargs | Dict of constant arguments |
def lsst.ctrl.pool.pool.PoolMaster.storeClear | ( | self, | |
context | |||
) |
Reset data store for a particular context on master and slaves
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
def lsst.ctrl.pool.pool.PoolMaster.storeDel | ( | self, | |
context, | |||
* | nameList | ||
) |
Delete stored data on slave for a particular context
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
def lsst.ctrl.pool.pool.PoolMaster.storeList | ( | self, | |
context | |||
) |
List store contents for a particular context on master and slaves
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
def lsst.ctrl.pool.pool.PoolMaster.storeSet | ( | self, | |
context, | |||
** | kwargs | ||
) |
Store data on slave for a particular context.
The data is made available to functions through the cache. The stored data differs from the cache in that it is identical for all operations, whereas the cache is specific to the data being operated upon.
context | namespace for store |
kwargs | dict of name=value pairs |
Reimplemented from lsst.ctrl.pool.pool.PoolNode.