lsst.ctrl.pool  13.0-5-g9cf35e0+9
 All Classes Namespaces Files Functions Variables Pages
Classes | Functions | Variables
lsst.ctrl.pool.pool Namespace Reference

Classes

class  PickleHolder
 
class  Comm
 
class  NoOp
 
class  Tags
 
class  Cache
 
class  SingletonMeta
 Metaclass to produce a singleton. More...
 
class  Debugger
 
class  ReductionThread
 
class  PoolNode
 
class  PoolMaster
 
class  PoolSlave
 
class  PoolWrapperMeta
 
class  PoolWrapper
 
class  Pool
 

Functions

def unpickleInstanceMethod
 
def pickleInstanceMethod
 
def unpickleFunction
 
def pickleFunction
 
def getBatchType
 
def setBatchType
 
def abortOnError
 
def guessPickleObj
 
def pickleSniffer
 
def catchPicklingError
 
def startPool
 Start a process pool. More...
 

Variables

list __all__ = ["Comm", "Pool", "startPool", "setBatchType", "getBatchType", "abortOnError", "NODE", ]
 
string NODE = "%s:%d"
 
string _batchType = "unknown"
 

Function Documentation

def lsst.ctrl.pool.pool.abortOnError (   func)
Function decorator to throw an MPI abort on an unhandled exception

Definition at line 108 of file pool.py.

def lsst.ctrl.pool.pool.catchPicklingError (   func)
Function decorator to catch errors in pickling and print something useful

Definition at line 232 of file pool.py.

def lsst.ctrl.pool.pool.getBatchType ( )
Return a string giving the type of batch system in use

Definition at line 99 of file pool.py.

def lsst.ctrl.pool.pool.guessPickleObj ( )
Try to guess what's not pickling after an exception

This tends to work if the problem is coming from the
regular pickle module.  If it's coming from the bowels
of mpi4py, there's not much that can be done.

Definition at line 161 of file pool.py.

def lsst.ctrl.pool.pool.pickleFunction (   function)
Pickle a function

This assumes that we can recreate the function object by grabbing
it from the proper module.  This may be violated if the function
is a lambda or in __main__.  In that case, I recommend recasting
the function as an object with a __call__ method.

Another problematic case may be a wrapped (e.g., decorated) method
in a class: the 'method' is then a function, and recreating it is
not as easy as we assume here.

Definition at line 76 of file pool.py.

def lsst.ctrl.pool.pool.pickleInstanceMethod (   method)
Pickle an instance method

The instance method is divided into the object and the
method name.

Definition at line 52 of file pool.py.

def lsst.ctrl.pool.pool.pickleSniffer (   abort = False)
Context manager to sniff out pickle problems

If there's a pickle error, you're normally told what the problem
class is.  However, all SWIG objects are reported as "SwigPyObject".
In order to figure out which actual SWIG-ed class is causing
problems, we need to go digging.

Use like this:

    with pickleSniffer():
        someOperationInvolvingPickle()

If 'abort' is True, will call MPI abort in the event of problems.

Definition at line 186 of file pool.py.

def lsst.ctrl.pool.pool.setBatchType (   batchType)
Return a string giving the type of batch system in use

Definition at line 103 of file pool.py.

def lsst.ctrl.pool.pool.startPool (   comm = None,
  root = 0,
  killSlaves = True 
)

Start a process pool.

Process Pool

Use this class to automatically provide 'context' to
the PoolMaster class.  If you want to call functions
that don't take a 'cache' object, use the PoolMaster
class directly, and specify context=None.

Returns a PoolMaster object for the master node. Slave nodes are run and then optionally killed.

If you elect not to kill the slaves, note that they will emerge at the point this function was called, which is likely very different from the point the master is at, so it will likely be necessary to put in some rank dependent code (e.g., look at the 'rank' attribute of the returned pools).

Note that the pool objects should be deleted (either by going out of scope or explicit 'del') before program termination to avoid a segmentation fault.

Parameters
comm,:MPI communicator
root,:Rank of root/master node
killSlaves,:Kill slaves on completion?

Definition at line 1214 of file pool.py.

def lsst.ctrl.pool.pool.unpickleFunction (   moduleName,
  funcName 
)
Unpickle a function

This has to be a named function rather than a lambda because
pickle needs to find it.

Definition at line 65 of file pool.py.

def lsst.ctrl.pool.pool.unpickleInstanceMethod (   obj,
  name 
)
Unpickle an instance method

This has to be a named function rather than a lambda because
pickle needs to find it.

Definition at line 43 of file pool.py.

Variable Documentation

list lsst.ctrl.pool.pool.__all__ = ["Comm", "Pool", "startPool", "setBatchType", "getBatchType", "abortOnError", "NODE", ]

Definition at line 38 of file pool.py.

string lsst.ctrl.pool.pool._batchType = "unknown"

Definition at line 97 of file pool.py.

string lsst.ctrl.pool.pool.NODE = "%s:%d"

Definition at line 40 of file pool.py.