lsst.ctrl.pool  22.0.1-2-g92698f7+296381303e
Classes | Functions | Variables
lsst.ctrl.pool.pool Namespace Reference

Classes

class  PickleHolder
 
class  Comm
 
class  NoOp
 
class  Tags
 
class  Cache
 
class  SingletonMeta
 Metaclass to produce a singleton. More...
 
class  Debugger
 
class  ReductionThread
 
class  PoolNode
 
class  PoolMaster
 
class  PoolSlave
 
class  PoolWrapperMeta
 
class  PoolWrapper
 
class  Pool
 

Functions

def unpickleInstanceMethod (obj, name)
 
def pickleInstanceMethod (method)
 
def unpickleFunction (moduleName, funcName)
 
def pickleFunction (function)
 
def getBatchType ()
 
def setBatchType (batchType)
 
def abortOnError (func)
 
def guessPickleObj ()
 
def pickleSniffer (abort=False)
 
def catchPicklingError (func)
 
def startPool (comm=None, root=0, killSlaves=True)
 Start a process pool. More...
 

Variables

string NODE = "%s:%d" % (os.uname()[1], os.getpid())
 

Function Documentation

◆ abortOnError()

def lsst.ctrl.pool.pool.abortOnError (   func)
Function decorator to throw an MPI abort on an unhandled exception

Definition at line 107 of file pool.py.

◆ catchPicklingError()

def lsst.ctrl.pool.pool.catchPicklingError (   func)
Function decorator to catch errors in pickling and print something useful

Definition at line 234 of file pool.py.

◆ getBatchType()

def lsst.ctrl.pool.pool.getBatchType ( )
Return a string giving the type of batch system in use

Definition at line 96 of file pool.py.

◆ guessPickleObj()

def lsst.ctrl.pool.pool.guessPickleObj ( )
Try to guess what's not pickling after an exception

This tends to work if the problem is coming from the
regular pickle module.  If it's coming from the bowels
of mpi4py, there's not much that can be done.

Definition at line 162 of file pool.py.

◆ pickleFunction()

def lsst.ctrl.pool.pool.pickleFunction (   function)
Pickle a function

This assumes that we can recreate the function object by grabbing
it from the proper module.  This may be violated if the function
is a lambda or in __main__.  In that case, I recommend recasting
the function as an object with a __call__ method.

Another problematic case may be a wrapped (e.g., decorated) method
in a class: the 'method' is then a function, and recreating it is
not as easy as we assume here.

Definition at line 71 of file pool.py.

◆ pickleInstanceMethod()

def lsst.ctrl.pool.pool.pickleInstanceMethod (   method)
Pickle an instance method

The instance method is divided into the object and the
method name.

Definition at line 46 of file pool.py.

◆ pickleSniffer()

def lsst.ctrl.pool.pool.pickleSniffer (   abort = False)
Context manager to sniff out pickle problems

If there's a pickle error, you're normally told what the problem
class is.  However, all SWIG objects are reported as "SwigPyObject".
In order to figure out which actual SWIG-ed class is causing
problems, we need to go digging.

Use like this:

    with pickleSniffer():
        someOperationInvolvingPickle()

If 'abort' is True, will call MPI abort in the event of problems.

Definition at line 187 of file pool.py.

◆ setBatchType()

def lsst.ctrl.pool.pool.setBatchType (   batchType)
Return a string giving the type of batch system in use

Definition at line 101 of file pool.py.

◆ startPool()

def lsst.ctrl.pool.pool.startPool (   comm = None,
  root = 0,
  killSlaves = True 
)

Start a process pool.

Returns a PoolMaster object for the master node.
Slave nodes are run and then optionally killed.

If you elect not to kill the slaves, note that they
will emerge at the point this function was called,
which is likely very different from the point the
master is at, so it will likely be necessary to put
in some rank dependent code (e.g., look at the 'rank'
attribute of the returned pools).

Note that the pool objects should be deleted (either
by going out of scope or explicit 'del') before program
termination to avoid a segmentation fault.

@param comm: MPI communicator
@param root: Rank of root/master node
@param killSlaves: Kill slaves on completion?

Definition at line 1216 of file pool.py.

◆ unpickleFunction()

def lsst.ctrl.pool.pool.unpickleFunction (   moduleName,
  funcName 
)
Unpickle a function

This has to be a named function rather than a lambda because
pickle needs to find it.

Definition at line 60 of file pool.py.

◆ unpickleInstanceMethod()

def lsst.ctrl.pool.pool.unpickleInstanceMethod (   obj,
  name 
)
Unpickle an instance method

This has to be a named function rather than a lambda because
pickle needs to find it.

Definition at line 37 of file pool.py.

Variable Documentation

◆ NODE

string lsst.ctrl.pool.pool.NODE = "%s:%d" % (os.uname()[1], os.getpid())

Definition at line 34 of file pool.py.