syncopy.shared.kwarg_decorators.process_io

syncopy.shared.kwarg_decorators.process_io(func)

Decorator for handling parallel execution of a computeFunction()

Parameters:

func (callable) – A Syncopy computeFunction()

Returns:

wrapper_io – Wrapped function; wrapper_io changes the way it invokes the wrapped computeFunction and processes its output based on the type of the provided first positional argument trl_dat.

• trl_dat : dict Wrapped computeFunction is executed concurrently; trl_dat was assembled by compute_parallel() and contains information for parallel workers (particularly, paths and dataset indices of HDF5 files for reading source data and writing results). Nothing is returned (the output of the wrapped computeFunction is directly written to disk).

• trl_dat : numpy.ndarray or FauxTrial object Wrapped computeFunction is executed sequentially (either during dry- run phase or in purely sequential computations); trl_dat is directly propagated to the wrapped computeFunction and its output is returned (either a tuple or numpy.ndarray, depending on the value of noCompute, see computeFunction() for details)

Return type:

callable

Notes

Parallel execution supports two writing modes: concurrent storage of results in multiple HDF5 files or sequential writing of array blocks in a single output HDF5 file. In both situations, the output array returned by computeFunction() is immediately written to disk and not propagated back to the caller to avoid inter-worker network communication.

In case of parallel writing, trial-channel blocks are stored in individual HDF5 files (virtual sources) that are consolidated into a single h5py.VirtualLayout which is subsequently used to allocate a virtual dataset inside a newly created HDF5 file (located in Syncopy’s temporary storage folder).

Conversely, in case of sequential writing, each resulting array is written sequentially to an existing single output HDF5 file using a distributed mutex for access control to prevent write collisions.

See also

unwrap_cfg

Decorator for processing cfg “structs”