syncopy.shared.kwarg_decorators.detect_parallel_client#

syncopy.shared.kwarg_decorators.detect_parallel_client(func)[source]#

Decorator for handling parallelization via parallel keyword/client detection

Any already initialized Dask cluster always takes precedence with both parallel=True and parallel=None. This gets checked via dd.get_client(), and hence if a Dask cluster was set up before, Syncopy (and also potentially ACME later) will just pass-through this one to the compute classes. In case no cluster is running, only a dedicated parallel=True will spawn either a new Dask cluster down the road via ACME (if on a slurm cluster) or a new LocalCluster as a default fallback. The LocalCluster gets closed again after the wrapped function exited.

If parallel is None:

First attempts to connect to a running dask parallel processing client. If successful, parallel is set to True and updated in func’s keyword argument dict. If no client is found parallel is set to False

If parallel is True and ACME is installed AND we are on a slurm cluster:

Do nothing and forward all the parallelization setup with parallel=True to the CR and ultimately ACME

If parallel is True and ACME is NOT installed OR we ar NOT on a slurm cluster:

Fire up a standard dask LocalCluster and forward parallel=True to func

Parameters:

func (callable) – Typically a Syncopy metafunction such as freqanalysis()

Returns:

parallel_client_detector – Wrapped function; parallel_client_detector attempts to extract parallel from keywords provided to func.

After successfully calling func with the modified input arguments, parallel_client_detector modifies func itself:

  1. The “Parameters” section in the docstring of func is amended by an entry explaining the usage of parallel. Note: func’s docstring is only extended if it has a “Parameters” section.

  2. If not already present, parallel is added as optional keyword (with default value None) to the signature of func.

Return type:

callable

Notes

This decorator assumes that func has already been processed by unwrap_cfg() and hence expects func to obey standard Python call signature func(*args, **kwargs). In other words, detect_parallel_client() is intended as “inner” decorator of, e.g., metafunctions. See Notes in the docstring of unwrap_select() for further details.

See also

unwrap_select

extract select keyword and process in-place data-selections

unwrap_cfg

Decorator for processing cfg “structs”