syncopy.shared.kwarg_decorators.detect_parallel_client#
- syncopy.shared.kwarg_decorators.detect_parallel_client(func)[source]#
Decorator for handling parallelization via parallel keyword/client detection
Any already initialized Dask cluster always takes precedence with both parallel=True and parallel=None. This gets checked via dd.get_client(), and hence if a Dask cluster was set up before, Syncopy (and also potentially ACME later) will just pass-through this one to the compute classes. In case no cluster is running, only a dedicated parallel=True will spawn either a new Dask cluster down the road via ACME (if on a slurm cluster) or a new LocalCluster as a default fallback. The LocalCluster gets closed again after the wrapped function exited.
- If parallel is None:
First attempts to connect to a running dask parallel processing client. If successful, parallel is set to True and updated in func’s keyword argument dict. If no client is found parallel is set to False
- If parallel is True and ACME is installed AND we are on a slurm cluster:
Do nothing and forward all the parallelization setup with parallel=True to the CR and ultimately ACME
- If parallel is True and ACME is NOT installed OR we ar NOT on a slurm cluster:
Fire up a standard dask LocalCluster and forward parallel=True to func
- Parameters:
func (callable) – Typically a Syncopy metafunction such as
freqanalysis()
- Returns:
parallel_client_detector – Wrapped function; parallel_client_detector attempts to extract parallel from keywords provided to func.
After successfully calling func with the modified input arguments, parallel_client_detector modifies func itself:
The “Parameters” section in the docstring of func is amended by an entry explaining the usage of parallel. Note: func’s docstring is only extended if it has a “Parameters” section.
If not already present, parallel is added as optional keyword (with default value None) to the signature of func.
- Return type:
callable
Notes
This decorator assumes that func has already been processed by
unwrap_cfg()
and hence expects func to obey standard Python call signaturefunc(*args, **kwargs)
. In other words,detect_parallel_client()
is intended as “inner” decorator of, e.g., metafunctions. See Notes in the docstring ofunwrap_select()
for further details.See also
unwrap_select
extract select keyword and process in-place data-selections
unwrap_cfg
Decorator for processing cfg “structs”