executor
Custom version of Python ThreadPoolExecutor).
FractalThreadPoolExecutor
¶
Bases: ThreadPoolExecutor
Custom version of
ThreadPoolExecutor)
that overrides the submit
and map
methods
Source code in fractal_server/app/runner/v1/_local/executor.py
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 |
|
map(fn, *iterables, local_backend_config=None)
¶
Custom version of the Executor.map
method
The main change with the respect to the original map
method is that
the list of tasks to be executed is split into chunks, and then
super().map
is called (sequentially) on each chunk. The goal of this
change is to limit parallelism, e.g. due to limited computational
resources.
Other changes from the concurrent.futures
map
method:
- Removed
timeout
argument; - Removed
chunksize
; - All iterators (both inputs and output ones) are transformed into lists.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
fn |
Callable
|
A callable function. |
required |
iterables |
Sequence[Iterable]
|
The argument iterables (one iterable per argument of
|
()
|
local_backend_config: The backend configuration, needed to extract
parallel_tasks_per_job
.
Source code in fractal_server/app/runner/v1/_local/executor.py
45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 |
|
submit(*args, local_backend_config=None, **kwargs)
¶
Compared to the ThreadPoolExecutor
method, here we accept an addition
keyword argument (local_backend_config
), which is then simply
ignored.
Source code in fractal_server/app/runner/v1/_local/executor.py
32 33 34 35 36 37 38 39 40 41 42 43 |
|