tardis.utilities.asyncbulkcall module

class tardis.utilities.asyncbulkcall.AsyncBulkCall(command: BulkCommand[T, R], size: int, delay: float, concurrent: Optional[int] = None)[source]

Bases: Generic[T, R]

Framework for queueing and executing several tasks via bulk commands

Parameters

  • command – async callable that executes several tasks

  • size – maximum number of tasks to execute in one bulk

  • delay – maximum time window for tasks to execute in one bulk

  • concurrent – how often the command may be executed at the same time

Given some bulk-task callable (T, ...) -> (R, ...) (the command), BulkExecution represents a single-task callable (T) -> R. Single-task calls are buffered for a moment according to size and delay, then executed in bulk with concurrent calls to command.

Each BulkExecution should represent a different command (for example, rm or mkdir) collecting similar tasks (for example, rm foo and rm bar to rm foo bar). The command is an arbitrary async callable and can freely decide how to handle its tasks. The BulkExecution takes care of collecting individual tasks, partitioning them to bulks, and translating the results of bulk execution back to individual tasks.

Both size and delay control how long to queue tasks at most before starting to execute them. The concurrent parameter controls how many bulks may run at once; when concurrency is low tasks may be waiting for execution even past size and delay. Possible values for concurrent are None for unlimited concurrency or an integer above 0 to set a precise concurrency limit.

Note

If the command requires additional arguments, wrap it via partial(), for example AsyncBulkCall(partial(async_rm, force=True), ...).

class tardis.utilities.asyncbulkcall.BulkCommand(*args, **kwargs)[source]

Bases: Protocol[T, R]

Protocol of callables suitable for BulkExecution

A bulk command must take an arbitrary number of tasks and is expected to provide an iterable of one result per task. Alternatively, it may provide a single None value to indicate that there is no result. An unhandled Exception means that all tasks failed with that Exception.