TARDIS - The Transparent Adaptive Resource Dynamic Integration System¶
Welcome to the TARDIS documentation!¶
The TARDIS
resource manager enables the dynamic integration of resources provided by different
resource providers into one overlaybatch system.
TARDIS
relies on COBalD - the Opportunistic Balancing Daemon in order
to balance opportunistic resources. Hence, TARDIS
is implemented as a COBalD
service.
Installation¶
The TARDIS
resource manager is available via PyPI, so you can use
pip to install TARDIS
and all its dependencies.
python3 -m pip install cobald-tardis
Configuration of COBalD¶
In order to run TARDIS
a COBalD
configuration is needed. Details about the available options and syntax can
be found in the COBalD component configuration documentation.
The pools to be
managed by TARDIS
are created by a factory function
create_composite_pool()
, which is registered as a
COBalD YAML plugin
and, therefore, can be included using the !TardisPoolFactory
YAML tag.
Example configuration
pipeline:
# Makes decision to add remove resources based utilisation and allocation
- !LinearController
low_utilisation: 0.90
high_allocation: 0.90
rate: 1
# Limits the demand for a resource
- !Limiter
minimum: 1
# Log changes
- !Logger
name: 'changes'
# Factory function to create composite resource pool
- !TardisPoolFactory
configuration: 'tardis.yml'
Configuration of TARDIS¶
In addition to the above mentioned COBalD
configuration a TARDIS
configuration is necessary as well,
which is provided to the factory function create_composite_pool()
via
its configuration parameter.
The TARDIS
YAML configuration supports the following sections:
Section |
Short Description |
Requirement |
---|---|---|
Plugins |
Configuration of the Plugins to use (see Plugins) |
Optional |
BatchSystem |
The overlay batch system to use (see BatchSystemAdapter) |
Required |
Sites |
List of sites to create (see Generic Site Configuration) |
Required |
Site Sections |
Configuration options for each site (see Generic Site Configuration) |
Required |
Example configuration
Plugins:
SqliteRegistry:
db_file: drone_registry.db
BatchSystem:
adapter: FakeBatchSystem
allocation: 1.0
utilisation: !TardisPeriodicValue
period: 3600
amplitude: 0.5
offset: 0.5
phase: 0.
machine_status: Available
Sites:
- name: Fake
adapter: FakeSite
quota: 8000 # CPU core quota
Fake:
api_response_delay: !TardisRandomGauss
mu: 0.1
sigma: 0.01
resource_boot_time: !TardisRandomGauss
mu: 60
sigma: 10
MachineTypes:
- m1.infinity
MachineTypeConfiguration:
m1.infinity:
MachineMetaData:
m1.infinity:
Cores: 8
Memory: 16
Disk: 160
Unified Configuration¶
Alternatively a unified COBalD
and TARDIS
configuration can be used. In this case, the TARDIS
part of the configuration is represented by a tardis
MappingNode.
Warning
In case of the unified configuration you can currently not use the yaml tag !TardisPoolFactory
to initialize
the pool factory, please use the COBalD legacy object initialisation
__type__: tardis.resources.poolfactory.create_composite_pool
instead!
Example configuration
pipeline:
# Makes decision to add remove resources based utilisation and allocation
- !LinearController
low_utilisation: 0.90
high_allocation: 0.90
rate: 1
# Limits the demand for a resource
- !Limiter
minimum: 1
# Log changes
- !Logger
name: 'changes'
# Factory function to create composite resource pool
- __type__: tardis.resources.poolfactory.create_composite_pool
tardis:
Plugins:
SqliteRegistry:
db_file: drone_registry.db
BatchSystem:
adapter: FakeBatchSystem
allocation: 1.0
utilization: !TardisPeriodicValue
period: 3600
amplitude: 0.5
offset: 0.5
phase: 0.
machine_status: Available
Sites:
- name: Fake
adapter: FakeSite
quota: 8000 # CPU core quota
Fake:
api_response_delay: !TardisRandomGauss
mu: 0.1
sigma: 0.01
resource_boot_time: !TardisRandomGauss
mu: 60
sigma: 10
MachineTypes:
- m1.infinity
MachineTypeConfiguration:
m1.infinity:
MachineMetaData:
m1.infinity:
Cores: 8
Memory: 16
Disk: 160
Start-up your instance¶
To start-up your instance you can run the following command:
python3 -m cobald.daemon cobald.yml
However, it is recommended to start COBalD
using systemd as decribed in the
COBalD Systemd Configuration documentation.
Running your instance in Docker¶
For your convenience and to try out COBalD
/TARDIS
ready to use docker containers are provided via Dockerhub.
To start a demo setup managing fake resources and a fake batch system simply run the following command.
docker run matterminers/cobald-tardis:latest
To run your own COBalD
/TARDIS
configuration you can run the following command.
docker run --mount type=bind,source=</your_path/cobald.yml>,target=/srv/cobald.yml matterminers/cobald-tardis:latest
To use the Sqlite Database to persistently store the state of your resources you can export an entire path to the container by running the following command.
docker run -v </your_path>:/srv matterminers/cobald-tardis:latest
Note
This path must contain a valid cobald.yml file. The persistent database will then be stored on the local machine.
Running your instance using HTCondor as overlay batch system in Docker¶
For your convenience and for systems without pre-compiled HTCondor support a ready to use docker container is provided via Dockerhub.
You can configure COBalD
/TARDIS
as described in this documentation and bind mount the directory containing
the configuration into the containers /srv directory.
Note
This path must contain at least a valid cobald.yml file.
docker run -v $PWD/configuration:/srv matterminers/cobald-tardis-htcondor:latest
HTCondor Token Support¶
Starting with the production release series 9.0, HTCondor introduces a new security configuration, which is no longer host-based. The security configuration is now user-based and requires authentication to access the HTCondor pool. This is also true for read-only operations such as condor_status. Therefore, this docker image supports the IDTOKENS authentication method introduced with the HTCondor 9.0 series.
In order to use ID tokens, add any tokens provided by the operator of the overlay batch system to a tokens.d directory and bind mount it to /etc/condor/tokens.d. HTCondor is automatically using them to authenticate against the pool.
docker run -v $PWD/configuration:/srv -v $PWD/tokens.d:/etc/condor/tokens.d matterminers/cobald-tardis-htcondor:latest
Note
Since COBalD
/TARDIS
uses the condor_status command, the token added needs at least the ALLOW_READ
privilege to access the HTCondor Collector and to query the status of resources.
In addition, COBalD
/TARDIS
uses the condor_drain command to release under utilized resources.
Therefore, a second token to access the HTCondor StartD of the Drone is needed.
Usually, both tokens are provided by the operator of the HTCondor overlay batch system.