API Reference

The recommended way to use the API is to start by looking at the final function write_config_to_file() and work backwards to all the constituent functions.

distribute_compute_config.apptainer_config(meta: Meta, description: Description) → ApptainerConfig

assemble the metadata() and description() into a config file that can be written to disk

Parameters:

You can write this description object to disk with write_config_to_file()

distribute_compute_config.metadata(namespace: str, batch_name: str, capabilities: List[str], matrix_username: str | None = None) → Meta

construct the metadata Meta object for information about what this job batch will run

Parameters:

  • namespace (str) – namespace where several batch_name runs may live

  • batch_name (str) – the name of this batch of jobs

  • capabilities (List[str]) – required capabilities for your job

  • matrix_username (Optional[str]) – a matrix username in the format @your_username:homeserver_url

for an apptainer job the capabilities are is simply ["apptainer"]. If you need GPU capabilities, use ["apptainer", "gpu"].

Example:

import distribute_compute_config as distribute
matrix_user = "@matrx_id:matrix.org"
batch_name = "test_batch"
namespace = "test_namespace"
capabilities = ["apptainer"]

meta = distribute.metadata(namespace, batch_name, capabilities, matrix_user)
distribute_compute_config.description(initialize: Initialize, jobs: List[Job]) → Description

combines initialization information with a list of jobs to describe the solver files, how they should be started, and what they will run

Parameters:

  • initialize – You can create an Initialize struct with initialize()

  • jobs – information for each job can be created from the job() function.

Example:

import distribute_compute_config as distribute

initialize = distribute.initialize(
    sif_path="./some/path/to/file.sif",
    required_files=[],
    required_mounts=[]
)

job_1_config_file = distribute.file(
    "./path/to/config1.json",
    alias="config.json",
    relative=True
)

job_1 = distribute.job("job_1", [job_1_config_file])

description = distribute.description(initialize, jobs=[job_1])
distribute_compute_config.file(path: str, relative=False, alias=None) → File

create a file to appear in the /input directory of the solver

Parameters:

  • path – a path to the file on disk. path should be an absolute, unless relative=True is also specified. If relative=True is not speficied, the full directory structure to the file must already exist.

  • relative – a flag for if the path specified is a relative path or not. Defaults to False

  • alias – how the file should be renamed when it appears in the /input directory of your solver. If no alias is specified, the current name of the file will be used.

For example, a file with a path ./path/to/config_1.json will appear as /input/config_1.json. with alias="config.json", this file will appear as /input/config.json

Example

import distribute_compute_config as config

# config1.json appears as `/input/config.json`
config_file_1 = distribute.file(
    "./path/to/config1.json",
    alias="config.json",
    relative=True
)

# config2.json appears as `/input/config2.json`
config_file_2 = distribute.file(
    "./path/to/config2.json",
    relative=True
)

# config3.json appears as `/input/config3.json`, folder structure
# `/root/path/to/` must already exist
config_file_3 = distribute.file("/root/path/to/config3.json")
distribute_compute_config.initialize(sif_path: str, required_files: List[File], required_mounts: List[str]) → Initialize

create the initialize section for loading apptainer .sif files, required container mounts, and input files present in all runs.

Also see the documentation for creating a File with the file() function

Parameters:

  • sif_path – The path to the .sif file produced by apptainer build

  • required_files – These files appear in the /input directory of every job, in addition to the File specified in each Job

  • required_mounts – a list of strings to paths inside the container that should be mutable.

Example:

import distribute_compute_config as distribute

sif_path = "./path/to/some/container.sif"

initial_condition = distribute.file(
    "./path/to/some/file.h5",
    relative=True,
    alias="initial_condition.h5"
)
required_files = [initial_condition]

required_mounts = ["/solver/extra_mount"]

initialize = distribute.initialize(sif_path, required_files, required_mounts)
distribute_compute_config.job(name: str, required_files: List[File]) → Job

creates a job from its name and the required input files

once you have a list of jobs that should be run in the batch, move on to creating a description()

Also see the documentation for creating a File with the file() function

Parameters:

  • name – the name of the job. It should be unique in combination with the batch_name of this job.

  • required_files – a python list of files that should appear in the /input directory when this job is run, along with the required_files specified in initialize().

name should therefore be unique to this batch since batch_name remains constant. the name should be slightly descriptive of the content of what the job will be running. This will make it easier to use distribute pull to download the files later

File types can be constructed with the file() function

Example:

import distribute_compute_config as distribute

job_1_config_file = distribute.file(
    "./path/to/config1.json",
    alias="config.json",
    relative=True
)
job_1_required_files = [job_1_config_file]

job_1 = distribute.job("job_1", job_1_required_files)
distribute_compute_config.write_config_to_file(config: ApptainerConfig, path: str)

write an apptainer_config() to a path

Parameters:

  • config – output of apptainer_config() function

  • path – the path that the config file should be written to, usually with the name distribute-jobs.yaml

Example:

See the User Documentation page in on the python api for a worked example