Experiments Scheduler#

Introduction#

To easily schedule experiment runs palaestrAI offers an experiment scheduler script that allows you to call one or more experiment files and run them in seperate docker containers.

This document serves both as an introduction to how to use the script as well as an more extensive look into the options and components of it.

Options#

The script offers a number of flags and options to customize the scheduler. All of them offer default values if not explicitly set.:

palaestrai-scheduler [OPTION]

  • --build-image (flag) The scheduler needs to have access to a docker-image that comes with all the necessary components to run experiments. This option allows you to build the palaestrAI full-stack image off our repository. Note that you need to have internet access and that it might take awhile.

    By default this builds the up-to-date development branch of palaestrAI, if you want to build a different branch please refer to the command –branch

    Usage example: palaestrai-scheduler –build-image

  • --pull-image (flag) This downloads the pre-built docker-image from the palestrAI repository. If you don’t want to locally build the image this is your easy-to-use alternative. Note that you need to have internet access and that it might take awhile.

    Usage example: palaestrai-scheduler –pull-image

  • --force-download (flag) This also downloads the pre-built docker-image from the palestrAI repository but will delete the local image of the same name, useful if you want to make sure to make a fresh start.

    Usage example: palaestrai-scheduler –force-download

  • --branch (string) Specifies the palaestrAI branch you want to build/download.

    Default value: development

    Usage example: palaestrai-scheduler –build-image –branch=master

  • --docker-image (string) Name of the docker-image you want to utilize for the experiment runs. Please make sure that the image provides all necessary packages, you could build our palaestrAI full-stack image available through containers/palaestrai.Dockerfile

    Default value: palaestrai/palaestrai

    Usage example: –docker-image=docker_image_42

  • --docker-volume (string) Name of the docker-volume you want to utilize for the experiment runs. If not provided the script will create a docker volume on the fly and work with that.

    Usage example: –docker-volume=palaestrai_volume

  • --experiments (string) A path to the experiment(s) you want to run. This can be a specific file or a directory. Note that the scheduler, if provided with an directory, will look up any y*ml file in the path recursively and checks them for the proper experiment-file syntax.

    Default value: Current working directory

    Usage example: –experiments=~/my_experiments

  • --gpu-dialog (flag) If you call the script with this flag it will trigger a dialogue that lets you choose the GPUs you want to be utilized by the scheduler. Note that choosing your GPUs with this interactive dialogue requires you to run the script in an active terminal window. The available GPUs will be listed in the terminal with their corresponding IDs. Type in the IDs of the GPUs you want to choose (e.g. 2,4,6-8)

    Usage example : palaestrai-scheduler –experiments=~/my_experiments –gpu-dialog

  • --gpus (flag) This allows you to explicitly choose the GPUs you want to utilize by providing their IDs (see –gpu-dialog). This is useful if you don’t want to go through the gpu-dialog each time and/or you’re not using an interactive terminal.

    Usage example: palaestrai-scheduler –experiments=~/my_experiments –gpus=1,4,7

  • --experiments-per-gpu (integer) This sets the amount of experiments that can be run in parallel per GPU. The scheduler will check for finished experiments and distribute more experiments accordingly.

    Default value: 4

    Usage example: palaestrai-scheduler –experiments=~/my_experiments –experiments-per-gpu=6

  • --memory-limit (integer) Before starting a new experiment the scheduler will check for the available memory. This value indicates the amount of memory in percent, that needs to be free to continue scheduling experiments.

    Default value: 35 (e.g. 35% of memory has to be free)

    Usage example: –memory-limit=20

  • --help Prints a help page to the terminal that shows an overview of the available options and flags.

Full example run:

palaestrai-scheduler --experiments=~/my_experiments --experiments-per-gpu=5 --gpus