Install and use
Note
You may use Docker to run the benchmarks, which will likely be easier. See the Docker section of this documentation for more information.
To install, clone the repo:
# You may need to upgrade pip
pip install pip -U
git clone git@github.com:mila-iqia/milabench.git
cd milabench
# <Activate virtual environment>
# Install in editable mode
pip install -e .
This will install two commands, milabench
and voir
.
Before running the benchmarks
Set the
$MILABENCH_BASE
environment variable to the base directory in which all the code, virtual environments and data should be put.Set the
$MILABENCH_CONFIG
environment variable to the configuration file that represents the benchmark suite you want to run. Normally it should be set toconfig/standard.yaml
.milabench install
: Install the individual benchmarks in virtual environments.milabench prepare
: Download the datasets, weights, etc.
If the machine has both NVIDIA/CUDA and AMD/ROCm GPUs, you may have to set the MILABENCH_GPU_ARCH
environment variable as well, to either cuda
or rocm
.
Run milabench
The following command will run the whole benchmark and will put the results in a new directory in $MILABENCH_BASE/runs
(the path will be printed to stdout).
milabench run
Here are a few useful options for milabench run
:
# Only run the bert benchmark
milabench run --select bert
# Run all benchmarks EXCEPT bert and stargan
milabench run --exclude bert,stargan
# Run the benchmark suite three times in a row
milabench run --repeat 3
Reports
The following command will print out a report of the tests that ran, the metrics and if there were any failures. It will also produce an HTML report that contains more detailed information about errors if there are any.
milabench report --runs $MILABENCH_BASE/runs/some_specific_run --html report.html
The report will also print out a score based on a weighting of the metrics, as defined in the file $MILABENCH_CONFIG
points to.