MORL-Baselines: A collection of multi-objective reinforcement learning algorithms.

Multiple policies

MORL-Baselines is a library of Multi-Objective Reinforcement Learning (MORL) algorithms. This repository aims to contain reliable MORL algorithms implementations in PyTorch.

It strictly follows MO-Gymnasium API, which differs from the standard Gymnasium API only in that the environment returns a numpy array as the reward.

For details on multi-objective MDPs (MOMDPs) and other MORL definitions, we suggest reading A practical guide to multi-objective reinforcement learning and planning. An overview of some techniques used in various MORL algorithms is also provided in Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Framework.

A tutorial on MO-Gymnasium and MORL-Baselines is also available: Open in Colab

Features of MORL-Baselines

  • Single and multi-policy algorithms under both SER and ESR criteria are implemented.

  • All algorithms follow the MO-Gymnasium API.

  • Performances are automatically reported in Weights and Biases dashboards.

  • Linting and formatting are enforced by pre-commit hooks.

  • Code is well documented.

  • All algorithms are automatically tested.

  • Utility functions are provided e.g. pareto pruning, experience buffers, etc.

  • Performances have been tested and reported in a reproducible manner.

  • Hyperparameter optimization available.

Benchmarks

MORL-Baselines participates to Open RL Benchmark which contains tracked experiments from popular RL libraries such as cleanRL and Stable Baselines 3.

We have run experiments of our algorithms on various environments from MO-Gymnasium. The results can be found here: https://wandb.ai/openrlbenchmark/MORL-Baselines. An issue tracking all the settings is available at #43. Some design documentation for the experimentation protocol are also available on our Documentation website.

Here is an example of our dashboards with Pareto support in WandB:

WandB dashboards

Citing MORL-Baselines

If you use MORL-Baselines in your research, please cite our NeurIPS 2023 paper:

@inproceedings{felten_toolkit_2023,
	author = {Felten, Florian and Alegre, Lucas N. and Now{\'e}, Ann and Bazzan, Ana L. C. and Talbi, El Ghazali and Danoy, Gr{\'e}goire and Silva, Bruno Castro da},
	title = {A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning},
	booktitle = {Proceedings of the 37th Conference on Neural Information Processing Systems ({NeurIPS} 2023)},
	year = {2023}
}