Sample-Efficient Multi-Task and
Multi-Objective Reinforcement Learning
by Combining Multiple Behaviors
Advisor: Ana L. C. Bazzan
Co-advisor: Bruno C. da Silva
Reinforcement Learning (RL)
Mastering the game of Go with deep neural networks and tree search. Silver, D., Huang, A., Maddison, C. et al.. Nature 529, (2016)
Autonomous navigation of stratospheric balloons using reinforcement learning. Bellemare, M.G., Candido, S., Castro, P.S. et al.. Nature 588, (2020)
Walk, Run, Crawl, RL Fun. Boston Dynamics (2025)
… and many other success stories (e.g., ChatGPT)
A Long-Term Goal of AI:
How to build generalist agents capable of learning multiple behaviors and solving multiple tasks?
The Real World is Multi-Objective
PhD (2021 – 2025)
- Advisors:
- Prof. Ana Bazzan
- Prof. Bruno C. da Silva (Univ. of Massachusetts)
- Thesis: “Sample-Efficient Multi-Task and Multi-Objective Reinforcement Learning by Combining Multiple Behaviors”
Hypothesis:
It is possible to combine the foundations and ideas of
Multi-Task RL (MTRL) and Multi-Objective RL (MORL)
to address this research problem.
Learning and Combining Behaviors
We introduce novel multi-policy methods that enable RL agents to:
- Learn multiple behaviors
- each specialized in a task or in a preference over objectives
\Pi =
\pi_1
\pi_2
…
\pi_n
- Combine previously-learned behaviors
- to quickly identify solutions to new tasks
\Pi
+
new task
zero-shot
transfer
new behavior
Research Challenges
1. How to construct a set of policies such that combining them directly yields the optimal policy for any task?
Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer (ICML 2022)
2. How to learn a set of policies that solves MORL problems in a robust and sample-efficient way?
Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization (AAMAS 2023)
+ JAAMAS Extended Paper
3. How to use approximate models of the environment to improve zero-shot policy transfer?
Multi-Step Generalized Policy Improvement by Leveraging Approximate Models (NeurIPS 2023)
Contributions
[Contrib. 1] Formal characterization of the connections between MTRL and MORL [ICML 22]
- Extends theoretical and algorithmic ideas across both areas
- Allows solving the Optimal Policy Transfer problem
- SFOLS (Successor Features Optimistic Linear Support)
[Contrib. 2] Novel algorithms that use GPI to define prioritization schemes [AAMAS 23]
- Improve sample efficiency in MORL
- Guarantee identification of the optimal set of policies
- GPI-LS (GPI Linear Support) and GPI-PD (GPI Prioritized Dyna)
- uGPI (Uncertainty-Aware GPI)
[Contrib. 3] Robust model-based method for zero-shot policy transfer [NeurIPS 23]
- Uses approximate models of the environment
- Interpolates between model-free GPI and model-based planning
- h-GPI (Multi-Step GPI)
Two Important Problems in RL
- Multi-task RL (MTRL)
- MDPs (tasks) drawn from a distribution
- Tasks defined by their reward function
r_1 =
r_2 = …
r_3 = +
- Multi-objective RL (MORL)
- Instead of optimizing a single reward function, identifies policies that simultaneously optimize multiple conflicting objectives
These problems have been investigated independently!
\boldsymbol{r} = [, , ]
Each reward component must be traded off!
Combining Behaviors via GPI
\textcolor{#00B050}{\pi_1}
w = [1,0,0]
\textcolor{#0070C0}{\pi_2}
w = [0,1,0]
zero-shot
\pi^{GPI}(s;\textcolor{#ED7D31}{w}) \in \operatorname*{arg\,max}_{\textcolor{#C00000}{a}\in\mathcal{A}}\ \max_{\pi\in\{\textcolor{#00B050}{\pi_1},\textcolor{#0070C0}{\pi_2}\}} q^{\pi}_{\textcolor{#ED7D31}{w}}(s,\textcolor{#C00000}{a})
\textcolor{#ED7D31}{w} = [\textcolor{#00B050}{1},\textcolor{#0070C0}{1},0]
GPI Theorem
q^{GPI}_{\textcolor{#ED7D31}{w}}(s,a) \geq \max_{\textcolor{#0070C0}{\pi}\in\Pi} q^{\textcolor{#0070C0}{\pi}}_{\textcolor{#ED7D31}{w}}(s,a) \qquad \textit{for any } \textcolor{#ED7D31}{w} \in \mathcal{W}
Multi-Objective RL
Agent
Action A_t
Next State S_{t+1}
Reward vector
\textcolor{#ED7D31}{\boldsymbol{R}_t = [R_1 \dots R_m]}
Environment
Multi-Objective RL
Convex Coverage Set (CCS)
\text{CCS} \triangleq \{\mathbf{v}^{\pi} \in \mathcal{F} \mid \exists\, \mathbf{w} \text{ s.t. } \forall \mathbf{v}^{\pi'} \in \mathcal{F},\ \mathbf{v}^{\pi} \cdot \mathbf{w} \geq \mathbf{v}^{\pi'} \cdot \mathbf{w}\}
The optimal policy for any
linear combination of rewards
is in the CCS!
Optimal Policy Transfer
[Contrib. 1]
We can combine ideas from SFs and multi-objective RL
to solve the following previously open problem:
Optimal Policy Transfer
How to construct a set of policies \Pi such that combining them via GPI yields the optimal policy for any new linear task \boldsymbol{w} \in \mathcal{W}?
\Pi =
new task
+\;\;\boldsymbol{w}
GPI
optimal policy
\pi^*_{\boldsymbol{w}}
Contributions
[Contrib. 1] First formal characterization of the connections between multi-objective and multi-task RL
- Enables direct transfer of all theoretical results and algorithms between the two areas
SFOLS: Solves the previously open Optimal Policy Transfer problem:
Reuses previous solutions to directly identify optimal solutions for any new task
\Pi =
New Task
+\;\;\boldsymbol{w}
GPI
Optimal Policy
\pi^*_{\boldsymbol{w}}
Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer (ICML 2022)
Qualis A1
SFOLS: Successor Features Optimistic Linear Support (our method)
[Contrib. 1]
optimistic maximal improvement
- Extension of OLS (Roijers, 2016)
- Corner weights
- optimistic maximal improvement
- Iteratively:
- Learns a new policy \pi_{\textcolor{#ED7D31}{w}}
- Adds \pi_{\textcolor{#ED7D31}{w}} to \Pi
- Converges to a CCS
Experiments: Reacher Task
[Contrib. 1]
\phi(s,a,s') \in \mathbb{R}^4
- SFOLS quickly learns a set of “base” policies that perform well across the whole task space.
Contributions
[Contrib. 2] Novel algorithms for optimal multi-objective decision-making with theoretical guarantees
- Guarantee: optimal (zero-shot) solution for any linear preference over the objectives
- Up to 10x more efficient at identifying optimal solutions than the state of the art
Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization (AAMAS 2023)
+ JAAMAS Extended Paper
Qualis A1
Scaling GPI-LS to Image-Based Tasks
[Contrib. 2]
- MO Super Mario Bros.
- 4 objectives
Time vs. Killing Enemies
- Performance improves as the size of the ensemble of value functions increases
Contributions
[Contrib. 3] Novel method for zero-shot transfer of solutions
- First technique able to interpolate between model-free and model-based approaches
- Robust to approximate models of the system → bounded approximation errors guaranteed
h steps of planning with an approximate model
zero-shot GPI on states reachable in h steps
Training Tasks
zero-shot
Test Task
Multi-Step Generalized Policy Improvement by Leveraging Approximate Models (NeurIPS 2023)
Qualis A1
FourRoom – Zero-Shot Performance
[Contrib. 3]
- h-GPI outperforms the SF-MPC baseline using 10x fewer planning steps
FetchPush – Zero-Shot Performance
[Contrib. 3]
h-GPI outperforms competitors for all values of h using a learned model
Requires a significantly shorter planning horizon
Contributions
[Contrib. 1] Formal characterization of the connections between MTRL and MORL [ICML 22]
- Extends theoretical and algorithmic ideas across both areas
- Allows solving the Optimal Policy Transfer problem
- SFOLS (Successor Features Optimistic Linear Support)
[Contrib. 2] Novel algorithms that use GPI to define prioritization schemes [AAMAS 23]
- Improve sample efficiency in MORL
- Guarantee identification of the optimal set of policies
- GPI-LS (GPI Linear Support) and GPI-PD (GPI Prioritized Dyna)
- uGPI (Uncertainty-Aware GPI)
[Contrib. 3] Robust model-based method for zero-shot policy transfer [NeurIPS 23]
- Uses approximate models of the environment
- Interpolates between model-free GPI and model-based planning
- h-GPI (Multi-Step GPI)
Open-Source Tools
![]()
- API and library of multi-objective decision-making environments
- MORL-Baselines
- Library of multi-objective reinforcement learning algorithms
Widely used by the community and industry:
- >300k downloads and >100 citations
- Transport Network Design (City of Amsterdam), Control of Antennas (Huawei)
A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning (NeurIPS 2023)
Qualis A1
Visiting PhD (2022–2023)
- AI Lab – Vrije Universiteit Brussel (VUB)
- Supervisor: Prof. Ann Nowé
- Group where Multi-Objective RL (MORL) originated
- Publications and Talks
- BNAIC
- EWRL
- ALA
- AAMAS
- NeurIPS (2)
Disney Research
- Novel method that controls the preferences over objectives on-the-fly
- Paper presented at SIGGRAPH 2025
Other Highlights
- Reviewer for the top AI and ML conferences
- NeurIPS Top Reviewer and ICLR Highlighted Reviewer
- Heidelberg Laureate Forum 2024 – (1 of 200 students selected worldwide)
- Supervision of undergraduate and master’s students
- Approved in the public selection for a faculty position (UFRGS)
Acknowledgments
- Advisors
- Prof. Ana L. C. Bazzan (UFRGS)
- Prof. Bruno C. da Silva (UMass)
- Prof. Ann Nowé (VUB)
- Thesis Defense Committee
- Prof. Fredrik Heintz, Prof. Anderson Tavares, Prof. Patrick Mannion
- Collaborators and Colleagues
- Florian Felten, André Barreto, Diederik Roijers, Grégoire Danoy, Willem Röpke, Raphael Avalos, Mathieu Reymond, Florent Delgrange, Roxana Radulescu, Gabriel Ramos, Louis Bagot, Moritz Bächer & Disney Research Zürich, Farama Foundation, …
Sample-Efficient Multi-Task and
Multi-Objective Reinforcement Learning
by Combining Multiple Behaviors
Thank you!
39th CTD – CSBC 2026
July 22, 2026
