Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning

Saeed Hedayatian1  ·  Harsh Shah1  ·  Celine Mneimneh2  ·  Stefanos Nikolaidis1

1Thomas Lord Department of Computer Science, University of Southern California    2Independent Researcher

Preprint, 2026

TL;DR RePPO turns the gradient of a quality-diversity objective into a per-policy shaped reward, so that a small population of ordinary PPO learners ends up both high-performing and behaviorally distinct, without an archive or evolutionary operators. It gives the best quality–diversity tradeoff on four Brax tasks and scales to Atari from pixels and to LLM fine-tuning.

Diagram: a population of policies in a two-dimensional behavior space before and after training. A zoom on one member shows its repulsion direction from nearby high-quality members and the direction of higher return; the shaped reward is the task reward plus the repulsion term.
Figure 1. One RePPO iteration in behavior space, shaded by return. Each member k is pushed away from nearby high-return members (zoom), and PPO maximizes its shaped reward \(\tilde r_k = r + p_k^{\top} m\). After training, the members occupy separate high-return regions.

Foot contact: left foot right foot

Figure 2. Policies found by RePPO (populations of N = 32) on four Brax tasks, in real time. Under each clip are the policy's mean return and the fraction of time each foot is on the ground (full bar = 100%), both over 256 evaluation episodes. Within each task, the clips come from one training run, except those marked †, which come from other seeds.

The problem

Standard RL looks for one policy that maximizes expected return, even when a task admits qualitatively different solutions. Quality-Diversity RL (QD-RL) instead asks for a population of high-performing policies that differ along a user-defined behavior descriptor, such as a robot's foot-contact pattern or where an Atari agent plays. The descriptor states which kind of diversity is wanted and makes it an explicit part of the learning problem.

Most QD-RL methods organize the search around an archive in the style of MAP-Elites: behavior space is partitioned into cells, each cell keeps its best solution, and evolutionary operators, increasingly combined with RL gradients, fill it. The number of cells grows quickly with descriptor dimension and resolution, and every cell holds a full policy, which becomes impractical once policies are convolutional networks or language-model adapters. We therefore ask whether quality and diversity can be optimized directly over a compact population, with standard RL updates and no discretization of behavior space.

How RePPO works

A continuous objective over a finite population. We start from the Soft QD lower bound (Hedayatian & Nikolaidis, 2026). For policies with qualities \(f_n > 0\) (expected return) and descriptors \(b_n \in \mathbb{R}^d\),

\[ S(\theta) \;=\; \sum_{n=1}^{N} f_n \;-\; \sum_{i<j} \sqrt{f_i f_j}\;\exp\!\Big(-\tfrac{\|b_i - b_j\|_2^2}{\ell^2}\Big). \]

The first term rewards individual quality. The second penalizes pairs of policies that are both good and behaviorally close, and the bandwidth \(\ell\) sets how far this interaction reaches. Diversity comes from interactions among a fixed number of members rather than from archive cells.

Population gradients are shaped rewards. Behavior cannot be differentiated through the environment. But when a descriptor is the expected sum of a per-step behavior signal \(m\), \(b(\theta) = \mathbb{E}\big[\sum_t \gamma^t m(s_t,a_t)\big]\), it has the same form as a return. Changing a policy's behavior in a direction \(v\) is then an ordinary policy gradient with reward \(v^{\top} m\). Differentiating \(S\) gives each policy a direction that points away from its nearby high-quality neighbors. RePPO computes this direction \(p_k\) from smoothed episode statistics, with a few stabilizers described in the paper. Each policy's shaped reward is

\[ \tilde r_k(s,a) \;=\; r(s,a) \;+\; p_k^{\top} m(s,a). \]

Then each member runs ordinary PPO. Each policy has a task critic and a behavior critic. Their advantages are combined with the weights \(p_k\) and fed to the usual clipped PPO update. One iteration:

  1. Collect on-policy rollouts for all N policies, recording task and behavior signals.
  2. Update each policy's smoothed return and descriptor, and compute \(p_k\) from the current population.
  3. Run PPO epochs for each policy with \(p_k\) held fixed.

Setting \(p_k = 0\) recovers N independent PPO runs, which we call Ind-PPO.

Results

We describe a population with two numbers. Its mean fitness (average return) measures quality. The Vendi Score of its descriptors measures diversity and can be read as the effective number of distinct behaviors. Their product, the quality-weighted Vendi Score (qVS), summarizes both. All numbers are computed after filtering out policies that fail at the task. Throughout, blue is RePPO and green is Ind-PPO.

Continuous control

On four Brax locomotion tasks, we compare RePPO with Ind-PPO and four archive-based QD-RL methods. RePPO attains the highest qVS on all four tasks. Ind-PPO finds strong policies that all behave alike. The archive-based methods cover behavior space, but most of their policies are weak. RePPO gets both quality and diversity.

Four scatter plots, one per Brax task, of mean fitness against Vendi Score for RePPO, DCRL-ME, PPGA, PGA-ME, Sep-CMA-MAE and Ind-PPO. RePPO lies furthest up and to the right of the constant-qVS curves on every task.
Figure 3. Quality (mean fitness) against diversity (Vendi Score) of the final populations. Large markers are means over seeds, faint dots are individual runs, and gray curves are levels of constant qVS. Up and to the right is better.

Repulsion costs some mean fitness relative to Ind-PPO. The best policy of each RePPO population, however, stays close to the best Ind-PPO policy.

Atari from pixels

In four Atari games, RePPO and Ind-PPO train populations of 8 policies from pixels. Descriptors record where the agent plays or how it approaches the game. Archives of such large policies are costly, so the QD baseline here is RefQD, which shares one network body across its archive. RePPO keeps scores close to Ind-PPO's in every game while being more diverse than both baselines. RefQD's policies score far lower.

Four scatter plots, one per Atari game, of mean score against Vendi Score for RePPO, RefQD and Ind-PPO. RePPO matches Ind-PPO's score at a higher Vendi Score; RefQD scores far lower.
Figure 4. The same plot for the four Atari games.

Diversity can also uncover strategies that independent training misses. In Seaquest, Ind-PPO and RefQD policies stay near the sea floor. Repulsion pushes some RePPO policies toward shallower water. In four of five seeds, the best RePPO policy operates just below the surface and outscores every Ind-PPO policy.

Scatter plot of Seaquest score against mean depth for every policy. Ind-PPO and RefQD policies cluster near the sea floor; RePPO policies spread across depths, and the circled best policies of four RePPO seeds sit just below the surface with the highest scores.
Figure 5. Score and mean depth of every competent Seaquest policy, with the best policy of each seed circled.

Fine-tuning language models

Finally, we fine-tune Qwen3-1.7B-Base and Granite-4.1-3B-Base on MATH problems with a correctness reward. Each population has 5 members, and each member is a LoRA adapter on a shared frozen model. Four descriptors capture how a solution is written: its use of Python code, self-verification, LaTeX symbols and markdown. To our knowledge, this is the first application of QD-RL to language-model fine-tuning.

RePPO's members develop distinct writing styles while nearly matching the accuracy of independent training. A simple classifier can tell which member wrote a solution about 80% of the time, against 20% by chance. Asking the base model for different styles through prompting changes its outputs far less and costs much more accuracy.

Methodpass@1VendiIdentifiability
Qwen3-1.7B
Base model0.320––
Prompt (corners)0.2741.1151%
Prompt (emphasis)0.2751.0844%
Ind-PPO0.4231.0441%
RePPO0.4131.8582%
Granite-4.1-3B
Base model0.233––
Prompt (corners)0.1631.0131%
Prompt (emphasis)0.1941.0337%
Ind-PPO0.5601.0353%
RePPO0.5502.5781%

Table 1. Populations on the test problems (MATH-500, Minerva, OlympiadBench). Vendi is the Vendi Score of the members' mean descriptors. Identifiability is the accuracy of a classifier that names a solution's author from its text.

Heatmaps of the mean descriptor of each of five members per method and model, with pass@1 alongside. Ind-PPO rows are nearly identical; RePPO rows differ strongly in code, verification, symbols and markdown while pass@1 stays similar.
Figure 6. Mean descriptor of each member, relative to the base model, with pass@1 in the gray column. Ind-PPO's members are nearly identical, while each RePPO member settles on its own style.

The two solutions below come from two Qwen3 RePPO members on the same problem. Both follow the same strategy and differ only in presentation.

RePPO · Member 1 — correct, 290 tokens

code −0.19 · verification −0.18 · symbols −0.74 · markdown −0.89

To solve the problem, we can treat the two boys as a single “block” or “unit”. This means we have 7 units to arrange: the block of 2 boys and the 6 girls. The number of ways to arrange these 7 units is 7! (7 factorial).

After arranging the 7 units, the 2 boys within their block can switch places. There are 2! (2 factorial) ways to arrange the boys within their block.

Therefore, the total number of ways to arrange the 6 girls and 2 boys such that the 2 boys sit next to each other is:

7! × 2!

Calculating 7!:

7! = 7 × 6 × 5 × 4 × 3 × 2 × 1 = 5040

Calculating 2!:

2! = 2 × 1 = 2

Multiplying these together gives:

7! × 2! = 5040 × 2 = 10080

Thus, the number of ways the 6 girls and 2 boys can sit in a row if the 2 boys insist on sitting next to each other is:

10080

RePPO · Member 3 — correct, 388 tokens

code +0.99 · verification −0.18 · symbols −0.86 · markdown +0.19

To solve this problem, we need to consider the boys as a single “block” since they insist on sitting next to each other. This means we can treat the two boys as one unit, and then we need to arrange this block along with the 6 girls.

Here’s the step-by-step reasoning:

1. Treat the two boys as a single unit. This means we now have 7 units to arrange: the block of two boys and the 6 girls.
2. The number of ways to arrange these 7 units is 7! (7 factorial), which is 7 × 6 × 5 × 4 × 3 × 2 × 1 = 5040.
3. Within the block, the two boys can switch places, so there are 2! (2 factorial) ways to arrange the two boys within their block, which is 2 × 1 = 2.

So the total number of arrangements is 7! × 2! = 5040 × 2 = 10080.

Let’s confirm this with Python code.

import math
# Number of ways to arrange 7 units (1 block of 2 boys and 6 girls)
arrangements_of_units = math.factorial(7)
# Number of ways to arrange the 2 boys within their block
arrangements_within_block = math.factorial(2)
# Total number of arrangements
total_arrangements = arrangements_of_units * arrangements_within_block
print(total_arrangements)
10080

The total number of ways to arrange 6 girls and 2 boys in a row, with the 2 boys insisting on sitting next to each other, is 10080.

Problem: In how many ways can 6 girls and 2 boys sit in a row if the 2 boys insist on sitting next to each other? (answer 10,080). Each box's header gives the solution's four descriptors. The “output” block after the code is written by the model itself, since no interpreter is available.

The appendix of the paper shows more domains, including locomotion from pixels and LunarLander.

The quality–diversity knob

The repulsion cap \(C_{\max}\) sets how strongly diversity enters each update, and \(C_{\max} = 0\) recovers Ind-PPO. Up to the default of 1, raising it buys diversity at little cost. Beyond that, diversity saturates and fitness starts to drop.

Two line plots over the repulsion cap C_max from 0.1 to 8 on Humanoid: mean and best fitness (left) stay flat up to about 1 and then decline; Vendi Score (right) rises steeply from 3.9 and saturates near 6.7.
Figure 7. Fitness (left) and diversity (right) on Humanoid as the repulsion cap grows. The dotted line marks the default.

When to use RePPO

Citation

@article{hedayatian2026reppo,
  title   = {Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning},
  author  = {Hedayatian, Saeed and Shah, Harsh and Mneimneh, Celine and Nikolaidis, Stefanos},
  journal = {arXiv preprint arXiv:2610.36425},
  year    = {2026},
  eprint  = {2610.36425},
  archivePrefix = {arXiv}
}