The Reward Engineering Problem: How Saturating Rewards Starve Average-Reward RL

Average-reward reinforcement learning subtracts a running baseline ρ from every reward it receives. This is the mechanism that removes the discount-factor hyperparameter and makes the algorithm theoretically correct on continuing, infinite-horizon tasks. It is also, in the wrong environment, a trap: when rewards saturate (converging to a near-constant value), the differential collapses to zero, the gradient signal disappears, and the agent stops improving entirely. This post dissects the failure, demonstrates a 20× performance reversal by redesigning the reward signal, and derives a practical checklist for deploying ARPPO in production.

A Brief Recap: ARPPO and the Differential Reward

Standard discounted PPO [1] maximises the expected discounted sum of rewards G_t = ∑_k≥0 γ^k r_t+k. The discount factor γ is an engineering approximation: it makes the sum finite and gives the agent a preference for sooner rewards, but on a continuing task it introduces a tunable horizon that has no physical meaning in the problem. Different values of γ produce different optima.

ARPPO (Average-Reward PPO) [2] replaces the discounted objective with the long-run average reward per step:

The algorithm maintains a running estimate ρ of this average reward and subtracts it from every transition reward before computing the advantage:

You can think of the ρ-update as a slow exponential moving average of the reward: as training progresses, ρ drifts toward the agent’s current average performance, and the differential r_t − ρ tells the policy whether each step was above or below that running level. Here α is a small, decaying step size:

On a continuing task, ρ tracks the policy’s current average quality. Good transitions (those with r_t > ρ) produce a positive differential and positive advantage; poor transitions produce a negative one. The signal is always relative to the agent’s current performance level, not to a fixed zero baseline.

The Saturation Trap

The mechanism works elegantly, provided the reward signal varies meaningfully across state quality. The trap appears when it does not.

Consider the classic flat survival reward: the agent receives r_t = +1 for every surviving step and r_t = 0 (or the episode terminates) on failure. This is the standard CartPole-v1 signal: a thoughtless default that rewards the agent for staying alive, without telling it anything about how well it is balancing. Once the policy is reasonably good, episodes grow long and almost all transitions yield r_t = 1. The ρ-estimate converges toward 1. The differential becomes:

The advantage A_t collapses to a pure temporal-difference residual γ V(s_t+1) − V(s_t) with no reward component. In an environment where all surviving states look equally good to the value function, the policy gradient vanishes entirely. The only informative transitions are the rare failures (r_t = 0, producing δ_t = −1), but once the policy is already avoiding most failures, these are too infrequent to drive further improvement.

This is not a tuning problem. It is intrinsic to the average-reward formulation: the differential reward is only meaningful relative to the running mean, and a near-constant reward produces a near-zero differential regardless of learning rate, network size, or any other hyperparameter. PPO, using a discounted objective, does not face this failure mode: even when r_t = 1 every step, the discounted return varies with episode length (shorter episodes give lower returns), and the advantage remains informative.

The DoublePole Experiment

The two-pole cart (DoublePole) is a challenging Markovian control task: a single cart must balance two independent poles of different length and mass simultaneously: a long pole (1.0 m, 0.1 kg) and a short pole (0.1 m, 0.01 kg). The short pole reacts far faster than the long one, making the coupled system substantially harder to stabilise than single-pole CartPole. The full 6-dimensional state [x, ˙x, θ₁, ˙θ₁, θ₂, ˙θ₂] is observed (fully Markovian), and the agent applies a bang-bang force (±10 N). Episodes terminate if either pole falls beyond 36° or the cart leaves the track (±2.4 m); truncation occurs at 1 000 steps, so a perfectly balanced episode returns exactly 1 000.

We ran the same ARPPO configuration against two different reward signals, holding all hyperparameters fixed. Five replications per condition (pilot, July 2026).

Condition 1: Flat Survival Reward

The agent receives r_t = +1 for every step both poles remain upright, identical to the standard CartPole-v1 signal. It is the unthinking default: reward survival, say nothing about quality.

The failure is visible in the training diagnostics: ρ climbs sharply to 0.98–0.99 within the first 10k steps (the survival rate is high from episode 1 because even a random policy on DoublePole occasionally survives a few seconds). Once ρ reaches that level, the differential flatlines and the policy stops improving. Meanwhile PPO, operating with a discounted return that grows with episode length, continues receiving a gradient signal and reaches 873 by the end of training.

Condition 2: Dense Balance-Quality Reward

The reward is redesigned as a per-step balance-quality score in [0, 1]:

This signal encodes how well the agent is balancing at every single step, not merely whether it is still alive. A perfectly vertical, centred pole gives r_t = 1; a pole at the fall threshold gives r_t = 0. Intermediate states produce intermediate values. The signal varies continuously as the poles tilt and recover; it is dense, bounded, and non-saturating relative to the policy’s current performance level.

The reversal is not marginal: PPO’s performance falls slightly from the flat-reward condition (873.6 → 737.8) because the dense reward turns a binary survival task into a continuous control-quality task with a harder objective ceiling. ARPPO, which was catastrophically bad under the flat reward, reaches near-ceiling performance under the dense signal. The outcome is driven entirely by the reward engineering, not by any change to the algorithm or its hyperparameters.

Validated Results at Scale (10 Seeds)

The pilot used 5 replications per condition. A subsequent full benchmark run at n=10 seeds, using the same tuned ARPPO config (configs/arppo/arppo_doublepole.yaml) with dense balance-quality reward and independent per-sub-environment seed streams, confirms the result:

Algorithm	Mean ± Std	Best seed	N	Notes
ARPPO	989.90 ± 24.86	999.81	10	9/10 seeds above 991; lowest 919.72. 5× tighter spread than PPO.
LARA PPO	767.19 ± 130.62	987.60	10	Range 514.73–987.60; large seed variance.
SB3 PPO (reference)	837.90 ± 159.36	n/a	10	Independent anchor (Gymnasium port, matched hyperparams).

Welch’s t-test: ARPPO vs LARA PPO, p ≈ 0.0004. ARPPO also beats the SB3 reference (p ≈ 0.015). The seed variance for ARPPO is five times tighter than PPO’s (±24.86 vs ±130.62), a consistent property of average-reward algorithms on well-suited tasks: once ρ is tracking quality, the policy converges to the same near-ceiling solution from almost any initialisation.

The Double-Centering Trap

Knowing that dense rewards help, a natural next step is to also apply reward normalisation: subtract the running mean, divide by the running standard deviation. This is what LARA’s RewardCenteredScaled mode does, and it is a useful stabilisation technique for discounted PPO (it prevents exploding value estimates when the reward scale is large or unknown). What happens if we apply it to ARPPO?

ARPPO’s own ρ-subtraction is a form of mean-centering. Applying RewardCenteredScaled first means the signal has already been centred before ρ sees it: this is double-centering. The outer normalisation removes the very structure that ρ needs to track: the long-run mean of the raw reward. After normalisation, the reward distribution is zero-centred by construction; ρ chases zero instead of the actual quality level; the differential loses its interpretation as “how much better than average is this step?”

This asymmetry is important: RewardCenteredScaled is a valid ablation arm for discounted PPO (it can improve stability by keeping value targets in a bounded range), but it is destructive for ARPPO. The two transformations are not independently beneficial when stacked; the outer centering strips out the information ARPPO’s baseline depends on.

CartPole-LQR: Dense Reward as a Control-Quality Metric

The same dense-reward principle applies to single-pole CartPole. Rather than the flat +1/step survival bonus, we use a quadratic verticality cost inspired by LQR (Linear-Quadratic Regulator) design:

ARPPO on CartPole-LQR: 485.1 mean return (5/5 seeds solve, range 481–489) ≈ near the 500-step theoretical ceiling. The dense quadratic cost gives ρ a fine-grained quality signal at every time step, and ARPPO converges rapidly and reliably.

The Special Case: Flat Reward with rho_init

The full ARPPO benchmark suite (10 environments, n=10 seeds) includes standard CartPole-v1 with its flat +1/step reward. ARPPO achieves 500.00 ± 0.00, a perfect score on all 10 seeds. This appears to contradict the saturation argument above. The resolution is the rho_init hyperparameter.

With rho_init: 0 and a very slow ρ step of 1e-5, ρ is initialised at zero and updated so slowly that it stays well below the near-constant reward of 1 throughout the entire training run (ending around 0.47 at 100k steps). The differential r_t − ρ ≈ 0.53 remains positive and informative on every surviving step. Survival is incentivised consistently, the gradient is always positive for surviving steps, and ARPPO converges to the optimal policy.

Connection to Discounted RL

Average-reward and discounted RL are not mutually exclusive frameworks. Setting rho_init: 0 and the ρ step size to zero freezes the baseline at 0 permanently, giving a differential r_t − 0 = r_t. Combined with a discount factor γ < 1, this recovers standard discounted PPO exactly: the algorithm is ARPPO with a permanently-zero baseline and a finite horizon. In other words, discounted RL is a special case of average-reward RL where the baseline update is switched off. This makes ARPPO a strict generalisation: when the task genuinely has a meaningful average-reward structure, let ρ adapt; when the discounted objective is appropriate, freeze ρ at zero.

Why Average-Reward RL Has a Competitive Advantage on Dense Continuing Tasks

The DoublePole result is not only a cautionary tale; it is also a demonstration of where average-reward RL wins. When the reward is dense and non-saturating, the ρ-baseline acts as an adaptive reference that automatically tracks the policy’s current competence level. This has two consequences that discounted PPO cannot match:

1. Automatic difficulty scaling. As the policy improves, ρ rises to match. A step that would have been above-average at the start of training becomes below-average once the policy is near-optimal: the agent is always being asked to do better than its recent self, not better than a fixed zero baseline. This is a built-in curriculum.

2. No horizon bias. Discounted PPO’s advantage estimate is biased by the choice of γ. A long-horizon task (DoublePole at 1 000 steps) with γ = 0.98 discounts rewards 1 000 steps in the future by a factor of 0.98¹⁰⁰⁰ ≈ 10⁻⁹: they are effectively invisible. The algorithm cares only about the immediate future and has no incentive to maintain balance further ahead. ARPPO has no discount term: the average-reward objective weights all time steps equally, giving the agent an incentive to maintain quality indefinitely.

Together, these properties explain the DoublePole outcome: ARPPO with dense reward achieves 989.90 ± 24.86 (mean 9/10 seeds near the ceiling) while PPO achieves 767.19 ± 130.62 with 5× larger seed variance. PPO’s variance reflects sensitivity to the initialisation-dependent horizon artefact; ARPPO’s tighter spread reflects the convergence property of a well-specified average-reward signal.

Practical Reward Engineering Checklist

Based on the DoublePole experiments and the broader ARPPO suite (9 environments across episodic, continuing, discrete, and continuous settings), the following rules govern whether ARPPO will converge:

Rule	Correct	Incorrect	Consequence if violated
1. Non-saturating reward	Reward varies continuously with state quality	Binary survival (+1/0) or sparse events	ρ → max reward; differential collapses
2. RewardNativeOnly	sbRewardMode: RewardNativeOnly	RewardCenteredScaled	Double-centering: ρ chases zero; gradient collapses
3. rho_init	rho_init: 0 for episodic / non-negative reward	Seeded ρ at the expected mean	Seeded ρ can starve the differential from step 1
4. Bounded reward	Reward clipped or normalised to a finite range	Unbounded cumulative costs	ρ diverges; critic targets blow up
5. zero_terminal_reward	zero_terminal_reward: true for episodic tasks	Allowing the done step to contribute reward to the continuing objective	Terminal transition inflates ρ incorrectly

The most common failure mode for new ARPPO deployments is Rule 1: practitioners port the reward function from a working discounted PPO config without modification. If that config uses a flat survival signal, ARPPO will fail, and the failure will look like a tuning problem (varying lr, α, γ makes little difference) rather than a reward design problem.

Diagnostic: Is Your Reward Saturating?

LARA logs ρ after every update. A converging-to-constant ρ early in training is the diagnostic signature of reward saturation; the earlier it saturates, the worse the situation. In a healthy ARPPO run on a control task, ρ should rise gradually as the policy improves, tracking quality, and end at a value that reflects the policy’s actual competence level. On DoublePole with dense reward, ρ ends at 0.9998: the agent has learned to balance near-perfectly, and the baseline accurately reflects this.

Implications for Production Deployment

The reward design principle here is actually simple and clarifying: specify what you want to optimise, not just whether a threshold was crossed. The RL algorithm handles the rest. A reward function that says “how much?” is almost always better than one that says “did it happen?”.

Domain	Binary (breaks ARPPO)	Dense (what you actually want)
Inventory	Was there a stockout today?	How much stock was there? (fill rate, backlog fraction)
Finance	Was profit > 0?	How much was the profit?
Scheduling	Was the job completed on time?	How much lateness remains? (continuous tardiness cost)
Pricing	Did revenue increase vs. yesterday?	What was the revenue per unit sold?

The dense version is not just better for the algorithm; it is more precise about what the business actually cares about. “Was there a stockout today?” treats a warehouse with one unit left identically to one with a full buffer. “How much stock was there?” captures the difference. The reward function becomes a legible specification of the objective, and ARPPO does the optimisation.

Summary

References

1. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
2. Schneckenreither, M. (2025). Average-Reward Proximal Policy Optimization (ARPPO). LARA Technical Report.
3. Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.), Chapter 10: On-policy Control with Approximation. MIT Press.
4. Mahadevan, S. (1996). Average Reward Reinforcement Learning: Foundations, Algorithms, and Empirical Results. Machine Learning, 22(1–3), 159–195.
5. Naik, A., Shariff, R., Hegde, N., Yin, H., Ghavamzadeh, M., & Sutton, R. S. (2019). Discounted Reinforcement Learning is Not an Optimization Problem. arXiv:1910.02140.
6. Wieland, A. P. (1990). Evolving controls for unstable systems. In Connectionist Models: Proceedings of the 1990 Summer School, 91–102.
7. Gomez, F., & Miikkulainen, R. (1999). Solving Non-Markovian Control Tasks with Neuroevolution. IJCAI-99.
8. Mnih, V., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533.
9. Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018). Deep Reinforcement Learning That Matters. AAAI-18.