Proximal Policy Optimization (PPO) has become the default choice for deep reinforcement learning: reliable, well-understood, and competitive across many benchmarks. But PPO uses discounted reward, which creates a fundamental mismatch with continuing operational tasks. Our ARPPO algorithm [Schneckenreither, 2020; Schneckenreither & Moser, 2025] addresses this directly.
Algorithm Comparison
| Property | PPO | ARPPO |
|---|---|---|
| Objective | Discounted cumulative reward | Long-run average reward |
| Advantage computation | GAE with discount factor | Differential GAE (subtracts running average ρ̂) |
| Hyperparameters | Requires discount factor tuning | No discount factor; uses ρ learning rate |
| Best for | Episodic tasks (games, navigation) | Continuing tasks (business ops) |
| Seasonal pattern capture | Limited by effective horizon 1/(1−γ) | Unlimited (average over all history) |
When PPO Wins
PPO performs best on tasks with natural episode boundaries (games, one-off decisions), strong preference for near-term outcomes, and simple reward structures. Standard RL benchmarks such as CartPole, LunarLander, and Atari [Gymnasium] are designed for episodic settings, which is why PPO excels there. Our benchmarks follow the evaluation protocol of rl-baselines3-zoo.
When ARPPO Wins
ARPPO outperforms PPO on tasks with no natural episode boundary (inventory systems, pricing engines, facility management), long-horizon seasonal patterns, sensitivity to average-case performance over years rather than episodes, and where a stable steady-state policy is required .
Practical Guidance
Use PPO when: your problem naturally resets (batch job scheduling, one-shot pricing decisions).
Use ARPPO when: your system runs continuously, performance is measured in steady-state throughput, and long-term results matter as much as short-term ones.
Not sure which is right for your use case? We are happy to discuss the specifics of your operational problem.