Hyperparameter tuning is one of the most frustrating parts of deploying reinforcement learning in production. Among all hyperparameters, the discount factor γ in Proximal Policy Optimization (PPO) is arguably the most consequential — and the most misunderstood. Set it too low and the agent ignores future consequences. Set it too high and training becomes numerically unstable. There is rarely a principled way to choose it.
ARPPO — our Average Reward Policy Optimization framework — eliminates this problem by design. Instead of discounting future rewards, ARPPO directly optimizes the long-run average reward per time step. The result is an algorithm that is structurally less sensitive to reward scale, planning horizon, and discount geometry.
The Discount Factor Problem in PPO
PPO maximizes the discounted sum of future rewards:
JPPO(π) = 𝔼[Σt=0∞ γt rt]
This formulation has a hidden assumption: that the problem has a finite planning horizon proportional to 1/(1-γ). For γ=0.99, the effective horizon is 100 steps. For γ=0.999, it’s 1,000 steps.
In practice, choosing γ requires you to implicitly answer: “How many steps ahead should my agent plan?” For inventory optimization, should that be 50 days? 500 days? For energy grid dispatch, is the horizon 24 hours or 6 months? These decisions profoundly affect the optimal policy — and there’s no principled way to set γ without domain expertise and extensive ablation studies.
Beyond horizon selection, γ introduces a second instability: reward scale sensitivity. The discounted return grows without bound if rewards are large and γ is close to 1. This forces practitioners to normalize rewards, clip them, or use reward shaping — each of which introduces its own hyperparameters.
ARPPO’s Alternative: Optimize Rho Directly
ARPPO replaces discounted returns with the average reward criterion:
ρ(π) = limT→∞ (1/T) 𝔼[Σt=0T-1 rt]
This is the long-run average reward per step under policy π. Crucially, ρ is horizon-free: it doesn’t ask you to specify how many steps ahead the agent should plan. In a stationary Markov environment, ρ is a well-defined quantity that captures steady-state performance without approximation.
The differential value function V(s) in ARPPO satisfies the Bellman equation:
V(s) = r(s,a) – ρ + 𝔼[V(s’)]
Notice: no γ multiplying V(s’). The average reward ρ is subtracted instead, centering the value function around zero. This means V(s) remains bounded regardless of reward magnitude — eliminating the scale sensitivity that plagues discounted formulations.
Which Hyperparameters Actually Matter in ARPPO?
ARPPO still has hyperparameters — every learning algorithm does. But the set is smaller and less sensitive:
| Hyperparameter | PPO | ARPPO |
|---|---|---|
| Discount factor γ | Critical — must tune per task | Not needed |
| Learning rate (policy) | Moderate sensitivity | Similar sensitivity |
| KL divergence budget δ | Clip ratio ε (proxy) | Direct KL bound — more principled |
| Reward normalization | Often required (scale sensitivity) | Online via Welford — automatic |
| Number of epochs | Typically 3–10 | KL-bounded — adaptive |
| Number of environments | Affects sample efficiency | Same — benefits from vectorization |
The key structural difference: ARPPO eliminates the three hyperparameters most prone to cross-task interaction (γ, reward scale, planning horizon), replacing them with problem-independent defaults.
KL Divergence as a Trust Region (Replacing Clip)
PPO uses a clipped surrogate objective to limit policy updates. The clipping ratio ε is another hyperparameter: too small and learning is slow; too large and training destabilizes. In practice, ε=0.2 works well for many tasks — but it’s a heuristic.
ARPPO uses a hard KL divergence constraint as its trust region. Rather than clipping, it computes the exact KL between the new and old policy distributions and rejects any update that exceeds a threshold δ. This is more principled because:
- KL divergence is policy-invariant — the same δ means the same “policy change” regardless of state distribution
- The clipped PPO objective approximates a trust region but doesn’t guarantee it — large updates can still occur at states where the ratio is near 1
- Adaptive KL threshold (increase δ if updates are too conservative, decrease if divergence spikes) is straightforward to implement
For an in-depth look at how ARPPO’s KL constraint works across training epochs, see our post on KL Divergence and Epochs in ARPPO Training.
Reward Normalization: Automatic vs. Manual
PPO is sensitive to reward magnitude because large rewards make discounted returns large, which make value function targets large, which makes learning unstable. The standard fix is to normalize rewards — but how? Min-max scaling? Z-score? Running mean? Each choice is a hyperparameter.
ARPPO uses Welford’s online algorithm to maintain a running mean and variance of observed rewards, then normalizes using the running statistics. This is:
- Automatic: no need to pre-specify reward range or standard deviation
- Adaptive: normalization adapts as the agent explores new regions of the state space
- Numerically stable: Welford’s algorithm avoids catastrophic cancellation in variance estimation
We covered the Welford normalization in detail in Reward Normalization Done Right: Welford Statistics for Stable Training.
Practical Implications for Operational AI
For teams deploying RL in production — inventory systems, energy dispatch, production scheduling — the reduced hyperparameter surface of ARPPO has a direct business impact:
- Faster time-to-production: fewer hyperparameter sweeps mean less compute and less human time spent debugging
- Better transfer across environments: when the reward scale or planning horizon changes (new SKU, new warehouse, new product line), ARPPO requires fewer re-tuning cycles
- More stable long-term operation: the absence of γ means no accumulated discount drift in environments that run indefinitely
This is why ARPPO is the foundation of our LARA platform — not PPO. For ongoing operational environments with no natural episode boundaries, average reward is the right formulation, and ARPPO is the right algorithm.
When Does PPO Still Win?
To be fair: PPO remains the right choice in several scenarios:
- Episodic tasks with clear terminal states: game playing, robot manipulation, navigation with defined endpoints. In these cases, the discounted return has a clean interpretation and γ is easy to set (typically γ=1 for finite-horizon, γ=0.99 for long-horizon episodic).
- Transfer to pre-trained models: PPO is the backbone of RLHF (reinforcement learning from human feedback) in large language models. The existing ecosystem favors discounted formulations.
- Short-horizon problems: if the effective planning horizon is 20–50 steps, the difference between average and discounted reward is small, and PPO’s stability is well-established.
ARPPO’s advantage grows with horizon length and environment stationarity — which is exactly the regime of continuous operational optimization.
Summary
ARPPO reduces hyperparameter sensitivity relative to PPO through three structural choices: eliminating γ (replacing it with average reward), replacing clip-based trust regions with a principled KL divergence bound, and automating reward normalization via Welford statistics. The result is an algorithm that is easier to tune, more robust to reward scale, and better suited to the infinite-horizon structure of real operational environments.
See also our comparison of ARPPO vs. PPO: When Average Reward Wins for empirical results, and Average Reward vs. Discounted Reward in Business Optimization for the mathematical foundations.