Average Reward vs. Discounted Reward: Why It Matters for Business Optimization

When building AI systems for business operations, one of the most consequential design choices is often overlooked: how do you measure success? Most reinforcement learning textbooks and libraries default to discounted reward . For games and simulated environments, this works fine. For real operational problems, it creates a systematic bias that undermines performance.

The Discounting Problem

Standard RL algorithms optimize for discounted cumulative reward. With a discount factor of 0.99 (a common setting), a reward received 100 steps in the future is worth only 37% (0.99¹⁰⁰ ≈ 0.366) of the same reward today. The formal relationship is: the effective horizon of a discount factor γ is T_eff = 1/(1−γ). With γ = 0.99: T_eff = 1/(1−0.99) = 100. This means the algorithm assigns most of its optimization weight to the next ≈100 steps; beyond that horizon, rewards are increasingly discounted toward zero. After 500 steps, the discount factor 0.99⁵⁰⁰ ≈ 0.007, meaning future rewards contribute less than 1% of their face value. This makes sense for a video game where episodes end. It makes much less sense for an inventory management system that runs 24/7/365 — a system where annual cycles span far beyond this effective horizon.

The Average Reward Alternative

Average reward RL maximizes the long-run average performance ρ* [Schneckenreither, 2020; Schneckenreither & Moser, 2025]. Every time step contributes equally. A policy optimized this way maximizes steady-state throughput, which is exactly what operational efficiency means.

ARPPO: Bringing Average Reward to Modern Deep RL

Our proprietary ARPPO (Average Reward PPO) algorithm combines the stable training properties of Proximal Policy Optimization with the average reward objective . The key mechanism is a running estimate of the average reward ρ̂, used to compute differential advantages via Generalized Advantage Estimation . This subtracts the baseline average reward from each transition, centering the learning signal around zero and enabling stable training without discount factor tuning.

Practical Impact

In inventory optimization problems with non-stationary demand, ARPPO adapts faster than discounted-reward PPO because it does not underweight the impact of long-horizon demand cycles . The result: lower average inventory costs, fewer stockouts, and a system that improves continuously as it observes more data.

Curious how this applies to your operations? Contact us for a no-obligation discovery conversation.