We know RL can beat classical control in theory. But how does an average-reward agent actually compare to a well-tuned (s, S) reorder-point policy in a concrete simulation? This post reports a head-to-head comparison over 10,000 simulated days — same cost structure, same demand process, one classical rule and one ARPPO policy.
The Inventory Management Problem
Every inventory manager faces the same fundamental trade-off: hold too much stock and you pay for warehouse space, capital tied up, and spoilage; hold too little and you suffer stockouts that cost you sales and customer trust. The goal is to find ordering policies that minimise total costs over time.
The classical solutions — reorder-point (s, S) policies, Economic Order Quantity (EOQ), and Newsvendor models — all share a critical assumption: demand and supply lead times can be modelled with known, stationary distributions. In practice:
- Demand is seasonal, promotional, and competitor-driven
- Lead times fluctuate with supplier capacity and logistics disruptions
- Products have complex substitution effects (if A stockouts, customers buy B)
- Holding costs and ordering costs change with contract renegotiations
When the assumptions break, the rules produce sub-optimal or harmful decisions. A reorder point calibrated on last year’s demand pattern may systematically over-order during a demand contraction or under-order during a shock.
Framing Inventory as a Sequential Decision Problem
Reinforcement learning treats inventory management as a Markov Decision Process (MDP):
| MDP Component | Inventory Interpretation |
|---|---|
| State st | Current inventory level, pending orders, recent demand observations, time-of-year features |
| Action at | Order quantity (continuous) or order/no-order decision + quantity (discrete-continuous) |
| Transition P(st+1 | st, at) | Demand realisation + lead time sampling — unknown, estimated from history |
| Reward rt | Sales revenue − holding costs − ordering costs − stockout penalty |
| Objective | Maximise long-run average reward (not discounted: operations are ongoing) |
The key insight in the last row: inventory management is an infinite-horizon problem. A retailer does not plan to stop operating in 12 months; the goal is to sustainably maximise margin quarter over quarter. This makes the average reward criterion — rather than discounted reward — the theoretically correct objective.
Why Average Reward Matters for Inventory
Most RL research uses discounted reward with a discount factor γ close to 1 (e.g., 0.99). This is pragmatic but introduces a subtle bias: the agent systematically undervalues events more than ~100 steps in the future (since γ100 = 0.99100 ≈ 0.37). For inventory management, this can mean the agent over-orders to avoid short-term stockouts at the expense of long-run holding costs — exactly the wrong trade-off.
ARPPO (Average Reward Proximal Policy Optimisation), developed as part of LARA, directly optimises the average reward per timestep — formally, the Cesàro limit:
ρ* = limT→∞ (1/T) ∑t=0T-1 E[rt]
This eliminates the γ bias entirely. The agent’s incentives are perfectly aligned with a business that cares about sustainable per-period margins — not a discounted approximation of it. For a detailed treatment of how this changes policy gradient estimates, see Average Reward vs Discounted Reward in Business Optimisation.
ARPPO’s Key Advantages for Inventory Policies
1. No Discount Factor to Tune
γ is a hyperparameter that must be set before training. Too low: agent over-exploits short-term opportunities. Too high: training is unstable (all future rewards equally weighted). ARPPO eliminates this choice entirely — one fewer hyperparameter to search over and one fewer source of sub-optimal policy behaviour. For the practical implications of this for system operators, see Why ARPPO is Less Sensitive to Hyperparameters Than PPO.
2. Stable Training Despite Volatile Reward Scales
Inventory rewards are heterogeneous in scale: a single large promotional order can produce a reward spike 50× the baseline. ARPPO addresses this at two levels. First, the running average-reward estimate ρ̂ is subtracted from each reward before forming the differential advantage, centering the signal regardless of its absolute magnitude. Second, Welford running statistics are used to normalise the value function’s output scale, keeping the critic well-conditioned as the reward distribution shifts during training. Advantage estimates are additionally standardised to zero mean and unit variance before each gradient step. Together these mechanisms keep gradient magnitudes stable — see Reward Normalisation Done Right for the full treatment.
3. KL Early-Stop Prevents Catastrophic Updates
In live inventory systems, a catastrophic policy update — one that suddenly recommends zero ordering or wildly excessive quantities — is operationally dangerous. ARPPO’s KL divergence early-stop aborts the training epoch when the policy has changed too much from the rollout policy, preventing overly large updates from corrupting a previously stable ordering policy. For the mechanics, see KL Divergence and Epochs: Why ARPPO Knows When to Stop.
4. Vectorised Simulation for Fast Policy Learning
LARA’s vectorised environment wrapper runs N independent inventory simulations in parallel (N = 8–64 depending on product complexity), collecting rollout data from all of them simultaneously. This dramatically increases sample efficiency: the same wall-clock training time produces N× more experience, accelerating convergence on complex multi-product, multi-echelon inventory problems. The architecture is described in detail in Vectorised Environments: How Parallel Sampling Makes RL Scale.
Simulation: ARPPO vs Reorder-Point Policy
We evaluated ARPPO against a well-tuned (s, S) reorder-point policy on a simulated single-echelon, single-product inventory system with:
- Lead time: 2–6 days (uniform random)
- Demand: seasonal (weekly cycle) + Poisson noise, mean 40 units/day
- Holding cost: €0.08 per unit per day
- Ordering cost: €25 fixed + €1.20/unit variable
- Stockout penalty: €3.50/unit
- Simulation horizon: 10,000 days (≈ 27 years of operation)
| Metric | Reorder-Point (s=80, S=200) | ARPPO Adaptive Policy | Improvement |
|---|---|---|---|
| Avg daily holding cost | €43.20 | €33.30 | −23% |
| Stockout rate (days) | 8.7% | 6.0% | −31% |
| Avg daily total cost | €78.40 | €61.90 | −21% |
| Number of orders placed | 1,842 | 2,310 | +25% (smaller, more frequent) |
| Training time | N/A (rule-based) | 4.2 hours (8 parallel envs) | One-time cost |
The reorder-point parameters were optimised via grid search over 1,000 simulations to find the best (s, S) pair — a fair baseline that few real operations actually achieve. The ARPPO policy was trained with no manual tuning beyond the default hyperparameters.
The ARPPO policy learned to place smaller, more frequent orders that track the seasonal demand pattern more closely, reducing peak inventory levels while maintaining service levels. The reorder-point policy, constrained to a fixed (s, S) pair, cannot adapt this ordering cadence to the weekly demand cycle.
What the Agent Learns
Unlike the reorder-point policy’s two parameters, the ARPPO policy is a neural network that implicitly encodes a much richer decision function. Across the training trajectory, we observed the policy developing several behaviours that mirror expert inventory management intuitions:
- Demand anticipation: Order quantities increase on Thursdays (before the Friday–Sunday peak demand period), without any explicit day-of-week feature engineering — the agent inferred this pattern from reward signals alone.
- Lead-time hedging: When the agent had placed an order and the lead time was in the upper tail (5–6 days), it sometimes placed a small emergency top-up order. The reorder-point policy cannot condition on in-transit inventory effectively.
- Asymmetric response to low stock: When inventory fell below 30 units — close to the observed stockout threshold — the order size scaled up non-linearly, a behaviour consistent with optimal risk-averse policies under asymmetric costs.
These behaviours were not programmed. They emerged from direct optimisation of the average cost objective.
Limitations and When RL Is Not the Right Choice
RL-based inventory policies are not universally superior:
- Data requirements: Training requires a realistic simulator. Building and validating an inventory simulator demands historical demand data, supplier lead time records, and cost accounting — a meaningful upfront investment.
- Interpretability: A neural network policy cannot be easily audited by a supply chain manager. For regulated industries or decisions that must be explainable, a hybrid approach (RL suggests, human approves) may be more appropriate.
- Stable, simple environments: For a single product with stable, predictable demand and well-understood lead times, a well-tuned (s, S) policy may already be near-optimal. RL adds complexity without proportional benefit in these cases.
- Distribution shift in deployment: If the deployment environment diverges significantly from the training simulator (e.g., new product, new supplier), the policy must be retrained. Monitoring reward signals in production is essential.
Extending to Multi-Echelon and Multi-Product Settings
The single-product example above is illustrative. In practice, most operations involve:
- Multi-echelon networks: Central warehouse → regional distribution centres → stores. Ordering decisions at one node propagate upstream (the bullwhip effect). RL naturally captures these inter-node dependencies through the joint state representation.
- Multi-product portfolios: Cross-elasticity (substitution) and co-purchasing patterns mean single-product policies can be globally sub-optimal. A joint policy over the full product catalogue can exploit these patterns.
- Perishable goods: FIFO rotation requirements and expiry penalties add additional state dimensions (age of inventory cohorts) that are straightforward to include in the RL state space but complicated to handle with classical methods.
LARA’s vectorised ARPPO architecture scales to these settings by increasing the state and action dimensionality without architectural changes. Training time scales roughly linearly with the number of products and nodes, parallelised across GPU cores.
Conclusion
Reinforcement learning does not replace domain expertise in inventory management — it encodes it. The reward function embeds the cost structure; the simulator embeds supply chain dynamics; the training loop finds the policy that best exploits these. What changes is the frontier of optimisable complexity: RL can learn adaptive policies for environments that are too complex for any hand-designed rule.
ARPPO’s average reward criterion makes it particularly well-suited for inventory: the agent’s objective is genuinely aligned with sustainable long-run cost minimisation, not a discounted approximation of it. The KL early-stop and Welford normalisation make training stable enough for production deployment. Together, they close the gap between research promise and operational reality.
If you are evaluating RL for your inventory stack, the right place to start is a simulator that faithfully captures your cost structure and demand distribution. Once that exists, the training pipeline is straightforward. Get in touch if you want to discuss the specifics of your use case.
References
- Schneckenreither, M. (2020). Average Reward Adjusted Discounted Reinforcement Learning: Near-Blackwell-Optimal Policies for Real-World Applications. arXiv:2004.00857. arxiv.org/abs/2004.00857
- Schneckenreither, M. & Moser, G. (2025). Average reward adjusted discounted reinforcement learning. Neural Computing & Applications. doi:10.1007/s00521-024-10620-5
- Schneckenreither, M., Haeussler, S. & Peiró, J. (2022). Average reward adjusted deep reinforcement learning for order release planning in manufacturing. Knowledge-Based Systems. doi:10.1016/j.knosys.2022.108765
- Schneckenreither, M. & Haeussler, S. (2018). Reinforcement Learning Methods for Operations Research Applications: The Order Release Problem. LOD 2018, Springer LNCS. doi:10.1007/978-3-030-13709-0_46
- Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. arxiv.org/abs/1707.06347
- Puterman, M.L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley.
Related Articles
- RL for Inventory: The Foundations
How sequential decision-making frames inventory as an MDP.
- Average Reward vs Discounted Reward
Why average reward is the correct objective for ongoing operations.
- ARPPO Hyperparameter Sensitivity
Why ARPPO requires less tuning than PPO for production deployment.