Inventory optimization is one of the oldest problems in operations research. Classical solutions such as EOQ, the newsvendor model, and (s,S) policies assume simplified demand distributions and static cost structures. In the real world, demand is seasonal, trends shift, and supplier lead times vary. Reinforcement learning handles all of this naturally .
The RL Formulation
An inventory system maps naturally to a Markov Decision Process:
- State: current inventory level, recent demand history, lead time, season indicator, price signals
- Action: order quantity (continuous or discrete)
- Reward: negative of (holding cost times inventory, plus stockout cost, plus fixed order cost when ordering)
The RL agent learns to maximize average profit across all these trade-offs simultaneously, without explicit demand distribution assumptions.
Why ARPPO Outperforms Standard RL
Standard PPO with discounted reward underperforms for two reasons: (1) with a discount factor of 0.99, the effective optimization horizon is only 1/(1−0.99) = 100 steps, making annual seasonal cycles (365+ days) largely invisible to the learning signal; (2) different cost ratios require different discount values, demanding expensive hyperparameter tuning. ARPPO eliminates both problems via its average reward formulation [Schneckenreither, 2020; Schneckenreither et al., 2022; Schneckenreither & Moser, 2025].
Implementation
Our LARA library implements inventory environments as Gymnasium-compatible environments , enabling parallel training across multiple inventory simulations, GPU-accelerated neural network updates via batched inference, continuous action spaces for fine-grained order quantity control, and LSTM-based state encoders for capturing demand history. A typical deployment trains on at least 12–24 months of historical data, then runs in shadow mode for 2 to 4 weeks before full automation.
Want to see what RL-based inventory optimization could do for your supply chain? Talk to our team.