ServiceNow AI researchers successfully restored training parity when migrating their PipelineRL framework from vLLM 0.8.5 to vLLM 0.18.1 by systematically resolving four backend divergence points. The engineering team demonstrated that fixing inference backend correctness must precede any algorithmic corrections to the reinforcement learning objective. This systematic overhaul eliminated severe divergence across core training metrics, including clip rate, KL divergence, entropy, and reward curves.
Resolving Backend Divergence Points
When using vLLM for rollout token generation in online reinforcement learning (RL) frameworks like GSPO, PPO, or GRPO, discrepancies between rollout logprobs and trainer calculations disrupt optimization dynamics. During ServiceNow's migration, initial attempts showed trainer metrics rapidly veering off course due to implicit engine behavior changes between vLLM versions.
To establish numerical equivalence before altering RL objectives, researchers isolated four key backend mechanisms:
- Processed Logprobs: Configured
logprobs-mode=processed_logprobsso returned logprobs reflected post-sampling distributions (after temperature and top-k/top-p filtering) rather than raw model logits. - Explicit Runtime Flags: Disabled
enable-prefix-cachingandasync-schedulingdefaults in vLLM V1 to prevent state reuse across dynamic weight update boundaries. - Inflight Weight Synchronization: Adapted weight loading using
pause_generation(mode="keep", clear_cache=False)to prevent premature cache invalidations during online updates. - FP32 Projection Head: Matched trainer-side mathematical precision by running the
lm_headfinal output layer in 32-bit floating point precision.
Mechanism and Practical Trade-Offs
Small logit shifts directly alter token logprobs, propagating errors into policy ratios and clipping calculations. By aligning projection precision and fixing backend defaults, ServiceNow mirrored the precision approaches cited in the MiniMax-M1 report and ScaleRL architecture.
However, restoring backend parity required specific operational trade-offs. Disabling runtime features like prefix caching and async scheduling eliminates V1-specific inference optimizations, temporarily foregoing throughput gains to guarantee total determinism. Practitioners building online RL pipelines must weigh these performance defaults against the risk of policy drift.
Why it matters
Attempting to fix rollout divergence using algorithmic off-policy adjustments often masks underlying infrastructure bugs. Resolving low-level engine discrepancies first ensures that optimization updates reflect true policy improvements rather than execution artifacts.