IBM has open-sourced its Granite 4.1 language model family under an Apache 2.0 license, featuring dense architectures scaling up to 30 billion parameters. Trained on approximately 15 trillion tokens across a five-phase pipeline, the new models extend context windows up to 512K tokens. Enterprise developers gain access to efficient models designed to deliver strong code and reasoning performance with reduced computational overhead.
Five-Phase Pre-Training and 512K Context Support
IBM built the Granite 4.1 dense decoder-only models—available in 3B, 8B, and 30B sizes—by prioritizing data quality and structured refinement over sheer compute scaling. The 15-trillion-token pre-training workflow progresses through foundational training on broad web data, heavy math and code enrichment, high-quality data annealing with chain-of-thought samples, and a final Long Context Extension (LCE) phase.
The LCE phase scales context capability from 4K up to 512K tokens using a mixture of 80% books and 20% repository code for the 8B and 30B variants. To maintain balance across varying sequence lengths, IBM performed model merges after each long-context training stage.
Key Granite 4.1 Technical Specifications
- Parameter Variants: 3B, 8B, and 30B dense decoder-only models released under Apache 2.0.
- Architecture: Grouped Query Attention (GQA), Rotary Position Embeddings (RoPE), SwiGLU activations, and RMSNorm.
- Pre-Training Corpus: ~15 trillion tokens across 5 phases, including web, technical, code, math, and long-context data.
- Fine-Tuning Setup: Supervised fine-tuning on ~4.1M curated samples using 16 compute nodes equipped with 4x NVIDIA GB200 GPUs per node.
- Efficiency Benchmark: The dense 8B instruct model matches or surpasses the previous 32B Granite 4.0-H-Small MoE model.
Data Filtering and Multi-Stage Reinforcement Learning
To prevent hallucinated behaviors during supervised fine-tuning, IBM deployed an LLM-as-Judge framework paired with rule-based deduplication and schema checks. The judge evaluated responses across six key criteria, applying hard-reject filters to automatically eliminate false premises, incorrect math, or ungrounded retrieval claims.
Following SFT, IBM used an on-policy Group Relative Policy Optimization (GRPO) pipeline with DAPO loss built on the SkyRL stack. The training spanned four sequential stages: Multi-domain RL across nine domain mixtures, RLHF on conversational chat prompts, Identity and Knowledge-Calibration RL, and a dedicated Math RL pass. The RLHF phase alone drove an average performance bump of ~18.9 points on Alpaca-Eval across all three model sizes.