Hugging Face and the UAE's Technology Innovation Institute (TII) have launched QIMMA, an open-source evaluation platform designed to accurately benchmark Arabic Large Language Models (LLMs). The project consolidates over 52,000 validated samples across 109 benchmark subsets to eliminate systemic errors found in translated datasets. AI developers building regional models can now access transparent, per-sample inference outputs to assess performance across seven key domains.
Automated and Human Quality Validation
Standard Arabic evaluations often suffer from direct English translations, incorrect ground-truth labels, and encoding errors. To resolve these issues, the researchers created a multi-stage validation pipeline applied to every evaluation sample before testing any models.
First, two automated LLMs—Qwen3-235B-A22B-Instruct and DeepSeek-V3-671B—independently score each sample on a 10-point rubric assessing text, formatting, and answer accuracy. Samples scoring below 7/10 by both models are dropped immediately, while samples with conflicting scores proceed to native Arabic human annotators. This pipeline revealed that even established datasets contained systematic flaws; for example, 3.1% of samples in ArabicMMLU and 2.3% in MizanQA were discarded due to quality defects.
Code Evaluation and Benchmark Rankings
QIMMA consolidates 109 subsets from 14 source benchmarks into over 52,000 samples with 99% native Arabic content. It is also the first Arabic leaderboard to incorporate code generation benchmarks, adapting HumanEval+ and MBPP+ with Arabic problem statements.
Key details from the benchmark suite and initial evaluation include:
- Massive Prompt Refinement: The team modified 88% of HumanEval+ prompts and 81% of MBPP+ prompts in 3LM's Arabic adaptations to normalize grammar and standardize mathematical terms.
- Top Overall Model: Qwen3.5-397B-A17B-FP8 achieved the highest overall average score of 68.06 across all tasks.
- Specialized Native Performance: Arabic-tailored models like Jais-2-70B-Chat led in domain-specific tasks, taking top ranks in ArabicMMLU (81.29) and ArabCulture (83.24).
- Open Framework: Evaluation scripts run on LightEval, EvalPlus, and FannOrFlop, with per-sample outputs made public on GitHub.
Despite these advancements, an honest limitation remains in technical coding tasks. Specialized Arabic models continue to lag behind large multilingual baselines in code generation, where Qwen3.5-397B led both coding benchmarks. Additionally, ensuring dialectal accuracy across diverse Arab regions still requires time-consuming manual human review.
Why It Matters
For engineering teams building Arabic-capable models, QIMMA provides a reliable testing standard that eliminates corrupted gold answers and cultural translation artifacts. By shifting from noisy aggregate benchmarks to validated datasets with public inference logs, developers can now pinpoint genuine reasoning and linguistic capabilities in regional AI deployment.