Research from Dharma AI published on Hugging Face reveals that applying Direct Preference Optimization (DPO) to vision-language OCR models cuts text degeneration loops by an average of 59.4%. By feeding models their own repetition failures as rejected training pairs across 23,726 Brazilian Portuguese documents, researchers successfully bypassed the strict limitations of supervised fine-tuning. The findings demonstrate how alignment techniques originally designed for conversational chatbots can directly eliminate structural errors in objective, non-chat AI workflows.
Targeting Repetition Loops at the Completion Level
Standard supervised fine-tuning (SFT) trains models on a token-by-token basis, evaluating predictions in isolation. Because each step is evaluated locally, SFT fails to penalize full-sequence repetition loops, allowing models to enter high-probability "attractor regions" during inference that sustain output loops until hitting maximum token limits.
To break this geometry, Dharma AI inverted the conventional data curation process during the development of DharmaOCR. Instead of filtering out model-generated repetition loops as low-quality noise, researchers preserved those specific failure modes as rejected examples in DPO training pairs.
By contrasting clean transcriptions against the model's own degenerate outputs, the post-SFT DPO stage functions as implicit unlikelihood training. This forces the model to evaluate completions as entire sequences rather than isolated local token probabilities.
Results Across Five Vision-Language Architectures
The Dharma AI team evaluated this approach across five distinct open-source vision-language model families to measure its impact on structured text extraction.
- DPO reduced text degeneration in every tested model family by an average of 59.4%, with a peak reduction of 87.6% on Nanonets-OCR2-3B.
- Vanilla degeneration rates ranged from under 1% to over 33% across tested open-source families prior to tuning.
- The benchmark dataset comprised 23,726 Brazilian Portuguese structured documents evaluated by an automated LLM judge.
- In Qwen2.5-VL-3B, vanilla degeneration initially rose from 0.60% to 3.23% after SFT as task capability grew, before dropping to 1.41% following DPO.
Implementation Trade-Offs and Limitations
While effective, the technique introduces specific engineering trade-offs. The pipeline relies on an automated LLM judge to pair clean transcriptions with failure outputs, as mechanical rule-based mechanisms cannot validate whether non-looping text is factual.
Additionally, the study highlights that task capability and degeneration resistance move independently during SFT. As models become more capable of generating long structured sequences, their exposure to degeneration attractors can briefly increase before the second-stage optimization corrects the underlying geometry.