CycleDistill

WAT 2025 Mumbai, India · ACL Anthology

CycleDistill

Bootstrapping machine translation with language models that learn from their own cycles.

Read the paper View code

Deepon HalderThanmay JayakumarRaj Dabre

हिन्दी বাংলা മലയാളം
model M0 pretrained
01 Generate synthetic pairs
02 Distill hard + soft signals
03 Repeat refine the model
20–30chrF gain after cycle one
1–4parallel examples needed
03Indic languages evaluated
03distillation cycles studied
Explore
01 / Overview The starting point

The bottleneck is not language.

It is parallel data.

Dedicated translation systems excel when bilingual corpora are abundant. For low-resource languages, those aligned sentence pairs are often scarce—or do not exist at all.

CycleDistill changes the starting point. A capable language model translates monolingual text, learns from the synthetic pairs it just created, then repeats the process. Each pass turns unlabeled language into a stronger translation system.

Start with Monolingual corpus + just 1–4 examples
Cycle through Generate · Distill synthetic supervision
Arrive at Stronger MT model or a smaller student
01

Self-supervised MT

Bootstraps translation quality from monolingual corpora with minimal parallel supervision.

02

Richer distillation

Preserves top-k token probabilities so the student learns more than the final decoded sequence.

03

Model compression

Transfers translation ability from 9B→2B and 8B→3B models while improving student performance.

02 / Method One loop, richer each time

Generate.
Distill. Repeat.

Three rounds are enough to expose the pattern: a large first jump, smaller gains after that, and a plateau by the third cycle.

  1. 1
    Mi → Di

    Generate synthetic translations

    The current model translates 20,000 monolingual Indic sentences into English, creating a fresh parallel corpus for the next pass.

  2. 2
    Di → Mi+1

    Distill into the next model

    The synthetic pairs refine the same architecture and train a smaller student. CycleDistill studies both sequence-level targets and token-level probability distributions.

  3. 3
    Mi+1 ↻

    Promote and cycle again

    The newly distilled model becomes the next teacher. Its translations seed another dataset, closing the loop for up to three iterations.

Soft distribution-preserving distillation

Do not only copy the answer.
Copy the uncertainty.

Instead of retaining only the teacher’s chosen token, CycleDistill records the top 20 candidates and their probabilities. The student matches this richer probability landscape with KL divergence loss.

Teacher · token 2
read0.60
likes0.30
reading0.10
Student · token 2
read0.50
likes0.40
prefers0.10
03 / Results Table 1, made explorable

The first cycle does the heavy lifting.

Compare zero-shot baseline chrF with the first iterative-distillation cycle across Bengali, Hindi, and Malayalam. Values below are taken directly from Table 1.

Base After cycle 1

24.6
50.9
BENBengali
28.8
58.4
HINHindi
23.8
48.3
MALMalayalam

0-shot · iterative distillation (DD1) · IN22 Gen benchmark

A

One example beats four.

One-shot prompting produced the best overall performance. Longer four-shot prompts diluted context, with sharper declines for morphologically rich Bengali and Malayalam.

1-shot
B

Small models can absorb efficiently.

LLaMA 3B’s average absorption rate was 1.190, versus 0.628 for Gemma 2B.

C

There is a clear boundary.

CycleDistill improved Nepali, but not Manipuri—if the base model cannot produce usable translations, the cycle has no reliable signal to amplify.

D

More cycles are not always better.

Performance often plateaus after cycle two; errors can compound when synthetic data is misaligned.

04 / Paper figures Original views

See the method as published.

Selected figures from the WAT 2025 paper. Open the PDF for complete tables, analyses, and appendices.

CycleDistill framework from initial model through synthetic data generation to self and smaller student models
Figure 2 Overview of the CycleDistill framework.
Teacher and student token distributions connected by KL divergence loss
Figure 3 Soft distribution-preserving distillation.
chrF rises sharply after the first distillation iteration and then plateaus
Figure 1 chrF over distillation cycles.

Figures reproduced from Halder et al., WAT 2025. © Association for Computational Linguistics.

05 / Citation Build on the work

Use it.
Test it. Cite it.

DOI ↗ Code ↗ ACL Anthology ↗
BibTeX 
@inproceedings{halder-etal-2025-cycledistill,
  title     = {{CycleDistill}: Bootstrapping Machine
               Translation using {LLM}s with Cyclical Distillation},
  author    = {Halder, Deepon and Jayakumar, Thanmay
               and Dabre, Raj},
  booktitle = {Proceedings of the Twelfth Workshop
               on Asian Translation},
  month     = dec,
  year      = {2025},
  address   = {Mumbai, India},
  publisher = {Association for Computational Linguistics},
  pages     = {78--92},
  doi       = {10.18653/v1/2025.wat-1.7}
}