AvinabhDutta-Dev/bangla-paraphrase-indicbartss

Bengali Paraphrase Benchmark - IndicBARTSS

TL;DR: The weakest performer among the two full-fine-tune models in this study — lower BLEU/BERTScore than BanglaT5 and the highest meaning-reversal error rate (5/100) of all four architectures tested.

A full fine-tune of ai4bharat/IndicBARTSS for Bengali paraphrase generation, trained on the BanglaParaphrase dataset.

This model is one of four architectures compared in a broader study, "Beyond BLEU: Evaluating Automated Metric Reliability for Bengali Paraphrase Generation" (GitHub repo), which benchmarks full fine-tuning vs. QLoRA and validates standard automated metrics (BLEU, ROUGE-L, BERTScore, PINC) against an LLM-as-a-judge.

Model Details

Base model	ai4bharat/IndicBARTSS
Architecture	mBART-style encoder-decoder (seq2seq)
Fine-tuning method	Full fine-tuning
Training data	BanglaParaphrase, 100,000-pair subset
Max sequence length	96 tokens
Effective batch size	32
Trained on	Google Colab (free-tier T4 GPU)
License	MIT (matches base model)

Trained on a single free-tier Colab T4 GPU; exact carbon footprint not tracked. Despite comparable full-fine-tuning compute cost to BanglaT5, results did not match BanglaT5's performance — see Limitations for discussion.

Intended Use

Generating Bengali paraphrases of an input sentence — e.g., data augmentation for downstream Bengali NLP tasks, or research on low-resource paraphrase generation. Evaluated only on the BanglaParaphrase test distribution; not evaluated on other domains, dialects, or languages.

Out-of-Scope Use

• Not intended for languages other than Bengali, or for code-mixed/transliterated text.
• Not intended for high-stakes or unreviewed automated use — this study's own LLM-judge evaluation found a measurable meaning-reversal error rate (5/100 judged examples; see Limitations), so outputs should be reviewed before use in any context where semantic accuracy matters.
• Not a general-purpose Bengali text generation or summarization model — it is trained specifically for sentence-level paraphrasing and has not been evaluated for other generation tasks.

How to Use

from transformers import MBartForConditionalGeneration, AlbertTokenizer

model_name = "AvinabhDutta-Dev/bangla-paraphrase-indicbartss"
tokenizer = AlbertTokenizer.from_pretrained(model_name, do_lower_case=False, use_fast=False, keep_accents=True)
model = MBartForConditionalGeneration.from_pretrained(model_name)

text = "এই বাড়িতে সাধারণত কোনো ঝগড়াঝাঁটি হয় না।"
inputs = tokenizer(text, return_tensors="pt")
bn_tag_id = tokenizer._convert_token_to_id_with_added_voc("<2bn>")
output_ids = model.generate(**inputs, decoder_start_token_id=bn_tag_id, max_length=96)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Sample Outputs

Real outputs from this model on the held-out test set, judged 5/5 for both meaning preservation and fluency by the study's LLM-judge evaluation:

Source	Generated
ইউনিভার্সিটির ল্যাবে কাজ করতেন।	তিনি ইউনিভার্সিটির ল্যাবে কাজ করতেন।
"স্যামুরাই এক্স" এনিমে সিরিজ: বাস্তবতা নাকি শুধুই ফিকশন?	"স্যামুরাই এক্স" এনিমে সিরিজ: বাস্তবতা কি শুধুই ফিকশন?

Evaluation Results

Evaluated on the full, held-out BanglaParaphrase test set (23,332 examples), using the same pipeline and test data across all four models in this study.

Metric	Score
BLEU	14.07
ROUGE-L	0.417
BERTScore (F1, BanglaBERT)	0.939
PINC (lexical novelty vs. source)	0.565

LLM-as-a-Judge Validation

A 400-pair, four-model symmetric subset (100 examples per model, identical sources) was independently scored by Gemini 3.5 Flash-Lite on semantic meaning preservation and fluency (1–5 scale), with a binary meaning-reversal flag.

Metric	Score
Mean judge meaning-preservation score	4.32 / 5
Mean judge fluency score	4.80 / 5

Error breakdown (out of 100 judged examples):

Category	Count
Valid paraphrase	87
Hallucination / semantic drift	8
Meaning reversal	5
Severe disfluency	0

Related Models (this study)

• BanglaT5 (full fine-tune)
• TigerLLM-1B (QLoRA)
• Qwen2.5-1.5B (QLoRA)

Limitations

• Of the four models in this study, IndicBARTSS showed the highest rate of meaning-reversal errors (5/100 judged examples) — cases where the generated paraphrase inverted or contradicted the source sentence's meaning (e.g., describing subjects returning when the source said they were departing), despite being grammatically fluent. This is a real failure mode to be aware of, not just a benchmark statistic — outputs should not be used without review in contexts where semantic accuracy is critical.

• Trained on a 100K-example subset (not the full ~420K available training pairs) and evaluated with greedy decoding, for compute tractability.

• BLEU/ROUGE-L are known to correlate only weakly with actual paraphrase quality (see the parent study's Notebook 07); BERTScore and the LLM-judge results are more informative for assessing this model.

• Reproducibility: Trained on a fixed 100,000-pair subset of BanglaParaphrase with a documented train/val/test split (see processed_data/metadata.json in the GitHub repo); decoding was fully deterministic (greedy decoding), so generation outputs are reproducible given the same checkpoint and inputs.

Citation

If you use this model, please cite both the original IndicBARTSS/BanglaParaphrase work and this study:

@misc{dabre2021indicbart,
  title={IndicBART: A Pre-trained Model for Indic Natural Language Generation},
  author={Dabre, Raj and others},
  year={2021}
}

@misc{banglaparaphrase2022,
  title={BanglaParaphrase: A High-Quality Bengali Paraphrase Dataset},
  author={Akil, Ajwad and others},
  year={2022}
}

See the project GitHub repository for the full paper, code, and citation for this study.

Framework Versions

• Transformers 5.13.1