abdullah693's picture
Challenge-entry card: pipeline + validated-effect figures, professional prose
e637de6 verified
|
Raw
History Blame Contribute Delete
5.94 kB
metadata
license: gemma
base_model: google/gemma-3-4b-it
language:
  - ur
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - urdu
  - pakistan
  - gemma3
  - education
  - reasoning
  - instruction-tuning
  - adaption
  - autoscientist
  - adaptive-data
  - urdummlu
datasets:
  - abdullah693/adaption-urdu-edu-cultural-reasoning
model-index:
  - name: gemma-3-4b-it-urdu-edu-reasoning
    results:
      - task:
          type: multiple-choice
          name: Multiple Choice QA
        dataset:
          name: UrduMMLU
          type: MBZUAI/UrduMMLU
        metrics:
          - type: accuracy
            value: 46.21
            name: UrduMMLU accuracy (Urdu, zero-shot)

Gemma-3-4B — Urdu Education & Reasoning

Submission to the Adaption Labs AutoScientist Challenge (Urdu language track). This is a Gemma-3-4B model adapted for Urdu. The training data was produced by translating and localising English knowledge corpora into Urdu with Adaption AutoScientist and the Adaptive Data pipeline. The model is evaluated on UrduMMLU, a 26,431-question benchmark written natively in Urdu.

Live demo: https://huggingface.co/spaces/abdullah693/urdu-edu-reasoning

Summary of results

On UrduMMLU (Urdu prompt, zero-shot), the model scores 46.21%, compared with 44.96% for the base gemma-3-4b-it. Accuracy improves in every domain that carries transferable knowledge (STEM, professional, social sciences, general knowledge). The only domain that does not improve is Urdu literature, which is discussed under Limitations.

overall

What this entry validates

The challenge asks whether Adaption's adaptive-data tooling can build useful, locally-relevant models. We tested a specific claim: adapting English knowledge corpora into Urdu with AutoScientist produces measurable gains on a native Urdu benchmark for knowledge that is language-independent. The result supports the claim and also marks its boundary.

validated effect

  • Every transferable domain improved (STEM +5.9, professional +3.6, other +3.6, social sciences +2.7), and the model exceeded its base overall (+1.33).
  • The effect does not extend to language-intrinsic content. Urdu literature declined by 2.5 points, because that knowledge cannot be obtained by translating English sources and requires native Urdu data.

The practical conclusion for low-resource adaptation: AutoScientist-based translation is effective for the science, mathematics, reasoning, and social-knowledge portions of a benchmark, and should be paired with native-language collection for culturally specific content.

Per-domain results

Domain Base Gemma-3-4B This model Change
STEM 46.9 52.8 +5.9
Professional 49.1 52.7 +3.6
Other 42.9 46.5 +3.6
Social sciences 48.7 51.4 +2.7
Urdu literature / Humanities 41.3 38.8 −2.5
Overall 44.96 46.21 +1.33

per-domain

Overall accuracy was measured on the full 26,431-question test set with zero unparsed responses. Per-domain base figures are from a 1,499-question stratified sample.

Method

pipeline

  1. Source assembly. About 40,000 examples were drawn from open English datasets that cover the UrduMMLU subject areas (MMLU, GSM8K, MATH, ARC, AQuA, CommonsenseQA) together with native-Urdu instruction, grammar, and literature data. Training splits only.
  2. Adaptation (Adaptive Data + AutoScientist). English rows were translated and localised into Pakistani Urdu. The adaptation step also produced a reformulated prompt, an answer with explanation, and an English reasoning trace for each row.
  3. Fine-tuning. Gemma-3-4B was supervised-fine-tuned on the adapted set.
  4. Evaluation. Generation with answer parsing on UrduMMLU, Urdu prompt, zero-shot.

Training data: abdullah693/adaption-urdu-edu-cultural-reasoning (~39,913 examples).

data composition

Reproducibility

  • Harness validation. Our evaluation reproduces the base gemma-3-4b-it at 44.96%, within 0.1 point of the 44.88% reported in the UrduMMLU paper, which confirms the protocol matches.
  • Contamination. UrduMMLU is the held-out benchmark and was not used in training.
  • Protocol. Urdu prompt template from the UrduMMLU repository, generation plus answer parsing, zero-shot, full test set.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
mid = "abdullah693/gemma-3-4b-it-urdu-edu-reasoning"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": "نظامِ شمسی میں کتنے سیارے ہیں؟"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0][ids.shape[1]:], skip_special_tokens=True))

Limitations

  • Urdu literature and other language-intrinsic content are weaker than the base model; native Urdu literary data is required to address this.
  • At 4 billion parameters the model has limited factual depth and can produce incorrect answers, particularly on long-tail facts.
  • It inherits the biases and knowledge cutoff of Gemma-3.
  • Intended for research and education. Not a reliable source for examinations, religious rulings, or legal and medical advice.

Citation

@misc{gemma3_4b_urdu_edu_2026,
  title  = {Gemma-3-4B Urdu Education and Reasoning},
  author = {abdullah693},
  year   = {2026},
  note   = {Adapted with Adaption AutoScientist; evaluated on UrduMMLU},
  url    = {https://huggingface.co/abdullah693/gemma-3-4b-it-urdu-edu-reasoning}
}