README.md

---
license: gemma
base_model: google/gemma-3-4b-it
language: [ur, en]
library_name: transformers
pipeline_tag: text-generation
tags: [urdu, pakistan, gemma3, education, reasoning, instruction-tuning, adaption, autoscientist, adaptive-data, urdummlu]
datasets: [abdullah693/adaption-urdu-edu-cultural-reasoning]
model-index:
  - name: gemma-3-4b-it-urdu-edu-reasoning
    results:
      - task: {type: multiple-choice, name: Multiple Choice QA}
        dataset: {name: UrduMMLU, type: MBZUAI/UrduMMLU}
        metrics:
          - {type: accuracy, value: 46.21, name: "UrduMMLU accuracy (Urdu, zero-shot)"}
---

# Gemma-3-4B — Urdu Education & Reasoning

Submission to the **Adaption Labs AutoScientist Challenge** (Urdu language track). This is a
Gemma-3-4B model adapted for Urdu. The training data was produced by translating and localising
English knowledge corpora into Urdu with **Adaption AutoScientist** and the **Adaptive Data** pipeline.
The model is evaluated on **[UrduMMLU](https://huggingface.co/datasets/MBZUAI/UrduMMLU)**, a 26,431-question
benchmark written natively in Urdu.

Live demo: https://huggingface.co/spaces/abdullah693/urdu-edu-reasoning

## Summary of results

On UrduMMLU (Urdu prompt, zero-shot), the model scores **46.21%**, compared with **44.96%** for the
base `gemma-3-4b-it`. Accuracy improves in every domain that carries transferable knowledge (STEM,
professional, social sciences, general knowledge). The only domain that does not improve is Urdu
literature, which is discussed under Limitations.

![overall](assets/fig_overall.png)

## What this entry validates

The challenge asks whether Adaption's adaptive-data tooling can build useful, locally-relevant models.
We tested a specific claim: **adapting English knowledge corpora into Urdu with AutoScientist produces
measurable gains on a native Urdu benchmark for knowledge that is language-independent.** The result
supports the claim and also marks its boundary.

![validated effect](assets/fig_validated.png)

- Every transferable domain improved (STEM +5.9, professional +3.6, other +3.6, social sciences +2.7),
  and the model exceeded its base overall (+1.33).
- The effect does not extend to language-intrinsic content. Urdu literature declined by 2.5 points,
  because that knowledge cannot be obtained by translating English sources and requires native Urdu data.

The practical conclusion for low-resource adaptation: AutoScientist-based translation is effective for
the science, mathematics, reasoning, and social-knowledge portions of a benchmark, and should be paired
with native-language collection for culturally specific content.

## Per-domain results

| Domain | Base Gemma-3-4B | This model | Change |
|---|---|---|---|
| STEM | 46.9 | 52.8 | +5.9 |
| Professional | 49.1 | 52.7 | +3.6 |
| Other | 42.9 | 46.5 | +3.6 |
| Social sciences | 48.7 | 51.4 | +2.7 |
| Urdu literature / Humanities | 41.3 | 38.8 | −2.5 |
| **Overall** | **44.96** | **46.21** | **+1.33** |

![per-domain](assets/fig_domains.png)

Overall accuracy was measured on the full 26,431-question test set with zero unparsed responses.
Per-domain base figures are from a 1,499-question stratified sample.

## Method

![pipeline](assets/fig_pipeline.png)

1. **Source assembly.** About 40,000 examples were drawn from open English datasets that cover the
   UrduMMLU subject areas (MMLU, GSM8K, MATH, ARC, AQuA, CommonsenseQA) together with native-Urdu
   instruction, grammar, and literature data. Training splits only.
2. **Adaptation (Adaptive Data + AutoScientist).** English rows were translated and localised into
   Pakistani Urdu. The adaptation step also produced a reformulated prompt, an answer with explanation,
   and an English reasoning trace for each row.
3. **Fine-tuning.** Gemma-3-4B was supervised-fine-tuned on the adapted set.
4. **Evaluation.** Generation with answer parsing on UrduMMLU, Urdu prompt, zero-shot.

Training data: [abdullah693/adaption-urdu-edu-cultural-reasoning](https://huggingface.co/datasets/abdullah693/adaption-urdu-edu-cultural-reasoning) (~39,913 examples).

![data composition](assets/fig_data.png)

## Reproducibility

- **Harness validation.** Our evaluation reproduces the base `gemma-3-4b-it` at 44.96%, within 0.1 point
  of the 44.88% reported in the UrduMMLU paper, which confirms the protocol matches.
- **Contamination.** UrduMMLU is the held-out benchmark and was not used in training.
- **Protocol.** Urdu prompt template from the UrduMMLU repository, generation plus answer parsing,
  zero-shot, full test set.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
mid = "abdullah693/gemma-3-4b-it-urdu-edu-reasoning"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": "نظامِ شمسی میں کتنے سیارے ہیں؟"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0][ids.shape[1]:], skip_special_tokens=True))
```

## Limitations

- Urdu literature and other language-intrinsic content are weaker than the base model; native Urdu
  literary data is required to address this.
- At 4 billion parameters the model has limited factual depth and can produce incorrect answers,
  particularly on long-tail facts.
- It inherits the biases and knowledge cutoff of Gemma-3.
- Intended for research and education. Not a reliable source for examinations, religious rulings, or
  legal and medical advice.

## Citation

```bibtex
@misc{gemma3_4b_urdu_edu_2026,
  title  = {Gemma-3-4B Urdu Education and Reasoning},
  author = {abdullah693},
  year   = {2026},
  note   = {Adapted with Adaption AutoScientist; evaluated on UrduMMLU},
  url    = {https://huggingface.co/abdullah693/gemma-3-4b-it-urdu-edu-reasoning}
}
```