subidhkhanal/sarvam-1-hindi-citizen-profile-lora

Sarvam-1 · Hindi → Citizen Profile JSON (LoRA)

A QLoRA fine-tune of sarvamai/sarvam-1 that converts spoken-Hindi self-descriptions into a strict 20-field CitizenProfile JSON schema. Designed for form-filling assistants at government service centers, banking KYC, and welfare-scheme applications, where applicants describe themselves in natural Hindi and the downstream form needs structured fields.

Headline (290-entry held-out test set):

Metric	Value
Parse rate	290 / 290 (100.0%)
Schema validation rate	290 / 290 (100.0%)
Field accuracy (strict exact-match)	5692 / 5800 (98.1%)
synthetic_dense gold-NN field acc (target distribution)	98.3%

Source code, data preparation pipeline, and training/eval scripts: github.com/subidhkhanal/hindi-form-agent.

Quick start

import json
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

BASE = "sarvamai/sarvam-1"
ADAPTER = "subidhkhanal/sarvam-1-hindi-citizen-profile-lora"

tokenizer = AutoTokenizer.from_pretrained(BASE)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

base = AutoModelForCausalLM.from_pretrained(
    BASE,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True,
    ),
    device_map="auto",
)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()

# Exact prompt format used at training time
INPUT_DELIMITER = "हिंदी पाठ:"
OUTPUT_DELIMITER = "संरचित JSON:"

hindi_text = (
    "मेरा नाम रामलाल है। मेरी उम्र चालीस साल है। "
    "मैं बिहार के मधुबनी जिले में रहता हूं। खेत मजदूरी का काम करता हूं, "
    "महीने में लगभग पच्चीस हजार रुपये कमाता हूं। आधार कार्ड और बैंक खाता है।"
)
prompt = f"{INPUT_DELIMITER}\n{hindi_text}\n\n{OUTPUT_DELIMITER}\n"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(
        **inputs, max_new_tokens=512, do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )
gen = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

# Model may emit trailing junk after the JSON; raw_decode handles it
predicted, _ = json.JSONDecoder().raw_decode(gen)
print(json.dumps(predicted, ensure_ascii=False, indent=2))

Expected output:

{
  "full_name": "रामलाल",
  "age": 40,
  "gender": "male",
  "marital_status": null,
  "district": "मधुबनी",
  "state": "बिहार",
  "occupation": "खेत मजदूर",
  "monthly_income_inr": 25000,
  "has_aadhaar": true,
  "has_bank_account": true,
  "bank_name": null
}

(plus null for the remaining fields)

Output schema

20-field Pydantic CitizenProfile (full definition in repo). All fields are Optional — the model is trained to emit null for fields the input does not mention, rather than hallucinate.

Closed-set (Literal) fields use canonical labels:

• gender ∈ {"male", "female", "other"}
• marital_status ∈ {"single", "married", "widowed", "divorced"}
• caste_category ∈ {"general", "obc", "sc", "st"}
• has_aadhaar, has_pan, has_voter_id, has_ration_card, has_bank_account ∈ {true, false}

Government ID fields are booleans only — actual ID numbers are never captured in training data (privacy by design).

Training

Knob	Value
Base model	sarvamai/sarvam-1 (2.5 B params)
Trainable params	6,422,528 (0.25%)
Method	QLoRA: 4-bit NF4 + LoRA
LoRA r / α / dropout	16 / 32 / 0.05
LoRA target modules	q_proj, k_proj, v_proj, o_proj
Epochs	3
Effective batch	8 (2 × grad-accum 4)
LR	2e-4 (cosine, 5% warmup)
Max seq length	1024
Mixed precision	fp16 (T4 = Turing, no bf16)
Optimizer	paged AdamW 8-bit
Loss masking	Completion-only (DataCollatorForCompletionOnlyLM)
Hardware	Kaggle T4 single GPU, ~58 min

Training data

2,270 train / 290 val / 290 test entries from four sources:

Source	n	Description
handcrafted_seed	10	Dense first-person anchor examples
hiner (subsampled)	2,000	AI4Bharat's HiNER-collapsed — real-Hindi distributional signal, sparse
synthetic_dense	800	First-person Devanagari citizen profiles generated against a persona matrix
hardcases	40	Stress tests: disfluency, negation, approximate values, mixed scripts, etc.

Full data prep methodology and the persona/domain references live in data_prep/ in the GitHub repo.

Evaluation

Strict exact-match field accuracy on a held-out test set (290 entries, never seen during training). Test set is itself produced by a deterministic random.seed(42) split, fully reproducible.

By source

Source	n	Parse	Schema	Field acc	Gold-NN field acc
synthetic_dense (target distribution)	80	100.0%	100.0%	98.7%	98.3%
hardcases	10	100.0%	100.0%	95.5%	93.4%
hiner	200	100.0%	100.0%	98.0%	80.8%

By field (gold-non-null accuracy, sorted)

Field	Overall	Gold-NN	n (gold-NN)
marital_status	100.0%	100.0%	80
father_or_husband_name	100.0%	100.0%	59
number_of_dependents	99.3%	100.0%	3
pincode	100.0%	100.0%	10
caste_category	100.0%	100.0%	78
religion	100.0%	100.0%	80
has_aadhaar	100.0%	100.0%	81
has_pan	100.0%	100.0%	9
has_voter_id	100.0%	100.0%	9
has_ration_card	100.0%	100.0%	26
has_bank_account	100.0%	100.0%	51
bank_name	100.0%	100.0%	25
age	99.7%	98.9%	90
gender	99.7%	98.9%	90
state	99.0%	97.8%	90
village_or_town	99.0%	95.1%	41
monthly_income_inr	98.6%	94.3%	70
full_name	88.6%	88.6%	290
occupation	94.8%	86.0%	86
district	84.1%	82.4%	250

Highlights:

• Every closed-set (Literal-typed) field at 100% gold-NN — the model canonicalizes free Hindi text to enum values reliably (अनुसूचित जाति → sc, मुस्लिम → इस्लाम, विधवा → widowed).
• bank_name at 100% (25/25) — the conditional rule (extract iff the speaker named the bank, else null) generalized perfectly.
• Devanagari number → int: चौवालीस → 44, इकहत्तर → 71, बावन → 52, etc.

Weak spots (district, full_name, occupation) are bottlenecked by:

1. HiNER label noise (party names mistagged as PER, locations truncated).
2. Strict exact-match: predicting "स्टेशनरी दुकानदार" when gold is "दुकानदार" counts as wrong even though both are correct. Real semantic accuracy is higher.

Intended use

Demo / research / educational. Drop-in inference component for Hindi form-filling assistants, voice-to-form pipelines, and structured-extraction research.

Not for production use without further validation. Government ID-related fields are booleans only (never extracts ID numbers — that's a deliberate privacy boundary, not an oversight).

Limitations

1. Strict exact-match underreports. Predictions semantically equivalent to gold (canonical spelling variants, more-specific occupations) score as wrong. A normalized or semantic-match metric would raise the field-accuracy number meaningfully without changing the underlying capability.
2. HiNER label noise. The 2,000 HiNER training entries provide real-Hindi distributional signal but carry inherited labeling quirks (party names as full_name, multi-word locations truncated). The data-prep stoplist filters the worst cases; some residual noise reaches the model.
3. Generation may over-run past the JSON. The model occasionally emits duplicate JSON blocks or Hindi narrative after the closing }. Always parse with json.JSONDecoder().raw_decode() (takes the first valid object) or use the StoppingCriteria provided in the repo's evaluate.py.
4. First-person dense inputs are the sweet spot. Sparse third-person news sentences (HiNER-style inputs) still produce well-formed JSON but value accuracy drops — that's a property of HiNER labels, not a model failure.

License

The adapter inherits the base model's license: Sarvam non-commercial license. Not for commercial use.

Citation / acknowledgement

Built on top of Sarvam-1 by Sarvam AI, and the HiNER-collapsed NER dataset by AI4Bharat / CFILT. See the GitHub repo for the full data prep acknowledgements and per-source methodology.

Full source, data, and reproducible training scripts: github.com/subidhkhanal/hindi-form-agent.