Blame for README.md

---
language:
- en
- hi
- bn
- mr
- ta
library_name: transformers
license: other
pipeline_tag: text-generation
tags:
- small-language-model
- slm
- indic
- multilingual
- edge-ai
- logistics
- customer-support
- llama
---

# Indic-Delivery-Rider-SLM (40M parameters)

A small, from-scratch, Llama-style language model for **offline**
assistance of delivery partners and drivers (e.g., Swiggy, Zomato,
Porter) in India.

Languages: English, Hindi, Bengali, Marathi, Tamil.

> **Experimental research model.** Trained for \~28 minutes on one T4
> GPU on a small dataset. Non-English domain data is
> **machine-translated** and has not been reviewed by native speakers.
> Not for payments, safety or legal advice.

## Intended Use Case

-   Offline edge execution in areas with poor connectivity.
-   Quick responses for address changes, payment status checks, COD
    handling, and order issues.

## Architecture

### Model Architecture

- **Architecture:** Decoder-only Transformer
- **Total Parameters:** 39.7 million
- **Transformer Layers:** 10
- **Hidden Dimension:** 512
- **Attention Heads:** 8
- **Key-Value Heads:** 4 (Grouped-Query Attention, GQA)
- **Feed-Forward Network:** SwiGLU-activated MLP with an intermediate dimension of 1,408
- **Normalization:** Root Mean Square Layer Normalization (RMSNorm)
- **Positional Encoding:** Rotary Position Embeddings (RoPE)
- **Embedding Configuration:** Tied input and output embeddings
- **Maximum Context Length:** 384 tokens
- **Tokenizer:** SentencePiece BPE with a 20,000-token vocabulary and byte fallback
- **Hugging Face Compatibility:** Exports as `LlamaForCausalLM`, with output logits verified against the original implementation.

**Tokenizer fertility on held-out text:** 
- **English:** 1.28 tokens/word (GPT-2 BPE: 1.30)
- **Hindi:** 1.26 tokens/word (GPT-2 BPE: 7.76)
- **Bengali:** 1.45 tokens/word (GPT-2 BPE: 12.79)
- **Marathi:** 1.57 tokens/word (GPT-2BPE: 10.50)
- **Tamil:** 1.81 tokens/word (GPT-2 BPE: 23.44)

## Prompt format

`<s><|user|>{question}<|assistant|><|lang|>` then the model writes the
answer and `</s>`.

`<|lang|>` is one of `<|en|>`, `<|hi|>`, `<|bn|>`, `<|mr|>`, `<|ta|>`;
prefill it to force the reply language, or omit it to let the model
choose.

## Usage

``` python
import torch, sentencepiece as spm
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM

path = snapshot_download("learner-subodh/Indic-Delivery-Rider-SLM")
sp = spm.SentencePieceProcessor(model_file=f"{path}/tokenizer.model")
model = AutoModelForCausalLM.from_pretrained(path).eval()

q = "Customer wants to change the delivery address after pickup. What should I tell them?"
ids = [sp.bos_id(), sp.piece_to_id("<|user|>")] + sp.encode(q) + [sp.piece_to_id("<|assistant|>"), sp.piece_to_id("<|en|>")]
out = model.generate(torch.tensor([ids]), max_new_tokens=160, do_sample=True, temperature=0.6, top_k=40, top_p=0.9,
                     repetition_penalty=1.15, eos_token_id=sp.eos_id(), pad_token_id=sp.pad_id())
print(sp.decode(out[0][len(ids):].tolist()))
```

## Training data

-   **Bitext customer-support dataset** (English,
    `bitext/Bitext-customer-support-llm-chatbot-training-dataset`)
-   **Sentence-level machine translation of a stratified subset into the
    other languages with NLLB-200 (600M distilled)**, filtered for
    placeholders, script, repetition and length
-   **Native-script instruction data from the Aya dataset**
    (`CohereForAI/aya_dataset`, Apache-2.0)

## Evaluation (held-out)

### Language Modelling

Bits per byte (BPB) measures the model's answer-text prediction performance, normalised by text bytes (lower is better). Perplexity (PPL) is reported as a supplementary metric.

- **`domain`:** Support-style data.
- **`general`:** Aya dataset.
- **`resp_tokens`:** Number of response tokens evaluated.

| Language | Data Split | Bits per Byte | Perplexity (PPL) | Response Tokens |
|----------|------------|--------------:|-----------------:|----------------:|
| Bengali (bn) | Domain | 0.204 | 6.614 | 10,821 |
| Bengali (bn) | General | 0.825 | 193.146 | 5,206 |
| English (en) | Domain | 0.258 | 2.297 | 52,493 |
| Hindi (hi) | Domain | 0.199 | 4.506 | 12,750 |
| Hindi (hi) | General | 0.694 | 51.689 | 3,170 |
| Marathi (mr) | Domain | 0.217 | 7.363 | 11,822 |
| Marathi (mr) | General | 0.668 | 36.486 | 4,682 |
| Tamil (ta) | Domain | 0.177 | 7.704 | 12,050 |
| Tamil (ta) | General | 0.731 | 119.382 | 9,661 |

### Multilingual Generation

Generation checks on hand-written rider prompts (3 per language; tiny
sample, indicative only).

-   **`lang_fidelity`:** reply is in the expected script (and Hindi/Marathi
    marker words).
-   **`auto_lang_id`:** model picks the right language tag unprompted.
-   **`distinct2`:** 1 - repetition (higher indicates greater lexical
    diversity).
-   **`stop_rate`:** fraction of generations terminating with the expected
    stop token.
-   **`avg_words`:** average generated words per response.

| Language | Lang Fidelity | Auto Lang ID | Distinct-2 | Stop Rate | Avg. Words |
|----------|--------------:|-------------:|-----------:|----------:|-----------:|
| Bengali (bn) | 1.00 | 1.00 | 0.95 | 0.67 | 80.00 |
| English (en) | 1.00 | 1.00 | 0.98 | 0.67 | 75.00 |
| Hindi (hi) | 1.00 | 1.00 | 0.94 | 1.00 | 91.67 |
| Marathi (mr) | 0.67 | 1.00 | 0.65 | 0.67 | 57.67 |
| Tamil (ta) | 1.00 | 1.00 | 0.99 | 0.67 | 89.67 |

### Evaluation Scope

-   **Language modelling:** Evaluates next-token prediction on held-out
    domain and general-language text; does not establish factual
    correctness.
-   **Generation:** Preliminary assessment of language consistency and
    generation behaviour using 15 manually written prompts.
-   **Tokenizer:** Fertility comparison measures tokenisation efficiency
    against GPT-2 BPE, not language understanding.
-   **Task performance:** Intent classification Macro-F1, entity
    extraction F1 and task completion rate have not been independently
    benchmarked.
-   **Deployment:** CPU inference latency, peak RAM, throughput and
    quantised inference performance have not yet been benchmarked.
-   **Human evaluation:** No comprehensive native-speaker or human
    assessment has been conducted.

## Limitations

-   Small capacity, small data: it can state wrong policies fluently,
    blur Hindi and Marathi (same script), and repeat itself.
-   Machine-translated training text carries translation errors.
-   General-language performance is limited, as reflected in held-out
    perplexity.
-   Generation checks use only three prompts per language and are not
    statistically representative.
-   Route anything involving money, injuries or disputes to a human or a
    fixed template.

## Licensing note

-   Mixed provenance: Bitext (CDLA-Sharing-1.0), Aya (Apache-2.0), and
    translations produced by NLLB-200 (model weights CC-BY-NC-4.0).
-   Check these terms with your own counsel before any commercial use.
-   Weights are released for education, research and evaluation.