--- language: - en - hi - bn - mr - ta library_name: transformers license: other pipeline_tag: text-generation tags: - small-language-model - slm - indic - multilingual - edge-ai - logistics - customer-support - llama --- # Indic-Delivery-Rider-SLM (40M parameters) A small, from-scratch, Llama-style language model for **offline** assistance of delivery partners and drivers (e.g., Swiggy, Zomato, Porter) in India. Languages: English, Hindi, Bengali, Marathi, Tamil. > **Experimental research model.** Trained for \~28 minutes on one T4 > GPU on a small dataset. Non-English domain data is > **machine-translated** and has not been reviewed by native speakers. > Not for payments, safety or legal advice. ## Intended Use Case - Offline edge execution in areas with poor connectivity. - Quick responses for address changes, payment status checks, COD handling, and order issues. ## Architecture ### Model Architecture - **Architecture:** Decoder-only Transformer - **Total Parameters:** 39.7 million - **Transformer Layers:** 10 - **Hidden Dimension:** 512 - **Attention Heads:** 8 - **Key-Value Heads:** 4 (Grouped-Query Attention, GQA) - **Feed-Forward Network:** SwiGLU-activated MLP with an intermediate dimension of 1,408 - **Normalization:** Root Mean Square Layer Normalization (RMSNorm) - **Positional Encoding:** Rotary Position Embeddings (RoPE) - **Embedding Configuration:** Tied input and output embeddings - **Maximum Context Length:** 384 tokens - **Tokenizer:** SentencePiece BPE with a 20,000-token vocabulary and byte fallback - **Hugging Face Compatibility:** Exports as `LlamaForCausalLM`, with output logits verified against the original implementation. **Tokenizer fertility on held-out text:** - **English:** 1.28 tokens/word (GPT-2 BPE: 1.30) - **Hindi:** 1.26 tokens/word (GPT-2 BPE: 7.76) - **Bengali:** 1.45 tokens/word (GPT-2 BPE: 12.79) - **Marathi:** 1.57 tokens/word (GPT-2BPE: 10.50) - **Tamil:** 1.81 tokens/word (GPT-2 BPE: 23.44) ## Prompt format `<|user|>{question}<|assistant|><|lang|>` then the model writes the answer and ``. `<|lang|>` is one of `<|en|>`, `<|hi|>`, `<|bn|>`, `<|mr|>`, `<|ta|>`; prefill it to force the reply language, or omit it to let the model choose. ## Usage ``` python import torch, sentencepiece as spm from huggingface_hub import snapshot_download from transformers import AutoModelForCausalLM path = snapshot_download("learner-subodh/Indic-Delivery-Rider-SLM") sp = spm.SentencePieceProcessor(model_file=f"{path}/tokenizer.model") model = AutoModelForCausalLM.from_pretrained(path).eval() q = "Customer wants to change the delivery address after pickup. What should I tell them?" ids = [sp.bos_id(), sp.piece_to_id("<|user|>")] + sp.encode(q) + [sp.piece_to_id("<|assistant|>"), sp.piece_to_id("<|en|>")] out = model.generate(torch.tensor([ids]), max_new_tokens=160, do_sample=True, temperature=0.6, top_k=40, top_p=0.9, repetition_penalty=1.15, eos_token_id=sp.eos_id(), pad_token_id=sp.pad_id()) print(sp.decode(out[0][len(ids):].tolist())) ``` ## Training data - **Bitext customer-support dataset** (English, `bitext/Bitext-customer-support-llm-chatbot-training-dataset`) - **Sentence-level machine translation of a stratified subset into the other languages with NLLB-200 (600M distilled)**, filtered for placeholders, script, repetition and length - **Native-script instruction data from the Aya dataset** (`CohereForAI/aya_dataset`, Apache-2.0) ## Evaluation (held-out) ### Language Modelling Bits per byte (BPB) measures the model's answer-text prediction performance, normalised by text bytes (lower is better). Perplexity (PPL) is reported as a supplementary metric. - **`domain`:** Support-style data. - **`general`:** Aya dataset. - **`resp_tokens`:** Number of response tokens evaluated. | Language | Data Split | Bits per Byte | Perplexity (PPL) | Response Tokens | |----------|------------|--------------:|-----------------:|----------------:| | Bengali (bn) | Domain | 0.204 | 6.614 | 10,821 | | Bengali (bn) | General | 0.825 | 193.146 | 5,206 | | English (en) | Domain | 0.258 | 2.297 | 52,493 | | Hindi (hi) | Domain | 0.199 | 4.506 | 12,750 | | Hindi (hi) | General | 0.694 | 51.689 | 3,170 | | Marathi (mr) | Domain | 0.217 | 7.363 | 11,822 | | Marathi (mr) | General | 0.668 | 36.486 | 4,682 | | Tamil (ta) | Domain | 0.177 | 7.704 | 12,050 | | Tamil (ta) | General | 0.731 | 119.382 | 9,661 | ### Multilingual Generation Generation checks on hand-written rider prompts (3 per language; tiny sample, indicative only). - **`lang_fidelity`:** reply is in the expected script (and Hindi/Marathi marker words). - **`auto_lang_id`:** model picks the right language tag unprompted. - **`distinct2`:** 1 - repetition (higher indicates greater lexical diversity). - **`stop_rate`:** fraction of generations terminating with the expected stop token. - **`avg_words`:** average generated words per response. | Language | Lang Fidelity | Auto Lang ID | Distinct-2 | Stop Rate | Avg. Words | |----------|--------------:|-------------:|-----------:|----------:|-----------:| | Bengali (bn) | 1.00 | 1.00 | 0.95 | 0.67 | 80.00 | | English (en) | 1.00 | 1.00 | 0.98 | 0.67 | 75.00 | | Hindi (hi) | 1.00 | 1.00 | 0.94 | 1.00 | 91.67 | | Marathi (mr) | 0.67 | 1.00 | 0.65 | 0.67 | 57.67 | | Tamil (ta) | 1.00 | 1.00 | 0.99 | 0.67 | 89.67 | ### Evaluation Scope - **Language modelling:** Evaluates next-token prediction on held-out domain and general-language text; does not establish factual correctness. - **Generation:** Preliminary assessment of language consistency and generation behaviour using 15 manually written prompts. - **Tokenizer:** Fertility comparison measures tokenisation efficiency against GPT-2 BPE, not language understanding. - **Task performance:** Intent classification Macro-F1, entity extraction F1 and task completion rate have not been independently benchmarked. - **Deployment:** CPU inference latency, peak RAM, throughput and quantised inference performance have not yet been benchmarked. - **Human evaluation:** No comprehensive native-speaker or human assessment has been conducted. ## Limitations - Small capacity, small data: it can state wrong policies fluently, blur Hindi and Marathi (same script), and repeat itself. - Machine-translated training text carries translation errors. - General-language performance is limited, as reflected in held-out perplexity. - Generation checks use only three prompts per language and are not statistically representative. - Route anything involving money, injuries or disputes to a human or a fixed template. ## Licensing note - Mixed provenance: Bitext (CDLA-Sharing-1.0), Aya (Apache-2.0), and translations produced by NLLB-200 (model weights CC-BY-NC-4.0). - Check these terms with your own counsel before any commercial use. - Weights are released for education, research and evaluation.