Akhorika E2B Base (ananddey/akhorika-e2b-base)

Akhorika E2B Base (আখৰিকা) is a foundational 2.69B parameter language model (1.91B active text decoder parameters) continued-pretrained (CPT) on Assamese language corpora, based on Google's Gemma 4 E2B architecture.

Model Details

• Model Name: Akhorika E2B Base
• Base Architecture: Gemma4ForConditionalGeneration (Google Gemma 4 E2B)
• Parameters: 2.69B total (1.91B text decoder parameters)
• Tokenizer: Custom 32,000 vocab Assamese-first SentencePiece Unigram tokenizer (replacing the native 256k tokenizer)
• Embedding Initialization: FOCUS (semantic subword surface-form projection from base Gemma embeddings)
• Languages: Assamese (as), English (en)
• Training Type: Continued Pretraining (CPT)
• Checkpoint: Step 821 / 821 (epoch 1 complete)
• Eval Loss: 4.09 (PPL ~60 on the held-out Assamese eval set)

Quickstart & Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ananddey/akhorika-e2b-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa"
)

prompt = "অসম ভাৰতৰ উত্তৰ-পূব অঞ্চলৰ"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=0,
        eos_token_id=2
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))