Instructions to use kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8")
model = AutoModelForMultimodalLM.from_pretrained("kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8

SGLang

How to use kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8 with Docker Model Runner:
```
docker model run hf.co/kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8
```

Gemma 4 31B Tamil + Devanagari OCR — FP8_BLOCK

This 32 GB vLLM-oriented artifact is quantized from the workspace's merged BF16 OCR model with LLM Compressor's Gemma 4 FP8_BLOCK recipe. Language-layer linear weights use 128x128 FP8 blocks with dynamic group-128 activations. The vision tower, multimodal projector, embeddings, and LM head remain BF16.

Use BF16 when quality is the only priority. FP8 is the compact option: on the independent 80-page gate it had essentially the same median gCER (11.01% versus 11.12%) and the same 7/80 catastrophic count, but BF16 won more paired pages (42 BF16 wins, 25 FP8 wins, 13 ties) and had slightly better clipped mean gCER. The FP8 raw mean happened to be lower because its repetition failures occurred on different pages and avoided one extreme BF16 loop; that difference was not statistically resolved.

Gate	gCER mean	gCER median	gCER clipped at 100%	Catastrophic	4,096-token caps
Independent 80-page gate	21.69%	11.01%	17.70%	7/80	4/80
Untouched base, same pages	374.74%	17.56%	25.99%	11/80	6/80

On the untouched public benchmark's 350 natural-text cases, FP8 scored 3.151% gCER (Tamil 5.366%, Devanagari 0.936%) and 53.71% exact match. Its aggregate gCER difference from the base was not statistically resolved.

vLLM

From the workspace root:

OCR_MAX_NUM_SEQS=8 scripts/serve_vllm.sh \
  artifacts/gemma4-31b-tamil-devanagari-ocr-fp8
.venv-serve/bin/python scripts/ocr_vllm_client.py page.png --output page.txt

The same source-data, license, Sanskrit-script coverage, scholarly-review, and repetition-loop limitations described in the BF16 model card apply here. The artifact pins the 1,120-token image budget in both model and processor configuration, so ordinary vLLM loading matches the fine-tuning input budget.

Downloads last month: 27

Safetensors

Model size

31B params

Tensor type

BF16

F8_E4M3

Model tree for kailasa-ngpt/gemma-4-31b-tamil-devanagari-ocr-fp8

Base model

google/gemma-4-31B

Finetuned

google/gemma-4-31B-it

Quantized

(346)

this model