India Open Government Data — Bharat Tables Embeddings
Collection
Research and non-commercial use (CC BY-NC-SA 4.0). Commercial: surya@judicialmind.ai / hello@judicialmind.ai • 1 item • Updated
You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
This model is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 licence (CC BY-NC-SA 4.0): research, academic, personal and other non-commercial use is permitted with attribution; adaptations must be shared under the same licence; commercial use is not permitted. The original public source data remains under the Government Open Data License - India (GODL-India). Commercial use: contact surya@judicialmind.ai or hello@judicialmind.ai.
Log in or Sign Up to review the conditions and access this model content.
IndiaTables-Embed-0.6B
Licence: CC BY-NC-SA 4.0 — research and non-commercial use only; derivatives must be shared under the same licence. Commercial use: contact surya@judicialmind.ai or hello@judicialmind.ai
Access is approved automatically when the email you enter matches your Hugging Face account email and belongs to the organisation you name; other requests are reviewed manually.
Retriever for Indian government data tables: given a question in English, Hindi or Hinglish, it finds the table
(from India's Open Government Data platform and NITI Aayog's NDAP) that answers it. Fine-tuned from
perplexity-ai/pplx-embed-v1-0.6b on row-checked questions over IndiaTables.
1024-dimensional embeddings, cosine similarity, up to 512 tokens.
Headline result: on IndiaTables-Bench table retrieval (23,961 test questions over 266,707 tables) it reaches nDCG@10 0.8388 and R@1 0.7225, against 0.6825 / 0.5415 for its base model pplx-embed-v1-0.6b, 0.6208 / 0.4762 for BAAI/bge-m3 and 0.4900 / 0.3514 for BM25.
Quickstart
Input: questions (plain text, no prefix) and table cards (plain text: title, source, level, years, columns, a
few rows; the card column of IndiaTables
cards.parquet has this format). Output: one 1024-d vector per text; higher cosine = better match.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("judicialmind/IndiaTables-Embed-0.6B", trust_remote_code=True)
model.max_seq_length = 512
queries = ["How much groundwater was extracted for irrigation in Punjab in 2020?",
"साक्षरता दर राज्यवार 2021"]
cards = ["Title: State/UTs-wise Groundwater Resource and their Overall Extraction Level of India during 2020\n"
"Level: state | Years: 2020 | Rows: 38\nColumns: Sl. No., States/Union Territories, "
"Current Annual Ground Water Extraction (in BCM) - Irrigation, Stage of Ground Water Extraction (%)",
"Title: State/UT-wise Literacy rate (in per cent) of persons of different age groups\n"
"Level: state | Years: 2017, 2018, 2019, 2020, 2021, 2022, 2023 | Rows: 4,530\n"
"Columns: Country, StateName, Year, GENDER",
"Title: District-wise Rainfall Distribution\nLevel: district | Years: 2021, 2022, 2023 | Rows: 1,246,210\n"
"Columns: StateName, DistrictName, Year, CalendarDay"]
q = model.encode(queries, normalize_embeddings=True)
d = model.encode(cards, normalize_embeddings=True)
scores = model.similarity(q, d)
print(scores)
print(scores.argmax(dim=1).tolist())
Expected output: a 2 x 3 score matrix and [0, 1] (the groundwater question matches the groundwater card, the
Hindi literacy question the literacy card).
Search over all IndiaTables cards with the bundled search.py (pip install -r requirements.txt; encoding all
267K cards needs a GPU, --limit N encodes the first N):
huggingface-cli download judicialmind/IndiaTables cards.parquet --repo-type dataset --local-dir .
python search.py "district-wise rainfall in Karnataka 2019" --cards cards.parquet --k 5 --save card_emb.npy
It prints score uuid title for the top k tables; the uuid names the table file in IndiaTables
(tables/<first two characters>/<uuid>.parquet, NDAP tables under tables/nd/). From Python:
import pyarrow.parquet as pq
from search import TableSearch, load_model
cards = pq.read_table("cards.parquet", columns=["uuid", "card"]).slice(0, 300) # first 300 cards
model = load_model("judicialmind/IndiaTables-Embed-0.6B")
ts = TableSearch.from_cards(model, cards.column("uuid").to_pylist(), cards.column("card").to_pylist())
print(ts.search("district-wise rainfall in Karnataka 2019", k=5)) # [(uuid, score), ...]
Files
| File | What |
|---|---|
model.safetensors, config.json, configuration.py, modeling.py |
Bidirectional Qwen3 encoder (custom code, needs trust_remote_code=True) |
tokenizer.json, tokenizer_config.json |
Tokenizer |
modules.json, sentence_bert_config.json, 1_Pooling/, 2_FlexibleQuantizer/, st_quantize.py, config_sentence_transformers.json |
sentence-transformers pipeline: mean pooling, then int8 (default), binary or packed-binary output quantization |
metrics.json |
Evaluation results (table below) |
search.py, requirements.txt |
Search helper and pinned runtime dependencies |
Results
IndiaTables-Bench task 5 (table retrieval), test split: 23,961 questions over the full IndiaTables card corpus, held-out ministries and template families.
| model | nDCG@10 | R@1 | R@10 | MRR@10 |
|---|---|---|---|---|
| bm25 | 0.4900 | 0.3514 | 0.6335 | 0.4443 |
| BAAI/bge-m3 | 0.6208 | 0.4762 | 0.7712 | 0.5729 |
| Qwen/Qwen3-Embedding-0.6B | 0.5808 | 0.4378 | 0.7336 | 0.5324 |
| perplexity-ai/pplx-embed-v1-0.6b | 0.6825 | 0.5415 | 0.8238 | 0.6373 |
| IndiaTables-Embed-0.6B | 0.8388 | 0.7225 | 0.9455 | 0.8041 |
The benchmark and its scoring script are in IndiaTables-Bench.
Coverage
Tables from data.gov.in and NDAP (snapshot May 2026), all ministries and geographic levels from national to village. Questions in English, Hindi (Devanagari) and Hinglish.
Source data
Trained and evaluated on table cards and questions derived from IndiaTables (data.gov.in and NITI Aayog NDAP, snapshot May 2026).
Original data: published by the ministries, departments and State governments named per table in IndiaTables on the Open Government Data Platform India (data.gov.in) and, where marked, NITI Aayog's National Data and Analytics Platform (NDAP), under the Government Open Data License - India (GODL-India). The original public data remains under GODL-India; the CC BY-NC-SA 4.0 licence of this repository covers JudicialMind's compilation, derived fields, annotations, benchmarks and models.
Citation
@misc{judicialmind2026indiatablesembed,
title={IndiaTables-Embed-0.6B: a retriever for Indian government data tables},
author={JudicialMind},
year={2026},
publisher={Hugging Face},
howpublished={\url{https://huggingface.co/judicialmind/IndiaTables-Embed-0.6B}}
}
License
CC BY-NC-SA 4.0 (see LICENSE): research, academic, personal and other non-commercial use, with attribution to JudicialMind and to the original data publishers; adaptations must be shared under the same licence. Commercial use requires a separate licence.
Commercial use: contact surya@judicialmind.ai or hello@judicialmind.ai.
Base model(s): perplexity-ai/pplx-embed-v1-0.6b (mit). Their licence terms and notices continue to apply to the upstream components; see THIRD_PARTY_NOTICES.md.
- Downloads last month
- -
Model tree for judicialmind/IndiaTables-Embed-0.6B
Base model
perplexity-ai/pplx-embed-v1-0.6b