MeshariAlsh/arabic1800s-qwen3-8b

إرث | Irth-LM

Irth is a language model that speaks as an Arabic writer from the 19th century. It answers in the language, style and outlook of the Nahda era, and it knows nothing of what came after its time.

Try it in the Irth-Chat Space.

Model details

Developed by	Meshari Alshammari
Base model	Qwen3-8B Base
Language	Arabic
License	Apache 2.0
Code	GitHub
Corpus	arabic1800s

This repository holds the full merged model, so it loads directly with transformers without any adapters.

How it was built

Stage 1, continued pretraining

The base model was trained further on the training split of the arabic1800s corpus, about 980,000 words of 19th-century Arabic from ten sources including books, newspapers and magazines (2.35 million tokens). The last 5% of every source was held out and never seen during training.

Model	Held-out loss	Perplexity
Qwen3-8B Base	2.851	17.3
After continued pretraining	2.736	15.4

Stage 2, supervised fine-tuning

The model was then trained on conversations built from the corpus. An LLM wrote the questions, and every answer is a passage copied word for word from the corpus and checked automatically, so the model learns to reply in genuine period language. Two narrative works were left out of this stage because their text is mostly character dialogue.

To this data, 13 hand-written conversations were added and repeated five times. They teach the model to decline religious rulings, to decline quoting the Quran or hadith from memory, and to say it knows nothing of later inventions. These are the only answers in the training data that were not written in the 19th century.

Training settings

Setting	Continued pretraining	Fine-tuning
Method	LoRA, rank 16, alpha 32	LoRA, rank 16, alpha 32
Target modules	All attention and feed-forward projections	Same
Learning rate	2e-4, cosine	1e-4, cosine
Batch	8 blocks of 2,048 tokens	8 conversations
Epochs	2	3
Checkpoint	Lowest held-out loss	Lowest held-out loss
Hardware	1 × A100	1 × A100

Both adapters were merged into the weights of this model.

Usage

The model was trained with a plain-text chat format, not the Qwen chat template. Use the format below exactly, or the answers will degrade.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "MeshariAlsh/arabic1800s-qwen3-8b"
PERSONA = "أنت كاتب عربي من القرن التاسع عشر. تجيب بلغة عصرك وأسلوبه، ولا تعرف شيئًا مما حدث بعد زمانك."

tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.bfloat16, device_map="auto")
EOS_ID = tok.eos_token_id


def seg(text):
    return tok(text, add_special_tokens=False)["input_ids"]


def ask(question):
    ids = seg(PERSONA + "\n") + seg(f"\nسؤال:\n{question}\n") + seg("جواب:\n")
    ids = torch.tensor([ids], device=model.device)
    out = model.generate(input_ids=ids, attention_mask=torch.ones_like(ids), max_new_tokens=300,
                         do_sample=True, temperature=0.5, top_p=0.9, repetition_penalty=1.1,
                         eos_token_id=EOS_ID, pad_token_id=EOS_ID)
    return tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True).strip()


print(ask("ما رأيك في تعليم البنات؟"))

Each part of the prompt is tokenised separately, as in training. Every answer ends with the <|endoftext|> token, which stops generation.

For a multi-turn conversation, add each earlier exchange as seg(f"\nسؤال:\n{question}\n") + seg("جواب:\n") + seg(answer) + [EOS_ID] before the new question. The Space code on GitHub shows a full example.

Limitations

• It invents things. It can make up facts, dates, numbers and events, and it does so with confidence.
• It reflects its sources. The texts carry the views of their time, including prejudices about women, other peoples and other religions, and the model can repeat them.
• Its opinions are not historical evidence. On complex topics it often stitches phrases together into answers that are incoherent or unrepresentative. Do not use it to learn what people of the period believed.
• Religious content. It is designed to decline religious rulings and quoting scripture, but it does not always succeed. It has produced invented rulings and misquoted verses. Never rely on it for religious matters. The Irth-Chat Space adds a keyword filter for these questions, which this model does not have on its own.
• It is not fully cut off from the modern world. The base model knows about later events, and some of that knowledge can still leak into answers.
• The corpus is small and uneven. Ten sources cannot represent every region, community or point of view of the 19th-century Arab world.

Intended use

Irth is for exploration, education and research into how language models can take on historical voices. It is not suitable as a source of facts, religious guidance or historical evidence.