Eraly-ml/Kimi-K3-0.40B-Kazakh-CPT-59M

Kimi-K3-0.40B Kazakh CPT 59M

An experimental Kazakh continued-pretraining checkpoint based on inference-optimization/Kimi-K3-0.40B.

The model has seen approximately 59 million cumulative Kazakh tokens. It is a base causal language model, not an instruction-tuned or chat model.

Бұл — қазақ тіліндегі мәтіндермен қосымша оқытылған эксперименттік базалық тілдік модель. Модель нұсқауларды орындауға немесе диалог жүргізуге арнайы бейімделмеген.

Model summary

Property	Value
Language-model parameters	388.6M
Architecture	Hybrid KDA + MLA + MoE
Transformer layers	8
Routed experts	8
Active routed experts per token	2
Shared experts	1
Previous CPT steps	200
Current CPT stage	7,000 steps
Cumulative optimizer steps	7,200
Current-stage training tokens	57,344,000
Cumulative training tokens	58,982,400
Training sequence length	512 tokens
Training precision	FP16
Hardware used	1× Tesla T4

Architecture

The upstream experimental Kimi-K3-0.40B configuration contains:

• 8 transformer layers;
• KDA layers at positions 0–2 and 4–6;
• MLA layers at positions 3 and 7;
• MoE blocks in layers 1–7;
• 8 routed experts, with 2 experts activated per token;
• 1 shared expert.

This checkpoint uses the experimental 0.40B Kimi-K3 architecture. It is not a compressed or distilled version of the full production Kimi K3 model.

Training data

Continued pretraining used the kazakhBooks.csv portion of kz-transformers/multidomain-kazakh-dataset.

Data statistic	Value
Prepared training tokens	60,000,000
Prepared validation tokens	500,000
Scanned documents	8,423
Accepted documents	2,823
Rejected documents	4,247
Detected duplicates	6

Documents were filtered using predicted language and Kazakh-specific character signals. Accepted texts were cleaned, deduplicated, tokenized and packed into fixed-length sequences.

The resulting corpus is primarily composed of books, educational materials and academic text. Therefore, the checkpoint may inherit domain, style and factual biases from these sources.

Training configuration

Parameter	Value
Sequence length	512
Micro-batch size	1
Gradient accumulation steps	16
Effective batch size	8,192 tokens
Optimizer	AdamW 8-bit
Peak learning rate	1e-4
Final learning rate	1e-5
Scheduler	Cosine decay
Warmup steps	70
Weight decay	0.1
Gradient clipping	1.0
Runtime	416.9 minutes

The Kaggle session provided two Tesla T4 GPUs, but only one GPU was used for training. This run did not use Distributed Data Parallel.

Evaluation

The directly comparable periodic evaluations used the same procedure and 50 validation batches at the beginning and end of the training stage.

Checkpoint	Validation loss	Perplexity
Stage start	5.4689	237.20
Stage step 7,000	3.3502	28.51

A separate extended final evaluation over 100 batches produced:

Evaluation	Loss	Perplexity
Extended final evaluation	2.8585	17.43

The extended evaluation used a different number of batches and covered a different portion of the validation data. Therefore, it should not be directly compared with the 50-batch periodic evaluations.

Loss and perplexity measure next-token prediction on this specific validation corpus. They do not directly measure factual accuracy, reasoning ability, instruction following or general Kazakh-language proficiency.

Training curve

Installation

The following configuration was used to test this checkpoint:

pip install "transformers==5.14.1" accelerate fla-core tiktoken safetensors

Usage

This repository contains custom modeling code, so trust_remote_code=True is required.

import torch
from transformers import AutoModel, AutoTokenizer

MODEL_ID = "Eraly-ml/Kimi-K3-0.40B-Kazakh-CPT-59M"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
)

model = AutoModel.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    dtype=torch.float16,
    low_cpu_mem_usage=True,
    device_map={"": 0},
)

language_model = getattr(model, "language_model", model)
language_model.eval()

prompt = "Қазақстан —"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    add_special_tokens=False,
)

device = next(language_model.parameters()).device

inputs = {
    key: value.to(device)
    for key, value in inputs.items()
}

with torch.inference_mode():
    output = language_model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        top_k=50,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

generated_text = tokenizer.decode(
    output[0],
    skip_special_tokens=True,
)

print(generated_text)

The first forward pass may take several minutes while the FLA/KDA CUDA components initialize or compile.

Because this is a base language model, it works best when given the beginning of a document or sentence rather than a chat-style instruction.

Example prompts:

Қазақстан —
Қазақ тілінің басты ерекшеліктерінің бірі —
Жасанды интеллект дегеніміз
Бір күні ауылға жас ғалым келді.

Intended use

This checkpoint is intended for:

• continued-pretraining experiments;
• Kazakh next-token prediction research;
• studying hybrid KDA, MLA and MoE architectures;
• analyzing expert routing on Kazakh text;
• further supervised fine-tuning or instruction tuning;
• educational experiments with small language models.

Limitations

• The model is still undertrained.
• Generated text may look grammatically plausible while remaining incoherent.
• The model may invent words, names, facts and references.
• The training corpus is strongly biased toward books and academic text.
• The model has not been instruction-tuned.
• The model has not been safety-aligned.
• It should not be used as a reliable source of factual information.
• Low validation loss does not guarantee meaningful or factually correct generation.
• The multimodal components were not adapted for Kazakh vision-language tasks.

Training lineage

inference-optimization/Kimi-K3-0.40B
    └── Eraly-ml/Kimi-K3-0.40B-Kazakh-CPT-step200
        └── Eraly-ml/Kimi-K3-0.40B-Kazakh-CPT-59M

Reproducibility

The repository includes:

• training_metadata.json — training configuration and statistics;
• eval_results.json — recorded evaluation results;
• training_curve.png — training and validation loss curves;
• custom Kimi-K3 modeling and tokenizer code;
• SHA256 information for the model weights.

Model weights SHA256:

cda0b0c966632df61de0784b92b21a73f775aa86b4464aa2454fb78f81f6f3bd

License

This checkpoint is distributed under the MIT license, following the upstream Kimi-K3-0.40B repository.

Users are responsible for reviewing the licenses and usage conditions of the upstream model and training dataset.

Acknowledgements

• Moonshot AI — Kimi architecture;
• inference-optimization — experimental Kimi-K3-0.40B implementation;
• kz-transformers — Kazakh multidomain dataset;
• Hugging Face — model and dataset infrastructure;
• Gigatoken — high-performance tokenization tooling.