SmolLM2-135M Noval β€” Domain CPT

HuggingFaceTB/SmolLM2-135M after Continued Pre-Training on the fictional OMC + Mars Express domain corpus. First-round (135M) proof of the pipeline.

Overview

  • Stage: Continued Pre-Training (full fine-tune, CLM)
  • Lineage: SmolLM2-135M β†’ CPT (this model)
  • Method: Continued Pre-Training (next-token, full fine-tune) on a 30/70 mars/orbital interleaved corpus (tokenized + packed).
  • Domain: fictional β€” Orbital Mining Corporation (OMC) technical docs + Mars Express telemetry.

Training

Corpus noval-corp-training-corpus (30% Mars / 70% OMC, packed input_ids)
LR / schedule 5e-4 cosine, warmup 200 steps
Duration / seq max_steps 5000, seq 1024, eff_batch 32, T4 fp16

Evaluation

No task metrics for this checkpoint (intermediate / no-training stage).

Intended use & limitations

Domain backbone for the 135M line (knows the domain; not instruction-tuned).

Limitations:

  • 135M is intrinsically limited β€” expectations should be modest.
  • Fictional domain; no chat template / instruction following.

Card generated by noval-corp/scripts/gen_model_cards.py (standardized across the noval-corp model family).

Downloads last month
23
Safetensors
Model size
0.1B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for atenareply/smollm2-135m-noval

Finetuned
(992)
this model

Dataset used to train atenareply/smollm2-135m-noval