VelkroLM/hausa-tokenizer

Hausa tokenizer

This repository contains the Hausa SentencePiece tokenizer artifacts hausa_sp.model and hausa_sp.vocab.

Intended use

Use this tokenizer for Hausa text preprocessing and compatible SentencePiece-based experiments. Check the tokenizer configuration used by the training pipeline before loading it into a model.

Files

• hausa_sp.model: SentencePiece model.
• hausa_sp.vocab: SentencePiece vocabulary.

License and provenance

The tokenizer artifacts in this repository are released under the MIT License. See the repository license metadata before redistribution or commercial use.