NaijaVox-2.0 — merged standalone model (epoch 3, r=64, SpecAugment)

README.md

@@ -0,0 +1,374 @@
+---
+language:
+- yo
+- ha
+- ig
+- en
+- pcm
+license: apache-2.0
+tags:
+- whisper
+- speech
+- asr
+- automatic-speech-recognition
+- yoruba
+- hausa
+- igbo
+- nigerian-english
+- nigerian-pidgin
+- nigeria
+- african-languages
+- audio
+base_model: openai/whisper-large-v3
+datasets:
+- google/fleurs
+- benjaminogbonna/nigerian_accented_english_dataset
+- asr-nigerian-pidgin/nigerian-pidgin-1.0
+- Tundragoon/IroyinSpeech
+- google/WaxalNLP
+- benjaminogbonna/nigerian_common_voice_dataset
+- vpetukhov/bible_tts_hausa
+pipeline_tag: automatic-speech-recognition
+---
+
+<p align="center">
+  <img src="https://huggingface.co/Axiveri/NaijaVox-V1/resolve/main/naijavox_logo.png" width="100%"/>
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/Model-NaijaVox--2.0-2ea44f?style=flat-square"/>
+  <img src="https://img.shields.io/badge/License-Apache%202.0-2196F3?style=flat-square"/>
+  <img src="https://img.shields.io/badge/Base-Whisper--Large--v3-8B5CF6?style=flat-square"/>
+  <img src="https://img.shields.io/badge/Version-2.0-FF9800?style=flat-square"/>
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/Languages-Yoruba%20%7C%20Hausa%20%7C%20Igbo%20%7C%20Pidgin%20%7C%20Naija%20English-FF6B35?style=flat-square"/>
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/Built%20by-Axiveri-e11d48?style=flat-square"/>
+  <img src="https://img.shields.io/badge/GPU-Tesla%20T4%20x2-64748b?style=flat-square"/>
+</p>
+
+<div align="center">
+
+| Language | WER | vs V1 |
+|---|---|---|
+| 🇳🇬 Pidgin | **14.7%** | ↓ 2.1pp |
+| 🇳🇬 Nigerian English | **19.6%** | ↓ 1.5pp |
+| 🇳🇬 Yoruba | **22.3%** | ↓ 6.5pp |
+| 🇳🇬 Hausa | **25.8%** | ↓ 5.2pp |
+| 🇳🇬 Igbo | **30.5%** | ↓ 11.4pp |
+
+</div>
+
+---
+
+## Nigeria's Voice in AI. Now Sharper.
+
+**NaijaVox-2.0** is the second generation of Axiveri's open-weight automatic speech recognition model for Nigerian languages — Yoruba (with full diacritics), Hausa, Igbo, Nigerian Pidgin, and Nigerian-accented English. Built on OpenAI Whisper-large-v3 with PEFT LoRA fine-tuning, NaijaVox-2.0 delivers significant accuracy gains over V1 through a larger and more diverse training corpus (25,866 samples across 7 datasets), deeper LoRA adaptation (r=64 targeting attention and feed-forward layers), SpecAugment, and realistic noise augmentation for real-world robustness.
+
+> *"Every Nigerian deserves to be heard and understood by AI — in their own language, with their own voice."*
+
+**[← NaijaVox-V1](https://huggingface.co/Axiveri/NaijaVox-V1)** — the original model
+
+---
+
+## 📈 V1 → V2 Improvement
+
+Evaluated on identical test sets with identical methodology (50 samples/language, strict WER, no normalization):
+
+| Language | V1 WER | V2 WER | Absolute Δ | Relative Gain |
+|---|---|---|---|---|
+| 🇳🇬 Yoruba | 28.8% | **22.3%** | −6.5pp | **+22.6%** |
+| 🇳🇬 Hausa | 31.0% | **25.8%** | −5.2pp | **+16.8%** |
+| 🇳🇬 Igbo | 41.9% | **30.5%** | −11.4pp | **+27.2%** |
+| 🇳🇬 Nigerian English | 21.1% | **19.6%** | −1.5pp | **+7.1%** |
+| 🇳🇬 Nigerian Pidgin | 16.8% | **14.7%** | −2.1pp | **+12.5%** |
+| **Average** | **27.9%** | **22.58%** | **−5.3pp** | **+19.1%** |
+
+> Igbo sees the largest jump (+27.2% relative) — driven by WaxalNLP Igbo TTS data and Nigerian Common Voice Igbo samples, combined with SpecAugment frequency masking.
+
+---
+
+## 🗣️ Languages Supported
+
+| Language | ISO Code | Script | Token |
+|---|---|---|---|
+| Yoruba | `yo` | Latin + full diacritics (ẹ, ọ, ṣ, à, á, etc.) | `<|yo|>` |
+| Hausa | `ha` | Latin + special chars (ƙ, ƴ, ɗ, etc.) | `<|ha|>` |
+| Igbo | `ig` | Latin + diacritics | `<|ig|>` |
+| Nigerian Pidgin | `pcm` | Latin | `<|pcm|>` |
+| Nigerian English | `en` | Latin | `<|en|>` |
+
+> **Note:** `<|ig|>` and `<|pcm|>` are custom language tokens added to the Whisper vocabulary. The extended tokenizer is included in this repository.
+
+---
+
+## 🚀 Quick Start
+
+```python
+from transformers import pipeline
+
+pipe = pipeline(
+    "automatic-speech-recognition",
+    model="Axiveri/NaijaVox-2.0",
+    device=0  # use GPU, or remove for CPU
+)
+
+result = pipe("your_audio.wav")
+print(result["text"])
+```
+
+### Specifying Language
+
+```python
+from transformers import WhisperProcessor, WhisperForConditionalGeneration
+import torch
+
+model     = WhisperForConditionalGeneration.from_pretrained("Axiveri/NaijaVox-2.0")
+processor = WhisperProcessor.from_pretrained("Axiveri/NaijaVox-2.0")
+vocab     = processor.tokenizer.get_vocab()
+
+LANG_TOKENS = {
+    "yoruba":           "<|yo|>",
+    "hausa":            "<|ha|>",
+    "igbo":             "<|ig|>",
+    "nigerian_english": "<|en|>",
+    "pidgin":           "<|pcm|>",
+}
+
+def transcribe(audio_array, sampling_rate, language="yoruba"):
+    lang_id      = vocab[LANG_TOKENS[language]]
+    transcribe   = vocab["<|transcribe|>"]
+    notimestamps = vocab["<|notimestamps|>"]
+    forced_ids   = [[1, lang_id], [2, transcribe], [3, notimestamps]]
+
+    inputs = processor.feature_extractor(
+        audio_array, sampling_rate=sampling_rate, return_tensors="pt"
+    ).input_features
+
+    with torch.no_grad():
+        generated = model.generate(
+            input_features=inputs,
+            forced_decoder_ids=forced_ids,
+            max_new_tokens=448
+        )
+    return processor.tokenizer.decode(generated[0], skip_special_tokens=True)
+```
+
+---
+
+## 📊 Benchmark Results
+
+Evaluated on FLEURS test splits (Yoruba, Hausa, Igbo), Nigerian Pidgin ASR test set, and Nigerian Accented English dataset. 50 samples per language, greedy decoding, strict WER via `jiwer` (no text normalization). **Identical methodology to V1 for direct comparison.**
+
+| Language | WER (%) | Accuracy (%) | Test Set | Samples |
+|---|---|---|---|---|
+| 🇳🇬 Nigerian Pidgin | **14.7** | **85.3** | asr-nigerian-pidgin/nigerian-pidgin-1.0 | 50 |
+| 🇳🇬 Nigerian English | **19.6** | **80.4** | benjaminogbonna/nigerian_accented_english | 50 |
+| 🇳🇬 Yoruba | **22.3** | **77.7** | google/fleurs yo_ng | 50 |
+| 🇳🇬 Hausa | **25.8** | **74.2** | google/fleurs ha_ng | 50 |
+| 🇳🇬 Igbo | **30.5** | **70.5** | google/fleurs ig_ng | 50 |
+| **Average** | **22.58** | **77.62** | — | 250 |
+
+> Lower WER = better. Human-level transcription ≈ 5–10%.
+
+---
+
+## 🛡️ Robustness Improvements over V1
+
+NaijaVox-2.0 was trained with two techniques not present in V1:
+
+### SpecAugment
+Frequency masking (up to 27 mel bins) and time masking (up to 100 time steps) applied to mel spectrograms during training. This prevents over-reliance on specific frequency bands or time positions, improving generalization to real-world recordings where background noise occupies variable frequency ranges.
+
+### Noise Augmentation
+30% of training samples received realistic background noise injection at random SNR levels before mel extraction. This directly trains the model for common Nigerian recording conditions — market noise, phone compression artifacts, outdoor ambient sound, and crowd audio — conditions where V1 degraded noticeably.
+
+### Code-Switching Robustness
+NaijaVox-2.0 was trained on Nigerian Pidgin and Nigerian English together with Yoruba, Hausa, and Igbo — all of which contain natural Yoruba-English, Igbo-English, and Pidgin-English code-switching patterns present in everyday Nigerian speech, media, and social content. The model handles mid-sentence language shifts without explicit code-switching supervision.
+
+---
+
+## 🎙️ Sample Transcriptions
+
+> Real audio samples from FLEURS test, Nigerian English, and Pidgin datasets — data the model **never saw during training**. Transcriptions generated live by the published merged model.
+
+### Yoruba
+
+| Reference | Audio | NaijaVox-2.0 Output |
+|---|:---:|---|
+| *àwọn èyàn ti mọ̀ nípa àwọn kemika pepe bí wúrà fàdákà àti kọ́pa àtijọ́ torípé a lè rí wọn* | <audio controls><source src="https://huggingface.co/Axiveri/NaijaVox-2.0/resolve/main/audio_samples/yoruba_sample1.wav" type="audio/wav"></audio> | àwọn èèyàn ti mọ̀ nípa àwọn kẹmíkà pèèpèé bí wúrà fàdákà àti kọpa àtijọ́ torí pé a lè rí wọn |
+| *àwọn ara ìrano lo kọ́kọ́ bẹ̀rẹ̀ si ni sin ewure ní bíi ọdún 15,0000 sẹ́yìn ní oke sagrosi* | <audio controls><source src="https://huggingface.co/Axiveri/NaijaVox-2.0/resolve/main/audio_samples/yoruba_sample2.wav" type="audio/wav"></audio> | àwọn ará ìrà náà ló kọ́kọ́ bẹ̀rẹ̀ sí ní sin ewúrẹ́ ní bí ọdún 1500 sẹ́yìn ní òkè sagrosi |
+
+### Hausa
+
+| Reference | Audio | NaijaVox-2.0 Output |
+|---|:---:|---|
+| *an kwatanta faretin gine-ginen da ke yin sararin samaniyar hong kong da ginshiƙi mai walƙi* | <audio controls><source src="https://huggingface.co/Axiveri/NaijaVox-2.0/resolve/main/audio_samples/hausa_sample1.wav" type="audio/wav"></audio> | an kwatanta feretin gine-ginen da ke yin sararin samaniya hong kong da ginshiki mai walƙiy |
+| *aristotle masanin falsafa ne yayi tunanin cewa komai ya kunshi cakuda daya ko fiye daga ab* | <audio controls><source src="https://huggingface.co/Axiveri/NaijaVox-2.0/resolve/main/audio_samples/hausa_sample2.wav" type="audio/wav"></audio> | aristotle masanin falsafani ya yi tunanin cewa kome ya kunshi ca kuda daya ko fiye daga ab |
+
+### Igbo
+
+| Reference | Audio | NaijaVox-2.0 Output |
+|---|:---:|---|
+| *ka akara rossby na-adị obere karịa ka arụmarụ na-adịkwu obere nke kpakpando n'ikwanye ugwu* | <audio controls><source src="https://huggingface.co/Axiveri/NaijaVox-2.0/resolve/main/audio_samples/igbo_sample1.wav" type="audio/wav"></audio> | akara rossby na-adị obere karịa ka arụmarụ na-adịkwa obere nke kpakpando n'ịkwà nye monto |
+| *ka agha dara mba britenị jiri ndị agha elu mmiri gbochie ndị jamani inweta enyemaka* | <audio controls><source src="https://huggingface.co/Axiveri/NaijaVox-2.0/resolve/main/audio_samples/igbo_sample2.wav" type="audio/wav"></audio> | ka agha adara mba briten jiri ndị agha elu mmiri gbochie ndị jamanị inweta enyemaka |
+
+### Nigerian English
+
+| Reference | Audio | NaijaVox-2.0 Output |
+|---|:---:|---|
+| *Did it change plain?  Yes. yes. Ok that means he was correct so this is if he's right that* | <audio controls><source src="https://huggingface.co/Axiveri/NaijaVox-2.0/resolve/main/audio_samples/english_sample1.wav" type="audio/wav"></audio> | Did it change green? Yes. Ok that means she was correct. So this is if its red then its no |
+| *Ebube Nwagbo studied Mass Communication at Nnamdi Azikiwe University.* | <audio controls><source src="https://huggingface.co/Axiveri/NaijaVox-2.0/resolve/main/audio_samples/english_sample2.wav" type="audio/wav"></audio> | Ebube Nwagbo studied Mass Communication at Nnamdi Azikiwe University. |
+
+### Nigerian Pidgin
+
+| Reference | Audio | NaijaVox-2.0 Output |
+|---|:---:|---|
+| *on top di injury her uncle no even carry her go hospital for treatment* | <audio controls><source src="https://huggingface.co/Axiveri/NaijaVox-2.0/resolve/main/audio_samples/pidgin_sample1.wav" type="audio/wav"></audio> | on top di injury and her uncle no even carry her go hospital for treatment |
+| *she tell don jazzy for december 2016 say as she be* | <audio controls><source src="https://huggingface.co/Axiveri/NaijaVox-2.0/resolve/main/audio_samples/pidgin_sample2.wav" type="audio/wav"></audio> | she tell don jazzy for december 2016 say i should be |
+
+---
+
+## 🏗️ Model Architecture
+
+```
+Input Audio (16kHz)
+        │
+        ▼
+Whisper-large-v3 Encoder  (frozen during fine-tuning)
+        │  1500 × 1280 features
+        ▼
+Whisper Decoder + LoRA    (r=64, alpha=128, fine-tuned)
+  target modules: q_proj, k_proj, v_proj, out_proj, fc1, fc2
+  V1 target: attention only (q/k/v/out)
+  V2 target: attention + feed-forward (adds fc1/fc2)
+        │
+        ▼
+Extended Tokenizer         (vocab: 51,868 tokens)
+  + <|ig|> Igbo token
+  + <|pcm|> Nigerian Pidgin token
+        │
+        ▼
+Transcript
+```
+
+> V2 publishes a **fully merged standalone model** — no PEFT dependency required. Load directly with `transformers`.
+
+---
+
+## 📦 Training Details
+
+| Parameter | V1 | V2 |
+|---|---|---|
+| Base model | openai/whisper-large-v3 | openai/whisper-large-v3 |
+| Fine-tuning method | LoRA (PEFT) | LoRA (PEFT) |
+| LoRA rank | 32 | **64** |
+| LoRA alpha | 64 | **128** |
+| Target modules | q/k/v/out_proj | **q/k/v/out_proj + fc1/fc2** |
+| LoRA dropout | 0.05 | 0.05 |
+| Training precision | fp16 | fp16 |
+| Effective batch size | 16 | **32** |
+| Learning rate | 1e-3 | **5e-4** |
+| Warmup steps | 50 | **200** |
+| Epochs (best) | 2 | **3 of 5** |
+| SpecAugment | ❌ | **✅** |
+| Noise augmentation | ❌ | **✅ (30% of samples)** |
+| Total training samples | 13,866 | **25,866** |
+| GPU | Tesla T4 × 2 (Kaggle) | Tesla T4 × 2 (Kaggle) |
+| Total training time | ~20 hours | ~40 hours |
+
+### Training Datasets
+
+| Dataset | Language(s) | Samples | License | New in V2 |
+|---|---|---|---|---|
+| google/fleurs (yo_ng) | Yoruba | 2,339 | CC-BY 4.0 | — |
+| google/fleurs (ha_ng) | Hausa | 3,259 | CC-BY 4.0 | — |
+| google/fleurs (ig_ng) | Igbo | 2,839 | CC-BY 4.0 | — |
+| benjaminogbonna/nigerian_accented_english_dataset | Nigerian English | 2,721 | Apache 2.0 | — |
+| asr-nigerian-pidgin/nigerian-pidgin-1.0 | Nigerian Pidgin | 2,708 | CC-BY 4.0 | — |
+| Tundragoon/IroyinSpeech | Yoruba | 2,500 | CC-BY 4.0 | ✅ |
+| google/WaxalNLP (ha/ig/yo/pcm) | Hausa, Igbo, Yoruba, Pidgin | 6,000 | CC-BY-SA 4.0 | ✅ |
+| benjaminogbonna/nigerian_common_voice_dataset | en/ha/ig/yo | 2,000 | Apache 2.0 | ✅ |
+| vpetukhov/bible_tts_hausa | Hausa | 1,500 | CC-BY-SA | ✅ |
+| **Total** | **5 languages** | **25,866** | | |
+
+---
+
+## ✅ Intended Use
+
+NaijaVox-2.0 is designed for:
+
+- 🏦 **Fintech & banking** — voice-based transactions and customer service in Nigerian languages
+- 📱 **Mobile apps** — voice input for Yoruba, Hausa, Igbo, and Pidgin speakers
+- 🎙️ **Media & journalism** — transcribing interviews and broadcasts in Nigerian languages
+- 🏥 **Healthcare** — patient intake and medical documentation
+- 📚 **Education** — language learning tools and accessibility
+- 🔬 **Research** — low-resource ASR study for West African languages
+- ♿ **Accessibility** — assistive technology for Nigerians with disabilities
+
+---
+
+## 🚫 Prohibited Use
+
+The following uses are explicitly **prohibited** under this model's responsible use policy:
+
+- ❌ **Non-consensual surveillance** — transcribing calls or conversations without consent of all parties
+- ❌ **Fraud facilitation** — using transcription output to forge spoken statements or support advance-fee fraud
+- ❌ **Deepfake pipelines** — combining with TTS to create fake audio-text pairs attributed to real people
+- ❌ **Discriminatory systems** — denying services based on language or accent identification from this model
+- ❌ **Political disinformation** — generating or verifying false transcripts of political speech
+
+While Apache 2.0 permits broad commercial use, these restrictions apply as a binding behavioral restriction under Axiveri's Responsible AI Use Policy.
+
+---
+
+## 👤 Creator
+
+**Emmanuel Ariyo (Ememzyvisuals)** — Founder, Axiveri
+
+NaijaVox is conceived, built, and trained by Emmanuel Ariyo — combining ML engineering with a Nigerian cultural design identity to bring open-weight speech recognition to Yoruba, Hausa, Igbo, Nigerian Pidgin, and Nigerian English speakers.
+
+---
+
+## 👥 About Axiveri
+
+**Axiveri** is building Africa's AI infrastructure — open models, open data, and open tools for African languages and developers.
+
+- 🌍 [Africlaude Series](https://huggingface.co/Axiveri) — African language models
+- 🗣️ [NaijaVox Collection](https://huggingface.co/collections/Axiveri/naijavox-nigerian-speech-recognition) — all NaijaVox versions
+
+---
+
+## 📄 Citation
+
+```bibtex
+@misc{naijavox2026,
+  title        = {NaijaVox-2.0: Open-Weight Speech Recognition for Nigerian Languages},
+  author       = {Ariyo, Emmanuel (Ememzyvisuals)},
+  year         = {2026},
+  publisher    = {HuggingFace},
+  howpublished = {\url{https://huggingface.co/Axiveri/NaijaVox-2.0}},
+  note         = {Whisper-large-v3 fine-tuned on Yoruba, Hausa, Igbo,
+                  Nigerian Pidgin and Nigerian-accented English.
+                  LoRA r=64, SpecAugment, noise augmentation, 25,866 samples.}
+}
+```
+
+---
+
+## 📜 License
+
+**Apache 2.0** — free for commercial and research use with attribution.
+Additional behavioral restrictions apply as described in the Prohibited Use section above.
+
+---
+
+<p align="center">
+  <i>Built in Nigeria 🇳🇬 — for Nigeria and the world.</i><br/>
+  <i>Created by <a href="https://huggingface.co/ememzyvisuals">Emmanuel Ariyo (Ememzyvisuals)</a></i><br/>
+  <i>Second model in the NaijaVox series by <a href="https://huggingface.co/Axiveri">Axiveri</a></i>
+</p>

config.json

@@ -0,0 +1,47 @@
+{
+  "activation_dropout": 0.0,
+  "activation_function": "gelu",
+  "apply_spec_augment": false,
+  "architectures": [
+    "WhisperForConditionalGeneration"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 50257,
+  "classifier_proj_size": 256,
+  "d_model": 1280,
+  "decoder_attention_heads": 20,
+  "decoder_ffn_dim": 5120,
+  "decoder_layerdrop": 0.0,
+  "decoder_layers": 32,
+  "decoder_start_token_id": 50258,
+  "dropout": 0.0,
+  "dtype": "float16",
+  "encoder_attention_heads": 20,
+  "encoder_ffn_dim": 5120,
+  "encoder_layerdrop": 0.0,
+  "encoder_layers": 32,
+  "eos_token_id": 50257,
+  "forced_decoder_ids": null,
+  "init_std": 0.02,
+  "is_encoder_decoder": true,
+  "mask_feature_length": 10,
+  "mask_feature_min_masks": 0,
+  "mask_feature_prob": 0.0,
+  "mask_time_length": 10,
+  "mask_time_min_masks": 2,
+  "mask_time_prob": 0.05,
+  "max_source_positions": 1500,
+  "max_target_positions": 448,
+  "median_filter_width": 7,
+  "model_type": "whisper",
+  "num_hidden_layers": 32,
+  "num_mel_bins": 128,
+  "pad_token_id": 50256,
+  "scale_embedding": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.0.0",
+  "use_cache": true,
+  "use_weighted_layer_sum": false,
+  "vocab_size": 51868,
+  "suppress_tokens": []
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "bos_token_id": 50257,
+  "decoder_start_token_id": 50258,
+  "eos_token_id": 50257,
+  "forced_decoder_ids": null,
+  "max_new_tokens": 448,
+  "suppress_tokens": [],
+  "transformers_version": "5.0.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d026e7088605bf2cfa35c8e00985ab4284f6a02ef29e80c2d2465556a6309f7
+size 3087136096

processor_config.json

@@ -0,0 +1,17 @@
+{
+  "feature_extractor": {
+    "chunk_length": 30,
+    "dither": 0.0,
+    "feature_extractor_type": "WhisperFeatureExtractor",
+    "feature_size": 128,
+    "hop_length": 160,
+    "n_fft": 400,
+    "n_samples": 480000,
+    "nb_max_frames": 3000,
+    "padding_side": "right",
+    "padding_value": 0.0,
+    "return_attention_mask": false,
+    "sampling_rate": 16000
+  },
+  "processor_class": "WhisperProcessor"
+}

tokenizer_config.json

@@ -0,0 +1,126 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|startoftranscript|>",
+    "<|en|>",
+    "<|zh|>",
+    "<|de|>",
+    "<|es|>",
+    "<|ru|>",
+    "<|ko|>",
+    "<|fr|>",
+    "<|ja|>",
+    "<|pt|>",
+    "<|tr|>",
+    "<|pl|>",
+    "<|ca|>",
+    "<|nl|>",
+    "<|ar|>",
+    "<|sv|>",
+    "<|it|>",
+    "<|id|>",
+    "<|hi|>",
+    "<|fi|>",
+    "<|vi|>",
+    "<|he|>",
+    "<|uk|>",
+    "<|el|>",
+    "<|ms|>",
+    "<|cs|>",
+    "<|ro|>",
+    "<|da|>",
+    "<|hu|>",
+    "<|ta|>",
+    "<|no|>",
+    "<|th|>",
+    "<|ur|>",
+    "<|hr|>",
+    "<|bg|>",
+    "<|lt|>",
+    "<|la|>",
+    "<|mi|>",
+    "<|ml|>",
+    "<|cy|>",
+    "<|sk|>",
+    "<|te|>",
+    "<|fa|>",
+    "<|lv|>",
+    "<|bn|>",
+    "<|sr|>",
+    "<|az|>",
+    "<|sl|>",
+    "<|kn|>",
+    "<|et|>",
+    "<|mk|>",
+    "<|br|>",
+    "<|eu|>",
+    "<|is|>",
+    "<|hy|>",
+    "<|ne|>",
+    "<|mn|>",
+    "<|bs|>",
+    "<|kk|>",
+    "<|sq|>",
+    "<|sw|>",
+    "<|gl|>",
+    "<|mr|>",
+    "<|pa|>",
+    "<|si|>",
+    "<|km|>",
+    "<|sn|>",
+    "<|yo|>",
+    "<|so|>",
+    "<|af|>",
+    "<|oc|>",
+    "<|ka|>",
+    "<|be|>",
+    "<|tg|>",
+    "<|sd|>",
+    "<|gu|>",
+    "<|am|>",
+    "<|yi|>",
+    "<|lo|>",
+    "<|uz|>",
+    "<|fo|>",
+    "<|ht|>",
+    "<|ps|>",
+    "<|tk|>",
+    "<|nn|>",
+    "<|mt|>",
+    "<|sa|>",
+    "<|lb|>",
+    "<|my|>",
+    "<|bo|>",
+    "<|tl|>",
+    "<|mg|>",
+    "<|as|>",
+    "<|tt|>",
+    "<|haw|>",
+    "<|ln|>",
+    "<|ha|>",
+    "<|ba|>",
+    "<|jw|>",
+    "<|su|>",
+    "<|yue|>",
+    "<|translate|>",
+    "<|transcribe|>",
+    "<|startoflm|>",
+    "<|startofprev|>",
+    "<|nospeech|>",
+    "<|notimestamps|>"
+  ],
+  "is_local": false,
+  "language": null,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "predict_timestamps": false,
+  "processor_class": "WhisperProcessor",
+  "task": null,
+  "tokenizer_class": "WhisperTokenizer",
+  "unk_token": "<|endoftext|>"
+}