Chatterbox Turbo finetuned for Hindi/Hinglish TTS (romanized text)

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+reference.wav filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,147 @@
+---
+language:
+- hi
+- en
+license: mit
+tags:
+- tts
+- text-to-speech
+- voice-cloning
+- hindi
+- hinglish
+- chatterbox
+base_model: ResembleAI/chatterbox-turbo
+pipeline_tag: text-to-speech
+---
+
+# Chatterbox Turbo — Hindi/Hinglish Finetuned
+
+Finetuned [Chatterbox Turbo](https://huggingface.co/ResembleAI/chatterbox-turbo) (350M, GPT-2 backbone) for **Hindi (romanized) and English** text-to-speech with voice cloning.
+
+## Key Features
+- **Bilingual**: Speaks both Hindi (romanized Latin script) and English
+- **Hinglish**: Handles code-mixed Hindi-English seamlessly
+- **Voice Cloning**: Provide any 5-10s reference audio to clone the voice
+- **Fast**: Single-step decoder, ~6x faster than real-time on GPU
+
+## How It Works
+Hindi text is written in **romanized form** (Latin script), not Devanagari. This allows the GPT-2 BPE tokenizer to handle it natively without any vocabulary extension.
+
+Example: `"bharat ke kisan bahut mehnat karte hai"` instead of `"भारत के किसान बहुत मेहनत करते हैं"`
+
+## Usage
+
+### Prerequisites
+```bash
+pip install chatterbox-tts safetensors torch torchaudio soundfile
+```
+
+### Quick Inference
+```python
+import torch
+import soundfile as sf
+from safetensors.torch import load_file
+from chatterbox.tts_turbo import ChatterboxTurboTTS
+from chatterbox.models.t3.t3 import T3
+
+# Load base Chatterbox Turbo
+engine = ChatterboxTurboTTS.from_pretrained(device="cuda")
+
+# Load finetuned T3 weights
+t3_config = engine.t3.hp
+t3_config.text_tokens_dict_size = 50276
+new_t3 = T3(hp=t3_config)
+if hasattr(new_t3.tfmr, "wte"):
+    del new_t3.tfmr.wte
+
+state_dict = load_file("t3_turbo_finetuned.safetensors", device="cpu")
+new_t3.load_state_dict(state_dict, strict=True)
+
+engine.t3 = new_t3
+engine.t3.to("cuda").eval()
+
+# Generate speech
+wav = engine.generate(
+    text="yeh ek bahut acchi baat hai ki hum sab milkar kaam kar rahe hai.",
+    audio_prompt_path="reference.wav",  # 5-10s reference clip of target voice
+    temperature=0.5,
+)
+sf.write("output.wav", wav.squeeze().cpu().numpy(), 24000)
+```
+
+### Text Format
+- **Hindi**: Use romanized text (Latin script). Example: `"namaste, mera naam Ketav hai"`
+- **English**: Use as-is. Example: `"Hello, my name is Ketav"`
+- **Hinglish**: Mix freely. Example: `"mujhe lagta hai ki yeh project bahut successful hoga"`
+
+### Romanization Guide
+Common Hindi romanization patterns used in training:
+
+| Hindi | Romanized |
+|-------|-----------|
+| है | hai |
+| में | mein |
+| यह | yeh |
+| वो | voh |
+| नहीं | nahi |
+| बहुत | bahut |
+| क्योंकि | kyonki |
+
+### Inference Tips
+- **Temperature 0.5** recommended (lower = more precise pronunciation)
+- Reference audio must be **>5 seconds**
+- Clean reference audio with minimal background noise works best
+
+## Training Details
+
+### Data
+- **14,085 samples** (~20.4 hours) from a single male Hindi/English speaker
+- 7,320 Hindi samples (romanized via IndicXlit + loanword dictionary)
+- 6,765 English samples (original text)
+- Duration filtered to 1-15 seconds per clip
+
+### Text Processing Pipeline
+1. Indic Normalize (DevanagariNormalizer)
+2. English loanword replacement (23,019 entry dictionary)
+3. IndicXlit transliteration cache (235,973 entries)
+4. Lowercase + standardize romanization (62 rules)
+
+### Hyperparameters
+- Base model: `ResembleAI/chatterbox-turbo`
+- Vocab size: 50,276 (original GPT-2, no extension)
+- Batch size: 16, gradient accumulation: 2 (effective 32)
+- Learning rate: 5e-5
+- Epochs: 100
+- Best checkpoint: step 38,000, loss 0.6685
+- GPU: NVIDIA RTX 3090 (24GB)
+- Training time: ~14 hours
+
+### Loss Curve
+| Epoch | Loss |
+|-------|------|
+| 1 | 7.204 |
+| 10 | 3.672 |
+| 20 | 2.162 |
+| 30 | 1.519 |
+| 50 | 0.938 |
+| 80 | 0.669 |
+
+## Files
+| File | Description |
+|------|-------------|
+| `t3_turbo_finetuned.safetensors` | Finetuned T3 model weights (1.6 GB) |
+| `inference.py` | Inference script with test sentences |
+| `reference.wav` | Sample reference audio for voice cloning |
+| `config.py` | Training configuration used |
+| `TRAINING_NOTES.md` | Detailed training documentation |
+
+## Limitations
+- Only handles **romanized** Hindi text, not Devanagari script
+- Voice quality depends on reference audio quality
+- May merge words at high temperature (use 0.5)
+- Trained on single male speaker — works for voice cloning of any voice, but Hindi pronunciation patterns are from one speaker
+
+## Acknowledgments
+- [Resemble AI](https://www.resemble.ai/) for Chatterbox Turbo
+- [gokhaneraslan/chatterbox-finetuning](https://github.com/gokhaneraslan/chatterbox-finetuning) for the finetuning toolkit
+- [AI4Bharat IndicXlit](https://github.com/AI4Bharat/IndicXlit) for transliteration

TRAINING_NOTES.md

@@ -0,0 +1,135 @@
+# Chatterbox Turbo Hinglish Finetuning — Training Notes
+
+## Overview
+Finetuning Chatterbox Turbo (350M, GPT-2 backbone) for **Hindi (romanized) + English** TTS using a single male speaker.
+
+## Data Pipeline
+
+### Source Data
+- **Hindi male**: 7,320 files from `/workspace/data/bilingual/hindi/wav_22k/` (train_hindimale_*.wav)
+- **English male**: 6,765 files from `/workspace/data/bilingual/english/wav_22k/` (train_hindifullmale_*.wav)
+- **Total**: 14,085 files, ~21.3 hours → filtered to 13,941 (1-15s duration), ~20.4 hours
+- **Audio format**: 22kHz mono WAV (auto-resampled to 16kHz by Chatterbox)
+- **Speaker**: Single male speaker (speaker ID 1)
+
+### Text Processing Pipeline (Hindi only, English left intact)
+1. **Indic Normalize** — DevanagariNormalizer from IndicNLP (nukta decomposition, visarga fixes)
+2. **Loanword Replacement** — 23,019 entry Devanagari→English dictionary (`scripts/utils/english_loanwords.py` from asr-main repo)
+3. **Transliteration** — xlit_cache_merged.json (235,973 entries) maps remaining Devanagari words to romanized Latin
+4. **Lowercase** — all Hindi text lowercased
+5. **Standardize Romanization** — 62 rules for common variations (hain→hai, vo→voh, bahot→bahut, etc.)
+6. **English text** — left completely untouched from source manifests
+
+### Coverage
+- xlit cache: 91,323 tokens (96.9%)
+- Loanword dict: 2,857 tokens (3.0%)
+- Missing: 1 token (junk `__वें`)
+- **Total: 100% coverage**
+
+### Dataset Format
+- **LJSpeech format**: `metadata.csv` (filename|text|text) + `wavs/` directory (symlinks)
+- Sorted by duration (ascending) to minimize padding waste
+- Uploaded to HuggingFace: `ketav/hinglish-tts-data/romanized_transcripts_hindimale.zip`
+
+## Model Configuration
+
+### No Vocab Extension Needed
+- All text is Latin characters (romanized Hindi + English)
+- GPT-2 BPE tokenizer handles this natively
+- `new_vocab_size = 50276` (matches pretrained T3 turbo exactly)
+- Skipped the `setup.py` vocab merging step entirely
+
+### Training Config
+```python
+is_turbo = True
+new_vocab_size = 50276
+batch_size = 16
+grad_accum = 2        # effective batch = 32
+learning_rate = 5e-5
+num_epochs = 100
+save_steps = 1000
+max_speech_len = 850
+max_text_len = 256
+prompt_duration = 3.0  # seconds
+```
+
+### Speed Optimizations Applied
+- `torch.backends.cuda.matmul.allow_tf32 = True` (~15% faster)
+- `torch.set_float32_matmul_precision('high')`
+- `bf16 = True` (mixed precision)
+- `gradient_checkpointing = True` (VRAM savings)
+- Duration-sorted data (less padding waste)
+- Filtered clips to 1-15s (removed 144 extreme outliers)
+- `torch.compile` — tried but incompatible with T3 model (CUDAGraphs conflict)
+
+### GPU
+- NVIDIA RTX 3090 (24GB)
+- VRAM usage: ~13.8 GB peak (with batch_size=16)
+- GPU utilization: 96-100%
+- Training speed: ~1.17-1.23 s/step
+
+## Training Progress
+
+### Loss Curve
+| Epoch | Loss   |
+|-------|--------|
+| 1     | 7.204  |
+| 5     | 5.042  |
+| 10    | 3.672  |
+| 20    | 2.162  |
+| 30    | 1.519  |
+| 40    | 1.127  |
+| 50    | 0.938  |
+| 60    | 0.822  |
+
+### Checkpoints
+Best checkpoints saved to `best_models/` by automated checkpoint_saver.py script.
+Backup copy at `/workspace/best_models_backup/`.
+
+## Inference
+
+### Reference Audio
+- Must be >5 seconds (Chatterbox Turbo requirement)
+- Using `train_hindimale_04569.wav` (6.28s Hindi male clip)
+
+### Best Settings
+```python
+temperature = 0.5   # lower = more precise, less word-merging artifacts
+# exaggeration and repetition_penalty are ignored by Turbo mode
+```
+
+### Known Issues
+- **Word merging**: At temperature 0.8, words can blend together (e.g., "company ka growth" → "compgrowth")
+- **Fix**: Lower temperature to 0.5 resolves this
+- Turbo mode ignores `exaggeration`, `cfg_weight`, and `min_p` parameters
+
+### Test Sentences Used
+```
+Hindi:    "yeh ek bahut acchi baat hai ki hum sab milkar kaam kar rahe hai."
+Hindi:    "bharat ke kisan bahut mehnat karte hai lekin unhe sahi daam nahi milta."
+English:  "The weather today is beautiful and I feel very happy about it."
+English:  "Technology has changed our lives in ways we never imagined before."
+Hinglish: "mujhe lagta hai ki yeh project bahut successful hoga because humne bahut hard work kiya hai."
+Hinglish: "aaj meeting mein boss ne kaha ki next quarter mein company ka growth double hoga."
+```
+
+## Monitoring
+- **Telegram bot**: Sends training updates every 30 min (loss, LR, epoch, progress)
+- **Checkpoint saver**: Keeps best 3 checkpoints by loss, detects overfitting (3 consecutive loss increases)
+- **TensorBoard**: `tensorboard --logdir chatterbox_output --port 6006`
+
+## Files Created
+| File | Purpose |
+|------|---------|
+| `src/config.py` | Training configuration |
+| `train.py` | Modified with resume support, tf32, torch.load patch |
+| `test_inference.py` | Mid-training inference from best checkpoint |
+| `telegram_monitor.py` | Telegram training status updates |
+| `checkpoint_saver.py` | Best checkpoint tracking + overfitting detection |
+| `TRAINING_NOTES.md` | This file |
+
+## Key Repos & Data
+- **Finetuning repo**: gokhaneraslan/chatterbox-finetuning
+- **ASR pipeline (text processing)**: `/workspace/asr-main/asr-main/`
+- **HF dataset**: `ketav/hinglish-tts-data` (private)
+- **HF parakeet data**: `ketav/hinglish-parakeet-tarred`, `ketav/hinglish-v2-sorted-tarred`

config.py

@@ -0,0 +1,42 @@
+from dataclasses import dataclass
+
+@dataclass
+class TrainConfig:
+    # --- Paths ---
+    model_dir: str = "./pretrained_models"
+    csv_path: str = "/workspace/chatterbox_dataset/metadata.csv"
+    metadata_path: str = "./metadata.json"
+    wav_dir: str = "/workspace/chatterbox_dataset/wavs"
+    preprocessed_dir = "/workspace/chatterbox_dataset/preprocess"
+    output_dir: str = "./chatterbox_output"
+
+    is_inference = False
+    inference_prompt_path: str = "./speaker_reference/reference.wav"
+    inference_test_text: str = "yeh ek bahut acchi baat hai ki hum sab milkar kaam kar rahe hai."
+
+    ljspeech = True
+    json_format = False
+    preprocess = False
+
+    is_turbo: bool = True
+
+    # --- Vocabulary ---
+    # Match pretrained T3 turbo vocab size exactly (GPT-2 50257 + 19 special tokens)
+    new_vocab_size: int = 50276
+
+    # --- Hyperparameters ---
+    batch_size: int = 16         # RTX 3090 24GB — plenty of headroom
+    grad_accum: int = 2          # Effective batch = 32
+    learning_rate: float = 5e-5
+    num_epochs: int = 100
+
+    save_steps: int = 1000
+    save_total_limit: int = 20
+    dataloader_num_workers: int = 8
+
+    # --- Constraints ---
+    start_text_token = 255
+    stop_text_token = 0
+    max_text_len: int = 256
+    max_speech_len: int = 850
+    prompt_duration: float = 3.0

inference.py

@@ -0,0 +1,107 @@
+#!/usr/bin/env python3
+"""Quick inference test using best checkpoint. Runs separately from training."""
+
+import os
+import glob
+import torch
+import numpy as np
+import soundfile as sf
+from safetensors.torch import load_file
+
+from src.config import TrainConfig
+from src.chatterbox_.tts_turbo import ChatterboxTurboTTS
+from src.chatterbox_.models.t3.t3 import T3
+from src.utils import trim_silence_with_vad
+
+# Patch torch.load for compatibility
+_original_torch_load = torch.load
+torch.load = lambda *args, **kwargs: _original_torch_load(*args, **{**kwargs, "weights_only": kwargs.get("weights_only", False)})
+
+cfg = TrainConfig()
+DEVICE = "cuda"
+REFERENCE_AUDIO = "./speaker_reference/reference.wav"
+OUTPUT_DIR = "/workspace/inference_test_v3"
+os.makedirs(OUTPUT_DIR, exist_ok=True)
+
+# Test sentences
+TEST_SENTENCES = {
+    "hindi_1": "yeh ek bahut acchi baat hai ki hum sab milkar kaam kar rahe hai.",
+    "hindi_2": "bharat ke kisan bahut mehnat karte hai lekin unhe sahi daam nahi milta.",
+    "english_1": "The weather today is beautiful and I feel very happy about it.",
+    "english_2": "Technology has changed our lives in ways we never imagined before.",
+    "hinglish_1": "mujhe lagta hai ki yeh project bahut successful hoga because humne bahut hard work kiya hai.",
+    "hinglish_2": "aaj meeting mein boss ne kaha ki next quarter mein company ka growth double hoga.",
+}
+
+def find_best_checkpoint():
+    """Find the best checkpoint from best_models/ or latest from training."""
+    best_dir = "/workspace/chatterbox-finetuning/best_models"
+    if os.path.exists(best_dir):
+        ckpts = sorted(glob.glob(f"{best_dir}/best_loss*"), key=lambda x: float(x.split("loss")[1].split("_")[0]))
+        if ckpts:
+            # Find the safetensors file inside
+            for f in os.listdir(ckpts[0]):
+                if f.endswith(".safetensors") and "model" in f:
+                    return os.path.join(ckpts[0], f)
+            # Try trainer format
+            model_path = os.path.join(ckpts[0], "model.safetensors")
+            if os.path.exists(model_path):
+                return model_path
+    return None
+
+def load_engine(checkpoint_path):
+    print(f"Loading base model...")
+    engine = ChatterboxTurboTTS.from_local(cfg.model_dir, device="cpu")
+
+    print(f"Creating T3 with vocab size: {cfg.new_vocab_size}")
+    t3_config = engine.t3.hp
+    t3_config.text_tokens_dict_size = cfg.new_vocab_size
+    new_t3 = T3(hp=t3_config)
+
+    if hasattr(new_t3.tfmr, "wte"):
+        del new_t3.tfmr.wte
+
+    print(f"Loading checkpoint: {checkpoint_path}")
+    state_dict = load_file(checkpoint_path, device="cpu")
+    # Strip "t3." prefix if saved by HF Trainer wrapper
+    if any(k.startswith("t3.") for k in state_dict):
+        state_dict = {k.replace("t3.", "", 1): v for k, v in state_dict.items()}
+    new_t3.load_state_dict(state_dict, strict=True)
+
+    engine.t3 = new_t3
+    engine.t3.to(DEVICE).eval()
+    engine.s3gen.to(DEVICE).eval()
+    engine.ve.to(DEVICE).eval()
+    engine.device = DEVICE
+
+    return engine
+
+def generate(engine, text, output_path):
+    with torch.no_grad():
+        wav = engine.generate(
+            text=text,
+            audio_prompt_path=REFERENCE_AUDIO,
+            temperature=0.5,
+        )
+    wav_np = wav.squeeze().cpu().numpy()
+    wav_np = trim_silence_with_vad(wav_np, engine.sr)
+    sf.write(output_path, wav_np, engine.sr)
+    return len(wav_np) / engine.sr
+
+if __name__ == "__main__":
+    ckpt = find_best_checkpoint()
+    if not ckpt:
+        print("No checkpoint found!")
+        exit(1)
+
+    print(f"Best checkpoint: {ckpt}")
+    engine = load_engine(ckpt)
+
+    print(f"\nGenerating {len(TEST_SENTENCES)} test sentences...\n")
+    for name, text in TEST_SENTENCES.items():
+        out_path = os.path.join(OUTPUT_DIR, f"{name}.wav")
+        dur = generate(engine, text, out_path)
+        print(f"  {name}: {dur:.1f}s -> {out_path}")
+        print(f"    Text: {text}")
+
+    print(f"\nDone! Audio files in {OUTPUT_DIR}/")

reference.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad65adc16d23ab41aa6e2f32ec19e51400acdb862e1055a724a776bd02997f71
+size 277188

t3_turbo_finetuned.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0ce1dd72db44baa59f4aa1996a64ca4ac5f69113117927f93b4236021f6c87f
+size 1709549468