README.md

---
language:
- hi
- en
license: mit
tags:
- tts
- text-to-speech
- voice-cloning
- hindi
- hinglish
- chatterbox
base_model: ResembleAI/chatterbox-turbo
pipeline_tag: text-to-speech
---

# Chatterbox Turbo — Hindi/Hinglish Finetuned

Finetuned [Chatterbox Turbo](https://huggingface.co/ResembleAI/chatterbox-turbo) (350M, GPT-2 backbone) for **Hindi (romanized) and English** text-to-speech with voice cloning.

## Key Features
- **Bilingual**: Speaks both Hindi (romanized Latin script) and English
- **Hinglish**: Handles code-mixed Hindi-English seamlessly
- **Voice Cloning**: Provide any 5-10s reference audio to clone the voice
- **Fast**: Single-step decoder, ~6x faster than real-time on GPU

## How It Works
Hindi text is written in **romanized form** (Latin script), not Devanagari. This allows the GPT-2 BPE tokenizer to handle it natively without any vocabulary extension.

Example: `"bharat ke kisan bahut mehnat karte hai"` instead of `"भारत के किसान बहुत मेहनत करते हैं"`

## Usage

### Prerequisites
```bash
pip install chatterbox-tts safetensors torch torchaudio soundfile
```

### Quick Inference
```python
import torch
import soundfile as sf
from safetensors.torch import load_file
from chatterbox.tts_turbo import ChatterboxTurboTTS
from chatterbox.models.t3.t3 import T3

# Load base Chatterbox Turbo
engine = ChatterboxTurboTTS.from_pretrained(device="cuda")

# Load finetuned T3 weights
t3_config = engine.t3.hp
t3_config.text_tokens_dict_size = 50276
new_t3 = T3(hp=t3_config)
if hasattr(new_t3.tfmr, "wte"):
    del new_t3.tfmr.wte

state_dict = load_file("t3_turbo_finetuned.safetensors", device="cpu")
new_t3.load_state_dict(state_dict, strict=True)

engine.t3 = new_t3
engine.t3.to("cuda").eval()

# Generate speech
wav = engine.generate(
    text="yeh ek bahut acchi baat hai ki hum sab milkar kaam kar rahe hai.",
    audio_prompt_path="reference.wav",  # 5-10s reference clip of target voice
    temperature=0.5,
)
sf.write("output.wav", wav.squeeze().cpu().numpy(), 24000)
```

### Text Format
- **Hindi**: Use romanized text (Latin script). Example: `"namaste, mera naam Ketav hai"`
- **English**: Use as-is. Example: `"Hello, my name is Ketav"`
- **Hinglish**: Mix freely. Example: `"mujhe lagta hai ki yeh project bahut successful hoga"`

### Romanization Guide
Common Hindi romanization patterns used in training:

| Hindi | Romanized |
|-------|-----------|
| है | hai |
| में | mein |
| यह | yeh |
| वो | voh |
| नहीं | nahi |
| बहुत | bahut |
| क्योंकि | kyonki |

### Inference Tips
- **Temperature 0.5** recommended (lower = more precise pronunciation)
- Reference audio must be **>5 seconds**
- Clean reference audio with minimal background noise works best

## Training Details

### Data
- **14,085 samples** (~20.4 hours) from a single male Hindi/English speaker
- 7,320 Hindi samples (romanized via IndicXlit + loanword dictionary)
- 6,765 English samples (original text)
- Duration filtered to 1-15 seconds per clip

### Text Processing Pipeline
1. Indic Normalize (DevanagariNormalizer)
2. English loanword replacement (23,019 entry dictionary)
3. IndicXlit transliteration cache (235,973 entries)
4. Lowercase + standardize romanization (62 rules)

### Hyperparameters
- Base model: `ResembleAI/chatterbox-turbo`
- Vocab size: 50,276 (original GPT-2, no extension)
- Batch size: 16, gradient accumulation: 2 (effective 32)
- Learning rate: 5e-5
- Epochs: 100
- Best checkpoint: step 38,000, loss 0.6685
- GPU: NVIDIA RTX 3090 (24GB)
- Training time: ~14 hours

### Loss Curve
| Epoch | Loss |
|-------|------|
| 1 | 7.204 |
| 10 | 3.672 |
| 20 | 2.162 |
| 30 | 1.519 |
| 50 | 0.938 |
| 80 | 0.669 |

## Files
| File | Description |
|------|-------------|
| `t3_turbo_finetuned.safetensors` | Finetuned T3 model weights (1.6 GB) |
| `inference.py` | Inference script with test sentences |
| `reference.wav` | Sample reference audio for voice cloning |
| `config.py` | Training configuration used |
| `TRAINING_NOTES.md` | Detailed training documentation |

## Limitations
- Only handles **romanized** Hindi text, not Devanagari script
- Voice quality depends on reference audio quality
- May merge words at high temperature (use 0.5)
- Trained on single male speaker — works for voice cloning of any voice, but Hindi pronunciation patterns are from one speaker

## Acknowledgments
- [Resemble AI](https://www.resemble.ai/) for Chatterbox Turbo
- [gokhaneraslan/chatterbox-finetuning](https://github.com/gokhaneraslan/chatterbox-finetuning) for the finetuning toolkit
- [AI4Bharat IndicXlit](https://github.com/AI4Bharat/IndicXlit) for transliteration