chatterbox-nano-coreml

+MIT License
+Copyright (c) 2025 Resemble AI
+The models in this repository are Core ML conversions of
+ResembleAI/chatterbox-nano (https://huggingface.co/ResembleAI/chatterbox-nano),
+which is distributed under the MIT License. The conversion scripts and the
+resulting .mlpackage artifacts are redistributed under the same terms.
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md CHANGED Viewed

@@ -1,206 +1,82 @@
 ---
 license: mit
 language:
 - en
 pipeline_tag: text-to-speech
 tags:
 - text-to-speech
-- speech
-- speech-generation
-- voice-cloning
 ---
-![resemble+cbturbo-1600x900](https://storage.googleapis.com/chatterbox-demo-samples/turbo/turbo-banner.jpg)
-# Chatterbox TTS
-<div style="display: flex; align-items: center; gap: 12px">
-  <a href="https://resemble-ai.github.io/chatterbox_turbo_demopage/">
-    <img src="https://img.shields.io/badge/listen-demo_samples-blue" alt="Listen to Demo Samples" />
-  </a>
-  <a href="https://huggingface.co/spaces/ResembleAI/chatterbox-nano-demo">
-    <img src="https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm.svg" alt="Open in HF Spaces" />
-  </a>
-  <a href="https://podonos.com/resembleai/chatterbox">
-    <img src="https://static-public.podonos.com/badges/insight-on-pdns-sm-dark.svg" alt="Insight on Podos" />
-  </a>
-</div>
-<div style="display: flex; align-items: center; gap: 8px;">
-  <span style="font-style: italic;white-space: pre-wrap">Made with  ❤️  by</span>
-  <img width="100" alt="resemble-logo-horizontal" src="https://github.com/user-attachments/assets/35cf756b-3506-4943-9c72-c05ddfa4e525" />
-</div>
-**Chatterbox** is a family of four state-of-the-art, open-source text-to-speech models by Resemble AI.
-We are excited to introduce **Chatterbox-Nano**, our most efficient model yet. Built on a streamlined 110M parameter architecture, **Nano** delivers high-quality speech on CPU - 3x faster than realtime on 8 cores. We have also distilled the speech-token-to-mel decoder, previously a bottleneck, reducing generation from 10 steps to just **one**, while retaining high-fidelity audio output.
-**Paralinguistic tags** are now native to both Nano and Turbo models, allowing you to use `[cough]`, `[laugh]`, `[chuckle]`, and more to add distinct realism. While Turbo was built primarily for low-latency voice agents, it excels at narration and creative workflows.
-If you like the model but need to scale or tune it for higher accuracy, check out our competitively priced TTS service (<a href="https://resemble.ai">link</a>). It delivers reliable performance with ultra-low latency of sub 200ms—ideal for production use in agents, applications, or interactive media.
-<img width="1200" height="600" alt="Podonos Turbo Eval" src="https://storage.googleapis.com/chatterbox-demo-samples/turbo/podonos_turbo.png" />
-### ⚡ Model Zoo
-Choose the right model for your application.
-| Model                                                                                                           | Size | Languages | Key Features                                            | Best For                                     | 🤗                                                                  | Examples |
-|:----------------------------------------------------------------------------------------------------------------| :--- | :--- |:--------------------------------------------------------|:---------------------------------------------|:--------------------------------------------------------------------------| :--- |
-| **Chatterbox-Nano**                                                                                            | **110M** | **English** | Paralinguistic Tags (`[laugh]`), 3x faster than realtime on 8-core CPU | On-device / CPU inference, tight latency & memory budgets | [Demo](https://huggingface.co/spaces/ResembleAI/chatterbox-nano-demo)        | [Listen](https://resemble-ai.github.io/chatterbox_turbo_demopage/) |
-| Chatterbox-Turbo                                                                                            | 350M | English | Paralinguistic Tags (`[laugh]`), Lower Compute and VRAM | Zero-shot voice agents,  Production          | [Demo](https://huggingface.co/spaces/ResembleAI/chatterbox-turbo-demo)        | [Listen](https://resemble-ai.github.io/chatterbox_turbo_demopage/) |
-| Chatterbox-Multilingual [(Language list)](#supported-languages)                                                 | 500M | 23+ | Zero-shot cloning, Multiple Languages                   | Global applications, Localization            | [Demo](https://huggingface.co/spaces/ResembleAI/Chatterbox-Multilingual-TTS) | [Listen](https://resemble-ai.github.io/chatterbox_demopage/) |
-| Chatterbox [(Tips and Tricks)](#original-chatterbox-tips)                                                       | 500M | English | CFG & Exaggeration tuning                               | General zero-shot TTS with creative controls | [Demo](https://huggingface.co/spaces/ResembleAI/Chatterbox)              | [Listen](https://resemble-ai.github.io/chatterbox_demopage/) |
-## Installation
-```shell
-pip install chatterbox-tts
-```
-Alternatively, you can install from source:
-```shell
-# conda create -yn chatterbox python=3.11
-# conda activate chatterbox
-git clone https://github.com/resemble-ai/chatterbox.git
-cd chatterbox
-pip install -e .
-```
-We developed and tested Chatterbox on Python 3.11 on Debian 11 OS; the versions of the dependencies are pinned in `pyproject.toml` to ensure consistency. You can modify the code or dependencies in this installation mode.
-## Usage
-##### Chatterbox-Nano
-Nano shares Turbo's architecture and is loaded through the same `ChatterboxTurboTTS` class by passing `nano=True`:
-```python
-import torchaudio as ta
-import torch
-from chatterbox.tts_turbo import ChatterboxTurboTTS
-# Load the Nano model (also runs on CPU: device="cpu")
-model = ChatterboxTurboTTS.from_pretrained(device="cuda", nano=True)
-# Generate with Paralinguistic Tags
-text = "Hi there, Sarah here from MochaFone calling you back [chuckle], have you got one minute to chat about the billing issue?"
-# Generate audio (requires a reference clip for voice cloning)
-wav = model.generate(text, audio_prompt_path="your_10s_ref_clip.wav")
-ta.save("test-nano.wav", wav, model.sr)
-```
-##### Chatterbox-Turbo
-```python
-import torchaudio as ta
-import torch
-from chatterbox.tts_turbo import ChatterboxTurboTTS
-# Load the Turbo model
-model = ChatterboxTurboTTS.from_pretrained(device="cuda")
-# Generate with Paralinguistic Tags
-text = "Hi there, Sarah here from MochaFone calling you back [chuckle], have you got one minute to chat about the billing issue?"
-# Generate audio (requires a reference clip for voice cloning)
-wav = model.generate(text, audio_prompt_path="your_10s_ref_clip.wav")
-ta.save("test-turbo.wav", wav, model.sr)
-```
-##### Chatterbox and Chatterbox-Multilingual
-```python
-import torchaudio as ta
-from chatterbox.tts import ChatterboxTTS
-from chatterbox.mtl_tts import ChatterboxMultilingualTTS
-device = "cuda"  # or "cpu" / "mps"
-# English example
-model = ChatterboxTTS.from_pretrained(device=device)
-text = "Ezreal and Jinx teamed up with Ahri, Yasuo, and Teemo to take down the enemy's Nexus in an epic late-game pentakill."
-wav = model.generate(text)
-ta.save("test-english.wav", wav, model.sr)
-# Multilingual examples
-multilingual_model = ChatterboxMultilingualTTS.from_pretrained(device=device)
-french_text = "Bonjour, comment ça va? Ceci est le modèle de synthèse vocale multilingue Chatterbox, il prend en charge 23 langues."
-wav_french = multilingual_model.generate(french_text, language_id="fr")
-ta.save("test-french.wav", wav_french, multilingual_model.sr)
-chinese_text = "你好，今天天气真不错，希望你有一个愉快的周末。"
-wav_chinese = multilingual_model.generate(chinese_text, language_id="zh")
-ta.save("test-chinese.wav", wav_chinese, multilingual_model.sr)
-# If you want to synthesize with a different voice, specify the audio prompt
-AUDIO_PROMPT_PATH = "YOUR_FILE.wav"
-wav = model.generate(text, audio_prompt_path=AUDIO_PROMPT_PATH)
-ta.save("test-2.wav", wav, model.sr)
-```
-See `example_tts.py`, `example_tts_turbo.py`, `example_tts_nano.py`, and `example_vc.py` for more examples.
-## Supported Languages
-Arabic (ar) • Danish (da) • German (de) • Greek (el) • English (en) • Spanish (es) • Finnish (fi) • French (fr) • Hebrew (he) • Hindi (hi) • Italian (it) • Japanese (ja) • Korean (ko) • Malay (ms) • Dutch (nl) • Norwegian (no) • Polish (pl) • Portuguese (pt) • Russian (ru) • Swedish (sv) • Swahili (sw) • Turkish (tr) • Chinese (zh)
-## Original Chatterbox Tips
-- **General Use (TTS and Voice Agents):**
-  - Ensure that the reference clip matches the specified language tag. Otherwise, language transfer outputs may inherit the accent of the reference clip’s language. To mitigate this, set `cfg_weight` to `0`.
-  - The default settings (`exaggeration=0.5`, `cfg_weight=0.5`) work well for most prompts across all languages.
-  - If the reference speaker has a fast speaking style, lowering `cfg_weight` to around `0.3` can improve pacing.
-- **Expressive or Dramatic Speech:**
-  - Try lower `cfg_weight` values (e.g. `~0.3`) and increase `exaggeration` to around `0.7` or higher.
-  - Higher `exaggeration` tends to speed up speech; reducing `cfg_weight` helps compensate with slower, more deliberate pacing.
-## Built-in PerTh Watermarking for Responsible AI
-Every audio file generated by Chatterbox includes [Resemble AI's Perth (Perceptual Threshold) Watermarker](https://github.com/resemble-ai/perth) - imperceptible neural watermarks that survive MP3 compression, audio editing, and common manipulations while maintaining nearly 100% detection accuracy.
-## Watermark extraction
-You can look for the watermark using the following script.
-```python
-import perth
-import librosa
-AUDIO_PATH = "YOUR_FILE.wav"
-# Load the watermarked audio
-watermarked_audio, sr = librosa.load(AUDIO_PATH, sr=None)
-# Initialize watermarker (same as used for embedding)
-watermarker = perth.PerthImplicitWatermarker()
-# Extract watermark
-watermark = watermarker.get_watermark(watermarked_audio, sample_rate=sr)
-print(f"Extracted watermark: {watermark}")
-# Output: 0.0 (no watermark) or 1.0 (watermarked)
-```
-## Official Discord
-👋 Join us on [Discord](https://discord.gg/rJq9cRJBJ6) and let's build something awesome together!
-## Acknowledgements
-- [Cosyvoice](https://github.com/FunAudioLLM/CosyVoice)
-- [Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)
-- [HiFT-GAN](https://github.com/yl4579/HiFTNet)
-- [Llama 3](https://github.com/meta-llama/llama3)
-- [S3Tokenizer](https://github.com/xingchensong/S3Tokenizer)
-## Citation
-If you find this model useful, please consider citing.
 ```
 @misc{chatterboxtts2025,
   author       = {{Resemble AI}},
@@ -210,5 +86,3 @@ If you find this model useful, please consider citing.
   note         = {GitHub repository}
 }
 ```
-## Disclaimer
-Don't use this model to do bad things. Prompts are sourced from freely available data on the internet.

 ---
 license: mit
+base_model: ResembleAI/chatterbox-nano
 language:
 - en
+library_name: coreml
 pipeline_tag: text-to-speech
 tags:
 - text-to-speech
+- coreml
+- ios
+- neural-engine
+- chatterbox
 ---
+# Chatterbox Nano — Core ML
+Core ML conversions of [ResembleAI/chatterbox-nano](https://huggingface.co/ResembleAI/chatterbox-nano),
+built for on-device text-to-speech on iOS. Used by [Annologue](https://github.com/seanlaidlaw),
+an iOS audiobook player, to synthesise chapters locally.
+These run **faster than real time on an iPhone**, with the two heaviest stages executing
+almost entirely on the Neural Engine.
+## Contents
+| File | Precision | Size | Compute units | Measured device placement |
+|---|---|---|---|---|
+| `t3_prefill_ane.mlpackage` | fp16 | 173 MB | CPU + GPU | ANE cannot compile this graph (`ANECCompile() FAILED`); it is ~1 % of wall time |
+| `t3_decode_ane.mlpackage` | fp16 | 173 MB | CPU + **ANE** | **525/537 ops = 97.8 % ANE** |
+| `s3gen_flow.mlpackage` | fp16 | 230 MB | CPU + **ANE** | **3300/3317 ops = 99.5 % ANE** |
+| `s3gen_vocoder.mlpackage` | fp16 (selective) | 41 MB | CPU + GPU | 179 convolutions over 288 000 samples exceed the ANE's per-dimension limits |
+| `t3_embeddings.bin` | fp16 | 86 MB | — | Text/speech/positional embedding tables, gathered on the CPU |
+| `nano_conds.bin` | — | 1.3 MB | — | Default speaker conditionals, with `cond_emb` precomputed |
+| `tokenizer.json`, `tokenizer_config.json` | — | 3.4 MB | — | — |
+`.mlpackage` files are uncompiled. Compile them on-device with
+`MLModel.compileModel(at:)` and cache the result — compilation takes roughly a minute
+in total and only needs to happen once per install.
+## How this differs from upstream
+**T3 is split into two packages.** Upstream is one autoregressive transformer; here prefill
+and decode are separate `MLModel`s with independent `MLState`s, and the KV prefix is copied
+across per utterance. This costs +173 MB of disk (the two halves do not share weights) and
+buys decode its own compute-unit placement, which is the entire reason decode can reach the
+Neural Engine while prefill cannot.
+**Decode's attention is written out rather than fused.** A fused
+`scaled_dot_product_attention` places on the ANE but the ANE **silently ignores its mask
+operand**, so attention runs over the whole KV cache including the ~500 slots holding
+nothing. The output stays fluent and is simply the wrong words. Spelling attention as
+`matmul → scale → add → softmax → matmul` makes the mask an ordinary `add` the compiler
+cannot drop. This cost no latency and lost no ANE placement.
+**S3Gen is split into flow and vocoder halves** so the flow half can be fp16 on the ANE
+while the vocoder stays on the GPU. The vocoder's HiFi-GAN upsamplers are re-expressed as
+sub-pixel (reshape + convolution) rather than transposed convolutions, which is numerically
+exact and removes ops that fell back to the CPU. Its `cumsum` phase accumulator is kept in
+fp32 — that one is a genuine fp16 hazard.
+**Quantisation stops at fp16.** int8 on S3Gen was measured and rejected: log-spectral
+distance against fp32 went from 0.034 to 0.354, and `MLComputePlan` stopped loading the
+model. int8 on T3 halves the weights but was measured at 0.95×/0.99× — a RAM lever, not a
+speed one.
+## ⚠️ No watermarking
+Upstream Chatterbox applies [Resemble AI's PerTh watermarker](https://github.com/resemble-ai/perth)
+to every generated file. **These conversions are the T3 and S3Gen models only — audio
+generated from them is not watermarked.** If you need the watermark, apply `perth` to the
+output yourself. Please use these responsibly.
+## Licence and attribution
+MIT, inherited from the upstream model. Copyright © 2025 Resemble AI — see `LICENSE`.
+The underlying model, training and weights are Resemble AI's work; this repository
+contributes only the Core ML conversion.
 ```
 @misc{chatterboxtts2025,
   author       = {{Resemble AI}},
   note         = {GitHub repository}
 }
 ```