VoxMel
Wavelet-factorized autoregressive Mel-spectrogram modeling for zero-shot speech synthesis
Audio demos · GitHub · Technical report
VoxMel is an open-weight text-to-speech model from VoxDance Lab for English and Mandarin zero-shot voice cloning. It uses a short reference recording and its transcript to synthesize new text in the reference voice.
VoxMel models Mel spectrograms directly without a learned acoustic tokenizer. A fixed two-axis Haar transform separates the coarse spectrogram from its temporal and frequency details. The model generates the coarse frames one at a time and recovers the details in two subsequent passes.
VoxMel inference.
Models and languages
| Checkpoint | Parameters | Languages |
|---|---|---|
| VoxMel-0.6B | 0.6B | English and Mandarin |
| VoxMel-0.3B | 0.3B | English and Mandarin |
Quick start
Listen. The project page pairs reference recordings with synthesized speech. Open audio demos
Run locally. Use Python 3.10–3.12 and install matching PyTorch and torchaudio builds for your device. Clone the repository and install the package.
git clone https://github.com/VoxDance-Lab/VoxMel.git
cd VoxMel
pip install -e .
Download your VoxMel checkpoint and its config.yaml, plus the 24 kHz Vocos weights and config. The example below expects the VoxMel checkpoint under its default filename, model_L_final.pt. See Prepare Model Files for the directory layout.
import soundfile as sf
from voxcore import VoxMel
model = VoxMel.from_pretrained(
'/path/to/voxmel-experiment',
tokenizer_path='examples/voxmel/qwen_tokenizer',
vocos_path='/path/to/vocos-mel-24khz',
device='auto',
)
waveform = model.generate(
'This is a demonstration of zero-shot voice cloning.',
prompt_wav_path='reference.wav',
prompt_text='Replace this with the exact words spoken in reference.wav.',
seed=42,
)
sf.write('voxmel-output.wav', waveform, model.sample_rate)
The reference transcript must match the words spoken in the recording. For a local interface, use the Gradio demo instructions.
Architecture
| Stage | What it does |
|---|---|
| Acoustic representation | A fixed, invertible Haar transform splits the training Mel spectrogram along frequency and time into Coarse-Mel and two residuals. |
| Autoregressive generation | A decoder-only Transformer and a flow matching head predict the next continuous Coarse-Mel frame from the conditioning inputs and preceding coarse frames. |
| Temporal reconstruction | A non-autoregressive tier predicts the temporal residual. Inverse time-axis Haar reconstructs the frequency-averaged Mel spectrogram. |
| Frequency reconstruction | A second non-autoregressive tier predicts the frequency residual. Inverse frequency-axis Haar recovers the full Mel spectrogram. |
| Waveform synthesis | A frozen Vocos vocoder renders the reconstructed Mel spectrogram as 24 kHz audio. |
The text frontend uses the Qwen2.5 tokenizer. Invertibility applies to the fixed Haar transform. Audio quality depends on the generated Mel features and vocoder.
Scope and license
VoxMel was evaluated primarily on short-form English and Mandarin speech. We’re working on long-form generation, multi-speaker dialogue, and support for more languages. Follow VoxDance Lab for updates.
VoxMel code and weights are released under Apache 2.0. Third-party components have their own licenses.
Citation
@misc{guan2026voxmel,
title = {VoxMel TTS Technical Report},
author = {Guan, Wenhao and Ye, Jiaxin and Li, Haitao and Duan, Yifan and
Liu, Junxi and Zhou, Wangjin and Li, Lin and Hong, Qingyang and Chen, Xie},
year = {2026},
eprint = {2611.20254},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.20254}
}
- Downloads last month
- 274
