VoxMel

Wavelet-factorized autoregressive Mel-spectrogram modeling for zero-shot speech synthesis

Audio demos · GitHub · Technical report

VoxMel is an open-weight text-to-speech model from VoxDance Lab for English and Mandarin zero-shot voice cloning. It uses a short reference recording and its transcript to synthesize new text in the reference voice.

VoxMel models Mel spectrograms directly without a learned acoustic tokenizer. A fixed two-axis Haar transform separates the coarse spectrogram from its temporal and frequency details. The model generates the coarse frames one at a time and recovers the details in two subsequent passes.

VoxMel inference animation showing coarse-frame feedback, temporal and frequency reconstruction, and waveform synthesis

VoxMel inference.

Models and languages

Checkpoint Parameters Languages
VoxMel-0.6B 0.6B English and Mandarin
VoxMel-0.3B 0.3B English and Mandarin

Quick start

Listen. The project page pairs reference recordings with synthesized speech. Open audio demos

Run locally. Use Python 3.10–3.12 and install matching PyTorch and torchaudio builds for your device. Clone the repository and install the package.

git clone https://github.com/VoxDance-Lab/VoxMel.git
cd VoxMel
pip install -e .

Download your VoxMel checkpoint and its config.yaml, plus the 24 kHz Vocos weights and config. The example below expects the VoxMel checkpoint under its default filename, model_L_final.pt. See Prepare Model Files for the directory layout.

import soundfile as sf
from voxcore import VoxMel

model = VoxMel.from_pretrained(
    '/path/to/voxmel-experiment',
    tokenizer_path='examples/voxmel/qwen_tokenizer',
    vocos_path='/path/to/vocos-mel-24khz',
    device='auto',
)

waveform = model.generate(
    'This is a demonstration of zero-shot voice cloning.',
    prompt_wav_path='reference.wav',
    prompt_text='Replace this with the exact words spoken in reference.wav.',
    seed=42,
)
sf.write('voxmel-output.wav', waveform, model.sample_rate)

The reference transcript must match the words spoken in the recording. For a local interface, use the Gradio demo instructions.

Architecture

Stage What it does
Acoustic representation A fixed, invertible Haar transform splits the training Mel spectrogram along frequency and time into Coarse-Mel and two residuals.
Autoregressive generation A decoder-only Transformer and a flow matching head predict the next continuous Coarse-Mel frame from the conditioning inputs and preceding coarse frames.
Temporal reconstruction A non-autoregressive tier predicts the temporal residual. Inverse time-axis Haar reconstructs the frequency-averaged Mel spectrogram.
Frequency reconstruction A second non-autoregressive tier predicts the frequency residual. Inverse frequency-axis Haar recovers the full Mel spectrogram.
Waveform synthesis A frozen Vocos vocoder renders the reconstructed Mel spectrogram as 24 kHz audio.

The text frontend uses the Qwen2.5 tokenizer. Invertibility applies to the fixed Haar transform. Audio quality depends on the generated Mel features and vocoder.

Scope and license

VoxMel was evaluated primarily on short-form English and Mandarin speech. We’re working on long-form generation, multi-speaker dialogue, and support for more languages. Follow VoxDance Lab for updates.

VoxMel code and weights are released under Apache 2.0. Third-party components have their own licenses.

Citation

@misc{guan2026voxmel,
  title  = {VoxMel TTS Technical Report},
  author = {Guan, Wenhao and Ye, Jiaxin and Li, Haitao and Duan, Yifan and
            Liu, Junxi and Zhou, Wangjin and Li, Lin and Hong, Qingyang and Chen, Xie},
  year   = {2026},
  eprint        = {2611.20254},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2611.20254}
}
Downloads last month
274
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Space using VoxDance-Lab/VoxMel-0.3B 1

Paper for VoxDance-Lab/VoxMel-0.3B