a43992899 commited on
Commit
29b3558
·
verified ·
1 Parent(s): 1a96eca

Sync Suno v6 WildSongBench results and frontier figure

Browse files

Use competitive with Suno v5/v6; refresh the September 12 comparison and matching figure. Keep inference, weights, prompts and evaluation code unchanged.

Files changed (4) hide show
  1. README.md +7 -3
  2. assets/figure1.pdf +0 -0
  3. assets/figure1.png +2 -2
  4. assets/figure1.svg +0 -0
README.md CHANGED
@@ -52,14 +52,16 @@ tags:
52
  <a href="https://huggingface.co/m-a-p/SheetSage2"><img alt="SheetSage2" src="https://img.shields.io/badge/SheetSage2-374151?logo=huggingface&amp;logoColor=FFD21E" height="20" /></a>
53
  </p>
54
 
55
- **YuE2 is an open music generation model that rivals Suno v5.** Turn lyrics and a style prompt into a complete song with vocals and accompaniment, then shape its melody and chords through an editable score.
56
 
57
- **State-of-the-art results on WildSongBench.** YuE2 (best-of-8) achieves the highest SongBench average among all evaluated open and proprietary models: **6.9632**, compared with **6.8721** for Suno v5.
58
 
59
  ![YuE2 song quality and text alignment on WildSongBench](assets/figure1.png)
60
 
61
  *Frontier song quality and text alignment on 192 WildSongBench prompts. YuE2 uses symbolic planning; Bo8 means best-of-8.*
62
 
 
 
63
  - **Compose and edit:** melody + chords, melody-only, or direct generation; bring your own ABC score.
64
  - **Edit with an agent:** turn musical feedback into score, style and lyric revisions, then let YuE2 render the next version. [Hear the editing process](https://map-yue2.github.io/#agentic-music-editing).
65
  - **Run locally:** 48 kHz stereo songs on a 24GB GPU, without quantization.
@@ -304,6 +306,8 @@ Server: 32 songs per row; AR/NAR use PyTorch 2.10 and Triton 3.6, VAE uses PyTor
304
  | Suno v5 | 5.9918 | 6.8721 | **0.5428** | **0.4353** | 4.5907 | 8.10% |
305
  | Suno v4.5 | 5.8317 | 6.6995 | 0.5022 | 0.3873 | 4.4149 | **5.80%** |
306
  | Suno v5.5 | 5.8087 | 6.7150 | 0.5089 | 0.3917 | 4.5914 | 5.96% |
 
 
307
  | MiniMax Music 2.6 | 5.4437 | 6.3222 | 0.4251 | 0.3670 | 4.5688 | 24.55% |
308
  | Mureka 9 | 6.0488 | 6.9377 | 0.4394 | 0.4102 | 4.6368 | 11.69% |
309
  | **YuE2** | 5.9075 | 6.7316 | 0.5068 | 0.4054 | 4.6819 | 8.44% |
@@ -328,7 +332,7 @@ Server: 32 songs per row; AR/NAR use PyTorch 2.10 and Triton 3.6, VAE uses PyTor
328
  <details>
329
  <summary>📐 Evaluation protocols and metric definitions</summary>
330
 
331
- WSB: SongBench Avg averages seven dimensions; Q3O measures prompt adherence on a 0–5 scale; PER is phoneme error rate. YuE2 selects the lower-PER candidate from two. Best-of-8 selects by Musicality → Q3O → PER. Each candidate's PER uses the lowest-PER of four ASR passes. A pipeline call generates one candidate; selection is separate. Q3O weights differ on 10 of 192 prompts between the two YuE2 settings. Bold marks the best value within each table. Open baselines use two candidates and four ASR passes; proprietary systems retain their delivered-candidate protocols. MiniMax Music 3 uses its official caption rewriter. SongBloom uses a fixed audio prompt rather than a style-text input.
332
 
333
  SHS100K: CLEWS and Discogs-VINet measure preserved song identity against 10,545 recordings after source exclusion. MuLan measures target-style similarity; Musicality is from SongBench. Identity and quality should be read together. Every displayed metric covers all 3,792 outputs per method; incomplete Q3O scores are omitted. Source-score extraction is separate from this kit.
334
 
 
52
  <a href="https://huggingface.co/m-a-p/SheetSage2"><img alt="SheetSage2" src="https://img.shields.io/badge/SheetSage2-374151?logo=huggingface&amp;logoColor=FFD21E" height="20" /></a>
53
  </p>
54
 
55
+ **YuE2 is an open music generation model with frontier song quality competitive with Suno v5/v6.** Turn lyrics and a style prompt into a complete song with vocals and accompaniment, then shape its melody and chords through an editable score.
56
 
57
+ **State-of-the-art results on WildSongBench.** YuE2 (best-of-8) achieves the highest SongBench average among all evaluated open and proprietary models: **6.9632**, compared with **6.8721** for Suno v5, **6.5562** for Suno v6, and **6.4195** for Suno v6 Wild.
58
 
59
  ![YuE2 song quality and text alignment on WildSongBench](assets/figure1.png)
60
 
61
  *Frontier song quality and text alignment on 192 WildSongBench prompts. YuE2 uses symbolic planning; Bo8 means best-of-8.*
62
 
63
+ September 12, 2026 comparison: 17 settings, including Suno v6 and Suno v6 Wild. [Vector PDF](assets/figure1.pdf) · [SVG](assets/figure1.svg) · [All nine metrics and protocols](https://huggingface.co/datasets/m-a-p/WildSongBench/blob/main/benchmark/results.md) · [Full-precision CSV](https://huggingface.co/datasets/m-a-p/WildSongBench/resolve/main/benchmark/benchmark-results.csv).
64
+
65
  - **Compose and edit:** melody + chords, melody-only, or direct generation; bring your own ABC score.
66
  - **Edit with an agent:** turn musical feedback into score, style and lyric revisions, then let YuE2 render the next version. [Hear the editing process](https://map-yue2.github.io/#agentic-music-editing).
67
  - **Run locally:** 48 kHz stereo songs on a 24GB GPU, without quantization.
 
306
  | Suno v5 | 5.9918 | 6.8721 | **0.5428** | **0.4353** | 4.5907 | 8.10% |
307
  | Suno v4.5 | 5.8317 | 6.6995 | 0.5022 | 0.3873 | 4.4149 | **5.80%** |
308
  | Suno v5.5 | 5.8087 | 6.7150 | 0.5089 | 0.3917 | 4.5914 | 5.96% |
309
+ | Suno v6 | 5.6558 | 6.5562 | 0.4916 | 0.4305 | 4.6258 | 7.58% |
310
+ | Suno v6 Wild | 5.5644 | 6.4195 | 0.4999 | 0.4316 | 4.5898 | 7.45% |
311
  | MiniMax Music 2.6 | 5.4437 | 6.3222 | 0.4251 | 0.3670 | 4.5688 | 24.55% |
312
  | Mureka 9 | 6.0488 | 6.9377 | 0.4394 | 0.4102 | 4.6368 | 11.69% |
313
  | **YuE2** | 5.9075 | 6.7316 | 0.5068 | 0.4054 | 4.6819 | 8.44% |
 
332
  <details>
333
  <summary>📐 Evaluation protocols and metric definitions</summary>
334
 
335
+ WSB: SongBench Avg averages seven dimensions; Q3O measures prompt adherence on a 0–5 scale; PER is phoneme error rate. YuE2 selects the lower-PER candidate from two. Best-of-8 selects by Musicality → Q3O → PER. Each candidate's PER uses the lowest-PER of four ASR passes. A pipeline call generates one candidate; selection is separate. Q3O weights differ on 10 of 192 prompts between the two YuE2 settings. Bold marks the best value within each table. Open baselines, Suno v6, and Suno v6 Wild use two candidates and four ASR passes per candidate, followed by lower-PER selection; earlier proprietary systems retain their delivered-candidate protocols. MiniMax Music 3 uses its official caption rewriter. SongBloom uses a fixed audio prompt rather than a style-text input.
336
 
337
  SHS100K: CLEWS and Discogs-VINet measure preserved song identity against 10,545 recordings after source exclusion. MuLan measures target-style similarity; Musicality is from SongBench. Identity and quality should be read together. Every displayed metric covers all 3,792 outputs per method; incomplete Q3O scores are omitted. Source-score extraction is separate from this kit.
338
 
assets/figure1.pdf ADDED
Binary file (31.4 kB). View file
 
assets/figure1.png CHANGED

Git LFS Details

  • SHA256: c52f18c1aaad89b6411db96ad2625f9fd3547d08577fe071940f4ca5aac798e5
  • Pointer size: 131 Bytes
  • Size of remote file: 147 kB

Git LFS Details

  • SHA256: 0f7e859bfee94eb456a1c38650eb6474568721356349ef490ac4a7e8bc49c68f
  • Pointer size: 131 Bytes
  • Size of remote file: 172 kB
assets/figure1.svg ADDED