Sync Suno v6 WildSongBench results and frontier figure
Browse filesUse competitive with Suno v5/v6; refresh the September 12 comparison and matching figure. Keep inference, weights, prompts and evaluation code unchanged.
- README.md +7 -3
- assets/figure1.pdf +0 -0
- assets/figure1.png +2 -2
- assets/figure1.svg +0 -0
README.md
CHANGED
|
@@ -52,14 +52,16 @@ tags:
|
|
| 52 |
<a href="https://huggingface.co/m-a-p/SheetSage2"><img alt="SheetSage2" src="https://img.shields.io/badge/SheetSage2-374151?logo=huggingface&logoColor=FFD21E" height="20" /></a>
|
| 53 |
</p>
|
| 54 |
|
| 55 |
-
**YuE2 is an open music generation model
|
| 56 |
|
| 57 |
-
**State-of-the-art results on WildSongBench.** YuE2 (best-of-8) achieves the highest SongBench average among all evaluated open and proprietary models: **6.9632**, compared with **6.8721** for Suno v5.
|
| 58 |
|
| 59 |

|
| 60 |
|
| 61 |
*Frontier song quality and text alignment on 192 WildSongBench prompts. YuE2 uses symbolic planning; Bo8 means best-of-8.*
|
| 62 |
|
|
|
|
|
|
|
| 63 |
- **Compose and edit:** melody + chords, melody-only, or direct generation; bring your own ABC score.
|
| 64 |
- **Edit with an agent:** turn musical feedback into score, style and lyric revisions, then let YuE2 render the next version. [Hear the editing process](https://map-yue2.github.io/#agentic-music-editing).
|
| 65 |
- **Run locally:** 48 kHz stereo songs on a 24GB GPU, without quantization.
|
|
@@ -304,6 +306,8 @@ Server: 32 songs per row; AR/NAR use PyTorch 2.10 and Triton 3.6, VAE uses PyTor
|
|
| 304 |
| Suno v5 | 5.9918 | 6.8721 | **0.5428** | **0.4353** | 4.5907 | 8.10% |
|
| 305 |
| Suno v4.5 | 5.8317 | 6.6995 | 0.5022 | 0.3873 | 4.4149 | **5.80%** |
|
| 306 |
| Suno v5.5 | 5.8087 | 6.7150 | 0.5089 | 0.3917 | 4.5914 | 5.96% |
|
|
|
|
|
|
|
| 307 |
| MiniMax Music 2.6 | 5.4437 | 6.3222 | 0.4251 | 0.3670 | 4.5688 | 24.55% |
|
| 308 |
| Mureka 9 | 6.0488 | 6.9377 | 0.4394 | 0.4102 | 4.6368 | 11.69% |
|
| 309 |
| **YuE2** | 5.9075 | 6.7316 | 0.5068 | 0.4054 | 4.6819 | 8.44% |
|
|
@@ -328,7 +332,7 @@ Server: 32 songs per row; AR/NAR use PyTorch 2.10 and Triton 3.6, VAE uses PyTor
|
|
| 328 |
<details>
|
| 329 |
<summary>📐 Evaluation protocols and metric definitions</summary>
|
| 330 |
|
| 331 |
-
WSB: SongBench Avg averages seven dimensions; Q3O measures prompt adherence on a 0–5 scale; PER is phoneme error rate. YuE2 selects the lower-PER candidate from two. Best-of-8 selects by Musicality → Q3O → PER. Each candidate's PER uses the lowest-PER of four ASR passes. A pipeline call generates one candidate; selection is separate. Q3O weights differ on 10 of 192 prompts between the two YuE2 settings. Bold marks the best value within each table. Open baselines use two candidates and four ASR passes; proprietary systems retain their delivered-candidate protocols. MiniMax Music 3 uses its official caption rewriter. SongBloom uses a fixed audio prompt rather than a style-text input.
|
| 332 |
|
| 333 |
SHS100K: CLEWS and Discogs-VINet measure preserved song identity against 10,545 recordings after source exclusion. MuLan measures target-style similarity; Musicality is from SongBench. Identity and quality should be read together. Every displayed metric covers all 3,792 outputs per method; incomplete Q3O scores are omitted. Source-score extraction is separate from this kit.
|
| 334 |
|
|
|
|
| 52 |
<a href="https://huggingface.co/m-a-p/SheetSage2"><img alt="SheetSage2" src="https://img.shields.io/badge/SheetSage2-374151?logo=huggingface&logoColor=FFD21E" height="20" /></a>
|
| 53 |
</p>
|
| 54 |
|
| 55 |
+
**YuE2 is an open music generation model with frontier song quality competitive with Suno v5/v6.** Turn lyrics and a style prompt into a complete song with vocals and accompaniment, then shape its melody and chords through an editable score.
|
| 56 |
|
| 57 |
+
**State-of-the-art results on WildSongBench.** YuE2 (best-of-8) achieves the highest SongBench average among all evaluated open and proprietary models: **6.9632**, compared with **6.8721** for Suno v5, **6.5562** for Suno v6, and **6.4195** for Suno v6 Wild.
|
| 58 |
|
| 59 |

|
| 60 |
|
| 61 |
*Frontier song quality and text alignment on 192 WildSongBench prompts. YuE2 uses symbolic planning; Bo8 means best-of-8.*
|
| 62 |
|
| 63 |
+
September 12, 2026 comparison: 17 settings, including Suno v6 and Suno v6 Wild. [Vector PDF](assets/figure1.pdf) · [SVG](assets/figure1.svg) · [All nine metrics and protocols](https://huggingface.co/datasets/m-a-p/WildSongBench/blob/main/benchmark/results.md) · [Full-precision CSV](https://huggingface.co/datasets/m-a-p/WildSongBench/resolve/main/benchmark/benchmark-results.csv).
|
| 64 |
+
|
| 65 |
- **Compose and edit:** melody + chords, melody-only, or direct generation; bring your own ABC score.
|
| 66 |
- **Edit with an agent:** turn musical feedback into score, style and lyric revisions, then let YuE2 render the next version. [Hear the editing process](https://map-yue2.github.io/#agentic-music-editing).
|
| 67 |
- **Run locally:** 48 kHz stereo songs on a 24GB GPU, without quantization.
|
|
|
|
| 306 |
| Suno v5 | 5.9918 | 6.8721 | **0.5428** | **0.4353** | 4.5907 | 8.10% |
|
| 307 |
| Suno v4.5 | 5.8317 | 6.6995 | 0.5022 | 0.3873 | 4.4149 | **5.80%** |
|
| 308 |
| Suno v5.5 | 5.8087 | 6.7150 | 0.5089 | 0.3917 | 4.5914 | 5.96% |
|
| 309 |
+
| Suno v6 | 5.6558 | 6.5562 | 0.4916 | 0.4305 | 4.6258 | 7.58% |
|
| 310 |
+
| Suno v6 Wild | 5.5644 | 6.4195 | 0.4999 | 0.4316 | 4.5898 | 7.45% |
|
| 311 |
| MiniMax Music 2.6 | 5.4437 | 6.3222 | 0.4251 | 0.3670 | 4.5688 | 24.55% |
|
| 312 |
| Mureka 9 | 6.0488 | 6.9377 | 0.4394 | 0.4102 | 4.6368 | 11.69% |
|
| 313 |
| **YuE2** | 5.9075 | 6.7316 | 0.5068 | 0.4054 | 4.6819 | 8.44% |
|
|
|
|
| 332 |
<details>
|
| 333 |
<summary>📐 Evaluation protocols and metric definitions</summary>
|
| 334 |
|
| 335 |
+
WSB: SongBench Avg averages seven dimensions; Q3O measures prompt adherence on a 0–5 scale; PER is phoneme error rate. YuE2 selects the lower-PER candidate from two. Best-of-8 selects by Musicality → Q3O → PER. Each candidate's PER uses the lowest-PER of four ASR passes. A pipeline call generates one candidate; selection is separate. Q3O weights differ on 10 of 192 prompts between the two YuE2 settings. Bold marks the best value within each table. Open baselines, Suno v6, and Suno v6 Wild use two candidates and four ASR passes per candidate, followed by lower-PER selection; earlier proprietary systems retain their delivered-candidate protocols. MiniMax Music 3 uses its official caption rewriter. SongBloom uses a fixed audio prompt rather than a style-text input.
|
| 336 |
|
| 337 |
SHS100K: CLEWS and Discogs-VINet measure preserved song identity against 10,545 recordings after source exclusion. MuLan measures target-style similarity; Musicality is from SongBench. Identity and quality should be read together. Every displayed metric covers all 3,792 outputs per method; incomplete Q3O scores are omitted. Source-score extraction is separate from this kit.
|
| 338 |
|
assets/figure1.pdf
ADDED
|
Binary file (31.4 kB). View file
|
|
|
assets/figure1.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/figure1.svg
ADDED
|
|