music-recsys-2026-retriever-4b

Fine-tuned Qwen/Qwen3-Embedding-4B, a two-tower (Siamese) conversation→track bi-encoder for the RecSys 2026 Music Conversational Recommendation Challenge. It maps a conversation and a track into the same 2560-dim space so the right track for a turn is the nearest catalog vector.

Part of our challenge submission — full pipeline, training code, and reproduction instructions: github.com/npatta01/music-conversational-music-recomender-2026 (architecture doc: docs/architectures/biencoder.md).

Used two ways in the pipeline:

1. dense.b1 retrieval branch — top-k ANN over the 47k-track catalog (a candidate source)
2. b1_cos reranker feature — cosine(conversation vector, candidate vector), fed into the LightGBM reranker judge

The deployed config uses it as a reranker feature only (b1_cos); the standalone retrieval branch added negligible union recall for the cost of a second ANN call.

Architecture

Single shared encoder (same fine-tuned weights on both towers) — the only asymmetry is an INSTRUCT prefix on the query side.

            QUERY TOWER (live, per turn)                 DOC TOWER (precomputed, 47k tracks)
        conversation turns (raw text)                     track metadata (catalog row)
        prev user turn / now / prev played track          artist, title, year, tags, "known for" line
                       │                                               │
     "[prev] <u_t-1> [now] <u_t> [prev_track] <artist — title>"   "Music track: <artist> — <title>
                       │                                          (<year>) | tags: a,b,c | known for: …"
                       │ + INSTRUCT prefix                            │  (doc_nokf = drop known-for line)
                       ▼                                               ▼
             Qwen3-Embedding-4B (shared, bf16), last-token pool (left-padded), L2-normalize
                       │                                               │
                 q_vec ∈ ℝ²⁵⁶⁰ (‖q‖=1)                          d_vec ∈ ℝ²⁵⁶⁰ (‖d‖=1)
                       └───────────────────────┬───────────────────────┘
                                                ▼
                        score = cos(q_vec, d_vec) ∈ [-1, 1]

Input / output

	Query tower	Doc tower
Raw input	conversation up to turn t (raw user turns + last played track)	one catalog track row
Rendered text	[prev] <user turn t-1> [now] <user turn t> [prev_track] <artist — title> (goal-free)	Music track: <artist> — <title> (<year>) | tags: <≤5 cleaned> | known for: <LLM-generated line>
Prefix	Instruct: Given a music recommendation conversation, retrieve relevant track metadata passages that match the listener request and prior music preferences.\nQuery:	none
Output	q_vec — 2560-dim unit vector	d_vec — 2560-dim unit vector

Goal-free by design: the query renderer emits no conversation_goal / goal_progress / thoughts fields (unavailable at test time on Blind-B), so the model trains on exactly the signal it sees at serve time.

Training recipe

• Base: Qwen/Qwen3-Embedding-4B, full fine-tune (whole encoder), bf16, gradient checkpointing
• Objective: MNRL / in-batch InfoNCE — logits = (Q @ D.T) * scale (scale=20), loss = cross_entropy(logits, diagonal), each query's positive must beat every other in-batch doc plus n_hardneg=4 appended hard negatives
• Positives: MOVES-only — a (turn, track) pair is positive iff its goal-progress label is MOVES_TOWARD_GOAL, with an off-by-one label correction (assessment[t+1] grades track[t]). 53,885 training pairs.
• Known-for field dropout: 30% of the time the doc tower sees the doc text without the "known for" line, so the model doesn't over-rely on a field that tail artists lack.
• Batch size 64, 1 epoch, last-token pooling, max_len=2048
• Training code: scripts/rerank/train_biencoder.py / scripts/rerank/modal_train_biencoder.py

Evaluation (devset, corrected labels)

As a standalone retriever (r@k over the 47k catalog):

r@20	r@100	r@1000	median rank
0.327	0.540	0.829	72

Bimodal by conversation type — strong on continuation (r@20 0.58) and turn_1 (r@20 0.36), weak on hard_pivot (r@20 0.10, a hard DJ-pivot ceiling shared by all retrievers in this pipeline).

As a reranker feature (b1_cos, held-out OOF nDCG@20):

	OOF full	OOF lockbox
prod judge (with artist-consensus crutch)	0.1970	0.2023
+ b1_cos	0.2032 (+0.0061)	0.2116 (+0.0093)
crutch removed, no b1_cos	0.1901 (−0.0070)	0.1972
crutch removed, + b1_cos	0.1991 (+0.0091)	0.2107

b1_cos's marginal value is larger once the artist-consensus crutch is removed — it's a learned, generalizable replacement for that crutch, recovering most of the prod-level nDCG without it.

Usage

This checkpoint is a plain transformers AutoModel (not a sentence-transformers model — pooling and normalization are manual, matching mcrs/embeddings/qwen3_embedding.py in the source repo):

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

MODEL = "Npatta01/music-recsys-2026-retriever-4b"
tok = AutoTokenizer.from_pretrained(MODEL, padding_side="left")  # left-pad required for last-token pooling
model = AutoModel.from_pretrained(MODEL, dtype=torch.bfloat16).eval()

def encode(texts: list[str]) -> torch.Tensor:
    batch = tok(texts, padding=True, truncation=True, max_length=2048, return_tensors="pt")
    with torch.inference_mode():
        out = model(**batch)
        last = out.last_hidden_state[:, -1]          # last-token pool (left-padded)
        return F.normalize(last, p=2, dim=1)          # L2-normalize -> cosine == dot product

QUERY_INSTRUCT = ("Instruct: Given a music recommendation conversation, retrieve relevant "
                   "track metadata passages that match the listener request and prior music "
                   "preferences.\nQuery: ")

q = encode([QUERY_INSTRUCT + "[prev] looking for something upbeat [now] more like that last one "
            "[prev_track] Daft Punk — One More Time"])
d = encode(["Music track: Justice — D.A.N.C.E. (2007) | tags: french house, funk | "
            "known for: a playful homage to Michael Jackson-era pop"])

score = (q @ d.T).item()  # cosine similarity in [-1, 1]

Doc-side vectors for the full challenge catalog are precomputed once and served from a vector store (LanceDB column b1_vstructpt_4b) — there's no need to re-encode the catalog at query time in the reference implementation.

Provenance

• fp32 source-of-truth (Modal scout-models volume): sha256 c78eb09b2ab012f4fb8f3b32b95fea18eed50de81a5e9e088099a30bc61308ef
• bf16 weights (this repo, model.safetensors): 8.04 GB, sha256 79aa4d26be96a117d749c6f067d31306b21a1e34fb1525a5aa9246537fe4c9a9
• Promoted 2026-06-25 (fp32 → bf16 cast, ~halves size; retrieval quality unaffected at this precision)

License

Base model Qwen/Qwen3-Embedding-4B is Apache-2.0; this fine-tune is released under the same license.