UIBK-Embedding-v1: A Dense and a Late-Interaction Retriever in One Model, One Index and One Search

We are releasing UIBK-Embedding-v1, an open English retrieval model with 179M parameters. It holds a dense retriever and a late-interaction (ColBERT) retriever in one encoder and adds their scores inside one standard MaxSim search. It reaches 59.79 nDCG@10 on BEIR, the best score among the models under 1B parameters with complete official MTEB results, and 70.55 on NanoBEIR. It is ahead of LateOn on all 15 BEIR datasets and stays ahead on the decontaminated BEIR. It loads as a normal PyLate ColBERT model and runs in a PLAID index with no change to the index or the scorer. The weights are available under Apache 2.0.

Model: DataScience-UIBK/uibk-embedding-v1

1. Introduction

Retrieval models come in two families. Dense bi-encoders turn a text into a single vector and compare a query with a document by one cosine. Late-interaction models, the ColBERT family, keep one vector per token and compare with MaxSim: every query token looks for its best-matching document token, and the matches are summed.

The two families are usually presented as a choice. In practice they are good at different things. One vector captures what a text is about, which is what counts when a long query has to be matched as a whole. Token-level matching finds the exact entity or phrase, which is what counts for factoid and multi-hop questions.

People use both today through hybrid retrieval: two models, two indexes, two searches, and a fusion step at the end. Models such as BGE-M3 return several representations from one encoder, which saves the second model, but each representation still has its own index and its own scoring function.

UIBK-Embedding-v1 puts both retrievers in one model and both scores in one search. The dense embedding is written into the list of token vectors, and a one-dimensional tag keeps the two kinds of vectors apart, so that the unmodified MaxSim of any ColBERT index returns the token-level score plus a weighted dense score.

The result is a small model that sits well above its size class. With 179M parameters it reaches 59.79 on BEIR. Among the 156 models with complete BEIR results in the official MTEB results repository, only nine score higher, and they have between 1.5B and 7.6B parameters, at least eight times more.

1.1 Four ways to retrieve

(a) A dense bi-encoder stores one vector per text. (b) A late-interaction model stores one vector per token. (c) Hybrid retrieval runs both and fuses their scores. (d) UIBK-Embedding computes both scores with one encoder, one index and one MaxSim.

1.2 Training pipeline

The model is trained in three stages on open data, with an open cross-encoder as the teacher:

1. Contrastive pre-training of ModernBERT-base on a 331M-pair sample of LightOn's public pre-training collection.
2. Two single-head models fine-tuned from this backbone: a dense model and a ColBERT model.
3. The unified model: both heads on one encoder, trained with contrastive losses, cross-encoder distillation into each head and into the fused score, and feature anchors to the two single-head models.

LateOn and DenseOn were trained without knowledge distillation. We add it, and we apply it to the score that is actually used at inference. Sections 4 and 5 give the details.

2. Two Families That Fail in Different Places

Before building one model, we checked that the two signals really are complementary. We fine-tuned a dense model and a ColBERT model from the same backbone on the same data and compared them on the 13 NanoBEIR datasets.

Each bar is the difference of their nDCG@10. Orange: the dense model is better. Blue: the late-interaction model is better.

The two models win on different datasets. The dense model is 14.7 points better on ArguAna, where the query is a whole argument, and 7.7 points better on SCIDOCS. The late-interaction model is 5.0 points better on NQ and 4.7 points better on MS MARCO, where a single entity or phrase decides. Neither family is simply better. That is the reason to want both scores, and the reason a hybrid pipeline is worth its cost today.

3. How It Works

A text is encoded once. The output is an ordinary list of 128-dimensional unit vectors, exactly what a ColBERT index expects.

3.1 One trunk, two tops

The backbone is ModernBERT-base with 22 layers. The embeddings and the first 18 layers form a trunk that both heads share. Each head then has its own copy of the last four layers: the token top is the original one, the dense top is a copy that is trained separately. The two heads can specialise in their last layers and still share most of the computation.

The second top adds 20.1M parameters to the 149.0M of the backbone. One pass costs 26 layer evaluations instead of 22.

3.2 Token head and dense head

The token head is the usual ColBERT projection: a linear layer maps every token state to 128 dimensions.

The dense head uses the latent-attention pooling of NV-Embed, here on an encoder-only model. Every token state of the dense top attends to 512 trainable latent vectors, a feed-forward layer follows, and the mean over the tokens is the sentence vector with 768 dimensions. In a controlled comparison with the same data, steps and seed, this pooling was 0.8 NanoBEIR points better than [CLS] pooling.

3.3 Slots: a 768-dimensional vector in a 128-dimensional index

A late-interaction index stores 128-dimensional vectors, and the sentence vector has 768. We project it with a fixed random matrix with orthonormal rows to 381 = 3 × 127 dimensions, cut the result into three slices and normalise each one. These three vectors are the slots. The sum of the three slice cosines of two texts approximates the cosine of their sentence vectors.

The slots are written over the first three positions of every text, which hold the [CLS] token, the query or document marker and the prompt word. The list is therefore not longer than that of a normal ColBERT model.

3.4 The tag: one dimension that keeps the two kinds apart

If slots and tokens simply sat in the same list, a query token could match a document slot and the two signals would mix. To prevent this, every vector gets one more coordinate, the tag, with opposite signs for tokens and slots:

token vector    [ +sqrt(0.1)   ; sqrt(0.9)   * u_t ]     u_t: the projected token state, 127-d
slot vector s   [ -sqrt(0.775) ; sqrt(0.225) * g_s ]     g_s: slice s of the projected sentence vector, 127-d

token x token   =  0.1   + 0.9   * cos(u, u')
slot  x slot    =  0.775 + 0.225 * cos(g_s, g'_s)
token x slot    = -0.28  + 0.45  * cos(u, g_s)

All vectors still have unit norm. A pair of the same kind gets a positive constant, a mixed pair a negative one. Three properties follow:

• A slot never selects a token. A slot-slot product is at least 0.55 and a slot-token product is at most 0.17, for every input.
• A token never selects a slot as soon as its best token match has a cosine above 0.08.
• A slot selects its own slot, because the three slots hold different coordinates of the sentence vector.

In practice the separation is complete. On NanoBEIR, over all pairs of a query vector and a document, no query token selects a slot and no slot selects a token.

3.5 One MaxSim, two scores

Put the tagged vectors into the standard MaxSim and the sum splits by itself:

MaxSim(q, d)  =  C(q)  +  0.9 * [ token MaxSim  +  0.75 * mean slot cosine ]
                                  late interaction        dense

C(q) depends only on the query, so the ranking is that of the bracket: a late-interaction score plus 0.75 times a dense score. No second index, no second search, no fusion step.

The weight 0.75 is a property of the stored vectors. It can be chosen when the model is packaged, without any training.

3.6 What this means in practice

• The model is saved as a ColBERT model and runs in PLAID through PyLate as it is.
• Queries are read up to 256 tokens. The dense head reads all of them; the token head uses the first 64.
• Documents are read up to 2048 tokens.
• The prompts query: and document: are required.

4. Training

4.1 Three stages

1. Pre-training. ModernBERT-base with latent-attention pooling, contrastive loss with in-batch negatives (temperature 0.02), 33,000 steps with a batch of 16,384 pairs, on a 331M-pair sample of LightOn's curated pre-training collection. This takes about 75 hours on four H100 GPUs.
2. Single-head models. From this backbone we fine-tune a dense model (the weight average of four runs) and a ColBERT model. They are the two models of Section 2.
3. UIBK-Embedding-v1. The two encoders of stage 2 are fine-tunes of one backbone, so a hybrid model can be initialised from their weight average. It is trained with both heads for three epochs. Its last four layers are then duplicated into the two tops, and the two-top model is trained for one final epoch: 16,597 steps, a batch of 128 queries with 14 hard negatives each, learning rate 7e-5 with 5% warm-up, queries up to 256 and documents up to 512 tokens. The final epoch takes 4.3 hours on four H100 GPUs.

4.2 The loss

For a query with its positive and 14 mined hard negatives, the model produces a token-level score, a dense score and the fused score of Section 3.5. The teacher is the public cross-encoder mxbai-rerank-large-v2. The loss is the sum of five terms:

L  =  NCE(token score)  +  NCE(dense score)
    + KL(teacher || dense score)  +  KL(teacher || fused score)
    + 10 * anchor

• NCE is the contrastive loss over all documents of the batch (temperature 0.02), one per head.
• KL compares the model's distribution over the candidates of a query with the teacher's.
• anchor keeps the vectors of each head close to those of the corresponding single-head model of stage 2.

The term KL(teacher || fused score) is the one that matters most. It is computed on the score that ranks documents at inference, and it is the only term in which the two heads appear together. The model is not trained to have two good heads; it is trained to have a good sum.

5. Training Data

Everything is built from public data.

Supervised collection. LightOn's public English fine-tuning collection: 1.56M queries from MS MARCO, NQ, HotpotQA, FEVER, FiQA, SQuAD v2, TriviaQA and MIRACL, each with mined negatives and cross-encoder scores.

Re-mined negatives. Hard negatives are most useful when they are hard for the model being trained. For every query we retrieve five more candidates with our own dense model, score them with the cross-encoder and keep the four hardest whose score is at least one logit below that of the positive. This adds 3.59M negatives, so every query has 14 negatives with teacher scores.

Auxiliary pairs. The supervised collection is made of questions over Wikipedia and web passages. To cover duplicate questions, community forums and scientific literature as well, we mine 412k pairs from the pre-training collection.

Long queries. The supervised queries are short questions, but a query can also be a whole paragraph, as in argument retrieval. We add 152k pairs whose queries are long. None of them comes from the sources of ArguAna or Touché.

Overlap with BEIR. Rows whose query equals a BEIR test or development query were removed from the training data: 37,391 rows of the pre-training sample and 228 rows of the fine-tuning collection. For the mined pairs, a pair was removed if its query or its document equals a BEIR query.

6. Results

All BEIR and NanoBEIR numbers of UIBK-Embedding-v1 come from MTEB with its standard PyLate wrapper and a PLAID index with default settings. Nothing in the evaluation is specific to our model. The numbers of the other models are taken from the official MTEB results repository. Best in bold.

6.1 BEIR

nDCG@10 on the 15 BEIR datasets, for the models with at most 250M parameters.

UIBK-Embedding-v1 reaches 59.79, against 57.22 for LateOn, the best other model of its size class, 56.20 for DenseOn and 56.06 for jina-embeddings-v5-text-nano. It is also ahead of larger models of up to 600M parameters, such as mLateOn (57.56, 307M), jina-embeddings-v5-text-small (56.67, 596M) and Qwen3-Embedding-0.6B (55.52). The table with the larger models is in the appendix.

On the ten retrieval tasks of MTEB(eng, v2) it reaches 62.47, the best of the 45 models with at most 250M parameters that have all ten tasks.

6.2 Ahead of LateOn and DenseOn

LateOn and DenseOn are the strongest late-interaction and dense models of this size. UIBK-Embedding-v1 is ahead of LateOn on all 15 datasets and ahead of DenseOn on 13 of 15 (it is 0.1 behind on FiQA and 0.2 behind on NFCorpus).

The gains are largest on Touché (+9.3 over LateOn), ArguAna (+6.0), Climate-FEVER (+5.2), NQ (+3.9) and DBPedia (+3.2). These are datasets of both kinds: ArguAna and Touché are the long-query, argument-style datasets where dense models lead, NQ and DBPedia are entity-centred datasets where late interaction leads. The model does not trade one family's strength for the other's.

Not an effect of document length. LateOn's reported numbers use documents of up to 300 tokens, and our model reads up to 2048. With 2048-token documents LateOn reaches 57.69 in our evaluation, so the gap at equal document length is still 2.10 points. For a fair reading, note also that our model has 30M more parameters than LateOn (179M against 149M).

6.3 NanoBEIR

NanoBEIR consists of 50-query samples of 13 BEIR datasets. UIBK-Embedding-v1 reaches 70.55, second among the 20 models with official results. Only inf-retriever-v1, with 7.1B parameters, scores higher (72.04).

LateOn has no official NanoBEIR results. With exact MaxSim search it scores 67.50 in our evaluation, against 70.43 for UIBK-Embedding-v1.

6.4 Decontaminated BEIR

A high BEIR score can come from overlap between training data and the benchmark. LightOn built a decontaminated BEIR to test this: 12 BEIR datasets from which the documents and queries that overlap with common training collections were removed. We evaluate on it with PyLate and a PLAID index, as MTEB does.

UIBK-Embedding-v1 reaches 61.55, against 60.36 for LateOn as published, and is ahead on 8 of the 12 datasets. When we run LateOn ourselves under the same protocol it reaches 59.69; the difference from its published number comes almost entirely from ArguAna. The gain over LateOn is therefore between 1.2 and 1.9 points on data the models cannot have seen.

6.5 Compared with multi-output encoders

BGE-M3 and jina-embeddings-v4 also return dense and multi-vector outputs from one model. The difference is where the scores are combined: they combine them outside the index, we combine them inside one MaxSim.

We evaluated BGE-M3 (568M parameters) on NanoBEIR in each of its scoring modes, following its own inference code and with exact search over the whole corpus.

Combining its dense and multi-vector scores gives BGE-M3 62.91, and adding its sparse score 64.19. UIBK-Embedding-v1 reaches 70.55 with one search in one index, with less than a third of the parameters, and is ahead of every mode of BGE-M3 on every dataset.

For jina-embeddings-v4 (3.8B parameters) the official MTEB results give 56.15 on the ten retrieval tasks of MTEB(eng, v2), for its single-vector output, against 62.47 for our model.

6.6 Beyond BEIR: LoTTE and LIMIT

LoTTE has five domains, each with search queries and forum queries. On its test split UIBK-Embedding-v1 reaches a success@5 of 83.77, the highest among the models with official MTEB results. LoTTE is drawn from StackExchange, which is part of the public pre-training collection used by our model and by the LightOn models.

LIMIT was built to show what a single vector cannot represent. On LIMIT small our model reaches a recall@2 of 91.4, above GTE-ModernColBERT (83.5) and 37 points above the best single-vector model, which has 8B parameters. The lexical baseline BM25 reaches 97.8 on this benchmark.

6.7 Cost

One model instead of two also means one pass and one index. We measured cost on FiQA (57,638 documents, one H100 GPU) with the same input lengths for every model: queries of 32 and documents of 300 tokens.

Encoding is 9% slower than LateOn, which is the price of the four additional layers of the dense top, and 1.7 times faster than the two passes of a LateOn and DenseOn hybrid. The index has the size of LateOn's and is 22% smaller than the two indexes of the hybrid. Late-interaction models use a PLAID index with default settings; DenseOn uses a flat index that is searched exhaustively.

7. What We Learned

Left: each head alone and both together (NanoBEIR, exact MaxSim). Middle: the weight of the dense score. Right: the learning rate of the last stage.

7.1 Both heads are needed

In every setting we tried, the fused score is above the better of the two heads. For the released model, exact MaxSim gives 69.00 for the dense head alone, 69.81 for the token head alone and 70.43 for both. The reason is that each head alone falls far behind on some datasets: the token head reaches 51.66 on ArguAna against 67.39 for the dense head, and the dense head reaches 34.26 on NFCorpus against 40.38 for the token head.

7.2 Two tops are better than one

Giving each head its own copy of the last four layers raises each head on its own by about one NanoBEIR point: from 65.56 to 66.42 for the dense head and from 66.67 to 67.68 for the token head. The cost is 20.1M parameters.

7.3 Distil the score you evaluate

Adding the term KL(teacher || fused score) to the losses of the two heads, with the data and the schedule unchanged, raises the BEIR average from 58.61 to 59.08. The gain is largest on Climate-FEVER (+1.6), ArguAna (+1.1) and NQ (+1.0), and with this term the model is ahead of LateOn on all 15 datasets.

7.4 The weight of the dense score is forgiving

The fused score is flat for a dense weight between 0.75 and 1 for every checkpoint we tested, so the exact value does not need tuning. We use 0.75, which also leaves a wide margin in the guarantees of Section 3.4.

7.5 Pooling matters

Latent-attention pooling was 0.8 NanoBEIR points better than [CLS] pooling in a controlled comparison in which the two training-loss curves coincide. LightOn's blog also reports that the pooling method matters for a dense model; our comparison points the same way.

7.6 The learning rate of the last stage matters

A learning rate of 7e-5 in the last stage is 0.54 BEIR points above 1e-5. Two seeds of one configuration differ by about 0.2 points, which is the resolution of our comparisons.

7.7 Longer documents at inference

The model is trained with documents of up to 512 tokens and reads longer ones at inference. A two-top model scores 57.61 on BEIR with 512 tokens, 58.14 with 2048 and 58.04 with 8192, so the gain is complete at 2048 tokens.

8. How to Use It

The model is used with PyLate. Two things differ from an ordinary ColBERT model: it needs trust_remote_code=True, because two small modules in the repository define the two-top encoder and the output layer, and the prompts must be passed when encoding.

pip install -U pylate "transformers>=5.3" "sentence-transformers>=5.3"

Quick start

from pylate import models, rank

model = models.ColBERT("DataScience-UIBK/uibk-embedding-v1", trust_remote_code=True)

query = "Which planet is known as the Red Planet?"
documents = [
    "Venus is often called Earth's twin because of its similar size and proximity.",
    "Mars, known for its reddish appearance, is often referred to as the Red Planet.",
    "Jupiter, the largest planet in our solar system, has a prominent red spot.",
    "Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",
]

queries_embeddings = model.encode([query], is_query=True, prompt_name="query")
documents_embeddings = model.encode(documents, is_query=False, prompt_name="document")

# MaxSim late-interaction scoring (higher is more relevant)
ranking = rank.rerank(
    documents_ids=[[1, 2, 3, 4]],
    queries_embeddings=queries_embeddings,
    documents_embeddings=[documents_embeddings],
)
print([(d["id"], round(d["score"], 2)) for d in ranking[0]])

Index and search

from pylate import indexes, models, retrieve

model = models.ColBERT("DataScience-UIBK/uibk-embedding-v1", trust_remote_code=True)

# A standard PLAID index: nothing in it is specific to this model
index = indexes.PLAID(index_folder="pylate-index", index_name="index", override=True)

documents_ids = ["1", "2", "3"]
documents = ["document 1 text", "document 2 text", "document 3 text"]
documents_embeddings = model.encode(documents, is_query=False, prompt_name="document")
index.add_documents(documents_ids=documents_ids, documents_embeddings=documents_embeddings)

retriever = retrieve.ColBERT(index=index)
queries_embeddings = model.encode(["query for document 3"], is_query=True, prompt_name="query")
scores = retriever.retrieve(queries_embeddings=queries_embeddings, k=10)

For large collections of long documents, pass batch_size="auto" to indexes.PLAID(...).

9. Limitations

• The model is English only. The construction does not depend on the language, but we have not trained a multilingual model.
• We study one backbone size, ModernBERT-base, and retrieval only.
• Like every late-interaction model, it stores one vector per token, so its index is larger than that of a single-vector model. The three slots are written over existing positions and do not lengthen it.
• The guarantees of Section 3.4 are statements about exact inner products, and we evaluate with PLAID. Indexes that transform the vectors in other ways are untested.
• It needs trust_remote_code=True, transformers >= 5.3 and the two prompts.

10. Conclusion

UIBK-Embedding-v1 shows that a dense retriever and a late-interaction retriever do not need two models, two indexes and a fusion step. Three slots and a one-dimensional tag are enough for a standard MaxSim to add the two scores, and distilling a cross-encoder into that sum turns two heads into one retriever.

With 179M parameters the model reaches 59.79 on BEIR, 2.57 points above LateOn and ahead of it on all 15 datasets; only models with at least 1.5B parameters score higher. It stays ahead on the decontaminated BEIR (61.55), and reaches 70.55 on NanoBEIR, 83.77 on LoTTE and a recall@2 of 91.4 on LIMIT small.

Get started

• Model: DataScience-UIBK/uibk-embedding-v1
• Library: PyLate and FastPLAID
• Leaderboards: BEIR and NanoBEIR on MTEB

Citation

soon

Acknowledgements

This model stands on open work. We thank LightOn for PyLate, FastPLAID, the public pre-training and fine-tuning collections, the decontaminated BEIR and LateOn, the reference that set the bar; Answer.AI and LightOn for ModernBERT; Mixedbread for the cross-encoder used as the teacher; the authors of NV-Embed for latent-attention pooling; and the teams behind Sentence Transformers, MTEB and BEIR.

Appendix

A. BEIR with larger models

The models of Section 6.1 with larger models for reference. Bold: best among the models with at most 250M parameters.

B. LoTTE per domain

Success@5 of UIBK-Embedding-v1 on the test split of LoTTE.

C. LIMIT small

Recall@2 of neural retrievers. The numbers of the other models are those of the LIMIT paper.

D. Settings

E. How the guarantees depend on the weight

The weight of the dense score can be changed without training. The table gives the worst-case inner products of Section 3.4 for other weights. A slot is guaranteed never to select a token as long as the smallest slot-slot product is above the largest slot-token product, which holds up to a weight of 1.

References

1. Khattab and Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. 2020.
2. Santhanam et al. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. 2022.
3. Santhanam et al. PLAID: An Efficient Engine for Late Interaction Retrieval. 2022.
4. Warner et al. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder (ModernBERT). 2024.
5. Lee et al. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. 2024.
6. Sourty et al. DenseOn with the LateOn: Open State-of-the-Art Single and Multi-Vector Models. 2026.
7. Chaffin and Sourty. PyLate: Flexible Training and Retrieval for Late Interaction Models. 2025.
8. Chen et al. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (BGE-M3). 2024.
9. Günther et al. jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval. 2025.
10. Thakur et al. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. 2021.
11. Muennighoff et al. MTEB: Massive Text Embedding Benchmark. 2023.
12. Zeta Alpha. NanoBEIR. 2024.
13. Weller et al. On the Theoretical Limitations of Embedding-Based Retrieval (LIMIT). 2025.
14. Moreira et al. NV-Retriever: Improving Text Embedding Models with Effective Hard-Negative Mining. 2024.
15. Mixedbread. mxbai-rerank-large-v2. 2025.
16. Wortsman et al. Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy without Increasing Inference Time. 2022.