Too much of a good thing: Why 20 candidates beat 200 when reranking search results

Get hands-on with Elasticsearch: Dive into our sample notebooks in the Elasticsearch Labs repo, start a free cloud trial, or try Elastic on your local machine now.

We recently tested  jina-reranker-v3.5 across 25 retrieval benchmarks to find out what candidate pool size provides the highest accuracy vs token cost. Reranking the top 100 candidates from Elasticsearch BM25 cut error by about 21% on average, and halving that pool to 50 cost roughly 2% accuracy for half the tokens. Semantic search with jina-embeddings-v5-text-small peaked much earlier, at 20 candidates and an 8.5% error reduction. Feeding it 200 scored worse on average than running no reranker at all. More candidates is not better. The worse your initial retrieval, the more of them a reranker needs, and the more it has left to fix.

What is a reranker?

Reranker AI models take queries and candidate matches as input and independently produce new rankings, ordering the candidates from best to least matching.

In theory, you could use them by themselves for retrieval by giving them your entire database to rank. But using a reranker that way is prohibitively expensive, both in computing resources and in token costs via an API. It amounts to having an AI model read your database every time you query it. In practice, rerankers are almost always used in conjunction with some more efficient retrieval process, such as embeddings, that selects candidate matches for a reranker to then reevaluate.

Rerankers come in broadly three types: pointwise, pairwise, and listwise. The difference is how many candidates you can feed them at once, how many times you run the model, and whether you get a reliable ordering out.

Table 1: A comparison of reranker types.

Pointwise rerankers are models that take as input a query and a single candidate match and produce a score. Multiple retrieval candidates are then sorted by their scores. They let you evaluate any number of candidates but require you to run the entire model for each candidate. This kind of reranker is best used when there is a large set of candidates to evaluate, like when there is no high-quality first-stage retrieval, or scenarios like federated search, where candidates come from multiple sources that don’t reveal their scores or ranks to you. When you can do a good job of restricting the candidate pool to a small number of high-probability best candidates, it’s a relatively expensive solution compared to the alternatives.

jina-colbert-v2 is an example of a pointwise reranker. It takes one query and document pair and returns a score, although it’s most efficiently used with batches of documents and returning scores for each so that the query doesn’t have to be re-encoded for every document independently. 

Pairwise rerankers take a query and just two candidates, and they tell you which of the two is a better match. You can use them to rerank any number of candidates by comparing them two at a time, like an elimination tournament. But this means that if you have N candidate matches, you need to run the model N-1 times, almost as many as for a pointwise reranker. Furthermore, they don’t guarantee a consistent ordering. If Candidate A is better than Candidate B, and Candidate B is better than Candidate C, according to the reranker, it’s possible that Candidate C will be ranked better than Candidate A.

PairReranker is an example of a pairwise reranker designed to rank candidate outputs from text generation models.

Listwise rerankers take a query and a collection of candidates and produce an ordering on them, from best match to worst. They only let you process as many candidates as you can fit into the model’s input, and they’re typically somewhat larger models than pointwise ones. However, you only need to invoke the model once to evaluate an entire candidate pool.

Listwise reranking generally outperforms pointwise and pairwise reranking in head-to-head comparisons, but it can only work with a small number of candidate matches. In cases with large numbers of candidates, or where the relatively large resource requirements of listwise rerankers pose a problem, pointwise and pairwise rerankers may be better solutions. 

In this article, we use jina-reranker-v3.5, the latest listwise reranker from Elastic | Jina AI, to look at two variables:

• How is listwise reranker output influenced by the quality of the initial retrieval step and the quality of the candidate pool?

• How many candidate matches does jina-reranker-v3.5 need to return optimal matches?

It turns out that these two variables aren’t entirely independent of each other.

How reranking improves Elasticsearch BM25 search

BM25 is the classic text information retrieval algorithm. It doesn’t use AI models, but it’s very fast, has low computational demands, and is built into Elasticsearch. It remains in very widespread use, despite generally superior performance from recent AI-driven search, in part because it’s very cheap and fast, and in part because this over-30-year-old algorithm has trained many users to expect search systems to perform the way that BM25 does.

We used a collection of standard text retrieval benchmarks, drawn from four widely used benchmark suites, to evaluate the performance of jina-reranker-v3.5 in conjunction with retrieval candidates found using BM25:

• Benchmarking-IR (BEIR) is the classic English text information retrieval benchmark, widely used and containing a variety of materials from different domains. We used a subset that, we admit, excluded the largest datasets because it would have taken a lot more time to run tests on them.

• CQADupStack is a collection of question-and-answer datasets, primarily relating to information technologies, drawn from forums on the Stack Exchange website.

• Code Information Retrieval Benchmark (CoIR) is a collection of code retrieval datasets. We only used three datasets from this collection, all of which are relatively small and involve natural language queries or documents.

• BRIGHT is a dataset designed to be very difficult for conventional retrieval algorithms. Its authors created it to test the reasoning abilities of text retrieval systems, ensuring that lexical retrieval schemes, like BM25, and embeddings-based semantic retrieval would struggle to find matches. Most conventional retrieval systems, including semantic search, score poorly on BRIGHT. We selected these datasets specifically to see whether jina-reranker-v3.5 adds value under extremely difficult conditions.

The table below shows the retrieval scores, measured at nDCG@10, for each dataset, using Elasticsearch’s BM25 implementation by itself (Elasticsearch Serverless on Elastic Cloud, English analyzer, default BM25 settings) and with the reranker at different candidate sizes: 10, 20, 50, 100, and 200. Our BM25 baseline was configured to retrieve only 200 candidates, so we didn’t test with larger candidate lists. 

Table 2: BM25 search with jina-reranker-v3.5 and varying candidate pool sizes.

Reranking with jina-reranker-v3.5 can improve on the base BM25 score for every benchmark, but for listwise rerankers, it depends on the candidate pool size. If the candidate pool is too large, adding the reranker can sometimes make things worse than using no reranker at all. In Table 2, you can see this happen with the  LeetCode and Webis-Touche2020 benchmarks.

If we had used pointwise or pairwise reranking, this wouldn’t be possible. They produce the same ordering no matter how many candidates they’re given. If Candidate A is better than Candidate B, adding Candidate C would never change that. But listwise rerankers look at the entire list of candidates at once, and,when given too much information, they suffer from attention dilution and noise accumulation. AI isn’t magic. Attention-based models have finite resources, and they perform more poorly when those resources are stretched thinner.

We see here that giving jina-reranker-v3.5 200 candidates produced worse outcomes than giving it just the top 100. Only the three CoIR benchmarks have higher scores at 200 than for smaller candidate pools. In some cases, using only 50 candidates scores better than 100. For two benchmarks, scores peaked at 20. 

The variation in optimal sizes highlights a major consideration in deploying AI, in any scenario; that is, benchmarks are only very indirectly indicative of performance on your data, in your pipeline, and it’s important to track the performance of the AI models you choose for your use case and data.

However, if we look at how much jina-reranker-v3.5 reduced error across different benchmarks, we do get a rough picture of what kinds of results we can expect:

Table 3: Error reduction for BM25 across candidate pool sizes.

We can see that in this scenario, using Elasticsearch BM25, performance peaks around 100 candidates, but that using 50 candidates only slightly increases the error (circa 2% using nDCG@10). Halving the number of candidates sent to the reranker reduces the token count by half, cutting the cost by 50% and reducing latency. Only two of the 25 benchmarks had a more than 10% increase in error when we shrunk the candidate pool from 100 to 50: BEIR TREC-COVID and CoIR StackOverflow-QA. In both cases, this was because the error rate was already very low at 50 candidates. 

These tests suggest that your best bargain, using jina-reranker-v3.5 with Elasticsearch’s BM25 implementation, is to use only 50 candidates.

How reranking performs with semantic vector search

But what if we have a better initial search scheme? Do these findings hold for semantic search with AI-generated embeddings?

TL;DR: No.

We ran the same tests using jina-embeddings-v5-text-small embeddings and vector search instead of BM25. First, semantic search using this model produces considerably better results than BM25, reducing error on average by 25.7% across the chosen benchmarks. On one benchmark, anomalously, Jina’s semantic search does more poorly, but not by a large margin:

However, using jina-reranker-v3.5 at 100 candidates raises BM25’s scores to become competitive with Jina’s best semantic retrieval:

Overall, jina-embeddings-v5-text-small performs better on these retrieval benchmarks, but the margin is much smaller. On average, across these benchmarks, jina-embeddings-v5-text-small reduces error by 4.2%, compared to using BM25 in conjunction with jina-reranker-v3.5. On half of the BEIR benchmarks, BM25 with reranking actually beats semantic search.

But if a good listwise reranker can bring lower-quality retrieval close to matching frontier semantic embeddings-based retrieval, how much more can it add when we use the best retrieval first? 

We repeated our tests using semantic search with jina-embeddings-v5-text-small, with the same candidate pool sizes: 10, 20, 50, 100, and 200. The results follow:

* CoSQA scores vary by about 0.01 between runs. Table 4: Semantic vector search using jina-embeddings-v5-text-small, combined with jina-reranker-v3.5 and varying candidate pool sizes.

Better first-stage retrieval changes the optimal candidate pool size. On average, it’s considerably smaller, and optimal candidate pool sizes are much more spread out over different benchmarks. Using 200 candidates actually made results worse on average than not using a reranker at all. For two benchmarks, BRIGHT LeetCode and CoIR CoSQA, reranking scored below the retriever alone at every pool size.

Table 5: Error reduction for semantic vector search across candidate pool sizes.

Reranking using 20 candidates maximized performance, when used with jina-embeddings-v5-text-small, in contrast to 100 with BM25. This indicates that using better strategies to get the initial candidate pool reduces the cost of using a reranker afterward. However, using a better initial search system also reduces the added value of the reranker. Reranking the best 100 candidates found by BM25 reduced error on average by over 20%. Reranking the best 20 candidates from jina-embeddings-v5-text-small (the best performing candidate pool size) only reduced error by approximately 8.5% on average. 

If we compare BM25 with jina-reranker-v3.5 to jina-embeddings-v5-text-small with jina-reranker-v3.5, using semantic search and an intelligent reranker in combination scores higher on every benchmark.

However, the difference is smaller than you might expect. Using jina-embeddings-v5-text-small with jina-reranker-v3.5 reduced error by 13.26% on average, compared to BM25 with jina-reranker-v3.5. This is roughly half the error reduction we get comparing jina-embeddings-v5-text-small to BM25 without using a reranker.

What matters when tuning a listwise reranker

In this article, we use one high-performance listwise reranker. Some of the calculations here might not apply equally to other rerankers. However, listwise rerankers in general will pose similar problems to the ones described here. We haven’t been scientifically rigorous here, but we can distill from these results some general principles for deploying listwise rerankers:

How many candidates should you send to a reranker?

There are significant differences in outcomes, depending on how many candidates you send to the reranker. We only tested strategies involving fixed numbers. Depending on what information you retrieve from the initial search, you might be able to discern how many candidates are worth sending to the reranker. BM25 and semantic embeddings-based search both return scores with their results. There may be some policy, like a score threshold, that would improve reranker performance. We didn’t test this possibility.

But we do find definitively that you can give too many candidates to the reranker. More is not always better. It’s often worse.

The better the initial search, the less a reranker adds

This result was one we expected. When a retrieval system makes fewer mistakes, there’s less room for reranking to improve it. But now, we have some empirical evidence to back that up. We also show that a good reranker can improve enough on a relatively poor initial search to make it competitive with high-performance semantic search. There are many practical use cases where this is an important result. BM25, in particular, is very cheap, efficient, fast, and deeply entrenched in information retrieval, and Elasticsearch contains an enhanced BM25 implementation that’s already a very strong retrieval system. It may be easier to add jina-reranker-v3.5 to a BM25 pipeline than to reengineer the entire retrieval process around semantic search.

The worse the initial search, the larger the candidate pool needs to be

Two data points aren't definitive, but the gap between optimal candidate pool sizes in our tests (100 for BM25 versus 20 for jina-embeddings-v5-text-small) is suggestive. It means that any savings you might make in compute, latency, or cost by using BM25 with a reranker over using semantic search may be partly offset by the increased cost of having to pass a much larger candidate pool to your reranker.

How much do reranking results vary between datasets?

The impact of reranking and the optimal candidate pool sizes can vary a lot for different benchmarks, which means it can vary at least as much for different real-world use cases. We encourage you to test with your own data and try to optimize your pipeline yourself, because no selection of benchmarks is identical to your pipeline. Both jina-reranker-v3.5 and jina-embeddings-v5-text-small are available for Elasticsearch through Elastic Inference Service (EIS). You can sign up for a free Elastic Cloud instance for 14 days to try it yourself.

How to add jina-reranker-v3.5 to your Elasticsearch pipeline

You can test all these findings on your own data without hosting any models yourself. Using Elastic Cloud means that Elastic hosts everything and manages the entire search pipeline and AI stack.

If you already use BM25 search in Elasticsearch, you can add jina-reranker-v3.5 without reindexing anything: 

1. Create an Elastic Inference API endpoint for the reranker.

2. Add a text_similarity_reranker retriever to your existing queries. 

3. Set rank_window_size to 100. This parameter controls how many candidates go to the reranker, which is the variable we tested here. 

4. Rerun with rank_window_size set to 50, and compare the results. In our tests, the smaller pool cost very little accuracy for half the tokens.

If you use semantic vector search, the setup changes and so does the starting point:

1. Index your data with jina-embeddings-v5-text-small, using the semantic_text field.

2. Add a text_similarity_reranker retriever with jina-reranker-v3.5. 

3. Set rank_window_size to 20, not 100. Better initial retrieval needs a much smaller candidate pool, and for some data, the best size will be smaller still. 

4. Don't assume that more candidates help. We found that reranking 200 semantic search candidates did worse on average than not reranking at all.

See the Elasticsearch documentation on text reranking for more information about using jina-reranker-v3.5 with your existing Elastic search pipeline.

Both jina-embeddings-v5-text-small and jina-reranker-v3.5 are also available through the Jina API, with free tokens to try them out, and as Jina on-prem containers that run on your own hardware. The weights for both are on Hugging Face for testing and research under a noncommercial license. For commercial use, contact Elastic sales.

The best candidate pool size for your pipeline depends on your data, so measure it rather than trusting any default, including ours.