• Geometry-preserving Embeddings: jina-embeddings-v5-omni sits atop the foundation laid by jina-embeddings-v5-text. That original text embedding space is completely unchanged, with its geometry left wholly intact.

• Locked: Synonymous with "frozen." All of the towers in this architecture have their weights locked. 

• Aligned: Aligning the other modalities with the text model's vector space, allowing for cross-modal comparison.

• TOwers: Modality component that converts one type of raw input into vectors.

The model comes in two variants: small and nano. The former has more parameters (1.57 billion) than the latter (0.95 billion), but functionally their architectures are nearly identical. For the sake of brevity, we mostly focus on jina-embeddings-v5-omni-small in this article.

What are vectors, towers and frozen encoders?

jina-embeddings-v5-omni relies on three core machine learning (ML) concepts: vectors, towers, and frozen weights. Here's what each means; feel free to skip ahead if you're already familiar with them. 

How vectors represent data in embedding models

How does AI understand abstract concepts? Can a machine comprehend what "ice cream" is? Does it understand that "chocolate fudge" and "rocky road" have more in common with each other than "sorbet"? The answer, surprisingly, is yes. The mechanism that makes it possible is the vector.

Take a look at this diagram.

If you tried visualizing a way to organize all ice cream flavors, you may end up with something like this: a simplex (triangle) with three vertices, each representing a base flavor. Each flavor is closer to or farther from each vertex, depending on how much of the corresponding base makes up that particular flavor. Chocolate ice cream is all chocolate, so it hugs the top vertex. Vanilla has a similar affinity for the bottom-left vertex. But cookies and cream is roughly a 50/50 mix, so it's about equidistant from both. Neapolitan blends all three, so it sits at the center of the triangle.

This is functionally how vectors work. These flavors get funneled into an ML model (more specifically known as an embedding model) that will then generate coordinates for each of these flavors along this simplex. By measuring the distance between coordinates, software can parse how related or unrelated two flavors are. It's easy to see that "chocolate" and "chocolate brownie fudge" are closely related flavors because they sit close to one another, but "strawberry" and "cookies and cream" are far away from each other, so we can infer that they aren’t similar.

It won't be quite this simple though. Rather than three labeled points, real embeddings run along hundreds or even thousands of dimensions. Nor will they have nice, human-readable labels; the vertex markers are something that only the model understands. The benefit, though, is that we can graph basically anything like this.

To clarify some jargon: These coordinates = vectors = embeddings. For the rest of this article, we use these terms interchangeably.

Much like how embeddings go by many different names, so too do the models that create them.

What is a tower in multimodal embedding models?

The term tower comes from Contrastive Language-Image Pre-training (CLIP), a model released by OpenAI in 2021 that was one of the first to learn a shared embedding space across text and images. In CLIP, each modality is handled by a completely separate model. On an architecture schematic, these models look like towers standing side by side, each taking one type of input and producing vectors in a shared space. The name stuck, and you'll see it used broadly across multimodal ML. 

In the context of jina-embeddings-v5-omni, the word is used a bit more loosely. Its architecture doesn't have true parallel towers in the CLIP sense. All modalities ultimately funnel into a single central text model, rather than sitting as equals beside it. 

With that caveat in place: A tower (or modality component) is a pipeline that converts one type of raw input into vectors. A text tower vectorizes strings, and a vision tower generates image embeddings. An audio tower does the same for sound.

Why freeze a tower instead of training it?

If we want multimodal capabilities, could we Frankenstein multiple towers that handle each of those inputs together into one model? The issue with this approach is that vectors from different models aren’t intelligible to each other. Images will exist in one vector space and audio in another, for example. This means that we have no ability to compare across different modalities. Think of the vectors outputted from one model as existing in their own language. Let's say our audio tower outputs Spanish and our image tower outputs English. Conceptually, the vectors can be describing the same things, but downstream tools that try to make use of these embeddings are "monolingual," so we're out of luck.

The CLIP-style approach is to take several towers and train them together, letting them all reshape each other until their outputs agree. It's sort of like having Spanish and English speakers try to communicate for long enough that they eventually all start speaking Spanglish.

This approach works, but it has a side effect: Towers that you already had working get remodeled in the process. Any embeddings that they produced before are now incompatible with any that are produced by the older version of the tower. If you had a text tower and generated 100 million embeddings with it, you would now need to re-embed all of those strings.

To combat this issue, you can freeze certain towers. This locks their weights, which are the knobs and dials that influence how they behave. This way, training never changes them. What you train instead is a small projector that translates one tower's output into another tower's language. Many different models train some portion of projectors and towers while leaving some others frozen. What makes jina-embeddings-v5-omniunique is that we froze every encoder and trained only the projectors and delimiter tokens. By the end of this article, you’ll understand exactly how that works.

You can think of these frozen and trainable components as clusters of neurons or isolated regions of the mind, and the whole embedding model (jina-embeddings-v5-omni) as the entire brain. Between these learning-capable components sit fixed math operations, such as merging, squashing, selecting, and rescaling numbers on their way from one tower to the next. They have no weights, so there’s nothing in them to freeze or train. 

With that groundwork laid, here’s the full architecture.

How the jina embeddings multimodal architecture works

The architecture routes all modalities through frozen encoders and small trainable projectors into a single shared text embedding space.

Why the text tower is the backbone

The first thing to notice here is the general flow of data in this diagram. Image, video, and audio inputs all (eventually) end up in the same space as text inputs. Why is the architecture set up like this?

jina-embeddings-v5-omni builds on top of jina-embeddings-v5-text: It retains its text-processing backbone and extends it with pretrained vision and audio components.

This has two main advantages:

1. The text model is already state of the art for its size and what it does, and it stays completely untouched. There’s no need to fix what isn't broken. We also don't need to re-embed anything we already embedded with jina-embeddings-v5-text.

2. A single shared vector space is what makes cross-modal search work at all. Because image, audio, and text all resolve to vectors in the same geometry, you can query an image with text, or audio with text, and compare them directly with cosine similarity (a similarity measure based on the angle between two vectors). If each modality lived in its own separate space, those comparisons would be meaningless.

How vision and audio encoders feed into the text model

The first step in building upon the foundation set by jina-embeddings-v5-text is integrating vision and audio encoders into this architecture. In this case, we use the existing Qwen3.5 vision encoders and the Qwen2.5-Omni audio encoder, which themselves have been adapted from SigLIP2 and Whisper-large-v3, respectively. They’re ultimately what’s responsible for generating raw vectors for all vision- and audio-based data. The emphasis is on raw here, since much transformation still needs to be done afterward.

How the vision encoder processes images

In the case of images, we’re borrowing more from Qwen than just the encoder. Additional plumbing inherited from Qwen is attached to the output of the encoder. Let's walk through what comes out of the encoder and how the inherited downstream components transform that output.

Vision encoder (frozen)

Let's use an image as our primary example, since the visual component of video is basically identical. Take this image of a banana split.

Rather than generating one single, clean vector embedding for this image, the vision encoder breaks the image up into 14-pixel by 14-pixel sections and generates a tiny patch token (basically a mini-vector) for each of these sections.

While still inside of the vision encoder, each token looks at every other token that makes up the image and pulls in information from the ones relevant to it, updating its own vector based on that context. This process allows us to preserve fine-grained details. Now, instead of one single vector for the whole image, we have multiple, smaller vectors that represent particulars of the whole dish.

LayerNorm (frozen)

LayerNorm rescales each patch's numbers so they sit in a consistent range before anything else touches them. It stops some patches from being wildly larger than others and drowning out the rest.

2x2 merge 

We saw in the vision encoder section that we split up the image into small, 14-pixel by 14-pixel squares. However, processing this many patch tokens will become expensive downstream. For this reason, the 2x2 merge operation consolidates four patch tokens into one. The squares are now 28 pixels by 28 pixels. The corresponding patch tokens are similarly consolidated.

fc_vision_1 (frozen) 

fc_vision_1 is the first of two matrix multiplies; it mixes the merged patch numbers into a new set. This layer is inherited from Qwen and left as is.

GELU 

Gaussian Error Linear Unit (GELU) is a gate applied to each number. It lets useful signals through and squashes the rest toward zero. It’s the one nonlinear step and is ultimately what lets the two fc_vision layers together learn shapes that a single flat multiply couldn't.

fc_vision_2 (trainable) 

As mentioned earlier, trying to put Qwen image embeddings directly into the same vector space as Jina text embeddings would be like trying to include a Spanish sentence in an English novel. Outside of its native context, its meaning is totally lost.

That is what the trainable projector fc_vision_2 is here to fix. It learns to translate the image embedding into something that jina-embeddings-v5-text can understand. For this reason, you can think of a trainable projector as a translator.

The emphasis belongs on trainable: This is the first component we’ve encountered in our walkthrough so far that isn’t frozen. Many components of this architecture are frozen, meaning that their weights are locked and never change during training, but fc_vision_2 is one of the few parts that actually gets updated, because it has to learn how to translate Qwen's image dimensions into a form that lands meaningfully inside Jina's text space.

× 4

By now, you may have noticed that fc_vision_2 has a "× 4" marked on the bottom, along with fc_audio, both encoders, and the ″Embedding text″ section within jina-embeddings-v5-text. In this case, it represents four different instances of fc_vision_2, each optimized and tuned to one of four slightly different tasks, outlined below.

Task

What it facilitates

Example user input

Example end result

Note

Retrieval

Finds a similar match for the input (that is, standard Google search)

"melting ice cream" as a string/text

Picture of a fallen ice cream cone on asphalt

Text-matching

Judges how similar inputs are

A text string "melting ice cream" and an image of a fallen ice cream cone on asphalt

Score judging how similar the two inputs are

The name of this task is a bit of a misnomer. It's called text-matching, but it works for any modality, not only text.

Clustering

Groups data into clusters

A large array of ice cream images

Lets the user discover natural groups, like "sundaes" and "popsicles"

Classification

Places data into predefined buckets

Two string labels: "melting" and "intact", along with a large array of ice cream images

Sorts the array of ice cream images into the two provided categories, based on their proximity in vector space to the label embeddings

To clarify, the "Example end result" column is the takeaway after some additional math and processing happens once the output vector is generated. The point is that  jina-embeddings-v5-omni only generates vectors. Those vectors take on a mildly different form to optimize for the selected task type.

These task types are also explicitly outlined in the Low-Rank Adaptation (LoRA) component of the architecture diagram, which we'll cover in a moment.

How the audio encoder processes sound

Before we go any deeper into the trenches of our model architecture, let's back up and see how the audio-oriented path differs and how it stays the same. If you were able to follow along during the vision section, the audio portion will be a breeze. We have no extra inherited plumbing from Qwen this time, only the audio encoder and one trainable projector.

Audio encoder (frozen)

Before audio can enter the encoder, it needs to be converted into a form that the encoder can work with. Raw audio is a one-dimensional wave, which isn't particularly useful to a neural network on its own. Instead, the audio is first transformed into a mel spectrogram: a 2D representation that maps frequency against time, weighted to emphasize the frequency ranges that the human ear is most sensitive to. Think of it as a visual fingerprint of the sound. Below is a mel spectrogram of a person saying, "I scream, you scream, we all scream for ice cream."

That spectrogram is then sliced into fixed-length 40ms chunks, analogous to how the vision encoder breaks an image into 14×14 pixel-tiles. The encoder then produces one token per chunk.

This matters for the same reason it does in the vision pipeline: Fine-grained detail would otherwise be lost. A single embedding for the full phrase "I scream, you scream, we all scream for ice cream" would smear everything into one blurry vector. Slicing it into short chunks keeps each fragment of sound intact, so the model can later distinguish "scream" from "cream" rather than collapsing them into an average. The same applies to music, animal noises, environmental sounds, and more.

fc_audio (trainable)

Once the encoder has produced its tokens, fc_audio performs the same translation role that fc_vision_2 does for images: It projects each audio token from the encoder's 1280-dimensional output space into jina-embeddings-v5-text’s hidden dimension (1024 for small, 768 for nano). Like fc_vision_2, it carries a "× 4" in the architecture diagram, meaning that there are four instances, each optimized for a specific task type (retrieval, text-matching, clustering, and classification).

Delimiters (trainable)

We need to convey to jina-embeddings-v5-text that this embedding represents an unexpected modality. We can do this pretty easily with delimiters. In the world of HTML, this looks like:

The first and last <p> tag conveys that everything in between them is a paragraph element.

In our case, things are a bit more complicated. For audio, the architecture diagram shows the delimiters as <aud_start> and <aud_end>, but that isn't quite accurate. The delimiters themselves are actually vectors rather than hard-coded strings.

Each task type has its own pair of delimiter vectors (hence the "4 × special tokens"). These vectors are identical every time rather than being different for each audio embedding. So a task type of retrieval with an audio input type always gets start delimiter vector X and end delimiter vector Y; text-match with audio input always gets a start delimiter vector A and an end delimiter vector B, and so on.

This is necessary because the Transformer layers component only understands vectors. So, by the time the audio embedding makes its way there, it looks like:

The same pattern applies to images:

Video is where this gets interesting. Up to 32 evenly spaced-out frames are pulled from the video and fed into the vision encoder. 

Rather than producing one delimiter-wrapped segment, each sampled frame gets its own <vis_start> / <vis_end> wrapper, and these per-frame segments are concatenated into one long sequence:

This is what makes multi-frame video work: Rather than averaging frames or treating them separately, the transformer receives the whole video as one token stream, allowing its attention to relate tokens across frames. This means that earlier frames can inform how later ones are interpreted. If the video does have an audio component, it's pulled out and fed into the audio encoder and ultimately prepended to the frame sequence.

The transformer layers component then processes this entire concatenated sequence as a single input.

Jina text transformer layers (frozen) 

We’ve generated patch tokens for our images, videos, and audio files. We’ve also wrapped them inside of vector delimiters, all for the sake of having them understood by these layers. They’ll allow each patch token to examine the other patch tokens and determine whether they need to update themselves based on the surrounding context. I know what you're thinking:

Didn't we already do this inside of the encoder? We split up the image into 14-pixel by 14-pixel sections and generated patch tokens for each section, and then the encoder updated each patch token based on surrounding context within the same image.

And you're right! We did. But there's a key difference now.

Originally, that recontextualization ran on the attention of Qwen's vision encoder. The operation within the Transformer layers runs the frozen Jina text transformer's attention. It’s the same operation with different learned parameters, so it transforms the tokens differently.

Think of it like a move: The projector is the flight and the moving trucks. It physically relocates you from vision land to text land, landing you in the right city and even the right neighborhood. The transformer's attention is what happens after you've unpacked. You're already home; you spend the next few weeks figuring out exactly where you fit, meeting the neighbors, finding your bearings, and adjusting your exact spot based on who's actually around you. You did the macro move already. This is the micro fine-tuning.

Lastly, the <vis_end_vector_delimiter> will absorb all the information from the patch tokens it wraps.

LoRA (frozen) 

LoRA is a way of fine-tuning an existing model without completely retraining it. It’s a small set of extra adjustment knobs bolted onto the transformer that nudges its behavior to optimize for one of the specific tasks (such as retrieval or classification).

Last-token pooling 

Since <vis_end_vector_delimiter> absorbed all the other patch tokens into itself, we don't need to consider anything except it, so we throw the rest away. It acts as a stand-alone embedding that represents a summary of the whole.

L2 normalization 

<vis_end_vector_delimiter> could be any length now, which is no good. This step shrinks or stretches it so its length is exactly 1, without changing the direction it points. This is tidying so that comparing it to other vectors later is a fair, clean angle comparison. It changes only the vector’s scale, not what it means.

Enough about ice cream

As we've journeyed our way through jina-embeddings-v5-omni, we’ve been careful to outline which components are frozen and which ones are trainable. By now, you may have noticed that every single tower has been frozen. In fact, only the small projectors (translators) and delimiter tokens have been trainable. We dubbed this architecture pattern GELATO. This makes the entire training process significantly cheaper.

To be clear, we didn't invent the concept of frozen towers. Prior work on Locked-image Tuning (LiT), VISTA, and Multi-modAl Retrieval model via Visual modulE pLugin (MARVEL) froze one side or the other. What no one had done before GELATO was push the idea to its limit: text, image, video, and audio all in one model with every encoder frozen. The only trained pieces are a single projector layer per modality and a handful of delimiter tokens.

But is it any good?

Benchmark results: jina embeddings vs. other multimodal embedding models

There's no point in building out a model and releasing it if you don't even know if it's any good, especially compared to the competition. That's why we have benchmarks and evaluation frameworks. The benchmarks that jina-embeddings-v5-omni was run against are Massive Image Embedding Benchmark (MIEB), Massive Audio Embedding Benchmark (MAEB), Massive Multimodal Embedding Benchmark–Video (MMEB-Video), and Massive Multilingual Text Embedding Benchmark (MMTEB).

As for the models we compare against, it's important not to make apples and oranges comparisons. For that reason, we’re specifically using open-weight omni-style models with support for the same media types:

• LanguageBind

• Omni-Embed-Nemotron-3B

• LCO-Embedding-Omni-3B

• LCO-Embedding-Omni-7B

Evaluation

The table above is sorted by parameter count. Models with the fewest parameters are clustered at the top, and models with the most parameters hug the bottom. This context is important because performance alone isn't the final variable. If a $100 ice cream with gold flakes and the finest dairy milk tastes the same as (or worse than) the average gallon of ice cream that you can buy from your local grocery store, it would be silly to buy it, because you're paying a massive premium for nothing.

A similar situation is unfolding here. jina-embeddings-v5-omni-small and nano outperform every other model on text, despite ranking low to middle in terms of parameter count.

Audio performance is strong, as well. jina-embeddings-v5-omni-small and nano beat out all other models except those from LCO, which they both trail by about 2 to 3 points.

The gap shrinks when considering image performance, particularly with jina-embeddings-v5-omni-small. It beats both LanguageBind and Omni-Embed-Nemotron-3B. It lags less than a point behind both LCO models, despite the fact that they have 4.70 billion and 8.93 billion, respectively, compared to Jina’s 1.57 billion.

Video is the weakest performer for our models, though even in that case jina-embeddings-v5-omni-small still beats Omni-Embed-Nemotron-3B, which has three times as many parameters. Ultimately, when these scores are averaged out, you get the following rankings:

Model

Number of parameters (B)

Average score

LCO-Embedding-Omni-7B

8.93

54.43

jina-embeddings-v5-omni-small

1.57

54.04

LCO-Embedding-Omni-3B

4.70

53.83

jina-embeddings-v5-omni-nano

0.95

47.49

Omni-Embed-Nemotron-3B

4.70

41.21

LanguageBind

1.14

35.82

LCO-Embedding-Omni-7B has nearly six times the number of parameters as jina-embeddings-v5-omni-small but barely squeaks past it in average performance.

One benchmark deserves a special callout for anyone building search or retrieval augmented generation (RAG): visual document retrieval, measured on the ViDoRe benchmark. 

Here, jina-embeddings-v5-omni-small scores 79.25 using only 0.92 billion active text-and-image parameters, ahead of LCO-Embedding-Omni-3B (78.24) and within striking distance of LCO-Embedding-Omni-7B (80.32), a model nearly 10 times its size on that path. nano matches that exact same 79.25 score with just 0.31 billion active parameters. The larger Omni-Embed-Nemotron-3B does take the top spot at 85.64, but it carries roughly five times the active parameters of jina-embeddings-v5-omni-small, so our models remain the most parameter-efficient of the group. If your workload is retrieving pages of documents by their layout and text, this is the number to weigh.

Limitations of frozen-tower multimodal embeddings

GELATO's frozen-tower design delivers strong results at low training cost, but it comes with trade-offs worth naming plainly. As already mentioned, the most consistent weak spot is video. jina-embeddings-v5-omni-small trails the LCO models on video, and moment retrieval (locating a specific event within a clip) is the weakest subtask of all. This is partly structural, since each frame produces its own token set before everything is concatenated and pooled into a single final embedding. Packing that much information into one embedding means that the early dimensions carry a heavier load, so video embeddings degrade faster than image embeddings when truncated to smaller sizes.

Audio has its own gap. While retrieval and classification scores are competitive, audio clustering is the weakest audio subtask (6.13 for jina-embeddings-v5-omni-small).

Cross-modal audio–text retrieval trails LCO-Omni-7B by 11–15 percentage points, a larger gap than the 6–7 points seen on the image–text (I-T) pair. The fc_audio projector is the natural next target for additional trainable parameters, suggesting the audio–text (A-T) alignment path has more room to grow than the multilayer vision pipeline. 

How multimodal embeddings distribute in vector space

We've already discussed performance via benchmarks, but what about how the actual embeddings are distributed in vector space? How does that tangibly differ from model to model?

In the above illustration, data from video clips is funneled into each model and graphed in vector space. It’s then compressed down to two dimensions via the Uniform Manifold Approximation and Projection (UMAP) method for easy visualization. Each modality corresponds to a different component of the video:

Modality

Component of the video

Image

Frame from the middle of the video

Video

The full video

Audio

Audio track from the video

Text

Description of the video

Immediately, some interesting patterns stand out.

Our models and the LCO models seem to have different modalities all mixed together, while LanguageBind and Omni-Embed-Nemotron-3B seem to lean more toward having their embeddings separated by modality.

Our models and the LCO models exhibit interleaved geometry for these vectors. This means that different modalities aren't clearly separated in vector space, but instead intermingle in similar areas. This is less pronounced with Omni-Embed-Nemotron-3B, since only image and video seem to occupy a similar space.

LanguageBind is fully separated, with different modalities occupying entirely different spaces. This is known as the modality-gap pattern.

So which one is better? In practice, interleaved geometry tends to be the more useful of the two, and it’s worth noticing that the strongest models in our benchmarks (ours and LCO's) all exhibit it. However, there are trade-offs.

Interleaved geometry excels at cross-modal retrieval, since everything is jumbled up together in the vector space and, therefore, much closer. It's easier to find a matching picture for the text "strawberry ice cream" when the text and image embeddings sit so close together in vector space.

When you're trying to do a same-modality task though, the image that was so conveniently within reach is now in the way. However, in practice, this is easily mitigated by metadata filtering on something like a “modality” field. 

No such workaround exists for the issues inherent to models that exhibit the modality-gap pattern. It’s easy to find another video of syrup poured on ice cream, since all the videos are sitting together in isolation. But having modalities confined into clusters like that makes finding an accompanying image much harder.

Why this architecture?

GELATO gives a lot of performance for very little training. You keep all your towers as they are and train only small projectors and delimiter tokens, which is significantly cheaper than the alternatives. To put concrete numbers on "cheaper": for jina-embeddings-v5-omni-small, training just the vision projector updates 4.20 million parameters instead of the 920.6 million a full fine-tune would touch. At the same 15,000-step budget, that projector-only run finishes about 1.8 times faster and peaks at 7.52 GiB of GPU memory instead of 12.96 GiB. The audio path shows an even wider gap, with projector-only training running 3.2 to 3.9 times faster than full training. But how did we conclude this was the way to go? We used a process known as ablation.

Ablation is when you remove or change one piece of a system to see how much it actually mattered. Imagine you've been working on an ice cream recipe. Every time you make a tweak, like doubling the milk, swapping brown sugar for white, using vanilla beans instead of extract, or taking out the chocolate chunks, that's ablation.

Ablation in ML functions much in the same way. It asks whether removing, rearranging, freezing, or unfreezing certain components makes the whole system more, less, or equally as performant. In this case, we’re particularly interested in whether unfreezing certain components, and in what order, may affect performance. We conducted five ablation studies on the Qwen3.5 vision stack. The results are measured in mean nDCG@10 (normalized Discounted Cumulative Gain), a standard score for ranking quality where higher is better.

Overall, nearly every ablation study yielded basically identical results, except for case #3, which performed terribly. Before we explain why, let's look at the equivalent ablation diagram for audio.

In this instance, ablation case #2 performs the worst. Do you see the commonality between the worst performer here and the worst performer among the vision ablations? Across both modalities, the same rule holds: If you unfreeze the encoder before the projector has been trained, you’ll see worse performance.

For both modalities, we ultimately chose ablation case #1 for the final architecture. Both had relatively high scores. In vision's case, the configurations that edged out case #1 did so by margins too small to justify their added training stages and extra per-task artifacts. A similar story unfolds for audio, with case #3 beating out case #1 by a small margin but requiring more per-task artifacts.

Ablation validates the GELATO approach: It's cheaper and nearly identical in quality to train a dedicated translator (rather than the speaker).

Summary: why frozen encoders make multimodal embeddings cheaper

Rather than expensively retraining multiple towers to achieve multimodal capabilities, GELATO allows us to minimize cost by freezing our already functioning towers and training only small projectors to translate embeddings. These embeddings get funneled into jina-embeddings-v5-text, ultimately allowing all the output vectors to exist in the same, interleaved geometry. We can now compare text, audio, images, and video at a fraction of the cost of the competition. 

Both jina-embeddings-v5-omni-small and jina-embeddings-v5-omni-nano are open-weight for personal use and available now. You can download them from the Jina AI collection on Hugging Face and start generating multimodal embeddings today, or read the full technical report for the complete set of benchmarks and ablations.

The palette takes shape: how multimodal search in Elasticsearch evolved from semantic_text

The semantic field is a convergence of several complementary features we've introduced over the past couple of years, bringing them together to create a cohesive multimodal search experience. Each solved an important piece of the semantic search puzzle on its own; together they enable native multimodal search.

The first brushstroke was semantic_text. Before it, running semantic search meant manually configuring mappings, wiring up ingest pipelines with an ML model, manually chunking content, and generating query-time embeddings yourself. The semantic_text field folds all of that away: it performs inference automatically at ingest time, chunks long documents for you, and simplifies the queries you write against it. Introduced in Elasticsearch 8.15 and released as GA in Elasticsearch 8.18, it has become the foundation for semantic search on the platform.

Next came the model to power multimodal search. jina-embeddings-v5-omni is our family of multimodal embedding models, capable of embedding text, images, video, audio, and PDFs into a shared vector space. Because those embeddings are semantically compatible across modalities, you can store diverse media in a single index and query across all of it at once, such as retrieving an image via a text description or matching audio against a written phrase, all without maintaining a separate pipeline for each content type. For more detailed information about how these embeddings are generated, see the model documentation.

We added the embedding query vector builder in Elasticsearch 9.4 to handle multimodal inputs at query time. Query vector builders are general-purpose tools you can use to convert input to a vector at query time as part of your request. For example, we have the text_embedding query vector builder for text-only models and input, and the lookup query vector builder for getting a vector from an existing document. The embedding query vector builder is a new type that works with multimodal models and accepts multimodal input, including text or base64-encoded binaries. This allows you to pose a query in whatever modality fits, and Elasticsearch generates the matching vector on the fly.

The final piece was multimodal ingest. The semantic_text field brought automatic embedding to text; the semantic field extends that same automatic experience to images, audio, video, and PDFs from ingest through query.

Painting the picture: creating an index with the semantic field

Let’s create an index with a semantic field. This is as simple as setting the field type to semantic and defining the inference endpoint you want to use:

In this example, we use the .jina-embeddings-v5-omni-small inference endpoint. This is our built-in jina-embeddings-v5-omni inference service, and it is available in all environments with access to the Elastic Inference Service (EIS). This includes:

• Serverless.

• Elastic Cloud Hosted (ECH).

• Self-managed with Cloud Connected Mode (CCM).

Indexing images, audio, video and PDFs

To index an image, provide an object with a type of image and a value containing the image as a base64-encoded data URL:

Arrays of objects are also accepted, allowing you to index multiple images in a single field value:

The semantic field also supports text values, just like semantic_text. You can provide such values standalone or intermix them with image values:

Text values are handled just like they are with semantic_text: long passages are chunked according to the chunking settings configured on either the inference service or field mapping. Multimodal values, such as images, are not chunked. Each multimodal value is represented as one chunk.

Other modalities are supported as well. Change the type value to match your content’s modality. Currently we support:

• image

• audio

• video

• pdf

For example, to index a video, the request would look like:

Image search and cross-modal retrieval with a text query

To find multimodal content using a text description, run a match query on the semantic field:

Just like with semantic_text, Elasticsearch automatically generates an embedding for the query text using the inference endpoint associated with the field. That query embedding is used to return semantically similar matches.

This query pattern enables easy text-to-image search. Just index an image and use a match query to retrieve it via text description! It also works for any other modality: index the multimodal input and search by description to retrieve it.

Querying with images, video, and other multimodal inputs

We can also search using a multimodal input by using the knn query with an embedding query vector builder. For example, we can search using an image:

The input object format is the same as when providing an image to index: set the type to image and value to a base64-encoded data URL.

Similar to when querying by text description, Elasticsearch automatically generates an embedding for the query image using the inference endpoint associated with the field. That query embedding is used to return semantically similar matches.

Just like with indexing, other modalities are supported, but are limited to those supported by your inference endpoint. For example, a search using a video clip would look like:

Extending the composition: highlighting, retrievers, and other semantic field features

The semantic field didn't start from a blank canvas. It's built on the same foundation as semantic_text, inheriting its behavior and its ergonomics, and extending them to multimodal content. In practice, that means nearly everything you already know about working with semantic_text carries over unchanged. If you've built with semantic_text before, the semantic field will feel immediately familiar.

Here’s a selection of the features that come along for the ride. See the documentation for a complete list.

Highlighting the best-matching chunks

If you index multiple values in a semantic field, you may want to know which value best matches the query. The semantic highlighter can be used to return the most relevant chunks as highlight fragments:

Setting order to score returns the fragments ranked by relevance, while number_of_fragments caps how many chunks come back. The response looks like:

Note how highlighted multimodal values are represented using their data URLs.

Controlling vector quantisation with index options

The semantic field stores its embeddings in an underlying vector field, and index_options lets you control how that vector field is indexed. For example, choosing a non-default quantization strategy:

Multi-field retrievers

The semantic field participates in the multi-field query format supported by the linear and rrf retrievers. Rather than hand-writing an inner retriever per field, you supply a single query and a list of fields, mixing lexical fields and semantic fields freely:

The retriever automatically separates lexical fields from semantic fields, queries each group, and normalizes the results so that each group contributes equally to the final ranking, preventing lexical matches from drowning out semantic ones.

Cross-cluster search

The semantic field supports cross-cluster search (CCS), enabling use of the field in large, multi-cluster deployments. Simply list the indices to query using the standard <cluster>:<index> format:

The fields queried across indices and clusters can use a mix of different inference endpoints that produce different query embeddings. The search request will automatically apply the proper query embedding to each individual field queried.

Off the easel, into the world: optimising multimodal embeddings for production

When you move multimodal search from experiment to production, the size of your multimodal inputs becomes a practical concern. Multimodal data is supplied as base64-encoded data URLs, and that data is stored in the index. Those strings can grow large in a hurry: a single high-resolution file can balloon into several megabytes of encoded text, which has several side effects:

• The index size on disk can increase significantly.

• Requests and responses containing multimodal data are larger, increasing transmission time and ingress/egress costs.

• Inference on larger multimodal inputs is slower.

The good news is that you don’t need that much fidelity. Multimodal embedding models reduce each input to a compact representation before generating a vector anyway, so a smaller, lower-fidelity version of a multimodal input (such as a downscaled image or a lower-bitrate audio clip) produces a very similar embedding, and similar search quality, to its full-size original. This also applies to PDF input. PDFs are generally processed visually by multimodal models, so the quality only needs to be good enough to perform operations like image embedding and OCR. Long PDFs should be broken up into chunks of smaller inputs, so the embeddings generated more accurately represent each chunk. Feeding the model small inputs keeps your documents lean, trims index and response sizes, and speeds up ingestion, all without meaningfully affecting relevance. 

Elasticsearch reinforces this practice with a guardrail: the indices.inference.max_binary_input_size cluster setting caps the size of each binary input, defaulting to 1 MB. Any individual value that exceeds the limit is rejected with a clear error, so oversized inputs surface as an actionable problem at index time rather than as silent bloat. This setting is adjustable in self-hosted and ECH through the cluster settings API. It is not adjustable in our serverless offering, where 1 MB is the hard limit for binary sizes.

When possible, it is also advised to use source filtering to exclude semantic fields from responses. For example:

This makes responses smaller, more performant, and easier to parse because multimodal data is not returned with each.

Try out the semantic field

The semantic field is available in Elasticsearch 9.5 and Serverless. Start a free trial and try it out today.

What’s a reranker and how does it work?

A reranker is an AI model used near the end of an information retrieval pipeline, after other modules have assembled a short list of candidate matches to a query. It’s trained to order the candidate list from best matching to least. Using a specialized model focused purely on ranking candidate matches can improve result quality dramatically.

Jina AI’s latest rerankers use a technique called late interaction, where queries and documents are encoded separately into lists of token embeddings that reflect each token’s semantics in context and then compared to each other.

This is an AI analog of lexical and grammatical disambiguation.

For example, consider the meaning of the word match in these two sentences:

• She looked for a match to light the candle.

• She looked for a match on Tinder.

The first sentence might be a match for queries about matchboxes; the second for queries about romance.

Transformer-based models do this kind of in-context disambiguation but bring much richer information into the token embeddings they produce. The word match might have a semantic embedding near to words like fire or illumination in the first sentence, while in the second, it might be closer to smartphone or swipe.

Late interaction rerankers generate these context-enriched token embeddings for both the query and the candidate documents and then compare them to produce sortable scores. They’re completely agnostic about how candidate match lists are created. The reranker works exactly the same when combined with lexical search schemes, like BM25, AI-driven semantic embeddings-based retrieval, or hybrid and federated search systems that may retrieve multiple candidate lists from different sources or using different algorithms. Of course, the results always depend on the quality of the candidates, so a reranker can’t fix bad first-stage retrieval, but it almost always improves whatever you’ve got.

jina-reranker-v3.5 is a listwise reranker using the last-but-not-late technique developed for jina-reranker-v3. The query and a list of candidate matches are passed into the model together and processed in one pass, returning a numerical score for each candidate. This enables the model to use context information from the query and the full candidate list to make sense of the entire input, producing better results because of the richer information available to it.

jina-reranker-v3 proved that listwise rerankers with last-but-not-late interaction can compete with the largest models on general reranking benchmarks. Only jina-reranker-v3.5 and models with over four billion parameters beat it on Massive Text Embedding Benchmark (MTEB) reranker tasks. However, this approach places strict limits on candidate list sizes. The query and all candidate matches must fit in the input context window of the model.

What problems does Jina Reranker v3.5 solve?

Despite having frontier-level performance overall, jina-reranker-v3 has some notable performance gaps:

Domain-specific text retrieval

jina-reranker-v3 was trained on general text corpora and, as a result, it underperforms on important use cases, particularly:

• Legal texts, like case law and contract clauses.

• Medical literature, like clinical trials and patient records.

• Financial datasets and other texts full of important numbers.

• Computer programming and IT documentation.

• Product catalogs full of technical terminology and specifications.

Structured data: Tables, JSON, and key-value records

Vast quantities of essential, real-world data is encoded in spreadsheets, tables, key-value lists, and structured records, like JSON data. However, rerankers trained purely for textual comparison, like jina-reranker-v3, perform poorly on this kind of data.

Compute costs for long candidate lists

The self-attention architecture at the core of most text-processing AI models means that memory and compute requirements grow quadratically with the size of its input. This makes jina-reranker-v3, like other AI models, very computationally expensive to run with a full input context window. But, to make the most effective use of the model, we want to put as many match candidates as possible into its input. When it’s at its most useful, it’s also slower and more expensive to run.

We’ve developed jina-reranker-v3.5 specifically to address these issues without reducing its performance on general purpose text retrieval.

What’s new in Jina Reranker v3.5?

jina-reranker-v3.5 contains a modified self-attention mechanism that enhances performance, increases processing speed, and reduces the resources required at inference time to process a full input context window. We’ve also introduced a new three-stage self-distillation training process better suited to the sliding-window architecture of large input context models.

We’ve also curated and used training data focusing on the performance gaps we identified in jina-reranker-v3, including:

• Multilingual legal texts drawn from diverse international sources.

• Medical texts drawn largely from scientific literature and materials used for other AI projects, including a collection of Chinese medical question-answer pairs.

• Financial industry data, including investment-related question-answer pairs, regulations, and tables with numbers and associated texts.

• Structured data, especially from ecommerce sources and public corpora of tables.

• Expanded multilingual and cross-language texts.

For details on the data sources and technical innovations in jina-reranker-v3.5, see our technical report.

How Jina Reranker 3.5 performs on retrieval benchmarks

Parameters

597 million

Input modalities

Text only

Context window size

131,072 tokens

Maximum number of candidate matches

No fixed limit, but all candidates and query must fit in the context window.

Languages

Training in 52 languages

General text reranking performance (BEIR and MIRACL)

jina-reranker-v3.5 improves on jina-reranker-v3’s performance on general text reranking benchmarks. On the English-language Benchmarking Information Retrieval (BEIR) benchmark, the average score has increased enough to surpass the frontier Qwen3-Reranker-4B and 0.6B models and Mixedbread AI’s rerankers.

We’ve also improved jina-reranker-v3’s multilingual reranking performance on the Multilingual Information Retrieval Across a Continuum of Languages (MIRACL) benchmark. Only the four billion parameter Qwen3 reranker regularly beats jina-reranker-v3.5’s score.

Legal, medical, and financial reranking

The Retrieval Embedding Benchmark (RTEB) suite consists of diverse domain-specific retrieval benchmarks. jina-reranker-v3.5 outperforms jina-reranker-v3 on all RTEB tasks related to law, medicine, and finance. Only the large Qwen3 reranker, at almost seven times as many parameters, has better average performance in those three domains.

The new model shows particularly strong improvements for legal data, beating jina-reranker-v3’s score by over 50% on case law retrieval tasks.

Task

Reranker v3

Reranker v3.5

Improvement v3 to v3.5

AILA-Case

20.82

32.55

+11.73 (56%)

AILA-Statute

32.15

46.16

+14.01 (44%)

LegalQuAD

81.84

83.09

+1.25 (1.5%)

LegalSum

69.64

70.99

+1.33 (1.9%)

Structured data reranking (Struct-IR and STaRK benchmarks)

We evaluated jina-reranker-v3.5's structured data reranking on two benchmarks: Struct-IR and STaRK. Both benchmarks contain AI-generated JSON text data covering a variety of applications, including product records, scientific papers, and biomedical knowledge bases.

jina-reranker-v3.5 improves substantially on jina-reranker-v3’s score on the Struct-IR benchmark, once again only exceeded by Qwen3-Reranker-4B. On the STaRK benchmark, jina-reranker-v3.5 beats all the other models we tested, of any size.

Inference speed: Latency benchmarks for short and long documents

The longer the candidate documents get, the more significant the architectural improvements we’ve brought to jina-reranker-v3.5 are. To verify this, we used two retrieval datasets distinguished by large differences in the average document length:

Dataset

Avg. doc length

jina-reranker-v3

jina-reranker-v3.5

Speedup

BEIR Natural Questions

145.5 tokens

371.1 ms

305.3 ms

22%

RTEB AILAcasedocs

1,904.0 tokens

16,064.9 ms

10,290.9 ms

56%

jina-reranker-v3.5 is significantly faster in both cases. On the Natural Questions benchmark, there’s a 22% speedup compared to jina-reranker-v3 with average request latency falling from 371.1 ms to 305.3 ms. Each query from the AILAcasedocs benchmark is much larger (more than 10 times larger on average) so it naturally takes longer to rerank on average: 16,064.9 ms for jina-reranker-v3 and 10,290.9 ms for jina-reranker-v3.5. This represents a 56% speedup for the newer model, representing less latency for applications and lower computer costs.

When should you use Jina Reranker v3.5?

Reranking improves search precision in practically every case, and jina-reranker-v3.5 has applications in a wide variety of information retrieval contexts. However, it has some limitations. The table below summarizes our best-practice advice:

Use case

Recommendation

General text retrieval in common international languages

Use `jina-reranker-v3.5`.

Legal, financial, and medical domain retrieval

Use `jina-reranker-v3.5`.

Semi-structured data, tables, product information texts for ecommerce

Use `jina-reranker-v3.5`.

Non-text or mixed-media data

Use `jina-reranker-m0`, which supports both text and image input.

How to use Jina Reranker 3.5 with the Elastic Inference API

jina-reranker-v3.5 is available via the Jina API with free tokens to try it out. It’s also available via the Elastic Inference API and Elastic Inference Service.

If you’re already using jina-reranker-v3, all you have to do is change the name of the model in the model field of your request to the Jina API or model_id field when configuring an Elastic Inference API endpoint. The two models have completely identical interfaces.

You can install jina-reranker-v3.5 as a Jina On-Prem container to get a completely self-contained server that runs on your own hardware. The model weights are also available to download for testing and research. Follow the instructions on the model’s page at Hugging Face. In both cases, the model is available under a CC BY-NC-4.0 license, so you’re free to try it out for testing, building prototypes, or doing scientific research. For commercial use, please contact Elastic sales.

The most powerful AI models run on remote cloud installations with access via a web API, meaning that you have to trust your AI service provider for security, service availability, and stable prices. You can’t easily align reasonable demands for reliability, privacy, manageable costs, and good data governance with increasingly powerful, sophisticated, and resource-intensive AI usage.

Government regulation, court rulings, and business considerations made in someone else’s interest have all recently resulted in restricting access to specific services. And even if you can switch to other services, AI models aren’t components that can just be swapped out whenever you want. Applications that use semantic embeddings depend on having access to the same models at query time as at data ingestion time. To lose access to your embedding model means your search system comes to a halt.

AI pricing models compound that risk. Recent financial disclosures from major AI vendors give customers good reason to be concerned about potential price hikes. Reliance on products with unpredictable costs adds more risk to capital-intensive AI investments that may not produce clear returns.

Jina On-Prem is Elastic’s answer to these challenges.

Who needs on-premises AI?

Local hosting and direct control over your AI models support a variety of technical demands, industry requirements, and business interests.

Local installation reduces what you pay your AI service providers, but it puts the cost of hardware and reliable access on your organization. Depending on your volume of use, it may simply be cheaper. But there are additional pressing reasons to consider running your own AI. If any of the issues described below concern your enterprise, consider a local AI solution like Jina On-Prem. This list is not exhaustive.

Use case

Why on-prem

Example

Air-gapped / high-security

No outbound data transmission; complete network isolation

Defence, intelligence, classified research

Regulatory compliance

Data sovereignty; no cross-border transmission or third-party exposure

Healthcare (Health Insurance Portability and Accountability Act [HIPAA]), finance, EU enterprises (General Data Protection Regulation [GDPR])

Latency-critical

Zero network dependency; no tolerance for connection failures

Robotics, edge computing, vehicles, ships

Cost predictability

Fixed infrastructure cost vs. per-token pricing with uncertain future rates

High-volume continuous inference workloads

Liability reduction

No third-party data exposure; maintains legal privilege and duty of care

Law firms, government agencies

Why air-gapped and firewalled systems need on-prem AI

Air-gapped and firewalled systems cannot use external AI APIs. Jina On-Prem runs entirely within your infrastructure with no outbound connections.

For organizations managing especially sensitive data, security and privacy considerations are paramount. It does little good to invest in protecting your sensitive data if you promptly turn it over to some remote third party that may have insufficient security in place or might be subject to the demands of a foreign government.

Employees in organizations that handle sensitive data often receive some training in secure data handling, but this isn’t very effective when they all have web browsers that may be open to any page on the internet while they handle that data. Isolation is the most effective security measure available, either through air-gapping or very restrictive firewalls, but that makes it difficult to use external services of any kind.

On-prem AI for latency-sensitive and high-availability systems

Software as a service and cloud computing represent a compromise between the cost of offering highly accessible, reliable services on your own computers and outsourcing the problem to someone else. But they come with variable latency, outages, and a complete loss of control when things go wrong. AI services aren’t the exception. If your search system goes offline when you can’t access your embedding model, it may no longer look like a good compromise.

Furthermore, relying on external AI will always involve risks that you can’t easily foresee or manage. Internet access and network latency can degrade without notice, as a result of political events, bad weather, or ships dragging their anchors over underwater fiber-optic cables. Governments can, and recently have, used export bans to suddenly block access to AI models. AI service providers sometimes withdraw models to induce you to switch to newer ones. The flexibility and managed costs of external services have to be balanced against the risks of dependency.

On-prem AI for GDPR, HIPAA, and data sovereignty compliance

Organizations that collect personal data are subject to increasingly stringent regulations which often differ between jurisdictions and may have contradictory requirements. Notably, HIPAA rules place very strict data protections on American healthcare providers, and strong general data protection laws in Canada, the European Union, and many Asian jurisdictions require all enterprises that handle personal information to do so securely and to limit the transmission of that data to other parties or other jurisdictions. These rules can even impose obligations on foreign entities if they have any customers in those jurisdictions. Financial institutions are frequently subject to even stricter rules and bear the same direct liability for information security that they have to protect against other forms of criminal activity.

Regulatory compliance can be incompatible with third-party AI services, especially if using them involves cross-border data transmission.

Furthermore, recent events show that rules restricting the physical location of data stores may not be a reliable source of protection when international cloud operators are subject to pressure from foreign governments. Local laws may conflict between jurisdictions, requiring local data storage and processing and making third-party services impossible to use. In some cases, the only solution is to take all the parts of your processes in house, including your AI systems.

AI liability risks from third-party data transmission

Data protection laws and recognized duties of care toward sensitive data routinely have liability implications, sometimes very severe ones. You can be liable for third-party service providers’ handling of your data. While courts and legal procedures might provide some retrospective protections from insecure service providers, those remedies are not available nor generally effective against national security actors, law enforcement, or criminal hackers.

For governments, there have already been instances of cross-border cloud service providers releasing sensitive state information to foreign actors.

But even if you don’t worry about foreign governments or hackers, and if your external AI service providers are themselves secure, just the fact that they’re external can create liabilities.

For example, in most jurisdictions, lawyers’ communications with their clients enjoy special legal protections, and law offices have strict liabilities when recording or storing this information. In the United States, this “attorney-client privilege” is so famous, it’s central to movie and TV plots. But one of the ways that privilege can be lost is by communicating information with someone who is not privileged, and recent developments suggest that external AI service providers might qualify.

It’s possible, at least in the United States, that just using third-party AI services over an internet API, like embedding models that provide indexing services, might violate critical confidentiality rules. A law firm might be sued, disciplined, or disbarred just for using externally hosted software, even if no security breach occurs.

On-prem AI for offline, edge, and physically isolated systems

Computer systems aren’t just isolated for security reasons. For example, moving vehicles cannot rely on internet access for any essential functions. Ships and aircraft have very extensive onboard computer systems that have to function without internet connections and therefore cannot use external AI services. Offshore platforms, remote facilities in wilderness areas, computer services in the Arctic, Antarctic and on small islands without adequate physical connections to global networks are all examples of installations that benefit from locally hosting all the services they need. As AI’s role in enterprise computing grows, these limitations become more important to address.

Emerging applications of AI to physical systems (robotics and other spatially confined or external-world–focused use cases, like logistics management systems or even supermarket checkouts) may be connected to the global internet, but they have no tolerance for connection failures or spikes in latency. If they rely on an AI system to operate, that AI system needs to be as local and reliable as possible.

Who doesn’t need on-premises AI?

Remote software services and off-site AI do have benefits. Running AI models can require expensive, power-hungry processors with notoriously short lifespans. Access to high-quality hardware is particularly difficult right now due to market factors and external economic shocks. Under the circumstances, it may make sense to pay by the token to use an external API instead of supporting the steep capital costs of local AI.

External APIs make the most sense for intermittent users. If you use AI models primarily to batch process data for analysis, rather than running a search system that has to be online all the time, it makes little sense to invest in capital-intensive hardware and local installations.

Furthermore, when your data processing is already cloud-based, for example, an ecommerce website hosted in the cloud for reliability and accessibility reasons, using AI services located in the same cloud infrastructure may provide a better value for money than introducing your own licensed AI model deployment. You’re already dependent on your cloud service provider, so being dependent on its AI services doesn’t add much risk.

If your use case sounds like it fits that description, Jina AI models are available on EIS, AWS Marketplace, and the Google Cloud Platform specifically to meet your needs.

The table below summarizes the key factors. Your answer depends on your data, infrastructure and usage pattern.

Factor

On-prem favored

Cloud API favored

Usage pattern

Continuous or high-volume inference

Intermittent or batch processing

Data sensitivity

Regulated, sovereign, or classified

No cross-border or third-party restrictions

Network environment

Air-gapped, firewalled, or unreliable

Stable, always-on internet

Existing infrastructure

Own or can procure GPU hardware

Already cloud-hosted with colocated AI

Cost model

Fixed hardware + license; predictable at scale

Per-token; lower up-front, variable long-term

Latency tolerance

None (robotics, edge, real-time)

Network variability is acceptable

Operational responsibility

Your team manages hardware and availability

Provider manages hardware and updates; you manage integration

You have to consider the costs and benefits in light of your particular circumstances and use cases, taking into account the issues highlighted in the previous section that apply to you. The cost-benefit analysis will doubtless change over time. We can’t predict the future of the AI industry or hardware prices even in the short term.

Introducing Jina On-Prem

For users who can benefit from local AI services, we’re introducing Jina On-Prem, a fully self-contained installation suite for Jina AI’s high-performance models.

Jina AI’s models match the accuracy of embedding models many times their size, reducing compute costs, memory footprints, and hardware requirements. This makes them an ideal choice for users who want or need to keep their AI on-premises. Commercial licenses are available with scalable, proportionately priced solutions for use cases of all sizes.

What API schemas does Jina On-Prem support?

• Available as a complete collection of dependencies for local installation or as a Docker container that you can install and run in minutes.

• Jina On-Prem installations do not call out to outside systems.

  • No call to Hugging Face Hub or any model registry (HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 are baked in).

  • There’s no license server.

  • There are no telemetry or logging endpoints.

• Supports both CPU and GPU hardware, with GPU autodetection.

• All 28 Jina AI models available, including the latest jina-embeddings-v5-omni multimodal embedding models and jina-reranker-v3.

• Access via standard AI API schemas: Jina API, OpenAI, Cohere, Voyage AI, and Gemini. Jina On-Prem is a drop-in solution for applications built on those schemas.

• Drop-in replacement for models served by the EIS. Jina On-Prem integrates directly with air-gapped Elastic deployments.

Hardware requirements for Jina AI on-prem models

The hardware requirements vary for different Jina models. The table below shows the recommendations for the most recent models using GPU settings. You don’t need anything more powerful than an NVIDIA L4 GPU, although an A100 is recommended for the v5 embedding models. Our latest embedding model currently requires a minimum of 8 GB of VRAM.

Model

Minimum VRAM

Recommended GPU

jina-embeddings-v5-text-nano

2 GB

T4 / L4

jina-embeddings-v5-text-small

3 GB

L4 / A10G

jina-embeddings-v5-omni-small

8 GB

L4 / A10G / A100

jina-reranker-v3

3 GB

L4

jina-clip-v2

4 GB

L4

jina-code-embeddings-1.5b

4 GB

L4

ReaderLM-v2

4 GB

L4

If you use more than one model at a time, the VRAM requirements will increase. Please see the Sizing and Hardware page for more information.

How to install Jina On-Prem with Docker

The quickest way to get started is to install Docker (if you haven’t already) and follow the instructions on the Jina On-Prem Quick Start page.

There are pre-composed Docker containers for all 28 Jina models. Download one and transfer it to your installation target, and you can have Jina AI models running in under five minutes.

For multimodal or custom builds, or to download the complete dependency set for installation outside of a container, follow the steps outlined in the bundling guide.

Your Jina On-Prem installation supports all Jina API and EIS functionality and embedding generation via OpenAI, Cohere, Voyage AI, and Gemini APIs, so it can integrate into preexisting applications using standard interfaces. See the API documentation for more information.

Jina models, including models installed with Jina On-Prem, are available on various licensing terms, with the latest models free for noncommercial use under a CC BY-NC 4.0 license. To license Jina On-Prem for commercial use, please contact Elastic Sales.

Combining image and text into one embedding beats either alone, and the gap isn't small. In our tests, averaged image and text embeddings put the correct product in the top spot up to 1.5 times as often as image embeddings alone. We benchmarked 5,000 real apparel and footwear products in English and German across two Jina embedding models, jina-clip-v2 and jina-embeddings-v5-omni-small, to see which model and which indexing strategy actually wins for ecommerce search. The older, narrower Contrastive Language–Image Pre-training–style (CLIP-style) model beat the newer, more general one, and that wasn't what we expected. This post walks through the data, the method, and what we'd recommend doing with it.

jina-clip-v2 vs. jina-embeddings-v5-omni-small: What's different

Multimodal embedding models work by generating representative semantic vectors for inputs of different kinds in a single high-dimensional space.

We used two models that do this:

jina-clip-v2

jina-embeddings-v5-omni-small

Architecture

Dual encoder (separate text + image towers)

Single shared backbone + frozen encoders

Parameters

~865M total

~1.74B total

Embedding dimensions

1024

1024

Max input

512×512 images, 8k tokens text

32k tokens

Language coverage

Broad multilingual

~100 languages

Modality handling

Text ↔ image alignment (purpose-built)

Text, image, audio, video via projectors

jina-clip-v2 is a CLIP-style dual encoder: a text tower (Jina XLM-RoBERTa, 561M parameters) and a separate image tower (EVA02-L14, 304M parameters), about 865M parameters in total. The two towers are independently trained but fine-tuned to output to a common semantic space. It produces 1024-dimensional embeddings, handles images up to 512×512 and up to 8k tokens of text, and has broad multilingual support. It has been engineered specifically to support text-to-image, image-to-text, and text-to-text matching.

jina-embeddings-v5-omni-small has a broader scope. It extends the jina-embeddings-v5-text model to support images, audio, and video by attaching frozen vision and audio encoders to the frozen text backbone. The encoders connect through cross-modal projectors, small trained layers that map each encoder's output into the text model's embedding space. These projectors are the only part of the model to receive additional training. The resulting model produces 1024-dimensional embeddings, supports a 32k-token input context, and covers roughly 100 languages. It encodes queries and documents asymmetrically: a query with the retrieval.query task, a document with retrieval.passage.

There’s an important functional difference between the two models: jina-clip-v2 is really two separate models trained to work together, but jina-embeddings-v5-omni-small uses a single shared backbone that produces embeddings for all its supported media types. Every modality maps into one shared vector space. In principle, it can handle text, images, audio, or video, or combine materials of different media types into one input, yielding one embedding that encompasses all the data. However, there are two important caveats when using jina-embeddings-v5-omni-small: Combining image and text into one input is a documented weak spot for the model, and the Jina API only allows users to embed one modality per request. Theoretically, it can create a joint image and text vector, but in practice, you can’t with the API (and shouldn’t anyway).

The ecommerce product dataset we used

For this article, we downloaded the Fashion Product Images dataset from Kaggle. It contains roughly 44,000 catalog entries for products from a real fashion retailer, each with a high-resolution photo and structured metadata. We only used the Apparel and Footwear categories (about 30,600 products) and sampled 5,000 from them with a fixed random seed.

For each product, the dataset contains three records:

• Each product is pictured in a 1800×2400 JPEG against a clean background.

• Metadata with the labels gender, masterCategory, subCategory, articleType, baseColour, season, year, usage, and productDisplayName.

• A collection of additional informationwith labels like Neckline, Pattern, Sleeve Length, Fit, and Fabric, and a free-text description in English.

For example, item #13885 is labelled "Scullers Men Check Black Shirts," with an accompanying image (see below) and a description that reads "Black and white checked shirt, made of 100% cotton, full length buttoned placket, long sleeves with buttoned cuffs."

How we generated the search queries

We generated our test queries without an AI language model, using rules and substitution lists.

For each product, we started from its color and article type and added one descriptive modifier, drawn at random from the product’s available metadata: neckline, pattern, sleeve length, length, surface styling, fit, fabric, season, or usage. We then reworded the query by substituting words from a fixed synonym table. This avoids making queries that reuse the catalog's own words. For example, ″t-shirt″ becomes ″tee″, ″regular fit″ becomes ″classic cut″, ″printed″ becomes ″with graphic design″, ″sleeveless″ becomes ″no sleeves″. The longest phrase with a synonym was replaced first, so we swapped ″sports shoes″ rather than ″shoes″. Because the pipeline is rule-based and seeded, query production is reproducible and all variation is accounted for. German queries were generated the same way, using a German term table, and then a native speaker corrected them for natural retail phrasing. Some examples:

The German queries used their own term table, mapping the same catalog attributes to natural German retail phrasing (later checked by a native speaker). Some examples:

The six embedding configurations we tested

We tested retrieval in six configurations, using the same text queries in each test condition and the same 5,000 product indexed dataset. For both jina-clip-v2 and jina-embeddings-v5-omni-small, we tested three different ways of generating embeddings for indexing:

1. Image-only. We generated embeddings from the images alone without any other data.

2. Text-only. We generated embeddings for the free text descriptions alone.

3. Averaged image and text. For each product, we generated embeddings for the image and text description separately and then averaged the two vectors into one.

We used the Jina API to generate document and query embeddings for product images and free text descriptions, as shown in the code below. All images were resized to fit into a 512x512px square before processing.

We combined images and texts by embedding them separately, averaging the two vectors, and then normalizing the result so we can speed up calculating cosines. This works for multimodal models because both embeddings share the same semantic space. The sum of the two vectors is a new vector with the semantic features of both.

This is very easy to do using the numpy package in Python. We used the code below:

For this article, we did exact retrieval, calculating the cosine between queries and all 5,000 stored product embeddings. In Elasticsearch, we would use a shortcut to approximate the same result. From the ranked results, we calculate Recall@1, Recall@5, Recall@10, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain at position 10 (nDCG@10).

Each query has exactly one correct answer, so Recall@K is the share of queries whose product lands in the top K. MRR scores the results by how close the correct answer is to the top. nDCG@10 is a standard metric that penalizes putting the best answer lower on the results list.

Product search benchmark results

The table below is the German cross-lingual run. We evaluated German queries to find products with English descriptions:

Query benchmark results for jina-clip-v2 and jina-embeddings-v5-omni-small using German-language text queries and the images and English descriptions of garments.

The averaged image/text embeddings score the best, both using jina-clip-v2 and jina-embeddings-v5-omni-small. Surprisingly, averaged vectors from jina-clip-v2 lead the table. It ranks the correct product first about 1.5 times as often as the image-only setup and significantly more often than the text-only setup. Furthermore, it beats every jina-embeddings-v5-omni-small condition. Results for the same tests using English-language queries:

Query benchmark results for jina-clip-v2 and jina-embeddings-v5-omni-small using English-language text queries and the images and English descriptions of garments.

The English run, on the same 5,000 products, tells the same story. The main difference is that the jina-embeddings-v5-omni-small scores are significantly closer to those of jina-clip-v2, although still lower.

What the benchmark scores actually mean

The relatively low scores in the German and English benchmark tables above are to be expected. This is a real-world dataset full of near duplicates. A search for a "black tee with classic cut" has to sort through dozens of basic black T-shirts, and even on the best ecommerce sites, you would expect a result like this. The important thing to understand is the difference in scores between the different conditions, not their absolute values. Our key finding is that combining text and image embeddings yields better performance than either one alone, highlighting how multimodal AI-driven search can use different information sources to produce better performance than non-multimodal strategies.

When a picture is worth a thousand words, and when it isn’t

The gain from adding images to text embeddings isn’t evenly distributed. We did a deep dive to see if there was a pattern to the results and discovered a few things:

• Images add a lot for visually distinctive items and attributes. For example, on footwear, image-only retrieval is on par with text-only: 0.029 versus 0.028 for Recall@1. Shoe styles have distinctive shapes, so the picture does the work (image-only 0.029 versus text 0.028). The same holds for visible attributes more broadly (color, pattern, sleeve length), where fusing image and text gives the biggest lift over text-only (0.085 versus 0.073).

• Images help the least with things the model can’t see. For example, if we query for fabric types, adding images to embeddings adds next to nothing. Humans and AI models alike struggle to see that something is or isn’t made of linen or polyester or some other fabric type. That information is only in the text description and metadata.

As a rule of thumb, we find that searches for clothing lean more heavily on accurate text descriptions, while footwear leans more on the semantics of images. But in both cases, merging the two embeddings either improves results or doesn’t make them worse. This highlights how use-case–specific considerations drive optimal search strategies.

Cross-language queries gain a lot from multimodal embeddings

The gap in performance between text-only and combined image and text embeddings using jina-clip-v2 is much larger for German queries (0.074 versus 0.065 Recall@1) than for English ones (0.076 versus 0.075). This implies that English queries can take advantage of being in the same language as the product descriptions. Whether that’s due to overlaps in the words or that the model is simply more competent with single-language semantics than cross-language doesn’t matter. But adding images to the text embeddings compensates almost completely for the model’s shortcomings in cross-language retrieval.

This gap is even larger for jina-embeddings-v5-omni-small. In any kind of cross-language or multilingual context, multimodal embeddings seem to significantly improve retrieval performance.

Can AI-generated product descriptions replace human ones?

AI-generated descriptions scored worse than human-written ones in our tests. We tried replacing human-authored product descriptions with ones written by jina-vlm based on the image. For this test, we used a 1,000-product random subset. The results were much worse than with the original human text. This was what we expected: The automatically generated description was less accurate and less oriented toward the salient features of the product than the human authored one.

So it turns out that not everyone’s job can be replaced by AI. People who write blurbs for catalogs ought to be safe for now.

How should you index your ecommerce data?

Our tests aren’t totally scientifically rigorous, but they do offer some insights into the issues you might face if you have similar data. We offer the following as provisional conclusions:

• If you have aligned texts and images (and most catalogs do), combine them in your embeddings. In every case, using a multimodal embedding model like the ones Jina AI by Elastic provides and then averaging the image and text embeddings significantly outperformed all other options. The combination adds no computing costs at inference time but does create additional costs at embedding time. For each product, you’ll need to generate two embeddings and combine them, roughly doubling the cost.

• Use the right model. You need to identify a model that supports all the modalities and languages you plan to use. All the inputs have to be embedded in the same semantic vector space or none of this will work. It won’t do to get two single-modality models or multiple language-specific text models, average their outputs, and hope for the best. Jina AI by Elastic currently supports texts in up to 100 languages, including computer code and technical terminology images of all kinds, such as scans and infographics, as well as audio and video data. You can change your mind about your models later, but only if you’re willing to reindex all your data.

• Whatever you pick, test it on your own data. The only way to know what the best model is for you, your data, and your use case is to try them out. We were very surprised that our older CLIP-style model outperformed our latest on this dataset, but it was trained almost specifically for this use case. Your data and use case could easily show the opposite. This result is from one catalog, with one style of photography and one kind of query. The ranking between two models can flip with a different domain, image style, or query mix. Run the same sort of benchmark on a sample of your own products before you commit. It’s the only way to know which model really fits your case.

• Use generated descriptions to fill gaps, not to replace good text. AI isn’t a replacement for good work done by conscientious people. Replacing human-made descriptions with machine-made ones made results worse. AI should only replace humans when it has to, like when data is missing or needs to be augmented and it’s impractical to have humans fill in the gaps. Yes, we have tools that work in those situations, but they aren’t necessarily good substitutes. They’re OK substitutes, sometimes.

Average your embeddings. Semantic embeddings are very robust, and averaging them is a relatively cheap solution that doesn’t affect inference-time costs at all. This is a real boon over methods that index each modality separately and require multiple queries to satisfy a single request. But they do require compatible multimodal models.

Limitations of this product search benchmark

A few things to keep in mind before generalizing too much from this experiment:

This article doesn’t perfectly match real-world use cases. Human users make messier queries and have more ambiguous matching criteria. The queries we used were generated specifically for this data. A test with actual customer-made queries from system logs would be a better one.

Embedding averaging isn’t the same as a true joint embedding. Embedding models rely on the different parts of their input to interact in order to extract a semantic representation of the whole. The approach used here is a bit of a hack, one that relies on the robust nature of semantic embedding spaces to get the job done. We expect future models from Jina AI to produce better embeddings by supporting more than one input modality at a time.

This is one dataset in one domain with distinctive features. Fashion photography is very foreground-focused and the descriptions are attribute-rich. Other kinds of materials, even for ecommerce, may look very different. It’s important to test as much as possible with your own data or something very similar.

How to get started with multimodal product search embeddings

jina-clip-v2 and jina-embeddings-v5-omni-small are available through the Jina API, Elastic Inference Service (EIS), and Hugging Face. The omni models are free to download under a CC-BY-NC-4.0 license and free for noncommercial use, with commercial licensing through Elastic. If you use Elasticsearch, EIS exposes both models through the semantic field type, with non-text media in Base64 encoding.

The takeaway from this article is intended to be practical and actionable: For product search, a picture and its description aren’t the same signal. Both add information, and you don’t have to pick one. Average your multimodal embeddings, and benchmark the results with your own data to get a good picture of the kinds of results you can expect.

Prerequisites

• Elasticsearch 9.x with jina-embeddings-v5-text-small inference endpoint available.

• Python 3.10+,

• Elasticsearch API key,

What is quantization?

An embedding is a list of numbers. By default, each number is a float32, which uses 4 bytes. Quantization stores each number with fewer bits, trading precision for space.

Like a JPEG, a quantized vector is a smaller, lower-fidelity copy of the original that still gets the job done.

Name

Bytes / dim

1024-d vector

Compression

`Float` (Baseline)

4

4096 B

1x

`int8`

1

1024 B

4x

`int4`

0.5

512 B

8x

`bbq`

~0.14

142 B

~29x

What is BBQ?

Better Binary Quantization (BBQ) is Elasticsearch's 1-bit quantization mode for dense vectors. Each dimension of the vector is stored as a single bit, plus a few corrective bytes per vector. Then, a rescoring step is applied at query time. This keeps the final retrieval quality close to a full precision search.

For the math behind each level, see Scalar quantization 101, Optimized Scalar Quantization, and the BBQ deep dive.

How does BBQ preserve search accuracy?

Plain 1-bit quantization leads to too high a search quality degradation on its own. BBQ maintains high retrieval quality through three mechanisms:

1. Asymmetric precision: Stored vectors use 1 bit per dimension.

2. Corrective factors: A few floats per vector record the rounding error and correct distances at scoring time.

3. Oversample and rescore: BBQ scans candidates with the bits and then reranks the top ones with higher precision. Fetching the top 10 means scanning about 30 candidates.

The result is the vectors that are roughly 32x smaller, with retrieval quality close to full precision. In the next section of the article, we’ll measure the memory savings and the recall on a real corpus.

How Jina embeddings v5 works

Jina embeddings v5 is a multilingual embedding model with quantization-aware training, which makes it a natural fit for BBQ in Elasticsearch: The 1024-dimensional vectors from jina-embeddings-v5-text-small sit above the dimensional floor where binary quantization stays accurate, and the model is trained so that 1-bit quantization loses little quality. Its main features are:

• One model for many tasks: v5 uses small Low-Rank Adaptation (LoRA) adapters on top of a single base model, one for each task: retrieval, text-matching, clustering, and classification. Elasticsearch picks the right adapter automatically at index and query time.

• Matryoshka dimensions: v5 is trained so you can truncate the vector (1024, 512 to 256) and minimize search quality reduction. This is another way to shrink vectors, independent of quantization.

• Quantization-aware training: v5 is trained to work with BBQ, so its 1-bit vectors lose little accuracy.

We use jina-embeddings-v5-text-small. This model is available through Elastic Inference Service (EIS) and outputs 1024 dimensions with 32k token context and is multilingual across 93 languages. That puts it above the 384-dimension threshold, below which Elasticsearch no longer defaults to bbq_hnsw.

Full model details are in the Jina v5 article on Search Labs.

Setting up the BBQ vs. float32 comparison

We’ll create two indices: Both share mappings, and what changes is the index_options.type parameter, which tells Elasticsearch how to store the dense vector field (as raw float32 HNSW or as 1-bit BBQ):

Index

`index_options`

Loaded into memory

`vectors-float32`

`hnsw`

Raw `float32` with no quantization (baseline)

`vectors-bbq`

`bbq_hnsw`

1-bit BBQ quantization + corrective factors

We then embed the corpus once with Jina v5, index those same vectors into both, and compare them on disk usage, memory footprint, and recall. You can follow along with the full supporting blog content notebook.

Connect to Elasticsearch

Create the two indices

Note: In production, you can use semantic_text to let Elasticsearch manage the mapping and inference endpoint automatically.

Point at the Jina v5 inference endpoint

We call the model jina-embeddings-v5-text-small directly (no need to create an inference endpoint) to turn text into vectors.

As result of the test, we got:

Load a multilingual news dataset

We stream real news articles from hotchpotch/multilingual_cc_news, a parquet mirror of CC-News. We take about 1,000 articles from five languages (around 3,000 docs total), plus a small held-out set of headlines to use as search queries. Using multiple languages also lets Jina v5 show its multilingual strength.

Generate the embeddings and bulk index

We embed the corpus a single time and feed those exact vectors into both indices.

We force-merge to a single segment so the storage numbers are stable and comparable.

Results: Disk versus memory

The disk usage API reports how many bytes each index spends on vectors (knn_vectors).

Result:

On disk, the two indices are about the same size. A quantized index still keeps the raw float32 vectors (needed for rescoring and requantization during merges) and adds the 1-bit vectors on top, so BBQ ends up slightly larger on disk.

The real savings is in memory. The HNSW scan only needs the 1-bit vectors in RAM, while the raw floats are read from disk to rescore the top candidates. We size that footprint using the documented kNN memory formulas: float uses num_vectors × dims × 4 and bbq uses num_vectors × (dims/8 + 14).

BBQ's extra bytes on disk should match the 1-bit payload we computed for memory. Here, that’s 13.25 - 12.80 = 0.45 MB versus the computed 0.44 MB. They line up.

Results: Recall

To check whether the quantized index returns results similar to the float baseline, we use recall:

recall@k = | BBQ top-k ∩ float32 top-k | / k, averaged over all queries.

We vary the oversampling factor (num_candidates / k) that’s the number of candidates BBQ scans with 1-bit vectors before reranking the top ones against the original floats to find the lowest value that still matches float32.

As a result, we have:

BBQ starts at 0.994 recall@10 at 1x oversampling, holds there up to 3x, and then settles at 0.989 at higher factors, meaning it returns at least 98.9% of the same top-10 documents as float32 across all oversampling values. For more on how recall varies across datasets under quantization, see Fast vs. accurate: Measuring the recall of quantized vector search.

BBQ quantization results summary

The same vectors, two storage formats, and one experiment:

• Disk: Roughly the same (12.80 MB versus 13.25 MB). BBQ keeps the raw floats around for rescoring and merging.

• Memory: 29x smaller (12.71 MB versus 0.44 MB). This is the number that decides whether your cluster fits the corpus.

• Recall@10: 0.994 at 1x oversampling. Quantization-aware training pays off.

When to enable BBQ: If your dimension count is above the 384-dim floor, if your vectors are the dominant memory cost, and if you can afford a few extra candidates to rescore. For Jina v5 specifically, the model is trained for it, so the recall hit on most corpora is small.

Further reading on BBQ and vector quantization

• Run the full notebook from this article in the supporting blog content repo.

• For the math behind BBQ, see Better Binary Quantization in Lucene and Elasticsearch.

• For more on Jina v5's architecture, see Jina embeddings v5 on Search Labs.

• For a broader walkthrough on adopting BBQ, see How to implement BBQ into your use case.

• For the original research behind BBQ, see the paper RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search.

Text queries retrieve images, screenshots retrieve documentation, and PDFs, charts, and infographics index into the same vector space. The model supports Matryoshka truncation, so you can drop from 1,024 to 512 or 256 dimensions when storage matters, with minimal quality loss.

jina-clip-v2 is one of several Jina embedding models now available on EIS. For workloads that also span video and audio, jina-embeddings-v5-omni covers all four modalities in a single index: nearly 100 languages and a 0.67B parameter base small enough to run on conventional GPU servers. jina-clip-v2 remains the focused option for cross-modal retrieval between text and images.

How multimodal search works in jina-clip-v2

jina-clip-v2 is a dual-encoder model where separate text and image encoders produce embeddings in the same vector space. This allows text and images to be retrieved interchangeably. A query like “red sports car” can return matching images, an image can surface relevant product descriptions or documentation, and screenshots can map directly to tickets, dashboards, or logs. This isn’t a stitched pipeline of models. It’s a single, shared embedding space across modalities, combining a multilingual Jina-XLM-RoBERTa text encoder with an EVA02-L vision encoder.

Multilingual and document-aware by design

Unlike traditional CLIP models that focus primarily on short English captions, jina-clip-v2 is trained on multilingual text-text and text-image pairs, across 89 languages, and on visually complex datasets at progressively higher resolutions.

EIS allows you to run managed models directly inside Elasticsearch. There’s no separate model hosting layer to provision, no GPU infrastructure to manage, and no external embedding service to maintain.

With jina-clip-v2 on EIS, you can:

• Generate text and image embeddings where your data already lives.

• Index multimodal vectors alongside structured and unstructured content.

• Combine vector search with BM25 using hybrid retrieval.

• Power multimodal retrieval augmented generation (RAG) pipelines grounded in images and documents.

How to run multimodal search with jina-clip-v2 on EIS

The jina-clip-v2 endpoint is preconfigured on Elastic Inference Service. To generate embeddings, call the inference endpoint from the Elasticsearch dev console:

This is the response:

Using jina-clip-v2 embeddings in a search query:

Get endpoint config

Basic text request

Multimodal batch (text + image as separate vectors)

The example below shows how to send both a text and an image input as separate items, each producing its own embedding:

Create custom endpoint with minimum dimensions

Multimodal search in Elasticsearch, from text to images to RAG

By making jina-clip-v2 available on EIS, multimodal search becomes a first-class capability inside Elasticsearch.

Text and images can be indexed into the same vector space. Queries can retrieve across modalities and languages. Hybrid search can combine lexical precision with multimodal semantics. RAG systems can ground responses in charts, screenshots, and document layouts, not just plain text.

All Elastic Cloud trials have access to Elastic Inference Service. Try it now on Elastic Cloud Serverless or Elastic Cloud Hosted, or use EIS via Cloud Connect with your self-managed cluster.

jina-vlm is a 2.4B-parameter vision-language model that pairs a SigLIP2 vision encoder with a Qwen3 language decoder, using attention pooling over image tiles for token-efficient handling of arbitrary-resolution inputs. Beyond the model itself, the paper's main contribution is its “leave-one-out” ablative data-mixture: By removing one task, domain, modality, or language category at a time during training, you can figure out which slices of data are significant or redundant, and whether learning in one domain transfers to others. The result is a compact model that, despite its size, achieves state-of-the-art multilingual VQA performance.

Rio delivered everything you'd hope for: warm, sunny beach weather, the easy walk between Copacabana and Ipanema, the view from Christ the Redeemer, the colors of Escadaria Selarón. A welcome contrast to a still-chilly European spring.

What was trending at ICLR 2026: RLVR, test-time compute and retrieval

Conferences like ICLR give everyone a chance to take the field’s pulse and find out what’s hot, what’s not, and what’s coming up. After a few days of walking the aisles at the poster sessions and dropping in on oral sessions, you start to get a sense for things. You start to see the same words on poster after poster, and you notice which sessions are the most crowded.

Here are a few things we picked up on:

Reinforcement Learning with Verifiable Rewards (RLVR) is now the dominant paradigm for post-training refinement. Almost every reasoning-focused poster we stopped at was using some form of Group Relative Policy Optimization (GRPO) for math correctness, code execution, and formal-logic checks, rather than Reinforcement Learning from Human Feedback (RLHF). Direct Preference Optimization (DPO) fine-tuning, which felt like the default a year ago, was conspicuously rare. It makes sense: If you can use code to check for correctness, you no longer need to get annotated data and the training loop goes much faster.

Test-time compute has stopped being a curiosity and become a design problem. Test-time compute – the time a system spends generating a response – is an increasingly important study variable. Papers now measure it as part of their experimental setup and developers try to optimize for it. Models are now built with the expectation that inference will be expensive and clever, not just a single forward pass through a neural network.

Vision-Language Models (VLMs) are everywhere, and Vision-Language-Action models (VLAs) are not far behind. A big chunk of the conference was about how to make multimodal AI work better, like better tokenization for images, better positional encodings for non-text media, and more efficient ways to compress visual information before it overwhelms your model. Vision-Language-Action models that extend multimodal AI recipes to robotics and embodied agents are no longer niche research. They brought in the crowds at their presentations and hosted vibrant debates.

Reports of the death of State-Space Models (SSMs) have been greatly exaggerated. Attention models still dominate AI, but Mamba, SSM variants and recurrent neural networks still draw attention and research, both as full replacements for Transformers and as components inside hybrid attention-based stacks. Whether they'll ever genuinely displace Transformers is an open question, but the line of research is alive and well.

Agentic AI safety is taken very seriously. A lot of papers and presentations discussed problems like machine unlearning and jailbreaking, and some of the most interesting work was on prompt injection through agentic tool use, like when a model dutifully follows instructions hidden in a webpage or an API response it just fetched. A repeated, slightly unsettling observation: models that follow instructions better tend to be more vulnerable to this kind of attack, not less. This capability-vulnerability tension is going to define a lot of the next few years of safety research.

Hallucination and factuality are increasingly framed as retrieval problems. Several talks made that point explicitly: A generative model that has to invent facts will inevitably hallucinate them, while a model that retrieves information can ground its responses in verifiable ways. That framing is, of course, exactly the bet that search AI engineers have been making all along.

ICLR 2026 invited talks: hidden universe imaging and open AI development

Two of the invited talks stood out to us, albeit for very different reasons:

Images of the Hidden Universe

Katie Bouman presented a tour of how physics, prior knowledge, and machine learning combine to reconstruct information that the universe never gives us directly, like the silhouettes of supermassive black holes and the invisible dark matter structures. She walked us through the Event Horizon Telescope's imaging of M87 and Sagittarius A, building images up from indirect and incomplete radio measurements, and then extended the same machinery to mapping dark matter through gravitational lensing.

This talk was a useful reminder of why machine learning matters outside the LLM bubble. The more you already know, the more you can learn from a little bit more information. This principle generalizes beyond astronomy to knowledge in general, and to machine learning in particular. Any decision system that uses sparse, noisy observations is confronted with it.

_____________________________________________________________________________________

Marin: Open Development of Frontier AI

Percy Liang opened his presentation with a blunt observation: As AI capabilities skyrocket, openness plummets. His response is Marin, a platform for community-driven AI research where every experiment is open, every suggestion or discussion is on public fora, and anyone can review or rerun a result.

What makes Marin interesting isn't just creating open weight models - plenty of projects do that - but creating an open process for making models. Project pre-registration, peer review, and reproducibility have long been part of the natural sciences, and Marin attempts to maintain that tradition for AI. Model training is treated as a matter of public scientific record.

The talk presented concrete scientific results from this approach (optimizer findings and scaling-law results), suggesting that community-scale science isn't just an aspiration but a workable methodology.

_____________________________________________________________________________________

Bouman and Liang made a pleasingly complementary pair: one a reminder of how much ML has to offer the world outside ML, the other a challenge to how the field organizes itself.

ICLR 2026 research highlights: embedding models, retrievers and sparse representations

We attended many oral presentations and poster sessions. The papers below stood out because of their potential to impact how we make and use embedding models.

Rethinking pretraining for representations

Decoder-only models have dominated the LLM leaderboards for years, but one paper makes a case for encoder models.

Seq vs Seq: An Open Suite of Paired Encoders and Decoders does a repeatable, open-data, architecture-controlled comparison of encoder-only and decoder-only models trained identically. They used the same data, same architecture, same training recipe, and differed only in their training paradigms: Bidirectional Masked Language Modeling (MLM), typically associated with encoders, vs. Causal Language Modeling (CLM), usually used in decoders. Their results confirm prior findings that encoders excel at classification and retrieval while decoders excel at generation. A key finding is that cross-objective continuous pretraining does not close the performance gap between the encoders and decoders. A 400M parameter encoder beats a 1B parameter decoder in classification and retrieval, and vice versa for generative tasks. All artifacts including data, checkpoints, and code are open-sourced.

Their study delivers a definitive empirical finding for the AI community: Encoder-only pretraining is substantially more efficient for classification and retrieval tasks than adapting decoders to act like encoders, even with post-training on high-quality data. This challenges the recent trend of adapting large decoder LLMs (like LLM2Vec) for embedding tasks. Dedicated encoder pretraining from scratch remains the most reliable path to strong retrieval performance. Additionally, the public release of 200+ checkpoints with batch-ordered training data makes their work an invaluable resource for studying how retrieval-relevant representations emerge during training and how they scale with parameter count and tokens.

New paradigms for training retrievers and embedders

Revela: Dense Retriever Learning via Language Modeling reframes dense retriever training as a language modeling problem. Rather than using supervised training with query-document pairs, it trains a retriever model jointly with a language model by conditioning next-token prediction on all the other documents in the batch. This innovative in-batch attention mechanism modifies the model’s Transformer blocks by injecting the similarity scores of documents in each batch into the cross-document attention weights. Training is done on raw text, without query-document pairs, hard negatives, or synthetic data generation. The resulting 3B parameter model outperforms E5-Mistral-7B-Instruct (with 7B parameters) as well as proprietary closed-weight embedding models like OpenAI, Cohere, and Voyage. On retrieval benchmarks, it matches E5 despite using roughly 1000 times less training data and approximately 10 times less compute.

This demonstrates that next-token prediction can still serve as an effective training objective for high-quality dense retrieval AI. This is important because plain text data – what you need for next-token prediction – is widespread and inexpensive and this paper shows that it’s all you need to train competitive embedding models.

Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinementadvances the proposition that LLMs should learn to "speak an embedding language," i.e., generate sequences of “soft tokens” optimized for semantic representation rather than human readability. They outline innovative loss functions and objectives in support of this goal, and show that the resulting models have very competitive performance, while generating only a handful of additional tokens.They also show that generating more tokens at inference time steadily improves embedding quality in a way analogous to chain-of-thought scaling in reasoning LLMs. KV-caching reduces the computational overhead of the generation process to within 1.1 times that of standard single-pass embedding models. This approach represents a new paradigm for representation learning, complementary to encoder-only and single-pass approaches.

Towards Improved Sentence Representations using Token Graphs frames the problem of generating embeddings for sentences from token-level representations as a relational learning problem rather than a compression problem. Instead of pooling tokens, it uses a supplementary neural network that processes a dynamically constructed graph made from output token similarities. This added network is compact, with very few trainable parameters, and can be implemented without doing any additional training on the main language model. The result is competitive with current frontier models.

This approach can be dropped into any language model at a very reasonable additional training cost, giving it immediate practical significance. Furthermore, the resulting models hold up well in the presence of noise, a known problem, especially for long-context models.

Sparse and ultra-efficient embeddings

LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference introduces an asymmetric dual-encoder architecture for embeddings-based retrieval in which the query encoder is much smaller and faster than the document one. The key insight is that while document embeddings benefit from the modeling power of a large language model, query embeddings are much less demanding. During training, they propose to learn per-token query embeddings, then, at query time, those embeddings are retrieved and averaged to produce a full query embedding. Documents must still be encoded at storage time using a potentially large encoder, but there is no need to invoke an embedding model at query time at all. The result retains approximately 95% of the performance of the query encoder it replaced. This has immediate implications for computational constrained, time-sensitive, or resource-efficient text information retrieval systems.

CSRv2: Unlocking Ultra-Sparse Embeddings addresses the computational cost of embeddings-based retrieval using dense, high-dimensional vectors. It tackles that cost with Contrastive Sparse Representation (CSR), which maps dense vectors into a much higher-dimensional space where only a few vector entries are non-zero, so that search can use highly efficient sparse-vector search techniques like inverted-indexes.

CSR approaches tend to break down when the number of dimensions with non-zero values becomes very low. This paper addresses this problem with an innovative training approach that makes ultra-sparse representations viable, opening up the possibility of much faster, less computationally demanding retrieval without loss of accuracy.

Multi-step and multimodal retrieval

Q-RAG: Long-Context Multi-Step Retrieval via Value-Based Embedder Training frames the problem of multi-step retrieval-augmented generation (RAG) in terms of optimizing the embeddings used in RAG search. RAG systems are typically based on a single retrieval step: Input to an LLM becomes a query to a vector store, and a selection of the results are presented to the LLM as a basis for composing a response. However, agentic approaches that involve multi-step interactions between the LLM and vector store can improve RAG performance significantly, especially for large input contexts that might contain millions of tokens. This paper seeks to optimize the embedding model used for retrieval to better support this usage scenario with Reinforcement Learning with Verifiable Rewards (RLVR).

This paper is one of the more elegant intersections of two of the conference's biggest themes — RLVR and retrieval — and it gives a glimpse of what retrieval looks like when it has to operate inside an agentic loop, not just before one.

Foundations and evaluation

HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks undertakes the unusual task of systematically measuring human performance on the Massive Text Embedding Benchmark (MTEB), the most widely used benchmark for embeddings-based information retrieval. Using 16 datasets in 5 languages, they find that average human retrieval accuracy is 77.6%, while the best embedding models currently score over 80%. However, this performance gap is uneven. Models may outperform humans on standard tasks but fall apart when faced with low-resource languages, where human intuition still holds a significant lead.

This paper also shows that "superhuman" scores on low-agreement tasks are mostly artifacts of fitting noise, not genuine capability. This underlines the problem of our current suite of embedding benchmarks: New models are not improving benchmark performance very much. To make progress, we need new, harder challenges and a total rethink of how we evaluate models.

Training dynamics for foundation models

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining identifies a significant but underexplored problem in AI model training. Large training sets can create a problem with models forgetting things that they’ve learned as they’re presented with more data. Curriculum-based pretraining — sorting data from low to high quality — should help, but in practice the results have been disappointing. The reason, the authors argue, is that the model encounters the highest-quality data late in the training schedule when the learning rate is at its lowest. Its gradient contribution is therefore greatly reduced. They confirm that hypothesis empirically by showing that curriculum training significantly beats random shuffling if training uses a constant learning rate.

They propose two simple strategies to fix this: Let the learning rate decay more slowly, or replace learning rate decay with weight-averaging over the multiple final checkpoints. Combining the two yields a 1.64% average benchmark improvement over standard practices with no additional data refinement. The broader message - that data composition and optimization schedule need to be co-designed - applies well beyond pretraining, and is a useful frame for embedding training too.

What ICLR 2026 means for retrieval and embedding research

Science has always been conducted through print and publication, but in-person conferences are still the only way to put people together in a room. Over five days, we met a steady stream of researchers from very different backgrounds — academia and industry, large labs and small startups, half a dozen countries — and conversations ranged from research trends to philosophical questions that have haunted AI from the beginning. Are LLMs really reasoning, or are they doing something more like very high-dimensional memorization with interpolation? Where exactly is the line, and does it matter for what we can build on top of them?

These conversations rarely produce answers, but they sharpen the questions, which is most of what good research is.

For the information retrieval work we do at Jina by Elastic, the future looks bright. Retrieval, long relegated to merely applied research, is increasingly recognized as the engine for keeping language models grounded. Better encoders, better embedding training paradigms, sparser representations, and retrieval that operates at the core of reasoning loops – these things matter to us all. What we saw and heard at ICLR 2026 convinces us that this is where a meaningful share of the next round of progress will come from.

We're already looking forward to seeing where the field is next year.

With this integration, Elasticsearch users gain a new deployment option that keeps data inside their security perimeter, delivers predictable infrastructure costs, and runs natively on Google Cloud. At the same time, the broader Google Cloud ecosystem gains access to Jina's purpose-built, state-of-the-art search and retrieval models.

This is the first stage of a broader rollout. Together with the models coming next, the lineup will form a complete retrieval stack: Embed your data, embed queries, retrieve and rerank candidates, and extend search to images with multimodal embeddings, all on infrastructure you control. You can start today with jina-embeddings-v3, the model already powering production search pipelines across the Elasticsearch ecosystem via Elastic Inference Service (EIS).

Model

Type

Parameters

Key capability

Status on Model Garden

`jina-embeddings-v3`

Text embedding

572M

Proven multilingual workhorse, 8K context, 1024 dim output, truncatable to 32

Available now

`jina-embeddings-v5-text-small`

Text embedding

677M

State-of-the-art sub-1B multilingual, 32K context, 1024 dim output, truncatable to 32

Coming soon

`jina-embeddings-v5-text-nano`

Text embedding

239M

Best-in-class under 500M params, 8K context, 768 dim output, truncatable to 32

Coming soon

`jina-reranker-v3`

Reranker

600M

Listwise reranker, 131K context, up to 64 documents

Coming soon

`jina-clip-v2`

Multimodal embedding

900M

Text + image in shared space, 89 languages, and 8K text context, 512×512 images

Coming soon

Every model runs on a single NVIDIA L4 (24 GB), the most cost-efficient GPU tier on Google Cloud. Most other embedding models on Google Cloud Model Garden require an A100 80 GB or H100, roughly three times the per-hour instance cost before you even start counting tokens.

No additional commercial license is required when deployed through Vertex AI.

Why Model Garden?

Why deploy through Model Garden instead of hitting an API? It comes down to three things: control, cost, and context.

Your data never leaves the house

The biggest draw for most developers is the self-deploy architecture. When you deploy a Jina model through Model Garden, the weights run on GPU instances inside your own Google Cloud project and your own VPC. This is a game-changer for anyone working in industries with data security concerns, like finance or healthcare. Because there are no external API calls, your sensitive data stays within your security perimeter.

Scaling with prediction

Instead of paying every time you embed a sentence or rerank a document, you pay a flat hourly instance cost. And because every Jina model can run on a single NVIDIA L4, the most affordable GPU tier on Google Cloud, the barrier to entry is low. Whether you process a thousand requests or a billion, your infrastructure bill stays predictable. This is a setup that actually rewards you for growing your traffic rather than taxing you for it.

Everything under one roof

If your data is already sitting in Elasticsearch on Google Cloud, BigQuery, or Cloud Storage, it makes sense to keep your inference engines nearby. By deploying through Model Garden, Jina search foundation models inherit all the enterprise features you are already using: identity and access management (IAM) for access control, unified billing on your existing Google Cloud invoice, and the ability to plug into Vertex AI Pipelines for machine learning operations (MLOps) workflows.

While the Jina AI Cloud API and Elastic Cloud offer the fastest path for bursty traffic or existing search workflows, Model Garden is ideal for enterprise applications requiring strict data security and predictable costs at scale. Elastic wants to meet you where you are.

Jina AI models

jina-embeddings-v3

Our proven multilingual embedding model with 572M parameters and 8K token context. Scores 65.5 on Massive Text Embedding Benchmark (MTEB) English. Supports five task-specific Low-Rank Adaptation (LoRA) adapters (retrieval query/passage, text-matching, classification, clustering) and Matryoshka truncation from 1024 to 64 dimensions. Already widely adopted across the Elasticsearch ecosystem via EIS.

We’re leading with v3 because many production systems already depend on it. If you’re migrating a v3-based pipeline to Google Cloud, you can now run the same model natively without changing your embedding dimensions or reindexing.

jina-embeddings-v5-text (small and nano)

Our fifth-generation text embedding models, released February 2026, achieve top-tier performance, competing with models many times their size.

v5-text-small (677M) scores 67.0 on the Multilingual MTEB (MMTEB) benchmark suite, encompassing 131 tasks of nine task types, and 71.7 on the MTEB English benchmark. It’s the strongest sub-1B multilingual embedding model on the MTEB Leaderboard.

v5-text-nano (239M) scores 65.5 on MMTEB. No other model under 500M parameters reaches this level. At less than half the size of most comparable models, it’s the natural choice for edge and latency-sensitive deployments.

Both models support:

• Four task-specific LoRA adapters: Retrieval, text-matching, classification, clustering. Selecting an appropriate adapter via task parameter at inference time.

• Matryoshka dimension truncation: Reduce embedding dimensions from 1024 (or 768 for nano) down to 32. Quality loss is minimal at moderate truncation (for example, 256 dims). Halving dimensions roughly halves storage.

• Binary quantization: Compress 1024-dim embeddings from 2KB to 128 bytes with binarization. Special training makes this compression minimal losses.

• Multilingual: 119 languages (small) and 93 (nano).

jina-reranker-v3

A 0.6B parameter multilingual listwise reranker built using a last but not late interaction architecture. The query and up to 64 candidate matches are entered into a single 131K-token context window, and the model performs cross-document comparison before scoring. Jina Reranker v3 achieves 61.94 nDCG@10 on BEIR, outperforming the model being 6× smaller in size.This is fundamentally different from pointwise rerankers that score each document in isolation, producing better results, especially for passage retrieval from single documents.

jina-clip-v2

A 0.9B multimodal, multilingual embedding model that maps text and images into a shared 1024-dimensional space. It supports:

• 89 languages for text-image retrieval.

• 512×512 image resolution.

• 8K token text input.

• Matryoshka truncation from 1024 to 64 dimensions for both modalities.

Highly competitive on image-to-text benchmarks, including multilingual tasks.

Getting started

Jina Embeddings v3 is live on Model Garden today. Here’s how to get it running.

You need a Google Cloud project with the Vertex AI API enabled and enough GPU quota for at least one g2-standard-8 instance (NVIDIA L4). If you’re new to Google Cloud, start with the setup guide.

The Model Garden page for Jina Embeddings v3 walks you through the full flow: Upload the model, create an endpoint, pick your machine type, and deploy. Open it in your own project, and follow the guided steps. A100 and H100 machines are also available where region and quota allow, but L4 is all you need to start.

From click to first embedding, the whole process takes a few minutes.

What comes next

Jina Embeddings v3 is the starting point. In the coming weeks, we’ll bring the rest of the Jina retrieval stack to Model Garden: v5 text embeddings (small and nano), jina-reranker-v3, and jina-clip-v2 for multimodal search. All will run on a single L4 GPU with the same self-deploy model.

Jina AI, which recently joined Elastic via acquisition, is a leader in open-source multilingual and multimodal embeddings, rerankers, and small language models. Jina brings deep expertise in search foundation models that help developers build high-quality retrieval and RAG systems across text, images, code, and long multilingual content.

Elastic Inference Service (EIS) makes it easy for developers to add fast, high-quality, and reliable semantic retrieval for search, RAG, and agentic applications with predictable, consumption-based pricing. EIS offers fully-managed GPU-powered inference with ready-to-use models, no additional setup or hosting complexity required.

jina-embeddings-v3 supports high-quality multilingual retrieval with long-context reasoning and task-tuned modes for RAG and agents. It provides developers fast dense embeddings across a broad range of languages without the operational overhead.

Why jina-embeddings-v3?

jina-embeddings-v3 is a text embedding model that supports 32 languages and up to 8192-token context, high relevance at lower cost, and GPU-powered inference through EIS.

Key capabilities

• Multilinguality: Closes the language gap and aligns meaning across 32 languages, including Arabic, Bengali, Chinese, Danish, Dutch, English, Finnish, French, Georgian, German, Greek, Hindi, Indonesian, Italian, Japanese, Korean, Latvian, Norwegian, Polish, Portuguese, Romanian, Russian, Slovak, Spanish, Swedish, Thai, Turkish, Ukrainian, Urdu, and Vietnamese.

• Parameter-efficiency: Delivers higher performance with only 570M parameters, achieving comparable performance to much larger LLM-based embeddings at lower costs.

• Dimensionality control: Default in 1024 dimensions, and with Matryoshka representation support, it lets developers dial the embedding size all the way down to 32 dimensions, giving flexibility to balance accuracy, latency, and storage based on your needs.

• Task-specific optimization: Features task-specific Low-Rank Adaptation (LoRA) adapters, enabling it to generate high-quality embeddings for various tasks including query-document retrieval, clustering, classification, and text matching.

Get started

The response:

What’s next

Alongside these new models, EIS continues to evolve to support more users and simplify semantic search across environments.

Cloud Connect for EIS: Cloud Connect for EIS will soon bring EIS to self-managed environments, reducing operational overhead and enabling hybrid architectures and scaling where it works best for you.

semantic_text defaults to jina-embeddings-v3 on EIS: semantic_text currently uses ELSER as the embeddings model behind the scenes, but will default to the jina-embeddings-v3 endpoint on EIS in the near future. This change will provide built-in inference at ingestion time, making it easier to adopt multilingual search without additional configuration.

More models: We’re expanding the EIS model catalog to meet the rising inference demands of our customers. In the coming months, we’ll introduce new models that support an even broader set of search and inference workloads. Hot on the heels of jina-embeddings-v3, the next models to follow are jina-reranker-v2-base-multilingualand jina-reranker-v3. Both Jina AI models greatly improve precision through multilingual reranking for RAG and AI agents.

Conclusion

With jina-embeddings-v3 on EIS, you can build multilingual, high-precision retrieval pipelines without managing models, GPUs, or infrastructure. You get fast dense retrieval and tight integration with Elasticsearch’s relevance stack, all in one platform.

Whether you are building global RAG systems, search, or agentic workflows that need reliable context, Elastic now gives you a high-performance model out-of-the-box, and the operational simplicity to move from prototype to production with confidence.

All Elastic Cloud trials have access to the Elastic Inference Service. Try it now on Elastic Cloud Serverless and Elastic Cloud Hosted.