We compared production traffic before and after the move and ran controlled tests for cases production can't isolate. Latency percentiles describe the slow end of response times: At the 99th percentile, 99 of every 100 requests finished faster than the value shown. Slow requests are the ones that users notice.

Workload	Test	Metric	NVIDIA L4 Tensor Core GPU	NVIDIA RTXPRO 6000	Change
Embedding	Production	99th-percentile latency	5.86 s	0.73 s	~8x
Reranking	Production with about 40% more average request volume	90th-percentile latency	732 ms	171 ms	4.3x
Large reranking job	Controlled test; same software	Completion time	702 ms (three L4s)	160 ms (one GPU)	4.4x
Small reranking requests alongside large jobs	Mixed-traffic test on a shared GPU pool	Latency	708 ms	66 ms	10.7x
Document OCR (jina-ocr-v1)	Same set of real document pages	Time to first output token	0.50 s	0.20 s	2.5x
Document OCR (jina-ocr-v1)	Same set of real document pages	Page conversion speed	454 pages/hr	1,671 pages/hr	3.7x faster

Running small models doesn't, by itself, simplify capacity planning for a shared service. The workload mix matters as much as the model size in deciding how much GPU capacity is needed. On NVIDIA Pro 6000 GPUs, small requests stayed fast while large jobs ran beside them. For our customers, that means a query doesn't slow down because a big job is running on the same service.

To learn more about Jina models in Elastic Inference Service, check out the documentation. Try out the Jina API.

Context engineering

Vector database

Search powered applications

Logs

Threat protection

Workflows

Elasticsearch

Kibana (Discover, Dashboards)

Elastic Agent Builder

AutoOps

Piped query language

Jina AI search models

Elastic Cloud Serverless

Elastic Cloud Hosted

Self-managed Elasticsearch

Ecommerce search

Customer support search

Search-driven apps

Log analytics

Infrastructure monitoring

Digital experience monitoring

App performance monitoring

AI SRE

LLM observability

SIEM

Agentic SOAR

XDR and endpoint security

AI for security

10x your data's value

Cloud providers

Elastic AI Ecosystem

Search AI Partner Program

AV-Comparatives

Forrester Wave™ XDR

2026 Gartner® Magic Quadrant™

IDC MarketScape

Search

Security

Observability

Get started

Demo gallery

Downloads

Integrations

Docs

Elasticsearch Labs

Elastic Security Labs

Elastic Observability Labs

Blog

Community

Events

Webinars

Discuss

Training

Support

Consulting

Author

Articles by Gregory Chase

Principal Product Marketing Manager, Jina

Elastic looks to speed up Jina Model Inference by 8x using NVIDIA RTX GPUs

Sign up for Elastic Cloud free trial