How RTL Germany tripled retention with Elastic Observability's frozen tier, with zero downtime in the cutover

RTL Germany reimagined its centralized data lake, moving from 30-day to 90-day standard retention, cutting node count nearly in half, and finishing the migration with zero data loss.

RTL Germany runs a company-wide Elastic data lake that has grown for nearly a decade into the standard platform for logging, application data, and business intelligence across departments. A 100 TB on-premises storage cap had pinned standard retention at roughly 30 days, which blocked teams that wanted to look at how KPIs evolved over months and forced compliance data to live in separate S3 buckets outside the searchable lake. After upgrading from Platinum to Enterprise and redesigning around searchable snapshots and the frozen tier, standard retention tripled to 90 days, combined capacity grew to 235 TB on the same licensed footprint, the hypervisor footprint dropped from 18 to 10 physical nodes, and the entire cutover finished with zero downtime and zero data loss.

RTL Germany's Elastic data lake covers nearly a decade of company-wide logging, application data, and business intelligence across the broadcasting and media business. By 2024, the lake was hitting a wall. Roughly 100 TB of on-premises storage capped standard retention at about 30 days, but several departments needed months of history to track how KPIs evolved over time, and compliance data subject to a six-month retention expectation under internal policy was living outside the searchable platform in separate S3 buckets. After upgrading from Platinum to Enterprise and redesigning around searchable snapshots and the frozen tier, RTL Germany tripled standard retention to 90 days, combined capacity grew to 235 TB on the same licensed footprint, reduced the hypervisor footprint from 18 to 10 physical nodes, and the entire cutover finished with zero downtime and zero data loss.

What 30 days blocks is not just longer history but a class of questions: quarter-over-quarter comparisons, year-over-year seasonal cycles, baselines long enough to make a trend line credible. For RTL Germany's BI users, the lake had the operational data but stopped well short of where the analytical questions started. With 90 days of history standard and a full year on some indices, that class of question can now be answered inside the same data lake the operational teams are watching live, against the same data, without exporting it somewhere else.

That outcome rests on two distinct transitions over the platform's lifetime: a 2023 consolidation onto Elastic after a corporate merger brought in a second logging platform, and the later license upgrade that reshaped the storage architecture so retention could triple without growing the licensed footprint.

RTL Germany's Elastic environment started as a shared infrastructure tool nearly a decade ago and grew, team by team, into the company-wide data lake. Two events shaped the platform as it stands today.

The first was a post-merger consolidation in 2023. A newly joined company entity arrived running a different logging platform. RTL Germany's Elastic environment was already on premises with capacity to spare, and running two platforms in parallel did not make financial or operational sense. The migration off the legacy logging tool ran about five months, with the heaviest lift being communication, rebuilding the dashboards and notification workflows the team had built up over years, and reworking ingest pipelines.

The second was the storage ceiling, and the path out of it. After seeing searchable snapshots and the frozen tier at Elastic{ON} Munich, the team saw a design that matched the storage problem directly: Keep the licensed footprint, move long-tail data to S3-backed frozen storage, and use the freed hot-tier headroom to extend retention. Before committing, the team built a proof of concept focused on the single question that mattered: How fast would queries run against older data on the frozen tier? The result was fast enough that the decision to redesign the whole cluster from hot-and-cold to hot-and-frozen was straightforward.

Before the redesign, RTL Germany's standard retention was capped at roughly 30 days by 100 TB of on-premises storage. The data lake was healthy, but three operational realities were pushing against the ceiling at once.

Business intelligence needs ran past 30 days. Some departments wanted to track how KPIs evolved over time. Thirty days of history was not enough to do that work in the data lake. Teams had the data they needed for operations, but not the window they needed for analysis.

Compliance data lived outside the lake. Firewall data subject to a six-month retention expectation from the internal policy was stored in separate S3 buckets, not in the searchable platform. The data was retained, but accessing it during an investigation meant pulling it out of object storage and loading it into something searchable, which the team described as a pretty big pain in any real case.

Hardware was aging. The cluster was running on servers that were roughly five or six years old, with standard SSDs rather than NVMe. The team knew the hardware refresh was coming whether or not the architecture changed.

The team's time was going into keeping the existing platform healthy at the cap, not into extending what it could do for the rest of the business.

The team upgraded from Platinum to Enterprise to unlock searchable snapshots and the frozen tier, and then redesigned the storage layout from a hot-and-cold model to hot-and-frozen, backed by S3.

The end-to-end data flow stayed familiar. Logstash nodes ingest from data sources across the organization and write into Elastic. Hot indices live on local NVMe SSDs on the new cluster. Older indices move to the frozen tier, where they are stored in S3 and pulled into a local cache on demand when queried. Cross-cluster search bridged the old and new clusters during the cutover so users could reach historical data without switching environments.

The performance tradeoff proved manageable. The first query against an index in the frozen tier takes roughly 30 seconds while the relevant data loads into the frozen nodes' cache. After that, queries against the same data run at hot-tier speed.

• Hot-tier query speed for current data: 35 TB of NVMe SSD on newer servers, running on KVM.
• Months-to-years of retention at low cost: 200 TB of S3-backed frozen storage, served via searchable snapshots.
• Tripled capacity on the same licensed footprint: 235 TB combined (35 TB hot plus 200 TB frozen), no increase in licensed resource units.
• Smaller, faster cluster: Hypervisor footprint reduced from 18 to 10 physical nodes, on newer hardware with more cores per node.
• Workload-by-workload cutover with no pipeline rearchitecture: Logstash continued to handle ingestion; destinations were redirected per data source during the migration.
• No loss of access to historical data mid-migration: Cross-cluster search active throughout, so users kept reading older indices on the legacy cluster while new data flowed to the new one.
• Hot-tier speed against older data after the first query: ~30 seconds for frozen-tier cache hydration on the initial query; subsequent queries run at hot-tier speed.

Rather than a big-bang switch, the team stood up the new cluster alongside the old one and migrated workload by workload, starting with the largest data sources. Because ingestion runs through Logstash, moving a workload was often as simple as changing the destination on the Logstash node from the old cluster's master to the new one. Cross-cluster search kept users in the new environment with read access to data still on the old cluster, so historical data stayed reachable throughout.

The team also took the opportunity to migrate custom ingest pipelines over to Elastic's maintained integrations where possible, which they describe as the better path for the future.

Elastic support extended the overlap period when hardware shipments slipped, which the team called out specifically as one of the things that made the timeline work.

The proof of concept itself took about two weeks. Most of the six to eight week PoC timeline went to procuring and provisioning hardware, not to testing. The full cutover ran about three months and finished without downtime or data loss.

For the departments that use the data lake day to day, the change was nearly invisible. Users had to point their bookmarks at the new URL during the temporary overlap and switch back to the standard domain after the cutover. Cross-cluster search kept historical data reachable from the new cluster throughout, so no one had to learn a new workflow or query two systems to find old data.

On the compliance side, the data that previously lived in separate S3 buckets now sits inside the searchable data lake. Six-month firewall retention is no longer a workflow problem. If a department needs to look back across the full retention window, they query the same way they would for anything else in the lake.

A few practical lessons stood out from the project.

Use managed integrations where possible. RTL Germany had years of custom ingest pipelines and migrated many to Elastic's maintained integrations during the cutover. The team described this as a better path for the future, not just for this project.

Treat a migration as a facelift. Replacing aging hardware, modernizing the architecture, and revisiting integrations at the same time multiplied the value of the project versus doing any one of them alone.

Evaluate the hosting model honestly. On-premises made sense given RTL Germany's existing resources and commercial considerations. For smaller organizations without the in-house capacity to stand up and run a cluster, the team would point toward Elastic's SaaS option to skip the operational overhead.

The hard part is volume, not complexity. Redirecting ingestion from one cluster to another was straightforward. The real effort was the sheer number of pipelines and the scale of data that had to pass through that simple change.

Two outcomes shape what comes next. The first: Cut the time it takes new team members to find the right data view, by connecting the Elastic AI Agent to RTL Germany's internal large language models (LLMs). This will help newer team members navigate the platform and find the right data view faster, which has historically been one of the harder onboarding steps.

The second: Absorb even more application logs into the data lake, without revisiting the licensing footprint. This is an internal program that will onboard even more application logs from every application into the data lake. The new architecture was deliberately designed with this growth in mind. With 200 TB of S3-backed capacity and the ability to extend that storage easily, the team has room to absorb the volume without revisiting the licensing footprint.

Your organization may not be running a decade-old data lake covering an entire broadcasting and media business today, but the same principles apply whether you are starting with a few terabytes of log data or scaling toward a 235 TB hot-and-frozen architecture: The licensed footprint does not have to grow with the retention window.

RTL Germany is part of the RTL Group, the largest broadcasting and media company in Germany, with operations spanning television, streaming, and content production.

See how Elastic Observability's frozen tier extends retention without adding cost, or start now with a free trial.

• Elastic Observability
• Log monitoring
• Elasticsearch
• Logstash
• Data integrations
• Observability Labs
• Elastic documentation

Topics: Elastic Observability, Elasticsearch, Logstash, frozen tier, searchable snapshots, tiered storage, cross-cluster search, log analytics, data lake, compliance data retention, media and broadcasting, Media & Entertainment