Why Nine Publishing rebuilt their infrastructure using search solutions

This post is a recap of a community talk given at a recent Elastic{ON} Tour event. Interested in seeing more talks like this? Check out the conference archive or find out when the Elastic{ON} Tour is coming to a city near you.

What does the revamped, modern technology stack of a 185-year-old Australian news corporation look like? In an industry that faces serious competition from anyone with a WordPress site and something to say, Nine Publishing, producer of Australia’s three largest news sites, sought to preserve a competitive edge by understanding how to enrich the experiences of their readers and journalists, while simultaneously identifying pain points and spotting problems before they happen. Driven by the challenges of reporting and monetizing news, Nine looked at their entire system — not just from a technological perspective, but from content management and business insights mindsets — to map out a new, modern infrastructure with the Elastic Stack.

The resulting architecture was conceived of and structured around search solutions, centralized application search, logging, and analytics. Using Elasticsearch, Beats, Logstash, and Kibana, Nine made metrics related to performance visible in one space and shared across teams.

Beyond search: Cost performance, business insights, and observability

Nine and its team started to layer Elastic solutions on top of a fairly standard content publishing architecture. Michael Lorant, principal systems engineer for Nine and a recent guest at Elastic{ON} Tour Sydney, led the construction of this new infrastructure. He and a few members of his team channeled a startup mentality, moving to a new, unfamiliar office to face their true greenfield environment. They leveraged Elastic products in three main ways:

• Search: Elasticsearch provided quick and relevant search results for readers querying an extensive archive of articles. On the back end, it helped journalists find the right media.

• Content analytics: Kibana gave journalists visible performance metrics, letting them know not only if the content was engaging, but also who exactly reads what content.

• Security and reporting: Centralizing the ingest and reporting of different types of logs opened an understanding of the way requests flowed through Nine’s apps, and how the devs and engineers could tweak or optimize their software.

Sometimes you have to disrupt yourself

Nine’s new system had to be flexible enough to update in shorter periods of time and agile enough to do daily releases (instead of every three weeks). And they needed to keep things lean, making sure that the new infrastructure was less costly. This led to best practices for node design and maintenance, which Lorant shared with the Elastic{ON} Tour Sydney audience. For example, to ingest data as varied as APM, cache hit rates, and request locations, Nine used Logstash to create small pipelines instead of one giant pipeline. By separating collection from the processing in a loosely coupled system, they were also able to simply reprocess a queue if something went wrong.

For more about the challenges Nine Publishing faced and how they responded (including an indispensable node maintenance tip), watch the full presentation.