Powering ITV’s DevOps engine with the Elastic stack
ITV is the oldest commercial TV network in the United Kingdom. It is an integrated producer broadcaster, operating the largest commercial family of channels in the UK. In addition to traditional broadcasting on our channels, we deliver our content on demand through numerous platforms, both directly and via the ITV Hub. Through ITV Studios we produce content for both our own channels and third parties in the UK and increasingly overseas. In our team we focus in the infrastructure behind the non aerial distribution of content to the various platforms. The Elastic products power core functions like searching & logging in our technology stacks.
At ITV we have several legacy products and several teams working on them. At the same time the teams are trying to deliver new and exciting products. Those systems emit logs to standard locations and designing a logging system was mostly a task for the product itself. There were different teams acting in a SILO mode, delivering a solution that worked only for them. We wanted a solution that works for all our products and different teams can easily inherit.
This is the solution we came up with at ITV:
- Create a Common Platform team
- Build a culture
- Set an objective
Our objective was simple:
Design, build and operate the next-generation hosting platform for ITV’s modernised applications. Based upon a standard set of tools and techniques it will deliver high quality, low cost environments to our customers.
One of the problems we tried to solve was around logging. Logging is one of the things that is constantly changing along with the product life. Traditionally we used to store logs in the file system and create reporting applications along with custom dashboards & interfaces for both development and management teams to search the logs. The problem with that approach is that it’s not repeatable, it’s not common and it is difficult to develop additional dashboards. We don’t want to SSH into servers to search application logs in 2015. We want to surface information to a higher level interface and give access to users through dashboards to search logs.
The Elastic Stack to the Rescue
We started using the Elastic technology stack 1.5 years ago and recently we made the decision to put the full Elastic stack deep in our Common Platform specification as the standard logging mechanism for all ITV products hosted by the platform. We tried and tested every component. It had to play nicely with our configuration management and our cloud environment before we adopted it. Elasticsearch’s ability to index & search million of log entries daily was impressive. The ability to create dashboards and share them using Kibana was unbelievably easy compared to the past. Finally, Logstash convinced us handling multiple sources with so many filter capabilities. All of that together lead to the fact that the Elastic stack won a thumbs up by our operations team.
Using the Elastic Stack in our Brand New ITV HUB
ITV has replaced both ITV Player and ITV.COM with a new destination for catch-up content and online services, the ITV HUB. With over 11 million registered users and 726 million long form requests last year we have a web platform that generates lots of logs. We use the Elastic stack to create useful dashboards for Ops & Devs to display operation metrics. Our Elasticsearch instances live on AWS and already handle millions of logs every day. Logstash ability to handle logs from multiple sources allow us to handle AWS logs such as Cloudtrail & ELB logs in the same Elastic stack. Elasticsearch’s curator keeps things tidy and everything is managed through our configuration management in a scalable solution.
Below we have an example for some useful dashboard’s created by our developers to help them debug production issues:
Fig. 1: Watching live website traffic and performing real time search queries is very easy. An operation available only to the Ops team.
Fig. 2: Visualising the data has never been easier. Created an error reporting dashboard for our operation team in just a minute. The Elastic stack increased visibility into our infrastructure by far!
ITV Hub is only one of the many products within ITV that uses the Elastic stack. We already started building other products based on Kibana 4. Also we want to try the new Elasticsearch 2.0 and the time series data store as a potential replacement of our existing data store. It looks very promising and some of the early demos are exciting! Our core logging system is powered by the Elastic stack so all our product teams directly benefit from having a logging solution covered right at the beginning of the life of a product. System & Application dashboards really tear down the wall between Dev & Ops and the logs stream very smooth with the Elastic stack :)
Efstathios Xagoraris is a Platform Engineer at ITV. Before joining ITV he was working as a Unix Administrator in Greece; where he is from. His experience comes from working in large ISP's, universities & startups. He joined ITV in 2013 and since then he has been involved in core modernisation projects for the company. Since starting his role at ITV, he has been working with Elastic products and was the first to adopt the Elastic stack for ITV, moving it all the way to production.