Learn how the Perform Group used the Elastic Stack to make their user-facing APIs centralized and scalable.

<html><head></head><body><h2>About Perform Group</h2>Perform Group is a sports media company based in London that offers a broad portfolio of products for both B2B and B2C clients. Perform Group&apos;s direct-to-consumer brands include the OTT (Over The Top) streaming service DAZN, websites such as goal.com and Sporting News, and the advertising-supported video-on-demand service ePlayer. In the B2B world, Perform Group is a major provider of sports data through its Opta &amp; RunningBall brands, operates a sports news agency called Omnisport, and provides live video streaming services to the largest bookmakers in the betting industry via its Watch &amp; Bet product.
<h2>Background</h2>As the company grew over the years, we produced a variety of external-facing web-pull APIs that were specific to each of our clients or products. The result of this was a set of custom APIs that had the following drawbacks:
<ul>
	<li>No centralised control of the distribution of content</li>
	<li>Inconsistencies in function, data structure and interface</li>
	<li>Varying methods of security</li>
	<li>Inflexibility</li>
	<li>Lack of re-usability</li>
	<li>Lack of scalability</li>
	<li>Difficult maintenance</li>
	<li>An architecture with an unscalable, single-point-of-failure: an Oracle RDBMS</li>
</ul>The above situation meant that we didn&apos;t have the scalability to support our rapidly growing user-base and highly variable load generated by servicing the online sports industry. Also, the inflexibility of these custom APIs meant that server-side development was often required for even minor features, and the lack of re-usability resulted in a lot of duplicated effort. These factors slowed our time-to-market for new products and features,&#xA0;resulting in financial implications and reputational damage to Perform Group.
To address this, and to satisfy the company&apos;s future ambitions, we decided to revamp our web-pull API strategy with the following objectives
<ul>
	<li>A consistent set of end-points that followed a common design pattern (ReST)</li>
	<li>A consistent architecture</li>
	<li>Centralised and detailed control of content distribution</li>
	<li>Consistent usage and appearance</li>
	<li>High performance and flexibility</li>
	<li>Horizontal scalability</li>
	<li>Resilience</li>
	<li>Security</li>
	<li>Deployability in a variety of hosting environments, both on-premise and in the cloud.</li>
</ul>The revamped API strategy was referred to as Perform Feeds. These feeds would need to support a variety of query types from our diverse suite of products and services, including queries by date range, wildcarded free-text searches, keyword search, and search by ID for specific items, with responses being available in a range of formats, such as XML, JSON and RSS. Today, Perform Feeds form the backbone of every product that Perform Group offers to its numerous B2B and B2C clients
<h2>Architecture</h2>To meet the non-functional requirements of scalability, resilience and portability, while offering the required functional flexibility, we quickly realised that our existing architecture, which relied on synchronous requests to a single on-premise Oracle database, was no longer suitable.
Due to our investment in the technology, Oracle was to remain as the single point of truth for the data that we create in our content management system (CMS). However, we wanted implementations of the Perform Feeds APIs to be decoupled from this database, and instead be backed by a data store that would satisfy both the functional and non-functional requirements mentioned above. Back in 2012, we tested and prototyped several candidate technologies, with Elasticsearch (release 0.20 at the time) coming out on top for search functionality, performance and horizontal scalability.
The Perform Feeds APIs were implemented in Java as standard web applications and deployed to Tomcat application servers with an Elasticsearch cluster deployed to the same data centre to provide the data store. Communication between the web applications and Elasticsearch was via the transport client.
At this point, the missing piece was the mechanism to populate Elasticsearch from Oracle. To implement this, we used RabbitMQ to notify us whenever an item was changed in the Oracle database. Queue consumer components were developed to receive these notification queue messages, query Oracle to obtain additional metadata related to the item, denormalise the data, then build an HTTP PUT, POST or DELETE request to the relevant Perform Feeds end-point. The implementation of the PUT, POST and DELETE end-points would then execute the corresponding Elasticsearch index or delete operation.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0ba59e988f7aec63/5c6c47ef2b22f9f161d6d356/image1.png" data-sys-asset-uid="blt0ba59e988f7aec63" alt="" style="display: block; margin: auto;">
Our initial deployment in late 2012 was limited to one feed of editorial data, but we quickly expanded in subsequent months to offer feeds for video-on-demand meta-data, live streaming metadata, and sports fixtures. After Perform Group&apos;s acquisition of Opta in 2013, Perform Feeds APIs were developed to provide our clients with detailed Opta sports data, then in 2015 a major new version of our ePlayer VOD product was released, which was 100% dependent on Perform Feeds APIs. In 2016, Perform Group&apos;s new direct-to-consumer streaming service, DAZN, which is fundamentally dependent on Perform Feeds, launched in Germany, Austria and Switzerland. More recently, our goal.com and Sporting News websites have been rebuilt to obtain their content via Perform Feeds, and a new Perform Feeds-based version of our betting industry live streaming product, Watch &amp; Bet, is currently being rolled out to early adopter clients.
As the use of Perform Feeds has grown, we&apos;ve scaled up the number of Elasticsearch clusters to meet demand. At the time of writing (late 2018), we have 8 clusters, ranging in size from 3 to 11 data nodes. Each data node runs on a virtual server with 32GB of RAM, 300GB SSD disk and an 8-way CPU. We&apos;ve kept pace with new Elasticsearch releases, making the jump from 0.20 to 1.3, then to 5.2 and 5.6 at the appropriate times. Our next planned migration is to 6.x in the coming months.
On a typical weekend in the European football season, Perform Feeds serves over 100 million client HTTP GET requests, each of which generates a corresponding Elasticsearch query; and over 17 million PUT, POST or DELETE requests, each of which results in an index operation. Perform Feeds has 99.99% availability over the past 12 months, with 95% of GET responses being served in under 100 milliseconds. Our biggest Perform Feeds Elasticsearch index contains nearly 300 million documents. This level of performance has only been possible because of the features and scalability offered by Elasticsearch.
As you can see from the above, Perform Feeds APIs, are now fundamental to our business, with at least 8 different development teams in 3 separate locations working on enhancing and maintaining the feeds.
<h2>Beyond Perform Feeds</h2>We also use the Elastic Stack to perform log aggregation from all parts of our platform, with logs being collected from nearly 250 different servers and stored in an Elasticsearch cluster with a hot-warm architecture. This has proved to be a real time-saver for our support teams, making problem resolution far easier than before. The log aggregation architecture includes Logstash and Beats to gather the log data, RabbitMQ to propagate it to Elasticsearch, and Kibana to visualise it.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta962e1544a97b739/5c6c47d2763b502362e5007e/image2.png" data-sys-asset-uid="blta962e1544a97b739" alt="" style="display: block; margin: auto;">
We&apos;re continually investigating how we might use the Elastic Stack in other ways. Currently under consideration is another log analysis use case, involving the generation of aggregated reports of live stream usage data. We&apos;re also evaluating the Elastic Stack for a security analysis use case, and machine learning for identifying anomalous load patterns on our servers. </body></html>

perform_banner.png

Perform-Group-Logo.png

Perform Feeds: Using the Elastic Stack to Improve the API Client Experience

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Jeremy Kivell

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

O conteúdo desta página não está disponível no idioma selecionado. A Elastic está trabalhando para garantir que o conteúdo esteja disponível em vários idiomas. Agradecemos a compreensão.

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/elastic-cloud-account-terms" target="_self">Termos de Serviço</a> e também concorda em receber <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">comunicações especiais</a> por meio dos dados de contato que forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/serverless-preview-terms" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Ao enviar, você concorda com os <a href="/pt/agreements/global/cloud-services-monthly" rel="nofollow">Termos de Serviço do Elastic Cloud</a>. Seus dados pessoais serão tratados de acordo com a <a href="/pt/legal/privacy-statement">Declaração de Privacidade da Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/terms-of-use" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

Explore similar demos

Explore outras demonstrações semelhantes

Overview

Load More

Speakers

Close

See more insights

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Consultar a hora de início do webinar no meu fuso horário

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Não vai conseguir participar? Inscreva-se e nós lhe enviaremos a gravação. Você também receberá um email com conteúdo relacionado.

Author

Articles by Jeremy Kivell

Perform Feeds: Using the Elastic Stack to Improve the API Client Experience

Sign up for Elastic Cloud free trial