Learn how the Perform Group used the Elastic Stack to make their user-facing APIs centralized and scalable.

<html><head></head><body><h2>About Perform Group</h2>Perform Group is a sports media company based in London that offers a broad portfolio of products for both B2B and B2C clients. Perform Group&apos;s direct-to-consumer brands include the OTT (Over The Top) streaming service DAZN, websites such as goal.com and Sporting News, and the advertising-supported video-on-demand service ePlayer. In the B2B world, Perform Group is a major provider of sports data through its Opta &amp; RunningBall brands, operates a sports news agency called Omnisport, and provides live video streaming services to the largest bookmakers in the betting industry via its Watch &amp; Bet product.
<h2>Background</h2>As the company grew over the years, we produced a variety of external-facing web-pull APIs that were specific to each of our clients or products. The result of this was a set of custom APIs that had the following drawbacks:
<ul>
	<li>No centralised control of the distribution of content</li>
	<li>Inconsistencies in function, data structure and interface</li>
	<li>Varying methods of security</li>
	<li>Inflexibility</li>
	<li>Lack of re-usability</li>
	<li>Lack of scalability</li>
	<li>Difficult maintenance</li>
	<li>An architecture with an unscalable, single-point-of-failure: an Oracle RDBMS</li>
</ul>The above situation meant that we didn&apos;t have the scalability to support our rapidly growing user-base and highly variable load generated by servicing the online sports industry. Also, the inflexibility of these custom APIs meant that server-side development was often required for even minor features, and the lack of re-usability resulted in a lot of duplicated effort. These factors slowed our time-to-market for new products and features,&#xA0;resulting in financial implications and reputational damage to Perform Group.
To address this, and to satisfy the company&apos;s future ambitions, we decided to revamp our web-pull API strategy with the following objectives
<ul>
	<li>A consistent set of end-points that followed a common design pattern (ReST)</li>
	<li>A consistent architecture</li>
	<li>Centralised and detailed control of content distribution</li>
	<li>Consistent usage and appearance</li>
	<li>High performance and flexibility</li>
	<li>Horizontal scalability</li>
	<li>Resilience</li>
	<li>Security</li>
	<li>Deployability in a variety of hosting environments, both on-premise and in the cloud.</li>
</ul>The revamped API strategy was referred to as Perform Feeds. These feeds would need to support a variety of query types from our diverse suite of products and services, including queries by date range, wildcarded free-text searches, keyword search, and search by ID for specific items, with responses being available in a range of formats, such as XML, JSON and RSS. Today, Perform Feeds form the backbone of every product that Perform Group offers to its numerous B2B and B2C clients
<h2>Architecture</h2>To meet the non-functional requirements of scalability, resilience and portability, while offering the required functional flexibility, we quickly realised that our existing architecture, which relied on synchronous requests to a single on-premise Oracle database, was no longer suitable.
Due to our investment in the technology, Oracle was to remain as the single point of truth for the data that we create in our content management system (CMS). However, we wanted implementations of the Perform Feeds APIs to be decoupled from this database, and instead be backed by a data store that would satisfy both the functional and non-functional requirements mentioned above. Back in 2012, we tested and prototyped several candidate technologies, with Elasticsearch (release 0.20 at the time) coming out on top for search functionality, performance and horizontal scalability.
The Perform Feeds APIs were implemented in Java as standard web applications and deployed to Tomcat application servers with an Elasticsearch cluster deployed to the same data centre to provide the data store. Communication between the web applications and Elasticsearch was via the transport client.
At this point, the missing piece was the mechanism to populate Elasticsearch from Oracle. To implement this, we used RabbitMQ to notify us whenever an item was changed in the Oracle database. Queue consumer components were developed to receive these notification queue messages, query Oracle to obtain additional metadata related to the item, denormalise the data, then build an HTTP PUT, POST or DELETE request to the relevant Perform Feeds end-point. The implementation of the PUT, POST and DELETE end-points would then execute the corresponding Elasticsearch index or delete operation.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0ba59e988f7aec63/5c6c47ef2b22f9f161d6d356/image1.png" data-sys-asset-uid="blt0ba59e988f7aec63" alt="" style="display: block; margin: auto;">
Our initial deployment in late 2012 was limited to one feed of editorial data, but we quickly expanded in subsequent months to offer feeds for video-on-demand meta-data, live streaming metadata, and sports fixtures. After Perform Group&apos;s acquisition of Opta in 2013, Perform Feeds APIs were developed to provide our clients with detailed Opta sports data, then in 2015 a major new version of our ePlayer VOD product was released, which was 100% dependent on Perform Feeds APIs. In 2016, Perform Group&apos;s new direct-to-consumer streaming service, DAZN, which is fundamentally dependent on Perform Feeds, launched in Germany, Austria and Switzerland. More recently, our goal.com and Sporting News websites have been rebuilt to obtain their content via Perform Feeds, and a new Perform Feeds-based version of our betting industry live streaming product, Watch &amp; Bet, is currently being rolled out to early adopter clients.
As the use of Perform Feeds has grown, we&apos;ve scaled up the number of Elasticsearch clusters to meet demand. At the time of writing (late 2018), we have 8 clusters, ranging in size from 3 to 11 data nodes. Each data node runs on a virtual server with 32GB of RAM, 300GB SSD disk and an 8-way CPU. We&apos;ve kept pace with new Elasticsearch releases, making the jump from 0.20 to 1.3, then to 5.2 and 5.6 at the appropriate times. Our next planned migration is to 6.x in the coming months.
On a typical weekend in the European football season, Perform Feeds serves over 100 million client HTTP GET requests, each of which generates a corresponding Elasticsearch query; and over 17 million PUT, POST or DELETE requests, each of which results in an index operation. Perform Feeds has 99.99% availability over the past 12 months, with 95% of GET responses being served in under 100 milliseconds. Our biggest Perform Feeds Elasticsearch index contains nearly 300 million documents. This level of performance has only been possible because of the features and scalability offered by Elasticsearch.
As you can see from the above, Perform Feeds APIs, are now fundamental to our business, with at least 8 different development teams in 3 separate locations working on enhancing and maintaining the feeds.
<h2>Beyond Perform Feeds</h2>We also use the Elastic Stack to perform log aggregation from all parts of our platform, with logs being collected from nearly 250 different servers and stored in an Elasticsearch cluster with a hot-warm architecture. This has proved to be a real time-saver for our support teams, making problem resolution far easier than before. The log aggregation architecture includes Logstash and Beats to gather the log data, RabbitMQ to propagate it to Elasticsearch, and Kibana to visualise it.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta962e1544a97b739/5c6c47d2763b502362e5007e/image2.png" data-sys-asset-uid="blta962e1544a97b739" alt="" style="display: block; margin: auto;">
We&apos;re continually investigating how we might use the Elastic Stack in other ways. Currently under consideration is another log analysis use case, involving the generation of aggregated reports of live stream usage data. We&apos;re also evaluating the Elastic Stack for a security analysis use case, and machine learning for identifying anomalous load patterns on our servers. </body></html>

perform_banner.png

Perform-Group-Logo.png

Perform Feeds: Using the Elastic Stack to Improve the API Client Experience

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Jeremy Kivell

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

제출하면 귀하가 <a href="/kr/legal/elastic-cloud-account-terms" target="_self">Elastic 서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 연락처로 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">선택한 커뮤니케이션</a>을 수신하는 데 동의한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/serverless-preview-terms" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

제출하면 <a href="/kr/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 서비스 약관</a>에 동의하시게 됩니다. 개인 데이터는 <a href="/kr/legal/privacy-statement">Elastic의 개인정보 취급방침</a>에 따라 처리됩니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/terms-of-use" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

이 페이지의 콘텐츠는 선택하신 언어로 제공되지 않습니다. Elastic은 다양한 언어로 콘텐츠를 제공하기 위해 최선을 다하고 있습니다.조금만 더 기다려주세요!

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

내 표준 시간대로 웨비나 시작 시간 확인하기

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

참여가 어려우신가요? 등록하시면 녹화 자료를 보내드리겠습니다. 이메일로 관련 콘텐츠도 받아보실 수 있습니다.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

You'll also receive an email with related content

이메일을 통해 관련 내용을 보내드리겠습니다.

Thank you for registering. We will send you a confirmation email soon.

등록해 주셔서 감사합니다. 곧 확인 이메일을 보내드리겠습니다.

Author

Articles by Jeremy Kivell

Perform Feeds: Using the Elastic Stack to Improve the API Client Experience

Sign up for Elastic Cloud free trial