This post is a recap of a community talk given at a recent <a href="/elasticon">Elastic{ON}</a>&nbsp;Tour event. Interested in seeing more talks like this? Check out the <a href="/elasticon/tour/2019">Elastic{ON} Tour</a>&nbsp;page to see when a stop is coming to a city near you.Bell Canada, one of Canada’s largest telecommunications companies, offers mobile phone, television, internet, and landline services to big corporations, small and medium-sized businesses, and individuals across the country. Bell Canada’s security operations center (SOC) covers every Bell office and business unit coast to coast and they rely on logs to detect cybersecurity threats.Sylvain Proulx, Bell Canada’s Senior Security Manager, says the business units — like Bell TV, Bell Internet, Bell Media, or Bell Mobility — that deliver services to their customers all use different technologies and applications, so the logs they collect are diverse and uncommon. Logs come from routers, firewalls, web logs, OS logs, application logs, and many other devices, some of which get ‘chatty’ and generate a lot of data.The SOC had performed log and event correlation and incident response and reporting using only an ArcSight Security Information and Event Management (SIEM) solution. But over time, as the volume of logs increased, normalizing many new types of logs from a variety of devices bogged down the system. Their SIEM solution also provided only rule-based detection with no machine learning, so it generated a high ratio of false-positive incidents, which threatened to alert-fatigue their analysts.Proulx said they’d hit their SIEM’s limit. They found no single vendor solution that would let them ingest more data faster, build threat detection models, and normalize many new types of logs while also retaining ownership of their data. So, the SOC got to work augmenting their ArcSight SIEM with tools like the Elastic Stack to handle high log volume and traffic spikes automatically and generate meaningful security data that wouldn’t overwhelm analysts.Bell Canada gets data from bare metal servers, virtual machines, and, increasingly, from container infrastructure&nbsp;with&nbsp;Docker and Kubernetes. They needed a log shipper that was simple, lightweight, and straightforward to automate, so they turned to <a href="/products/beats">Beats</a>. They use <a href="/products/beats/filebeat">Filebeat</a> and <a href="/products/beats/winlogbeat">Winlogbeat</a> to ship logs because they’re easy to configure, test, and deploy. Plus, they can version control their configurations and there is no loss of data in case of a network outage.After the data is queued in Kafka, the SOC must parse and normalize their logs in all their various formats in order to perform security analysis. Running <a href="/products/logstash">Logstash</a> instances on OpenShift has helped them scale quickly and automatically in case of traffic spikes without dropping logs, and it consumes less resources than multiple virtual machines. An additional advantage they’ve found to having Logstash in a container is that they can easily run it through RSpec for testing before moving to production.Once the logs are normalized, the SOC stores them in <a href="/products/elasticsearch">Elasticsearch</a>. Bell Canada’s previous solution was unable to handle increasing log volumes and scale without losing logs. The SOC now does this with Elasticsearch, which allows them to scale quickly and horizontally, making&nbsp;their job a lot easier.The day events are logged, the SOC searches the data with multiple queries and processes, which puts a heavy load on the cluster, so they’ve implemented a hot-warm architecture with automated deployment of new nodes. The beefier nodes are ingesting and being searched constantly, but when the logs lose their value, they’re shipped to warm nodes for aggregation and lighter analysis. “If you lose a node in Elasticsearch, you can still keep working. Not a problem. You can fix it later,” says Mathew Vandystadt, Bell Canada’s Security Specialist Software Engineer.Securing data is a top priority for the SOC. Role-based access control (RBAC) is a must, but RBAC can be painful to manage. With the <a href="/products/stack/security">security features of the Elastic Stack</a>, the SOC has control over who has access to the data, they can add a layer of encryption over data transportation, and they can easily perform RBAC management — and it also ties in easily with their existing LDAP, meaning they don’t need to spend extra time redefining group roles and can focus on their security mandate.Once they have their data where they want it, the SOC analysts can use it to find security incidents — and good visualization is key. <a href="/products/kibana">Kibana</a>’s straightforward interface means their busy specialists don’t spend a lot of extra time learning new query languages.<blockquote>“[Kibana] works great in our use cases. It’s simple. We don’t have to do a lot of training and we get a nice visualization.” Mathew Vandystadt, Security Specialist Software Engineer | Bell Canada</blockquote>Bell Canada found a flexible alerting solution for rule-based detection, but their analysts also needed smart detection that works with different algorithms, so they developed in-house machine learning with open source, ML-centric libraries. Their containers in OpenShift let them easily spin Python containers tied to Kafka or Elasticsearch so all data is accessible. Then they use their ArcSight SIEM for event aggregation and correlation to get a higher ratio of true positives to false positives, sparing analysts from alert bombardment.They’ve built this whole pipeline using different software from different vendors, made possible because Elastic allows for simple integration with open security protocols. In the future, the SOC plans to merge a Cyber <a href="https://www.elastic.co/security/tip" target="_self">Threat Intelligence platform</a> with their new security architecture. “Having that infrastructure right there allows us to do more than we were a year and a half ago with only the ArcSight solution,” says Proulx.Ready to learn more about the team’s tools and pipeline? <a href="/elasticon/tour/2018/toronto/security-events-logging-bell-canada">Watch this October 2018 Elastic{ON}Tour presentation</a> and discover how they accomplish their security mission, including how Bell Canada handles long-term log retention and fast forensic data retrieval without using public clouds.<a href="/elasticon/tour/2018/toronto/security-events-logging-bell-canada"><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltc586b844830b7d7f/5c66ddb4f6d704c65fb28add/download" data-sys-asset-uid="bltc586b844830b7d7f" alt="Bell Canada is using Elastic to drive improved security analysis in their SOC."/></a>

bell-canada-banner.jpg

bell-canada-thumb.jpg

Security Events Logging at Bell Canada

Learn how AmericanAncestors.org worked with the Elastic support team to boost performance and deliver a highly scalable service with minimal overhead.

<html><head></head><body>This post is a recap of a community talk given at a recent <a href="/elasticon">Elastic{ON} Tour</a> event. Interested in seeing more talks like this? Check out the conference archive or find out when the <a href="/elasticon">Elastic{ON} Tour</a> is coming to a city near you. 
<a href="https://www.americanancestors.org">AmericanAncestors.org</a>, the website of the nonprofit New England Historical Genealogical Society, is one of the leading online family history research sites in the United States. They maintain one terabyte of data in <a href="/cloud/elasticsearch-service">Elastic&#x2019;s Elasticsearch Service</a> that their more than 250,000 members can search in real time, including over 1.4 billion names and an electronic collection of over 40 million pages of centuries-old, handwritten documents.
The average AmericanAncestors.org member is sixty-five and retired. They have names and locations to research and pages of elegant, faded quill script to decipher. They don&#x2019;t want to fiddle with a slow, rigid search feature. They want intelligent search that delivers results fast.
Before switching to Elasticsearch, AmericanAncestors.org had used SQL for their search capability, but that solution didn&#x2019;t have the flexibility their users needed. When Don LeClair, Associate Director of Database Search and Systems, started at AmericanAncestors.org in 2016, they had been running Elasticsearch 1.3 on five physical servers set up by external contractors. A year later, with those contractors long gone, their clusters were no longer optimized, search was slow, and the environment was unstable. LeClair was learning Elastic on the job and, unfortunately, becoming proficient at full cluster restarts.
LeClair and AmericanAncestors.org needed to increase availability and boost performance, but they also wanted to add expertise and reduce system administration time and costs, so they purchased a Platinum subscription to the Elasticsearch Service. Additionally, with their new subscription, they were able to implement <a href="/products/stack/security">Elasticsearch security</a> definitions so queries would come through with appropriate authentication. Another major advantage they found in choosing the Elasticsearch Service was they could create a separate cluster to monitor production and visualize the data in Kibana.
<blockquote>&#x201C;For the first time we had visibility and facts about how the system was performing rather than opinions.&#x201D;&#xA0; Don LeClair,&#xA0;Associate Director of Database Search and Systems | AmericanAncestors.org
</blockquote>Once the clusters were optimized, LeClair was able to turn his attention to new challenges. The genealogy research industry was trending toward search that offered suggestions. At the time, AmericanAncestors.org&#x2019;s approach to search was that results must fit search criteria, but that led to many frustrating member searches that delivered no results.
So they changed to a best fit results approach with prioritized rankings. This change improved the user experience, but also increased the server workload. And that workload increased even more dramatically after <a href="https://www.csoonline.com/article/3247868/vulnerabilities/spectre-and-meltdown-explained-what-they-are-how-they-work-whats-at-risk.html">Spectre and Meltdown</a> vulnerability fixes were applied to their systems, leading to a performance decline and even an outage. With a new optimization challenge to handle, AmericanAncestors.org decided to work closely with Elastic&#x2019;s support team to drive improvements. 
Through this work, LeClair discovered that Elastic support team&#x2019;s ethos went deeper than simply working to help mitigate break/fix issues. The Elastic support team helped identify problem queries, performed code reviews on breaking changes between versions, and altered their sharding strategy to increase speed. Support also identified that over 35% of records in their index were logically deleted documents. With this revelation, AmericanAncestry.org changed tools and processes to avoid excessive re-indexing. They also implemented and maintain a full-data test cluster to simplify upgrade testing and enable them to stay current. The test cluster helps during development so AmericanAncestors.org can roll into production with minimal outage and inconvenience.
In the end, with the help of the Elastic team, they built a highly scalable service with minimal overhead. They reduced disk usage by 60% and CPU by 50%, making search 2-3 times faster. This speed enabled AmericanAncestors.org to offer a major promotion with free signup and search for three days. The system handled a 400% traffic increase with excellent response time. &#x201C;It was &#x2026; perfect,&#x201D; said LeClair. &#x201C;In fact, it was so smooth on the second day I stopped watching it on the monitor.&#x201D; Their current solution was dramatically simpler than their previous physical servers, and, even with premium support and a full-data test cluster, it cost 25% less than their old solution.
Learn about how AmericanAncestors.org is leveraging Elasticsearch Service to drastically improve their service and the lessons they learned along the way implementing by watching <a href="/elasticon/tour/2018/boston/elastic-at-nehgs">Scalability &amp; Support Using the Elasticsearch Service: Elastic @ NEHGS</a> from Elastic{ON} Tour Boston.
 
<a href="/elasticon/tour/2018/boston/elastic-at-nehgs"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0a1b033b9e9e32b9/5c5837924c9f4feb0b4c5a73/american-ancestors-boston-tour-architecture-thumbnail.png" data-sys-asset-uid="blt0a1b033b9e9e32b9" alt="american-ancestors-boston-tour-architecture-thumbnail.png" "=""></a>
</body></html>

the-green-dragon-boston-elasticsearch-american-ancestors-fullbleed-2.jpg

the-green-dragon-boston-elasticsearch-american-ancestors-thumbnail.jpg

A Fast, Satisfying Search Experience with AmericanAncestors.org

How Merck is using Elasticsearch to tame a rich, but wild jungle of genetic data and getting it to perform tricks — like predicting disease causes.

<html><head></head><body>The Human Genome Project was supposed to cure all the diseases. At least, that&apos;s what the general public thought back in 2001 when the results of the project were first shared. But scientists suspected all along that uncovering the root of all human disease would be a bit more complicated &#x2014; and they were right. The data could be collected, but making use of it was going to take years of grueling and tedious work.
Daniel Myung and Bhasker Bokuri are two scientific computing experts at <a href="https://www.elastic.co/elasticon/conf/2017/sf/integrating-human-genetic-data-to-help-drive-drug-discovery-elastic-at-merck">Merck Pharmaceuticals who presented at Elastic{ON}</a> on March 9, 2017 about how their innovative approach to managing genomic data with the Elastic Stack opens new vantages for geneticists. Using the Elastic Stack, scientists can see their data in new ways that uncover safer, more effective treatments for diseases like cancer, Alzheimer&#x2019;s, and diabetes.
&#x201C;The pharmaceutical industry still has a pretty low batting average,&#x201D; said Myung. &#x201C;There&#x2019;s all this work and all this effort. It&#x2019;s pretty expensive and there&#x2019;s still this low hit rate of [a product idea] actually being safe and effective.&#x201D; They want to use genetic data to better predict in the discovery phase how a drug will affect the body long before expensive trials. The goal is to speed the production of effective treatments and make drugs cheaper for all.
<blockquote>They want to use genetic data to better predict in the discovery phase how a drug will affect the body long before expensive trials.</blockquote>As sequencing costs have fallen from tens of millions of dollars to around $1,000 to sequence a whole genome, thousands of studies have been pouring in from academia, public sources, and consortium efforts among multiple pharmaceutical companies. The consortium efforts are the source of most of the de-identified, aggregate data in Merck&apos;s&#xA0;database. The problem Myung and Bokuri face is that the data is unwieldy. There is so much of it, it&#x2019;s in multiple formats from diverse sources, and the methods that have produced this 13 years-and-counting&#x2019;s worth of data change every few years. Previous methods of handling this data, as Myung and Bokuri described, were tedious, manual, and required a significant amount of expert integration. It could take two days to add a single field.
Myung and Bokuri have used Elastic products to make a pipeline that ingests and harmonizes the diverse data into a rapidly searchable database and creates a universal coordinate system for genetic variants. The size and the weird shape of their data fit well into Elastic.
<blockquote>The size and the weird shape of their data fit well into Elastic.</blockquote>They approach it like a weather map, but instead of using location, temperature, humidity, pressure, and wind speed data to make weather predictions, they use genetic variant locations, epigenetic effects, phenotype expression, GWAS, and eQTL data to help researchers predict the efficacy of a drug. It&#x2019;s not perfect, just like weather forecasts aren&#x2019;t perfect, but it presents search results in hours that used to take weeks. And it helps researchers find better places to aim their questions and hopefully raise that pharmaceutical batting average.
Interestingly, this method works in retrospect. Myung displayed&#xA0;a graph from the genetic database of the gene targeted by the successful cholesterol drug Lipitor &#x2014; a known pharmaceutical home run.The graph displayed some interesting effects. If geneticists had run a search on this gene using the new Elastic database, they would have seen these effects and recommended this gene as a target for new medications.
The database is promising, but it&#x2019;s not done yet. &#x201C;This is a start,&#x201D; said Myung. &#x201C;We&#x2019;ve put something in a database and you can search things now.&#x201D; Myung and Bokuri are still looking for ways to support the geneticists who make those data-driven decisions. In the future they hope to apply machine learning or statistics to the data.
If you&#x2019;re interested in their&#xA0;approach to working with complex and important data, dive in and <a href="https://www.elastic.co/elasticon/conf/2017/sf/integrating-human-genetic-data-to-help-drive-drug-discovery-elastic-at-merck">check out their Elastic{ON} presentation and slide deck</a>.
 
<a href="https://www.elastic.co/elasticon/conf/2017/sf/integrating-human-genetic-data-to-help-drive-drug-discovery-elastic-at-merck"><img src="https://api.contentstack.io/v2/uploads/58f7c026c543ea35133a52a7/download?uid=blt3cfe3817e5d29e11" data-sys-asset-uid="blt3cfe3817e5d29e11" alt="merck-elasticsearch-elasticon-thumbnail.jpg"></a>
</body></html>

Stretching-the-Elastic-Stack-to-Fit-the-Human-Genome-full-bleed.jpg

Elastic-Stack-to-Fit-the-Human-Genome-thumbnail.jpg

Stretching the Elastic Stack to Fit the (Very Large and Complicated) Human Genome

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Emily Mosher

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/elastic-cloud-account-terms" target="_self">Conditions générales d'utilisation</a>, et vous acceptez de recevoir les <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">types de notification sélectionnés</a> à l'adresse que vous avez fournie plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/serverless-preview-terms" target="_self">Conditions générales d'utilisation</a>, et vous permettez à Elastic de <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">vous contacter</a> à propos de produits et services similaires en utilisant les informations que vous avez fournies plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

En envoyant votre demande d'inscription, vous acceptez les <a href="/fr/agreements/global/cloud-services-monthly" rel="nofollow">conditions générales d'utilisation d'Elastic Cloud</a>. Vos données personnelles seront traitées conformément à la <a href="/fr/legal/privacy-statement">déclaration de confidentialité d'Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/terms-of-use" target="_self">Conditions générales d'utilisation</a>, et vous permettez à Elastic de <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">vous contacter</a> à propos de produits et services similaires en utilisant les informations que vous avez fournies plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.

Explore similar demos

Overview

You'll also receive an email with related content.

You'll also receive an email with related content

Nous vous enverrons plus de contenu susceptible de vous intéresser.

Speakers

Close

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Le contenu de cette page n'est pas disponible dans la langue sélectionnée. Chez Elastic, nous mettons tout en œuvre pour vous proposer du contenu dans différentes langues. En attendant, nous vous remercions de votre patience !

See more insights

Articles by

Author

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Voir à quelle heure débute le webinar dans mon fuseau horaire

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Si vous ne pouvez pas vous libérer le jour J, inscrivez-vous et nous vous enverrons l'enregistrement. Vous recevrez aussi un e-mail contenant des ressources connexes.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

Thank you for registering. We will send you a confirmation email soon.

Nous vous remercions de votre inscription. Vous allez recevoir un e-mail de confirmation.

Author

Articles by Emily Mosher

Security Events Logging at Bell Canada

A Fast, Satisfying Search Experience with AmericanAncestors.org

Stretching the Elastic Stack to Fit the (Very Large and Complicated) Human Genome

Sign up for Elastic Cloud free trial