This post is a recap of a community talk given at a recent <a href="/elasticon">Elastic{ON}</a>&nbsp;Tour event. Interested in seeing more talks like this? Check out the <a href="/elasticon/tour/2019">Elastic{ON} Tour</a>&nbsp;page to see when a stop is coming to a city near you.Bell Canada, one of Canada’s largest telecommunications companies, offers mobile phone, television, internet, and landline services to big corporations, small and medium-sized businesses, and individuals across the country. Bell Canada’s security operations center (SOC) covers every Bell office and business unit coast to coast and they rely on logs to detect cybersecurity threats.Sylvain Proulx, Bell Canada’s Senior Security Manager, says the business units — like Bell TV, Bell Internet, Bell Media, or Bell Mobility — that deliver services to their customers all use different technologies and applications, so the logs they collect are diverse and uncommon. Logs come from routers, firewalls, web logs, OS logs, application logs, and many other devices, some of which get ‘chatty’ and generate a lot of data.The SOC had performed log and event correlation and incident response and reporting using only an ArcSight Security Information and Event Management (SIEM) solution. But over time, as the volume of logs increased, normalizing many new types of logs from a variety of devices bogged down the system. Their SIEM solution also provided only rule-based detection with no machine learning, so it generated a high ratio of false-positive incidents, which threatened to alert-fatigue their analysts.Proulx said they’d hit their SIEM’s limit. They found no single vendor solution that would let them ingest more data faster, build threat detection models, and normalize many new types of logs while also retaining ownership of their data. So, the SOC got to work augmenting their ArcSight SIEM with tools like the Elastic Stack to handle high log volume and traffic spikes automatically and generate meaningful security data that wouldn’t overwhelm analysts.Bell Canada gets data from bare metal servers, virtual machines, and, increasingly, from container infrastructure&nbsp;with&nbsp;Docker and Kubernetes. They needed a log shipper that was simple, lightweight, and straightforward to automate, so they turned to <a href="/products/beats">Beats</a>. They use <a href="/products/beats/filebeat">Filebeat</a> and <a href="/products/beats/winlogbeat">Winlogbeat</a> to ship logs because they’re easy to configure, test, and deploy. Plus, they can version control their configurations and there is no loss of data in case of a network outage.After the data is queued in Kafka, the SOC must parse and normalize their logs in all their various formats in order to perform security analysis. Running <a href="/products/logstash">Logstash</a> instances on OpenShift has helped them scale quickly and automatically in case of traffic spikes without dropping logs, and it consumes less resources than multiple virtual machines. An additional advantage they’ve found to having Logstash in a container is that they can easily run it through RSpec for testing before moving to production.Once the logs are normalized, the SOC stores them in <a href="/products/elasticsearch">Elasticsearch</a>. Bell Canada’s previous solution was unable to handle increasing log volumes and scale without losing logs. The SOC now does this with Elasticsearch, which allows them to scale quickly and horizontally, making&nbsp;their job a lot easier.The day events are logged, the SOC searches the data with multiple queries and processes, which puts a heavy load on the cluster, so they’ve implemented a hot-warm architecture with automated deployment of new nodes. The beefier nodes are ingesting and being searched constantly, but when the logs lose their value, they’re shipped to warm nodes for aggregation and lighter analysis. “If you lose a node in Elasticsearch, you can still keep working. Not a problem. You can fix it later,” says Mathew Vandystadt, Bell Canada’s Security Specialist Software Engineer.Securing data is a top priority for the SOC. Role-based access control (RBAC) is a must, but RBAC can be painful to manage. With the <a href="/products/stack/security">security features of the Elastic Stack</a>, the SOC has control over who has access to the data, they can add a layer of encryption over data transportation, and they can easily perform RBAC management — and it also ties in easily with their existing LDAP, meaning they don’t need to spend extra time redefining group roles and can focus on their security mandate.Once they have their data where they want it, the SOC analysts can use it to find security incidents — and good visualization is key. <a href="/products/kibana">Kibana</a>’s straightforward interface means their busy specialists don’t spend a lot of extra time learning new query languages.<blockquote>“[Kibana] works great in our use cases. It’s simple. We don’t have to do a lot of training and we get a nice visualization.” Mathew Vandystadt, Security Specialist Software Engineer | Bell Canada</blockquote>Bell Canada found a flexible alerting solution for rule-based detection, but their analysts also needed smart detection that works with different algorithms, so they developed in-house machine learning with open source, ML-centric libraries. Their containers in OpenShift let them easily spin Python containers tied to Kafka or Elasticsearch so all data is accessible. Then they use their ArcSight SIEM for event aggregation and correlation to get a higher ratio of true positives to false positives, sparing analysts from alert bombardment.They’ve built this whole pipeline using different software from different vendors, made possible because Elastic allows for simple integration with open security protocols. In the future, the SOC plans to merge a Cyber <a href="https://www.elastic.co/security/tip" target="_self">Threat Intelligence platform</a> with their new security architecture. “Having that infrastructure right there allows us to do more than we were a year and a half ago with only the ArcSight solution,” says Proulx.Ready to learn more about the team’s tools and pipeline? <a href="/elasticon/tour/2018/toronto/security-events-logging-bell-canada">Watch this October 2018 Elastic{ON}Tour presentation</a> and discover how they accomplish their security mission, including how Bell Canada handles long-term log retention and fast forensic data retrieval without using public clouds.<a href="/elasticon/tour/2018/toronto/security-events-logging-bell-canada"><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltc586b844830b7d7f/5c66ddb4f6d704c65fb28add/download" data-sys-asset-uid="bltc586b844830b7d7f" alt="Bell Canada is using Elastic to drive improved security analysis in their SOC."/></a>

bell-canada-banner.jpg

bell-canada-thumb.jpg

Security Events Logging at Bell Canada

Learn how AmericanAncestors.org worked with the Elastic support team to boost performance and deliver a highly scalable service with minimal overhead.

<html><head></head><body>This post is a recap of a community talk given at a recent <a href="/elasticon">Elastic{ON} Tour</a> event. Interested in seeing more talks like this? Check out the conference archive or find out when the <a href="/elasticon">Elastic{ON} Tour</a> is coming to a city near you. 
<a href="https://www.americanancestors.org">AmericanAncestors.org</a>, the website of the nonprofit New England Historical Genealogical Society, is one of the leading online family history research sites in the United States. They maintain one terabyte of data in <a href="/cloud/elasticsearch-service">Elastic&#x2019;s Elasticsearch Service</a> that their more than 250,000 members can search in real time, including over 1.4 billion names and an electronic collection of over 40 million pages of centuries-old, handwritten documents.
The average AmericanAncestors.org member is sixty-five and retired. They have names and locations to research and pages of elegant, faded quill script to decipher. They don&#x2019;t want to fiddle with a slow, rigid search feature. They want intelligent search that delivers results fast.
Before switching to Elasticsearch, AmericanAncestors.org had used SQL for their search capability, but that solution didn&#x2019;t have the flexibility their users needed. When Don LeClair, Associate Director of Database Search and Systems, started at AmericanAncestors.org in 2016, they had been running Elasticsearch 1.3 on five physical servers set up by external contractors. A year later, with those contractors long gone, their clusters were no longer optimized, search was slow, and the environment was unstable. LeClair was learning Elastic on the job and, unfortunately, becoming proficient at full cluster restarts.
LeClair and AmericanAncestors.org needed to increase availability and boost performance, but they also wanted to add expertise and reduce system administration time and costs, so they purchased a Platinum subscription to the Elasticsearch Service. Additionally, with their new subscription, they were able to implement <a href="/products/stack/security">Elasticsearch security</a> definitions so queries would come through with appropriate authentication. Another major advantage they found in choosing the Elasticsearch Service was they could create a separate cluster to monitor production and visualize the data in Kibana.
<blockquote>&#x201C;For the first time we had visibility and facts about how the system was performing rather than opinions.&#x201D;&#xA0; Don LeClair,&#xA0;Associate Director of Database Search and Systems | AmericanAncestors.org
</blockquote>Once the clusters were optimized, LeClair was able to turn his attention to new challenges. The genealogy research industry was trending toward search that offered suggestions. At the time, AmericanAncestors.org&#x2019;s approach to search was that results must fit search criteria, but that led to many frustrating member searches that delivered no results.
So they changed to a best fit results approach with prioritized rankings. This change improved the user experience, but also increased the server workload. And that workload increased even more dramatically after <a href="https://www.csoonline.com/article/3247868/vulnerabilities/spectre-and-meltdown-explained-what-they-are-how-they-work-whats-at-risk.html">Spectre and Meltdown</a> vulnerability fixes were applied to their systems, leading to a performance decline and even an outage. With a new optimization challenge to handle, AmericanAncestors.org decided to work closely with Elastic&#x2019;s support team to drive improvements. 
Through this work, LeClair discovered that Elastic support team&#x2019;s ethos went deeper than simply working to help mitigate break/fix issues. The Elastic support team helped identify problem queries, performed code reviews on breaking changes between versions, and altered their sharding strategy to increase speed. Support also identified that over 35% of records in their index were logically deleted documents. With this revelation, AmericanAncestry.org changed tools and processes to avoid excessive re-indexing. They also implemented and maintain a full-data test cluster to simplify upgrade testing and enable them to stay current. The test cluster helps during development so AmericanAncestors.org can roll into production with minimal outage and inconvenience.
In the end, with the help of the Elastic team, they built a highly scalable service with minimal overhead. They reduced disk usage by 60% and CPU by 50%, making search 2-3 times faster. This speed enabled AmericanAncestors.org to offer a major promotion with free signup and search for three days. The system handled a 400% traffic increase with excellent response time. &#x201C;It was &#x2026; perfect,&#x201D; said LeClair. &#x201C;In fact, it was so smooth on the second day I stopped watching it on the monitor.&#x201D; Their current solution was dramatically simpler than their previous physical servers, and, even with premium support and a full-data test cluster, it cost 25% less than their old solution.
Learn about how AmericanAncestors.org is leveraging Elasticsearch Service to drastically improve their service and the lessons they learned along the way implementing by watching <a href="/elasticon/tour/2018/boston/elastic-at-nehgs">Scalability &amp; Support Using the Elasticsearch Service: Elastic @ NEHGS</a> from Elastic{ON} Tour Boston.
 
<a href="/elasticon/tour/2018/boston/elastic-at-nehgs"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0a1b033b9e9e32b9/5c5837924c9f4feb0b4c5a73/american-ancestors-boston-tour-architecture-thumbnail.png" data-sys-asset-uid="blt0a1b033b9e9e32b9" alt="american-ancestors-boston-tour-architecture-thumbnail.png" "=""></a>
</body></html>

the-green-dragon-boston-elasticsearch-american-ancestors-fullbleed-2.jpg

the-green-dragon-boston-elasticsearch-american-ancestors-thumbnail.jpg

A Fast, Satisfying Search Experience with AmericanAncestors.org

How Merck is using Elasticsearch to tame a rich, but wild jungle of genetic data and getting it to perform tricks — like predicting disease causes.

<html><head></head><body>The Human Genome Project was supposed to cure all the diseases. At least, that&apos;s what the general public thought back in 2001 when the results of the project were first shared. But scientists suspected all along that uncovering the root of all human disease would be a bit more complicated &#x2014; and they were right. The data could be collected, but making use of it was going to take years of grueling and tedious work.
Daniel Myung and Bhasker Bokuri are two scientific computing experts at <a href="https://www.elastic.co/elasticon/conf/2017/sf/integrating-human-genetic-data-to-help-drive-drug-discovery-elastic-at-merck">Merck Pharmaceuticals who presented at Elastic{ON}</a> on March 9, 2017 about how their innovative approach to managing genomic data with the Elastic Stack opens new vantages for geneticists. Using the Elastic Stack, scientists can see their data in new ways that uncover safer, more effective treatments for diseases like cancer, Alzheimer&#x2019;s, and diabetes.
&#x201C;The pharmaceutical industry still has a pretty low batting average,&#x201D; said Myung. &#x201C;There&#x2019;s all this work and all this effort. It&#x2019;s pretty expensive and there&#x2019;s still this low hit rate of [a product idea] actually being safe and effective.&#x201D; They want to use genetic data to better predict in the discovery phase how a drug will affect the body long before expensive trials. The goal is to speed the production of effective treatments and make drugs cheaper for all.
<blockquote>They want to use genetic data to better predict in the discovery phase how a drug will affect the body long before expensive trials.</blockquote>As sequencing costs have fallen from tens of millions of dollars to around $1,000 to sequence a whole genome, thousands of studies have been pouring in from academia, public sources, and consortium efforts among multiple pharmaceutical companies. The consortium efforts are the source of most of the de-identified, aggregate data in Merck&apos;s&#xA0;database. The problem Myung and Bokuri face is that the data is unwieldy. There is so much of it, it&#x2019;s in multiple formats from diverse sources, and the methods that have produced this 13 years-and-counting&#x2019;s worth of data change every few years. Previous methods of handling this data, as Myung and Bokuri described, were tedious, manual, and required a significant amount of expert integration. It could take two days to add a single field.
Myung and Bokuri have used Elastic products to make a pipeline that ingests and harmonizes the diverse data into a rapidly searchable database and creates a universal coordinate system for genetic variants. The size and the weird shape of their data fit well into Elastic.
<blockquote>The size and the weird shape of their data fit well into Elastic.</blockquote>They approach it like a weather map, but instead of using location, temperature, humidity, pressure, and wind speed data to make weather predictions, they use genetic variant locations, epigenetic effects, phenotype expression, GWAS, and eQTL data to help researchers predict the efficacy of a drug. It&#x2019;s not perfect, just like weather forecasts aren&#x2019;t perfect, but it presents search results in hours that used to take weeks. And it helps researchers find better places to aim their questions and hopefully raise that pharmaceutical batting average.
Interestingly, this method works in retrospect. Myung displayed&#xA0;a graph from the genetic database of the gene targeted by the successful cholesterol drug Lipitor &#x2014; a known pharmaceutical home run.The graph displayed some interesting effects. If geneticists had run a search on this gene using the new Elastic database, they would have seen these effects and recommended this gene as a target for new medications.
The database is promising, but it&#x2019;s not done yet. &#x201C;This is a start,&#x201D; said Myung. &#x201C;We&#x2019;ve put something in a database and you can search things now.&#x201D; Myung and Bokuri are still looking for ways to support the geneticists who make those data-driven decisions. In the future they hope to apply machine learning or statistics to the data.
If you&#x2019;re interested in their&#xA0;approach to working with complex and important data, dive in and <a href="https://www.elastic.co/elasticon/conf/2017/sf/integrating-human-genetic-data-to-help-drive-drug-discovery-elastic-at-merck">check out their Elastic{ON} presentation and slide deck</a>.
 
<a href="https://www.elastic.co/elasticon/conf/2017/sf/integrating-human-genetic-data-to-help-drive-drug-discovery-elastic-at-merck"><img src="https://api.contentstack.io/v2/uploads/58f7c026c543ea35133a52a7/download?uid=blt3cfe3817e5d29e11" data-sys-asset-uid="blt3cfe3817e5d29e11" alt="merck-elasticsearch-elasticon-thumbnail.jpg"></a>
</body></html>

Stretching-the-Elastic-Stack-to-Fit-the-Human-Genome-full-bleed.jpg

Elastic-Stack-to-Fit-the-Human-Genome-thumbnail.jpg

Stretching the Elastic Stack to Fit the (Very Large and Complicated) Human Genome

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Emily Mosher

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

제출하면 귀하가 <a href="/kr/legal/elastic-cloud-account-terms" target="_self">Elastic 서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 연락처로 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">선택한 커뮤니케이션</a>을 수신하는 데 동의한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/serverless-preview-terms" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

제출하면 <a href="/kr/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 서비스 약관</a>에 동의하시게 됩니다. 개인 데이터는 <a href="/kr/legal/privacy-statement">Elastic의 개인정보 취급방침</a>에 따라 처리됩니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/terms-of-use" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

이 페이지의 콘텐츠는 선택하신 언어로 제공되지 않습니다. Elastic은 다양한 언어로 콘텐츠를 제공하기 위해 최선을 다하고 있습니다.조금만 더 기다려주세요!

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

내 표준 시간대로 웨비나 시작 시간 확인하기

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

참여가 어려우신가요? 등록하시면 녹화 자료를 보내드리겠습니다. 이메일로 관련 콘텐츠도 받아보실 수 있습니다.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

You'll also receive an email with related content

이메일을 통해 관련 내용을 보내드리겠습니다.

Thank you for registering. We will send you a confirmation email soon.

등록해 주셔서 감사합니다. 곧 확인 이메일을 보내드리겠습니다.

Author

Articles by Emily Mosher

Security Events Logging at Bell Canada

A Fast, Satisfying Search Experience with AmericanAncestors.org

Stretching the Elastic Stack to Fit the (Very Large and Complicated) Human Genome

Sign up for Elastic Cloud free trial