Elastic{ON} 2017

IBM discusses the Bluemix platform and how it enables logging with the Elastic Stack. Take your interconnected, cloud-hosted micro services and bring them together quickly and simply so you spend less time instrumenting your app for logging applications and more time enhancing it.

Elasticsearch team and tech leads give an overview of the changes already released in 5.x series, and a taste of the new features coming in 6.0.

Machine Zone is the company behind massively-multiplayer online games Game of War, Fire Age, and Mobile Strike. Real-time operations is essential for a great user experience. The MZ engineering team shares how they use the Elastic Stack to track player activity to support customer service and more.

Did you know that Walmart sells about 156 bananas per second? Thanks to the Elastic Stack, they do. What started as an engineering hack to prove the banana myth is now a fully-operational self-service analytics platform for Walmart’s data scientists.

NVIDIA collects metrics for every frame streamed, every second spent on GeForce NOW, its streaming video game platform, using the Elastic Stack – enabling business, engineering, operations, and quality assurance teams to assess quality and debug issues in real time.

Search-based analytics are a critical function for any organization. Ryft leverages FPGA/x86 heterogeneous compute technology to eliminate indexing and transformation, in addition to providing enhanced Elasticsearch functionality that accelerates workflows, increases the speed of search and analysis, and enhances wildcard searches.

Kibana team and tech leads cover the latest news in Kibana — heatmaps, log context, pipeline aggregations, CSV export, and more. They look back at what's happened since 5.0 and forward to what's coming in future releases.

Cisco's Commerce Platform powers product configuration, pricing, quoting, export compliance, credit checks, and order booking across all product lines. Learn how they've implemented a 180+ node Elasticsearch deployment to improve customer experience and business agility.

General Mills replaced their Endeca-based search and service-oriented architecture with Elasticsearch on core website properties such as Pillsbury.com, BettyCrocker.com, and more. Their use cases continue to expand as they move away from a relational database model to a flexible, document-based model.

Beats are a family of lightweight shippers that send data from edge machines to Elasticsearch. Beats creators Monica and Tudor walk through the latest, including Filebeat modules, which simplify the collection and parsing of common log files down to a single command, and Heartbeat, for uptime monitoring.

The NERSC data collect system provides access to 30 TB of logs and time-series data generated by the supercomputers at Berkeley Lab. Explore how NERSC uses Elasticsearch as a large, long-term data storage engine, including index allocation tagging, use of index aliases, and more.

The distributed systems movement, and open source more broadly, is fueled by solving a series of complex problems: consensus, leader election, failure semantics, among others. How does a distributed team building distributed systems, at Elastic, function?

As healthcare institutions generate more data, they need a way to search through electronic health records (EHR) and find meaningful insights. UCLA Health has chosen Elasticsearch as its tool of choice to index, search, and produce more thorough, actionable results for clinicians and researchers.

Learn how a team of two with a vision for the future grew into a group of more than 30 engineers operating as the de-facto Elasticsearch Center of Excellence at XPO Logistics.

We assembled a panel of executives and engineers to talk about their experiences in the software and technology space. They discussed approaches to growing their careers, navigated key challenges and difficult decisions, and how they have promoted diversity and equality in tech.

Logstash team and tech leads address questions like: Can we go faster? What is the persistent queue? How do I monitor Logstash? What is the future of Logstash configuration? How does the team keep systadmins and DevOps in mind when working on the product? And what's in store for the Logstash UI?

In spite of being close to 20 years old, the Lucene project keeps innovating. Hear stories of the latest features in Lucene 6, how they impacted Elasticsearch, and what to expect in Lucene 7.

How do you build a network of devices and log taps that monitor security at the world's largest conservation non-profit? Can it be done at low budget scale across over 100 offices effectively? Can defending against attacks on a laptop in Pennsylvania help to influence river flow metric collection? It can.

IBM and their customers identified multiple languages localization support as a key capability needed in all Kibana visualizations and dashboards. Learn about the key aspects of Kibana globalization from design to capability delivered, as well as some of the design journey they took to get there.

Adding security, alerting, monitoring, and graph exploration capabilities to the Elastic Stack with X-Pack has never been easier. Over the last year, X-Pack usability has improved, including a better getting started experience and several UI features. The engineering team tells all.

Tinder relies on the Elastic Stack to analyze, visualize, and predict not only which people a user will swipe right on, or which people will swipe right on that user, but also when there's a mutual swipe match. Hear how the service is growing into a global platform for social discovery in many facets of life.

Blackboard uses Elasticsearch to power SafeAssign, its plagiarism detection software. Learn about their internal web search engine, how they transitioned from a Solr deployment to a multi-terabyte, full-text search engine powered by Elasticsearch, and their experiences running their infrastructure on AWS.

Learn how to apply machine learning features to the Elastic Stack and what business problems they will help you solve. These new capabilities let you answer new questions like "are users exfiltrating data unusually?" and "is my website response time atypical?"

Terradue develops and operates large cloud solutions for processing Earth observation satellite data, and uses Elasticsearch to empower users to expound it. Learn how they use the Elasticsearch .NET client to tackle challenges such as geohazards for rapid response monitoring of volcanoes or earthquakes...

Learn how Blizzard Entertainment — makers of Overwatch and World of Warcraft — leverages Elasticsearch, Kibana, Logstash, Kafka, tribes, and Node.js to generate actionable value from gamer and server events.

As of Logstash 5.1.1, you can easily connect any device that supports the CEF data format as a codec to the Elastic Stack via files, Kafka, or syslog, allowing you to extend and complement your existing ArcSight deployment with the Elastic Stack.

The ability to effectively browse, scroll, and sift through raw log files is critical for IT teams and developers to identify errors, perform root cause analysis, and troubleshoot. Learn about a new open source Kibana plugin from Search Technologies that enables this.

Exciting developments are coming for those who want to deploy and manage the Elastic Stack with the click of a button on AWS — or choose to run it the way they want, in the environment they want.

Barclays utilizes Elasticsearch in key data analytics initiatives to enable cyber security and cyber defense.

With countless business-critical text search and analytics use cases that utilize Elasticsearch, eBay has created a custom 'Elasticsearch as Service' platform. Learn about sizing, provisioning, configuring, maintaining, auto-scaling, and decommissioning states for every Elasticsearch cluster.

Metrics and logs are meant to be together. Why do we insist on keeping them apart? Learn about our mission to reunite them, in the process deriving powerful operational insights using brand-new Kibana visualizations and machine learning techniques.

The Elastic Cloud team runs thousands of clusters and is growing rapidly, while maintaining solid SLAs and allowing users to scale, upgrade, and reliably monitor their clusters. Ever wonder how it works?

SQL for Elasticsearch is coming. Learn how this feature converts a SQL statement into an Elasticsearch query, executes a SQL query and return the results in tabular form, provides a console to explore data, and more.

Distributed systems rely on consensus algorithms. Choosing among Paxos and its variants determines the underlying system's performance and fault-tolerance. Learn about the mechanics of quorum-based consensus algorithms and tradeoffs compared to the primary-backup approach.

The few thousands operations per second indexing requirement is now few millions; a single use case has grown to multiple; one user group has become tens of user groups. Learn about Elastic Stack topologies to meet evolving use case(s), and deployment patterns for projects of various sizes.

IBM Bluemix platform logs to the Elastic Stack. Learn about the flexibility of their logging solutions in microservice environments. See Docker and Cloud Foundry logging — no user configuration necessary. Runtime logs are easily added. Additionally, see an example logging from a Watson service.

Dive into the new machine learning technologies available in the Elastic Stack and how to apply them. Explore the unsupervised machine learning techniques Elastic uses, and the challenges and constraints that exist in order to provide operationally useful insight when applying them to time series data.

How do you take lessons from ballet and performance into distributed systems, and how do you take lessons from distributed systems into biometric and kinetic sensing wearables?

The Elastic Docker team discusses running the Elastic Stack in container environments using the official images they maintain and publish.

The ability to monitor Docker environments is critical. Metricbeat offers two different possibilities to do so with cgroups and the Docker API. Learn about the differences and similarities between the two approaches.

Learn how Workday Search expanded their Elastic Stack deployment by implementing a robust, easy-to-use metrics processing pipeline for over 1 billion logs.

Walgreens is creating a search platform for its website powered by Elasticsearch and is expanding to use X-Pack for log management and analysis. Learn about their migration from Endeca, and their technical and architectural lessons and successes.

Storing metrics and events in Postgres wasn’t enough for Datadog, a SaaS-based infrastructure monitoring company. Learn how and why they moved to Elasticsearch to create a fast and efficient environment for thousands of customers.

Monitoring malicious activity and handling the resulting alerts is vital to the success of a defensive security program. Slack talks tools everyone should consider to monitor their infrastructure, including Elasticsearch, and how to create a reliable logging pipeline to handle data from thousands of hosts.

Get introduced to extending Beats, the platform for lightweight data shippers to Elasticsearch and Logstash, and learn how to extend it to deal with your particular use case.

The folks behind the Elastic curtain shows how they use the Elastic Stack to enhance their internal technology stack. They show you from start to finish how they query and expose license data, enhance the support engineers’ experiences in their console, and how to bring it all together in Salesforce.

Elasticsearch enabled Merck to harmonize a data ingestion pipeline and create a universal coordination system for genetic variants as a backbone to help scientists uncover new insights on human genetics across a spectrum of diseases and aid in the discovery and validation of new therapies.

Improvements are coming to Elasticsearch including range fields, removing the _all field, a unified highlighter, and the synonym graph filter.

Elasticsearch plays a key role in Uber's Marketplace Dynamics core data system, aggregating business metrics to control critical marketplace behaviors like dynamic (surge) pricing, supply positioning, and assess overall marketplace diagnostics — all in real time. Learn how they do it.

In order to understand their outbound marketing and campaigns, Fandango deployed the Elastic Stack to monitor and analyze over 5 billion web logs monthly. In one weekend, the FandangoNOW team redesigned and re-architected their on-premise deployment onto Elastic Cloud to analyze 500 million records per day.

Knowing what sort of data makes sense to put in Graph and how to prepare it is often a challenge for new users. This session walks through examples of how to model your Elasticsearch data in order to start exploring the interesting connections it contains using this X‑Pack feature.

Get introduced to the basics of ES-Hadoop's native Spark Integration, touch upon the other features that the connector brings to the table (including native integrations with Hive, Storm, Pig, Cascading, and MapReduce), have a look at the internals to see how it works, and see what's to come.

When monitoring met alerting, the average troubleshooting time went down and the average sleep time went up. True story. X-Pack brings both features together to enable built-in cluster alerts. Learn about the latest in monitoring and management and how to solve real-world problems using monitoring data.

Get a detailed walkthrough of Tagcloud and Heatmap (new visualizations in Kibana 5.2), see what's coming with future geospatial visualizations, and learn about dedicated UIs for time series visualizations in Timelion and a new visual builder for pipeline aggregations.

Learn how to write plugins for Logstash and what goodies the 5.X version line will bring to the plugin developer. We cover the basics, as well as how to write a Java-based plugin, how to instrument your plugin with metrics, and more!

Elastic believes technology enables us to progress toward a better future. Inspired by the people applying Elastic software this way, Elastic recognized three projects using the Elastic Stack to advance the greater good, improve the human condition, and help the planet.