On-demand webinar

Loading large datasets into the ELK Stack using Filebeat

Hosted by

Michael Heldebrant
Michael Heldebrant

Ph.D, Distinguished Architect

Elastic

Overview

In this video, we show you how to load two FDA Datasets, MAUDE and FAERS, into Elasticsearch. MAUDE contains medical device adverse event reports submitted by mandatory reporters (manufacturers, importers and device user facilities) and voluntary reporters (health care professionals, patients, and consumers). The FDA Adverse Event Reporting System (FAERS) contains information on adverse event and medication error reports submitted to the FDA.

Filebeat is a lightweight shipper for loading data into Elasticsearch, the heart of the Elastic Stack (formerly known as the ELK Stack). You can use Filebeat to ingest JSON data in bulk.

Highlights:

  • Don’t overthink it. Filebeat can do all the heavy lifting — reading from files line by line, retries, checkpoints, and watching whole directories of files at a time.
  • Learn how to do neat things with JSON documents that are difficult to load, and pick up some tricks for processing weird things that you might encounter.
  • This is reusable. If you can create line by line JSON data like this, then you can use the same process to get your data into Elasticsearch.

Additional Resources:

Register to watch

You'll also receive an email with related content.