Loading large datasets into the ELK Stack using Filebeat

In this video, we show you how to load two FDA Datasets, MAUDE and FAERS, into Elasticsearch. MAUDE contains medical device adverse event reports submitted by mandatory reporters (manufacturers, importers and device user facilities) and voluntary reporters (health care professionals, patients, and consumers). The FDA Adverse Event Reporting System (FAERS) contains information on adverse event and medication error reports submitted to the FDA.

Filebeat is a lightweight shipper for loading data into Elasticsearch, the heart of the Elastic Stack (formerly known as the ELK Stack). You can use Filebeat to ingest JSON data in bulk.

Highlights:

  • Don’t overthink it. Filebeat can do all the heavy lifting — reading from files line by line, retries, checkpoints, and watching whole directories of files at a time.
  • Learn how to do neat things with JSON documents that are difficult to load, and pick up some tricks for processing weird things that you might encounter.
  • This is reusable. If you can create line by line JSON data like this, then you can use the same process to get your data into Elasticsearch.

Additional Resources:

ビデオをみる

You'll also receive an email with related content

Michael Heldebrant

Michael architects the world’s most interesting projects. He has 15+ years of experience in information technology spanning biomedical research, logistics, digital Hollywood production, and electronic health records. Michael Heldebrant is a Solutions Architect at Elastic where he works with customers on architecting real-time data ingest, search, and analytics solutions using the Elastic Stack.