(Elastic) Searching for Big Data

Presentation by Government Consultants Brian Dolbeare & Satish Kolli from the Washington DC Elasticsearch Meet Up Group hosted by OpenGov Hub.  Brian and Satish's presentation covered how they implemented Elasticsearch and how it worked along side of Hadoop, as well as some of the lessons they learned through the process.  The topics covered included: Covers different document types (nested, parent-child)-External document versioning scheme; Storing full document and fetching partial fields; Easy and self-contained cluster setup; Elasticsearch Hadoop integration; How we use Elasticsearch Hadoop plugins (Pig, Java M/R) to ingest billions of documents into ES index; Tuning Elasticsearch for heavy indexing; How we configured Elasticsearch internal queues and buffers for heavy indexing operations; Rolling restarts and Node outages; How we perform OS/ES upgrades with little to no outage; Affect of auto balancing and number of parallel recoveries on indexing/search.