Reinventing Fermilab's Scientific Computing Grid Accounting with the Elastic Stack

In 2017, Fermilab launched a project to re-engineer an aging MySQL-based grid accounting system in their scientific computing division to a distributed system based around Elasticsearch, RabbitMQ, and Grafana. Leveraging this suite of open source tools, Fermilab was able to deploy a more flexible, scalable, and responsive system, all while saving resources and man hours.

Today, their scientific computing division is using the Elastic Stack to collect, search through, and analyze data on the state and usage of their distributed computing system that connects thousands of researchers with hundreds of computing resources around the world. Discover what motivated this huge change, get details on the migration, and hear lessons learned along the way.

Kevin Retzke

Application Developer and System Analyst, Fermilab

In the Scientific Computing division at Fermilab, Kevin supports hundreds of scientists running globally distributed computing jobs, which may analyze the tiniest particles or search the breadth of the universe, by developing tools to monitor the usage and performance of computing resources around the world.