Box: Deploying the Elastic Stack for observability — one microservice at a time

In recent years, Box's engineering team was growing concerned that its legacy backend for reporting was not scaling. And as Box's roadmap was growing, the Observability team was interested in adopting a more reliable and cost effective solution for application and operational logging than what Splunk provided.

These were mission-critical processes taking on heightened importance — especially as Box continued on its path of transforming a monolithic infrastructure into hundreds of microservices to enable it to grow, innovate, and deliver new customer-facing features.

Because Box's legacy logging solution was priced on the amount of data ingested, sometimes Box would have to cut logging projects to limit costs or Box engineers would decide not to log events from newly deployed microservices.

That was a fact of life at the time, and was at odds with Box's mission to rapidly transform itself into a leading Cloud Content Management platform. This transformation required Box to break down its monolith into microservices, a direction that requires more and comprehensive logging — not less.

What's more, Box's use of MySQL for enterprise reporting had its own problems. Among them was the inability to render usage logs data for large enterprises — such as when files were opened, moved to a different folder, or even shared — because of the enormous amounts of events that were being generated.

So beginning in 2015, after surveying the competitive landscape for pricing, expandability, support, and security, Box went with Elastic.

By harnessing the Elastic Stack, Box has adopted a future-proof, scalable approach of observability with a pricing model rooted in the amount of memory under management instead of data ingest. This changeover to Elastic, while strengthening Box's reporting capabilities and dramatically reducing costs, also captures logs from within Box's microsystems to understand performance and behavior. These logs are available at a moment's notice, too.

The reporting use case was the entry point for Elastic at Box. That success in replacing MySQL for reporting with Elastic seeded confidence and trust in Elastic. These were the major factors prompting Box to begin replacing its logging solution with Elastic, one new microservice at a time.

Even so, a new reporting and logging system must generate buy-in from engineers or face potential failure or slow adoption. With Box's choosing of the Elastic Stack, engineers are now more motivated than ever to run reports at scale and to program logs for existing or new microservices — all of which positions Box to succeed even more.

All the while, Box's initial journey started with a few terabytes of storage and a handful of developers on the Storage team developing and adopting Box's new reporting features alongside the help of on-site Elastic consultants and Elastic training sessions. Today, about 500 engineers across a wide swath of teams at Box are embracing the Elastic Stack for both reporting and logging — and visualizing terabytes of data daily on hundreds of Kibana dashboards.

As we know, Box's first move with Elastic was to migrate enterprise logs to Elasticsearch for reporting purposes and to serve as the backend for data analytics. This data pipeline would later evolve to support Box's new, Elastic-backed logging environment.

For the reporting project, the company is taking advantage of mandated security features such as role-based access control and user authentication. These and other features — such as being able to tell who accessed what Box file when and where — enabled Box to satisfy compliance obligations involving security and privacy.

The initial switch to this Elasticsearch reporting project, known internally as "Newsroom," cured issues surrounding filtering problems and inconsistencies between enterprise logs and business analytics. No longer was Box having issues producing reports that it was meeting its SLAs, for example. For large enterprises, sometimes the reports didn't load at all, and for medium enterprises, they took too long to return.

Other fixes included:

• Gained ability to efficiently filter usage logs based on users/folders
• Able to acquire events related to specific users in the enterprise or a specific folder
• Statistics reports shown in admin console are unbroken
• Resolved inconsistency of enterprise logs and business analytics

Box's former logging solution was priced on the amount of data ingested, which at the time was about 7 terabytes a day. That has since grown to around 20 terabytes a day, and is expected to increase even more. So as not to break the bank, that meant Box had to pick and choose where to implement observability features on its ever-growing microservices platform — an untenable issue that switching to Elastic has resolved.

Cost wasn't the only concern. Under its previous logging platform, indexers would sometimes fail. Latency, too, was a problem and alerting was not reliable.

With that in mind, and as the scale of data from daily logging was exploding, the company re-engineered its logging infrastructure in 2017.

Box is continually moving all operational and application logging securely into Elasticsearch with a trifurcated approach of separating retention, ad hoc, and interactive logging via a pipeline internally codenamed Almost-Real-Time-Analytics, or "Arta."

Box says producing and maintaining enterprise-grade systems performing at the guaranteed SLAs is critical. So Box engineers must have visibility into their systems and the logs they are generating. Ahmed says Elastic has provided the Observability team at Box with a logging system offering sophisticated querying capabilities at minimal indexing overhead — all at a reasonable cost as Box scales.

With this Elastic-driven observability, in addition to the hundreds of Kibana dashboards to visualize this data, Box engineers have insights at their fingertips. They can see whether there are issues for customers opening their Box files, if collaborators were added to files, if files were added to a folder — and you name it. Logging also provides engineers a quick route to fix issues — such as customer file upload latency — in order to stay on target of uptime SLA requirements.

In addition, to ensure that Box customers continue with a great user experience, logging is central to the act of coding at Box to ensure that code changes have been committed, says Dave Ward, senior director of engineering at Box.

"When you make a change to add new features or fix, did my change produce the desired result?" Ward asks. "Am I seeing errors when pushing code? If the observability pipeline isn't functioning as intended, we could not guarantee the sanity of our systems. Arta leveraging Elasticsearch is mission critical to sustaining our agility and release cadence at scale."

Today, Box is on the path to running much of its production-level, developer-focused logging streams in Arta. As the company continues to add new and existing logging streams from its ongoing development of microservices, the Box team is looking into creating further improvements in stability, resiliency, operational efficiency, and bringing the costs down even more.

For the future, Box is considering centralizing more of its tracing metrics into one platform under the Elastic umbrella. Box also is actively partnering with Elastic engineers and refining and sharing product feedback.

The company is interested in other Elastic features like Elastic APM for application performance monitoring, machine learning to help detect and alert to issues, as well as Elastic's geoIP mapping capabilities so Box can visualize in Kibana where in the world queries are coming from, starting at the IP address.

Box is also exploring the Elastic SIEM to help drive security operations as well as beautifying their Kibana dashboards with Canvas, in addition to surveying many other Elastic features.

In financial terms, the adoption of Elastic has helped reduce the price by half when it comes to data loads. In addition, reporting problems and hiccups under MySQL have been shored up. And the latency caused, in part, by its former logging solution indexing on read compared to Elastic indexing on write — is becoming a thing of the past.

"Boil the ocean or boil a pond to get the data." That's how Wadhwani describes the difference in latency. "Querying in Kibana with Elasticsearch is so much faster," Wadhwani says.

On a more unquantifiable level, when it comes to ROI, Box is actively encouraging its engineers to produce the right amount of logs to achieve the level of observability required for building enterprise grade products.

These and other nuanced changes are helping Box meet internal and external SLA agreements while at the same time they are encouraging Box's developers to innovate as Box scales.