KPN Harnesses the Elastic Stack to Thwart Security Data Overload

This post is a recap of a community talk given at a recent Elastic{ON} Tour event. Interested in seeing more talks like this? Check out the conference archive or find out when the Elastic{ON} Tour is coming to a city near you.

KPN Security Services is a SOC/SIEM security incident and threat-event management company based in the Netherlands. Having the philosophy that everyone should be able to connect online free of exploits and vulnerabilities, the company’s security offerings include preventing, detecting, responding to, and monitoring for threats.

Han Pieterse, a KPN senior solution architect, and Marius Iversen, who at the time was a KPN senior platform engineer, presented at Elastic{ON) Tour Amsterdam and discussed how the Elastic Stack was leveraged to power security operations for KPN and their global customers.

The problem facing KPN was that while its customer base was growing and scaling, the amount of data it had to manage was endlessly multiplying as well. Knowing that data volumes would never cease growing and would become chaotic to manage, the company embraced the Elastic Stack as the solution.

Visibility gaps

Among other reasons, the move to the Elastic Stack was to prevent “visibility gaps in our day-to-day business” and to enable the 400-member KPN security team to stick to the core function of preventing and thwarting attacks on its customers instead of dealing with data management and data collection headaches, Pieterse says.

Pieterse says that an issue the company was confronting boiled down to “security data overload.” That data glut could bring “risk in our business” such as misinterpretation of threats and mismanagement of collected machine data, in addition to delays in threat detection, he says.

“To be secure, to be also operational in the future, we have to handle that volume,” he says.

The Elastic Stack is assisting KPN Security Solutions toward attaining several goals:

  • Controlling “data chaos,” complexity, and costs
  • Gaining more complete, and accurate, security analysis and visibility
  • Managing data sources, streams, and destinations with flexibility
  • Using security resources more efficiently

Pre-Elastic Stack

The high-level view of the company, before adopting the Elastic Stack, was essentially a setup that was single-tenant, manually configured, had a basic API, and normalized data only.

A key limitation that would eventually throttle growth was KPN’s single-tenant implementation for each customer, Iversen says.

“As we get more and more customers, it's going to get really hard to manage because you always have to grow your own team of engineers at the same rate as you get customers because you get so many more clusters — there's so much manual configuration. And that’s not going to work in the long run,” Iversen says.

KPN’s deployment supports non-CEF, harnesses the ArcSight Logstash plugin, TLS, and seizes on the benefits of having multiple pipelines funneling through Kibana dashboards.Data ingestion with the ArcSight Logstash plugin, Iversen says, was “a lot easier than we anticipated.”

“We only really needed to flip the switch. We really didn't need to do anything else. We just set up Logstash, installed the plugin, and it already works,” he says. “We’re only going to scale this larger, and larger.”

Machine Learning as a Service

Iversen says KPN is going to build interesting use cases with the Elastic Stack machine learning capabilities — for both non-security and security purposes alike.

“It's going to be big for us on security breaches,” he says, noting that machine learning can monitor hundreds or thousands of servers simultaneously “for new processes, file changes, and so on.”

And if there is a breach, he says, “We can much faster come to the conclusion of how far the breach went.”

Machine learning is also good for security monitoring in finance by looking for irregularities in money transfers. And it can also look for irregularities in employee behavior, from access to buildings via monitoring security badges to finding anomalies in worker data usage, he says.

When it comes to machine learning outside the security context, Iversen says, “We want to use machine learning to run forecasts, depending on both the performance and resource requirements for our platform.”

One key issue KPN wants to head off is a nightmare for any organization: running out of memory space.

As part of KPN’s machine learning, Iversen says, they’ll ask the program: “How much memory do you think we’re going to need for the next 6 months?”

Watch and listen to Iversen and Pieterse as they get into more of the details on indexing, sharding, censoring data, storage over time, benchmarking, multi-tenancy implementation, encryption, naming conventions, third-party integrations, event brokering and alerting, and Kibana pipeline management.

Watch KPN Elasticon Talk