How a European national police force is uniting data to better serve the public with Elastic
A European national police force of 65,000 officers and 2,000 IT professionals was faced with the challenge to build a central, and searchable repository of all logs for infrastructure, applications and audit data.
To meet this challenge, the agency chose the Elastic platform to build a data lake, which will eventually become the central hub for all log data. The agency’s goal is to fill the lake with log data from as many as 350 applications over the next two years.
“What we're looking at two years from now is some 75TB of ingest each and every day. As a frame of reference, this is nearly 20 times more data captured in one day than the total amount that can be stored in Apple’s personal cloud storage plan. And at this moment we are already processing some 2.5TB coming in from the sources we have currently connected. So being prepared for growth is essential to our system,” says the agency’s product line manager.
Logging for DevOps, investigators, and digital security personnel
“Obviously we have the DevOps teams who will be facilitated by us in storing and retaining the log information. And in doing that, they will be able to build dashboarding for their own systems based upon an excellent source,” says the product line manager.
A second group of users of the data lake will include investigators examining suspected “misuse of police data.”
“For example, if an officer wants to check on her daughter’s boyfriend,to see if he has a record, that's not proper use of police data. It should be related to the task,” the product line manager says.
A third group of users, the product line manager says, are the developers at the Security Operations Center “who will be looking for anomalies in the use of data.”
“For example, if an officer logs in at their terminal in one city location and at the same time somebody walks into the police station in another city that is located hours away and uses that officer’s badge, that will trigger an anomaly at the Security Operations center and is observed in the logging of the systems”
To maintain control of the data, users of the data lake are to be granted view rights via role-based-access, which is easily configurable with Elastic.
Data retention: warm, hot, cold, and frozen
Because the data lake will be retaining and logging 75TB of data daily when complete, the agency is going to take advantage of Elastic’s tiered storage levels.
Depending on the data type, retention policies, and regulations, the agency stores its data between 1-5 years.
The business benefits of this type of storage architecture is immense. The colder the data storage, the less costly it becomes to maintain. This means older data, or data kept for regulatory purposes, can be kept in cheaper, colder storage that uses less computing power.
For this police agency, this makes it more “efficient” to manage growing data volumes in Elastic, and opens the door to new use cases.
“In order to be able to handle that amount of data in a somewhat efficient way, we made the split in the design between warm, hot, cold, and frozen storage,” the product line manager says.
Engaging Elastic Consulting
To deploy the project more smoothly, the agency engaged Elastic Consulting in building the initial design and for implementing best practices.