Tech Topics

Introduction to GDPR with Elasticsearch

Editor's Note (August 3, 2021): This post uses deprecated features. Please reference the map custom regions with reverse geocoding documentation for current instructions.

We get this question a lot: "We have a lot of data in our Elasticsearch cluster. Some of this data requires GDPR compliance. What advice do you have?"

In this first post of our multi-part blog series on the EU General Data Protection Regulation, we take a look at the background of GDPR, offer a brief primer, and introduce a simple model of the process an organization may take in readying for compliance with this new regulation. We've also provided this page around GDPR Compliance with Elasticsearch and the Elastic Stack.


Replacing the previous 1995 EU Data Protection Directive, GDPR was developed in recognition of the increasing need to protect the rights and personal data of each individual EU resident. As the volume, depth, and breadth of personal data proliferates, it's easy to see why. GDPR is becoming increasingly recognized as regulation that will be leveraged to stem the increasing number of damaging data breaches reported across a variety of sectors. While previously compliant organizations may find many similarities to the earlier Directive, GDPR brings in some significant changes to the way personal data can be handled, rules on how breaches must be reported, and hefty penalties for non-compliance.

The European Commission believes that by homogenizing data protection laws throughout the single market, and by putting in place a more transparent and simpler legal landscape in which to operate, this will save corporations €2.3 billion each year, collectively.

With the deadline for compliance fast approaching, some claim corporations have a long way to go with preparations. According to recently published research from international law firm Paul Hastings, as of early January 2018, less than 39% of organizations (and 47% in the US) have created internal GDPR task forces. In addition, less than 40 percent are working with third-parties to conduct gap analyses and give counsel on compliance.

According to the UK Information Commissioner's Office, if an organization is already compliant with the current EU directive, then most of its approach to compliance will remain valid under GDPR and can be used as the starting point from which to build. However, there are new elements and significant enhancements to take into consideration.

GDPR Affected Establishments

EU and Non-EU establishments may be affected by GDPR depending on their business models, geographical reach, and the subjects from which they control or process data. An easy way to think about this is to apply one of the following four classifications to organizations:


  • EU Establishments (includes UK for now)
  • Super-regional or global establishments with EU presence
  • Establishments without EU presence, but with EU customers/visitors/subjects

Not GDPR-Affected:

  • Establishments with no EU presence and no EU customers/visitors/subjects

Relevant GDPR Entities

GDPR defines roles or personas in terms of Data Subjects, Data Controllers, Data Processors, Sub-processors, and Authorities. Also, it defines a formalized role within the organization called the Data Protection Officer (DPO). The key roles are listed below:

  • Data Subject: Persons in the EU
  • Data Controller: Controls purpose and means of processing. Direct responsibility to data subject and data protection authority
  • Data Processor: Acts on instructions of Data Controller. Direct responsibility to data subject and data protection authority

Data In Scope for GDPR

Unlike other regulations, GDPR does not use the term "PII" (Personally Identifiable Information), but rather uses the term "Personal Data." In this white paper, we'll use capitalized "Personal Data" to refer to personal data as defined by GDPR.

Personal Data means any information relating to an identified or identifiable natural person ("data subject").

GDPR also defines "special categories" of personal data that have additional restrictions and requirements (racial, ethnic, religious, political, biometric, etc.). Additionally, Recital 10 of the GDPR equates these special categories of personal data with the term "sensitive data." In this white paper, we'll use the capitalized "Sensitive Personal Data" to refer to data that falls into these special categories.

"Identifiable Person" is a natural person who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural, or social identity of that person. [Chapter 1, Article 4(1)]

Protecting the Rights of Data Subjects

GDPR seeks to build on some of the key pillars of the current Data Protection Directive by significantly enhancing the rules around the processing and storage of personal data. GDPR includes the following rights for individuals:

  • The right to be informed
  • The right of access
  • The right to rectification
  • The right to erasure
  • The right to restrict processing
  • The right to data portability
  • The right to object
  • The right not to be subject to automated decision-making including profiling

Many of these rights, existing within the older EU Directive, have been strengthened. What's new in GDPR are three key rights:

  1. The Right to Erasure: This is designed to give individuals the right to be forgotten if the data held on them is no longer needed; the individual withdraws consent or objects to its use; there's a legal reason to remove it or it has been processed unlawfully; the data is related to a child / children. There is a responsibility for the operator to remove data and inform third-parties of the change.
  2. The Right to Restriction of Processing: This is designed to make it easier to contest the accuracy or lawful processing of data where there is an objection regarding the legitimacy of the processing. If an individual asserts this right, his or her data can only continue to be processed for defense of legal claims or with the consent of the individual or for the protection of the rights of another person or an important public interest.
  3. The Right to Data Portability: This protects the rights of individuals to access all data held by third-parties and to be able to view it in a commonly used format. Under this right, individuals can ask one company to share personal data with another.

Granting New Access

GDPR requires that data controllers and data processors must be more transparent about how they collect data, how they process it, and how they intend to store it. This must be communicated in a clear and unambiguous way. Under GDPR, individuals have the right to access any information an organization holds on them, to know why it's being processed, who it can be accessed by, plus where (and for how long) it is stored. GDPR expects organizations to provide direct, secure access for people to review what information is held.

Handling Data Breaches

According to a report from IBM Security and the Ponemon Institute, the average total cost of a data breach is an estimated $3.62 million (or €2.9 million). The rules for handling data breaches within the GDPR framework are clear: organizations must inform their local data protection authority of a breach within 72 hours of detection.

As many organizations fail to detect breaches in a timely manner, it is seen as a difficult criteria to follow and the onus will be on IT departments to put in alerting systems that are capable of handling larger amounts of structured and unstructured data that come from multiple different sources. Next-generation fraud detection and alerting methodologies will become crucial in detecting threats before they have time to do damage.

The Cost of Non-Compliance

A major change in GDPR is the way that non-compliance will be penalized; the severity of the penalties should not be underestimated. Those who fail to comply could face a penalty of up to €20 million or 4% of a company's annual worldwide revenue, whichever is higher. When looking at recent Information Commissioner's Office (ICO) fines issued (maximum penalty of £500,000) then scaling according to GDPR rules, it's clear how much harsher the penalties for non-compliance could be. Incredibly, the total fines issued by the ICO in 2016 amounted to £880,500 — this amount would become tens of million under the new GDPR rules. It's pays to be compliant!

Securing Personal Data

The security requirements in GDPR are general in nature and avoid prescribing specific technologies or practices.

"...the controller and the processor shall implement appropriate technical and organisational measures to ensure a level of security appropriate to the risk.…" [Chapter IV, Section 2, Article 32]

Those familiar with other information security frameworks such as SOC and ISO 27000, will recognize the terms "technical and organizational measures" to be a sweeping term describing all the things an organization's people, processes, and technology must do in order to provide the confidentiality, integrity, and availability of data.

Transferring Personal Data

Moving Personal Data across borders can be complex. Transfers of Personal Data out of the EU to a country that is not deemed to provide an adequate level of protection are only permitted if the controller or processor provide appropriate safeguards as described in the GDPR. These safeguards may include standard data protection clauses adopted by the European Commission (i.e., "Model Clauses"), binding corporate rules, or an approved self-certification program such as the EU-US Privacy Shield.

GDPR Handling of Personal Data in a Nutshell

The simplified model below summarizes the decision process a GDPR Affected organization may consider when determining how it treats Personal Data.


In future posts in this series, we'll cover additional GDPR-related topics such as data onboarding, GDPR pseudonymization (say that three times quickly!), and access controls for GDPR.

For additional reading now, please check out our new white paper, GDPR Compliance and the Elastic Stack, or get in touch with an Elastic expert.

We'll be holding a GDPR Birds of a Feather discussion at our Elastic{ON} user conference next week.  If you'll be attending, make sure to stop by to participate.