Introducing unified API keys for Elastic Cloud Serverless and Elasticsearch

Free yourself from operations with Elastic Cloud Serverless. Scale automatically, handle load spikes, and focus on building—start a 14-day free trial to test it out yourself!

You can follow these guides to build an AI-Powered search experience or search across business systems and software.

Imagine you're a site reliability engineer (SRE) responsible for a growing fleet of Elastic Cloud Serverless projects: Elastic Observability for your production infrastructure, Elastic Security for your security operations center (SOC) team, and Elasticsearch for your customer-facing application. Each project has its own Elasticsearch API key. Your continuous integration and continuous delivery (CI/CD) pipeline needs a separate Cloud API key to provision and manage those projects. Rotation day arrives every quarter: You walk through each project, mint fresh keys, update your Terraform state, redeploy your pipelines, and hope nothing falls through the cracks. When an incident hits at 2 a.m. and you need to revoke access fast, you're cross-referencing a spreadsheet of credentials to figure out which key belongs to which project and which service.

Today, that story gets a lot simpler. Elastic Cloud API keys can now be used to authenticate directly against Elasticsearch and Kibana APIs on Elastic Cloud Serverless. You can now use a single credential to manage your organization's resources and run data operations, like Elasticsearch Query Language (ES|QL) queries, data ingestion, and alerting.

Let’s look at why we built this, how we engineered a globally distributed identity layer to make it possible, and how it lays the foundation for cross-project search.

The secret management burden

Building reliable CI/CD pipelines, GitOps workflows, or Terraform automation around data platforms comes with a hidden cost: secret sprawl.

In the previous model, developers faced a disjointed authentication story:

• Control plane (Elastic Cloud API keys): Organization-scoped keys used to create projects, invite users, and manage billing via the Elastic Cloud API.
• Data plane (Elasticsearch API keys): Project-scoped keys created inside a specific Serverless project to interact with Elasticsearch and Kibana APIs.

This meant that your deployment script had to authenticate to Elastic Cloud, provision a Serverless project, extract a newly minted Elasticsearch API key from that specific project, and then inject that second key into the downstream application or automation tool, resulting in complex pipelines, fragmented audit logs, and a higher risk of credential leaks.

Unified authentication in Elastic Cloud Serverless

With this release, the split is gone for Serverless projects. You can now create an Elastic Cloud API key that’s explicitly authorized for Cloud, Elasticsearch, and Kibana APIs.

• Before: An Elastic Cloud API key was strictly a control plane token. It could create projects, manage billing, and invite users, but it had a hard boundary; it couldn’t be used to call the Elasticsearch or Kibana APIs inside those projects. You always needed a second, project-specific key for data operations.
• Now: By opting into Cloud, Elasticsearch, and Kibana API access when creating an Elastic Cloud API key, the hard boundary is removed for Serverless. That API key becomes a truly unified credential. It retains its ability to manage your organization's infrastructure, while simultaneously gaining native access to query, ingest, and analyze data across any authorized Serverless project.

By unifying this under a single Elastic Cloud API key, you gain a single identity that can be scoped, audited, rotated, and revoked as one unit. Every API call, whether it provisions a new project or runs an ES|QL query, appears under the same credential in your audit logs, giving you a single trail to follow during incident investigations or compliance reviews. Credential rotation becomes a one-step operation instead of a coordinated update across separate control-plane and data-plane secrets. And because role assignments are per-project, a single key can span several projects, managing ingestion in your observability project and running queries in your security project, without juggling separate credentials for each.

Importantly, unified does not mean all-powerful. By using the role_assignments payload, you can scope a unified key strictly to a single project and a specific role (such as read-only), ensuring the blast radius remains completely contained if a credential is ever exposed. If a developer leaves or an application is decommissioned, you can revoke a single key from the Elastic Cloud Console, immediately terminating access across both the control plane and all associated Elasticsearch projects.

(Note: For Elastic Cloud Hosted/managed deployments, Cloud API keys still only manage the control plane. Support for extending this to hosted stack APIs is planned for a future release.)

Automating your workflows

Getting started is simple. You can configure this entirely through the Elastic Cloud console or automate it using the Elastic Cloud API.

The UI process remains the same, but now you can select Cloud, Elasticsearch, and Kibana API access under the project role assignment.

Here’s how you create a unified key programmatically using the Elastic Cloud API. Notice the application_roles array, as this is what grants the key native access to the Elasticsearch data plane:

Once created, you simply pass this exact same key in the Authorization: ApiKey header to both api.elastic-cloud.com and your specific Serverless Elasticsearch endpoints.

Under the hood: Building a distributed identity layer

Making a Cloud API key work across both the control plane and the data plane isn't as simple as passing a token. It requires solving a fundamental distributed systems challenge.

Historically, Cloud API keys lived in a centralized global security cluster. That works fine for control plane operations where a higher latency is acceptable. However, Elasticsearch data requests require ultra-low latency. We cannot afford a round trip across the globe to a central control plane to validate every single search query or ingest request.

To solve this, we introduced a new authentication architecture backed by a globally distributed datastore. The following sequence diagram shows a client sending an Elasticsearch query using an Elastic Cloud API key, illustrating how authentication happens entirely within the local region, without a round trip to the global control plane. Elasticsearch delegates authentication to the Regional IAM Service, which validates the key and resolves its role assignments against a local replica of the globally distributed database. Once authorized, Elasticsearch executes the query and returns results to the client.

Globally distributed persistence

Instead of relying solely on a centralized security cluster, Elastic Cloud API keys and their associated role definitions are now persisted in a globally distributed, highly available database. This database synchronizes identity and access management (IAM) data across the global control plane and the regional data planes where your Serverless projects actually run.

Local validation with regional IAM

When your client sends a request to Elasticsearch using an Elastic Cloud API key, the request doesn't go back to the global control plane. Instead, it gets routed to the new regional IAM service. It validates the key against the local database replica, ensuring that authentication happens with near-zero latency and is completely insulated from global control plane outages.

Dynamic role mapping

Authentication is only half the battle; the system also needs to authorize the request. The regional IAM service instantly translates your Cloud-level role assignments for example, application_roles) into native Elasticsearch privileges. Elasticsearch can then authorize and execute the request locally, without ever needing a local .security index.

The foundation for Cross-Project Search

This distributed identity architecture is a foundational building block for the future of the Elastic platform.

Because identity and access are now unified and globally synchronized, we have the framework required to securely pass your identity between different projects. This enables the upcoming Cross-Project Search (CPS) capabilities for Serverless.

With CPS, you'll be able to query data spanning multiple remote Serverless projects, such as combining security and observability workloads, as easily as if they were a single dataset. By relying on unified API keys, the system can automatically evaluate your permissions across all projects simultaneously without requiring you to configure complex trust relationships, certificates, or duplicate credentials on every target project.

Learn more

Ready to simplify your stack?

• Read the Elastic Cloud API keys documentation to learn how to assign stack access.
• Check out the Create API key (Elastic Cloud API) reference to automate key generation.
• Review Elastic API keys for a full comparison of key types across the Elastic platform.

Start or continue building in Elastic Cloud today.

Disclaimer

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.