API keysedit

API keys are sent as plain-text, so they only provide security when used in combination with TLS.

When enabled, API keys are used to authorize requests to the APM Server. API keys are not applicable for APM agents running on clients, like the RUM agent, as there is no way to prevent them from being publicly exposed.

You can assign one or more unique privileges to each API key:

  • Agent configuration (config_agent:read): Required for agents to read Agent configuration remotely.
  • Ingest (event:write): Required for ingesting agent events.

To secure the communication between APM Agents and the APM Server with API keys, make sure TLS is enabled, then complete these steps:

Enable API keysedit

Enable API key authorization in the API key authentication options. You should also set a limit on the number of unique API keys that APM Server allows per minute; this value should be the number of unique API keys configured in your monitored services.

Create an API key user in Kibanaedit

API keys can only have the same or lower access rights than the user that creates them. Instead of using a superuser account to create API keys, you can create a role with the minimum required privileges.

The user creating an APM agent API key must have at least the manage_own_api_key cluster privilege and the APM application-level privileges that it wishes to grant. In addition, when creating an API key from the APM app, you’ll need the appropriate Kibana Space and Feature privileges.

The example below uses the Kibana role management API to create a role named apm_agent_key_role.

POST /_security/role/apm_agent_key_role
{
   "cluster": [ "manage_own_api_key" ],
   "applications": [
      {
         "application":"apm",
         "privileges":[
            "event:write",
            "config_agent:read"
         ],
         "resources":[ "*" ]
      },
      {
         "application":"kibana-.kibana",
         "privileges":[ "feature_apm.all" ],
         "resources":[ "space:default" ] 
      }
   ]
}

This example assigns privileges for the default space.

Assign the newly created apm_agent_key_role role to any user that wishes to create APM agent API keys.

Create an API key in the APM appedit

The APM app has a built-in workflow that you can use to easily create and view APM agent API keys. Only API keys created in the APM app will show up here.

Using a superuser account, or a user with the role created in the previous step, open Kibana and navigate to Observability > APM > Settings > Agent keys. Enter a name for your API key and select at least one privilege.

For example, to create an API key that can be used to ingest APM events and read agent central configuration, select config_agent:read and event:write.

Click Create APM Agent key and copy the Base64 encoded API key. You will need this for the next step, and you will not be able to view it again.

APM app API key
Set the API key in your APM agentsedit

You can now apply your newly created API keys in the configuration of each of your APM agents. See the relevant agent documentation for additional information:

Alternate API key creation methodsedit

API keys can also be created and validated outside of Kibana:

APM Server API key workflowedit

This API creation method only works with the APM Server binary.

Deprecated in 8.6.0.

Users should create API Keys through Kibana or the Elasticsearch REST API

APM Server provides a command line interface for creating, retrieving, invalidating, and verifying API keys. Keys created using this method can only be used for communication with APM Server.

apikey subcommandsedit
create

Create an API Key with the specified privilege(s). No required flags.

The user requesting to create an API Key needs to have APM privileges used by the APM Server. A superuser, by default, has these privileges. For other users, you can create them. See create an API key user for required privileges.

info
Query API Key(s). --id or --name required.
invalidate
Invalidate API Key(s). --id or --name required.
verify
Check if a credentials string has the given privilege(s). --credentials required.
Privilegesedit

If privileges are not specified at creation time, the created key will have all privileges.

  • --agent-config grants the config_agent:read privilege
  • --ingest grants the event:write privilege
  • --sourcemap grants the sourcemap:write privilege
Create an API keyedit

Create an API key with the create subcommand.

The following example creates an API key with a name of java-001, and gives the "agent configuration" and "ingest" privileges.

apm-server apikey create --ingest --agent-config --name java-001

The response will look similar to this:

Name ........... java-001
Expiration ..... never
Id ............. qT4tz28B1g59zC3uAXfW
API Key ........ rH55zKd5QT6wvs3UbbkxOA (won't be shown again)
Credentials .... cVQ0dHoyOEIxZzU5ekMzdUFYZlc6ckg1NXpLZDVRVDZ3dnMzVWJia3hPQQ== (won't be shown again)

You should always verify the privileges of an API key after creating it. Verification can be done using the verify subcommand.

The following example verifies that the java-001 API key has the "agent configuration" and "ingest" privileges.

apm-server apikey verify --agent-config --ingest --credentials cVQ0dHoyOEIxZzU5ekMzdUFYZlc6ckg1NXpLZDVRVDZ3dnMzVWJia3hPQQ==

If the API key has the requested privileges, the response will look similar to this:

Authorized for privilege "event:write"...:          Yes
Authorized for privilege "config_agent:read"...:    Yes

To invalidate an API key, use the invalidate subcommand. Due to Elasticsearch caching, there may be a delay between when this subcommand is executed and when it takes effect.

The following example invalidates the java-001 API key.

apm-server apikey invalidate --name java-001

The response will look similar to this:

Invalidated keys ... qT4tz28B1g59zC3uAXfW
Error count ........ 0

A full list of apikey subcommands and flags is available in the API key command reference.

Elasticsearch API key workflowedit

It is also possible to create API keys using the Elasticsearch create API key API.

This example creates an API key named java-002:

POST /_security/api_key
{
  "name": "java-002", 
  "expiration": "1d", 
  "role_descriptors": {
    "apm": {
      "applications": [
        {
          "application": "apm",
          "privileges": ["sourcemap:write", "event:write", "config_agent:read"], 
          "resources": ["*"]
        }
      ]
    }
  }
}

The name of the API key

The expiration time of the API key

Any assigned privileges

The response will look similar to this:

{
  "id" : "GnrUT3QB7yZbSNxKET6d",
  "name" : "java-002",
  "expiration" : 1599153532262,
  "api_key" : "RhHKisTmQ1aPCHC_TPwOvw"
}

The credential string, which is what agents use to communicate with APM Server, is a base64 encoded representation of the API key’s id:api_key. It can be created like this:

echo -n GnrUT3QB7yZbSNxKET6d:RhHKisTmQ1aPCHC_TPwOvw | base64

You can verify your API key has been base64-encoded correctly with the Authenticate API:

curl -H "Authorization: ApiKey R0gzRWIzUUI3eVpiU054S3pYSy06bXQyQWl4TlZUeEcyUjd4cUZDS0NlUQ==" localhost:9200/_security/_authenticate

If the API key has been encoded correctly, you’ll see a response similar to the following:

{
   "username":"1325298603",
   "roles":[],
   "full_name":null,
   "email":null,
   "metadata":{
      "saml_nameid_format":"urn:oasis:names:tc:SAML:2.0:nameid-format:transient",
      "saml(http://saml.elastic-cloud.com/attributes/principal)":[
         "1325298603"
      ],
      "saml_roles":[
         "superuser"
      ],
      "saml_principal":[
         "1325298603"
      ],
      "saml_nameid":"_7b0ab93bbdbc21d825edf7dca9879bd8d44c0be2",
      "saml(http://saml.elastic-cloud.com/attributes/roles)":[
         "superuser"
      ]
   },
   "enabled":true,
   "authentication_realm":{
      "name":"_es_api_key",
      "type":"_es_api_key"
   },
   "lookup_realm":{
      "name":"_es_api_key",
      "type":"_es_api_key"
   }
}

You can then use the APM Server CLI to verify that the API key has the requested privileges:

apm-server apikey verify --credentials R25yVVQzUUI3eVpiU054S0VUNmQ6UmhIS2lzVG1RMWFQQ0hDX1RQd092dw==

If the API key has the requested privileges, the response will look similar to this:

Authorized for privilege "config_agent:read"...:  Yes
Authorized for privilege "event:write"...:        Yes
Authorized for privilege "sourcemap:write"...:    Yes