Loading

Authentication, delivery, and failure handling with managed inputs

This page covers what all managed inputs share: how you authenticate, how data is stored for a limited time, how it is delivered, what happens when indexing fails, and network limitations on Elastic Cloud Hosted. For protocol-specific setup, choose your endpoint in Next steps.

Managed inputs authenticate with an Elasticsearch API key that includes the event:write privilege for the apm application. The same event:write / apm privilege applies to every managed endpoint, including the OTLP, Prometheus Remote Write, and _bulk endpoints.

You can create this API key in one of the following ways:

From the Add data flow
  1. Select Add data in the navigation menu of your Serverless Observability project or Elastic Cloud Hosted deployment.
  2. In the Connect directly to the endpoint section, select the managed endpoint for which you want to create an API key.
  3. Select Create key, then copy the encoded value from the API key field.
From the API keys management page in Kibana

  1. Go to the API keys management page in the navigation menu or use the global search field.

  2. Select Create API key, enter a name for the key, and enable Control security privileges.

  3. In the role descriptors box, enter the following privileges:

    {
  "managed-inputs-writer": {
    "applications": [
      {
        "application": "apm",
        "resources": ["*"],
        "privileges": ["event:write"]
      }
    ]
  }
}

  4. Select Create API key and copy the encoded value.

For more details, refer to Elasticsearch API keys and Serverless project API keys.

Using the Create API key API

Use the Create API key API. The same request works for both Elastic Cloud Serverless and Elastic Cloud Hosted. For example, to create an API key named managed-inputs-api-key:

				POST /_security/api_key
					{
  "name": "managed-inputs-api-key",
  "role_descriptors": {
    "managed-inputs-writer": {
      "applications": [
        {
          "application": "apm",
          "resources": ["*"],
          "privileges": ["event:write"]
        }
      ]
    }
  }
}

Send the encoded API key in the Authorization header of each request to the managed endpoint as ApiKey <encoded-api-key>. For example:

Authorization: ApiKey <api-key>
Note

Index-level privilege scoping is not supported for managed inputs.

Managed inputs provide a durable ingest layer in front of Elasticsearch:

  • Incoming data is stored (buffered) in a durable ingest layer before it reaches your Elasticsearch cluster. Buffered data is held for a limited time before it must be delivered.
  • When capacity controls reject data, endpoints can respond with 429 Too Many Requests, so clients should retry with backoff. Other temporary service failures can return 503 Service Unavailable. Refer to Managed inputs rate limiting.
Note

For the Managed Elasticsearch _bulk endpoint, a batch is atomic: the endpoint accepts or rejects the whole batch, and a 201 per item means the data is durably enqueued, not indexed. For details, refer to Delivery behavior.

To confirm your data was indexed, verify that documents landed in the destination data stream, and use Data Set Quality to monitor and triage indexing issues.

A successful accept response from a managed input means the data was durably accepted for processing, not that Elasticsearch has indexed it. Indexing errors, such as mapping conflicts or ingest pipeline errors, can happen asynchronously after the data is accepted, and aren't reported back to the client.

Managed inputs don't enable or manage the failure store. The failure store is an Elasticsearch data stream setting. If the destination data stream has it enabled, documents that fail indexing are written there. If it isn't enabled, those documents aren't captured.

Warning

If a document fails indexing and the destination data stream doesn't have the failure store enabled, the document is dropped. Because indexing errors happen after the data is accepted, your shipper still reports success, so this data loss is silent.

In Elastic Cloud Hosted deployments, the following limitations apply to managed inputs:

  • IP filters do not apply to managed endpoints.
  • Managed endpoints are not available over a private connection. When private connectivity is configured, the public managed endpoint is still available.

After you understand authentication and delivery, configure the endpoint for your protocol: