Loading

Ingest data with the Managed Elasticsearch _bulk endpoint

The Managed Elasticsearch _bulk endpoint ingests data sent in the Elasticsearch _bulk API format. It accepts _bulk traffic natively, so shippers that already write to Elasticsearch can send data through managed inputs by pointing their existing Elasticsearch output at the endpoint. It's a dedicated managed input exposed on the /_es path of the same ingest host as the Managed OTLP Endpoint and the Managed Prometheus Remote Write endpoint.

The endpoint is Elasticsearch-compatible: it emulates a subset of the _bulk API, so most shippers need only a new endpoint and credentials to start sending data. The managed input then durably buffers the data and routes it into Elasticsearch. The endpoint accepts only log data. Bulk actions must use the create action.

Use the Managed Elasticsearch _bulk endpoint to bring data shippers that rely on the Elasticsearch _bulk API into managed ingestion, including:

  • Beats
  • Elastic Agent
  • Logstash
  • Any other shipper that sends data using the Elasticsearch _bulk API

Compared to sending _bulk requests directly to Elasticsearch, the Managed Elasticsearch _bulk endpoint provides:

  • A single ingest endpoint and API key shared with the other managed inputs.
  • Durable buffering, back-pressure, and automatic retries before data reaches Elasticsearch.
  • A low-friction path for existing Beats, Elastic Agent, and Logstash deployments to adopt managed ingestion without re-architecting their pipelines.
Note

On Elastic Cloud Serverless and Elastic Cloud Hosted, use the Managed Elasticsearch _bulk endpoint instead of sending _bulk requests directly to Elasticsearch. Direct ingest bypasses managed inputs, so it has no durable buffering or managed processing before data reaches Elasticsearch, and it authenticates with Elasticsearch credentials or an API key with index privileges instead of an API key for managed inputs.

  • An Elastic Cloud Serverless Observability or Security project, or an Elastic Cloud Hosted deployment on Elastic Stack version 9.0 or later.
  • A _bulk-compatible shipper that can send create actions, such as Beats, Elastic Agent, Logstash, or another shipper with an Elasticsearch output.
  • An API key with the event:write privilege for the apm application. Refer to Authentication for the required key format and generation steps.
  • Any Elasticsearch index templates and Kibana assets your shipper relies on, installed beforehand. The endpoint doesn't install them for you. Refer to Limitations.

The Managed Elasticsearch _bulk endpoint uses the same ingest host as the Managed OTLP Endpoint, with the /_es path appended. Its exact format depends on your deployment type. Copy the exact value from the Elastic Cloud Console, as described in the following steps.

  1. Find your endpoint

    To find your Managed Elasticsearch _bulk endpoint:

    1. Log in to the Elastic Cloud Console.
    2. Do one of the following:
      • Elastic Cloud Serverless: Find your project and select Manage.
      • Elastic Cloud Hosted: Find your deployment in Hosted deployments and select Manage.
    3. In the Application endpoints, cluster and component IDs section, select Elasticsearch, then copy the _bulk endpoint value.

    This endpoint value is shown as <managed-_bulk-endpoint> in the examples that follow. Your shipper appends the _bulk path (and any target index path) itself, so requests are sent to POST <managed-_bulk-endpoint>/_bulk.

  2. Authenticate

    Create an API key suitable for managed inputs as described in Authentication. Send this key in each request's Authorization header as ApiKey <api-key>:

    Authorization: ApiKey <api-key>

    Shippers that call GET /_es or GET /_es/_license at startup can use the same API key. You don't need to grant additional privileges for those requests.

  3. Configure your shipper and send data

    To send data, configure your shipper's Elasticsearch output with the following:

    • Endpoint: Use your Managed Elasticsearch _bulk endpoint value (<managed-_bulk-endpoint>) in the host format for your deployment type. If your shipper has a separate path or URL-prefix setting, set it to /_es instead of including it in the host.
    • Authentication: The API key you created for the managed inputs, sent as the HTTP header Authorization: ApiKey <api-key>.
    • Action: Use create. Shippers that write to data streams already use this action.

    The following example configures a Logstash elasticsearch output. Setting names and the exact way to pass credentials vary by shipper and version, so validate the configuration for your shipper:

    output {
  elasticsearch {
    hosts => ["<managed-_bulk-endpoint>"]
    custom_headers => {
      "Authorization" => "ApiKey <api-key>"
    }
    action => "create"
  }
}

    Beats and Elastic Agent configure their Elasticsearch output the same way: they point the output hosts at <managed-_bulk-endpoint> and provide their API key. If your shipper uses the index, update, or delete action, switch to create or target a data stream. Refer to Limitations.

    To confirm your setup is working, check that your shipper reports successful (2xx) responses with no authentication errors, then open Discover and verify that new documents are landing in your target data stream.

    If documents don't appear, they might have failed during asynchronous indexing. These errors aren't reported in the bulk response, so use Data Set Quality to monitor and triage indexing issues. For more details, refer to Indexing errors and the failure store.

Each action in a _bulk request can specify its target through the _index field, so data lands in the data stream or index that your shipper already targets. For example, a shipper writing nginx access logs to logs-nginx.access-default continues to land there. If your shipper sets a fallback target in the request path (/_es/<target>/_bulk), that target is used for actions that omit _index.

The Managed Elasticsearch _bulk endpoint emulates the Elasticsearch _bulk API, but because it ingests through managed inputs, it behaves differently from indexing directly into Elasticsearch. Keep the following in mind:

  • Batches are atomic. The endpoint either durably enqueues the entire batch and returns success, or rejects the entire request. There's no per-document partial success or failure. If a valid batch can't be enqueued, the whole request fails with 503 Service Unavailable. Malformed requests, unsupported actions, or missing targets fail with 400 Bad Request.
  • A success response means the data is durably enqueued, not indexed. A successful response returns an Elasticsearch-compatible body in which each item reports a 201 status. This confirms the managed input durably accepted the document, not that Elasticsearch has indexed it. Errors that occur later during indexing, such as mapping conflicts, happen asynchronously and aren't reported in the bulk response.
  • Compressed requests are supported. The endpoint accepts Content-Encoding: gzip request bodies.

For shared buffering and delivery behavior across managed inputs, refer to Buffering and delivery.

Because indexing happens asynchronously, indexing failures such as mapping conflicts aren't reported in the bulk response, and the endpoint doesn't provide client-side visibility into them. To confirm your data was indexed, verify that documents landed in the destination data stream and use Data Set Quality to monitor indexing issues. For more detail on how indexing errors are handled, refer to Indexing errors and the failure store.

Under load, or when the service can't accept more data, the endpoint can respond with 429 Too Many Requests or 503 Service Unavailable. Configure your shipper to retry these responses with backoff and to queue data locally during transient rejections. For how rate limiting works and how it differs between Elastic Cloud Serverless and Elastic Cloud Hosted, refer to Managed inputs rate limiting.

The following limitations apply when using the Managed Elasticsearch _bulk endpoint:

  • Only create actions are supported. Requests that use index, update, or delete actions are rejected with 400 Bad Request. This endpoint accepts only log data. For Logstash, the elasticsearch output must use action => "create". Features that depend on other actions, such as scripted upserts, aren't supported.
  • Duplicate detection isn't applied. The endpoint doesn't deduplicate documents by _id, so client retries can produce duplicate documents.
  • No client-side visibility into indexing outcomes. Because batches are accepted asynchronously, the bulk response confirms only that data was enqueued, not indexed. Shippers don't receive per-document indexing errors, such as mapping conflicts, in their responses, so monitor indexing separately in Kibana with Data Set Quality.
  • Index templates, index lifecycle management (ILM) policies, and Kibana assets can't be installed through the endpoint, which serves only the root (/_es), license (/_es/_license), and _bulk paths (/_es/_bulk and /_es/<target>/_bulk). Beats, Elastic Agent, and Logstash setup steps that create index templates or load dashboards must run against Elasticsearch and Kibana directly before you send data.
  • For Elastic Cloud Hosted network limitations that apply to all managed inputs, refer to Elastic Cloud Hosted limitations.