Troubleshoot cloud credential problems

A cross-project search datafeed stores an internal cloud API key so periodic searches can read linked projects on your behalf. Elasticsearch mints that key only when you create or update the datafeed in Kibana, or through the API with an Elastic Cloud API key, not during a scheduled extraction cycle. Failures fall into three paths covered on this page: the key could not be created, an existing key no longer authorizes search, or the key was cleared or never minted.

Use these sources to gather diagnostic information:

  • Anomaly detection job job messages in Kibana: Open Machine Learning → Anomaly Detection, select the job, and review the Job messages tab for audit entries and warnings about linked projects, credentials, or scope changes. On the Datafeed tab, View datafeed counts opens the datafeed chart flyout for extraction timing.

    The same entries are stored in .ml-notifications-*.

  • GET _ml/datafeeds/{datafeed_id}: Shows the effective project_routing value and, when an internal cloud API key exists, authorization.cloud_api_key.id.

  • GET _ml/datafeeds/{datafeed_id}/_stats: While the datafeed runs, shows remote_cluster_stats with total_clusters, available_clusters, skipped_clusters, availability_ratio, stabilized_cluster_aliases, and per_cluster_consecutive_skips. The object is absent until the first search cycle establishes a baseline.

  • .ml-annotations-read: Scope-change annotations for the job. The annotation event field carries search_scope_changed (not the separate type field).

  • GET /_project/tags: Lists linked projects and their tags so you can compare them with a routing expression.

  • Elastic Cloud console: Review linked projects in Link and manage projects.

If extraction failures are ongoing, check Job messages first. remote_cluster_stats from get datafeed stats only updates after a cycle completes.

Elasticsearch probes the search with the caller's credential before storing a new internal key. Probe failures fail the create or update synchronously.

Missing index privileges in a linked project:

User lacks the required permissions to read datafeed indices on project [...].

On a specific index in the origin project or a qualified index pattern:

User lacks the required permissions to read datafeed index [...].

The bracketed project alias or index name varies. When no single index is named:

User lacks the required permissions to read from the datafeed indices.

Other probe failures surface as:

Datafeed search probe failed with status [FORBIDDEN]

The bracket shows a status name such as UNAUTHORIZED or FORBIDDEN, not an HTTP status code.

Routing that matches no linked project is reported separately. See Project scope problems.

Resolve missing index privileges, routing that matches nothing, or other probe errors before retrying create or update. Retry in Kibana, or through the API with an Elastic Cloud API key. Elasticsearch API keys are scoped to a single project. They cannot mint or re-key the internal credential, and they clear it instead.

The create or update succeeds. Job messages show Internal cloud API key minted for cross-project datafeed. GET _ml/datafeeds/{datafeed_id} includes authorization.cloud_api_key.id.

After minting succeeds, periodic searches can fail when the stored credential is invalid or lacks privileges. Job messages record:

Invalidated or expired key:

Internal cloud API key [abc123def456] failed authentication during datafeed search; it may have been revoked or expired. Re-key by issuing a cloud-authenticated POST _ml/datafeeds/_update on this datafeed

The bracketed value is the key identifier (the same authorization.cloud_api_key.id from GET _ml/datafeeds/{datafeed_id}), not the credential secret. Elasticsearch never logs or returns the internal key text.

Insufficient privileges:

Datafeed search was denied (forbidden) while using internal cloud API key [abc123def456]; the key's privileges or the requesting user's cross-project access may be insufficient. Verify the key and the datafeed owner's project privileges, then re-key with a cloud-authenticated update if the key is the cause

A mint or re-key followed by a runtime failure means the key worked at create or update time but later stopped authorizing search.

Stop the datafeed and issue a force re-key:

				POST _ml/datafeeds/my-datafeed/_update
					{
  "_force_rekeying": true
}

Replace my-datafeed with your datafeed id. Call _force_rekeying from Console while signed in to Kibana, or through the API with an Elastic Cloud API key. Elasticsearch API keys return a validation error.

Elasticsearch also re-keys automatically when you change the cross-project search surface from Kibana, or through the API with an Elastic Cloud API key: project_routing, indices, or indices_options. A no-op update without _force_rekeying does not replace an existing key.

				GET _ml/datafeeds/{datafeed_id}
				GET _ml/datafeeds/{datafeed_id}/_stats

Expect a new authorization.cloud_api_key.id, a re-key entry in job messages, and successful extraction cycles in _stats once linked projects authorize search again.

After re-keying, Elasticsearch best-effort revokes the old key. If revocation fails, job messages might show Failed to revoke internal cloud API key [abc123def456]. Search uses the new key. The message is informational and does not block recovery.

When authorization.cloud_api_key.id is missing, one of two things happened:

  • Credential cleared: job messages record Internal cloud API key cleared on datafeed update with non-cloud credentials after an update that used an Elasticsearch API key or other non-Elastic Cloud credential removed a stored key.
  • Never minted: the datafeed was created or last updated outside Kibana without an Elastic Cloud API key, so no key was ever stored and the cleared-key message never appears. Job messages lack Internal cloud API key minted for cross-project datafeed.

After clearing, GET _ml/datafeeds/{datafeed_id} omits authorization.cloud_api_key.id. Cross-project portions of the search fail until you update the datafeed in Kibana, or through the API with an Elastic Cloud API key.

Update the datafeed in Kibana, or through the API with an Elastic Cloud API key. When the credential was cleared, that update mints a new key even if the configuration is unchanged.

GET _ml/datafeeds/{datafeed_id} includes a new authorization.cloud_api_key.id. Job messages show Internal cloud API key minted for cross-project datafeed or Internal cloud API key re-keyed for cross-project datafeed update.

If a force re-key or surface-changing update in Kibana, or through the API with an Elastic Cloud API key, completes but cross-project search still fails, contact Elastic support with:

  • Origin project id
  • anomaly detection job and datafeed ids
  • authorization.cloud_api_key.id before and after the update
  • Relevant job message excerpts (probe errors, runtime failures, or lifecycle entries)
Edit page