Troubleshoot linked project unavailable

When a linked project in the datafeed's scope is skipped during a search cycle, Elasticsearch fails the entire extraction cycle instead of continuing on the remaining projects. Partial cross-project results could produce misleading anomalies, so the cycle produces no data. The error is audited and the datafeed retries on its next scheduled cycle. You see repeated extraction errors in job messages and gaps in results for the affected time buckets until every project in scope is reachable again, or you narrow project_routing. A hard cluster failure with status FAILED rather than SKIPPED surfaces through the ordinary search-failure path with different job message text.

Use these sources to gather diagnostic information:

  • Anomaly detection job job messages in Kibana: Open Machine Learning → Anomaly Detection, select the job, and review the Job messages tab for audit entries and warnings about linked projects, credentials, or scope changes. On the Datafeed tab, View datafeed counts opens the datafeed chart flyout for extraction timing.

    The same entries are stored in .ml-notifications-*.

  • GET _ml/datafeeds/{datafeed_id}: Shows the effective project_routing value and, when an internal cloud API key exists, authorization.cloud_api_key.id.

  • GET _ml/datafeeds/{datafeed_id}/_stats: While the datafeed runs, shows remote_cluster_stats with total_clusters, available_clusters, skipped_clusters, availability_ratio, stabilized_cluster_aliases, and per_cluster_consecutive_skips. The object is absent until the first search cycle establishes a baseline.

  • .ml-annotations-read: Scope-change annotations for the job. The annotation event field carries search_scope_changed (not the separate type field).

  • GET /_project/tags: Lists linked projects and their tags so you can compare them with a routing expression.

  • Elastic Cloud console: Review linked projects in Link and manage projects.

If extraction failures are ongoing, check Job messages first. remote_cluster_stats from get datafeed stats only updates after a cycle completes.

Skipped or failed linked projects surface in the anomaly detection job's Job messages tab or .ml-notifications-*. The outer audit entry wraps the skip summary as its cause:

Datafeed is encountering errors extracting data: [1] remote clusters out of [3] were skipped when performing datafeed search

The bracketed skip and total counts vary with how many linked projects are in scope. The wrapped text is the message thrown when any linked project is skipped during the search.

Inspect remote_cluster_stats using get datafeed stats.

During an active skip outage, skipped_clusters can remain 0 while skip errors repeat in job messages because extraction aborts before stats are updated.

After cycles complete successfully, compare per_cluster_consecutive_skips and availability_ratio with project_routing from GET _ml/datafeeds/{datafeed_id} and linked projects from GET /_project/tags or the Cloud console. Field names and structure match the stats API, but values vary with your configuration:

"remote_cluster_stats": {
  "total_clusters": 3,
  "available_clusters": 3,
  "skipped_clusters": 0,
  "availability_ratio": 1.0,
  "stabilized_cluster_aliases": ["origin", "production", "staging"],
  "per_cluster_consecutive_skips": {}
}
Pattern Likely cause
One datafeed skips a project that other datafeeds search successfully. project_routing names an unlinked alias or a project that is no longer linked Configuration: stale or wrong routing
Many unrelated datafeeds skip the same linked project at the same time. Cross-project queries fail broadly Platform outage or regional cross-project search connectivity degradation

For routing or stale alias problems, see Project scope problems.

An internal cloud API key problem can also stop cross-project searches. The distinguishing signal is a credential-specific message in job messages, for example an authentication failure on the internal cloud API key or a forbidden response while using that key, rather than a skip summary. See Cloud credential problems.

If the linked project was removed or never linked, re-establish it in Link and manage projects, then wait for the next extraction cycle or restart the datafeed.

When a project stays unreachable and you need the datafeed to keep producing results from the remaining scope, narrow project_routing to an _alias: expression that excludes the unavailable project. Stop the datafeed and close the anomaly detection job before running this update.

				POST _ml/datafeeds/{datafeed_id}/_stop
				POST _ml/anomaly_detectors/{job_id}/_close
				POST _ml/datafeeds/{datafeed_id}/_update
					{
  "project_routing": "_alias:production-*"
}

Use an expression that matches only the linked projects you still need. Changing scope can affect the model. See Changing project scope.

When the underlying link or platform issue clears, the datafeed recovers on the next successful cycle without configuration changes. Job messages might show:

Datafeed has recovered data extraction and analysis

After a long period with no ingested data, you might also see:

Datafeed has started retrieving data again

Each failed extraction cycle leaves a gap in the time series the anomaly detection job analyzes. If a project stays out of scope for an extended period, whether because of outages or because you narrowed routing, the model adapts to the changed data distribution. See Scope changed and model is reacting for rollback options.

Job messages stop reporting new extraction errors for the skip cause. After at least one successful completed cycle, remote_cluster_stats shows skipped_clusters at 0 and no non-zero entries in per_cluster_consecutive_skips.

If many unrelated datafeeds still skip the same linked project after you confirm the project link and project_routing are correct, contact Elastic support with the project id, datafeed id, and the extraction error message text.

Edit page