Troubleshoot project scope changes

Changing project_routing (the Project scope field in Kibana) decides which linked projects a cross-project search datafeed searches. This page covers three related problems: Elasticsearch or Kibana rejects the change, a bulk update succeeds for some jobs but not others, or the change took effect and the anomaly detection job model is reacting to a different data set.

Use these sources to gather diagnostic information:

  • Anomaly detection job job messages in Kibana: Open Machine Learning → Anomaly Detection, select the job, and review the Job messages tab for audit entries and warnings about linked projects, credentials, or scope changes. On the Datafeed tab, View datafeed counts opens the datafeed chart flyout for extraction timing.

    The same entries are stored in .ml-notifications-*.

  • GET _ml/datafeeds/{datafeed_id}: Shows the effective project_routing value and, when an internal cloud API key exists, authorization.cloud_api_key.id.

  • GET _ml/datafeeds/{datafeed_id}/_stats: While the datafeed runs, shows remote_cluster_stats with total_clusters, available_clusters, skipped_clusters, availability_ratio, stabilized_cluster_aliases, and per_cluster_consecutive_skips. The object is absent until the first search cycle establishes a baseline.

  • .ml-annotations-read: Scope-change annotations for the job. The annotation event field carries search_scope_changed (not the separate type field).

  • GET /_project/tags: Lists linked projects and their tags so you can compare them with a routing expression.

  • Elastic Cloud console: Review linked projects in Link and manage projects.

If extraction failures are ongoing, check Job messages first. remote_cluster_stats from get datafeed stats only updates after a cycle completes.

If the datafeed is still running, the update fails with:

Cannot update datafeed [my-datafeed] while its status is started

If the anomaly detection job is still open, the update fails with:

Cannot update project_routing for datafeed [my-datafeed] while job [my-job] is opened. Close the job so a rollback model snapshot can be retained.

The bracketed datafeed id, job id, and job state vary with your configuration. From the API, a rejected update returns HTTP 409.

A job with no model snapshot can still change scope. Elasticsearch retains a rollback snapshot only when one already exists.

Assigning _alias:_origin for the first time to a datafeed that had no project_routing preserves local-only scope. That update does not require a closed job.

A bulk update stops and restarts a running datafeed for you, but it does not close the anomaly detection job. Because a project_routing change requires a closed job, open jobs fail the bulk update even though their datafeed was stopped automatically. Close those jobs first, then run the update again.

When you try to update a single job while the datafeed is running, the datafeed tab shows:

Datafeed settings cannot be edited while the datafeed is running. Please stop the job if you wish to edit these settings.
Entry point Fix
API update Stop the datafeed, close the anomaly detection job, then retry POST _ml/datafeeds/{datafeed_id}/_update.
Kibana bulk Change project scope Close every open job in the selection (the bulk action does not close jobs for you), then re-run the update.
Kibana single-job Datafeed tab Stop the datafeed so Project scope becomes editable, close the job if you are changing to a new effective scope, then save. 
				POST _ml/datafeeds/{datafeed_id}/_stop
				POST _ml/anomaly_detectors/{job_id}/_close

For routing expressions that match no project or reference stale aliases, see Project scope problems.

The update succeeds without HTTP 409. GET _ml/datafeeds/{datafeed_id} returns the new project_routing value.

When you bulk update multiple jobs, the bulk update might not be completely successful.

Failed jobs display the job id and a cross icon ( ). To learn why a job failed, open Job messages for that job or retry the update through the API and read the error response.

If the flyout cannot load the job list, Kibana shows Could not load jobs for project routing update. If the bulk API call fails before per-job results are returned, Kibana shows Project routing update failed.

A separate message appears if Kibana could not restart a datafeed after a successful routing change: Failed to restart datafeed for job job id.

After submission, a bulk update status message appears:

  • Partial: Project routing was updated for X of Y jobs. Any jobs that were previously running will need to be restarted if their update failed.

Jobs that fail show a cross icon in the flyout job list. Jobs that succeed show a check icon.

  1. Note every job ID that shows a cross icon in the flyout (or that appeared in the partial-failure message).
  2. For each failed job, open Job messages and read the API error. Common causes are an open job, invalid project_routing, or a credential problem.
  3. Fix the underlying cause per job. For credential failures, see Cloud credential problems.
  4. Re-run the update for the failed jobs only. Select just those jobs and use Change project scope again.
  5. When one job keeps failing, use the single-job path: edit the job, open the Datafeed tab, set Project scope, and save after closing the job.

Example retry for one job after stopping the datafeed and closing the job:

				POST _ml/datafeeds/{datafeed_id}/_stop
				POST _ml/anomaly_detectors/{job_id}/_close
				POST _ml/datafeeds/{datafeed_id}/_update
					{
  "project_routing": "_alias:production-*"
}

Adjust project_routing to the scope you chose in the bulk flyout.

Every selected job shows a check icon in the flyout, or the success toast reports all jobs updated.

When linked projects are added or removed (whether because you changed project_routing, or linked or unlinked a project in Elastic Cloud), the datafeed might search a different set of projects than when the model was trained. Elasticsearch records scope changes only after they stabilize. Temporary anomalies are expected while the model adapts.

Scope-change confirmations appear in job messages once stabilization completes. Examples with realistic alias names (linked/unlinked aliases vary):

When a project is newly linked:

Datafeed search scope changed: [staging] linked. Data distribution may have changed due to new data sources, which can cause temporary anomalies while the model adapts. If detection quality degrades, consider specifying the source clusters explicitly and reviewing recent model snapshots for potential rollback.

When a project is unlinked:

Datafeed search scope changed: [staging] unlinked. Data distribution may have changed due to removed data sources, which can cause temporary anomalies as patterns the model learned are no longer present. If detection quality degrades, consider specifying the source clusters explicitly and reviewing recent model snapshots for potential rollback.

When both happen in the same stabilization window:

Datafeed search scope changed: [production] linked, [staging] unlinked. Data distribution may have changed, which can cause temporary anomalies while the model adapts. If detection quality degrades, consider specifying the source clusters explicitly and reviewing recent model snapshots for potential rollback.

If anomaly scores spike after a confirmed change, job messages might also show:

Elevated anomaly scores detected after search scope change at [2026-07-28T10:15:00.000Z] (production linked). [12] buckets with anomaly score >= 75 observed since the scope change. This is likely caused by the data distribution shift. Consider reviewing model snapshots if the anomalies are not meaningful.

The timestamp, change summary in parentheses, and bucket count vary with your job.

When Elasticsearch retains a rollback snapshot, job messages include an entry such as:

Rollback model snapshot [1720000000] retained before project_routing scope change: Automatic rollback snapshot retained before project_routing scope change [_alias:_origin] -> [_alias:prod-*]

If the datafeed had no stored routing, the left side of the pair is empty. If the job had no snapshot, this message does not appear and the update still succeeds.

By default, Elasticsearch confirms a scope change only after the linked-project set stays stable for 12 consecutive extraction cycles and at least 5 minutes since the change was first observed. Until both conditions are met, you do not see scope-change messages or annotations even if projects were linked or unlinked in Elastic Cloud.

Elasticsearch writes scope-change annotations to .ml-annotations-read. The annotation event field carries search_scope_changed. Search that index for the job id and filter on event: search_scope_changed to see when scope stabilized.

If the scope change was intentional, allow several extraction cycles for the model to adapt. Monitor job messages and annotations until elevated-score warnings stop.

If the change was unintentional (for example a project was unlinked in Elastic Cloud), restore the link or update project_routing to the intended expression. See Project scope problems for routing syntax.

If detection quality degrades and anomalies are not meaningful, revert to the model snapshot Elasticsearch retained before the scope change, or to an earlier snapshot:

  1. Close the anomaly detection job.
  2. In Kibana, open Machine Learning → Anomaly Detection → your job → Model snapshots, select a snapshot from before the scope change (or the automatic rollback snapshot), and revert.
  3. Confirm project_routing on the datafeed matches your intended scope with GET _ml/datafeeds/{datafeed_id}.
  4. Start the datafeed.

New search_scope_changed annotations stop appearing and job messages no longer report scope-change or elevated-score warnings on every cycle. GET _ml/datafeeds/{datafeed_id}/_stats shows successful extraction cycles with remote_cluster_stats listing only the intended projects.

Edit page