Elastic ML predicts when your disk will fill up: How to make it alert you

Elastic ML can already forecast disk usage 14 days out, with a confidence band on the upper and lower bounds. Kibana Workflows in Elastic 9.4 puts that forecast into a daily routine. One YAML file checks the confidence band against a 90% threshold and sends a Slack alert listing every host and mount point about to breach it before it happens.

Here's the full picture of what we're building:

What is Elastic ML forecasting?

Elastic's anomaly detection jobs model the normal behavior of a time series. Once the model has learned the pattern, the forecast API can project it forward with a confidence interval. A POST /_ml/anomaly_detectors/{job_id}/_forecast call runs the projection for a specified duration and writes the results into the job's results index as model_forecast documents.

Each document carries three values: forecast_prediction (the point estimate), forecast_lower, and forecast_upper (the bounds of a 95% confidence interval).

For capacity alerting, we want forecast_upper, not forecast_prediction. Think of it like a weather forecast: we'd rather carry an umbrella on a day that turns out sunny than get caught in the rain because we trusted the optimistic estimate.

Setting up the ML job

The ML job is a one-time setup. It can live for the lifetime of the cluster. It needs at least a few days of data before forecasts become meaningful; for disk capacity trends, plan on a week or two.

We could simply use the built-in job max_disk_utilization_ecs available in the supplied anomaly detection configurations in Kibana.

For the purposes of this demonstration, a slightly more advanced job was created that additionally supplies the mounting point information. This way, the alert will contain not only the information about which host is running out of storage, but also which mounting point is affected.

{
  "job_id": "disk-usage-forecast",
  "analysis_config": {
    "bucket_span": "1h",
    "detectors": [
      {
        "function": "max",
        "field_name": "system.filesystem.used.pct",
        "partition_field_name": "host.name",
        "by_field_name": "system.filesystem.mount_point"
      }
    ],
    "influencers": ["host.name", "system.filesystem.mount_point"]
  },
  "data_description": {
    "time_field": "@timestamp"
  },
  "datafeed_config": {
    "datafeed_id": "datafeed-disk-usage-forecast",
    "indices": ["metrics-system.filesystem-*"],
    "query": {
      "bool": {
        "filter": [
          { "exists": { "field": "system.filesystem.used.pct" } },
          { "exists": { "field": "host.name" } },
          { "exists": { "field": "system.filesystem.mount_point" } }
        ]
      }
    }
  },
  "model_plot_config": { "enabled": false },
  "results_index_name": "disk-usage-forecast",
  "results_retention_days": 28
}

A few choices worth calling out:

1. Double-split by host and mount. partition_field_name: host.name and by_field_name: system.filesystem.mount_point gives each host-mount combination its own independent model. A host with five partitions gets five independent trend lines. Field implementations have targeted this pattern at ~50,000 splits (roughly 10,000 hosts x 5 mount points). At that scale, bucket-span tuning and job splitting become necessary (see the scaling section below).

2. model_plot_config.enabled: false. The Kibana UI warns at around 100 series for the model plot; that warning doesn't apply once the model plot is off. We're alerting on the forecast results index, not on model plot data, so we don't need it.

3. Bucket span. The config uses 1h, which gives granular breach timestamps in the Slack alert. For production, many teams prefer 6h or 1d for disk capacity monitoring as it reduces the results index volume and smooths short-term noise.

4. Adjust the index pattern (metrics-system.filesystem-*) to match your Elastic Agent or Metricbeat setup.

After creating the job, let's open it and start the datafeed:

POST /_ml/anomaly_detectors/disk-usage-forecast/_open
PUT /_ml/datafeeds/datafeed-disk-usage-forecast/_start

Then wait for the model to process enough historical data before running the workflow.

Why forecast rather than alert on current disk usage?

A threshold rule fires when a disk hits 90%. At that point, we're in incident-response mode, scrambling to free up space before services fail. Forecasting gives us lead time.

Going back to the weather analogy: a threshold alert is a rain sensor that triggers when we're already soaked. An ML forecast is the weather forecast that tells us to bring an umbrella tomorrow. With a 14-day forecast running daily, we have time to act: add storage, move data, and clean up logs before anything breaks.

One caveat to keep in mind: ML forecasting assumes trend continuation. If a host's usage changes suddenly because a new application deploys or a log rotation kicks in, the forecast will lag behind the new reality. Run the forecast daily; it reflects today's data. Skip the regeneration, and you're alerting on a trend that may no longer exist.

The Kibana Workflows YAML for disk capacity alerts

All three active steps live in a single YAML file. The consts section at the top is where we set the job ID, threshold, and Slack connector. The rest of the logic stays untouched across environments.

name: disk-capacity-forecast-alert
description: >
  Daily ML forecast on disk-usage-forecast job; alert Slack when the upper
  confidence bound crosses 90% at a future timestamp.
enabled: true
tags:
  - machine-learning
  - capacity-planning
  - observability

consts:
  job_id: disk-usage-forecast
  results_index: .ml-anomalies-disk-usage-forecast
  forecast_duration: 14d
  forecast_expires_in: 24h
  forecast_max_model_memory: 200mb     # capped at min(500mb, 40% of job model_memory_limit)
  breach_threshold: 0.9
  max_partitions_in_alert: 20          # cap the list; total count is reported separately
  slack_connector_id: REPLACE_WITH_SLACK_CONNECTOR_ID
  slack_channel: REPLACE_WITH_SLACK_CHANNEL_ID

triggers:
  - type: scheduled
    with:
      rrule:
        freq: DAILY
        interval: 1
        tzid: UTC
        byhour: [6]
        byminute: [0]

steps:
  # Step 1: generate a fresh 14-day forecast
  - name: run_forecast
    type: elasticsearch.request
    with:
      method: POST
      path: "/_ml/anomaly_detectors/{{ consts.job_id }}/_forecast"
      body:
        duration: "{{ consts.forecast_duration }}"
        expires_in: "{{ consts.forecast_expires_in }}"
        max_model_memory: "{{ consts.forecast_max_model_memory }}"

  # Step 2: poll until the forecast finishes and its docs are searchable
  # (a fixed sleep races at high cardinality; this waits exactly as long as needed)
  - name: wait_for_forecast
    type: while
    condition: "steps.check_forecast.output.hits.total.value < 1"
    max-iterations:
      limit: 60
      on-limit: fail
    steps:
      - name: backoff
        type: wait
        with:
          duration: 2s
      - name: check_forecast
        type: elasticsearch.request
        with:
          method: POST
          path: "/{{ consts.results_index }}/_search"
          body:
            size: 0
            query:
              bool:
                filter:
                  - term: { job_id: "{{ consts.job_id }}" }
                  - term: { result_type: model_forecast_request_stats }
                  - term: { forecast_id: "{{ steps.run_forecast.output.forecast_id }}" }
                  - term: { forecast_status: finished }

  # Step 3: find future buckets where the upper bound crosses the threshold
  - name: search_breaches
    type: elasticsearch.request
    with:
      method: POST
      path: "/{{ consts.results_index }}/_search"
      body:
        size: 0
        runtime_mappings:
          host_mount:
            type: keyword
            script:
              source: "emit(doc['partition_field_value'].value + '|' + doc['by_field_value'].value)"
        query:
          bool:
            filter:
              - term: { job_id: "{{ consts.job_id }}" }
              - term: { result_type: model_forecast }
              - term: { forecast_id: "{{ steps.run_forecast.output.forecast_id }}" }
              - range: { forecast_upper: { gte: "{{ consts.breach_threshold }}" } }
              - range: { timestamp: { gte: now } }   # future buckets only
        aggs:
          total_affected:                            # true distinct host×mount count
            cardinality: { field: host_mount }
          top_partitions:                            # bounded, most-urgent-first
            multi_terms:
              size: "{{ consts.max_partitions_in_alert }}"
              terms:
                - field: partition_field_value
                - field: by_field_value
              order: { earliest_breach: asc }
            aggs:
              earliest_breach: { min: { field: timestamp } }
              peak_forecast: { max: { field: forecast_upper } }

  # Step 4: alert only when breaches exist
  - name: notify_if_breach
    type: if
    condition: "steps.search_breaches.output.hits.total.value > 0"
    steps:
      - name: slack_alert
        type: slack.postMessage
        connector-id: "{{ consts.slack_connector_id }}"
        with:
          channel: "{{ consts.slack_channel }}"
          text: |
            Disk capacity forecast alert (job: {{ consts.job_id }})
            Future breach buckets (forecast_upper >= {{ consts.breach_threshold }}): {{ steps.search_breaches.output.hits.total.value }} across {{ steps.search_breaches.output.aggregations.total_affected.value }} partition(s)
            Horizon: {{ consts.forecast_duration }}

            Showing top {{ steps.search_breaches.output.aggregations.top_partitions.buckets | size }} partition(s), soonest breach first:
            {% for b in steps.search_breaches.output.aggregations.top_partitions.buckets %}- {{ b.key[0] }} {{ b.key[1] }} - peak {{ b.peak_forecast.value | times: 100 | round: 1 }}%, first crosses on {{ b.earliest_breach.value_as_string | date: "%Y-%m-%d %H:%M UTC" }} ({{ b.doc_count }} future bucket(s))
            {% endfor %}
    else:
      - name: no_breach_log
        type: console
        with:
          message: "No disk forecast breaches above {{ consts.breach_threshold }} for {{ consts.job_id }}"

A few things in this YAML are worth understanding before you run it:

• elasticsearch.request is what actually returns the full response here. In 9.4.2 testing, elasticsearch.search strips out aggregations and only exposes took, timed_out, _shards, and hits. Since the Slack message is built by iterating over aggregation buckets, method: POST on elasticsearch.request is what gets you there instead.
• The message template runs on Liquid. Workflows doesn't support Handlebars, so {{#each}} fails workflow validation if you're coming from a stack that uses it. The {% for %} loop and filters like | times: 100 | round: 1 and | date all work fine in 9.4.2.
• This polls until the forecast is actually done, instead of guessing with a fixed wait. POST _forecast returns a forecast_id almost immediately, but the result documents land asynchronously. A flat 5 second wait races against that at high cardinality. If the forecast isn't finished, the breach search comes back with 0 hits regardless of what the data says. The while step polls model_forecast_request_stats until forecast_status: finished instead, so it waits exactly as long as the forecast takes and no longer. max-iterations with on-limit: fail means a stuck forecast fails loudly rather than spinning into the 2000-iteration default.
• The breach aggregation runs on multi_terms because composite can't order by a sub-aggregation metric. A composite agg would return partitions alphabetically, and a fixed size would quietly hide the worst offenders. multi_terms sorts by earliest_breach and caps the list at max_partitions_in_alert, while a separate total_affected cardinality aggregation reports the true distinct-partition count, so the message reads "top 20 of 137" instead of truncating without saying so. multi_terms bucket keys come back as arrays too, so they're indexed as b.key[0] (host) and b.key[1] (mount). The timestamp >= now filter matters for the same reason: without it, the query would alert on historical buckets from the same forecast run, not just future ones.
• Model memory and expiry get sized together. max_model_memory for the forecast defaults to 20 MB and is capped at min(500mb, 40% of the job's model_memory_limit). A forecast that exceeds that budget spools to disk and runs slower, so raise both the job's model_memory_limit and forecast_max_model_memory together for large fleets. Each daily run also writes horizon_buckets × partitions documents to the results index, which is why expires_in: 24h is what keeps it from growing unbounded.

What the Slack message looks like

When the workflow finds breaches, the alert looks like this:

Disk capacity forecast alert (job: disk-usage-forecast)
Future breach buckets (forecast_upper >= 0.9): 545 across 2 partition(s)
Horizon: 14d

Showing top 2 partition(s), soonest breach first:
- host-a / : peak 113.2%, first crosses on 2026-06-28 09:00 UTC (334 future bucket(s))
- host-b /data : peak 97.4%, first crosses on 2026-07-01 14:00 UTC (211 future bucket(s))

"Peak 113.2%" looks alarming, but it's mathematically correct: the upper confidence band can exceed 100% when the model's variance is wide. The useful number is "first crosses on". That's the runway we have to act. On a large fleet, the list is capped at max_partitions_in_alert (and ordered soonest-first), while "across N partition(s)" still reports the true total, so nothing is hidden.

Operational considerations

• Cardinality. At tens of thousands of partitions (hosts × mount points), model_plot_config.enabled: false is not optional. The model plot data volume at that scale is significant and unnecessary for alerting. The breach search uses multi_terms capped at max_partitions_in_alert (default 20), ordered soonest-breach-first, and reports the true distinct-partition count via a cardinality aggregation, so the alert stays readable even when thousands of partitions breach.

• Wait timeout. The while step's max-iterations limit (60 iterations at 2s each) means forecasts can take up to 2 minutes to complete. At very high cardinality, this may not be enough; raise the limit if your forecasts legitimately run longer.

• Forecast quality. ML forecasting works best for metrics with consistent, learnable trends. Disk usage typically qualifies. Memory is noisier and can produce wide confidence bands, especially on hosts with variable workloads. If the Slack alert regularly shows "peak 400%," the model may be underfitting on some partitions, and a tighter filter on forecast_upper max value can help reduce noise.

Scaling beyond ~1,000 partitions

The formula the scale guide makes explicit:

Forecasting writes a new document to the result index for every forecasted element for every bucket.

So: result docs per run = horizon buckets × (hosts × mounts). With the blog's 1h bucket span and a 14d horizon, that's 336 docs per partition per daily run:

At ~10 M docs per run, the indexing load and results-index size become significant, even with expires_in: 24h. The search_breaches query stays fast (it's size: 0, purely aggregation), but ingesting 10 M docs per day into .ml-anomalies-* alongside normal workload adds real pressure.

The single biggest lever is bucket span. Disk capacity planning doesn't need hourly granularity over a 14-day horizon. "First breach on Tuesday" is actionable; "first breach between 10:00 and 11:00 UTC" usually isn't. Increasing bucket span reduces document volume multiplicatively:

For production fleets of 10k+ partitions, 6h or 1d is the right default. It requires cloning and retraining the job.

Beyond bucket span, there are two more options:

• Shorter horizon. A 7-day forecast at 6h gives 28 docs per partition per run — 12× fewer than the blog's default. Most capacity incidents have more than a week of warning; 14 days is comfortable headroom, not a hard requirement.

• Split the job. The single-job pattern breaks down above ~100k partitions. Split by datacenter, cloud region, or team — each job covers a bounded cardinality slice. The workflow YAML remains identical across all of them; the only changes are consts.job_id and consts.results_index. You can run one workflow per job or fan out with workflow.executeAsync.

There is also a hard compute ceiling: the docs state that forecasts needing more than 500 MB of working memory (the spool cap) will not start at all. This is a constraint on model complexity during the projection phase, not on the number of result docs. For disk-usage patterns, it's rarely the first limit you hit — index volume and ML node memory for the anomaly job itself usually bite first.

For teams already running Elastic for observability, this pattern keeps capacity alerting inside the same stack as the rest of the data. One less product to evaluate, one less agent to deploy.

How to enable disk capacity forecasting alerts in Kibana

Enable Workflows in Kibana under Stack Management > Workflows (requires Kibana 9.4+ and the manage_workflows privilege). Then:

1. Create the ML job using the configuration above and start the datafeed.
2. Wait for the model to accumulate a week's worth of disk metrics.
3. Set slack_connector_id and slack_channel in the consts section of disc-capacity-forecast-alert.yaml. Import the YAML in Kibana and enable the workflow.

On the first morning run, we'll either get a Slack alert listing hosts to investigate or a console log confirming everything is below the threshold. No more pesky surprise disk-full incidents.