Machine Learning Anomaly Scoring and Elasticsearch - How it Works | Elastic Blog
エンジニアリング

機械学習の異常スコアリングとElasticsearchの仕組み

3recordIPinfluencerbucket

Record

 Admin300 300% IPIP

1e-308便

IP

0-100UIanomaly score

UI75100critical5075major2550minor025warning4

enter image description here

Single Metric Viewer90criticalSeverity thresholdInterval

APIfarequote_count5query for record results

GET /_xpack/ml/anomaly_detectors/farequote_count/results/records?human
{
  "sort": "record_score",
  "desc": true,
  "start": "2016-02-09T16:15:00.000Z",
  "end" :"2016-02-09T16:20:00.000Z"
}

{
  "count": 1,
  "records": [
    {
      "job_id": "farequote_count",
      "result_type": "record",
      "probability": 1.75744e-11,
      "record_score": 90.6954,
      "initial_record_score": 85.0643,
      "bucket_span": 300,
      "detector_index": 0,
      "is_interim": false,
      "timestamp_string": "2016-02-09T16:15:00.000Z",
      "timestamp": 1455034500000,
      "function": "count",
      "function_description": "count",
      "typical": [
        59.9827
      ],
      "actual": [
        179
      ]
    }
  ]
}

bucket_spanrecord_score90.6954100)probability1.75744e-1117960

UIprobability1.75744e-110-100

UIdescritption(descriptiontypical

Influencer

influencerinfluencer

influencerinfluencer

infuluencerdetector

influencerbucket_timeinfluencer使

APIdetectorinfluencer

airlineAPI回数 airlineAPI平均API

airlineinfluencer

Anomaly Explorer

enter image description here

influencerTop influencersinfluencerinfluencerinfluencerAALinfluencer97influener184influencerinfluencer97influencerAALAnomaliesrecord scores

influencerAPI調

GET _xpack/ml/anomaly_detectors/farequote_count_and_responsetime_by_airline/results/influencers?human
{
  "start": "2016-02-09T16:15:00.000Z",
  "end" :"2016-02-09T16:20:00.000Z"
}

{
  "count": 2,
  "influencers": [
    {
      "job_id": "farequote_count_and_responsetime_by_airline",
      "result_type": "influencer",
      "influencer_field_name": "airline",
      "influencer_field_value": "AAL",
      "airline": "AAL",
      "influencer_score": 97.1547,
      "initial_influencer_score": 98.5096,
      "probability": 6.56622e-40,
      "bucket_span": 300,
      "is_interim": false,
      "timestamp_string": "2016-02-09T16:15:00.000Z",
      "timestamp": 1455034500000
    },
    {
      "job_id": "farequote_count_and_responsetime_by_airline",
      "result_type": "influencer",
      "influencer_field_name": "airline",
      "influencer_field_value": "AWE",
      "airline": "AWE",
      "influencer_score": 0,
      "initial_influencer_score": 0,
      "probability": 0.0499957,
      "bucket_span": 300,
      "is_interim": false,
      "timestamp_string": "2016-02-09T16:15:00.000Z",
      "timestamp": 1455034500000
    }
  ]
}

AALinfluencer_score97.15497Anomaly ExplorerUIprobability6.56622e-40influencer_score

initial_influencer_score98.506997.1547調2influencerAWEAWEinfluencer0

influencer_scoreAPIactualtypical

Bucket

3bucket_span

byfieldspartitionfields使detector使

bucketinfluencer

detector

airlinecount(回数) airlinemean(平均レスポンスタイム)

Anomaly Explorer enter image description here

Anomaly timelineoverallUIbucket_spanUI

909895critical

API調

GET _xpack/ml/anomaly_detectors/farequote_count_and_responsetime_by_airline/results/buckets?human
{
  "start": "2016-02-09T16:15:00.000Z",
  "end" :"2016-02-09T16:20:00.000Z"
}

{
  "count": 1,
  "buckets": [
    {
      "job_id": "farequote_count_and_responsetime_by_airline",
      "timestamp_string": "2016-02-09T16:15:00.000Z",
      "timestamp": 1455034500000,
      "anomaly_score": 90.7,
      "bucket_span": 300,
      "initial_anomaly_score": 85.08,
      "event_count": 179,
      "is_interim": false,
      "bucket_influencers": [
        {
          "job_id": "farequote_count_and_responsetime_by_airline",
          "result_type": "bucket_influencer",
          "influencer_field_name": "airline",
          "initial_anomaly_score": 85.08,
          "anomaly_score": 90.7,
          "raw_anomaly_score": 37.3875,
          "probability": 6.92338e-39,
          "timestamp_string": "2016-02-09T16:15:00.000Z",
          "timestamp": 1455034500000,
          "bucket_span": 300,
          "is_interim": false
        },
        {
          "job_id": "farequote_count_and_responsetime_by_airline",
          "result_type": "bucket_influencer",
          "influencer_field_name": "bucket_time",
          "initial_anomaly_score": 85.08,
          "anomaly_score": 90.7,
          "raw_anomaly_score": 37.3875,
          "probability": 6.92338e-39,
          "timestamp_string": "2016-02-09T16:15:00.000Z",
          "timestamp": 1455034500000,
          "bucket_span": 300,
          "is_interim": false
        }
      ],
      "processing_time_ms": 17,
      "result_type": "bucket"
    }
  ]
}

anomaly_score 90.7 initial_anomaly_score anomaly_scoreanomaly_scoreinitial_anomaly_scoreUI bucket_influencers influencerinfluencerinfluencer_field_name:airlineinfluencer_field_name:bucket_timebuilt-in influencerinfluencerAPIinfluencerrecord

recordinfluencer便

influencer_scorerecord_score

1record_score使使

Alerting on Machine Learning Jobs in Elasticsearch v5.5Explaining the Bucket Span in Machine Learning for ElasticsearchResults resources, Machine Learning Docs