Machine Learning Anomaly Scoring and Elasticsearch - How it Works | Elastic Blog
엔지니어링

머신 러닝 변칙 점수 평가와 Elasticsearch, 그 작동 방법

Elastic . , . , (ML) .

"( ) . (") , IP (") , (") . .

, . :

  • = 1 300
  • 300%
  • IP IP

, (1e-308 ). , .

  • ? , ?
  • ( , IP ) ?

ML 0~100 . UI " .

, UI " . 75 100 (critical)", 50 75 (major)", 25 50 ,(minor)", 0 25 (warning)" , .

여기에 이미지 설명을 입력하세요

. 90 " . (Severity threshold)" , (Interval)" .

5 ML API ( farequote_count),

GET /_xpack/ml/anomaly_detectors/farequote_count/results/records?human
{
  "sort": "record_score",
  "desc": true,
  "start": "2016-02-09T16:15:00.000Z",
  "end" :"2016-02-09T16:20:00.000Z"
}

.

{
  "count": 1,
  "records": [
    {
      "job_id": "farequote_count",
      "result_type": "record",
      "probability": 1.75744e-11,
      "record_score": 90.6954,
      "initial_record_score": 85.0643,
      "bucket_span": 300,
      "detector_index": 0,
      "is_interim": false,
      "timestamp_string": "2016-02-09T16:15:00.000Z",
      "timestamp": 1455034500000,
      "function": "count",
      "function_description": "count",
      "typical": [
        59.9827
      ],
      "actual": [
        179
      ]
    }
  ]
}

5 ( bucket_span) record_score 90.6954(100 ) '` 1.75744e-11 . 5 " , 60 179 .

UI . 개연성 1.75744e-11 , , . 0 100 . , . . , .

UI " ( 3 ) . . " " .

. . ML " . , . . , .

, , ML , ( ) " " . ( ) , .

, . ML . , .

ML , `bucket_time . .

, ML API .

  • 항공사’에서 분할되고/파티션된 API 호출`
  • 항공사’에서 분할되고/파티션된 API 호출( )`

항공사 .

,

여기에 이미지 설명을 입력하세요

(Top influencers) . , ( ) ( ). , AAL 97 , 184. , , 97 . AAL " " .

API ,

GET _xpack/ml/anomaly_detectors/farequote_count_and_responsetime_by_airline/results/influencers?human
{
  "start": "2016-02-09T16:15:00.000Z",
  "end" :"2016-02-09T16:20:00.000Z"
}

.

{
  "count": 2,
  "influencers": [
    {
      "job_id": "farequote_count_and_responsetime_by_airline",
      "result_type": "influencer",
      "influencer_field_name": "airline",
      "influencer_field_value": "AAL",
      "airline": "AAL",
      "influencer_score": 97.1547,
      "initial_influencer_score": 98.5096,
      "probability": 6.56622e-40,
      "bucket_span": 300,
      "is_interim": false,
      "timestamp_string": "2016-02-09T16:15:00.000Z",
      "timestamp": 1455034500000
    },
    {
      "job_id": "farequote_count_and_responsetime_by_airline",
      "result_type": "influencer",
      "influencer_field_name": "airline",
      "influencer_field_value": "AWE",
      "airline": "AWE",
      "influencer_score": 0,
      "initial_influencer_score": 0,
      "probability": 0.0499957,
      "bucket_span": 300,
      "is_interim": false,
      "timestamp_string": "2016-02-09T16:15:00.000Z",
      "timestamp": 1455034500000
    }
  ]
}

UI influencer_score 97.1547( 97) AAL . 개연성 6.56622e-40 influencer_score ( ) , .

initial_influencer_score 98.5096 , . 97.1547 . ML / . AWE , ( 0) .

influencer_score , API . , , .

, , ML bucket_span . ( ) ( ) .

, .

  • ()
  • () byfields / partitionfields " , .

, .

, ML ,

  • 항공사’에서 분할되고/파티션된`
  • 항공사’에서 분할되고/파티션된( )`

, 여기에 이미지 설명을 입력하세요

" . . UI ML bucket_span , UI " .

90, , 98 95 .

API ,

GET _xpack/ml/anomaly_detectors/farequote_count_and_responsetime_by_airline/results/buckets?human
{
  "start": "2016-02-09T16:15:00.000Z",
  "end" :"2016-02-09T16:20:00.000Z"
}

.

{
  "count": 1,
  "buckets": [
    {
      "job_id": "farequote_count_and_responsetime_by_airline",
      "timestamp_string": "2016-02-09T16:15:00.000Z",
      "timestamp": 1455034500000,
      "anomaly_score": 90.7,
      "bucket_span": 300,
      "initial_anomaly_score": 85.08,
      "event_count": 179,
      "is_interim": false,
      "bucket_influencers": [
        {
          "job_id": "farequote_count_and_responsetime_by_airline",
          "result_type": "bucket_influencer",
          "influencer_field_name": "airline",
          "initial_anomaly_score": 85.08,
          "anomaly_score": 90.7,
          "raw_anomaly_score": 37.3875,
          "probability": 6.92338e-39,
          "timestamp_string": "2016-02-09T16:15:00.000Z",
          "timestamp": 1455034500000,
          "bucket_span": 300,
          "is_interim": false
        },
        {
          "job_id": "farequote_count_and_responsetime_by_airline",
          "result_type": "bucket_influencer",
          "influencer_field_name": "bucket_time",
          "initial_anomaly_score": 85.08,
          "anomaly_score": 90.7,
          "raw_anomaly_score": 37.3875,
          "probability": 6.92338e-39,
          "timestamp_string": "2016-02-09T16:15:00.000Z",
          "timestamp": 1455034500000,
          "bucket_span": 300,
          "is_interim": false
        }
      ],
      "processing_time_ms": 17,
      "result_type": "bucket"
    }
  ]
}

, .

  • anomaly_score - ( 90.7)
  • initial_anomaly_score - anomaly_score(, anomaly_score ). initial_anomaly_score UI .
  • bucket_influencers - . , influencer_field_name:airline influencer_field_name:bucket_time( ) . (, ) , API .

( , , ) ? , .

, . influencer_score’를 사용하는 것을 고려해 봐야 합니다. 또는 시간 창 내에서 가장 비정상적인 변칙을 감지하고 알림을 받고자 한다면, 보고 또는 알림의 기반으로 개별record_score` .

, . , bucketspan 1 . , `recordscore , , . .

:

Elasticsearch v5.5 Elasticsearch ,