Evaluate data frame analytics Generally available; Added in 7.3.0

View as markdown
POST /_ml/data_frame/_evaluate
Api key auth Basic auth Bearer auth

The API packages together commonly used evaluation metrics for various types of machine learning features. This has been designed for use on indexes created by data frame analytics. Evaluation requires both a ground truth field and an analytics result field to be present.

Required authorization

  • Cluster privileges: monitor_ml
application/json

Body Required

  • evaluation object Required

    Defines the type of evaluation you want to perform.

    Hide evaluation attributes Show evaluation attributes object
    • classification object

      Classification evaluation evaluates the results of a classification analysis which outputs a prediction that identifies to which of the classes each document belongs.

      Hide classification attributes Show classification attributes object
      • actual_field string Required

        The field of the index which contains the ground truth. The data type of this field can be boolean or integer. If the data type is integer, the value has to be either 0 (false) or 1 (true).

      • predicted_field string

        The field in the index which contains the predicted value, in other words the results of the classification analysis.

      • top_classes_field string

        The field of the index which is an array of documents of the form { "class_name": XXX, "class_probability": YYY }. This field must be defined as nested in the mappings.

      • metrics object
    • outlier_detection object

      Outlier detection evaluates the results of an outlier detection analysis which outputs the probability that each document is an outlier.

      Hide outlier_detection attributes Show outlier_detection attributes object
      • actual_field string Required

        The field of the index which contains the ground truth. The data type of this field can be boolean or integer. If the data type is integer, the value has to be either 0 (false) or 1 (true).

      • predicted_probability_field string Required

        The field of the index that defines the probability of whether the item belongs to the class in question or not. It’s the field that contains the results of the analysis.

      • metrics object
    • regression object

      Regression evaluation evaluates the results of a regression analysis which outputs a prediction of values.

      Hide regression attributes Show regression attributes object
      • actual_field string Required

        The field of the index which contains the ground truth. The data type of this field must be numerical.

      • predicted_field string Required

        The field in the index that contains the predicted value, in other words the results of the regression analysis.

      • metrics object

        Specifies the metrics that are used for the evaluation. For more information on mse, msle, and huber, consult the Jupyter notebook on regression loss functions.

        Hide metrics attributes Show metrics attributes object
        • mse object

          Average squared difference between the predicted values and the actual (ground truth) value. For more information, read this wiki article.

        • r_squared object

          Proportion of the variance in the dependent variable that is predictable from the independent variables.

  • index string Required

    Defines the index in which the evaluation will be performed.

  • query object

    A query clause that retrieves a subset of data from the source index.

    External documentation

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • classification object

      Evaluation results for a classification analysis. It outputs a prediction that identifies to which of the classes each document belongs.

      Hide classification attributes Show classification attributes object
      • auc_roc object

        The AUC ROC (area under the curve of the receiver operating characteristic) score and optionally the curve. It is calculated for a specific class (provided as "class_name") treated as positive.

        Hide auc_roc attributes Show auc_roc attributes object
        • value number Required
        • curve array[object]
      • accuracy object

        Accuracy of predictions (per-class and overall).

        Hide accuracy attributes Show accuracy attributes object
        • classes array[object] Required
        • overall_accuracy number Required
      • multiclass_confusion_matrix object

        Multiclass confusion matrix.

        Hide multiclass_confusion_matrix attributes Show multiclass_confusion_matrix attributes object
        • confusion_matrix array[object] Required
        • other_actual_class_count number Required
      • precision object

        Precision of predictions (per-class and average).

        Hide precision attributes Show precision attributes object
        • classes array[object] Required
        • avg_precision number Required
      • recall object

        Recall of predictions (per-class and average).

        Hide recall attributes Show recall attributes object
        • classes array[object] Required
        • avg_recall number Required
    • outlier_detection object

      Evaluation results for an outlier detection analysis. It outputs the probability that each document is an outlier.

      Hide outlier_detection attributes Show outlier_detection attributes object
      • auc_roc object

        The AUC ROC (area under the curve of the receiver operating characteristic) score and optionally the curve.

        Hide auc_roc attributes Show auc_roc attributes object
        • value number Required
        • curve array[object]
      • precision object

        Set the different thresholds of the outlier score at where the metric is calculated.

        Hide precision attribute Show precision attribute object
        • * number Additional properties
      • recall object

        Set the different thresholds of the outlier score at where the metric is calculated.

        Hide recall attribute Show recall attribute object
        • * number Additional properties
      • confusion_matrix object

        Set the different thresholds of the outlier score at where the metrics (tp - true positive, fp - false positive, tn - true negative, fn - false negative) are calculated.

        Hide confusion_matrix attribute Show confusion_matrix attribute object
        • * object Additional properties
          Hide * attributes Show * attributes object
          • tp number Required

            True Positive

          • fp number Required

            False Positive

          • tn number Required

            True Negative

          • fn number Required

            False Negative

    • regression object

      Evaluation results for a regression analysis which outputs a prediction of values.

      Hide regression attributes Show regression attributes object
      • huber object

        Pseudo Huber loss function.

        Hide huber attribute Show huber attribute object
        • value number Required
      • mse object

        Average squared difference between the predicted values and the actual (ground truth) value.

        Hide mse attribute Show mse attribute object
        • value number Required
      • msle object

        Average squared difference between the logarithm of the predicted values and the logarithm of the actual (ground truth) value.

        Hide msle attribute Show msle attribute object
        • value number Required
      • r_squared object

        Proportion of the variance in the dependent variable that is predictable from the independent variables.

        Hide r_squared attribute Show r_squared attribute object
        • value number Required
POST /_ml/data_frame/_evaluate 
POST _ml/data_frame/_evaluate
{
  "index": "animal_classification",
  "evaluation": {
    "classification": {
      "actual_field": "animal_class",
      "predicted_field": "ml.animal_class_prediction",
      "metrics": {
        "multiclass_confusion_matrix": {}
      }
    }
  }
}
resp = client.ml.evaluate_data_frame(
    index="animal_classification",
    evaluation={
        "classification": {
            "actual_field": "animal_class",
            "predicted_field": "ml.animal_class_prediction",
            "metrics": {
                "multiclass_confusion_matrix": {}
            }
        }
    },
)
const response = await client.ml.evaluateDataFrame({
  index: "animal_classification",
  evaluation: {
    classification: {
      actual_field: "animal_class",
      predicted_field: "ml.animal_class_prediction",
      metrics: {
        multiclass_confusion_matrix: {},
      },
    },
  },
});
response = client.ml.evaluate_data_frame(
  body: {
    "index": "animal_classification",
    "evaluation": {
      "classification": {
        "actual_field": "animal_class",
        "predicted_field": "ml.animal_class_prediction",
        "metrics": {
          "multiclass_confusion_matrix": {}
        }
      }
    }
  }
)
$resp = $client->ml()->evaluateDataFrame([
    "body" => [
        "index" => "animal_classification",
        "evaluation" => [
            "classification" => [
                "actual_field" => "animal_class",
                "predicted_field" => "ml.animal_class_prediction",
                "metrics" => [
                    "multiclass_confusion_matrix" => new ArrayObject([]),
                ],
            ],
        ],
    ],
]);
curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"index":"animal_classification","evaluation":{"classification":{"actual_field":"animal_class","predicted_field":"ml.animal_class_prediction","metrics":{"multiclass_confusion_matrix":{}}}}}' "$ELASTICSEARCH_URL/_ml/data_frame/_evaluate"
client.ml().evaluateDataFrame(e -> e
    .evaluation(ev -> ev
        .classification(c -> c
            .actualField("animal_class")
            .predictedField("ml.animal_class_prediction")
            .metrics(m -> m)
        )
    )
    .index("animal_classification")
);
Request examples
Run `POST _ml/data_frame/_evaluate` to evaluate a a classification job for an annotated index. The `actual_field` contains the ground truth for classification. The `predicted_field` contains the predicted value calculated by the classification analysis. 
{
  "index": "animal_classification",
  "evaluation": {
    "classification": {
      "actual_field": "animal_class",
      "predicted_field": "ml.animal_class_prediction",
      "metrics": {
        "multiclass_confusion_matrix": {}
      }
    }
  }
}
Run `POST _ml/data_frame/_evaluate` to evaluate a classification job with AUC ROC metrics for an annotated index. The `actual_field` contains the ground truth value for the actual animal classification. This is required in order to evaluate results. The `class_name` specifies the class name that is treated as positive during the evaluation, all the other classes are treated as negative. 
{
  "index": "animal_classification",
  "evaluation": {
    "classification": {
      "actual_field": "animal_class",
      "metrics": {
        "auc_roc": {
          "class_name": "dog"
        }
      }
    }
  }
}
Run `POST _ml/data_frame/_evaluate` to evaluate an outlier detection job for an annotated index. 
{
  "index": "my_analytics_dest_index",
  "evaluation": {
    "outlier_detection": {
      "actual_field": "is_outlier",
      "predicted_probability_field": "ml.outlier_score"
    }
  }
}
Run `POST _ml/data_frame/_evaluate` to evaluate the testing error of a regression job for an annotated index. The term query in the body limits evaluation to be performed on the test split only. The `actual_field` contains the ground truth for house prices. The `predicted_field` contains the house price calculated by the regression analysis. 
{
  "index": "house_price_predictions",
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "ml.is_training": false
          }
        }
      ]
    }
  },
  "evaluation": {
    "regression": {
      "actual_field": "price",
      "predicted_field": "ml.price_prediction",
      "metrics": {
        "r_squared": {},
        "mse": {},
        "msle": {
          "offset": 10
        },
        "huber": {
          "delta": 1.5
        }
      }
    }
  }
}
Run `POST _ml/data_frame/_evaluate` to evaluate the training error of a regression job for an annotated index. The term query in the body limits evaluation to be performed on the training split only. The `actual_field` contains the ground truth for house prices. The `predicted_field` contains the house price calculated by the regression analysis. 
{
  "index": "house_price_predictions",
  "query": {
    "term": {
      "ml.is_training": {
        "value": true
      }
    }
  },
  "evaluation": {
    "regression": {
      "actual_field": "price",
      "predicted_field": "ml.price_prediction",
      "metrics": {
        "r_squared": {},
        "mse": {},
        "msle": {},
        "huber": {}
      }
    }
  }
}
Response examples (200)
A succesful response from `POST _ml/data_frame/_evaluate` to evaluate a classification analysis job for an annotated index. The `actual_class` contains the name of the class the analysis tried to predict. The `actual_class_doc_count` is the number of documents in the index belonging to the `actual_class`. The `predicted_classes` object contains the list of the predicted classes and the number of predictions associated with the class. 
{
  "classification": {
    "multiclass_confusion_matrix": {
      "confusion_matrix": [
        {
          "actual_class": "cat",
          "actual_class_doc_count": 12,
          "predicted_classes": [
            {
              "predicted_class": "cat",
              "count": 12
            },
            {
              "predicted_class": "dog",
              "count": 0
            }
          ],
          "other_predicted_class_doc_count": 0
        },
        {
          "actual_class": "dog",
          "actual_class_doc_count": 11,
          "predicted_classes": [
            {
              "predicted_class": "dog",
              "count": 7
            },
            {
              "predicted_class": "cat",
              "count": 4
            }
          ],
          "other_predicted_class_doc_count": 0
        }
      ],
      "other_actual_class_count": 0
    }
  }
}
A succesful response from `POST _ml/data_frame/_evaluate` to evaluate a classification analysis job with the AUC ROC metrics for an annotated index. 
{
  "classification": {
    "auc_roc": {
      "value": 0.8941788639536681
    }
  }
}
A successful response from `POST _ml/data_frame/_evaluate` to evaluate an outlier detection job. 
{
  "outlier_detection": {
    "auc_roc": {
      "value": 0.9258475774641445
    },
    "confusion_matrix": {
      "0.25": {
        "tp": 5,
        "fp": 9,
        "tn": 204,
        "fn": 5
      },
      "0.5": {
        "tp": 1,
        "fp": 5,
        "tn": 208,
        "fn": 9
      },
      "0.75": {
        "tp": 0,
        "fp": 4,
        "tn": 209,
        "fn": 10
      }
    },
    "precision": {
      "0.25": 0.35714285714285715,
      "0.5": 0.16666666666666666,
      "0.75": 0
    },
    "recall": {
      "0.25": 0.5,
      "0.5": 0.1,
      "0.75": 0
    }
  }
}