Predicting delayed flights with classification analysis

Each document in the sample flight data set contains details for a single flight, so this data is ready for analysis; it is already in a two-dimensional entity-based data structure. In general, you often need to transform the data into an entity-centric index before you can analyze the data.

In order to be analyzed, a document must contain at least one field with a supported data type (numeric, boolean, text, keyword or ip) and must not contain arrays with more than one item. If your source data consists of some documents that contain the dependent variable and some that do not, the model is trained on the subset of documents that contain it.

{
  "_index": "kibana_sample_data_flights",
  "_type": "_doc",
  "_id": "S-JS1W0BJ7wufFIaPAHe",
  "_version": 1,
  "_seq_no": 3356,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "FlightNum": "N32FE9T",
    "DestCountry": "JP",
    "OriginWeather": "Thunder & Lightning",
    "OriginCityName": "Adelaide",
    "AvgTicketPrice": 499.08518599798685,
    "DistanceMiles": 4802.864932998549,
    "FlightDelay": false,
    "DestWeather": "Sunny",
    "Dest": "Chubu Centrair International Airport",
    "FlightDelayType": "No Delay",
    "OriginCountry": "AU",
    "dayOfWeek": 3,
    "DistanceKilometers": 7729.461862731618,
    "timestamp": "2019-10-17T11:12:29",
    "DestLocation": {
      "lat": "34.85839844",
      "lon": "136.8049927"
    },
    "DestAirportID": "NGO",
    "Carrier": "ES-Air",
    "Cancelled": false,
    "FlightTimeMin": 454.6742272195069,
    "Origin": "Adelaide International Airport",
    "OriginLocation": {
      "lat": "-34.945",
      "lon": "138.531006"
    },
    "DestRegion": "SE-BD",
    "OriginAirportID": "ADL",
    "OriginRegion": "SE-BD",
    "DestCityName": "Tokoname",
    "FlightTimeHour": 7.577903786991782,
    "FlightDelayMin": 0
  }
}

To predict whether a specific flight is delayed:

Now you have a new index that contains a copy of your source data with predictions for your dependent variable.

When you view the classification results in Kibana, it shows the contents of the destination index in a tabular format. It also provides information about the analysis details, model evaluation metrics, total feature importance values, and a scatterplot matrix. Let’s start by looking at the results table:

In this example, the table shows a column for the dependent variable (FlightDelay), which contains the ground truth values that you are trying to predict. It also shows a column for the predicted values (ml.FlightDelay_prediction), which were generated by the classification analysis. The ml.is_training column indicates whether the document was used in the training or testing data set. You can use the Training and Testing filter options to refine the contents of the results table. You can also enable histogram charts to get a better understanding of the distribution of values in your data.

If you want to understand how certain the model is about each prediction, you can examine its probability and score (ml.prediction_probability and ml.prediction_score). The higher these values are, the more confident the model is that the data point belongs to the named class. If you examine the destination index more closely in the Discover app in Kibana or use the standard Elasticsearch search command, you can see that the analysis predicts the probability of all possible classes for the dependent variable. The top_classes object contains the predicted classes with the highest scores.

GET model-flight-delays-classification/_search

          ...
          "FlightDelay" : false,
          ...
          "ml" : {
            "FlightDelay_prediction" : false,
            "top_classes" : [ 
              {
                "class_name" : false,
                "class_probability" : 0.9427605087816684,
                "class_score" : 0.3462468700158476
              },
              {
                "class_name" : true,
                "class_probability" : 0.057239491218331606,
                "class_score" : 0.057239491218331606
              }
            ],
            "prediction_probability" : 0.9427605087816684,
            "prediction_score" : 0.3462468700158476,
            ...

If you chose to calculate feature importance, the destination index also contains ml.feature_importance objects. Every field that is included in the classification analysis (known as a feature of the data point) is assigned a feature importance value. This value has both a magnitude and a direction (positive or negative), which indicates how each field affects a particular prediction. Only the most significant values (in this case, the top 10) are stored in the index. However, the trained model metadata also contains the average magnitude of the feature importance values for each field across all the training data. You can view this summarized information in Kibana:

You can also see the feature importance values for each individual prediction in the form of a decision plot:

In Kibana, the decision path shows the relative impact of each feature on the probability of the prediction. The features with the most significant positive or negative impact appear at the top of the decision plot. Thus in this example, the features related to flight time and distance had the most significant influence on the probability value for this prediction. This type of information can help you to understand how models arrive at their predictions. It can also indicate which aspects of your data set are most influential or least useful when you are training and tuning your model.

If you do not use Kibana, you can see the summarized feature importance values by using the get trained model API and the individual values by searching the destination index.

GET _ml/trained_models/model-flight-delays-classification*?include=total_feature_importance

{
  "count" : 1,
  "trained_model_configs" : [
    {
      "model_id" : "model-flight-delays-classification-1601405047985",
      ...
      "metadata" : {
        ...
        "feature_importance_baseline" : { 
          "classes" : [
            {
              "class_name" : true,
              "baseline" : -1.5869016940485443
            },
            {
              "class_name" : false,
              "baseline" : 1.5869016940485443
            }
          ]
        },
        "total_feature_importance" : [
          {
            "feature_name" : "dayOfWeek",
            "classes" : [
              {
                "class_name" : false,
                "importance" : {
                  "mean_magnitude" : 0.037513174351966404, 
                  "min" : -0.20132653028125566, 
                  "max" : 0.20132653028125566 
                }
              },
              {
                "class_name" : true,
                "importance" : {
                  "mean_magnitude" : 0.037513174351966404,
                  "min" : -0.20132653028125566,
                  "max" : 0.20132653028125566
                }
              }
            ]
          },
          {
            "feature_name" : "OriginWeather",
            "classes" : [
              {
                "class_name" : false,
                "importance" : {
                  "mean_magnitude" : 0.05486662317369895,
                  "min" : -0.3337477336556598,
                  "max" : 0.3337477336556598
                }
              },
              {
                "class_name" : true,
                "importance" : {
                  "mean_magnitude" : 0.05486662317369895,
                  "min" : -0.3337477336556598,
                  "max" : 0.3337477336556598
                }
              }
            ]
          },
          ...

GET model-flight-delays-classification/_search

          ...
          "FlightDelay" : false,
          ...
          "ml" : {
            "FlightDelay_prediction" : false,
            ...
            "prediction_probability" : 0.9427605087816684,
            "prediction_score" : 0.3462468700158476,
            "feature_importance" : [
              {
                "feature_name" : "DistanceMiles",
                "classes" : [
                  {
                    "class_name" : false,
                    "importance" : -1.4766536146534828
                  },
                  {
                    "class_name" : true,
                    "importance" : 1.4766536146534828
                  }
                ]
              },
              {
                "feature_name" : "FlightTimeMin",
                "classes" : [
                  {
                    "class_name" : false,
                    "importance" : 1.0919201754729184
                  },
                  {
                    "class_name" : true,
                    "importance" : -1.0919201754729184
                  }
                ]
              },
              ...

Lastly, Kibana provides a scatterplot matrix in the results. It has the same functionality as the matrix that you saw in the job wizard. Its purpose is to likewise help you visualize and explore the relationships between the numeric fields and the dependent variable in your data.

Though you can look at individual results and compare the predicted value (ml.FlightDelay_prediction) to the actual value (FlightDelay), you typically need to evaluate the success of your classification model as a whole.

Kibana provides a normalized confusion matrix that contains the percentage of occurrences where the analysis classified data points correctly with their actual class and the percentage of occurrences where it misclassified them.

If you want to see the exact number of occurrences, select a quadrant in the matrix. You can also use the Training and Testing filter options to refine the contents of the matrix. Thus you can see how well the model performs on previously unseen data. In this example, there are 2952 documents in the testing data that have the true class. 794 of them are predicted as false; this is called a false negative. 2158 are predicted correctly as true; this is called a true positive. The confusion matrix therefore shows us that 27% of the actual true values were correctly predicted and 73% were incorrectly predicted in the test data set.

Likewise if you select other quadrants in the matrix, it shows the number of documents that have the false class as their actual value in the testing data set. In this example, the model labeled 7262 documents out of 8802 correctly as false; this is called a true negative. 1540 documents are predicted incorrectly as true; this is called a false positive. Thus 83% of the actual false values were correctly predicted and 17% were incorrectly predicted in the test data set. When you perform classification analysis on your own data, it might take multiple iterations before you are satisfied with the results and ready to deploy the model.

Kibana also provides the receiver operating characteristic (ROC) curve as part of the model evaluation. The plot compares the true positive rate (y-axis) to the false positive rate (x-axis) for each class; in this example, true and false. From this plot, the area under the curve (AUC) value is computed. It is a number between 0 and 1. The higher the AUC, the better the model is at predicting the classes correctly.

You can also generate these metrics with the data frame analytics evaluate API. For more information about interpreting the evaluation metrics, see Classification evaluation.

POST _ml/data_frame/_evaluate
{
 "index": "model-flight-delays-classification",
   "query": {
    "term": {
      "ml.is_training": {
        "value": true  
      }
    }
  },
 "evaluation": {
   "classification": {
     "actual_field": "FlightDelay",
     "predicted_field": "ml.FlightDelay_prediction",
     "metrics": {
       "multiclass_confusion_matrix" : {}
     }
   }
 }
}

POST _ml/data_frame/_evaluate
{
 "index": "model-flight-delays-classification",
   "query": {
    "term": {
      "ml.is_training": {
        "value": false  
      }
    }
  },
 "evaluation": {
   "classification": {
     "actual_field": "FlightDelay",
     "predicted_field": "ml.FlightDelay_prediction",
     "metrics": {
       "multiclass_confusion_matrix" : {}
     }
   }
 }
}

{
  "classification" : {
    "multiclass_confusion_matrix" : {
      "confusion_matrix" : [
        {
          "actual_class" : "false", 
          "actual_class_doc_count" : 8802, 
          "predicted_classes" : [
            {
              "predicted_class" : "false", 
              "count" : 7262 
            },
            {
              "predicted_class" : "true",
              "count" : 1540
            }
          ],
          "other_predicted_class_doc_count" : 0
        },
        {
          "actual_class" : "true",
          "actual_class_doc_count" : 2952,
          "predicted_classes" : [
            {
              "predicted_class" : "false",
              "count" : 794
            },
            {
              "predicted_class" : "true",
              "count" : 2158
            }
          ],
          "other_predicted_class_doc_count" : 0
        }
      ],
      "other_actual_class_count" : 0
    }
  }
}

When you have trained a satisfactory model, you can deploy it to make predictions about new data. Those steps are not covered in this example. See Inference.

If you don’t want to keep the data frame analytics job, you can delete it in Kibana or by using the delete data frame analytics job API. When you delete data frame analytics jobs in Kibana, you have the option to also remove the destination indices and index patterns.