**IMPORTANT**: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

# Regression

edit## Regression

editThis functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

Regression analysis is a machine learning process for estimating the relationships among different fields in your data, then making further predictions based on these relationships.

For example, suppose we are interested in finding the relationship between apartment size and monthly rent in a city. Our imaginary data set consists of three data points:

Size (m2) |
Monthly rent |

44 |
1600 |

24 |
1055 |

63 |
2300 |

After the model determines the relationship between the apartment size and the rent, it can make predictions such as the monthly rent of a hundred square meter-size apartment.

This is a simple example. Usually regression problems are multi-dimensional,
so the relationships that regression analysis tries to find are between multiple
fields. To extend our example, a more complex regression analysis could take into
account additional factors such as the location of the apartment in the city, on
which floor it is, and whether the apartment has a riverside view or not, and so
on. All of these factors can be considered *features*; they are measurable
properties or characteristics of the phenomenon we’re studying.

#### Feature variables

editWhen you perform regression analysis, you must identify a subset of fields that you
want to use to create a model for predicting other fields. We refer to these
fields as *feature variables* and *dependent variables*, respectively.
Feature variables are the values that the dependent variable value depends on. If one or
more of the feature variables changes, the dependent variable value also changes. There are
three different types of feature variables that you can use with our regression
algorithm:

- Numerical. In our example, the size of the apartment was a numerical feature variable.
- Categorical. A variable that can have one value from a set of values. The value set has a fixed and limited number of possible items. In the example, the location of the apartment in the city (borough) is a categorical variable.
- Boolean. The riverside view in the example is a boolean value because an apartment either has a riverside view or doesn’t have one. Arrays are not supported.

#### Training the regression model

editRegression is a supervised machine learning method, which means that you need to supply a labeled training data set that has some feature variables and a dependent variable. The regression algorithm identifies the relationships between the feature variables and the dependent variable. Once you’ve trained the model on your training data set, you can reuse the knowledge that the model has learned to make inferences about new data.

The relationships between the feature variables and the dependent variable are described as a mathematical function. Regression analysis tries to find the best prediction for the dependent variable by combining the predictions from multiple base learners – algorithms that generalize from the data set. The performance of an ensemble is usually better than the performance of each individual base learner because the individual learners will make different errors. These average out when their predictions are combined.

Regression works as a batch analysis. If new data comes into your index, you must restart the data frame analytics job.

##### Regression algorithms

editThe ensemble learning technique that we use in the Elastic Stack is a type of boosting called extreme gradient boost (XGboost) which combines decision trees with gradient boosting methodologies.

#### Feature importance

editFeature importance is calculated for supervised machine learning methods such as regression and classification. This value provides further insight into the results of a data frame analytics job and therefore helps interpret these results. As we mentioned, there are multiple features of a data point that are analyzed during data frame analytics. These features are responsible for a particular prediction to varying degrees. Feature importance shows to what degree a given feature of a data point contributes to the prediction. The feature importance value of a feature can be either positive or negative depending on its effect on the prediction. If the feature reduces the prediction value, the value is negative. If the feature increases the prediction, the feature importance value positive. The magnitude of the feature importance value shows how significantly the feature affects the prediction both locally (for a given data point) or generally (for the whole data set).

Feature importance in the Elastic Stack is calculated using the SHAP (SHapley Additive exPlanations) method as described in Lundberg, S. M., & Lee, S.-I. A Unified Approach to Interpreting Model Predictions. In NeurIPS 2017.

By default, feature importance values are not calculated. To generate this
information, when you create a data frame analytics job you must specify the
`num_top_feature_importance_values`

property. The feature importance values are
stored in the destination index in fields prefixed by `ml.feature_importance`

.

The number of feature importance values for each document might be less
than the `num_top_feature_importance_values`

property value. For example, it
returns only features that had a positive or negative effect on the prediction.

#### Measuring model performance

editYou can measure how well the model has performed on your training data set by
using the `regression`

evaluation type of the
evaluate data frame analytics API. The mean squared
error (MSE) value that the evaluation provides you on the training data set is
the *training error*. Training the regression model means finding the
combination of model parameters that produces the lowest possible training
error.

Another crucial measurement is how well your model performs on unseen
data points. To assess how well the trained model will perform on data it has
never seen before, you must set aside a proportion of the training data set for
testing. This split of the data set is the testing data set. Once the model has
been trained, you can let the model
predict the value of the data points it has never seen before and compare the
prediction to the actual value. This test provides an estimate of a quantity
known as the *model generalization error*.

Two concepts describe how well the regression algorithm was able to learn the
relationship between the feature variables and the dependent variable. *Underfitting* is when
the model cannot capture the complexity of the data set. *Overfitting* is when
the model is too specific to the training data set and is capturing details
which do not generalize to new data. A model that overfits the data has a
low MSE value on the training data set and a high MSE value on the testing
data set. For more information about the evaluation metrics, see
Regression evaluation.

### Further readings

editFeature importance for data frame analytics (Jupyter notebook)