The release of supervised machine learning in Elastic Stack 7.6 closes the loop for an end-to-end machine learning pipeline. Learn how to get started with it in this example using binary classification.

<p>The 7.6 release of the Elastic Stack delivered the last piece required for an end-to-end machine learning pipeline. Previously, machine learning focused on unsupervised techniques with<a href="https://www.elastic.co/guide/en/machine-learning/7.8/ml-overview.html"> anomaly detection</a>. However, several features have been released over the 7.x releases. In<a href="https://www.elastic.co/blog/elastic-stack-7-2-0-released"> 7.2</a> Elasticsearch released <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/transforms.html">transforms</a> for turning raw indices into a feature index. Then <a href="https://www.elastic.co/blog/elastic-stack-7-3-0-released">7.3</a>, <a href="https://www.elastic.co/blog/elastic-stack-7-4-0-released">7.4</a>, and <a href="https://www.elastic.co/blog/elastic-stack-7-5-0-released">7.5</a> released<a href="https://www.elastic.co/guide/en/machine-learning/7.8/dfa-outlier-detection.html"> outlier detection</a>,&nbsp;<a href="https://www.elastic.co/guide/en/machine-learning/7.8/dfa-regression.html">regression</a>, and<a href="https://www.elastic.co/guide/en/machine-learning/7.8/dfa-classification.html"> classification</a>, respectively. Finally with 7.6 you can use regression and classification models with the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/inference-processor.html">inference ingest processor</a> to enrich documents. All this together closes the loop for an end-to-end machine learning pipeline. All within the Elastic Stack.
</p> <strong><h2>Why supervised learning?&nbsp;</h2></strong><p>Supervised machine learning trains a model from labeled data. This is in contrast to unsupervised learning where the model learns from unlabeled data. In <a href="https://www.elastic.co/guide/en/machine-learning/7.8/ml-dfa-overview.html">data frame analytics</a>, we give you the toolset to create supervised models without knowledge of the underlying algorithms. You can then use the trained models to make predictions on unobserved data (via inference). The model applies what it learned through training to provide a prediction.
</p><p>Why use this instead of anomaly detection? <a href="https://www.elastic.co/guide/en/machine-learning/7.8/xpack-ml.html">Anomaly detection</a>, an example of unsupervised learning, is excellent at learning patterns and predicting metrics based on time series data. This is powerful for many use cases, but what if we wanted to identify bad actors by <a href="https://www.elastic.co/blog/machine-learning-in-cybersecurity-training-supervised-models-to-detect-dga-activity">detecting suspicious domains</a>? What about <a href="https://www.elastic.co/blog/multilingual-search-using-language-identification-in-elasticsearch">automatically detecting language</a> and applying the appropriate analyzers for better search?&nbsp;
</p><p>These use cases need richer feature sets to provide accurate predictions. They also have known examples from which to learn. Meaning, there exists a training dataset where the data is labeled with the correct prediction. <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/transforms.html">Transforms</a> supplies the tools needed to build complex feature sets. <a href="https://www.elastic.co/guide/en/machine-learning/7.8/ml-dfa-overview.html">Data frame analytics</a> can train supervised machine learning models. We can use both to deploy the feature generation and supervised models for production use on streaming data.&nbsp;
</p> <strong><h2>Supervised learning example: binary classification</h2></strong><p>Let’s build and use a binary classification model from scratch. Don’t be intimidated. No data science degree is necessary. We will use the building blocks within the Elastic Stack to create our feature set, train our model, and deploy it in an ingest pipeline to perform inference. Using an ingest processor allows our data to be enriched with the model’s predictions before being indexed.
</p><p>In this example, we will predict whether&nbsp;a telecom customer is likely to churn based on features of their customer metadata and call logs. We’ll use examples of churned and not churned customers as the training dataset.&nbsp;
</p><strong><h3>Setting up your environment</h3></strong><p>To follow along, start a <a href="https://www.elastic.co/elasticsearch/service">free trial of Elastic Cloud</a>, spin up a <a href="https://www.elastic.co/guide/en/cloud/current/ec-create-deployment.html">new deployment</a>, download the <a href="https://storage.googleapis.com/elastic-ml-public/supervised_ml_webinar_20200225/calls.csv">calls.csv</a> and <a href="https://storage.googleapis.com/elastic-ml-public/supervised_ml_webinar_20200225/customers.csv">customers.csv</a> files, and upload them via the <a href="https://www.elastic.co/blog/importing-csv-and-log-data-into-elasticsearch-with-file-data-visualizer">CSV upload feature</a>. Data is derived from a dataset referenced in various sources:<a href="https://www.openml.org/d/40701"> openml</a>,<a href="https://www.kaggle.com/becksddf/churn-in-telecoms-dataset"> kaggle</a>,<a href="http://dataminingconsultant.com/DKD2e_data_sets.zip"> Larose 2014</a>. This dataset was disaggregated into line items, and random features were added. Phone numbers and other data are fabricated and any resemblance to real data is coincidental.&nbsp;
</p><p>Once uploaded you should have data that looks like this:
</p><p>In the customers index:
</p><pre class="prettyprint">{
  "column1" : 0,
  "number_vmail_messages" : 25,
  "account_length" : 128,
  "churn" : 0,
  "customer_service_calls" : 1,
  "voice_mail_plan" : "yes",
  "phone_number" : "415-382-4657",
  "state" : "KS",
  "international_plan" : "no"
}
</pre><p>In the calls index:
</p><pre class="prettyprint">{
  "column1" : 20000,
  "dialled_number" : "408-408-6496",
  "call_charges" : 0.1980211744513012,
  "call_duration" : 2.0445331385440126,
  "@timestamp" : "2019-08-08T13:49:06.850-04:00",
  "phone_number" : "408-365-9011",
  "timestamp" : "2019-08-08 13:49:06.850150"
}
</pre><strong><h3>Building the feature set</h3></strong><p>First, we want to build out our feature set. We have two sources of data: customer metadata and call logs. This requires the use of <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/transforms.html">transforms</a> and the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ingest-enriching-data.html">enrich processor</a>. This combination allows us to merge the metadata along with the call information in the other index.
</p><p>The enrichment policy and pipeline. (Execute the commands in the <a href="https://www.elastic.co/guide/en/kibana/7.8/console-kibana.html">Kibana dev console</a>)
</p><pre class="prettyprint"># Enrichment policy for customer metadata
PUT /_enrich/policy/customer_metadata
{
    "match": {
        "indices": "customers",
        "match_field": "phone_number",
        "enrich_fields": ["account_length", "churn", "customer_service_calls", "international_plan","number_vmail_messages", "state", "voice_mail_plan"]
    }
}
# Execute the policy so we can populate with the metadata
POST /_enrich/policy/customer_metadata/_execute
# Our enrichment pipeline for generating features
PUT _ingest/pipeline/customer_metadata
{
  "description": "Adds metadata about customers by phone_number", 
  "processors": [
      {
        "enrich": {
          "policy_name": "customer_metadata",
          "field": "phone_number",
          "target_field": "customer",
          "max_matches": 1
        }
      }
    ]
}
</pre><p>We can now build a transform that utilizes our new pipeline. Here is the transform definition:
</p><pre class="prettyprint"># transform for enriching the data for training
PUT _transform/customer_churn_transform
{
 "source": {
   "index": [
     "calls"
   ]
 },
 "dest": {
   "index": "churn",
   "pipeline": "customer_metadata"
 },
 "pivot": {
   "group_by": {
     "phone_number": {
       "terms": {
         "field": "phone_number"
       }
     }
   },
   "aggregations": {
     "call_charges": {
       "sum": {
         "field": "call_charges"
       }
     },
     "call_duration": {
       "sum": {
         "field": "call_duration"
       }
     },
     "call_count": {
       "value_count": {
         "field": "dialled_number"
       }
     }
   }
 }
}
</pre><p>Now that the transform is created, we can start it and see its progress on the Transforms page under Stack Management.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt775cd16542ef4d91/5f12125caa875d7f39eaed33/blog-binary-class-transforms.png" data-sys-asset-uid="blt775cd16542ef4d91" alt="Transforms in Kibana" style="display: block; margin: auto;">
</p><p>We have built sums of call charges, number of calls, and call durations. These features paired with our customer metadata should be enough to build a churn prediction model.
</p><p>We have a total of nine features in our feature set, three of which are categorical features. Additionally, we have the field <code>customer.churn</code>. This represents historical data of customers that have churned in the past. It will be the label our supervised learning utilizes to build our predictive model.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1ae0a97f476a6b7e/5f12127b2cd8e87f92bdeeba/blog-binary-class-fields.png" data-sys-asset-uid="blt1ae0a97f476a6b7e" alt="The fields we'll use for our job" style="display: block; margin: auto;">
</p><strong><h3>Building the model</h3></strong><p>Once we have our feature set, we can start building our supervised model. We will predict churn based on our customer calling data.&nbsp;
</p><p><strong>Note:</strong> An <a href="https://www.elastic.co/guide/en/kibana/7.8/tutorial-define-index.html">index pattern</a> referencing the newly pivoted data is required. The pattern I chose was <code>churn*</code>.
</p><p>Navigate to the <a href="https://www.elastic.co/guide/en/kibana/7.8/xpack-ml-dfanalytics.html">Data Frame Analytics</a> tab in the Machine Learning app&nbsp;in Kibana.
</p><p>To create the job let’s first:
</p><table style="border:none; text-align:left; width:100%">
<tbody>
<tr>
	<td style="border:none; vertical-align:middle; width:33%;"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb4865245be261382/5f1210ee9be1ce01c562ba17/blog-binary-class-job-meta.png" data-sys-asset-uid="bltb4865245be261382" alt="Transform analytics meta data" style="display: block; margin: auto;">
	</td>
	<td style="border:none; vertical-align:middle; width:67%;">
		<ul>
			<li>Select the “classification” job type for predicting the two classes of “churn”</li>
			<li>Give the job a unique ID</li>
			<li>Add a nice description</li>
			<li>Specify where the training data resides</li>
			<li>Specify where our predictions will be written</li>
		</ul>
	</td>
</tr>
</tbody>
</table><p>Now that our job metadata is out of the way, let's choose our features and training.
</p><table style="border:none; text-align:left; width:100%">
<tbody>
<tr>
	<td style="border:none; vertical-align:middle; width:33%;"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt708ac0089aacefc3/5f121105aa5d18016392a0ea/blog-binary-class-job-features.png" data-sys-asset-uid="blt708ac0089aacefc3" alt="Transform analytics job features and training" style="display: block; margin: auto;">
	</td>
	<td style="border:none; vertical-align:middle; width:67%;">
		<ul>
			<li>The dependent variable is “customer.churn” as that is what we are trying to predict</li>
			<li>It's fine to keep the default training percent of 80</li>
			<li>We should exclude fields we know don’t add information. In this case, “customer.phone_number” and “phone_number” are unique per entry and don’t contain useful information.</li>
			<li>Keep the model memory limit as it is. This allows the job executor to execute on a node that can support at least this amount of memory.</li>
		</ul>
	</td>
</tr>
</tbody>
</table><p>This analytics job will determine the following:
</p><ul>
	<li>The best encodings for each feature</li>
	<li>Which features give the best information or which should be ignored</li>
	<li>The optimal hyperparameters with which to build the best model</li>
</ul><p>Once it has finished running, we can see the model error rates given its testing and training data.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0ef040498813e8b8/5f1212a2ceebbf7e87c31cca/blog-binary-class-error-rates.png" data-sys-asset-uid="blt0ef040498813e8b8" alt="Error rates from our job" style="display: block; margin: auto;">
</p><p>Pretty good accuracy! If you are following along, your numbers might differ slightly. When the model is trained, it uses a random subset of the data.
</p><p>We now have a model that we can use in an ingest pipeline.
</p> <strong><h3>Using the model</h3></strong><p>Since we know which data frame analytics job created this model, we can see its ID and various settings with this API call:
</p><pre class="prettyprint">GET _ml/inference/customer_churn*?human=true
</pre><p>With the model ID in hand we can now make predictions. We built the original feature set through transforms. We will do the same for all data that we send through the inference processor.&nbsp;
</p><pre class="prettyprint"># Enrichment + prediction pipeline
# NOTE: model_id will be different for your data
PUT _ingest/pipeline/customer_churn_enrich_and_predict
{
  "description": "enriches the data with the customer info if known and makes a churn prediction", 
  "processors": [
      {
        "enrich": {
          "policy_name": "customer_metadata",
          "field": "phone_number",
          "target_field": "customer",
          "max_matches": 1,
          "tag": "customer_data_enrichment"
        }
      },
      {
        "inference": {
          "model_id": "customer_churn-1581693287679",
          "inference_config": {
            "classification": {}
          },
          "field_map": {},
          "tag": "chrun_prediction"
        }
      }
    ]
}
</pre><p>Time to put it all in a continuous transform.
</p><pre class="prettyprint">PUT _transform/continuous-customer-churn-prediction
{
 "sync": {
   "time": {
     "field": "@timestamp",
     "delay": "10m"
   }
 },
 "source": {
   "index": [
     "calls"
   ]
 },
 "dest": {
   "index": "churn_predictions",
   "pipeline": "customer_churn_enrich_and_predict"
 },
 "pivot": {
   "group_by": {
     "phone_number": {
       "terms": {
         "field": "phone_number"
       }
     }
   },
   "aggregations": {
     "call_charges": {
       "sum": {
         "field": "call_charges"
       }
     },
     "call_duration": {
       "sum": {
         "field": "call_duration"
       }
     },
     "call_count": {
       "value_count": {
         "field": "dialled_number"
       }
     }
   }
 }
}
</pre><p>As the transform sees new data, it creates our feature set and infers against the model. The prediction along with the enriched data is indexed into <code>churn_predictions</code>. This all occurs continuously. As new customer call data comes in, the transform updates our churn predictions. The predictions are data in an index. This means we can operationalize via <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/watcher-getting-started.html">alerting</a>, build visualizations with <a href="https://www.elastic.co/guide/en/kibana/7.8/lens.html">Lens</a>, or build a custom dashboard.&nbsp;
</p><strong><h2>Conclusion</h2></strong>Data pipelining for feature creation, model training, and model inference is now possible inside Elastic. This means that data in <a href="https://www.elastic.co/apm">Elastic APM</a>, <a href="https://www.elastic.co/siem">SIEM</a>, or <a href="https://www.elastic.co/log-monitoring">Logs</a> can be used to build supervised machine learning models. This opens the door for many use cases in&nbsp;security, observability, logistics, or good ol' curiosity. Try out the new machine learning features today in a <a href="https://www.elastic.co/elasticsearch/service">free trial of Elastic Cloud</a>. What questions will machine learning answer for you?

blog-banner-charts-monitor.png

blog-thumb-charts-monitor.png

Using Elastic supervised machine learning for binary classification

Benjamin Trent

Articles By

Register to Attend

Sign In to Attend

Register to Watch

You'll also receive an email with related content

Watch Now

Learn More

Contact Info

See All Posts

What to explore next...

Load more news

Load more press releases

Load More

View more learning opportunities

More

Date

Location

Agenda

Hosted by

Related workshops

Translate Content

Articles by Benjamin Trent

Using Elastic supervised machine learning for binary classification

Products & Solutions

Company

Resources

Featured topics

News

Articles by Benjamin Trent

Using Elastic supervised machine learning for binary classification