Document enrichment with MLedit

Starting in 8.5, Enterprise Search creates ingestion pipelines by default to work with indices created for search use cases. The ML inference pipeline uses inference processors to analyze fields and enrich documents with the output. Inference processors use ML trained models, so you need to deploy a trained model in your cluster to use this feature.

This guide focuses on the ML inference pipeline, its use, and how to manage it.

NLP use casesedit

Natural Language Processing (NLP) allows developers to create rich search experiences that go beyond the standards of lexical search. A few examples of ways to improve search experiences through the use of NLP models:

Named entity recognition (NER)edit

Most commonly used to detect entities such as People, Places, and Organization information from text, NER can be used to extract key information from text and group results based on that information. A sports news media site could use NER to automatically extract names of professional athletes, stadiums, and sports teams in their articles and link to season stats or schedules.

Text classificationedit

Text classification is commonly used for sentiment analysis and can be used for similar tasks, such as labeling content as containing hate speech in public forums, or triaging and labeling support tickets so they reach the correct level of escalation automatically.

Text embeddingedit

Analyzing a text field using a Text embedding model will generate a dense_vector representation of the text. This array of numeric values encodes the semantic meaning of the text. Using the same model with a user’s search query will produce a vector that can then be used to search, ranking results based on vector similarity - semantic similarity - as opposed to traditional word or text similarity.

A common use case is a user searching FAQs, or a support agent searching a knowledge base, where semantically similar content may be indexed with little similarity in phrasing.

NLP in Enterprise Searchedit

Overview of ML inference pipeline in Enterprise Searchedit

The diagram below shows how documents are processed during ingestion.

Find, deploy, and manage trained modelsedit

This feature is intended to make it easier to use your ML trained models. First, you need to figure out which model works best for your data. Make sure to use a compatible third party NLP model. Since these are publicly available, it is not possible to fine-tune models before deploying them.

Trained models must be available in the current Kibana Space and running in order to use them. By default, models should be available in all Kibana Spaces that have the Analytics > Machine Learning feature enabled. To manage your trained models, use the Kibana UI and navigate to Stack Management → Machine Learning → Trained Models. Spaces can be controlled in the spaces column. To stop or start a model, go to the Machine Learning tab in the Analytics menu of Kibana and click Trained Models in the Model Management section.

Add inference processors to your ML inference pipelineedit

To create the index-specific ML inference pipeline, go to Enterprise Search → Content → Indices → <your index> → Pipelines in the Kibana UI.

If you only see the ent-search-generic-ingestion pipeline, you will need to open the Settings and choose Copy and customize to create index-specific pipelines. This will create the {index_name}@ml-inference pipeline.

Once your index-specific ML inference pipeline is ready, you can add inference processors that use your ML trained models. To add an inference processor to the ML inference pipeline, click the Add inference pipeline in the ML inference pipelines card.

Here, you’ll be able to:

Manage and delete inference processors from your ML inference pipelineedit

Inference processors added to your index-specific ML inference pipelines are normal Elasticsearch pipelines. Once created, each processor will have options to View in Stack Management and Delete Pipeline. Deleting an inference processor from within the Enterprise Search view deletes the pipeline and also removes its reference from your index-specific ML inference pipeline.

These pipelines can also be viewed, edited, and deleted in Kibana via Stack Management → Ingest Pipelines, just like all other Elasticsearch ingest pipelines. You may also use the Ingest pipeline APIs. If you delete any of these pipelines outside of the Enterprise Search product in Kibana, please make sure to edit the ML inference pipelines that reference them.

Test your ML inference pipelineedit

To ensure the ML inference pipeline will be run when ingesting documents, you must make sure the documents you are ingesting have a field named _run_ml_inference that is set to true and you must set the pipeline to {index_name}. For connector and crawler indices, this will happen automatically if you’ve configured the settings appropriately for the pipeline name {index_name}. To manage these settings:

Learn More:edit