Classify text

These NLP tasks enable you to identify the language of text and classify or label unstructured input text:

Language identification enables you to determine the language of text.

A language identification model is provided in your cluster, which you can use in an inference processor of an ingest pipeline by using its model ID (lang_ident_model_1). For an example, refer to Add NLP inference to ingest pipelines.

The longer the text passed into the language identification model, the more accurately the model can identify the language. It is fairly accurate on short samples (for example, 50 character-long streams) in certain languages, but languages that are similar to each other are harder to identify based on a short character stream. If there is no valid text from which the identity can be inferred, the model returns the special language code zxx. If you prefer to use a different default value, you can adjust your ingest pipeline to replace zxx predictions with your preferred value.

Language identification takes into account Unicode boundaries when the feature set is built. If the text has diacritical marks, then the model uses that information for identifying the language of the text. In certain cases, the model can detect the source language even if it is not written in the script that the language traditionally uses. These languages are marked in the supported languages table (see below) with the Latn subtag. Language identification supports Unicode input.

The table below contains the ISO codes and the English names of the languages that language identification supports. If a language has a 2-letter ISO 639-1 code, the table contains that identifier. Otherwise, the 3-letter ISO 639-2 code is used. The Latn subtag indicates that the language is transliterated into Latin script.

Text classification assigns the input text to one of multiple classes that best describe the text. The classes used depend on the model and the data set that was used to train it. Based on the number of classes, two main types of classification exist: binary classification, where the number of classes is exactly two, and multi-class classification, where the number of classes is more than two.

This task can help you analyze text for markers of positive or negative sentiment or classify text into various topics. For example, you might use a trained model to perform sentiment analysis and determine whether the following text is "POSITIVE" or "NEGATIVE":

{
    docs: [{"text_field": "This was the best movie I’ve seen in the last decade!"}]
}
...

Likewise, you might use a trained model to perform multi-class classification and determine whether the following text is a news topic related to "SPORTS", "BUSINESS", "LOCAL", or "ENTERTAINMENT":

{
    docs: [{"text_field": "The Blue Jays played their final game in Toronto last night and came out with a win over the Yankees, highlighting just how far the team has come this season."}]
}
...

The zero-shot classification task offers the ability to classify text without training a model on a specific set of classes. Instead, you provide the classes when you deploy the model or at inference time. It uses a model trained on a large data set that has gained a general language understanding and asks the model how well the labels you provided fit with your text.

This task enables you to analyze and classify your input text even when you don’t have sufficient training data to train a text classification model.

For example, you might want to perform multi-class classification and determine whether a news topic is related to "SPORTS", "BUSINESS", "LOCAL", or "ENTERTAINMENT". However, in this case the model is not trained specifically for news classification; instead, the possible labels are provided together with the input text at inference time:

{
    docs: [{"text_field": "The S&P 500 gained a meager 12 points in the day’s trading. Trade volumes remain consistent with those of the past week while investors await word from the Fed about possible rate increases."}],
    "inference_config": {
        "zero_shot_classification": {
            "labels": ["SPORTS", "BUSINESS", "LOCAL", "ENTERTAINMENT"]
        }
    }
}

The task returns the following result:

...
{
    "predicted_value": "BUSINESS"
    ...
}
...

You can use the same model to perform inference with different classes, such as:

{
    docs: [{"text_field": "Hello support team. I’m writing to inquire about the possibility of sending my broadband router in for repairs. The internet is really slow and the router keeps rebooting! It’s a big problem because I’m in the middle of binge-watching The Mandalorian!"}]
    "inference_config": {
        "zero_shot_classification": {
            "labels": ["urgent", "internet", "phone", "cable", "mobile", "tv"]
        }
    }
}

The task returns the following result:

...
{
    "predicted_value": ["urgent", "internet", "tv"]
    ...
}
...

Since you can adjust the labels while you perform inference, this type of task is exceptionally flexible. If you are consistently using the same labels, however, it might be better to use a fine-tuned text classification model.