Upload custom plugins, dictionaries, and scripts

There are several cases where you might need your own files to be made available to your Elasticsearch cluster’s nodes:

To facilitate this, we make it possible to upload a ZIP file that contains the files you want to make available. Uploaded files are stored using Amazon’s highly-available S3 service. This is necessary so we do not have to rely on the availability of third-party services, such as the official plugin repository, when provisioning nodes.

Custom plugins, dictionaries and scripts are collectively referred to as extensions.

Before you begin

The selected plugins/bundles are downloaded and provided when a node starts. Changing a plugin does not change it for nodes already running it. See Updating Plugins and Bundles.

With great power comes great responsibility: Your plugins can extend your cluster with new functionality, but also make it break. Be careful. We obviously cannot guarantee that your custom code works.

You cannot edit or delete a custom extension after it has been used in a deployment. To remove it from your deployment, you can disable the extension and update your deployment configuration.

Uploaded files cannot be bigger than 20 MB for most subscription levels, for Platinum the limit is 1GB.

It is important that plugins and dictionaries that you reference in mappings and configurations are available at all times. For example, if you try to upgrade Elasticsearch and de-select a dictionary that is referenced in your mapping, the new nodes will be unable to recover the cluster state and function. This is true even if the dictionary is referenced by an empty index you do not actually use.

Prepare your files for upload

Files you upload need to be in ZIP file format. You need to choose whether your uploaded file should be treated as a plugin or as a bundle. Bundles are not installed as plugins. If you need to upload both a custom plugin and custom dictionaries, upload them separately.

To prepare your files, create one of the following:

Plugins

A plugin is a ZIP file that is installable as a plugin using Elasticsearch’s plugin tool. When configuring the node of your cluster, that is exactly what we do. We invoke the install-tool with the URL of the uploaded ZIP file. This makes it easy for you to test that your uploaded ZIP file works locally: just see if you can run ./bin/elasticsearch-plugin install file:///path/to/plugin.zip yourself.

Elasticsearch’s plugin tool assumes that the uploaded ZIP file contains binaries. If it finds any source code, it will fail with an error message, causing provisioning to fail. Make sure you upload binaries, and not source code.

Bundles

The entire content of a bundle is made available to the node. This is useful to make custom dictionaries and scripts available, for example. You Place scripts in a /scripts folder and dictionaries in a /dictionaries folder in the root path of your ZIP file.

Here is an example of two bundles which contain a script and a dictionary of synonyms:

$ tree .
.
└── scripts
    └── test.groovy

The script test.groovy can be referred in queries as "script": "test".

$ tree .
.
└── dictionaries
    └── synonyms.txt

The dictionary synonyms.txt can be used as synonyms.txt in the synonyms_path of the synonym-filter.

To learn more about analyzing with synonyms, see Synonym token filter and Formatting Synonyms.

Add your extension

You must upload your files before you can apply them to your cluster configuration:

  1. Log in to the Elasticsearch Service Console.
  2. Select your deployment on the home page in the Elasticsearch Service card or go to the deployments page.

    Narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list.

  3. Go to the Extensions page.
  4. Click Create extension.
  5. Complete the extension fields, including the Elasticsearch version. For bundle extensions, you can use a wildcard for a version integer.
  6. Select the extension Type.
  7. Under Plugin file, choose the file to upload.
  8. Click Create extension.

After creating your extension, you can enable them for existing Elasticsearch deployments or enable them when creating new deployments.

Update your deployment configuration

After uploading your files, you can select to enable them when creating a new Elasticsearch deployment. For existing deployments, you must update your deployment configuration to use the new files:

  1. Log in to the Elasticsearch Service Console.
  2. Select your deployment on the home page in the Elasticsearch Service card or go to the deployments page.

    Narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list.

  3. From your deployment menu, click Edit.
  4. From the Elasticsearch plugins, extensions, and settings list, select the custom extension.
  5. Click Save.

The selected plugins or bundles are downloaded and provisioned when each node in your Elasticsearch cluster starts.

Update your extension

While you can update the ZIP file for any plugin or bundle, this will not update the files on nodes that are already using the plugin or bundle. These are downloaded and made available when a node is started.

  1. Prepare a new plugin or bundle.
  2. Make your new files available by uploading them.
  3. On the Extensions page, update your extension.
  4. Select your deployment on the home page in the Elasticsearch Service card or go to the deployments page.

On the Edit page, the new extension appears under Elasticsearch plugins, extensions, and settings.

Using scripts

To use scripts, you need to upload a bundle containing the scripts. A bundle is a ZIP file that you upload and associate with your cluster’s configuration.

The bundle needs to contain a folder named scripts/, with the scripts inside. Scripts in sub-folders must be referred to with _ as the delimiter, and not /. For example, the script scripts/foo/bar.grovy must be referred to as foo_bar.

Here’s an example of three scripts and how to invoke them:

$ tree scripts # Show files and their placement
scripts
├── module1
│   └── script_name.groovy
├── module2
│   └── other_script.groovy
└── top_level.groovy

$ cat scripts/top_level.groovy
123 * my_var

$ cat scripts/module1/script_name.groovy
42 + my_var

$ cat scripts/module2/other_script.groovy
42 * my_var

$ zip my-scripts.zip -r scripts # make a zip-bundle of the scripts
  adding: scripts/ (stored 0%)
  adding: scripts/module1/ (stored 0%)
  adding: scripts/module1/script_name.groovy (stored 0%)
  adding: scripts/module2/ (stored 0%)
  adding: scripts/module2/other_script.groovy (stored 0%)
  adding: scripts/top_level.groovy (stored 0%)

With these scripts, you can upload my-scripts.zip as a bundle and associate them with a cluster.

Then, you can use them like the following:

  • The script scripts/top_level.groovy can be referred to as top_level:
{
    "size": 1,
    "script_fields": {
        "foo": {
            "script": "top_level",
            "params": {
                "my_var": 2
            }
        }
    }
}
  • The script scripts/module1/script_name.groovy can be referred to as module1_script_name
{
    "size": 1,
    "script_fields": {
        "foo": {
            "script": "module1_script_name",
            "params": {
                "my_var": 2
            }
        }
    }
}
  • The script scripts/module2/other_script.groovy can be referred to as module2_other_script:
{
    "size": 1,
    "script_fields": {
        "sample_field": {
            "script": "module2_other_script",
            "params": {
                "my_var": 2
            }
        }
    }
}