Uploading Custom Plugins, Dictionaries, and Scripts

Tip

Only Gold and Platinum subscriptions, running version 2.4.6 or later, have access to uploading custom plugins. All subscription levels, including Standard, can upload scripts and dictionaries.

There are several cases where you might need your own files to be made available to your Elasticsearch cluster’s nodes:

To facilitate this, we make it possible to upload a ZIP file that contains the files you want to make available. Uploaded files are stored using Amazon’s highly-available S3 service. This is necessary so we do not have to rely on the availability of third-party services, such as the official plugin repository, when provisioning nodes.

Plugins versus Bundles

Files you upload are expected to be a ZIP file. Also, you need to choose whether your uploaded file should be treated as a plugin or as a bundle.

A plugin is a ZIP file that is installable as a plugin using Elasticsearch’s plugin tool. When configuring the node of your cluster, that is exactly what we do. We invoke the install-tool with the URL of the uploaded ZIP file. This makes it easy for you to test that your uploaded ZIP file works locally: just see if you can run bin/plugin --url file:///path/to/plugin --install plugin-name yourself.

Elasticsearch’s plugin tool assumes that the uploaded ZIP file contains binaries. If it finds any source code, it will fail with an error message, causing provisioning to fail. Make sure you upload binaries, and not source code.

For bundles, the entire contents are made available to the node. This is useful to make custom dictionaries, scripts, etc. available.

Bundles are not installed as plugins. If you need to upload both a custom plugin and custom dictionaries, you will need to upload the plugin and dictionary separately.

Restrictions and Caveats

The selected plugins/bundles are downloaded and provided when a node starts. Changing a plugin does not change it for nodes already running it. See Updating Plugins and Bundles.

With great power comes great responsibility: Your plugins can extend your cluster with new functionality, but also make it break. Be careful. We obviously cannot guarantee that your custom code works.

Important

You cannot edit or delete a custom plugin after it has been uploaded. To remove it from your cluster, you can disable the plugin and update your cluster configuration.

Uploaded files cannot be bigger than 20 MB, i.e. 20971520 bytes.

It is important that plugins and dictionaries that you reference in mappings and configurations are available at all times. For example, if you try to upgrade Elasticsearch and de-select a dictionary that is referenced in your mapping, the new nodes will be unable to recover the cluster state and function. This is true even if the dictionary is referenced by an empty index you do not actually use.

File Preparation

Files you upload need to be in ZIP file format. You need to choose whether your uploaded file should be treated as a plugin or as a bundle.

To prepare your files, create one of the following:

Plugins

A plugin is a ZIP file that is installable as a plugin using Elasticsearch’s plugin tool. When configuring the node of your cluster, that is exactly what we do. We invoke the install-tool with the URL of the uploaded ZIP file. This makes it easy for you to test that your uploaded ZIP file works locally: just see if you can run bin/plugin --url file:///path/to/plugin --install plugin-name yourself.

Elasticsearch’s plugin tool assumes that the uploaded ZIP file contains binaries. If it finds any source code, it will fail with an error message, causing provisioning to fail. Make sure you upload binaries, and not source code.

Bundles

The entire content of a bundle is made available to the node. This is useful to make custom dictionaries and scripts available, for example. Elastic Cloud expects scripts to be under a /scripts path and dictionaries to be under a /dictionaries path in the root of your ZIP file.

Here is an example of two bundles which contain a script and a dictionary of synonyms:

$ tree .
.
└── scripts
    └── test.groovy

The script test.groovy can be referred in queries as "script": "test".

$ tree .
.
└── dictionaries
    └── synonyms.txt

The dictionary synonyms.txt can be used as synonyms.txt in the synonyms_path of the synonym-filter.

Note

Bundles are not installed as plugins. If you need to upload both a custom plugin and custom dictionaries, upload them separately.

Upload Your Files

You must upload your files before you can apply them to your cluster configuration:

  1. Log into the Elastic Cloud Console.
  2. Go to the Custom plugins page.
  3. Click Add plugin.
  4. Complete the plugin fields, including the Elasticsearch version. For bundle plugins, you can use a wildcard for a version integer.
  5. Select the Plugin type.
  6. Click Create plugin.
  7. Go back to the Custom plugins page and click on the new plugin.
  8. Under Plugin file, choose the file to upload.

After uploading your files is complete, you can enable them for existing Elasticsearch clusters or enable them when creating new clusters.

Update Your Cluster Configuration

After uploading your files, you can select to enable them when creating a new Elasticsearch cluster. For existing clusters, you must update your cluster configuration to use the new files:

  1. Log into the Elastic Cloud Console.
  2. On the Deployments page, select your deployment.

    Narrow the list of the deployments by name, ID, or choose from several other filters. Use a combination of them to further define the list.

  3. From your deployment menu, go to Elasticsearch and click Edit.
  4. From the Plugins list, select the custom file.
  5. Click Update.

The selected plugins or bundles are downloaded and provisioned when each node in your Elasticsearch cluster starts.

Updating Plugins and Bundles

While you can update the ZIP file for any plugin or bundle, this will not update the files on nodes that are already using the plugin or bundle. These are downloaded and made available when a node is started.

  1. Prepare a new plugin or bundle.
  2. Make your new files available by uploading them.
  3. Re-configure your cluster to use the new plugin or bundle.