Upload custom plugins and bundles

Upload a ZIP file when you need a custom or third-party plugin that Elastic Cloud Hosted does not provide, or custom configuration files such as dictionaries and SAML metadata. In the Elastic Cloud console and API, these uploads are extensions.

Uploaded files are stored in highly available object storage so Elastic Cloud does not depend on third-party services, such as a public plugin repository, when provisioning nodes.

Before you upload your first custom plugin or bundle, review the following considerations:

  • The selected plugins and bundles are downloaded and provided when a node starts. Changing a plugin does not change it for nodes already running it. Refer to Replace an extension.

  • Custom plugins can add capabilities to your deployment, but they can also cause failures. Elastic does not guarantee that custom code will work correctly.

  • You cannot edit or delete a custom extension after it has been used in a deployment. To remove it from your deployment, you can disable the extension and update your deployment configuration.

  • Your extension file size limit depends on your subscription level. For Platinum and Enterprise subscriptions, the limit is 8GB. For all other subscription levels, the limit is 20MB.

  • It is important that plugins and dictionaries that you reference in mappings and configurations are available at all times. For example, if you try to upgrade Elasticsearch and de-select a dictionary that is referenced in your mapping, the new nodes will be unable to recover the cluster state and function. This is true even if the dictionary is referenced by an empty index you do not actually use.

Plugins are uploaded as ZIP files. You need to choose whether your uploaded file should be treated as a plugin or as a bundle. Bundles are not installed as plugins. If you need to upload both a custom plugin and custom dictionaries, upload them separately.

To prepare your files, create one of the following:

Plugins

Use a plugin to add functionality to Elasticsearch: a custom or third-party plugin that Elastic Cloud Hosted does not provide, or one that you write yourself.

A plugin is a ZIP file that contains a plugin descriptor file and binaries. The descriptor file is called stable-plugin-descriptor.properties for plugins built against the stable plugin API, or plugin-descriptor.properties for plugins built against the classic plugin API. A plugin ZIP file should contain only one descriptor file.

Elasticsearch assumes that the ZIP file contains binaries. If it finds any source code, it fails with an error message, causing provisioning to fail. Make sure the ZIP file contains binaries, and not source code.

Note

Plugins larger than 5GB should have the plugin descriptor file at the top of the archive. This order can be achieved by specifying at time of creating the ZIP file:

zip -r name-of-plugin.zip name-of-descriptor-file.properties *
Bundles

Use a bundle to make configuration files, such as custom dictionaries or SAML metadata, available to every node.

A bundle is a ZIP file whose entire content is extracted to the /app/config directory of every Elasticsearch node. The directory structure inside the ZIP file determines where the files land: folders such as truststore, saml, and ingest-geoip keep their names, so a bundle that contains truststore/keystore.ks makes the keystore available at /app/config/truststore/keystore.ks. Use that full path when you reference the file in your Elasticsearch settings.

Dictionaries are the exception. Place them in a /dictionaries folder in the root path of your ZIP file, and their contents are extracted directly to /app/config rather than to an /app/config/dictionaries subfolder.

Here are some examples of bundles:

Dictionary of synonyms

$ tree .
.
└── dictionaries
    └── synonyms.txt

The dictionary synonyms.txt can be used as synonyms.txt or using the full path /app/config/synonyms.txt in the synonyms_path of the synonym token filter.

To learn more about analyzing with synonyms, check Synonym token filter and Formatting Synonyms.

GeoIP database bundle

$ tree .
.
└── ingest-geoip
    └── MyGeoLite2-City.mmdb

Note that the extension must be -(City|Country|ASN).mmdb, and it must be a different name than the original file name GeoLite2-City.mmdb which already exists in Elastic Cloud Hosted. To use this bundle, you can refer it in the GeoIP ingest pipeline as MyGeoLite2-City.mmdb under database_file.

You must upload your files before you can apply them to your cluster configuration:

  1. Log in to Elastic Cloud.

  2. From the navigation menu, select Extensions.

  3. Click Create extension.

  4. Complete the extension fields, including the Elasticsearch version.

    • Plugins must use full version notation down to the patch level, such as 7.10.1. You cannot use wildcards. This version notation should match the version in your plugin’s plugin descriptor file. For classic plugins, it should also match the target deployment version.
    • Bundles should specify major or minor versions with wildcards, such as 7.* or *. Wildcards are recommended to ensure the bundle is compatible across all versions of these releases.

  5. Click Create extension.

After creating your extension, you can enable it on an existing Elasticsearch deployment or enable it when creating new deployments.

Note

Creating extensions larger than 200MB must be done through the API. Refer to Upload an extension using the API.

After uploading your files, you can enable them when creating a new Elasticsearch deployment. For existing deployments, enable them from the deployment edit page:

  1. Log in to Elastic Cloud.

  2. Find your deployment on the home page or on the Hosted deployments page, then select Manage to access its settings menus.

    On the Hosted deployments page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list.

  3. From the Actions dropdown, select Edit deployment.

  4. Select Manage user settings and extensions.

  5. Select the Extensions tab.

  6. Select the plugins or extensions that you want to enable.

  7. Select Back.

  8. Select Save. The Elasticsearch cluster is then updated with new nodes that have the plugin installed.

While you can update the ZIP file for any plugin or bundle, these are downloaded and made available only when a node is started.

If the extension is not in use by any deployments, you can update the files or extension details. However, if the extension is in use, and if you need to update it with a new file, it is recommended to create a new extension rather than updating the existing one that is in use.

By following this method, only the one node would be down even if the extension file is faulty. This would ensure that HA clusters remain available.

This method also supports having a test/staging deployment to test out the extension changes before applying them on a production deployment.

You may delete the old extension after updating the deployment successfully.

To replace an extension with a new file version:

  1. Prepare a new plugin or bundle.
  2. On the Extensions page, upload a new extension.
  3. Follow the steps in Enable extensions on a deployment. On the Extensions tab, select the new extension and deselect the old one before you save.
  • Be careful when updating an extension. If you update an existing extension with a new file, and if the file is broken for any reason, all the nodes could be impacted, as either a restart or a move node could make even HA clusters non-available. Also, shards of your indices may become unassigned if there's anything wrong with the bundle, for example if a file referenced by an index is missing due to the update.
  • If you need to update your extension, instead of updating an existing extension with a new file directly, create a new extension to test the behavior first, verify its validity, and then apply it to your deployment.

Use the extensions API to upload plugins and bundles programmatically. You must use the API for extensions larger than 200MB; the Cloud UI supports uploads up to that size. You must also use the API for automation or when your ZIP file is not reachable from a public URL in a single request.

Before you start, create an Elastic Cloud API key. You can then create an extension in one of two ways:

To add an extension to a deployment, update its metadata, or delete it afterwards using the Elastic Cloud API, refer to Manage plugins and extensions through the Elastic Cloud API. For the complete HTTP reference, see Extensions API.