Upgrading from Enterprise Search 7.11 and earlieredit

Enterprise Search 7.12 brings changes to the underlying indexing and querying mechanisms, for significant improvements in ingestion times, storage requirements, and search relevance. None of these improvements affect the publicly available APIs and interfaces: all enhancements are found under the hood.

With this newer architecture, Enterprise Search brings an updated and optimized data structure, thus temporarily requiring additional storage capacity during the upgrade phase. Once completed, certain indices can be purged, resulting in a smaller, more efficient deployment.

Summary of the enhancements in Enterprise Search 7.12edit

Enterprise Search packages and focuses the power of Elasticsearch for the most common and demanding search use cases. It is designed to give you the tools to easily deploy quality search experiences without the initial configuration and trial-and-error stages of deployment. For that to occur, Enterprise Search relies on advanced application logic, optimized analysis and query pipelines, as well as simplified public APIs specific to the requirements of typical search use cases.

With the 7.12 release comes significant improvements to the application logic layer, and underlying data models. These changes are applied transparently: APIs and UIs operate in the exact same way, but the revamped internals provide meaningful enhancements:

  • Up to 40% improvement in document ingestion times;
  • Up to 80% improvement in storage requirements;
  • Instant field type changes, schema updates no longer require re-indexing;
  • Increased query precision, with fewer long-tail results;
  • Improved typo-tolerance, for more resilient search.

For example, using a real-world deployment, benchmarking data shows significant gains in a number of key areas. Using the Enron Email Dataset, we can draw the following comparison between older versions and Enterprise Search 7.12 (and beyond):

  • Storage need for indexed documents for a 7.10 deployment: 11.47 GB, across two distinct datastores
  • Storage need for indexed documents for a 7.12+ deployment: 2.9 GB, with a single datastore

Preparing your deployment for the upgradeedit

To upgrade from App Search 7.6 or earlier, you must first migrate to Enterprise Search 7.9.3. See Migrating from App Search on Elastic Cloud or Migrating from self-managed App Search. Then continue with the following steps.

Given the significant underlying changes, upgrading from Enterprise Search 7.11 or earlier requires a migration at upgrade time. There are a few key items to consider when preparing for the upgrade:

  1. The infrastructure on which the existing Enterprise Search deployment exists must have at least 50% of disk space available. The migration requires duplication of documents and content, and the upgrade will not be initiated should the capacity be below this threshold.
  2. If you have a large number of engines (e.g. more than a hundred), or share the same Elasticsearch cluster for additional non-Enterprise Search use cases, you will want to ensure you do not exceed the max shards per node. Follow the instructions in the Checking shards section of this guide.
  3. The migration can take anywhere from a few minutes to a few hours for large deployments. The existing deployment remains available in read-only mode, and search queries will be served normally, with no degradation in experience. Note: Search analytics events will not be recorded for the window of migration.
  4. It is recommended to create a snapshot of the deployment as a safety measure in the very unlikely event of non-recoverable failure from the upgrade. (Please note that this is recommended for any upgrade).

Upgrading to Enterprise Search from 7.11 or earlier for Elastic Cloud deploymentsedit

To upgrade from App Search 7.6 or earlier, you must first migrate to Enterprise Search 7.9.3. See Migrating from App Search on Elastic Cloud or Migrating from self-managed App Search. Then continue with the following steps.

To upgrade your Enterprise Search deployment to 7.16 on Elastic Cloud:

  1. Log in to the Elasticsearch Service Console.
  2. Select the Enterprise Search deployment you want to upgrade to 7.16.
  3. Ensure you have enough storage for the upgrade:

    1. Scroll down to the Instances section and check each of your Elasticsearch instances.
    2. The disk allocation of your Elasticsearch instances should be under 40%.
    3. If any are not under 40%, you will have to temporarily increase your Elasticsearch instance size.
  4. Select Upgrade and then in the dialog that opens, select the latest version of 7.16, then click Upgrade.

This will kick off the automated migration process to 7.16.

Once 7.16 is up, validate that your Enterprise Search data is available: do any performance, functional, or other tests that you might need.

You can then purge the old indices to free up storage space. Once the obsolete indices are removed, you can return your Elasticsearch instance sizes to their original configuration if you had increased them for the migration.

If the upgrade fails, it can be safely retried using the steps above. Check FAQ and troubleshooting for solving common upgrade problems.

Upgrading to Enterprise Search from 7.11 or earlier for self-managed deploymentsedit

To upgrade from App Search 7.6 or earlier, you must first migrate to Enterprise Search 7.9.3. See Migrating from App Search on Elastic Cloud or Migrating from self-managed App Search. Then continue with the following steps.

When upgrading Enterprise Search, you must also upgrade Elasticsearch. Both services must be running the same minor version.

Once you’ve ensured that you have the additional storage space required in your deployment, you can follow any of the options in Upgrading self-managed deployments.

The migration process will start automatically when 7.16 loads.

After upgrading, validate that your Enterprise Search data is available. You can then purge the old indices to free up storage space.

If the upgrade fails, it can be safely retried by restarting any of your 7.16 instances. The upgrade process is coordinated between instances to ensure that only one instance will trigger the migration process. Check FAQ and troubleshooting for solving common upgrade problems.

Purging old data and documentsedit

In order to benefit from the significant storage improvements after upgrading, you must remove the now obsoletely structured search indices. Worry not: this process is straightforward.

Step 1 - Once you have successfully upgraded your deployment, verify that you are able to log into the Enterprise Search experience and access either the Workplace Search or App Search dashboards. Also verify that indexing and search are working as they did before the upgrade.

Step 2 - Find your Elasticsearch endpoint and credentials. In the following steps, ELASTICSEARCH_URL is a combination of your Elasticsearch host and credentials (e.g., ELASTICSEARCH_URL=http://elastic:changeme@localhost:9200).

  • For Elastic Cloud deployments, the Elasticsearch endpoint can be found under Elasticsearch > Copy endpoint. The password for the elastic user was provided when the deployment was created. Alternatively, you can execute these requests in the Elastic Cloud Elasticsearch API console, under Elasticsearch > API Console.
  • For self-managed deployments, (depending on your setup) you may be able to use the values from your Enterprise Search configuration: elasticsearch.host, elasticsearch.username, elasticsearch.password.

Step 3 - List the obsolete indices.

curl "$ELASTICSEARCH_URL/_cat/indices/.ent-search-engine-*,.ent-search-document-backend-*,-.ent-search-engine-documents-*?v&s=index"

Prior to 7.12+, engine documents were stored in three indices per engine:

  • .ent-search-document-backend-$ENGINE_ID
  • .ent-search-document-backend-$ENGINE_ID-document_type_id-external_id-unique-constraint
  • .ent-search-engine-$ENGINE_ID

Step 4 - List the new indices and compare document counts to verify the migration completed successfully

curl "$ELASTICSEARCH_URL/_cat/indices/.ent-search-engine-documents-*?v&s=index"

In 7.12+, the three engine document indices were combined into one:

  • .ent-search-engine-documents-$ENGINE_NAME

Step 5 - Delete each obsolete index. Be sure to take a snapshot before completing this step as it cannot be undone. Also be careful not to delete the new 7.12+ indices, which match the format: .ent-search-engine-documents-*

curl "$ELASTICSEARCH_URL/.ent-search-engine-$ENGINE_ID" -X DELETE
curl "$ELASTICSEARCH_URL/.ent-search-document-backend-$ENGINE_ID" -X DELETE
curl "$ELASTICSEARCH_URL/.ent-search-document-backend-$ENGINE_ID-document_type_id-external_id-unique-constraint" -X DELETE

Checking and increasing the max shards per nodeedit

If you have more than a hundred engines or additional indices on your Elasticsearch cluster, during the upgrade your shard count may approach the maximum per-node set in Elasticsearch.

Step 1 - Determine how many additional shards the upgrade process will require: take the number of App Search engines plus Workplace Search content sources, multiply that by 4 (two shards x one replica), and divide the result by the number of Elasticsearch data nodes.

Step 2 - Check your Elasticsearch _cat allocation API to determine the current number of shards per data node:

curl -XGET "$ELASTICSEARCH_URL/_cat/allocation?v"

Step 3 - Check the Elasticsearch cluster.max_shards_per_node setting. (If you haven’t customized this setting, it defaults to 1000 shards per data node):

# If you have previously customized the setting using dynamic cluster settings
curl -XGET "$ELASTICSEARCH_URL/_cluster/settings"

# If you have previously customized the setting from the elasticsearch.yml file
curl -XGET "$ELASTICSEARCH_URL/_nodes/settings"

Step 4 - If adding the number of additional shards (Step 1) to the current number of shards (Step 2) exceeds the max_shards_per_node limit (Step 3), you will have to temporarily relax the cluster.max_shards_per_node limit in Elasticsearch before upgrading. As an example, this command increases the limit from the default 1000 to 1500:

curl -XPUT "$ELASTICSEARCH_URL/_cluster/settings" -H 'Content-Type: application/json' -d'{  "persistent": {    "cluster.max_shards_per_node": 1500  }}'

After the migration has completed successfully, and you’ve purged the old indices, you can revert the setting:

curl -XPUT "$ELASTICSEARCH_URL/_cluster/settings" -H 'Content-Type: application/json' -d'{  "persistent": {    "cluster.max_shards_per_node": null  }}'

FAQ and troubleshootingedit

The upgrade keeps failing. Anything I should check for?

The most common hurdle to upgrading from Enterprise Search 7.11 to a newer version is available storage capacity for the deployment. Given the significant changes made to the search index architecture powering Enterprise Search, all data structures must be duplicated temporarily at upgrade time, and therefore available deployment storage must be at least twice as much as the deployment size itself. Once the deployment upgrade has successfully completed, you may remove the obsolete search indices following the purging instructions above.

Another common challenge is deployment scale. The bigger the deployment, the longer the migration time. In very rare cases (where a deployment is unusually large), timeouts may occur. Restarting the upgrade will pick up the migration where the timeout occurred and should complete successfully.

I’ve upgraded successfully but my deployment is much larger than it used to be, when I was expecting it to be much smaller. What’s wrong?

This is expected. Given the significant changes made to the search index architecture powering Enterprise Search, all data structures must be duplicated temporarily at upgrade time. Once the deployment upgrade has successfully completed, you may remove the obsolete search indices following the purging instructions above.

The search experience seems to be returning fewer results for all queries. Is this normal?

In light of the valuable feedback of hundreds of organizations and users, newer versions of Enterprise Search bring an updated relevance model that promotes precision in favor of recall. Don’t worry, all your documents are still there, but your search experience just got a whole lot more tidy and precise.