Migrating from self-managed App Search

Since version 7.7.0, App Search is distributed as Elastic Enterprise Search, a solution which combines App Search and Workplace Search into a single package. There is therefore no direct upgrade path from App Search 7.6 to App Search 7.7. Instead, App Search 7.6 users must migrate to Enterprise Search. The following document outlines the migration process.

Although this document describes a migration, the process resembles a standard Enterprise Search upgrade. We recommend first reviewing the Enterprise Search upgrade documentation.

Migration process overview

When an App Search deployment is migrated to the new Enterprise Search solution, the App Search data stored in Elasticsearch needs to be converted into a new set of indexes compatible with Enterprise Search. This process is non-destructive (meaning App Search data is never removed or modified during an upgrade), allowing operators a reliable way to roll back in case of a failed upgrade attempt. Combined with the usage of the read-only mode flag, this means that an upgrade can be performed in-place without downtime.

Limitations of the migration process

While the migration process is designed to help you with your move to the Enterprise Search solution without any data loss, it has some limitations. Review these limitations before you proceed:

  • Disk Space – your Elasticsearch cluster needs to have enough disk space to accommodate two copies of your data for the period of the migration (since your data is re-indexed in a new format).
  • Analytics and Application Logs are not migrated – to reduce disk space requirements and the time it takes to perform a migration, we have decided not to support old Analytics and Application request logs in Enterprise Search. Your old analytics data will remain stored in Elasticsearch and could be accessed with Kibana, but the Enterprise Search analytics dashboard will not be able to see that data.
  • You cannot skip any migration steps – when deploying the new Enterprise Search solution on your Elasticsearch cluster that used to run App Search, you have to go through the migration process completely. If Enterprise Search detects any App Search indexes on the cluster, a migration will be attempted for each of them automatically and the service will be unavailable until the migration is complete.

If you do not want to migrate your App Search data, you can use a new Elasticsearch cluster or, as a last resort, you can take an Elasticsearch snapshot and then delete all App Search indexes (.app-search*) from the cluster before attempting an Enterprise Search deployment.

Before you begin

Before attempting a migration of an App Search cluster to a new Enterprise Search solution, you need to take a few steps to guarantee your data safety and increase the probability of a successful upgrade:

  • Before you migrate production servers, test the migration process in a development environment to familiarize yourself with the process. Using an Elasticsearch snapshot created from a production deployment may be the best option to completely test the upgrade process without risking your service availability or data consistency.
  • Stop writing to your Elasticsearch cluster. If running App Search version 7.6 or later, enable the read-only mode to guarantee a consistent snapshot of your data. For versions before 7.6, manually stop all write/indexing operations to your cluster at the source.
  • Back up your data with Elasticsearch snapshots. Having a backup of your data will ensure your ability to safely roll back in case of a failed Enterprise Search upgrade.

Migrating from self-managed App Search deployments

The following sections describe different options for migrating on-prem/self-managed App Search deployments.

When dealing with self-managed deployments, you have a lot of flexibility on how you want to approach the migration process. The options below are ordered by their complexity, which often is in reverse proportion to the option’s risk and associated downtime. Which option you choose will depend on your tolerance for downtime, the resources available to you and so on.

In-place migration to Enterprise Search

For cases when you can tolerate some amount of downtime during the upgrade process, an in-place upgrade option may be the easiest one to attempt. Here are the steps you will need to take:

  1. Stop writing to your cluster (enable read-only mode if on the latest App Search version).
  2. Back up your data using an Elasticsearch snapshot.
  3. Stop your App Search instance.
  4. Deploy an Enterprise Search instance using the same Elasticsearch credentials and other settings from app-search.yml in your new enterprise-search.yml configuration file.
  5. Start Enterprise Search and allow it to perform the migration as needed.
  6. After Enterprise Search starts, you should have access to the App Search UI and APIs using your old credentials as usual.
  7. Remove read-only mode from Enterprise Search.
Live in-place migrations

If you can only use one Elasticsearch cluster, but really need to keep your App Search downtime to a minimum, you can slightly change the migration process to perform an upgrade on a live App Search deployment:

  1. Stop writing to your cluster (enable read-only mode if on the latest App Search version).
  2. Back up your data using an Elasticsearch snapshot.
  3. Deploy an Enterprise Search instance using the same Elasticsearch credentials and other settings from your old app-search.yml in your new enterprise-search.yml configuration file. If the instance is going to be running on the same server(s) as App Search, make sure to change the port used by Enterprise Search (ent_search.listen_port) so that your instances could run in parallel. Also set the port in ent_search.external_url to match the new ent_search.listen_port port number.
  4. Start Enterprise Search and allow it to perform the migration as needed.
  5. Use your newly deployed Enterprise Search instance to verify that everything is working.
  6. Switch your production traffic to the new instance (by shifting traffic to the new port via a load-balancer or by changing client configuration).
  7. Shut down App Search when you are sure your deployment works as expected.
  8. Remove read-only mode from Enterprise Search.
Migrating to a new infrastructure

Finally, the safest way to migrate from App Search to Enterprise Search is by migrating to a completely new infrastructure using the backups taken from the existing one:

  1. Stop writing to your cluster (enable read-only mode if on the latest App Search version).
  2. Back up your data using an Elasticsearch snapshot.
  3. Provision a new Elasticsearch cluster using the snapshot taken from the old cluster.
  4. Deploy an Enterprise Search instance using the new Elasticsearch cluster.
  5. Start Enterprise Search and allow it to perform the migration as needed.
  6. After Enterprise Search starts, you should have access to the App Search UI and APIs using your old credentials as usual.
  7. Switch your production traffic to the new product by shifting traffic to the instances via a load-balancer or by changing client configuration.
  8. Shut down App Search when you are sure your deployment works as expected.
  9. Remove read-only mode from Enterprise Search.