Migrating your OpenShift Elasticsearch 6.x cluster to Elastic Cloud on Kubernetes (ECK)

New to Elasticsearch? Join our getting started with Elasticsearch webinar. You can also start a free cloud trial or try Elastic on your machine now.

Red Hat's OpenShift platform has long been a trusted foundation for enterprise Kubernetes workloads, and for years, its built-in Elasticsearch Operator has made log management simple. But all good things evolve: The OpenShift Elasticsearch Operator reached the end of its supported lifecycle with Red Hat OpenShift Container Platform (OCP) 4.13, and the Elasticsearch 6.x clusters it managed are long out of support. Working closely with Red Hat, we've put together this step-by-step guide to help you move from that legacy setup to Elastic Cloud on Kubernetes (ECK), the modern, full-featured operator maintained directly by Elastic. The migration path we've designed respects the OpenShift-native tooling you already rely on, keeps disruption to a minimum, and leaves you on a solid foundation for future upgrades to 8.x, 9.x, and beyond.

Why this matters

• Security and support: The last Elasticsearch 6.x patch release was on January 13, 2022. ECK lets you upgrade at your own pace, with a supported operator from the creators of Elasticsearch. Remaining on an old Elasticsearch version exposes you to supportability risks or well-known security issues.
• Features you’ve been missing: Autoscaling, data tiers, machine learning (ML) jobs, searchable snapshots. None of these are available in the legacy operator.

Future-proof operations: ECK ships day-and-date with every new Elastic release, so you’re never stuck waiting again.

High-level plan

Feel free to bookmark this list. Each milestone is small, reversible, and validated before you move on.

0. Preflight checks

A. Health first: Run /_cat/health and make sure you’re green.

B. Disk watermarks: Keep at least 20% free before starting a migration.

C. Final snapshot: S3, GCS, NFS: It doesn’t matter, as long as you can mount the same repo in the new cluster.

1. If you don’t have object storage handy in your environment, you can use this solution-post by Red Hat to snapshot your data to local storage on the OpenShift cluster.

D. Review the documentation: Elastic provides thorough documentation for migrating data between Elasticsearch clusters.

1. Installing ECK 2.16.1 (your “bridge” operator)

ECK 2.16.1 is the last release that still accepts spec.version: 6.8.x, which makes it the ideal bridge between past and future Elasticsearch versions.

You can keep the Red Hat operator in place; the two watch different Custom Resource Definitions (CRDs), so they won’t step on each other’s toes.

Keep in mind that with OpenShift, ECK might display some Transport Layer Security (TLS) errors in its logs as OpenShift tries to connect to its healthcheck webhook endpoint via HTTP, but ECK allows TLS communication only. This is a well-known issue and shouldn’t pose a problem.

You can refer further to Elastic documentation, in case you need to make a local namespaced installation.

2. Launching a 6.x cluster under ECK

Below is a starter Kubernetes manifest that balances resiliency (separate masters) with cost (three hot-tier data nodes). Swap storage class names, resources, and snapshot credentials to match your environment.

Note: The syntax used below is a bit different than what it would be for newer Elasticsearch versions on ECK.

Deploy it, watch pods come up, and you’re ready for data.

3. Moving the data

To move data from one Elasticsearch cluster to another, you can also further consult this guide in the Elastic documentation. For the purpose of this post, we’re assuming that snapshot and restore are used.

Snapshot and restore are quickest:

Can’t share an object store? Use remote re-index (slower, but works everywhere; has drawbacks in terms of not migrating index templates, component templates, and more) or pump logs through a one-off Logstash job.

4. Configuring ClusterLogging operator

First, we’ll need to decommission our Red Hat operator–managed Elasticsearch cluster. We’ll modify our ClusterLogging like so:

Then we’ll define a ClusterLogForwarder to direct the logs from fluentd to our newly built Elasticsearch 6.x cluster managed by ECK. We’ll need to create a secret with the Elasticsearch credentials:

For configuring TLS (as recommended), you’ll need to create a ConfigMap for ClusterLogForwarder to trust the ECK ca certificates. Further guidance can be found here. We’ll run the command: 

To create the certificate secret, and then we’ll reference it in the ClusterLogging CRD:

⚠️ If you’re troubleshooting connectivity issues, you can temporarily set tls.insecureSkipVerify: true, but this shouldn’t be used long term.

Because we’re restoring legacy indices into a fresh ECK-managed cluster, OpenShift Logging will not automatically recreate the legacy index layout or aliases. You must ensure that write aliases exist and point to writable indices. In my case, I needed to verify that I have proper aliases, set up as:

• app-write
• infra-write
• audit-write

Pointing to indices with dynamic mappings (not recommended) for minimizing errors and troubleshooting steps. 

Repeat for app-write and audit-write with their respective backing indices.We should see data start flowing now toward our new ECK managed cluster.

5. Rolling upgrade to 7.17.29, and verify

Now you can finally leave 6.x behind.

A. Check _xpack/migration/deprecations?pretty using curl against Elasticsearch, to tackle deprecations. This API will return either warnings or critical things to attend to before you upgrade.

B. Patch the CRD to upgrade it to the latest 7.x version. I’m using 7.17.29.

C. ECK restarts nodes one at a time. Your cluster should be online throughout.

D. Give cluster tasks and shard recoveries time to settle before pressing on.

E. Don’t forget to upgrade Kibana in the same way.

Once complete, check your Elasticsearch version and Kibana version, as well as the health state:

6. Operator upgrade: ECK 2.16.1 → 3.3.1

ECK upgrades are pleasantly boring:

Watch the operator pod roll. Your Elasticsearch cluster keeps running; only the controller restarts.

Verify that the upgrade is successful by looking at the operator logs and ensuring that no major errors appear:

And then verifying the new version of the operator (will now be 3.3.1):

7. Your roadmap to 8.x and 9.x (when you’re ready)

You’re now on:

• ECK Operator: 3.3.1
• Elastic Stack: 7.17.29

That pair is fully supported and serves as the official launchpad for 8.x. It’s important to first go through the Elastic upgrade documentation.

We’ll again go through the procedure of checking for any hard-breaking changes between our 7.17.29 and the latest 8 version (8.19.9):

It's important to look through the result of this query carefully and to go through necessary steps, like re-indexing indices and changing mappings, among others.

Once you’ve addressed all required changes from 7.17.29 to 8.x:

ECK will handle the rest. Just remember to upgrade Beats, Logstash pipelines, and client libraries in lockstep to avoid wire-protocol surprises.

Repeat the process again to migrate to the latest 9.x version.

8. Cleanup

• Remove the Red Hat Elasticsearch operator.

Now that you’re no longer using the Red Hat Elasticsearch operator, you can remove it from your cluster. You can do that via the following steps:

A. In the OpenShift Console, go to Operators and then to Installed Operators.

B. In the Filter By Name field, enter “Elasticsearch” to find the installed Red Hat Elasticsearch operator.

C. On the Operator Details page, select Uninstall Operator from the Actions list.

D. On the Uninstall Operator? dialog box, select Uninstall. This removes the operator, the operator deployments, and the pods. After this step, the operator stops running and will no longer receive updates.

All of these steps can be found in this link from Red Hat OpenShift documentation.

Wrapping up

By installing ECK 2.16.1 as a bridge, snapshot-restoring into a new cluster, and stepping cleanly through 7.x before landing on ECK 3.3, you’ve transformed an aging, unsupported logging back end into a modern, secure, first-class Elastic deployment, without surprises or downtime.