Kibana's Road to 6.0 and the Removal of Mapping Types

When Elasticsearch removed mapping types in 6.0.0, it was a major breaking change that affected any application that uses indices with multiple types. Kibana is one such application, relying on multiple types to store objects like visualizations, dashboards, index patterns, and more. How did we successfully migrate Kibana to be compatible with Elasticsearch 6? We share this strategy with you below, and hope that it gives you ideas for how to convert your multi-type indices to single-type.

First, deciding what the new mapping was going to look like. There are a few common alternatives to mapping types:

• Using a single index per type
• Collapsing all properties under a single type
• Creating a custom type field, and nesting all types under a single type.

We chose adding a custom type field and nesting the data under the name of the type. This allowed us to have fields with the same name under different types, which was previously a limitation.

Here is a visual of what this Elasticsearch document transformation looks like:

{
   "_type":"index-pattern",
   "_id":"abc123",
   "_source":{
      "title":"Test pattern"
   }
}
        

{
   "_type":"doc",
   "_id":"index-pattern:abc123",
   "_source":{
      "type":"index-pattern",
      "index-pattern":{
         "title":"Test pattern"
      }
   }
}
        

Once we had the mapping, the next step was migrating the data. When creating the new index, we need to ensure index.mapper.dynamic is false, otherwise we will not be able to fall-back to the single type. Additionally, 5.6 has an option index.mapping.single_type which mimics the behavior in 6.0. After creating the new index with the desired mapping, we reindexed the data into the new format. To do this, we leverage the reindex API to transform the documents, setting the type field and nesting the previous data under the name of the type. Since IDs are only unique within a type, we also prefix the ID with the type. Using an alias, we can swap out the index in a single atomic action without downtime. Details of this full migration process can be found in our manual migration steps.

Kibana 5.6 is considered a compatibility release, allowing users to perform a rolling upgrade to 6.0 without downtime. This means Kibana 5.6 needs to seamlessly handle both single and multiple mapping types. To allow for this, we introduced a saved object API which has become the preferred way of programmatically interacting with Kibana data. This allows for a consistent interface, regardless of the underlying Elasticsearch document structure.

In 5.6, we still default to using multiple types, but fall back to a single type when we identify the data has been migrated. Here are a few examples of how we built a fallback system.

Create

When creating a document we will receive a type_missing_exception if the type does not exist. We receive this error after the data has been migrated, allowing us to re-try with the single-type format.

POST /.kibana/doc/index-pattern:abc123/_create
{
   "type":"index-pattern",
   "index-pattern":{
      "title":"Test pattern"
   }
}

Get

Instead of making two requests, we can perform a single search to capture the document. Here we use a boolean query containing both of the formats.

POST /.kibana/_search
{
  "query":{
    "bool":{
      "should":[{
        "bool":{
          "must":[{
            "term":{
              "_id":"abc123"
            }
          }, {
            "term":{
              "_type":"index-pattern"
            }
          }]
        }
      }, {
        "bool":{
          "must":[{
            "term":{
              "_id":"index-pattern:abc123"
            }
          }, {
            "term":{
              "type":"index-pattern"
            }
          }]
        }
      }]
    }
  }
}

Delete

When deleting a document, we use _delete_by_query, providing the same query used for fetching a document.

Update

When using the _update API, we receive a document_missing_exception if the document is missing. We receive this after the data has been migrated, allowing us to re-try with the single-type format.

POST /.kibana/doc/index-pattern:abc123/_update
{
   "index-pattern":{
      "title":"My new title"
   }
}

These changes allowed us to migrate the Kibana index at any time during a 5.6 installation. In X-Pack, we have made this process even easier for users through our Upgrade Assistant. If you have indices created in 2.x, you must reindex them before upgrading to Elasticsearch 6.0. In addition, the internal Kibana and X-Pack indices must be reindexed to upgrade them to the format required in 6.0. The Reindex Helper identifies these indices and provides a button to perform the reindex.

Have a special use case or strategy for how to migrate your multiple type indices to single type? Share it with us! You can also ask us for advice on our forum.