Rollup jobsedit

A rollup job is a periodic task that aggregates data from indices specified by an index pattern and rolls it into a new index. Rollup indices are a good way to compactly store months or years of historical data for use in visualizations and reports.

You’ll find Rollup Jobs under Management > Elasticsearch. With this UI, you can:

][List of currently active rollup jobs

Before using this feature, you should be familiar with how rollups work. Rolling up historical data is a good source for more detailed information.

Create a rollup jobedit

Kibana makes it easy for you to create a rollup job by walking you through the process. You fill in the name, data flow, and how often you want to roll up the data. Then you define a date histogram aggregation for the rollup job and optionally terms, histogram, and metrics aggregations.

When defining the index pattern, you must enter a name that is different than the output rollup index. Otherwise, the job will attempt to capture the data in the rollup index. For example, if your index pattern is metricbeat-*, you can name your rollup index rollup-metricbeat, but not metricbeat-rollup.

][Wizard that walks you through creation of a rollup job

Start, stop, and delete rollup jobsedit

Once you’ve saved a rollup job, you’ll see it the Rollup Jobs overview page, where you can drill down for further investigation. The Manage menu in the lower right enables you to start, stop, and delete the rollup job. You must first stop a rollup job before deleting it.

][Rollup job details

You can’t change a rollup job after you’ve created it. To select additional fields or redefine terms, you must delete the existing job, and then create a new one with the updated specifications. Be sure to use a different name for the new rollup job—reusing the same name can lead to problems with mismatched job configurations. You can read more at rollup job configuration.

Try it: Create and visualize rolled up dataedit

This example creates a rollup job to capture log data from sample web logs. To follow along, add the sample web logs data set.

In this example, you want data that is older than 7 days in the target index pattern kibana_sample_data_logs to roll up once a day into the index rollup_logstash. You’ll bucket the rolled up data on an hourly basis, using 60m for the time bucket configuration. This allows for more granular queries, such as 2h and 12h.

Create the rollup jobedit

As you walk through the Create rollup job UI, enter the data shown in the table below. The terms, histogram, and metrics fields reflect the key information to retain in the rolled up data: where visitors are from (geo.src), what operating system they are using (machine.os.keyword), and how much data is being sent (bytes).

FieldValue

Name

logs_job

Index pattern

kibana_sample_data_logs

Rollup index name

rollup_logstash

Frequency

Every day at midnight

Page size

1000

Delay (latency buffer)

7d

Date field

@timestamp

Time bucket size

60m

Time zone

UTC

Terms

geo.src, machine.os.keyword

Histogram

bytes, memory

Histogram interval

1000

Metrics

bytes (average)

You can now use the rolled up data for analysis at a fraction of the storage cost of the original index. The original data can live side by side with the new rollup index, or you can remove or archive it using Index Lifecycle Management.

Visualize the rolled up dataedit

Your next step is to visualize your rolled up data in a vertical bar chart. Most visualizations support rolled up data, with the exception of Timelion, TSVB, and Vega visualizations.

Using the information from the example rollup configuration described above, you can use rollup_logstash to match the rolled up index pattern, and kibana_sample_data_logs to match the index pattern for raw data. The notation for a combination index pattern with both raw and rolled up data is rollup_logstash,kibana_sample_data_logs.

][Visualization of rolled up data

You can then create a dashboard that contains visualizations of the rolled up data, raw data, or both. See Using rolled up data in a visualization for more information.

][Dashboard with rolled up data