Optimize is a bit of a misnomer. It is in actuality a Lucene forceMerge operation. With time-series data in a per-day index, Lucene does a good job of keeping the number of segments low. However, if no new data is being ingested, no further segment merging will happen. There are some minor performance benefits from merging segments down to a smaller count, but a greater benefit when it comes to restarts [e.g. version upgrades, etc.] after a shutdown: with fewer segments to have to validate, the cluster comes back up sooner.
Curator counts the number of segments per shard before performing an optimize call. If the count is at or below the threshold (default is 2 segments per shard) it will skip optimizing that index. In this manner, indices are not optimized repeatedly.
Extra disk space requirededit
Performing this forceMerge requires a lot of extra disk space: 3x what the original segments consume. Fortunately, this extra space is only needed during the forceMerge operation. Be sure to keep enough disk space available so that this does not become an issue for you.
A word about timeoutsedit
With some commands (e.g.
optimize) the default behavior is to wait until the
operation is complete before proceeding with the next step. Since these
operations can take quite a long time it is advisable to set
to a high value. If one is not set, a default of 6 hours (21600 seconds) will be
$ curator optimize --help Usage: curator optimize [OPTIONS] COMMAND [ARGS]... Optimize Indices Options: --delay INTEGER Pause *n* seconds after optimizing an index. [default: 0] --max_num_segments INTEGER Merge to this number of segments per shard. [default: 2] --request_timeout INTEGER Allow this many seconds before the transaction times out. [default: 21600] --help Show this message and exit. Commands: indices Index selection.
Optimize matching indices to 2 segments per shard (the default):
curator optimize indices <<index selection parameters>>
Optimize matching indices older than 2 days to 1 segment per shard, and delay 120 seconds between indices:
curator optimize --max_num_segments 1 --delay 120 indices <<index selection parameters>>