Welcome to This Week in Elasticsearch and Apache Lucene! With this weekly series, we're bringing you an update on all things Elasticsearch and Apache Lucene at Elastic, including the latest on commits, releases and other learning resources.
Configurable translog retention for fast ops-based shard recovery
Up to 6.0 the primary purpose of the translog was to persist all operations that hadn't yet been persisted to Lucene. Once Lucene files were committed (i.e., fsync) we would clear the translog and start writing to it again (a.k.a
_flush). While this behaviour is optimal from a storage perspective, it is not ideal for the new ops-based fast recovery introduced in 6.0. In order to bring a replica up to speed without copying segments, the primary needs to have all the required operations available in its translog. That meant that fast ops recoveries weren't possible during full restarts as we flush when we recover the primary, or if a node left the cluster long enough for the primary to flush while it was away. This now changes with the introduction of a translog retention policy - we will now keep translog files (up 512MB and for at least 12 hours by default) even if the data they contain is already committed to lucene. This is also very important for the future xDCR feature where we need the ops to be streamed off cluster.
Augmenting Painless functionality
Scripts in Elasticsearch are now run in a particular "context", which defines the return value type and the variables and functions that are enabled in a particular script, e.g. an update script should have access to
_source, while a fast search script shouldn't. Now, other classes can extend the functionality available to Painless scripts, instead of them being restricted to just a hardcoded list of functions. This will allow Painless scripts to expose utility functions which are only useful in certain contexts.
Removal of default passwords
We will no longer be using the default
changeme password for internal users in Elasticsearch 6.0 with security. Instead, a cluster in production mode will start with no passwords, but will only accept requests to change the password for the
elastic user from localhost. This can be done via the API, but we provide a command line tool which will generate strong passwords for all internal users and output them to the console. Alternatively, the user can specify their own passwords using the same tool. For Docker, where this CLI tool is not accessible, we allow a temporary password to be set via an environment variable, which will cease to work as soon as the user changes the password via the API.
Longer user and role names Some users have long requested longer user and role names in security, which were previously limited to just 30 characters. The new limits are as follows:
A valid username's length must be at least 1 and no more than 1024 characters. It may not contain leading or trailing whitespace. All characters in the name must be be alphanumeric (a-z, A-Z, 0-9), printable punctuation or symbols in the Basic Latin (ASCII) block, or the space character.
Changes in 5.5:
index.internal.versionsetting will be used by internal indices to track the version of the index definition.
- The MD5 checksum in the S3 repository plugin was broken and redundant, and so has been removed.
- The secure settings keystore CLI was adding files as strings instead of as files.
- Cross-cluster search should not fall back to a local search if a remote cluster index pattern resolves to no indices.
- Snapshot/restore verbose info output should return all fields, even if empty.
- Settings deprecation logging should only be emitted the first time the setting is used, and a new LRU cache reduces the frequency of deprecation logs.
WrapperQueryBuildershould rewrite wrapped queries.
Changes in 5.x:
_cat/nodesnow returns info about disk usage.
TemplateUpgraderscan now be called during a rolling restart, in addition to full cluster restart.
- When promoting a replica to be a primary, the replica on the highest node version should be preferred.
- Secure settings should not be serialised.
- On Windows, use the short path name for starting/stopping native controllers like ML in case the path is too long.
- A cluster allocation explain request should return a 400 error when no shards are unallocated, instead of a 500 error.
Changes in master:
- [BREAKING] Replaced the
path.confsetting with a
--path.confcommand line flag so that it cannot be set in
- [BREAKING] Plugins should have an explicit constructor instead of relying on reflection to find one.
- [BREAKING] Removed the
default.path.logssettings in favour of adding these settings to the packaged
- [BREAKING] Index expressions may no longer user a
+prefix to indicate inclusion, as this is redundant.
SearchHitFieldreturn the same information and so have been unified into
- [BREAKING] 32 bit architectures are no longer supported.
- Painless scripts can have new functionality added by other classes.
- Synonyms are now parsed with the same analysis chain as other terms, so that (eg) lowercasing and edge-ngram filters will be applied automatically.
- Sequence IDs:
- When using sequence IDs for recovery, a primary shard can now stream all changes since the global checkpoint to replica shards to ensure that the replica includes all changes from the primary. (Replicas may still include unacknowledged changes not present on the primary)
- When a primary shard is relocated to another node, the old primary now passes information about the shards that it knows about to the new primary, to ensure that the global checkpoint calculation remains the same.
- When a replica sees a new primary term, it should also update its global checkpoint.
- Multiple concurrent recoveries of the same target shard (which can happen due to network disconnects) should not be allowed as it makes global checkpoint tracking more complicated.
- Shrunk indices should use a higher primary term and max unsafe auto ID than that used by the source index.
- Cleaned up network/transport settings which fallback to
network.*to be explicit and correct.
document_typeparameter to the
percolatequery is deprecated as new indices in 6.0 can only have a single type.
We're moving towards getting Lucene 7 released. The 7.0 and 7.x branches are expected to be cut very soon. A change in the version string format of Java 9 affects Hadoop, which Solr relies on, and we are looking at the best way to get things fixed since it will be important for Lucene 7 to support Java 9.
Want to watch Lucene-related content? There were 3 Lucene-related talks at Berlin Buzzwords this year:
- "What is coming to Lucene 7" by Uwe Schindler (https://www.youtube.com/watch?v=zigqdXhxLK0)
- "How does a Lucene query actually work" by Alan Woodward (https://www.youtube.com/watch?v=Z-yG-KvIuD8)
- "Running slow queries with Lucene" by me (https://www.youtube.com/watch?v=p51vIDWHWqk)
Mike has been exploring using multiple threads to resolve doc ids for the deleted terms, which results in a 50% update throughput increase in some cases.
- We are slowly moving code to the new ValuesSource API.
- Should we use doc values rather than postings in order to encode block joins?
- There is disagreement about whether the maximum number of boolean clauses should be configurable per query.
- Multi-term synonyms would parse to the wrong query when the entire query string matches a synonym.
- DisjunctionMaxQuery could rewrite to a BooleanQuery when the tiebreaker is 1, allowing it to use BooleanQuery's optimized bulk scorer.
- TopDocs.totalHits should be a long if we don't want it to overflow when merging results from multiple shards that have more than 2B matches in total.
- Removing the Fields abstraction makes things simpler.
- The Token class has been deprecated since Lucene 2.9, we should look into finally removing it.
- Can we save disk space when multiple terms have the same postings list?
Watch This Space
Stay tuned to this blog, where we'll share more news on the whole Elastic ecosystem including news, learning resources and cool use cases!