This week in Elasticsearch and Apache Lucene: Elasticsearch Core and Lucene tips
Welcome to This Week in Elasticsearch and Apache Lucene! With this weekly series, we're bringing you an update on all things Elasticsearch and Apache Lucene at Elastic, including the latest on commits, releases and other learning resources.
Top News
Elasticsearch Core
- Internal: Refactor SSD/FileStore logic out of NodeEnvironment (#10755, 2.0.0)
- Snapshot/Restore: remove obsolete
expand_wildcards_open
and <code>expand_wildcards_close options (#10744, 2.0.0) - Bulk:
_default_
mapping should be picked up from index template during auto create index (#10762, 2.0.0, 1.6.0, 1.5.2, 1.4.5) - Internal: Refactor
TransportShardReplicationOperationAction
(#10749, 2.0.0, 1.6.0) - Core: Simplify IndexStore and friends (#10773, 2.0.0)
- Startup: Use Tuple only as return value in
Bootstrap
(#10784, 2.0.0) - Mappings: Join
MergeResults
with <code>MergeContext since they are almost the same (#10765, 2.0.0) - Build: Disable security manager when running with
mvn exec:exec
(#10792, 2.0.0) - Stats:
CommitStats
doesn't need to allow for null values in commit user data (#10774, 2.0.0, 1.6.0) - Internal: change
BigArrays
to not extend <code>AbstractComponent (#10798, 2.0.0, 1.6.0) - HTTP: Ensure url path expansion only works inside of plugins (#10815, 2.0.0, 1.6.0, 1.5.2, 1.4.5)
- Refactoring: Reduce code duplication in
TransportIndexAction
/<code>TransportShardBulkAction (#10819, 2.0.0) - Query enhancement: Enable Lucene ranking behaviour for numeric queries (#10790, 2.0.0)
- Build: Fix to let eclipse ignore the antigun plugin in it's maven integration (#10822, 2.0.0, 1.6.0, 1.5.3)
- Shutdown API: Remove
_shutdown
API (#10831, 2.0.0) - Circuit Breaker: Don't create a new
BigArrays
instance for every call of <code>withCircuitBreaking (#10800, 2.0.0, 1.6.0) - Backcompat: Add test for missing filter (#10842, 2.0.0, 1.6.0, 1.5.3)
- Internal: Remove deprecated
metadata
cluster block level (#10779, 2.0.0) - Analysis: Fix wrong position number by analyze API (#10771, 2.0.0)
- Tests: Add backward compatibility test for cluster state with blocks (#10754, 2.0.0, 1.6.0)
- Security manager cleanups (#10844, 2.0.0)
- Mappings: Remove dead code after previous refactorings (#10666, 2.0.0)
- Internal: Ensure that explanation descriptions are not null on serialization (#10689, 2.0.0, 1.6.0)
- Matched queries: Remove redundant and broken code (#10694, 2.0.0)
- Dependencies: Upgrade to Lucene 5.2 r1675100 (#10699, 2.0.0)
- Geo: Update
tree_level
and precision parameter priorities (#10679, 1.5.2) - Mappings: Simplified mapper lookups (#10705, 2.0.0)
- Stats: Add
CommitStats
to supply information about the current commit point on shard level (#10687, 2.0.0, 1.6.0) - FileStore: Make
getFileStore
a bit more defensive (#10696, 2.0.0, 1.6.0) - Dependencies: Upgrade to lucene-5.2-snapshot-1675363. (#10727, 2.0.0)
- Rest: Add more utilities for source/body handling in RestAction (#10724, 2.0.0)
- Engine: Implement retries for
ShadowEngine
creation (#10688, 2.0.0, 1.6.0, 1.5.2) - Mappings: simplify dynamic mappings updates (#10720, 2.0.0)
- Mappings: Restrict murmur3 field type to sane options (#10738, 2.0.0)
- Refacotoring: refactor and cleanup transport request handling (#10730, 2.0.0)
- Startup: Enable securitymanager (#10717, 2.0.0)
- Remove working directory (#10672, 2.0.0)
- Settings: validate number_of_shards/_replicas without index setting prefix (#10701, 2.0.0, 1.6.0)
- Snapshot/Restore: Improve the error message when attempting to snapshot a closed index (#10608, 2.0.0, 1.6.0)
- Mappings: Validate dynamic mappings updates on the master node (#10634, 2.0.0)
- Search: Replace deprecated filters with equivalent queries (#10531, 2.0.0)
- Geo: Fix OOM for high precision exotic shapes (#10652, 1.6.0)
- Geo: Make
GeoContext
mapping idempotent (#10602, 2.0.0, 1.6.0, 1.5.2) - Query Template: Fix updating templates (#10526, 2.0.0, 1.6.0, 1.5.2)
- Queries: Single value numeric queries shouldn't be handled by
NumericRangeQuery
(#10648, 2.0.0, 1.6.0) - Core: Allow rebalancing primary shards on shared filesystems (#10585, 1.6.0, 1.5.2)
- Scripting: Added a new script construct (#10649, 2.0.0)
- Stats: Add field stats api (#10523, 2.0.0, 1.6.0)
- REST: Render REST errors in a structural way (#10643, 2.0.0)
- Build: Update package repositories when creating a release (#10507, 2.0.0, 1.5.3, 1.4.5)
- TransportClient: Ensure netty I/O thread is not blocked (#10644, 2.0.0, 1.6.0, 1.5.3)
- Aggregations: New
sampler
aggregation (#10221, 2.0.0) - Geo Coordinates can contain more than two elements (x,y) in
GeoJSON
parser (#9542, 2.0.0, 1.6.0) - Rest: Add JSON in request body to scroll, clear scroll, and analyze API (#9076, 2.0.0)
- Snapshot/Restore: Expand wildcards in snapshot (#9903, 2.0.0, 1.6.0)
- Internal: Add
METADATA_READ
and <code>METADATA_WRITE blocks (#9203, 2.0.0, 1.6.0) - Cluster state: Add support for cluster state diffs (#10212, 2.0.0)
- Java API: Added
toString
methods for <code>SearchRequestBuilder andCountRequestBuilder
(#9944, 2.0.0, 1.6.0, 1.5.3)
Apache Lucene
- Now you can call non-static zero-argument methods from Lucene's expressions, e.g. date.getMonth()
- Span queries get better two-phased iterator support
PackedQuadPrefixTree
gives more compact encoding for geo quad-tree terms, and uses less transient memory during indexing - Highlighter can now enforce a token offset limit when pulling tokens from term vectors
- Explanation API is simpler
- Make sure filters implement hashCode and equals
ShuffleFS
, randomly swapped into Lucene's tests, changes the order of directory listings- The "who tests the tester" test cases for Lucene's test-framework finally move to the right place in the source tree
- Upgrade to version 1.8 of forbidden APIs.
- Show only changed settings when tests randomly tweak
IndexWriterConfig
, and other verbose tests improvements - Some slow tests get faster when not running nightly
- Block joins can now carry the minimum child score up as the parent's score
- Tests can avoid fsync using the new
SuppressFsync
annotation ConcurrentMergeScheduler
's dynamic defaults are now repeatable during tests based on the test seed- Add
GeoPointField
and shape queries for simple common-case geo searching - Detect invalid ghost bits in Lucene's bit sets
- Tests should not depend on the full paths of their temporary directories
How to sync transformed data from #MongoDB to #Elasticsearch using Transporter http://t.co/eSv7HWjbtN @composeio pic.twitter.com/SftVAZspVG
— DigitalOcean (@digitalocean) April 26, 2015
Watch This Space
Stay tuned to this blog, where we'll share more news on the whole ELK ecosystem including news, learning resources and cool use cases!