Search & Analyze in Real Time

Elasticsearch is a distributed, open source search and analytics engine, designed for horizontal scalability, reliability, and easy management.

Elasticsearch 1.5.0

Elasticsearch can also be installed from our repositories using apt or yum. See Repositories in the Guide.

Installation Steps

  • Download and unzip the latest Elasticsearch distribution

  • Run bin/elasticsearch on Unix or bin/elasticsearch.bat on Windows

  • Run curl -X GET http://localhost:9200/
see release notes

Breaking changes

  • Aliases:
    • Aliases: Throw exception if index is null or missing when creating an alias #8240 (issues: #7863, #7976)
  • Benchmark:
    • Benchmark api: removed leftovers #10180
  • Engine:
    • [ENGINE] Remove full flush / FlushType.NEW_WRITER #9559
  • Index APIs:
    • Change behaviour of indices segments api to allow no indices #9219 (issue: #5856)
  • Plugins:
    • Plugins: Don't overwrite plugin configuration when removing/upgrading plugins #7890 (issue: #5064)
  • Query DSL:
    • [QUERY] Remove lowercase_expanded_terms and locale options #10086 (issues: #9973, #9978)
  • Recovery:
    • Recovery: RecoveryState clean up #9811 (issue: #6644)
  • Scripting:
    • Scripting: cleanup ScriptService & friends in preparation for #6418 #9992 (issue: #6418)
    • Disable dynamic Groovy scripting by marking Groovy as not sandboxed [ISSUE] #9655
    • Scripting: Script with _score: remove dependency of DocLookup and scorer #7819 (issues: #7043, #7487)

Deprecations

  • CRUD:
  • Core:
    • Core: deprecate index.fail_on_merge_failure #10084
  • Mapping:
    • Mappings: Deprecate _analyzer and per query analyzers #9383 (issue: #9279)
  • More Like This:
    • Deprecation: MLT Field Query #8253
  • Network:
    • Deprecated the thrift and memcached transports #10167 (issue: #10166)
  • Search:
    • Core: add deprecation messages for delete-by-query #10082

New features

  • Aggregations:
    • New aggregations feature - “PercentageScore" heuristic for significant_terms #9747 (issue: #9720)
    • significant terms: add scriptable significance heuristic #7850
  • CAT API:
    • Cat API: show open and closed indices in _cat/indices #7936 (issue: #7907)
  • Circuit Breakers:
    • Circuit Breakers: Add NoopCircuitBreaker used in NoneCircuitBreakerService #8063
  • Engine:
    • Shadow replicas on shared filesystems #9727 (issue: #8976)
  • More Like This:
    • MLT Query: Support for artificial documents #7725
  • Query DSL:
    • Add time_zone setting for query_string #8164 (issue: #7880)
    • Search: add format support for date range filter and queries #7821 (issue: #7189)
    • Add min_score parameter to function score query to only match docs above this threshold #7814 (issue: #6952)
  • Search:
    • Add inner hits to nested and parent/child queries #8153 (issues: #3022, #3152)
  • Store:
  • Term Vectors:
    • Term Vectors/MLT Query: support for different analyzers than default at the field #7801

Enhancements

  • Aggregations:
    • Aggregations: deprecate pre_zone and post_zone in date_histogram #9722 (issue: #9062)
    • Aggregations: Add 'offset' option to date_histogram #9614 (issue: #9062)
    • Aggregations: Numeric metric aggregations are now formattable #9032 (issue: #6812)
    • Aggregations: Adds methods to get to/from as Strings for Range Aggs #9026 (issue: #9003)
    • Made the nested, reverse_nested and children aggs ignore unmapped nested fields or unmapped child / parent types. #8808 (issue: #8760)
    • Fixed #6061 size parsing consistent for strings #8645
    • Aggregations: Do not take deleted documents into account in aggregations filters. #8540
    • Aggs - support for arrays of numeric values in include/exclude clauses #7727 (issue: #7714)
    • Aggregations: Adds ability to sort on multiple criteria #7662 (issues: #6917, #7588)
    • Aggregations: Properly support top_hits aggregation in a nested and reverse_nested aggregations. #7164 (issue: #3022)
  • Allocation:
    • Allocation: Speed-up disk-threshold decider #8803 (issue: #6372)
    • Allocation: DiskThresholdDecider#remain(...) should take shards relocating away into account #8659 (issue: #8538)
    • Allocation: Take percentage watermarks into account for reroute listener #8382 (issues: #8367, #8368)
    • Allocation: Reroute shards automatically when high disk watermark is exceeded #8270 (issue: #8146)
    • Add option to take currently relocating shards' sizes into account #7785 (issues: #6168, #7753)
  • CAT API:
    • Mark shadow replicas with 's' in _cat/shards output #10023 (issue: #9772)
    • Admin: add total index memory in _cat/indices #7824 (issue: #7008)
    • Add file descriptor details to cat/nodes #7655 (issue: #7652)
  • Circuit Breakers:
    • Circuit Breakers: Log if CircuitBreaker is tripping #8050
  • Core:
    • Core: don't rethrow already handled merge exceptions #10083
    • [ENV] NodeEnv should lock all shards for an index #9799
    • [CORE] Allow primary promotion on shadow replica without failing the shard #9786
    • [INDICES] Retry if shard deletes fail due to IOExceptions #9784
    • Core: only do a single listAll from FileSwitchDir #9666 (issue: #6636)
    • Core: Remove ability to run optimize and upgrade async #9640 (issue: #9638)
    • Update joda-time to v2.7 #9610
    • [CORE] Consolidate index / shard deletion in IndicesService #9605
    • Minor hygiene, Removed Redundant inheritance #9427
    • Core: don't throttle recovery indexing operations #9396 (issue: #9394)
    • Core: increase default xlog flush size from 200mb to 512 mb #9341 (issue: #9265)
    • Pass through all exceptions in IndicesLifecycleListeners #9330
    • Pass index settings to IndicesLifecycle#beforeIndexCreated and #afterIndexShardClosed #9245
    • Core: Fix Store.checkIntegrity() for lucene 3.x index files. #9234
    • Core: Don't verify adler32 for lucene 3.x terms dict/terms index. #9142
    • [CORE] Delete shard content under lock #9083 (issues: #8608, #9009)
    • [SEARCH] close active contexts on SearchService#close() #8947 (issue: #8940)
    • Core: use Lucene's defaults for compound file format #8934 (issue: #8919)
    • [CORE] Remove explicit .cleanUp() on cache clear #8924
    • Core: clarify index removal log message #8641
    • [CORE] Ensure shards are deleted under lock on close #8579
    • [INDEX] Add before/after indexDeleted callbacks to IndicesLifecycle #8569 (issue: #8551)
    • [CORE] Free pending search contexts if index is closed #8551
    • [CORE] Ban all usage of Future#cancel(true) #8494
    • Core: set bloom default to false even when Directory doesn't have a codecService #8442
    • [CORE] Introduce shard level locks to prevent concurrent shard modifications #8436
    • Discovery: Don't wait joinThread when stopping #8359 (issue: #8327)
    • [STATE] Observe cluster state on health request #8350
    • [UTILITIES] Introduce a RefCounted interface and basic impl #8210
    • Use 1 instead of 0 as filler version value for nested docs #8145
    • [CORE] Add cluster and index state checksums #8010 (issue: #7586)
    • Resiliency: Perform write consistency check just before writing on the primary shard #7873
    • [CORE] Add ThreadPool.terminate to streamline shutdown #7868
    • [CORE] Add ActionRunnable support to ThreadPool to simplify async operation on bounded threadpools #7765
    • Internal: Add missing cluster blocks handling for master operations #7763 (issue: #7740)
  • Discovery:
    • [Discovery] Prevent stale master nodes from sharing dated cluster states to nodes that have moved to a different master node #9632
    • Discovery: publishing timeout to log at WARN and indicate pending nodes #9551
    • Discovery: Concurrent node failures can cause unneeded cluster state publishing #8933 (issue: #8804)
    • [CLIENT] Only fetch the node info during node sampling #8685
    • Discovery: don't accept a dynamic update to min_master_nodes which s larger then current master node count #8321
    • Discovery: Improve handling of multicast binding exceptions #8243 (issue: #8225)
    • Discovery: Simplify discovery node initialization if version is unknown #8055 (issue: #8051)
    • Discovery: Close ping handler's executor service properly #7903
  • Engine:
    • [CORE] move InternalEngine.segmentStats() into abstract Engine #9728 (issue: #9727)
    • [ENGINE] Move more methods into abstract Engine #9717
    • [CORE] Move as much as possible into abstract Engine #9678
    • [ENGINE] Factor out settings updates from Engine #9625
    • [ENGINE] Close Engine immediately if a tragic event strikes. #9616 (issue: #9517)
    • [CORE] Refactor InternalEngine into abstract Engine and classes #9585
    • [ENGINE] Remove FlushType and make resources final in InternalEngine #9565
    • [ENGINE] Simplify Engine construction and ref counting #9211
    • [CORE] Fold engine into IndexShard #9181
    • Core: Don't acquire Engine's readLock in segmentsStats #8910 (issue: #8905)
    • [ENGINE] Remove engine related command classes #8900
    • Internal: allow InternalEngine to be stopped and started #8784 (issue: #8720)
    • [ENGINE] Flush IndexWriter to disk on close and shutdown #7563
  • Geo:
    • Revert "[GEO] Update GeoPolygonFilter to handle ambiguous polygons" #9463 (issues: #5968, #9304, #9339, #9462)
    • Geo: Update GeoPolygonFilter to handle polygons crossing the dateline #9339 (issues: #5968, #8672, #9304)
    • Geo: GeoPolygonFilter not properly handling dateline and pole crossing #9171 (issue: #5968)
    • Geo: Removing unnecessary orientation enumerators #9036 (issues: #8978, #9035)
    • Geo: Add optional left/right parameter to GeoJSON #8978 (issue: #8764)
    • Geo: Feature/Fix for OGC compliant polygons failing with ambiguity #8762 (issue: #8672)
    • Geo: Fixes BoundingBox across complete longitudinal range #7340 (issue: #5218)
  • Index Templates:
    • Provide template usage information on index creation #8646 (issue: #7421)
  • Internal:
    • Core: fix typo when primary is not available to index a document (UnavailableShardsException) #10140
    • [SHARD] make assert less strict to ensure local node is not null #10076
    • Use provided cluster state for indices service validations #10014
    • some more simple fs cleanups. #9827
    • Removed 'Master' from names.txt #9698 (issue: #9687)
    • Internal: Introduce TimedPrioritizedRunnable base class to all commands that go into InternalClusterService.updateTasksExecutor #9671 (issues: #8077, #9354)
    • Update Groovy dependency to 2.4.0 #9520
    • Add beforeIndexAddedToCluster callback #9514
    • [CACHE] Use a smaller expected size when serializing query results #9485
    • [CACHE] Use correct number of bytes in query cache accounting #9479
    • Internal: add AliasesRequest interface to mark requests that manage aliases #9460
    • Internal: ClusterInfoService should wipe local cache upon unknown exceptions #9449
    • Internal: fix shard state tranport action names #9440 (issue: #7105)
    • Mappings: Remove includeExisting flag from adding ObjectMapper and FieldMapper listeners #9184
    • Internal: assert that we do not call blocking code from transport threads #9164
    • [EXEC] Remove reduced stack size #9158 (issue: #9135)
    • Search: parse terms filters on a single term as a term filter. #9014
    • Internal: remove IndexCloseListener & Store.OnCloseListener #9009 (issues: #8436, #8608)
    • Core: add ignore_idle_threads (default: true) to hot threads #8985 (issue: #8908)
    • Remove circular dependency between IndicesService and IndicesStore #8918
    • Remove some Internal* abstractions #8904
    • [HEALTH] Don't mark health as timed out if desired state is reached #8683
    • Internal: Use a 1024 byte minimum weight for filter cache entries #8304 (issues: #8249, #8268)
    • Immediately remove filter cache entries on cache clear #8289 (issue: #8285)
    • Internal: Make indexQueryParserService available from ParseContext #8252 (issue: #8248)
    • Internal: Allow to configure custom thread pools #8247
    • Internal: Expose concurrency_level setting on all caches #8112 (issue: #7836)
    • Resiliency: Be more conservative if index.version.created is not set #8018
    • Internal: split internal fetch request used within scroll and search #7870 (issues: #6933, #7319, #7856)
    • Internal: split internal free context request used after scroll and search #7856 (issues: #6933, #7319)
    • Internal: Clarify when a shard search request gets created to be only used locally #7855
    • Internal: Add a listener thread pool #7837
    • Internal: remove ForceSyncDirectory #7804
    • Internal: Check if from + size don't cause overflow and fail with a better error #7778 (issue: #7774)
    • Internal: make sure that internally generated percolate request re-uses the original headers and request context #7767
    • Internal: make sure that update internal requests share the same original headers and request context #7766
    • Internal: Nest original exception while creating NoShardAvailableActionException #7757 (issue: #7756)
    • Internal: make sure that all delete mapping internal requests share the same original headers and context #7736
    • Internal: make sure that original headers are used when executing search as part of put warmer #7711
  • Java API:
    • Java API: package private getters to become public if there have corresponding public setters #9273
    • java: QueryBuilders cleanup (add and deprecate) #8667
    • Added utility method #8594
    • Java API: Enabled overriding the request headers in the clients #8258
    • Internal: add indices setter to IndicesRequest interface #7734
  • Logging:
    • Store: renaming temp files should log errors of delete in trace #9933
    • Fix logging a RoutingNode object, log an object with a good .toString instead #9863
    • Logging: improve logging messages added in #9562 #9603
    • Logging: add logging around gateway shard allocation #9562
    • Logging: Log byte and doc size for slow merges #8855 (issue: #8853)
    • Core: separately log file deletions #8662 (issue: #8603)
    • Logging: Add log4j-extras dependency #8464 (issue: #7927)
    • Core: log how long IW.rollback took, and when MockFSDir starts its check index #8388
    • Logs: Change log level for mpercolate #8306
    • Logging: suppress long mapping logging during mapping updates (unless in TRACE) #7949
    • Add log4j-extras dependency #7947 (issue: #7927)
    • Core: Add EnhancedPatternLayout to logging.yml options #4991
  • Mapping:
    • Mappings: Add enabled flag for _field_names to replace disabling through index=no #9893
    • [Mapper] Add ignore_missing option to timestamp #9104 (issues: #8882, #9049)
    • Mapping: Include currentFieldName into ObjectMapper errors #9020
    • Mappings: Make lookup structures immutable. #7486
  • More Like This:
    • MLT Query: Support for when all fields are deprecated #8067
    • MLT Query: use minimum should match more extensive syntax #7898
  • NOT CLASSIFIED:
    • [LIFECYCLE] Add before/afterIndexShardDelete callback #10173
  • Network:
    • Netty HTTP Transport: Change access modifiers to protected #9724
    • Transport: added a simple request tracer, logging incoming and outgoing requests #9286
    • Netty Transport: Add profiles to transport infos #9134
    • Netty: Support binding on multiple host/port pair #8098
    • Internal: Chunk direct buffer usage by networking layer #7811
    • Netty: Make sure channel closing never happens on i/o thread #7726
  • Packaging:
    • Windows: makes elasticsearch.bat more friendly to automated processes #9160 (issue: #8913)
    • Shutdown: Add support for Ctrl-Close event on Windows platforms to grace... #8993
    • Packaging: Add java7/8 java-package paths to init script #8815 (issue: #7383)
    • Packaging: Check if proc file exists before calling sysctl #8793 (issue: #4978)
    • deb: add systemd service config for upcoming Jessie #8765 (issue: #8493)
    • bin/elasticsearch: add help, fix endless loop #8729 (issues: #2168, #7104)
    • Packaging: Introduce elasticsearch.in.bat (i.e. es.in for Windows) #8244 (issue: #8237)
    • add more checks to build_release.py #7913
    • Prevent init script from returning when the service isn't actually started #6909
    • Boostrap: Log startup exception to console if needed and to file as error #6581
    • Packaging: Use the new command line syntax in the init script #5033
    • Startup: Add ES_HOME to ES_INCLUDE search path #4958
  • Parent/Child:
    • Parent/child: Reduce memory usage in top children query #8165
  • Percolator:
    • Introduce index option named 'index.percolator.map_unmapped_fields_as_string' #9054 (issues: #9025, #9053)
  • Plugins:
    • [PLUGINS] Always send current ES version when downloading plugins #10131
  • Query DSL:
    • Add support for minimum_should_match to simple_query_string #9864 (issue: #6449)
    • inner hits: Nested parent field should be resolved based on the parent inner hit definition #9251 (issue: #8153)
    • Raise an exception on an array of values being sent as the factor for a ... #9246 (issue: #7408)
    • inner_hits: Added another more compact syntax for inner hits. #8770
    • function_score: use query and filter together #8675 (issue: #8638)
    • Query: add option for analyzing wildcard/prefix to simple_query_strinq #8422 (issue: #787)
    • Query DSL: Expose max_determinized_states in regexp query, filter #8384 (issue: #8357)
    • FunctionScore: RandomScoreFunction now accepts long, as well a strings. #8311 (issue: #8267)
    • Be stricter parsing ids for ids query #7945 (issue: #7686)
    • Adding "min" score mode to parent-child queries #7771 (issue: #7603)
    • Query DSL: Throw parsing exception if terms filter or query has more than one field #5137 (issue: #5014)
    • Query DSL: Expose dist/pre/post options for SpanNotQuery #4452
  • REST:
    • [HTTP] add option to only return simple exception messages #10117
    • Rest: expose master_timeout flag on GET _template & HEAD _template #9688
    • Rest: Adding support of multi-index query parameters for _cluster/state #9295 (issue: #5229)
    • Adds parameters to API endpoint cluster put settings specification #8769
    • Change IndexPrimaryShardNotAllocatedException from 409 to 500 #7987 (issue: #7632)
    • Percolator: Added missing percolate API parameters to the rest spec #7173
  • Recovery:
    • Recovery: add throttle stats #10097
    • Engine: update index buffer size during recovery and allow configuring version map size #10046 (issues: #6363, #6667)
    • Recovery: unify RecoveryState management to IndexShard and clean up semantics #9902 (issue: #9503)
    • [RECOVERY] Only iterate the files that we recovered from the commit #9761
    • Recovery: add a timeout to local mapping change check #9575
    • Recovery: flush immediately after a remote recovery finishes (unless there are ongoing ones) #9439
    • [RECOVERY] Release store lock before blocking on mapping updates #9102
    • [RECOVERY] Ensure shards are identical after recovery #8723
    • Recovery: be more resilient to partial network partitions #8720
    • [RECOVERY] Throw IndexShardClosedException if shard is closed #8648
    • Backport ShardRecoveryHandlerrefactorings to 1.x #8570
    • Recovery: refactor RecoveryTarget state management #8092 (issues: #7315, #7893)
  • Scripting:
    • Scripting: Add String to the default whitelisted receivers #9837 (issue: #8866)
    • Scripting: Make script.groovy.sandbox.method_blacklist_patch truly append-only #9473
    • Scripting: Make groovy sandbox method blacklist dynamically additive #9470
    • Scripting: Add explainable script again #8665 (issues: #7245, #8561)
    • Scripting: Created a parameter parser to standardise script options #7977
    • Provide more context variables in update scripts #5724 (issues: #1607, #2230, #2231)
  • Search:
    • Added nested scope to parse context that keeps track the current nested level during search request parsing #9692 (issue: #9305)
    • Search: Reduce memory usage during fetch source sub phase #8138
    • Internal: Don't let took be negative. #7968
    • Core: switch to Lucene QueryRescorer #7707 (issue: #6232)
  • Settings:
    • Add 'http.publish_port' setting to the HTTP module #8807 (issue: #8137)
    • Settings: Validates bool values in yaml for node settings #8186 (issue: #8097)
  • Snapshot/Restore:
    • [RESTORE] Refactor how restore cleans up files after snapshot was restored #9770
    • Snapshot/Restore: add support for changing index settings during restore... #9285 (issue: #7887)
    • Snapshot/Restore: Override write(byte[] b, int off, int len) in FilterOutputStream for better performance #8749 (issue: #8748)
    • Snapshot/Restore: Allow custom metadata to specify whether or not it sho... #7901 (issue: #7900)
  • Stats:
    • Recovery: add total operations to the _recovery API #10042 (issue: #9368)
    • API: add pending tasks count to cluster health #9877
    • Stats : Add time in index throttle to stats. #7896 (issue: #7861)
  • Store:
    • [INDICES] Schedule pending delete if index store delete fails #9856
    • [STORE] Improve safety when deleting files from the store #9801
    • [STORE] Add simple cache for StoreStats #9709 (issue: #9683)
    • [STORE] use Directory#fileLength() less during calculating checksums #9689
    • [STORE] Cache fileLength for fully written files #9683
    • [STORE] Synchronize operations that modify file mappings on DistributorDirectory #8408
    • [STORE] Cut over MetaDataStateFormat to NIO Path API #8297
    • [STORE] Remove special file handling from DistributorDirecotry #8276
    • [STORE] Fold two hashFile implemenation into one #7720
  • Term Vectors:
    • Term Vectors: requests are now timed #9583
    • Term Vectors: support for realtime #7846
  • Translog:
    • [CORE] Handle truncated translog gracefully #9797 (issue: #9699)
    • Internal: Write translog opSize twice #7735
  • Upgrade:
    • Core: upgrade to Lucene 4.10.4 bugfix release #9960
    • Internal: Log when upgrade starts and stops #9229 (issue: #9227)
    • Dependencies: Upgrade netty to 3.10.0.Final #9132
    • upgrade to lucene 4.10.3 release #9100
    • Internal: Upgrade to Jackson 2.4.2 #7934 (issue: #7932)
    • Internal: Upgrade to Lucene 4.10.1 snapshot #7844 (issue: #7905)

Bug fixes

  • Aggregations:
    • Aggregations: Be lenient when converting local to utc time in time zone roundings #10031 (issue: #10025)
    • Aggs: Fix rounding issue using date_histogram with pre_zone_adjust_large_interval #9828 (issue: #8209)
    • Aggs: Fix rounding issues when using date_histogram and time zones #9790 (issues: #7673, #9491)
    • Aggregations: Prevent negative intervals in date_histogram #9690 (issue: #9634)
    • Aggs: Remove limitation on field access within aggs to the types provided in the search #9487
    • Aggs: nested agg needs to reset root doc between segments. #9441 (issues: #9436, #9437)
    • Aggs: Fix handling of multiple buckets being emitted for the same parent doc id in nested aggregation #9346 (issues: #8454, #9317)
    • Aggs: In reverse nested aggregation, fix handling of the same child doc id being processed multiple times. #9345 (issues: #9263, #9346)
    • Aggs: The parent filter of the nested aggregator isn't resolved correctly all the time #9335 (issue: #9280)
    • Aggs: post collection the children agg should also invoke that phase on its wrapped child aggs. #9291 (issue: #9271)
    • Aggs: Validate the aggregation order on unmapped terms in terms agg. #8952 (issue: #8946)
    • Aggregations: Fix date_histogram issues during a timezone DST switch #8655 (issue: #8339)
    • Fielddata: Fix iterator over global ordinals. #8627 (issue: #8580)
    • Aggregations: Fix geohash grid doc counts computation on multi-valued fields #8574 (issue: #8512)
    • Aggregations: Parser throws NullPointerException when Filter aggregation clause is empty #8527 (issue: #8438)
    • Aggregations: Fix geohash grid aggregation on multi-valued fields. #8513 (issue: #8507)
    • Aggregations: Change nested agg to execute in doc id order #8454
    • Aggregations: The children agg didn't take deleted document into account #8180
    • Aggregations: Fixes scripted metrics aggregation when used as a sub aggregation #8037 (issue: #8036)
    • Aggregations: Makes script params consistent with other APIs in scripted_metric #7969
    • Aggregations: Significant terms can throw error on index with deleted docs. #7960 (issue: #7951)
    • Aggregations: size property parsing inconsistent [ISSUE] #6061
  • Aliases:
    • Aliases: Take into account _default_ mapping in index template when parsing alias filter in index templates. #8577 (issue: #8473)
  • Allocation:
    • Allocation: Weight deltas must be absolute deltas #9149 (issue: #9023)
    • Allocation: Fix handling of dangling_timeout set to 0 and auto_import_dangled true #8257
    • Allocation: Enable ClusterInfoService by default #8206
  • Bulk:
    • Bulk API: Handle failed request when auto create index is disabled #8163 (issue: #8125)
    • Bulk API: Bulk operation can create duplicates on primary relocation #7729
  • CAT API:
    • Stats: Fix NPE in /_cat/nodes #7815 (issue: #6297)
  • Circuit Breakers:
    • Circuit Breaker: Only set breaker when stats are retrieved #7721
  • Core:
    • Core: also throttle delete by query when merges fall behind #9986
    • Internal: promptly cleanup updateTask timeout handler #9621
    • Core: Disable auto gen id optimization #9468 (issue: #8788)
    • Core: Verify the index state of concrete indices after alias resolution #9057
    • Core: ignore_unavailable shouldn't ignore closed indices #9047 (issue: #7153)
    • Core: Terms filter lookup caching should cache values, not filters. #9027 (issues: #1, #2)
    • Core: Don't forcefully release IW lock (this can cause corruption) #8892 (issue: #8588)
    • Core: Hard wire utf-8 encoding, so unicode filenames work #8847
    • Gateway: GatewayService should register cluster state listener before checking for current state #8789
    • Core: Remove unnecessary index removal on index creation #8639
    • [GATEWAY] Remove _state directory if index has been deleted #8610
    • disable bloom filters #8572 (issues: #8564, #8571)
    • Core: Change default eager loading behaviour for nested fields and parent/child in bitset cache #8440
    • Core: Don't eagerly load NestedDocsFilter in bitset filter cache, because it is never used. #8414 (issue: #8394)
    • Internal: Extend refresh-mapping logic to the _default_ type #8413 (issue: #4760)
    • Internal: Translog leaks filehandles if it's corrupted or truncated #8372
    • Return 0 instead of -1 for unknown/non-exposed ramBytesUsed() #8291 (issue: #8239)
    • Resiliency: Don't catch FNF/NSF exception when reading metadata #8207
    • Internal: Don't handle FNF exceptions when reading snapshot #8086
  • Dates:
    • DateMath: Use time zone when rounding. #9885 (issue: #9814)
    • Query DSL: Resolve now in date ranges in percolator and alias filters at search time instead of parse time #8534 (issue: #8474)
  • Discovery:
    • Zen: Node receiving a cluster state with a wrong master node should reject and throw an error #9963
    • Discovery: check index uuid when merging incoming cluster state into the local one #9541 (issue: #9489)
    • Discovery: only retry join when other node is not (yet) a master #8979 (issue: #8972)
    • Discovery: Removed unnecessary DiscoveryService reference from LocalDiscovery #8415 (issue: #8539)
    • Discovery: Improve the lifecycle management of the join control thread in zen discovery. #8327
  • Engine:
    • [ENGINE] Upgrade 3.x segments on engine startup #9899
    • Core: SerialMergeScheduler never triggers index throttling #8405
    • Internal: Add current translog ID to commit meta before closing #8245
  • Geo:
    • [GEO] Fix validate_* merge policy for GeoPointFieldMapper [OPEN] #10165 (issue: #10164)
    • Geo: Correct bounding box logic for GeometryCollection type #9550 (issue: #9360)
    • Geo: Throw helpful exception for Polygons with holes outside of shell #9105 (issue: #9071)
    • Geo: GIS envelope validation #9091 (issues: #2544, #8672, #9067, #9079, #9080)
    • Geo: Fix for NPE enclosed in SearchParseException for a "geo_shape" filter or query #8785 (issue: #8432)
    • Geo: Fix for geohash neighbors when geohash length is even. #8529 (issue: #8526)
    • Geo: Fix for ArithmeticException[/ by zero] when parsing a polygon #8475 (issue: #8433)
    • geo sort: remove unnecessary code from geo distance builder #8338
    • Geo: Fix IndexedGeoBoundingBoxFilter to not modify the bits of other filters. #8325
  • Index APIs:
    • Indices API: Fix to make GET Index API consistent with docs #9178 (issue: #9148)
    • Indices API: Fixed backward compatibility issue #8387 (issue: #8364)
    • Internal: Fix optimize behavior with 'force' and 'flush' flags. #7920 (issues: #7886, #7904)
    • Indices API: fixes GET Alias API backwards compatibility #7892 (issue: #7793)
  • Indexed Scripts/Templates:
    • Scripting: ScriptService can deadlock entire nodes if script index is recovering #8901
    • Indexed Scripts/Templates: GetIndexedScript call can deadlock #8266
    • Make template params take arrays #8255
    • Indexed Scripts/Templates: Cleaned up various issues #7787 (issues: #7559, #7560, #7567, #7568, #7647)
  • Internal:
    • Internal: Fix PageCacheRecycler's max page size computation. #10087 (issue: #10077)
    • Ensure that we don't pass negative timeInQueue to writeVLong #9662 (issue: #8077)
    • Snapshot status api: make sure headers are handed over to inner nodes request #9409
    • fix equality check of timevalue after serialization #9218
    • Internal: AdapterActionFuture should not set currentThread().interrupt() #9141 (issue: #9001)
    • Internal: PlainTransportFuture should not set currentThread().interrupt() #9001
    • Internal: IndexService - synchronize close to prevent race condition with shard creation #8557
    • Index API: BWC layer for GetIndex should not block in a listener #8496
    • Internal: Don't pass acceptdocs down to nonNestedDocsFilter in ParentsFilter #8463 (issue: #8461)
    • Indices API: Fix GET index API always running all features #8452
    • Internal: When corruption strikes, don't create exceptions with circular references #8331
    • Internal: Fixed bitset filter cache leftover in nested filter #8303
    • Internal: Cleanup non nested filter to not flip the FixedBitSet returned by the wrapped filter. #8232 (issue: #8227)
    • Internal: Dynamic changes to max_merge_count are now picked up by index throttling #8136 (issue: #8132)
    • Internal: Only schedule another refresh if refresh_interval is positive #8087 (issue: #8085)
    • Internal: Fix serialization of PendingClusterTask.timeInQueue. #8077
    • Internal: dangling indices import ignores aliases #8059
    • Fix serialization of short[] arays. #8025 (issue: #7845)
    • Internal: Make close() synchronized during node shutdown #7885
  • Java API:
    • Mappings: Fix Get field mapping api with pretty flag #8806 (issue: #6552)
    • Settings: Ensure fields are overriden and not merged when using arrays #8381 (issue: #6887)
    • Fixing SearchRequestBuilder aggregations call to facets #8121 (issue: #8120)
    • Fixing copy/paste mistake in SearchRequest.extraSource's exception message #8118 (issue: #8117)
    • Aggregations: Added missing module registration in TransportClient for Significant Terms #7852 (issue: #7840)
  • Logging:
    • [STORE] Add filename to corruption message #10110 (issue: #10062)
    • Transport: fix potential NPE in new tracer log if request timeout #9994 (issue: #9286)
    • Fix example in logging daily rotate configuration #8550 (issue: #8464)
    • Internal: Fix location information for loggers #8052 (issue: #5130)
  • Mapping:
    • Fixes ignore_malformed behaviour for ip fields #10112
    • Mapping: Throw StrictDynamicMappingException exception #9445 (issue: #9444)
    • Mapping: Using default=null for _timestamp field creates a index loss on restart #9233 (issues: #9104, #9223)
    • Mapping: Reencode transformed result with same xcontent #8974 (issue: #8959)
    • Mapping: serialize doc values settings for _timestamp #8967 (issue: #8893)
    • mappings: update cluster state with type mapping also for failed indexin... #8692 (issue: #8650)
    • Mapping: Fix conflict when updating mapping with _all disabled #8426 (issues: #7377, #8423)
    • Mappings: Generate dynamic mappings for empty strings. #8329 (issue: #8198)
    • Mapping: Throw exception if null_value is set to null #7978 (issue: #7273)
    • Mapping: Posting a mapping with default analyzer fails #7902 (issue: #2716)
  • More Like This:
    • MLT Query: Fix exclude with artificial documents #8679
  • Network:
    • Netty: Add HTTP pipelining support #8299 (issue: #2665)
  • Packaging:
    • Packaging: Add antlr and asm dependencies #9696
    • Packaging: Added quotes to allow spaces in installation path #8428 (issue: #8441)
    • Packaging: Move forbidden api signature files to dev-tools. #7921 (issue: #7917)
    • Packaging: Export JAVA_HOME in RPM init script #5434
  • Parent/Child:
    • Parent/child: Fix concurrency issues of the _parent field data. #9030 (issue: #8396)
    • Parent/child: Fixed p/c filters not being able to be used in alias filters. #8649 (issues: #5916, #8628)
    • Bulk API: Missing parent routing causes NullPointerException #8506 (issue: #8365)
    • Parent/child: has_parent filter must take parent filter into account when executing the inner query/filter #8020 (issue: #7362)
  • Percolator:
    • Percolator: Support encoded body as query string param consistently #9628
    • Percolator: Take filters from aliases into account #9590 (issue: #6241)
    • Percolator: Fixed bug when using multi percolate api with routing #9161 (issue: #6214)
  • Plugins:
    • Plugins: Installation failed when directories are on different file systems #9011 (issue: #8999)
    • Plugins: NPE when plugins dir is inaccessible #8839 (issue: #8837)
    • Plugins: Support usage of ES_JAVA_OPTS in plugin commands #8288
    • PluginManager: Fix config path extraction from plugin handle #7935
  • Query DSL:
    • Query DSL: Fix wrong error messages in MultiMatchQueryParser #8597
    • DateMath: Fix semantics of rounding with inclusive/exclusive ranges. #8556 (issue: #8424)
    • Query DSL: Fix NumberFormatException in Simple Query String Query #7876 (issue: #7875)
  • REST:
    • REST: Add fielddata_fields to the REST spec #9399 (issues: #4492, #9398)
    • Search: Passing fieddata_fields as a non array causes OOM #8203
  • Recovery:
    • Gateway: improve assertion at the end of shard recovery #10028
    • Recovery: synchronize RecoveryState.timer methods #9943
    • [RECOVERY] Don't recover from buggy version #9925 (issues: #7210, #9922)
    • Engine: close snapshots before recovery counter #9760 (issue: #9439)
    • [ENGINE] Fix deadlock problems when API flush and finish recovery happens concurrently #9648
    • [RECOVERY] Handle corruptions during recovery finalization #9619
    • Core: Mapping update task back references already closed index shard #9607
    • Recovery: update access time of ongoing recoveries #9506 (issue: #8720)
    • Recovery: cleaner interrupt handling during cancellation #9000
    • Internal: harden recovery for old segments #8399
    • Gateway: Prefer recovering the state file that uses the latest format. #8343
    • Recovery: change check for finished to a ref count check #8271 (issue: #8092)
    • Recovery: RecoveriesCollection.findRecoveryByShard should call recoveryStatus.tryIncRef before accessing fields #8231 (issue: #8092)
  • Scripting:
    • Scripting: File scripts cache key to include language and prevent conflicts #10033
    • script_file does not work with _update [ISSUE] #10007
    • Internal: Avoid unnecessary utf8 conversion when creating ScriptDocValues for a string field. #9557 (issue: #6908)
    • Scripting: Disallow method pointer expressions in Groovy scripting #9509
    • Scripting: Make _score in groovy scripts comparable #9094 (issue: #8828)
    • Search: Function score and optional weight : avg score is wrong #9004 (issue: #8992)
    • Scripting: Return new lists on calls to getValues. #8591 (issue: #8576)
    • Scripting: Add score() back to AbstractSearchScript #8417 (issues: #8377, #8416)
    • Scripting: Clear the GroovyClassLoader cache before compiling #8062 (issues: #7658, #8073)
    • Scripting: Fix NPE in ScriptService when script file with no extension is deleted #7953 (issue: #7689)
  • Search:
    • inner hits: Don't fail if an object is specified as a nested value instead of an array. #9743 (issue: #9723)
    • Query cache: Remove query-cache serialization optimization. #9500 (issue: #9294)
    • inner_hits: Make sure inner hits on has_parent query resolve hits properly #9384
    • Query cache: Queries are never cached when date math expressions are used (including exact dates) #9269 (issue: #9225)
    • Search: Make sure that named filters/ queries defined in a wrapped query/filters aren't lost #9166 (issue: #6871)
    • Search: Fix paging on strings sorted in ascending order. #9157 (issue: #9136)
    • Refactor term analysis for simple_query_string prefix queries #8435
  • Settings:
    • Settings: Reset TieredMP settings only if the value actually changed #9497 (issue: #8890)
    • Settings: cluster.routing.allocation.disk.threshold_enabled accepts wrong values [ISSUE] #9309
    • Settings: Tab characters in YAML should throw an exception #8355 (issue: #8259)
    • Settings: LogConfigurator resolveConfig also reads .rpmnew or .bak files #7457
  • Snapshot/Restore:
    • Snapshot/Restore: delete operation should ignore finalizing shards on no... #9981 (issue: #9924)
    • Snapshot/Restore: Allow deletion of snapshots with corrupted snapshot files #9569 (issue: #9534)
    • Snapshot/Restore: better handling of index deletion during snapshot #9418 (issue: #9024)
    • Snapshot/Restore: add validation of restored persistent settings #9051 (issue: #8830)
    • Snapshot/Restore: restore with wait_for_completion=true should wait for succesfully restored shards to get started #8545 (issue: #8340)
    • Snapshot/Restore: keep the last legacy checksums file at the end of rest... #8358 (issue: #8119)
    • Snapshot/Restore: restore of indices that are only partially available i... #8341 (issue: #8224)
    • Snapshot/Restore: Fix snapshotting of a single closed index #8047 (issue: #8046)
    • Snapshot/Restore: Make it possible to delete snapshots with missing metadata file #7981 (issue: #7980)
    • Snapshot/Restore: Make sure indices cannot be renamed into restored aliases #7918 (issue: #7915)
  • Stats:
    • Translog: make sure stats's op count and size are in sync #10041
    • Nodes Stats: Fix open file descriptors count on Windows #9397 (issue: #1563)
    • Stats: Relax restrictions on filesystem size reporting in DiskUsage #9283 (issues: #9249, #9260)
    • Indices API: Fix wrong search stats groups #8950 (issue: #7644)
    • Stats: Fix Bloom filter ram usage calculation #8584 (issue: #8564)
    • Stats: _status with #shards >> queue capacity failing with BroadcastShardOperationFailedException [ISSUE] #7916
  • Store:
    • Internal: only ack index store deletion on data nodes #9672 (issue: #9605)
    • [Store] Only fail recovery if files are inconsistent #8779
    • Store: Use Lucene checksums if segment version is >= 4.9.0 #8599 (issue: #8587)
    • Store: Calculate Alder32 Checksums for legacy files in Store#checkIntegrity #8407
  • Suggesters:
    • return an HTTP error code when a suggest request failed instead of 200 #10104
    • Completion Suggester: Fix CompletionFieldMapper to correctly parse weight #8197 (issue: #8090)
  • Translog:
    • Translog: stats fail to serialize size #10105
    • [GATEWAY] copy translog file if rename fails after retries. #9980
    • Translog: Never delete translog file from a snapshot #8917
  • Tribe Node:
    • Tribe node: system properties and configuration settings must not be forwarded to tribe clients #9721 (issue: #9576)
    • Tribe node: remove closed indices from cluster state #9334
  • Upgrade:
    • Upgrade: Fix version check in bytes to upgrade that spans major versions #9340

Regression

  • Mapping:
    • Mapping: Mapping With a null Default Timestamp Causes NullPointerException on Merge #9205 (issue: #9204)

Docs

  • Geo:
    • [GEO] fix docs for geo_point "validate" option [OPEN] #10168 (issue: #9539)
  • Java API:
    • A getHits() was forgotten! #9320
    • [Doc] Java API: add search templates #8716 (issue: #7321)
    • Docs: Add java documentation for aggregations #8694
  • NOT CLASSIFIED:
    • Need additional line in config file, to work tutorial's example. [ISSUE] #10074
    • fix typo #10045
    • Fix some docs n typos #10029
    • Documented default value for node_initial_primaries_recoveries #9955
    • [DOCS] added a note for the default shard_size value #9873
    • [DOCS] Default units for Geo Distance Agg is meters not km [ISSUE] #9812
    • [doc] Link mapper-attachment type documentation to its repo #9756 (issue: #101)
    • Clarify no master block [ISSUE] #9739
    • Add warning to settings documentation because setting number_of_replicas on a closed can lead to index beeing not openable again #9591 (issue: #9566)
    • Docs: Use the new experimental annotation. #9563
    • Testing classes don't support Parameterized tests? [ISSUE] #9423
    • Updates the command to add the repo #9261
    • Fix typo in sample json of keyword marker tokenfilter #9253
    • [TEST] Add missing docs and tests for '_cat/segments' #9179 (issue: #5856)
    • [Doc] Java API: add information on JBoss EAP #8766 (issue: #3445)
    • [DOCS] Document ActionNames class and related tests #8759
    • Fix error in documentation #8605
    • Update DiskThresholdDecider javadoc #8585 (issue: #6201)
    • Docs: Fix a typo in a javadoc comment #8575
    • Missing quote in the example #8565
    • randomizedtesting jar versions mismatch [ISSUE] #8450
    • Docs: note about confusing disk threshold settings #8437
    • Docs: document action.replication_type setting #8290
    • MLT Query: use ParseField#withAllDeprecated for percent_terms_to_match #8241
    • Docs: Clarify that index.routing.allocation.total_shards_per_node applies to replica and primary #8002
    • Docs: rolling upgrade process seems incorrect [ISSUE] #7973
    • null_value and dynamic_templates not working correctly [ISSUE] #7874
    • Docs : Filtering based on exact values not working [ISSUE] #7807
    • Docs: Document the most important changes to zen discovery. #7746
    • Inconsistent ceiling round up in range search for "lt" (less than) [ISSUE] #7203
  • Packaging:
    • Packaging: Added a command to start elasticsearch on bootup. #8600

Tests

  • Aggregations:
    • Adding tests for TimezoneRounding in 1.x #9796
  • NOT CLASSIFIED:
    • [TEST] Add tests that triggers truncated files on recovery #10155
    • [TESTS] Make sure test end with ..Tests #9947 (issue: #9945)
    • [TEST] Work around URI encode limitations in RestClient #9946 (issue: #9769)
    • Tests: Add back tests.cluster #9916
    • [TEST] Fix NPE in ElasticsearchIntegrationTest if no indexer is provided #9909 (issue: #9907)
    • Tests: check node ports availability when using unicast discovery #9886
    • Tests: Use all found index files instead of static list for static bwc tests #9854
    • Tests: Remove thread leak filter #9843
    • [TEST] Don't reuse single node tests node instance across tests #9813
    • Kernel32LibraryTests creates JNA proxy classes [ISSUE] #9802
    • Tests: Remove global shared cluster #9781 (issue: #8854)
    • Test: Remove use of globalTempDir() and forbid it from future use. #9777
    • Testing: Added REST tests to check if expression scripting works #9750
    • Testing: Ensure RepeatOnException rule is available in test-jar #9675
    • [TEST] Introduce basic validation of our REST spec #9629
    • Tests: Add static indexes to fill out 0.20.x coverage. #9537
    • [TEST] support stashed values within property names in our REST tests #9533
    • Testing: Add test rule to repeat tests on binding exceptions #9527 (issue: #9010)
    • Tests: Add type-unrestricted version of field mapper getter to SearchContext #9490
    • Test: use local gateway by default #9476
    • Tests: Rename tests with the same name #9370
    • Internal: Change snapshot state for unreleased versions and add validation tests for constants #9228
    • Tests: Add upgrade step to static bwc tests #9207
    • Add 0.20.x index. #9146
    • Don't enable custom data_path for backwards compatibility tests #9119
    • Fix TransportNodesListShardStoreMetaData for custom data paths #9101
    • [TEST] Make sure we restart the suite cluster after each test failure #9015
    • [TEST] upgrade randomized runner to 2.1.11 #8930
    • [TEST] LoggingListener to restore the initial logger levels after any modification #8845 (issue: #8820)
    • Test: fix InternalEngineTest.testIndexWriterIFDInfoStream to pass in IntelliJ #8822
    • [TEST] Pass class level test logging to external nodes #8820 (issue: #8552)
    • [TEST] Add unit tests for DiskThresholdDecider settings #8816
    • [TEST] guarantee REST tests execution order #8745
    • [TEST] make sure rest tests info is printed for any @Rest annotated test #8680 (issue: #7795)
    • [TEST] split base settings in ClusterDiscoveryConfiguration between node and transport client #8653
    • [TEST] Extend unicast ports generation to support more concurrent clusters #8634
    • [TEST] Register data.path for all nodes on close in InternalTestCluster #8519
    • Tests: Skip bwc analysis tests if using turkish or azuri locale. #8492
    • Allow -SNAPSHOT versions to be parsed by Version.fromString #8379
    • [TEST] move ClusterDiscoveryConfiguration to org.elasticsearch.test.discovery #8337
    • Tests: Pass through locale and timezone to test runner, and print in repro command line. #8315
    • [TEST] Remove redundant call to setTemplateType() #8295
    • [TEST] initialize SUITE | GLOBAL scope cluster in a private random context #8250
    • [TEST] Scope.SUITE is not reproducible due to late cluster initialization #8220
    • [TEST] Add simple BWC tests that start from pre-build indices [ISSUE] #8065
    • Tests: dump all thread stacks on failure #8033
    • [TEST] fix CurrentTestFailedMarker to reset its state after each test #8008
    • [TEST] create client nodes using node.client: true #7911
    • [TEST] Use Shutdown API only if nodes are on 1.3.3 or newer to prevent shutdown problems #7910 (issue: #7885)
    • [TEST] add regular scroll REST test #7860
    • Internal: MulticastChannel should wait on receiver thread to stop during shutdown #7835
    • Tests: Add a more restrictive thread leaks filter #7825 (issue: #7833)
    • [TEST] move REST tests to their own test group #7795
    • [TEST] Only reset test cluster if a test actually failed #7775
    • [TEST] Improve HTTP support in TestClusters #7738
    • [TEST] Update REST client before each test in our REST tests #7737 (issues: #6734, #7723)
    • Test: always run CheckIndex after a test, and fail the test if it detects corruption #7730
    • Tests: always run CheckIndex on all shards created by the test [ISSUE] #7724
    • [TEST] Minor REST tests infra cleanup #7723
    • [TEST] Expose ability to provide http headers when sending requests in our REST tests #7710
    • Tools: Add script to grab ES version for BWC tests. #7653
    • Geo: [TEST] Adds tests for GeoUtils #7510
    • Benchmark: BenchmarkIntegrationTest failure fixed [ISSUE] #6094
    • Test: Integration tests for benchmark API. #6023 (issue: #6003)
    • Test: Start nodes using node.bench true to enable benchmark api #5920 (issue: #5910)
  • Recovery:
    • [RECOVERY] Make recovery delay configurable #8739
  • Snapshot/Restore:
    • Tests: Make sure snapshots created with old version of elasticsearch can... #8968

Not classified

  • NOT CLASSIFIED:
    • Core: check only call rate limiter every minPauseCheckBytes during recovery [OPEN] #10185 (issue: #6018)
    • Build: Check that all packages contain the same jar files #9757
    • Rename Maven repository id #9733
    • [BUILD] Add marvel and license to the release script #9697
    • ignore intellij project/workspace files #9044
    • Switch to forbidden-apis 1.7 #8620
  • Recovery:
    • Recovery: node shut down during the last phase of recovery needlessly fails shard [ISSUE] #9496