Elasticsearch 1.3.0

Release Notes

Breaking changes

  • Allocation: Enable disk-space allocation decider by default #6200
  • Analysis: Improvements to StemmerTokenFilter #6452
  • Internal:
    • Make transport action name available in TransportAction base class #6860
    • Thread pool rejection status code should be 429 #6629
    • Remove unnecessary intermediate interfaces #6517
    • Remove unsafe unaligned memory access - illegal on SPARC #6962
  • Mappings: Update mapping on master in async manner #6648
  • REST API: Replace error code 503 with 429 when appropriate to tell client to back off #6627
  • Security: Disable JSONP by default #6795

New features

  • Aggregations:
    • Created infrastructure for changing easily the significance terms heuristic #6561
    • Added the percentiles_rank aggregation #6386
    • Deferred aggregations prevent combinatorial explosion #6128
    • Added top_hits aggregation #6124
    • Support bounding box aggregation on geo_shape/geo_point data types #5634
  • Scripting:
    • Add script engine for Lucene expressions #6819
    • Add Groovy as a scripting language, add groovy sandboxing #6233
    • Allow search templates stored in an index to be retrieved and used at search time #5921
    • Allow to run scripts/templates stored in .scripts index #5484
  • Mappings: Add transform to document before index #6599
  • Suggester: Phrase suggest option to limit suggestions to existing phrases #3482

Enhancements

  • Resiliency:
    • Resend failed shard messages when receiving a cluster state still referring to the failed shards #6881
    • Send shard exists requests if shard exists locally but is not allocated to the node #6870
    • Don't attempt to start or fail shard if no master node can be found #6841
    • Improve handling of failed primary replica handling #6825
    • During discovery, verify connect when sending a rejoin cluster request #6779
    • During relocation, process pending mapping update in phase 2 #6762
    • Set a default of 5m to recover_after_time when any of the expected*Nodes is set #6742
    • Start master/node fault detection pinging immediately during discovery #6706
    • Before deleting shard verify that another node holds an active shard instance #6692
    • Ensure index.version.created is consistent #6660
    • Cancel recovery if shard on the target node closes during recovery operation #6645
    • Raise proper failure if not fully reading translog entry #6562
    • Wait till node is part of cluster state for join process #6480
    • Have a dedicated join timeout that is higher than ping.timeout for join #6342
    • Before deleting a local unused shard copy, verify we're connected to the node it's supposed to be on #6191
    • Use Lucene built-in checksumming #5924
  • Indexing:
    • Set default translog flush_threshold_ops to unlimited, to flush by byte size by default and not penalize tiny documents #6783
    • Don't acquire dirtyLock on autoid for create #6584
    • Clear versionMap on refresh not flush #6379
    • If versionMap is too large we should trigger refresh #6378
    • Reuse Lucene's TermsEnum for faster _uid/version lookup during indexing #6298
    • Don't use AllTokenStream if no fields were boosted #6187
  • Analysis:
    • Improve Hunspell error messages #6850
    • Share numeric date analyzer instances between mappings #6843
    • Added missing pre built analysis components #6770
    • PatternAnalyzer should use PatternTokenFilter instead #6717
    • More resource efficient analysis wrapping usage #6714
    • Added additional Analyzers, Tokenizers, and TokenFilters from Lucene #6693
  • Administration & Monitoring:
    • Improve pending API to include current executing class #6744
    • Stats: Expose IndexWriter and versionMap RAM usage #6483
    • Improve indices stats options #6390
    • Improve cluster update settings api #6244
    • Remove field names in stats url #6054
    • Index template API: Unified PUT/POST behaviour in relation to create parameter #6429
  • Snapshot/Restore:
    • Restore an index without restoring its aliases #6457
    • Add ability to restore partial snapshots #5742
  • Aggregations:
    • Extend allowed characters in aggregation name #6702
    • Add from support to top_hits aggregator #6299
    • Moved BucketsAggregator#docCounts field to IntArray #6529
    • GlobalOrdinalsStringTermsAggregator is inefficient for high-cardinality fields #6518
    • Remove ordinals execution hint. #6499
    • Delegation of nextReader calls #6477
  • Fielddata:
    • Make BytesValues.WithOrdinals more similar to Lucene's SortedSetDocValues #6524
    • Don't expose hashes in Fielddata anymore. #6500
  • Allocation:
    • Allow primaries that have never been allocated to be allocated if under the low watermark #6209
    • Change high/low watermark defaults for disk based allocation decider #6201
    • Disk-aware allocation decider should allow initial primary allocation if under the high watermark #6196
  • Mapping:
    • Improve performance for many new fields introduction in mapping #6707
    • Better logic on sending mapping update new type introduction #6669
    • Wait for mapping updates during local recovery #6666
  • Scripting:
    • Add a transformer to translate constant BigDecimal to double #6609
    • Add Groovy sandboxing for GString-based method invocation #6596
    • Fix optional default script loading #6582
    • Exposed _uid, _id and _type fields as stored fields (_fields notation) #6406
  • Search:
    • Wrap filter only once in ApplyAcceptedDocsFilter #6873
    • Remove Queries#optimizeQuery - already handled in BooleanQuery #6743
    • Speed up exists and missing filters on high-cardinality fields #5659
    • Parent/Child: Support min_children and max_children on has_child query/filter #6019
    • Geo: Added caching support to geohash filter #6478
  • Highlighting:
    • Make the HighlightQuery class public #6446
    • Highlight fields in request order #6178
    • Plain highlighter does not honor _analyzer #5497
  • More Like This Query:
    • Ensure selection of best terms is indeed O(n) #6657
    • creates only one MLT query per field for all queried items #6404
    • Values of a multi-value fields are compared at the same level #6310
    • replaced exclude with include to avoid double ... #6275
    • allow for both like_text and docs/ids to be specified. #6246
  • Plugins:
    • Enables plugins to define default logging configuration for their needs. #6805
    • bin/plugin tests for missing plugin name when passing --url #6013
  • Build:
    • Check for tabs and nocommits in the code on validate #6474
    • Generate source jars for tests #6445
    • Packaging: Remove java-6 directories from debian init script #6350
    • Startup: Reset locale to C in bin/elasticsearch #6047
    • Windows: Modify command window title (windows) #6752
  • REST API:
    • Copy the headers from REST requests to the corresponding TransportRequest(s) #6513
    • Ensure 503 responses lead to retry on another node #4066
    • Security: Make JSONP responses optional. #6164
  • Internal:
    • Make XContentBuilder implement Releasable #6869
    • Don't close/reopen IndexWriter when changing RAM buffer size #6856
    • Added more utility methods to Settings #6840
    • Cleanup of the transport request/response messages #6834
    • Add local node to cluster state #6811
    • Control whether MapperService docMapper iterator should contain DEFAULT_MAPPING #6793
    • Upgrade to Jackson 2.4.1.1 #6789
    • Don't replace indices within ActionRequest and check blocks against concrete indices #6777
    • Improve Settings#get lookup for camel case support #6765
    • Improve large bytes request handling by detecting content composite buffer #6756
    • Call callback on actual mapping processed #6748
    • Remove intern calls on FieldMapper#Names for better performance #6747
    • Clean shard bulk mapping update to only use type name #6695
    • Disable explicit GC by default #6637
    • Make a hybrid directory default using mmapfs/niofs #6636
    • Make sure we don't reuse arrays when sending an error back #6631
    • Wrap RateLimiter rather than copy RateLimitedIndexOutput #6625
    • Upgrade to Lucene 4.9 #6623
    • Re-shade MVEL as a dependency #6570
    • Refactored AckedClusterStateUpdateTask & co. to remove code repetitions in subclasses #6559
    • Better default size for global index to alias map #6504
    • Suppress Unsafe is an internal proprietary API... compilation warnings #6423
    • use ConcurrentHashMapV8 for lower memory overhead #6400
    • Base64 decode parsing detects more errors #6334
    • Upgrade to netty 3.9.1 #6331
    • Added plugins to .gitignore #6324
    • Change the default type of the page recycler to CONCURRENT instead of SOFT_CONCURRENT #6320
    • Entirely cut over to TopDocs#merge for merging shard docs in the reduce phase #6197
    • Add support for Byte and BytesRef to the XContentBuilder #6127
    • Make Node and Client interfaces Closeable #4355
    • Settings: Throw error when incorrect setting applied to auto_expand_replicas #5752
    • TransportClient: Improve logging, fix minor issue #6376
    • Java API: Client intermediate interfaces removal follow-up #6563

Bug fixes

  • Resiliency:
    • Increment Store refcount on RecoveryTarget #6844
    • Improve handling of failed primary replica handling #6816
    • Do not ignore ConnectTransportException for shard replication operations #6813
    • Recovering replicas might get stuck in initializing state #6808
    • During recovery, only send mapping updates to master if needed #6772
    • If the node initialisation fails, make sure the node environment is closed correctly #6715
    • Handle ConnectionTransportException during a Master/Node fault detection ping during discovery #6686
    • Recovery from local gateway should re-introduce new mappings #6659
    • Also send Refresh and Flush actions to relocation targets #6545
    • Honor time delay when retrying recoveries #6226
  • Aggregations:
    • Histogram Aggregation key bug #6655
    • Fix JSON response for significant terms #6535
    • Fix reducing of range aggregations. #6484
    • Fix cardinality aggregation when doc values field is empty #6413
    • ReverseNestedAggregator does not compute parent documents correctly #6278
    • ClassCastException when sibling aggregations have the same name #6255
    • date_histogram aggregation breaks on date fields with multiple formats #6239
    • DateHistogramBuilder uses wrong data type for pre_offset and post_offset #5586
  • Administration & Monitoring:
    • Admin: Recovery API should also report ongoing relocation recoveries #6585
    • Cat API: Fix NullPointerException in recovery API #6190
    • Index Templates API: GET templates doesn't honor the flat_settings parameter. #6671
    • Snapshot/Restore: Allow deleting of interrupted snapshot #6383
  • Indexing:
    • Translog: Better support for partial buffer reads/writes in translog infrastructure #6576
    • Force refresh when versionMap is using too much RAM #6443
    • Lower the translog flush triggers to workaround #6377
    • Clear versionMap on refresh not flush #6363
    • DocumentMissingException is uncaught if thrown during retry of update request #6355
    • Geo: Valid polygon crossing dateline fails to parse #6179
  • Search:
    • The query_string cache should returned cloned Query instances. #6733
    • Replace empty bool queries with match_all to prevent NullPointerException #6722
    • Match query with operator and, cutoff_frequency and stacked tokens #6573
    • Search template not replacing parameter after initial failure in parameter substitution #6318
    • Nested: queries/filters/aggregations expect FixedBitSets, yet it isn't the case with NoneFilterCache #6279
  • Query DSL:
    • Fix MatchQueryParser not parsing fuzzy_transpositions #6300
    • Function score without function throws NPE #6292
    • Improved explanation for match_phrase_prefix #2449
  • Mapping:
    • MapperParsingException when create default mapping with 'include_in_all' nested #6304
    • Fix possibility of losing meta configuration on field mapping update #5053
    • Allow _version to use disk as a doc values format. #6523
  • Analysis:
    • stem_exclusion as array not working in language analyzers #6237
    • Default analyzer includes stopwords #5974
  • Bulk API:
    • Fix return of wrong request type on failed updates #6630
    • Bulk request which try and fail to create multiple indices may never return #6436
    • BulkProcessor does not call afterBulk when bulk throws eg NoNodeAvailableException #5038
  • Percolator:
    • Fix memory leak when percolating with nested documents #6578
    • Fix handling of nested documents #6540
    • Allow significant terms and geo hash grid aggregations in the percolator #6037
  • Plugins:
    • bin/plugin removes itself #6745
    • Removing plugin does not fail when plugin dir is read only #6546
    • Fix github download link when using specific version #6321
  • Java API:
    • Fix source excludes setting if no includes were provided #6632
    • BulkRequest#add(Iterable) to support UpdateRequests #6551
  • Internal:
    • Fix possible NPE during shutdown for requests using timeouts #6849
    • Prevent NPE if engine is closed while version map is checked #6786
    • Fixes Logger class for BackgroundIndexer #6781
    • Check for index blocks against concrete indices on master operations #6694
    • IndexingMemoryController should only update buffer settings of fully recovered shards #6667
    • Fix possible race condition in checksum name generator #6662
    • Lucene: Use XNativeFSLockFactory instead of the buggy Lucene 4.8.1 version #6424
    • Disable circuit breaking for BigArrays #6381
    • Guava doesn't explicitly remove entries when clearing the entire cache #6296
    • Filter cache size limit not honored for 32GB or over #6268
    • Threadpool Info: Allow to serialize negative thread pool sizes #6486
    • Core: The ignore_unavailable option should also ignore indices that are closed #6475
    • Remove indicesLifecycle.Listener from IndexingMemoryController #6892
    • Store: delete unallocated shards under a cluster state task #6902
    • Routing: Restore shard routing #6393
    • Scripting: Wrap groovy script exceptions in a serializable Exception object #6628

Tests

  • Resiliency: Add basic Backwards Compatibility Tests #6497
  • Add test case verifying that dynamically enabling/disabling merge IO throttling works #6842
  • Added ability to provide settings for external nodes in backwards compatibility tests #6809
  • Add assertBusy helper test method #6753
  • Reset all cluster if a test hit a failure #6734
  • Take compatibility version into account for XContentType randomization #6691
  • Prevent usage of system properties in the InternalTestCluster #6663
  • Randomize netty worker and connection parameters #6635
  • Change es.node.mode default for tests to local #6624
  • Verify all threads created by node and client have the node name #6516
  • Fix Test - Cluster naming #6510
  • Add test for accessing _score in scripts #6509
  • Randomly disable the filter cache #6280
  • Allow to disable plugin loading from classpath #6242
  • Add dummy docs injection to indexRandom #6235