See also Elasticsearch 6.5.0 release notes.
Audit security events in new structured logsedit
By default, when you enable auditing, it uses the
logfile audit output, which persists events to
<clustername>_audit.log file in the logs directory. This file is new to
version 6.5 and prints audit entries as JSON documents.
For backwards compatibility purposes, a
<clustername>_access.log with the old style of
formatting is also generated. See also Settings changes. Due to this,
the logging throughput effectively doubles. It is recommended that you enable only the
new audit format in production (by adjusting the settings in the
Discover the structure of text filesedit
[experimental] This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features. The new find file structure API determines the structure of a text file. It also returns statistics about the most common values of the detected fields and mappings that you can use when you ingest the file into Elasticsearch. If you want to influence the output, you can specify optional parameters. For example, if the file contains semi-structured text, you can specify a Grok pattern or let the tool generate it. This API is also used by the new File Data Visualizer in Kibana.
Improved machine learning results for partitioned multi-metric jobsedit
If you use the
by_field_name parameters in a machine learning job (or the
Split Data field in the Kibana multi-metric job wizard), it generates many
simultaneous analyses that are modeled independently. In 6.5, we have decreased
the influence of the anomaly scores in each partition on other partitions' scores.
As a result, record- and influencer-level results produce a much wider range of scores,
but the overall anomaly score still uses sophisticated aggregation to ensure rate-limited
alerts for the job.
Find multi-bucket anomalies in machine learning jobsedit
Sometimes events are not interesting or anomalous in the context of a single
bucket. However, they become interesting when you take into consideration a
sequence of buckets as a whole. In 6.5, there is a new machine learning
multi-bucket analysis, which uses features from multiple contiguous buckets
for anomaly detection. The final anomaly score is now a combination of values
from both the “standard” single-bucket analysis and the new multi-bucket
analysis. A new
multi_bucket_impact property in the
record results indicates how strongly either form of
analysis influences the score. In Kibana, anomalies with medium or high
multi-bucket impact are depicted in the Anomaly Explorer and the
Single Metric Viewer with a cross symbol instead of a dot.
Create source-only snapshotsedit
Source-only snapshots save space on disk by only saving stored fields and index metadata. This can result in up to 50% reduction in size. Index and doc values are not included, so a restored source-only snapshot is not searchable. You must reindex the data into a new index after it’s restored.
Apply token filters conditionallyedit
The conditional token filter enables you to use inline Painless scripts to filter out terms. Previously, conditionally applying token filters required creating and using custom analysis plugins.
Use ODBC to connect to Elasticsearch SQLedit
Elasticsearch SQL now supports ODBC mode in addition to JDBC. The functionality is currently the same as JDBC. An alpha version of the ODBC client is now available for download.
Delegate authorization to other realmsedit
If you enable the Elasticsearch security features, some realms now have the
ability to perform authentication internally then delegate authorization to
another realm. For example, you could authenticate using PKI then delegate to an
LDAP realm for role information. The realms that support this feature have a
authorization_realms setting that you can configure in the
elasticsearch.yml file. For more information, see
Delegating authorization to another realm and Realm settings.
Cross-cluster replication (beta*)edit
Cross-cluster replication enables you to replicate indices that exist in remote clusters to your local cluster. You create an index in your local cluster (known as a follower index) that follows an index (known as a leader index) in a remote cluster. You can also automatically follow indices in a remote cluster that match a pattern. The individual write operations that occur on the leader indices are then replayed on the follower indices. This functionality is useful for replicating data to a second cluster for disaster recovery purposes and for geo-proximity so that reads can be served locally.
Monitor Elasticsearch with Metricbeat (beta*)edit
In 6.4 and later, you can use Metricbeat to collect data about Kibana and ship it directly to your monitoring cluster, rather than routing it through Elasticsearch. Now in 6.5, you can also use Metricbeat to collect and ship data about Elasticsearch. If you are monitoring Logstash or Beats, at this time you must still use exporters to route the data. See Collecting monitoring data with Metricbeat and How it works.
* This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Please try this functionality in your test and development environments and provide feedback in the Elastic community forums.