world-map.jpg

hello.jpg

Categorizing Non-English Log Messages in Machine Learning for Elasticsearch

<h2>Grab Your Cartometers</h2><p>In Kibana 5.5 we added the ability to define your own <a href="/guide/en/kibana/5.5/regionmap.html">region maps</a>, which are also also known as choropleth maps. This allows users to define custom geo-boundaries as <a href="http://geojson.org/">geojson format</a>, then overlay them on the Elastic Map Service to display aggregations for custom geographical areas natively in Kibana.
</p><p>This is a simple yet very powerful method for providing localised insights from your datasets. We've touched on this previously in a <a href="/blog/region-maps-gauge-kibana">blog post</a>, but let's dig into this some more and run through a practical example of how to deploy it step by step in the Elastic Stack!
</p><p>If you'd like to replicate this blog post as you read it, the code for doing so is all <a href="https://gist.github.com/markwalkom/4f1c3dbfba74eb6fcca53aa8cd3e1ddb">here</a>.
</p><h3>Ready, Set….</h3><p>Before we start, let's define a few words that we will use throughout this post that you may be unfamiliar with, or have heard elsewhere in different contexts;
</p><br><ul>
	<li>geojson - a json file that contains one or more geoshapes that are specified in a <a href="http://geojson.org/">standardised format</a></li>
	<li>geoshape - an array of points that represent latitude and longitudes to form a polygon</li>
	<li>regionmap - a visualisation in Kibana used to show the results of aggregations that have been bucketed into geoshapes and then geographically mapped</li>
</ul><h2>Sourcing Your Map</h2><p><a href="https://exploratory.io/map"><span></span></a><a href="https://exploratory.io/map">https://exploratory.io/map</a> has a number of high quality maps and for this post we'll be using the Australian State file. If you download the file, extract it and take a look in it, you will find a repetitive structure, with the geoshapes we need and other fields that we will use later.
</p><p>There are a number of other sources out there that build and provide geojson files for you to download and use, your favourite search engine can help locate them for you.
</p><p>Ultimately, if you can't find what you want on the internet then you can build your own geojson files. That's outside the scope of this blog post, but luckily exploratory.io themselves have an excellent blog post on how to do that right <a href="https://blog.exploratory.io/creating-geojson-out-of-shapefile-in-r-40bc0005857d#.5pwo258dm">here</a>!
</p><h2>Configuring Your Source</h2><p>Now that we have our geojson file, we need to let Kibana know how to read it and map the geoshapes it holds onto the tiles that the Elastic Map Service provides.
</p><h3>kibana.yml</h3><p>Using the extracted aus_state.geojson from the "Australia States" archive, we need to configure some custom settings in your kibana.yml file. Open the yaml file in your favourite editor and add this on the end;
</p><pre class="prettyprint"># Custom Region Maps
regionmap:
  layers:
     - name: "Australian States"
       url: "http://localhost:8000/aus_state.geojson"
       attribution: "https://exploratory.io/maps"
       fields:
          - name: "STATE_NAME"
            description: "State Name"
</pre><p>These settings are explained in <a href="/guide/en/kibana/5.6/settings.html#regionmap-settings">this page</a> of the documentation, but let's break it down here;
</p><ul>
	<li>name: a description of the geojson file we are defining. Also used in the Options tab as we will see later.</li>
	<li>url: is where Kibana can retrieve the geojson file from. For now it needs to live on a webserver but we are working on other options, such as a file.</li>
	<li>attribution: it's always nice to provide a reference to the source of the file where we can!</li>
	<li>fields.name: this field is used to setup an inner join between the Elasticsearch document and the geojson file. More details on that soon.</li>
	<li>fields.description: is a human readable label that is displayed in the Options tab when building the visualisation. Just like with name, we will see that below.</li>
</ul><p>The <span style="font-family: Helvetica;"></span>fields.name we define above is extremely important. It tells Kibana how to take the documents in Elasticsearch and then figure out which shapes in the geojson file it needs to place that document for the aggregation results to be displayed. We will run through this in detail and step by step, so don't worry if that's a bit confusing for right now.
</p><h3>Serving Up The Geojson File</h3><p>The last step we need to do here is to serve the geojson file from a web server, as was defined in url. For this post we kept it simple and used this awesome little tool from npm, <a href="https://www.npmjs.com/package/http-server">http-server</a>, and ran this command via our shell from the same directory that the aus_state.geojson lives in;
</p><pre class="prettyprint">http-server --cors='*' -p 8000
</pre><p>You can test this by opening <a href="http://localhost:8000/aus_lga.geojson"><span></span></a><a href="http://localhost:8000/aus_lga.geojson">http://localhost:8000/aus_lga.geojson</a> from your browser, and you should see the contents of the geojson file.
</p><p>There are also dedicated products to serve your geo-data, like <a href="http://docs.geoserver.org/latest/en/user/production/container.html">Geoserver</a> that would be better suited for ongoing, production level deployments. We'd definitely recommend investigating them as more permanent solutions.
</p><h2>Deploying Your Configuration</h2><p>It's all been pretty text heavy so far, especially for what is ultimately a visualisation. So let's take a look at what this all translates to in the Kibana UI.
</p><h3>The Aggregation Settings</h3><p>As with most things in Kibana, we need to build an aggregation to show our data on our custom geoshapes. For this example dataset we will change the default Count Aggregation, under Metrics, to a Max and select the Population field to run the agg on. Then we Bucket the data using a Terms Aggregation on the Name field.
</p><p>Here's what we mean;
</p><div style="width:50%; float:left;"><img src="https://api.contentstack.io/v2/uploads/5a2ff5836bb36abe28075a02/download?uid=bltf099326ffed5478e" data-sys-asset-uid="bltf099326ffed5478e" alt="1.png" style="max-width:350px;">
</div><div style="width:50%; float:left;"><img src="https://api.contentstack.io/v2/uploads/5a2ff588d69e14e528c7796a/download?uid=blt3b7f4e870d2f2a1a" data-sys-asset-uid="blt3b7f4e870d2f2a1a" alt="2.png" style="max-width:350px;">
</div><p>Note - these are shown side-by-side to save room on this&nbsp;page. In Kibana they are displayed as a single vertical column on the left of the browser window.
</p><h3>The Map Settings</h3><p>Now we have to tell Kibana to use the regionmap settings as we defined in the kibana.yml.
</p><p>Under the Options tab we need to change the default Vector map to point to the one we configured, this is where the name field from above is used, so select Australian States. After that we select the Join field, which you have probably already guessed, will show the fields.description value of State Name.
</p><p><img src="https://api.contentstack.io/v2/uploads/5a2ff58c73340ef831ac65ac/download?uid=blta3a46fc28a4237ee" data-sys-asset-uid="blta3a46fc28a4237ee" alt="3.png" width="350">
</p><p>Then we hit the Apply changes button and….
</p><h2>The Outcome</h2><p>Let's break and explain something we briefly touched on earlier, remember when we talked about the inner-join? Well, at this point Kibana has;
</p><br><ol>
	<li>Loaded the geojson file from the webserver we specified.</li>
	<li>Built and run our aggregation to retrieve the results.</li>
	<li>Mapped those results into our custom geoshapes.</li>
	<li>And if you're playing along using the link we provided above, you can probably see the map already (lucky you!).</li>
</ol><p>However! Most importantly to all this is what happens between steps 2 and 3, where we run this inner join we keep referring to. Let's break that down as it's super important to understand if you want to start building these maps for your own datasets.
</p><h3>Joint Pain</h3><p>In our aus_state.geojson file we have a number of geoshapes that represent the boundaries of the Australian States (and Territories). Each of those has a STATE_NAME field with, funnily enough, the name of the state/territory in it.
</p><p>For any custom regionmap to display the data correctly, we need a field in each document we've indexed into Elasticsearch to have one of the same values as is in the specified join field from the geojson file.
</p><p>Therefore, if we have these values in the geojson file;
</p><pre class="prettyprint">grep STATE_NAME aus_state.geojson
        "STATE_NAME": "New South Wales",
        "STATE_NAME": "Victoria",
        "STATE_NAME": "Queensland",
        "STATE_NAME": "South Australia",
        "STATE_NAME": "Western Australia",
        "STATE_NAME": "Tasmania",
        "STATE_NAME": "Northern Territory",
        "STATE_NAME": "Australian Capital Territory",
        "STATE_NAME": "Other Territories",

</pre><p>And if we create a document in Elasticsearch like so;
</p><pre class="prettyprint">PUT aussiestates/doc/nsw
{
  "Name": "New South Wales",
  "Population": 7757800
}
</pre><p>We then create a terms aggregation on the `Name` field (i.e. step 2). Kibana takes the value of the `Name` field and look in the geojson file against the field we defined in `fields.name`, which is `STATE_NAME`. Given it finds an exact match, it adds the max value in the `Population` field to the `New South Wales` bucket (i.e. step 3). Then it takes the custom geoshape and transposes it onto the tilemap service with the results of the aggregation. Then it repeats this over and over until all the documents have been processed and we have the final results being displayed (step 4).
</p><p>And what does that all translate to?
</p><p><img src="https://api.contentstack.io/v2/uploads/5a2ff590d69e14e528c77970/download?uid=blt0a0624e71f5e2ef1" data-sys-asset-uid="blt0a0624e71f5e2ef1" alt="4.png">
</p><p>Glorious, sunburnt Australia! Some places more than others&nbsp;;)<br>
</p><h2>Troubleshooting</h2><h3>space-not-tabs.yml</h3><p>One general point we will start with is that the configuration file for Kibana is yaml formatted. That means that spaces are super important for nested definitions like the one we use here. If you run into problems then it's always good to check your indentation is correct and uses spaces and not tabs.
</p><h3>There's Nothing On My Map!</h3><p>If you've still gotten this far but your maps are still blank, remember that the values in the geojson file and the Elasticsearch document need to be the same. Not just similar, exactly the same. If you aren't already adjusting your mappings, make sure you do and set the field that contains the value to a <a href="/guide/en/elasticsearch/reference/6.0/keyword.html">keyword</a> to ensure it doesn't get split and/or lower-cased like a <a href="/guide/en/elasticsearch/reference/6.0/text.html">text</a> field would. Ultimately you need make sure your processing pipeline normalises the field values to the geojson file values.
</p><p>From 6.1 there will be&nbsp;improvements on how we handle failed joins. We will allow users to turn off warnings for failed joins, so when some terms aren't present in the geojson source, the dashboard won't be overloaded with error warnings.
</p><h3>CORS Mate!</h3><p>CORS stands for Cross-Origin Resource Sharing and is an important feature of the browser's security model. CORS configurations define how a browser decides what content any given web-application has access to, based on the domain (or origin) of where that web application is hosted. The default behavior is simple; Browsers do NOT allow Javascript to load or post any content cross-origin. If this were so the browser would make data-theft very easy! &nbsp;After all, as administrators, we generally do not want a web-application to retrieve (possibly malicious) code or post (possibly private) content from or to another domain we do not control. Our users certainly would not want this!
</p><p>For example, javascript-code from the Kibana application hosted on the <a href="http://engineering.example.org"><span></span></a><a href="http://engineering.example.org">http://engineering.example.org</a> cannot execute a request to fetch files hosted on <a href="http://accounting.example.org"><span></span></a><a href="http://accounting.example.org">http://accounting.example.org</a>, as they are on two different domains.
</p><p>To work around this, servers can advertise which other domains they "trust". This is a server's CORS configuration. As an example, the admins of accounting.example.org can configure the CORS-headers of their server so it will accept requests from the engineer.example.org domain.
</p><p>The browser's behavior would then look something like this: When users visit Kibana running on the on engineering.example.org domain, the browser will first sent a preliminary 'test'-request to accounting.example.org. When the server on accounting.example.org responds that it accepts requests from websites hosted on the engineering.example.org domain, the browser will then send the "real" request for the data to accounting.example.org.
</p><p>So why did we decide to require users to setup CORS? After all, why can't the Kibana server serve the geojson file so we can avoid this whole CORS business? The reason for this is that geodata comes from many different sources, and a lot of organisations already have an established stack for geo-data deployed, products like Geoserver or ArcGIS server. It is very common that these dedicated services would be running on a different domain from where Kibana will be hosted.
</p><h2>And That's It!</h2><p>One last thing that we will note is that the Region map provides a zoom level of 10 by default. To increase the zoom levels, install X-Pack alongside Kibana and Elasticsearch and then register for <a href="https://register.elastic.co/">a free Basic license</a> for all 18 levels!
</p><p>We hope this has been useful and given you some great ideas on how to use custom maps with &nbsp;the flexibility built natively into the Elastic Stack. Imagine extending this to show voting levels based on your state/district/electoral areas, or grades for schools based on their coverage area, or even showing <a href="https://discuss.elastic.co/t/region-map-polygon-line-widths-and-legend-background-customization/107088/5">soil type, soil infiltration capability, land use, and the depth of the seasonal high water table!</a> It's now even easier if you have the data.
</p><p>Feel free to head on over to <a href="https://discuss.elastic.co">our forums</a> if you have any further questions, or if you just want to share your awesome custom Region map successes, and thanks for reading.
</p>

blog-region-maps-fb.jpg

blog-region-maps-thumb.jpg

Custom Region Maps in Kibana 6.0

<p>One of my favorite parts of our Elastic{ON} user conference is the Birds of a Feather sessions. There’s something about the community of Elastic users, and their willingness to share stories and experiences, that makes these sessions interesting and fun, *even* if the topic is something relatively dry, such as regulatory compliance!
</p><p>I had the opportunity to moderate one of these sessions where the topic was GDPR, the European Union’s General Data Protection Regulation, which goes into effect on May 25, 2018.
</p><p>As compliance discussions go, this one was unusual in the sense that not one person shared an audit story! Of course this makes sense, since the GDPR regulation is not yet being enforced, and there’s no history of audit findings, court rulings, or fines levied.
</p><p>I estimate that there were about 40-50 people in the room. The format was an open discussion, but I had the opportunity to sprinkle in a few questions and jot down some notes. While this is the furthest thing from a scientific poll, I thought I’d share some of the discussion points and anecdotes in the form of a Q&A, with the A’s representing my perception of the sentiment of the group.
</p><p>Q1: How would you describe your Elastic Stack use case that is affected by GDPR?
</p><p>A1: While GDPR affects all kinds of personal data, this discussion focused on logging-related use cases, with IT operational analytics and security analytics cited by many participants. Others mentioned application and APM logging. Interestingly, several participants said their GDPR-affected use of the Elastic Stack spanned multiple use cases.
</p><p>Q2: With respect to GDPR, is your organization a data processor, a data controller, or both?
</p><p>A2: While a few birds claimed only the data processor role or the data controller role, the vast majority responded that their organization played both data processor and data controller roles.
</p><p>Q3: GDPR recommends pseudonymization as one method to protect personal data - What kinds of Personal Data are you planning to pseudonymize that might not be obvious?
</p><p>A3: A few respondents mentioned telephone numbers and user IDs. Many responded with session ID information, such as that coming from&nbsp;application logs, load balancer logs, web servers logs, and APM. Others mentioned file-paths in access logs (e.g., user/mikepaquette), and even latitude/longitude coordinates and other geoIP information. IP address treatment was a major topic of discussion.
</p><p>Q4: Again with respect to pseudonymization and other treatment of personal data, what methods are you considering to pseudonymize data?
</p><p>A4: Respondents discussed a number of techniques, including removing it from logs altogether, masking the data (almost all reported they plan to use this approach), and marking/tagging the data and controlling access. Some attendees reported “all of the above,” depending on the data.
</p><p>Q5: Are you using any automated means to “detect” personal data on an ongoing basis (e.g., checking fields that don’t normally contain personal data)?
</p><p>A5: An interesting related topic was scanning application error logs that might contain personal data (username/password, transaction details etc.) before allowing the logs to be seen at by analysts. One respondent had built scripts and queries to try to identify common types of data that might be present.
</p><p>Q6: Do you plan to use encryption at rest on the systems on which the Elastic Stack was installed?
</p><p>A6: Most folks in the room were unsure of the necessity of employing encryption at rest on their Elasticsearch clusters systems. Participants said that this issue was closely related to the pseudonymization discussion. They asked if the personal data was cryptographically pseudonymized, why do you need to encrypt the disk too? Although others mentioned that good security control might include disk-based encryption. A couple participants presumed that volume-based encryption might be too slow.
</p><p>Q7: What kind of access controls do you expect to deploy on Elastic Stack systems processing Personal Data?
</p><p>A7: Most attendees said they planned to use X-Pack <a href="http://www.elastic.co/products/x-pack/security"> security features </a> to implement access controls to Elasticsearch clusters that processed personal data, including X-Pack authentication and role-based access controls (RBAC). An interesting idea came up regarding geo-based access controls for limiting access to data depending on where the user was located. Our product team was there to take that idea back for investigation!
</p><p>Q8: If you’ve determined that you need to pseudonymize IP addresses in your logs, are you considering or using any special treatment of IP addresses to compensate for analyst inability to use them in detections and investigations (e.g., range calculations, subnet matching)?
</p><p>A8: This topic prompted lots of discussion. Most approaches involved simply hashing the IP address fields (e.g., using <a href="https://www.elastic.co/guide/en/logstash-versioned-plugins/versioned_plugin_docs/v3.1.2-plugins-filters-fingerprint.html#v3.1.2-plugins-filters-fingerprint-options"> Logstash fingerprint filter</a>), so correlation can still happen within the Elastic Stack, but limiting the ability to view the actual IP address to certain staff. Many mentioned similar hashing, but also adding additional enrichment (e.g., geoIP, including ASN) to events to compensate for the inability to see the actual IP address. Technical approaches ranged from using strong encryption to using “special” pseudonymization functions that preserve subnet and range visibility.
</p><p>Q9: Do you plan to use automated retention period enforcement techniques on your data in the Elastic Stack?
</p><p>A9: Only one person described how their organization had automated the removal of personal data by automatic index deletion after the retention period had transpired. Many seemed to be approaching this with manual processes at first, and would consider automation at a later time, when processes were more mature.
</p><p>Q10: We know that GDPR recommends securing personal data according to its risk. One could argue that in some cases, the disclosure of an IP address that was present in a log file would be a pretty minor privacy risk. Is your organization considering NOT pseudonymizing IP addresses based on this logic?
</p><p>A10: Although several folks agreed that the impact to privacy was probably low, it seemed that the main reason for still pseudonymizing IP addresses was worry that their legal team might not agree with them!
</p><p>So there you have it - a quick flight over some some current GDPR topics as shared by our birds of a feather. The 45-minutes seemed to “fly” by!
</p><p>We’ll be discussing some of these topics and more during our upcoming <a href="https://www.elastic.co/webinars/gdpr-compliance-and-elasticsearch"> GDPR webinar</a>. Why not participate to learn more and join in the live Q&A?
</p><p>In the meantime, please check out our <a href="https://www.elastic.co/gdpr"> GDPR white paper</a>, in which we provide some GDPR background information and describe a process that organizations can use to prepare for GDPR.
</p>

GDPR_BoF_2Kx415 copy.jpg

GDPR_Flight2.jpg

A Quick Flight over GDPR: Ten topics from our GDPR & Elasticsearch session at Elastic{ON} 2018

<p>At Elastic, we are big fans of providing self-service tools to successfully deploy and maintain our software in production. One of the most popular features of Elasticsearch is the APIs used to retrieve it's operational data. While Logstash was always easy to get started with, it was a bit of a black box when it came to managing it in production. Administrators had to rely on external monitoring systems to make sure Logstash was functioning without any issues. A core theme in 5.0 release was to make it easier to manage Logstash. In this post, I'll show you how to monitor your Logstash filters using the inbuilt APIs and the <a href="https://www.elastic.co/products">Elastic Stack</a> itself.&nbsp;
</p><h2>What is my ingestion rate?</h2><p>If you have deployed Logstash as part of a high-volume data ingestion pipeline, one of the most important metrics to monitor is “how fast is Logstash ingesting data”. Previously, to answer this question, you had to install the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-metrics.html">metrics filter</a> to generate the ingestion stats. Starting <a href="https://www.elastic.co/blog/logstash-5-2-0-released">with 5.2</a>, we've made it really easy to monitor Logstash using the Elastic Stack itself in addition to the <a href="https://www.elastic.co/guide/en/logstash/current/monitoring.html">native APIs</a>&nbsp;which were introduced in Logstash 5.0. The <a href="https://www.elastic.co/downloads/x-pack">x-pack plugin</a> for Elastic Stack bundles an agent that you can install on all your production Logstash nodes. It also ships with charts for the most important metrics information that let you visualize Logstash monitoring data out of the box. The monitoring feature in x-pack is under the <a href="https://www.elastic.co/subscriptions">Basic license</a> which means you get all this awesomeness for free. The APIs mentioned here are all part of the open source Logstash distribution.&nbsp;
</p><p>Let's get started:<br>
</p><ul>
	<li>Install x-pack on your production Logstash nodes. This assumes you've already configured your production cluster for monitoring. If not, you can find the details <a href="https://www.elastic.co/guide/en/x-pack/current/monitoring-cluster.html">here</a>.</li>
</ul><pre class="prettyprint">cd $LS_HOME
bin/logstash-plugin install x-pack
</pre><p>This plugin should be installed on <i>all</i> Logstash nodes you wish to monitor. The x-pack plugin extends your Logstash to periodically ship all the metrics it collects to your production Elasticsearch cluster. Once installed, you can configure the shipping agent using the <code>logstash.yml</code> file.<br>
</p><ul>
	<li>Configure your Elasticsearch end-point.</li>
</ul><p> <code>xpack.monitoring.elasticsearch.url: "http://my-es-prod-node:9200"</code>
</p><p>We strongly recommend that you enable security for your production Elasticsearch clusters. Once you've installed x-pack, you can use the extensive <a href="https://www.elastic.co/products/x-pack/security">security features</a> that ships with it.<br>
</p><ul>
	<li>Restart Logstash and navigate to <code>http://kibana-url:5600</code></li>
</ul><p>Pick the Logstash you want to get detailed monitoring info:
</p><p><img src="https://api.contentstack.io/v2/uploads/588febd1c387e5ea18b34892/download?uid=blt166210739a9e46e6" data-sys-asset-uid="blt166210739a9e46e6" alt="nodes_list.png" width="568" height="151" style="width: 568px; height: 151px;">
</p><p><img src="https://api.contentstack.io/v2/uploads/588febf1f6ea3cf85f2c933c/download?uid=blt90f828912b55c071" data-sys-asset-uid="blt90f828912b55c071" alt="events_rate.png" width="575" height="311" style="width: 575px; height: 311px;">
</p><h2>What are my slowest filters?</h2><p>Filters are the workhorses of Logstash, which means they can be the slowest stage of your pipeline, compared to the inputs and outputs stage. The performance of a filter is also dependent on the type of data it processes. For example, the grok filter, a very popular one, uses a regular expression engine to extract fields. And certain types of event data can throw your grok filter into an expensive backtracking loop. Before we attempt to tweak the filters for performance, the first step if to actually figure out which one of them is the slowest:<br>
</p><pre class="prettyprint">curl localhost:9600/_node/stats/pipeline?pretty
</pre><pre class="prettyprint">...
"filters":[  
  {  
     "id":"2178468f89ab43c6f32e934d3cb587e3c850d93a-4",
     "events":{  
        "duration_in_millis":24005,
        "in":49482,
        "out":49480
     },
     "name":"geoip"
  },
  {  
     "id":"2178468f89ab43c6f32e934d3cb587e3c850d93a-2",
     "events":{  
        "duration_in_millis":84193,
        "in":49483,
        "out":49482
     },
     "matches":29400,
     "failures":20083, 
     "patterns_per_field":{  
        "message":1
     },
     "name":"grok"
  },
  {  
     "id":"2178468f89ab43c6f32e934d3cb587e3c850d93a-5",
     "events":{  
        "duration_in_millis":44353,
        "in":49480,
        "out":49477
     },
     "name":"useragent"
  }
  ...
</pre><p>For all the plugins in our pipeline, this API provides a per-pipeline break down of event stats. One particular field of interest  is the <code>duration_in_millis</code> field which reports the total amount of time spent in a filter for all the events that passed through it. If you divide <code>duration_in_millis</code> by <code>events.in</code>, you should get the average time events have spent in this filter. From the response above, it looks like the grok filter is the slowest, followed by the useragent filter.  Zooming in on the grok filters, we can also see that there are quite a bit of grok match failures. It is plausible that the regular expression engine tried parsing the pattern(s), and may have gone into a backtracking loop wasting a lot of CPU cycles, contributing to the overall slowness.
</p><p><img src="https://api.contentstack.io/v2/uploads/588fed461dd6e41e6ff76159/download?uid=blt6092a17c7ccb1cba" data-sys-asset-uid="blt6092a17c7ccb1cba" alt="tip.png" style="background-color: initial; float: left; margin: 0px;">&nbsp; An important tip to our readers — if you see the response above, the ID field contains a randomly &nbsp; &nbsp; &nbsp; generated UUID which identifies each of the filters. You can also, and its highly recommended that &nbsp; &nbsp; &nbsp; you set the ID field in your config file to label plugins appropriately. This can help a lot when&nbsp;debugging a pipeline with multiple grok filters. For example, in your config file :
</p><pre class="prettyprint">filter {
  grok {
    id =&gt; "apache_common_log_praser"
    pattern =&gt; ..
  }
  grok {
    id =&gt; "java_exception_logs_parser"
    pattern =&gt; ..
  }
}
</pre><p>Now that we've identified the grok filter as the slowest, we can tweak the regular expression pattern to optimize the performance. We have <a href="https://www.elastic.co/blog/do-you-grok-grok">previously posted</a> about various ways to tame your Grok filter, which is a highly recommended read.
</p><p>If you head over to the monitoring UI, the events latency graph shows you aggregated information on how long it takes on average for events to be processed in your pipeline.<br>
</p><p><img src="https://api.contentstack.io/v2/uploads/588fec3b647b87d73bfa45bd/download?uid=blt7676c01a7e3493a7" data-sys-asset-uid="blt7676c01a7e3493a7" alt="Screen Shot 2017-01-29 at 08.37.54.png" width="426" height="141" style="width: 426px; height: 141px;">
</p><p>One of my favorite parts of shipping monitoring data to Elasticsearch is the access to historical information. As you iterate over your configs, you can validate from these graphs if your changes made a difference to the overall pipeline latency. This historical information can also be used to spot performance regressions over time in your production setting. You are no longer flying in the dark!<br>
</p><h2>Slowlog</h2><p>
	In our previous section, we showed you how to identify the slow sections of the pipeline. But, this doesn't pinpoint which events <i>may </i>have contributed to the slowdown<i>. </i>Enter the <a href="https://www.elastic.co/guide/en/logstash/current/logging.html#_slowlog">slowlog feature</a><i>.</i> <br>
	<br>
	Slowlog adds the ability to log when a specific event takes an abnormal amount of time to make its way through the pipeline. You can define the criteria when you want the slowlog to be triggered. Just like normal application logs, you can find slow-logs in your <code>—path.logs</code> directory<i>. </i>In your logstash.yml file, add this setting:<br>
</p><pre class="prettyprint">slowlog.threshold.warn: 2s
</pre><p>
	When events take more than 2 seconds to be processed, you should see:<br>
</p><pre class="prettyprint">[2016-10-27T13:48:19,308][WARN ][slowlog.logstash.filters.grok] event processing time {:took_in_millis=&gt; 2130, :event=&gt;"{\"sequence\":1,\"@timestamp\":\"2016-10-27T20:48:18.150Z\",\"@version\":\"1\",\"host\":\"elasticbox-2.local\",\"mes
sage: \"217.69.133.237 - - [11/Sep/2014:21:58:29 +0000] "GET /scripts/netcat-webserver HTTP/1.1" 304 - "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)""\"}\"}
</pre><h2>Hot Spots</h2><p>
	If the CPU on your Logstash instance is running red hot, you can issue a command like <code>top -H</code> to analyze what parts of the Logstash pipeline is consuming the most CPU. Logstash threads are labeled based on the stage and plugin name, so from the output of top, one can quickly find out which parts are CPU intensive. For example, <code>[main]&gt;worker1</code> denotes a filter + output thread, while <code>[main]&lt;generator</code> denotes a generator input.<br>
</p><p><img src="https://api.contentstack.io/v2/uploads/588fed9e2886d7f63f63b5d7/download?uid=blt1f9f15aed960df49" data-sys-asset-uid="blt1f9f15aed960df49" alt="top_output.png" width="396" height="96" style="width: 396px; height: 96px;">
</p><p>While this is super useful, it would be nice to zoom into those threads to find the expensive parts. We have an API just for that — <a href="https://www.elastic.co/guide/en/logstash/current/hot-threads-api.html">the hot threads API</a> provides a thread dump or a snapshot of the java threads sorted by the time spent.<br>
</p><pre class="prettyprint">curl localhost:9600/_node/hot_threads?human
::: {}
Hot threads at 2017-01-29T04:18:55+00:00, busiestThreads=3:
================================================================================--------------------------------------------------------------------------------
64.07 % of cpu usage, state: runnable, thread name: '[main]&gt;worker0'
    org.jruby.RubyRegexp$SearchMatchTask.run(RubyRegexp.java:273)
    org.jruby.RubyThread.executeBlockingTask(RubyThread.java:1066)
    org.jruby.RubyRegexp.matcherSearch(RubyRegexp.java:235)
    org.jruby.RubyRegexp.search19(RubyRegexp.java:1780)
    org.jruby.RubyRegexp.matchPos(RubyRegexp.java:1720)
    org.jruby.RubyRegexp.match19Common(RubyRegexp.java:1701)
    org.jruby.RubyRegexp.match_m19(RubyRegexp.java:1680)
    org.jruby.RubyRegexp$INVOKER$i$match_m19.call(RubyRegexp$INVOKER$i$match_m19.gen)
    org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:350)
    org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
</pre><p>
	In the above example response, we see again that the regex engine used by the Grok filter is the busiest thread. You could also use this API to find out if your Logstash is stuck performing I/O operations.<br>
</p><h2>Conclusion</h2><p>
	In this post, we introduced you to our new monitoring UI feature and showed a few ways to identify slow parts of Logstash. We are not done yet — our roadmap has exciting monitoring features that will make Logstash and your pipelines even more of a breeze to manage. This post is part of a series on monitoring, so stay tuned for our next one and all our product updates.<br>
</p>

events_rate.png

Monitoring Logstash Filters: X-ray glasses included

<p>A little over a year ago I&nbsp;<a href="https://www.elastic.co/blog/performance-considerations-elasticsearch-indexing">described how to maximize indexing throughput</a>&nbsp;with Elasticsearch, but a lot has changed since then so here I explain the changes that affect indexing performance in Elasticsearch 2.0.0.
</p><p><img src="https://api.contentstack.io/v2/assets/575e4cce58208ba076e28b50/download?uid=blt2cbb4778383fcda9?uid=blt2cbb4778383fcda9" data-sys-asset-uid="blt07af3496603329d7" alt="small-auto-throttle.jpg" style="max-width:100%;">
</p><h2>Store throttling is now automatic</h2><p>Prior to 2.0.0, Elasticsearch throttled ongoing merges at fixed rate (20 MB/sec by default), but this was often far too low and would easily&nbsp;lead to merges falling behind and subsequent index throttling.&nbsp;<br>
</p><p>As of 2.0.0, Elasticsearch now&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/9243">uses adaptive merge IO throttling</a>: when merges are starting to fall behind, the allowed IO rate is increased, and decreased when merges are keeping up. This means a sudden but rare large merge on a low-throughput indexing use case should not saturate all available IO on the node, minimizing impact to ongoing searches and indexing.
</p><p>But for heavy indexing, the allowed merge IO will increase to match the ongoing indexing.
</p><p>This is good news! &nbsp;It&nbsp;means you shouldn't need to touch any throttling settings: just use the defaults.<br>
</p><h2>Multiple&nbsp;<code>path.data</code></h2><p>Using multiple IO devices (by specifying multiple&nbsp;<code>path.data</code>&nbsp;paths)&nbsp;to hold the shards on your node is useful for increasing total storage space, and improving IO performance, if that's a bottleneck for your Elasticsearch usage.
</p><p>With 2.0, there is an important change in how Elasticsearch spreads the IO load across multiple paths: previously, each low-level index file was sent to the best (default: most empty) path, but now that switch is per-shard instead. &nbsp;When a shard is allocated to the node, the node will pick&nbsp;which path&nbsp;will hold all files for that shard.&nbsp;
</p><p>The improves resiliency to IO device failures:&nbsp;if one of your IO devices crashes, now&nbsp;you'll only lose the shards that were on it, whereas before 2.0&nbsp;you would lose any shard that had at least one file on the affected device (typically this means nearly all shards on that node).
</p><p>Note that&nbsp;an OS-level RAID 0 device is also a poor choice as&nbsp;you'll lose all shards on that node when any device fails, since files are striped at the block level,&nbsp;so multiple&nbsp;<code>path.data</code>&nbsp;is the recommended approach.<br>
</p><p>You should still have at least 1 replica for your indices so you can recover the lost shards from other nodes&nbsp;without any data loss.
</p><h2>The Auto-ID optimization is removed</h2><p>Previously, Elasticsearch optimized the auto-id case (when you didn't provide your own id for&nbsp;each indexed document) to use append-only Lucene APIs under-the-hood, but this proved&nbsp;problematic in error cases and so&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/9468">we removed it</a>.
</p><p>We also&nbsp;<a href="https://github.com/elastic/elasticsearch/issues/6212">greatly improved ID lookup performance</a>&nbsp;so that this optimization was no longer so important, and this allowed us to&nbsp;<a href="https://github.com/elastic/elasticsearch/issues/8571">remove bloom filters</a>&nbsp;entirely as they consumed non-trivial heap but didn't offer much ID lookup performance gain.
</p><p>Finally, as of 1.4.0, Elasticsearch&nbsp;<a href="https://github.com/elasticsearch/elasticsearch/issues/5941">now uses Flake IDs</a>&nbsp;to generate its IDs, for better lookup performance over the previous UUIDs.
</p><p>All of this means you should feel free to use your own id field, with no performance penalty due to the now removed auto-id handling,&nbsp;but just remember that your choice of&nbsp;id field values&nbsp;<a href="http://blog.mikemccandless.com/2014/05/choosing-fast-unique-identifier-uuid.html">does affect indexing performance</a>.
</p><h2>More indexing changes...</h2><p>Beyond these changes there have been a great&nbsp;many additional 2.0&nbsp;indexing&nbsp;changes&nbsp;such as:&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/10209" target="_blank">enabling doc&nbsp;values by default</a> (instead of using CPU- and heap-costly field data at search time), moving&nbsp;the dangerous delete-by-query API from a core API to a safe&nbsp;<a href="https://github.com/elastic/elasticsearch/issues/7052" target="_blank">plugin on top of the bulk indexing API</a>, exposing control over&nbsp;<a href="https://github.com/elastic/elasticsearch/pull/8863" target="_blank">increased stored fields&nbsp;compression while indexing</a>, better <a href="https://issues.apache.org/jira/browse/LUCENE-6668" target="_blank">compression of sparse doc values</a> and <a href="https://issues.apache.org/jira/browse/LUCENE-6030">norms</a>,&nbsp;reducing heap required when merging&nbsp;and a number of improvements in resiliency such as <a href="https://issues.apache.org/jira/browse/LUCENE-5969" target="_blank">detecting&nbsp;pre-existing&nbsp;index corruption when merging</a>.
</p><p>If you are curious about the low-level Lucene actions while indexing, add&nbsp;<code>index.engine.lucene.iw: TRACE</code>&nbsp;to your&nbsp;<code>logging.yml</code>, but be warned: this generates impressively copious&nbsp;output!
</p><p>Our&nbsp;<a href="http://benchmarks.elasticsearch.org/">nightly indexing performance charts</a>&nbsp;shows that we are generally&nbsp;moving in the right direction with our defaults&nbsp;over time: the docs/sec and segment counts&nbsp;for the defaults (4 GB) heap has caught up to the&nbsp;<code>fast</code>&nbsp;performance line.
</p>

airbus-auto-throttle-levers-banner.jpg

airbus-auto-throttle-levers-thumb.jpg

Performance Considerations for Elasticsearch 2.0 Indexing

<h2>First, some context</h2><p><a href="https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x">Hot-Warm</a> architectures are often used when we want to get the most out of our hardware. It is particularly useful when we have time-based data, like <a href="https://www.elastic.co/blog/observability-with-the-elastic-stack">logs, metrics, and APM data</a>. Most of these setups rely on the fact that this data is read-only (after ingest) and that indices can be time(or size)-based. So they can be easily deleted based on our desired retention period. In this architecture, we categorize Elasticsearch nodes into two types: 'hot' and 'warm'.
</p><p><strong>Hot nodes</strong> hold the most recent data and thus handle all indexing load. Since recent data is usually the most frequently queried, these nodes will be the most powerful in our cluster: fast storage, high memory and CPU. But that extra power gets expensive, so it doesn’t make sense to store older data that isn’t queried as often on a hot node.
</p><p>On the other hand, <strong>warm nodes</strong> will be the ones dedicated to long-term storage in a more cost-efficient way. Data on the warm nodes is not as likely to be queried often and data within the cluster will move from hot to warm nodes based on our planned retention (achieved through <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/shard-allocation-filtering.html">shard allocation filtering</a>), while still being available online for queries.
</p><p>Starting with Elastic Stack 6.3, we've been building new features to enhance hot-warm architectures and simplify working with time-based data.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte8ad3502d7fba4c1/5c98d0a5c9ab24634dcd6d86/hot-warm-features-frozen-indices-transp.png" data-sys-asset-uid="blte8ad3502d7fba4c1" alt="hot-warm-features-frozen-indices-transp.png" width="550" style="margin: auto; display: block;">
</p><p><strong><a href="https://www.elastic.co/blog/data-rollups-in-elasticsearch-you-know-for-saving-space">Data rollups</a></strong> were first introduced in version 6.3, to save storage. In time-series data, we want fine-grained detail for the most recent data. But it is very unlikely that we will need the same for historical data, where we will typically look at datasets as a whole. And this is where rollups came in, since starting at version 6.5 we can <a href="https://www.elastic.co/blog/how-to-create-manage-and-visualize-elasticsearch-rollup-data-in-kibana">create, manage and visualize rollup data in Kibana</a>.
</p><p>Shortly after, we added <strong><a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/modules-snapshots.html#_source_only_repository">source-only snapshots</a></strong>. This minimal snapshots will provide a significant reduction of snapshot's storage, with the tradeoff of having to reindex data if we want to restore and query. This has been available since version 6.5.
</p><p>In version 6.6, we released two powerful features, <strong>Index Lifecycle Management (ILM)</strong> and <strong>Frozen Indices</strong>.
</p><p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/index-lifecycle-management.html">ILM</a> provides the means to automate your indices management over time. It simplifies moving indices from hot to warm, allows deletion when indices are too old, or automates force merging indices down to one segment.
</p><p>And for the rest of this blog, we’ll talk about frozen indices.
</p><h2>Why freeze an index?</h2><p>One of the biggest pain points with “old” data is that, regardless of age, indices still have a significant memory footprint. Even if we place them on cold nodes, they still use heap.
</p><p>A possible solution could be to close the index. If we close an index, it won’t require memory, but we will need to re-open it to run a search. Reopening indices will incur an operational cost and also require the heap it was using before being closed.
</p><p>On each node, there is a memory (heap) to storage ratio that will limit the amount of storage available per node. It may vary from as low as 1:8 (memory:data) for memory intensive scenarios, to something close to 1:100 for less demanding memory use cases.
</p><p>This is where frozen indices come in. What if we could have indices that are still open —  keeping them searchable —  but do not occupy heap? We could add more storage to data nodes that hold frozen indices, and break the 1:100 ratio, understanding the tradeoff that searches might be slower.
</p><p>When we freeze an index, it becomes read-only and its transient data structures are dropped from memory. In turn, when we run a query on frozen indices, we will have to load the data structures to memory. Searching a frozen index doesn’t have to be slow. Lucene heavily depends on the filesystem cache which might have enough capacity to retain significant portions of your index in memory. In such a case searches are comparable in speed per shard. Yet, a frozen index is still throttled such that only one frozen shard is executing per node at the same time. This aspect might slow down searches compared to unfrozen indices.
</p><h2>How freezing works</h2><p>Frozen indices are searched through a dedicated, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/modules-threadpool.html#search-throttled">searched-throttled</a> threadpool. This by default uses a single thread, to ensure that frozen indices are loaded into memory one at a time. If concurrent searches are happening, they will get in the queue to add additional protections to prevent nodes from running out of memory.
</p><p>So, in a hot-warm architecture, we will now be able to transition indices from hot to warm, and then be able to freeze them before archiving or deleting them, allowing us to reduce our hardware requirements.
</p><p>Before frozen indices, to reduce infrastructure cost we had to snapshot and archive our data, which adds a significant operational cost. We would have to restore data if we needed to search again. Now, we can keep our historical data available for search, without a significant memory overhead. And if we need to write again on an already froze index, we can just <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/unfreeze-index-api.html">unfreeze it</a>.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta19e364f7d3f763a/5c98d09ec9ab24634dcd6d80/index-states-transp.png" data-sys-asset-uid="blta19e364f7d3f763a" alt="index-states-transp.png" width="400" style="margin: auto; display: block;">
</p><h2>How to freeze an Elasticsearch index</h2><p>Frozen indices are easy to implement in your cluster, so let’s get started on how to use the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/freeze-index-api.html">Freeze index API</a> and how to <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/_searching_a_frozen_index.html">search on frozen indices</a>.
</p><p>First, we'll start by creating some sample data on a test index.
</p><pre class="prettyprint">POST /sampledata/_doc
{
    "name": "Jane",
    "lastname": "Doe"
}
POST /sampledata/_doc
{
    "name": "John",
    "lastname": "Doe"
}
</pre><p>And then check that our data has been ingested. This should return two hits:
</p><pre class="prettyprint">GET /sampledata/_search
</pre><p>As a best practice, before you freeze an index it’s recommended to first run a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html">force_merge</a>. This will ensure that each shard has only a single segment on disk. It will also provide much better compression and simplifies the data structures we will need when running an aggregation or a sorted search request on the frozen index. Running searches on a frozen index with multiple segments can have a significant performance overhead up to multiple orders of magnitude.
</p><pre class="prettyprint">POST /sampledata/_forcemerge?max_num_segments=1
</pre><p>The next step is to just invoke a freeze on our index via de Freeze index API endpoint.
</p><pre class="prettyprint">POST /sampledata/_freeze
</pre><h2>Searching frozen indices</h2><p>Now that it’s frozen, you’ll see that regular searches won’t work. The reason for this is that, to limit memory consumption per node, frozen indices are&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/searching_a_frozen_index.html">throttled</a>. &nbsp;Since we could target a frozen index by mistake, we’ll prevent accidental slowdowns by specifically adding <code>ignore_throttled=false</code> to the request.
</p><pre class="prettyprint">GET /sampledata/_search?ignore_throttled=false
{
 "query": {
   "match": {
     "name": "jane"
   }
 }
}
</pre><p>Now we can check the status of our new index, by running the following request:
</p><pre class="prettyprint">GET _cat/indices/sampledata?v&h=health,status,index,pri,rep,docs.count,store.size
</pre><p>This will return a result similar to the following, with the index status being ‘open’:
</p><pre class="prettyprint">health status index      pri rep docs.count store.size
green  open   sampledata   5   1          2     17.8kb
</pre><p>As mentioned above, we must protect the cluster from running out of memory, thus there is a limit in the number of frozen indices we can concurrently load for search on a node. The number of threads in the search-throttled threadpool defaults to 1, with a default queue of 100. This means that if we run more than one request, they will be queued up to a hundred. We can monitor the&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/cat-thread-pool.html#cat-thread-pool">threadpool status</a>, to check queues and rejections, with the following request:
</p><pre class="prettyprint">GET _cat/thread_pool/search_throttled?v&h=node_name,name,active,rejected,queue,completed&s=node_name
</pre><p>Which should return a response similar to:
</p><pre class="prettyprint">node_name             name             active rejected queue completed
instance-0000000000   search_throttled      0        0     0        25
instance-0000000001   search_throttled      0        0     0        22
instance-0000000002   search_throttled      0        0     0         0
</pre><p>Frozen indices might be slower, but they can be pre-filtered in a very efficient manner. It is also recommended to set the request parameter <code>pre_filter_shard_size</code> to 1.
</p><pre class="prettyprint">GET /sampledata/_search?ignore_throttled=false&pre_filter_shard_size=1
{
 "query": {
   "match": {
     "name": "jane"
   }
 }
}
</pre><p>This will not add a significant overhead to the query and will allow us to take advantage of the usual scenario. For example, when searching on a date range on time-series indices, not all shards will match.
</p><h2>How to write to a frozen Elasticsearch index</h2><p>What will happen if we try to write on an already frozen index? Let’s go for it and find out.
</p><pre class="prettyprint">POST /sampledata/_doc
{
  "name": "Janie",
  "lastname": "Doe"
}
</pre><p>What happened? Frozen indices are read-only, so writing is blocked. We can check these in the index settings:
</p><pre class="prettyprint">GET /sampledata/_settings?flat_settings=true
</pre><p>Which will return:
</p><pre class="prettyprint">{
 "sampledata" : {
   "settings" : {
     "index.blocks.write" : "true",
     "index.frozen" : "true",
     ....
   }
 }
}
</pre><p>We have to use the&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/unfreeze-index-api.html">Unfreeze index API</a>, invoking the unfreeze endpoint on the index.
</p><pre class="prettyprint">POST /sampledata/_unfreeze
</pre><p>And now we'll be able to create a third document and search for it.
</p><pre class="prettyprint">POST /sampledata/_doc
{
 "name": "Janie",
 "lastname": "Doe"
}
GET /sampledata/_search
{
 "query": {
   "match": {
     "name": "janie"
   }
 }
}
</pre><p>Unfreezing should be done only under exceptional situations. And remember to always run a `force_merge` before freezing the index again to ensure optimal performance.
</p><h2>Using frozen indices in Kibana</h2><p>To begin with, we will need to load some sample data, like the <a href="https://www.elastic.co/guide/en/kibana/6.6/tutorial-sample-data.html">sample flight data</a>.
</p><p>Click on the “Add” button for Sample flight data.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt142abc81bf2e4ac0/5c98d104632578d5055bb49f/kibana-load-data.png" alt="kibana-load-data.png" width="700" style="margin: auto; display: block;" data-sys-asset-uid="blt142abc81bf2e4ac0">
</p><p>We should now be able to see the loaded data by clicking the “View data” button. The dashboard will be similar to this one.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt70ea061b7b398da9/5c98d0ff026179ca592ee205/flight-dashboard.png" width="100%" data-sys-asset-uid="blt70ea061b7b398da9" alt="flight-dashboard.png" style="margin: auto; display: block;">
</p><p>Now we can test freezing the index:
</p><pre class="prettyprint">POST /kibana_sample_data_flights/_forcemerge?max_num_segments=1
POST /kibana_sample_data_flights/_freeze
</pre><p>And if we go back to our dashboard, we’ll notice that the data has apparently “disappeared”.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0a68db6e72e372e7/5c98d0f9acf2956c56cb1953/empty-flight-dashboard.png" data-sys-asset-uid="blt0a68db6e72e372e7" alt="empty-flight-dashboard.png" width="100%" style="margin: auto; display: block;">
</p><p>We need to tell Kibana to allow searches on frozen indices, which is disabled by default.
</p><p>Go to Kibana Management, choose Advanced Settings. In the Search section, you will find that “Search in frozen indices” is disabled. Toggle to enable and save the changes.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf5f1d58c21e9ba65/5c98d0f249a201165fca1028/frozen-indices-kibana-settings.png" data-sys-asset-uid="bltf5f1d58c21e9ba65" alt="frozen-indices-kibana-settings.png" width="100%" style="margin: auto; display: block;">
</p><p>And the flight's dashboard will show the data again.
</p><h2>Wrapping up</h2><p>Frozen Indices are a very powerful tool in hot-warm architectures. They enable a more cost-effective solution for increased retention while retaining online search. I recommend that you test your search latency with your hardware and data, to come with the right sizing and search latency for your frozen indices.
</p><p>Check out the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/frozen-indices.html">Elasticsearch documentation</a> to learn more about the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/freeze-index-api.html">Freeze index API</a>. And as always, if you have any questions, reach out on our <a href="https://discuss.elastic.co/">Discuss</a> forums. <strong>Happy freezing</strong>!
</p>

frozen-indices-freeze-index-api.jpg

frozen-indices-freeze-index-api-small.jpg

Creating frozen indices with the Elasticsearch Freeze index API

<p>The Unix world has a variety of RAID solutions, ranging from specialized distributions, such as FreeNAS, to hardware RAID, as well as OS-level solutions such as
	<code>graid</code> on BSD and <code>mdadm</code> on Linux. These systems all report a variety of state data and metrics through a wide array of interfaces. With the release of 7.2, <a href="/products/beats/metricbeat">Metricbeat</a> now reports expanded metrics on Linux software RAID. In this post, we’ll look at the software RAID metrics available in Linux, and how to ingest them with Metricbeat.
</p><h2>Linux software RAID metrics</h2><p>Linux software RAID devices, usually created using <code>mdadm</code>, expose a variety of metric interfaces. One of these is via <code>ioctl</code>, which uses the <code>GET_DISK_INFO</code> and <code>GET_ARRAY_INFO</code> request codes. Users don’t often interact with <code>ioctl</code>, but you can see the data they return using tools like <code>mdadm --status</code>:
</p><pre class="”prettyprint”">sudo mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Tue Jun  4 15:29:29 2019
        Raid Level : raid1
        Array Size : 101376 (99.00 MiB 103.81 MB)
     Used Dev Size : 101376 (99.00 MiB 103.81 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent
       Update Time : Tue Jun  4 15:29:31 2019
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0
Consistency Policy : resync
              Name : ubuntu-bionic:0  (local to host ubuntu-bionic)
              UUID : 03183b64:f3ec0a8e:f0e1d1fc:ea0f2bba
            Events : 17
    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       34        1      active sync   /dev/sdc2
</pre><p>Another is <code>/proc/mdstat</code>, which provides a human-readable procfs interface:
</p><pre class="”prettyprint”">Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdc2[1] sdc1[0]
      101376 blocks super 1.2 [2/2] [UU]
unused devices: &lt;none&gt;
</pre><p>The final interface, and the one that Metricbeat utilizes as of 7.2, is <code>/sys/block/mdX</code>:
</p><pre class="”prettyprint”">ls -l /sys/block/md0/
total 0
drwxr-xr-x  9 root root    0 Jun  4 15:56 ./
drwxr-xr-x 11 root root    0 Jun  4 15:23 ../
-r--r--r--  1 root root 4096 Jun  4 15:53 alignment_offset
lrwxrwxrwx  1 root root    0 Jun  4 15:53 bdi -&gt; ../../bdi/9:0/
-r--r--r--  1 root root 4096 Jun  4 15:53 capability
-r--r--r--  1 root root 4096 Jun  4 15:29 dev
-r--r--r--  1 root root 4096 Jun  4 15:53 discard_alignment
-r--r--r--  1 root root 4096 Jun  4 15:53 ext_range
-r--r--r--  1 root root 4096 Jun  4 15:53 hidden
drwxr-xr-x  2 root root    0 Jun  4 15:53 holders/
-r--r--r--  1 root root 4096 Jun  4 15:53 inflight
drwxr-xr-x  2 root root    0 Jun  4 15:53 integrity/
drwxr-xr-x  5 root root    0 Jun  4 15:56 md/
drwxr-xr-x  2 root root    0 Jun  4 15:53 power/
drwxr-xr-x  2 root root    0 Jun  4 15:53 queue/
-r--r--r--  1 root root 4096 Jun  4 15:53 range
-r--r--r--  1 root root 4096 Jun  4 15:53 removable
-r--r--r--  1 root root 4096 Jun  4 15:53 ro
-r--r--r--  1 root root 4096 Jun  4 15:53 size
drwxr-xr-x  2 root root    0 Jun  4 15:53 slaves/
-r--r--r--  1 root root 4096 Jun  4 15:53 stat
lrwxrwxrwx  1 root root    0 Jun  4 15:53 subsystem -&gt; ../../../../class/block/
drwxr-xr-x  2 root root    0 Jun  4 15:53 trace/
-rw-r--r--  1 root root 4096 Jun  4 15:29 uevent
</pre><p>This sysfs interface provides <a href="https://www.kernel.org/doc/html/v4.15/admin-guide/md.html#md-devices-in-sysfs">a number of metrics</a> based on the RAID level of the array. As of 7.2, Metricbeat only reads from a few of these files: <code>size</code>, <code>md/array_state</code>, <code>md/level</code>,<code> md/dev-X/state</code>,  <code>md/sync_completed</code>, and <code>md/sync_action</code>. These last two are only available if the RAID array is redundant; that is, a 1, 4, 5, 6, or 10 array.
</p><h2>Ingesting Linux software RAID data with Metricbeat</h2><p>If you’re already using the Metricbeat system module, enable the RAID metricset. By default, the configuration is present, but commented out in <code>modules.d/system.yml</code>. Uncomment the lines to get started:
</p><pre class="”prettyprint”">#- module: system
#  period: 5m
#  metricsets:
#    - raid
#  raid.mount_point: '/'
</pre><p>If you’re running Metricbeat inside of Docker, <a href="/guide/en/beats/metricbeat/7.2/running-on-docker.html#monitoring-host">configure Metricbeat to monitor the host machine</a>, and set the <code>hostfs</code> variable in <code>system.yml</code> to reflect the sysfs mount point.
</p><p>After you configure the metricset, start Metricbeat and look for events from the RAID metricset:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta3d61732b9ae329c/5cffd97ae646a764172fe094/image2.png" data-sys-asset-uid="blta3d61732b9ae329c" alt="JSON result of looking for events from the RAID metricset">
</p><p>Metricbeat normalizes disk states into four categories: <code>active</code>, <code>total</code>, <code>spare</code>, and <code>failed</code>. The <code>states</code> object reports the raw states from the operating system. This structure allows us to keep metrics consistent as we add more platforms to the metricset. The <code>blocks</code> object will show the state of any sync action in the array, or the total block size if no action is taking place.
</p><h2>Example: Finding failed disks in Kibana</h2><p>Once you’re ingesting RAID data into Elasticsearch, it’s easy to find RAID arrays with failed disks or other problems using the Discover app.
</p><p>Just create a new filter rule using the system.raid.disks.failed field, and filter for values other than 0:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt78efbedad1f38d3f/5cffd99b6a96af7017dfe419/image3.png" data-sys-asset-uid="blt78efbedad1f38d3f" alt="Creating a new filter rule in the Kibana Discover app">
</p><p>We can now see that the md1 array has a failed disk that should be addressed:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte6ea2f0b8137b596/5cffd9b8e646a764172fe09c/image1.png" data-sys-asset-uid="blte6ea2f0b8137b596" alt="Kibana Discover app showing that the md1 array has a failed disk">
</p><h2>Wrapping up</h2><p>And it’s that simple to start monitoring your Linux software RAIDs with Metricbeat and the Elastic Stack. You can try it out in a 14-day free trial of the <a href="/cloud/">Elasticsearch Service</a>, or <a href="/downloads">download</a> and run it locally today. If you run into problems, be sure to reach out on our <a href="https://discuss.elastic.co/c/beats">Discuss forum</a>.
</p>

kibana-filter-export-raid.jpg

linux-raid-metrics-thumb.jpg

Monitoring Linux software RAID metrics with Metricbeat and Elasticsearch

Monitoring Linux software RAID metrics with Metricbeat

<p>Part of the success of Elasticsearch has probably been its large number of supported language clients right from the start. Its powerful REST API can be accessed using HTTP and JSON, which makes it easy to integrate with using any language that provides those capabilities.
</p>
<p>Nevertheless, there hasn’t been an official Elastic Java REST client for a long time. This was partly due to the existence of the  <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.6/client.html">native Java Client</a> (also sometimes called the TransportClient), that doesn’t communicate with the cluster via REST but via a binary protocol. However, there are several drawbacks of this client, like security aspects or tighter coupling to JVM and cluster versions. For more details, we recommend reading the <a href="https://www.elastic.co/blog/state-of-the-official-elasticsearch-java-clients">State of the Official Elasticsearch Java Clients</a> blog post, where we also described our future plans around the Java clients for Elasticsearch.
</p>
<h3>Enter the Low-Level Client </h3>
<p>As a first step to improve the situation, the low-level REST client was released with version 5.0 of the Elastic Stack.
 It solved the compatibility and dependency problems by providing strict access via the REST API. However, this came at the price of losing all the comforts of the Java API, which people were used to when using the transport client, like search request and query builders and response objects that are convenient to use in Java. That’s why next we started working on the high-level REST client: a client that sends requests using the low-level client, but that is also able to accept 
higher level request objects and returns response objects similar to those the native client already uses.
</p>
<h3>Let's Level Up</h3>
<p>Right after the release of version 5.0 we started working on the high-level client, with the primary goal to make migration from the current transport client easy and to initially support the core APIs index, bulk, get, delete and search. It soon became clear that in order to progress and to ease migration at the same time, we would still need the
 current search and query builders and support the current response objects. The work then mostly involved implementing parsing Elasticsearch REST responses back to Java objects, a tedious issue if you consider the richness especially of the search response with all its varying aggregation responses. One of the 
	<a href="https://github.com/elastic/elasticsearch/issues/23331">tracking issues</a> gives a bit of an insight into the work that has gone into this.
</p>
<p>For this reason, the high-level client still depends on Elasticsearch, like the Java API does today. This may not be ideal, as it still ties users of the client to depend on a certain version of Elasticsearch, but this decision allows users to migrate away more easily from the transport client. We would like to get rid of this direct dependency in the future, but since this is a separate long-term project, we didn’t want this to affect the timing of the client’s first release.
</p>
<p>The first release of the Java high-level REST client went out with the <a href="https://www.elastic.co/blog/elastic-stack-6-0-0-beta1-released">6.0.0-beta1 release</a> of the Elastic Stack. It allows you to communicate via REST with Elasticsearch nodes running 6.0.0-beta1, but it is not fully compatible with 5.x nodes. The <a href="https://www.elastic.co/blog/elastic-stack-5-6-0-released">5.6.0 release</a> of the Elastic Stack recently went out and includes the 5.6.0 version of the high-level client, which works against Elasticsearch nodes running version 5.6. It is unfortunately not compatible with previous 5.x versions of Elasticsearch.
</p>
<h3>What’s Up Next?</h3>
<p>The high-level client will eventually replace the transport client in the future, hence we invite all our Java users to try it out and migrate to it if possible. The current release supports the following <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high-supported-apis.html">Elasticsearch APIs</a>:
</p>
<p> index, get, delete, update, bulk, search, scroll and clear scroll. Support for other APIs will be added step by step, also according to the feedback received from the community. As described in our <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high-level-migration.html">migration guide</a>, it is possible meanwhile to use the low-level REST client to perform requests that are not yet supported by the high-level one.
</p>
<p>The documentation for the high-level REST client can be found <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high.html">here</a>, and its javadoc <a href="https://artifacts.elastic.co/javadoc/org/elasticsearch/client/elasticsearch-rest-high-level-client/5.6.0/index.html">here</a>. We are looking forward to your feedback! Even better: become an <a href="https://www.elastic.co/blog/elastic-stack-6-0-0-alpha1-released">Elastic Pioneer</a> and receive a special Elastic gift package as our thanks for trying out the high level client in one of the 6.0 pre-releases and report any bug you find on our GitHub page.
</p>
<p><em>Banner Image Credit: <a href="https://pixabay.com/users/Hans-2/">Hans</a></em> <em>via <a href="https://pixabay.com/photos/elevator-lift-staircase-futuristic-1271928/">Pixabay</a></em> (CC0)<br>
</p>

elevator-elasticsearch-high-level-java-client-fullbleed.jpg

The Elasticsearch Java High-Level Rest Client is Out

<table style="background: #FFFFD2;">
<tbody>
<tr>
	<td><strong>July 9, 2018:</strong> This blog has been updated to include recommendations on <a href="#not-trim"> when to <u>not trim</u> disks</a> in order to prevent major slowdowns in production environments.
	</td>
</tr>
</tbody>
</table><h2> Is your Elasticsearch "Trimmed"?</h2><p>Here at Elastic we regularly benchmark the performance of Elasticsearch. The results are <a href="https://elasticsearch-benchmarks.elastic.co/">publicly available</a>. Looking at the results, we have observed a recurring pattern of performance degradation:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt96c807239f2b4dac/5bbedf1d7498f42b0c1a67f9/Recurring_Elasticsearch_Slowness.png" data-sys-asset-uid="blt96c807239f2b4dac" alt="Recurring Elasticsearch slowness" "="">
</p><p>In the picture above we can see peak performance on 2016/09/26 gradually dropping until it recovers again on 2016/10/03, followed by another steady performance drop recovering on 2016/10/10. Have you spotted the pattern yet?
</p><table>
<tbody>
<tr>
	<td><strong>Date</strong>
	</td>
	<td><strong>Day of the Week</strong>
	</td>
</tr>
<tr>
	<td>20160926
	</td>
	<td>Monday
	</td>
</tr>
<tr>
	<td>20161003
	</td>
	<td>Monday
	</td>
</tr>
<tr>
	<td>20161010
	</td>
	<td>Monday
	</td>
</tr>
</tbody>
</table><p>Dismissing the possible influence of periodic astronomical effects, we started looking at the scheduled weekly jobs on the operating system instead, for an explanation.
</p><p>Our benchmarks run on bare metal servers, use two SSD drives in software RAID-0 configuration and have Ubuntu 16.04 installed.
</p><p>Following is the content of <code>cron.weekly</code> on a Ubuntu Server 16.04 LTS based VM. Notice the <code>fstrim</code> job.
</p><pre class="prettyprint">vagrant@ubuntu1604vbox:/etc/cron.weekly$ ls
fstrim  man-db
</pre><h2>What on earth is TRIM?</h2><p>Quoting the <a href="https://en.wikipedia.org/wiki/Trim_(computing)">Wikipedia article</a>:
</p><blockquote>A trim command (known as TRIM in the ATA command set, and UNMAP in the SCSI command set) allows an operating system to inform a solid-state drive (SSD) which blocks of data are no longer considered in use and can be wiped internally.
</blockquote><p>SSD drives are composed of groups of Flash (NAND) memory cells. Three major differences from traditional magnetic disks can affect the performance of SSD drives:
</p><ol>
	<li>Memory cells cannot be overwritten in one operation: existing data need to be erased first before they can be replaced with new data.</li>
	<li>Memory cells support a limited number of program/erase cycles.</li>
	<li>Memory cells are organized in pages and blocks and SSD drives can <strong>only erase entire blocks</strong>!</li>
</ol><p>These limitations have forced manufacturers to implement <a href="http://www.thessdreview.com/daily-news/latest-buzz/garbage-collection-and-trim-in-ssds-explained-an-ssd-primer">Garbage Collection</a> where blocks containing <em>both</em> pages with data and pages marked for deletion are emptied <em>after</em> copying pages with data to free blocks. Therefore SSD drives, internally, create more IO than explicitly generated by the OS for write operations; this phenomenon is called <a href="https://en.wikipedia.org/wiki/Write_amplification">write amplification</a>.
</p><p>When the OS gets a request to delete a file, the filesystem only updates metadata and doesn't erase the disk addresses holding the actual data. This worsens write amplification on SSD drives during GC runs.
</p><p>To address this problem, OSes can issue the TRIM command to make the SSD drive aware of erased data; this brings two advantages:
</p><ul>
	<li>Prevents unnecessary data movement (of data already erased by the OS) during GC runs
	</li>
	<li>Improves the efficiency of GC due to more free space available</li>
</ul><h2>What does it mean practically?</h2><p>It means that your SSD drives will operate faster if the filesystem passes information to them about files that have been deleted.
</p><p>This what the <code>fstrim</code> crontab process does on Ubuntu.
</p><h2>Does it really make such a big difference?</h2><p>You bet! After moving the <code>fstrim</code> cronjob (after 2016/10/17) to a daily schedule we can see the earlier graph stabilizing:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltada930ad8a473466/5bbedf207498f42b0c1a67ff/Elasticsearch_performance_after_trim_v2.png" data-sys-asset-uid="bltada930ad8a473466" alt="Elasticsearch performance after trim" "="">
</p><p>The detrimental effect of no TRIM can be seen running ten consecutive runs of a different (PMC) benchmark below. The filesystem got TRIMmed only once, right before starting the benchmarks:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt58f2b5a2fe83450f/5bbedf233dbe2e845d52ae33/ten_rally_iterations_nossd.png" data-sys-asset-uid="blt58f2b5a2fe83450f" alt="Ten Rally iterations" "="">
</p><h2 id="not-trim">Is there any reason NOT to run trim?</h2><p>While trimming disks before executing benchmarks makes sense (as one example), there may be reasons you don't want to do it on productions environments.
</p><p>Depending on how long ago disks were trimmed, disk performance may severely degrade during trim execution and the trim may take a long time to finish.</p><ul>
	<li>For Elastic Cloud Enterprise (ECE) installations, the control plane may become unstable during lengthy trims with system wide repercussions — as a result we recommend avoiding trimming ECE disks manually.</li>
	<li>In other production cases, it’s strongly recommended to schedule execution only during a quiet traffic period or, if it's constantly running hot, disable periodic trimming entirely.</li>
</ul><h2>Caveats</h2><p>On CentOS-7 the <code>fstrim.service</code> and associated systemd timer is disabled by default, so you will need to enable the service and probably override the default timer schedule as well.
</p><p>If you are using LVM and frequently perform resize operations you will need to set <code>issue_discards</code> to 1 in <code>/etc/lvm/lvm.conf</code>.
</p><p>The <code>dm-crypt</code> Linux encryption layer, if used, also requires special handling. Instructions for configuring TRIM under <code>lvm</code>/<code>dm-crypt</code> can be found at the ArchLinux <a href="https://wiki.archlinux.org/index.php/Solid_State_Drives#TRIM">wiki page</a>.
</p><p>The TRIM issue affects Cloud instances as well, if they provide direct access to SSD disks. This is the case with Instance Store-Backed Linux AMIs on <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html#InstanceStoreTrimSupport">some Amazon Instances</a>.
</p><h2>Conclusion</h2><p>Elasticsearch is an IO-intensive application. If you are running it on SSD storage you should check if TRIM is enabled and, depending on your load, adjust the frequency of the TRIM job. You will be surprised with the performance gains!
</p>

trim-elasticsearch-ssd.jpg

is-your-elasticsearch-trimmed-ssd.jpg

Is your Elasticsearch "Trimmed"?

<p><a href="https://github.com/hyperledger/fabric#releases">Hyperledger Fabric</a> is a blockchain framework by the Linux Foundation that uses some open source projects to build a modular blockchain application. And do you know what? You can monitor your Hyperledger Fabric applications with the Elastic Stack. :)
</p><p>We are going to assume a specific Hyperledger Fabric setup. With Hyperledger Fabric you can deploy your application using different "blocks" to achieve the same result. For example, you can use a Hyperledger Fabric internal implementation, Kafka, or <a href="https://github.com/etcd-io/etcd">etcd</a> for consensus operations, and you can use GoLevelDB or CouchDB as database.
</p><p>The services deployed for this example are:
</p><ul>
	<li>Apache Kafka for ordering, monitored with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-kafka.html">Metricbeat Kafka module</a></li>
	<li>Apache Zookeeper as a requirement of Apache Kafka, monitored with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-zookeeper.html">Metricbeat Zookeeper module</a></li>
	<li>CouchDB as the source of truth database, monitored with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-couchdb.html">Metricbeat CouchDB module</a></li>
	<li>Peers monitored with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-prometheus.html">Metricbeat Prometheus module</a></li>
	<li>Orderer service written in Go, monitored with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-golang.html">Metricbeat Golang module</a></li>
</ul><p>The result is a very rich dashboard with an overall view of a Hyperledger Fabric network:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0c455b80c85170db/5d3e0f08165116375f8ad5e0/hyperledger-observability-overall-dashboard-blog.jpg" data-sys-asset-uid="blt0c455b80c85170db" alt="Dashboard with overall view" style="display: block; margin: auto;">
</p><p>In the screenshot above we see a portion of a dashboard specifically built for Hyperledger Fabric. We can also see Metricbeat, Filebeat, and Heartbeat working in harmony. Each peer inside the Hyperledger Fabric network is monitored by configuring them to expose Prometheus metrics. <a href="https://www.elastic.co/guide/en/beats/heartbeat/7.2/heartbeat-getting-started.html">Heartbeat</a> is also used; in this case, Heartbeat supports ICMP, TCP, and HTTP protocols so we have configured it to monitor all HTTP endpoints that were available.
</p><p>The network is deployed in Docker so the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-docker.html">Metricbeat Docker module</a> has been configured to monitor it. But also, a Filebeat configured with <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-input-container.html">Container as input</a> (previously called Docker input) and with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/add-docker-metadata.html">add_docker_metadata</a> processor allow us to extract and enrich Docker generated logs. Because we are not using any Filebeat module to process Docker log lines, the configuration can’t be simpler:
</p><pre class="prettyprint">filebeat.inputs:
  - type: container
    enabled: true
    paths:
      - /path_to_your_docker_logs/*/*.log
    json.keys_under_root: true
output.elasticsearch:
  hosts: ["your_elasticsearch:9200"]
processors:
  - add_host_metadata: ~
  - add_docker_metadata:
      host: "unix:///var/run/docker.sock"
</pre><p>The config above points the container input in Filebeat to the logs path (in this case, they were Docker containers so it points to the Docker logs path). Docker outputs log lines as JSON so we set to true <code>json.keys_under_root</code> so that they are not “stringified.” We configure our Elasticsearch cluster and add two very useful processors: <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/add-host-metadata.html">add_host_metadata</a>, which enriches the JSON with useful data coming from the host machine where Filebeat is running; and <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/add-docker-metadata.html">add_docker_metadata</a> which, in turn, enriches the JSON with useful information about the Docker engine that is running the containers.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc0b18b2bb3e8e634/5d3e0f32165116375f8ad5e6/hyperledger-observability-container-json-data-blog.jpg" data-sys-asset-uid="bltc0b18b2bb3e8e634" alt="JSON enriched with useful Docker data" style="display: block; margin: auto;">
</p><p>Apart from the Discover tab, you can also use the Logs UI to have a never-ending stream of logs with customized columns and data:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt88ca3aff539e63e1/5d3e0f8965a8bf360426aea8/hyperledger-observability-logs-ui-blog.jpg" data-sys-asset-uid="blt88ca3aff539e63e1" alt="Logs UI with customized columns and data" style="display: block; margin: auto;">
</p><p>Finally, if logs, metrics, and uptime data playing together weren’t enough, you can also use the Infrastructure app. There you can see a different view of the infrastructure; in this case, we grouped the containers by image name to see their CPU usage.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb54a114ddfb52861/5d3e0fcc1f1c20368b7349af/hyperledger-observability-infra-ui-blog.jpg" data-sys-asset-uid="bltb54a114ddfb52861" alt="Infrastructure UI with containers grouped by image name" style="display: block; margin: auto;">
</p><p>The Elastic Stack for observability is critical for any complex distributed system, but don't forget that you must also instrument your apps with Elastic APM solution to "close the circle" of observability inside your product: logs, metrics, uptime, and tracing.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte7cf30b41ea796bf/5d3e1015cc2cb63e6348cb4d/hyperledger-observability-apm-blog.png" data-sys-asset-uid="blte7cf30b41ea796bf" alt="APM dashboard showing transaction sample" style="display: block; margin: auto;">
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt17ddfa9b106db882/5d3e104527d09d3764c6df44/hyperledger-observability-apm-txns-blog.jpg" data-sys-asset-uid="blt17ddfa9b106db882" alt="APM dashboard showing transaction graphs" style="display: block; margin: auto;">
</p><p>Bonus! Did you know that Kibana now has dark mode? If you want to give it a try, just go to Management - Advanced Settings and activate it! :) Here we can see the Logs UI in dark mode:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbc31b85a4e1952cd/5d3e10a18d62573e6d6c6f15/hyperledger-observability-dark-mode.jpg" data-sys-asset-uid="bltbc31b85a4e1952cd" alt="Kibana in dark mode" style="display: block; margin: auto;">
</p><h2>Want to see it in action?</h2><p>You can start monitoring your Hyperledger Fabric today by signing up for a free trial of the <a href="https://www.elastic.co/cloud/elasticsearch-service">Elasticsearch Service</a> on Elastic Cloud, or you can <a href="https://www.elastic.co/downloads/">download</a> it as part of the default distribution of the Elastic Stack. Grab the <a href="https://www.elastic.co/downloads/beats/metricbeat">Metricbeat</a>&nbsp;and&nbsp;<a href="https://www.elastic.co/downloads/beats/filebeat">Filebeat</a> executables appropriate for your platform and start monitoring!
</p>

hyperledger-observability-blog-banner.jpg

hyperledger-observability-blog-thumb.jpg

Observability on blockchain and the Hyperledger project

<p>We’re thrilled to announce that with the release of Elastic Cloud Enterprise (ECE) 2.3, role-based access control and authentication with external sources are now generally available. These features allow you to have multiple users whose access to the ECE platform is controlled by role. You can add users natively in ECE and connect your own directory servers or identity providers to give access to your existing users.
</p>
<p>We initially added beta support for these features in <a href="/blog/elastic-cloud-enterprise-2-2-0-released" target="_blank">ECE 2.2</a>. As well as ironing out bugs with 2.3, we’ve added support for Active Directory in addition to the existing LDAP and SAML options.
</p>
<h2>How does it work?</h2>
<p>Enabling RBAC for ECE adds a security deployment, which is a system deployment that manages all of the authentication configuration and permissions. When a user attempts to log in, ECE verifies the authentication using the security deployment, falling back to the system users if necessary. If the user successfully authenticates, ECE applies their assigned roles and translates them into fine-grained permissions that control what data each user can see and what actions they can take.
</p>
<p>Note that a user’s ECE roles are separate from whatever credentials the user holds for the deployments hosted in ECE. A user could have no access to ECE but administrative access to the hosted deployments, and vice versa.
</p>
<h2>What roles are available?</h2>
<p>ECE offers a rich set of operations, both at the platform and deployment level. In order to save administrators the effort of defining and maintaining their own role definitions, ECE provides a set of predefined roles that cover the most common use cases. These roles will be kept updated as more features are delivered, so you don’t need to worry about keeping your own definitions up to date.
</p>
<p>The roles are described below. Users can hold more than one role and combine them as needed — for example, a “Platform admin” can do anything, so there is no need to give them another role. However, a “Platform viewer” can see everything but cannot change anything, so you might choose to combine it with the “Deployments manager” role.
</p>
<h3>Platform admin</h3>
<p>This role allows a user to see all data and perform any operation in ECE in the same way as the system-level <code>admin</code> user (or <code>root</code> in ECE 1.x) that is created during the install process. This role would typically only be held by administrators that are responsible for the entire ECE platform. The “Platform” section in the UI is a good example, as it gives information about e.g. allocators and their deployments, and the ability to vacate an allocator or put it into maintenance mode.
</p>
<h3>Platform viewer</h3>
<p>This role gives view-only permission for the entire platform and hosted deployments. The associated permissions are the same as those held by the <code>readonly</code> system-level user. This is useful for automation, for example, monitoring ECE’s status.
</p>
<h3>Deployments manager</h3>
<p>This role allows a user to create and manage deployments on the platform. A user with this role can perform all actions on a deployment — scale up, scale down, configure snapshots, restart nodes, reset passwords, and more. This role does not allow a user to access any platform-level operations and resources such as deployment templates, instance configurations, allocators, system deployments, etc.
</p>
<p>This is a suitable role for anyone who has responsibility for managing deployments, but has no requirement to see platform-level information, such as development team leads.
</p>
<h3>Deployments viewer</h3>
<p>A user holding this role can view deployments, but cannot modify them in any way. This role is suitable for support staff or development team members.
</p>
<h2>Managing native users
</h2>
<p>The simplest way to get started with RBAC in ECE is to create native users. These are held in the security deployment, in the Elasticsearch native realm. They only support a limited number of attributes: username, full name, email, password, roles, and whether they are currently enabled.
</p>
<p>Click on "Users" in the navigation menu to view your authentication providers, one of which is the "Native users" profile. Opening the profile takes you to a list of all native users. The list includes the two system users that are created by the ECE installer. These users can’t be edited or deleted, and you can’t reset their passwords here — see the <a href="/guide/en/cloud-enterprise/current/ece-manage-system-passwords.html">documentation</a> for further instructions on how to reset passwords.
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6263f031f9bbb57d/5d39c1b65cb66453b127ab26/rbac-ece-2-3-native-users-demo-blog.gif" data-sys-asset-uid="blt6263f031f9bbb57d" alt="Demo of creating native users" style="display: block; margin: auto;">
</p>
<p>From the "Native users" page you can create, edit, and delete native users. These users can log in to ECE just like the system users, and their access is controlled by the roles you assign to them.
</p>
<h2>User settings page
</h2>
<p>ECE 2.3 also adds a user settings page. Click on the user icon at the top-right of the page, and click "Settings". If you’re logged in as a native user, you can edit your name and email, or change your password. If you’re logged in with a user from an external authentication provider, you’ll see a read-only page with some basic information, the name and type of the authentication profile, and the roles you hold.
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt29033bf68c6711b2/5d39c1f1c3a6d753ac37c0b5/rbac-ece-2-3-user-settings-blog.gif" data-sys-asset-uid="blt29033bf68c6711b2" alt="Demo of editing user settings" style="display: block; margin: auto;">
</p>
<h2>External authentication providers
</h2>
<p>If you already have an LDAP, Active Directory, or SAML server, you can configure ECE to use them for authentication and authorization. You can even configure multiple servers, and authentication will be attempted in the order you set <a href="/guide/en/elastic-stack-overview/7.2/realm-chains.html" target="_blank">in the same way as Elasticsearch</a>. Using your existing authentication sources means that you only have to manage users in one place. The role mapping configuration for an external provider lets you map users’ attributes to roles in ECE, so any changes to users’ attributes, such as group membership, gets picked up by ECE automatically.
</p>
<p>From the auth providers overview page, click "Add provider" and pick a type. On the next page you’ll configure the provider. Let’s walk through the configuration options for a basic LDAP setup.
</p>
<h2></h2>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb2810d6c8646f88f/5d39c2291f2f0457e07e3c9a/rbac-ece-2-3-create-ldap-provider-blog.gif" data-sys-asset-uid="bltb2810d6c8646f88f" alt="Demo of creating an LDAP provider" style="display: block; margin: auto;">
</p>
<h3>LDAP auth provider
</h3>
<p>General settings:
</p>
<ol>
	<li>Every profile has a name. Besides using it to label the profile, the name is also used to generate a realm ID. Note that once a profile is created, the realm ID never changes.</li>
	<li>At least one LDAP server must be set, including the <code>ldap:</code> or <code>ldaps:</code> protocol at the start. If you choose a DNS-based load-balancing strategy, you can only specify a single server.</li>
	<li>Choose a load-balancing strategy, keeping in mind the above restriction.</li>
</ol>
<p>Trusted certificates:
</p>
<ol start="4">
	<li>If your LDAP server is secured in such a way that clients need to hold particular SSL/TLS certificates, you’ll need to prepare a bundle file and make it available to ECE via a URL. See the <a href="/guide/en/cloud-enterprise/current/ece-create-ldap-profiles.html#ece-prepare-ldap-certificates" target="_blank">documentation</a> for more details.</li>
	<li>If your bundle is password protected, supply the password here.</li>
</ol>
<p>Bind credentials:
</p>
<ol start="6">
	<li>If credentials are required to bind to the LDAP server, you can set them here.</li>
	<li>Alternatively, if credentials are not required then click the “Bind anonymously” toggle.</li>
</ol>
<p>Search mode settings:
</p>
<ol start="8">
	<li>You can specify each of the details for a user search. See the <a href="/guide/en/cloud-enterprise/current/ece-create-ldap-profiles.html#ece-ldap-search-mode" target="_blank">documentation</a> for information on these fields. At a minimum, you’ll probably want to set the “Base DN for users”, for example “cn=users,dc=example,dc=com”.</li>
	<li>Alternatively, if you need to use a template to perform LDAP queries instead, click the “Template” radio button, and supply one or more templates.</li>
</ol>
<p>Group search settings:
</p>
<ol start="10">
	<li>Much like the search mode settings, you can configure how ECE should search for a user’s groups. You’ll probably want to set the “Base DN for groups”, for example “ou=groups,dc=example,dc=com”.</li>
</ol>
<p>Role mappings:
</p>
<ol start="11">
	<li>A user needs to hold one or more roles in order to do anything with ECE. You can specify default roles, which will be assigned to all users that successfully authenticate. For example, you could grant all users the “Deployments viewer” role, so that they could see all the deployments that are hosted in ECE, without being able to edit anything.</li>
	<li>The other method of assigning roles is through role mapping. These are simply rules that say if the user DN or group DN matches some value, then the specified role or roles are assigned. You can have as many mappings as you require — for example, a mapping that give IT Ops users “Deployment viewer”, a mapping that gives all developers “Deployment manager”, and a mapping that gives all administrators “Platform viewer”.</li>
</ol>
<p>When you’re done, click <strong>Create profile</strong>, and ECE reconfigures the security deployment. You should now log in with a variety of LDAP users in order to check that they can authenticate and that their roles are correct. You can explicitly check the roles from the user settings page, as well as by navigating through the UI and ensuring that what they are able to see and do is what you expect.
</p>
<h3>Active Directory and SAML auth providers
</h3>
<p>From a high level, the process is similar to LDAP from ECE’s perspective. You create a SAML or Active Directory auth provider, name it, specify how ECE should communicate with the server, and define the mappings that should be applied.
</p>
<p>See the documentation for a full description of configuring a <a href="/guide/en/cloud-enterprise/current/ece-create-saml-profiles.html" target="_blank">SAML auth provider</a> or an <a href="/guide/en/cloud-enterprise/current/ece-create-ad-profiles.html" target="_blank">Active Directory auth provider</a>.
</p>
<h3>REST API support
</h3>
<p>All of the above operations can also be carried out using the <a href="/guide/en/cloud-enterprise/current/ece-restful-api.html" target="_blank">REST API</a>. For example, to fetch a list of all users, even those that are currently disabled:
</p>
<pre class="prettyprint">GET /api/v1/users?include_disabled=true
</pre>
<p>Let’s say you need to give access to a new sysadmin, Sarah. You could create a new native user for her as follows:
</p>
<pre class="prettyprint">POST /api/v1/users
{
  "user_name": "sarah",
  "security": {
    "roles": ["ece_platform_admin"],
    "password": "deadb33f"
  }
}
</pre>
<p>If you then wanted to change Sarah’s access, you could send a PATCH request with just the field you want to change, in this case the roles:
</p>
<pre class="prettyprint">PATCH /api/v1/users/sarah
{
  "security": {
    "roles": ["ece_platform_viewer"]
  }
}
</pre>
<p>Finally, you could remove Sarah’s account with a DELETE request:
</p>
<pre class="prettyprint">DELETE /api/v1/users/sarah
</pre>
<p>See the <a href="/guide/en/cloud-enterprise/current/Platform_-_Configuration_-_Security_Realms.html" target="_blank">REST API documentation</a> for details and examples of the auth provider endpoints.
</p>
<h3 style="background-color: rgb(255, 255, 255);">Get started today</h3>
<p style="background-color: rgb(255, 255, 255);">For a complete list of the changes in ECE 2.3, be sure to check out the <a href="/guide/en/cloud-enterprise/2.3/index.html" target="_blank">release notes</a>, and if you want to try it out yourself, get started today with a <a href="/downloads/enterprise" target="_blank">free 30-day trial</a>.
</p>

pile-of-old-keys-full-bleed.jpg

pile-of-old-keys-thumbnail.jpg

Role-based access control and external authentication is GA in ECE 2.3

<table style="background: #FFFFD2;">
<tbody>
<tr>
	<td><strong>Important note:</strong> Please see our documentation for updated information on <a href="/guide/en/elastic-stack-overview/current/xpack-ccr.html" style="color:#0077CC">cross-cluster replication capabilities</a> in the Elastic Stack.
	</td>
</tr>
</tbody>
</table><h3>Overview</h3><p>Organizations often produce and consume data in multiple regions, sometimes within a country and sometimes globally. Regions often have local data centers to meet local security, privacy or performance requirements. Data created in one region may need to be accessed in other regions. Requirements that may play a role in determining an appropriate architecture include high availability, fault tolerance, disaster recovery, ingestion latency and search and access latency. Limited network bandwidth and high latency between data centers can be key considerations for determining a multi-region and multi-data center architecture.
</p><p>Queueing data is a central theme in distributed architectures for reasons including guaranteed message delivery, low network bandwidth and message throughput variation. The purpose of this blog is to show and discuss proposed solutions to a use case where distributed departments produce data locally that must be replicated across data centers and be made accessible for search and access.
</p><h3>Use Case</h3><p>Assume a creative agency with offices in New York and London where media assets, such as images, videos, HTML documents, CSS documents, etc are created in the respective offices and maintained in separate Digital Asset Management (DAM) systems. Media assets are shared between New York and London and are discoverable through a search user interface.
</p><p><img src="https://api.contentstack.io/v2/uploads/598883ff8853622f0911f4f7/download?uid=blt4a926926851aef57" data-sys-asset-uid="blt4a926926851aef57" alt="img1.jpg">
</p><h3><strong>Requirements</strong></h3><strong><ul>
	<li dir="ltr">Media assets are created locally at both data centers and the data must be replicated to between data centers</li>
	<li dir="ltr">Both New York's and London's indices should be eventually consistent</li>
	<li dir="ltr">Search latency is critical as media authors need the ability to rapidly discover and repurpose existing assets</li>
	<li dir="ltr">Indexing latency, while important, isn't as critical and eventual consistency across data centers is acceptable</li>
</ul><h4>Considerations</h4><p>The blog post <a href="/blog/clustering_across_multiple_data_centers">Clustering Across Multiple Data Centers</a> addresses key concerns and provides guidance around the following topics:
</p><ul>
	<li dir="ltr">High Latency Between Data Centers</li>
	<li dir="ltr">Limited or Unreliable Connectivity</li>
	<li dir="ltr">Data Availability</li>
</ul><p>This post elaborates upon the guidance by incrementally refining architectures for the proposed use case.
</p><h3>Potential Architectures</h3><h4>Single Shared Elasticsearch Cluster Distributed Across Data Centers</h4><p>One might start by defining a single shared Elasticsearch cluster where nodes are distributed between the New York and London data centers. Such a configuration is discouraged; it’s a mistake you will regret. Elasticsearch clustering was designed assuming high speed and highly reliable networking. Elastic recommends clusters to be located in the same datacenter and rack, preferably. This assumes network failures in the same data center are infrequent whereas network failures between different geographic areas are more frequent. If the Elasticsearch minimum master nodes are configured incorrectly, a network failure or even a high latency network can result in a situation where master nodes are elected at both data centers, indices become unsynchronized and data is potentially lost.
</p><h4>Independent Elasticsearch Clusters Searchable by Tribe Nodes</h4><p>An alternative architecture is to have independent Elasticsearch clusters in each data center where each cluster indexes local media assets and&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-tribe.html">Tribe Nodes</a> federate searches across New York’s and London’s indices. There are a couple potential issues with this architecture. One being, if the network connection fails, then New York content authors won’t be able to search London assets and vice-versa. Another issue being the requirement to support rapid discovery will not be achieved in a high latency network environment; search response times will be determined by the slowest data center connection.
</p><h4>Independent Elasticsearch Clusters and A Shared Kafka Cluster</h4><p>In this architecture there are independent Elasticsearch clusters in each data center where each cluster indexes assets from a common Kafka queue which is shared across data centers. &nbsp;There is a single Kafka cluster, distributed between New York’s and London’s data centers. A concern with this architecture is that Kafka uses Zookeeper for cluster management.
</p><p>Similar to the concern of a minimum master nodes misconfiguration, provisioning a Zookeeper ensemble across data centers, especially in high latency environments, can result in partitioning; see&nbsp;<a href="https://zookeeper.apache.org/doc/r3.4.1/zookeeperObservers.html">ZooKeeper Observers</a>. If the two data centers are disconnected, or if there is a just some network lag, leader election may occur. In the data center where a quorum is determined and a leader is elected, read and write operations will continue. In the non-quorum data center, reads will continue, but writes will not be possible.
</p><h4>Independent Elasticsearch and Kafka Clusters</h4><p>Given the limitations of the other architectures, an alternative is to maintain two independent environments and to synchronize data between data centers. What is required is the ability to replicate data reliably between data centers. In this architecture, each data center operates independently, allowing authors to author content into and search a local Elasticsearch cluster having a complete data set. Changes that are made in the local data center are queued locally for remote consumption.
</p><p>There are two proposed architectures for replicating local changes to the remote data center. The first is to use Kafka MirrorMaker to replicate the queues to the remote data center. The second is to use Logstash to pull data from the remote data center’s queue.
</p><p>In the first proposed architecture the local queue is replicated to the remote (mirror) data center using MirrorMaker. A local instance of Logstash pulls data from the mirror of the remote Kafka Queue. MirrorMaker is a useful tool, having a small footprint, requiring no configuration and through a single command line request, MirrorMaker mirrors data between Kafka clusters. In this architecture local Logstash instances read from the local mirrored queues.
</p><p><img src="https://api.contentstack.io/v2/uploads/59888422d4a55d5c099ad498/download?uid=blt56091b764d11e002" data-sys-asset-uid="blt56091b764d11e002" alt="Multi Data Center MirrorMaker and Logstash">
</p><p>In the second proposed architecture a simplification is made and MirrorMaker is taken out of the picture as it isn’t needed. MirrorMaker doesn’t address feeding ElasticSearch; it serves purely as a queue replication technology. There may be cases where MirrorMaker is useful, for example, if there are consumers in addition to Logstash, but in this case Logstash is the only consumer. Logstash is able to pull data from the remote data center.
</p><p><img src="https://api.contentstack.io/v2/uploads/5988842f4558bdc02f8c3ed6/download?uid=blt4d3f03fcb8aa292d" data-sys-asset-uid="blt4d3f03fcb8aa292d" alt="Multi Data Center Logstash Pull Remote">
</p><p>There are additional benefits to having independent and decoupled installations. If there is a catastrophic failure at one location the other location remains fully operational. &nbsp;The operational location continues uninterrupted. And, if required, users of the failure site can access the remote location. Additionally, the fully operational location can serve as a backup to be restored to the failure location.
</p><h3>Summary</h3><p>While Elasticsearch currently doesn’t support reliable and eventually consistent replication across a high latency network, the combination of Elasticsearch and Kafka supports the configuration of an architecture that supports the proposed use case.
</p></strong>

data-centre.jpg

Scaling Elasticsearch Across Data Centers With Kafka

elasticsearch-blog-banner.jpg

Elasticsearch Blog Thumb.jpg

What's New with Elasticsearch Ingest Node in 6.5

full-bleed-2k.jpg

thumb.jpeg

How to Develop Your Own Security Extensions and Custom Realms for Elasticsearch

<p><em>Want to learn more about the differences between the Amazon Elasticsearch Service and our official Elasticsearch Service? Visit our <a href="https://www.elastic.co/aws-elasticsearch-service">AWS Elasticsearch</a> comparison page.</em>
</p><p>On August 1, Elastic <a href="/blog/the-next-generation-elasticsearch-service-hot-warm-clusters-machine-learning-more-hardware-choices-and-new-pricing">launched some exciting new features</a> in <a href="/cloud/elasticsearch-service">Elasticsearch Service</a> on <a href="/cloud">Elastic Cloud</a> that help users better match their workloads to their cloud infrastructure. In this post we’ll take a look at how some of these new features work in Elasticsearch Service and also how they can be set up and further customized in <a href="/products/ece">Elastic Cloud Enterprise</a>.
</p><h2>Where We Came From</h2><p>Elasticsearch Service is known for its ease of use when it comes to performing everyday tasks such as creating, upgrading, configuring, and scaling Elasticsearch deployments. Creating a highly available cluster from scratch takes minutes, and growing a cluster is just a few clicks. But while Elasticsearch provides a lot of flexibility in how it can be used, the Elasticsearch Service previously provided a homogeneous experience in terms of which types of machines were used under the covers, and which Elasticsearch roles could be assigned to those machines.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta69490dfb1888a24/5c57e178a669c2e50b763c87/pre-dnt-landscape.png" data-sys-asset-uid="blta69490dfb1888a24" alt="pre-dnt-landscape.png" width="661" height="350" style="width: 661px; height: 350px;">
</p><p>Additionally, Elasticsearch Service deployments have traditionally been all about Elasticsearch and Kibana, but we wanted to allow for additional products and solutions to be included in Elastic Cloud deployments, including <a href="/products/stack/machine-learning">machine learning</a> and <a href="/solutions/apm">APM</a>, and these called for different hardware than we typically used.
</p><h2>Where We Are At</h2><p>Elasticsearch Service on Elastic Cloud now allows you to choose use case specific hardware profiles that match the way you want to use Elasticsearch. We currently offer four deployment profile options: I/O, compute, memory, and hot-warm.
</p><p>The I/O optimized profile provides a balance of compute, memory, and SSD-based storage that is suitable for general purpose workloads, or workloads with frequent write operations. The compute optimized profile is suitable for CPU-intensive workloads where more computing power&nbsp;is needed relative to storage. The memory optimized profile is ideal for memory-intensive operations, such as frequent aggregations, where more memory is needed relative to storage. And the hot-warm profile is a cost-effective solution for storing time-based logs, where indices are migrated from I/O optimized to lower cost storage optimized hardware as they age.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt508fddddb6d90730/5c57e172a4f817ef0b0e80df/select-a-template.png" data-sys-asset-uid="blt508fddddb6d90730" alt="select-a-template.png" "="">
</p><p>When choosing a profile, the underlying AWS, GCP, or Azure machine types and hardware specs are described, so you always know what you’re getting and what performance to expect.
</p><h2>Customize Your Deployment</h2><p>Choosing a hardware profile is only the first step in creating a deployment. You can also choose to customize your deployment by independently sizing and configuring the hardware on which different nodes and instances within your deployment will run. For example, for a larger deployment you may choose to configure dedicated master nodes. You can also choose to enable and run a dedicated machine learning node. And of course, you can independently specify the size and level of fault tolerance for your Elasticsearch nodes and Kibana instances.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0b0830402078b07e/5c57e0cfa4f817ef0b0e80d9/customize-deployment-data.png" data-sys-asset-uid="blt0b0830402078b07e" alt="customize-deployment-data.png" "="">
</p><p>For hot-warm deployments you can also choose to configure index curation to automatically migrate indices from I/O optimized data nodes to storage optimized data nodes after they’ve reached a specified age. This architecture is well suited for storing time-based logs, as it allows you to take advantage of lower cost hardware for storing older indices that do not need to be frequently updated, while reserving your more performant hardware for newer indices that are under heavier I/O load. The index curation configuration allows you to specify a pattern to match the names of any indices you wish to be curated, along with the time period after which curation should take place for a new index.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltca9f26f42dfc4e7c/5c57e15a8d25f6030ca04359/index-curation-config.png" data-sys-asset-uid="bltca9f26f42dfc4e7c" alt="index-curation-config.png" "="">
</p><h2>Optimized Performance</h2><p>However you customize your deployment, the machines that are used under the covers are matched to the hardware profile you choose. Overall, Elasticsearch Service now leverages a wider variety of hardware for deployments, and for the different roles within a deployment, providing a more optimized and cost-effective experience across the board.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb29aa716a54ce030/5c57e0ac713ebdec0ba06f8d/post-dnt-landscape.png" data-sys-asset-uid="bltb29aa716a54ce030" alt="post-dnt-landscape.png" width="663" height="353" style="width: 663px; height: 353px;">
</p><p>One of the optimizations that this separation of responsibilities onto different hardware allows for, is the ability for Elastic Cloud to route an Elasticsearch request that enters our public endpoint directly to the node that is most likely to process it. For example, read requests such as searches can be routed directly to data nodes. Write requests can be routed directly to ingest nodes. And cluster configuration requests can be routed directly to the master node.
</p><h2>A Peek Behind the Curtain</h2><p>The features we’ve discussed here represent a new experience for users of the Elasticsearch Service on Elastic Cloud, but there’s another side of this functionality that a cloud administrator will experience, and in particular, an Elastic Cloud Enterprise administrator. The customization features introduced to Elasticsearch Service are set to be included in the next <a href="/products/ece">Elastic Cloud Enterprise</a> release, so let’s take a peek at how things will work from the cloud administrator’s perspective.
</p><p>One of the first things an Elastic Cloud Enterprise administrator will notice in the next release is the ability to characterize available hardware for Elastic Stack deployments by tagging it. Tags are simple ways of describing the hardware, and are used later on to help determine which nodes and instances should be placed on what hardware.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2571f4d7d8c352ca/5c57e142a209c1ca0be9dcdf/allocator-tagging.png" data-sys-asset-uid="blt2571f4d7d8c352ca" alt="allocator-tagging.png" width="449" height="217" style="width: 449px; height: 217px;">
</p><p>Next, an administrator can create instance configurations that represent the use of hardware for particular nodes or instances, or for a particular use case. The first part of this configuration involves expressing which machines you’d like the configuration to match. This is done by building an expression that matches the tags on machines.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt994df3fa0c358c7f/5c57e13ca669c2e50b763c81/allocator-filter.png" data-sys-asset-uid="blt994df3fa0c358c7f" alt="allocator-filter.png" "="">
</p><p>You can then specify the instance and node types that the configuration should support, as well as the supported memory and disk sizing for nodes or instances that use this configuration.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd990d43e1e221761/5c57e135516e21cf0b2a0fbb/instance-config-types.png" data-sys-asset-uid="bltd990d43e1e221761" alt="instance-config-types.png" "=""><br>
	<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt571f13cb440bbcf4/5c57e1300da9acd10bcc8e82/instance-config-sizing.png" data-sys-asset-uid="blt571f13cb440bbcf4" alt="instance-config-sizing.png" "="">
</p><p>Finally, give your instance configuration a name that represents what it should be used for.
</p><h2>Tying it all Together</h2><p>Having constructed some configurations to represent different use cases for different instance and node types, Elastic Cloud Enterprise administrators will have the ability to assemble them together into a template that can be used for creating entire deployments.
</p><p>Creating a template is similar to creating a deployment but with a bit more power. To start, you can choose which types of nodes or instances, along with which instances configurations, are eligible to be included in deployments created from the template. The instance configurations you choose will drive the compatible sizes for each node or instance, and also will affect the hardware that they’re matched to. You can specify default sizing and fault tolerance for each of the nodes and instances in the deployment, and you can also configure default settings or plugins for deployments.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd82e9d494b941e23/5c57e11fc545afde0bc70874/create-template-1.png" data-sys-asset-uid="bltd82e9d494b941e23" alt="create-template-1.png" "="">
</p><p>Hot-warm style deployment templates can be created by adding multiple data configurations to a template, and when multiple data configurations are present you can also specify index curation settings.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd63f143f7b42c48f/5c57e0ee02ac90e80ba0a8e5/create-template-index-curation.png" data-sys-asset-uid="bltd63f143f7b42c48f" alt="create-template-index-curation.png" width="633" height="255" style="width: 633px; height: 255px;">
</p><p>Give your template a name and now it’s ready to be used for creating deployments.
</p><h2>Summary</h2><p>The new features in <a href="/products/elasticsearch/service">Elasticsearch Service</a> on Elastic Cloud allow you to customize your deployments to match your use cases, including placing nodes and instances on different hardware as desired, and allowing users to create more efficient, cost-effective deployments than ever before. This same experience will be coming to soon to Elastic Cloud Enterprise, and will allow cloud administrators to more effectively manage their infrastructure than previously possible. We hope you <a href="/cloud/elasticsearch-service/signup">try out </a>these new features and tell us what you think.
</p><p><em>Editor’s Note (September 3, 2019): This blog has been updated to include Microsoft Azure as a cloud provider for Elasticsearch Service on Elastic Cloud. For more information, please read the <a href="/blog/elasticsearch-service-on-elastic-cloud-now-available-on-microsoft-azure">announcement blog</a>.</em></p>

dnt_bannerv2.jpg

dnt_thumb.jpg

Stretching the Cloud: Flexibility in Elastic Cloud Deployments

blog-banner-apple-sliced.jpg

blog-thumb-apple-sliced.jpg

Mac system extensions for threat detection: Part 2

We've recently finished adding a number of layers to Elastic Maps Service, which completes the European continent for all second-level national boundaries.

<p>At Elastic, we are adding data <em>layers</em> to our <a href="https://www.elastic.co/elastic-maps-service">Maps Service</a> on a regular basis. We are proud to announce that we have recently finished adding a number of layers that complete the European continent for all second level national boundaries. The list of new layers are Albania, Andorra, Bosnia and Herzegovina, Bulgaria, Czechia, Greece, Greenland, Iceland, Latvia, Liechtenstein, Lithuania, North Macedonia, Moldova, Montenegro, Romania, Serbia, and Ukraine. With this release, the total number of available datasets in the Elastic Map Service is <strong>65</strong>!&nbsp;</p><p>In addition to having the layers available for Kibana region map visualizations and the Maps app, this data is all available at our <a href="https://maps.elastic.co/">Elastic Maps Service</a> page, where you can explore and download individual datasets for free. This is not just the boundaries and their geographical extent, but also the alphanumeric data associated with each boundary: their <a href="https://www.iso.org/obp/ui/#search">ISO codes</a> and any other widely used region identifier, and their name representation in English and any other official languages for that country in particular.</p><div class="embed-responsive embed-responsive-16by9 shadow m-t-40 m-b-40"><video  autoplay="autoplay" playsinline="playsinline" muted="muted" loop="loop"><source src="https://assets.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt808f7a84c9148f5d/5e62b026a9742c061bdae127/maps-europe-layer-update.mp4" type="video/mp4" /></video></div><figcaption>North Macedonia municipalities are available with labels in English, Macedonian, and Albanian.</figcaption><p>The data is curated using <a href="https://openstreetmap.org">OpenStreetMap</a> and <a href="https://www.wikidata.org">Wikidata</a> databases, two well-known open data repositories that are maintained by companies and individuals all across the world, including us!</p><p>The layers added to Elastic Maps Service are meant to be used by our community joining those geographies to their business data; all you need to do is use <a href="https://www.iso.org/obp/ui">ISO codes</a> as term join identifiers, unless other stable identifiers are available. For example, the Logstash <a href="https://www.elastic.co/guide/en/logstash/7.6/plugins-filters-geoip.html">Geoip filter</a> and Elasticsearch <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.6/geoip-processor.html">Geoip processor</a> will include these ISO codes automatically to help you aggregate your data by country or region.</p><p>Downloading and indexing the datasets, instead of using the Elastic Maps Service boundaries source, can be useful for those that want to leverage this information directly in their cluster, or for those that don’t have access to EMS on air gapped environments. <a href="https://www.elastic.co/guide/en/kibana/7.6/geojson-upload.html">GeoJSON upload</a> would be a great resource to get this data quickly and ready to be used for joins, data enrichment, etc., but you can also upload them using the GDAL tool ogr2ogr as we covered in our <a href="https://www.elastic.co/blog/how-to-ingest-geospatial-data-into-elasticsearch-with-gdal">Ingest geospatial data into Elasticsearch with GDAL</a> blog.</p>We hope you’re as excited about these new boundary layers as we are. And if you haven’t used <a href="https://www.elastic.co/maps">Elastic Maps</a>, take it for a spin with a <a href="https://www.elastic.co/elasticsearch/service">free 14-day trial of Elasticsearch Service</a>.

map-service-europe-update-blog-banner.jpg

map-service-europe-update-blog-thumb.png

Europe regions are complete on Elastic Maps Service

<p>As we enter 2017, there has been one particular user experience (UX) trend that designers can no longer ignore: chatbots.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3abb97e966ee23ce/5dfc020d973ece5c8871325c/intelligent-assistant-pt1-graphic-blog.jpg" data-sys-asset-uid="blt3abb97e966ee23ce" alt="intelligent-assistant-pt1-graphic-blog.jpg" style="display: block;margin: auto;"/></p><p>While some <a href="http://www.businessinsider.com/businesses-are-skeptical-about-chat-bots-2016-4">debate their practicality</a>, others believe it to be the <a href="https://medium.com/@tedlivingston/the-future-of-chat-isn-t-ai-b07f65bc252#.v85ob9yyw">internet’s next wave of innovation</a>.&nbsp; And despite some noticably <a href="http://www.businessinsider.com/microsoft-deletes-racist-genocidal-tweets-from-ai-chatbot-tay-2016-3">terrible attempts at chatbot implementation</a>, there is a huge surge of virtual assistants coming your way in 2017, whether we like it or not.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt47a0d9dcd8bcc47b/5dfc04cac1df3b5a1f071395/intelligent-assistant-pt1-clippy-blog.png" data-sys-asset-uid="blt47a0d9dcd8bcc47b" alt="intelligent-assistant-pt1-clippy-blog.png" style="display: block;margin: auto;"/></p><figcaption>The Original Digital Assistant</figcaption><p>Most discussions on chatbots emphasize the underlying artificial intelligence aspect, but the UX component is equally important.&nbsp; Chatbots need to be effective in capturing a user’s needs, and quickly deliver the expected results without added complication or the use of complex methods to input commands.&nbsp; This is true of any successful application, where the ease of workflow is seamless. As Joe Sparano noted, ‘Good design is obvious, great design is transparent’. For designers, ‘a simple workflow’ is the result of countless hours of research, implementation, and expert feedback, repeated in an endless loop. If the chatbot is more of a nuisance than an assistant, ignoring instead of augmenting the natural workflow, users skeptical of chatbots will view the technology as yet another tech fad that is over-hyped and under delivers, and would be quickly cast aside.</p><p>Knowing well the promise but also potential landmines associated with chatbots, we began the journey of designing our new intelligent assistant for cyber defense, Artemis. I’ll briefly walk through our use case and decision to integrate a chatbot into our platform&nbsp; (which will be expanded upon in a follow-on post), and the key design principles we implemented and researched along the way.</p><h2>Is a Bot For You?</h2><p>Infosec is non-intuitively a natural use case for a chatbot. Infosec operators and analysts generally are overwhelmed with data, often have limited time and resources, and need their questions answered ASAP. As UX designers, we understood the key pain points and requirements for the analysts and operators who would be using our platform. These included anything from alert fatigue to data navigation to eliminating the necessity to learn complex and proprietary machine languages to perform even the most basic of tasks. In addition to significant design innovations throughout the interface, a chatbot seemed like a logical component to augment the user’s workflow.</p><p>As we designed Artemis, we constantly sought best practices and design principles to ensure Artemis augments the SOC analyst. Let’s take a look at some of our key take-aways that are relevant for anyone who wants to jump on one of the year’s hottest tech trends and ensure it truly augments the user’s experience.</p><h2>Creating a Successful Bot Workflow</h2><p>Below are several notable design interactions and useful guidance our UX team has integrated while creating Artemis.</p><p>There’s no need to reinvent the wheel</p><p>Chat is a pretty well-defined medium. There are countless apps in the market today that pull from preexisting chat paradigms.&nbsp; Every user is familiar with the basics of call and response: The chatbot asks a question and in return you provide a response. We built upon this established foundation to instead focus our efforts on improving optimization to the user.<span></span></p><p><strong>Provide your user guide rails</strong></p><p>Chatbots can be characteristically ambiguous without proper guidance. Let’s imagine the user’s first interaction with an unprompted chatbot - they will either be confused about what exactly they should type or confused about what the chatbot can offer them. From the very start, provide your users a short introduction that bridges directly to quick call-to-actions. While we allow the user to engage in simple conversation with the Artemis, we purposely drive each starting conversation with contextual call-to-actions that map to and are relevant for the specific view within the interface. Start thinking of ways to subtly sprinkle ‘guide rails’ throughout your chat UI to enable your user to take action, such as those in the graphic below.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltcd96337b638b1e9f/5dfc02766ab6275c8e16c9e7/intelligent-assistant-pt1-guide-rails-blog.png" data-sys-asset-uid="bltcd96337b638b1e9f" alt="intelligent-assistant-pt1-guide-rails-blog.png" style="display: block;margin: auto;"/></p><figcaption>Guard Rails Provide Context for Users</figcaption><p><strong>Get to the point&nbsp;&nbsp;</strong></p><p>When designing a utility chatbot, the user will most likely have a goal in mind the moment they initiate a conversation.&nbsp; Your user could be ordering flowers, booking a hotel, or in our particular case, gathering sensor information around active endpoints to look for anomalous activity in their environment.&nbsp; It’s important to remember to avoid letting conversation get in the way of action. We want to allow the user to drive as fast and efficiently as possible. The last thing they want to do is navigate through rhetorical questions and ‘cute’ persona-based responses. And unlike booking a hotel, our particular use case has users interacting with the chatbot throughout the day, gathering information on their endpoints or triaging suspicious activity in their environment. Having the chatbot initiate even just a few lines of human-like rhetoric can be quite annoying after your fiftieth query prompt. In cybersecurity, even seconds matter - there is no room for cuteness when the proper interaction may impact the dwell time of malicious activity.</p><p><strong>Don’t design for the edge case&nbsp;&nbsp;</strong></p><p>Designing for every interaction can be a cumbersome task when creating a chatbot. It can be quite daunting to try to account for every input from a user and provide a custom response back.&nbsp; In fact, the more branches you open in a conversational tree, the higher likelihood your user will hit a dead end. For a <a href="https://www.inc.com/christine-outram/5-chatbots-well-all-be-building-in-2017.html">utility model</a>, structure your user’s expected input until you have reached safe place to explore in plain text.</p><p><strong>Provide a way out of the rabbithole&nbsp;&nbsp;</strong></p><p>Expect the user to make mistakes. Your chatbot may have correctly guided your user down a logical path of action, but your user may not have started with a set of correct base facts.&nbsp; Build in a set of interactions for the user to make changes, walk back up the conversational path, or completely restart the conversation over again.</p><p><strong>Rethink the role of traditional design</strong></p><p>Are you a designer working on a chatbot? Great! Now roll up your sleeves, because we are now going outside the comfort of pixel pushing and into writing conversational dialog. Most conversational chatbots try to mirror human speech, while assigning it an identity that pushes the company’s brand identity. For instance, our users are often purpose driven to a particular set of unique tasks. This required our designers to create a dialog that both educates the user on their system’s environment and optimizes the speed at which they can set tasks, while also maintaining our brand identity.&nbsp; So it goes to show, as a designer, you should be heavily involved in the writing process. Your chatbot’s conversational skills are only as strong as your writing. Writing purposeful and engaging conversations to your user is now your UX. Lorem ipsum is no longer your friend (<a href="http://thenextweb.com/dd/2015/04/09/why-designers-should-never-use-fake-text/">but really, it never was</a>).</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6beb4e69ee72e0ac/5dfc031bc80d975c7c97fa6b/intelligent-assistant-pt1-conversation-blog.png" data-sys-asset-uid="blt6beb4e69ee72e0ac" alt="intelligent-assistant-pt1-conversation-blog.png" style="display: block;margin: auto;"/></p><figcaption>Beyond Lorem Ipsum</figcaption><p><strong>Incorporate feedback loops</strong></p><p>As UX designers, we are always thinking how can we optimize the experience for the user.&nbsp; Traditional user-research heavily involves understanding your customer’s need through a variety of <a href="https://www.nngroup.com/articles/which-ux-research-methods/">user research methods</a>.&nbsp; Defining what works and what doesn’t is the sole foundation for great design.</p><p>An easy UX win, when designing any chatbot, is to include a <a href="http://www.topbots.com/best-way-boost-chatbot-bot-engagement-retention/">feedback loop</a> at the end of every successful (or unsuccessful) interaction.&nbsp; Your chatbot may save an entire history of conversational information, but providing a purposeful feedback outlet to the user can cut the root of the problem. After the user completes a successful query, the chatbot can prompt the user to optionally input feedback on the most recent conversational path.&nbsp; A simple prompt from your chatbot could capture frustrations from the user, workflow gaps in our design or even bugs.&nbsp;&nbsp;</p><h2>To Bot or Not?</h2><p>While some are <a href="https://www.forbes.com/sites/tomaslaurinavicius/2016/12/04/ux-trends-2017/#2c8009ac7c94">betting</a> on 2017 as the year of the chatbot, it is important to remember that chatbots take an extreme amount of thought and process. It’s easy to make a terrible experience for the user, and extremely difficult to get it right without the proper design principles. The most important first step is defining your business needs and deciding whether your user truly needs a bot. If the answer is yes, be sure to keep in mind the design principles summarized above.&nbsp;&nbsp;</p><p>With those in place, the next step is to gather key lessons learned from those in other industries. In the upcoming weeks, I’ll address those lessons learned and apply them to our specific use case in endpoint security. And remember, as UX designers, we never compromise at the expense of the user. We know from experience that often times it takes a whole lot of effort to create the effortless.&nbsp;</p>

blog-banner-generic-black.png

blog-thumb-generic-black.png

Designing the intelligent assistant: Part 1, design principles

2000x415.jpg

720x480.jpg

Building real-time address search with the Australian G-NAF dataset

<p>Welcome to part 2 of our multi-part <a href="https://kafka.apache.org/">Apache Kafka</a> and Elastic Stack post. In our <a href="/blog/just-enough-kafka-for-the-elastic-stack-part1">previous post</a>, we introduced use cases of Kafka for the Elastic Stack and shared knowledge about designing your system for time based and user based data flow. In this post, we'll focus on the operation aspects: tips for running Kafka and&nbsp;Logstash in production to ingest massive amounts of data.<span></span></p>
<h2><strong>Capacity Planning</strong></h2>
<p>Before we dive deep, a reminder that we are mostly talking about Kafka 0.8, and Logstash 2.x which is the current stable version. There are newer versions of Kafka — 0.9 and recently, 0.10, but the core concepts discussed here can be applied to any Kafka versions. Without further ado, let’s start by discussing the different systems at play here:</p>
<p><strong>Apache ZooKeeper: </strong>Kafka has a dependency on <a href="https://zookeeper.apache.org/">ZooKeeper </a>(ZK) — brokers need it to form a cluster, topic configuration is stored in ZK nodes, etc. Plus, in version 2.x of Logstash, the input offsets are stored in ZK as they get acknowledged. Newer versions of Kafka have decoupled the clients — consumers and producers — from having to communicate with ZooKeeper. In Kafka 0.9 and 0.10, offsets are stored in topics by default instead of in ZK. Either way, you still need ZooKeeper to run Kafka brokers. Our general advice is to run 3 ZK instances to achieve a quorum configuration, and all of them on separate hardware. For more information on operationalizing ZK, refer to <a href="http://kafka.apache.org/documentation.html#zk">this excellent section</a> in Kafka docs. From our experience, ZK itself does not need much hand-holding once set up. You just have to make sure the instances are up and are monitored.<br /></p>
<p><strong>Kafka Brokers: </strong>Number of Kafka brokers you need typically depends on data retention and replication strategy. The more brokers you add, more data you can store in Kafka. In terms of resources, Kafka is typically IO bound. Performance will be limited by disk speed and file system cache — good SSD drives and file system cache can easily allow millions of messages/sec to be supported per second. You can use <a href="/downloads/beats/topbeat">topbeat</a> to monitor these information.<br /></p>
<p><strong>Logstash: </strong>How many Logstash instances do you need to process the data in Kafka? It is really hard to magically place a number for this, because frankly, it depends on a lot of variables. Questions like: how many filters do you have? How expensive are your filters, needs to be answered. Remember, it is really easy to end up with a complex Grok pattern with multiple conditionals to process your data! What is your volume of data you expect? What are all your outputs? As you see, there's a lot of information we need to gather before providing a number. Often times, it is the outputs (external systems) where you have to focus your capacity planning, not Logstash itself! That being said, you can easily scale Logstash and Elasticsearch horizontally. Therefore, our advice is to start small, and continue to add nodes or new LS instances as your data needs grow.<br /></p>
<p>In particular, for data in Kafka that Logstash consumes, you can group multiple instances into consumer groups. Each group shares the load and instances will handle data exclusively, i.e. messages will be consumed only once by one client in the group. This design lends very cleanly to our original proposition — start small and scale iteratively. Using topics, you can design your workflow such that data which needs more complex transformations, or data that needs to be stored in a slower output is isolated from other fast-moving data. Remember, in Logstash, a single slow output can block all other outputs which are configured to run after it.<br /></p>
<p><strong>Elasticsearch:&nbsp;</strong>As we've mentioned before, Elasticsearch is truly elastic in that you can scale up easily. Capacity planning for Elasticsearch is an entire blog&nbsp;post by itself, and beyond the scope of this article. We recommend you read the following posts which cover the concepts of&nbsp;scaling and sizing Elasticsearch --&nbsp;<a href="/blog/found-sizing-elasticsearch">Sizing Elasticsearch</a>, <a href="/blog/performance-indexing-2-0">Performance Considerations for Elasticsearch Indexing</a>, and <a href="/blog/category/solutions">others</a>.&nbsp; &nbsp;<strong>&nbsp;</strong></p>
<h2><strong>Data Retention</strong></h2>
<p>If your Kafka instance is running out of disk space, chances are that your retention time for Kafka logs is too high. In Kafka, you can configure data retention based on 2 criteria: age and size, using log.retention.bytes and log.retention.hours broker settings respectively. If either one of these criteria is met, Kafka broker will start deleting messages starting from the oldest, regardless of whether Logstash has consumed it.</p>
<p>It is tempting to design data recovery and retention for Elasticsearch by using Kafka's retention tools. From our experience, it is best to use a tool like <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html">Curator</a> to manage Elasticsearch's time-based indexes, in addition to configuring a snapshot strategy for restoring indexes on any catastrophic failures. Most often, data in Kafka is raw, unfiltered content, and has multiple destinations, so it is good to not have it tightly coupled to one downstream component.<br /></p>
<h2><strong>Offset Management and Message Delivery Guarantees</strong></h2>
<p>From Kafka’s documentation:</p>
<p style="margin-left: 20px;" rel="margin-left: 20px;"><em>The messages in the partitions are each assigned a sequential id number called the offset that uniquely identifies each message within the partition. Offset is controlled by the consumer: normally a consumer will advance its offset linearly as it reads messages</em><br /></p>
<p>Kafka input keeps track of offset information using ZooKeeper. As Logstash pulls messages from the topic and processes it, it periodically commits to ZK. This process is called check-pointing or committing. By default, Logstash check-points to ZK every minute. You can control this frequency by using the auto_commit_interval_ms setting. Be aware that longer times for this setting could lead to data loss in case Logstash is forcefully stopped, or the process crashes. On the other hand, small time value means increase writes per client, which could overwhelm the ZK cluster.<br /></p>
<p>If you restart Logstash, it will first read the offset information stored in ZK and start fetching messages from the previous commit point. Kafka is designed to follow <em>at-least-once</em> semantics — messages are guaranteed to be not lost, but may be redelivered. This means there could be scenarios where Logstash crashes, while the offset is still in memory, and not committed. This <em>can</em> cause messages to be re-delivered, or in other words, duplicated. If this is a concern in your use&nbsp;case, you can workaround the potential duplication by generating/using unique IDs in a field for your messages. Whether you write your own code to create these IDs, or use the<a href="http://www.elastic.co/guide/en/logstash/current/plugins-filters-uuid.html"> uuid</a> filter in Logstash, you will have to do this prior to the messages entering Kafka. On the "shipper"&nbsp;side of Logstash, you can map this event ID to<a href="http://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id"> document_id</a> option in the Elasticsearch output plugin. This means Elasticsearch will overwrite the indexed document which has the same ID, which is generally preferred over producing multiple documents with same content!<br /></p>
<p>This is also useful if you ever have to replay content, should you lose data downstream. You can use a different consumer group to replay data at its own pace.<br /></p><pre class="prettyprint">input {<br />  kafka {<br />    zk_connect =&gt; "kafka:2181"<br />    group_id =&gt; "logstash"<br />    topic_id =&gt; "apache_logs"<br />    consumer_threads =&gt; 16<br />  }<br />}<br />....<br />output {<br />  elasticsearch {<br />    document_id =&gt; "%{my_uuid}"<br />  }<br />}<br /></pre>
<p>Where my_uuid is an existing field in the Event.<br /></p>
<h2><strong>Monitoring: How do I know if there is a&nbsp;lag?</strong></h2>
<p>One of the important things to monitor when using Kafka is how many messages are backed up and waiting to&nbsp;be consumed by Logstash. There are plenty of tools to monitor this information; below are some options:<br /></p>
<h3>CLI tool bundled with Kafka</h3>
<p>Simple, command line tool to check offsets. You can run it on a cronjob periodically and alert using your favorite alerting software.<br /></p><pre class="prettyprint">/usr/bin/kafka-consumer-offset-checker --group logstash --topic apache_logs --zookeeper localhost:2181<br /></pre>
<p>Sample response:<br /></p><pre class="prettyprint">Group    Topic       Pid Offset  logSize Lag     Owner<br />logstash apache_logs 0   145833  300000  154167  none<br />logstash apache_logs 1   145720  300000  154280  none<br />logstash apache_logs 2   145799  300000  154201  none<br />logstash apache_logs 3   146267  300000  153733  none<br /></pre>
<p>The Lag column tells you how many messages you are lagging by.<br /></p>
<h3><strong>JMX</strong></h3>
<p>Kafka can be easily monitored via JMX with <a href="https://docs.oracle.com/javase/8/docs/technotes/guides/management/jconsole.html">JConsole</a>. To attach JMX to monitor Logstash, you can set these extra Java options before starting Logstash:<br /></p><pre class="prettyprint">export LS_JAVA_OPTS="<br /> -Dcom.sun.management.jmxremote.authenticate=false<br /> -Dcom.sun.management.jmxremote.port=3000<br /> -Dcom.sun.management.jmxremote.ssl=false<br /> -Dcom.sun.management.jmxremote.authenticate=false"<br /></pre>
<p>If you are running on AWS, don't forget to use the external hostname or IP of the server.<br /></p><pre class="prettyprint">-Djava.rmi.server.hostname=ec2-107-X-X-X.compute-1.amazonaws.com<br /></pre>
<h3><strong>Elastic Stack</strong></h3>
<p>Why yes, you can monitor Kafka&nbsp;by using the Elastic Stack itself. Obviously, its our favorite option! For this specific case, we use the purpose built <a href="/beats">Beat</a>&nbsp;called Kafkabeat which is written by our very own <a href="https://github.com/gingerwizard">Dale McDiarmid</a><strong>. </strong>This Beat<strong> </strong>gathers offset and other topic&nbsp;information and stores it in&nbsp;Elasticsearch. You can then analyze consumer lag using Kibana. Coupled with <a href="/downloads/beats/topbeat">topbeat</a> that captures system level stats like disk throughput, CPU and memory, we have a powerful solution to monitor Kafka. So now you have all the data in one place&nbsp;to&nbsp;convince your boss to replace those old crufty spinning disks with brand new SSDs! Oh, and in 5.0.0, it gets even better. All this critical information -- application and system level monitoring -- gets rolled into one Beat,&nbsp;called <a href="/blog/beats-5-0-0-alpha3-released">Metricbeat</a>.&nbsp;Savvy?&nbsp;</p>
<p><span></span>Ok, back to Kafkabeat for now. Here's how you get&nbsp;started with it:</p>
<ol>
  <li>Clone <a href="https://github.com/gingerwizard/kafkabeat">https://github.com/gingerwizard/kafkabeat</a></li>
  <li>Run make inside kafkabeat directory.</li>
  <li>Deploy Kafkabeat to your Kafka brokers and run it by ./kafkabeat -c kafkabeat.yml</li>
</ol>This will collect offset information from all Kafka topics for this broker and index into Elasticsearch with the following document structure<pre class="prettyprint">"@timestamp": "2016-06-22T01:00:43.033Z",<br />  "beat": {<br />    "hostname": "Suyogs-MBP-2",<br />    "name": "Suyogs-MBP-2"<br />  },<br />  "type": "consumer",<br />  "partition": 0,<br />  "topic": "apache_logs_test",<br />  "group": "logstash",<br />  "offset": 3245<br />  "lag": 60235<br />}<br /></pre>
<p>Once the data is in Elasticsearch, it is straightforward to visualize it with Kibana.&nbsp;I've used the hot-off-the-press&nbsp;<a href="/blog/kibana-5-0-0-alpha3">Kibana 5.0.0-alpha3</a>&nbsp;to create a dashboard to&nbsp;graph&nbsp;consumer lag field over timestamp.<br /></p>
<p><img src="https://api.contentstack.io/v2/assets/576b138a173b7f5d50603618/download?uid=blt047b13b470bf1421" data-sys-asset-uid="blt047b13b470bf1421" alt="Kibana_Kafka2.png" style="width: 719px;height: 539px;" width="719" height="539" /></p>
<h3>Kafka Manager<br /></h3>
<p>This is<strong> </strong>an open source <a href="https://github.com/yahoo/kafka-manager">UI tool</a> to manage Kafka&nbsp;in its entirety. You can create topics, track metrics, manage offsets and more. Be aware that this tool takes a while to compile and build, but if you need an end-to-end management solution for Kafka&nbsp;you can give this a try! Once you start the Kafka Manager tool, follow instructions to create a new cluster to monitor&nbsp;by pointing it to your ZK instance.&nbsp;</p>
<p><strong>Consumer View</strong></p>
<p><img src="https://api.contentstack.io/v2/assets/576b1383173b7f5d50603613/download?uid=bltc03906c98ff40ede" data-sys-asset-uid="bltc03906c98ff40ede" alt="kafka UI.png" style="width: 580px;height: 316px;" width="580" height="316" /></p>
<p><strong>Topic View</strong></p>
<p><img src="https://api.contentstack.io/v2/assets/576b137c173b7f5d5060360d/download?uid=bltd8755b986c0532c8" data-sys-asset-uid="bltd8755b986c0532c8" alt="Kafka UI 2.png" style="width: 577px;height: 396px;" width="577" height="396" /></p>
<h2><strong>Conclusion</strong></h2>
<p>In this post, we provided tips to operationalize Kafka and Logstash so you can ingest data from multiple sources into Elasticsearch. There's more to come!</p>
<p>Last year, Kafka released <a href="http://www.confluent.io/blog/apache-kafka-0.9-is-released">0.9.0</a> version, and recently <a href="http://www.confluent.io/blog/announcing-apache-kafka-0.10-and-confluent-platform-3.0">0.10.0</a> which is packed with new features like inbuilt security, new consumer implementation, data quotas and more. We've updated Logstash input and output so you can use these features with Logstash! In the next post, we'll cover the new features in Kafka, and in particular, end-to-end security using Kafka and the Elastic Stack.</p>
<p>'till next time!<br /></p>

Kibana_Kafka1.png

just-enough-kafka-for-the-elastic-stack-part2.jpg

Just Enough Kafka For The Elastic Stack, Part 2

malware-detection-blog-banner.jpg

malware-detection-blog-thumb.jpg

Catching malware with Elastic outlier detection

full_bleed_image.png

image.png

We are out of memory (or: Why systemd process limits ruined my day)

<p>As users adopt Kubernetes, Elastic products move fast to support their evolving needs. Whether you are trying to run Elasticsearch workloads to Kubernetes or want to gain better visibility into container workloads running across different environments, we are doubling down on building cloud native tools to support these efforts.
</p><p>This blog covers recent advancements to support Kubernetes initiatives:
</p><ul>
	<li><strong>Elastic Observability</strong> for Kubernetes, including integration with Prometheus and Jaeger</li>
	<li><strong>Elastic Cloud for Kubernetes (ECK)</strong> — the best way to run and orchestrate Elasticsearch on Kubernetes</li>
	<li><strong>Elastic Security</strong> for Kubernetes, including Elastic SIEM and Endpoint</li>
</ul><p>For additional details on these topics, read on, or see a walkthrough in the <a href="/webinars/kubernetes-and-elastic-from-operation-to-observability">Kubernetes and Elastic: From operation to observability</a> webinar.
</p><h2>Elastic Observability for Kubernetes</h2><p>For over two years, Elastic has invested in engineering the best monitoring experience for containerized environments with a particular focus on Kubernetes. Our efforts are aimed to address common challenges we’ve seen with observing Kubernetes workloads:
</p><ul>
	<li>Separate data silos for logs, metrics, and APM</li>
	<li>Poor options for bridging visibility in hybrid environments</li>
	<li>Pricing models incompatible with increasingly dynamic environments</li>
</ul><p>When adopting cloud native initiatives, organizations still end up with separate data silos for logs, metrics, and APM data. Challenges correlating across these data silos are even more pronounced in increasingly dynamic environments. Elastic’s answer to this challenge is an <a href="/blog/observability-with-the-elastic-stack">open and flexible data layer for logs, metrics, and traces</a>, where each data type is just another index in Elasticsearch, tied together using Elastic Common Schema. Under the covers, we optimize storage and retention per data type, but we don't limit what you can ultimately do with the data (such as perform high cardinality queries on top of container labels). Kibana provides a single user interface for curated observability workflows as well as ad hoc data visualizations of this data.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt103d7e9fc0c3efdc/5dc422e35ffacd0409556e36/apm.png" data-sys-asset-uid="blt103d7e9fc0c3efdc" alt="apm.png">
</p><p>Users adopting cloud native tools embark on a journey, often committing to a multi-year transformation of their business and operations. During this time, teams responsible for observability across enterprise applications must find ways to unify visibility across hybrid environments, comprised of traditional on-prem workloads, public cloud workloads, and workloads orchestrated by Kubernetes. Traditional approaches force users to centralize all telemetry in a single place, which is both fragile and costly. Elastic’s answer is to keep the telemetry local to the workloads by offering <a href="/products/#run-elastic-products-your-way">flexible deployment models</a> across on-prem deployments, private and public cloud providers, then unifying visibility across environments using the cross-cluster search feature of Elasticsearch.
</p><p>The final challenge teams struggle with is the cost of observability initiatives, based on traditional pricing models that charge per-ingest on logging workloads, per-host on metrics workloads, and per-agent on APM workloads. These pricing models are punitive, especially in the context of increasingly modular and dynamic Kubernetes workloads. Elastic’s <a href="/pricing">resource-based pricing model</a> scales with resources you use to store and search telemetry. We put the user in control, removing artificial barriers to adoption. If granular observability data is valuable, store it and search it for as long as you like. If the data stops being valuable, roll it up or archive it, using index lifecycle management policies. You can apply these policies per-tenant, so you do not need to have a “one size fits all” answer for your applications.
</p><h2>Recent product announcements for Kubernetes Observability</h2><p>Elastic has been investing heavily in helping users monitor logs, metrics, and application traces in Kubernetes. This includes out-of-the-box monitoring solutions for core Kubernetes primitives such as kube-state-metrics, controller manager, and kube-proxy, as well as CNCF landscape projects — CNCF graduated (CoreDNS and Envoy) and CNCF incubating (NATS and etcd) alike. Additionally, Elastic APM supports OpenTracing, and we are in close collaboration with OpenTelemetry working group aimed to further evolve open standards-based approaches to instrumenting applications.
</p><p>At KubeCon North American 2019, we are excited to share several new advancements in Elastic Observability:
</p><ul>
	<li>Elastic Metrics for Prometheus</li>
	<li>Elastic APM for Jaeger</li>
	<li>Elastic Uptime for Kubernetes</li>
</ul><p>Elastic users have many options to ingest data from Kubernetes environments, including both Elastic native tools and cloud native tools. For many years, users have had their pick of using either FIlebeat or Fluentd for ingesting logs into Elasticsearch. Earlier this year, we announced <a href="/blog/elasticsearch-observability-embracing-prometheus-and-openmetrics-standards-for-metrics">Elastic Metrics for Prometheus</a>, which allows users to directly query metrics from Prometheus exporters and servers and combine them with logs and traces in Elasticsearch. Throughout the year, we continued to evolve this integration based on feedback from many users, including customers like <a href="/elasticon/conf/2018/sf/monitoring-anything-and-everything-with-beats-at-ebay">eBay</a> and <a href="/elasticon/tour/2019/sydney/nine-publishing-building-a-modern-infrastructure-with-the-elastic-stack">Nine Entertainment</a>, which run Elastic for their Kubernetes observability needs at scale.
</p><p>When it comes to Jaeger, <a href="https://www.jaegertracing.io/docs/1.11/deployment/#elasticsearch">Elasticsearch is already a popular backend store for Jaeger traces</a>. This year, we are happy to announce that with the Jaeger graduation announcement, <a href="https://github.com/elastic/apm-server/issues/2886">we are announcing an initiative around tighter integration between Jaeger and Elastic APM</a>. The goal of this integration is to enable users adopting Jaeger to ingest Jaeger traces into Elasticsearch in a schema compatible with Elastic APM, allowing them to visualize Jaeger traces directly in the Elastic APM UI.
</p><p>Finally, cloud-native environments and endpoints aren't static — containers are created, scaled, moved, and destroyed. Elastic Uptime, combined with the recent addition of <a href="/blog/docker-and-kubernetes-hints-based-autodiscover-with-beats">Docker and Kubernetes hints-based autodiscovery</a> in Heartbeat allows users to effectively monitor network and web endpoints and service APIs for availability and correctness and combine these synthetic checks with real user data behind a single pane of glass.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5984b6810c27a6ea/5dc3409a288a9501827774fd/image1.png" data-sys-asset-uid="blt5984b6810c27a6ea" alt="image1.png">
</p><h2>From observing to operating and orchestrating<br>
</h2><p>Observability is only part of Elastic's Kubernetes story. Elastic joined CNCF last year and introduced Elasticsearch Helm Charts to simplify the experience of deploying Elasticsearch on Kubernetes. We continued these efforts with the release of <a href="/blog/introducing-elastic-cloud-on-kubernetes-the-elasticsearch-operator-and-beyond">Elastic Cloud on Kubernetes</a> (ECK), the official Elasticsearch Operator. ECK gives users full-lifecycle control when running and managing Elasticsearch on Kubernetes, including orchestration for upgrades and high-availability.
</p><p>As Michael Lorant, Principal Systems Engineer at Nine Publishing pointed out in his <a href="/elasticon/tour/2019/sydney/nine-publishing-building-a-modern-infrastructure-with-the-elastic-stack">Elastic{ON} presentation</a>, orchestrating complex technology stacks is hard. You have to weigh in the right mix of price versus performance, while balancing simplicity and automation. ECK brings this simplicity and power to a reality and is available for download <a href="/blog/elastic-cloud-on-kubernetes-1-0-0-beta1-released">today</a>.
</p><p>To hear more about Elastic’s innovation in orchestrating complex solutions in Kubernetes, you can attend the KubeCon talk, "<a href="https://kccncna19.sched.com/event/UaeV">Writing a Kubernetes Operator: the Hard Parts</a>", presented by Sebastien Guilloux, a Senior Software Engineer at Elastic, on Wednesday, November 20, 2019 at 3:20pm.
</p><p>The best part of Elastic Cloud for Kubernetes is that every ECK deployment of Elasticsearch includes full access to Elastic’s complete suite of Basic (and free) features. This includes our curated observability applications in Kibana — purpose-built applications for viewing Logs, Metrics, and APM traces, along with the Uptime app — but also includes core Elastic security features, such as authentication, role-based access control, encrypted communications between nodes and agents, API key management, and Kibana Spaces.
</p><h2>To protect and (ob)serve</h2><p>The line between security monitoring and application monitoring isn't one of the "double-yellow" variety — more like the "dotted-white" kind, with each side free to cross back and forth. Data from application traces might be used to identify malicious actors attempting SQL injection, and Endpoint Security might be leveraged to identify slowdowns with internal apps. While you are collecting these logs, metrics, and traces, Elastic lets you use them to protect. Elastic SIEM also supports logs generated by <a href="https://www.envoyproxy.io/">envoy proxy</a>, an edge and service proxy designed for cloud native applications.
</p><p>Combining Kubernetes logs, metrics, and traces with Elastic Endpoint Security goes beyond traditional observability. While you observe, why not protect? With the average threat dwell time exceeding 100 days, shipping, scaling, and storing data efficiently in Elasticsearch makes searching through all of this disparate security-related data practical, easy, and fast.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb1827846ca93267a/5dc340daff1920021896edd7/image2.png" data-sys-asset-uid="bltb1827846ca93267a" alt="image2.png">
</p><p>In <a href="/blog/observability-with-the-elastic-stack">Observability with the Elastic Stack</a>, Tanya Bragin, Senior Director of Product Management here at Elastic, notes, "Observability is not something that a vendor delivers in a box — it is an attribute of a system you build, much like usability, high availability, and stability". Vendors don't know which data is going to be important to you in the long run, which is why Elastic products follow a simple, unified pricing model that we’ve had since the beginning. We apply this model to all use cases, including in the contexts of SIEM and <a href="/blog/elastic-endpoint-security-what-does-no-per-endpoint-pricing-mean">Endpoint data</a>, putting you in control of transforming your security operations.
</p><h2>Get started</h2><p>You can get started with Elasticsearch on Kubernetes today using <a href="/guide/en/cloud-on-k8s/current/index.html">Elastic Cloud on Kubernetes</a> (ECK), and you always have access to the latest version of the Elastic Stack when using the <a href="/cloud/elasticsearch-service">Elasticsearch Service</a>, where you can sign up for a free trial to create your first cluster, or you can download it as part of the <a href="/downloads/">default distribution</a> of the Elastic Stack.
</p><p><span></span>
</p><p><span></span>
</p>

blog-banner-release-observability.png

blog-thumb-release-observability.png

KubeCon 2019:  Elastic Doubles Down on Observability and Orchestration for Kubernetes

<p>We are pleased to announce the general availability of <em>external collection</em> for Elastic Stack Monitoring. With this announcement comes the ability to monitor Elasticsearch, Kibana, Logstash, APM server, and Beats all via Metricbeat modules.
</p><p>Using external collection, users now have the capability to collect and send monitoring data for their Elastic Stack&nbsp;without having to depend on the health of the monitored services. This release dramatically improves the reliability and flexibility of options for monitoring the Elastic Stack. Using external collection is a recommended approach for all users of Elastic Stack Monitoring.
</p><h2>Background</h2><p>Previously, services in the Elastic Stack gathered and shipped their own monitoring data — a process known as <em>internal collection</em>.
</p><p>With the introduction of external collection, users gain the ability to run Metricbeat alongside each type of monitored services, such as Elasticsearch, to gather and ship monitoring data about its health and performance directly to the monitoring cluster.
</p><p>In the past, Stack Monitoring has required that one send all monitoring data first to the production cluster where it would then be forwarded to the monitoring cluster. This requirement was undesirable because, of course, monitoring is of the most value when the production cluster is suspected to be under duress!
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt62630b8198e2572f/5de6974dd826760c7d0303c1/elastic-stack-monitoring-blog-internal-collection-v2.png" data-sys-asset-uid="blt62630b8198e2572f" alt="Internal collection for Elastic Stack Monitoring" style="display: block; margin: auto;"></p><figcaption>Past setups which required that monitoring data be routed through a production cluster
</figcaption><p>With the introduction of Metricbeat as the agent for collecting and shipping monitoring data, the need to route monitoring data through the production cluster has been removed. Monitoring data can now be collected by an independent, lightweight monitoring agent and sent directly to the monitoring cluster.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blted87d6b45f566066/5ddde63b3d222f0c421f02be/elastic-stack-monitoring-blog-external-collection.png" data-sys-asset-uid="blted87d6b45f566066" alt="External collection for Elastic Stack Monitoring using Metricbeat" style="display: block; margin: auto;">
</p><figcaption>Monitoring with Metricbeat, which routes all data directly to the monitoring cluster without needing to touch the production cluster at all</figcaption><p>Using Metricbeat for monitoring collection means less work for the production cluster and more resilience for your monitoring setup.
</p><h2>How do I use it?</h2><p>If you’ve never used Stack Monitoring to track the health and performance of your Elastic Stack services, now is a great time to try! To get started with Stack Monitoring, <a href="/guide/en/elasticsearch/reference/7.5/monitor-elasticsearch-cluster.html">follow the instructions</a>. When prompted in the Kibana interface, choose <strong>Set up with Metricbeat.</strong>
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9d32d52bb00c83ff/5ddde6707819930c2c70a726/elastic-stack-monitoring-blog-no-data.png" data-sys-asset-uid="blt9d32d52bb00c83ff" alt="Setting up external collections with Metricbeat" style="display: block; margin: auto; max-width: 75%;">
</p><figcaption>Click <strong>Set up monitoring with Metricbeat</strong> to configure external collection</figcaption><p>From there, you’ll be walked through step-by-step instructions on how to configure Metricbeat for external collection and a configuration will be generated for you which you can simply copy and paste into a configuration file on your system!
</p><p>If you’ve set up Stack Monitoring in the past, you’ve likely used internal collection, which was enabled via a single button-click in the Stack Monitoring interface.
</p><p>Luckily, migrating from internal collection to external collection for Stack Monitoring is very simple.
</p><p>With the 7.5 release of the Elastic Stack, a new Migration Wizard is available in Stack Monitoring for users of internal collection. Using the Migration Wizard, users are guided through the various steps necessary for moving from internal collection to external collection with Metricbeat.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt80422a5205be07c8/5ddde6c00a6fc50c16d70e16/elastic-stack-monitoring-blog-setup-mode.png" data-sys-asset-uid="blt80422a5205be07c8" alt="Click Enter Setup Mode to transition from internal to external collections" style="display: block; margin: auto; max-width: 75%;">
</p><figcaption>Click <strong>Enter Setup Mode</strong> to transition from internal to external collections</figcaption><p>Current Stack Monitoring users can enable Setup Mode for existing, simply click on the link in the upper-right corner of the Stack Monitoring application.
</p><h2>Important upcoming changes</h2><p>A future major version release of the Elastic Stack will remove internal collection entirely. All users are encouraged to switch to external collection to ensure that they are well-prepared for the transition away from internal collection. Of course, Elastic will keep users well-informed about the upcoming changes along the way so that there is plenty of time to prepare but with the release of external collection there is no reason to wait! Elastic encourages all users to migrate to Metricbeat for Stack Monitoring as soon as they are able to.
</p><h2>Conclusion</h2><p>Users can see significant gains in the reliability of monitoring by switching from internal collection to external collection. Using either the Migration Wizard or the directions available in the <a href="/guide/en/elasticsearch/reference/7.5/monitor-elasticsearch-cluster.html">Elastic Stack documentation</a>, it’s easy to see immediate gains from the change and to be well-prepared for the future!
</p>

blog-banner-generic-blue.png

elastic-stack-monitoring-blog-thumb.png

External collection for Elastic Stack Monitoring is now available via Metricbeat

<p>Hard on the heels of our <a href="https://www.elastic.co/blog/master-time-with-kibanas-new-time-series-visual-builder">first time series video tutorial</a>, Chris Cowan is back with another video show off how powerful and versatile this new UI is. If you'd like to follow along, go ahead and download <a href="https://www.elastic.co/downloads/beats/metricbeat">Metricbeat</a> to start ingesting metrics from your systems or services. You'll also want to make sure you have the latest version of <a href="https://www.elastic.co/downloads/kibana">Kibana</a> installed.
</p>
<p><img src="https://api.contentstack.io/v2/uploads/59513a0a4b6c69cd2af4f1e5/download?uid=bltdf18bc128e3f72ff" data-sys-asset-uid="bltdf18bc128e3f72ff" alt="tsvb2-metric.png" style="background-color: initial;">&nbsp;
</p>
<p>In this video, you will learn to do the following:
</p><span id="docs-internal-guid-03274d3e-e548-d7cf-270b-d247e3f10432">
<ul>
	<li dir="ltr">Quickly change visualization types with the same data</li>
	<li dir="ltr">Use the overall functions to aggregate over the total time interval in single metric</li>
	<li dir="ltr">Take advantage of the data formatter easily switch between data types</li>
	<li dir="ltr">Conditionally style a metric visualization based on the most recent value of your time series data</li>
</ul>
<p>Ready to dive in? Check out the video tutorial below.
</p>
<div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/4vZEBRZ33apj1j1hqXKJCP.jpg" data-uuid="4vZEBRZ33apj1j1hqXKJCP" data-v="4" data-type="inline">
</div>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Be on the lookout for our next tutorial with the new time series visual builder!
</p></span>

tsvb2-banner.png

tsvb2-metric.jpg

Kibana's New Time Series Visual Builder - Part 2

<p>As another year is coming to an end, let's take time to look at what happened in Lucene land in 2019.
</p> <strong><h2>Active community</h2></strong><p>There are not that many projects that are 20+ years old and still as dynamic as <a href="https://lucene.apache.org/">Apache Lucene</a>. In 2019:
</p><ul>
	<li>7 community members were made committers</li>
	<li>4 committers became Project Management Committee (PMC) members</li>
	<li>9 new versions were released, including a major version: 8.0</li>
</ul><p>In particular, we are glad to report that two Elasticians became PMC members this year, <a href="https://github.com/iverase">Ignacio Vera</a> and <a href="https://github.com/nknize">Nick Knize</a>. Congratulations, Ignacio and Nick!
</p><p>One metric that has especially improved over the last few years is the number of unique authors in the git log:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt27c917834edaaa66/5e09ed8cc848085fa46e6c9e/apache-2019-review-blog-stats.png" data-sys-asset-uid="blt27c917834edaaa66" alt="Chart showing the number of unique authors in the git log" style="display: block; margin: auto;"></p><p>To be fair, part of this increase is due to the <a href="https://lists.apache.org/thread.html/ad50c9c6cb8a3f8695e04328376e909c95cfc6d8b9d8e22f034659b3%401453209417%40%3Cdev.lucene.apache.org%3E">migration of Lucene's repository from Subversion to Git</a> in 2016, which made it easier to preserve the name of the author of a contribution. Before that, contributions would always be attributed to the committer who merged the contribution in the git log, and the contributor would only be mentioned in the changelog. But there have also been efforts to make contributing to Lucene easier. While contributions used to require creating patches manually and attaching them to a JIRA issue, it is now also possible to open a pull request on Github, which any committer can merge by just clicking on a button. This simpler process, which also looks much more similar to a majority of open-source projects, certainly helps attract new contributors.
</p> <strong><h2>Many new features and improvements</h2></strong> <strong><h3>Block-Max WAND</h3></strong><p><a href="https://github.com/rmuir">Robert Muir</a>, <a href="https://github.com/jimczi">Jim Ferenczi</a> and I worked on adding support for <a href="https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand">Block-Max WAND</a> to Lucene, which was Lucene 8.0's main release highlight. Block-Max WAND is a retrieval technique that allows to retrieve the top hits of a disjunction (bool query that only has <code>should</code> clauses) without visiting all matches. This technique can make some queries hundreds of times faster with the only downside that the hit count is not computed.
</p><p>It was released alongside a new <a href="https://lucene.apache.org/core/8_3_0/core/org/apache/lucene/document/FeatureField.html">FeatureField</a>, which allows to incorporate static scoring signals into the final score while still taking advantage of Block-Max WAND. For instance, one could index a <code>pagerank</code> as a FeatureField and could use this field to compute scores as <code>bm25_relevance + log(pagerank)</code>. FeatureField is exposed in Elasticsearch via the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/rank-feature.html">rank_feature</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/rank-features.html">rank_features</a> fields.
</p> <strong><h3>Luke has become a Lucene module</h3></strong><p>The Luke tool is well known from Lucene users: it allows you to introspect the content of an index. For instance Luke can be used to list the content of the terms dictionary. One major challenge for Luke users was that versions were lagging a bit behind, so if you upgraded to a recent version of Lucene soon after its release, there might be a couple weeks until a new version of Luke was released for that version. But thanks to <a href="https://github.com/mocobeta">Tomoko Uchida</a> and <a href="https://github.com/uschindler">Uwe Schindler</a>, <a href="https://medium.com/@mocobeta/luke-become-an-apache-lucene-module-as-of-lucene-8-1-7d139c998b2">Luke is now a Lucene module</a>, which gets released alongside every new Lucene version and can be used right away.
</p> <strong><h3>Lucene got a new "monitor" module</h3></strong><p><a href="https://github.com/romseygeek">Alan Woodward</a> and his former colleagues at FlaxSearch decided to donate Luwak, a framework that helps find all queries that would match a document, like Elasticsearch's percolator. It has been incorporated into Lucene as module called "<a href="https://lucene.apache.org/core/8_3_0/monitor/org/apache/lucene/monitor/package-summary.html">monitor</a>".
</p> <strong><h3>Much faster indexing of multi-dimensional points</h3></strong><p>Constructing a BKD tree requires recursively partitioning a set of multi-dimensional points around the median value of the dimension that has the widest span of values. Computing the median value used to sort values offline with a merge sort. <a href="https://github.com/iverase">Ignacio Vera</a> optimized it by taking inspiration from the radix sort algorithm in order to more efficiently compute the median value. This translated into significant indexing speedup for geo points, range fields and geo shapes. See annotations AE, AF and AG on the <a href="https://people.apache.org/~mikemccand/geobench.html#index-times">nightly geo benchmarks</a>.
</p> <strong><h3>Increased efficiency</h3></strong><p>Sometimes there are small improvements that give a lot of value when combined together. This year again, Lucene received a number of neat optimizations:
</p><ul>
	<li><a href="https://github.com/msokolov">Mike Sokolov</a> and <a href="https://github.com/bruno-roustant">Bruno Roustant</a> helped improve the efficiency of FST lookups on dense nodes by <a href="https://issues.apache.org/jira/browse/LUCENE-8920">indexing outgoing labels using a bitset</a> instead of performing a binary search.</li>
	<li>Ankit Jain introduced the ability to <a href="https://issues.apache.org/jira/browse/LUCENE-8635">read FSTs directly from disk</a>, which can be especially useful as the terms index is an FST and usually drives the memory usage of Lucene indices.</li>
	<li><a href="https://github.com/jdconrad">Jack Conradson</a> did a <a href="https://issues.apache.org/jira/browse/LUCENE-8932">similar change for BKD trees</a>, which helps save even more memory.</li>
	<li>I worked on a change to <a href="https://issues.apache.org/jira/browse/LUCENE-9027">how postings are encoded</a> so that the decoding logic gets <a href="https://en.wikipedia.org/wiki/Automatic_vectorization">auto-vectorized</a>.</li>
</ul><p>Thanks for following along for the last 12 months, and let's wish Lucene another great year in 2020!
</p>

blog-banner-fireworks.jpg

blog-thumb-fireworks.jpg

This year in Apache Lucene - 2019

<p>
	With the much-anticipated release of Kibana 4, we were thinking about how to help users get up to speed quickly with the new feature set.
</p>
<p>
	As a first step, we wrote up comprehensive 
	<a href="/guide/en/kibana/current/index.html">documentation</a> for Kibana 4, which is a great reference. However, we felt that the power and ease-of-use of the new interface was best conveyed in a series of short video tutorials, ranging from a high-level overview to panel-by-panel recommendations on how to migrate visualizations from Kibana 3 to Kibana 4.
</p>
<p>
	Today, we publish the first four of these videos, with more to come!
</p>
<p>
	The first video introduces you to Kibana 4 navigation at a very high-level. Whether you are coming over from Kibana 3 or just starting with Kibana, this one is worth while to check out.
</p>
<div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/FneaLb8ynzyUdJRBTDs68v.jpg" data-uuid="FneaLb8ynzyUdJRBTDs68v" data-v="4" data-type="inline">
</div>
<p>
	The second video walks you through how to build a specific visualization in Kibana 4, in this case a pie chart.
</p>
<div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/yVN92wPs5P4Mdzwzhiavs5.jpg" data-uuid="yVN92wPs5P4Mdzwzhiavs5" data-v="4" data-type="inline">
</div>
<p>
	The third video shows how you can re-create Kibana 3 "hits" panels in Kibana 4.
</p>
<div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/knGSJWDRy1ha5sYNAkwTP1.jpg" data-uuid="knGSJWDRy1ha5sYNAkwTP1" data-v="4" data-type="inline">
</div>
<p>
	The fourth video shows how you can re-create Kibana 3 "terms" panels in Kibana 4.
</p>
<div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/GV85h17A8GhaK7V28DzYfu.jpg" data-uuid="GV85h17A8GhaK7V28DzYfu" data-v="4" data-type="inline">
</div>
<p>
	And we're always on the lookout for a good Kibana story. Send us yours to 
	<a href="mailto:stories@elastic.co">stories@elastic.co</a>.
</p>
<p><br>
</p>
<p style="line-height: 22.3999996185303px;"><span style="font-weight: 700;">Links to&nbsp;other&nbsp;parts of&nbsp;Kibana 4&nbsp;Video Tutorials series:</span>
</p>
<ul>
	<li><a href="https://www.elastic.co/blog/kibana-4-video-tutorials-part-1">Part 1 - Navigation Overview, Kibana&nbsp;4&nbsp;Pie&nbsp;Charts, Kibana&nbsp;3 Hits and Terms&nbsp;Panels</a></li>
	<li><a href="https://www.elastic.co/blog/kibana-4-video-tutorials-part-2">Part 2 - Kibana 4&nbsp;Data Discovery, Bar Chart, Line Chart</a></li>
	<li><a href="https://www.elastic.co/blog/kibana-4-video-tutorials-part-3">Part 3 - Kibana 4 Tile Maps</a></li>
	<li><a href="http://elastic.co/blog/kibana-4-video-tutorials-part-4">Part 4&nbsp;- Embedding Kibana 4&nbsp;Visualizations</a><span></span></li>
</ul>

Kibana 4 Video Tutorials, Part 1

<p>Curious about how to write more idiomatic concurrent code in Go? It’s not always easy or intuitive, even if you’ve done lots of concurrent programming in other languages. I’ve been lucky to have worked in a <a href="https://github.com/elastic/beats">well-written code base</a>, and had the expert advice of Beats core area lead <a href="https://github.com/urso">Steffen Siering</a>&nbsp;along the way.&nbsp;
</p><p>In this post I’ll walk you through how we implemented a <a href="https://github.com/elastic/beats/pull/14890">new scheduler</a>&nbsp;for <a href="https://discuss.elastic.co/c/beats/heartbeat">Heartbeat</a>&nbsp;that is part of our upcoming 7.6.0 release. I’ll talk about the high level concurrency strategy for implementing this scheduler, and some of the challenges faced therein. This post assumes a general familiarity with <a href="https://en.wikipedia.org/wiki/Concurrent_computing">concurrent programming</a>, and focuses on the high level approach to concurrency within our scheduler.
</p><p>If you don’t already know, Heartbeat is the agent behind the <a href="https://www.elastic.co/products/uptime-monitoring">Uptime app</a>. It regularly pings a site via ICMP, TCP, or HTTP on a schedule to determine whether it’s up or down, and also reports some stats on its performance. The Heartbeat scheduler is in charge of dispatching those tasks at a predetermined interval. So our scheduler performs a job much like cron or <code>java.util.Timer</code>.
</p><p>For Heartbeat’s purposes we need a scheduler that:
</p><ul>
	<li>Is reliable (our old one had some issues that surfaced in high concurrency)</li>
	<li>Is fast and memory efficient</li>
	<li>Supports both the addition and removal of jobs without stopping the scheduler</li>
	<li>Manages spawned subtasks (essentially continuations) from multi-part jobs</li>
	<li>Supports limiting the maximum number of running sub-tasks</li>
</ul><h2>A naive initial attempt
</h2><p>My initial attempt to refactor the scheduler was going to be so easy to write. I had the bright idea to lean on the lightweightness of goroutines and take an approach that while only seemingly naive, would be&nbsp;beautifully minimal and leverage the efficiency of Golang’s goroutine scheduler and <code>time.Timer</code>. I was very wrong here.
</p><p>In this scheme, for each scheduled task we’d launch a goroutine that immediately blocked on a <code>time.Timer</code>, looping after each task invocation. This meant that for N defined jobs, we’d always have N goroutines, mostly in a blocked state waiting for their timers to trigger. My thinking was that goroutines are super-lightweight, and <code>time.Timer</code> is efficient. So this should just work, would be easy to implement, would check for correctness, and would be the sort of thing that you could get working in some sort of rough shape in an afternoon. Well, it turns out only the last part was correct. After building a PoC of this it turned out that goroutines are not as light as I’d hoped, and neither is the <code>time.Timer</code> type.
</p><p>This naive approach used much more memory (at least 20% more) than the old timer, ran slower, and used more CPU in intensive use cases. I didn’t keep exact stats because the results were clearly terrible, and it was obvious that we’d need to take a more robust approach. My hope that lightweight Go internals would let me write dead simple code was completely off.
</p><h2>A more robust approach
</h2><p>After trying, and failing to find a silver bullet to fix the previous approach, I realized I needed to start over. It became apparent that we had to manage the resources more explicitly, and try to minimize <code>time.Timer</code> instances and goroutines. The standard approach to this is to use a priority queue to keep track of which timer is executing next. This is a classic approach to this problem used in other timer libraries. Implementing it in a thread-safe way was quite a bit more effort. After a lot more work than my naive approach, we wound up with something that performed significantly better than the old scheduler in terms of both CPU and memory utilization.
</p><p>The new implementation combines:
</p><ul>
	<li>A priority queue implemented via a heap, storing of functions for the dispatch of tasks.&nbsp;</li>
	<li>Go’s <a href="https://godoc.org/golang.org/x/sync/semaphore">WeightedSemaphore</a>&nbsp;for limiting concurrent task execution.</li>
	<li>Go’s <a href="https://golang.org/pkg/sync/#WaitGroup">WaitGroup</a>&nbsp;to wait for all of a job’s sub-tasks to return.</li>
	<li>Go’s atomic <a href="https://golang.org/pkg/sync/atomic/#CompareAndSwapInt32">CompareAndSwap</a>&nbsp;functions to transition between states.</li>
	<li>Go’s <a href="https://golang.org/pkg/context/">Context</a>&nbsp;to provide an easy method of terminating and ongoing tasks from the root of the code.</li>
</ul><h2>General structure
</h2><p>Our <a href="https://github.com/elastic/beats/tree/master/heartbeat/scheduler">scheduler code</a>&nbsp;is split into several packages:
</p><ul>
	<li>scheduler contains the high level scheduler which accepts new jobs, and handles their removal. It also contains the high level logic around requeuing jobs, executing their sub-tasks, and limiting concurrency.</li>
	<li>scheduler/schedule represents the definition of a schedule, and offers a <code>Next(now) time.Time</code> method, that returns the next scheduled execution based on a given now value.</li>
	<li>scheduler/timerqueue is a thread-safe queue that lets you schedule tasks for a single execution making efficient use of timers.</li>
</ul><h2>Minimizing timer instances with a priority queue
</h2><p>A key part of keeping things efficient is minimizing the number of running <code>time.Timer</code> instances. We only really need one timer, activating when the next scheduled task is ready, and then a check to see how many tasks are due to be executed. We only need to reset this <code>time.Timer</code> if a new task is scheduled that’s due to be executed before any other tasks. So, what we need is a way of tracking our tasks in order of execution.
</p><p>The perfect data structure for this is a <em>priority queue</em>, or heap. You can find our implementation in Heartbeat’s <a href="https://github.com/elastic/beats/blob/master/heartbeat/scheduler/timerqueue/heap.go">timerqueue/heap.go</a>. With a heap, we get O(log n) insertion and removal of elements. O(1) ‘peek’ing at the highest priority item, and O(log n) ‘pop’ing of the highest priority removal (peek + delete).
</p><p>This means every time a new item is scheduled, adding it is a matter of:
</p><ol>
	<li>Peeking to see if it is scheduled to run before the current highest priority item O(1)</li>
	<li>Pushing it onto the queue O(log n)</li>
</ol><p>Go comes with a <a href="https://golang.org/pkg/container/heap/">heap implementation</a>, but Go’s heap implementation is not thread-safe. The <a href="https://github.com/elastic/beats/blob/master/heartbeat/scheduler/timerqueue/queue.go">TimerQueue</a>&nbsp;type we created provides thread-safe methods built on top of Go’s heap operations. We spawn a background goroutine that listens for events that mutate the heap, thus serializing access to the underlying data structure. If you’re familiar with the <em>active object</em> pattern, this should ring a bell.
</p><h2>Running recursive tasks using weighted semaphores and wait groups
</h2><p>When Heartbeat executes a job, there may be more than one sub-task invoked as a result. For instance, if <code>mode:all</code> is set, Heartbeat will ping each IP returned from the DNS lookup as a separate sub-task. Heartbeat supports limiting the number of running sub-tasks (not jobs) via the <code>scheduler.limit</code> option. The core logic around this can be found in <a href="https://github.com/elastic/beats/blob/master/heartbeat/scheduler/scheduler.go#L242">scheduler.runRecursiveTask</a>.
</p><p>Let’s examine the way Heartbeat limits task execution here. You may be wondering why we limit tasks, not jobs. The reason is that we want users to be able to limit the number of file descriptors in use at one time. One task will use at most one file descriptor. This lets users plan around OS file descriptor limits with more predictability. Golang’s <a href="https://godoc.org/golang.org/x/sync/semaphore">WeightedSemaphore</a>&nbsp;is a great fit for our needs here. It lets you define a quantity of some resource, then reserve some portion of that capacity. If a goroutine attempts to grab more resources than are available it blocks, waiting for that resource to be released. In our case we grab one unit per sub-task.
</p><p>We use Go’s <a href="https://golang.org/pkg/sync/#WaitGroup">WaitGroup</a>&nbsp;to determine when all of a job’s sub-tasks are done so we can do some post-processing. <code>WaitGroup</code> lets you block a goroutine until a counter tracking some number of inflight tasks reaches zero. One nice property of <code>WaitGroup</code> is that if you’re kicking off N tasks, you can just increment it by N, instead of invoking <code>add</code> N times.
</p><h2>Using atomic operations to handle state transitions
</h2><p>The scheduler has a simple life cycle defined by a simple state machine. It can be in either a <code>preRunning</code>, <code>running</code>, or <code>stopped state</code>, with no backwards transitions. To manage these transitions, we use atomic compare and set (CAS) operations, which are wrapped in Beats’ utility atomic&nbsp;package. We use this utility package to ease some of the awkwardness of using the Go <a href="https://golang.org/pkg/sync/atomic/">atomic</a>&nbsp;package, which is a bit cumbersome.
</p><p>The nice thing about using CAS for a state machine is that you can ensure that a transition happens exactly once. See <a href="https://github.com/elastic/beats/blob/master/heartbeat/scheduler/scheduler.go#L120">Scheduler.Start</a>&nbsp;for an example of how we use it to start the scheduler at most once, no matter how many times <code>Start()</code> is invoked.
</p><h2>Using context to shut down spawned tasks
</h2><p>The scheduler controls the lifecycle of most Heartbeat operations. As a result, it’s responsible for shutting down all user defined monitors when asked. There’s no way to say ‘shut down any goroutines you may have spawned’ in Go. You have to let those goroutines gently know it’s time to shut down.
</p><p>An idiomatic way to do this is by passing Go’s <a href="https://golang.org/pkg/context/">context.Context</a> throughout your code. By passing this type throughout the scheduling code, we can easily abort the entire tree of downstream processes by cancelling the top level context. The next time a spawned Go process does a select, it can attempt a channel read off <code>(Context).Done()</code> and stop processing immediately. This does require you to always check <code>(Context).Done()</code> any time you may block in a select, though. It’s a little cumbersome, and can potentially be tricky to integrate with non-channel based concurrency like mutexes and weighted semaphores, but it’s a great way to write code that can respond to shutdown requests correctly. Additionally, network/io libraries usually take a context, meaning that our shutdown can be correctly handled outside of code we’ve directly written.
</p><h2>Wrapping things up
</h2><p>I hope this has been a useful post in terms of looking at the high level concurrency strategy for go code. If you have any questions, comments, or feedback drop us a line on the <a href="https://discuss.elastic.co/c/beats/heartbeat">Heartbeat topic on Discuss</a>. And if you’d like to see Heartbeat in action, check it out on <a href="https://demo.elastic.co/app/uptime">demo.elastic.co</a>.
</p>

blog-banner-train-track-junction.jpg

blog-thumb-train-track-junction.jpg

A tour of Go concurrency patterns via the new Heartbeat scheduler

<p>So you want to install <a href="https://www.elastic.co/ece" target="_blank">Elastic Cloud Enterprise</a> (you know, the orchestration solution for the Elastic Stack that simplifies and standardizes how you deploy, upgrade, resize, configure, and monitor one to many clusters from a single UI/API)
</p><p>Installing ECE on one host isn’t tough. Installing it on two isn’t much harder. However, when you start dealing with 3, 5, 7, 11, etc., the complexity grows, as does the work involved in operating and maintaining (upgrading!) it all. The typical way to deal with the increase in complexity brought by scaling is to start automating deployment and configuration.
</p><p>There are many tools that you can use to automate both infrastructure provisioning as well as application deployment. In this case I’m going to focus on the latter, and in particular on <a href="https://docs.ansible.com/ansible/latest/index.html" target="_blank">Ansible</a>.
</p><p>Why Ansible? Because we began providing an <a href="https://github.com/elastic/ansible-elastic-cloud-enterprise" target="_blank">Ansible role for ECE</a> deployments with version 2.2.
</p><p>Why me? I’m totally new to Ansible, but wanted to play with it a bit to understand how it worked and get a feel for how hard it was to use to deploy and configure a multi-node ECE environment. The good news is, it’s pretty easy.
</p><p>In this blog I’m going to set up a small ECE environment that could be used as a proof of concept, a test environment, or to host a very small production cluster. Typically, ECE users will attach many large machines so that they can host many different clusters and apps within it.
</p><h2>Tasks</h2><p>We’re going to follow the <a href="https://www.elastic.co/guide/en/cloud-enterprise/2.2/ece-topology-example1.html" target="_blank">small baseline installation</a> example in the ECE docs. To do so we’ll need to:
</p><ol>
	<li>Create a Security Group</li>
	<li>Launch three EC2 instances</li>
	<li>Install ansible on your machine</li>
	<li>Download the ECE Ansible role</li>
	<li>Setup an Ansible project for our ECE deployment</li>
	<li>Create an Ansible inventory for our ECE environment</li>
	<li>Create an ECE playbook</li>
	<li>Run the playbook</li>
	<li>Celebrate</li>
</ol><h3>Create a Security Group</h3><p>Not the most exciting way to get started, but is kinda necessary. Alternatively, you can open up all inbound and outbound ports (1-65535) to the world (please don’t) or to your own IP (better). We’re going to do it <em>right</em> though, because <a href="https://en.wikiquote.org/wiki/Philip_Stanhope,_4th_Earl_of_Chesterfield" target="_blank">whatever is worth doing at all is worth doing well</a>.
</p><p>The <a href="https://www.elastic.co/guide/en/cloud-enterprise/2.4/ece-prereqs-networking.html" target="_blank">port documentation</a> will be our guide to wellness.
</p><p>It’s worth noting that we’re only creating a single Security Group because each of our hosts are running all the ECE components. In a larger production environment, you’d likely have different Security Groups for each role (e.g. “ece-director”, “ece-allocator”, “ece-proxy”, etc) to ensure the fewest open ports possible. The single group also means the documentation will repeat some of the port ranges since they apply to multiple ECE components.
</p><ol>
	<li>Log in to the AWS console</li>
	<li>Navigate to the Security Groups page on the EC2 network and security menu.</li>
	<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfee9bc8dd7456db8/5de9202894bf745993ed5213/ece-ansible-blog-security-group.jpg" data-sys-asset-uid="bltfee9bc8dd7456db8" alt="Navigate to Security Groups" width="155" height="148" style="display: block; margin: auto; width: 155px; height: 148px;">
	<li>Click the <strong>Create Security Group</strong> button.</li>
	<li>Fill out the basic form with a name (maybe put “ECE” in there somewhere), a description (for fun, totally avoid the use of “ECE”), and the right VPC.</li>
	<li>Add all the outbound traffic ports (HTTP and HTTPS) from "Anywhere".</li>
	<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltef42adc8bd450fcf/5de920648651f15abe07f681/ece-ansible-blog-output-ports.jpg" data-sys-asset-uid="bltef42adc8bd450fcf" alt="Add outbound traffic ports" style="display: block; margin: auto;">
	<p>I just use the text from the Purpose column in the documentation for the rule Description
	</p>
	<li>Add <em>SOME</em> inbound traffic ports.<br>At this point, we only want to add the inbound ports that need to be allowed <a href="https://www.elastic.co/guide/en/cloud-enterprise/2.4/ece-prereqs-networking.html#ece_inbound_traffic_from_any_source" target="_blank">from any source</a>.</li>
	<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb851b1c81f5933a4/5de920b889b250596f300804/ece-ansible-blog-inbound-ports.jpg" data-sys-asset-uid="bltb851b1c81f5933a4" alt="Add necessary inbound ports" style="display: block; margin: auto;">
	<li>Click <strong>Create</strong>.</li>
	<li>Select the Security Group you just created, and click the <strong>Edit</strong> button on the Inbound tab at the bottom of the screen.</li>
	<li>Add the <a href="https://www.elastic.co/guide/en/cloud-enterprise/2.4/ece-prereqs-networking.html#ece_inbound_traffic_from_internal_components_of_ece" target="_blank">internal inbound ports</a> using the Security Group ID as the Source.
	<ol>
		<li>Click <strong>Add Rule</strong> at the bottom of the popup modal window.</li>
		<li>Add everything else. It’ll look something like this:</li>
		<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltcd73d45cedb1d3b9/5de92130546f955ab898e047/ece-ansible-blog-all-inbound-ports.jpg" data-sys-asset-uid="bltcd73d45cedb1d3b9" alt="Add additional internal inbound ports" style="display: block; margin: auto;">
	</ol></li>
	<li>Click <strong>Save</strong></li>
</ol><h3>Launch three EC2 instance
</h3><p>We recommend installing Elasticsearch in three separate zones (racks or cloud availability zones, not data centers or regions) for production clusters to provide high availability. To deploy a three zone Elasticsearch cluster within ECE, we need to have ECE machines in three separate zones. Makes sense! Since we’re in AWS, these “zones” are <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html" target="_blank">availability zones</a> within an AWS region.
</p><ol>
	<li>In the AWS console, navigate to the EC2 service dashboard, and click the <strong>Launch Instances</strong> button.</li>
	<li>Select an AMI. I’m going to use CentOS.</li>
	<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt50f7ca8683aee7f1/5de9215d89b250596f30080a/ece-ansible-blog-ami.jpg" data-sys-asset-uid="blt50f7ca8683aee7f1" alt="Select an AMI" style="display: block; margin: auto;">
	<li>Select an instance type.</li>
	<p>The ECE docs recommend that host machines have a minimum of 128GB of memory for small deployments. Since I’m just spinning up a small test environment, and am interested in reducing my spend, I’m going a touch smaller. ECE also <a href="https://www.elastic.co/guide/en/cloud-enterprise/2.4/ece-planning-hw.html#ece-planning-hw" target="_blank">recommends SSDs</a> for hosts running the ECE management services.
	</p>
	<p>Given the above, I’ve selected the storage optimized i3.xl instances which give us 30.5GB of memory and 950GB of SSD disk.<br>
	</p>
	<li>Click <strong>Review and Launch</strong>.</li>
	<li>Select the Security Group.</li>
	<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt808dc29961c73d54/5de9217594bf745993ed522f/ece-ansible-blog-security-group-2.jpg" data-sys-asset-uid="blt808dc29961c73d54" alt="Select the security group" style="display: block; margin: auto;">
	<p>The default Security Group isn’t going to be enough for our needs, so we’ll use the one we just created.
	</p>
	<ol>
		<li>Click the <strong>Edit Security Groups</strong> link.</li>
		<li>Select the option to select an existing group.</li>
		<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt951a1dad6e8db10f/5de921ea94bf745993ed5237/ece-ansible-blog-existing-security-group.jpg" data-sys-asset-uid="blt951a1dad6e8db10f" alt="Select an existing group" width="389" height="52" style="display: block; margin: auto; width: 389px; height: 52px;">
		<li>Select the group you created.</li>
		<li>Click <strong>Review and Launch</strong>.</li>
	</ol>
	<li>Click the <strong>Launch</strong> button.</li>
	<li>Select a key pair and <strong>Launch</strong> (yet again...).</li>
	<p>If you don’t have one yet, follow the directions to create a new one. Otherwise, select an existing one and check the checkbox if you do actually have the private key.
	</p>
	<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb1c90528d482f4f6/5de921808d1dce6ad6bee8fa/ece-ansible-blog-keys.jpg" data-sys-asset-uid="bltb1c90528d482f4f6" alt="Select an existing key pair" style="display: block; margin: auto;">
	<li>Launch another instance “like this”.</li>
	<p>To launch another instances like the last, navigate back to the EC2 dashboard, select your newly launched (or launching) instance (make a note of the availability zone it’s in), click the <strong>Actions</strong> button, and select <strong>Launch More Like This.</strong>
	</p>
	<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7194452642146054/5de92196d7e18e0c7662dcb4/ece-ansible-blog-more-like-this.jpg" data-sys-asset-uid="blt7194452642146054" alt="Launch another instance like this" width="195" height="227" style="width: 195px; height: 227px; display: block; margin: auto;">
	<li>Deploy the new instance to a different availability zone.
	<ol>
		<li>Click the <strong>Edit Instance Details</strong>.</li>
		<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt34509e8519d79218/5de9220394bf745993ed523d/ece-ansible-blog-instance-details.jpg" data-sys-asset-uid="blt34509e8519d79218" alt="Click Edit Instance Details" style="display: block; margin: auto;">
		<li>Select a subnet in a different zone than the first instance.</li>
		<li>Click the <strong>Review and Launch</strong> button, then the <strong>Launch</strong> button, select the key pair, and click Launch again.
		</li>
	</ol>
	</li>
	<li>Repeat steps 8 & 9 to deploy the third instance into yet another availability zone.</li>
</ol><p>At this point you should have three i3.2xls running in different AZs within a single region, just itching to get some ECE magic <a href="https://media.giphy.com/media/l4Jz3a8jO92crUlWM/giphy.gif" target="_blank">sprinkled on them</a>.<br>
</p><h3>Install Ansible on your machine
</h3><p>I’m not going to walk through this one step by step. Check out the <a href="https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html">Ansible install guide</a> for install guidance. As a Mac guy, I’ll do this
</p><pre class="prettyprint">brew install ansible
</pre><h3>Download the ECE Ansible role</h3><p>The Elastic provides an <a href="https://github.com/elastic/ansible-elastic-cloud-enterprise" target="_blank">Ansible role for ECE</a> on GitHub.
</p><p>The easiest way to get it is to use the Ansible Galaxy command-line tool.
</p><pre class="prettyprint">ansible-galaxy install git+https://github.com/elastic/ansible-elastic-cloud-enterprise.git
</pre><p>On MacOS, the role ends up in <code>~/$HOME/.ansible/roles/ansible-elastic-cloud-enterprise</code>
</p><h3>Setup an Ansible project for our ECE deployment</h3><p>Ansible appears to like operating inside of a filesystem folder that has all the bits it expects. I could be wrong. I don’t know, I’m just starting out, back off!
</p><p>What worked for me was creating a folder with a couple configuration files: one that described my AWS hosts and one that described the actions to perform, i.e. the playbook.
</p><ol>
	<li>Create a directory.</li>
	<pre class="prettyprint">mkdir -p /workspace/ansible/ece-test
</pre>
	<p>Yeah, I <code>-P</code>’d it: you will do things my way! ...or you can do it yours and put it wherever you like.
	</p>
</ol><h3>Create an Ansible inventory for our ECE environment</h3><p><p">For our needs, the only thing we need to do is setup the <a href="https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html" target="_blank">inventory</a>. There’s a ton you can do with an Ansible inventory, but our needs are basic, and so is our inventory setup</p">
</p><ol>
	<li>Create a new file named hosts in our project directory.</li>
	<pre class="prettyprint">vi /workspace/ansible/ece-test/hosts
</pre>
	<li>Build the inventory.
	</li>
	<p>Use the below as reference, substituting your public DNS info, AZ info, and private keys.
	</p>
	<pre class="prettyprint">[primary]
ec2-100-26-213-139.compute-1.amazonaws.com

[primary:vars]
availability_zone=us-east-1e

[secondary]
ec2-3-83-78-122.compute-1.amazonaws.com

[secondary:vars]
availability_zone=us-east-1d

[tertiary]
ec2-54-204-78-126.compute-1.amazonaws.com

[tertiary:vars]
availability_zone=us-east-1a

[aws:children]
primary
secondary
tertiary

[aws:vars]
ansible_ssh_private_key_file=/Users/barretta/.ssh/barretta-aws-east.pem
ansible_user=centos
ansible_become=yes
device_name=nvme0n1
</pre>
</ol><p>What this does is:
</p><ul>
	<li>Define three host groups, one for each availability zone - we must use “primary” as the group name with our primary host.</li>
	<li>Set the avalilability_zone variable for each host group.
	<ul>
		<li>These can be set to any values, but I used the actual AZ name where my hosts reside.</li>
	</ul></li>
	<li>Group the groups under a parent named “aws”.</li>
	<li>Setup some variables on that parent so Ansible will:
	<ul>
		<li>ssh into our machines with our key.</li>
		<li>use the centos user when connecting.</li>
		<li>su to root</li>
	</ul></li>
	<li>Tell the elastic-cloud-enterprise role to use <code>/dev/nvme0n1</code> as the main filesystem device. This value depends entirely on your environment, but is the right value for i3s in AWS.</li>
</ul><p>It’s also important to use the public DNS for your hosts vs using the IP. This is because of how AWS networking works for intra-security groups rules. Briefly, when you create a security group rule and have it apply for hosts within the same security group, the private IPs of the hosts are used, not the public IPs. So while you could specify the public IP CIDR block when defining some outbound security group rules, this method is faster and easier IMHO.
</p><h3>Create an ECE playbook
</h3><p>We’re almost there. We’re going to look back to the <a href="https://github.com/elastic/ansible-elastic-cloud-enterprise" target="_blank">GitHub project page</a> and grab the example playbook it contains as part of the README. Basically, we’re going to copy and paste the playbook into a file.
</p><ol>
	<li>Create a new file named <code>ece.yml</code> in our project directory.
	</li>
	<pre class="prettyprint">vi /workspace/ansible/ece-test/ece.yml
</pre>
	<li>Paste in what’s below:</li>
	<pre class="prettyprint">---
- hosts: primary
  gather_facts: true
  roles:
    - ansible-elastic-cloud-enterprise
  vars:
    ece_primary: true

- hosts: secondary
  gather_facts: true
  roles:
    - ansible-elastic-cloud-enterprise
  vars:
    ece_roles: [director, coordinator, proxy, allocator]

- hosts: tertiary
  gather_facts: true
  roles:
    - ansible-elastic-cloud-enterprise
  vars:
    ece_roles: [director, coordinator, proxy, allocator]
</pre>
</ol><p>The playbook is pretty straight forward: we list our three hosts by the names we’ve given them in the inventory file, mapped each host to the Elastic-created Ansible role we installed via Ansible Galaxy to do the ECE install and configuration, and specified which specific ECE roles each host should take. As you can see, by assigning the “ece_primary” role, that host becomes the First Host (which is important in ECE installations) and also implicitly gains all ECE roles.
</p><h3>Run the playbook
</h3><p><p<a href="https://thumbs.gfycat.com/ComfortableUnsteadyIlladopsis-size_restricted.gif" target="_blank">And here we go! Nono, this will go smoothly.</p<a>
</p><ol>
	<li>Run it</li>
	<pre class="prettyprint">ansible-playbook -i hosts ece.yml
</pre>
</ol><h3>Celebrate!
</h3><p><a href="https://giphy.com/gifs/hulu-happy-celebration-seinfeld-3oz8xHEisOJuebgmhq/fullscreen" target="_blank">Hooray!</a>
</p><h3>Or fix any mistakes...doh
</h3><p>If you messed up a bit as you went (like me), and hit an error like “<code>Container frc-directors-director was not found</code>” then the simplest thing to do is go in and delete all the docker images and delete the data directory thusly:
</p><ol><li>Log in to the host with the error (you can see the IP from preceding log lines).</li><li>Switch to the elastic user.</li><pre class="prettyprint">sudo su - elastic
</pre><li>Kill all the docker containers.</li><pre class="prettyprint">docker kill $(docker ps -q)
</pre><li>Delete the data directory a resource busy error is ok.</li><pre class="prettyprint">sudo rm -rf /mnt/data
</pre><li>Switch back to your "main" machine and run the playbook again.</li><pre class="prettyprint">ansible-playbook -i hosts ece.yml
</pre><li>Return to previous section and celebrate.</li></ol><h2>Conclusion
</h2><p>At this point, you should be the proud new owner of a three node, three zone ECE environment. Congrats!
</p><p>Open a browser and navigate to the console using the info Ansible output at the end of the playbook execution. Mine looks like:
</p><pre class="prettyprint">TASK [elastic-cloud-enterprise : debug] 
***************************************************************
ok: [ec2-54-204-78-126.compute-1.amazonaws.com] =&gt; {
    "msg": "Adminconsole is reachable at: https://ec2-100-26-213-139.compute-1.amazonaws.com:12443"
}
TASK [elastic-cloud-enterprise : debug] 
***************************************************************
ok: [ec2-54-204-78-126.compute-1.amazonaws.com] =&gt; {
    "msg": "Adminconsole password is: somethinglongandnarly"
}
</pre><p>Once in, click on the Platform link to verify you have three machines in three zones.
</p><p>You now face a choice: bask in the glow of your newfound powers over the dark forces of modern devops technology or use ECE to <a href="https://www.elastic.co/guide/en/cloud-enterprise/2.4/ece-create-deployment.html" target="_blank">spin up some Elasticsearch clusters</a>. Or, do one first and then the other...that’s what I’d do, perhaps spending too long basking in the <a href="https://comb.io/SVS4kW" target="_blank">warm glowing warming glow</a>.
</p>

colour-1885352.jpg

ece-aws-ansible.jpg

Automating the install of Elastic Cloud Enterprise on AWS with Ansible

<p>Anomaly detection, one of the <a href="/what-is/elastic-stack-machine-learning">Elastic machine learning features</a>, has been a great way of signaling when there’s an issue in a dataset. Until now, there’s been no easy way to focus in on anomalies for a particular field. This is especially true when the field of interest is not one of the top N shown in the results view.
</p><p>The release of Kibana 7.2 introduces a new query bar in the Machine Learning app in Kibana to make it easier for users to search the anomaly results for specific potential influencers.
</p><h2>Using the query bar with NGINX events</h2><p>The data used in the following example contains anonymized data from elastic.co web access logs processed by Filebeat’s <a href="/guide/en/beats/filebeat/7.2/filebeat-module-nginx.html">NGINX</a> module configuration (14,479,391 events collected over 38 days).
</p><p>In this example, we’ve created a job to detect unusual activity in the number of incoming requests — something that could be indicative of an attack.
</p><p>A normal user is limited by how quickly they can manually navigate to website pages, whereas an attacker or bot may be able to flood a website with requests. Assuming the majority of clients are normal users, we can detect atypical behavior by looking at unusually high event rate for a client:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4ceb11958b75308f/5d37569c5cb66453b1279a50/ml-anomaly-explorer-high-event-rate-blog.jpg" data-sys-asset-uid="blt4ceb11958b75308f" alt="Display of IP addresses with high event rates" style="display: block; margin: auto;">
</p><p>The results show several anomalous clients; 185.78.31.85 especially seems to show anomalous activity over a longer period of time.
</p><p>In this case, we are interested in only viewing results for that specific IP. Using the query bar, we can narrow the results for 185.78.31.85.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2cedaad33047c466/5d3757711f2f0457e07e2b43/ml-anomaly-explorer-ip-search-blog.gif" data-sys-asset-uid="blt2cedaad33047c466" alt="Display narrowing results to specific IP address" style="display: block; margin: auto;">
</p><p>With the query bar, we now have the ability to search through anomaly results to focus in on one or more selected <a href="/guide/en/kibana/7.2/job-tips.html#influencers">influencers</a>. The query bar suggests influencer field names and values for selected jobs as it receives input. Auto-suggest makes it easy to view possible fields and values we can search by; we’re no longer limited to seeing only the top ten shown in the results view.
</p><p>The query bar accepts input in <a href="/guide/en/kibana/7.2/kuery-query.html">KQL syntax</a>. Users can combine queries using AND or OR keywords and can filter by field value only.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt256ed8acee0a410b/5d3757ad1f2f0457e07e2b49/ml-anomaly-explorer-combine-queries-blog.png" data-sys-asset-uid="blt256ed8acee0a410b" alt="Combining queries to show two different IP addresses" style="display: block; margin: auto;">
</p><p>Wildcard queries are supported. Searching for <code>source.address: 185*</code>, for example, narrows the displayed results to all IP values beginning with <code>185</code>.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltca1e436f81fccf3f/5d375899e2285157535437e7/ml-anomaly-explorer-wildcard-query-blog.png" data-sys-asset-uid="bltca1e436f81fccf3f" alt="Query including a wildcard" style="display: block; margin: auto;">
</p><p>If the influencer of interest is listed in Top Influencers, clicking the plus icon next to it will add that field/value combination to the query bar and apply the search. Removing the search can be accomplished by clicking the minus icon. This can also be done from the results table.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc36e08b0df9ee8cf/5d3758d32a506857d64efc9a/ml-anomaly-explorer-influencers-blog.gif" data-sys-asset-uid="bltc36e08b0df9ee8cf" alt="Using Top Influencers to alter query" style="display: block; margin: auto;">
</p><p>At Elastic, we like to build search into everything we do. The query bar provides an easy way to search through anomalies for one or more particular influencers, allowing for easier result navigation. This is especially useful when a field is not one of the top N shown in the results view.
</p><p>Interested in trying it out? Nab a <a href="/elasticsearch/service">free 14-day trial of the Elasticsearch Service</a>. If you have feedback or questions we invite you to join the conversation in the machine learning <a href="https://discuss.elastic.co/">Discuss forum</a>.
</p>.

kibana-improved-querying.jpg

kibana-query-cowboy.jpg

Find influencers faster with the machine learning Anomaly Explorer query bar

<p>You may have noticed that sometimes the anomaly results you get from our anomaly detector are marked with crosses instead of circles. These crosses indicate a special type of anomaly: a multi-bucket impact anomaly. We first introduced these new types of results in <a href="/blog/changes-to-elastic-machine-learning-anomaly-scoring-in-6-5">our blog post on the 6.5 release</a>. In short, this means that the anomaly was not due to a single anomalous value in the current bucket, but rather to an anomalous sequence of values in the previous 12 buckets. The values themselves may not be unusual and may even be within the model bounds, but in a sequence they make up an anomalous progression of values with respect to the history of the dataset.
</p><p>Let’s take a closer look at these anomalies to understand them better.
</p><h2>Why are multi-bucket anomalies useful?
</h2><p>A key requirement for using ML anomaly detection in the Elastic Stack is deciding on a bucket span. This value determines the granularity at which our anomaly detection algorithms observe and alert on the data. What happens if we have a dataset where anomalies usually occur on a five-minute range, but we are also interested in detecting anomalies that occur on a longer time range? For example, we would like to be alerted in sudden drops in visitor count over a five-minute range, but would also want to know if we experience consecutive drops in visitor counts over an hour. This is where multi-bucket anomalies come in handy. Instead of focusing on single anomalous buckets, multi-bucket anomalies alert us to the presence of anomalous regions of time where multiple sequential buckets exhibit unusual behavior. If you look back to the blog post we published with our <a href="/blog/changes-to-elastic-machine-learning-anomaly-scoring-in-6-5">6.5 release,</a> we showed how the introduction of multi-bucket impact anomalies allowed us to detect deviations from a usual pattern in a timeseries.
</p><h2>How do I interpret multi-bucket anomalies?
</h2><p>In order to understand why your dataset contains a multi-bucket anomaly, look at the behavior of the data in the last 12 bucket spans. To illustrate some of the concepts discussed in this blog post, we will be using an example dataset that illustrates the aggregate number of requests a company’s server receives. Let’s take a moment to look at the general characteristics of this dataset in Figure 1. The dataset exhibits a periodic weekly pattern as well as a daily pattern.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdaa4ed52bb43ac2c/5d360b9e241468159d6fb25a/multibucket-anomaly-image1-blog.jpg" data-sys-asset-uid="bltdaa4ed52bb43ac2c" alt="Time series analysis of server requests" width="1200" style="display: block; margin: auto;">
</p><figcaption>Figure 1: Timeseries of the number of requests received by a company’s server.
</figcaption><p>As one might expect, there are two temporal patterns that are immediately obvious: a daily pattern exhibiting high activity between 7am and 7pm and a weekly pattern exhibiting spikes of activity that correspond to the five working days with low to zero activity on the weekends. In addition to the variations at different times during the day and weekly variations, there are also hourly variations.
</p><p>Suppose we want to take a very granular view of this data and set the bucket span to be 30 minutes. This will mean we will detect anomalies that occur on a 30-minute bucket timescale. But, we might also be interested in detecting things on a broader timescale — say we might be interested in days that are unusual or weeks that are unusual.
</p><p>This is a prime use case for multi-bucket impact anomalies — we will run a machine learning job that detects anomalies in the count of requests sent to the company’s server with a 30-minute bucket span, which will provide insights into anomalies that occur on short time scales, while also detecting unusual periods of up to 6 hours (12 buckets * 30 minutes). To show some subtle effects that multi-bucket impact anomalies introduce into machine learning jobs with “sided” configurations (high_* or low_* detectors), we will also run the same job with a low_count and a high_count detector and compare the differences in the results in the following sections of this post.
</p><p>
	Let’s take a look at what the count detector was able to find in our dataset. As we can see from Figure 2, something strange took place during the working week starting on Monday 2017-07-10. During a system upgrade (which started on Thursday afternoon and continued well into Friday), there was an outage, which is detected as a series of multi-bucket impact anomalies (marked with crosses in the UI). The activity profiles on Thursday and Friday differ from those of a typical working week.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8094091f265b3b92/5d360bec241468159d6fb260/multibucket-anomaly-image2a-blog.jpg" data-sys-asset-uid="blt8094091f265b3b92" alt="Time series analysis showing multi-bucket anomalies" width="1200" style="display: block; margin: auto;">
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt553f3ffd33e21a6f/5d360c3103710e1970f3da11/multibucket-anomaly-image2b-blog.jpg" data-sys-asset-uid="blt553f3ffd33e21a6f" alt="Normal time series analysis for comparison" width="1200" style="display: block; margin: auto;">
</p><figcaption>Figure 2: Timeseries of a working week during which the company upgraded to a new internal system. For comparison, the activity during a normal working week is also included.
</figcaption><p>Zooming in to a section of time on Friday (Figure 3), we can see that in the 12 bucket period including and preceding the anomaly annotated with A, the values on each bucket are trailing below the midpoint of the model bounds, which is an approximate indicator of how we decide whether or not to mark something as a multi-bucket impact anomaly.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6e45898260338425/5d360c65c780b1161fc58983/multibucket-anomaly-image3-blog.jpg" data-sys-asset-uid="blt6e45898260338425" alt="Zoomed-in view of graph of multi-bucket anomalies" width="1200" style="display: block; margin: auto;">
</p><figcaption>Figure 3: A zoomed in view of the series of multi-bucket anomalies occurring during a company-wide system upgrade and subsequent outage. Approximate midpoint of model bounds marked with a dashed black line.
</figcaption><p>Another thing to observe is that the multi-bucket impact anomalies can occur on values that are within model bounds — sometimes even on values that are very close to the typical value observed for this region of the data. This, while counterintuitive, is a possible scenario, since we are alerting on an anomalous region instead of an anomalous bucket. It might not be too unusual to observe one bucket value that is slightly higher than is typical, but if we observe several buckets in a row, where the values are slightly higher than is typical, this could be a potential anomalous region to be investigated.
</p><p>Thus when you are examining multi-bucket impact results, it is important to bear in mind that these results are for a region of buckets instead of a single bucket and adjust your investigations accordingly.
</p><p>Finally, it’s good to note that when the multi-bucket anomaly detection algorithm examines a sequence of buckets, it gives more weight to the buckets that are closer in time to the bucket it is analyzing.
</p><h2>Do multi-bucket impact anomalies respect high_* and low_* sided detectors?
</h2><p>A common point of confusion when interpreting multi-bucket impact anomalies concerns their behaviors with sided detectors (for example a high_count or a low_count detector). Let’s examine an anomalous region picked out by the high_count detector with a 30-minute bucket span (see Figure 4).&nbsp;</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6ac713cce2357ebd/5d360ca68d63c919f994e452/multibucket-anomaly-image4-blog.jpg" data-sys-asset-uid="blt6ac713cce2357ebd" alt="30-minute bucket span of anomalous region" width="1200" style="display: block; margin: auto;">
</p><figcaption>Figure 4: Do multi-bucket impact anomalies respect the side of the detector? 
</figcaption><p>The first thing to notice here is the anomaly occurs when the actual value dips below the typical value for this region of the data. This may seem counterintuitive. After all, our anomaly detector was configured using the high_count function, so it should be looking for anomalies where the actual value exceeds the typical. This would indeed be true for regular anomalies, but not for multi-bucket anomalies. Since the latter depend on the observed values in multiple buckets as opposed to a single bucket, it is possible to be alerted about an anomalous region even if the actual value in the bucket where the multi-bucket anomaly is marked is below the typical.
</p><p>We do, however, respect the side (high_* or low_*) of the detector in the region as a whole. We would only mark something as a multi-bucket impact anomaly if the values of the buckets in the region as a whole were higher than usual. If you examine the anomaly marked with a black arrow in Figure 4, you will notice that the actual value in the current bucket is lower than the typical, yet the preceding buckets contain values that are above the midpoint of the model bounds, which leads the high_count detector to mark this as an anomalous region in time. In summary, this means that the region can contain individual buckets where the actual value is lower than typical (see the highlighted bucket in Figure 4).
</p><p>In the UI, the multi-bucket impact anomalies are marked with crosses in the charts but the same symbol is not carried over into the anomaly results table (as you can see in Figure 4 — the same anomaly is marked with a cross in the UI and with a circle in the chart). To view the multi-bucket impact in the anomaly results table, you can expand the row, which may not be obvious to first-time users.
</p><p>An example that displays that multi-bucket impact anomalies do respect detector side is shown in Figure 5. In this figure, we are displaying the results of the low_count and high_count detectors (both with a 30-minute bucket span) on July 5th, 2017.
</p><p>As you can see from Figure 5, the low_count detector detects a series of multi-bucket impact anomalies, but in the high_count detector we no longer detect those, because in the region of interest the bucket values are on average lower than typical and thus our high_* sided detector will not pick those up. Even though multi-bucket impact anomalies can occur on buckets where the individual bucket value is on the wrong side of the detector (that is, a lower-than-typical value marked as anomalous in a high_* detector), the region flagged by the multi-bucket anomaly on the whole respects the sidedness of the detector.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbc5904f57c491aed/5d360cc86e329a11cbb64aae/multibucket-anomaly-image5a-blog.jpg" data-sys-asset-uid="bltbc5904f57c491aed" alt="Graph showing low-count detectors" width="1200" style="display: block; margin: auto;">
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2d30e5911adab276/5d360d0b03710e1970f3da19/multibucket-anomaly-image5b-blog.jpg" data-sys-asset-uid="blt2d30e5911adab276" alt="Graph using high-count detector" width="1200" style="display: block; margin: auto;">
</p><figcaption>Figure 5: A low_count and a high_count detector on the same dataset pick out different multi-bucket impact anomalies.
</figcaption><h2>How are multi-bucket impact anomalies scored?
</h2><p>If you examine multi-bucket impact anomalies in the UI, you will notice that they are graded from high multi-bucket impact to low multi-bucket impact. If you are using the API to retrieve machine learning results, multi-bucket impact anomalies will have a <code>multi_bucket_impact</code> field, which will score the anomaly from –5 to 5. Anomalies with a <code>multi_bucket_impact</code> value of greater than or equal to 2 are considered multi-bucket impact anomalies and are marked with crosses in our UI. This score is independent of the anomaly score, which determines the severity of the alert and thus the colour of the multi-bucket impact cross in the UI.
</p><p>To further understand how we compute this score, consider the following. Every single bucket of data we examine with our anomaly detector can be anomalous in two ways: as part of a sequence of buckets that make up an anomalous region in time or on its own as a single bucket. For each bucket of data we examine, we compute two probabilities. The first probability captures the likelihood of observing a bucket value as extreme or more extreme based on the model of the data we have built up until this point in time. The second probability is based on a model of a multi-bucket feature — a kind of weighted average with a window of 12 buckets (the current bucket and the 11 buckets preceding it) that attempts to capture the anomalousness of a whole region the bucket is part of.
</p><p>We then examine these probabilities in relation to one another. If both the single bucket and the multi-bucket probability are roughly on the same order of magnitude, we assign a multi-bucket impact score of 0 — that is, there is no significant contribution to this anomaly from the region of bucket preceding it. On the other hand, if the single bucket probability is high (that is, it is a value that is likely to occur based on our model of the series) and the multi-bucket impact probability is low (the bucket is part of an anomalous sequence of buckets that is very unlikely to occur), we would assign a high multi-bucket impact value to indicate that the anomalousness of this bucket results from an anomalous region of time as opposed to a single anomalous bucket value.
</p><h2>Conclusion
</h2><p>Multi-bucket impact anomalies were developed to give you two perspectives of your data: a granular perspective determined by your bucket span configuration, and a more zoomed-out view, which is able to pick out anomalies in longer periods of time. Therefore, a key thing to keep in mind when examining multi-bucket impact anomalies in your results is that they are referring to the anomalousness of a group of buckets in a region of time, instead of a single bucket and thus it is entirely possible for a multi-bucket anomaly to occur at a bucket value that is within model bounds or on a bucket value that is lower than typical for a detector designed to pick out values higher than typical (and vice versa).
</p><p>
	Want to learn more? Check out our <a href="/guide/en/elastic-stack-overview/7.2/xpack-ml.html"> machine learning documentation</a>. Or hop in and try it out yourself with a <a href="/elasticsearch/service">free 14-day trial of the Elasticsearch Service</a>.
</p>

multibucket-anomaly-blog-banner.jpg

multibucket-anomaly-blog-thumb.jpg

Interpreting multi-bucket impact anomalies using Elastic machine learning features

<p>I’m pleased to announce the release of a brand new Logstash input: <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html">HTTP Poller</a>. With this new input you’ll be able to repeatedly poll one or more HTTP endpoints and turn the response into Logstash events. There are a number of practical uses for this plugin, like:
</p><ul>
	<li>Monitoring a daemon such as HAProxy or Apache’s HTTP stats end points for metrics such as total open connections, or the number of busy workers</li>
	<li>Checking that your website is up and responding in a timely manner</li>
	<li>Hitting a custom metrics endpoint in a webapp to gain deep insight into some process not exposed through logs</li>
</ul><p>The syntax for this plugin is dead simple to boot, as seen in the example below:
</p><pre class="prettyprint">input {
  http_poller {
    # List of urls to hit
    # URLs can either have a simple format for a get request
    # Or use more complex HTTP features
    urls =&gt; {
      some_service =&gt; "http://localhost:8000"
      some_other_service =&gt; {
        method =&gt; "POST"
        url =&gt; "http://localhost:8000/foo"
      }
    }
    # Maximum amount of time to wait for a request to complete
    request_timeout =&gt; 30
    # How far apart requests should be
    interval =&gt; 60
    # Decode the results as JSON
    codec =&gt; "json"
    # Store metadata about the request in this key
    metadata_target =&gt; "http_poller_metadata"
  }
}
output {
  stdout {
    codec =&gt; rubydebug
  }
}
</pre><p>This request will put the HTTP responses of the polled endpoints into the <code>message</code> field and provide metadata like response timing and HTTP response headers in the <code>http_poller_metadata</code> field.
</p><h2><a id="Using_HTTP_Poller_to_monitor_website_status_43"></a>Using HTTP Poller to monitor website status</h2><p>We’ll start with a simple example; using HTTP Poller to monitor whether a given URL is up, down, or responding slowly. The following Logstash config will hit a webserver on <code>http://localhost:8000</code>. If you don’t have one up, you can start one that takes a variable length of time to return a JSON response by running <code>ruby -rsinatra -e 'set :port, 8000; get("/") { n= rand(10) / 10.0; sleep n; "{\"t\": #{n}}" }'</code> in your console (assuming you have ruby installed and have installed the sinatra gem with <code>gem install sinatra</code>). After that, try running Logstash with the sample config below. The sample config has been heavily annotated to make reading it easy, even for a complete logstash novice.
</p><p>Using the config below, you can generate Kibana charts like the one just underneath this paragraph, showing the ratio of slow to fast requests to your service over time. If you have Elastic's <a href="https://www.elastic.co/what-is/elasticsearch-alerting">Watcher</a> set&nbsp;up, you can use that to automatically send you alerts when you receive slow requests as well
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c930342dfd738263220/download?uid=blt1258163f65d4cf37?uid=blt1258163f65d4cf37" data-sys-asset-uid="blt8c4282f417a5443e" style="max-width:100%;">
</p><pre class="prettyprint">input {
  http_poller {
    urls =&gt; {
      "localhost" =&gt; "http://localhost:8000"
    }
    automatic_retries =&gt; 0
    # Check the site every 10s
    interval =&gt; 10
    # Wait no longer than 8 seconds for the request to complete
    request_timeout =&gt; 8
    # Store metadata about the request in this field
    metadata_target =&gt; http_poller_metadata
    # Tag this request so that we can throttle it in a filter
    tags =&gt; website_healthcheck
  }
}
filter {
  # The poller doesn't set an '@host' field because it may or may not have meaning
  # In this case we can set it to the 'name' of the host which will be 'localhost'
  # The name is the key used in the poller's 'url' config
  if [http_poller_metadata] {
    mutate {
      add_field =&gt; {
        "@host" =&gt; "%{http_poller_metadata[name]}"
      }
    }
  }
  # Classify slow requests
  if [http_poller_metadata][runtime_seconds] and [http_poller_metadata][runtime_seconds] &gt; 0.5 {
    mutate {
      add_tag =&gt; "slow_request"
    }
  }
  # Classify requests that can't connect or have an unexpected response code
  if [http_request_failure] or
     [http_poller_metadata][code] != 200 {
     # Tag all these events as being bad
     mutate {
       add_tag =&gt; "bad_request"
     }
 }
 if "bad_request" in [tags] {
    # Tag all but the first message every 10m as "_throttled_poller_alert"
    # We will later drop messages tagged as such.
    throttle {
      key =&gt; "%{@host}-RequestFailure"
      period =&gt; 600
      before_count =&gt; -1
      after_count =&gt; 1
      add_tag =&gt; "throttled_poller_alert"
    }
    # Drop all throttled events
    if "throttled_poller_alert" in [tags] {
      drop {}
    }
    # The SNS output plugin requires special fields to send its messages
    # This should be fixed soon, but for now we need to set them here
    # For a more robust  and flexible solution (tolerant of logstash restarts)
    # Logging to elasticsearch and using the Watcher plugin is advised
    mutate {
      add_field =&gt; {
        sns_subject =&gt; "%{@host} is not so healthy! %{@tags}"
        sns_message =&gt; '%{http_request_failure}'
        codec =&gt; json
      }
    }
  }
}
output {
  # Catch throttled messages for request failures
  # If we hit one of these, send the output to stdout
  # as well as an AWS SNS Topic
  # UNCOMMENT THIS TO ENABLE SNS SUPPORT
  #if "http_request_failure" in [tags] {
  #  sns {
  #    codec =&gt; json
  #    access_key_id =&gt; "YOURKEY"
  #    secret_access_key =&gt; "YOURSECRET"
  #    arn =&gt; "arn:aws:sns:us-east-1:773216979769:logstash-test-topic"
  #  }
  #}
  elasticsearch {
    protocol =&gt; http
  }
  stdout {
    codec =&gt; rubydebug
  }
}
</pre><h2><a id="Using_HTTP_Poller_to_monitor_HAProxy_stats_and_Apache_serverstatus_pages_154"></a>Using HTTP Poller to monitor HAProxy stats and Apache server-status pages</h2><p>Both <a href="http://www.haproxy.org/">HAProxy</a> and <a href="http://httpd.apache.org/">Apache HTTPD</a> support stats API endpoints for to get information like the number of open connections. In the following example I’ll show how to setup Logstash to record this information to elasticsearch.
</p><p>The key takeaway here is that you can use the HTTP Poller to monitor the health of HAProxy and Apache with greater insight than you’d get with logs alone. Additionally, you can use it to trigger alerts via AWS SNS topics when those thresholds are passed. Those SNS topics can be configured to send texts or emails to alert an operator.
</p><p>Implementing this requires you to <a href="http://serverfault.com/a/198271">enable the stats port on HAProxy</a> as well as <a href="http://httpd.apache.org/docs/2.2/mod/mod_status.html">enable mod_status on apache</a>. To make this easier to try out I’ve prepared a script that will launch a set of docker machines with this stuff all setup. To run it you’ll just need bash, docker, and <a href="https://docs.docker.com/machine/">docker-machine</a>. Try checking out all the code in <a href="https://github.com/logstash-plugins/logstash-input-http_poller/tree/master/examples/ha-apache">this directory</a>. After you have the code run <code>buildit.sh</code>, which will launch the docker machines and write out a sample <code>logstash.conf</code> file. After you’ve done that, just run <code>logstash -f logstash.conf</code> to see it with action. If you hit the haproxy server (whose address will be printed out by buildit) with traffic and load the the <code>kibana.elasticdump</code> file into <code>.kibana</code> with <a href="https://github.com/taskrabbit/elasticsearch-dump">elasticdump</a>,  you should see something like the kibana dashboard below.
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c933dc542cb38c083a3/download?uid=bltfebb17b70ea68e55?uid=bltfebb17b70ea68e55" data-sys-asset-uid="bltc9e6da078bab0eae" style="max-width:100%;"><br>
</p><p>Notice that we can graph such things as HAProxy sessions, the response times of polling requests (which rise as the server is more and more saturated, and which HAProxy services are active. All things that cannot be exposed via plain log data, but can be reached via HTTP polling.
</p><p>If you’d rather not run the examples to see the config used, I’ve reproduced a well commented version of it below:
</p><pre class="prettyprint">input {
  # Setup one poller for httpd, we keep these separate to tag them differently
  http_poller {
    urls =&gt; {
      "custom_httpd_t1" =&gt; { url =&gt; "http://192.168.99.100:8001/server-status?auto"}
      "custom_httpd_t2" =&gt; { url =&gt; "http://192.168.99.100:8002/server-status?auto"}
      "custom_httpd_t3" =&gt; { url =&gt; "http://192.168.99.100:8003/server-status?auto"}
     }
     tags =&gt; apache_stats
     codec =&gt; plain
     metadata_target =&gt; http_poller_metadata
     interval =&gt; 1
  }
  # Another poller, this time for haproxy
  http_poller {
    urls =&gt; {
      ha_proxy_stats =&gt; "http://statsguy:statspass@192.168.99.100:1936/;csv"
    }
    tags =&gt; haproxy_stats
    codec =&gt; plain
    metadata_target =&gt; http_poller_metadata
    interval =&gt; 1
   }
    # Pull the regular Apache/HAProxy logs via docker commands
    # This is a hack for the purposes of this example
  pipe {
    command =&gt; "docker logs -f custom_httpd_t1"
    tags =&gt; [ "apache" ]
    add_field =&gt; { "@host" =&gt; "custom_httpd_t1" }
  }
  pipe {
    command =&gt; "docker logs -f custom_httpd_t2"
    tags =&gt; [ "apache" ]
    add_field =&gt; { "@host" =&gt; "custom_httpd_t2" }
  }
  pipe {
    command =&gt; "docker logs -f custom_httpd_t3"
    tags =&gt; [ "apache" ]
    add_field =&gt; { "@host" =&gt; "custom_httpd_t3" }
  }
    pipe {
      command =&gt; "docker logs -f custom_haproxy"
      tags =&gt; [ "haproxy" ]
      add_field =&gt; { "@host" =&gt; "custom_haproxy" }
    }
}
filter {
  if [http_poller_metadata] {
    # Properly set the '@host' field based on the poller's metadat
    mutate {
      add_field =&gt; {
        "@host" =&gt; "%{http_poller_metadata[name]}"
      }
    }
  }
  # Processed polled apache data
  if "apache_stats" in [tags] {
    # Apache stats uses inconsistent key names. Make sure all fields are camel cased, no spaces
    mutate {
      gsub =&gt; ["message", "^Total ", "Total"]
    }
    # Parse the keys/values in the apache stats, they're separated by ": '
    kv {
      source =&gt; message
      target =&gt; apache_stats
      field_split =&gt; "\n"
      value_split =&gt; ":\ "
      trim =&gt; " "
    }
    # We can make educated guesses that strings with mixes of numbers and dots
    # are numbers, cast them for better behavior in Elasticsearch/Kibana
    ruby {
      code =&gt; "h=event['apache_stats']; h.each {|k,v| h[k] = v.to_f if v =~ /\A-?[0-9\.]+\Z/}"
    }
  }
  # Process polled HAProxy data
  if "haproxy_stats" in [tags] {
    split {}
    # We can't read the haproxy csv header, so we define it statically
    # This is because we're working line by line, and so have no header context
    csv {
       target =&gt; "haproxy_stats"
       columns =&gt; [ pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime ]
    }
    # Drop the haproxy CSV header, which always has this special value
    if [haproxy_stats][pxname] == "# pxname" {
      drop{}
    }
    # We no longer need the message field as the CSV filter has created separate
    # fields for data.
    mutate {
      remove_field =&gt; message
    }
    # Same as the cast we did for apache
    ruby {
      code =&gt; "h=event['haproxy_stats']; h.each {|k,v| h[k] = v.to_f if v =~ /\A-?[0-9\.]+\Z/}"
    }
  }
  # Process the regular apache logs we captured from the docker pipes
  if "apache" in [tags] {
    grok {
      match =&gt; [ "message", "%{COMMONAPACHELOG:apache}" ]
    }
  }
  # We're going to email ourselves on error, but we want to throttle the emails
  # so we don't get so many. This says only send one every 5 minutes
  if "_http_request_failure" in [tags] {
    throttle {
      key =&gt; "%{@host}-RequestFailure"
      period =&gt; 600
      before_count =&gt; -1
      after_count =&gt; 1
      add_tag =&gt; "_throttled_poller_alert"
    }
    # Drop all throttled events
    if "_throttled_poller_alert" in [tags] {
      drop {}
    }
    # The SNS output plugin requires special fields to send its messages
    # This should be fixed soon, but for now we need to set them here
    mutate {
      add_field =&gt; {
        sns_subject =&gt; "%{@host} unreachable via HTTP"
        sns_message =&gt; "%{http_request_failure}"
      }
    }
  }
}
output {
  # Store everything in the local elasticsearch
  elasticsearch {
    protocol =&gt; http
  }
  # Catch throttled messages for request failures
  # If we hit one of these, send the output to stdout
  # as well as an AWS SNS Topic
  # UNCOMMENT TO ENABLE SNS
  #if "_http_request_failure" in [tags] {
  #  sns {
  #    codec =&gt; json
  #    access_key_id =&gt; "YOURKEY"
  #    secret_access_key =&gt; "YOURSECRET"
  #    arn =&gt; "arn:aws:sns:us-east-1:773216979769:logstash-test-topic"
  #  }
    stdout {
      codec =&gt; rubydebug
    }
  }
}
</pre><h2><a id="Using_the_HTTP_Client_Mixin_in_Your_Own_Plugin_342"></a>Using the HTTP Client Mixin in Your Own Plugin</h2><p>HTTP Poller is the first plugin to use <a href="https://github.com/logstash-plugins/logstash-mixin-http_client">logstash-mixin-http_client</a>. If you need to add an HTTP client to a plugin you’re writing consider using the <code>HttpClient</code> mixin. This mixin will add a bunch of well validated configuration options and sane defaults to your plugin for free. Using it is as simple as adding <code>include LogStash::PluginMixins::HttpClient</code> to the body of your plugin. This will expose a new <code>client</code> method in your plugin class, which is an instance of the <a href="https://github.com/cheald/manticore">Manticore</a> http client. Manticore is a well written and performant client is based on Apache Commons HTTP. Of note is Manticore’s ability to execute requests asynchronously using thread pools with a simple API
</p><h2><a id="Wrapping_Up_346"></a>Wrapping Up</h2><p>I hope these examples have been useful! If you find any other uses for the http input poller, <a href="https://discuss.elastic.co/">let us know</a>! If you think you’ve found a bug in it, please <a href="https://github.com/logstash-plugins/logstash-input-http_poller">submit an issue</a>.
</p>

HTTP Poller, Opening up a New World for Logstash

<p>Earlier this week, an <a href="https://issues.apache.org/jira/browse/LOG4J2-1863">advisory</a> was released detailing an object deserialization security flaw in the way Apache Log4j version 2 processes input data (<a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5645">CVE-2017-5645</a>). This flaw would give a remote attacker the ability to execute code of their choosing within the JVM process listening for Log4j events.
</p><span id="docs-internal-guid-8a44ce56-8bfe-40df-4279-06f3786077fd"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;" rel="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">In a default Logstash install, the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html">Log4j plugin</a> is installed but not enabled. If you aren't explicitly using this plugin in your configuration, you are not affected by this issue.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">When used in the Logstash pipeline, the Logstash log4j input plugin accepts Log4j version 1 data from remote applications. Often, any client is able to connect to Logstash because the connection offers no authentication. Given the very purpose of Logstash is to be an endpoint for receiving log data for an organization, it may not be practical to firewall the system as a form of protection. It is our expectation that Logstash will be easily reachable in most environments, and will accept whatever data is passed into it.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">The currently known exploits for Java object deserialization do not work against default Logstash deployments, but the vulnerability is still present even without a known exploit. We recognize that this doesn’t necessarily mean a Logstash isn’t vulnerable to this flaw, it simply means we’re not aware of a weaponized exploit.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">The Elastic Security Team does not believe the log4j input can be made 100% invulnerable &nbsp;given the way it receives log data from arbitrary sources. By its very nature, object deserialization is difficult to secure and may be impossible to secure when parsing remote untrusted data. The general consensus in the security community is that if you must do object deserialization, it should only be done between systems that have a high level of trust. We do not have this level of trust with expected Logstash clients.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><h3 dir="ltr">Remediation</h3><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">We have patched the version of Log4j shipped in Logstash against this particular attack using a variant of the patch from the updated version of Apache Log4j. Updates for Logstash will be included in a future release. This will improve the security of the Log4j input, but we continue to have reservations about its security given the prior paragraph.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Existing Logstash v5.x and v2.4 users can upgrade the log4j input to receive this fix today by doing the following:
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><pre style="code">bin/logstash-plugin update logstash-input-log4j
</pre><pre style="code">% bin/logstash-plugin update logstash-input-log4j
Updating logstash-input-log4j
Updated logstash-input-log4j 3.0.3 to 3.0.5
</pre><h3 dir="ltr">Deprecation Notice
</h3><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Based on the reasons stated above, we are <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html#_deprecation_notice">deprecating</a> the Log4j input. Our recommendation is for current Log4j input users to stop using log4j’s SocketAppender in their applications. For safe transport of log4j logs, users should configure log4j to write logs to disk and use <a href="https://www.elastic.co/beats/filebeat">Filebeat</a> to forward to log information to Logstash. Setting up Filebeat to ship your local logs&nbsp;is easy and we've provided migration steps&nbsp;in <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html">these&nbsp;docs</a>. This solution removes object deserialization from being used in an insecure manner. Additionally we have marked the Log4j plugin as deprecated and are going to remove Log4j support in Logstash 6.0.<br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Elastic would like to thank Marcio Almeida de Macedo of Red Team at Telstra for alerting us of this issue.
</p></span>

blog-logstash-banner.jpg

The future of Log4j input in Logstash

<p>Throughout the day, news spread of a <a href="https://www.nytimes.com/2017/06/27/technology/ransomware-hackers.html">ransomware attack</a> hitting Europe and Russia, before making its way into the United States. By one estimate, it has already impacted tens of thousands computers, including pharmaceutical giant Merck, Russian oil company Rosneft, Danish shipping company AP Moller-Maersk, and a <a href="https://www.independent.co.uk/news/world/europe/chernobyl-ukraine-petya-cyber-attack-hack-nuclear-power-plant-danger-latest-a7810941.html">Ukrainian company</a> responsible for monitoring Chernobyl’s radiation system. Today’s ransomware attack follows closely on the heels of the widespread <a href="https://www.endgame.com/blog/technical-blog/dont-wcry-youve-got-endgame">WannaCry ransomware</a> infection, which struck <a href="http://money.cnn.com/2017/05/15/technology/wannacry-hack-responsible-hackers/index.html?iid=hp-toplead-dom">300,000 machines</a> in May, and just last week debilitated computers at <a href="https://www.engadget.com/2017/06/21/wannacry-ransomware-honda-shut-down/">Honda.</a>. Given the widespread impact and potential for propagation, we quickly tested the Endgame platform against this latest Petya-like ransomware. Endgame’s layered approach once again proved up to the task, catching and remediating the ransomware before it could inflict damage.</p><h2>Ransomware - Today’s Hottest Attack Trend</h2><p>2017 already has proven to be a stellar year for ransomware, and we’re just half-way through it. Experts are divided whether this is a variant of Petya or just Petya-like, and some have recently dubbed it Goldeneye, which is a hybrid of Petya and another ransomware variant, Misha. For the sake of simplicity, we will refer to it as Petya. Despite this nomenclature disagreement, there is general consensus that it has already had a large impact.  </p><p>As our ransomware timeline below illustrates, the Petya ransomware variant emerged last year. It is just one of the many new variants of ransomware that continue to shape the threat landscape. Petya overwrites the master boot record (MBR) with a custom bootloader from user space by the DLL and then the system is rebooted. Upon reboot, the custom bootloader that overwrote the MBR is loaded, which will encrypt the MFT and then display the ransom note, demanding $300 in bitcoin (which is proving problematic).</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt20ba1687cf93407e/5df84ff9f913b6068acf8e74/blog-ransomware-timeline-0.png" data-sys-asset-uid="blt20ba1687cf93407e" alt="blog-ransomware-timeline-0.png"></p><p>While Petya is similar to WannaCry, there are a few significant differences. With WannaCry, a kill switch was discovered relatively early, helping limit its propagation. There were signs late Tuesday that one was discovered for Petya as well, but it works more as a local inoculation than a global kill switch. The efficacy is also questionable because it is filename dependent.</p><p>Despite being spotted in the wild in early 2016, it is not terribly surprising to see Petya reappear. This time around, Petya takes advantage of the <a href="https://arstechnica.com/security/2017/04/nsa-leaking-shadow-brokers-just-dumped-its-most-damaging-release-yet/">EternalBlue exploit</a> released by the Shadow Brokers, similar to WannaCry.  It includes additional propagation methods as well, taking advantage of at least one other ShadowBrokers exploit and leveraging credential harvesting and usage of legitimate administrative tools to move laterally and inflict damage more broadly inside a network.  While many thought WannaCry was yet another <a href="https://www.ft.com/content/5ba47f70-2426-11e7-a34a-538b4cb30025?mhq5j=e1">wake-up call</a> to companies and governments, the quick diffusion of Petya demonstrates yet another wake-up call that went ignored.</p><h2>Protecting Against Petya and Beyond</h2><p>As we’ve discussed in previous <a href="https://www.endgame.com/blog/technical-blog/so-you-wanna-stop-ransomware-detailing-endgame-ransomware-protection">posts</a>, Endgame stops ransomware attacks by combining several layered, signatureless preventions including exploit prevention, malware prevention, and a variety of behavioral preventions.  Endgame’s <a href="https://www.endgame.com/blog/technical-blog/world-meet-malwarescore">MalwareScore TM</a>  catches Petya immediately, identifying the malware as malicious. MalwareScoreTM is a machine-learning based approach to detection and has proven highly effective at detecting known and unknown malware. Unlike some other machine learning-based solutions, MalwareScore does not require a cloud connection to be effective, protecting even those critical assets without an internet connection. We also tested Petya against many early versions of our MalwareScoreTM model, and all detect Petya, proving the protective power of our signatureless machine learning approach yet again. With Endgame deployed on a system, Petya is prevented without any prior knowledge of the ransomware variant. This is especially important because there is near certainty that Petya will not be the last, highly impactful ransomware attack. As long as these attacks remain profitable, they will continue.</p><p>Despite being highly effective, MalwareScoreTM comprises just one layer of Endgame’s multi-layer approach to protection. We know machine learning is not a silver bullet, and as impactful as ransomware is, there are other forms of malware-less and file-less attacks that require additional protection capabilities. In just a few weeks, our team will be in full force at BSidesLV, Black Hat, and DEF CON, talking about our research and the protection capabilities within our platform. Swing by our Black Hat booth 1360, and attend our conference talks covering everything from prevention to machine learning to the essential role of design and user experience in facilitating the analyst workflow.</p>

blog-thumb-generic-blue.png

Catching Petya: How Endgame Protects Against Another Global Attack

<p>In an attempt to appeal to information security executives and practitioners, some vendors have positioned Machine Learning (ML) – often liberally decorated as “Artificial Intelligence” (AI) – as a panacea for information security’s challenges.  In many cases, the hype has gone well beyond reality and become marketing nonsense.  Is there a useful place in information security for machine learning?  Most definitely. Will algorithms replace domain experts?  No.  Is machine learning automagically better than other carefully crafted detections because, well, math?  No.</p><p>Whether you’re on the vendor side (as I am) or on the user side (as many of you surely are), there is far more nuance to effectively using ML to achieve your security objectives.   ML can be extremely powerful, but it is not always the answer.  Even when it is the best tool for the job, it is very easy to screw up a ML model due to the use of bad data, incomplete data, the wrong features, or a variety of other factors.  Worse, vendors’ poorly explained metrics and unverified claims can make it impossible for users to recognize bad implementations.</p><p>Even the concepts themselves are misused and mischaracterized interchangeably, leading to the necessity for articles to ‘<a href="http://www.securityweek.com/demystifying-machine-learning-how-turn-buzzword-real-benefits-endpoint-security?imm_mid=0f5b54&cmp=em-webops-na-na-newsltr_security_20170829">demystify</a>’ AI and ML in security.   To <a href="https://www.linkedin.com/pulse/artificial-intelligence-machine-learning-deep-all-jiang-ciso-cissp">simplify</a>, AI is the practice of making computers behave or reason in an “intelligent” manner.  ML is a subset of AI, in which the computer makes predictions or insights from data.  In security, most discussions related to AI are actually just ML so we will stick to that terminology here for consistency.    </p><p>As ML remains a hot buzzword in security, the misconceptions surrounding it only seem to be increasing. I’ll first address many of the misconceptions of ML in security, and then highlight the advantages of good ML-driven solutions when properly implemented. Given the range of pitfalls and considerations required to optimize a ML-driven solution, a hybrid approach of domain expertise combined with ML is necessary. It is this interplay that has made our research and development so successful, and continues to innovate our detection and protection capabilities within the Endgame platform.  </p><h2>Common Misconceptions</h2><p>Various myths about ML are gaining a foothold in the security community.  Many industry veterans are rightly skeptical and growing increasingly cynical as they watch vendors claim their magical ML solutions will solve all security problems. Machine learning is <a href="http://sdtimes.com/black-hat-usa-2017-machine-learning-not-silver-bullet-security/">not the silver bullet</a> often claimed.  Below are  four of the biggest misconceptions in security that result from that type of messaging.</p><ol><li>ML REPLACES THE NEED FOR SKILLED EMPLOYEES: This is not likely in infosec. According to some projections, the industry will face close to a <a href="http://www.techrepublic.com/article/the-world-needs-more-cybersecurity-pros-but-millennials-arent-interested-in-the-field/">two million</a> workforce shortage by 2022.  Demand for talent is likely to outstrip supply for a long time.  Vendors need experts well versed in both ML and security to properly implement ML-driven security products.  In addition, security teams need people to interpret the results and take actions via their tools, whether they are ML-driven or not.  Some organizations may be able to reduce the need for additional resources, but don’t expect a significant drop.  ML can augment your current workforce, but the value of your human capital isn’t going down anytime soon.</li><li>ML IS INHERENTLY BETTER THAN HUMANS AT FINDING AND STOPPING INTRUSIONS: There are instances where this is true, but it is not universal. To be effective, solutions which are implemented using ML need to be trained, tuned, and tested by domain experts who understand the problem.  Solutions which lack this rigor tend to be ineffective, noisy, and ultimately shelved by users.</li><li>IN ML, ALGORITHMS ARE THE MOST IMPORTANT FACTORS: We hear a lot about deep learning, neural networks, and a host of other fancy sounding terms.  Most people don’t know what these words mean, but they sound great!  The truth is that the quality of the data you feed to any ML technology greatly determines the real-world efficacy of the resulting model.  Data curation, cleaning and labeling is hard and extremely time consuming.  It requires deep domain knowledge and collaboration between data scientists and security experts.</li><li>ML IN SECURITY IS ONLY ABOUT DETECTION: It is true that ML is an excellent way to solve some detection problems, but security is much bigger than your appliance blocking or finding something bad.  False positives happen.  An alert rarely tells the entire story of an attack.  Security teams often need help deciding what to do next and how to respond.  The industry’s narrow focus on ML only for detection has hindered potential advances in other key areas such as triage, response, and workflow.</li></ol><h2>Advantages of Machine Learning in Security</h2><p>Despite these challenges with ML, when properly implemented, ML can be powerful in security.  In fact, when used correctly, it has significant advantages over non-ML approaches in solving certain problems.  These include:</p><ol><li>GENERALIZATION: Models can generalize to never-before-seen techniques, sequences of adversary actions, or malware samples based on structural or behavioral relationships within the data that are not obviously constructed by hand.  Human-constructed signatures or heuristics tend to be very specific and reactionary, giving rise to subsequent high false negative rates because of evolving adversary tools and tradecraft.  ML solutions, if done right, can consistently detect never-before-seen evil.</li><li>SCALE AND AUTOMATION: ML solutions scale well with increasing volumes of data, a pervasive problem in the industry. They can aggregate, synthesize, and analyze disparate data sources automatically. When a novel attack occurs or new malware sample emerges, just adding it to the training set can improve the model.  An army of malware analysts and signature developers is not necessary.</li><li>DEEP INSIGHTS: ML learns from the data what constitutes malicious and benign content or behaviors.  A human does not dictate exactly where the decision boundary between benign and malicious lies.  This can lead to surprising and sometimes non-intuitive ways the underlying data can be sliced to make detection decisions – things which would not occur to a domain expert.</li><li>INFREQUENT UPDATES: A need for constant signature updates can be a substantial operational burden and put systems, especially offline or off-corporate-network systems, at rapidly increasing risk of compromise.  Well-constructed ML solutions generalize well to new threats and thus require far less frequent updates to be effective.</li></ol><h2>Considerations When Building ML/AI Models</h2><p>Endgame has a number of powerful ML-based detection solutions in our product, with many more in development.  Where we have chosen to implement ML, the advantages are significant.  However, in building these capabilities, we are always extremely aware of both the pitfalls (ensuring we avoid them) and opportunities (ensuring we optimize them) inherent within ML-based solution. For others looking to evaluate a ML model, or thinking about building their own, I’ve compiled some of the key considerations to keep in mind when building ML-based solutions.</p><ol><li>GARBAGE IN/GARBAGE OUT: This is the most significant issue facing security researchers building models.  First, gathering the right representative data, both malicious and benign, is a huge challenge.  Any model will have holes in its global understanding of data and behavior, but the bigger the holes, the worse the model will perform in the real-world. In addition, unsupervised machine learning - where the model alone derives inferences from the data without human labeling - generally is inadequate for most security use cases. Instead,  most models are built through supervised machine learning – that is, feeding the tool a lot of training data which is labeled either good or bad.  The model then learns whether to call future unlabeled content or behavior fed to it by the security application for evaluation good or bad.  Labeling of training data may sound simple, but in practice it is very hard.  There are significant edge cases, like how to deal with things like adware, or legitimate tools which can look like malware that does similar things, such as remote access solutions.  Worse, attackers can and do sneak into the benign training set, usually not through some conscious effort but through labeling laziness or mistakes.  A file has zero malicious detections in VirusTotal?  Scoop up that piece of advanced malware, call it benign, and your model will ignore it forever.  </li><li>FEATURE ENGINEERING: Models are usually built upon features which describe the data.  Feature extraction is the process by which input data, such as files, are transformed deterministically into a representation of that data comprising many features. The features are engineered to encode domain knowledge about the problem to be solved. For example, Windows executable files are transformed into thousands of floating point numbers during our MalwareScore™ feature extraction process.   Feature engineering, that is, researching and implementing the right feature set, is an enormously important part of the model building process.  To generate features that can generalize to the future, an understanding of how and why things work is required: operating systems, networks, adversary tradecraft, etc.  For example, we can simply make a feature that represents whether a binary imports functions for keylogging, but what if an adversary decides to dynamically load functions associated with keylogging to evade detection?  The feature space needs to be diverse and account for evasion techniques to generalize well.  Featureless learning is an exciting area of research, but it is nascent, and just as vulnerable to the garbage in/garbage out problem. In many security domains (e.g., static malware detection) hand-crafted features still represent the state of the art, allowing designers to encode decades of domain knowledge that have yet to be replicated through end-to-end deep learning.</li><li>USERS WILL DO THINGS YOU DO NOT EXPECT: Your model needs to be false positive intolerant to have a chance at being successful in production.  Many models look great when tested against known data, but when deployed they explode with false positives.  This is often because users don’t act in predictable ways.  Administrators may create new accounts, bounce between systems aggressively, and use tools, such as Powershell, which are often co-opted by hackers. Someone from accounting might log in at unpredictable times or practice new things on her computer discovered online.  Software your model doesn’t know about might get installed.  Researchers must be aware of these issues and expect the unexpected in the real-world. In addition, researchers must be aware of scale.  A 1 in 10,000 false positive rate might seem great, but what if you’re observing a billion events per day?</li><li>OVERFITTING: Overtraining isn’t only an issue at the gym.  It is a real issue in data science.  It is possible to build a model that is tuned exceptionally well at detecting known data but won’t extrapolate well for unknown data.  This leads to a loss of generalization and sub-par, real-world efficacy.  Researchers need to avoid falling into a trap of seeking perfection on results against a training set and instead constantly assess model performance against representative slices of withheld data; that is, data which the model did not test on in training. This helps avoid the trap of performing perfectly on the known data set, while failing when applied to real-world, unknown data.</li><li>MISLEADING METRICS: This is mostly an industry problem, but also is a potential issue for internal teams building and “selling” custom models as they seek funding and production deployment on their own networks.  There are always tradeoffs in model performance. Researchers and engineers must select cut offs between good and bad. It’s quite easy to manipulate the numbers and market a certain level of performance. For instance, it’s easy to detect 99.99 % of all malware, but at what false positive rate?  Probably way too high.  This is a dangerous but unfortunately common behavior by vendors, and the impact is compounded by how hard it can be for customers and users to validate claims in this industry.  Test against real, representative data, be transparent, and demand that solutions submit to <a href="https://www.endgame.com/blog/technical-blog/transparency-third-party-testing">third-party validation</a> whenever possible.</li><li>BURN-IN TIME: Some solutions require months to tune an environment’s baseline of normal, otherwise known as burn-in time. Beyond forcing users to wait a few months before there is any value, a lengthy burn-in time increases the opportunity for malicious data or behavior to creep into the benign training set and does not eliminate the issue of holes in understanding what is normal.  Efforts should be taken to minimize burn-in time or avoid it all together.</li></ol><h2>Moving Beyond Detection</h2><p>In addition to these considerations, we must also hold up the actual use case to tight scrutiny. To date, ML is largely only applied to detection. For those who spend their time protecting systems day in, day out, you may by this point be asking yourself, “do I even need more alerts?”  “Is detection even my biggest problem?”  I find that the answer to both questions is often no.  Security products need to be much broader than pure detection.  They need to provide context, tie information together, and guide the practitioner through incident triage, scoping, and response.  Today’s products generally are severely lacking in these areas. It is time to consider how AI and ML can improve the workflow including, but beyond, detection.</p><p>ML and AI can help automatically gather and correlate data, can suggest or even automatically execute response actions, remember what you did before, and make the process of asking questions of a mountain of available data straightforward and accessible to users.  We need to collectively think bigger about AI in security and demand tools which apply AI and ML not just to create more alerts about things that might be malicious, but make the process of doing our jobs as security analysts easier.</p><p>At Endgame, through our own experiences as operators and analysts, we know the key pain points and hurdles, and bake ML into our endpoint security product in effective ways.  Our research and development team combines experienced data scientists, reverse engineers, incident responders, and threat experts hailing from diverse industry and government backgrounds.  Because domain expertise is so critical, security domain experts are paired with data scientists to craft, evaluate, and red-team solutions.  We constantly question our assumptions, endlessly seek and fill gaps in our data sets and features, and get way into the weeds in analyzing our results to identify and fix potential shortcomings. The fruits of this tireless labor include our ML-powered <a href="https://www.endgame.com/blog/technical-blog/world-meet-malwarescore">MalwareScore™</a> feature, which we’d put up against anyone in the industry (and you can check it out yourself in <a href="https://www.virustotal.com/#/home/upload">VirusTotal</a>).</p><p>We are also very comfortable in saying that ML is not the answer to all our detection woes.  Our researchers and engineers have built world-class capabilities our customers use every day to block exploits, stop process injection, find injected code, eliminate ransomware, and much more.  The techniques we use are very powerful and no less interesting or revolutionary because of the absence of ML – our growing patent portfolio and publicly shared research in these areas attests to this.</p><p>We also treat the <a href="https://www.endgame.com/blog/technical-blog/integrating-ux-big-data-and-cyber-security-improved-protection">usability</a> and workflow problem as a serious R&D challenge in need of a solution.  By pairing researchers with user experience experts, our product addresses those key workflow pain points with novel, yet intuitive to use, ways to fix them. The result is <a href="https://www.endgame.com/blog/technical-blog/augmenting-analysts-bot-not">Artemis®</a>, our AI powered chatbot which eases search, triage, and response throughout our endpoint protection platform.  This does not eliminate the need for highly skilled users, but it does facilitate training of new users and removes friction for more advanced users.  Under the hood remains a powerful two-way API which is fully documented and open for any user, extending our commitment to usability and extensibility.</p><h2>The Bottom Line</h2><p>Looking beyond the hype and buzzwords, when implemented well ML can be a powerful piece of a security program or tool. The key challenge is to build it correctly, taking into account the range of factors I’ve addressed in this post.  The marketplace is littered with poorly implemented ML-based solutions fronted by aggressive marketing.  To help cut through the noise, the graphic below provides a cheat sheet for key questions to ask when evaluating ML-driven solutions.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte3d7fb05732aaf49/5dfc4cc258fc4a5aad8ebc3d/blog-ml-ground-truth-cheklist-endgame.png" data-sys-asset-uid="blte3d7fb05732aaf49" alt="blog-ml-ground-truth-cheklist-endgame.png"></p><p>The key to effective implementation of ML is domain expertise.  In security, that domain expertise needs to drive dataset curation, labeling, feature selection, and evaluation.  It also is equally effective to apply that domain expertise directly via other methods of detection.  You need real time, inline analysis of data and actions on monitored systems looking for behaviors across the range of adversary techniques.  ML isn’t the answer when domain expertise is required to go deep into the kernel and boil adversary actions like process injection down to their essential essence and block that in real-time.</p><p>Most importantly, remember that machine learning is a powerful tool, but is not inherently good or even better than alternatives.  The appropriate use case, data, parameters, and domain expertise (just to name a few of the challenges) all impact the efficacy of ML-based solutions. Moreover, beyond improving detection, as a community we must think bigger and address our industry’s usability problems and how ML could help alleviate some of the major workflow challenges.  To this end, at Endgame we will continue to research and operationalize powerful new features, both ML and non-ML based. Stay tuned!</p>

Beyond The Math: Effective Machine Learning In Security

<p>Last year, we introduced <a href="https://www.endgame.com/blog/technical-blog/world-meet-malwarescore">Endgame MalwareScore</a>®, a machine learning malware detection and protection engine for Windows Portable Executable (PE) files. Since its release, MalwareScore has proven capable in <a href="https://www.endgame.com/blog/executive-blog/catching-petya-how-endgame-protects-against-another-global-attack">detecting</a> emergent malware and resilient against bypass. For example, <a href="https://www.recordedfuture.com/code-signing-certificates/">recent research from Recorded Future</a> stated that MalwareScore was the only classifier in <a href="https://virustotal.com/">VirusTotal </a>that detected malware signed with particular counterfeit certificates. Beginning today, MalwareScore now supports macOS. We are extremely confident in our new Mac support, which has been released in VirusTotal for the world to see!
</p><p>Mac support is a major enhancement which required overcoming many challenges that we didn’t encounter when we created MalwareScore for Windows. We faced three primary challenges during development. First, there is a lack of good open source tools for parsing Mach-O (binaries run on macOS) files. Second, our internal data pipelines and infrastructure were built with the specific use case of processing PE files (Windows binaries). Third, and most difficult, we had limited training data. This post will walk through each of these challenges and how we overcame them.
</p><h2>Lack of Open Source Mach-O Parsers</h2><p>When looking at Windows PE files, there are tools that do a tremendous job parsing the files. We instantly gravitated towards <a href="https://github.com/erocarrera/pefile">Ero Carrera's PEFile parser</a> for our initial R&D efforts, allowing Endgame data scientists to jump right into in-depth feature engineering. Once we were ready to productionize our Windows classifier, we implemented our own highly optimized PE parser to ship with our product. Unfortunately, there is no parallel in the world of Mac Mach-O files. There are some nice command line utilities such as <a href="http://www.newosxbook.com/tools/jtool.html">Jonathan Levin’s JTool</a> for parsing Mach-O files, but they weren’t built for our use case and would not scale to the order of millions. Other tools like <a href="https://lief.quarkslab.com/">Quarkslab’s LIEF</a> have a lot of potential, but were not mature enough when we first started our research into a Mach-O classifier. Ultimately, we had to roll our own static parser to best support our research.
</p><p>Fortunately, there are a lot of good references for learning about Mach-O file formats. We recommend reading the <a href="https://github.com/aidansteele/osx-abi-macho-file-format-reference">Mach-O file format reverence</a> and Jonathan Levin’s book <a href="https://www.amazon.com/MacOS-iOS-Internals-III-Insecurity/dp/0991055535/ref=as_sl_pc_qf_sp_asin_til?tag=newosxbookcom-20&linkCode=w00&linkId=0b61c945365c9c37cd3cf88f10a5f629&creativeASIN=0991055535">macOS and iOS Internals, Volume III: Security & Insecurity</a>. We used these heavily in implementing our static parsers and as a reference during feature engineering.
</p><h2>Retooling Our Data Pipeline</h2><p>There are significantly fewer unique Mac Mach-O files in the wild than PE files, so scale in our pipeline was of little concern. However, there were small details that forced significant changes in how we process, store, and format data. The first detail that required refactoring was the paradigm of <a href="https://en.wikipedia.org/wiki/Mach-O">Mach-O Universal Binaries (or Fat files)</a>. These files are essentially a small header with one to many standard Mach-O files concatenated together. The issue is that it's possible to have both malicious and benign Mach-O files contained in the same Fat file! Additionally, Endgame only supports Intel architectures (macOS), so other architectures such as ARM-based iOS and older Mac architectures using PowerPC packaged in the same Fat file are of little to no use to us in our training data. Sifting through all this created some complexity we didn’t anticipate going into the effort.
</p><p>In addition, much of our pipeline is backed by SQL, which makes for fast and expressive querying, but does not handle fundamental changes in data formatting. Instead of bolting on a change, we fundamentally changed our table structures for better support of new file types. Extensibility gives us the option to extend to new types in the future but, of course, led us to a huge data migration. As anyone that has done data migration will tell you, it is a very long and painful process, especially when you have deadlines to meet!
</p><p>Another seemingly small but important detail is that of magic bytes. Magic bytes occur at the beginning of a file so that their type can be inferred from something other than the file extension. For example, Fat Mach-O files begin with 0xcafebabe. Unfortunately, Java class files also start with the <a href="https://en.wikipedia.org/wiki/Java_class_file">same four bytes</a>. This means a deeper parsing needs to occur when data is being mined. A naïve approach of just pulling in files with those magic bytes will leave you with a large amount of useless data, and you’ll wonder why the number of samples with successfully extracted features is much smaller than expected (I speak from experience!).
</p><h2>Lack of Training Data</h2><p>Many of us were skeptical when we first started out on this journey to create MalwareScore for Mac. Our primary concern was the lack of data. More specifically, we were legitimately concerned about a lack of malicious data, and rightly so. To demonstrate the stark difference in data availability between PE files and Mach-O files, we took three day’s worth of data in VirusTotal for both PE files and Mach-O files (Intel Architecture) and compared the breakdown of malicious versus benign. We define malicious files as those with at least five detections. This is not a perfect labeling scheme, but good enough for a high-level analysis and for demonstration purposes.
</p><p>The two pie charts below demonstrate the general disparity in file types and are scaled to represent the total number of samples for each file type (2.7 million PE files and 94,000 Mach-O files). There are two key conclusions to ascertain from these charts. First, there are far more files in VirusTotal for Windows than Mac. Second, there is very little Mac malware. About 46% of PE files submitting during this time are malicious while 1.5% of Mach-O files are malicious.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta88db2f33f9bf285/5dfd53df028cba3ef4ec660a/endgame-pie-chart.png" data-sys-asset-uid="blta88db2f33f9bf285" alt="endgame-pie-chart.png" style="display: block; margin: auto;"></p><p><em>The pie chart on the left reflects the malware distribution for PE files, while the smaller one on the right reflects the Mac distribution. The size of the pie charts reflects data availability and the small population problem when classifying Mac malware.</em>
</p><p>We took several steps to help ameliorate this situation. As is common practice for imbalanced learning problems, we incorporated class weights when training our classifier. However, we discovered that adjusting the class weights to simulate a fifty-fifty class balance was suboptimal in our experiments. Instead, we performed a grid search on class weights and discovered that our problem preferred much stronger weighting of benign files, while still achieving high true positive rates.
</p><p>We also expanded data sources beyond narrowly scoped or biased sample distributions. As researchers, we must fight the urge to only grab as much data as we can from VirusTotal and other similarly biased data sources to build our classifiers. This is obviously easier said than done, but was very important for us to do. Three easy ways to potentially expand data sources include grabbing Mach-O files from a clean and freshly installed macOS, adding known benign open source software, and incorporating customer data. The problem with these solutions is that they do little to help class imbalance as they heavily favor benign data. Without adding additional and diverse malicious Mach-O files, your classifier is likely to overtrain and not generalize to new malware. You’ll need to be more creative to make a production-level classifier!
</p><h2>Conclusion</h2><p>Endgame MalwareScore® for Mac is now live in VirusTotal! This release was the culmination of lots of hard work by many engineers, data scientists, and other researchers. This post outlined several challenges we faced during its development and we hope it will help others as they extend their machine learning AV to support Mac. We are proud of Endgame MalwareScore® and now MalwareScore for Mac, and believe it is important to provide transparency into the range of data challenges we encounter in building product-grade machine learning malware classifiers. In the coming weeks we’ll post additional details on performance!
</p>

The Making Of Endgame MalwareScore® For Mac

<p>Phishing continues to be one of the most effective methods of compromise according to <a href="https://www.verizonenterprise.com/verizon-insights-lab/dbir/">Verizon’s Data Breach Investigations Report</a>. Adversaries often use crafted documents containing malicious macros and a deceptive lure to achieve initial access to target users and networks. These macro-based attacks remain difficult to stop for numerous reasons. Adversaries are becoming more clever with their phishing schemes, and mail filtering will never stop a determined adversary from delivering payloads to inboxes. In addition, human nature and a lot of history tells us that users will open and interact with malicious attachments. Finally, security products do a poor job in providing the necessary safety net of detection and prevention without prior knowledge of the specific attack.
</p><p>Today, we are introducing MalwareScore for macros - a machine learning-based detector for malicious macro-enabled Microsoft Office documents - to protect against phishing designed to gain a foothold on a targeted system. This new capability was just released into VirusTotal and prevents compromise before malicious macro documents are even opened. Endgame already provides <a href="https://www.prnewswire.com/news-releases/independent-testing-confirms-endgame-prevention-certified-to-protect-business-environments-300535124.html">extensive signatureless protection</a> against this class of attack, including prevention against documents exploiting a vulnerability in rendering software like Adobe Reader or Microsoft Word, inline protections <a href="https://www.endgame.com/blog/technical-blog/bug-feature-debate-back-yet-again-ddeauto-root-cause-analysis">against non-macro attacks</a>, tradecraft analytics on suspicious behaviors of commonly targeted software like Office, and our full suite of protections against payloads delivered via the initial macro-based attack. With the addition of MalwareScore for macros, Endgame provides unparalleled protection against phishing campaigns with macro-enabled documents that seek to gain access to endpoints.
</p><h2>World, Meet MalwareScore for Macros</h2><p>Endgame has already <a href="https://www.endgame.com/news/press-releases/av-comparatives-validates-endgame-most-effective-next-generation-anti-virus">proven excellence</a> in known and unknown malware detection with MalwareScore for <a href="https://www.endgame.com/blog/technical-blog/world-meet-malwarescore">Windows PE files </a>and <a href="https://www.endgame.com/blog/technical-blog/making-endgame-malwarescore-mac">Mac executable files</a>. To counter malicious macros, Endgame Research built MalwareScore for macros, a static machine-learning driven malware protection for malicious Office macros. This high-efficacy macro malware classifier was released today in VirusTotal and will soon be available to Endgame customers in our 3.0 product release.
</p><p>In creating MalwareScore for macros, we applied many lessons-learned from building and maintaining MalwareScore for Windows and Mac. However, creating a classifier for macros has its own quirks and challenges to overcome. There are multiple file types to consider, unique issues related to file similarity, totally new feature engineering requirements, difficulty in gathering large benign and malicious datasets, and difficulty generating high quality labels on the training and test data. These are just a few of the challenges we encountered. Before addressing how we overcame each of these challenges in creating MalwareScore for macros, let’s take a quick look at why this protection is so essential in the first place.
</p><h2>Macros Gone Bad</h2><p>Visual Basic for Applications (VBA) has been used in Office documents since it was introduced in Excel in 1993. One of the first widespread macro viruses, Melissa, appeared in 1999 and forced several tech giants to shut down their email systems to prevent the virus from spreading. Since then, malicious Word and Excel documents with seemingly important information have been flowing freely to inboxes everywhere with cleverly constructed content encouraging the user to click “Enable Content”. The most effective of today’s macro-enabled attacks are not the easy to spot scam emails of previous eras, but rather are extremely sophisticated. They leverage everything from modifications to <a href="https://www.cyberscoop.com/dnc-hackers-impersonated-nato-attempt-hack-romanian-government/">real documents</a> to the vast wealth of personal information available online to blurry text and fake mentions of unlocking encryption to successfully deliver both targeted and widespread phishing attacks.
</p><p>Email remains the most frequent delivery mechanism for phishing attacks, but social media also is an increasingly popular attack vector. Groups such as Iran’s Cobalt Gypsy/Oil Rig <a href="https://money.cnn.com/2017/07/27/technology/business/cobalt-gypsy-oilrig-linkedin-photographer/index.html">target individuals</a> at strategic organizations, connecting with them via social media and eventually convincing them to download malicious macro-enabled documents onto their corporate networks. Earlier this year, the Pyeongchang Winter Olympics <a href="https://www.scmagazine.com/2018-winter-olympics-being-used-as-phishing-attack-bait/article/735639/">served as a decoy</a> to target organizations with a macro-based phishing campaign, hoping the malicious document would enable compromise and access to corporate information. <a href="https://researchcenter.paloaltonetworks.com/2018/02/unit42-sofacy-attacks-multiple-government-entities/">APT28</a> and other aggressive Russian actors have consistently used macro-enabled documents to gain access to their highest value targets in the US and abroad.
</p><p>We could cite dozens or hundreds of additional examples, but campaigns have similarities. The weaponized macro-based documents often evade detection because they are multi-stage, take advantage of legitimate functionality within Windows and the Office toolsuite, and leverage credible-looking user prompts for execution. Given how frequently the attack documents change and how legitimate these attacks appear, a machine learning-based classification approach can drastically improve prevention rates when modeled carefully and robustly. However, this is not a trivial task.
</p> <h2></h2> <h2>Challenges with Creating A Macro-Based Classifier</h2><p>When creating a macro-based classifier, there are a range of unique considerations. Many years of evolving versions of Office which the classifier must support, several distinct file types, and the absence of trustworthy labels on samples, especially newly in-the-wild malicious documents, are some of the special challenges to overcome. There also are the typical considerations of guaranteeing high detection efficacy, negligible false positives, and performing at scale and speed. I’ll address four of the most important challenges we solved when creating this new capability: 1) Parsing macros effectively; 2) Feature engineering; 3) Lack of solid labels; and 4) Identifying similar samples across different documents.
</p> <h3>PARSING MACROS EFFECTIVELY</h3><p>Parsing macro-enabled Office docs is an exercise in enumeration and iteration. Not only are there multiple file types such as Word documents, Excel spreadsheets, and other document types to deal with, there are also multiple Office versions that have different ideas of file structure. Pre-2007 Office versions use a <a href="https://msdn.microsoft.com/es-es/library/office/gg615407(v=office.14).aspx">binary file format</a> whereas those after 2007 use an <a href="https://support.office.com/en-us/article/open-xml-formats-and-file-name-extensions-5200d93c-3449-4380-8e11-31ef14555b18">XML file format</a>, which is effectively a zip file to house the contents.
</p><p>By checking for specific combinations of byte strings in the file, we can determine the type and format of the file and thus the the location of relevant code within the file. With that in hand, the OLE (Object Linking and Embedding) Streams must be parsed to get the full account of macro text. OLE Streams in documents are analogous to an internal filesystem and can be comprised of very few streams or many thousands. Parsing these streams helps determine which contain macro code and collecting that as text for analysis.
</p><p>Finally, once all code streams (see Figure 1 below) are parsed, the text segments are passed to our feature generation process, described next.
</p><figcaption><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdfd4def0ef902e68/5dfd0971ba09063e77b60272/endgame-office-vba-file-format-structure.png" data-sys-asset-uid="bltdfd4def0ef902e68" alt="endgame-office-vba-file-format-structure.png" style="display: block; margin: auto;"><br>Figure 1: Office VBA File Format Structure (Source: <a href="https://msdn.microsoft.com/en-us/library/office/cc313094(v=office.12).aspx">Microsoft</a>)</figcaption><h3><br></h3><h3>FEATURE ENGINEERING</h3><p>As with most applied machine learning problems, feature engineering for malicious macro classification is one of the most important and impactful, and thus guarded, steps in the model creation process. Feature engineering means converting raw input, in this case the code streams we parsed in the previous step, into an array of numbers which our chosen modeling software can use for building a model.
</p><p>For this feature, our feature engineering focused on analysis of code streams driven by close collaboration between reverse engineers, threat researchers, and data scientists. Some of our features depend on counting reserved keywords such as “connect” and “thisdocument”. Others collect string metrics, perhaps to detect obfuscation or encoding, while others conduct more involved textual analysis. All-in-all, we generate hundreds of features per sample for analysis. The final feature set is the end product of months of iteration, experimentation, and testing to ensure our desired levels of efficacy.
</p> <h3>LACK OF SOLID LABELS</h3><p>Supervised classifiers require large, reliably labeled datasets. This is a significant challenge for any machine learning problem in security, but good labels for macros are especially hard to come by. Having no AVs call a file bad is not a stellar indicator of non-maliciousness for any file in security, and that’s an issue we’ve dealt with successfully with past classifiers. In the realm of macro enabled documents, this issue is especially challenging.
</p><p>To increase certainty in a label, the industry often relies on internal, and sometimes external, crowdsourcing to generate signatures and compile blacklists. We sought to build upon that idea by developing a framework we dubbed Active Labeling to make it quick and easy for Endgame reverse engineers to provide a label for a given sample, and have that feed back into the training pipeline.
</p><p>First, we generated the list of samples that would make the “biggest” impact to classification performance, specifically targeting samples that have significant uncertainty or that can most influence the decision boundary of our classifier. These samples are often scored at the good/bad threshold for a given model (e.g. 0.49-0.51). Next, we automated the extraction of the macro, IOCs, and any other metadata that could aid in the labeling of a sample and display to the analyst in a web UI. This provides an intuitive interface for the analysts to grab a sample and make a judgement as quickly as possible. These "human labeled” samples are fed back into our machine learning training to further improve performance. Active Labeling allows us not only to detect troublesome samples, but to efficiently enhance and refine our future machine learning models to better predict new and unknown samples. Our classifier would not have been shippable without significant effort in this area.
</p> <h3>IDENTIFYING SIMILAR SAMPLES</h3><p>One of the few things everyone (mostly) agrees about in security is how to identify a file. The file hash, such as sha256, uniquely identifies each malware sample. With PE or Macho executable malware, because of polymorphic malware and code modifications over time, looking for a hash you already know about is an imperfect method for finding that malware in the future. Despite it not being a great solution for robust future detections, a hash is a very useful and in fact an industry-standard quality to key off of when it comes to whitelisting, blacklisting, and similar actions. We need to think a little differently about macro-enabled documents.
</p><p>Just like with executables, the hash of a document can be used to find that exact same file in a network. However, if someone changes even one cell in the fake spreadsheet the entire file hash changes while the malicious macro, the part we are most concerned about, remains unaffected. We should instead look at the macro itself, not the phishing content, as our anchor for sameness..
</p><p>We’ve implemented an idea similar to <a href="https://www.fireeye.com/blog/threat-research/2014/01/tracking-malware-import-hashing.html">ImpHash</a> which we internally call MacroHash. Instead of hashing the entire file, we perform some light sanitation on the OLE Streams, order them, and hash them in aggregate. This way we uniquely fingerprint the same combination of macros across multiple host Office documents, and are unaffected by changes in the file contents and thus file hash when seeking identical samples in our training set or providing necessary customer-facing features like whitelisting.
</p><h2>Conclusion</h2><p>MalwareScore for macros is now live in VirusTotal! Macro-enabled phishing attacks aren’t going away anytime soon. They continue to be the easiest way into many target networks, and defenses have been woefully inadequate. Creating MalwareScore for macros was truly a collaborative process across Endgame requiring significant cross-functional innovation. We’re very excited to share it with the wider security community through inclusion in VirusTotal. The description of challenges we faced when building MalwareScore for macros furthers our commitment to demystifying security and provides transparency to our current and future customers about how Endgame takes features from idea to product.
</p>

How Endgame Protects Against Phishing From Macro-Enabled Documents

<p>Bonjour les amis. We're constantly adding new vector layers to our <a href="/elastic-maps-service">Elastic Maps Service</a>. Most recently, we've added support for France department boundary vector layers to region maps. You may match results of terms in your Elasticsearch data to department names in either French or English. We have also included INSEE codes for each department as defined by <a href="https://www.insee.fr/">Institut national de la statistique et des études économiques (INSEE)</a> [National Institute of Statistics and Economic Studies]. INSEE collects, analyzes, and distributes information on the French economy and society.
</p><h4>#TalkPay</h4><p>Kaggle has an interesting dataset from INSEE on <a href="https://www.kaggle.com/etiennelq/french-employment-by-town">employment, salaries, and population per town in France</a>. We can import this dataset into Elasticsearch and visualize in Kibana region maps. For this project we will look at salary information.
</p><h4>Prepare the data</h4><p>There are several data files in the Kaggle dataset that require a little massaging before we can import into Elasticsearch. The net_salary_per_town_categories.csv file includes a column called CODGEO which the dataset describes as the INSEE code for the Town. This can be joined to the code_insee column in the name_geographic_information.csv file. The name_geographic_information.csv file includes INSEE codes for the administrative regions, departments, and prefectures for each town. This normalization strategy works well for importing into relational databases, but we will want to join these tables (de-normalize) for importing into Elasticsearch.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6fef0732095723f2/5c584199516e21cf0b2a1246/blog-france-salary-tables-1.png" data-sys-asset-uid="blt6fef0732095723f2" alt="blog-france-salary-tables-1.png">
</p><p><a href="https://www.sqlite.org/">SQLite</a> is an incredibly useful and versatile database engine that we can use to join these tables. We’ll create a database file and import the CSV files into it.
</p><pre class="prettyprint">sqlite3 france-salary.db
sqlite&gt;.mode csv
sqlite&gt;.import data/net_salary_per_town_categories.csv salary
sqlite&gt;.import data/name_geographic_information.csv names
</pre><p>Let’s preview our data.
</p><pre class="prettyprint">sqlite&gt;.headers on
sqlite&gt;SELECT * FROM salary LIMIT 5;
</pre><table style="font-family:Menlo, Monaco, Consolas, Courier New, monospace;">
<tbody>
<tr>
	<td>CODGEO
	</td>
	<td>LIBGEO
	</td>
	<td>SNHM14
	</td>
	<td>SNHMC14
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>01004
	</td>
	<td>Ambérieu-en-Bugey
	</td>
	<td>13.7
	</td>
	<td>24.2
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>01007
	</td>
	<td>Ambronay
	</td>
	<td>13.5
	</td>
	<td>22.1
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>01014
	</td>
	<td>Arbent
	</td>
	<td>13.5
	</td>
	<td>27.6
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>01024
	</td>
	<td>Attignat
	</td>
	<td>12.9
	</td>
	<td>21.8
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>01025
	</td>
	<td>Bâgé-la-Ville
	</td>
	<td>13
	</td>
	<td>22.8
	</td>
	<td>...
	</td>
</tr>
</tbody>
</table><pre class="prettyprint">sqlite&gt;SELECT * FROM names LIMIT 5;
</pre><table style="font-family:Menlo, Monaco, Consolas, Courier New, monospace;">
<tbody>
<tr>
	<td>code_insee
	</td>
	<td>nom_commune
	</td>
	<td>code_région
	</td>
	<td>nom_région
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>1004
	</td>
	<td>Ambérieu-en-Bugey
	</td>
	<td>82
	</td>
	<td>Rhône-Alpes
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>1007
	</td>
	<td>Ambronay
	</td>
	<td>82
	</td>
	<td>Rhône-Alpes
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>1014
	</td>
	<td>Arbent
	</td>
	<td>82
	</td>
	<td>Rhône-Alpes
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>1024
	</td>
	<td>Attignat
	</td>
	<td>82
	</td>
	<td>Rhône-Alpes
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>1025
	</td>
	<td>Bâgé-la-Ville
	</td>
	<td>82
	</td>
	<td>Rhône-Alpes
	</td>
	<td>...
	</td>
</tr>
</tbody>
</table><p>Notice that the CODGEO column has a leading zero, but the code_insee column does not. This discrepancy appears in the CSV tables. In order to join the two tables, we’ll need to pad the code_insee column with a leading zero. We can do this with a little SQLite magic.
</p><pre class="prettyprint">sqlite&gt;CREATE TABLE names_ready as SELECT substr(‘00000’ || code_insee, -5, 5) as insee_town, * as FROM names;
sqlite&gt;SELECT * FROM names_ready LIMIT 5;
</pre><table style="font-family:Menlo, Monaco, Consolas, Courier New, monospace;">
<tbody>
<tr>
	<td>insee_town
	</td>
	<td>code_insee
	</td>
	<td>nom_commune
	</td>
	<td>code_région
	</td>
	<td>nom_région
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>01004
	</td>
	<td>1004
	</td>
	<td>Ambérieu-en-Bugey
	</td>
	<td>82
	</td>
	<td>Rhône-Alpes
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>01007
	</td>
	<td>1007
	</td>
	<td>Ambronay
	</td>
	<td>82
	</td>
	<td>Rhône-Alpes
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>01014
	</td>
	<td>1014
	</td>
	<td>Arbent
	</td>
	<td>82
	</td>
	<td>Rhône-Alpes
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>01024
	</td>
	<td>1024
	</td>
	<td>Attignat
	</td>
	<td>82
	</td>
	<td>Rhône-Alpes
	</td>
	<td>...
	</td>
</tr>
<tr>
	<td>01025
	</td>
	<td>1025
	</td>
	<td>Bâgé-la-Ville
	</td>
	<td>82
	</td>
	<td>Rhône-Alpes
	</td>
	<td>...
	</td>
</tr>
</tbody>
</table><p>Now the insee_town column in the new names_ready table can be joined to the CODGEO column in the salary table. But first, let’s create indexes on those columns.
</p><pre class="prettyprint">sqlite&gt;CREATE INDEX idx_insee_town ON names_ready (insee_town);
sqlite&gt;CREATE INDEX idx_codgeo ON salary (CODGEO);
</pre><p>Then we will create a table from a select query to join the salary and names tables and create useful aliases for the field names based on the metadata in the Kaggle dataset.
</p><pre class="prettyprint">sqlite&gt;CREATE TABLE france_salary as 
    SELECT 
    CODGEO as Town_INSEE,
    Nom_commune as Town_Name
    code_région as Region_INSEE,
    nom_région as Region_Name,
    numéro_département as Department_INSEE,
    nom_département as Department_Name,
    Préfecture as Prefecture_Name,
    Numéro_circonscription as Conscription_Number,
    Codes_postaux as Postal_Code,
    Latitude,
    Longitude,
    SNHM14 as mean_net_salary,
    SNHMF14 as mean_net_salary_female,
    SNHMH14 as mean_net_salary_male,
    SNHMC14 as mean_net_salary_hour_executive,
    SNHMFC14 as mean_net_salary_hour_executive_female,
    SNHMHC14 as mean_net_salary_hour_executive_male,
    SNHMP14 as mean_net_salary_hour_middlemanager,
    SNHMFP14 as mean_net_salary_hour_middlemanager_female,
    SNHMHP14 as mean_net_salary_hour_middlemanager_male,
    SNHME14 as mean_net_salary_hour_employee,
    SNHMFE13 as mean_net_salary_hour_employee_female,
    SNHMHE14 as mean_net_salary_hour_employee_male,
    SNHMO14 as mean_net_salary_hour_worker,
    SNHMFO14 as mean_net_salary_hour_worker_female,
    SNHMHO14 as mean_net_salary_hour_worker_male,
    SNHM1814 as mean_net_salary_hour_year18-25,
    SNHMF1814 as mean_net_salary_hour_year18-25_female,
    SNHMH1814 as mean_net_salary_hour_year18-25_male,
    SNHM2614 as mean_net_salary_hour_year26-50,
    SNHMF2614 as mean_net_salary_hour_year26-50_female,
    SNHMH2614 as mean_net_salary_hour_year26-50_male,
    SNHM5014 as mean_net_salary_hour_year50+,
    SNHMF5014 as mean_net_salary_hour_year50+_female,
    SNHMH5014 as mean_net_salary_hour_year50+_male
    FROM salary
    LEFT OUTER JOIN names_ready
    ON salary.CODGEO = names_ready.insee_town;
</pre><h4>Ingest into Elasticsearch</h4><p><a href="/logstash">Logstash</a> is a great way to transform and ingest data for Elasticsearch. A multitude of plugins are available for Logstash. For this project we will use the CSV filter plugin to import data from a CSV file.
</p><p>First, let’s generate a new CSV file called salary_data.csv from the table we created above.
</p><pre class="prettyprint">sqlite&gt;.headers on
sqlite&gt;.mode csv
sqlite&gt;.once salary_data.csv
sqlite&gt;SELECT * from france_salary;
</pre><p>Next we’ll create a config file for Logstash to load this CSV file.
</p><p>Here is our definition for the file input plugin. Change the path to the absolute path of your salary_data.csv file. Specifying /dev/null as the sincedb_path will ensure that Logstash always reads from the top of the file, but beware of this in production as it could create duplicate records.
</p><pre class="prettyprint">input {
   file {
       path =&gt; "/path/to/france-salary/salary_data.csv"
       start_position =&gt; "beginning"
       codec =&gt; plain {
           charset =&gt; "UTF-8"
       }
       sincedb_path =&gt; "/dev/null"
   }
}
</pre><p>Then we must specify how to handle the CSV file using the CSV plugin. Columns in CSV files are always assumed to be text, so we need to use the convert parameter to change numeric fields into float for Elasticsearch.
</p><pre class="prettyprint">filter {
   csv {
       separator =&gt; ","
       skip_header =&gt; "true"
       columns =&gt; [
           "Town_INSEE",
           "Town_Name",
           "Region_INSEE",
           "Region_Name",
           "Department_INSEE",
           "Department_Name",
           "Prefecture_Name",
           "Conscription_Number",
           "Postal_Code",
           "latitude",
           "longitude",
           "mean_net_salary",
           "mean_net_salary_female",
           "mean_net_salary_male",
           "mean_net_salary_hour_executive",
           "mean_net_salary_hour_executive_female",
           "mean_net_salary_hour_executive_male",
           "mean_net_salary_hour_middlemanager",
           "mean_net_salary_hour_middlemanager_female",
           "mean_net_salary_hour_middlemanager_male",
           "mean_net_salary_hour_employee",
           "mean_net_salary_hour_employee_female",
           "mean_net_salary_hour_employee_male",
           "mean_net_salary_hour_worker",
           "mean_net_salary_hour_worker_female",
           "mean_net_salary_hour_worker_male",
           "mean_net_salary_hour_year18_25",
           "mean_net_salary_hour_year18_25_female",
           "mean_net_salary_hour_year18_25_male",
           "mean_net_salary_hour_year26_50",
           "mean_net_salary_hour_year26_50_female",
           "mean_net_salary_hour_year26_50_male",
           "mean_net_salary_hour_year50over",
           "mean_net_salary_hour_year50over_female",
           "mean_net_salary_hour_year50over_male"
       ]
       convert =&gt; {
           "mean_net_salary" =&gt; "float"
           "mean_net_salary_female" =&gt; "float"
           "mean_net_salary_male" =&gt; "float"
           "mean_net_salary_hour_executive" =&gt; "float"
           "mean_net_salary_hour_executive_female" =&gt; "float"
           "mean_net_salary_hour_executive_male" =&gt; "float"
           "mean_net_salary_hour_middlemanager" =&gt; "float"
           "mean_net_salary_hour_middlemanager_female" =&gt; "float"
           "mean_net_salary_hour_middlemanager_male" =&gt; "float"
           "mean_net_salary_hour_employee" =&gt; "float"
           "mean_net_salary_hour_employee_female" =&gt; "float"
           "mean_net_salary_hour_employee_male" =&gt; "float"
           "mean_net_salary_hour_worker" =&gt; "float"
           "mean_net_salary_hour_worker_female" =&gt; "float"
           "mean_net_salary_hour_worker_male" =&gt; "float"
           "mean_net_salary_hour_year18_25" =&gt; "float"
           "mean_net_salary_hour_year18_25_female" =&gt; "float"
           "mean_net_salary_hour_year18_25_male" =&gt; "float"
           "mean_net_salary_hour_year26_50" =&gt; "float"
           "mean_net_salary_hour_year26_50_female" =&gt; "float"
           "mean_net_salary_hour_year26_50_male" =&gt; "float"
           "mean_net_salary_hour_year50over" =&gt; "float"
           "mean_net_salary_hour_year50over_female" =&gt; "float"
           "mean_net_salary_hour_year50over_male" =&gt; "float"
       }
   }
}
</pre><p>The hosts parameter in the output plugin section is the URL to your running Elasticsearch instance. We also specify stdout in json format as an output for debugging. The stdout parameters are not required for ingesting the data.
</p><pre class="prettyprint">output {
   stdout {
codec =&gt; json
   }
   elasticsearch {
       hosts =&gt; ["localhost:9200"]
       index =&gt; "france-example-salary"
   }
}
</pre><p>Let’s call this file france-salary.conf and tell Logstash to use it for ingestion.
</p><pre class="prettyprint">./bin/logstash -f /path/to/france-example-salary.conf
</pre><p>Logstash does not terminate automatically when it reaches the end of the file. Instead it will continue to monitor the CSV file for new data. Since we don’t expect to add more data to the CSV file you can safely stop Logstash using Ctrl-C when the output stops.
</p><h4>Create Maps of Salary by France Department</h4><p>Create a new Index pattern in Kibana to match france-example-*. Then check out the Discover tab to make sure our data is there. It should look something like this.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt48e3902d3813eab5/5c584194a4f817ef0b0e8380/blog-france-salary-kibana-1.png" data-sys-asset-uid="blt48e3902d3813eab5" alt="blog-france-salary-kibana-1.png">
</p><p>Switch to the Visualize tab and create a new region map visualization using the france-example-* index. Set the value to average the mean_net_salary. The shape field should be a terms aggregation on the Department_Name keyword. This will average the mean_net_salary for all towns in each department. You may also want to set the size to large number (e.g. 100) in order to display all department data.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt85d63fa43a3eb131/5c58418e122247d10be8ecf3/blog-france-salary-kibana-2.png" data-sys-asset-uid="blt85d63fa43a3eb131" alt="blog-france-salary-kibana-2.png">
</p><p>Nothing will appear on the map yet because we haven’t defined which map layer to use. Switch to the Options tab and change the vector map to France Departments and the Join Field to Department name (French). Click the “Apply Changes” button and each department on the map will automatically be colored depending on the value. You may have to zoom in to France for a better look.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9c36d74723265a67/5c5841888c741b010cbf6d26/blog-france-salary-kibana-3.jpg" data-sys-asset-uid="blt9c36d74723265a67" alt="blog-france-salary-kibana-3.jpg">
</p><p>It appears that the departments around Paris have a higher mean net salary. That’s probably not a big surprise though. Let’s try another visualization.
</p><p>Our data has mean net salaries for males and females. So we can check for a gender pay gap for each department. To do this, we’ll need to first create a scripted field. Open the Management tab on the sidebar, select “Index Patterns” and choose the france-example-* pattern if necessary.
</p><p>Open the scripted fields tab and click the “Add a Scripted Field” button. Set the name of the scripted field to Gender Salary Ratio and enter
<code>doc['mean_net_salary_female'].value / doc['mean_net_salary_male'].value
</code> as the script. This will calculate the ratio of female salary to male salary. You may also wish to change the format to percentage. Click the “Create Field” button to add this new field to the index.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltff71054bbfb8f9ed/5c584182a209c1ca0be9df64/blog-france-salary-kibana-4.png" data-sys-asset-uid="bltff71054bbfb8f9ed" alt="blog-france-salary-kibana-4.png">
</p><p>Now create a new region map visualization using the new Gender Salary Ratio field and aggregate by department name. Now we can see which departments appear to have towns with higher and lower disparities between male and female salaries.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt907d42af64d14c06/5c58417d773ad9030c5c2e6c/blog-france-salary-kibana-5.jpg" data-sys-asset-uid="blt907d42af64d14c06" alt="blog-france-salary-kibana-5.jpg">
</p><p>You can explore further by creating other scripted fields for different position levels and age ranges.
</p><p>Region map visualizations are available on Kibana version 5.5 and up. We are working to release more administrative regions soon. Please create an issue on the <a href="https://github.com/elastic/kibana">Kibana GitHub repository</a> if there’s a region dataset that should be included in Kibana.
</p>

blog-france-salary-kibana-fullbleed.jpg

blog-france-salary-kibana-5.jpeg

Visualizing France Salary Data with Region Maps in Kibana

<p>The presence of domain names created by a Domain Generation Algorithm (DGA) is a telling indicator of compromise. For example, the domain <em>xeogrhxquuubt.com</em> is a DGA generated domain created by the Cryptolocker ransomware. If a process attempts to connect to this domain, then your network is probably infected with ransomware. Domain blacklists are commonly used in security to prevent connections to such domains, but this blacklisting approach doesn’t generalize to new strains of malware or modified DGA algorithms. We have been researching DGAs extensively at Endgame, and recently posted a <a href="https://arxiv.org/abs/1611.00791">paper on arxiv</a> that describes our ability to predict domain generation algorithms using deep learning.
</p><p>In this blogpost, we’ll briefly summarize our paper, presenting a basic yet powerful technique to detect DGA generated domains that performs far better than “state-of-the-art” techniques presented in academic conferences and journals. “How?” you may ask. Our approach is to use neural networks (more popularly called deep learning) and more specifically, Long Short-Term Memory networks (LSTMs). We’ll briefly discuss the benefits of this specific form of deep learning and then cover our simple yet powerful approach for detecting DGA algorithms.
</p><p>If you are unfamiliar with machine learning, you may want to jump over to our <a href="https://www.endgame.com/blog/machine-learning-you-gotta-tame-beast-you-let-it-out-its-cage">three part intro series</a> on machine learning before continuing on.
</p><h2>The Benefits of Long Short-Term Memory Networks</h2><p>Deep learning is a recent buzzword in the machine learning community. Deep learning refers to deeply-layered neural networks (one type of machine learning model), in which feature representations are learned by the model rather than hand-crafted by a user. With roots that go back decades, deep learning has become wildly popular in the last four-five years in large part due to improvements in hardware (such as improved parallel processing with GPUs) and optimization tricks that make training complex networks feasible. LSTMs are one such trick for implementing recurrent neural networks, meaning a neural network that contains cycles. LSTMs are extremely good at learning patterns in long sequences, such as text and speech. In our case, we use them to learn patterns in sequences of characters (domain names) so that we can classify them as DGA-generated or not DGA generated.
</p><p>The benefit of using deep learning over more conventional machine learning algorithms is that we do not require any feature engineering. Conventional approaches generate a long list of features (such as length, vowel to consonant ratio, and n-gram counts) and use these features to distinguish between DGA-generated and not DGA-generated domains (our colleague wrote about this in a previous <a href="https://www.endgame.com/blog/hunting-networks-part-2-higher-order-patterns">post</a>). If an adversary knows your features, they can easily update their DGA to avoid detection. This requires us, as security professionals, to go through a long and arduous process of creating new features to stay ahead. In deep learning, on the other hand, the model learns the feature representations automatically allowing our models to adapt more quickly to an ever-changing adversary.
</p><p>Another advantage to our technique is that we classify solely on the domain name without the use of any contextual features such as NXDomains and domain reputation. Contextual features can be expensive to generate and often require additional infrastructure (such as network sensors and third party reputations systems). Surprisingly, LSTMs without contextual information perform significantly better than current state of the art techniques that do use contextual information! For those who want to know more about LSTMs, tutorials are plentiful. Two that we recommend: <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/">colah’s blog</a> and <a href="http://deeplearning.net/tutorial/lstm.html">deeplearning.net</a>, and of course our <a href="https://arxiv.org/abs/1611.00791">paper</a> on arxiv.
</p><h2>What is a DGA?</h2><p>First, a quick note on what a DGA is and why detecting DGAs is important. Adversaries often use domain names to connect malware to command and control servers. These domain names are coded into malware and give the adversary some flexibility in where the C2 is actually hosted in that it’s easy for them to update the domain name to point to a new IP. However, hardcoded domain names are easy to blacklist or sinkhole.
</p><p>Adversaries use DGAs to evade blacklisting and sinkholing by creating an algorithm (DGA) that creates pseudorandom strings that can be used as domain names. Pseudorandom means that the sequences of strings appear to be random but are actually repeatable given some initial state (or seed). This algorithm is used by both malware that runs on a victim’s machine as well as on some remote software used by the adversary.
</p><p>On the adversary side, the attacker runs the algorithm and randomly selects a small number of domains (sometimes just one) which he knows will be predicted by the DGA, registers them, and points them at C2 servers. On the victim side, malware runs the DGA and checks an outputted domain to see if it’s live. If a domain is registered, the malware uses this domain as its Command and Control (C2) server. If not, it checks another. This can happen hundreds or thousands of times.
</p><p>If security researchers gather samples and can reverse engineer the DGA, they can generate lists of all possible domains or pre-register domains which will be predicted in the future and use them for blacklisting or sinkholing, but this doesn’t scale particularly well since thousands or more domains can be generated by a DGA for a given day, with an entirely new list each following day. As you can see, a more generalized approach for predicting whether a given domain name is likely generated by a DGA would be ideal. That generalized approach is what we’re describing here.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltff6e7b941805f9d6/5e0e011df5916f4381bc9717/endgame-detecting-dgas-discovery-process-blog.jpeg" data-sys-asset-uid="bltff6e7b941805f9d6" alt="endgame-detecting-dgas-discovery-process-blog.jpeg" style="display: block; margin: auto;"></p><figcaption>Figure 1 demonstrates the discovery process used by many types of malware. In this case, the malware attempts three domains: asdfg.com, wedcf.com, and bjgkre.com. The first two domains are not registered and receive an NXDomain response from the DNS server. The third domain is registered and the malware uses this domain to call home.
</figcaption><h2>Building the LSTM</h2><h3>Training Data</h3><p>Any machine learning model requires training data. Luckily, training data for this task is easy to find. We used the Alexa top 1 million sites for <a href="https://s3.amazonaws.com/alexa-static/top-1m.csv.zip">benign data</a>. We also put together a few DGA algorithms in Python that you can grab on our <a href="https://github.com/endgameinc/dga_predict">github site</a>. We use these to generate malicious data. These will generate all the data needed to reproduce the results in this blog.
</p><h3>Tools and Frameworks</h3><p>We used the <a href="https://keras.io/">Keras</a> toolbox, which is a Python library that makes coding neural networks easy. There are other tools that we like, but Keras is the easiest to demo and understand. Keras is stable and production ready as it uses either <a href="http://deeplearning.net/software/theano/">Theano</a> or <a href="https://www.tensorflow.org/">TensorFlow</a> under the hood.
</p><h3>Model Code</h3><p>The following is our Python code for building our model:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blted5613c682037f91/5e0e015d4dfaba3e76b44caa/endgame-detecting-dgas-python-model-blog.png" data-sys-asset-uid="blted5613c682037f91" alt="endgame-detecting-dgas-python-model-blog.png" style="display: block; margin: auto;"></p><p>Yup, that’s it! Here is a very simple breakdown of what we’re doing (with links to additional information):
</p><ul>
	<li>We create a basic neural network model on the first line. The next line adds an embedding layer. This layer converts each character into a vector of 128 floats (128 is not a magical number. We chose it as it gave us the best numbers consistently). Each character essentially goes through a lookup once this layer is trained (input character and output 128 floats). <em>max_features</em> defines the number of valid characters. <em>input_length</em> is the maximum length string that we will ever pass to our neural network.</li>
	<li>The next line adds an LSTM layer. This is the main workhorse of our technique. <em>128</em> represents the number of dimensions in our internal state (this happens to be the same size as our previous embedding layer by coincidence). More dimensions mean a larger more descriptive model, and we found 128 to work quite well for our task at hand.</li>
	<li>The <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network#Dropout"><em>Dropout</em></a> layer is a trick used in deep learning to prevent overtraining. You can probably remove this, but we found it useful.</li>
	<li>This Dropout layer precedes a <em>Dense</em> layer (<a href="https://en.wikipedia.org/wiki/Convolutional_neural_network#Fully_connected_layer">fully connected layer</a>) of size 1.</li>
	<li>We added a <a href="https://en.wikipedia.org/wiki/Sigmoid_function">sigmoid activation</a> function to squash the output of this layer between 0 and 1, which represent, respectively, benign and malicious.</li>
	<li>We optimize using the <a href="https://en.wikipedia.org/wiki/Cross_entropy">cross entropy loss</a> function with the <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent#RMSProp">RMSProp</a> optimizer. RMSProp is a variant of <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent">stochastic gradient descent</a> and tends to work very well for recurrent neural networks.</li>
</ul><h2>Preprocessing Code</h2><p>Before we start training, we must do some basic data preprocessing. Each string should be converted into an array of ints that represents each possible character. This encoding is arbitrary, but should start at 1 (we reserve 0 for the end-of-sequence token) and be contiguous. An example of how this can be done is given below.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb3b552010d417f4f/5e0e01945f3f935f859311ff/endgame-detecting-dgas-converting-string-blog.png" data-sys-asset-uid="bltb3b552010d417f4f" alt="endgame-detecting-dgas-converting-string-blog.png" style="display: block; margin: auto;"></p><p>Next, we pad each array of ints to the same length. Padding is a requirement of our toolboxes to better optimize calculations (theoretically, no padding is needed for LSTMs). Fortunately, Keras has a nice function for this:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt945811a5c049fe8a/5e0e01b7631b8140b1f058bb/endgame-detecting-dgas-padding-blog.png" data-sys-asset-uid="blt945811a5c049fe8a" alt="endgame-detecting-dgas-padding-blog.png" style="display: block; margin: auto;"></p><p><em>maxlen</em> represents the length that each array should be. This function pads with 0’s and crops when an array is too long. It’s important that your integer encoding from earlier starts at 1 as the LSTM should learn the difference between padding and characters.
</p><p>From here, we can divide out our testing and training set, train, and evaluate our performance using a ROC curve.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt524ea7fd61ab74f7/5e0e01e3631b8140b1f058c1/endgame-detecting-dgas-roc-curve-blog.png" data-sys-asset-uid="blt524ea7fd61ab74f7" alt="endgame-detecting-dgas-roc-curve-blog.png" style="display: block; margin: auto;"></p><h4></h4><h2>Comparisons</h2><p>We compared our simple LSTM technique to three other techniques in our arxiv <a href="https://arxiv.org/abs/1611.00791">paper</a>. To keep things simple for this blog, we only compare the results with a single method that uses <a href="https://en.wikipedia.org/wiki/Bigram">bigram</a> distributions with <a href="https://en.wikipedia.org/wiki/Logistic_regression">logistic regression</a>. This technique also works better than the current state of the art (but not as good as an LSTM) and is surprisingly fast and simple. It is a more conventional feature-based approach where features are the histogram (or raw count) of all bigrams contained within a domain name. The code for this technique is also on our <a href="https://github.com/endgameinc/dga_predict">github</a> site.
</p><h2>Results</h2><p>We are finally ready to see some results! And here they are:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt776c174b97d89c32/5e0e020567e97c652d3c85c1/endgame-detecting-dgas-results-blog.png" data-sys-asset-uid="blt776c174b97d89c32" alt="endgame-detecting-dgas-results-blog.png" style="display: block; margin: auto;"></p><p>Nice! An AUC of 0.9977 with just a few lines of code! All of this is featureless with a simple and straightforward implementation. We actually did a much deeper analysis on a larger and more diverse dataset and observed 90% detection with a 1/10,000 false positive rate, and this can be combined with other approaches outside the scope of this post to improve detection even further. With our LSTM, we even did quite well on multiclass classification, meaning we could classify a piece of malware just by the domains it generated.
</p><h2>Conclusion</h2><p>We presented a simple technique using neural networks to detect DGAs. The technique uses no contextual information (such as NXDomains or domain reputation) and performs far better than state of the art. While this was a brief summary of our DGA research, feel free to read our complete <a href="https://arxiv.org/abs/1611.00791">paper on arxiv</a> and explore our code on <a href="https://github.com/endgameinc/dga_predict">github</a>. As always, comments and suggestions are welcome on our github site!
</p>

Using deep learning to detect DGAs

<p>Natural Language Processing (<a href="https://en.wikipedia.org/wiki/Natural_language_processing">NLP</a>) is a diverse field in computer science dedicated to automatically parsing and processing human language. NLP has been used to perform authorship attribution and sentiment analysis, as well as being a core function of IBM’s Watson and Apple’s Siri. NLP research is thriving due to the massive amounts of diverse text sources (e.g., Twitter and Wikipedia) and multiple disciplines using text analytics to derive insights. However, NLP can be used for more than human language processing and can be applied to any written text. Data scientists at Endgame apply NLP to security by building upon advanced NLP techniques to better identify and understand malicious code, moving toward an NLP methodology specifically designed for malware analysis—a Malicious Language Processing framework. The goal of this Malicious Language Processing framework is to operationalize NLP to address one of the security domain’s most challenging big data problems by automating and expediting the identification of malicious code hidden within benign code.
</p><h3>How is NLP used in InfoSec?
</h3><p>Before we delve into how Endgame leverages NLP, let’s explore a few different ways others have used it to tackle information security problems:
</p><ul>
	<li><a href="http://conferences.sigcomm.org/imc/2010/papers/p48.pdf">Domain Generation Algorithm classification</a> – Using NLP to identify malicious domains (e.g., blbwpvcyztrepfue.ru) from benign domains (e.g., cnn.com)</li>
	<li><a href="https://www.usenix.org/legacy/events/woot11/tech/slides/yamaguchi.pdf">Source Code Vulnerability Analysis</a> – Determining function patterns associated with known vulnerabilities, then using NLP to identify other potentially vulnerable code segments.</li>
	<li><a href="http://nlp.uned.es/~lurdes/araujo/eswa13_malicious_tweets.pdf">Phishing Identification</a> – A bag-of-words model determines the probability an email message contains a phishing attempt or not.</li>
	<li>Malware Family Analysis –Topic modeling techniques assign samples of malware to families, as discussed in my colleague Phil Roth’s previous <a href="https://www.endgame.com/blog/examining-malware-python">blog</a>.</li>
</ul><p>Over the rest of this post, I’ll discuss how Endgame data scientists are using Malicious Language Processing to discover malicious language hidden within benign code.
</p><h3>Data Acquisition/Corpus Building
</h3><p>In order to perform NLP you must have a corpus, or collection of documents. While this is relatively straightforward in traditional NLP (e.g., APIs and web scraping) it is not necessarily the same in malware analysis. There are two primary techniques used to get data from malicious binaries: static and dynamic analysis.
</p><figcaption><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt63811a93bc14888b/5dfd4fbb15c5fe6533ad76e7/endgame-nlp-disassembled-source-code.png" data-sys-asset-uid="blt63811a93bc14888b" alt="endgame-nlp-disassembled-source-code.png" style="display: block; margin: auto;"><br>Fig 1. Disassembled source code</figcaption><p>Static analysis, also called source code analysis, is performed using a<a href="https://en.wikipedia.org/wiki/Disassembler">disassembler</a> providing output similar to the above (Fig 1). The disassembler presents a flat view of a binary, however structurally we lose important contextual information by not clearly delineating the logical order of instructions. In disassembly, jmp or call instructions should lead to different blocks of code that a standard flat file misrepresents. Luckily, static analysis tools exist that can provide call graphs that provide logical flow of instructions via a directed graph, like <a href="https://github.com/vivisect/vivisect">this</a> and <a href="https://radare.org/r/">this</a>.<br></p><p>Dynamic analysis, often called behavioral analysis, is the collection of metadata from an executed binary in a sandbox environment. Dynamic analysis can provide data such as network access, registry/file activity, and API function monitoring. While dynamic analysis is often more informative, it is also more resource intensive, requiring a suite of collection tools and a sandboxed virtual environment. Alternatively, static analysis can be automated to generate disassembly over a large set of binaries generating a corpus ready for the NLP pipeline. At Endgame we have engineered a hybrid approach that automates the analysis of malicious binaries providing data scientists with metadata from both static and dynamic analysis.
</p><h3>Lexical Parsing
</h3><p>Lexical parsing is paramount to the NLP process as it provides the ability to turn large bodies of text into individual tokens. The goal of Malicious Language Processing is to parse a binary the same way an NLP researcher would parse a document:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3fd50bed57da2cb1/5dfd4fdd631b8140b1f0417d/endgame-nlp-chart.png" data-sys-asset-uid="blt3fd50bed57da2cb1" alt="endgame-nlp-chart.png" style="display: block; margin: auto;"></p><p>To generate the “words” in this process we must perform a few traditional NLP techniques. First is tokenization, the process of breaking down a string of text into meaningful segments called tokens.Segmenting on whitespace, new line characters, punctuation or regular expressions can generate tokens. (Fig 2)
</p><figcaption><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc81d810e282f29ab/5dfd4fec4dfaba3e76b434cb/endgame-tokenized-disassembly.png" data-sys-asset-uid="bltc81d810e282f29ab" alt="endgame-tokenized-disassembly.png" style="display: block; margin: auto;"><br>Fig 2. Tokenized disassembly</figcaption><p><br></p><p>The next step in the lexical parsing process is to merge families of derivationally related words with similar meaning or text normalization. The two forms of this process are called stemming and lemmatization.
</p><p>Stemming seeks to reduce a word to its functional stem. For example, in malware analysis this could reduce SetWindowTextA or SetWindowTextW to SetWindowText (Windows API), or JE, JLE, JNZ to JMP (x86 instructions) accounting for multiple variations of the essentially the same function.
</p><p>Lemmatization is more difficult in general because it requires context or the part-of-speech tag of a word (e.g., noun, verb, etc.). In English, the word “better” has “good” as its lemma. In malware we do not yet have the luxury of parts-of-speech tagging, so lemmatization is not yet applicable. However, a rules-based dictionary that associates Windows API equivalents of C runtime functions may provide a step towards lemmatization, such as mapping _fread to ReadFile or _popen to CreateProcess.
</p><h3>Semantic Networks
</h3><p>Semantic or associative networks represent the co-occurrence of words within a body of text to gain an understanding of the semantic relationship between words. For each unique word in a corpus, a node is created on a directed graph. Links between words are generated with an associated weight based on the frequency that the two words co-occurred. The resulting graph can then be clustered to derive cliques or communities of functions that have similar behavior.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2fc31c2a126bbc4d/5dfd500194b1d7403bbd3779/endgame-nlp.png" data-sys-asset-uid="blt2fc31c2a126bbc4d" alt="endgame-nlp.png" style="display: block; margin: auto;"></p><p>A malicious language semantic network could aid in the generation of a lexical database capability for malware similar to WordNet. WordNet is a lexical database of English nouns, verbs, and adjectives grouped into sets of cognitive synonyms. Endgame data scientists are in the incipient stages of exploring ways to search and identify synonyms or synsets of malicious functions. Additionally, we hope to leverage our version of WordNet in the development of lemmatization and the Parts-of-Speech tagging within the Malicious Language Processing framework.
</p><h3>Parts-of-Speech Tagging
</h3><p>Parts-of-Speech (POS) tagging is a piece of software capable of tagging a list of tokens in a string of text with the correct language annotation, such as noun, verb, etc. POS Tagging is crucial for gaining a better understanding of text and establishing semantic relationship within a corpus. Above I mentioned that there is currently no representation of POS tagging for malware. Source code may be too abstract to break down into nouns, prepositions or adjectives. However, it is possible to treat subroutines as “sentences” and gain an understanding of functions used as subjects, verb and predicates. Using pseudo code for a process injection in Windows, for example, would yield the following from a Malicious Language Processing POS-Tagger:
</p><h3><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd505b71f2097d85a/5dfd5014028cba3ef4ec65fe/endgame-process-injection.png" data-sys-asset-uid="bltd505b71f2097d85a" alt="endgame-process-injection.png" style="display: block; margin: auto;"><br></h3><h3>Closing Thoughts
</h3><p>While the majority of the concepts mentioned in this post are being leveraged by Endgame today to better understand malware behavior, there is still plenty of work to be done. The concept of Malicious Language Processing is still in its infancy. We are currently working hard to expand the Malicious Language Processing framework by developing a malicious stop word list (a list of the most common words/functions in a corpus of binaries) and creating an anomaly detector capable of determining which function(s) do not belong in a benign block of code. With more research and larger, more diverse corpuses, we will be able to understand the behavior and basic capabilities of a suspicious binary without executing or having a human reverse engineer it. We view NLP as an additional tool in a data scientist’s toolkit, and a powerful means by which we can apply data science to security problems, quickly parsing the malicious from the benign.
</p>

NLP For Security: Malicious Language Processing

Your package has been successfully encrypted: TeslaCrypt 4.1A and the malware attack chain

<p>When optimizing machine learning (ML) models, model performance often is prioritized over model interpretability. However, in the field of information security interpretability is a sought after feature, especially in the realm of malware classification. By understanding why a model classifies a binary as benign or malicious, security practitioners are better equipped to remediate an alert.
</p><p>We worked with the <a href="https://www.endgame.com/blog/technical-blog/introducing-ember-open-source-classifier-and-dataset">ember benchmark model</a>, a classifier that is currently not interpretable, to design a measure of interpretability that will determine which features contribute to the model’s predictions. This enables future ember users to quickly interpret and explain their findings, highlighting the value of focusing on both performance and interpretability in ML models.
</p><strong><h2><strong>Methods for Model Interpretability</strong></h2></strong><p>There are many ways to approach model interpretability, including partial dependence plots (<a href="https://christophm.github.io/interpretable-ml-book/pdp.html">PDP</a>), local interpretable model-agnostic explanations (<a href="https://christophm.github.io/interpretable-ml-book/lime.html">LIME</a>), and <a href="https://christophm.github.io/interpretable-ml-book/shapley.html">Shapley values</a>.
</p><p>The Shapley values method, which quantifies how much each model feature contributes to the model’s prediction, is the only one that can determine each feature effect on a global scale and thus give a full model explanation. For this reason we chose to use SHapley Additive exPlanation (SHAP) Values, an extension of the Shapley values method, to assess interpretability for the ember benchmark model. For a given model, all the SHAP values are summed to get the overall difference in model prediction in comparison to the baseline prediction without considering feature effect. The SHAP paper is available <a href="https://arxiv.org/abs/1705.07874">here</a> and the open source code we utilized is available <a href="https://github.com/slundberg/shap">here</a>.
</p><p>It is important to note how SHAP values are correlated to file classification. When interpreting malware classification models, lower SHAP values push the model towards classifying a file as benign, whereas higher SHAP values push the model towards classifying a file as malicious.
</p><strong><h2><strong>Applying SHAP for Model Interpretability</strong></h2></strong><p>We applied SHAP value-based interpretability to the ember model in two ways: model summary and single file. Each of these are discussed below.
</p><strong><h3><strong>Model summary</strong></h3></strong><p>The ember dataset consists of raw features extracted from over one million files/samples that are used as training/test sets for the ember model. Each file’s raw features are converted into 2351 vectorized features. By creating a SHAP summary plot for the model, we were able to determine the 20 features (out of 2351) that affected the model the most and how changes in these values affect the model’s prediction.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt650a3665cb2738e1/5e096a2867e97c652d3c7bc0/endgame-ml-black-box-model-summary-blog.png" data-sys-asset-uid="blt650a3665cb2738e1" alt="endgame-ml-black-box-model-summary-blog.png" style="display: block; margin: auto;"></p><figcaption>A summary for all the files/samples in the ember dataset of the correlation between feature values and SHAP values</figcaption><p>This plot shows how high and low feature values are related to SHAP value. For example, Feature 637 (machine type) is the most important model feature because high and low feature values directly correlate to high and low SHAP values. In comparison, other model features are less important because there is less distinction between high and low features values and their resulting SHAP values are closer to zero. This can be seen with Feature 504 (byte entropy histogram), as its corresponding SHAP values are closer to zero and thus has less impact on the model. We were able to use this plot to get a better understanding of how certain features affected the ember model decision.
</p><strong><h3><strong>Single file</strong></h3></strong><p>While understanding features’ effects on the model is important, ideally we want to provide a file and identify how the features in that file specifically influence the model’s prediction.&nbsp;&nbsp;
</p><p>Within the ember dataset, each file’s raw features are already converted into 2351 vectorized features and categorized into eight feature groups (Byte Histogram, Byte Entropy Histogram, String Extractor, General File Info, Header File Info, Section Info, Imports Info, and Exports Info).
</p><p>It would be possible to visualize how each of these 2351 features contributes to the model; however, we determined that the visualization would be too cluttered for a user to extract any meaningful information. Instead, it would be more informative to understand how each feature group contributed to the final prediction so that the user would know what part of the file to scrutinize more.
</p><p>We added together the SHAP values for all the features in each feature group to get an overall SHAP value for each group, and then created a force plot (see example below) to visualize each feature group’s overall effect on the model’s prediction.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3d6e223375255374/5e096a6c7651f865a36da390/endgame-ml-black-box-force-plot-blog.png" data-sys-asset-uid="blt3d6e223375255374" alt="endgame-ml-black-box-force-plot-blog.png" style="display: block; margin: auto;"></p><figcaption>Most of the eight feature groups in this specific file push the model towards classifying the file as malicious (red)</figcaption><p>In addition to feature group importance, we also provide a list of the top five features in the file that contribute to the prediction the most, as well that feature’s information. An example can be seen here:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7697a068b9df9a79/5e096aa7ba09063e77b61069/endgame-ml-black-box-top-5-features-blog.png" data-sys-asset-uid="blt7697a068b9df9a79" alt="endgame-ml-black-box-top-5-features-blog.png" style="display: block; margin: auto;"></p><p>While we can say which features contributed most to the model’s prediction and what groups/subgroups they are in, the next step would be to identify what exactly that feature represents. For example, from the visual above, we can say that Feature 620 is a part of the General File Information, specifically the exports section, but ideally, we would be able to determine exactly which export contributes to how the file is classified.
</p><strong><h2><strong>Conclusion</strong></h2></strong><p>With the possible extensions mentioned above, we now have a measure of model interpretability for the ember model that will help us better understand why it classifies certain files as benign or malicious. Ideally, this interpretability feature could be implemented for other classifiers so that researchers in information security can quickly identify and respond to a security alert.
</p><p>As shown through the ember model, interpretability provides more detailed insights about ML models. These insights will allow researchers to move past current ML black box predictions, thus highlighting the importance of model interpretability and transparency within ML models.
</p>

Opening the machine learning black box with model interpretability

<p>Search results come in all shapes and sizes. But for any search, it’s all about <strong>the goods</strong>: concise, rich results which represent larger documents. And it’s better when it’s fast.<strong></strong> For that, we’re releasing<strong> Result Settings for Elastic App Search.
</strong></p><p><br></p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt01caa5b56d869649/5d72ce3d9c44590436852c99/app-search-result-settings-blog-feature.png" data-sys-asset-uid="blt01caa5b56d869649" alt="Result Settings in Elastic App Search" style="box-shadow: .5px .5px 2px gray; border-radius: 5px; margin-left: auto; margin-right: auto; display: block;">
</p><strong><h2><br></h2><h2>What’s App-ening?</h2></strong><p><a href="https://www.elastic.co/app-search">Elastic App Search</a> is an API-centric search engine with accessible dashboard tools for humans of all kind. <strong>Quickly write search into all of your applications</strong>, from ecommerce to internal logging, game and mobile app functionality, automated content placement or just simple search; the applications are as broad as your imagination. And then use sleek visual tools to refine and perfect them.
</p><p>App Search is a <a href="https://www.elastic.co/app-search/service">managed service</a> by Elastic. Or it’s a self-managed distribution which you can <em>host yourself for free</em> with a basic Elasticsearch license. You can <a href="https://www.elastic.co/downloads/app-search">download it here</a>. We don’t even ask for an email!&nbsp;</p><p><em>Result Settings will be available in the self managed version in the 7.5.0 stack release.
</em></p> <strong><h2>Setting the standard</h2></strong><p><strong>Result Settings</strong> provide two main benefits for search creators. You can:
</p><ol>
	<li>Highlight query matches using <code>&lt;em&gt;</code> tags via snippet fields.</li>
	<li>Enable, disable, and customize fields to optimize query performance.</li>
</ol><p>In other words, you can trim out any excess from your search query responses so that they are as concise as they need to be. And then you can use snippets on your text fields to highlight query matches for improved scan-ability of search results.
</p><p>Setting a performant standard query defers deeper and more intensive for queries when you need them.&nbsp;</p><p>Searchers will experience snappy, rich search where they need it.
</p> <strong><h2>Buttons in action</h2></strong><p>Let’s take a deeper look at <strong>Result Settings</strong>. Imagine that we have a document like this:
</p><pre class="prettyprint">{ 
  “nps_link”: { 
    “raw”: “https://www.nps.gov/grsm/index.htm” 
  }, 
  “title”: { 
    “raw”: “Great Smoky Mountains” 
  }, 
  “date_established”: { 
    “raw”: “1934-06-15T05:00:00+00:00” 
  }, 
  “world_heritage_site”: { 
    “raw”: “true” 
  }, 
  “states”: { 
    “raw”: [ 
      “Tennessee”, 
      “North Carolina” 
    ] 
  }, 
  “description”: { 
    “raw”: “The Great Smoky Mountains, part of the Appalachian Mountains, span a wide range of elevations, making them home to over 400 vertebrate species, 100 tree species, and 5000 plant species. Hiking is the park’s main attraction, with over 800 miles (1,300 km) of trails, including 70 miles (110 km) of the Appalachian Trail. Other activities include fishing, horseback riding, and touring nearly 80 historic structures.” 
  }, 
  “visitors”: { 
    “raw”: 11312786 
  }, 
  “id”: { 
    “raw”: “park_great-smoky-mountains” 
  }, 
  “location”: { 
    “raw”: “35.68,-83.53” 
  }, 
  “square_km”: { 
    “raw”: 2114.2 
  }, 
  “acres”: { 
    “raw”: 522426.88 
  } 
}
</pre><p>It’s a US National Park!… Well, a JSON document <em>representing</em> a US National Park. By default, this result will appear in its whole, raw form. It’s quite large. And that isn’t ideal. </p><p>Most users would search for a  park by its name or&nbsp;a trait, something like “mountains”.&nbsp; The queries are often vague enough that multiple results will appear. If we were to return an object of such heft for each result, that would put undue strain on the search engine. And it would be more of a burden to manage on the front-end. <br></p><p>We want to return a smaller object, one that contains only the <code>title</code> of the park and a smaller subset of the <code>description</code> text. And we want to highlight query matches and HTML escape the responses  so that our searchers see results which clearly contain whatever it is they are seeking.  With such a small, polished object, search will be both quick and valuable.<br></p><p>To do this, open <strong>Result Settings</strong>, un-check the  fields which you do not want to include in your result object, and then set a concise character limit for the fields which you do want. And then make  a choice: are the text fields <strong>raw</strong> or <strong>snippets</strong>? If we select <strong>snippet</strong>, query matches are highlighted - encapsulated in <code>&lt;em&gt;</code> tags - so that they’re enriched for easy scanability.
</p><p>When complete, it would look like this:
</p><p><br></p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt20e72dd9dc324717/5d72ce9a9901373e4fb50209/app-search-result-settings-blog-usage.png" data-sys-asset-uid="blt20e72dd9dc324717" alt="Updating result set settings within Elastic App Search" style="box-shadow: .5px .5px 2px gray; border-radius: 5px; margin-left: auto; margin-right: auto; display: block;">
</p><p><br></p><p>After we click <strong>Save</strong>, the default search result becomes much tighter and quicker, helping your users get where they’re trying to go:
</p><pre class="prettyprint">{ 
  “title”: { 
    “snippet”: “Great Smoky &lt;em&gt;Mountains&lt;/em&gt;” 
  }, 
  “description”: { 
    “snippet”: “The Great Smoky &lt;em&gt;Mountains&lt;/em&gt;, part of the Appalachian &lt;em&gt;Mountains&lt;/em&gt;, span a wide range of elevations” 
  }, 
  “id”: { 
    “raw”: “park_great-smoky-mountains” 
  } 
}
</pre><p>Should you need to provide larger responses and filter or expand upon this slim foundation, that's no problem. The full, dynamic set of search APIs can surface whatever is needed.
</p> <strong><h2>Summary</h2></strong><p>Elastic App Search is the quickest and easiest way to get search built into a wide array of valuable applications. You can start a <a href="https://www.elastic.co/app-search/service?elektra=performant-rich-search-results-blog">14-day free trial of the managed service</a> or <a href="https://www.elastic.co/downloads/app-search">download it for free</a>. <strong>Result Settings</strong> will be there, ready to optimize and enrich your queries. Result Settings are available today within the managed service. <br>
</p><p>Read the <a href="https://swiftype.com/documentation/app-search/guides/result-settings">Result Settings Guide</a>&nbsp;in the documentation to learn more.&nbsp;</p><p><em>Not into the UI thing? You can do all of this <a href="https://swiftype.com/documentation/app-search/api/search-settings">through the API</a>.
</em></p>

app-search-result-settings-blog-banner.jpg

app-search-result-settings-blog-thumb.jpg

Elastic App Search: Performant, Rich Search Results with Result Settings

<p>Two years ago, we published a post near the time that we were <a href="https://www.elastic.co/blog/making-kibana-accessible">kicking off our accessibility work</a>. Since then, we’ve come a long way in our understanding of what it means to build an accessible product and best practices for doing so. With 7.6, we’ve shipped quite a few accessibility improvements and fixes all over Kibana, and thought now would be a good time to highlight some of our recent work.
</p>
<h2>Introducing our accessibility statement</h2>
<p>The goals for our accessibility efforts go beyond compliance: We want all humans to feel empowered to understand and effect the world when they use Elastic products. Not as focused on minutia as a traditional <a href="https://www.section508.gov/sell/vpat">VPAT</a> might be, we hope that by putting our accessibility work front-and-center in an easy-to-read format, we can show our commitment to building an inclusive product for everyone. You can find our statement under <a href="https://www.elastic.co/guide/en/kibana/master/accessibility.html">Accessibility</a> in the Kibana docs. (If you are interested in our formal VPAT, please message <a href="mailto:accessibility@elastic.co">accessibility@elastic.co</a>.)
</p>
<h2>Recent highlights</h2>
<h3>An accessible EUI</h3>
<p><a href="https://github.com/elastic/eui">EUI</a>, or Elastic UI, is the component library that’s behind all of the UI that we create at Elastic. An accessibility focus at this foundational layer of our work ensures solid building blocks supporting every piece of UI we create. By consolidating all of our most used patterns in one area, we’re able to provide consistent experiences and <a href="https://github.com/elastic/eui/issues/2262">handle lots of accessibility concerns</a> before the scores of implementing developers — from Kibana, to Elastic Cloud, to every other UI team, internal and external — even get to work. We still have a <a href="https://github.com/elastic/eui/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3Aaccessibility+">list of improvements</a> we're looking to implement in EUI, but we’re quickly approaching a point where we’ve checked off all of the obvious boxes and can move on to improving Elastic-specific use cases.
</p>
<p>To see some of the things we strive to instill in EUI and in our consuming app, take a look at our recently published <a href="https://elastic.github.io/eui/#/guidelines/accessibility">Accessibility Guidelines</a> or jump into <a href="https://github.com/elastic/eui/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3Aaccessibility">EUI’s Github issue tracker</a> to get involved with the details.
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5c3ff9f34f679ab4/5e4bf98d7c62095ce7b38116/accessibility-guidelines-eui-shot-blog.jpg" data-sys-asset-uid="blt5c3ff9f34f679ab4" alt="Elastic EUI Accessibility Guidelines" style="display: block; margin: auto;">
</p>
<h3>Navigating around Kibana</h3>
<p>Kibana’s big. And loaded with features. And built by a huge, diverse, and growing team. So, sometimes, we can miss the little things that make a UI that’s quick and easy to navigate. With 7.6, we’ve gone back and updated all the products within Kibana with a more complete <a href="https://github.com/elastic/kibana/issues/37539">heading</a> and <a href="https://github.com/elastic/kibana/pull/46569">landmark</a> structure to make navigating with assistive tech that much easier.
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbee971e4cdea641d/5e4bf9ba7c62095ce7b3811c/accessibility-guidelines-kibana-nav-blog.jpg" data-sys-asset-uid="bltbee971e4cdea641d" alt="Making Kibana navigation accessible" style="display: block; margin: auto;">
</p>
<h2>Mindset shift</h2>
<p>Years ago, we promised to continue improving our process around creating accessible experiences for everyone. Since then, we’ve been able to imbue many parts of our process with a larger accessibility focus:
</p>
<ul>
	<li>Members of our QA team have trained on how to use automated tools and screen readers to evaluate accessibility.</li>
	<li>Our design team is trained to think about accessibility up front, which means the design and UX is accessible from the ground up.</li>
	<li>Many PRs now start with accessibility already on the minds of implementing developers from our continued education efforts through Slack, team training, and engagement at company events.</li>
	<li>Accessibility specs for our more complicated components (e.g., <a href="https://github.com/elastic/eui/issues/2404">EuiSuggest</a>, <a href="https://docs.google.com/document/d/16Bje5R00cybqXeCvA0tLM4v8Dl3GwJG6o4etPCiChRM/edit?usp=sharing">EuiDataGrid</a>).</li>
</ul>
<p>But we’re most excited about our rollout of automated accessibility testing powered by <a href="https://www.deque.com/axe/">axe</a>. Automated testing can’t find every problem but it’s great at picking off that low-hanging fruit so that we can focus on the harder problems. <a href="https://github.com/elastic/kibana/pull/43584">Kibana is already up and running</a> with <a href="https://github.com/elastic/kibana/pull/54209">several high-level tests</a> and <a href="https://github.com/elastic/eui/pull/2569">EUI has kicked off the process</a> to follow suit.
</p>
<h2>Future work</h2>
<p>There’s always more work to do but here are some things that we’ve got our sights set on for the immediate future.
</p>
<h3>An accessible Charts library</h3>
<p>As Elastic Charts grows to become the future of all charting around Elastic, we recognize that it needs to work for all of our users. We’ve been talking about the best ways we can support sighted and vision-impaired screen reader users and have some concrete ways we plan to improve. This won’t be a one-and-done sort of effort, so we encourage you to monitor our progress at <a href="https://github.com/elastic/elastic-charts/issues/300">elastic-charts#300</a> and get involved.
</p>
<h3>Better navigation</h3>
<p>Navigating between pages in <a href="https://en.wikipedia.org/wiki/Single-page_application">SPAs</a> has always been a little more difficult for screen reader users than traditional websites have been. We have a <a href="https://github.com/elastic/kibana/issues/56232">big redesign</a> coming together for how to improve our navigation&nbsp;for everyone and are expressly tracking some of the <a href="https://github.com/elastic/kibana/issues/42379">high-level accessibility issues</a> that we plan on tackling at the same time.&nbsp;
</p>
<h2>How you can help</h2>
<p>Accessibility is a critical part of the user experience. If you'd like to help us make our interfaces more accessible, you can browse <a href="https://github.com/elastic/eui/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3Aaccessibility">EUI's</a> or <a href="https://github.com/elastic/kibana/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3AProject%3AAccessibility">Kibana’s</a> accessibility issues to find areas that need improvement. If you find a part of our UI that's inaccessible to you, please file an issue or <a href="mailto:accessibility@elastic.co">email</a> so we can work together to find a solution.
</p>

accessibility-blog-banner.png

accessibility-blog-thumb.png

Our accessibility journey: Mindset and tools

<article>
<p>Percolator is another feature that will be part of upcoming <strong>elasticsearch</strong> 0.15. The percolator allows you to register queries against an index, and then send <code>percolate</code> requests which include a doc, and getting back the queries that match on that doc out of the set of registered queries.
</p>
<p>Think of it as the reverse operation of what <strong>elasticsearch</strong> does by nature: instead of sending docs, indexing them, and then running queries, one sends queries, registers them, and then sends docs and finds out which queries match that doc.
</p>
<p>As an example, a user can register an interest (a query) on all tweets that contain the word “elasticsearch”. For every tweet, one can percolate the tweet against all registered user queries, and find out which ones matched.
</p>
<h2>Sample Usage</h2>
<p>First, create an index we will work with:
</p>
<pre class="prettyprint">curl -XPUT 'localhost:9200/twitter'
</pre>
<p>Next, we will register a percolator query with a specific name called <code>elasticsearch</code> against the <code>twitter</code> index:
</p>
<pre class="prettyprint">curl -XPUT localhost:9200/_percolator/twitter/elasticsearch -d '{
    "query" : {
        "field" : {
            "message" : "elasticsearch"
        }
    }
}'
</pre>
<p>And now, we can percolate a document and see which queries match on it (note, its not really indexed!):
</p>
<pre class="prettyprint">curl -XGET localhost:9200/twitter/tweet/_percolate -d '{
    "doc" : {
        "message" : "this new elasticsearch percolator feature is nice, borat style"
    }
}'
</pre>
<p>And the matched percolated queries are part of the response:
</p>
<pre class="prettyprint">{"ok":true,"matches":["elasticsearch"]}
</pre>
<h2>Percolate on Index / Bulk</h2>
<p>Percolation can also be done when executing an index request. Lets index the tweet from before, and also specify that it needs to be percolated by adding the <code>percolate</code> parameter:
</p>
<pre class="prettyprint">curl -XPUT 'localhost:9200/twitter/tweet/1?percolate=*' -d '{
    "message" : "this new elasticsearch percolator feature is nice, borat style"
}'
</pre>
<p>And in the response, we will also get the percolated queries it matched on:
</p>
<pre class="prettyprint">{
    "ok":true,
    "_index":"twitter",
    "_type":"tweet",
    "_id":"1",
    "_version":1,
    "matches":["elasticsearch"]
}
</pre>
<h2>Filtering Executed Queries</h2>
<p>Since the registered percolator queries are just docs in an index, one can filter the queries that will be used to percolate a doc. For example, we can add a color field to the registered query:
</p>
<pre class="prettyprint">curl -XPUT localhost:9200/_percolator/twitter/elasticsearch -d '{
    "color" : "green",
    "query" : {
        "field" : {
            "message" : "elasticsearch"
        }
    }
}'
</pre>
<p>And then, we can percolate a doc that only matches on <code>green</code> percolated queries:
</p>
<pre class="prettyprint">curl -XGET localhost:9200/twitter/tweet/_percolate -d '{
    "doc" : {
        "message" : "this new elasticsearch percolator feature is nice, borat style"
    },
    "query" : {
        "term" : {
            "color" : "green"
        }
    }
}'
</pre>
<p>With no matches received for <code>blue</code> percolated queries:
</p>
<pre class="prettyprint">curl -XGET localhost:9200/twitter/tweet/_percolate -d '{
    "doc" : {
        "message" : "this new elasticsearch percolator feature is nice, borat style"
    },
    "query" : {
        "term" : {
            "color" : "blue"
        }
    }
}'
</pre>
<p>The filtering can also be passed as a query string to the percolated parameter when indexing a doc, for example:
</p>
<pre class="prettyprint"># matches
curl -XPUT 'localhost:9200/twitter/tweet/1?percolate=color:green' -d '{
    "message" : "this new elasticsearch percolator feature is nice, borat style"
}'
# no match
curl -XPUT 'localhost:9200/twitter/tweet/1?percolate=color:blue' -d '{
    "message" : "this new elasticsearch percolator feature is nice, borat style"
}'
</pre>
<h2>How it Works</h2>
<p>The <code>_percolator</code> which holds the repository of registered queries is just a another index in ES. The query is registered under a concrete index that exists (or will exist) in ES. That index name is represented as the type in the _percolator index (a bit confusing, I know…).
</p>
<p>The fact that the queries are stored as docs in another index (<code>_percolator</code>) gives us both the persistency nature of it, and the ability to filter out queries to execute using another query 😊.
</p>
<p>The <code>_percolator</code> index uses the <code>index.auto_expand_replica</code> setting to make sure that each data node will have access locally to the registered queries, allowing for fast query executing to filter out queries to run against a percolated doc.
</p>
<p>The percolate API uses the whole number of shards as percolating processing “engines”, both primaries and replicas. In our above case, if the test index has 2 shards with 1 replica, 4 shards will round robing in handing percolate requests. (dynamically) increasing the number of replicas will increase the number of percolation power.
</p>
<p>Note, percolate request will prefer to be executed locally, and will not try and round robin across shards if a shard exists locally on a node that received a request (for example, from HTTP). Its important to do some roundrobin in the client code among ES nodes (in any case its recommended).
</p>
<aside id="post_tags">
</aside>
</article>

Percolator

<p><strong></strong>
</p><h2><strong>Comparing Elastic’s Elasticsearch Service costs to New Relic and DataDog</strong></h2><p>Thousands of customers leverage the official Elasticsearch Service (ESS) on top of Elastic Cloud to get the best experience for running Elasticsearch — including all of Elastic’s exclusive products such as <a href="/products/log-monitoring">Elastic Logs</a>, <a href="/products/apm">Elastic APM</a>, <a href="/products/siem">Elastic SIEM</a>, and more. ESS is the only managed Elasticsearch service that gives you the complete Elasticsearch experience — all the features, all the solutions, and support from the source — as well as a number of operational and deployment benefits that complement these products.
</p><p>In a <a href="/blog/elastic-endpoint-security-what-does-no-per-endpoint-pricing-mean">previous post</a>, we’ve explained the cost benefits of Elastic’s resource-based pricing model in comparison with other vendors’ more punitive pricing models. In this series, we’ll highlight additional operational and deployment benefits that allow you to further drive down costs — and increase the value you derive from ESS. This post covers how the extensive cloud provider and region presence of ESS across 22 regions (and counting) and 3 cloud providers (Azure, GCP, and AWS) allow you to benefit from data locality and significantly reduce your cloud provider data transfer costs.
</p><h2><strong>Why are my costs so high?</strong>
</h2><p>Some of the most overlooked expenses when running on Amazon, Azure, or Google are the outbound data transfer charges. These charges are incurred whenever data is sent outside a cloud region. This means every log file sent, metric data point shipped, and security event captured and sent outside of the cloud region your application or infrastructure are deployed on incurs an additional fee. While data entering into a cloud provider or moved within a region can be relatively cheap or free, <a href="https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer">Amazon data-out fees</a>, for example, can be nine times higher per GB than its in-region data transfer costs ($0.09 vs. $0.01 per GB).
</p><p>With exponentially growing data volumes in use cases such as logging, metrics, and APM in observability as well as networking and endpoint data in security, this can represent significant added costs over time. These fees are even more overwhelming when leveraging other SaaS solutions that do not have a local presence in the region your workloads are running on. For example, both DataDog and New Relic both only expose “US” and “EU” as endpoints to which you can ship your data. These endpoints give zero hints on what underlying cloud provider or which region you are sending data to, almost guaranteeing that you’ll face punitive data-out charges.
</p><h2><strong>How much can data transfer egress charges affect me?</strong>
</h2><p>To get a real sense of data transfer egress charges, we will run through some real-world scenarios of shipping various data types such as logs, metrics, and APM to ESS, New Relic, and DataDog. We’ll assume all data sent to services is uncompressed — the default setting in many agents — and is configurable depending on the method used to send data to each service.&nbsp;We’ll also assume our application runs on the Azure East US 2 region and our ESS deployment runs in the same region. With the other SaaS solutions, we’ll use their respective US options.
</p><p>When running the same Azure region, the bandwidth charges are currently $0 per GB to transfer data in and out of an availability zone. To ensure a fair comparison, we will also include bandwidth charge in case you were running on top of another cloud provider where you could incur in-region data transfer charges. As of this post, AWS charges $0.01 per GB for traffic within a region, and GCP charges the same.
</p><p>The Azure East US 2 region has different rates for outbound data transfer depending on the total amount of data. At the time of writing this blog, the first 5GB per month are free, the next 9.95TB per month are charged at $0.087, and the next 30TB are charged at $0.083. To ensure we get an accurate number and account for all the tiers properly, we’ll use the Azure pricing calculator’s bandwidth to generate the monthly cost and multiply by 12 to get the annual cost.
</p><h3><strong>Outbound data transfer costs: Logs (annual)</strong>
</h3><p>First let’s take an example of comparing outbound data transfer costs for one of the most popular and voluminous data sources, logs. We’ll use three different volumes: 50GB, 500GB, and 1TB per day from an application. For each log volume, we’ll first divide by 24 to get an hourly ingest, then multiply by 730 to get a monthly amount. We’ll then input this into the calculator to get the total monthly, which we will then multiply by 12 to get the total annual cost. These numbers are rounded to the nearest whole number:
</p><table style="width:100%">
<tbody>
<tr>
	<td>
	</td>
	<td style="background-color: rgb(232, 246, 254); text-align: center;" colspan="3">
		<strong>Logging ingest rate per day</strong>
	</td>
</tr>
<tr style="background-color:#E8F6FE;">
	<td>
		<strong>Service</strong>
	</td>
	<td>
		<strong>50GB</strong>
	</td>
	<td>
		<strong>500GB</strong>
	</td>
	<td>
		<strong>1TB</strong>
	</td>
</tr>
<tr>
	<td>
		Elasticsearch Service
	</td>
	<td>
		Up to $182
	</td>
	<td>
		Up to $1,825
	</td>
	<td>
		Up to $3,650
	</td>
</tr>
<tr style="background-color:#F6F7FA;">
	<td>
		New Relic / DataDog (US)
	</td>
	<td>
		$1,582
	</td>
	<td>
		$15,624
	</td>
	<td>
		$31,080
	</td>
</tr>
</tbody>
</table><p>We can easily see that data transfer costs are not something we can ignore when performing moderate scale logging.
</p><p>Logs, however, are only part of the equation to reaching full observability into your application and infrastructure. Customers typically also add metrics and APM traces to enhance their ability to troubleshoot and solve issues. Let’s consider a scenario of adding metrics on top of 50, 1,000, and 5,000 hosts and determine the data transfer cost.
</p><h3><strong>Outbound data transfer costs: Infrastructure metrics (annual)</strong>
</h3><p>Similar to the method above, we’ll first calculate total data generated from each host and then use that to generate total data sent out. While metric collection and storage may differ per provider, in an effort to normalize, we’ll say each agent collects 100 bytes per metric and 100 metrics every 10 seconds, system and custom. This totals to 31.5GB per host per year.
</p><table style="width:100%">
<tbody>
<tr>
	<td>
	</td>
	<td style="background-color: rgb(232, 246, 254); text-align: center;" colspan="3">
		<strong>Total number of hosts monitored</strong>
	</td>
</tr>
<tr style="background-color:#E8F6FE;">
	<td>
		<strong>Service</strong>
	</td>
	<td>
		<strong>50</strong>
	</td>
	<td>
		<strong>1,000</strong>
	</td>
	<td>
		<strong>5,000</strong>
	</td>
</tr>
<tr>
	<td>
		Elasticsearch Service
	</td>
	<td>
		Up to $16
	</td>
	<td>
		Up to $315
	</td>
	<td>
		Up to $1,575
	</td>
</tr>
<tr style="background-color:#F6F7FA;">
	<td>
		New Relic / DataDog (US)
	</td>
	<td>
		$131
	</td>
	<td>
		$2,709
	</td>
	<td>
		$13,745
	</td>
</tr>
</tbody>
</table><p>With the rise of containers and microservices, these numbers can grow enormously, as you can start to have thousands of hosts that last less than a minute and contain important metric information. With these next-generation data sources, additional custom metrics, or more frequent metric collection, these outbound data transfer costs also quickly add to your total bill.
</p><h3><strong>Outbound data transfer costs: APM (annual)</strong>
</h3><p>Last but not least, let’s add APM data to complete the observability trifecta. This example will use 10, 50, and 100 services, and each of these applications will have 10 medium-sized transactions (~200KB) every minute. This totals to 1.05TB per service per year:
</p><table><colgroup><col width="193"><col width="109"><col width="102"><col width="102"></colgroup>
</table><table style="width:100%">
<tbody>
<tr>
	<td>
	</td>
	<td style="background-color: rgb(232, 246, 254); text-align: center;" colspan="3">
		<strong>Total number of services</strong>
	</td>
</tr>
<tr style="background-color:#E8F6FE;">
	<td>
		<strong>Service</strong>
	</td>
	<td>
		<strong>10</strong>
	</td>
	<td>
		<strong>50</strong>
	</td>
	<td>
		<strong>100</strong>
	</td>
</tr>
<tr>
	<td>
		Elasticsearch Service
	</td>
	<td>
		Up to $105
	</td>
	<td>
		Up to $525
	</td>
	<td>
		Up to $1,050
	</td>
</tr>
<tr style="background-color:#F6F7FA;">
	<td>
		New Relic / DataDog (US)
	</td>
	<td>
		$909
	</td>
	<td>
		$4,677
	</td>
	<td>
		$9,360
	</td>
</tr>
</tbody>
</table><h2><strong>Conclusion</strong>
</h2><p>Overall you can see that sending data to a SaaS solution that’s not in the same region as your services, infrastructure, or logging devices incurs a hefty fee. These fees get compounded by these services’ complicated billing models for logs ingested, APM agents used, and metrics collected.
</p><p>In contrast, the Elasticsearch Service offers <a href="/guide/en/cloud/current/ec-reference-regions.html">22 distinct regions across 3 cloud providers with ESS today</a> and many more coming in the near future, plus a unique <a href="/blog/elastic-endpoint-security-what-does-no-per-endpoint-pricing-mean">resource-based pricing model</a> that doesn’t penalize you for using more APM agents, monitoring more hosts, or ingesting more logs.
</p><p>Want to experience all these benefits for yourself? Be sure to sign up for a <a href="/products/elasticsearch/service">14-day trial</a>.
</p>

blog-banner-cloud-network.jpg

blog-thumb-cloud-network.jpg

Remove hidden network costs for your logs, metrics, and APM data by using Elasticsearch Service

Remove hidden network costs for your logs, metrics, and APM data

Elastic SIEM is a great way to provide security analytics and monitoring capabilities to small businesses and homes. Check out Part 6 of this series.

blog-banner-castle-tower.jpg

Elastic SIEM for home and small business: Beats on Mac

We've extended the timeline to July 21, 2020 for ending support for TLS 1.0/1.1 and unencrypted (plaintext HTTP) traffic to Elasticsearch Service.

<p>As COVID19 continues to affect the world, we have decided to extend the timeline for <a href="https://www.elastic.co/blog/support-ending-for-tls-1-0-tls-1-1-http-traffic-to-elasticsearch-service">removing support for TLS 1.0/1.1 and unencrypted traffic for clients sending data to Elasticsearch Service</a> by another three months to July 21, 2020 from the initial date of April 21, 2020.
</p><p>In addition to extending the date, we have decided to do a trial run (by removing support for plaintext HTTP and TLS 1.0/1.1) prior to the final cutoff. We aim to do this between June 29 and July 10, 2020 for a short period of time (about 2 days) and will keep the <a href="https://cloud-status.elastic.co/">status page</a> updated. This will allow you to identify clients that might not yet be updated or even identified to be updated if you have monitoring setup.
</p><h2>What do you need to do?</h2><p>In order to understand if your clients need to be updated, read the <a href="https://www.elastic.co/blog/support-ending-for-tls-1-0-tls-1-1-http-traffic-to-elasticsearch-service">blog post</a> we published in January that has details on the client versions that support TLS 1.2. If you are still unable to determine if your clients support TLS 1.2, please reach out to our support team via <a href="https://support.elastic.co/customers/s/login/">our support portal</a> or by emailing <a href="mailto:support@elastic.co">support@elastic.co</a>.
</p><p>We understand that the current pandemic has no deadline, and we are keeping our eyes and ears open to what our community needs. We will align and evolve plans accordingly. If you have any further questions, please email <a href="mailto:support@elastic.co">support@elastic.co</a>.
</p><p>Thanks for being a valued Elasticsearch Service customer.
</p>

blog-banner-security-circuit-lock.jpg

blog-thumb-security-circuit-lock.jpg

Deadline extended for end of support for TLS 1.0/1.1 and unencrypted (plaintext HTTP) traffic to Elasticsearch Service

Apache Lucene has a robust feature set that's been growing and maturing with each version. From full-text search to numerics to geospatial data, find out what features and advancements contributors are most excited about.

<p><a href="/celebrating-lucene">Apache Lucene turned 20</a> this year, and to celebrate, we've reached out to folks involved in the project to talk about its past, present, and future. In this week's blog, we're going to take a look at some of the amazing&nbsp;<strong>features</strong>&nbsp;that have been implemented over the years.&nbsp;In addition to hearing from Lucene's&nbsp;founder, Doug Cutting, we'll&nbsp;also feature&nbsp;different Project Management Committee (PMC) members, committers, and contributors —&nbsp;highlighting their favorites. Let's start with Simon:
</p><blockquote>My favorite lucene feature is the DocValues feature, which may seem like a “must-have” and straightforward to implement. Up to a certain extent that's true, but the interesting part is the history behind it.<br><br>
After the great rewrite of Lucene 4, where we moved to something called flexible indexing (which allowed us to plug-in posting formats in the beginning&nbsp;and eventually entire codecs, including almost everything we write to disk), we started working on a feature called Column Stride Fields. Due to the name conflict, or rather, confusion with the Compound File System (CSF vs. CFS) it was renamed to DocValues. It was a rather simple typed, column-oriented mechanism to store per-field values, and at the same time was one of the most wanted and requested features to date. We started with an iterative API and quickly moved to a more flexible random access API. It had many many iterations until we had a stable API. Everybody was convinced this was the way to go.<br><br>
A couple of years later we saw many, many users of Elasticsearch suffer from that decision, since with a  random access API you also need to provide fast random access. Yet, with sparse data — which is what many users of Elasticsearch ingest into the system — you end up with huge amounts of space wasted on disk. Obviously the first thing you think about is compression, but to efficiently compress sparse data we needed to rethink our API decision. We move back to an iterator based API since Lucene’s recommended access pattern is document a time, which means we can move away from random access and require users to access DocValues in internal document ID order. This gave us the ability to quickly improve the situation on the sparse data significantly and at the same time allow for fast random access if the data is dense.<br><br>
This feature took about 5 to 7 years to mature. It teaches us that simple things are never simple and that revisiting a design decision is always worth it.<br><br>- Simon Willnauer, Apache Lucene PMC
</blockquote><p>Everyone has a favorite feature (or maybe a few). Let's hear from the community...
</p><div class="video embed-container" style="height: 319.725px;">
	<img class="vidyard-player-embed" src="https://play.vidyard.com/omCyk5R5NiWuMhH28qL7jn.jpg" data-uuid="omCyk5R5NiWuMhH28qL7jn" data-v="4" data-type="inline" style="width: 100%; margin: auto; display: block;">
</div><p>Visit our <a href="/celebrating-lucene">Apache Lucene timeline</a> to see how it became the search engine that powers digital experiences worldwide.
</p>

blog-lucene-birthday-cake-banner.png

blog-lucene-birthday-cake-thumb.png

Celebrating 20 years of Apache Lucene: Features

Apache Lucene has had a great first 20 years, and we're excited to see what happens in the coming decades. Check out this short blog and video to find out what developments the community sees on the horizon.

<p><a href="/celebrating-lucene">Apache Lucene turned 20</a> this year, and to celebrate, we've reached out to folks involved in the project to talk about its past, present, and future. In this week's blog, we're going to take a look at what the <strong>future</strong> may hold in store for Lucene.&nbsp;In addition to hearing from Lucene's&nbsp;founder, Doug Cutting, we'll also feature&nbsp;different Project Management Committee (PMC) members, committers, and contributors — each sharing their hopes for the future. Let's start with Adrien:
</p><blockquote>We have a good momentum around finding top hits without visiting all documents that match a query. This started a couple of years ago when we started implementing block-max WAND, and we're now working on generalizing the approach to queries sorted by field. This might not sound big, but there are so many deployments of Lucene around the world that even modest improvements have the potential to save lots of computing power! What's beyond that? I have honestly no idea. There are ongoing discussions about adding support for high-dimensionality data in Lucene, I'm curious where this will go. What I'm certain of however, is that Lucene has a bright future ahead.<br><br>- Adrien Grand, Apache Lucene PMC
</blockquote><p>Lucene's future is so bright, it has to wear shades. Let's hear what the community is looking forward to...
</p><div class="video embed-container" style="height: 319.725px;">
	<img class="vidyard-player-embed" src="https://play.vidyard.com/UWgKNwqhNudL39LtStDQuy.jpg" data-uuid="UWgKNwqhNudL39LtStDQuy" data-v="4" data-type="inline" style="width: 100%; margin: auto; display: block;">
</div><p>Visit our <a href="/celebrating-lucene">Apache Lucene timeline</a> to see how it became the search engine that powers digital experiences worldwide.
</p>

Celebrating 20 years of Apache Lucene: Future

Welcome to our latest newsletter blog edition, where you'll find information on upcoming events in your region, catch up on content, and stay up to date with product updates.

What’s new?

The new Kibana Console available on every page (Elastic Search 8.13)

Image_1_newsletter.png

Tech preview of editing and visualizing ES|QL queries on Elastic Observability 8.13

Image_2_newsletter_(1).png

Refined Cloud Security Posture Management (CSPM) benchmark rules experience on Elastic Security 8.13

Image_3_newsletter_(1).png

Blogs, videos, and interesting links

Featured blogs from the community

Upcoming events and meetups

America

Europe, Middle East, and Africa

Asia-Pacific

ELK/Elastic Stack

Elasticsearch

Community

community-blog-series-04_(1)_(1).png

DevRel newsletter April 2024

DevRel newsletter — April 2024

Learn how to implement a plugin and enable ChatGPT users to extend ChatGPT with any content indexed in Elasticsearch, using the Elastic documentation. 

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

Thank you!

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS