<a href="/app-search/service?elektra=advanced-search-queries-blog">Elastic App Search</a> is a solution built on top of Elasticsearch that offers a curated experience for search use cases right out of the box. It makes it much easier and faster to build common search features like autocomplete, faceting, and tune-able relevance. App Search provides this through a more concise API and an intuitive admin dashboard UI that’s accessible for both developers and non-developers alike.
App Search may expose fewer APIs than Elasticsearch, but don’t let that mislead you into thinking that App Search only supports the more basic features of search. The truth is, Elastic App Search can be used to support search use cases with more complex data models and relevance scoring requirements. This blog post highlights two of those features: Result Grouping and Boosting.
<h2>Result Grouping</h2>Result grouping is a query feature that groups or “rolls up” results by a higher level entity. For example, within an ecommerce search implementation, each document in the search engine may represent a unique product variant at the SKU level. One document will represent a shoe with Style ID 123, size 6, colour blue and another might represent represent Style ID 123, size 7, colour green.
<pre class="”prettyprint”">[
 {
 style_id: 123
 size: 6,
 colour: blue
 },
 {
 style_id: 123,
 size: 7,
 colour: green
 }
]
</pre>This SKU-based data model allows the user to drill down by SKU level attributes such as size and colour accurately. But when you return results on this SKU, you will overwhelm users with separate search result for each variant.
In practicality, consider an online store’s search experience when a user queries for “sneakers”. Within the first 8 results, there is one shoe in 6 different colours and a second shoe in 2 colours. This is not helpful — there are really only 2 different types of shoe, yet we have 8 results!
A much more helpful search experience would return 8 unique products. Each product variant would then be grouped together to provide a cleaner user experience. The user would see, at a glance, the colours, sizes, and so on, available for each individual product.
Applying a Result Grouped query is as simple as adding a parameter to specify the group key field alongside the other query parameters:
<pre class="”prettyprint”">{
 "query": "sneakers",
 "group": {
 "field": "style_id"
 }
}
</pre><h2>Value Boosting and Functional Boosting</h2>A common requirement is to have certain document types boosted in the results list. You might want restaurants who accept reservations to score higher than those that do not. Or, you may want to find a national park that is also a world heritage site, or a shoe that comes in velcro straps instead of laces.
You can accomplish this through Value Boosting. The following configuration states that documents where ‘accepts_reservations’ = ‘true’ should be boosted with an Impact of 2. The higher the Impact, the higher the score when the value is present:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt73e3d6039849121f/5c3047a8ff9c993a6eb5a3e8/relevance-tuning-value.png" data-sys-asset-uid="blt73e3d6039849121f" alt="relevance-tuning-value.png">
Elastic App Search also allows you to define Weights on a field. One may want the name or title field to be the more important field, compared to something like the body text or the URL. Similar to Boosts, Weights can be scaled with an Impact from 1-10. But Boosts are different from Weights in that they don’t have anything to do with how well the user’s query matches the document. Boosting is about influencing the score based on a field value within the document.
While the Value Boost influences the score based on a static value, Functional Boosts allow you to influence the score using a numeric field value on the document. In the next image, we’re boosting documents based on the value of the Rating field to help make sure highly rated restaurants show up closer to the top of the results:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltba443e11fe051bdf/5c3047a65a7621286e3e906f/relevance-tuning-logarithmic.png" data-sys-asset-uid="bltba443e11fe051bdf" alt="relevance-tuning-logarithmic.png">
App Search provides detailed control over Functional Boosts allowing you to select a Logarithmic, Exponential, or Linear function and a Multiple or Add operation. These different Boosting functions offer fine-tuned control over how a numeric field such as Number of Clicks, Number of Likes, and Gross Margin, should influence the scoring.
Finally, App Search offers a Proximity Boosting feature that allows you to influence scoring based on the difference between a given value and the document’s field value. App Search has four field types: text, number, date, and geolocation. Proximity Boosting works on number, date, and geolocation fields.
If we want restaurant result scores to gradually diminish as the distance increases between a restaurant and the user’s current location, we could configure a Proximity Boost that looks like this:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7971399eef9dbd71/5c3047a4c599e169662410b8/relevance-tuning-geo.png" data-sys-asset-uid="blt7971399eef9dbd71" alt="relevance-tuning-geo.png">
The Proximity Boost supports three different functions: Gaussian, Linear, or Exponential. A nice bonus is that these complex boosting features are configurable within the Admin Dashboard! And we have <a href="https://swiftype.com/documentation/app-search/guides/relevance-tuning">helpful documentation</a> to guide you through it. This means that even non-developers can tune these features without needing to alter the application code.
<h2>Summary</h2>Elastic App Search is a search solution that exposes powerful Elasticsearch features like Result Grouping and Functional Boosting through a simpler API and an administrative UI so you can integrate them into your application faster and more easily.&nbsp;
You can get started today with a <a href="/app-search/service?elektra=advanced-search-queries-blog" target="_blank">free 14 day trial</a> of our hosted version or <a href="https://elastic.co/downloads/app-search">download the beta</a> of the self-managed version. 
Happy Searching!

full-bleep.jpg

thumb-yarrrr.jpg

Advanced Search Queries in Elastic App Search

<table style="background: #FFFFD2;"><tbody><tr><td>NOTE: This article now contains outdated information. Check out this updated post about <a href="/blog/filebeat-modiles-access-logs-and-elasticsearch-storage-requirements">Elasticsearch storage requirements</a>.</td></tr></tbody></table>Several months ago we wrote about the <a href="https://www.elastic.co/blog/elasticsearch-storage-the-true-story" target="_blank">true story behind Elasticsearch storage requirements</a> and today we’re here to refresh those tests with the new <a href="https://www.elastic.co/blog/elasticsearch-2-0-0-beta1-released" target="_blank">Elasticsearch 2.0 beta-1</a> which includes several enhancements. For those of you looking for a quick answer, these tests showed an overall compression ratio of 0.732 (index size on disk / raw log file size) for those of you who are doing both search and visual analysis of your log data.But of course, the answer to “How much hardware will I need” is still “It depends!” Many factors need to be taken into account and both this post and our previous are meant to elaborate on these factors in detail so you can make informed choices for both the best performance and the right hardware purchases. <h2>Changes since last time</h2>When we wrote our last blog post, we ran our experiments using Elasticsearch 1.4. Since then, there have been hundreds of new features, enhancements and bug fixes made to Elasticsearch. With respect to disk storage requirements, the key changes are: 1) the addition of a best_compression option for stored fields and 2) doc_values enabled by default. In addition, we found a bug in our original experiments that had a significant impact on index size.&nbsp; <h3>DEFLATE: no footballs have been tampered in these tests</h3>As described in this <a href="https://www.elastic.co/blog/store-compression-in-lucene-and-elasticsearch" target="_blank">blog post</a>&nbsp;by Adrien Grand, one of our engineers and core Lucene committers, Lucene 5.0 added a new compression option using the DEFLATE algorithm (the same algorithm behind zip, gzip, and png). Elasticsearch 2.0 is the first version of Elasticsearch to rely on Lucene 5.x and thus is able to leverage the new compression option. This is exposed via the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.0/index-modules.html#_static_index_settings" target="_blank">index.codec configuration setting</a>.&nbsp;In our tests, using&nbsp;best_compression reduces our index size&nbsp;between 15-25% depending on the configuration. This is substantial, especially when looking at larger clusters. Many large Elasticsearch clusters are 50-100 nodes and more, where cluster size is primarily driven by sheer data volume. When you can cut down the amount of hardware by 15-25%, that’s a pretty significant change for the better. So what’s the catch? The stored fields (value of the _source field) are what’s compressed, so there’s only a performance penalty due to decompression when the stored fields are returned in the query response. When the size=0 parameter is added to the request (as is recommended for pure aggregations queries and what Kibana does under the covers), there is no decompression penalty. There is a small performance penalty at index time; in many cases, people will gladly give up&nbsp;the extra CPU required for compression in exchange for disk space but you’ll have to consider this in the context of your requirements.&nbsp;Alternatively, we expect many users will want to utilize a <a href="https://www.elastic.co/blog/hot-warm-architecture" target="_blank">“hot/warm” architecture</a> using the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-allocation.html" target="_blank">shard allocation filtering feature</a>. In this scenario, time-based indexes on hot nodes can be configured to be created with the default LZ4 compression;&nbsp;when they’re migrated to the warm nodes (configured with index.codec: best_compression in elasticsearch.yml), the indexes can&nbsp;be compressed by <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-optimize.html" target="_blank">optimizing that index</a>. It may be preferable to pay the CPU penalty of compression during this optimize process (which we often recommend executing when the cluster is known to be less utilized) than at the time of initial indexing. <h3>What's up doc_values?</h3><a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html" target="_blank">Doc values</a> were introduced with Elasticsearch 1.0 as an alternative means of storing data that is better suited for analytics workloads by using a columnar representation and can be accessed off the JVM heap. Initially, using doc values was only recommended in specific scenarios but over time, we’ve seen our users wanting to run analytics on continuously growing volumes of data that simply could not be executed under the constraints of the JVM heap. Fortunately, we’ve made significant enhancements to eliminate most of the performance gap between using field_data and doc_values, to the point where we felt comfortable <a href="https://github.com/elastic/elasticsearch/issues/8312" target="_blank">enabling doc_values by default</a> for fields of all types except analyzed string fields.There are two ways doc values can impact a cluster’s hardware requirements. First, enabling doc_values requires more disk but this should be offset by the gains from enabling best_compression as described above. Second, doc_values can reduce overall hardware footprint by not letting the limits of the JVM heap define the hardware profile of the nodes or the horizontal scaling point. For example, without doc_values enabled, a node might have been sized&nbsp;with a smaller indexed data volume to ensure that JVM heap utilization is kept under control to avoid frequent garbage collection. This becomes less of a concern when the data can be accessed off-heap.<h3>Dates and strings</h3>In our original tests, our Logstash configuration converts the string representation of the date/time into a date-typed value. We forgot to delete the original string representation of the date/time, which is safe to do if the date/time parsing is successful. This oversight is significant since we discovered the date/time string values made up about 20% of the overall index size in our test data set. Elasticsearch, as with most search engines, builds an inverted index to provide fast lookups; when you have a value that’s highly unique (such as a timestamp), this can greatly increase the size of that index. <h2>The details</h2>The test methodology hasn’t changed so you can check out the <a href="https://www.elastic.co/blog/elasticsearch-storage-the-true-story" target="_blank">old blog post</a> or the <a href="https://github.com/elastic/elk-index-size-tests/blob/master/README.md" target="_blank">README in the Github repo</a>&nbsp;for the details.&nbsp;<table><thead><tr><td>Test </td><td>String fields </td><td>_all </td><td>index size /w LZ4 </td><td>index size /w DEFLATE </td><td>expansion ratio /w LZ4 </td><td>expansion ratio /w DEFLATE </td><td>Impact of DEFLATE </td></tr></thead><tbody><tr><td>Structured data file. Original file size:&nbsp;67644119 </td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>1 </td><td>analyzed and not_analyzed&nbsp; </td><td>enabled </td><td>63047579 </td><td>53131592 </td><td>0.932 </td><td>0.785 </td><td>-0.157 </td></tr><tr><td>2 </td><td>analyzed and not_analyzed&nbsp; </td><td>disabled </td><td>48271433 </td><td>38327106 </td><td>0.713 </td><td>0.566 </td><td>-0.206 </td></tr><tr><td>3 </td><td>not_analyzed </td><td>disabled </td><td>38920800 </td><td>29014796 </td><td>0.575 </td><td>0.428 </td><td>-0.254 </td></tr><tr><td>3b </td><td>not_analyzed, except for 'message' field which is retained and analyzed </td><td>disabled </td><td>65382872 </td><td>49532858 </td><td>0.966 </td><td>0.732 </td><td>-0.242 </td></tr><tr><td>4 </td><td>not_analyzed, except for 'agent' field which is analyzed </td><td>disabled </td><td>43083702 </td><td>32063602 </td><td>0.636 </td><td>0.474 </td><td>-0.255 </td></tr><tr><td>Semi-structured data file. Original file size:&nbsp;75037027 </td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>1 </td><td>analyzed and not_analyzed&nbsp; </td><td>enabled </td><td>100478376 </td><td>82132782 </td><td>1.339 </td><td>1.094 </td><td>-0.182 </td></tr><tr><td>2 </td><td>analyzed and not_analyzed&nbsp; </td><td>disabled </td><td>75238480 </td><td>56911638 </td><td>1.002 </td><td>0.758 </td><td>-0.243 </td></tr><tr><td>3 </td><td>not_analyzed </td><td>disabled </td><td>71866672 </td><td>53553561 </td><td>0.957 </td><td>0.713 </td><td>-0.254 </td></tr><tr><td>3b </td><td>not_analyzed, except for 'message' field which is retained and analyzed </td><td>disabled </td><td>104638750 </td><td>83824398 </td><td>1.394 </td><td>1.117 </td><td>-0.198 </td></tr><tr><td>4 </td><td>not_analyzed, except for 'agent' field which is&nbsp;analyzed </td><td>disabled </td><td>72925624 </td><td>54603882 </td><td>0.971 </td><td>0.727 </td><td>-0.251</td></tr></tbody></table>With the standard LZ4-based compression, the indexed data size to raw data size ratio ranged from 0.575 to 1.394. After enabling DEFLATE-based compression using the best_compression index.codec option, the indexed data size to raw data size ratio range came down to 0.429 to 1.117. Enabling the best_compression option resulted in a 15.7% to 25.6% reduction in indexed data size depending on the test parameters.&nbsp;As you can see, the ratio of index size to raw data size can vary greatly based on your mapping configuration, what fields you decide to create/retain, and the characteristics of the data set itself. We encourage you to run similar tests yourself to determine what the data compression/expansion factor is for your data set and application requirements. <h2>Conclusion</h2>There were <a href="https://www.elastic.co/blog/elasticsearch-2-0-0-beta1-released" target="_blank">many amazing features</a> added to Elasticsearch 2.0 worth considering. As we’ve discussed, two of these new features in particular can reduce the hardware footprint required for an Elasticsearch cluster by 15-25% or more: 1) the addition of a best_compression option and 2) enabling doc_values by default. This allows us to get to compression ratios between 0.429 and 1.117.Being an open source company means a lot more to us than just the licensing model of our core products. It means being open about the capabilities (and limitations) of our products. Our hope is that these experiments encourage you to discover the truth about Elasticsearch’s capabilities with regards to storage requirements yourself and not just take our word (or anyone else’s) for it. We’d love to hear about the results of your own experiments using your own data and specific configuration or mapping settings that makes sense for your application. You can submit those config and data files or file issues at this Github repo: <a href="https://github.com/elastic/elk-index-size-tests" target="_blank">https://github.com/elastic/elk-index-size-tests</a>.

memory-computer-storage-elasticsearch-compression-full-bleed.jpg

memory-elasticsearch-compute-storage-computer-thumbnail.jpg

Part 2.0: The true story behind Elasticsearch storage requirements

blog-banner-generic-teal.png

The true story behind Elasticsearch storage requirements

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

peter-kim.jpg

Solutions Architect, Search Specialist

Peter Kim

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/elastic-cloud-account-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">ausgewählte Nachrichten und Informationen</a> an die oben von Ihnen bereitgestellten Kontaktangaben zu senden. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/serverless-preview-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Beim Absenden akzeptieren Sie die <a href="/de/agreements/global/cloud-services-monthly" rel="nofollow">Nutzungsbedingungen von Elastic Cloud</a>. Die Verarbeitung Ihrer personenbezogenen Daten unterliegt der <a href="/de/legal/privacy-statement">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/terms-of-use" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Der Inhalt dieser Seite ist in der ausgewählten Sprache nicht verfügbar. Wir bei Elastic arbeiten daran, die bereitgestellten Inhalte in verschiedenen Sprachen anzubieten. Bis dahin bitten wir Sie um etwas Geduld und hoffen auf Ihr Verständnis!

Author

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Sie können nicht dabei sein? Registrieren Sie sich, damit wir Ihnen die Aufzeichnung zukommen lassen können. Außerdem erhalten Sie eine E‑Mail mit Begleitmaterial.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Learn More

Read less

You'll also receive an email with related content

Wir schicken Ihnen zudem relevante Informationen, die von Interesse sein könnten.

Thank you for registering. We will send you a confirmation email soon.

Vielen Dank für Ihre Registrierung. Sie erhalten in Kürze eine Bestätigungs-E‑Mail von uns.

Author

Articles by Peter Kim

Solutions Architect, Search Specialist, Elastic

Advanced Search Queries in Elastic App Search

Part 2.0: The true story behind Elasticsearch storage requirements

The true story behind Elasticsearch storage requirements