Visitors to the Elastic{ON} 2018 Training booth had a chance to win a free Annual Online Training Subscription. See how we used Elasticsearch to pick a winner.

<html><head></head><body>The <a href="https://elastic.co/training">Elastic Training</a> team thanks everybody who came to see us at <a href="https://elastic.co/elasticon/conf/2018/sf">Elastic{ON} 2018</a> in San Francisco! Everyone that had their badge scanned at the Elastic Training booth was entered into the drawing for an <a href="https://www.elastic.co/training/online-annual-training-subscription">Online Annual Training Subscription</a>. As promised, we made a random selection to determine the winner after indexing the scanned badge data into Elasticsearch. Here&#x2019;s a quick breakdown of how we did it.
As with all development, we started with a workflow:
<ol>
	<li>Remove duplicate entries &#x2014; in case any of you tried to get sneaky and put multiple entries in there ;-)</li>
	<li>Filter out Elastic employees (we should know this stuff already, it&#x2019;s our job).</li>
	<li>Redact all Personally Identifiable Information (PII).</li>
	<li>Index the list of attendee emails as documents into Elasticsearch.</li>
	<li>Pick a lucky winner at random.</li>
</ol>To get started, we indexed the documents using the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docs-bulk.html">_bulk API</a>, making this a purely Elasticsearch solution. Then, for filtering, we first used the ticket ID to reconcile duplicates. Then we used the email domain to filter out elastic.co entries, routing those to a separate index. Finally, we redacted all PII.
For the domain extraction, we first needed to <a href="https://www.elastic.co/guide/en/elasticsearch/painless/6.2/painless-examples.html#modules-scripting-painless-regex">enable regex</a> in <code>elasticsearch.yml</code> and then restart the node:
<pre class="prettyprint"># permit regex
script.painless.regex.enabled: true
</pre>Then we added an <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/pipeline.html">ingest pipeline</a> which used <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/modules-scripting-painless.html">Painless</a> to:
<ul>
	<li>Set the document <code>_id</code> to the ticket ID (no PII)</li>
	<li>Extract the domain from the email address as a new field</li>
	<li>Route to index <code>employee_ineligible</code> or <code>lead_eligible</code> on the basis of domain</li>
	<li>Redact all potential PII (including the domain)</li>
</ul>Like this:
<pre class="prettyprint">PUT _ingest/pipeline/email_to_id_route_and_redact
{
 &quot;description&quot;: &quot;use Ticket_Reference_ID as _id to dedupe, route by domain and redact ALL potential PII&quot;,
 &quot;processors&quot;: [
 {
 &quot;script&quot;: {
 &quot;source&quot;: &quot;&quot;&quot;
// set document id to Ticket_Reference_ID
ctx._id = ctx.Ticket_Reference_ID;
// extract domain from email
def domain = /.*@/.matcher(ctx.Email).replaceAll(&apos;&apos;);
// if domain == elastic.co, route to ineligible index
ctx._index = (domain == &apos;elastic.co&apos;) ? 
 &apos;employee_ineligible&apos; : &apos;lead_eligible&apos;;
// get the keys (=document property names)
Set keys = new HashSet(ctx.keySet()); 
// iterate over the document properties these keys represent
Iterator properties = keys.iterator();
while (properties.hasNext()) {
 def property = properties.next();
 // extract the field prefix
 def prefix = property.substring(0,1);
 // redact fields not generated by ES, excepting Ticket_Reference_ID
 if(!(prefix.equals(&apos;_&apos;) || 
 prefix.equals(&apos;@&apos;) || 
 property.equals(&apos;Ticket_Reference_ID&apos;)))
 // set those fields to pii_redacted 
 ctx[property] = &apos;pii_redacted&apos;;
}
&quot;&quot;&quot;
 }
 }
 ]
}
</pre>As a best practice, we used the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/simulate-pipeline-api.html">_simulate API</a> to test our pipeline before running the full ingest, so we sent two test documents, from eligible and ineligible entries.
<pre class="prettyprint">POST _ingest/pipeline/email_to_id_route_and_redact/_simulate
{
 &quot;docs&quot; : [
 { &quot;_index&quot;: &quot;index&quot;,
 &quot;_type&quot;: &quot;doc&quot;,
 &quot;_source&quot;: { 
 &quot;field1&quot;:&quot;test&quot;, 
 &quot;field2&quot;:&quot;test&quot;, 
 &quot;Email&quot;:&quot;ineligible:-(@elastic.co&quot;, 
 &quot;Ticket_Reference_ID&quot;:&quot;ABCDE12345&quot;} },
 { 
 &quot;_index&quot;: &quot;index&quot;,
 &quot;_type&quot;: &quot;doc&quot;,
 &quot;_source&quot;: { 
 &quot;field1&quot;:&quot;test&quot;, 
 &quot;field2&quot;:&quot;test&quot;, 
 &quot;Email&quot;:&quot;eligible:-)@not_elastic.suffix&quot;, 
 &quot;Ticket_Reference_ID&quot;:&quot;FGHIJ67890&quot;} }
 ]
}
</pre>The output confirmed that the pipeline mapped the document <code>_id</code>, routed to the correct indices, and redacted all potential PII.
<pre class="prettyprint">{
 &quot;docs&quot;: [
 {
 &quot;doc&quot;: {
 &quot;_index&quot;: &quot;employee_ineligible&quot;,
 &quot;_type&quot;: &quot;doc&quot;,
 &quot;_id&quot;: &quot;ABCDE12345&quot;,
 &quot;_source&quot;: {
 &quot;field1&quot;: &quot;pii_redacted&quot;,
 &quot;Email&quot;: &quot;pii_redacted&quot;,
 &quot;field2&quot;: &quot;pii_redacted&quot;,
 &quot;Ticket_Reference_ID&quot;: &quot;ABCDE12345&quot;
 },
 &quot;_ingest&quot;: {
 &quot;timestamp&quot;: &quot;2018-04-19T16:24:13.071Z&quot;
 }
 }
 },
 {
 &quot;doc&quot;: {
 &quot;_index&quot;: &quot;lead_eligible&quot;,
 &quot;_type&quot;: &quot;doc&quot;,
 &quot;_id&quot;: &quot;FGHIJ67890&quot;,
 &quot;_source&quot;: {
 &quot;field1&quot;: &quot;pii_redacted&quot;,
 &quot;Email&quot;: &quot;pii_redacted&quot;,
 &quot;field2&quot;: &quot;pii_redacted&quot;,
 &quot;Ticket_Reference_ID&quot;: &quot;FGHIJ67890&quot;
 },
 &quot;_ingest&quot;: {
 &quot;timestamp&quot;: &quot;2018-04-19T16:24:13.071Z&quot;
 }
 }
 }
 ]
}
</pre>To avoid any inconsistency due to sharding or routing given the small data set, we added a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/dynamic-templates.html">dynamic template</a> to ensure single-sharded indexes:
<pre class="prettyprint"># clean up
DELETE *eligible*
PUT _template/single_shard
{
 &quot;index_patterns&quot; : &quot;*eligible&quot;,
 &quot;order&quot; : 1,
 &quot;settings&quot;: {
 &quot;number_of_shards&quot;: 1,
 &quot;number_of_replicas&quot;: 0
 }
}
</pre>And then Ingested the data using the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docs-bulk.html">_bulk API</a>:
<pre class="prettyprint">PUT _bulk
{ &quot;index&quot; : { &quot;_index&quot; : &quot;&quot;, &quot;_type&quot; : &quot;doc&quot;, &quot;pipeline&quot;: &quot;email_to_id_route_and_redact&quot;} }
{ &quot;Exhibitor_Name&quot;:&quot;Elastic&quot;, &quot;First_Name&quot;:&quot; ...
</pre>Next step was to count the eligible and ineligible candidates using a&#xA0;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/search-aggregations-bucket-terms-aggregation.html">terms aggregation</a>:
<pre class="prettyprint">GET *eligible/_search
{
 &quot;size&quot;: 0,
 &quot;aggs&quot; : {
 &quot;eligibility&quot; : {
 &quot;terms&quot; : { &quot;field&quot; : &quot;_index&quot; }
 }
 }
}
</pre>The results were:
<pre class="prettyprint">&quot;eligibility&quot;: 
{
 &quot;doc_count_error_upper_bound&quot;: 0,
 &quot;sum_other_doc_count&quot;: 0,
 &quot;buckets&quot;: [
 {
 &quot;key&quot;: &quot;lead_eligible&quot;,
 &quot;doc_count&quot;: 393
 },
 {
 &quot;key&quot;: &quot;employee_ineligible&quot;,
 &quot;doc_count&quot;: 9
 }
 ]
}
</pre>Finally, fairness and repeatability were ensured by sourcing the seed for the random score using a random number generated from terminal: <code>echo $RANDOM</code>. This returned the value <code>8121</code>.
To pick our lucky winner, we queried the documents using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/query-dsl-function-score-query.html">random_score function_score query</a>:
<pre class="prettyprint">GET lead_eligible/_search
{
 &quot;size&quot;: 1,
 &quot;_source&quot;: &quot;T*&quot;, 
 &quot;query&quot;: {
 &quot;function_score&quot;: {
 &quot;functions&quot;: [
 {
 &quot;random_score&quot;: {
 &quot;seed&quot;: 8121,
 &quot;field&quot;: &quot;_id&quot;
 }
 }
 ]
 }
 }
}
</pre>Which resulted in:
<pre class="prettyprint">&quot;hits&quot;: [
 {
 &quot;_index&quot;: &quot;lead_eligible&quot;,
 &quot;_type&quot;: &quot;doc&quot;,
 &quot;_id&quot;: &quot;FWNVN5VRB2L&quot;,
 &quot;_score&quot;: 0.9928309,
 &quot;_source&quot;: {
 &quot;Ticket_Type&quot;: &quot;pii_redacted&quot;,
 &quot;Transcription_Status&quot;: &quot;pii_redacted&quot;,
 &quot;Title&quot;: &quot;pii_redacted&quot;,
 &quot;Ticket_Reference_ID&quot;: &quot;FWNVN5VRB2L&quot;
 }
 }
 ]
</pre><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt042821875053d39f/5c5840ecdfaabbd40b9c2c48/and_the_winner_is.jpg" data-sys-asset-uid="blt042821875053d39f" alt="and_the_winner_is.jpg" style="display: block; margin: auto;">
Congratulations <code>FWNVN5VRB2L</code>, aka Stephen Steck! We look forward seeing you in a virtual classroom soon! 
For anyone interested in trying this out on their own, all of the above requests and some sample data can be found in <a href="https://gist.github.com/djp-search/73e3bd31b4157c16af82a17546195f6f">this Github Gist</a>.
</body></html>

training_banner.png

elasticon_2018_training_booth_thumb.jpg

And the Winner of the Elastic{ON} 2018 Training Subscription Drawing is...

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Education Architect

Dominic Page

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/elastic-cloud-account-terms" target="_self">服务条款</a>，并且您同意通过您提供的上述联系方式接收<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">所选的通信</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/serverless-preview-terms" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

提交即表示您同意 <a href="/cn/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 服务条款</a>。我们将会按照 <a href="/cn/legal/privacy-statement">Elastic 的隐私声明</a>处理您的个人数据。

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/terms-of-use" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

本页内容尚不支持所选语言。Elastic 正在不断努力，以实现对多种语言内容的支持。感谢您在此期间给予的耐心与陪伴！

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

查看本次网络研讨会在我的时区的开始时间

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

无法实时参加？注册后我们会给您发送录像。您还将收到相关内容的电子邮件。

Author

Articles by Dominic Page

Education Architect, Elastic

And the Winner of the Elastic{ON} 2018 Training Subscription Drawing is...