See how IST Research — a 2017 Elastic Cause honoree — has expanded their use of the Elasticsearch Service to help support global humanitarian efforts.

<a href="https://www.istresearch.com/">IST Research</a> uses technology and techniques to support humanitarian efforts and addresses global human security issues such as human and sex trafficking, violent extremism, and child soldier recruitment, and pushes new boundaries in understanding how information about these activities propagates in the online community. As an 2017 <a href="/elastic-search-awards">Elastic Cause</a> honoree, we continue to push this fight forward while evolving and expanding our use of the Elastic Stack.
Sometimes you need to look back from where you have come in order to make forward progress. In looking through the <a href="/elasticon/conf/2017/sf/closing-keynote">slide deck</a> we presented at Elastic{ON} 2017, we have made some fundamental and significant changes to our architecture, and have expanded our use of the Elastic Stack greatly in order to serve our flagship product, Pulse, to our range of federal and commercial customers.
<a href="https://www.istresearch.com/what-we-do/">Pulse</a> enables many different information channels to flow into a single unified platform. It allows users to engage directly with populations on the ground all over the world, while also understanding what that same population is saying online. At its core, all the data gathered by Pulse feeds into a number of different Elasticsearch clusters designed for different use cases.
For context, any one of our production instances can generate 500GB to 1TB of application data per day and over 100 million logs per day. The scale we need to run at, which is about three times larger than in 2017, is just one of the many challenges we face.
In this post I will walk you through the evolution of our Elastic implementation.
The diagram we used at <a href="/elasticon/">Elastic{ON}</a> 2017 during our award presentation looked something like this:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8a8df42782ac2503/5d1b2013e2b9b15368e39ef3/ElasticFocusArch.png" data-sys-asset-uid="blt8a8df42782ac2503" alt="IST Research data collection and ingestion pipeline" width="505" height="425" style="width: 505px; height: 425px; display: block; margin: auto;">
It shows a fairly straightforward data collection and ingestion pipeline, consisting of Elasticsearch, Logstash, Kibana, Curator, and percolator technologies being used to power <a href="https://www.istresearch.com/what-we-do/">Pulse</a>. We pumped all of our data into Elasticsearch, and integrated Kibana directly into our platform to allow our analysts to work with the raw data our enrichment pipeline produced.
This setup works great for serving dedicated customers and for using out of the box solutions. Over the past couple of years, however, we have continued to find new and novel use cases that have required us to expand our capabilities beyond a traditional stack. As seen in the diagram below, we’ve greatly expanded our use of the Elastic product offering, both complicating and empowering our own product growth as well.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt71ba81f82eb3f130/5d1b204ca3e03652eee9ff9a/Elastic_Blog_Diagram_2019-05(5).png" data-sys-asset-uid="blt71ba81f82eb3f130" alt="IST Research Elastic Stack implementation" width="740" height="353" style="width: 740px; height: 353px; display: block; margin: auto;">
The first thing to note is our use of <a href="/products/elasticsearch/service">Elasticsearch Service</a> on <a href="/cloud/">Elastic Cloud</a>, as we have transitioned completely away from running our own Elasticsearch clusters. With that comes the ability to manage the style, size, and capabilities of our cluster without having to worry about operations. With that, we also get the graces of the Elastic Support team and automatic index snapshots in the event of a self-induced problem.
The second notable thing is the segmentation and specialization of our Elasticsearch clusters. We found that tuning one cluster to meet all of our various demands was quite difficult, but using a segmented approach allows us to tune each cluster individually to meet the performance and scale desired. Both our <a href="/guide/en/elasticsearch/reference/7.1/query-dsl-percolate-query.html">Percolator</a> cluster for reverse querying and our <a href="https://pelias.io/">Pelias</a> cluster for reverse geocoding are largely static, enabling us to dial up search and query performance. Our data clusters use the newly released <a href="/guide/en/elasticsearch/reference/7.1/modules-cross-cluster-search.html">cross-cluster search</a>, allowing us to tune each cluster to meet our desired customer’s data retention, search analyzer, and performance needs, while still enabling us to conduct cross-cluster analysis across our entire platform.
To allow us to get better insight into our application, we installed Logstash everywhere and use a dedicated logging cluster. It has a short retention window of up to a week’s worth of log data, but can ingest large volumes of logs very quickly. Capping off these components with cluster management in <a href="https://github.com/lmenezes/cerebro">Cerebro</a> and data management via <a href="https://github.com/elastic/curator">Curator</a>, we get a large window into how our application and clusters are performing at each stage of the process.
The next big item in this architecture is our multi-instance Kibana setup. We have an open source <a href="https://github.com/istresearch/kibana-object-format">Kibana plugin</a> that we pair with our use of dashboards, <a href="/guide/en/kibana/7.1/canvas.html">Canvas</a>, and <a href="/guide/en/kibana/7.1/timelion.html">Timelion</a> to give our analysts a wide variety of ways to view our data. This architecture was actually created before <a href="/guide/en/kibana/7.2/xpack-spaces.html">Kibana Spaces</a> was released, as something of a workaround for our need to utilize access control, but continue to enable users with siloed Kibana instances. We are working on migrating our product to Spaces by year’s end, and we just purchased eight days of consulting to further integrate our product with the Elastic Stack.
Lastly, to create a consistent user interface experience, we also worked with the Elastic team to integrate the <a href="/elastic-maps-service">Elastic Maps Service</a> as our dedicated tile map server. This allows our customers to interact with maps that help them figure out where they are interested in collecting data, and then seamlessly transition over to the same experience to view the resulting data in Kibana.
It’s amazing to look at the evolution of our product stack that has come about just in the past two years, and we continue to look forward to new technologies on the Elastic horizon. We have a number of plans on our docket for the future, including:
<ul>
	<li dir="ltr">Upgrading all clusters to 6.7 and 7.0</li>
	<li dir="ltr">Utilizing Kibana Spaces to improve our Kibana RBAC experience</li>
	<li dir="ltr">Taking advantage of the newly released <a href="/guide/en/kibana/7.2/maps.html">Maps</a> app</li>
	<li dir="ltr">Revamping our application monitoring architecture to take advantage of <a href="/guide/en/kibana/7.2/xpack-apm.html">Elastic APM</a> and the <a href="/guide/en/kibana/7.2/xpack-infra.html">Infrastructure</a> app</li>
	<li dir="ltr">Integrating with <a href="/cloud/app-search-service">Elastic App Search</a> to make our user search experience even easier</li>
</ul>These capabilities and many more are already available to the community; we just need the time to integrate them! I can’t wait to see what the Elastic Engineers and the open source community have in store in the next few years.
As the Elastic Stack continues to progress, it enables <a href="https://www.istresearch.com/what-we-do/">IST Research’s Pulse Platform </a>to expand the boundaries of what is possible for the work we perform. By utilizing so many different features and components, it allows our engineers to continue to create new product capability in parallel.
I hope our use of the Elastic Stack and architecture design outlined above has given you new insight and additional things to think about when designing your own application at scale using the many different offerings included in the Elastic toolbox.
<table style="border: 0px; border-collapse: collapse;">
<tbody>
<tr>
	<td style="border: 0px; border-collapse: collapse; width: 30%; vertical-align: middle;"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9a5d5388766b4dae/5d1b21005f304656c2ba860b/Madison.jpeg" data-sys-asset-uid="blt9a5d5388766b4dae" alt="Photo of Madison Bahmer" width="222" height="222" style="width: 222px; height: 222px; margin: 0px; float: left;">
	</td>
	<td style="border: 0px; border-collapse: collapse; width: 70%; vertical-align: middle;"><a href="https://www.linkedin.com/in/madison-bahmer-39726965/">Madison Bahmer</a> is the Chief Technology Officer at <a href="https://www.istresearch.com/">IST Research</a>, a company focused on human security issues founded in 2008. He has been with the company for over six years, and enjoys creating artwork in his spare time.
	</td>
</tr>
</tbody>
</table>

ist-research-fullbleed.jpg

ist-research-thumbnail.jpg

IST Research uses Elastic Stack to aid humanitarian efforts globally

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-madison-bahmer-300x300.jpg

Madison Bahmer, IST Research

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

제출하면 귀하가 <a href="/kr/legal/elastic-cloud-account-terms" target="_self">Elastic 서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 연락처로 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">선택한 커뮤니케이션</a>을 수신하는 데 동의한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/serverless-preview-terms" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

제출하면 <a href="/kr/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 서비스 약관</a>에 동의하시게 됩니다. 개인 데이터는 <a href="/kr/legal/privacy-statement">Elastic의 개인정보 취급방침</a>에 따라 처리됩니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/terms-of-use" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

이 페이지의 콘텐츠는 선택하신 언어로 제공되지 않습니다. Elastic은 다양한 언어로 콘텐츠를 제공하기 위해 최선을 다하고 있습니다.조금만 더 기다려주세요!

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

내 표준 시간대로 웨비나 시작 시간 확인하기

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

참여가 어려우신가요? 등록하시면 녹화 자료를 보내드리겠습니다. 이메일로 관련 콘텐츠도 받아보실 수 있습니다.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

You'll also receive an email with related content

이메일을 통해 관련 내용을 보내드리겠습니다.

Thank you for registering. We will send you a confirmation email soon.

등록해 주셔서 감사합니다. 곧 확인 이메일을 보내드리겠습니다.

Author

Articles by Madison Bahmer

CTO, IST Research

IST Research uses Elastic Stack to aid humanitarian efforts globally