Guidestar by Candid: Powering the Search for a More Philanthropic World

Guidestar by Candid’s goal is to make the world a better place. They are the world’s largest source of nonprofit information, aggregating data from more than 2 million nonprofits to help more than 7 million donors and grantmakers make informed decisions around which nonprofits they support.

Guidestar by Candid initially relied on FAST ESP as its search engine, but once FAST became part of SharePoint in 2014, they knew they needed to make a change. They began by switching their APIs over to Elasticsearch and saw positive results, so when they decided to start moving their infrastructure to the cloud, they chose Elastic Cloud on AWS to continue the migration of their other search offerings off of FAST.

Once they made the switch to Elastic Cloud, not only did they see faster load times, an increase in user engagement, and experience major cost savings, but their engineers are also happier and more productive.

With data serving as the center of Guidestar by Candid's business, they need it to be readily available and easily accessible to their users. Early on, Guidestar by Candid concluded that fast and relevant search results were the gateway to connect users, donors, grantmakers, and nonprofit managers with the information they need to make confident decisions. It then became one of their top organizational priorities.

In 2008, they implemented FAST ESP as their enterprise search solution, which worked well until . . .

Microsoft, who acquired FAST ESP, announced they were going to stop supporting FAST as a standalone product and only offer it through SharePoint. FAST ESP also only supported XML, which Guidestar by Candid found to be too computationally-heavy for them to achieve the performance benchmarks they were looking for. And ultimately, after six years, it was time for Guidestar by Candid to refresh its software and data architecture.

Guidestar by Candid began researching alternatives that were open source, had native JSON support, were horizontally scalable, had user community engagement, and RESTful API endpoints for administrative tasks.

They started looking into Lucene. After playing around with it, they realized they needed more functionality, as well as support to ensure they were able to meet customer SLAs. They next looked at technologies that were built upon Lucene, and came across Elasticsearch.

Around the time Guidestar by Candid discovered Lucene and Elasticsearch, they were about to release the next generation of their APIs, which are used by more than 200 philanthropic websites and applications including American Express, BlackBaud, CrowdRise, JustGive, and NetworkForGood. Instead of ripping FAST out all at once, they thought this would be a good testbed to try a new search solution and see how things went.

Using Lucene required GuideStar's API users to construct complex search queries in order to produce relevant results. Guidestar by Candid believed that Elasticsearch could do a much better job without as much time being spent constructing search queries. In order to properly test this assumption, Guidestar by Candid decided to roll out a beta version of their Search API and test "one box" search functionality.

At this time, they also decided to purchase an Elastic subscription to help them architect the software the right way from the onset, as well as make sure they were able to adhere to their customer SLAs should anything go awry.

In January 2015, Guidestar by Candid released the beta of its Elasticsearch-powered Search API. The results were so overwhelmingly positive that GuideStar decided to pivot from their APIs and focus on the search capabilities of Guidestar.org.

By the middle of 2015, Guidestar by Candid was contemplating whether or not to renew its hardware leases and sign another contract with a co-location facility. At that time, the strategic decision was made to renew both contracts for just two years and start transitioning workloads to public cloud offerings, particularly AWS.

Shane Ward, Senior Director of Technology at Guidestar by Candid, attended Elastic{ON} 2015, Elastic's first user conference, and remembered hearing about Elastic Cloud, a hosted version of Elasticsearch that automates critical processes such as installation, configuration, maintenance, backup, and high-availability of Elasticsearch clusters. He contacted his Elastic account manager to see if they could give Elastic Cloud on AWS a spin. Elastic gladly provided Shane and his team with a free full production instance to test, and within a week, they were seeing fantastic results.

These results gave Shane and his team the confidence they needed to fully migrate from FAST to Elastic Cloud. Additionally, keeping their applications hosting consistent with their data sources and cloud platform meant Elastic Cloud on AWS was a great fit.

Shane and his team decided to start the migration to Elastic Cloud with two of their most popular search products: organizational search, which lets people find nonprofits by cause, and people search, which allows users to get information on nonprofit leaders and board members.

The transition itself was relatively easy due to the search abstraction layer they had in place with FAST. One of the first things they had to figure out was their cluster size – which they worked with their Elastic support engineer on: one master node, two data nodes, with two replicas for each index.

They also implemented X-Pack, which is part of their Elastic subscription, to secure their cluster with SSL encryption. This lets them prevent their data from being intercepted when communicating with the cluster in the cloud.

Their database administrator, Rich, who Guidestar by Candid refers to as the "grand poobah" of Elasticsearch, leveraged Elasticsearch's search templates to work on the product in isolation without impacting the live search application.

The product team at Guidestar by Candid used this opportunity as a chance to enhance the search UI. One of the big improvements they made was to preview how many results fit under certain filters, such as cause area, location, and financial information, by utilizing Elasticsearch's aggregations – which update in real-time. They didn't have this capability with FAST, and a user's search would often result in no matches as they couldn't get a feel for the results to expect without fully executing the search.

Before they went into production, user testing of the new search experience showed search results were a lot more relevant – and quick. Guidestar by Candid also did some automated load testing and found their Elastic Cloud cluster could handle 4x the load compared to when they were running FAST on-premise.

In August 2016, Guidestar by Candid went into production with Elastic Cloud on AWS and launched their new search products to the world.

Here's what Guidestar by Candid has experienced so far as a result of their move to Elastic Cloud:

4x Faster Load Times

Guidestar by Candid is seeing 2.5-4x faster load times with Elastic Cloud compared to FAST ESP.

Drastically Reduced IT Spend

When Guidestar by Candid used FAST ESP, they had 19 virtual machines dedicated to search, which was 18% of their entire co-located infrastructure. With Elastic Cloud, they are able to meet the same demand with a two-node cluster, which ultimately saves them hundreds of thousands of dollars in infrastructure, licensing, support, and labor costs. They're also no longer fighting for resources with other systems across the organization, like their internal CRM or Exchange email server.

34% Increase in User Engagement

Thanks to the real-time nature of Elasticsearch's aggregations, Guidestar by Candid enhanced the interactivity of their search filters and saw a 34% increase in user engagement. User engagement is a leading indicator of churn. If your customers aren't engaging with your product, they won't renew their subscriptions.

Leveraging Assistance From Elastic Support

Whenever they have a question, Guidestar by Candid doesn't hesitate to check in with Elastic Support – engaging with them on architecture guidance, validating ideas, and consulting on various other decisions they were making – all of which is possible due to Elastic's dedicated support model. Dedicated support means everyone at Guidestar by Candid interacts with the same Elastic support engineer each time they have a question or need help. After working together over a period of time, this engineer essentially becomes an extended part of a customer's team.

Upgrading is a Cinch

Elastic does fairly frequent product releases to get the latest features into users' hands as quickly as possible. In order to upgrade to the latest version of FAST, Guidestar by Candid had to stand up another cluster, test everything, and then schedule a night or weekend to switch everything over. With Elastic Cloud, upgrading is done with the single click of a button through the Cloud user console. This takes care of automatically spinning up a new cluster with the latest version of Elasticsearch, populating and configuring everything against it, replicating the data, and switching traffic over. No nights, no weekends. It's as simple as that.

Less Maintenance, More Time for Core Business (and Sleep!)

When using FAST, Guidestar by Candid had to engage with their clusters for management and performance monitoring on a daily basis. With Elastic Cloud, they check-in just once a month to make sure the cluster is healthy – they no longer have to spend time on hardware maintenance, or deal with alarms in the middle of the night to fix outages in their production environment.

Increased Agility

Guidestar by Candid is no longer scared to experiment as Elastic Cloud is less cumbersome than FAST and they know that nothing is going to fall over. When the product team wants to test a new set of features, Guidestar by Candid can stand up a brand new cluster and have it fully populated with the 16 million documents they have in 60-90 minutes. This would have taken a day and a half with FAST.

At the end of the day, Guidestar by Candid wants to work with Elastic because of shared values: we too want to make data more insightful. We too want to support our customers in the best way possible. We too embody the same passion for what we do, day in and day out. And now, we get to work together on achieving a shared goal, which feels just grand.

Learn more about Elastic Observability and our partnership with AWS. Get started today and try Elastic Observability on AWS for free.