OpenCrowd

Breaking down the silos

OpenCrowd is a design and technology services company that helps clients build custom applications leveraging cutting-edge technologies like Elasticsearch. Within their roster of clients, OpenCrowd has been working with several of the top global investment banks.

"We have worked with several investment banks that share a common problem of having to support legacy applications on disparate but related data," says Brooke Boyd, Business Development Manager for OpenCrowd.

"One of the most noticeable issues affecting the usability of their systems was performance. With one of our clients, wait times could be so long that the user could literally leave the building for coffee, and the query would still be processing when he came back. Often those queries would time out."

The main reason for the slow performance typically resulted from complex queries against a mix of traditional databases from multiple vendors consisting of structured and unstructured data. For one particular client seeking to build a global credit risk management system, OpenCrowd's goal was to consolidate up to 30 applications and databases into a common architecture and create a combined single application that streamlined the workflow.

Reducing query response from minutes to sub-second

For this client, OpenCrowd chose Elasticsearch as the core data layer component. The new system is comprised of various services that provide data to a front end user interface. Elasticsearch serves as the hub of a hub-and-spoke architecture. The majority of the data is fed into Elasticsearch, and then various systems access Elasticsearch to get at the data.

"The advantage we gain with Elasticsearch is that response time is much lower than the original system," explains John Bramlett, Chief Architect at OpenCrowd.

"In the previous system, it could take a minute or two for query results to come back. We are getting sub-second response times with Elasticsearch."

"With traditional databases, queries sometimes take a long time to produce results," adds David Feng, OpenCrowd Engagement Manager. "If your query has four or five joins on very large data sets, you could see a significant drag in query performance. Elasticsearch has eliminated that problem altogether."

Scale was another additional benefit of this new architecture. Leveraging Elasticsearch to build the data infrastructure has made it easier to linearly scale as new data stores are added in the future.

Making search smarter and improving workflow

Performance wasn't the only issue that OpenCrowd's client had. Users had to search for data in multiple applications and were required to be very prescriptive in how the data should be found.

"Previously, users would have to hunt and peck in one system, and then go to another system, and so on," Feng confirms. "Now, Elasticsearch helps drive the workflow, providing instant discovery."

"They had to use identifiers that were nonsensical," Bramlett explains. "Some people actually kept a hard copy list of unique identifiers in a spreadsheet. With Elasticsearch, we are able to let the users speak in terms they are familiar with. Because of Elasticsearch, they are able to find information very quickly using their own terminology. That is a very compelling benefit for our client because it allows them to move away from cryptic identifiers into a more natural language for search."

For example, in the previous system, when searching for a company, the user would have to know if the name included the term 'company' or 'companies'.

"In Elasticsearch, it does not matter," Bramlett notes. "The synonym feature in Elasticsearch allows for free form queries, which requires less precision on the part of the user and programmers no longer have to defensively program. Traditional databases cannot do any of this. This is a very helpful capability that increases our client's productivity. Through these types of features, Elasticsearch reduces search time from minutes to sub-seconds."

In addition, Elasticsearch Full Text Search enables multi-language support, context aware fuzzy search, auto complete and powerful scripting to customize relevancy scores and enrich data.

"Now that we have Elasticsearch, our client is able to create dynamic portfolios that allow them to slice and dice information in different ways," Bramlett adds. "Being able to make these very rich criteria-based selections is one of the most powerful advantages of Elasticsearch."

Consolidating applications and reducing cost

Today, companies are looking to cut costs anywhere they can, and IT is a focal point. Multiple applications require time, people and money to support them, so large enterprise organizations have a strong incentive to consolidate applications and decommission redundant legacy tools. OpenCrowd has been able to help clients accomplish this, and Elasticsearch helps make it possible.

"Elasticsearch provides a strong foundation for allowing the team to consolidate multiple applications into one," says Feng. " Elasticsearch allows us to unify the data that is used by these applications."

"The cost of maintaining multiple databases can be very high, because they require database administrators to set up and maintain them," Bramlett concludes.

"Elasticsearch has a much lower cost of ownership because a developer can easily set up a cluster. It is very simple to get Elasticsearch up and running. So the cost of putting Elasticsearch into your environment is relatively low. In addition, we've built a REST-based service on top of Elasticsearch to account for specific client customization requirements around data integrity, security, data masking and scalability."