Evolution of Workplace Search: Search your private data with Elasticsearch
Workplace Search functionality will merge with Elastic Search in the future. Here’s what you need to know.
Recent advancements in generative AI technologies have opened up a wave of possibilities with search. As developers build new experiences, users are adopting new ways of using search — from search queries written in natural language, to searching by uploading images or voice samples. At Elastic®, we have been heads-down to make sure Elasticsearch® has the right machine learning tools developers need to build modern search.
As we continue on this journey, we want to share some developments about Workplace Search:
- Elasticsearch is the recommended tool to build search experiences for internal knowledge search use cases going forward.
- The standalone Workplace Search product will continue to be supported in its current form. New features will be added to Elasticsearch, Workplace Search will receive security upgrades and fixes.
If you are working with the standalone Workplace Search product or are interested in building search experiences for workplace search use cases, this blog includes important information for you.
The standalone Workplace Search product became part of our offering when Swiftype joined forces with Elastic in 2017. Since then, we have added a number of content sources to popular databases, file systems, and other tools to Workplace Search. While users loved that Workplace Search included integrations for various data sources, there was clear feedback that developers wanted the transparency and flexibility to tune the search experience. These tools for tuning were only available when working directly with the Elasticsearch platform.
Internal knowledge search use cases (i.e., being able to search all your private content sources—databases, knowledge bases, file systems, collaboration, or ticketing tools) have increased in importance to our users with the recent popularity of ChatGPT and generative AI technologies. There is no doubt that search retrieval for private data and LLMs are better together.
Elasticsearch connectors are the future
Customers love using Workplace Search content sources
Customers that used the Workplace Search product loved the simplicity of ingesting data through content sources, using the indexed data in search experiences, and managing data and searches with built-in tools. These features are not going away but will evolve as part of our Elastic Search offering — some of this work began in early 8.x releases. Content sources have been replaced and elevated by connectors as a type of Elastic integration. These connectors write directly to search optimized Elasticsearch indices, instead of working with abstractions that use hidden indices. By doing so, the ability to utilize platform capabilities like integrating with third party transformer models to transform the indexed data is intuitive and easy to configure. As a result, setting up search experiences is more flexible than ever, and Elasticsearch native features provide multiple means of data and search management.
Typical deployment of Elastic connectors
The most common architecture for internal knowledge search use cases involves bringing private organizational data into Elastic through native connectors and/or self-managed connector clients. Then, a self-managed search application exposes data that your end users are authorized to see through a search experience.
Two options: native connectors on Elastic Cloud, or self-hosted clients
Connectors are available directly within your Elastic Cloud deployment as native connectors, where no additional infrastructure is required. If you prefer to self-host a native connector, these are available as connector clients to be easily deployed and self-managed on your own infrastructure with no development required. For this, you can choose to run connectors from source or deploy via Docker. Connector clients are open-code and can be customized to satisfy your specific needs. What’s more, you can leverage our connector framework to create new connector clients that integrate with your data sources that might not be covered by our current catalog.
Transform data to power your search
Synchronized content is indexed to regular search optimized Elasticsearch indices that can be used directly in your search experience. By default, connector specific logic and the bundled ent-search-generic-ingestion pipeline extract and transform your data, as configured in your deployment. For more advanced use cases, data can be filtered and transformed using tools like:
- Basic sync rules, which are identical for all connectors
- Advanced sync rules for remote filtering at the data source level
- Self-managed Extraction Service for connector clients
- Customized pipeline filtering, before persistence to Elasticsearch through ingest pipelines (Read about Ingest pipelines in Search for more details.)
- Ingest pipelines are especially powerful as they provide an extremely accessible integration point for ML models such as ELSER, for any private data indexed through connectors.
Building experiences with Elastic Search
Once you have your data in Elasticsearch, you can build your own custom search experience with projects like Search UI and SearchKit for frontend components. Our internal knowledge search example app shows how you can use existing tools to get started.
We're rapidly adding features to make search more powerful. Building your internal search systems with Elasticsearch gives you access to these capabilities as soon as they're released.
- Elasticsearch supports vector, semantic, hybrid, and generative AI search use cases. Check out some of our semantic and vector search capabilities, including a powerful out-of-the box sparse-vector model. In 8.11 this model is generally available and within a few clicks can be applied to data written using the Elastic connectors or web crawler.
- With a fully customizable Search Application API based on search templates, this new API abstracts away the learning curve of the Elasticsearch Query DSL, if that is required for your use case. You can dive into the details at any time with the Query DSL.
- New search management tools let you directly manage the results of specific queries, for example boost or bury results and work programmatically with the synonyms management API.
Workplace Search product and workplace search use case
The standalone Workplace Search product will continue to be a supported experience, so while migrating to the new features is recommended, it is not a requirement. Any user looking to elevate their internal knowledge search to include semantic search, vector search or chat prompt based experiences, would benefit from transitioning to use Elastic connectors. There are many compelling advancements for the range of workplace search use cases in the enterprise, supported by Elastic Search. Now would be a great time to think about upgrading your internal knowledge search to make it an amazing experience for your users!
Resources for you
- Example app to build a search experience for your internal knowledge sources
- Elastic Search and Workplace Search feature comparison (see table in our docs)
- NEW! Search Labs has code notebooks, engineering blogs, and sample projects on GitHub.
- Did you know that Elasticsearch is a vector database superset?
The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.
Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.