Activating the new Intelligence Community data strategy with Elastic as a unified foundation


The US Office of the Director of National Intelligence (ODNI) recently released its data strategy for 2023–2025. The document stresses the need for all 18 Intelligence Community (IC) elements to adopt common services, increase data interoperability and discoverability, and prepare teams and technology for artificial intelligence (AI).

This strategy follows in the (long) wake of ODNI’s 2017 data strategy. In the six years since the publication of that document, IC leaders say that data volume has become an increasing challenge, as well as the need for advanced data capabilities such as automation. According to a recent ThoughtLab/Elastic® study, public sector leaders estimate that their data volume will increase by 59% over the next three years, so this challenge is worth strategically solving for the long term.

What’s included in the 2023–2025 IC data strategy?

The mission of the IC data strategy is to “enable secure discovery, access, and use of IC data for mission value and insight at speed and scale.” Supporting this goal, the document highlights four strategic focus areas for agencies to prioritize and act on over the next few years:

  • Performing end-to-end data management
  • Delivering data interoperability and analytics at speed and scale
  • Advancing all partnerships for continued digital and data innovation
  • Transforming the IC workforce to be data-driven

Data as a strategic asset in IC mission success

As government agencies and private sector companies drastically innovate and transform their digital operations, data is an underlying foundation of success and should be viewed as a strategic asset. The challenge, though — in addition to the increasing data volume as mentioned above — is contending with data silos, tool sprawl, correlating different data types, and sharing data across teams and organizations. 

According to the Elastic/ThoughtLab study, half of public sector leaders surveyed say that availability of needed data is a top challenge at their organization, and 45% of public sector organizations say they are struggling to manage the volume of data they collect.

The authors of the IC strategy recognize the importance of data as a fundamental determinant of IC mission success in a dynamic competitive and technical environment:

Putting data at the center of an agency’s strategy (not just IT strategy) ensures that people, technology, and processes all have a single knowledge repository, a shared foundation for skills collaboration, and the ability to find the right answers at the right time leveraging state-of-the art technology like AI.

Why data management is essential for AI use cases

Artificial intelligence is already shaping the intelligence community globally. As a whole, public sector is still early in AI adoption: 80% of government organizations are at the initial or developing digital maturity stages. As adoption ramps up, agencies need to prioritize teeling up their data strategy and data management systems to be ready to take advantage of AI capabilities.

For the IC, the value of generative AI comes when organizations integrate their own internal data with large language models (LLMs). Publicly accessible AI platforms like ChatGPT are trained on publicly available data, so they’ll only be as relevant as the information that everyone else can access. But when LLMs can securely leverage an agency’s proprietary internal data, generative AI becomes infinitely more accurate, actionable, and mission-specific.

This means that in order to get the most value out of generative AI outputs, it’s key to have a unified data platform that allows you to search and access all your internal data. Having this unified data mesh in one place avoids data silos and data duplication, making it easier to quickly and accurately use data to extract mission value from generative AI.

How Elastic can help the IC align to the new data strategy

Elastic enables the IC to leverage enriched, secure data as a force multiplier to maximize situational awareness and safeguard national security interests through a dynamic, scalable platform. 

Specifically, Elastic offers tangible benefits in each of the four action areas as outlined in the IC strategy:

Performing end-to-end data management

The data strategy states that IC entities need to “Create end-to-end data management plans for the collection and acquisition of all data, to enable and reduce the time of secure data flow from collection to actionable insight.”

Elastic can ingest all forms of data — text, numbers, geospatial — no matter where that data is located or how it’s hosted. This can significantly streamline an agency’s end-to-end data management by serving as a single point of data collection.

Within seconds of data being ingested from sensors and systems, Elastic can normalize and index all data in optimized ways to allow for fast query analytics that don’t alter the  native characteristics of the data. This means that data is normalized for unified querying and analytics. 

Once the data has been ingested, Elastic can correlate and analyze all the data without having to copy and move large amounts of data (likely terabytes, maybe even petabytes) to a central location for access and analysis. Without this need for centralization, agencies can save time and bandwidth and ensure data accuracy throughout an end-to-end data lifecycle.

Delivering data interoperability and analytics at speed and scale

Elastic solves the data interoperability challenge by serving as a unifying “data mesh” that provides a single source of truth for every use case and each level of the organization.

As a data mesh layer for IC agencies, Elastic can not only collect, but also fuse, a growing amount of disparate data sets with minimal latency and highest security.

Just as Elastic can ingest all types of data, it can also correlate and analyze all that data without having to move it to a central location. This functionality, known as Cross-Cluster Search, saves time, reduces error, and makes sure that data stays in the control of the individual or team most knowledgeable about it. As a result of this data mesh approach, all data — no matter where it resides or what format it’s in — can feed into Elastic’s AI and ML-powered insight tools.

Advancing all partnerships for continued digital and data innovation

The strategy writers state that, “based on the speed of data and digital innovation and the changing and increasing national security attack surface, direct partnerships with the private sector and academia are more critical for the IC now than ever before.”

Elastic can facilitate collaboration with non-IC organizations through the open, flexible, accessible approach that forms the core of our company’s mission and values. 

Because of this collaborative approach, Elastic gives agencies the freedom to share code and architecture with other projects and systems when needed. Unlike the walled gardens of some proprietary systems, data is stored in non-proprietary formats, meaning both data and analytic methods are transferable within an enterprise architecture or to cross-agency initiatives. 

Data sharing does not come at the expense of security, however. By leveraging role-based access control (RBAC) within Elastic, an administrator can create secure dynamic data access that allows each role to have its own view of only the data that’s relevant or permitted to them.

Transforming the IC workforce to be data-driven

When an agency uses a unified data platform and data mesh, the entire workforce can align around a single tool set and data store, creating a reliable single source of truth. 

In alignment with Elastic’s free and open foundation, non-engineers can easily access mission intelligence data and drag-and-drop data visualization capabilities that leverage link analysis, AI, and ML — instead of access being limited to staff with specialized technical expertise. As a result, distributed teams can quickly come together around the same data and insights when making real-time decisions. As stated in the IC data strategy, “Data acumen has to become a core skill for every member of the workforce — not just data professionals.” 

With a unified data platform at the heart of agency strategy, there are fewer tools to learn, maintain, and stay up to date on. This leaves time for individuals to expand technical skills in specific areas or learn new skills as aligned with their professional growth plans or mission needs.

Talk to an IC expert about your data strategy

Reach out to an Elastic IC expert to discuss how we can help you align to the new IC data strategy. At Elastic, we have a proven track record of helping government and public sector customers find answers from all data at scale. Read this blog for some of our recent thinking on related topics.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use. 

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.