Responsible transformation: Agentic AI for the public sector

Here’s the question: Can AI agents be safely deployed in the public sector? The short answer is yes — but only if responsibility and governance are intentionally built into how they operate.

Data is the foundation of AI. But in the public sector, data is highly sensitive, and a breach can carry national security implications. That makes data security the single biggest barrier to adoption. While these concerns understandably create hesitation, they exist alongside growing anticipation about what AI can unlock.

When deployed responsibly, agentic AI can modernize traditional IT workflows, streamline internal processes, and help organizations operate more efficiently. AI agents work autonomously to operate with minimal human interaction.

In the public sector, agentic AI can enhance citizen services, strengthen transparency, and help close the trust gap between institutions and the public, while also improving the daily experience of the civil servants who deliver those services.

A Dutch defense organization — the DATA department of the Materiel and IT Command (COMMIT) — offers a compelling example. It developed an in-house, air-gapped large language model (LLM) operating entirely on a closed network. By isolating the system from the internet, it ensured sensitive information remained secure. 

There is no escaping AI. Generative AI (GenAI) tools are widely accessible to private citizens, and employees are already experimenting with them in their daily lives. As a result, public sector organizations cannot simply opt out.

Unrestricted access to public agentic AI tools can jeopardize highly sensitive citizen data. Without proper oversight, organizations expose themselves to privacy violations, compliance failures, and data leaks.

Complete lockdown often leads to shadow AI agents. Employees turn to unsanctioned tools without oversight — increasing the risk of data leakage, only now without visibility or governance.

The solution lies in making a deliberate, strategic choice about where your organization should sit on the AI access spectrum, based on its specific risk profile and tolerance.

Secure-by-design infrastructure is key:

• Air-gapped or locally hosted LLMs where necessary

• Controlled deployment pipelines

• Clear governance defining who can use which tools and how

Building on open standards further strengthens long-term resilience, prevents vendor lock-in, and future-proofs your solution.

Keep in mind: Future-proofing is both technical and organizational. Technology that cannot be understood, managed, or improved internally creates dependency on external vendors.

Building an internal hub of AI knowledge and expertise ensures that your organization develops the capability to evaluate tools critically, implement them responsibly, and continuously optimize their use. By investing in user education and internal expertise, you reduce reliance on external providers and create long-term resilience.

Implementing AI agents in the public sector hinges on key challenges: data security, controlled access to information, and a mission-critical need for relevance and accuracy. This is where retrieval augmented generation (RAG) comes in. 

A typical interaction with a GenAI model is susceptible to sensitive data exposure and can often produce hallucinations because the model relies on outdated internet data. By air-gapping your model and adding your knowledge base into the loop, you can control what data the model uses to produce outputs, consolidating access and enabling traceability in AI outputs.

In other words, RAG grounds agentic AI responses in an organization’s own verified knowledge base, allowing models to generate answers based only on approved internal data. 

But implementing RAG introduces a new challenge: Effective RAG deployment requires accessibility to data, and in most public sector organizations, that data is highly fragmented. Structured databases, NoSQL systems, and vast amounts of unstructured documents are scattered across the ecosystem. 

A data mesh approach can help connect these distributed datasets, enabling them to function as a unified knowledge layer for AI applications. The result is a secure, intelligent search and assistant capability — one that can power advanced use cases such as AI-driven search solutions or even attack discovery within a controlled RAG environment without compromising data sovereignty.

Agentic AI in government is possible, but it must be thoughtful and focused. To enable agentic AI in the public sector, you need to build a tooling ecosystem. Then, the LLM decides which tools to use for which answers. This can get quite complicated.

Intelligent integration requires that you:

• Find solutions that integrate with your current systems. Stand-alone systems that don’t integrate end up causing more issues. 

• Consider AI maturity when investigating products. Don’t get caught in the hype. 

• Integrate gradually. Small, specific deployments with regular checks against your success criteria ensure that you keep control.

• Keep humans in the loop. This is a matter of observability, traceability, and accountability. 

Sign up to watch the Putting the responsible back into RAG and agentic AI webinar on demand and dive deeper into AI adoption strategies, best practices, and use cases. Or contact us.