From dropped logs to continuous detection: How a major US city government scaled Elastic Security across 12,000+ agents and automated SOC response

After its prior SIEM began dropping log data it could not ingest fast enough, one of the largest city governments in the United States rebuilt its security operations on Elastic Security, bringing 12,000+ agents under continuous monitoring and collapsing multistep manual response into one-click automated actions.

A major US metropolitan government protects the digital services that hundreds of thousands of residents rely on, and its security operations center (SOC) monitors every part of that estate. Its prior SIEM could not keep pace with the environment's log volume, dropping data and leaving gaps that made reliable detection impossible. After migrating to Elastic Security on Elastic Cloud, the organization brought 12,000+ agents, roughly 1,000 network devices, and dozens of applications under continuous monitoring, all normalized to a common schema across cloud, network, and identity sources. Analysts now work pre-enriched cases where credential resets, host isolation, and firewall blocks execute in a single click, and the Elastic AI Agent surfaces system issues before they become outages or gaps in logging.

A large US metropolitan government runs the kind of infrastructure most private-sector organizations never have to think about: thousands of endpoints and servers across dozens of agencies, hundreds of network devices, multiple cloud environments, and a constellation of applications that keep essential city services running for residents. The security operations team is responsible for monitoring and protecting all of it, continuously, at whatever scale the city's operations demand.

For years the team carried that responsibility with a legacy SIEM at the center. It worked well enough at first. But as the city's digital footprint grew, a structural problem surfaced: The platform could not keep up with the volume of log data the environment generated. Logs were being dropped, and detection coverage had gaps the team could only partly see.

A SIEM with gaps is not really a SIEM. The team needed a platform that could ingest everything the city produced, reliably, and keep pace as the environment kept growing.

The organization had run a legacy SIEM as its primary detection platform before moving to Elastic. That platform's architecture was not built for the volume and variety a large metropolitan government produces: telemetry from network gear, cloud platforms, identity providers, custom syslog emitters, and a growing estate of cloud-hosted infrastructure. The result was not gradual degradation but a capacity mismatch. Events that were not ingested could not be detected, and cases could not be built on data that was never retained.

Case management already lived outside the SIEM. The team had long run an external case and automation layer rather than the built-in alert management of any single detection tool, so the workflow model itself was familiar. What had to change was the foundation underneath it:

• a platform that could ingest at the city's actual log volume without dropping data
• normalize a heterogeneous source catalog to one schema
• support detection content the team could build and own
• connect cleanly to the automation layer through an API

Elastic Security met all four.

Two constraints defined the day-to-day before the migration.

First, coverage was incomplete by definition. Logs the prior platform dropped were simply unavailable for investigation, so a detection was only ever as good as the fraction of data that survived ingestion.

Second, response was manual and varied from analyst to analyst. When an alert warranted action, analysts executed the steps by hand: resetting passwords, isolating hosts, revoking tokens, creating firewall blocks. None of it was standardized. Each analyst developed their own habits, and documentation was inconsistent between shifts.

The current Elastic Security deployment runs on Elastic Cloud with premium support, ingesting a broad, heterogeneous source catalog into a single pipeline. Every source normalizes to the Elastic Common Schema (ECS) with consistent timestamps, which is a meaningful engineering effort across inputs that span network devices, cloud APIs, identity providers, and bespoke syslog. Kibana is the analyst surface for detection review, investigation, dashboarding, and reporting.

Detections fire from 170+ rules, a mix of out-of-the-box content from Elastic Security Labs and custom rules the team wrote for its own environment. Alerts route automatically into Tines through an API, where enrichment runs before an analyst ever opens a case. When a case closes in Tines, that closure feeds back to Elastic automatically through the same API, so alert state stays in sync without manual reconciliation.

Technical highlights

• Elastic Cloud for production Security workloads, backed by premium support
• 12,000+ Elastic Agents collecting endpoint and server telemetry
• ~1,000 network devices ingested alongside cloud and identity sources
• ECS normalization with consistent timestamps across all source types
• 170+ active detection rules, custom plus out-of-the-box content from Elastic Security Labs
• Tines integration over API: automated enrichment, one-click response actions, and closed-loop case-status sync back to Elastic
• Change Management Database (CMDB) data indexed in Elasticsearch for security-coverage reporting
• Elastic AI Agent used for log analysis, ad hoc filter generation, and proactive anomaly detection
• Fleet for agent lifecycle management across the endpoint and server estate
• Elastic Observability collecting system performance metrics for servers and network devices

Continuous ingestion without dropped data

The first thing the migration fixed was the foundational problem: keeping pace with the log volume the city generates. Where the prior platform shed data under load, Elastic Security ingests the full stream across agents, network devices, cloud sources, and custom syslog, and ECS normalization lands all of it in a consistent schema with accurate timestamps. That continuous, complete coverage is the prerequisite for everything downstream: Rules fire on whole data, investigations run on clean timelines, and enriched cases arrive actionable rather than partial.

One-click response through Tines automation

The clearest change in the analyst's day is at the response layer. Alerts from the 170+ detection rules route into Tines automatically, and enrichment runs before a case is opened: indicator lookups completed, context populated. The analyst reviews a prebuilt case instead of assembling one and then acts on it directly. From that case, response actions run with a single click:

• Credential reset: Randomize a compromised user's password.
• Token revocation: Flush active user tokens to end unauthorized sessions.
• Host isolation: Quarantine an endpoint from the network.
• Firewall blocking: Create a network block for a flagged address.
• Automatic documentation: Every action writes back to the case record.

The variation in how individual analysts handled routine response once introduced both risk and rework. Standardizing those actions behind a single click, with documentation written automatically, removed both.

Coverage-gap visibility from CMDB data in Elasticsearch

Beyond detection and response, the team used the same platform to find blind spots in the security stack itself. The city's Change Management Database feeds into Elasticsearch, where reports flag systems missing critical security tooling: endpoints without an EDR agent, hosts without remote logging, systems without backup coverage. The result is a continuously updated, query-driven map of where coverage is thin, replacing the manual audits or spreadsheets that tracking would otherwise require. In a government estate where systems are added, decommissioned, and reconfigured at scale, that programmatic visibility is a standing capability rather than a periodic project.

Proactive issue detection with Elastic AI Agent

The team uses the Elastic AI Agent in live operations: analyzing unfamiliar log sources, building working Kibana filters without first documenting a schema, and surfacing anomalies early. When a new source comes online, an analyst can point the AI Agent at it, generate filters, and start building coverage immediately instead of waiting to reverse-engineer the format by hand.

“Elastic and the use of the AI Agent have been able to catch system issues before they become outages or cause gaps in logging. The AI Agent has pointed out problems and given solutions that were actionable.” — A senior member of the city's security operations team

The judgment stays with the analyst: The AI Agent surfaces the problem and proposes an action, and a person decides whether to take it. Problems the team would once have discovered only as an incident or a logging outage now get caught earlier in the cycle.

The operating-model shift is a shift in what analysts decide versus what they execute. Before, context was assembled by hand and each response step was run individually, with no standard workflow and inconsistent documentation. Now, context arrives preassembled in the case, response runs in a click, and every action is logged automatically. The human judgment, what to do about a compromised credential, a flagged host, a suspicious network event, stays with the analyst, who reviews the prebuilt case, decides, and approves. What the platform absorbed is the assembly and execution work around that judgment. In practice, the team reports the SOC has become more consistent and efficient, and analysts are freed to spend time on other projects and training rather than working through alerts by hand.

With detection continuous and response automated, the team is extending the same data layer further into operations. Elastic Observability already collects system performance metrics for servers and network devices, and the team has used that data to find bottlenecks, right-size virtual machines, and troubleshoot network outages. The next step is deepening that footprint, so the agents already deployed for security telemetry surface performance and availability data for the same infrastructure, without a separate rollout, platform, or budget line.

For a government organization managing cost and complexity across a large, heterogeneous estate, one unified data platform is an operational and budget argument as much as a technical one. The team began with a security problem, a platform dropping data and leaving gaps in coverage, and the foundation they chose to solve it, one that could ingest everything, normalize it consistently, and connect directly to automation, turned out to be the same foundation cross-domain operations run on.

Your organization may not be managing 12,000+ agents today, but the same principles apply whether you are starting with a few hundred endpoints or scaling to tens of thousands.

• Elastic Security
• Elastic Security SIEM
• AI for the SOC
• Elasticsearch
• Elastic Observability
• Elastic Security Labs
• Elastic Cloud

Topics: SIEM migration, Elastic Security, automated SOC response, Tines integration, ECS normalization, log ingestion, Elastic AI Agent, CMDB security coverage, Kibana, public sector security, government cybersecurity, endpoint detection, cloud security

See how Elastic Security gives public sector teams continuous detection and one-click automated response at city scale, or start now with a free trial.