A customer described it to me this way: A single transaction passes through three products on its way through the mainframe, and each one names the same field differently (system name, program name, user). Getting the data into Elastic isn't the hard part; getting it to correlate across products, so that Elastic's dashboards and ML jobs recognize it as the same data, is where most projects fall short.

Done right, Elastic becomes the speed layer that mainframe environments have never had: a near–real-time view across operational, transactional, and security data, while the authoritative systems of record stay exactly where they are.

This is the onboarding process I use with mainframe customers, built from architectures running in production at large financial institutions. It covers the ingest patterns that actually work, how Elastic Common Schema (ECS) alignment makes the data usable, and whether the data quality holds up or fails quietly.

Validate your mainframe source data before you write a pipeline

The most expensive failures I've seen in mainframe ingest projects are the ones that don't fail loudly. Pipelines run, data lands, dashboards render, and weeks later, someone notices that half the events from one logical partition (LPAR) never parsed or a quiet typing change has been silently corrupting a field.

Two mistakes come up frequently:

1. Format inconsistency across LPARs and time windows: Log formats vary across LPARs, between batch and online windows, and across shift changes. A format that parses cleanly in a dev LPAR may not match what production emits during peak batch. This is the single most common cause of partial parse failures I run into.

2. Sample configurations treated as production configurations: A common cause of "it broke overnight" incidents: The upstream collector configuration was based on a sample structure shipped by the vendor and then never replaced with a deliberate production configuration. When the vendor pushed an update, naming and typing changed (fields renamed, types shifted) and the downstream pipeline started rejecting records mid-flight. Treat sample configurations as exactly that, and replace them with a deliberate production configuration that doesn't move under you.

Before any pipeline development begins, walk through 24–48 hours of raw samples from each source with the mainframe team. This review should be treated as a recurring requirement rather than a one-off event, because the conditions that produce format drift (vendor updates, configuration changes, new message types) keep happening after the project goes live.

Worth knowing first: For mainframe environments, the Elastic integrations catalog is short. The IBM MQ integration is the most complete option (Queue Manager error logs and performance metrics, ECS-aligned, with out-of-the-box dashboards), though the metrics data stream requires the containerized MQ distribution rather than native z/OS MQ. If your architecture includes Customer Information Control System (CICS) workloads or you need end-to-end distributed tracing, assess IBM Z Observability Connect before building custom pipelines: It's the native OpenTelemetry (OTel) path and covers more ground than the architectures below. For everything else, read on.

ECS alignment: from data in Elastic to data Elastic can use

Before choosing an ingest strategy, it's worth understanding why ECS alignment comes first in practice, even if the pipeline gets built later. It's the decision that determines whether everything else pays off.

A mainframe team's core mission: Trace a single transaction from a REST call into z/OS Connect, through to an Information Management System (IMS) application, and back. That flow touches three products, each emitting telemetry with its own field names for the same concepts (system name, program name, user, transaction ID). Without normalization, correlating that transaction means writing queries that explicitly union three different field names per concept. That’s expensive to write and fragile when any product changes its schema.

ECS solves this. It defines a consistent target schema (host.name, process.name, user.name, event.code) that every source maps into. Once z/OS Connect, IMS Connect, and IMS data all land in the same ECS fields for the same logical concepts, that cross-product transaction trace becomes a single query.

There's a second reason this matters. Elastic's OOTB dashboards, alerting rules, anomaly detection, and ML jobs are all built against ECS field paths. A job_name field that Logstash extracted from a JES log is invisible to them. A process.name field carrying the same value is immediately recognized and processed. ECS alignment is what makes the platform's built-in capabilities recognize your data.

Skipping this step is the most common reason that ingest projects fall short, despite the data being technically present.

Map what fits to core ECS

The mapping below is a starting point drawn from what I've seen work across customer environments. Field names in your source data will vary, but the ECS targets are stable:

Reference: ECS process fields and ECS event fields.

Extend strategically with custom ECS fields

Mainframe-specific concepts have no ECS equivalent: job class, ASID, SMF record type and subtype, sysplex name, WTO routing codes, CICS transaction ID. Flattening these into labels.* as untyped strings destroys type information and makes them effectively unusable for queries and aggregations.

Define a zos.* custom namespace using ECS's documented extension mechanism. It keeps your core telemetry ECS-compliant while retaining the operational context your mainframe team needs for incident response.

Use ecs@mappings to stay current

Include ecs@mappings as a component template in your index template (available from Elasticsearch 8.9 for custom index templates and from 8.13 for Elastic Agent integration templates). It provides Elastic-maintained ECS field definitions automatically and keeps them current with each Elasticsearch release. For custom pipelines, this is what keeps your ECS alignment from drifting over time without manual upkeep.

One important caveat from the field: ecs@mappings provides the field definitions but doesn't enforce types at ingest. A return code arriving as a string is accepted and mapped as a string. Monitoring these discrepancies is critical, and they can be identified using the Data Quality dashboard. And because Elastic donated ECS to OpenTelemetry, the zos.* mappings you define here remain valid as OTel semantic conventions and ECS converge. The alignment work is the same whether data arrives via Logstash or OpenTelemetry Protocol (OTLP).

Choose the right architecture for the source

Most environments I work with run more than one of these ingest architectures, and different data sources have different latency, throughput, and licensing characteristics. A single architecture rarely covers everything. The table below maps common z/OS data sources to the architectures that work well for them.

Kafka is commonly added when there's a network resilience requirement between the mainframe and the Elastic cluster. If Kafka isn't already in your estate, the operational overhead of running Kafka should be weighed against the resilience benefits. From IBM MQ 9.4.3, Kafka Connect can run natively in z/OS UNIX System Services for MQ connector use cases, reducing the need for an off-platform Kafka Connect cluster.

When Kafka is used, Logstash is the recommended downstream consumer for the ingest paths in this guide. The Confluent Elasticsearch sink connector is an alternative; the self-managed v1 connector supports Elasticsearch 7.x and 8.x but is deprecated with end of life (EOL) in April 2027; and the v2 connector is Confluent Cloud only, making it unsuitable for on-premises and air-gapped environments.

Reference architecture: Kafka as middleware.

Flow diagram of the ingest paths from z/OS to Elastic.

IBM Z CDP: The workhorse for z/OS operational data

IBM Z CDP is the most widely deployed first-mile collector for z/OS operational data. It reads from SMF datasets in near-real time and forwards off-platform, handling the genuinely difficult part of getting data off z/OS without burdening performance-critical paths. In the environments I work with, it's the standard path for SMF type 30 job accounting, IMS statistical records, and z/OS SYSLOG.

CDP forwards to Logstash, which handles parsing, field extraction, and routing into Elasticsearch. Kafka is an optional middleware message queue:

• CDP → (Kafka →) Logstash → Elasticsearch

The trade-offs: CDP is a separately licensed IBM product, binary SMF records need preprocessing before Logstash can parse them, and the architecture isn't suited to sub-minute latency requirements.

Worth noting alongside CDP: IBM ODP plays the same collector role for performance and availability metrics from whichever OMEGAMON monitors are in your stack: CICS, IMS, Db2, z/OS, network, and storage. Unlike CDP's binary SMF output, ODP converts to JSON natively, so there's no preprocessing step. ODP consists of two components: OMEGAMON Data Broker (a Zowe cross-memory server plugin running on z/OS that collects attributes from OMEGAMON monitoring agents and forwards them to Data Connect); and OMEGAMON Data Connect (a Java application running on or off z/OS that receives data from Data Broker and forwards it to destinations including Elasticsearch; the destination settings are configured here). If OMEGAMON is already in your monitoring stack, ODP is the natural path for getting that telemetry into Elastic.

Reference architecture: Logstash to Elasticsearch.

IMS Connect Extension: For IMS workloads that can bypass CDP

Rocket Software's IMS Connect Extension journals IMS transaction activity directly as JSON, bypassing the SMF layer entirely. Events publish to Kafka, and Logstash consumes and indexes. Some organizations standardize all log streams through Kafka (rsyslog → Kafka → Logstash) as an optional resilience pattern.

• IMS Connect Extension → (Kafka →) Logstash → Elasticsearch

This works well for IMS transaction performance data and application-level event streams. JSON output removes the binary parsing problem. Kafka gives you decoupling, replay, and a buffer for downstream maintenance.

The trade-offs: IMS Connect Extension licensing, Kafka infrastructure to operate, and IMS-specific coverage that doesn't help with z/OS SYSLOG or other SMF types.

One thing I always validate before committing to this pattern is Kafka topic naming. Banks and regulated environments typically have strict topic naming policies, and IMS Connect Extension's default behavior of creating topics itself can clash with those policies. It’s cheaper to discover this before architecture commitment than after.

Batch export: For historical analysis and PoC

Export from IMS Problem Investigator or similar tooling to CSV, transfer off-platform, and ingest via Logstash or Elastic Agent file input. This approach has no real-time capability, and it doesn’t require any new z/OS software.

• Batch export → CSV/FTP → Logstash/Elastic Agent → Elasticsearch

This works well for historical analysis, initial PoC work, and demonstrating value before committing to a real-time pipeline. We also use this to get ECS mapping right before the production architecture is in place. It isn’t suitable as a long-term operational observability solution.

Linux on IBM Z: A separate and easier path

This path is often overlooked. Linux on IBM Z workloads can run standard Elastic Agent (Elastic Agent doesn’t run on native z/OS), no z/OS-specific tooling, no custom pipeline and the full Elastic integration catalog is available.

If you have Linux on IBM Z workloads in your estate, treat them as a separate (and considerably easier) onboarding path.

For all options, please see the reference architectures with Elastic Agent.

Throughput and air-gapped: The two questions every mainframe team asks

Throughput impact: Anything that touches z/OS performance-critical paths is a nonstarter for mainframe teams running thousands of transactions per second. All four architectures above use off-platform collection deliberately: CDP, ODP, Kafka/Logstash, batch export, or standard Linux agent. This is the right design for the environment, not a workaround.

Air-gapped environments: Most mainframe estates I work with are network-restricted to some degree. Elastic's air-gapped reference architecture is a documented, supported deployment path.

Build the pipeline, and don't start from scratch

For the CDP and IMS Connect Extension architectures, log data lands in Elastic reflecting the limited structure of its source. Mainframe log formats are installation-specific and partially structured at best; no off-the-shelf parser covers them, and writing a pipeline from scratch has historically been the largest time sink in any onboarding project. AI has changed that. For mainframe estates that can't call out to a hosted model (which is most of them), both tools below work with self-managed local large language models (LLMs), so the capability is available in air-gapped and network-restricted environments. See the local LLMs overview for supported options.

Streams: For data already landing in Elastic (available from 9.2).
Open the Processing tab for a stream in Kibana, and click Suggest pipeline. Within seconds, you're looking at a complete, validated pipeline (Grok or Dissect pattern, date normalization, type conversions, field cleanup) with a live preview of how your actual documents parse through it. Nothing writes to the stream until you confirm. Under the hood, generation runs in two stages: First, deterministic fingerprinting groups your log formats and picks the best parsing approach; second, a reasoning agent iterates to add normalization and cleanup, validating against hard thresholds before handing control to you. The result is a working pipeline you refine, not a starting point you rewrite. The technical detail is in How Streams Generates a Log Pipeline in Seconds.

Automatic Import: For building a new custom integration from the ground up (available from 8.18/9.0). Automatic Import takes a different path. You upload sample data, and it generates a complete, deployable Elastic Agent integration package (ingest pipeline, ECS field mappings, event categorization, and related.* field population), which you review and approve before it installs. Where the Streams Suggest Pipeline structures data already arriving in a stream, Automatic Import builds the entire collection path from scratch. Supported input formats include JSON, NDJSON, CSV, and syslog, which covers z/OS SYSLOG directly. Supported collection methods include Kafka, File Stream, TCP, and HTTP Endpoint, making it a natural fit for shops already routing data through Kafka or receiving ODP output over TCP. For mainframe shops adopting Elastic Agent, this removes what was previously weeks of custom integration work.

A less obvious benefit that applies to both tools is continuity. The engineer who wrote your custom GROK pattern eventually moves to another team. A tool that can regenerate a pipeline or integration from sample data is operational resilience.

In terms of scope, Streams works on text. Binary SMF records need to be converted to text or JSON upstream (via CDP or IBM-supplied utilities) before either tool can do anything with them. That conversion happens before Elastic is involved.

Configure the dead-letter queue from day one

Mainframe teams know the dead-letter queue pattern from MQ: When a message can't be delivered or processed, it goes to a holding queue rather than being silently dropped. Elasticsearch has the same concept for ingest pipelines, called the failure store. Configure it from day one, not after your first production incident.

Format drift is a recurring failure mode: vendor updates, sample-config-as-production, new message types appearing in batch windows. The failure store is how you find out about it before your dashboards lie to you. When a log line arrives in an unexpected format and the pipeline can't parse it, the failure store captures the original document with metadata about why it failed. You can query it, alert on its growth rate, and use the captured documents to fix the pipeline.

Without it, parse failures either fall to default handling (records indexed with raw message fields, expected query fields simply absent) or get dropped entirely. Either way, you don't know it's happening.

Configure retention based on how long it takes your team to triage drift, typically days to a couple of weeks. Pair it with an alert on document count or growth rate so the queue itself is the early warning, not something someone has to remember to check.

Verify mainframe data quality before you build on it

Don't build dashboards or alerting rules on data you haven't verified. The Data Quality dashboard tells you whether your ECS alignment is real or aspirational.

For mainframe data, silent type mismatches are common: return codes in hex mapped as keywords, elapsed times stored as strings, timestamps that never coerced to @timestamp. None of these fail at ingest. All of them silently break queries and alerting conditions.

Run the checker against real production data, not synthetic samples. z/OS log variation across batch windows and message types means edge cases only surface under real conditions. Expect to iterate: Find the mismatch, fix the pipeline, and run again. Two or three passes is normal for a new mainframe data stream.

Source quality validation, the failure store, and the Data Quality dashboard are three points on the same loop. Together they give you confidence that the dashboards reflect what's actually happening on the mainframe, not what you hoped your pipeline was producing.

Getting started with mainframe data onboarding

The mainframe is a first-class observability target, and the path there is more concrete than it was a few years ago. Managed integrations cover IBM MQ. CDP and Kafka-based architectures have well-understood deployment patterns. Streams and Automatic Import remove the blank-page problem for custom pipelines, including in restricted environments through local LLMs. IBM Z Observability Connect is there when the OTel path is in reach.

Recommended order of operations:

1. Validate the source data.
2. Use OOTB integrations where they exist.
3. Align to ECS early.
4. Choose architectures source by source.
5. Generate pipelines rather than write them from scratch.
6. Configure the failure store from day one.
7. Verify before building anything on top.

If your organization is working through this and you'd like to compare notes, or if you're hitting a specific blocker, reach out to your Elastic account team. For the OTel-native path, the End-to-End Observability from Cloud Native to Mainframe deep dive is the next read. To try the building blocks in your own environment, start a free Elastic Cloud trial.