This post builds that pipeline from a real debugging session and ends with Agent Builder diagnosing a Jellyfin playback failure from the enriched logs. It also covers the alternatives, journald and the OpenTelemetry Collector, and when to prefer them.

How rootless Podman logs differ from Docker logs

The Docker integration reads Docker's format, paths and socket, and rootless Podman uses different ones for all three. Each produced a distinct symptom during our debugging session.

The log format is CRI, not Docker JSON

Podman's default file-based log driver is k8s-file, and even --log-driver=json-file is just an alias for it.

Instead of Docker's one-JSON-object-per-line format, Podman writes the format used by Kubernetes container runtimes (CRI): a timestamp, the stream name, a partial/full flag, and then the message.

An example follows:

A parser expecting Docker JSON breaks on this immediately.

Where rootless Podman stores container logs

Rootful Docker writes logs under /var/lib/docker/containers/<container-id>/. Rootless Podman stores everything under the user's home instead:

Any integration with a hardcoded /var/lib/... path finds nothing.

The Podman API socket is per-user

In rootless Podman there is no /var/run/docker.sock. Instead, it exposes a Docker-compatible API on a user-scoped socket, typically /run/user/<uid>/podman/podman.sock.

Where Elastic Agent runs: host install vs containerized

This walkthrough assumes Elastic Agent is installed directly on the host, not in a container. In our case, the Agent is Fleet-managed and installed directly on a Ubuntu host. It is not running in a container. That matters because the paths in the Agent Policy are resolved by the Agent process itself: a containerized Agent only sees its own mount namespace, so the host directories above would have to be volume-mounted into the Agent container and referenced by their in-container paths. Running as root with default privileges is also what lets the Agent read files under another user's home directory.

How the rootless Podman log pipeline fits together

The setup we ended up with is simple once all the pieces are known:

1. Elastic Agent tails the raw CRI log files from the rootless user's container storage.
2. A container parser with format: cri strips the timestamp, stream, and flag prefix from each line.
3. A processor extracts the container ID from a fixed position in the log file path.
4. The add_docker_metadata processor resolves that ID against the rootless Podman socket and attaches container name, image, and labels.
5. Enriched events are shipped to Elasticsearch.

Let's now build the log ingestion pipeline step by step.

Step 1: Confirm the Podman log driver

First, check which log driver Podman uses on your host, because the default varies by distribution and containers.conf:

If it prints journald, either switch to the file-based driver or jump to the journald alternative at the end of this post. To make k8s-file the default for all containers of this user, set it in ~/.config/containers/containers.conf:

You can also set it per container with podman run --log-driver=k8s-file. Remember that json-file is accepted but silently behaves as k8s-file: do not expect Docker-formatted JSON from it. Containers pick up the driver at creation time, so you will need to recreate any containers that were started with a different driver.

Step 2: Enable the rootless Podman API socket

The metadata enrichment in step 4 needs the Podman API. Enable the user-scoped socket as the user that runs the containers:

The enable-linger call keeps the user's systemd instance, and with it the socket, alive when the user is not logged in. Verify the socket answers Docker-compatible API calls:

You should get a JSON array describing the running containers. This compatibility layer is exactly why a processor named add_docker_metadata will work against Podman later.

Step 3: Collect rootless Podman logs with a Custom Logs integration

In Fleet, add the Custom Logs (Filestream) integration to your agent policy instead of the Docker integration.

Set the file path pattern to the rootless storage location:

Then, in the advanced options, configure the parser so the agent understands the CRI format:

The container parser removes the <timestamp> <stream> <flag> prefix, reassembles partial lines (the P flag marks a message that was split), and stores the original timestamp and stream in the event.

For standalone agents, the equivalent input configuration looks like this:

At this point you should see clean log messages in Discover, but without any container context: no name, no image, no labels. A raw container ID buried in log.file.path is all you have, which makes the data hard to filter and nearly impossible to correlate.

Step 4: Enrich with container metadata using add_docker_metadata

The add_docker_metadata processor enriches our logs with information such as the container name. It extracts a container ID from the log file path, queries the Docker-compatible API for that container, and attaches the container's metadata to each event.

Add it to the integration's processors field (or under processors: in a standalone input):

The match_source_index value deserves an explanation, as it is not a commonly used one. The processor splits the log file path on /, discards the empty leading element, and picks the component at the given index as the container ID.

For the rootless Podman path, the indices work out like this:

Index

Component

0

home

1

<user-id>

2

.local

3

share

4

containers

5

storage

6

overlay-containers

7

<container-id>

8

userdata

The default value of 4 exists because Docker's path is /var/lib/docker/containers/<container-id>/..., where index 4 lands on the ID. However, as you can see from the table above, in our case the proper value is 7.

Note that the correct index depends on the depth of the home directory. /home/<user-id>/... puts the ID at index 7, but a nonstandard home location shifts it. Count the components of your actual path to verify whether 7 is the right value for you.

Once the ID resolves correctly, the processor calls the Podman socket and each event gains the familiar container.* fields. Here is a real document from a Jellyfin container managed with podman-compose:

The metadata is more complete than it first appears: alongside container.id and container.image.name, every label on the container arrives too, including the com_docker_compose_* and io_podman_compose_* labels set by podman-compose and the standard org_opencontainers_image_* labels baked into the image. Filtering all logs of a compose project is now one query on container.labels.com_docker_compose_project.

Step 5: Validate the rootless Podman log pipeline

Generate some traffic and check Discover:

A healthy pipeline produces events where:

• message contains only the application log line, with no CRI prefix.
• @timestamp matches the timestamp Podman wrote.
• log.file.path points into overlay-containers.
• container.id is the full 64-character ID, and container.name and container.image.name are populated.

If metadata is missing but messages parse fine, test the socket with the curl command from step 2 and re-check match_source_index. If messages still carry a timestamp prefix, the CRI parser is not applied; verify the parser YAML made it into the integration policy.

Alternatives for collecting rootless Podman logs

The file-based approach above yields the same ECS container.* fields as the Docker integration, but three alternatives are worth weighing first.

Approach

Metadata

ECS container.*

Throughput

Best when

Custom Logs + add_docker_metadata

Name, image, and all labels from the Podman socket

Native

High, plain file tailing

You want the same fields the Docker integration produces

journald log driver

Container ID, name, and image as journal fields

Needs a rename step

Lower, rate limited by RateLimitBurst

Containers run as systemd services via Quadlet

Docker integration on the compat socket

None for logs

Metrics datasets only

Not applicable to logs

You only need metrics, validated per dataset

OpenTelemetry Collector (filelog)

container.id via a regex operator, names and images need custom work

Partial

High, plain file tailing

You are standardizing on EDOT

Collect Podman logs with the journald log driver

Podman integrates with systemd natively, and on many distributions journald is already the default log driver. With --log-driver=journald, Podman writes each log line to the journal and attaches CONTAINER_ID_FULL, CONTAINER_NAME, and the image name as structured journal fields.

You can collect these with the Custom Journald logs integration. The metadata comes for free, with no socket, no path counting, and no match_source_index.

However, journald applies rate limiting that can drop bursts from chatty containers unless you raise RateLimitBurst. Throughput is lower than plain file tailing. The container fields also arrive with journald's naming, so you need a rename step (an ingest pipeline or processors) to get ECS-style container.* fields. Rootless containers also write to the per-user journal rather than the system one. An agent running as root still sees those entries, because reading the default journal as root includes user journals, but enable journald persistence (Storage=persistent in journald.conf) if you want container logs to survive a reboot.

If your containers are managed as systemd services via Quadlet, this is a natural fit.

Point the Docker integration at the Podman compat socket (metrics only)

Since Podman exposes a Docker-compatible API, it is tempting to point the Docker integration's host at unix:///run/user/<uid>/podman/podman.sock.

• Metrics datasets can work, because they only talk to the API.
• Logs do not, because the integration reads Docker's JSON log files from Docker's paths, and rootless Podman uses neither. Treat this as a partial option for metrics, and validate each dataset you enable.

Collect Podman logs with the OpenTelemetry Collector

If you are standardizing on OpenTelemetry, for instance by using EDOT (Elastic Distributions of OpenTelemetry) the Collector's filelog receiver reads the same files with its container operator, which auto-detects CRI-style formats:

Set add_metadata_from_filepath: false because that option expects Kubernetes pod log paths, which don't match Podman's layout.

Here you will miss out on some data enrichment: there is no Podman equivalent of the k8sattributes processor, so container names and labels are not attached automatically.. You can extract container.id from the file path with a regex operator, but resolving it to names and images requires custom work. The contrib Collector does include a podman_stats receiver for container metrics over the same socket, so a Collector-based setup covers metrics well and logs with reduced metadata.

Parsing application logs from Podman containers

The Custom Logs pipeline above ships raw log lines without application-level parsing. For instance, Nginx logs will end up in your cluster plainly, without proper processing. If you'd like to enable proper processing, you can use Ingest Pipelines to "redirect" specific containers (e.g. nginx) to the Ingest Pipeline defined by the Elastic Integration, and get full parsing of that data "for free".

Root cause analysis on Podman logs with Agent Builder

Was it worth the effort? Absolutely! Getting clean, metadata-rich container logs into Elasticsearch lets an LLM investigate issues against the data directly, and Elastic Agent Builder is the quickest way to see that in action.

Agent Builder, available on Elastic Cloud Serverless and the Enterprise tier for Elastic Cloud Hosted and self-managed, provides a chat interface in Kibana, backed by an LLM, with built-in tools to explore your indices and run ES|QL queries against them.

Here is a real example from the host we just configured. Among other containers, it runs Jellyfin, a media server, as a rootless Podman container. One day, playback of a title fails instantly on an Android TV client: the screen goes black and drops back to the menu, with no error apart from a cryptic "Playback error".

In Discover, filtering on container.name: "jellyfin" narrows the view to a few hundred verbose .NET log lines around the playback attempts. The histogram shows a burst of activity at each failed attempt, but the actual cause is buried somewhere in the noise.

Instead of scanning the messages manually, open the Agent Builder panel directly from Discover and ask it to investigate the issue. The agent queries the log data stream on its own, reconstructs the timeline, and returns a diagnosis:

In this run it worked out that the client requested playback six times. Jellyfin tried to convert a 4K Dolby Vision/HEVC video to H.264 through VA-API hardware transcoding while burning in a subtitle stream, and FFmpeg exited with code 187 each time before producing the first HLS segment. That is why playback stopped at zero milliseconds. It also stated its uncertainty honestly: the available logs don't contain FFmpeg's stderr, so it proposed concrete steps to discriminate between a broken transcoding combination and missing GPU access, including checking whether the container can reach the render device /dev/dri/renderD128.

The pipeline we just built made each step of this investigation possible. The agent can slice by container.name only because add_docker_metadata resolved it from the Podman socket, the message field is queryable because the CRI parser stripped the prefixes, and the timeline is trustworthy because @timestamp comes from Podman, not from ingestion time. Without that work, the LLM would face a pile of prefixed raw lines, likely resulting in worse performance and higher token usage.

Conclusion: three settings for rootless Podman logs

Rootless Podman is Docker-compatible where it matters the most: the API. For log collection it uses its own format, file paths, and socket location, which is what the Elastic Docker integration reads.

Once you know that, the fix is three configuration decisions: parse cri instead of Docker JSON, tail the user's overlay-containers storage, and point add_docker_metadata at the per-user Podman socket with a corrected match_source_index.

The same recipe applies to any rootless user on the host; only the home path and UID in the socket change.

If you want to try this workflow end to end to collect your Podman logs, spin up an Elastic Cloud trial, install the Elastic Agent on your host, and you can go from raw CRI files to fully enriched container logs in one integration policy. If you want to push this further, from a one-off chat to automated investigations that open cases with evidence attached, see automated root cause analysis with Elastic Agent Builder.

Why use Elastic Observability with Proxmox?

Here at Elastic, we are passionate about efficiently managing and monitoring infrastructure and applications. Many of us have fun playing with home labs, oftentimes running Proxmox VE, a powerful open-source virtualization platform used to run virtual machines and Linux Containers (LXCs) with ease. While Proxmox provides robust tools for managing virtualized resources, gaining deep insights into the performance and health of your LXCs, VMs, and hosts requires a comprehensive monitoring solution. This blog post will guide you through leveraging the power of Elastic Observability, in conjunction with Elastic Agent, to effectively monitor your Proxmox VE deployment, ensuring optimal performance and proactive issue resolution thanks to Kibana Alerts.

The homelab setup

Our homelab setup centers around an Intel N100 mini PC, serving as the host for Proxmox VE. This setup is simple and minimal, yet effective for showcasing a few interesting capabilities. On top of this mini PC, we run several Linux Containers (LXCs) for various services, along with a dedicated virtual machine for Home Assistant.

Elastic Agent installation and configuration

Before beginning, it is worth noting that there are numerous ways to install and configure the Elastic Agent. For the sake of simplicity, we will showcase a setup in which only one instance of the Elastic Agent is running on the host machine. The Elastic Agent reports to an Elastic Cloud Observability deployment and is managed via Fleet, which makes it tremendously easy to upgrade and re-configure it whenever needed.

Diving into the host

Kibana offers various panes that make it nice and easy to learn about a system's health at a quick glance.

As a first step, let's take a look at the Infrastructure > Hosts page in Kibana:

Here we can see various information about our Proxmox VE host (i.e. the mini PC). The top processes running on it are presented, including processes running in LXCs such as pia-daemon. We can also see a kvm process, specifically running a Home Assistant virtual machine, and a Proxmox pve-firewall process.

Let's now take a look at Universal Profiling > Flamegraph. This graph shows how much CPU time is consumed by different stack traces from processes running on the host system. You can drill down into specific processes using the search bar at the top. For instance, you can filter by kvm to only see information regarding this specific process.

The Observability AI Assistant

All the Kibana panes we visited so far have proved to be highly interesting, but they struggle to answer urgent questions such as:

• did anything happen in our mini PC recently?
• was there any significant change in functionality?
• is there any precious information hidden among the thousands of data points collected?

The Elastic Observability AI Assistant helps us by answering these questions in natural language. By default, on Elastic Cloud, it uses the Elastic-managed LLM connector, which means users do not need to configure anything to get started with it. It just works!

Let's go to the Observability > AI Assistant pane in Kibana and let's try to ask a generic prompt such as: "please give me an overview of the health of my prox host".

Let's then wait a minute so that it can dig into the data… et voilà, here comes lots of relevant information in the form of graphs and natural language explanations. The Observability AI Assistant understood our question, went through all the data for our Proxmox host, ran data analytics on it, and reported back in a matter of seconds!

Alerting upon disruption with Kibana Alerts

As a final step, let's try to define a Kibana Alert to help us understand whether our host is overloaded. Let's head to Observability > Alerts > Rules and create a new rule. We will create a Custom Threshold rule that will fire if CPU usage for the host is higher than 80% on average for the last 15 minutes. Kibana will send us an email in case the rule fires. The rule is also configured to fire if no data appears for the last 15 minutes, which is extremely helpful as it would imply the presence of some issues to be debugged: broken network or no electricity in the house, a faulty Agent deployment, or even a hardware issue with the mini PC.

Conclusion

In this blog post we showcased how to effectively use the Elastic Stack to monitor Proxmox VE deployments. If you would like to try out such a setup first-hand, you are more than welcome to enjoy Elastic Cloud's 14-days free trial.

In future blog posts, we will investigate how to dig deeper into LXCs and VMs to gather even more information from our home lab and create more tailored alerts. Stay tuned!