Why small log files were skipped

Filestream needs a stable identifier for every file it reads so that it can track how far it has read from each file. In earlier versions, we used filesystem metadata such as inode numbers. This was not enough. For example, filesystems such as ext4 can reuse an inode number after a file is deleted, leading to a new file being mistaken for the old one. This is why we introduced the fingerprint identity and made it the default in 9.0.

Fingerprinting means files are identified by their content, not their metadata. The first 1024 bytes (by default) are hashed and used as the unique identity of the file. A registry keeps track of all IDs we encounter beyond the agent's process lifetime. Restart your machine, change the underlying device, host the files over the network, it doesn't matter; Elastic Agent will remember.

The major downside is that we can't hash a file's first 1024 bytes if it doesn't have this many bytes to begin with. Imagine a critical Kubernetes pod that panics on startup and its small log output is the reason the panic never reaches your Kibana dashboard.

The main way we've dealt with this until now is through configuration: instead of 1024 bytes, we could hash 512. Or 256. Or even less. However, each step towards reducing the threshold leaves less file content available to distinguish one file from another. Distinct files are therefore more likely to share the same fingerprint material, making the resulting file identity less stable. This is where the new growing fingerprint can shine instead.

How growing fingerprint splits file identity into two phases

With the introduction of the growing fingerprint mode, filestream splits a file's identity into two phases based on its size. While a file is smaller than the 1024-byte fingerprint threshold, it's in "growing" mode. Filestream identifies it with a hash of everything the file contains so far. When the file grows past the threshold, it gets promoted to a static fingerprint, the one it would have had pre-9.5.

What about file-identity collisions? Two files with identical contents do share a growing fingerprint, but only for as long as they stay identical: the moment their contents diverge, so do their fingerprints.

Let's take a closer look inside the internals. Let's start with a simple 13-byte text file:

mkdir -p /tmp/demo
echo 'Hello world!' > /tmp/demo/file.txt

and start Filebeat, the component inside Elastic Agent that collects log files, with a minimal configuration:

cat >/tmp/demo/filebeat.yml <<YAML
path.home: /tmp/demo

filebeat.inputs:
  - type: filestream
    id: growing-demo
    paths: [ "/tmp/demo/file.txt" ]

output.file:
  path: /tmp/demo/out
logging.level: debug
YAML

filebeat -c /tmp/demo/filebeat.yml

Phase 1: the growing fingerprint of a 13-byte file

After a few moments, filestream will pick up our new file. Digging through the debug logs (vastly reduced here for simplicity), we see:

{"log.level":"debug","log.logger":"input.filestream.prospector","message":"A new file /tmp/demo/file.txt has been found"}

We can also directly check the registry write-ahead log (WAL) filestream writes to data/registry/filebeat/log.json:

{
  "k": "filestream::growing-demo::fingerprint::f04f9d5ad8c21a115cd207488c6aaadeb4d7db581ea675f48c7f5d23e038c805",
  "v": {
    "cursor": {
      "eof": false,
      "offset": 13
    },
    "meta": {
      "fingerprint_len": 13,
      "identifier_name": "fingerprint",
      "source": "/tmp/demo/file.txt"
    }
  }
}

The k field is the registry key filestream uses to identify this file state. Its final segment is the fingerprint hash, which we can reproduce from the file contents:

$ xxd -p /tmp/demo/file.txt | tr -d '\n' | sha256sum
f04f9d5ad8c21a115cd207488c6aaadeb4d7db581ea675f48c7f5d23e038c805  -

Still in phase 1: the fingerprint changes as the file grows

Let's write more lines to the file:

for i in {1..70}; do echo "Hello $i" >> /tmp/demo/file.txt; done

Filestream detects the write while the file remains below the threshold:

{"log.logger":"input.filestream.prospector","message":"File /tmp/demo/file.txt has been updated","operation":"write"}

In the registry, the larger file now has a new fingerprint and a 634-byte fingerprint_len:

{
  "k": "filestream::growing-demo::fingerprint::cb8e361971e7ba91e343d1b5bd8e6560be0bdadbbed2b2f89a6c50099e972d63",
  "v": {
    "cursor": {
      "eof": false,
      "offset": 634
    },
    "meta": {
      "fingerprint_len": 634,
      "identifier_name": "fingerprint",
      "source": "/tmp/demo/file.txt"
    }
  }
}

The updated key ends in the SHA-256 hash of the hexadecimal fingerprint material:

$ xxd -p /tmp/demo/file.txt | tr -d '\n' | sha256sum
cb8e361971e7ba91e343d1b5bd8e6560be0bdadbbed2b2f89a6c50099e972d63  -

Phase 2: promotion to a static 1024-byte fingerprint

Let's now grow the file beyond the 1024-byte threshold:

for i in {1..500}; do echo "More messages $i" >> /tmp/demo/file.txt; done

Once the file crosses the threshold, filestream reports another write:

{"log.logger":"input.filestream.prospector","message":"File /tmp/demo/file.txt has been updated","operation":"write"}

The registry key changes one final time as the growing fingerprint is promoted to its static form:

{
  "k": "filestream::growing-demo::fingerprint::c8e252d76d621a02161f92e7a59d1617e3079db666cefd6cfcab106329082d1e",
  "v": {
    "cursor": {
      "eof": false,
      "offset": 9526
    },
    "meta": {
      "identifier_name": "fingerprint",
      "source": "/tmp/demo/file.txt"
    }
  }
}

The promoted fingerprint hashes the first 1024 bytes. It is exactly what a pre-9.5 Filebeat would have computed for this file:

$ head -c 1024 /tmp/demo/file.txt | sha256sum
c8e252d76d621a02161f92e7a59d1617e3079db666cefd6cfcab106329082d1e

How filestream matches a growing fingerprint to an existing file

During each scan, filestream resolves a file's identity in this order:

1. Look for an exact registry key matching the file's current fingerprint. This part has not changed.
2. If no key is found, search the growing entries for a shorter fingerprint, trying the same path first. A prefix match means filestream saw the same file earlier, when it contained fewer bytes.
3. Move the existing registry state to the new key, preserving the read offset.

You might have noticed from the registry entries above that we don't store the raw file contents. That was our initial approach, but it quickly exploded the size of the registry. Instead, we use the SHA-256 hash as the registry key and store fingerprint_len, which records the number of raw bytes the fingerprint represents. This reduces the storage required per entry by nearly 100×.

When filestream looks for a prefix match, it first groups the growing fingerprints by length. It then streams the scanned file's hexadecimal fingerprint material through SHA-256, snapshotting the digest at each group length. Filestream compares each snapshot only with stored hashes for the same length. A match means that the stored fingerprint is a prefix of the scanned file.

Growing fingerprint benchmarks: memory and CPU at 10,000 log files

So what does all this extra work cost? We benchmark Elastic Agent 9.5 ingesting thousands of above-threshold and sub-threshold files. To make the benchmark fair, we compare the growing fingerprint against the workaround of reducing the fingerprint size to 128 bytes. Otherwise, we would be comparing doing something against doing nothing and nothing would always win.

The benchmarks run on an r8id.2xlarge EC2 instance (8 vCPUs, Ubuntu 24.04) and report the median of five runs. Memory numbers are the Elastic Agent's anonymous memory as reported by the kernel via cgroups.

At 10,000 files, the two modes compare as follows.

| Metric | Condition | 128-byte static | Growing | Difference | |---|---|---|---|---| | Peak memory | Files above 1 KiB | 1,057 MiB | 1,099 MiB | +4% | | Peak memory | Files below 1 KiB | 941 MiB | 1,012 MiB | +8% | | Peak memory | Files crossing the threshold | 968 MiB | 1,016 MiB | +5% | | Steady-state memory | Files above 1 KiB | 291 MiB | 290 MiB | none | | Steady-state memory | Files below 1 KiB | 270 MiB | 322 MiB | +52 MiB | | CPU, one core | Files above 1 KiB, idle | 4.1% | 4.1% | none | | CPU, one core | Files below 1 KiB, idle | 3.9% | 4.9% | +1.0pp | | CPU, one core | Files crossing the threshold | 16.8% | 25.5% | +8.7pp | | Registry size at rest | Files above and below 1 KiB | 8.6 MB | 8.6 MB | none |

Memory use at 100, 1,000 and 10,000 log files

At 100 and 1,000 files, 128-byte static and growing fingerprints differ by only a few MiB. At 10,000 files the peaks separate by a few percent, but the only lasting difference is for sub-threshold files: above the threshold both modes stabilize within a MiB of each other.

The most demanding case is when the fingerprints are still changing. We created 10,000 sub-threshold files and appended 2,000 lines/s across them until every file crossed the threshold. That repeatedly exercises the prefix-lookup path and moves registry state to new keys. The extra memory this costs is released once the files are past the threshold and their harvesters close: both modes settle under 300 MiB, within a few MiB of each other:

CPU use with and without writes

The CPU impact is larger: every scan re-hashes a sub-threshold file's full contents instead of a fixed 128-byte prefix. That costs one extra percentage point of a core with no writes coming in, and almost nine under the changing-fingerprint workload above:

What changes in 9.6

We are already working on improving these numbers. On current main (future 9.6 release), the same benchmarks peak at less than half the memory, and the growing fingerprint's steady-state overhead, in both memory and CPU, is roughly halved.

Upgrading and rolling back

Growing mode ships enabled by default in 9.5 through file_identity.fingerprint.growing. Set it to false to restore the pre-9.5 behavior. There is no need for a migration on upgrade; existing registry entries keep working and files that were previously too small get picked up on the next scan.

Rolling back is also possible, but you go back to pre-9.5 behaviour: small files are ignored and when they cross the threshold, they get re-ingested.

What growing fingerprint means for your log files

Elastic Agent 9.5 ingests log files smaller than 1024 bytes by default, with no configuration and no waiting for the file to grow. Shared identities between identical small files resolve themselves as soon as their contents diverge, which a static fingerprint cannot do. The cost stays small: at realistic node scales the growing fingerprint is indistinguishable from the old workarounds, and at a 10,000-file workload it costs a few percent of memory and single-digit percent of one CPU core. Remember when upgrading that small files that were invisible before count as new, however long they have been sitting in your paths. If the trade doesn't work for you, file_identity.fingerprint.growing: false brings back the old behavior.

References

• How to choose file identity for filestream covers all identity options and their trade-offs.
• Growing fingerprint reference documents the setting and the rollback caveat.
• PR #50566 is the implementation and PR #51784 the registry optimization; PR #51863 and PR #51914 trimmed per-scan allocations and fingerprint retention in 9.5.2.
• Optimizations coming up in 9.6: PR #51278 parks idle harvesters instead of keeping one goroutine per open file, PR #51694 scans each file in a single pass, and PR #52476 normalizes published events in place. PR #52103, which replaces the multiline timeout goroutine with a read deadline, is not exercised by these benchmarks.