fix(log-sources): attribute a file only to the process writing it #96

Merged
christianmanivong merged 1 commits from fix/log-writers-only into main 2026-11-06 17:52:19 +00:00
Owner

What

get_log_sources() named whichever process held a log file with the lowest pid as its writer. A log reader -- CrowdSec's acquisition, Promtail, Filebeat, rsyslog imfile, a tail -F -- holds the file too, so on paperless-01 the inventory said crowdsec writes /mnt/docroot/data/log/paperless.log (NetOrk/netork#2092).

Now only a process holding the file open for writing is its producer:

  • LOG_SOURCES_COMMAND prints a new [flags] section: the flags: line of /proc/<pid>/fdinfo/<fd> for every descriptor find listed. It is one xargs grep -H over all of them, in the same round trip -- no command per descriptor.
  • The parse takes the access mode (flags & 3): O_WRONLY/O_RDWR is a writer, O_RDONLY (0100000) a reader. The first writer by pid gives unit and process, whatever pid a reader has.
  • A file only readers hold stays listed, with unit and process None: it is still a log on the host (the CrowdSec idle check relies on the inventory listing what the agent reads).
  • A file only a reader inside a container holds is left out. Its path is that container's view of a mount; listing it with container_id kept would give it to the reader's container (the same misattribution), and listing it without would present a container path as a host path. The real file appears in its writer's namespace, or on the host.
  • A descriptor whose flags did not answer (the process ended in between) is no writer. A report without a [flags] section at all -- an older command -- is read as before, so a recorded fixture still parses the same.
  • Targets that are no path (socket:[…], pipe:[…]) are never a source.

No interface change; the drivers carry the command unchanged (_run_log_sources_command), so no driver needs a change. The linux, proxmox, openmediavault and qnap-qts driver tests pass against this branch.

Version

Proposed: 4.31.1 (PATCH -- a fix, no interface change). The entry is under [Unreleased].

Tests

Fixtures only, plus the command run on the test host itself:

  • a writer with 0102001 (O_WRONLY|O_APPEND), a reader with 0100000 and a lower pid on the same file: the writer is named;
  • a file only CrowdSec reads: listed, no producer, size and last write kept;
  • an O_RDWR writer (0100002); a reader only inside a container: left out;
  • a socket and a pipe descriptor: no source; a deleted file still left out though written;
  • a writer whose fdinfo did not answer; a report without [flags];
  • live: this test's process holds one file open for appending and one for reading; the command names a producer for the first and none for the second.

ruff, mypy, coverage (log_sources 100 %), napalm-standard check and ratchet pass.

Refs NetOrk/netork#2092

## What `get_log_sources()` named whichever process held a log file with the lowest pid as its writer. A log *reader* -- CrowdSec's acquisition, Promtail, Filebeat, rsyslog `imfile`, a `tail -F` -- holds the file too, so on paperless-01 the inventory said `crowdsec` writes `/mnt/docroot/data/log/paperless.log` (NetOrk/netork#2092). Now only a process holding the file open **for writing** is its producer: - `LOG_SOURCES_COMMAND` prints a new `[flags]` section: the `flags:` line of `/proc/<pid>/fdinfo/<fd>` for every descriptor `find` listed. It is one `xargs grep -H` over all of them, in the same round trip -- no command per descriptor. - The parse takes the access mode (`flags & 3`): `O_WRONLY`/`O_RDWR` is a writer, `O_RDONLY` (`0100000`) a reader. The first writer by pid gives `unit` and `process`, whatever pid a reader has. - A file only readers hold stays listed, with `unit` and `process` None: it is still a log on the host (the CrowdSec idle check relies on the inventory listing what the agent reads). - A file only a reader **inside a container** holds is left out. Its path is that container's view of a mount; listing it with `container_id` kept would give it to the reader's container (the same misattribution), and listing it without would present a container path as a host path. The real file appears in its writer's namespace, or on the host. - A descriptor whose flags did not answer (the process ended in between) is no writer. A report without a `[flags]` section at all -- an older command -- is read as before, so a recorded fixture still parses the same. - Targets that are no path (`socket:[…]`, `pipe:[…]`) are never a source. No interface change; the drivers carry the command unchanged (`_run_log_sources_command`), so no driver needs a change. The linux, proxmox, openmediavault and qnap-qts driver tests pass against this branch. ## Version Proposed: **4.31.1** (PATCH -- a fix, no interface change). The entry is under `[Unreleased]`. ## Tests Fixtures only, plus the command run on the test host itself: - a writer with `0102001` (O_WRONLY|O_APPEND), a reader with `0100000` and a lower pid on the same file: the writer is named; - a file only CrowdSec reads: listed, no producer, size and last write kept; - an `O_RDWR` writer (`0100002`); a reader only inside a container: left out; - a socket and a pipe descriptor: no source; a deleted file still left out though written; - a writer whose fdinfo did not answer; a report without `[flags]`; - live: this test's process holds one file open for appending and one for reading; the command names a producer for the first and none for the second. `ruff`, `mypy`, coverage (log_sources 100 %), `napalm-standard check` and `ratchet` pass. Refs NetOrk/netork#2092
christianmanivong added 1 commit 2026-11-06 17:45:46 +00:00
fix(log-sources): attribute a file only to the process writing it
driver / standard (pull_request) Successful in 16s
driver / lint (pull_request) Successful in 14s
driver / types (pull_request) Successful in 1m8s
driver / test (3.13) (pull_request) Successful in 1m8s
driver / test (3.12) (pull_request) Successful in 1m15s
driver / lowest (pull_request) Successful in 40s
driver / build (pull_request) Successful in 29s
driver / security (pull_request) Successful in 59s
CI / driver (pull_request) Successful in 2m25s
driver / publish (pull_request) Skipped
98d1dbef4c
A process that holds a log open only to read it -- CrowdSec's
acquisition, Promtail, rsyslog imfile, a tail -F -- was named as the
file's writer whenever it had the lower pid. The command now reads each
descriptor's flags from /proc/<pid>/fdinfo in the same round trip, and
only an O_WRONLY or O_RDWR holder is the producer. A file only readers
hold stays listed without one; a file only a reader inside a container
holds is its view of a mount and is left out.

Refs: NetOrk/netork#2092
christianmanivong merged commit f5a2811333 into main 2026-11-06 17:52:19 +00:00
christianmanivong deleted branch fix/log-writers-only 2026-11-06 17:52:19 +00:00
Sign in to join this conversation.