feat: which logs a host writes, and who writes each (4.30.0) #94

Merged
christianmanivong merged 2 commits from feature/log-sources-2049 into main 2026-11-04 23:41:03 +00:00
Owner

Summary

LogSourcesMixin.get_log_sources(): every log a Linux host writes, in one sh -c round trip, for NetOrk/netork#2049 (log-source inventory, epic NetOrk/netork#2048).

Changes

  • napalm_device_types/log_sources.py: LOG_SOURCES_COMMAND, parse_log_sources, LogSourcesMixin, LogSourcesUnavailable. It follows the template form of ListeningSocketsMixin: a driver supplies _run_log_sources_command(command, privileged=).
    • journald: journalctl -F _SYSTEMD_UNIT, services only, bounded by timeout 30.
    • Open files: find /proc/<pid>/fd for *.log, *.log.N and anything under /var/log.
      • Size and last write come from stat -L on the descriptor; the writer's service or container from /proc/<pid>/cgroup, reusing the listening helpers.
      • A file inside a container keeps the container's own path.
    • Container streams: the engine's file under /var/lib/docker/containers/<id>/ is that container's stream, not a file of dockerd.
    • Left out: journald's store, deleted files, and a database's write-ahead log (LevelDB's 000004.log).
    • Without root: it reads again and says so (attributed: false).
    • Without GNU find (busybox has no -lname/-printf): it raises LogSourcesUnavailable instead of reading as "no files".
  • models.py: LogSourceDict, LogSourcesDict.
  • README, CHANGELOG; version 4.30.0.

Related Issue

Refs NetOrk/netork#2049

Testing

  • tests/test_log_sources.py, 20 tests: parse, attribution, streams, exclusions, the GNU find check, the privileged fallback, sh -n, and a real run on the test host that finds a file the test holds open.
  • Full suite green; ruff and mypy clean.
## Summary `LogSourcesMixin.get_log_sources()`: every log a Linux host writes, in one `sh -c` round trip, for NetOrk/netork#2049 (log-source inventory, epic NetOrk/netork#2048). ## Changes - `napalm_device_types/log_sources.py`: `LOG_SOURCES_COMMAND`, `parse_log_sources`, `LogSourcesMixin`, `LogSourcesUnavailable`. It follows the template form of `ListeningSocketsMixin`: a driver supplies `_run_log_sources_command(command, privileged=)`. - **journald:** `journalctl -F _SYSTEMD_UNIT`, services only, bounded by `timeout 30`. - **Open files:** `find /proc/<pid>/fd` for `*.log`, `*.log.N` and anything under `/var/log`. - Size and last write come from `stat -L` on the descriptor; the writer's service or container from `/proc/<pid>/cgroup`, reusing the listening helpers. - A file inside a container keeps the container's own path. - **Container streams:** the engine's file under `/var/lib/docker/containers/<id>/` is that container's stream, not a file of `dockerd`. - **Left out:** journald's store, deleted files, and a database's write-ahead log (LevelDB's `000004.log`). - **Without root:** it reads again and says so (`attributed: false`). - **Without GNU `find`** (busybox has no `-lname`/`-printf`): it raises `LogSourcesUnavailable` instead of reading as "no files". - `models.py`: `LogSourceDict`, `LogSourcesDict`. - README, CHANGELOG; version 4.30.0. ## Related Issue Refs NetOrk/netork#2049 ## Testing - `tests/test_log_sources.py`, 20 tests: parse, attribution, streams, exclusions, the GNU `find` check, the privileged fallback, `sh -n`, and a real run on the test host that finds a file the test holds open. - Full suite green; ruff and mypy clean.
christianmanivong added 1 commit 2026-11-04 23:26:04 +00:00
feat: which logs a host writes, and who writes each (4.30.0)
driver / standard (pull_request) Failing after 19s
driver / lint (pull_request) Successful in 20s
driver / types (pull_request) Successful in 1m14s
driver / test (3.13) (pull_request) Successful in 1m11s
driver / test (3.12) (pull_request) Successful in 1m18s
driver / lowest (pull_request) Successful in 38s
driver / build (pull_request) Successful in 33s
driver / security (pull_request) Successful in 1m14s
CI / driver (pull_request) Failing after 2m45s
driver / publish (pull_request) Skipped
5daba5d37c
LogSourcesMixin.get_log_sources() reads, in one sh -c call as root, the
units journald holds entries for, the log files running processes hold
open (/proc/<pid>/fd: *.log, *.log.N, anything under /var/log) with size
and last write from stat -L on the descriptor, and the engine's own file
of each container's output as that container's stream. Each file names
the service or container of its first writer from /proc/<pid>/cgroup; a
file a container writes keeps the container's own path.

journald's store, deleted files and a database's write-ahead log
(LevelDB's 000004.log) are left out. Without root it reads again and says
so (attributed: false); a host whose find cannot list open files (busybox)
raises LogSourcesUnavailable rather than reading as no files.

Refs NetOrk/netork#2049
christianmanivong added 1 commit 2026-11-04 23:34:21 +00:00
refactor: keep the log source shapes in log_sources, not in models
driver / standard (pull_request) Successful in 16s
driver / lint (pull_request) Successful in 15s
driver / types (pull_request) Successful in 59s
driver / test (3.13) (pull_request) Successful in 1m17s
driver / test (3.12) (pull_request) Successful in 1m20s
driver / lowest (pull_request) Successful in 38s
driver / build (pull_request) Successful in 29s
driver / security (pull_request) Successful in 1m7s
CI / driver (pull_request) Successful in 2m42s
driver / publish (pull_request) Skipped
e622b26d42
models.py is over the file size limit and may not grow (quality ratchet);
the shapes sit with their mixin, as ScheduledJobDict does.

Refs NetOrk/netork#2049
christianmanivong merged commit 1b600416f0 into main 2026-11-04 23:41:03 +00:00
christianmanivong deleted branch feature/log-sources-2049 2026-11-04 23:41:03 +00:00
Sign in to join this conversation.