Compare commits

...
19 Commits
Author SHA1 Message Date
christianmanivong 278a707157 Merge pull request 'feat(running-backups): the backups running on a device right now' (#102) from feature/running-backups-2318 into main
driver / lint (push) Successful in 13s
driver / standard (push) Successful in 14s
driver / types (push) Successful in 59s
driver / test (3.13) (push) Successful in 58s
driver / test (3.12) (push) Successful in 1m13s
driver / lowest (push) Successful in 35s
driver / build (push) Successful in 26s
driver / security (push) Successful in 56s
driver / publish (push) Successful in 26s
CI / driver (push) Successful in 2m36s
2026-11-11 07:22:54 +00:00
christianmanivong 56e5e82a59 feat(running-backups): the backups running on a device right now
driver / lint (pull_request) Successful in 17s
driver / standard (pull_request) Successful in 19s
driver / types (pull_request) Successful in 1m2s
driver / test (3.13) (pull_request) Successful in 1m5s
driver / test (3.12) (pull_request) Successful in 1m15s
driver / lowest (pull_request) Successful in 37s
driver / build (pull_request) Successful in 29s
driver / security (pull_request) Successful in 59s
CI / driver (pull_request) Successful in 2m24s
driver / publish (pull_request) Skipped
A restart aborts every backup that runs on the device it restarts, and a
guest's restart breaks the backup its hypervisor is taking of it. netOrk
asks right before it restarts a host after patching (NetOrk/netork#2318),
so the reading belongs in the contract first: RunningBackupsMixin declares
get_running_backups(), opt-in like VmStopRequestMixin, so hasattr tells a
caller whether the driver can tell at all.

Release 4.34.0: the drivers that implement it need it on PyPI.
2026-11-10 23:39:17 +01:00
christianmanivong e6f145d4a3 Merge pull request 'chore(release): 4.33.0' (#101) from chore/release-4.33.0 into main
driver / types (push) Successful in 1m0s
driver / standard (push) Successful in 16s
driver / lint (push) Successful in 16s
driver / test (3.12) (push) Successful in 1m16s
driver / test (3.13) (push) Successful in 1m10s
driver / lowest (push) Successful in 54s
driver / build (push) Successful in 32s
driver / security (push) Successful in 1m5s
driver / publish (push) Successful in 33s
CI / driver (push) Successful in 3m28s
2026-11-07 19:23:16 +00:00
christianmanivong 66037e878e chore(release): 4.33.0
driver / standard (pull_request) Successful in 19s
driver / lint (pull_request) Successful in 14s
driver / types (pull_request) Successful in 1m7s
driver / test (3.12) (pull_request) Successful in 1m15s
driver / lowest (pull_request) Successful in 39s
driver / test (3.13) (pull_request) Successful in 1m22s
driver / build (pull_request) Successful in 44s
driver / security (pull_request) Successful in 1m9s
CI / driver (pull_request) Successful in 2m57s
driver / publish (pull_request) Skipped
2026-11-07 20:18:14 +01:00
christianmanivong afffd0915c Merge pull request 'feat(host-status): tell an interrupted dnf, yum, apk or opkg' (#98) from feat/interrupted-pm-851 into main
driver / standard (push) Successful in 17s
driver / lint (push) Successful in 16s
driver / types (push) Successful in 1m7s
driver / test (3.13) (push) Successful in 1m9s
driver / test (3.12) (push) Successful in 1m17s
driver / lowest (push) Successful in 41s
driver / build (push) Successful in 35s
driver / security (push) Successful in 1m7s
CI / driver (push) Successful in 2m34s
driver / publish (push) Skipped
2026-11-07 19:18:08 +00:00
christianmanivong 92891473be fix(host-status): keep models.py at its recorded size, one literal for the dnf query
driver / standard (pull_request) Successful in 14s
driver / lint (pull_request) Successful in 13s
driver / types (pull_request) Successful in 58s
driver / test (3.12) (pull_request) Successful in 1m11s
driver / test (3.13) (pull_request) Successful in 1m5s
driver / lowest (pull_request) Successful in 40s
driver / build (pull_request) Successful in 36s
driver / security (pull_request) Successful in 1m6s
CI / driver (pull_request) Successful in 2m38s
driver / publish (pull_request) Skipped
The quality ratchet records models.py at 1454 lines and the new comments
had made it 1460. bandit read the dnf history query, built with an
f-string for its path, as SQL put together from strings (B608); it is a
fixed query, now one literal with the path spelt out, and a test holds
it to DNF_HISTORY.

Refs NetOrk/netork#851
2026-11-07 20:14:11 +01:00
christianmanivong 658f1c8ab2 Merge remote-tracking branch 'origin/main' into feat/interrupted-pm-851
# Conflicts:
#	CHANGELOG.md
2026-11-07 20:14:11 +01:00
christianmanivong aacde6be3f Merge pull request 'chore(release): 4.32.0' (#100) from chore/release-4.32.0 into main
driver / standard (push) Successful in 17s
driver / lint (push) Successful in 14s
driver / types (push) Successful in 1m6s
driver / test (3.13) (push) Successful in 1m14s
driver / test (3.12) (push) Successful in 1m29s
driver / lowest (push) Successful in 37s
driver / build (push) Successful in 30s
driver / security (push) Successful in 1m7s
driver / publish (push) Successful in 35s
CI / driver (push) Successful in 3m14s
2026-11-07 18:49:52 +00:00
christianmanivong 58c7616401 chore(release): 4.32.0
driver / standard (pull_request) Successful in 18s
driver / lint (pull_request) Successful in 16s
driver / types (pull_request) Successful in 1m5s
driver / test (3.12) (pull_request) Successful in 1m23s
driver / test (3.13) (pull_request) Successful in 1m38s
driver / lowest (pull_request) Successful in 1m1s
driver / build (pull_request) Successful in 32s
driver / security (pull_request) Successful in 1m16s
CI / driver (pull_request) Successful in 3m13s
driver / publish (pull_request) Skipped
2026-11-07 19:44:49 +01:00
christianmanivong b7e31205d5 Merge pull request 'feat(ssh-server): read how sshd lets people log in' (#99) from feat/ssh-server-auth into main
driver / lint (push) Successful in 15s
driver / standard (push) Successful in 19s
driver / types (push) Successful in 1m2s
driver / test (3.13) (push) Successful in 1m16s
driver / test (3.12) (push) Successful in 1m22s
driver / lowest (push) Successful in 42s
driver / build (push) Successful in 33s
driver / security (push) Successful in 1m5s
CI / driver (push) Successful in 2m40s
driver / publish (push) Skipped
2026-11-07 18:44:43 +00:00
christianmanivong 1d71ff8f91 feat(ssh-server): read how sshd lets people log in
driver / lint (pull_request) Successful in 15s
driver / standard (pull_request) Successful in 17s
driver / types (pull_request) Successful in 1m10s
driver / test (3.13) (pull_request) Successful in 1m25s
driver / test (3.12) (pull_request) Successful in 1m32s
driver / lowest (pull_request) Successful in 44s
driver / build (pull_request) Successful in 32s
driver / security (pull_request) Successful in 1m5s
CI / driver (pull_request) Successful in 2m48s
driver / publish (pull_request) Skipped
SshServerAuthMixin.get_ssh_server_auth() reads the authentication settings
from sshd -T as root (password, keyboard-interactive, public key, host-based,
GSSAPI, AuthenticationMethods, PermitRootLogin, UsePAM) and, with PAM on, the
modules of sshd's PAM auth stack from /etc/pam.d with its includes followed.
It reports the facts; whether a module is a second factor is the consumer's
call. OpenSshServerMixin carries it and both read sshd -T by the same fixed
command, so every driver that can harden sshd can say this too.

Refs NetOrk/netork#662
2026-11-07 15:09:47 +01:00
christianmanivong 58ed0f0a94 feat(host-status): tell an interrupted dnf, yum, apk or opkg
driver / standard (pull_request) Failing after 18s
driver / lint (pull_request) Successful in 17s
driver / types (pull_request) Successful in 1m9s
driver / test (3.12) (pull_request) Successful in 1m22s
driver / lowest (pull_request) Successful in 39s
driver / build (pull_request) Successful in 36s
driver / test (3.13) (pull_request) Successful in 2m14s
driver / security (pull_request) Failing after 1m1s
CI / driver (pull_request) Failing after 2m42s
driver / publish (pull_request) Skipped
get_host_status() read only dpkg's state. It now also reads the rpm
database and dnf's history (or yum's journal), apk's installed database
and opkg's status file, without root, and names the manager it read in
the new package_manager field. While one of them runs, the answer is
unknown, as for dpkg.

Refs NetOrk/netork#851
2026-11-07 14:44:55 +01:00
christianmanivong c1a05d6dba Merge pull request 'chore(release): 4.31.1' (#97) from chore/release-4.31.1 into main
driver / types (push) Successful in 59s
driver / standard (push) Successful in 15s
driver / lint (push) Successful in 13s
driver / test (3.12) (push) Successful in 1m13s
driver / test (3.13) (push) Successful in 1m3s
driver / lowest (push) Successful in 37s
driver / build (push) Successful in 32s
driver / security (push) Successful in 55s
driver / publish (push) Successful in 27s
CI / driver (push) Successful in 3m3s
2026-11-06 17:57:27 +00:00
christianmanivong fccf902c07 chore(release): 4.31.1
driver / standard (pull_request) Successful in 18s
driver / lint (pull_request) Successful in 15s
driver / types (pull_request) Successful in 1m6s
driver / test (3.12) (pull_request) Successful in 1m23s
driver / lowest (pull_request) Successful in 39s
driver / test (3.13) (pull_request) Successful in 1m8s
driver / build (pull_request) Successful in 30s
driver / security (pull_request) Successful in 1m4s
CI / driver (pull_request) Successful in 3m0s
driver / publish (pull_request) Skipped
2026-11-06 18:52:24 +01:00
christianmanivong f5a2811333 Merge pull request 'fix(log-sources): attribute a file only to the process writing it' (#96) from fix/log-writers-only into main
driver / lint (push) Successful in 12s
driver / standard (push) Successful in 16s
driver / types (push) Successful in 56s
driver / test (3.13) (push) Successful in 1m4s
driver / test (3.12) (push) Successful in 1m12s
driver / lowest (push) Successful in 36s
driver / build (push) Successful in 35s
driver / security (push) Successful in 1m19s
CI / driver (push) Successful in 2m39s
driver / publish (push) Skipped
2026-11-06 17:52:18 +00:00
christianmanivong 98d1dbef4c fix(log-sources): attribute a file only to the process writing it
driver / standard (pull_request) Successful in 16s
driver / lint (pull_request) Successful in 14s
driver / types (pull_request) Successful in 1m8s
driver / test (3.13) (pull_request) Successful in 1m8s
driver / test (3.12) (pull_request) Successful in 1m15s
driver / lowest (pull_request) Successful in 40s
driver / build (pull_request) Successful in 29s
driver / security (pull_request) Successful in 59s
CI / driver (pull_request) Successful in 2m25s
driver / publish (pull_request) Skipped
A process that holds a log open only to read it -- CrowdSec's
acquisition, Promtail, rsyslog imfile, a tail -F -- was named as the
file's writer whenever it had the lower pid. The command now reads each
descriptor's flags from /proc/<pid>/fdinfo in the same round trip, and
only an O_WRONLY or O_RDWR holder is the producer. A file only readers
hold stays listed without one; a file only a reader inside a container
holds is its view of a mount and is left out.

Refs: NetOrk/netork#2092
2026-11-06 18:45:31 +01:00
christianmanivong 252ea3b81c Merge pull request 'feat: where a host's logs go beyond it (4.31.0)' (#95) from feat/log-forwarding-2053 into main
driver / lint (push) Successful in 13s
driver / standard (push) Successful in 16s
driver / types (push) Successful in 55s
driver / test (3.13) (push) Successful in 1m7s
driver / test (3.12) (push) Successful in 1m14s
driver / lowest (push) Successful in 37s
driver / build (push) Successful in 27s
driver / security (push) Successful in 53s
driver / publish (push) Successful in 27s
CI / driver (push) Successful in 2m44s
2026-11-06 10:14:19 +00:00
christianmanivong b685b200cc refactor: the rsyslog reader's cutter and action in their own steps
driver / standard (pull_request) Successful in 15s
driver / lint (pull_request) Successful in 13s
driver / types (pull_request) Successful in 53s
driver / test (3.12) (pull_request) Successful in 1m7s
driver / test (3.13) (pull_request) Successful in 1m2s
driver / lowest (pull_request) Successful in 37s
driver / build (pull_request) Successful in 26s
driver / security (pull_request) Successful in 56s
CI / driver (pull_request) Successful in 2m31s
driver / publish (pull_request) Skipped
Keeps _pieces and _Reader.call within the standard's complexity limit.
2026-11-06 11:10:57 +01:00
christianmanivong 8557710724 feat: where a host's logs go beyond it (4.31.0)
driver / lint (pull_request) Successful in 16s
driver / standard (pull_request) Failing after 17s
driver / types (pull_request) Successful in 1m8s
driver / test (3.13) (pull_request) Successful in 1m4s
driver / test (3.12) (pull_request) Successful in 1m15s
driver / lowest (pull_request) Successful in 38s
driver / build (pull_request) Successful in 30s
driver / security (pull_request) Successful in 1m2s
CI / driver (pull_request) Failing after 2m23s
driver / publish (pull_request) Skipped
get_log_sources() reads, in the same command, whether rsyslog runs, its
forwarding rules (legacy @/@@/:omrelp: and RainerScript omfwd/omrelp,
with selector, ruleset and whether they take every message), its imfile
inputs, and whether the journal reaches it (imjournal, or imuxsock with
journald's ForwardToSyslog; failing a setting, the default the
distribution ships commented out). Returned as `forwarding`;
parse_log_forwarding and parse_rsyslog are public (NetOrk/netork#2053).
2026-11-06 11:06:07 +01:00
18 changed files with 2233 additions and 46 deletions

No files matched your search

+54
View File
@@ -7,6 +7,60 @@ recorded here; their history is the git log.
## [Unreleased]
## [4.34.0] - 2026-11-10
### Added
- `RunningBackupsMixin.get_running_backups()`: the backups running on a device right now --
a hypervisor's backup of its guests (with the guests it is taking now), a backup server's
backup, sync, verify, garbage collection, prune or tape run, a host's own file-level backup.
Declared only; a driver mixes it in when it can tell, and a driver without it cannot tell.
`RunningBackupDict`, `RunningBackupKind` and `RUNNING_BACKUP_KINDS` are public. An
application asks right before it restarts a device (NetOrk/netork#2318).
## [4.33.0] - 2026-11-07
### Added
- `get_host_status()` tells an interrupted package manager on hosts without dpkg too, and
says which one it read in the new `package_manager` field (`dpkg`, `dnf`, `yum`, `apk`,
`opkg`, or None). dnf and yum: a package installed in two versions (not the kernels and
other packages dnf keeps side by side) as state `duplicate`, or a transaction that never
ended (dnf 4's history, yum's `transaction-all` journal). apk: a package apk marked with
broken files or a broken script (`broken-files`, `broken-script`), the ones `apk fix` would
take. opkg: a package left `unpacked`, `half-configured` or `half-installed`, or flagged for
reinstallation. All read without root; while one of them runs the answer is unknown
(NetOrk/netork#851).
## [4.32.0] - 2026-11-07
### Added
- `SshServerAuthMixin.get_ssh_server_auth()`: how an OpenSSH server lets people log in --
password, keyboard-interactive, public key, host-based and GSSAPI authentication,
`AuthenticationMethods` as lists of methods, `PermitRootLogin` and `UsePAM`, from `sshd -T`
as root, and sshd's PAM `auth` stack (each module with its control and arguments, includes
followed), from `/etc/pam.d` as the login user. `OpenSshServerMixin` carries it, so every
driver that can harden sshd has it. `parse_sshd_auth`, `parse_pam_auth`, `parse_pam_files`,
`pam_auth_stack` and `SSHD_EFFECTIVE_COMMAND` are public (NetOrk/netork#662).
## [4.31.1] - 2026-11-06
### Fixed
- `get_log_sources()` names as a file's writer only a process that holds it open for writing.
A log reader -- CrowdSec, Promtail, rsyslog's `imfile`, a `tail -F` -- was given the file
it reads as if it wrote it. A file only readers hold is now listed without a producer
(`unit` and `process` None); one only a reader inside a container holds is left out, as its
path is that container's view of a mount. The access mode comes from
`/proc/<pid>/fdinfo/<fd>`, read in the same command (NetOrk/netork#2092).
## [4.31.0] - 2026-11-06
### Added
- `get_log_sources()` also says where the logs go beyond the host: `forwarding` holds whether
rsyslog runs, its forwarding rules (legacy `@`/`@@`/`:omrelp:` and RainerScript
`omfwd`/`omrelp`, with selector, ruleset and whether they take every message), its `imfile`
inputs, and whether the journal reaches it (`imjournal`, or `imuxsock` with journald's
`ForwardToSyslog`). Read in the same command; `parse_log_forwarding` and `parse_rsyslog` are
public (NetOrk/netork#2053).
## [4.30.0] - 2026-11-04
### Added
+21 -3
View File
@@ -174,9 +174,12 @@ A driver supplies `_run_apt_command`, `_run_apt_root_command` and `_write_apt_fi
`HostStatusMixin` (`get_host_status`) is mixed in the same way: whether a Linux host needs
a reboot to finish an update (`/var/run/reboot-required`, `needs-restarting -r`, or a newer
kernel of the running flavour installed), whether it patches itself (unattended-upgrades,
dnf-automatic) and when that last succeeded (apt's `upgrade-stamp`), and whether dpkg was
left halfway (a journal in `/var/lib/dpkg/updates` or a half-configured package; unknown
while a package manager is running), and how full `/`, `/boot` and `/var` are (`df -Pk`,
dnf-automatic) and when that last succeeded (apt's `upgrade-stamp`), and whether its package
manager was left halfway (unknown while one is running): dpkg (a journal in
`/var/lib/dpkg/updates` or a half-configured package), dnf or yum (two versions of a package
in the rpm database, a transaction that never ended), apk (a package marked with broken files
or a broken script) or opkg (a package left `unpacked`), with `package_manager` naming which,
and how full `/`, `/boot` and `/var` are (`df -Pk`,
each filesystem once, in bytes; how full is too full is the caller's decision). `package_updates` holds the shared apt and dnf parsers: apt's suites become
an update's `origin`, a `-security` suite makes it a security update, and dnf's security
advisories do the same.
@@ -202,6 +205,11 @@ store, deleted files and a database's write-ahead log (LevelDB's `000004.log`) a
host whose `find` cannot list open files (busybox) raises `LogSourcesUnavailable`. A
driver supplies `_run_log_sources_command(command, privileged=)`; without root only the login
user's processes show their files, and the reading says so (`attributed: false`).
The same round trip reads where the logs go beyond the host (`forwarding`,
`napalm_device_types.log_forwarding`): whether rsyslog runs, its forwarding rules (`@`, `@@`,
`:omrelp:`, `action(type="omfwd"|"omrelp")`, each *complete* only for `*.*` with no condition
around it), the files it reads in (`imfile`), and whether the journal reaches it (`imjournal`,
or `imuxsock` with journald's `ForwardToSyslog`, the distribution's commented default counting).
LLDP neighbours are rows: `LldpNeighborRow` is one neighbour as its local port heard it
(chassis and port id with their type, name, description, management addresses,
@@ -290,6 +298,16 @@ A driver mixes it in and supplies `run_command`; a platform that regenerates ssh
itself (an appliance with its own database) must not, or opts out with
`harden_ssh_server = None`.
`SshServerAuthMixin` (`get_ssh_server_auth`) says how an OpenSSH server lets people log in:
password, keyboard-interactive, public key, host-based and GSSAPI authentication on or off,
`AuthenticationMethods` as lists of methods (empty for `any`), `PermitRootLogin` and
`UsePAM` (None where sshd has no PAM), from `sshd -T` as root -- the same fixed command the
hardening reads. With PAM on, it also names the modules of sshd's PAM `auth` stack
with each control and its arguments, read from `/etc/pam.d` in one listing as the
login user and followed through `@include`, `include` and `substack`. Which of them is a
second factor is the consumer's call. `Match` blocks are not read. `OpenSshServerMixin`
carries it; a driver can also mix it in alone and supply `run_command`.
`SudoHistoryMixin` (`get_history_sudo_lines`, `remove_history_sudo_lines`) deals with what
a driver left behind when it typed `echo '<password>' | sudo -S -p "" …` into an interactive
shell: that line, password and all, in the login user's shell history, and files its partial
+1
View File
@@ -151,6 +151,7 @@ the driver.
| Module | Knows |
|---|---|
| `host_status` | apt, dnf, dpkg, systemd timers, `df` |
| `package_managers` | rpm, dnf and yum history, apk's and opkg's databases, `/proc/<pid>/comm` |
| `kernel` | `/proc`, `/lib/modules`, gzip, base64 |
| `listening` | `ss`, `netstat`, `/proc/<pid>/cgroup` |
| `systemd` | `systemctl` |
+38
View File
@@ -56,8 +56,10 @@ instead of being restated on every role that happens to need it:
* :class:`~napalm_device_types.nat_vpn.NatVpnMixin`
* :class:`~napalm_device_types.packages.PackageManagementMixin`
* :class:`~napalm_device_types.ping_sweep.PingSweepMixin`
* :class:`~napalm_device_types.running_backups.RunningBackupsMixin`
* :class:`~napalm_device_types.service_clusters.ServiceClustersMixin`
* :class:`~napalm_device_types.service_packages.ServicePackagesMixin`
* :class:`~napalm_device_types.ssh_server_auth.SshServerAuthMixin`
* :class:`~napalm_device_types.sudo_history.SudoHistoryMixin`
* :class:`~napalm_device_types.scheduled_jobs.ScheduledJobsMixin`
* :class:`~napalm_device_types.scheduled_jobs_posix.PosixScheduledJobsMixin`
@@ -167,12 +169,19 @@ from napalm_device_types.local_accounts import (
LocalAccountDict,
LocalAccountsMixin,
)
from napalm_device_types.log_forwarding import (
FileInputDict,
ForwardRuleDict,
LogForwardingDict,
parse_rsyslog,
)
from napalm_device_types.log_sources import (
LOG_SOURCES_COMMAND,
LogSourceDict,
LogSourcesDict,
LogSourcesMixin,
LogSourcesUnavailable,
parse_log_forwarding,
parse_log_sources,
)
from napalm_device_types.mac_acl import MacAclMixin
@@ -216,6 +225,12 @@ from napalm_device_types.ping_sweep import PingSweepMixin, driver_supports_ping
from napalm_device_types.provisioning import VMProvisionIncomplete
from napalm_device_types.residential_gateway import ResidentialGatewayDriver
from napalm_device_types.roles import primary_role_of, role_keys_of, roles_of
from napalm_device_types.running_backups import (
RUNNING_BACKUP_KINDS,
RunningBackupDict,
RunningBackupKind,
RunningBackupsMixin,
)
from napalm_device_types.scheduled_jobs import (
ScheduledJobChanged,
ScheduledJobDict,
@@ -264,6 +279,14 @@ from napalm_device_types.ssh_server import (
SshHardeningResult,
plan_ssh_hardening,
)
from napalm_device_types.ssh_server_auth import (
SSHD_EFFECTIVE_COMMAND,
PamAuthEntryDict,
SshServerAuthDict,
SshServerAuthMixin,
pam_auth_stack,
parse_sshd_auth,
)
from napalm_device_types.storage import StorageDriver
from napalm_device_types.sudo_history import (
HistoryCleanupDict,
@@ -408,6 +431,11 @@ __all__ = [
"ListeningSocketsMixin",
"ListeningSocketsUnavailable",
"LOG_SOURCES_COMMAND",
"FileInputDict",
"ForwardRuleDict",
"LogForwardingDict",
"parse_log_forwarding",
"parse_rsyslog",
"LogSourceDict",
"LogSourcesDict",
"LogSourcesMixin",
@@ -447,6 +475,10 @@ __all__ = [
"VIRTUAL_IPS_COMMAND",
"VirtualIpsMixin",
"VmStopRequestMixin",
"RUNNING_BACKUP_KINDS",
"RunningBackupDict",
"RunningBackupKind",
"RunningBackupsMixin",
"VMProvisionIncomplete",
"SERVICE_CLUSTERS_COMMAND",
"ServiceClustersMixin",
@@ -457,6 +489,12 @@ __all__ = [
"is_placeholder_identity",
"parse_system_identity",
"OpenSshServerMixin",
"PamAuthEntryDict",
"SshServerAuthDict",
"SshServerAuthMixin",
"SSHD_EFFECTIVE_COMMAND",
"pam_auth_stack",
"parse_sshd_auth",
"SSH_ALGORITHM_CLASSES",
"SshHardeningPlan",
"SshHardeningResult",
+30 -8
View File
@@ -43,7 +43,9 @@ exits quietly every day (NetOrk/netork#738). While dpkg, apt, aptitude or
unattended-upgrade is running, packages are halfway on purpose -- also between
two of apt's dpkg calls -- so the answer is then ``None``, as on a host without
dpkg. unattended-upgrade is told from Ubuntu's idle shutdown helper by its
command line (:data:`BUSY_CHECK`).
command line (:data:`BUSY_CHECK`). On a host without dpkg, dnf or yum, apk and opkg
are read the same way (:mod:`napalm_device_types.package_managers`,
NetOrk/netork#851), and ``package_manager`` names the one the answer is about.
**Filesystems** are how full ``/``, ``/boot`` and ``/var`` are, from POSIX
``df -Pk``: where an upgrade unpacks a kernel, builds its initrd and keeps the
@@ -65,6 +67,11 @@ from napalm_device_types.models import (
HostStatusDict,
InterruptedPackageDict,
)
from napalm_device_types.package_managers import (
PACKAGE_MANAGERS_COMMAND,
PackageManagerReading,
parse_package_managers,
)
from napalm_device_types.terminal import strip_terminal_codes
_BEGIN = "HSTAT_BEGIN"
@@ -117,6 +124,7 @@ HOST_STATUS_COMMAND = (
f"echo '[dpkg-journal]'; ls -1 {_DPKG_JOURNAL} 2>/dev/null | head -n 20; "
"echo '[dpkg-audit]'; dpkg-query -W -f='${db:Status-Abbrev} ${Package}\\n' 2>/dev/null "
"| awk 'substr($0, 2, 1) ~ /[HUFWt]/ || substr($0, 3, 1) == \"R\"' | head -n 50; fi; "
f"{PACKAGE_MANAGERS_COMMAND}"
f"echo '[filesystems]'; df -Pk {_FILESYSTEMS}; "
f"echo '[timers]'; for u in {_APT_TIMER} {' '.join(_DNF_TIMERS)}; do "
'printf \'%s %s\\n\' "$u" "$(systemctl is-enabled "$u" 2>/dev/null)"; done; '
@@ -287,12 +295,24 @@ def _halfway(line: str) -> InterruptedPackageDict | None:
return {"name": name, "state": state} if name and state else None
def _dpkg(sections: dict[str, list[str]]) -> tuple[bool | None, list[InterruptedPackageDict]]:
if "dpkg-audit" not in sections or "dpkg-busy" in sections:
return None, []
def _dpkg(sections: dict[str, list[str]]) -> PackageManagerReading | None:
if "dpkg-audit" not in sections:
return None
if "dpkg-busy" in sections:
return PackageManagerReading("dpkg", None, [])
packages = [p for p in map(_halfway, sections["dpkg-audit"]) if p is not None]
journal = any(_JOURNAL_ENTRY.fullmatch(f) for f in sections.get("dpkg-journal") or [])
return bool(packages) or journal, packages
return PackageManagerReading("dpkg", bool(packages) or journal, packages)
def _package_manager(sections: dict[str, list[str]]) -> PackageManagerReading:
"""The reading that says most: one left halfway, then one that is not, then a busy one."""
readings = [r for r in (_dpkg(sections), *parse_package_managers(sections)) if r is not None]
for interrupted in (True, False):
found = next((r for r in readings if r.interrupted is interrupted), None)
if found is not None:
return found
return readings[0] if readings else PackageManagerReading(None, None, [])
def _filesystems(sections: dict[str, list[str]]) -> list[FilesystemUsageDict] | None:
@@ -318,15 +338,16 @@ def parse_host_status(output: str) -> HostStatusDict:
"""
sections = _sections(output)
required, reason = _reboot(sections)
interrupted, packages = _dpkg(sections)
manager = _package_manager(sections)
return {
"reboot_required": required,
"reboot_reason": reason,
"auto_updates": _auto_updates(sections),
"auto_updates_last_success": _last_success(sections),
"auto_updates_interval": _interval(sections),
"package_manager_interrupted": interrupted,
"interrupted_packages": packages,
"package_manager": manager.manager,
"package_manager_interrupted": manager.interrupted,
"interrupted_packages": manager.packages,
"filesystems": _filesystems(sections),
}
@@ -356,6 +377,7 @@ class HostStatusMixin:
* auto_updates (bool or None)
* auto_updates_last_success (Unix time or None)
* auto_updates_interval (seconds or None)
* package_manager (dpkg, dnf, yum, apk, opkg, or None)
* package_manager_interrupted (bool or None)
* interrupted_packages (list of {name, state})
* filesystems (list of {mountpoint, total, used, available} in bytes, or None)
+318
View File
@@ -0,0 +1,318 @@
"""Where a host's logs go beyond it: what rsyslog forwards, which files it reads
in, and whether it gets the journal (NetOrk/netork#2053).
A log source (:mod:`napalm_device_types.log_sources`) says who writes a log;
this says who reads it on. It is read in the same command, so a reading of
the logs a host writes also says which of them leave it.
**rsyslog's configuration, read as text.** Both of its syntaxes are read:
the legacy one (``authpriv.* @@logs:514``, ``& @host``, ``:programname, isequal,
"sshd" @host``, ``$InputFileName``) and RainerScript (``action(type="omfwd" …)``,
``input(type="imfile" …)``, ``ruleset(…) { … }``, ``if … then``). A rule is
*complete* when it takes every message of its ruleset -- ``*.*`` with no
condition around it. Anything narrower is partial, and keeps its selector or
condition for the reader to see: which facility a unit logs to is not
computed (decided in #2053).
**The journal reaches rsyslog** through ``imjournal``, or through ``imuxsock``
when journald forwards to syslog (``ForwardToSyslog``). With no setting of its
own a host runs on its distribution's default, which ``journald.conf`` ships
commented out: ``#ForwardToSyslog=yes``.
"""
from __future__ import annotations
import re
from typing import TypedDict
class ForwardRuleDict(TypedDict):
"""One rule sending messages to another host."""
protocol: str # "udp", "tcp" or "relp"
target: str # the host, as configured
port: int | None # None: the protocol's default
selector: str # "*.*", "authpriv.*", or the condition around it
complete: bool # every message of its ruleset goes
ruleset: str | None # None: the default ruleset, which gets the journal and syslog
class FileInputDict(TypedDict):
"""A file rsyslog reads in (``imfile``), and the ruleset it hands it to."""
path: str # as configured: may hold a glob
ruleset: str | None
class RsyslogDict(TypedDict):
rules: list[ForwardRuleDict]
file_inputs: list[FileInputDict]
imjournal: bool
imuxsock: bool
class LogForwardingDict(TypedDict):
"""Where a host's logs go beyond it.
``journal`` is None when it cannot be told whether the journal reaches
rsyslog: ``imuxsock`` is loaded but ``journald.conf`` says nothing.
"""
rsyslog: bool # rsyslogd runs
journal: bool | None # the journal reaches rsyslog
rules: list[ForwardRuleDict]
file_inputs: list[FileInputDict]
#: Selectors that take every message.
_EVERYTHING = {"*.*", "*.debug"}
_PARAM_RE = re.compile(r'([A-Za-z][\w.]*)\s*=\s*"([^"]*)"')
_CALL_RE = re.compile(
r"^(?P<before>.*?)\b(?P<name>action|input|module|ruleset)\s*\((?P<args>.*)\)\s*;?$", re.S
)
_IF_RE = re.compile(r"^(?:else\s+)?if\s+(?P<cond>.*?)\s+then\b\s*(?P<rest>.*)$", re.S)
_PROPERTY_RE = re.compile(r'^(?P<sel>:[^,]+,\s*!?[\w-]+,\s*"(?:[^"\\]|\\.)*")\s+(?P<action>.+)$')
_LEGACY_RE = re.compile(r"^(?P<sel>\S+)\s+(?P<action>.+)$")
_FORWARD_RE = re.compile(
r"^(?:(?P<at>@@?)(?:\([^)]*\))?|:omrelp:)"
r"(?:\[(?P<v6>[^\]]+)\]|(?P<host>[^:;\s]+))(?::(?P<port>\d+))?"
)
_FORWARD_TYPES = {"omfwd", "omrelp"}
_TRUE = {"yes", "true", "1", "on"}
def _uncomment(text: str) -> str:
"""*text* without its comments: ``#`` to the end of the line, outside quotes."""
out: list[str] = []
quoted = commented = False
for char in text:
if char == "\n":
quoted = commented = False
elif commented:
continue
elif char == '"':
quoted = not quoted
elif char == "#" and not quoted:
commented = True
continue
out.append(char)
return "".join(out)
class _Pieces:
"""Cuts a configuration into statements, ``{`` and ``}``: a line runs on
while a bracket is open, and a brace outside one stands as a piece of its own."""
def __init__(self) -> None:
self.pieces: list[str] = []
self.current: list[str] = []
self.depth = 0
self.quoted = False
def close(self) -> None:
piece = "".join(self.current).strip()
if piece:
self.pieces.append(piece)
self.current.clear()
def bracket(self, char: str) -> None:
if char in "([":
self.depth += 1
elif char in ")]":
self.depth = max(self.depth - 1, 0)
def feed(self, char: str) -> None:
if char == '"':
self.quoted = not self.quoted
elif not self.quoted:
self.bracket(char)
if self.depth == 0 and char in "{}\n":
self.close()
if char != "\n":
self.pieces.append(char)
return
self.current.append(char)
def _pieces(text: str) -> list[str]:
"""Statements, ``{`` and ``}`` in order (:class:`_Pieces`)."""
cutter = _Pieces()
for char in text:
cutter.feed(char)
cutter.close()
return cutter.pieces
def _params(args: str) -> dict[str, str]:
return {key.lower(): value for key, value in _PARAM_RE.findall(args)}
def _port(value: str | None) -> int | None:
return int(value) if value and value.isdigit() else None
class _Reader:
"""Walks the pieces of a configuration, keeping the blocks it is in."""
def __init__(self) -> None:
self.parsed: RsyslogDict = {
"rules": [],
"file_inputs": [],
"imjournal": False,
"imuxsock": False,
}
self.stack: list[tuple[str, str | None]] = []
self.pending: tuple[str, str | None] | None = None
self.legacy_ruleset: str | None = None
self.last_selector = "*.*"
def ruleset(self) -> str | None:
names = [value for kind, value in self.stack if kind == "ruleset"]
return names[-1] if names else self.legacy_ruleset
def condition(self, once: str | None) -> str | None:
conditions = [value for kind, value in self.stack if kind == "if"]
return once or (conditions[-1] if conditions else None)
def forward(
self, protocol: str, target: str, port: int | None, selector: str, once: str | None
) -> None:
condition = self.condition(once)
self.parsed["rules"].append(
{
"protocol": protocol,
"target": target,
"port": port,
"selector": condition or selector,
"complete": condition is None and selector in _EVERYTHING,
"ruleset": self.ruleset(),
}
)
def piece(self, piece: str) -> None:
if piece == "{":
self.stack.append(self.pending or ("block", None))
self.pending = None
return
if piece == "}":
if self.stack:
self.stack.pop()
return
if piece == "else":
self.pending = ("if", "else")
return
# A condition whose action follows on its own: the action is under it.
once = self.pending[1] if self.pending and self.pending[0] == "if" else None
self.pending = None
match = _IF_RE.match(piece)
if match:
if match["rest"]:
self.statement(match["rest"], once=match["cond"])
else:
self.pending = ("if", match["cond"])
return
self.statement(piece, once=once)
def statement(self, text: str, once: str | None) -> None:
call = _CALL_RE.match(text)
if call:
self.call(call["name"], _params(call["args"]), call["before"].strip(), once)
elif text.startswith("$"):
self.directive(text)
else:
self.legacy(text, once)
def call(self, name: str, params: dict[str, str], before: str, once: str | None) -> None:
if name == "module":
self.module(params.get("load", ""))
elif name == "ruleset":
self.pending = ("ruleset", params.get("name"))
elif name == "input" and params.get("type") == "imfile" and params.get("file"):
self.parsed["file_inputs"].append(
{"path": params["file"], "ruleset": params.get("ruleset")}
)
elif name == "action":
self.action(params, before, once)
def action(self, params: dict[str, str], before: str, once: str | None) -> None:
selector = self.last_selector if before == "&" else (before or "*.*")
self.last_selector = selector
kind = params.get("type", "")
if kind in _FORWARD_TYPES and params.get("target"):
protocol = "relp" if kind == "omrelp" else params.get("protocol", "udp").lower()
self.forward(protocol, params["target"], _port(params.get("port")), selector, once)
def module(self, name: str) -> None:
if name == "imjournal":
self.parsed["imjournal"] = True
elif name == "imuxsock":
self.parsed["imuxsock"] = True
def directive(self, text: str) -> None:
name, _, value = text.partition(" ")
name, value = name.lower(), value.strip()
if name == "$modload":
self.module(value)
elif name == "$inputfilename" and value:
self.parsed["file_inputs"].append({"path": value, "ruleset": self.legacy_ruleset})
elif name == "$ruleset":
self.legacy_ruleset = None if value == "RSYSLOG_DefaultRuleset" else value
def legacy(self, text: str, once: str | None) -> None:
match = _PROPERTY_RE.match(text) or _LEGACY_RE.match(text)
if not match:
return
selector = self.last_selector if match["sel"] == "&" else match["sel"]
self.last_selector = selector
forward = _FORWARD_RE.match(match["action"].strip())
if forward:
protocol = {"@": "udp", "@@": "tcp"}.get(forward["at"] or "", "relp")
self.forward(
protocol, forward["v6"] or forward["host"], _port(forward["port"]), selector, once
)
def parse_rsyslog(text: str) -> RsyslogDict:
"""rsyslog's configuration -- ``rsyslog.conf`` and ``rsyslog.d`` as one text --
read for its forwarding rules, its file inputs, and the inputs that take
the journal."""
reader = _Reader()
for piece in _pieces(_uncomment(text)):
reader.piece(piece)
return reader.parsed
def journal_to_syslog(lines: list[str]) -> bool | None:
"""Whether journald forwards to syslog, from its ``ForwardToSyslog`` lines in
the order journald reads them: the last setting wins; with none, the
default the distribution ships commented out; None when nothing says."""
setting = shipped = None
for line in lines:
commented = line.lstrip().startswith("#")
_, _, value = line.lstrip("# \t").partition("=")
if commented:
shipped = value.strip().lower() in _TRUE
else:
setting = value.strip().lower() in _TRUE
return setting if setting is not None else shipped
def log_forwarding(
running: bool, rsyslog: RsyslogDict, journald_lines: list[str]
) -> LogForwardingDict:
"""Where a host's logs go: *rsyslog* is its configuration, read whether it runs or not."""
if not running:
return {"rsyslog": False, "journal": False, "rules": [], "file_inputs": []}
journal: bool | None
if rsyslog["imjournal"]:
journal = True
elif rsyslog["imuxsock"]:
journal = journal_to_syslog(journald_lines)
else:
journal = False
return {
"rsyslog": True,
"journal": journal,
"rules": rsyslog["rules"],
"file_inputs": rsyslog["file_inputs"],
}
+124 -26
View File
@@ -14,10 +14,22 @@ whose path is the container's own. The engine's own file of a container's
output (``/var/lib/docker/containers/<id>/…``) is that container's stream, not
a file of ``dockerd``.
**Who writes it.** ``/proc/<pid>/cgroup`` says which systemd service or
container the first process holding a file runs in, as for listening sockets
(:mod:`napalm_device_types.listening`). journald's own files, files already
deleted and a database's write-ahead log (LevelDB's ``000004.log``) are left out.
**Who writes it.** Only a process holding the file open for writing writes
it: the access mode in ``/proc/<pid>/fdinfo/<fd>``'s ``flags`` is ``O_WRONLY`` or
``O_RDWR``. A reader -- CrowdSec, Promtail, rsyslog's ``imfile``, a ``tail -F``
-- holds it too, and is no producer (NetOrk/netork#2092). ``/proc/<pid>/cgroup``
says which systemd service or container the first writer runs in, as for
listening sockets (:mod:`napalm_device_types.listening`). A file only readers
hold is listed without a producer; one only readers in a container hold is left
out, its path being the reader's view of a mount. journald's own files, files
already deleted and a database's write-ahead log (LevelDB's ``000004.log``) are
left out.
**Where they go** is read in the same round trip
(:mod:`napalm_device_types.log_forwarding`, NetOrk/netork#2053): whether
rsyslog runs, its configuration -- each line behind ``| ``, so that none reads
as a section -- and journald's ``ForwardToSyslog`` lines in the order journald
reads them.
**One round trip, root, and without it.** As for listening sockets the script
goes as one ``sh -c`` argument. Without root only the login user's processes
@@ -36,6 +48,7 @@ from shlex import quote
from typing import TYPE_CHECKING, TypedDict
from napalm_device_types.listening import _cgroup_paths, _container, _sections, _unit
from napalm_device_types.log_forwarding import LogForwardingDict, log_forwarding, parse_rsyslog
from napalm_device_types.terminal import strip_terminal_codes
@@ -51,7 +64,7 @@ class LogSourceDict(TypedDict):
path: str | None # the file; None for a journal
unit: str | None # the journal's unit, or the file writer's service; without ".service"
container_id: str | None # the full container ID: the writer's, or the stream's
process: str | None # the first process holding the file
process: str | None # the first process writing the file; None when only readers hold it
size: int | None # bytes; None for a journal, or when the file could not be read
modified: str | None # the last write, ISO 8601 UTC; None as for size
@@ -68,6 +81,7 @@ class LogSourcesDict(TypedDict):
attributed: bool
journald: bool
sources: list[LogSourceDict]
forwarding: LogForwardingDict | None # where the logs go beyond the host
_BEGIN = "LOGS_BEGIN"
@@ -90,15 +104,28 @@ LOG_SOURCES_COMMAND = (
"echo '[open]'; printf '%s\\n' \"$o\"; "
"echo '[stat]'; printf '%s\\n' \"$o\" | cut -d' ' -f1 "
"| xargs -r stat -L -c '%n %s %Y' 2>/dev/null; "
"echo '[flags]'; printf '%s\\n' \"$o\" | cut -d' ' -f1 | sed 's|/fd/|/fdinfo/|' "
"| xargs -r grep -H '^flags:' 2>/dev/null; "
"pids=$(printf '%s\\n' \"$o\" | cut -d/ -f3 | sort -un); "
"echo '[cgroups]'; for p in $pids; do sed \"s|^|$p |\" /proc/$p/cgroup 2>/dev/null; done; "
"echo '[comm]'; for p in $pids; do c=$(cat /proc/$p/comm 2>/dev/null) && echo \"$p $c\"; done; "
"else echo '[no-find]'; fi; "
"echo '[rsyslog]'; { pidof rsyslogd || pgrep -x rsyslogd; } >/dev/null 2>&1 && echo running; "
"echo '[rsyslog-conf]'; cat /etc/rsyslog.conf /etc/rsyslog.d/*.conf 2>/dev/null "
"| sed 's/^/| /'; "
"echo '[journald-conf]'; cat /etc/systemd/journald.conf "
"/usr/lib/systemd/journald.conf.d/*.conf /run/systemd/journald.conf.d/*.conf "
"/etc/systemd/journald.conf.d/*.conf 2>/dev/null "
"| grep -i '^[#[:space:]]*ForwardToSyslog'; "
"printf '%s%s\\n' LOGS_ END"
)
#: ``/proc/900/fd/5`` -- the process and its descriptor.
_FD_RE = re.compile(r"^/proc/(\d+)/fd/\d+$")
#: ``/proc/900/fdinfo/5:flags:<tab>0102001`` -- the descriptor's open flags, octal.
_FLAGS_RE = re.compile(r"^/proc/(\d+)/fdinfo/(\d+):flags:\s*([0-7]+)$")
#: The access mode bits of the open flags: 0 is O_RDONLY, 1 O_WRONLY, 2 O_RDWR.
_ACCESS_MODE = 0o3
#: The engine's file of a container's output: json-file and local drivers.
_STREAM_RE = re.compile(r"^/var/lib/docker/containers/([0-9a-f]{64})/")
#: Files that are no log to read: journald's store, a file already deleted, and
@@ -159,39 +186,83 @@ def _journals(units: list[str]) -> list[LogSourceDict]:
def _held(lines: list[str]) -> list[tuple[int, str, str]]:
"""(pid, descriptor, target) for every log a process holds, by pid."""
"""(pid, descriptor, target) for every log file a process holds, by pid."""
held = []
for line in lines:
fd, _, target = line.partition(" ")
match = _FD_RE.match(fd)
if match and target and not _NOT_A_SOURCE.search(target):
if match and target.startswith("/") and not _NOT_A_SOURCE.search(target):
held.append((int(match.group(1)), fd, target))
return sorted(held)
def _writes(lines: list[str]) -> dict[str, bool]:
"""Whether each descriptor is open for writing, by its ``/proc/<pid>/fd/<fd>``."""
writes: dict[str, bool] = {}
for line in lines:
match = _FLAGS_RE.match(line)
if match:
pid, fd, flags = match.groups()
writes[f"/proc/{pid}/fd/{fd}"] = int(flags, 8) & _ACCESS_MODE != 0
return writes
_Key = tuple[str, str | None, str]
def _holders(lines: list[str], paths: dict[int, str]) -> dict[_Key, list[tuple[int, str]]]:
"""The (pid, descriptor) pairs holding each log, by kind, container and path."""
holders: dict[_Key, list[tuple[int, str]]] = {}
for pid, fd, target in _held(lines):
stream = _STREAM_RE.match(target)
kind, container = (
("container", stream.group(1)) if stream else ("file", _container(paths.get(pid)))
)
holders.setdefault((kind, container, target), []).append((pid, fd))
return holders
def _writer(held: list[tuple[int, str]], writes: dict[str, bool] | None) -> int | None:
"""The first process writing; the first holding for a report without flags,
as an older command printed it."""
if writes is None:
return held[0][0]
return next((pid for pid, fd in held if writes.get(fd)), None)
def _producer(
kind: str, writer: int | None, paths: dict[int, str], names: dict[int, str]
) -> tuple[str | None, str | None]:
"""The writer's service and name; a stream's service is its container."""
if writer is None:
return None, None
return (_unit(paths.get(writer)) if kind == "file" else None), names.get(writer)
def _files(sections: dict[str, list[str]]) -> list[LogSourceDict]:
stats = _stats(sections.get("stat", []))
paths = _cgroup_paths(sections.get("cgroups", []))
names = _names(sections.get("comm", []))
found: dict[tuple[str, str | None, str], LogSourceDict] = {}
for pid, fd, target in _held(sections.get("open", [])):
stream = _STREAM_RE.match(target)
cgroup = paths.get(pid)
kind, container = ("container", stream.group(1)) if stream else ("file", _container(cgroup))
source = found.get((kind, container, target))
if source is None:
source = found[(kind, container, target)] = {
writes = _writes(sections["flags"]) if "flags" in sections else None
found: list[LogSourceDict] = []
for (kind, container, target), held in _holders(sections.get("open", []), paths).items():
writer = _writer(held, writes)
if writer is None and kind == "file" and container is not None:
continue # a reader's view of a mount, inside its container
unit, process = _producer(kind, writer, paths, names)
size, modified = next((stats[fd] for _, fd in held if fd in stats), (None, None))
found.append(
{
"kind": kind,
"path": target,
"unit": None if stream else _unit(cgroup),
"unit": unit,
"container_id": container,
"process": names.get(pid),
"size": None,
"modified": None,
"process": process,
"size": size,
"modified": modified,
}
if source["size"] is None and fd in stats:
source["size"], source["modified"] = stats[fd]
return list(found.values())
)
return found
def _order(source: LogSourceDict) -> tuple[str, str, str]:
@@ -213,6 +284,33 @@ def parse_log_sources(output: str) -> tuple[bool, list[LogSourceDict]]:
return journald, sorted(sources, key=_order)
def _reading(output: str, *, attributed: bool) -> LogSourcesDict:
journald, sources = parse_log_sources(output)
forwarding = parse_log_forwarding(output)
return {
"attributed": attributed,
"journald": journald,
"sources": sources,
"forwarding": forwarding,
}
def parse_log_forwarding(output: str) -> LogForwardingDict | None:
"""Parse where the logs go from what :data:`LOG_SOURCES_COMMAND` printed;
None for a report without the sections, as an older command printed it.
:raises ValueError: when the output carries no intact report.
"""
sections = _sections(_frame(output))
if "rsyslog" not in sections:
return None
config = "\n".join(
line[2:] if line.startswith("| ") else line[1:] for line in sections.get("rsyslog-conf", [])
)
running = "running" in sections["rsyslog"]
return log_forwarding(running, parse_rsyslog(config), sections.get("journald-conf", []))
class LogSourcesMixin:
"""Adds :meth:`get_log_sources` to a driver that can run a command on a Linux host.
@@ -237,6 +335,8 @@ class LogSourcesMixin:
* attributed (bool) - read as root, so every process's files are listed
* journald (bool) - the host keeps systemd's journal
* sources (list) - see :class:`~napalm_device_types.models.LogSourceDict`
* forwarding (dict) - where the logs go beyond the host, see
:class:`~napalm_device_types.log_forwarding.LogForwardingDict`
Example::
@@ -258,10 +358,8 @@ class LogSourcesMixin:
command = f"sh -c {quote(LOG_SOURCES_COMMAND)}"
try:
output = self._run_log_sources_command(command, privileged=True)
journald, sources = parse_log_sources(output)
return {"attributed": True, "journald": journald, "sources": sources}
return _reading(output, attributed=True)
except ValueError:
pass
output = self._run_log_sources_command(command, privileged=False)
journald, sources = parse_log_sources(output)
return {"attributed": False, "journald": journald, "sources": sources}
return _reading(output, attributed=False)
+5 -5
View File
@@ -85,8 +85,8 @@ class InterruptedPackageDict(TypedDict):
"""A package an interrupted package manager run left halfway."""
name: str
#: dpkg's word for it: "half-installed", "unpacked", "half-configured",
#: "triggers-awaited", "triggers-pending", or "reinstall-required".
#: The manager's word for it: dpkg's and opkg's ("unpacked", "half-configured", ...),
#: rpm's "duplicate", apk's "broken-files" and "broken-script" (``package_managers``).
state: str
@@ -111,9 +111,9 @@ class HostStatusDict(TypedDict):
reboot_reason: str | None
#: True when the host installs updates on its own (unattended-upgrades, dnf-automatic).
auto_updates: bool | None
#: True when a package manager run was cut off and has to be finished
#: (``dpkg --configure -a``) before anything else installs; None when it
#: cannot tell, or a package manager is running right now.
#: The package manager read (dpkg, dnf, yum, apk, opkg; None: none) and whether a run
#: of it was cut off, left halfway (None: cannot tell, or one is running right now).
package_manager: NotRequired[str | None]
package_manager_interrupted: NotRequired[bool | None]
#: The packages that run left halfway; empty when only its journal says so.
interrupted_packages: NotRequired[list[InterruptedPackageDict]]
+194
View File
@@ -0,0 +1,194 @@
"""Whether dnf or yum, apk or opkg was left halfway: the host status's answer for
a host without dpkg (NetOrk/netork#851).
Read the same way as dpkg's state (:mod:`napalm_device_types.host_status`): in the
one host status command, without root, and changing nothing. Each manager is asked
only where it exists, and while one of its processes runs -- from
``/proc/<pid>/comm``, which every Linux has and BusyBox's ``ps`` cannot narrow to --
its packages are halfway on purpose and the answer is unknown.
**dnf and yum** (rpm). A transaction cut off after rpm installed the new version
of a package and before it erased the old one leaves both in the rpm database:
what ``dnf check --duplicates`` and ``package-cleanup --dupes`` report. Packages
dnf installs side by side on purpose do not count: its default ``installonlypkgs``
(``kernel``, ``installonlypkg(kernel)`` and the rest), plus any the host adds in
``dnf.conf``, asked of rpm by what they provide; and ``gpg-pubkey``, one entry
per imported key. Also interrupted is a transaction that never ended: dnf 4's
history (libdnf's ``trans`` table) holds its row with ``state`` 0 until
``finish()`` writes 1 (done) or 2 (error), so the newest row at 0 is one cut off;
yum leaves a ``transaction-all.*`` journal in ``/var/lib/yum``, which
``yum-complete-transaction`` reads (one renamed ``.disabled`` is given up). The
history is read with Python's sqlite3 module in read-only mode, as a stock EL host
has no ``sqlite3`` program and a ``dnf`` command loads every plugin and may reach
for its repositories. dnf 5's history lives elsewhere and is not read; its
duplicates are.
**apk** (Alpine, OpenWrt from 25.12). apk marks an installed package whose files
or scripts failed with ``f:`` in ``/lib/apk/db/installed`` -- ``f`` broken files,
``s`` a broken script; ``x`` (xattrs) and ``S`` (the checksum kind) are no fault --
and those are exactly the packages ``apk fix`` without arguments picks. A run
killed outright leaves no such mark; its next run usually repeats what it left.
**opkg** (OpenWrt before 25.12). ``/usr/lib/opkg/status`` holds
``Status: <want> <flags> <state>`` per package. A run cut off, or a postinst that
failed, leaves ``unpacked`` (or ``half-configured``, ``half-installed``), and the
``reinstreq`` flag asks for a reinstall; ``opkg configure`` finishes every unpacked
package.
What to do about it is not decided here.
"""
from __future__ import annotations
from typing import NamedTuple
from napalm_device_types.models import InterruptedPackageDict
#: dnf's default ``installonlypkgs``: names and what such packages provide.
_INSTALLONLY = (
"kernel",
"kernel-PAE",
"installonlypkg(kernel)",
"installonlypkg(kernel-module)",
"installonlypkg(vm)",
"multiversion(kernel)",
)
#: Where a host adds its own installonly packages, dnf's then yum's.
_DNF_CONF = "/etc/dnf/dnf.conf /etc/yum.conf"
DNF_HISTORY = "/var/lib/dnf/history.sqlite"
YUM_DIR = "/var/lib/yum"
APK_INSTALLED = "/lib/apk/db/installed"
OPKG_STATUS = "/usr/lib/opkg/status"
#: manager -> the process names that leave its packages halfway while they run.
_RUNNING = {
"rpm": frozenset({"rpm", "dnf", "dnf5", "yum", "dnf-automatic", "packagekitd"}),
"apk": frozenset({"apk"}),
"opkg": frozenset({"opkg"}),
}
_UNFINISHED = "0"
_YUM_JOURNAL = "transaction-all."
_YUM_GIVEN_UP = ".disabled"
_APK_FLAGS = (("f", "broken-files"), ("s", "broken-script"))
_OPKG_HALFWAY = frozenset({"unpacked", "half-configured", "half-installed"})
_OPKG_REINSTALL = "reinstreq"
#: The newest transaction in dnf's history as ``<id> <state>``; nothing when unreadable.
#: One literal, with :data:`DNF_HISTORY` spelt out: a fixed query, nothing put into it.
_DNF_HISTORY_PY = (
"import sqlite3; "
"c = sqlite3.connect('file:/var/lib/dnf/history.sqlite?mode=ro', uri=True); "
"r = c.execute('SELECT id, state FROM trans ORDER BY id DESC LIMIT 1').fetchone(); "
"r and print(*r)"
)
_RUNNING_PART = (
"echo '[pm-running]'; cat /proc/[0-9]*/comm 2>/dev/null | grep -xE '"
+ "|".join(sorted(set().union(*_RUNNING.values())))
+ "' | sort -u; "
)
_RPM_PART = (
"if command -v rpm >/dev/null 2>&1 && { command -v dnf || command -v yum; } >/dev/null 2>&1; "
"then echo '[rpm-frontend]'; { command -v dnf || command -v yum; } | sed 's#.*/##'; "
f"echo '[rpm-installonly]'; io=\"{' '.join(_INSTALLONLY)} "
f"$(sed -n 's/^[[:space:]]*installonlypkgs[[:space:]]*=//p' {_DNF_CONF} 2>/dev/null "
"| tr ',' ' ')\"; rpm -q --whatprovides $io --qf '%{NAME}\\n' 2>/dev/null "
"| grep -v ' ' | sort -u; "
"echo '[rpm-dupes]'; rpm -qa --qf '%{NAME} %{ARCH}\\n' 2>/dev/null "
"| grep -v '^gpg-pubkey ' | sort | uniq -d | head -n 50; "
f"echo '[yum-journal]'; ls -1 {YUM_DIR} 2>/dev/null | grep '^transaction-all\\.' "
"| head -n 20; "
"echo '[dnf-history]'; for py in /usr/libexec/platform-python python3; do "
f'if command -v $py >/dev/null 2>&1; then $py -c "{_DNF_HISTORY_PY}" 2>/dev/null; '
"break; fi; done; fi; "
)
_APK_PART = (
f"if command -v apk >/dev/null 2>&1 && [ -r {APK_INSTALLED} ]; then echo '[apk-audit]'; "
"awk '/^P:/ {p = substr($0, 3)} /^f:/ {f = substr($0, 3)} "
'/^$/ {if (f ~ /[fs]/) print f, p; p = ""; f = ""} END {if (f ~ /[fs]/) print f, p}\' '
f"{APK_INSTALLED} | head -n 50; fi; "
)
_OPKG_PART = (
f"if command -v opkg >/dev/null 2>&1 && [ -r {OPKG_STATUS} ]; then echo '[opkg-audit]'; "
"awk '/^Package:/ {p = $2} /^Status:/ && "
"($4 ~ /^(unpacked|half-configured|half-installed)$/ || $3 ~ /reinstreq/) "
f"{{print $3, $4, p}}' {OPKG_STATUS} | head -n 50; fi; "
)
#: The part of the host status command that reads them; POSIX ``sh``, BusyBox too.
PACKAGE_MANAGERS_COMMAND = _RUNNING_PART + _RPM_PART + _APK_PART + _OPKG_PART
class PackageManagerReading(NamedTuple):
"""One package manager's state: halfway or not, None while it is running."""
manager: str | None
interrupted: bool | None
packages: list[InterruptedPackageDict]
def _unfinished_transaction(sections: dict[str, list[str]]) -> bool:
newest = (sections.get("dnf-history") or [""])[0].split()
if len(newest) == 2 and newest[1] == _UNFINISHED:
return True
return any(
name.startswith(_YUM_JOURNAL) and not name.endswith(_YUM_GIVEN_UP)
for name in sections.get("yum-journal") or []
)
def _rpm(sections: dict[str, list[str]], running: set[str]) -> PackageManagerReading | None:
if "rpm-dupes" not in sections:
return None
manager = (sections.get("rpm-frontend") or ["dnf"])[0]
if running & _RUNNING["rpm"]:
return PackageManagerReading(manager, None, [])
installonly = set(sections.get("rpm-installonly") or [])
names = dict.fromkeys(line.split()[0] for line in sections["rpm-dupes"])
packages: list[InterruptedPackageDict] = [
{"name": name, "state": "duplicate"} for name in names if name not in installonly
]
return PackageManagerReading(
manager, bool(packages) or _unfinished_transaction(sections), packages
)
def _apk(sections: dict[str, list[str]], running: set[str]) -> PackageManagerReading | None:
if "apk-audit" not in sections:
return None
if running & _RUNNING["apk"]:
return PackageManagerReading("apk", None, [])
packages: list[InterruptedPackageDict] = []
for line in sections["apk-audit"]:
flags, _, name = line.partition(" ")
name = name.strip()
packages += [
{"name": name, "state": state} for flag, state in _APK_FLAGS if name and flag in flags
]
return PackageManagerReading("apk", bool(packages), packages)
def _opkg_package(line: str) -> InterruptedPackageDict | None:
parts = line.split()
if len(parts) != 3:
return None
flags, state, name = parts
if _OPKG_REINSTALL in flags.split(","):
return {"name": name, "state": "reinstall-required"}
return {"name": name, "state": state} if state in _OPKG_HALFWAY else None
def _opkg(sections: dict[str, list[str]], running: set[str]) -> PackageManagerReading | None:
if "opkg-audit" not in sections:
return None
if running & _RUNNING["opkg"]:
return PackageManagerReading("opkg", None, [])
packages = [p for p in map(_opkg_package, sections["opkg-audit"]) if p is not None]
return PackageManagerReading("opkg", bool(packages), packages)
def parse_package_managers(sections: dict[str, list[str]]) -> list[PackageManagerReading]:
"""What :data:`PACKAGE_MANAGERS_COMMAND` found, one reading per manager it asked."""
running = set(sections.get("pm-running") or [])
readings = (read(sections, running) for read in (_rpm, _apk, _opkg))
return [reading for reading in readings if reading is not None]
+69
View File
@@ -0,0 +1,69 @@
"""The backups running on a device right now.
A restart aborts every backup that runs on the device it restarts: a
hypervisor's backup of its guests, a backup server's backup, sync, verify or
garbage collection, a host's own file-level backup (restic, borg and the like).
A guest's restart breaks the backup its hypervisor is taking of it. So an
application that restarts a device asks first, right before the restart.
Declared under ``if TYPE_CHECKING``: a driver mixes this in when it can tell;
``hasattr`` tells a caller whether it did. No role base carries it, and a driver
without it cannot tell -- which is not the same as "nothing runs".
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Literal, TypedDict, get_args
#: What a running backup does: the backup itself, or one of a backup server's
#: jobs (``napalm_device_types.backups.BackupJobKind``).
RunningBackupKind = Literal["backup", "sync", "verify", "gc", "prune", "tape"]
RUNNING_BACKUP_KINDS: tuple[str, ...] = get_args(RunningBackupKind)
class RunningBackupDict(TypedDict):
"""One backup running on the device (``get_running_backups``)."""
kind: RunningBackupKind
job: str | None # the job it runs for, as the platform names it; None: started by hand
task: str | None # this run's id on the platform: a task id, a unit, a process id
guests: list[str] # the guests (get_vms' ids) it is backing up now; [] for the host's own
since: int | None # unix seconds it started; None when the platform does not say
class RunningBackupsMixin:
if TYPE_CHECKING:
def get_running_backups(self) -> list[RunningBackupDict]:
"""
Returns the backups running on the device now; ``[]`` when none is.
Every entry runs on this device: restarting the device aborts it.
A hypervisor names in ``guests`` the guests a backup is taking right
now -- a guest that is locked for its backup counts, with or without
a task the platform still shows -- so the application can tell a
guest's own restart not to break it.
Example::
[
{
"kind": "backup",
"job": "backup-3f2a9c1e-77d1",
"task": "UPID:pve1:0001F2A3:00C4B1D2:6731A0C0:vzdump::root@pam:",
"guests": ["101"],
"since": 1793660400,
},
{
"kind": "gc",
"job": "store1",
"task": "UPID:localhost:000004D2:...:garbage_collection:store1:root@pam:",
"guests": [],
"since": 1793664000,
},
]
:raises Exception: when the device could not be read; never an
empty list for a failed read.
"""
...
+7 -2
View File
@@ -53,6 +53,8 @@ from typing import (
TYPE_CHECKING,
)
from napalm_device_types.ssh_server_auth import SSHD_EFFECTIVE_COMMAND, SshServerAuthMixin
if TYPE_CHECKING: # pragma: no cover - for type checkers only
from napalm_device_types.channel import CommandResult
@@ -420,7 +422,7 @@ def _unchanged(
return result
class OpenSshServerMixin:
class OpenSshServerMixin(SshServerAuthMixin):
"""Adds :meth:`harden_ssh_server` to a driver that can run commands as root on an OpenSSH host.
Mixed in by the drivers whose host runs a stock OpenSSH, not by a base class,
@@ -430,6 +432,9 @@ class OpenSshServerMixin:
``harden_ssh_server = None``: its file would be overwritten on the next
apply. Where that generated file still includes the drop-in directory, the
driver sets :attr:`SSHD_CONFIG_GENERATED` instead and keeps the hardening.
It carries :class:`~napalm_device_types.ssh_server_auth.SshServerAuthMixin`:
a host whose sshd can be hardened can also say how it lets people log in.
"""
#: True where the platform writes sshd_config from its own database
@@ -513,7 +518,7 @@ class OpenSshServerMixin:
return parse_sshd_effective(self._sshd_effective_text())
def _sshd_effective_text(self) -> str:
result = self._ssh_run("PATH=$PATH:/usr/sbin:/sbin sshd -T")
result = self._ssh_run(SSHD_EFFECTIVE_COMMAND)
if result.exit_code != 0:
raise RuntimeError(f"sshd -T failed: {(result.stderr or result.stdout).strip()}")
return result.stdout
+281
View File
@@ -0,0 +1,281 @@
"""How an OpenSSH server lets people log in: the effective settings and sshd's PAM stack.
Whether a host takes a password, only a key, or a key *and* a second factor is
the same question on every host that runs OpenSSH, so the reading lives here,
and a driver only mixes in :class:`SshServerAuthMixin` and supplies
``run_command``.
**What sshd uses, not what a file says.** ``sshd -T`` prints the settings after
sshd's defaults, ``sshd_config``, every ``Include`` and first-match-wins -- the
same reading :class:`~napalm_device_types.ssh_server.OpenSshServerMixin` plans
its hardening on, by the same fixed command (:data:`SSHD_EFFECTIVE_COMMAND`). It
needs root: without it sshd cannot load its host keys and exits.
**PAM, as far as it can be read.** With ``UsePAM yes``, keyboard-interactive
asks whatever PAM's ``auth`` stack for sshd asks: a password from
``pam_unix.so``, a one-time code from an OTP module, or both. The stack is read
from ``/etc/pam.d`` in one listing as the login user (the files are
world-readable), and followed through ``@include``, ``include`` and
``substack``. Each module is reported with its control and arguments as
written; which of them is a second factor, and whether ``nullok`` lets a user
without one through, is the consumer's call.
Not read: ``Match`` blocks (``sshd -T`` without ``-C`` prints the global
settings), and what a PAM module's own configuration asks for.
"""
from __future__ import annotations
import re
from collections.abc import Mapping
from typing import TYPE_CHECKING, TypedDict
if TYPE_CHECKING: # pragma: no cover - for type checkers only
from napalm_device_types.channel import CommandResult
#: sshd's effective configuration. ``sshd`` lives in ``/usr/sbin``, outside a login
#: user's ``PATH``. One fixed command, run as root.
SSHD_EFFECTIVE_COMMAND = "PATH=$PATH:/usr/sbin:/sbin sshd -T"
PAM_DIR = "/etc/pam.d"
#: Every file in ``/etc/pam.d``, each after a ``==> path <==`` header (GNU, BSD and
#: busybox ``tail`` alike). One fixed command, as the login user.
PAM_FILES_COMMAND = f"tail -n +1 -- {PAM_DIR}/*"
#: The PAM service sshd runs as.
PAM_SERVICE = "sshd"
_TIMEOUT = 30
_HEADER_RE = re.compile(rf"^==> {re.escape(PAM_DIR)}/(?P<name>[^/]+) <==$")
#: A service name an include may name: a file in ``/etc/pam.d``, never a path out of it.
_SERVICE_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
_INCLUDE_CONTROLS = ("include", "substack")
class PamAuthEntryDict(TypedDict):
"""One line of a PAM ``auth`` stack: ``auth required pam_unix.so nullok``."""
#: The module's file name, without its directory: ``pam_unix.so``.
module: str
#: ``required``, ``requisite``, ``sufficient``, ``optional``, or a bracketed
#: ``[value=action ...]`` as written.
control: str
args: list[str]
class SshServerAuthDict(TypedDict):
"""How an OpenSSH server lets people log in, as ``get_ssh_server_auth`` read it.
The global settings ``sshd -T`` prints. A ``Match`` block can change them for
some users or addresses; ``sshd -T`` without ``-C`` does not show that.
``authentication_methods`` is ``AuthenticationMethods``: each inner list is one
way in, every method of it required in turn (``publickey,keyboard-interactive``
-> ``["publickey", "keyboard-interactive"]``); empty for ``any``, where any one
enabled method is enough. A method keeps its submethod
(``keyboard-interactive:pam``).
``use_pam`` is None where sshd has no PAM (OpenBSD). ``pam_auth_stack`` is the
``auth`` stack PAM runs for sshd, includes and substacks followed; None when
PAM is off or its files gave nothing for sshd.
"""
password_authentication: bool
#: ``KbdInteractiveAuthentication``; before OpenSSH 8.7, either it or
#: ``ChallengeResponseAuthentication``.
kbd_interactive_authentication: bool
pubkey_authentication: bool
hostbased_authentication: bool
#: False also where sshd was built without GSSAPI.
gssapi_authentication: bool
authentication_methods: list[list[str]]
#: ``yes``, ``prohibit-password`` (``without-password``), ``forced-commands-only`` or ``no``.
permit_root_login: str
use_pam: bool | None
pam_auth_stack: list[PamAuthEntryDict] | None
def _flag(settings: Mapping[str, str], keyword: str) -> bool:
return settings.get(keyword, "no") == "yes"
def _settings(text: str) -> dict[str, str]:
"""``keyword value`` per line, the first one of each keyword."""
found: dict[str, str] = {}
for raw in text.splitlines():
parts = raw.strip().split(None, 1)
if len(parts) == 2:
found.setdefault(parts[0].lower(), parts[1].strip())
return found
def _methods(value: str) -> list[list[str]]:
if value == "any":
return []
return [[m for m in alternative.split(",") if m] for alternative in value.split()]
def parse_sshd_auth(text: str) -> SshServerAuthDict:
"""The authentication settings from ``sshd -T`` output; ``pam_auth_stack`` left None.
:raises ValueError: if a setting every OpenSSH prints is missing -- not sshd -T
output, or cut short.
"""
settings = _settings(text)
missing = [
k
for k in ("passwordauthentication", "pubkeyauthentication", "permitrootlogin")
if k not in settings
]
if missing:
raise ValueError(f"sshd -T printed no {', '.join(missing)}")
use_pam = settings.get("usepam")
return {
"password_authentication": _flag(settings, "passwordauthentication"),
"kbd_interactive_authentication": _flag(settings, "kbdinteractiveauthentication")
or _flag(settings, "challengeresponseauthentication"),
"pubkey_authentication": _flag(settings, "pubkeyauthentication"),
"hostbased_authentication": _flag(settings, "hostbasedauthentication"),
"gssapi_authentication": _flag(settings, "gssapiauthentication"),
"authentication_methods": _methods(settings.get("authenticationmethods", "any")),
"permit_root_login": settings["permitrootlogin"],
"use_pam": None if use_pam is None else use_pam == "yes",
"pam_auth_stack": None,
}
def _pam_tokens(line: str) -> list[str]:
"""A PAM line's tokens, a bracketed control (``[success=1 default=ignore]``) as one."""
tokens: list[str] = []
for part in line.split():
if tokens and tokens[-1].startswith("[") and not tokens[-1].endswith("]"):
tokens[-1] += " " + part
else:
tokens.append(part)
return tokens
def _pam_entry(line: str) -> PamAuthEntryDict | None:
tokens = _pam_tokens(line.split("#", 1)[0])
if len(tokens) >= 2 and tokens[0] == "@include":
tokens = ["auth", "include", tokens[1]]
if len(tokens) < 3 or tokens[0].lstrip("-") != "auth":
return None
control, target, args = tokens[1], tokens[2], tokens[3:]
if control in _INCLUDE_CONTROLS:
if not _SERVICE_RE.match(target):
return None
return {"module": target, "control": control, "args": []}
return {"module": target.rsplit("/", 1)[-1], "control": control, "args": args}
def parse_pam_auth(text: str) -> list[PamAuthEntryDict]:
"""One PAM file's ``auth`` stack, in order.
A module by its file name (``pam_unix.so``), with its control and arguments.
An include is an entry whose control is ``include`` or ``substack`` and whose
module is the service it names: ``@include common-auth`` brings in a whole
file, ``auth include`` and ``auth substack`` its ``auth`` lines; both are
followed by :func:`pam_auth_stack`. An include that names a path rather than
a service is left out.
"""
return [entry for line in text.splitlines() if (entry := _pam_entry(line.strip()))]
def parse_pam_files(text: str) -> dict[str, str]:
""":data:`PAM_FILES_COMMAND`'s output, by file name."""
files: dict[str, list[str]] = {}
current: list[str] | None = None
for line in text.splitlines():
header = _HEADER_RE.match(line.strip())
if header:
current = files.setdefault(header["name"], [])
elif current is not None:
current.append(line)
return {name: "\n".join(lines).strip("\n") + "\n" for name, lines in files.items()}
def pam_auth_stack(
files: Mapping[str, str], service: str = PAM_SERVICE
) -> list[PamAuthEntryDict] | None:
"""*service*'s ``auth`` modules, includes and substacks followed, in stack order.
A file included twice -- or in a loop -- counts once; an include of a file
that is not there adds nothing, as with PAM. None without a file for *service*.
"""
if service not in files:
return None
stack: list[PamAuthEntryDict] = []
seen: set[str] = set()
def walk(name: str) -> None:
if name in seen or name not in files:
return
seen.add(name)
for entry in parse_pam_auth(files[name]):
if entry["control"] in _INCLUDE_CONTROLS:
walk(entry["module"])
else:
stack.append(entry)
walk(service)
return stack
class SshServerAuthMixin:
"""Adds :meth:`get_ssh_server_auth` to a driver that can run commands on an OpenSSH host.
Mixed in by the drivers whose host runs a stock OpenSSH --
:class:`~napalm_device_types.ssh_server.OpenSshServerMixin` carries it -- so
``hasattr(driver, "get_ssh_server_auth")`` stays a truthful answer. Reading,
not writing: a platform that generates sshd_config itself can carry it too.
"""
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
def run_command(
self,
command: str,
*,
privileged: bool = False,
timeout: float = 60,
stdin: bytes | None = None,
) -> CommandResult: ...
def get_ssh_server_auth(self) -> SshServerAuthDict:
"""
Returns how sshd lets people log in, from ``sshd -T`` and PAM's sshd stack.
Example::
{
"password_authentication": False,
"kbd_interactive_authentication": True,
"pubkey_authentication": True,
"hostbased_authentication": False,
"gssapi_authentication": False,
"authentication_methods": [["publickey", "keyboard-interactive"]],
"permit_root_login": "no",
"use_pam": True,
"pam_auth_stack": [
{"module": "pam_google_authenticator.so", "control": "required", "args": []}
],
}
:raises RuntimeError: if ``sshd -T`` fails -- no OpenSSH, or no root.
"""
result = self.run_command(SSHD_EFFECTIVE_COMMAND, privileged=True, timeout=_TIMEOUT)
if result.exit_code != 0:
raise RuntimeError(f"sshd -T failed: {(result.stderr or result.stdout).strip()}")
auth = parse_sshd_auth(result.stdout)
if auth["use_pam"]:
auth["pam_auth_stack"] = self._pam_auth_stack()
return auth
def _pam_auth_stack(self) -> list[PamAuthEntryDict] | None:
"""sshd's PAM auth stack; None when ``/etc/pam.d`` gave nothing for sshd.
``tail`` goes on past a file it cannot open and exits 1, so what it did
print is used whatever the exit status.
"""
result = self.run_command(PAM_FILES_COMMAND, privileged=False, timeout=_TIMEOUT)
return pam_auth_stack(parse_pam_files(result.stdout))
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "napalm-device-types"
version = "4.30.0"
version = "4.34.0"
description = "Abstract device-type base classes for NAPALM drivers"
readme = "README.md"
requires-python = ">=3.12"
+376
View File
@@ -13,6 +13,8 @@ from __future__ import annotations
import os
import re
import shutil
import sqlite3
import subprocess
import pytest
@@ -25,6 +27,12 @@ from napalm_device_types.host_status import (
kernel_reboot_pending,
parse_host_status,
)
from napalm_device_types.package_managers import (
APK_INSTALLED,
DNF_HISTORY,
OPKG_STATUS,
PACKAGE_MANAGERS_COMMAND,
)
def _wire(
@@ -570,3 +578,371 @@ class TestFilesystems:
mountpoints = [fs["mountpoint"] for fs in parse_host_status(out.stdout)["filesystems"]]
assert "/" in mountpoints
def _other(*sections: tuple[str, tuple[str, ...]], running: tuple[str, ...] = ()) -> str:
"""A report from a host without dpkg: the given sections before ``[timers]``."""
lines = ["[pm-running]", *running]
for name, body in sections:
lines += [f"[{name}]", *body]
return _wire().replace("[timers]", "\n".join([*lines, "[timers]"]))
class TestInterruptedRpm:
"""dnf and yum hosts: a transaction rpm was cut off in leaves the old and the
new version of a package both installed (``dnf check --duplicates``,
``package-cleanup --dupes``), and dnf's history records a transaction that
never ended. netOrk only reports it: there is no one repair that is safe on
every host (NetOrk/netork#851, decided 2026-11-07).
The outputs are shaped after the documented formats -- ``rpm -qa --qf``,
libdnf's ``trans`` table (``state`` 0 = UNKNOWN until ``finish()`` sets DONE
1 or ERROR 2), yum's ``transaction-all.*`` journal that
``yum-complete-transaction`` reads -- and still have to be checked against a
real Oracle Linux 9 host.
"""
INSTALLONLY = ("kernel-core", "kernel-modules", "kernel-uek-core")
#: An upgrade of glibc and openssl-libs cut off after the new versions were in.
CUT_OFF = ("glibc x86_64", "kernel-uek-core x86_64", "openssl-libs x86_64")
def _rpm(self, dupes=(), history=("42 1",), journal=(), frontend="dnf", running=()):
return _other(
("rpm-frontend", (frontend,)),
("rpm-installonly", self.INSTALLONLY),
("rpm-dupes", dupes),
("yum-journal", journal),
("dnf-history", history),
running=running,
)
def test_a_clean_rpm_database_is_not_interrupted(self):
status = parse_host_status(self._rpm())
assert status["package_manager_interrupted"] is False
assert status["interrupted_packages"] == []
assert status["package_manager"] == "dnf"
def test_two_versions_of_a_package_left_by_a_cut_off_transaction(self):
status = parse_host_status(self._rpm(dupes=self.CUT_OFF))
assert status["package_manager_interrupted"] is True
assert status["interrupted_packages"] == [
{"name": "glibc", "state": "duplicate"},
{"name": "openssl-libs", "state": "duplicate"},
]
def test_kernels_are_installed_side_by_side_on_purpose(self):
"""dnf's installonly packages: several versions are the rule, not a fault."""
status = parse_host_status(self._rpm(dupes=("kernel-uek-core x86_64",)))
assert status["package_manager_interrupted"] is False
def test_a_package_duplicated_in_two_arches_is_named_once(self):
status = parse_host_status(self._rpm(dupes=("glibc i686", "glibc x86_64")))
assert status["interrupted_packages"] == [{"name": "glibc", "state": "duplicate"}]
def test_the_last_transaction_never_finished(self):
status = parse_host_status(self._rpm(history=("43 0",)))
assert status["package_manager_interrupted"] is True
assert status["interrupted_packages"] == []
@pytest.mark.parametrize("history", [("44 1",), ("45 2",), ()])
def test_a_finished_or_unreadable_history_says_nothing(self, history):
"""A failed transaction (ERROR) did end; an unreadable history is no answer."""
assert parse_host_status(self._rpm(history=history))["package_manager_interrupted"] is False
def test_yum_left_its_transaction_journal(self):
journal = ("transaction-all.2026-11-01.10:12.33", "transaction-done.2026-11-01.10:12.33")
status = parse_host_status(self._rpm(frontend="yum", history=(), journal=journal))
assert status["package_manager_interrupted"] is True
assert status["package_manager"] == "yum"
def test_a_journal_yum_gave_up_on_is_none(self):
"""yum-complete-transaction skips a journal it renamed ``.disabled``."""
journal = ("transaction-all.2026-11-01.10:12.33.disabled",)
status = parse_host_status(self._rpm(frontend="yum", history=(), journal=journal))
assert status["package_manager_interrupted"] is False
@pytest.mark.parametrize("name", ["rpm", "dnf", "dnf5", "yum", "dnf-automatic", "packagekitd"])
def test_a_running_transaction_makes_it_unknown(self, name):
status = parse_host_status(self._rpm(dupes=self.CUT_OFF, running=(name,)))
assert status["package_manager_interrupted"] is None
assert status["interrupted_packages"] == []
assert status["package_manager"] == "dnf"
def test_another_managers_run_does_not(self):
status = parse_host_status(self._rpm(dupes=self.CUT_OFF, running=("opkg",)))
assert status["package_manager_interrupted"] is True
def test_the_command_asks_rpm_only_where_dnf_or_yum_exists(self):
assert "command -v rpm" in HOST_STATUS_COMMAND
assert "rpm -qa --qf" in HOST_STATUS_COMMAND
assert f"file:{DNF_HISTORY}?mode=ro" in HOST_STATUS_COMMAND
def test_it_never_repairs(self):
for word in ("distro-sync", "--cleandupes", "remove", "yum-complete-transaction", "rpm -e"):
assert word not in HOST_STATUS_COMMAND
class TestInterruptedApk:
"""Alpine and OpenWrt 25.12: apk marks a package whose files or scripts failed
in ``/lib/apk/db/installed`` (``f:`` with ``f`` broken files, ``s`` a broken
script; ``x`` xattrs and ``S`` the checksum kind are no fault), and ``apk fix``
with no arguments is what picks exactly those (apk-tools ``app_fix.c``).
netOrk only reports it (NetOrk/netork#851). Shaped after apk-tools'
``database.c``; to be checked against a real OpenWrt 25.12 host."""
def _apk(self, audit=(), running=()):
return _other(("apk-audit", audit), running=running)
def test_a_clean_database_is_not_interrupted(self):
status = parse_host_status(self._apk())
assert status["package_manager_interrupted"] is False
assert status["package_manager"] == "apk"
@pytest.mark.parametrize(
("flags", "states"),
[
("s", ["broken-script"]),
("f", ["broken-files"]),
("fsS", ["broken-files", "broken-script"]),
],
)
def test_a_package_apk_marked_broken(self, flags, states):
status = parse_host_status(self._apk(audit=(f"{flags} busybox",)))
assert status["package_manager_interrupted"] is True
assert status["interrupted_packages"] == [
{"name": "busybox", "state": state} for state in states
]
def test_a_running_apk_makes_it_unknown(self):
status = parse_host_status(self._apk(audit=("s busybox",), running=("apk",)))
assert status["package_manager_interrupted"] is None
def test_the_command_reads_the_installed_database(self):
assert "/lib/apk/db/installed" in HOST_STATUS_COMMAND
assert "apk fix" not in HOST_STATUS_COMMAND
class TestInterruptedOpkg:
"""OpenWrt before 25.12: opkg writes ``Status: <want> <flags> <state>`` per
package to ``/usr/lib/opkg/status``; a package whose run was cut off or whose
postinst failed stays ``unpacked``, and ``opkg configure`` finishes every
unpacked package (opkg-lede ``opkg_cmd.c``, ``pkg.c``). netOrk's update run
does that first (NetOrk/netork#851). To be checked against a real router."""
def _opkg(self, audit=(), running=()):
return _other(("opkg-audit", audit), running=running)
def test_a_clean_status_file_is_not_interrupted(self):
status = parse_host_status(self._opkg())
assert status["package_manager_interrupted"] is False
assert status["package_manager"] == "opkg"
@pytest.mark.parametrize(
("line", "state"),
[
("user unpacked luci-app-sqm", "unpacked"),
("ok half-configured luci-app-sqm", "half-configured"),
("ok half-installed luci-app-sqm", "half-installed"),
("reinstreq,user installed luci-app-sqm", "reinstall-required"),
],
)
def test_every_state_an_interrupted_run_leaves(self, line, state):
status = parse_host_status(self._opkg(audit=(line,)))
assert status["package_manager_interrupted"] is True
assert status["interrupted_packages"] == [{"name": "luci-app-sqm", "state": state}]
def test_a_running_opkg_makes_it_unknown(self):
status = parse_host_status(self._opkg(audit=("ok unpacked x",), running=("opkg",)))
assert status["package_manager_interrupted"] is None
def test_the_command_reads_the_status_file_and_never_configures(self):
assert "/usr/lib/opkg/status" in HOST_STATUS_COMMAND
assert "opkg configure" not in HOST_STATUS_COMMAND
class TestPackageManager:
def test_dpkg_says_so(self):
assert parse_host_status(_wire(dpkg=()))["package_manager"] == "dpkg"
def test_a_busy_dpkg_still_says_which(self):
status = parse_host_status(_wire(dpkg=(), busy=True))
assert (status["package_manager"], status["package_manager_interrupted"]) == (
"dpkg",
None,
)
def test_no_package_manager_read_is_none(self):
status = parse_host_status(_wire())
assert status["package_manager"] is None
assert status["package_manager_interrupted"] is None
#: An rpm that answers the two queries the command asks, as on an EL host whose
#: glibc upgrade was cut off; and a dnf that only has to exist.
_RPM_STUB = """#!/bin/sh
case "$*" in
*--whatprovides*) printf '%s\\n' kernel-core kernel-core 'no package provides kernel-PAE' ;;
-qa*) printf '%s\\n' 'glibc x86_64' 'kernel-core x86_64' 'gpg-pubkey (none)' \\
'glibc x86_64' 'kernel-core x86_64' 'gpg-pubkey (none)' 'bash x86_64' ;;
esac
"""
#: /lib/apk/db/installed shaped after apk-tools' database.c: busybox's script failed.
_APK_DB = """C:Q1abc=
P:alpine-baselayout
V:3.6.8-r1
A:x86_64
F:etc
R:fstab
C:Q1def=
P:busybox
V:1.37.0-r12
A:x86_64
f:s
F:bin
R:busybox
C:Q1ghi=
P:musl
V:1.2.5-r9
f:S
"""
#: /usr/lib/opkg/status as opkg writes it: luci-app-sqm's postinst never ran.
_OPKG_STATUS = """Package: libc
Version: 1.2.4-4
Depends: libgcc1
Status: install hold installed
Architecture: mipsel_24kc
Installed-Time: 1717000000
Package: luci-app-sqm
Version: 1.5.3-r1
Depends: libc, sqm-scripts
Status: install user unpacked
Architecture: all
Installed-Time: 1730000000
"""
def _stubs(tmp_path, *names: str, rpm: bool = False) -> dict:
bin_dir = tmp_path / "bin"
bin_dir.mkdir()
for name in names:
stub = bin_dir / name
stub.write_text(_RPM_STUB if name == "rpm" else "#!/bin/sh\nexit 0\n")
stub.chmod(0o755)
return {**os.environ, "PATH": f"{bin_dir}:/usr/bin:/bin"}
def _run(command: str, env: dict, shell: tuple[str, ...]) -> str:
report = f"printf '%s%s\\n' HSTAT_ BEGIN; {command} printf '%s%s\\n' HSTAT_ END"
program = shutil.which(shell[0]) or shell[0]
return subprocess.run(
[program, *shell[1:], report], capture_output=True, text=True, env=env, timeout=30
).stdout
_SHELLS = [("sh", "-c")] + ([("busybox", "sh", "-c")] if shutil.which("busybox") else [])
def _skip_busybox_rpm(shell: tuple[str, ...]) -> None:
"""A BusyBox built with its own ``rpm`` applet runs that before any stand-in."""
if shell[0] == "busybox":
applets = subprocess.run(["busybox", "--list"], capture_output=True, text=True).stdout
if "rpm" in applets.split():
pytest.skip("this busybox has an rpm applet of its own")
@pytest.mark.parametrize("shell", _SHELLS)
class TestThePackageManagerCommandOnARealShell:
"""The command part run for real, with stand-in programs and files."""
def test_rpm_duplicates_without_the_kernel_and_the_keys(self, tmp_path, shell):
_skip_busybox_rpm(shell)
env = _stubs(tmp_path, "rpm", "dnf")
out = _run(PACKAGE_MANAGERS_COMMAND, env, shell)
status = parse_host_status(out)
assert status["package_manager"] == "dnf"
assert status["interrupted_packages"] == [{"name": "glibc", "state": "duplicate"}]
def test_dnfs_last_transaction_that_never_ended(self, tmp_path, shell):
_skip_busybox_rpm(shell)
history = tmp_path / "history.sqlite"
with sqlite3.connect(history) as db:
db.execute("CREATE TABLE trans (id INTEGER PRIMARY KEY, state INTEGER NOT NULL)")
db.executemany("INSERT INTO trans VALUES (?, ?)", [(41, 1), (42, 0)])
env = _stubs(tmp_path, "rpm", "dnf")
out = _run(PACKAGE_MANAGERS_COMMAND.replace(DNF_HISTORY, str(history)), env, shell)
assert "[dnf-history]\n42 0\n" in out
def test_apk_names_the_package_it_marked(self, tmp_path, shell):
installed = tmp_path / "installed"
installed.write_text(_APK_DB)
env = _stubs(tmp_path, "apk")
out = _run(PACKAGE_MANAGERS_COMMAND.replace(APK_INSTALLED, str(installed)), env, shell)
status = parse_host_status(out)
assert status["package_manager"] == "apk"
assert status["interrupted_packages"] == [{"name": "busybox", "state": "broken-script"}]
def test_opkg_names_the_unpacked_package(self, tmp_path, shell):
status_file = tmp_path / "status"
status_file.write_text(_OPKG_STATUS)
env = _stubs(tmp_path, "opkg")
out = _run(PACKAGE_MANAGERS_COMMAND.replace(OPKG_STATUS, str(status_file)), env, shell)
status = parse_host_status(out)
assert status["package_manager"] == "opkg"
assert status["interrupted_packages"] == [{"name": "luci-app-sqm", "state": "unpacked"}]
def test_a_host_with_none_of_them_reads_none(self, tmp_path, shell):
env = _stubs(tmp_path)
env["PATH"] = str(tmp_path / "bin")
out = _run(PACKAGE_MANAGERS_COMMAND, env, shell)
assert parse_host_status(out)["package_manager"] is None
def test_the_running_processes_come_from_proc(self, tmp_path, shell):
"""The test runner's own process is there under its own name; no manager is."""
out = _run(PACKAGE_MANAGERS_COMMAND, _stubs(tmp_path), shell)
assert "[pm-running]\n" in out
class TestTheWholeCommand:
def test_it_runs_on_busybox(self):
"""OpenWrt's and Alpine's shell: still one intact report."""
if not shutil.which("busybox"):
pytest.skip("no busybox here")
out = subprocess.run(
["busybox", "sh", "-c", HOST_STATUS_COMMAND], capture_output=True, text=True, timeout=30
).stdout
assert parse_host_status(out)["filesystems"]
def test_the_package_managers_are_part_of_it(self):
assert PACKAGE_MANAGERS_COMMAND in HOST_STATUS_COMMAND
+158
View File
@@ -0,0 +1,158 @@
"""Where a host's logs go beyond it: rsyslog's forwarding, its file inputs, and
whether it gets the journal (NetOrk/netork#2053)."""
from __future__ import annotations
import pytest
from napalm_device_types.log_forwarding import journal_to_syslog, parse_rsyslog
DEBIAN = """\
module(load="imuxsock")
module(load="imklog")
$FileOwner root
$IncludeConfig /etc/rsyslog.d/*.conf
*.*;auth,authpriv.none\t\t-/var/log/syslog
auth,authpriv.*\t\t\t/var/log/auth.log
*.emerg\t\t\t\t:omusrmsg:*
"""
GRAYLOG = "*.* @graylog.example.net:1514;RSYSLOG_SyslogProtocol23Format\n"
def _rules(text: str) -> list[dict]:
return parse_rsyslog(text)["rules"]
class TestRules:
def test_a_debian_host_forwards_nothing(self):
assert _rules(DEBIAN) == []
def test_netorks_own_graylog_rule_forwards_everything_over_udp(self):
assert _rules(DEBIAN + GRAYLOG) == [
{
"protocol": "udp",
"target": "graylog.example.net",
"port": 1514,
"selector": "*.*",
"complete": True,
"ruleset": None,
},
]
def test_two_ats_are_tcp_and_a_missing_port_stays_unknown(self):
(rule,) = _rules("*.* @@logs.lan\n")
assert (rule["protocol"], rule["target"], rule["port"]) == ("tcp", "logs.lan", None)
def test_an_ipv6_target_keeps_its_brackets_off(self):
(rule,) = _rules("*.* @[2001:db8::5]:514\n")
assert (rule["target"], rule["port"]) == ("2001:db8::5", 514)
def test_a_narrower_selector_is_partial_and_says_which(self):
(rule,) = _rules("authpriv.* @10.0.0.5\n")
assert (rule["selector"], rule["complete"]) == ("authpriv.*", False)
def test_excluding_a_facility_is_partial_too(self):
(rule,) = _rules("*.*;mail.none @10.0.0.5\n")
assert rule["complete"] is False
def test_an_ampersand_line_forwards_what_the_line_before_selected(self):
rules = _rules("kern.* /var/log/kern.log\n& @10.0.0.5:514\n")
assert [(r["selector"], r["target"]) for r in rules] == [("kern.*", "10.0.0.5")]
def test_relp_is_its_own_protocol(self):
(rule,) = _rules("*.* :omrelp:10.0.0.6:2514\n")
assert (rule["protocol"], rule["target"], rule["port"]) == ("relp", "10.0.0.6", 2514)
def test_an_action_over_several_lines_is_read_whole(self):
text = '*.* action(type="omfwd"\n target="10.0.0.7" port="6514"\n protocol="tcp")\n'
(rule,) = _rules(text)
assert (rule["protocol"], rule["target"], rule["port"], rule["complete"]) == (
"tcp",
"10.0.0.7",
6514,
True,
)
def test_an_action_on_its_own_takes_every_message(self):
(rule,) = _rules('action(type="omfwd" Target="10.0.0.7")\n')
assert (rule["selector"], rule["complete"], rule["protocol"]) == ("*.*", True, "udp")
def test_an_action_under_a_condition_is_partial_and_names_it(self):
(rule,) = _rules(
'if $programname == \'nginx\' then action(type="omfwd" target="10.0.0.8")\n'
)
assert rule["complete"] is False
assert rule["selector"] == "$programname == 'nginx'"
def test_a_condition_whose_action_is_on_the_next_line_still_holds(self):
(rule,) = _rules('if $msg contains "x" then\n action(type="omfwd" target="10.0.0.14")\n')
assert (rule["selector"], rule["complete"]) == ('$msg contains "x"', False)
def test_a_block_under_a_condition_makes_each_of_its_actions_partial(self):
text = 'if $syslogseverity <= 4 then {\n action(type="omfwd" target="a")\n action(type="omrelp" target="b")\n}\n*.* @c\n'
rules = _rules(text)
assert [(r["target"], r["complete"]) for r in rules] == [
("a", False),
("b", False),
("c", True),
]
def test_a_property_filter_is_a_partial_selector(self):
(rule,) = _rules(':programname, isequal, "sshd" @10.0.0.9\n')
assert rule["complete"] is False
assert rule["selector"] == ':programname, isequal, "sshd"'
def test_an_action_in_a_ruleset_belongs_to_that_ruleset(self):
text = 'ruleset(name="remote") {\n action(type="omfwd" target="10.0.0.10" port="514")\n}\n'
(rule,) = _rules(text)
assert (rule["ruleset"], rule["complete"]) == ("remote", True)
def test_comments_are_no_rules(self):
assert _rules("# *.* @10.0.0.11\n*.* /var/log/all.log # @10.0.0.12\n") == []
def test_a_hash_inside_quotes_is_no_comment(self):
(rule,) = _rules('action(type="omfwd" target="10.0.0.13" template="a#b")\n')
assert rule["target"] == "10.0.0.13"
def test_a_local_action_is_no_forwarding(self):
assert _rules('action(type="omfile" file="/var/log/x.log")\n*.* /var/log/y.log\n') == []
class TestInputs:
def test_an_imfile_input_names_its_file_and_ruleset(self):
text = 'input(type="imfile" File="/var/log/nginx/*.log" Tag="nginx" Ruleset="remote")\n'
assert parse_rsyslog(text)["file_inputs"] == [
{"path": "/var/log/nginx/*.log", "ruleset": "remote"}
]
def test_a_legacy_imfile_input_goes_to_the_default_ruleset(self):
text = "$ModLoad imfile\n$InputFileName /var/log/app.log\n$InputFileTag app:\n$InputRunFileMonitor\n"
assert parse_rsyslog(text)["file_inputs"] == [{"path": "/var/log/app.log", "ruleset": None}]
class TestModules:
def test_imjournal_reads_the_journal(self):
assert parse_rsyslog('module(load="imjournal")\n')["imjournal"] is True
def test_legacy_modload_counts(self):
modules = parse_rsyslog("$ModLoad imuxsock\n")
assert (modules["imuxsock"], modules["imjournal"]) == (True, False)
class TestJournalToSyslog:
"""``ForwardToSyslog``: the last setting wins; with none, the shipped default
the distribution prints commented out in its ``journald.conf``."""
@pytest.mark.parametrize(
("lines", "expected"),
[
(["#ForwardToSyslog=yes"], True),
(["#ForwardToSyslog=no"], False),
(["#ForwardToSyslog=yes", "ForwardToSyslog=no"], False),
(["ForwardToSyslog=no", "ForwardToSyslog = yes"], True),
([], None),
],
)
def test_the_last_setting_wins_else_the_shipped_default(self, lines, expected):
assert journal_to_syslog(lines) is expected
+152 -1
View File
@@ -20,6 +20,7 @@ from napalm_device_types.log_sources import (
LOG_SOURCES_COMMAND,
LogSourcesMixin,
LogSourcesUnavailable,
parse_log_forwarding,
parse_log_sources,
)
@@ -44,6 +45,35 @@ OPEN = f"""/proc/610/fd/7 /var/log/syslog
/proc/1500/fd/3 /srv/app/log with space.log
/proc/1500/fd/4 /srv/app/Local Storage/leveldb/000004.log
/proc/1500/fd/5 /var/lib/rocksdb/012345.log
/proc/300/fd/11 /var/log/nginx/access.log
/proc/300/fd/12 /mnt/docroot/data/log/paperless.log
/proc/300/fd/13 socket:[48213]
/proc/300/fd/14 pipe:[48214]
/proc/1600/fd/6 /var/log/host/syslog
"""
#: Each descriptor's ``flags`` from ``/proc/<pid>/fdinfo/<fd>``, octal: the access
#: mode (``& 3``) is 1 or 2 for a writer -- 0102001 is O_WRONLY|O_APPEND|O_LARGEFILE
#: -- and 0 for a reader (0100000, O_RDONLY|O_LARGEFILE). CrowdSec (300) reads
#: nginx's log and a log on a mount; a promtail in a container (1600) reads the
#: host's syslog as its own /var/log/host/syslog. 1500 opens its log O_RDWR.
FLAGS = """/proc/610/fdinfo/7:flags:\t0102001
/proc/610/fdinfo/8:flags:\t0102001
/proc/620/fdinfo/9:flags:\t02100002
/proc/900/fdinfo/5:flags:\t0102001
/proc/900/fdinfo/6:flags:\t0102001
/proc/901/fdinfo/5:flags:\t0102001
/proc/700/fdinfo/4:flags:\t0102001
/proc/1100/fdinfo/30:flags:\t0102001
/proc/1100/fdinfo/31:flags:\t0102001
/proc/1400/fdinfo/3:flags:\t0102001
/proc/1500/fdinfo/3:flags:\t0100002
/proc/1500/fdinfo/4:flags:\t0100002
/proc/1500/fdinfo/5:flags:\t0100002
/proc/300/fdinfo/11:flags:\t0100000
/proc/300/fdinfo/12:flags:\t0100000
/proc/300/fdinfo/13:flags:\t02004002
/proc/300/fdinfo/14:flags:\t02004001
/proc/1600/fdinfo/6:flags:\t0100000
"""
STAT = """/proc/610/fd/7 99000 1762250000
/proc/610/fd/8 4100 1762250100
@@ -53,6 +83,8 @@ STAT = """/proc/610/fd/7 99000 1762250000
/proc/1100/fd/30 7000 1762250400
/proc/1100/fd/31 300 1762250500
/proc/1400/fd/3 0 1762250600
/proc/300/fd/11 2048 1762250200
/proc/300/fd/12 81920 1762250700
"""
CGROUPS = f"""610 0::/system.slice/rsyslog.service
900 0::/system.slice/nginx.service
@@ -60,6 +92,8 @@ CGROUPS = f"""610 0::/system.slice/rsyslog.service
1100 0::/system.slice/docker.service
1400 0::/system.slice/docker-{CID}.scope
1500 0::/user.slice/user-1000.slice/session-3.scope
300 0::/system.slice/crowdsec.service
1600 0::/system.slice/docker-{OTHER}.scope
"""
COMM = """610 rsyslogd
900 nginx
@@ -67,6 +101,8 @@ COMM = """610 rsyslogd
1100 dockerd
1400 python3
1500 app
300 crowdsec
1600 promtail
"""
@@ -77,14 +113,18 @@ def _wire(
stat: str = STAT,
cgroups: str = CGROUPS,
comm: str = COMM,
flags: str | None = FLAGS,
noise: str = "",
forwarding: str = "",
) -> str:
"""The report as the command prints it, framed."""
parts = [noise, "LOGS_BEGIN\n"]
if journald is not None:
parts += ["[journald]\n", journald]
parts += ["[open]\n", open_, "[stat]\n", stat, "[cgroups]\n", cgroups, "[comm]\n", comm]
parts.append("LOGS_END\n")
if flags is not None:
parts += ["[flags]\n", flags]
parts += [forwarding, "LOGS_END\n"]
return "".join(parts)
@@ -184,6 +224,100 @@ class TestParse:
parse_log_sources(cut)
class TestWriters:
"""A file is attributed to the process writing it, not to one reading it
(NetOrk/netork#2092): CrowdSec, Promtail or a ``tail -F`` holding a log is
no producer of it."""
def test_a_reader_with_the_lower_pid_does_not_take_the_writers_file(self) -> None:
access = _by_key(_sources())[("file", None, "/var/log/nginx/access.log")]
assert (access["unit"], access["process"]) == ("nginx", "nginx")
def test_a_file_only_readers_hold_is_listed_without_a_producer(self) -> None:
paperless = _by_key(_sources())[("file", None, "/mnt/docroot/data/log/paperless.log")]
assert paperless == {
"kind": "file",
"path": "/mnt/docroot/data/log/paperless.log",
"unit": None,
"container_id": None,
"process": None,
"size": 81920,
"modified": "2025-11-04T10:05:00+00:00",
}
def test_a_file_open_for_reading_and_writing_is_written(self) -> None:
spaced = _by_key(_sources())[("file", None, "/srv/app/log with space.log")]
assert spaced["process"] == "app"
def test_a_file_only_a_reader_in_a_container_holds_is_left_out(self) -> None:
# Its path is the reader's view of a mount: it names nothing on the host,
# and the container does not write it.
assert not [s for s in _sources() if s["path"] == "/var/log/host/syslog"]
def test_sockets_and_pipes_are_no_sources(self) -> None:
assert not [s for s in _sources() if not (s["path"] or "/").startswith("/")]
def test_a_deleted_file_stays_out_though_it_is_written(self) -> None:
assert not [s for s in _sources() if (s["path"] or "").startswith("/var/log/old")]
def test_a_holder_whose_flags_did_not_answer_is_no_writer(self) -> None:
# The process ended between the listing and the read of its fdinfo.
flags = FLAGS.replace("/proc/900/fdinfo/5:flags:\t0102001\n", "")
flags = flags.replace("/proc/901/fdinfo/5:flags:\t0102001\n", "")
access = _by_key(_sources(_wire(flags=flags)))[("file", None, "/var/log/nginx/access.log")]
assert (access["unit"], access["process"]) == (None, None)
def test_a_report_without_flags_names_the_first_holder_as_before(self) -> None:
# An older command printed no [flags]: who writes cannot be told apart.
access = _by_key(_sources(_wire(flags=None)))[("file", None, "/var/log/nginx/access.log")]
assert (access["unit"], access["process"]) == ("crowdsec", "crowdsec")
#: rsyslog running, forwarding to Graylog, its config prefixed so that no line reads as a section.
FORWARDING = """[rsyslog]
running
[rsyslog-conf]
| module(load="imuxsock")
| [comment-looking line]
|
| *.* @graylog.lan:1514;RSYSLOG_SyslogProtocol23Format
| input(type="imfile" File="/var/log/nginx/*.log" Tag="nginx")
[journald-conf]
#ForwardToSyslog=yes
"""
class TestForwarding:
"""Where the logs go beyond the host, read in the same command (NetOrk/netork#2053)."""
def test_rsyslog_forwarding_and_the_journal_reaching_it(self) -> None:
forwarding = parse_log_forwarding(_wire(forwarding=FORWARDING))
assert forwarding is not None
assert forwarding["rsyslog"] is True
assert forwarding["journal"] is True
assert [(r["target"], r["port"], r["complete"]) for r in forwarding["rules"]] == [
("graylog.lan", 1514, True)
]
assert forwarding["file_inputs"] == [{"path": "/var/log/nginx/*.log", "ruleset": None}]
def test_a_config_line_in_brackets_is_no_section(self) -> None:
forwarding = parse_log_forwarding(_wire(forwarding=FORWARDING))
assert forwarding is not None and forwarding["rules"]
def test_an_rsyslog_that_does_not_run_forwards_nothing(self) -> None:
stopped = FORWARDING.replace("running\n", "")
forwarding = parse_log_forwarding(_wire(forwarding=stopped))
assert forwarding == {"rsyslog": False, "journal": False, "rules": [], "file_inputs": []}
def test_a_report_without_the_sections_says_nothing(self) -> None:
assert parse_log_forwarding(_wire()) is None
def test_the_reading_carries_it(self) -> None:
reading = _Host({True: _wire(forwarding=FORWARDING)}).get_log_sources()
assert reading["forwarding"] is not None
assert reading["forwarding"]["rules"][0]["target"] == "graylog.lan"
class _Host(LogSourcesMixin):
"""Carries the command the way a driver does; *answers* per privilege."""
@@ -229,6 +363,21 @@ class TestCommand:
def test_it_is_valid_posix_sh(self) -> None:
assert subprocess.run(["sh", "-n", "-c", LOG_SOURCES_COMMAND]).returncode == 0
@pytest.mark.skipif(shutil.which("find") is None, reason="needs find on this host")
def test_here_a_file_held_for_writing_has_a_producer_and_one_held_for_reading_none(
self, tmp_path
) -> None:
written, read = tmp_path / "written.log", tmp_path / "read.log"
read.write_text("one line\n")
with written.open("a") as w, read.open() as r:
out = subprocess.run(
["sh", "-c", LOG_SOURCES_COMMAND], capture_output=True, text=True, timeout=60
).stdout
assert w.writable() and r.readable()
sources = {s["path"]: s for s in _sources(out)}
assert sources[str(written)]["process"] is not None
assert sources[str(read)]["process"] is None
@pytest.mark.skipif(shutil.which("find") is None, reason="needs find on this host")
def test_it_runs_here_and_parses(self, tmp_path) -> None:
log = tmp_path / "held.log"
@@ -241,3 +390,5 @@ class TestCommand:
# This test's own process holds the file, so it must be among the sources.
mine = [s for s in _sources(out) if s["path"] == str(log)]
assert mine and mine[0]["size"] == len("one line\n")
# Whether rsyslog runs here or not, the report says where logs go.
assert parse_log_forwarding(out) is not None
+51
View File
@@ -0,0 +1,51 @@
"""get_running_backups: the backups running on a device right now.
An application that restarts a device asks first, right before the restart: a
restart aborts a backup that runs on the device -- a hypervisor's vzdump, a
backup server's backup, sync, verify or garbage collection, a host's own restic
or borg run -- and a guest's restart breaks the backup its hypervisor is taking
of it (NetOrk/netork#2318). A driver mixes it in when it can tell; no role
demands it, and a driver without it cannot tell.
"""
from __future__ import annotations
import inspect
from typing import get_type_hints
from napalm_device_types import (
RUNNING_BACKUP_KINDS,
BackupServerDriver,
DeviceTypeDriver,
HypervisorDriver,
OSDriver,
RunningBackupDict,
RunningBackupsMixin,
)
def test_no_role_demands_it():
for role in (BackupServerDriver, HypervisorDriver, OSDriver, DeviceTypeDriver):
assert not issubclass(role, RunningBackupsMixin)
def test_declared_not_implemented():
"""hasattr is the caller's probe: only a driver that can tell answers True."""
assert not hasattr(RunningBackupsMixin, "get_running_backups")
def test_declaration_documents_the_contract():
source = inspect.getsource(RunningBackupsMixin)
assert "def get_running_backups(self) -> list[RunningBackupDict]" in source
assert "empty list for a failed read" in source
def test_the_kinds_cover_a_backup_servers_jobs_and_the_backup_itself():
assert set(RUNNING_BACKUP_KINDS) == {"backup", "sync", "verify", "gc", "prune", "tape"}
def test_the_shape():
hints = get_type_hints(RunningBackupDict)
assert set(hints) == {"kind", "job", "task", "guests", "since"}
assert hints["guests"] == list[str]
assert hints["since"] == int | None
+353
View File
@@ -0,0 +1,353 @@
"""How an OpenSSH server lets people in: the effective settings and sshd's PAM auth stack.
The parsers are pure. The getter runs against a fake host that answers ``sshd -T``
and the dump of ``/etc/pam.d`` from a dict, so the root read, the PAM includes
and a failed read are covered without a machine.
The ``sshd -T`` texts are abridged from what OpenSSH 9.2p1 on Debian 12 prints
(the keywords and their order as sshd writes them; the lines that say nothing
about authentication are left out). OpenSSH before 8.7 also prints
``challengeresponseauthentication``; OpenBSD's sshd has no ``usepam``.
"""
from __future__ import annotations
import pytest
from napalm_device_types.channel import CommandResult
from napalm_device_types.ssh_server import OpenSshServerMixin
from napalm_device_types.ssh_server_auth import (
PAM_FILES_COMMAND,
SSHD_EFFECTIVE_COMMAND,
SshServerAuthMixin,
pam_auth_stack,
parse_pam_auth,
parse_pam_files,
parse_sshd_auth,
)
#: Debian 12 as installed: password login on, PAM on, no AuthenticationMethods.
SSHD_T_DEBIAN = """\
port 22
addressfamily any
listenaddress [::]:22
listenaddress 0.0.0.0:22
usepam yes
logingracetime 120
x11displayoffset 10
maxauthtries 6
maxsessions 10
clientaliveinterval 0
clientalivecountmax 3
permitrootlogin without-password
ignorerhosts yes
ignoreuserknownhosts no
hostbasedauthentication no
hostbasedusesnamefrompacketonly no
pubkeyauthentication yes
kerberosauthentication no
kerberosorlocalpasswd yes
kerberosticketcleanup yes
gssapiauthentication no
gssapicleanupcredentials yes
gssapikeyexchange no
gssapistrictacceptorcheck yes
gssapistorecredentialsonrotate no
passwordauthentication yes
kbdinteractiveauthentication no
printmotd no
x11forwarding yes
authenticationmethods any
kexalgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256
"""
#: Key only: password and keyboard-interactive off.
SSHD_T_KEY_ONLY = SSHD_T_DEBIAN.replace(
"passwordauthentication yes", "passwordauthentication no"
).replace("permitrootlogin without-password", "permitrootlogin no")
#: A key and then an OTP through PAM, for everyone.
SSHD_T_KEY_AND_OTP = SSHD_T_KEY_ONLY.replace(
"kbdinteractiveauthentication no", "kbdinteractiveauthentication yes"
).replace("authenticationmethods any", "authenticationmethods publickey,keyboard-interactive")
#: OpenSSH 8.4p1 (Debian 11): the old keyword beside the new one.
SSHD_T_OLD = SSHD_T_KEY_ONLY.replace(
"kbdinteractiveauthentication no",
"kbdinteractiveauthentication yes\nchallengeresponseauthentication no",
)
#: OpenBSD 7.6: no usepam line.
SSHD_T_OPENBSD = SSHD_T_KEY_ONLY.replace("usepam yes\n", "")
#: Debian 12's /etc/pam.d/sshd, abridged: the auth stack comes from common-auth.
PAM_SSHD_DEBIAN = """\
# PAM configuration for the Secure Shell service
# Standard Un*x authentication.
@include common-auth
# Disallow non-root logins when /etc/nologin exists.
account required pam_nologin.so
# Standard Un*x authorization.
@include common-account
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so close
@include common-session
@include common-password
"""
PAM_COMMON_AUTH = """\
auth\t[success=1 default=ignore]\tpam_unix.so nullok
auth\trequisite\t\t\tpam_deny.so
auth\trequired\t\t\tpam_permit.so
"""
PAM_COMMON_ACCOUNT = "account [success=1 new_authtok_reqd=done default=ignore] pam_unix.so\n"
#: The usual way to add a TOTP: a line of its own in sshd's stack.
PAM_SSHD_OTP = PAM_SSHD_DEBIAN.replace(
"@include common-auth",
"#@include common-auth\nauth required pam_google_authenticator.so nullok",
)
#: Red Hat's form: ``auth substack password-auth``, a module by its full path.
PAM_SSHD_RHEL = """\
#%PAM-1.0
auth substack password-auth
auth include postlogin
-auth optional /usr/lib64/security/pam_reauthorize.so prepare
account required pam_sepermit.so
"""
PAM_PASSWORD_AUTH = "auth required pam_env.so\nauth sufficient pam_unix.so try_first_pass nullok\n"
PAM_POSTLOGIN = "session optional pam_umask.so silent\n"
# ── parse_sshd_auth ──────────────────────────────────────────────────────
def test_parse_reads_the_settings_that_decide_how_people_log_in():
auth = parse_sshd_auth(SSHD_T_DEBIAN)
assert auth == {
"password_authentication": True,
"kbd_interactive_authentication": False,
"pubkey_authentication": True,
"hostbased_authentication": False,
"gssapi_authentication": False,
"authentication_methods": [],
"permit_root_login": "without-password",
"use_pam": True,
"pam_auth_stack": None,
}
def test_parse_splits_authentication_methods_into_lists_of_methods():
auth = parse_sshd_auth(
SSHD_T_KEY_ONLY.replace(
"authenticationmethods any",
"authenticationmethods publickey,keyboard-interactive:pam publickey,password",
)
)
assert auth["authentication_methods"] == [
["publickey", "keyboard-interactive:pam"],
["publickey", "password"],
]
def test_parse_counts_the_old_challenge_response_keyword_as_keyboard_interactive():
"""Before 8.7 either keyword could switch PAM's prompts on: either one is enough."""
assert parse_sshd_auth(SSHD_T_OLD)["kbd_interactive_authentication"] is True
old_only = SSHD_T_KEY_ONLY.replace(
"kbdinteractiveauthentication no",
"kbdinteractiveauthentication no\nchallengeresponseauthentication yes",
)
assert parse_sshd_auth(old_only)["kbd_interactive_authentication"] is True
def test_parse_says_none_for_usepam_where_sshd_has_no_pam():
assert parse_sshd_auth(SSHD_T_OPENBSD)["use_pam"] is None
def test_parse_takes_a_method_sshd_was_built_without_as_off():
"""sshd built without GSSAPI prints no gssapiauthentication line: that method cannot be used."""
text = SSHD_T_DEBIAN.replace("gssapiauthentication no\n", "")
assert parse_sshd_auth(text)["gssapi_authentication"] is False
def test_parse_refuses_output_that_is_not_sshd_t():
with pytest.raises(ValueError, match="passwordauthentication"):
parse_sshd_auth("sshd: no hostkeys available -- exiting.\n")
# ── PAM ──────────────────────────────────────────────────────────────────
def _pairs(entries) -> list[tuple[str, str]]:
return [(e["control"], e["module"]) for e in entries]
def _modules(stack) -> list[str]:
return [e["module"] for e in stack]
def test_pam_auth_lists_modules_and_includes_in_stack_order():
# Every @include: which of them hold auth lines is for the included file to say.
assert _pairs(parse_pam_auth(PAM_SSHD_DEBIAN)) == [
("include", "common-auth"),
("include", "common-account"),
("include", "common-session"),
("include", "common-password"),
]
def test_pam_auth_reads_bracketed_controls_paths_and_optional_lines():
assert _pairs(parse_pam_auth(PAM_SSHD_RHEL)) == [
("substack", "password-auth"),
("include", "postlogin"),
("optional", "pam_reauthorize.so"),
]
assert parse_pam_auth(PAM_COMMON_AUTH) == [
{"module": "pam_unix.so", "control": "[success=1 default=ignore]", "args": ["nullok"]},
{"module": "pam_deny.so", "control": "requisite", "args": []},
{"module": "pam_permit.so", "control": "required", "args": []},
]
def test_pam_auth_keeps_a_modules_arguments():
"""``nullok`` lets a user without an OTP secret through: the consumer has to see it."""
assert parse_pam_auth(PAM_SSHD_OTP)[0] == {
"module": "pam_google_authenticator.so",
"control": "required",
"args": ["nullok"],
}
def test_pam_auth_skips_comments_and_other_types():
assert _pairs(parse_pam_auth(PAM_SSHD_OTP)) == [
("required", "pam_google_authenticator.so"),
("include", "common-account"),
("include", "common-session"),
("include", "common-password"),
]
def test_pam_auth_ignores_an_include_that_is_not_a_service_name():
assert parse_pam_auth("auth include ../../etc/shadow\n@include\nauth required\n") == []
def _dump(files: dict[str, str]) -> str:
"""What ``tail -n +1`` prints for several files: a header before each."""
return "\n".join(f"==> /etc/pam.d/{name} <==\n{text}" for name, text in files.items())
def test_pam_files_splits_the_dump_by_file():
files = {"sshd": PAM_SSHD_DEBIAN, "common-auth": PAM_COMMON_AUTH}
assert parse_pam_files(_dump(files)) == files
def test_auth_stack_follows_includes_and_substacks_in_stack_order():
files = {
"sshd": PAM_SSHD_RHEL + "auth include password-auth\n",
"password-auth": PAM_PASSWORD_AUTH,
"postlogin": PAM_POSTLOGIN,
}
assert _modules(pam_auth_stack(files)) == ["pam_env.so", "pam_unix.so", "pam_reauthorize.so"]
def test_auth_stack_stops_at_an_include_loop_and_skips_a_missing_file():
files = {
"sshd": "@include a\n@include gone\n",
"a": "@include sshd\nauth required pam_unix.so\n",
}
assert _modules(pam_auth_stack(files)) == ["pam_unix.so"]
def test_auth_stack_is_none_without_a_file_for_the_service():
assert pam_auth_stack({"login": PAM_COMMON_AUTH}) is None
# ── the getter ───────────────────────────────────────────────────────────
class _Host(SshServerAuthMixin):
"""Answers ``sshd -T`` (root only) and the dump of /etc/pam.d."""
def __init__(self, sshd_t: str | None, pam: dict[str, str] | None = None) -> None:
self.sshd_t = sshd_t
self.pam = pam
self.calls: list[tuple[str, bool]] = []
def run_command(
self,
command: str,
*,
privileged: bool = False,
timeout: float = 60,
stdin: bytes | None = None,
) -> CommandResult:
self.calls.append((command, privileged))
if command == SSHD_EFFECTIVE_COMMAND:
if self.sshd_t is None or not privileged:
return CommandResult("", "sshd: no hostkeys available -- exiting.", 1)
return CommandResult(self.sshd_t, "", 0)
assert command == PAM_FILES_COMMAND, command
if self.pam is None:
return CommandResult("", "tail: cannot open '/etc/pam.d/*' for reading", 1)
return CommandResult(_dump(self.pam), "", 0)
def test_getter_reads_sshd_t_as_root_and_pam_without():
host = _Host(
SSHD_T_DEBIAN,
{
"sshd": PAM_SSHD_DEBIAN,
"common-auth": PAM_COMMON_AUTH,
"common-account": PAM_COMMON_ACCOUNT,
},
)
auth = host.get_ssh_server_auth()
assert auth["password_authentication"] is True
assert _modules(auth["pam_auth_stack"]) == ["pam_unix.so", "pam_deny.so", "pam_permit.so"]
# sshd -T once, as root; PAM's files once, as the login user.
assert host.calls == [(SSHD_EFFECTIVE_COMMAND, True), (PAM_FILES_COMMAND, False)]
def test_getter_names_an_otp_module_in_sshds_own_stack():
host = _Host(SSHD_T_KEY_AND_OTP, {"sshd": PAM_SSHD_OTP})
stack = host.get_ssh_server_auth()["pam_auth_stack"]
assert _modules(stack) == ["pam_google_authenticator.so"]
def test_getter_keeps_what_it_read_when_one_pam_file_is_unreadable():
"""tail goes on past a file it cannot open and exits 1: the rest is still read."""
class _Partial(_Host):
def run_command(self, command: str, **kwargs) -> CommandResult: # type: ignore[override]
result = super().run_command(command, **kwargs)
if command == PAM_FILES_COMMAND:
return CommandResult(result.stdout, "tail: cannot open 'x'", 1)
return result
host = _Partial(SSHD_T_DEBIAN, {"sshd": "auth required pam_unix.so\n"})
assert _modules(host.get_ssh_server_auth()["pam_auth_stack"]) == ["pam_unix.so"]
def test_getter_leaves_pam_unread_without_usepam():
host = _Host(SSHD_T_OPENBSD)
auth = host.get_ssh_server_auth()
assert auth["pam_auth_stack"] is None
assert host.calls == [(SSHD_EFFECTIVE_COMMAND, True)]
def test_getter_says_none_when_pam_cannot_be_read():
assert _Host(SSHD_T_DEBIAN).get_ssh_server_auth()["pam_auth_stack"] is None
def test_getter_raises_when_sshd_t_fails():
with pytest.raises(RuntimeError, match="no hostkeys"):
_Host(None).get_ssh_server_auth()
def test_the_hardening_mixin_carries_the_reader():
"""Every driver that can harden sshd can also say how it lets people in."""
assert issubclass(OpenSshServerMixin, SshServerAuthMixin)