Compare commits

...
145 Commits
Author SHA1 Message Date
christianmanivong 2080ccbfd7 Merge pull request 'chore(release): 4.38.0' (#118) from chore/release-4.38.0 into main
driver / lint (push) Successful in 13s
driver / build (push) Successful in 28s
driver / publish (push) Successful in 27s
driver / test (3.13) (push) Successful in 1m1s
driver / security (push) Successful in 53s
driver / lowest (push) Successful in 37s
driver / types (push) Successful in 56s
driver / test (3.12) (push) Successful in 1m10s
driver / standard (push) Successful in 16s
CI / driver (push) Successful in 3m1s
2026-11-13 21:56:00 +00:00
christianmanivong ce1fe18405 chore(release): 4.38.0
driver / security (pull_request) Successful in 1m11s
driver / build (pull_request) Successful in 31s
CI / driver (pull_request) Successful in 2m59s
driver / test (3.13) (pull_request) Successful in 1m9s
driver / lowest (pull_request) Successful in 46s
driver / types (pull_request) Successful in 1m2s
driver / test (3.12) (pull_request) Successful in 1m18s
driver / standard (pull_request) Successful in 16s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 14s
2026-11-13 22:39:54 +01:00
christianmanivong e43926f1d5 Merge pull request 'feat(config-revisions): declare a device's own configuration history' (#117) from feat/config-revisions into main
driver / build (push) Successful in 29s
driver / publish (push) Skipped
driver / lint (push) Successful in 15s
driver / test (3.13) (push) Successful in 1m6s
driver / security (push) Successful in 1m5s
driver / lowest (push) Successful in 37s
driver / types (push) Successful in 1m11s
driver / test (3.12) (push) Successful in 1m27s
driver / standard (push) Successful in 16s
CI / driver (push) Successful in 2m58s
2026-11-13 21:39:38 +00:00
christianmanivong 43e0210ff6 feat(config-revisions): declare a device's own configuration history
driver / security (pull_request) Successful in 3m34s
CI / driver (pull_request) Successful in 5m2s
driver / lowest (pull_request) Successful in 47s
driver / build (pull_request) Successful in 34s
driver / test (3.13) (pull_request) Successful in 1m12s
driver / test (3.12) (pull_request) Successful in 1m18s
driver / standard (pull_request) Successful in 16s
driver / types (pull_request) Successful in 1m3s
driver / lint (pull_request) Successful in 13s
driver / publish (pull_request) Skipped
Some devices keep a copy of their configuration for every save and can
go back to one. An application restores a configuration there by
reverting to the revision that holds it; on OPNsense that is the only
way, since its API takes no uploaded config.xml (NetOrk/netork#2511).

ConfigRevisionsMixin declares get_config_revisions() (newest first, as
ConfigRevisionDict) and get_config_revision(revision_id).
revert_config_revision(revision_id) is a template over the driver's
_revert_to_config_revision step: it refuses an id that is not in the
history (UnknownConfigRevision), reads the revision before the revert
can rotate it out, and afterwards checks that the newest revision holds
its configuration (ConfigRevertFailed otherwise), since a device may
report a revert it did not make. config_revision_content(text) is the
hook where a driver leaves out what its device stamps into every save,
so revisions with the same configuration compare equal.

Refs NetOrk/netork#2511
2026-11-13 22:28:30 +01:00
christianmanivong 241acef6c1 Merge pull request 'feat(accounts): add_service_account takes the keys the new one replaces' (#115) from fix/service-account-replaces into main
driver / lint (push) Successful in 13s
driver / build (push) Successful in 28s
driver / publish (push) Skipped
driver / test (3.13) (push) Successful in 59s
driver / security (push) Successful in 57s
driver / lowest (push) Successful in 35s
driver / types (push) Successful in 55s
driver / test (3.12) (push) Successful in 1m10s
driver / standard (push) Successful in 16s
CI / driver (push) Successful in 2m35s
2026-11-13 08:55:08 +00:00
christianmanivong 4c0ba5b2c0 refactor(accounts): the account step of add_service_account in a helper
driver / security (pull_request) Successful in 54s
CI / driver (pull_request) Successful in 2m32s
driver / lowest (pull_request) Successful in 39s
driver / build (pull_request) Successful in 26s
driver / test (3.12) (pull_request) Successful in 1m8s
driver / test (3.13) (pull_request) Successful in 1m7s
driver / standard (pull_request) Successful in 16s
driver / types (pull_request) Successful in 52s
driver / lint (pull_request) Successful in 14s
driver / publish (pull_request) Skipped
replaces= made add_service_account 55 lines long, over the quality
ratchet's 50. Its account step -- none to take when the driver logs in
as the account already, else the create or reopen -- moves to
_accounts_service_account_step. Behaviour is unchanged; the tests of
add_service_account cover both branches.
2026-11-13 09:00:43 +01:00
christianmanivong e9da399568 Merge remote-tracking branch 'origin/main' into fix/service-account-replaces 2026-11-13 08:59:48 +01:00
christianmanivong 8673d1eda1 Merge pull request 'feat(log-forwarding): each forwarding rule names its template' (#116) from feat/forward-rule-template into main
driver / build (push) Successful in 31s
driver / publish (push) Skipped
driver / lint (push) Successful in 16s
driver / test (3.13) (push) Successful in 1m13s
driver / security (push) Successful in 1m3s
driver / lowest (push) Successful in 43s
driver / types (push) Successful in 1m8s
driver / test (3.12) (push) Successful in 1m22s
driver / standard (push) Successful in 17s
CI / driver (push) Successful in 2m33s
2026-11-12 22:42:45 +00:00
christianmanivong 46d7125aa6 refactor(log-forwarding): a legacy rule's template is read on its own
driver / security (pull_request) Successful in 1m4s
CI / driver (pull_request) Successful in 2m57s
driver / lowest (pull_request) Successful in 40s
driver / build (pull_request) Successful in 29s
driver / test (3.12) (pull_request) Successful in 1m21s
driver / test (3.13) (pull_request) Successful in 1m8s
driver / standard (pull_request) Successful in 17s
driver / types (pull_request) Successful in 58s
driver / lint (pull_request) Successful in 14s
driver / publish (pull_request) Skipped
The quality ratchet holds _Reader.legacy at complexity 10.
2026-11-12 23:38:28 +01:00
christianmanivong 8e43fe2ed9 feat(log-forwarding): each forwarding rule names its template
driver / security (pull_request) Successful in 1m5s
CI / driver (pull_request) Failing after 2m38s
driver / build (pull_request) Successful in 32s
driver / test (3.13) (pull_request) Successful in 1m15s
driver / test (3.12) (pull_request) Successful in 1m30s
driver / lowest (pull_request) Successful in 40s
driver / types (pull_request) Successful in 1m9s
driver / standard (pull_request) Failing after 17s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 16s
A rule's template decides the timestamp it sends: rsyslog's default, RFC 3164,
carries the host's local time without a zone, which a collector such as Graylog
files shifted by the host's UTC offset (NetOrk/netork#493). The rule now says
which template it uses: its own (`@host;Name`, `action(... template="...")`),
else the default set before it (`$ActionForwardDefaultTemplate`, or
`module(load="builtin:omfwd" template="...")` for omfwd actions), else None.
2026-11-12 23:36:38 +01:00
christianmanivong 679584766f feat(accounts): add_service_account takes the keys the new one replaces
driver / security (pull_request) Successful in 1m16s
CI / driver (pull_request) Failing after 2m45s
driver / build (pull_request) Successful in 38s
driver / test (3.13) (pull_request) Successful in 1m11s
driver / test (3.12) (pull_request) Successful in 1m19s
driver / lowest (pull_request) Successful in 46s
driver / standard (pull_request) Failing after 18s
driver / types (pull_request) Successful in 1m1s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 15s
netOrk moves a host from its installation-wide key to its site's
(NetOrk/netork#2462): the old key has to go from its own account, which
remove_local_account_keys refuses (the login's own account, a system
account). With replaces, the old keys go once the new key is in -- never
the new key itself, and not while it could not be added.
2026-11-12 21:55:13 +01:00
christianmanivong 1b04818982 Merge pull request 'fix(dhcp): a refused subnet no longer stops the rest of the set' (#114) from fix/dhcp-subnet-apply-tolerance into main
driver / lint (push) Successful in 13s
driver / build (push) Successful in 29s
driver / publish (push) Skipped
driver / test (3.13) (push) Successful in 1m6s
driver / security (push) Successful in 1m5s
driver / lowest (push) Successful in 43s
driver / types (push) Successful in 56s
driver / test (3.12) (push) Successful in 1m12s
driver / standard (push) Successful in 14s
CI / driver (push) Successful in 2m44s
2026-11-12 19:06:13 +00:00
christianmanivong 65991d0930 fix(dhcp): a refused subnet no longer stops the rest of the set
driver / security (pull_request) Successful in 1m13s
CI / driver (pull_request) Successful in 3m15s
driver / build (pull_request) Successful in 35s
driver / test (3.12) (pull_request) Successful in 1m32s
driver / test (3.13) (pull_request) Successful in 1m16s
driver / lowest (pull_request) Successful in 41s
driver / standard (pull_request) Successful in 16s
driver / types (pull_request) Successful in 1m19s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 16s
apply_dhcp_subnetset called apply_dhcp_subnet for each add and update
and committed only at the end, so one refused write raised before the
commit: the subnets written ahead of it stayed saved but inactive, and
as the device reports them as configured, no later diff committed them.

It now behaves like apply_dhcp_reservationset since 4.35.1: a refused
subnet gets a [failed] line, the others are written, any non-empty diff
is committed, then DhcpApplyError is raised. DhcpApplyError takes an
optional kind ("reservation" by default) for its message.

Closes #113
2026-11-12 19:19:03 +01:00
christianmanivong 9dfb5596bc Merge pull request 'fix(dhcp): release an address before another reservation takes it' (#112) from fix/dhcp-reservation-apply-order into main
driver / build (push) Successful in 30s
driver / publish (push) Skipped
driver / lint (push) Successful in 14s
driver / test (3.13) (push) Successful in 1m9s
driver / security (push) Successful in 1m9s
driver / lowest (push) Successful in 40s
driver / types (push) Successful in 1m1s
driver / test (3.12) (push) Successful in 1m17s
driver / standard (push) Successful in 17s
CI / driver (push) Successful in 2m59s
2026-11-12 18:14:11 +00:00
christianmanivong b4cdcfd347 fix(dhcp): release an address before another reservation takes it
driver / security (pull_request) Successful in 1m10s
driver / build (pull_request) Successful in 32s
CI / driver (pull_request) Successful in 3m6s
driver / test (3.12) (pull_request) Successful in 1m25s
driver / test (3.13) (pull_request) Successful in 1m13s
driver / lowest (pull_request) Successful in 45s
driver / types (pull_request) Successful in 1m3s
driver / standard (pull_request) Successful in 17s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 14s
apply_dhcp_reservationset wrote adds first, then updates in the order of
the desired set. Kea keeps addresses unique at each write, so when an
address moved from one reservation to another, the reservation taking it
was refused as a duplicate whenever it came first.

The changes are now ordered (_dhcp_order): an update that moves a
reservation off its address goes before the change that takes that
address, along a chain from its free end, and adds come after every
update. A swap or rotation of addresses has no such order; its changes
are still tried, so a server that checks only at the commit takes them,
and a refused one's [failed] line says it was a swap and with which
reservations. Nothing is moved to a temporary address: the contract
knows no free one, and a failed second write would leave it there.

The diff is computed against one read of the reservations, shared with
diff_dhcp_reservations. Refused rows still do not stop the others, the
rest is committed and DhcpApplyError follows, as since 4.35.1.

Closes #106
2026-11-12 19:07:31 +01:00
christianmanivong 5258a43dc9 Merge pull request 'feat(updates): the CVEs a host's installed advisories close (4.37.0)' (#111) from feature/installed-advisories into main
driver / build (push) Successful in 28s
driver / publish (push) Successful in 28s
driver / lint (push) Successful in 13s
driver / test (3.13) (push) Successful in 1m16s
driver / security (push) Successful in 1m6s
driver / lowest (push) Successful in 39s
driver / types (push) Successful in 59s
driver / test (3.12) (push) Successful in 1m12s
driver / standard (push) Successful in 15s
CI / driver (push) Successful in 6m6s
2026-11-12 11:12:53 +00:00
christianmanivong f3cb3601a8 feat(updates): the CVEs a host's installed advisories close (4.37.0)
driver / security (pull_request) Successful in 1m0s
driver / build (pull_request) Successful in 28s
CI / driver (pull_request) Successful in 2m29s
driver / test (3.13) (pull_request) Successful in 1m11s
driver / lowest (pull_request) Successful in 40s
driver / types (pull_request) Successful in 1m2s
driver / test (3.12) (pull_request) Successful in 1m24s
driver / standard (pull_request) Successful in 16s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 15s
UpdateMixin declares get_installed_advisory_cves(): per package, the CVEs
the installed security advisories close -- the distribution's word that a
backport is in although the upstream version did not move. The command
(DNF_INSTALLED_CVES_COMMAND, dnf 4, from the cache) and its parser
(parse_dnf_installed_cves) live here for every dnf driver, as the
security read's do. netOrk reads it where its knowledge base holds no
records of the host's release (NetOrk/netork#2394).
2026-11-12 12:09:37 +01:00
christianmanivong a8fedf299a Merge pull request 'fix(host-status): kernel releases compare the way rpm does (4.36.2)' (#110) from fix/rpm-kernel-version-order into main
driver / build (push) Successful in 40s
driver / publish (push) Successful in 30s
driver / lint (push) Successful in 14s
driver / test (3.13) (push) Successful in 1m13s
driver / security (push) Successful in 1m16s
driver / lowest (push) Successful in 46s
driver / types (push) Successful in 1m4s
driver / test (3.12) (push) Successful in 1m20s
driver / standard (push) Successful in 15s
CI / driver (push) Successful in 3m36s
2026-11-12 10:08:48 +00:00
christianmanivong c736fb30d0 refactor(host-status): compare one version segment in a helper of its own
driver / security (pull_request) Successful in 57s
driver / build (pull_request) Successful in 26s
CI / driver (pull_request) Successful in 2m30s
driver / test (3.13) (pull_request) Successful in 1m8s
driver / lowest (pull_request) Successful in 40s
driver / types (pull_request) Successful in 56s
driver / test (3.12) (pull_request) Successful in 1m8s
driver / standard (pull_request) Successful in 15s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 14s
The quality ratchet refused _version_cmp at complexity 12. Comparing one
segment moves to _segment_cmp, which also takes 01 and 1 as equal, as
rpm does.
2026-11-12 11:05:49 +01:00
christianmanivong cf555d006d fix(host-status): kernel releases compare the way rpm does (4.36.2)
driver / security (pull_request) Successful in 1m3s
driver / build (pull_request) Successful in 28s
CI / driver (pull_request) Failing after 2m32s
driver / test (3.13) (pull_request) Successful in 1m14s
driver / lowest (pull_request) Successful in 42s
driver / types (pull_request) Successful in 1m5s
driver / test (3.12) (pull_request) Successful in 1m19s
driver / standard (pull_request) Failing after 15s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 13s
_version_key compared the text between numbers as plain strings, so
'.el' outranked '.' and 6.12.0-208.112.5.el9uek counted as newer than
the running 6.12.0-208.112.5.1.el9uek: an Oracle Linux host asked to
reboot into an older kernel, and a reboot never cleared it. The
releases now compare like rpmvercmp: separators only separate, a
number is newer than letters in its place, and '~' comes before
anything. Checked against rpm.labelCompare on Oracle Linux 9.

Closes #109
2026-11-12 11:03:46 +01:00
christianmanivong 4e79cd1fc8 Merge pull request 'fix(host-status): an Oracle Linux host's UEK kernels are not an interrupted dnf (4.36.1)' (#108) from fix/dnf-installonly-defaults into main
driver / build (push) Successful in 30s
driver / publish (push) Successful in 30s
driver / lint (push) Successful in 13s
driver / test (3.13) (push) Successful in 1m12s
driver / security (push) Successful in 1m3s
driver / lowest (push) Successful in 41s
driver / types (push) Successful in 1m0s
driver / test (3.12) (push) Successful in 1m17s
driver / standard (push) Successful in 16s
CI / driver (push) Successful in 3m21s
2026-11-12 09:37:34 +00:00
christianmanivong 304f404567 fix(host-status): an Oracle Linux host's UEK kernels are not an interrupted dnf (4.36.1)
driver / security (pull_request) Successful in 56s
CI / driver (pull_request) Successful in 2m26s
driver / lowest (pull_request) Successful in 41s
driver / build (pull_request) Successful in 30s
driver / test (3.13) (pull_request) Successful in 1m12s
driver / test (3.12) (pull_request) Successful in 1m17s
driver / standard (pull_request) Successful in 18s
driver / types (pull_request) Successful in 59s
driver / lint (pull_request) Successful in 16s
driver / publish (pull_request) Skipped
The installonly packages were dnf's upstream defaults plus dnf.conf's.
Oracle builds kernel-uek and its parts into libdnf's defaults, and the
UEK packages provide nothing of upstream's list, so every host with more
than one UEK kernel was reported as left halfway. The defaults are now
also asked of the host's libdnf through its Python binding; upstream's
list stays the floor for dnf 5 and yum.

Closes #107
2026-11-12 10:30:48 +01:00
christianmanivong aaa9f0504a Merge pull request 'feat(time-sync): DHCP-provided NTP servers count and are read apart (4.36.0)' (#105) from feat/time-sync-dhcp-ntp-2380 into main
driver / build (push) Successful in 31s
driver / publish (push) Successful in 31s
driver / lint (push) Successful in 15s
driver / test (3.13) (push) Successful in 1m11s
driver / security (push) Successful in 1m2s
driver / lowest (push) Successful in 42s
driver / types (push) Successful in 59s
driver / test (3.12) (push) Successful in 1m16s
driver / standard (push) Successful in 18s
CI / driver (push) Successful in 3m17s
2026-11-11 21:25:27 +00:00
christianmanivong bc47220318 Merge remote-tracking branch 'origin/main' into feat/time-sync-dhcp-ntp-2380
driver / security (pull_request) Successful in 1m26s
driver / build (pull_request) Successful in 48s
CI / driver (pull_request) Successful in 3m17s
driver / test (3.13) (pull_request) Successful in 1m32s
driver / lowest (pull_request) Successful in 1m3s
driver / types (pull_request) Successful in 1m34s
driver / test (3.12) (pull_request) Successful in 1m52s
driver / standard (pull_request) Successful in 19s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 15s
# Conflicts:
#	CHANGELOG.md
#	pyproject.toml
2026-11-11 22:21:34 +01:00
christianmanivong 08a62838f7 Merge pull request 'fix(dhcp): one refused reservation no longer stops the apply' (#104) from fix/dhcp-reservations-partial-apply into main
driver / lint (push) Successful in 14s
driver / build (push) Successful in 33s
driver / publish (push) Successful in 32s
driver / test (3.13) (push) Successful in 1m9s
driver / security (push) Successful in 1m7s
driver / lowest (push) Successful in 41s
driver / types (push) Successful in 1m4s
driver / test (3.12) (push) Successful in 1m18s
driver / standard (push) Successful in 18s
CI / driver (push) Successful in 3m29s
2026-11-11 21:12:31 +00:00
christianmanivong ac38c68feb feat(time-sync): DHCP-provided NTP servers count and are read apart
driver / security (pull_request) Successful in 1m5s
driver / build (pull_request) Successful in 29s
CI / driver (pull_request) Successful in 3m22s
driver / test (3.13) (pull_request) Successful in 1m29s
driver / test (3.12) (pull_request) Successful in 1m33s
driver / lowest (pull_request) Successful in 41s
driver / types (pull_request) Successful in 1m16s
driver / standard (pull_request) Successful in 17s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 31s
A device that gets the site's time giver through DHCP option 42 is
aligned (NetOrk/netork#2380): most clients already learn it that way.
4.35.0 switched that off on every write -- chrony's DHCP sourcedir was
commented out, and systemd-timesyncd's link servers failed the verify
step -- and folded DHCP's servers into ntp_servers, so an application
could not tell configured from DHCP-provided servers.

TimeSyncDict gains dhcp_ntp_servers (not required): the servers the
device got by DHCP and uses, in its order; [] for none, absent or None
where the driver cannot tell. ntp_servers keeps only the configured ones.

PosixTimeSyncMixin reads chrony's sources from the DHCP client's
sourcedir (CHRONY_DHCP_DIRS, /run/chrony-dhcp by default) as chronyd
reads them, and systemd-timesyncd's LinkNTPServers; ntpd and OpenNTPD
keep DHCP's servers in a configuration of their own, so there it is
unknown. A chrony write now comments out only server, pool and peer
lines and leaves every sourcedir and the DHCP client's files alone. A
DHCP sourcedir that 4.35.0 commented out behind the mark is given back
on the next write, with a restart, so hosts it aligned hear DHCP again.

Version 4.36.0.
2026-11-11 22:08:52 +01:00
christianmanivong 8516f2a1ee refactor(dhcp): one reservation's write in a helper of its own
driver / security (pull_request) Successful in 1m9s
CI / driver (pull_request) Successful in 3m0s
driver / build (pull_request) Successful in 37s
driver / test (3.12) (pull_request) Successful in 1m19s
driver / test (3.13) (pull_request) Successful in 1m10s
driver / lowest (pull_request) Successful in 42s
driver / standard (pull_request) Successful in 17s
driver / types (pull_request) Successful in 1m6s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 15s
apply_dhcp_reservationset grew to 59 lines with the per-write handling;
the quality ratchet allows no new function that long.

Refs NetOrk/netork#2381
2026-11-11 22:08:50 +01:00
christianmanivong c74173523a fix(dhcp): one refused reservation no longer stops the apply
driver / security (pull_request) Successful in 1m11s
driver / build (pull_request) Successful in 38s
CI / driver (pull_request) Failing after 2m57s
driver / test (3.13) (pull_request) Successful in 1m26s
driver / lowest (pull_request) Successful in 38s
driver / types (pull_request) Successful in 1m22s
driver / test (3.12) (pull_request) Successful in 1m28s
driver / standard (pull_request) Failing after 22s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 18s
apply_dhcp_reservationset wrote the diff one reservation at a time and
committed only after the last. A write the device refused -- on
OPNsense, an address in a subnet Kea does not manage -- ended the run
before the commit: every later reservation went unwritten, and the
earlier ones stayed saved but inactive. The next diff read those back as
configured, reported no changes and never committed them.

A refused change now gets a "[failed] <mac> -> <ip>: <reason>" line and
the others are still written. Any non-empty diff commits, also one whose
every write was refused, so writes an interrupted earlier run saved take
effect. DhcpApplyError (a RuntimeError, exported) is raised after the
commit with how many of how many changes were refused.

Refs NetOrk/netork#2381
2026-11-11 22:03:52 +01:00
christianmanivong 6c7f164004 Merge pull request 'feat(time-sync): TimeSyncMixin, PosixTimeSyncMixin and the posix_tz helpers' (#103) from feature/time-sync-2361 into main
driver / build (push) Successful in 26s
driver / publish (push) Successful in 27s
driver / lint (push) Successful in 14s
driver / test (3.13) (push) Successful in 1m7s
driver / security (push) Successful in 54s
driver / lowest (push) Successful in 38s
driver / types (push) Successful in 59s
driver / test (3.12) (push) Successful in 1m12s
driver / standard (push) Successful in 16s
CI / driver (push) Successful in 3m3s
2026-11-11 14:05:02 +00:00
christianmanivong 073087da69 feat(time-sync): TimeSyncMixin, PosixTimeSyncMixin and the posix_tz helpers
driver / security (pull_request) Successful in 1m0s
CI / driver (pull_request) Successful in 2m23s
driver / lowest (pull_request) Successful in 38s
driver / build (pull_request) Successful in 26s
driver / test (3.13) (pull_request) Successful in 1m4s
driver / test (3.12) (pull_request) Successful in 1m14s
driver / standard (pull_request) Successful in 19s
driver / types (pull_request) Successful in 57s
driver / lint (pull_request) Successful in 19s
driver / publish (pull_request) Skipped
netOrk gets one list of NTP servers and a time zone per site and aligns
every device that differs (NetOrk/netork#2360). The contract comes first,
so the drivers can be built against it at the same time (#2361):

TimeSyncMixin declares get_time_sync() -> TimeSyncDict (servers in the
device's order, the zone as an IANA name, synchronised, the client) and
carries configure_time_sync(ntp_servers=None, timezone=None) as a fix
template: None leaves a value alone, the device ends up with exactly the
servers given, and a verify step reads it back. time_sync_writes says
which of the two values a driver writes; anything else, an unknown zone,
a malformed server or an empty list is refused before anything is
written.

PosixTimeSyncMixin implements it over run_command for Linux-like hosts:
one reading as the login user (timedatectl, chrony's configuration and
chronyc, timedatectl show-timesync, ntpd and OpenNTPD), the zone through
timedatectl set-timezone, chrony's servers in a file of their own with
every other source commented out, systemd-timesyncd's in a drop-in that
resets NTP= and empties FallbackNTP=. ntpd and OpenNTPD are read only.

posix_tz moves here from netOrk, so OpenWrt can write the POSIX rule and
a switch the UTC offset and summer-time rule (parse_posix_tz). It reads
the tzdata package, now a dependency.

Release 4.35.0: the drivers that implement it need it on PyPI.
2026-11-11 14:30:36 +01:00
christianmanivong 278a707157 Merge pull request 'feat(running-backups): the backups running on a device right now' (#102) from feature/running-backups-2318 into main
driver / build (push) Successful in 26s
driver / publish (push) Successful in 26s
driver / lint (push) Successful in 13s
driver / test (3.13) (push) Successful in 58s
driver / security (push) Successful in 56s
driver / lowest (push) Successful in 35s
driver / types (push) Successful in 59s
driver / test (3.12) (push) Successful in 1m13s
driver / standard (push) Successful in 14s
CI / driver (push) Successful in 2m36s
2026-11-11 07:22:54 +00:00
christianmanivong 56e5e82a59 feat(running-backups): the backups running on a device right now
driver / security (pull_request) Successful in 59s
CI / driver (pull_request) Successful in 2m24s
driver / lowest (pull_request) Successful in 37s
driver / build (pull_request) Successful in 29s
driver / test (3.13) (pull_request) Successful in 1m5s
driver / test (3.12) (pull_request) Successful in 1m15s
driver / standard (pull_request) Successful in 19s
driver / types (pull_request) Successful in 1m2s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 17s
A restart aborts every backup that runs on the device it restarts, and a
guest's restart breaks the backup its hypervisor is taking of it. netOrk
asks right before it restarts a host after patching (NetOrk/netork#2318),
so the reading belongs in the contract first: RunningBackupsMixin declares
get_running_backups(), opt-in like VmStopRequestMixin, so hasattr tells a
caller whether the driver can tell at all.

Release 4.34.0: the drivers that implement it need it on PyPI.
2026-11-10 23:39:17 +01:00
christianmanivong e6f145d4a3 Merge pull request 'chore(release): 4.33.0' (#101) from chore/release-4.33.0 into main
driver / build (push) Successful in 32s
driver / publish (push) Successful in 33s
driver / lint (push) Successful in 16s
driver / test (3.13) (push) Successful in 1m10s
driver / security (push) Successful in 1m5s
driver / lowest (push) Successful in 54s
driver / types (push) Successful in 1m0s
driver / test (3.12) (push) Successful in 1m16s
driver / standard (push) Successful in 16s
CI / driver (push) Successful in 3m28s
2026-11-07 19:23:16 +00:00
christianmanivong 66037e878e chore(release): 4.33.0
driver / security (pull_request) Successful in 1m9s
CI / driver (pull_request) Successful in 2m57s
driver / build (pull_request) Successful in 44s
driver / test (3.12) (pull_request) Successful in 1m15s
driver / lowest (pull_request) Successful in 39s
driver / test (3.13) (pull_request) Successful in 1m22s
driver / standard (pull_request) Successful in 19s
driver / types (pull_request) Successful in 1m7s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 14s
2026-11-07 20:18:14 +01:00
christianmanivong afffd0915c Merge pull request 'feat(host-status): tell an interrupted dnf, yum, apk or opkg' (#98) from feat/interrupted-pm-851 into main
driver / build (push) Successful in 35s
driver / publish (push) Skipped
driver / lint (push) Successful in 16s
driver / test (3.13) (push) Successful in 1m9s
driver / security (push) Successful in 1m7s
driver / lowest (push) Successful in 41s
driver / types (push) Successful in 1m7s
driver / test (3.12) (push) Successful in 1m17s
driver / standard (push) Successful in 17s
CI / driver (push) Successful in 2m34s
2026-11-07 19:18:08 +00:00
christianmanivong 92891473be fix(host-status): keep models.py at its recorded size, one literal for the dnf query
driver / security (pull_request) Successful in 1m6s
CI / driver (pull_request) Successful in 2m38s
driver / build (pull_request) Successful in 36s
driver / test (3.12) (pull_request) Successful in 1m11s
driver / test (3.13) (pull_request) Successful in 1m5s
driver / lowest (pull_request) Successful in 40s
driver / types (pull_request) Successful in 58s
driver / standard (pull_request) Successful in 14s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 13s
The quality ratchet records models.py at 1454 lines and the new comments
had made it 1460. bandit read the dnf history query, built with an
f-string for its path, as SQL put together from strings (B608); it is a
fixed query, now one literal with the path spelt out, and a test holds
it to DNF_HISTORY.

Refs NetOrk/netork#851
2026-11-07 20:14:11 +01:00
christianmanivong 658f1c8ab2 Merge remote-tracking branch 'origin/main' into feat/interrupted-pm-851
# Conflicts:
#	CHANGELOG.md
2026-11-07 20:14:11 +01:00
christianmanivong aacde6be3f Merge pull request 'chore(release): 4.32.0' (#100) from chore/release-4.32.0 into main
driver / build (push) Successful in 30s
driver / publish (push) Successful in 35s
driver / lint (push) Successful in 14s
driver / test (3.13) (push) Successful in 1m14s
driver / security (push) Successful in 1m7s
driver / lowest (push) Successful in 37s
driver / types (push) Successful in 1m6s
driver / test (3.12) (push) Successful in 1m29s
driver / standard (push) Successful in 17s
CI / driver (push) Successful in 3m14s
2026-11-07 18:49:52 +00:00
christianmanivong 58c7616401 chore(release): 4.32.0
driver / security (pull_request) Successful in 1m16s
CI / driver (pull_request) Successful in 3m13s
driver / build (pull_request) Successful in 32s
driver / test (3.12) (pull_request) Successful in 1m23s
driver / test (3.13) (pull_request) Successful in 1m38s
driver / lowest (pull_request) Successful in 1m1s
driver / standard (pull_request) Successful in 18s
driver / types (pull_request) Successful in 1m5s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 16s
2026-11-07 19:44:49 +01:00
christianmanivong b7e31205d5 Merge pull request 'feat(ssh-server): read how sshd lets people log in' (#99) from feat/ssh-server-auth into main
driver / build (push) Successful in 33s
driver / publish (push) Skipped
driver / lint (push) Successful in 15s
driver / test (3.13) (push) Successful in 1m16s
driver / security (push) Successful in 1m5s
driver / lowest (push) Successful in 42s
driver / types (push) Successful in 1m2s
driver / test (3.12) (push) Successful in 1m22s
driver / standard (push) Successful in 19s
CI / driver (push) Successful in 2m40s
2026-11-07 18:44:43 +00:00
christianmanivong 1d71ff8f91 feat(ssh-server): read how sshd lets people log in
driver / security (pull_request) Successful in 1m5s
CI / driver (pull_request) Successful in 2m48s
driver / lowest (pull_request) Successful in 44s
driver / build (pull_request) Successful in 32s
driver / test (3.13) (pull_request) Successful in 1m25s
driver / test (3.12) (pull_request) Successful in 1m32s
driver / standard (pull_request) Successful in 17s
driver / types (pull_request) Successful in 1m10s
driver / lint (pull_request) Successful in 15s
driver / publish (pull_request) Skipped
SshServerAuthMixin.get_ssh_server_auth() reads the authentication settings
from sshd -T as root (password, keyboard-interactive, public key, host-based,
GSSAPI, AuthenticationMethods, PermitRootLogin, UsePAM) and, with PAM on, the
modules of sshd's PAM auth stack from /etc/pam.d with its includes followed.
It reports the facts; whether a module is a second factor is the consumer's
call. OpenSshServerMixin carries it and both read sshd -T by the same fixed
command, so every driver that can harden sshd can say this too.

Refs NetOrk/netork#662
2026-11-07 15:09:47 +01:00
christianmanivong 58ed0f0a94 feat(host-status): tell an interrupted dnf, yum, apk or opkg
driver / security (pull_request) Failing after 1m1s
CI / driver (pull_request) Failing after 2m42s
driver / build (pull_request) Successful in 36s
driver / test (3.12) (pull_request) Successful in 1m22s
driver / lowest (pull_request) Successful in 39s
driver / test (3.13) (pull_request) Successful in 2m14s
driver / standard (pull_request) Failing after 18s
driver / types (pull_request) Successful in 1m9s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 17s
get_host_status() read only dpkg's state. It now also reads the rpm
database and dnf's history (or yum's journal), apk's installed database
and opkg's status file, without root, and names the manager it read in
the new package_manager field. While one of them runs, the answer is
unknown, as for dpkg.

Refs NetOrk/netork#851
2026-11-07 14:44:55 +01:00
christianmanivong c1a05d6dba Merge pull request 'chore(release): 4.31.1' (#97) from chore/release-4.31.1 into main
driver / build (push) Successful in 32s
driver / publish (push) Successful in 27s
driver / lint (push) Successful in 13s
driver / test (3.13) (push) Successful in 1m3s
driver / security (push) Successful in 55s
driver / lowest (push) Successful in 37s
driver / types (push) Successful in 59s
driver / test (3.12) (push) Successful in 1m13s
driver / standard (push) Successful in 15s
CI / driver (push) Successful in 3m3s
2026-11-06 17:57:27 +00:00
christianmanivong fccf902c07 chore(release): 4.31.1
driver / security (pull_request) Successful in 1m4s
driver / build (pull_request) Successful in 30s
CI / driver (pull_request) Successful in 3m0s
driver / lowest (pull_request) Successful in 39s
driver / test (3.13) (pull_request) Successful in 1m8s
driver / types (pull_request) Successful in 1m6s
driver / test (3.12) (pull_request) Successful in 1m23s
driver / standard (pull_request) Successful in 18s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 15s
2026-11-06 18:52:24 +01:00
christianmanivong f5a2811333 Merge pull request 'fix(log-sources): attribute a file only to the process writing it' (#96) from fix/log-writers-only into main
driver / build (push) Successful in 35s
driver / publish (push) Skipped
driver / lint (push) Successful in 12s
driver / test (3.13) (push) Successful in 1m4s
driver / security (push) Successful in 1m19s
driver / lowest (push) Successful in 36s
driver / types (push) Successful in 56s
driver / test (3.12) (push) Successful in 1m12s
driver / standard (push) Successful in 16s
CI / driver (push) Successful in 2m39s
2026-11-06 17:52:18 +00:00
christianmanivong 98d1dbef4c fix(log-sources): attribute a file only to the process writing it
driver / security (pull_request) Successful in 59s
CI / driver (pull_request) Successful in 2m25s
driver / build (pull_request) Successful in 29s
driver / test (3.13) (pull_request) Successful in 1m8s
driver / test (3.12) (pull_request) Successful in 1m15s
driver / lowest (pull_request) Successful in 40s
driver / types (pull_request) Successful in 1m8s
driver / standard (pull_request) Successful in 16s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 14s
A process that holds a log open only to read it -- CrowdSec's
acquisition, Promtail, rsyslog imfile, a tail -F -- was named as the
file's writer whenever it had the lower pid. The command now reads each
descriptor's flags from /proc/<pid>/fdinfo in the same round trip, and
only an O_WRONLY or O_RDWR holder is the producer. A file only readers
hold stays listed without one; a file only a reader inside a container
holds is its view of a mount and is left out.

Refs: NetOrk/netork#2092
2026-11-06 18:45:31 +01:00
christianmanivong 252ea3b81c Merge pull request 'feat: where a host's logs go beyond it (4.31.0)' (#95) from feat/log-forwarding-2053 into main
driver / build (push) Successful in 27s
driver / publish (push) Successful in 27s
driver / lint (push) Successful in 13s
driver / test (3.13) (push) Successful in 1m7s
driver / security (push) Successful in 53s
driver / lowest (push) Successful in 37s
driver / types (push) Successful in 55s
driver / test (3.12) (push) Successful in 1m14s
driver / standard (push) Successful in 16s
CI / driver (push) Successful in 2m44s
2026-11-06 10:14:19 +00:00
christianmanivong b685b200cc refactor: the rsyslog reader's cutter and action in their own steps
driver / security (pull_request) Successful in 56s
driver / build (pull_request) Successful in 26s
CI / driver (pull_request) Successful in 2m31s
driver / test (3.12) (pull_request) Successful in 1m7s
driver / test (3.13) (pull_request) Successful in 1m2s
driver / lowest (pull_request) Successful in 37s
driver / types (pull_request) Successful in 53s
driver / standard (pull_request) Successful in 15s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 13s
Keeps _pieces and _Reader.call within the standard's complexity limit.
2026-11-06 11:10:57 +01:00
christianmanivong 8557710724 feat: where a host's logs go beyond it (4.31.0)
driver / security (pull_request) Successful in 1m2s
CI / driver (pull_request) Failing after 2m23s
driver / build (pull_request) Successful in 30s
driver / test (3.13) (pull_request) Successful in 1m4s
driver / test (3.12) (pull_request) Successful in 1m15s
driver / lowest (pull_request) Successful in 38s
driver / standard (pull_request) Failing after 17s
driver / types (pull_request) Successful in 1m8s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 16s
get_log_sources() reads, in the same command, whether rsyslog runs, its
forwarding rules (legacy @/@@/:omrelp: and RainerScript omfwd/omrelp,
with selector, ruleset and whether they take every message), its imfile
inputs, and whether the journal reaches it (imjournal, or imuxsock with
journald's ForwardToSyslog; failing a setting, the default the
distribution ships commented out). Returned as `forwarding`;
parse_log_forwarding and parse_rsyslog are public (NetOrk/netork#2053).
2026-11-06 11:06:07 +01:00
christianmanivong 1b600416f0 Merge pull request 'feat: which logs a host writes, and who writes each (4.30.0)' (#94) from feature/log-sources-2049 into main
driver / lint (push) Successful in 13s
driver / build (push) Successful in 27s
driver / publish (push) Successful in 27s
driver / test (3.13) (push) Successful in 1m3s
driver / security (push) Successful in 55s
driver / lowest (push) Successful in 36s
driver / types (push) Successful in 1m4s
driver / test (3.12) (push) Successful in 1m13s
driver / standard (push) Successful in 15s
CI / driver (push) Successful in 2m41s
2026-11-04 23:41:03 +00:00
christianmanivong e622b26d42 refactor: keep the log source shapes in log_sources, not in models
driver / security (pull_request) Successful in 1m7s
driver / build (pull_request) Successful in 29s
CI / driver (pull_request) Successful in 2m42s
driver / test (3.13) (pull_request) Successful in 1m17s
driver / lowest (pull_request) Successful in 38s
driver / types (pull_request) Successful in 59s
driver / test (3.12) (pull_request) Successful in 1m20s
driver / standard (pull_request) Successful in 16s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 15s
models.py is over the file size limit and may not grow (quality ratchet);
the shapes sit with their mixin, as ScheduledJobDict does.

Refs NetOrk/netork#2049
2026-11-05 00:34:20 +01:00
christianmanivong 5daba5d37c feat: which logs a host writes, and who writes each (4.30.0)
driver / security (pull_request) Successful in 1m14s
CI / driver (pull_request) Failing after 2m45s
driver / build (pull_request) Successful in 33s
driver / test (3.13) (pull_request) Successful in 1m11s
driver / test (3.12) (pull_request) Successful in 1m18s
driver / lowest (pull_request) Successful in 38s
driver / standard (pull_request) Failing after 19s
driver / types (pull_request) Successful in 1m14s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 20s
LogSourcesMixin.get_log_sources() reads, in one sh -c call as root, the
units journald holds entries for, the log files running processes hold
open (/proc/<pid>/fd: *.log, *.log.N, anything under /var/log) with size
and last write from stat -L on the descriptor, and the engine's own file
of each container's output as that container's stream. Each file names
the service or container of its first writer from /proc/<pid>/cgroup; a
file a container writes keeps the container's own path.

journald's store, deleted files and a database's write-ahead log
(LevelDB's 000004.log) are left out. Without root it reads again and says
so (attributed: false); a host whose find cannot list open files (busybox)
raises LogSourcesUnavailable rather than reading as no files.

Refs NetOrk/netork#2049
2026-11-05 00:24:24 +01:00
christianmanivong dacd10fcce Merge pull request 'chore(release): 4.29.0' (#93) from chore/release-4.29.0 into main
driver / build (push) Successful in 36s
driver / publish (push) Successful in 34s
driver / lint (push) Successful in 17s
driver / test (3.13) (push) Successful in 1m21s
driver / security (push) Successful in 1m12s
driver / lowest (push) Successful in 38s
driver / types (push) Successful in 1m13s
driver / test (3.12) (push) Successful in 1m38s
driver / standard (push) Successful in 18s
CI / driver (push) Successful in 3m56s
2026-11-04 14:34:50 +00:00
christianmanivong 8a54674e40 chore(release): 4.29.0
driver / security (pull_request) Successful in 1m11s
CI / driver (pull_request) Successful in 2m58s
driver / build (pull_request) Successful in 37s
driver / test (3.12) (pull_request) Successful in 1m14s
driver / test (3.13) (pull_request) Successful in 1m6s
driver / lowest (pull_request) Successful in 42s
driver / types (pull_request) Successful in 1m6s
driver / standard (pull_request) Successful in 17s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 16s
2026-11-04 15:29:48 +01:00
christianmanivong d45455adf1 Merge pull request 'feat(local-accounts): remove_local_account_keys can clear the password too' (#92) from feat/remove-keys-clear-password into main
driver / build (push) Successful in 34s
driver / publish (push) Skipped
driver / lint (push) Successful in 16s
driver / test (3.13) (push) Successful in 1m11s
driver / security (push) Successful in 1m13s
driver / lowest (push) Successful in 42s
driver / types (push) Successful in 1m3s
driver / test (3.12) (push) Successful in 1m20s
driver / standard (push) Successful in 19s
CI / driver (push) Successful in 2m43s
2026-11-04 14:29:42 +00:00
christianmanivong ac49e2b5d4 feat(local-accounts): remove_local_account_keys can clear the password too
driver / security (pull_request) Successful in 1m2s
driver / build (pull_request) Successful in 30s
CI / driver (pull_request) Successful in 2m36s
driver / test (3.13) (pull_request) Successful in 1m15s
driver / test (3.12) (pull_request) Successful in 1m18s
driver / lowest (pull_request) Successful in 39s
driver / types (pull_request) Successful in 1m1s
driver / standard (pull_request) Successful in 19s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 16s
A person who deletes their key loses the password that went with it in netOrk, but the devices kept it, and sudo with it. remove_local_account_keys(name, keys, clear_password=True) takes the keys, the password and sudo in one call; the account stays open. Each step runs when another failed.

See NetOrk/netork#2033.
2026-11-04 15:21:01 +01:00
christianmanivong c967635cba Merge pull request 'chore(release): 4.28.0' (#91) from chore/release-4.28.0 into main
driver / build (push) Successful in 37s
driver / publish (push) Successful in 34s
driver / lint (push) Successful in 13s
driver / test (3.13) (push) Successful in 1m4s
driver / security (push) Successful in 1m17s
driver / lowest (push) Successful in 43s
driver / types (push) Successful in 1m2s
driver / test (3.12) (push) Successful in 1m12s
driver / standard (push) Successful in 17s
CI / driver (push) Successful in 3m33s
2026-11-04 11:04:06 +00:00
christianmanivong c470686305 chore(release): 4.28.0
driver / security (pull_request) Successful in 1m16s
driver / build (pull_request) Successful in 39s
CI / driver (pull_request) Successful in 3m2s
driver / test (3.13) (pull_request) Successful in 1m10s
driver / lowest (pull_request) Successful in 56s
driver / types (pull_request) Successful in 1m0s
driver / test (3.12) (pull_request) Successful in 1m16s
driver / standard (pull_request) Successful in 17s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 16s
2026-11-04 11:55:02 +01:00
christianmanivong 1edf458feb Merge pull request 'feat(local-accounts): replace and remove keys, clear the password' (#90) from fix/local-account-key-replace into main
driver / lint (push) Successful in 16s
driver / build (push) Successful in 31s
driver / publish (push) Skipped
driver / test (3.13) (push) Successful in 1m10s
driver / security (push) Successful in 1m7s
driver / lowest (push) Successful in 41s
driver / types (push) Successful in 1m14s
driver / test (3.12) (push) Successful in 1m14s
driver / standard (push) Successful in 18s
CI / driver (push) Successful in 2m55s
2026-11-04 10:54:06 +00:00
christianmanivong 00452d0d37 feat(local-accounts): replace and remove keys, clear the password
driver / security (pull_request) Successful in 1m16s
CI / driver (pull_request) Successful in 2m44s
driver / lowest (pull_request) Successful in 40s
driver / build (pull_request) Successful in 39s
driver / test (3.13) (pull_request) Successful in 1m10s
driver / test (3.12) (pull_request) Successful in 1m21s
driver / standard (pull_request) Successful in 18s
driver / types (pull_request) Successful in 1m1s
driver / lint (pull_request) Successful in 15s
driver / publish (pull_request) Skipped
A person who replaces a key, deletes one or removes their password had
no way to see that reach the device: add_local_account only ever added
a key and kept the password.

add_local_account takes replaces=[...] (the old keys go, matched by
their base64, the person's own keys stay) and clear_password=True (the
account logs in with a key only and cannot have sudo).
remove_local_account_keys takes keys off an account and leaves it open.
Drivers supply _account_clear_password and may override
_account_drop_keys for a key store of their own.

See NetOrk/netork#2013
2026-11-04 11:47:46 +01:00
christianmanivong 4e40f95bc1 Merge pull request 'feat(scheduled-jobs): ScheduledJobsMixin and its POSIX implementation (#86), release 4.27.0' (#87) from feature/scheduled-jobs into main
driver / build (push) Successful in 30s
driver / publish (push) Successful in 41s
driver / lint (push) Successful in 14s
driver / test (3.13) (push) Successful in 1m0s
driver / security (push) Successful in 54s
driver / lowest (push) Successful in 35s
driver / types (push) Successful in 59s
driver / test (3.12) (push) Successful in 1m6s
driver / standard (push) Successful in 15s
CI / driver (push) Successful in 3m7s
2026-11-03 07:37:25 +00:00
christianmanivong 70e1c9b218 chore(release): 4.27.0
driver / security (pull_request) Successful in 1m5s
CI / driver (pull_request) Successful in 2m33s
driver / build (pull_request) Successful in 34s
driver / test (3.13) (pull_request) Successful in 1m11s
driver / test (3.12) (pull_request) Successful in 1m20s
driver / lowest (pull_request) Successful in 40s
driver / standard (pull_request) Successful in 17s
driver / types (pull_request) Successful in 1m0s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 14s
Scheduled jobs (NetOrk/netork#1933): ScheduledJobsMixin, its POSIX
implementation and napalm_device_types.cron. 4.26.0 went out meanwhile
with the sshd drop-in change, so this is 4.27.0.
2026-11-03 08:34:14 +01:00
christianmanivong a70501ff60 Merge remote-tracking branch 'origin/main' into feature/scheduled-jobs
# Conflicts:
#	CHANGELOG.md
2026-11-03 08:33:52 +01:00
christianmanivong c6d119eb83 fix(scheduled-jobs): a monotonic timer as its unit file says it, OnUnitActiveSec
driver / security (pull_request) Successful in 1m8s
driver / build (pull_request) Successful in 27s
CI / driver (pull_request) Successful in 2m47s
driver / lowest (pull_request) Successful in 43s
driver / test (3.13) (pull_request) Successful in 1m14s
driver / types (pull_request) Successful in 57s
driver / test (3.12) (pull_request) Successful in 1m12s
driver / standard (pull_request) Successful in 14s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 13s
systemctl show names a timer's monotonic settings by their properties,
OnUnitActiveUSec=3h; the schedule now reads as the unit file writes it.
Found on a live Ubuntu host.

Refs #86
2026-11-03 08:11:48 +01:00
christianmanivong 8e9d21c368 Merge pull request 'feat(ssh): SSHD_CONFIG_GENERATED keeps the hardening to the drop-in (#88), release 4.26.0' (#89) from fix/ssh-hardening-generated-config into main
driver / build (push) Successful in 1m5s
driver / publish (push) Successful in 1m11s
driver / lint (push) Successful in 23s
driver / test (3.13) (push) Successful in 1m41s
driver / security (push) Successful in 1m41s
driver / lowest (push) Successful in 1m30s
driver / types (push) Successful in 2m7s
driver / test (3.12) (push) Successful in 1m35s
driver / standard (push) Successful in 17s
CI / driver (push) Successful in 8m25s
2026-11-03 07:00:40 +00:00
christianmanivong 25f28297e4 chore(release): 4.26.0
driver / security (pull_request) Successful in 1m9s
CI / driver (pull_request) Successful in 3m28s
driver / lowest (pull_request) Successful in 36s
driver / build (pull_request) Successful in 35s
driver / test (3.13) (pull_request) Successful in 2m2s
driver / test (3.12) (pull_request) Successful in 2m10s
driver / standard (pull_request) Successful in 17s
driver / types (pull_request) Successful in 2m9s
driver / lint (pull_request) Successful in 13s
driver / publish (pull_request) Skipped
2026-11-03 07:56:46 +01:00
christianmanivong 847c902193 feat(ssh): SSHD_CONFIG_GENERATED keeps the hardening to the drop-in (#88)
A platform that writes sshd_config from its own database (OpenMediaVault)
had to opt out of harden_ssh_server altogether. Where that generated file
includes /etc/ssh/sshd_config.d, the drop-in is safe; only the fallback
that rewrites sshd_config itself would be undone on the next apply. With
SSHD_CONFIG_GENERATED set, a sshd_config without the include is left alone
and the result says why.
2026-11-03 07:56:46 +01:00
christianmanivong ffbb194300 fix(scheduled-jobs): a file without a last newline no longer swallows the next header
driver / security (pull_request) Successful in 1m13s
driver / build (pull_request) Successful in 32s
CI / driver (pull_request) Successful in 2m43s
driver / test (3.13) (pull_request) Successful in 1m4s
driver / lowest (pull_request) Successful in 41s
driver / types (pull_request) Successful in 1m4s
driver / test (3.12) (pull_request) Successful in 1m6s
driver / standard (pull_request) Successful in 15s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 14s
GNU sed keeps a missing final newline, so the next section's header ran
on into the file's last line and its jobs were lost. The report ends
every such file with a newline of its own.

Refs #86
2026-11-03 05:50:44 +01:00
christianmanivong edef6cf07c fix(scheduled-jobs): write a linked crontab through its link, with the target's mode
driver / security (pull_request) Successful in 1m8s
driver / build (pull_request) Successful in 36s
CI / driver (pull_request) Successful in 2m54s
driver / test (3.13) (pull_request) Successful in 1m18s
driver / lowest (pull_request) Successful in 40s
driver / types (pull_request) Successful in 1m11s
driver / test (3.12) (pull_request) Successful in 1m25s
driver / standard (pull_request) Successful in 16s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 20s
stat without -L gave the link's own mode, 777, and install replaced the
link with a plain file of that mode: cron ignores a file anyone may write,
so every job in it stopped (/etc/cron.d/vzdump on older PVE links to
/etc/pve/vzdump.cron). The file the link names is written instead.

Refs #86
2026-11-03 05:39:57 +01:00
christianmanivong df476a12f2 feat(scheduled-jobs): the install command, time zone, owners and platform sources are the driver's
driver / security (pull_request) Successful in 1m6s
driver / build (pull_request) Successful in 30s
CI / driver (pull_request) Successful in 2m48s
driver / test (3.13) (pull_request) Successful in 1m4s
driver / lowest (pull_request) Successful in 39s
driver / types (pull_request) Successful in 1m7s
driver / test (3.12) (pull_request) Successful in 1m12s
driver / standard (pull_request) Successful in 17s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 13s
A BSD install takes -D as a destdir, OpenWrt keeps its time zone in uci
and its package database in opkg, and a platform's generated files are its
own; each is a class attribute a driver sets.

Refs #86
2026-11-03 05:33:14 +01:00
christianmanivong 834cb2e0ea fix(scheduled-jobs): an own job goes to /etc/cron.d only where the host has it
driver / security (pull_request) Successful in 1m7s
CI / driver (pull_request) Successful in 3m3s
driver / build (pull_request) Successful in 32s
driver / test (3.12) (pull_request) Successful in 1m21s
driver / lowest (pull_request) Successful in 39s
driver / test (3.13) (pull_request) Successful in 1m21s
driver / standard (pull_request) Successful in 19s
driver / types (pull_request) Successful in 1m12s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 15s
BusyBox crond reads no /etc/cron.d, so an Alpine host without systemd
takes the job in the user's crontab instead of a file it would never run.

Refs #86
2026-11-03 05:30:19 +01:00
christianmanivong f573fa53d2 feat(scheduled-jobs): ScheduledJobsMixin and its POSIX implementation (#86)
driver / security (pull_request) Successful in 1m27s
driver / build (pull_request) Successful in 43s
CI / driver (pull_request) Successful in 3m11s
driver / test (3.13) (pull_request) Successful in 1m18s
driver / lowest (pull_request) Successful in 46s
driver / types (pull_request) Successful in 1m16s
driver / test (3.12) (pull_request) Successful in 1m27s
driver / standard (pull_request) Successful in 20s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 23s
What a host runs on its own schedule as one shape: cron lines, run-parts
scripts, anacron, systemd timers, periodic, a platform's own jobs, Windows
tasks. Every job says who owns its definition and what the driver can do
to it; every write names the digest the caller saw and is refused when the
host's definition changed since.

PosixScheduledJobsMixin reads everything in one framed report as root,
switches a cron line behind #napalm-disabled:, a timer by systemctl and a
run-parts script by its execute bit, runs a job now bounded by timeout, and
writes an application's job as a timer where systemd runs, else in
/etc/cron.d or the user's crontab. napalm_device_types.cron parses cron
expressions, gives their next runs and writes them as OnCalendar lines.

get_cron_jobs and CronJobDict go: nothing called them.

Version 4.26.0, so drivers can require it before the release is tagged.

Closes #86
2026-11-03 05:29:27 +01:00
christianmanivong fdf2f89115 Merge pull request 'chore(release): 4.25.0' (#84) from chore/release-4.25.0 into main
driver / build (push) Successful in 29s
driver / publish (push) Successful in 29s
driver / lint (push) Successful in 14s
driver / test (3.13) (push) Successful in 1m3s
driver / security (push) Successful in 1m2s
driver / lowest (push) Successful in 34s
driver / types (push) Successful in 1m4s
driver / test (3.12) (push) Successful in 1m10s
driver / standard (push) Successful in 17s
CI / driver (push) Successful in 3m6s
2026-11-02 20:01:39 +00:00
christianmanivong d63abff467 chore(release): 4.25.0
driver / security (pull_request) Successful in 1m8s
driver / build (pull_request) Successful in 33s
CI / driver (pull_request) Successful in 3m4s
driver / lowest (pull_request) Successful in 36s
driver / test (3.13) (pull_request) Successful in 1m6s
driver / types (pull_request) Successful in 1m9s
driver / test (3.12) (pull_request) Successful in 1m20s
driver / standard (pull_request) Successful in 23s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 13s
2026-11-02 20:55:27 +01:00
christianmanivong 9d6a1a65ed Merge pull request 'feat(backup): BackupServerDriver, a role base for backup servers (#82)' (#83) from feat/backup-server-role into main
driver / build (push) Successful in 39s
driver / publish (push) Skipped
driver / lint (push) Successful in 15s
driver / test (3.13) (push) Successful in 1m8s
driver / security (push) Successful in 1m27s
driver / lowest (push) Successful in 42s
driver / types (push) Successful in 57s
driver / test (3.12) (push) Successful in 1m7s
driver / standard (push) Successful in 16s
CI / driver (push) Successful in 2m49s
2026-11-02 19:55:05 +00:00
christianmanivong 95e3aa1e26 feat(backup): BackupServerDriver, a role base for backup servers (#82)
driver / security (pull_request) Successful in 58s
CI / driver (pull_request) Successful in 2m22s
driver / build (pull_request) Successful in 29s
driver / test (3.13) (pull_request) Successful in 1m7s
driver / test (3.12) (pull_request) Successful in 1m15s
driver / lowest (pull_request) Successful in 30s
driver / types (pull_request) Successful in 1m6s
driver / standard (pull_request) Successful in 16s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 15s
A backup server keeps datastores, one backup group per source with its
snapshots, and the jobs that keep them healthy: garbage collection,
verify, prune, sync and tape. None of that is a pool, a volume or a
replication job, so it is a role of its own instead of being bent into
StorageDriver. A server that manages its own disks lists StorageDriver
second.

The role declares the reads (datastores, groups with their newest
snapshot, a group's snapshots, jobs, remotes) and the actions (garbage
collection, running a job, verifying, protecting and deleting a
snapshot), which start a task and return its id for get_backup_task.
The shapes live in napalm_device_types.backups.

ClusterStorageDict gains an optional backup_server: which server,
datastore and namespace a hypervisor's backup storage writes to, with
the fingerprint it pins. Matching that to a server is the application's.
2026-11-02 20:46:40 +01:00
christianmanivong a0901e3277 Merge pull request 'chore(release): 4.24.0' (#81) from chore/release-4.24.0 into main
driver / build (push) Successful in 31s
driver / publish (push) Successful in 32s
driver / lint (push) Successful in 17s
driver / test (3.13) (push) Successful in 1m8s
driver / security (push) Successful in 1m16s
driver / lowest (push) Successful in 36s
driver / types (push) Successful in 1m14s
driver / test (3.12) (push) Successful in 1m15s
driver / standard (push) Successful in 19s
CI / driver (push) Successful in 4m45s
2026-11-02 09:45:41 +00:00
christianmanivong 060e210130 chore(release): 4.24.0
driver / security (pull_request) Successful in 1m20s
CI / driver (pull_request) Successful in 3m17s
driver / lowest (pull_request) Successful in 35s
driver / build (pull_request) Successful in 34s
driver / test (3.12) (pull_request) Successful in 1m16s
driver / test (3.13) (pull_request) Successful in 1m18s
driver / standard (pull_request) Successful in 19s
driver / types (pull_request) Successful in 1m12s
driver / lint (pull_request) Successful in 22s
driver / publish (pull_request) Skipped
2026-11-02 10:38:38 +01:00
christianmanivong 31973d2d5f Merge pull request 'feat(lldp): one lldpctl keyvalue parser and the LLDP projections' (#78) from feat/lldpctl-keyvalue-parser into main
driver / build (push) Successful in 39s
driver / publish (push) Skipped
driver / lint (push) Successful in 17s
driver / test (3.13) (push) Successful in 1m13s
driver / security (push) Successful in 1m28s
driver / lowest (push) Successful in 36s
driver / types (push) Successful in 1m24s
driver / test (3.12) (push) Successful in 1m26s
driver / standard (push) Successful in 18s
CI / driver (push) Successful in 3m30s
2026-11-02 09:38:06 +00:00
christianmanivong abda0973e7 Merge remote-tracking branch 'origin/main' into feat/lldpctl-keyvalue-parser
driver / security (pull_request) Successful in 1m13s
driver / build (pull_request) Successful in 35s
CI / driver (pull_request) Successful in 2m55s
driver / test (3.12) (pull_request) Successful in 1m14s
driver / test (3.13) (pull_request) Successful in 1m12s
driver / lowest (pull_request) Successful in 38s
driver / types (pull_request) Successful in 1m7s
driver / standard (pull_request) Successful in 18s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 17s
2026-11-02 10:24:03 +01:00
christianmanivong 4eeaaef236 Merge pull request 'docs(structure): format the code examples as current ruff does' (#80) from fix/ruff-markdown-format into main
driver / build (push) Successful in 36s
driver / publish (push) Skipped
driver / lint (push) Successful in 17s
driver / test (3.13) (push) Successful in 1m7s
driver / security (push) Successful in 1m22s
driver / lowest (push) Successful in 44s
driver / types (push) Successful in 1m15s
driver / test (3.12) (push) Successful in 1m20s
driver / standard (push) Successful in 20s
CI / driver (push) Successful in 3m2s
2026-11-02 09:23:57 +00:00
christianmanivong 3490d8b62a docs(structure): format the code examples as current ruff does
driver / security (pull_request) Successful in 1m29s
CI / driver (pull_request) Successful in 3m21s
driver / lowest (pull_request) Successful in 34s
driver / build (pull_request) Successful in 32s
driver / test (3.12) (pull_request) Successful in 1m14s
driver / test (3.13) (pull_request) Successful in 1m7s
driver / standard (pull_request) Successful in 21s
driver / types (pull_request) Successful in 1m5s
driver / lint (pull_request) Successful in 17s
driver / publish (pull_request) Skipped
ruff 0.17 formats Python blocks in Markdown too, and CI installs ruff
unpinned, so the lint job on main failed on docs/STRUCTURE.md.

Closes #79
2026-11-02 09:18:42 +01:00
christianmanivong 2542081c6f feat(lldp): report the neighbour's chassis MAC in lldp_neighbors
driver / security (pull_request) Successful in 1m13s
CI / driver (pull_request) Failing after 3m1s
driver / lowest (pull_request) Successful in 38s
driver / build (pull_request) Successful in 31s
driver / test (3.12) (pull_request) Successful in 1m19s
driver / test (3.13) (pull_request) Successful in 1m12s
driver / standard (pull_request) Successful in 20s
driver / types (pull_request) Successful in 1m13s
driver / lint (pull_request) Failing after 15s
driver / publish (pull_request) Skipped
An application matches an LLDP neighbour by its chassis MAC first, and
napalm-linux and napalm-netgear report it as mac; the projection did
not, so every driver built on it would have reported neighbours by name
only (NAPALM/napalm-openwrt#31). lldp_neighbors now carries mac, lower
case, when the chassis id is a MAC address (LldpNeighborDict).
2026-11-02 08:50:50 +01:00
christianmanivong 4427be35cc feat(lldp): parse lldpctl keyvalue into one row per neighbour
driver / security (pull_request) Successful in 1m18s
CI / driver (pull_request) Failing after 3m7s
driver / lowest (pull_request) Successful in 50s
driver / build (pull_request) Successful in 37s
driver / test (3.12) (pull_request) Successful in 1m18s
driver / test (3.13) (pull_request) Successful in 1m3s
driver / standard (pull_request) Successful in 16s
driver / types (pull_request) Successful in 1m8s
driver / lint (pull_request) Failing after 16s
driver / publish (pull_request) Skipped
napalm-linux, napalm-openwrt and napalm-proxmox each parsed lldpd's
neighbours on their own, and the copies grouped lines by interface (two
neighbours on one port merged), cut the interface name at its first dot
and looked for chassis.id.value, a key lldpd never prints. The parse
lives here once now: neighbours are split at their via line, the
interface name runs up to the neighbour's first field, ids are read
under the type lldpd prints them with, and a failed lldpctl raises.

The rows project into NAPALM's two LLDP shapes with lldp_neighbors and
lldp_neighbors_detail, the projection part of #10.

Refs #10
2026-11-02 08:42:10 +01:00
christianmanivong ad9ec0fb4d Merge pull request 'docs: a structure that keeps device-types vendor- and product-agnostic' (#37) from docs/structure-plan into main
driver / build (push) Successful in 42s
driver / publish (push) Skipped
driver / lint (push) Failing after 13s
driver / test (3.13) (push) Successful in 1m2s
driver / security (push) Successful in 1m19s
driver / lowest (push) Successful in 36s
driver / types (push) Successful in 1m5s
driver / test (3.12) (push) Successful in 1m0s
driver / standard (push) Successful in 14s
CI / driver (push) Failing after 2m48s
Reviewed-on: #37
2026-11-02 07:19:02 +00:00
christianmanivong 8dd7e8ab9a Merge branch 'main' into docs/structure-plan
driver / security (pull_request) Successful in 58s
CI / driver (pull_request) Failing after 2m22s
driver / build (pull_request) Successful in 27s
driver / test (3.12) (pull_request) Successful in 58s
driver / test (3.13) (pull_request) Successful in 58s
driver / lowest (pull_request) Successful in 30s
driver / types (pull_request) Successful in 1m0s
driver / standard (pull_request) Successful in 14s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Failing after 13s
2026-11-02 07:18:47 +00:00
christianmanivong 935fad2a2d Merge pull request 'fix(updates): read dnf's security advisories from the cached metadata (#76)' (#77) from fix/dnf-security-cache-only into main
driver / build (push) Successful in 31s
driver / publish (push) Skipped
driver / lint (push) Successful in 14s
driver / test (3.13) (push) Successful in 55s
driver / security (push) Successful in 1m3s
driver / lowest (push) Successful in 31s
driver / types (push) Successful in 1m1s
driver / test (3.12) (push) Successful in 1m1s
driver / standard (push) Failing after 2s
CI / driver (push) Failing after 2m6s
Reviewed-on: #77
2026-11-02 07:18:33 +00:00
christianmanivong f337ae7fb8 fix(updates): read dnf's security advisories from the cached metadata
driver / security (pull_request) Successful in 57s
driver / build (pull_request) Successful in 25s
CI / driver (pull_request) Successful in 2m27s
driver / lowest (pull_request) Successful in 32s
driver / test (3.13) (pull_request) Successful in 1m4s
driver / types (pull_request) Successful in 55s
driver / test (3.12) (pull_request) Successful in 1m2s
driver / standard (pull_request) Successful in 15s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 13s
DNF_SECURITY_COMMAND ran `dnf updateinfo list --security`, which downloads
the repositories' metadata first whenever the cached copy has expired. On an
Oracle Linux host on OCI that took longer than netOrk's whole 120 s poll.
It now runs with --cacheonly, as APT_UPGRADABLE_COMMAND reads apt's local
index; refreshing stays `dnf makecache`. Without a cache dnf fails at once,
and the __DNF_RC marker already turns that into "security unknown".

Closes #76
2026-11-02 08:07:48 +01:00
christianmanivong e2aebeadeb Merge pull request 'chore(release): 4.23.0' (#75) from chore/release-4.23.0 into main
driver / build (push) Successful in 30s
driver / publish (push) Successful in 44s
driver / lint (push) Successful in 19s
driver / test (3.13) (push) Successful in 1m16s
driver / security (push) Successful in 1m6s
driver / lowest (push) Successful in 32s
driver / types (push) Successful in 1m18s
driver / test (3.12) (push) Successful in 1m24s
driver / standard (push) Successful in 21s
CI / driver (push) Successful in 3m29s
2026-11-01 23:26:04 +00:00
christianmanivong 36cf117b95 chore(release): 4.23.0
driver / security (pull_request) Successful in 1m0s
CI / driver (pull_request) Successful in 2m22s
driver / lowest (pull_request) Successful in 31s
driver / build (pull_request) Successful in 31s
driver / test (3.12) (pull_request) Successful in 1m2s
driver / test (3.13) (pull_request) Successful in 58s
driver / standard (pull_request) Successful in 16s
driver / types (pull_request) Successful in 57s
driver / lint (pull_request) Successful in 13s
driver / publish (pull_request) Skipped
2026-11-02 00:21:38 +01:00
christianmanivong 02d461036b Merge pull request 'feat: TLS certificates by trust on first use' (#74) from feat/tls-fingerprint-tofu into main
driver / lint (push) Successful in 15s
driver / build (push) Successful in 27s
driver / publish (push) Skipped
driver / test (3.13) (push) Successful in 57s
driver / security (push) Successful in 1m3s
driver / lowest (push) Successful in 34s
driver / types (push) Successful in 1m5s
driver / test (3.12) (push) Successful in 1m4s
driver / standard (push) Successful in 16s
CI / driver (push) Successful in 2m16s
2026-11-01 23:21:03 +00:00
christianmanivong 63039ce758 fix(types): the adapter's ignore holds with and without requests' stubs
driver / security (pull_request) Successful in 53s
CI / driver (pull_request) Successful in 2m12s
driver / build (pull_request) Successful in 26s
driver / test (3.13) (pull_request) Successful in 1m4s
driver / test (3.12) (pull_request) Successful in 1m10s
driver / lowest (pull_request) Successful in 30s
driver / standard (pull_request) Successful in 15s
driver / types (pull_request) Successful in 57s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 14s
CI installs types-requests, so HTTPAdapter is typed there and the
ignore for subclassing an untyped base is unused; locally it is needed.
2026-11-02 00:15:55 +01:00
christianmanivong 516fb18c0e Merge remote-tracking branch 'origin/main' into feat/tls-fingerprint-tofu
driver / security (pull_request) Successful in 59s
CI / driver (pull_request) Failing after 2m34s
driver / lowest (pull_request) Successful in 34s
driver / build (pull_request) Successful in 28s
driver / test (3.13) (pull_request) Successful in 1m14s
driver / test (3.12) (pull_request) Successful in 1m27s
driver / standard (pull_request) Successful in 15s
driver / types (pull_request) Failing after 1m35s
driver / lint (pull_request) Successful in 14s
driver / publish (pull_request) Skipped
# Conflicts:
#	CHANGELOG.md
2026-11-02 00:06:53 +01:00
christianmanivong 8d32e70cd7 Merge pull request 'feat(fixes): configure_lldp_agent(interfaces) template' (#73) from feat/lldp-agent-template into main
driver / build (push) Successful in 37s
driver / publish (push) Skipped
driver / lint (push) Successful in 25s
driver / test (3.13) (push) Successful in 1m8s
driver / security (push) Successful in 1m15s
driver / lowest (push) Successful in 34s
driver / types (push) Successful in 1m15s
driver / test (3.12) (push) Failing after 6s
driver / standard (push) Successful in 24s
CI / driver (push) Failing after 2m19s
2026-11-01 23:06:48 +00:00
christianmanivong 51e8c849bb Merge remote-tracking branch 'origin/main' into feat/lldp-agent-template
driver / security (pull_request) Successful in 1m10s
CI / driver (pull_request) Successful in 2m45s
driver / lowest (pull_request) Successful in 38s
driver / build (pull_request) Successful in 33s
driver / test (3.13) (pull_request) Successful in 1m6s
driver / test (3.12) (pull_request) Successful in 1m14s
driver / standard (pull_request) Successful in 18s
driver / types (pull_request) Successful in 1m6s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 14s
# Conflicts:
#	CHANGELOG.md
2026-11-02 00:01:46 +01:00
christianmanivong 6aadcc3d91 Merge pull request 'feat(snmp): let a health profile mark a value unknown' (#72) from feat/health-profile-unknown into main
driver / build (push) Successful in 31s
driver / publish (push) Skipped
driver / lint (push) Successful in 17s
driver / test (3.13) (push) Successful in 1m15s
driver / security (push) Failing after 3s
driver / lowest (push) Successful in 41s
driver / types (push) Successful in 1m4s
driver / test (3.12) (push) Successful in 1m17s
driver / standard (push) Successful in 19s
CI / driver (push) Failing after 2m6s
2026-11-01 23:01:40 +00:00
christianmanivong e98d4763f3 Merge remote-tracking branch 'origin/main' into feat/health-profile-unknown
driver / security (pull_request) Successful in 1m11s
driver / build (pull_request) Successful in 29s
CI / driver (pull_request) Successful in 2m57s
driver / test (3.13) (pull_request) Successful in 1m9s
driver / lowest (pull_request) Successful in 40s
driver / types (pull_request) Successful in 1m7s
driver / test (3.12) (pull_request) Successful in 1m8s
driver / standard (pull_request) Successful in 19s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 18s
# Conflicts:
#	CHANGELOG.md
2026-11-01 23:56:39 +01:00
christianmanivong 24f4be3b7e Merge pull request 'feat: VMProvisionIncomplete names the VM a failed creation left behind' (#70) from feat/vm-create-partial-failure into main
driver / lint (push) Successful in 18s
driver / build (push) Successful in 38s
driver / publish (push) Skipped
driver / test (3.13) (push) Successful in 1m9s
driver / security (push) Successful in 1m10s
driver / lowest (push) Successful in 35s
driver / types (push) Successful in 1m6s
driver / test (3.12) (push) Successful in 1m16s
driver / standard (push) Successful in 19s
CI / driver (push) Successful in 2m38s
2026-11-01 22:56:34 +00:00
christianmanivong f4354f498e refactor: the SNMP health profile shapes in snmp_health_profile.py
models.py is over the file size limit and may not grow; the unknown
key pushed it past its recorded length. SnmpHealthProfile,
SnmpValueSpec, SnmpMemorySpec and SnmpInterfacesSpec move to
napalm_device_types/snmp_health_profile.py. The package exports them
as before and models re-exports them, so existing imports keep working.
The baseline records the smaller models.py.
2026-11-01 23:05:13 +01:00
christianmanivong 73771aba00 docs: a tighter create_vm_from_cloud_init docstring
driver / security (pull_request) Successful in 1m6s
CI / driver (pull_request) Successful in 2m52s
driver / build (pull_request) Successful in 30s
driver / test (3.12) (pull_request) Successful in 1m12s
driver / lowest (pull_request) Successful in 35s
driver / test (3.13) (pull_request) Successful in 1m11s
driver / standard (pull_request) Successful in 16s
driver / types (pull_request) Successful in 1m7s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 18s
The VMProvisionIncomplete entry pushed the method past its recorded
length in the quality baseline. The image-cache note joins the
paragraph it repeated, and the Raises entry takes two lines; nothing
it says is dropped.
2026-11-01 23:02:46 +01:00
christianmanivong c9571d59f1 Merge remote-tracking branch 'origin/main' into feat/tls-fingerprint-tofu
# Conflicts:
#	CHANGELOG.md
2026-11-01 23:01:38 +01:00
christianmanivong cf862af026 Merge remote-tracking branch 'origin/main' into feat/lldp-agent-template
# Conflicts:
#	CHANGELOG.md
2026-11-01 23:01:38 +01:00
christianmanivong 83ea13f88e Merge remote-tracking branch 'origin/main' into feat/health-profile-unknown
# Conflicts:
#	CHANGELOG.md
2026-11-01 23:01:38 +01:00
christianmanivong 4938056d20 Merge remote-tracking branch 'origin/main' into feat/vm-create-partial-failure
# Conflicts:
#	CHANGELOG.md
2026-11-01 23:01:38 +01:00
christianmanivong 9c3be46d8a Merge pull request 'feat: HA state, backups, quorum votes, migrate_vm and node maintenance' (#71) from feat/vm-lifecycle into main
driver / build (push) Successful in 27s
driver / publish (push) Skipped
driver / lint (push) Successful in 15s
driver / test (3.13) (push) Successful in 53s
driver / security (push) Successful in 56s
driver / lowest (push) Successful in 30s
driver / types (push) Successful in 58s
driver / test (3.12) (push) Successful in 1m3s
driver / standard (push) Successful in 16s
CI / driver (push) Successful in 2m29s
2026-11-01 21:48:11 +00:00
christianmanivong a6289cb24a feat: HA state, backups, quorum votes, migrate_vm and node maintenance
driver / security (pull_request) Successful in 1m3s
driver / build (pull_request) Successful in 27s
CI / driver (pull_request) Successful in 2m30s
driver / test (3.12) (pull_request) Successful in 59s
driver / lowest (pull_request) Successful in 30s
driver / test (3.13) (pull_request) Successful in 58s
driver / types (pull_request) Successful in 58s
driver / standard (pull_request) Successful in 15s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 12s
get_clusters() may also report the cluster's HA resources, backup jobs, the
newest backup per guest and storage, and each node's quorum votes (all
NotRequired; None means not read). HypervisorDriver declares
get_vm_migration_preflight(), migrate_vm() and set_node_maintenance(); the
preflight shape lives in vm_migration.py.

Part of NetOrk/netork#1820 (stage 7: #759, #760, #761).
2026-11-01 22:15:01 +01:00
christianmanivong 46c1c9dc4e feat: TLS certificates by trust on first use
driver / security (pull_request) Successful in 1m13s
CI / driver (pull_request) Failing after 3m4s
driver / lowest (pull_request) Successful in 38s
driver / build (pull_request) Successful in 34s
driver / test (3.12) (pull_request) Successful in 1m14s
driver / test (3.13) (pull_request) Successful in 1m4s
driver / standard (pull_request) Successful in 17s
driver / types (pull_request) Failing after 1m10s
driver / lint (pull_request) Successful in 20s
driver / publish (pull_request) Skipped
Devices ship self-signed certificates, so a driver either fails the CA
check on every one of them or turns verification off and sends the
login to whatever answers on the address. tls_certificates pins the
certificate by its SHA-256 fingerprint instead, shaped like
ssh_host_keys: the caller passes the fingerprint to expect by port
(optional_args["tls_fingerprints"]), the driver refuses any other
certificate before a request carries credentials, and every certificate
seen goes to optional_args["tls_certificate_observer"]. Without a
fingerprint the first certificate is accepted.

CertificateCheck checks the certificate in a handshake of its own, then
pins a requests session to it (urllib3's assert_fingerprint, direct and
through a proxy), so no other host can step in between the check and
the login.

Refs NAPALM/napalm-hpe-aruba-procurve#17
2026-11-01 22:13:20 +01:00
christianmanivong bc17862a44 feat(fixes): configure_lldp_agent(interfaces) template
driver / security (pull_request) Successful in 58s
driver / build (pull_request) Successful in 28s
CI / driver (pull_request) Successful in 2m32s
driver / test (3.13) (pull_request) Successful in 56s
driver / lowest (pull_request) Successful in 30s
driver / types (pull_request) Successful in 57s
driver / test (3.12) (pull_request) Successful in 1m7s
driver / standard (pull_request) Successful in 15s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 12s
LLDP getters used to install, start and reconfigure lldpd on their own,
on every poll. Getters only read now, so starting the agent needs an
action the application calls. The template installs the agent through
ensure_package, has the driver write the interfaces it runs on, then
enables and restarts it through manage_service. The caller names the
interfaces: an agent on every interface also advertises on a WAN port.

Refs NAPALM/napalm-openwrt#8
2026-11-01 22:11:21 +01:00
christianmanivong 9f6ffa8a25 feat(snmp): let a health profile mark a value unknown
driver / security (pull_request) Successful in 1m1s
driver / build (pull_request) Successful in 27s
CI / driver (pull_request) Failing after 2m30s
driver / test (3.12) (pull_request) Successful in 1m3s
driver / test (3.13) (pull_request) Successful in 56s
driver / lowest (pull_request) Successful in 35s
driver / types (pull_request) Successful in 58s
driver / standard (pull_request) Failing after 15s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 13s
SnmpValueSpec gains `unknown`. A driver whose device's agent answers a
value wrongly says so in its SNMP_HEALTH_PROFILE, and the application
reports no value for it instead of the standard reading or a default.
OpenBSD's net-snmp reports the CPU as never idle, so a reading of
ssCpuIdle shows every OpenBSD host at 100 % (NAPALM/napalm-bsd#7).
2026-11-01 22:11:19 +01:00
christianmanivong 4150570a94 feat: VMProvisionIncomplete names the VM a failed creation left behind
driver / security (pull_request) Successful in 1m15s
CI / driver (pull_request) Failing after 3m1s
driver / lowest (pull_request) Successful in 37s
driver / build (pull_request) Successful in 35s
driver / test (3.12) (pull_request) Successful in 1m13s
driver / test (3.13) (pull_request) Successful in 1m6s
driver / standard (pull_request) Failing after 17s
driver / types (pull_request) Successful in 1m5s
driver / lint (pull_request) Successful in 16s
driver / publish (pull_request) Skipped
create_vm_from_cloud_init raised a plain error once the VM existed, so the
caller had no id to remove it by. The new RuntimeError carries vmid and
node, with the failure as its cause; a failure before the VM exists raises
as before.

Refs NAPALM/napalm-proxmox#28
2026-11-01 22:09:01 +01:00
christianmanivong a809e6c02f Merge pull request 'feat: get_clusters() and create_vm_from_cloud_init(node=)' (#68) from feat/cluster-inventory into main
driver / build (push) Successful in 35s
driver / publish (push) Skipped
driver / lint (push) Successful in 20s
driver / test (3.13) (push) Successful in 1m21s
driver / security (push) Successful in 1m16s
driver / lowest (push) Successful in 39s
driver / types (push) Successful in 1m30s
driver / test (3.12) (push) Successful in 1m26s
driver / standard (push) Failing after 7s
CI / driver (push) Failing after 2m49s
2026-11-01 18:41:05 +00:00
christianmanivong 5b894913e1 refactor: the cluster shapes in clusters.py, a shorter provisioning docstring
driver / security (pull_request) Successful in 1m4s
CI / driver (pull_request) Successful in 2m16s
driver / build (pull_request) Successful in 27s
driver / test (3.13) (pull_request) Successful in 57s
driver / test (3.12) (pull_request) Successful in 1m4s
driver / lowest (pull_request) Successful in 32s
driver / standard (pull_request) Successful in 15s
driver / types (pull_request) Successful in 1m1s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 14s
models.py is over the file size limit and may not grow, and the declaration
of create_vm_from_cloud_init() may not get longer: the four cluster shapes
move to napalm_device_types/clusters.py (exported from the package as
before), and the docstring says the same in fewer lines.

Refs NetOrk/netork#484
2026-11-01 17:08:10 +01:00
christianmanivong 00aa5efef0 feat: get_clusters() and create_vm_from_cloud_init(node=)
driver / security (pull_request) Successful in 1m0s
CI / driver (pull_request) Failing after 2m32s
driver / lowest (pull_request) Successful in 32s
driver / build (pull_request) Successful in 28s
driver / test (3.13) (pull_request) Successful in 1m13s
driver / test (3.12) (pull_request) Successful in 1m19s
driver / standard (pull_request) Failing after 19s
driver / types (pull_request) Successful in 1m10s
driver / lint (pull_request) Successful in 16s
driver / publish (pull_request) Skipped
A hypervisor reports every cluster its connection can see: the cluster's
name and quorum, its nodes (offline ones too, the local one marked) with
CPU and memory, and the storages and networks with the nodes that see
each. A standalone host is a cluster of one without a name; a manager
reports each cluster and standalone host it runs. Declared only, so a
capability check stays truthful.

create_vm_from_cloud_init() takes the node to create the VM on; a driver
that can only create on its own node refuses another.

Refs NetOrk/netork#484, NetOrk/netork#1822
2026-11-01 16:38:38 +01:00
christianmanivong 97cb8779c5 Merge pull request 'feat: netOrk's own account through the current login (NetOrk/netork#1817)' (#67) from feature/system-user-1817 into main
driver / build (push) Successful in 26s
driver / publish (push) Skipped
driver / lint (push) Successful in 15s
driver / test (3.13) (push) Successful in 1m0s
driver / security (push) Successful in 1m0s
driver / lowest (push) Successful in 30s
driver / types (push) Successful in 57s
driver / test (3.12) (push) Successful in 1m6s
driver / standard (push) Successful in 16s
CI / driver (push) Successful in 2m16s
2026-11-01 12:39:41 +00:00
christianmanivong 47751d12d6 fix: another service's account of the same name is left alone (NetOrk/netork#1817)
driver / security (pull_request) Successful in 55s
CI / driver (pull_request) Successful in 56s
driver / lowest (pull_request) Successful in 30s
driver / build (pull_request) Successful in 26s
driver / test (3.13) (pull_request) Successful in 59s
driver / test (3.12) (pull_request) Successful in 56s
driver / standard (pull_request) Successful in 15s
driver / types (pull_request) Successful in 1m1s
driver / lint (pull_request) Successful in 14s
driver / publish (pull_request) Skipped
On bw-01, netork is the satellite's own user (nologin, in docker).
add_service_account opened it, gave it the passwordless sudo rule and netOrk's
key, and only the login as it failed. An account that is there and cannot log
in (nologin, false) is now refused before anything changes.
2026-11-01 13:36:38 +01:00
christianmanivong 3f59023f46 feat: a fixed uid for netOrk's own account (NetOrk/netork#1817)
driver / security (pull_request) Successful in 1m3s
CI / driver (pull_request) Successful in 1m10s
driver / lowest (pull_request) Successful in 31s
driver / build (pull_request) Successful in 26s
driver / test (3.13) (pull_request) Successful in 1m2s
driver / test (3.12) (pull_request) Successful in 1m10s
driver / standard (pull_request) Successful in 16s
driver / types (pull_request) Successful in 1m8s
driver / lint (pull_request) Successful in 14s
driver / publish (pull_request) Skipped
add_service_account(uid=) makes a new account with that uid once no other
account has it, through the driver's _account_create_with_uid; an account that
is there keeps its own, as a uid is never changed under files that carry it.
The infrastructure's build standard wants the same uid on every host.
2026-11-01 12:42:30 +01:00
christianmanivong 666be091e3 feat: netOrk's own account through the current login (NetOrk/netork#1817)
driver / security (pull_request) Successful in 1m19s
CI / driver (pull_request) Failing after 2m35s
driver / build (pull_request) Successful in 38s
driver / test (3.12) (pull_request) Successful in 1m9s
driver / test (3.13) (pull_request) Successful in 1m6s
driver / lowest (pull_request) Successful in 50s
driver / standard (pull_request) Successful in 21s
driver / types (pull_request) Successful in 1m10s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Failing after 7s
add_service_account(name, public_key) creates or reopens the account through
the login the driver uses today: a key and no password, and a rule of its own
in 85-service-<name> that lets it run everything as root without a password,
checked with visudo and sudo -l. Root is refused; run again with the account
the driver already logs in with, it only adds what is missing.
2026-11-01 12:03:01 +01:00
christianmanivong 25cc264306 Merge pull request 'fix(apt_proxy): https sources take the http proxy too (NetOrk/netork#1811)' (#64) from fix/apt-https-proxy into main
driver / build (push) Successful in 26s
driver / publish (push) Skipped
driver / lint (push) Successful in 15s
driver / test (3.13) (push) Successful in 1m4s
driver / security (push) Successful in 59s
driver / lowest (push) Successful in 30s
driver / types (push) Successful in 1m4s
driver / test (3.12) (push) Successful in 1m10s
driver / standard (push) Successful in 15s
CI / driver (push) Successful in 2m18s
2026-11-01 07:25:56 +00:00
christianmanivong 86959b7599 Merge remote-tracking branch 'origin/main' into fix/apt-https-proxy
driver / security (pull_request) Successful in 1m7s
CI / driver (pull_request) Successful in 2m50s
driver / build (pull_request) Successful in 33s
driver / test (3.12) (pull_request) Successful in 1m12s
driver / test (3.13) (pull_request) Successful in 1m0s
driver / lowest (pull_request) Successful in 33s
driver / types (pull_request) Successful in 1m11s
driver / standard (pull_request) Successful in 16s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 15s
2026-11-01 08:21:18 +01:00
christianmanivong 8664accbc4 Merge pull request 'refactor(ssh_server): split three functions under the standard's line limit (#65)' (#66) from fix/ssh-server-ratchet into main
driver / build (push) Successful in 28s
driver / publish (push) Skipped
driver / lint (push) Successful in 17s
driver / test (3.13) (push) Successful in 1m7s
driver / security (push) Successful in 1m16s
driver / lowest (push) Successful in 31s
driver / types (push) Successful in 1m1s
driver / test (3.12) (push) Successful in 1m3s
driver / standard (push) Successful in 21s
CI / driver (push) Successful in 2m37s
2026-11-01 07:21:11 +00:00
christianmanivong 449a6b13ca refactor(ssh_server): split three functions under the standard's line limit (#65)
driver / security (pull_request) Successful in 1m9s
CI / driver (pull_request) Successful in 3m9s
driver / lowest (pull_request) Successful in 35s
driver / build (pull_request) Successful in 33s
driver / test (3.12) (pull_request) Successful in 1m28s
driver / test (3.13) (pull_request) Successful in 1m18s
driver / standard (pull_request) Successful in 17s
driver / types (pull_request) Successful in 1m18s
driver / lint (pull_request) Successful in 15s
driver / publish (pull_request) Skipped
harden_ssh_server (70), _apply_or_restore (51) and exempt_from_ssh_penalties
(81) had grown past the 50-line limit with #63, and `driver / standard` was
red on main. The reading and planning, the putting back, the session's
address, the reasons to leave the exemption and its write move into methods
of their own. Same behaviour, same tests; harden_ssh_server leaves the
baseline.
2026-11-01 08:17:27 +01:00
christianmanivong c71ddd7fe6 refactor(apt_proxy): smaller parts, under the standard's complexity limit
driver / security (pull_request) Successful in 1m33s
driver / build (pull_request) Successful in 55s
CI / driver (pull_request) Failing after 3m0s
driver / test (3.13) (pull_request) Successful in 1m6s
driver / lowest (pull_request) Successful in 47s
driver / types (pull_request) Successful in 1m8s
driver / test (3.12) (pull_request) Successful in 1m12s
driver / standard (pull_request) Failing after 20s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 17s
The quality ratchet counted _dump, parse_apt_proxy and _https_direct over
complexity 10 (NetOrk/netork#1811). Same behaviour, same tests.
2026-11-01 08:13:20 +01:00
christianmanivong 3b6c5f9585 fix(apt_proxy): https sources take the http proxy too (NetOrk/netork#1811)
driver / security (pull_request) Successful in 1m1s
CI / driver (pull_request) Failing after 2m22s
driver / lowest (pull_request) Successful in 33s
driver / build (pull_request) Successful in 28s
driver / test (3.13) (pull_request) Successful in 1m3s
driver / test (3.12) (pull_request) Successful in 1m7s
driver / standard (pull_request) Failing after 18s
driver / types (pull_request) Successful in 1m7s
driver / lint (pull_request) Successful in 17s
driver / publish (pull_request) Skipped
apt's https method asks Acquire::https::Proxy and then Acquire::http::Proxy,
for the host and then for all. An 00proxy with the http line alone therefore
sent download.docker.com and every other https source to apt-cacher-ng, which
answers 403 CONNECT denied without a PassThroughPattern, and the whole upgrade
failed.

- The report carries https_proxy, https_per_host and https_settings;
  https_proxy_for(report, host) is the proxy apt takes for one https host.
- check_https_source() fetches one https source the way apt does.
- configure_package_proxy gains an https step: one https source through the
  new proxy. Where the proxy refuses it, 00proxy also says
  Acquire::https::Proxy "DIRECT", later https lines to the same proxy are
  commented out, and the source is fetched direct. An HTTP error from the
  mirror counts as let through.
- apt_fetch_command takes https URLs; comment_out_command takes a scheme.
2026-11-01 07:58:05 +01:00
christianmanivong 1c71fe6896 Merge pull request 'feat: LocalAccountsMixin, personal accounts added and locked again (NetOrk/netork#1793)' (#62) from feat/local-accounts into main
driver / lint (push) Successful in 14s
driver / build (push) Successful in 31s
driver / publish (push) Skipped
driver / test (3.13) (push) Successful in 1m2s
driver / security (push) Successful in 1m11s
driver / lowest (push) Failing after 5s
driver / types (push) Successful in 1m6s
driver / test (3.12) (push) Successful in 1m4s
driver / standard (push) Failing after 16s
CI / driver (push) Failing after 2m47s
Reviewed-on: #62
2026-10-31 23:45:34 +00:00
christianmanivong 9159106837 Merge remote-tracking branch 'origin/main' into feat/local-accounts
driver / security (pull_request) Successful in 59s
CI / driver (pull_request) Failing after 2m35s
driver / build (pull_request) Successful in 32s
driver / test (3.12) (pull_request) Successful in 1m7s
driver / lowest (pull_request) Successful in 31s
driver / test (3.13) (pull_request) Successful in 58s
driver / types (pull_request) Successful in 59s
driver / standard (pull_request) Failing after 18s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 13s
# Conflicts:
#	CHANGELOG.md
2026-11-01 00:45:06 +01:00
christianmanivong 28ff14589b Merge pull request 'feat: the managing client and sshd's per-source penalties (NetOrk/netork#1800)' (#63) from feat/ssh-penalties-1800 into main
driver / build (push) Successful in 30s
driver / publish (push) Skipped
driver / lint (push) Successful in 14s
driver / test (3.13) (push) Successful in 58s
driver / security (push) Successful in 1m4s
driver / lowest (push) Successful in 30s
driver / types (push) Successful in 1m2s
driver / test (3.12) (push) Successful in 1m4s
driver / standard (push) Failing after 17s
CI / driver (push) Failing after 2m21s
Reviewed-on: #63
2026-10-31 23:29:09 +00:00
christianmanivong 5fffb3b886 feat: the managing client and sshd's per-source penalties (NetOrk/netork#1800)
driver / security (pull_request) Successful in 1m4s
CI / driver (pull_request) Failing after 2m24s
driver / build (pull_request) Successful in 30s
driver / test (3.12) (pull_request) Successful in 1m11s
driver / test (3.13) (pull_request) Successful in 1m7s
driver / lowest (pull_request) Successful in 33s
driver / standard (pull_request) Failing after 9s
driver / types (pull_request) Successful in 1m14s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 22s
OpenSSH 9.8+ penalises a source whose connections do not authenticate --
a scanner's probes -- and then drops every connection from it, the
managing client's included.

- OpenSshServerMixin.exempt_from_ssh_penalties puts the address the host
  sees the client at ($SSH_CONNECTION) into PerSourcePenaltyExemptList,
  next to the host's own exemptions, in its own drop-in. Only where sshd
  has the penalties (an older one refuses the directive) and includes the
  drop-in directory; checked with sshd -t, sshd -T and a new session, and
  taken back on any failure. The check-and-restore step of
  harden_ssh_server is shared for it.
- retry_refused tries a connection again after 1, 3, 7 and 15 s while
  sshd closes it before its banner; HostKeyCheck.connect uses it.
2026-11-01 00:24:11 +01:00
christianmanivong 2846824a21 feat: a host may add a key file sshd reads (NetOrk/netork#1793)
driver / security (pull_request) Successful in 1m5s
CI / driver (pull_request) Successful in 2m45s
driver / lowest (pull_request) Successful in 39s
driver / build (pull_request) Successful in 32s
driver / test (3.12) (pull_request) Successful in 1m14s
driver / test (3.13) (pull_request) Successful in 1m23s
driver / standard (pull_request) Successful in 17s
driver / types (pull_request) Successful in 1m6s
driver / lint (pull_request) Successful in 14s
driver / publish (pull_request) Skipped
_account_key_files says which files hold an account's keys; the default is
~/.ssh/authorized_keys and authorized_keys2. OpenMediaVault adds the file it
deploys under /var/lib/openmediavault/ssh/authorized_keys, so reading an
account back sees every key sshd would take.
2026-10-31 23:31:05 +01:00
christianmanivong 0545d1d814 feat: a driver may keep the account keys elsewhere (NetOrk/netork#1793)
driver / security (pull_request) Successful in 1m9s
CI / driver (pull_request) Successful in 2m44s
driver / lowest (pull_request) Successful in 36s
driver / build (pull_request) Successful in 34s
driver / test (3.13) (pull_request) Successful in 1m8s
driver / test (3.12) (pull_request) Successful in 1m14s
driver / standard (pull_request) Successful in 16s
driver / types (pull_request) Successful in 1m6s
driver / lint (pull_request) Successful in 13s
driver / publish (pull_request) Skipped
OpenMediaVault keeps a user's keys in its own database and deploys them
to /var/lib/openmediavault/ssh/authorized_keys, and creates a home only
when its home directory setting is on. The key store becomes three hooks
with the ~/.ssh default: _account_read_keys, _account_add_key and
_account_remove_keys, each given the account's home.
2026-10-31 23:29:46 +01:00
christianmanivong 49c65cbd5d feat: LocalAccountsMixin, personal accounts added and locked again (NetOrk/netork#1793)
driver / security (pull_request) Successful in 1m21s
CI / driver (pull_request) Successful in 3m6s
driver / lowest (pull_request) Successful in 46s
driver / build (pull_request) Successful in 36s
driver / test (3.13) (pull_request) Successful in 1m15s
driver / test (3.12) (pull_request) Successful in 1m41s
driver / standard (pull_request) Successful in 25s
driver / types (pull_request) Successful in 1m36s
driver / lint (pull_request) Successful in 30s
driver / publish (pull_request) Skipped
get_local_accounts(names) reads a person's account back: groups, lock,
sudo, key files. add_local_account() creates or reopens one with the
person's key, optional password hash and optional sudo rule;
lock_local_account() takes the keys and sudo away and refuses every
login, keeping the home.

Every command root runs is one fixed command, so a sudoers allowlist can
name it. The key files are read and written as the person (sudo -n -u),
never as root, since a link in a home the person owns may point anywhere.
The sudo rule is a drop-in of its own, checked with visudo before and
sudo -l after; a rule elsewhere (cloud-init's) is reported, not edited.
The login user's own account and system accounts are refused.

A driver supplies creating, opening, locking, reading the lock and
leaving a group.
2026-10-31 23:24:27 +01:00
christianmanivong 888a7229dc Merge pull request 'feat: SudoHistoryMixin, the sudo password left in shell histories (NetOrk/netork#1090)' (#60) from feat/sudo-history-cleanup into main
driver / build (push) Successful in 34s
driver / publish (push) Skipped
driver / lint (push) Successful in 16s
driver / test (3.13) (push) Successful in 1m3s
driver / security (push) Successful in 1m5s
driver / lowest (push) Successful in 32s
driver / types (push) Successful in 1m1s
driver / test (3.12) (push) Successful in 1m10s
driver / standard (push) Successful in 17s
CI / driver (push) Successful in 2m25s
2026-10-30 20:47:48 +00:00
christianmanivong 0f0064c2d7 test: the fake host never answers with this machine's uid
driver / security (pull_request) Successful in 1m2s
CI / driver (pull_request) Successful in 2m32s
driver / lowest (pull_request) Successful in 36s
driver / build (pull_request) Successful in 30s
driver / test (3.12) (pull_request) Successful in 1m14s
driver / test (3.13) (pull_request) Successful in 1m16s
driver / standard (pull_request) Successful in 19s
driver / types (pull_request) Successful in 1m14s
driver / lint (pull_request) Successful in 15s
driver / publish (pull_request) Skipped
CI runs as root, which the owner step skips, so three owner tests failed there
while they passed for a normal user.
2026-10-30 21:44:40 +01:00
christianmanivong cb90fa9835 feat: SudoHistoryMixin, the sudo password left in shell histories (NetOrk/netork#1090)
driver / security (pull_request) Successful in 1m8s
driver / build (pull_request) Successful in 29s
CI / driver (pull_request) Failing after 2m24s
driver / test (3.13) (pull_request) Failing after 1m3s
driver / lowest (pull_request) Failing after 33s
driver / types (pull_request) Successful in 1m8s
driver / test (3.12) (pull_request) Failing after 1m9s
driver / standard (pull_request) Failing after 7s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 20s
A driver that typed `echo '<password>' | sudo -S -p "" <command>` into an
interactive shell left that line, password and all, in the login user's
shell history once per command, and where only the part before the first
`&&` ran as root, files root should own were left to the login user.

get_history_sudo_lines() counts the lines per history file with grep -c,
as the login user, without reading any back. remove_history_sudo_lines()
deletes exactly those lines with sed -i, overwrites the backup with
shred -u, and as root gives the driver's SUDO_ROOT_FILES back to root:root
where each is a regular file, not a link, owned by the login user. It
reports each file's lines before and after. Nothing goes to stdin.

The driver names the marker and the files and supplies run_command.
2026-10-30 21:09:24 +01:00
christianmanivong 551ed0a29d Merge pull request 'feat: NetworkTargetDict.uplinks, the NICs a bridge leaves through (NetOrk/netork#1325)' (#58) from feat/network-target-uplinks into main
driver / build (push) Successful in 32s
driver / publish (push) Skipped
driver / lint (push) Successful in 16s
driver / test (3.13) (push) Successful in 1m8s
driver / security (push) Successful in 1m8s
driver / lowest (push) Successful in 31s
driver / types (push) Successful in 1m3s
driver / test (3.12) (push) Successful in 1m16s
driver / standard (push) Successful in 18s
CI / driver (push) Successful in 2m34s
2026-10-30 09:54:19 +00:00
christianmanivong 0dc1a598e2 Merge remote-tracking branch 'origin/main' into feat/network-target-uplinks
driver / security (pull_request) Successful in 1m5s
CI / driver (pull_request) Successful in 3m1s
driver / lowest (pull_request) Successful in 32s
driver / build (pull_request) Successful in 28s
driver / test (3.12) (pull_request) Successful in 1m9s
driver / test (3.13) (pull_request) Successful in 59s
driver / standard (pull_request) Successful in 24s
driver / types (pull_request) Successful in 1m2s
driver / lint (pull_request) Successful in 16s
driver / publish (pull_request) Skipped
# Conflicts:
#	CHANGELOG.md
2026-10-30 10:20:48 +01:00
christianmanivong 131346c228 Merge pull request 'feat: ARP_AGE_UNKNOWN, the age of a neighbour the device does not report' (#57) from fix/arp-unknown-age into main
driver / lint (push) Successful in 17s
driver / build (push) Successful in 42s
driver / publish (push) Skipped
driver / test (3.13) (push) Successful in 1m7s
driver / security (push) Successful in 1m16s
driver / lowest (push) Successful in 33s
driver / types (push) Successful in 1m5s
driver / test (3.12) (push) Successful in 1m12s
driver / standard (push) Successful in 19s
CI / driver (push) Successful in 3m5s
2026-10-30 09:20:21 +00:00
christianmanivong a5b7b540c6 feat: NetworkTargetDict.uplinks, the NICs a bridge leaves through (NetOrk/netork#1325)
driver / security (pull_request) Successful in 1m12s
driver / build (pull_request) Successful in 32s
CI / driver (pull_request) Successful in 2m47s
driver / test (3.12) (pull_request) Successful in 1m3s
driver / test (3.13) (pull_request) Successful in 59s
driver / lowest (pull_request) Successful in 36s
driver / types (pull_request) Successful in 1m1s
driver / standard (pull_request) Successful in 18s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 14s
A VM on a VLAN-aware bridge without a VLAN tag only gets an address when
the switch port behind the bridge carries an untagged network. To find
that port, the caller needs the bridge's physical NICs: their LLDP
neighbours name the switch and the port.

uplinks lists them in groups, one per way out of the bridge: a bond's
slaves together, a single NIC alone. A bond is one link, so a caller can
let one slave's neighbour speak for the others when lldpd sees only one.
Internal, VLAN and virtual ports are no uplinks. A missing key means the
driver cannot tell; an empty list is a bridge without uplink.
2026-10-30 09:27:23 +01:00
christianmanivong 874c6c5a92 feat: ARP_AGE_UNKNOWN, the age of a neighbour the device does not report
driver / security (pull_request) Successful in 1m15s
CI / driver (pull_request) Successful in 3m3s
driver / build (pull_request) Successful in 35s
driver / test (3.12) (pull_request) Successful in 1m13s
driver / test (3.13) (pull_request) Successful in 1m8s
driver / lowest (pull_request) Successful in 31s
driver / types (pull_request) Successful in 1m12s
driver / standard (pull_request) Successful in 17s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 18s
NAPALM requires an age on every entry of get_arp_table and
get_ipv6_neighbors_table. Many devices do not report one, and drivers
filled in 0.0, which means "confirmed just now". ARP_AGE_UNKNOWN (-1.0, as
NAPALM's IOS driver reports) is the value for "not reported"; the README
states the contract.

NetOrk/netork#1327
2026-10-30 08:37:08 +01:00
christianmanivong 145c2c668b Merge pull request 'chore: Python 3.12 is the floor (NAPALM/driver-standard#6)' (#56) from chore/python-312-floor into main
driver / lint (push) Successful in 13s
driver / build (push) Successful in 27s
driver / publish (push) Skipped
driver / test (3.13) (push) Successful in 57s
driver / security (push) Successful in 59s
driver / lowest (push) Successful in 30s
driver / types (push) Successful in 55s
driver / test (3.12) (push) Successful in 58s
driver / standard (push) Successful in 17s
CI / driver (push) Successful in 2m25s
2026-10-27 10:19:37 +00:00
Christian Manivong 1cef3385e8 fix: the floor on napalm is 5.1 (NAPALM/driver-standard#6)
driver / security (pull_request) Successful in 1m1s
CI / driver (pull_request) Successful in 2m49s
driver / lowest (pull_request) Successful in 28s
driver / build (pull_request) Successful in 29s
driver / test (3.12) (pull_request) Successful in 1m13s
driver / test (3.13) (pull_request) Successful in 55s
driver / standard (pull_request) Successful in 21s
driver / types (pull_request) Successful in 1m5s
driver / lint (pull_request) Successful in 14s
driver / publish (pull_request) Skipped
napalm 5.0 and older import pkg_resources, which a Python 3.12 venv no
longer has, so the lowest job failed on them.
2026-10-27 10:49:27 +01:00
Christian Manivong fbd3cf0129 chore: Python 3.12 is the floor (NAPALM/driver-standard#6)
driver / security (pull_request) Successful in 1m13s
driver / build (pull_request) Successful in 31s
CI / driver (pull_request) Failing after 2m50s
driver / lowest (pull_request) Failing after 30s
driver / test (3.13) (pull_request) Successful in 1m0s
driver / types (pull_request) Successful in 1m3s
driver / test (3.12) (pull_request) Successful in 1m2s
driver / standard (pull_request) Successful in 21s
driver / publish (pull_request) Skipped
driver / lint (pull_request) Successful in 14s
netOrk runs every driver in its own containers on Python 3.12, so 3.10
and 3.11 are no longer supported or tested. requires-python, mypy and
ruff move to 3.12, along with what ruff's pyupgrade rules then rewrite.
2026-10-27 10:18:55 +01:00
christianmanivong e400c4359c Merge pull request 'feat: shutdown_host and begin_stop_vm, shutting down without waiting' (#55) from feat/shutdown-requests into main
driver / build (push) Successful in 30s
driver / publish (push) Skipped
driver / lint (push) Successful in 14s
driver / test (3.13) (push) Successful in 1m7s
driver / security (push) Successful in 1m9s
driver / test (3.11) (push) Successful in 1m21s
driver / lowest (push) Successful in 34s
driver / types (push) Successful in 1m25s
driver / test (3.12) (push) Successful in 1m22s
driver / standard (push) Successful in 15s
driver / test (3.10) (push) Successful in 1m23s
CI / driver (push) Successful in 3m20s
2026-10-27 07:41:23 +00:00
christianmanivong 4d3574cd60 feat: shutdown_host and begin_stop_vm, shutting down without waiting
driver / security (pull_request) Successful in 1m8s
driver / test (3.10) (pull_request) Failing after 6s
driver / test (3.11) (pull_request) Successful in 1m13s
CI / driver (pull_request) Failing after 2m41s
driver / lowest (pull_request) Successful in 43s
driver / build (pull_request) Successful in 29s
driver / test (3.12) (pull_request) Successful in 1m8s
driver / test (3.13) (pull_request) Successful in 1m1s
driver / standard (pull_request) Successful in 20s
driver / types (pull_request) Successful in 1m4s
driver / lint (pull_request) Successful in 18s
driver / publish (pull_request) Skipped
HostShutdownMixin.shutdown_host(), on DeviceTypeDriver: a device shuts
itself down cleanly through its own means; declared, so hasattr says
whether a driver can. There is no forced variant.

VmStopRequestMixin.begin_stop_vm(name): ask for a VM's clean shutdown
and return at once, for a caller that waits on a timer and reads
get_vm_status instead of blocking in stop_vm. No role demands it.

For netOrk's graph workflows, which switch a hypervisor off when CI has
nothing for it (NetOrk/netork#1636).
2026-10-27 00:36:54 +01:00
christianmanivong c7ea15dbb4 docs: what the drivers need from the SNMP profile, opt-outs, and how getters behave
CI / test (3.11) (push) Successful in 24s
CI / test (3.13) (push) Successful in 25s
CI / test (3.10) (push) Successful in 26s
CI / test (3.10) (pull_request) Successful in 25s
CI / test (3.12) (pull_request) Successful in 26s
CI / test (3.13) (pull_request) Successful in 25s
CI / test (3.12) (push) Successful in 26s
CI / test (3.11) (pull_request) Successful in 24s
The SNMP profile has to express fallback OIDs, tables folded by first,
average or maximum, numbers inside text values, memory as used, free or
percent, and the uptime source. TX errors counting drops becomes
OPNsense's entry instead of every firewall's.

A driver opts out of an inherited template with None, so callers ask
callable(). USES_SSH, FULL_UPGRADE and platform stay as driver facts.
Getters do not write, a failed read raises, a write reports its real
result, and the application uses public driver methods only.
2026-10-14 10:56:17 +02:00
christianmanivong 01fe680456 docs: a structure that keeps device-types vendor- and product-agnostic
CI / test (3.11) (push) Successful in 26s
CI / test (3.13) (push) Successful in 25s
CI / test (3.10) (push) Successful in 28s
CI / test (3.10) (pull_request) Successful in 25s
CI / test (3.12) (pull_request) Successful in 25s
CI / test (3.13) (pull_request) Successful in 24s
CI / test (3.12) (push) Successful in 27s
CI / test (3.11) (pull_request) Successful in 24s
Roles and capabilities declare, templates run declared steps, reconcilers
stay free of I/O. POSIX host readers, the SSH exec transport and the
privilege layer move to a new package, napalm-unix. SNMP polling, ping
sweeps, discovery scoring, guest profiles, display labels and warnings
belong to the application; drivers declare SNMP OIDs as data and report
facts. Fixes become templates whose values come from the caller.

Says in which order this happens and where the open proposals #8-#14 and
#22 go instead.
2026-10-14 10:20:55 +02:00
90 changed files with 16876 additions and 345 deletions

No files matched your search

+439
View File
@@ -7,6 +7,445 @@ recorded here; their history is the git log.
## [Unreleased]
## [4.38.0] - 2026-11-13
### Added
- `ConfigRevisionsMixin`: a device's own configuration history (NetOrk/netork#2511).
`get_config_revisions()` (newest first, the first in place now, as
`ConfigRevisionDict`: id, UTC time, description, user, size) and
`get_config_revision(revision_id)` are declared. `revert_config_revision(revision_id)`
is a template over the driver's `_revert_to_config_revision`: it refuses an id that is
not in the history (`UnknownConfigRevision`), and after the revert checks that the
newest revision holds the revision's configuration, else `ConfigRevertFailed`; it
returns that revision. `config_revision_content(text)` leaves out what a device stamps
into every save, so an application can compare revisions with a configuration it has.
- `add_service_account(..., replaces=[...])`: netOrk's keys the new one takes the place of
go from its account once the new key is in it (step `old keys`), never the new key
itself; while the new key could not be added they stay, so netOrk keeps its way in.
netOrk calls it through a login with the new key, after trying that login, to move a
host from its installation-wide key to its site's (NetOrk/netork#2462).
- Each forwarding rule of `LogForwardingDict` names its `template`: the one the rule
gives (`@host;RSYSLOG_SyslogProtocol23Format`, `action(… template="…")`), else the
default set before it (`$ActionForwardDefaultTemplate`, or
`module(load="builtin:omfwd" template="…")` for omfwd actions), else None -- rsyslog's
own default, RFC 3164, which sends the host's local time without a time zone. A
reader can tell a rule whose timestamps a collector files shifted (NetOrk/netork#493).
### Fixed
- `apply_dhcp_reservationset` writes its changes so an address is released before it is
taken: an update that moves a reservation off its address goes before the change that
takes that address (along a chain, from its free end), and adds come after every
update. A server that keeps addresses unique at each write, such as Kea, no longer
refuses the second reservation as a duplicate when an address moves from one to
another. A swap -- reservations trading addresses, or a longer rotation -- has no such
order: its changes are still tried, and where the server refuses them the `[failed]`
line says it was a swap and with which reservations. Everything else is still written
and committed, and `DhcpApplyError` follows (#106).
- `apply_dhcp_subnetset` no longer stops at the first subnet the device refuses. As
`apply_dhcp_reservationset` has since 4.35.1, a refused subnet gets a `[failed]` line,
the others are still written, any non-empty diff is committed -- even one whose every
write was refused -- and `DhcpApplyError` follows. Before, the subnets written ahead of
the refused one stayed saved but inactive, and since the device reports them as
configured, no later apply committed them. `DhcpApplyError` takes an optional `kind`
(`"reservation"` by default, `"subnet"` here) that its message names (#113).
## [4.37.0] - 2026-11-12
### Added
- `UpdateMixin.get_installed_advisory_cves()` (declared): the CVEs a host's installed
security advisories close, per package. With `DNF_INSTALLED_CVES_COMMAND` (dnf 4,
read from the cache) and `parse_dnf_installed_cves()`, the parser every dnf driver
shares. netOrk reads it where its knowledge base holds no records of the host's
release, as the distribution's word that a backport is in (netOrk #2394).
## [4.36.2] - 2026-11-12
### Fixed
- `kernel_reboot_pending` compares kernel releases the way rpm does (`rpmvercmp`). A
number is newer than letters in the same place, so an Oracle Linux host running
`6.12.0-208.112.5.1.el9uek` no longer asks to reboot into the older
`6.12.0-208.112.5.el9uek`. A `~` (release candidate) comes before its release (#109).
## [4.36.1] - 2026-11-12
### Fixed
- The host status no longer reports an Oracle Linux host's UEK kernels as an interrupted
dnf transaction. The installonly packages are now also the ones the host's libdnf was
built with, read from its Python binding: Oracle adds `kernel-uek` and its parts there,
and their packages provide nothing of upstream's defaults (#107).
## [4.36.0] - 2026-11-11
### Added
- `TimeSyncDict.dhcp_ntp_servers` (not required): the NTP servers a device got by DHCP
(option 42) and uses, in its order; `[]` for none, absent or None where the driver cannot
tell. `ntp_servers` stays the servers configured on the device and no longer holds
DHCP's. A device that gets the site's time giver by DHCP is aligned, so an application
has to tell the two apart (NetOrk/netork#2380).
- `PosixTimeSyncMixin` reads them: chrony's sources from the DHCP client's `sourcedir`
(`CHRONY_DHCP_DIRS`, `/run/chrony-dhcp` by default), systemd-timesyncd's
`LinkNTPServers`. Where ntpd or OpenNTPD runs they are unknown (None).
`timesyncd_link_servers` is public.
### Changed
- `configure_time_sync` leaves DHCP-provided NTP servers alone. chrony: only `server`,
`pool` and `peer` lines are commented out; no `sourcedir` line is any more, and the DHCP
client's files are left alone. systemd-timesyncd: a link's servers stay and no longer
fail the `verify` step.
- A write gives back a DHCP client's `sourcedir` that 4.35.0 commented out
(`#napalm-time-sync: sourcedir /run/chrony-dhcp`), so a host aligned by 4.35.0 hears DHCP
again; chronyd restarts for it. Another `sourcedir` 4.35.0 took out stays out.
- `timesyncd_servers` returns the configured servers (runtime, then system) without a
link's; the fallback only while there is no server at all.
## [4.35.1] - 2026-11-11
### Fixed
- `apply_dhcp_reservationset()` no longer stops at the first reservation the device refuses.
One refused write (on OPNsense, an address in a subnet Kea does not manage) ended the run
before the commit: every later reservation went unwritten, and the earlier ones stayed
saved but inactive, where the next diff found them configured and never committed them. A
refused change now gets a `[failed] <mac> -> <ip>: <reason>` line, the others are written,
any non-empty diff commits, and `DhcpApplyError` (a `RuntimeError`, exported) is raised
after the commit with how many of how many were refused (NetOrk/netork#2381).
## [4.35.0] - 2026-11-11
### Added
- `TimeSyncMixin`: a device's time sources and time zone. `get_time_sync()` (declared only)
returns `TimeSyncDict` -- the configured NTP servers in the device's order, the zone as an
IANA name (None where the device keeps no name), whether the clock is synchronised now,
and the client (`chrony`, `systemd-timesyncd`, `ntpd`, `openntpd`; None for the device's
own). `configure_time_sync(ntp_servers=None, timezone=None)` is a fix template: None
leaves a value alone, the device ends up with exactly the servers given, in order, and a
`verify` step reads it back. An unknown zone, a malformed server, an empty list or a value
outside the driver's `time_sync_writes` is refused before anything is written.
`TIME_SYNC_VALUES` and `check_ntp_servers` are public (NetOrk/netork#2361).
- `PosixTimeSyncMixin` implements it over `run_command`: read in one round trip as the login
user (`timedatectl`, chrony's configuration and `chronyc -n sources`, `timedatectl
show-timesync`, ntpd's and OpenNTPD's configuration); the zone by `timedatectl
set-timezone`; chrony's servers in a file of their own (`sources.d`, else a `confdir` or
an included file) with every other source commented out, then `chronyc reload sources` or
a restart; systemd-timesyncd's in a drop-in that resets `NTP=` and empties `FallbackNTP=`.
ntpd and OpenNTPD are read, not written. `parse_time_sync` is public.
- `napalm_device_types.posix_tz` (from netOrk): `posix_tz`, `is_iana_zone`, `is_posix_tz` and
`UnknownTimezone`, and `parse_posix_tz`, which takes a POSIX rule apart into the UTC
offset and the summer-time transitions a switch wants (`PosixTzDict`,
`DstTransitionDict`).
### Changed
- Needs `tzdata` (2024.1 or later): the zones' rules come from its TZif files.
## [4.34.0] - 2026-11-10
### Added
- `RunningBackupsMixin.get_running_backups()`: the backups running on a device right now --
a hypervisor's backup of its guests (with the guests it is taking now), a backup server's
backup, sync, verify, garbage collection, prune or tape run, a host's own file-level backup.
Declared only; a driver mixes it in when it can tell, and a driver without it cannot tell.
`RunningBackupDict`, `RunningBackupKind` and `RUNNING_BACKUP_KINDS` are public. An
application asks right before it restarts a device (NetOrk/netork#2318).
## [4.33.0] - 2026-11-07
### Added
- `get_host_status()` tells an interrupted package manager on hosts without dpkg too, and
says which one it read in the new `package_manager` field (`dpkg`, `dnf`, `yum`, `apk`,
`opkg`, or None). dnf and yum: a package installed in two versions (not the kernels and
other packages dnf keeps side by side) as state `duplicate`, or a transaction that never
ended (dnf 4's history, yum's `transaction-all` journal). apk: a package apk marked with
broken files or a broken script (`broken-files`, `broken-script`), the ones `apk fix` would
take. opkg: a package left `unpacked`, `half-configured` or `half-installed`, or flagged for
reinstallation. All read without root; while one of them runs the answer is unknown
(NetOrk/netork#851).
## [4.32.0] - 2026-11-07
### Added
- `SshServerAuthMixin.get_ssh_server_auth()`: how an OpenSSH server lets people log in --
password, keyboard-interactive, public key, host-based and GSSAPI authentication,
`AuthenticationMethods` as lists of methods, `PermitRootLogin` and `UsePAM`, from `sshd -T`
as root, and sshd's PAM `auth` stack (each module with its control and arguments, includes
followed), from `/etc/pam.d` as the login user. `OpenSshServerMixin` carries it, so every
driver that can harden sshd has it. `parse_sshd_auth`, `parse_pam_auth`, `parse_pam_files`,
`pam_auth_stack` and `SSHD_EFFECTIVE_COMMAND` are public (NetOrk/netork#662).
## [4.31.1] - 2026-11-06
### Fixed
- `get_log_sources()` names as a file's writer only a process that holds it open for writing.
A log reader -- CrowdSec, Promtail, rsyslog's `imfile`, a `tail -F` -- was given the file
it reads as if it wrote it. A file only readers hold is now listed without a producer
(`unit` and `process` None); one only a reader inside a container holds is left out, as its
path is that container's view of a mount. The access mode comes from
`/proc/<pid>/fdinfo/<fd>`, read in the same command (NetOrk/netork#2092).
## [4.31.0] - 2026-11-06
### Added
- `get_log_sources()` also says where the logs go beyond the host: `forwarding` holds whether
rsyslog runs, its forwarding rules (legacy `@`/`@@`/`:omrelp:` and RainerScript
`omfwd`/`omrelp`, with selector, ruleset and whether they take every message), its `imfile`
inputs, and whether the journal reaches it (`imjournal`, or `imuxsock` with journald's
`ForwardToSyslog`). Read in the same command; `parse_log_forwarding` and `parse_rsyslog` are
public (NetOrk/netork#2053).
## [4.30.0] - 2026-11-04
### Added
- `LogSourcesMixin.get_log_sources()`: every log a Linux host writes. Sources are the units
journald holds entries for, the log files running processes hold open (with size, last write
and the service or container writing them), and the engine's file of each container's output.
A file a container writes is named by the container's own path. `LOG_SOURCES_COMMAND` and
`parse_log_sources` are public, and a host whose `find` cannot list open files (busybox)
raises `LogSourcesUnavailable`. A driver supplies `_run_log_sources_command(command,
privileged=)`, as for listening sockets (NetOrk/netork#2049).
## [4.29.0] - 2026-11-04
### Added
- `remove_local_account_keys(name, keys, *, clear_password=False)`: with `clear_password`
the account's password goes along with the keys, and so do its sudo rule and admin
groups, as sudo asks for that password. For a person who deleted their key and whose
password went with it. The account stays open; the steps are `keys`, `password` and
`sudo`, each run when another failed (NetOrk/netork#2033).
## [4.28.0] - 2026-11-04
### Added
- `LocalAccountsMixin.add_local_account(..., replaces=[...])`: the keys the new one takes
the place of go from the account, matched by their base64 whatever a line's options or
comment. Keys the person added on the host stay. The step is `old keys`; it runs even
when the new key could not be added (NetOrk/netork#2013).
- `add_local_account(..., clear_password=True)`: the account's password goes, and it logs
in with a key only. Step `password`. It cannot be combined with a `password_hash` or
with `admin=True`, as the sudo rule asks for that password. A driver supplies
`_account_clear_password`; one that does not fails the step and says so.
- `LocalAccountsMixin.remove_local_account_keys(name, keys)`: takes the named keys off an
account and leaves it open, its password and sudo as they are -- for a person who
deleted a key and has no new one. Step `keys`.
- A driver whose host keeps keys elsewhere overrides `_account_drop_keys` next to
`_account_add_key` and `_account_remove_keys`.
## [4.27.0] - 2026-11-03
### Added
- `ScheduledJobsMixin`: what a host runs on its own schedule as one shape,
`ScheduledJobDict` — cron lines, run-parts scripts, anacron, systemd timers, `periodic`,
a platform's own jobs, Windows tasks — with who owns each definition (`managed_by`) and
what the driver can do to it (`actions`). `get_scheduled_jobs` reads them with the host's
time zone; `manage_scheduled_job` enables, disables or runs one now; `put_scheduled_job`
and `delete_scheduled_job` create, change and delete a job an application writes. Every
write names the `digest` the caller saw (`job_digest`) and raises `ScheduledJobChanged`
when the host's definition differs; `ScheduledJobRefused` for a job that cannot take a
verb (NetOrk/netork#1933, #86).
- `PosixScheduledJobsMixin`: the implementation over the command channel. One framed report
as root reads `/etc/crontab`, `/etc/cron.d`, the user crontabs, the run-parts directories,
`/etc/anacrontab` and every systemd timer with the unit it starts, and which package owns
each definition. A cron line is switched off behind `#napalm-disabled: `; an
application's job is a timer where systemd runs, else a file in `/etc/cron.d` or marked
lines in the user's crontab.
- `napalm_device_types.cron`: cron expressions as Vixie cron reads them, their next runs in
a time zone, and as systemd `OnCalendar` lines that fire on exactly the same minutes.
### Removed
- `OSDriver.get_cron_jobs` and `CronJobDict`. No application called them; the readers in
napalm-linux and napalm-bsd go with them. `get_scheduled_jobs` replaces them.
## [4.26.0] - 2026-11-03
### Added
- `OpenSshServerMixin.SSHD_CONFIG_GENERATED`: a driver whose platform writes sshd_config
from its own database, but includes `/etc/ssh/sshd_config.d` (OpenMediaVault), sets it
and keeps `harden_ssh_server`. The hardening then goes only into the drop-in; a
sshd_config without that include is left alone and the result says why, since the
platform would write over a change to it on its next apply (#88, NetOrk/netork#1950).
## [4.25.0] - 2026-11-02
### Added
- `BackupServerDriver` (`ROLE = "backup_server"`), a role base for backup servers such as
Proxmox Backup Server (#82). It declares `get_backup_datastores`, `get_backup_groups`
(each group with its newest snapshot), `get_backup_snapshots`, `get_backup_jobs`
(garbage collection, verify, prune, sync and tape in one list) and `get_backup_remotes`,
and the actions `start_garbage_collection`, `run_backup_job`, `verify_backup_snapshot`,
`set_backup_snapshot_protected` and `delete_backup_snapshot`, which return a task to
follow with `get_backup_task`. The shapes live in `napalm_device_types.backups`. A
server that manages its own disks lists `StorageDriver` as its second role.
- `ClusterStorageDict.backup_server` (optional): which backup server, datastore and
namespace a hypervisor's backup storage writes to, with the certificate fingerprint it
pins, so an application can tie a server's backup groups to a cluster's guests.
## [4.24.0] - 2026-11-02
### Added
- `parse_lldpctl_keyvalue(output)` and `LLDPCTL_KEYVALUE_COMMAND`: lldpd's neighbours
(`lldpctl -f keyvalue`) as one `LldpNeighborRow` per neighbour, for every driver of a
host that runs lldpd. Several neighbours on one port (a phone and the PC behind it, an
unmanaged switch) stay apart, an interface name with dots (`eth0.1`) stays whole, the
chassis and port ids are read whatever their type (MAC, address, interface name, local
id), with management addresses and capabilities. A failed lldpctl (lldpd not running,
lldpctl missing) raises instead of reading as no neighbours.
- `lldp_neighbors(rows)` and `lldp_neighbors_detail(rows, interface="")`: NAPALM's
`get_lldp_neighbors` and `get_lldp_neighbors_detail` from those rows, so a driver no
longer builds the two shapes itself. A neighbour that names its chassis by a MAC
address carries it in `get_lldp_neighbors` as `mac`, in lower case
(`LldpNeighborDict`), so an application can match it by MAC.
### Fixed
- `DNF_SECURITY_COMMAND` reads the cached metadata only (`dnf --cacheonly`), as
`APT_UPGRADABLE_COMMAND` reads apt's local index (#76). With an expired cache dnf
downloaded the repositories' metadata first, and on an Oracle Linux host on OCI that
outlasted netOrk's whole poll budget. Refreshing the index stays a step of its own
(`dnf makecache`). Without a cache dnf fails at once, which already reads as
"security unknown".
## [4.23.0] - 2026-11-02
### Changed
- Python 3.12 is the floor (`requires-python = ">=3.12"`): netOrk runs every driver
on 3.12, so 3.10 and 3.11 are no longer tested (NAPALM/driver-standard#6).
- The floor on napalm is 5.1: older releases import `pkg_resources`, which a Python
3.12 venv no longer has.
- `requests` is a declared dependency (floor 2.25.0), for `tls_certificates`; it came
with napalm before.
- The SNMP health profile shapes (`SnmpHealthProfile`, `SnmpValueSpec`,
`SnmpMemorySpec`, `SnmpInterfacesSpec`) are defined in
`napalm_device_types.snmp_health_profile`. The package and `models` still export
them, so an existing import keeps working.
### Added
- `HypervisorDriver.get_vm_migration_preflight(name)`, `migrate_vm(name, target_node,
online=True, target_storage=None, timeout=3600)` and `set_node_maintenance(node,
enabled)`: which nodes of its cluster a guest could move to and what blocks it
(`MigrationPreflightDict`: `allowed_nodes`, `refused_nodes` with the platform's
reason, `local_disks` the migration copies, `local_resources` that block it), the
move itself, blocking until the platform's task ended and raising with the
platform's own cause, and a node's HA maintenance mode. Declared only; netOrk
probes each with `hasattr`.
- `get_clusters()` may report HA, backups and quorum votes: `ClusterDict.ha_resources`
(`HAResourceDict`: requested and current state, node, PVE 8 group, PVE 9 rules),
`backup_jobs` (`BackupJobDict`: schedule, storage, mode, the guests a job covers)
and `backups` (`BackupDict`: per guest and storage the newest backup, its size
and how many there are), and `ClusterNodeDict.votes`. All `NotRequired`, so
drivers written before stay valid; None means could not be read, `[]` read and
empty.
- `napalm_device_types.tls_certificates`: TLS certificates by trust on first use, the
counterpart of `ssh_host_keys` for a driver's HTTPS API. A driver takes the
fingerprint to expect from `optional_args["tls_fingerprints"]` (by port, the SHA-256
of the DER certificate, `SHA256:AB:CD:…`) and refuses any other certificate with
`TlsCertificateMismatch` (a `ConnectionException` naming both fingerprints) before a
password is sent; no CA is consulted, so a self-signed certificate verifies. Without
a fingerprint the first certificate is accepted. Every certificate seen goes to
`optional_args["tls_certificate_observer"]`. `CertificateCheck.verify_server()`
checks the certificate in a handshake of its own, `pin(session)` makes every later
connection of a `requests` session present the same one, `refusing()` turns a refusal
there into the mismatch, `observe()` reports it once the login stands
(NAPALM/napalm-hpe-aruba-procurve#17).
- `LldpAgentMixin.configure_lldp_agent(interfaces)` (`napalm_device_types.fixes`): run
an LLDP agent on the interfaces the caller names. The template installs the agent
(`ensure_package(LLDP_AGENT_PACKAGE)`), has the driver write the interface list
(`_write_lldp_agent_config`), then enables and restarts the service
(`manage_service(LLDP_AGENT_SERVICE, …)`); both names default to `lldpd`. No
interface is a ValueError before anything changes. It is the action for what LLDP
getters used to do on their own (NAPALM/napalm-openwrt#8).
- `SnmpValueSpec.unknown`: a driver whose device's agent answers a value wrongly says
so in its `SNMP_HEALTH_PROFILE` (`"cpu": {"unknown": True}`), and the application
reports no value for it -- not the standard reading, not 0. OpenBSD's net-snmp,
which reports the CPU as never idle, is the case (NAPALM/napalm-bsd#7).
- `VMProvisionIncomplete` (`napalm_device_types.provisioning`): what
`create_vm_from_cloud_init` raises when it fails after the VM was created. A
`RuntimeError` with the failure as its cause and the half-built VM's `vmid` and
`node`, so the caller can remove it with `destroy_vm` (NAPALM/napalm-proxmox#28).
- `HypervisorDriver.get_clusters()` and its shapes `ClusterDict`, `ClusterNodeDict`,
`ClusterStorageDict`, `ClusterNetworkDict`: every cluster the connection can see,
with its nodes (offline ones included, `local` on the node the connection is),
their CPU and memory, and the storages and networks with the nodes that see each.
A standalone host is a cluster of one with no name; a manager reports each cluster
and standalone host it runs. Declared only (NetOrk/netork#484, #1821).
- `create_vm_from_cloud_init(node=…)`: the cluster node to create the VM on. None
leaves it to the driver; a driver that can only create on its own node raises
ValueError for another (NetOrk/netork#1822).
- `LocalAccountsMixin.add_service_account(name, public_key)`: netOrk's own account,
created through the account the driver logs in with today. It takes a key and no
password, and a rule of its own in `85-service-<name>` lets it run everything as
root without a password (`NOPASSWD: ALL`, checked with `visudo` and `sudo -l`). An
account that is there is opened again and keeps its other keys; root is refused
(NetOrk/netork#1817). With `uid=`, a new account gets that uid once no other account
has it; one that is there keeps its own. A driver supplies
`_account_create_with_uid`. An account that is there and cannot log in (`nologin`,
`false`) is another service's, such as a netOrk satellite's `netork`, and is refused
before anything changes.
- `LocalAccountsMixin` (`napalm_device_types.local_accounts`): personal login accounts
on a host. `get_local_accounts(names)` reads them back (groups, lock, sudo, key
files), `add_local_account(name, public_key, password_hash=, admin=)` creates or
reopens one, `lock_local_account(name)` takes its keys and sudo away and refuses
every login while keeping its home. Every command root runs is one fixed command,
the key files are read and written as the account itself, and the sudo rule is a
drop-in of its own, checked with `visudo` and `sudo -l`. A driver supplies the
commands that create, open and lock an account and take it out of a group, and may
replace where the keys are kept (`_account_key_files`, `_account_read_keys`,
`_account_add_key`, `_account_remove_keys`; default `~/.ssh/authorized_keys`)
(NetOrk/netork#1793).
- `OpenSshServerMixin.exempt_from_ssh_penalties`: puts the address the host sees
the managing client at (`$SSH_CONNECTION`) into `PerSourcePenaltyExemptList`,
in `/etc/ssh/sshd_config.d/00-netork-penalty-exempt.conf`, next to the host's
own exemptions. Only where sshd has per-source penalties (OpenSSH 9.8+; an older
one refuses the directive) and sshd_config includes the drop-in directory;
checked with `sshd -t` and `sshd -T`, then a new session, and taken back on any
failure. Helpers: `parse_penalty_exemptions`, `client_address`, `is_exempt`
(NetOrk/netork#1800).
- `retry_refused` and `refused_before_banner` (`napalm_device_types.ssh_host_keys`):
a connection sshd closes before its banner -- what OpenSSH 9.8+ does to a source
it has just penalised -- is tried again after 1, 3, 7 and 15 s.
`HostKeyCheck.connect` does so itself (NetOrk/netork#1800).
- `ARP_AGE_UNKNOWN` (`napalm_device_types.arp`, `-1.0`): the `age` a driver reports
in `get_arp_table` and `get_ipv6_neighbors_table` when the device does not say how
old an entry is. `0.0` means "confirmed just now" and is no stand-in for it
(NetOrk/netork#1327).
- `NetworkTargetDict.uplinks` (optional): the physical NICs a bridge leaves the
host through, one group per way out -- a bond's slaves together, a single NIC
alone (`[["eno1", "eno2"]]`). A missing key means unknown, an empty list a
bridge without uplink. NetOrk follows their LLDP neighbours to the switch port
and its untagged VLAN; one slave's neighbour speaks for its bond
(NetOrk/netork#1325).
- `HostShutdownMixin.shutdown_host()`, on `DeviceTypeDriver`: shut the device itself
down cleanly through its own means; declared, so `hasattr` says whether a driver
can. No forced variant (NetOrk/netork#1636).
- `VmStopRequestMixin.begin_stop_vm(name)`: ask for a VM's clean shutdown and return
at once, for callers that wait on a timer instead of in `stop_vm`. No role demands
it (NetOrk/netork#1636).
- `SudoHistoryMixin` (`napalm_device_types.sudo_history`): the sudo password an earlier
driver typed into an interactive shell (`echo '<password>' | sudo -S -p "" …`) and
left in the login user's shell history, for a driver to mix in (NetOrk/netork#1090).
The driver says what its lines carry (`SUDO_HISTORY_MARKER`) and which files its
partial sudo left to the login user (`SUDO_ROOT_FILES`), and supplies `run_command`.
- `get_history_sudo_lines()` counts the lines per history file (`~/.bash_history`,
`~/.zsh_history`, `~/.ash_history`, `~/.sh_history`) with `grep -c` as the login
user. No line is read back; a file it cannot read raises.
- `remove_history_sudo_lines()` deletes exactly those lines as the login user (`sed
-i`, its backup overwritten with `shred -u`), and as root gives each of
`SUDO_ROOT_FILES` back to `root:root` where it is a regular file, not a symbolic
link, owned by the login user. It reports every file's lines before and after,
how many were removed and which files are root's again. Nothing goes to stdin.
### Fixed
- `AptProxyMixin` knows that apt's https sources take the http proxy too: apt's https
method asks `Acquire::https::Proxy` and then `Acquire::http::Proxy`, for the host and
then for all. An `00proxy` with the http line alone sent download.docker.com and
every other https source to apt-cacher-ng, which answers `403 CONNECT denied` without
a `PassThroughPattern`, and the whole upgrade failed (NetOrk/netork#1811).
- The report carries `https_proxy`, `https_per_host` and `https_settings`;
`https_proxy_for(report, host)` is the proxy apt takes for one https host.
- `check_https_source()` fetches one https source's `InRelease` the way apt does;
None where no https source goes through a proxy.
- `configure_package_proxy` gains an `https` step: one https source through the new
proxy. Where the proxy refuses it, `00proxy` also says `Acquire::https::Proxy
"DIRECT";`, later https lines to the same proxy are commented out, and the source
is fetched direct. An HTTP error from the mirror counts as let through.
- `apt_fetch_command` takes https URLs; `direct` sets the host's https setting too.
- `comment_out_command(..., scheme="https")`.
## [4.22.0] - 2026-10-23
### Added
+184 -14
View File
@@ -47,6 +47,18 @@ UUID and NIC MACs -- what ties the BMC to the host it manages. Its sensors come
NAPALM's `get_environment`, with `cpu` and `memory` empty, since a BMC does not know the
host OS's load; it has no `HealthMetricsMixin` (SNMP) and no command channel.
A backup server (Proxmox Backup Server) is a role of its own too, `BackupServerDriver`
(`backup_server`). Its datastores, the backup groups in them (each with its newest
snapshot), its jobs (garbage collection, verify, prune, sync, tape) and its remotes have
no counterpart in `StorageDriver`, so they are not bent into pools, volumes and
replication jobs. A server that also manages its own disks and pools lists
`StorageDriver` second: `class ProxmoxBackupDriver(BackupServerDriver, StorageDriver, ...)`.
Its actions (garbage collection, running a job, verifying, protecting or deleting a
snapshot) start a task and return its id at once; `get_backup_task` follows it. On the
hypervisor side, a cluster storage that writes to a backup server says which one
(`ClusterStorageDict.backup_server`: server, port, datastore, namespace, fingerprint), so
an application can tie the server's groups to the cluster's guests.
### Role bases declare; they never implement
**A role base must not contain a single runtime method — not even a
@@ -85,20 +97,35 @@ base; a mixin listed after it loses the MRO to NAPALM's stub. For the same reaso
`hasattr` says nothing about NAPALM's getters, and the OS role's account listing is
`get_os_users`, not `get_users` -- NAPALM's `get_users` has a different shape.
### NAPALM's getters: an unknown ARP age is `-1.0`
NAPALM requires an `age` on every entry of `get_arp_table` and
`get_ipv6_neighbors_table`: seconds since the device last confirmed that neighbour.
Many devices do not report one (`ip neigh`, `/proc/net/arp`, a router's host list).
A driver that cannot read the age reports `ARP_AGE_UNKNOWN` (`-1.0`, as NAPALM's IOS
driver does), never `0.0`: `0.0` means "confirmed just now", and an application that
ranks sightings by freshness takes it at its word (NetOrk/netork#1327). An expiry
countdown (FreeBSD's `expires in 1180 seconds`) is not an age either. A driver whose
device does report the age keeps reporting it.
## Function classes: what a device *can do*
Behaviour shared across roles lives in a function class, exactly once, and a role base
is a thin bundle over them — `PackageManagementMixin`, `HealthMetricsMixin`,
`ServiceControlMixin`, `UpdateMixin`, `NatVpnMixin`, `MacAclMixin`, `FirewallRuleMixin`,
`DhcpServerMixin`, `PingSweepMixin`, `ConfigLifecycleMixin`, `InterfaceFilterMixin`.
`HostRebootMixin` (`reboot_host`) is mixed into `DeviceTypeDriver` itself, since any
device may be restartable; like the others it only declares.
`HostRebootMixin` (`reboot_host`) and `HostShutdownMixin` (`shutdown_host`) are mixed
into `DeviceTypeDriver` itself, since any device may be restartable or able to shut itself
down; like the others they only declare. `VmStopRequestMixin` (`begin_stop_vm`) asks for a
VM's clean shutdown without waiting for it — a hypervisor driver mixes it in when it can.
**Fixes are templates** (`napalm_device_types.fixes`, since 4.13). A change the
application wants made on a device runs as fixed steps the driver implements, with values
the caller supplies: `SnmpAgentMixin.configure_snmp_agent(settings)` (install, configure,
firewall, start), `EnsurePackageMixin.ensure_package(name)`,
`PackageProxyMixin.configure_package_proxy(url)` and
firewall, start), `LldpAgentMixin.configure_lldp_agent(interfaces)` (install, write the
interfaces, enable and restart; the driver writes only the interfaces),
`EnsurePackageMixin.ensure_package(name)`, `PackageProxyMixin.configure_package_proxy(url)`,
`TimeSyncMixin.configure_time_sync(ntp_servers, timezone)` and
`ContainerEngineMixin.grant_container_engine_access(user)`. A driver whose device takes the
change in one call overrides the template method. A driver mixes in the fixes its device
can take; none is on a role base. They replace `run_device_action`, which is deprecated.
@@ -119,6 +146,9 @@ class MySwitchDriver(SwitchDriver):
A value is `oids` (the first that answers wins) or a `table` folded by `first`, `average`
or `max`, with an optional `pattern` for an agent that answers with text. What the profile
does not name, the application reads from the standard UCD-MIB and IF-MIB.
A value the agent gets wrong is `{"unknown": True}`: the application then reports no value
for it, rather than the standard reading or a default (OpenBSD's net-snmp reports its CPU
as never idle).
`KernelFactsMixin` (`get_kernel_facts`) is the exception that is mixed in by a driver
rather than by a role base: what a Linux kernel has built and loaded is read the same way
@@ -145,9 +175,12 @@ A driver supplies `_run_apt_command`, `_run_apt_root_command` and `_write_apt_fi
`HostStatusMixin` (`get_host_status`) is mixed in the same way: whether a Linux host needs
a reboot to finish an update (`/var/run/reboot-required`, `needs-restarting -r`, or a newer
kernel of the running flavour installed), whether it patches itself (unattended-upgrades,
dnf-automatic) and when that last succeeded (apt's `upgrade-stamp`), and whether dpkg was
left halfway (a journal in `/var/lib/dpkg/updates` or a half-configured package; unknown
while a package manager is running), and how full `/`, `/boot` and `/var` are (`df -Pk`,
dnf-automatic) and when that last succeeded (apt's `upgrade-stamp`), and whether its package
manager was left halfway (unknown while one is running): dpkg (a journal in
`/var/lib/dpkg/updates` or a half-configured package), dnf or yum (two versions of a package
in the rpm database, a transaction that never ended), apk (a package marked with broken files
or a broken script) or opkg (a package left `unpacked`), with `package_manager` naming which,
and how full `/`, `/boot` and `/var` are (`df -Pk`,
each filesystem once, in bytes; how full is too full is the caller's decision). `package_updates` holds the shared apt and dnf parsers: apt's suites become
an update's `origin`, a `-security` suite makes it a security update, and dnf's security
advisories do the same.
@@ -162,6 +195,33 @@ without `ss` is read with `netstat -lntup` (OpenWrt's busybox, old net-tools); o
cgroup names the procd service (`/services/<name>/<instance>`). A host with neither raises
`ListeningSocketsUnavailable`.
`LogSourcesMixin` (`get_log_sources`) is mixed in the same way: every log the host writes, in
one round trip. The units journald holds entries for (`journalctl -F _SYSTEMD_UNIT`, services
only), the log files running processes hold open (`/proc/<pid>/fd`: `*.log`, `*.log.N` and
anything under `/var/log`) with size and last write from `stat -L` on the descriptor, and the
engine's own file of a container's output (`/var/lib/docker/containers/<id>/…`) as that
container's stream. Each file names the service or container of its first writer from
`/proc/<pid>/cgroup`; a file a container writes carries the container's own path. journald's
store, deleted files and a database's write-ahead log (LevelDB's `000004.log`) are left out; a
host whose `find` cannot list open files (busybox) raises `LogSourcesUnavailable`. A
driver supplies `_run_log_sources_command(command, privileged=)`; without root only the login
user's processes show their files, and the reading says so (`attributed: false`).
The same round trip reads where the logs go beyond the host (`forwarding`,
`napalm_device_types.log_forwarding`): whether rsyslog runs, its forwarding rules (`@`, `@@`,
`:omrelp:`, `action(type="omfwd"|"omrelp")`, each *complete* only for `*.*` with no condition
around it), the files it reads in (`imfile`), and whether the journal reaches it (`imjournal`,
or `imuxsock` with journald's `ForwardToSyslog`, the distribution's commented default counting).
LLDP neighbours are rows: `LldpNeighborRow` is one neighbour as its local port heard it
(chassis and port id with their type, name, description, management addresses,
capabilities in NAPALM's names), and `lldp_neighbors(rows)` / `lldp_neighbors_detail(rows,
interface)` are NAPALM's two getters projected from them, a port with two neighbours
keeping both; a neighbour named by its chassis MAC carries it as `mac` (lower case). A driver for a host that runs lldpd sends `LLDPCTL_KEYVALUE_COMMAND` and
hands its output to `parse_lldpctl_keyvalue`: one row per neighbour block, an interface
name with dots (`eth0.1`) kept whole, ids read under the type lldpd prints them with
(`chassis.mac`, `port.local`, ...). When lldpctl fails (lldpd not running, no `lldpctl`)
the parse raises instead of reading as no neighbours.
`HostFirewallMixin` (`get_host_firewall`) reads a Linux host's own netfilter rules the same
way: `nft -j list ruleset`, with the tables iptables-nft manages taken from `iptables-save`
instead, because nft shows their extensions (`-m conntrack`, `-m set`) only as an opaque `xt`.
@@ -239,6 +299,26 @@ A driver mixes it in and supplies `run_command`; a platform that regenerates ssh
itself (an appliance with its own database) must not, or opts out with
`harden_ssh_server = None`.
`SshServerAuthMixin` (`get_ssh_server_auth`) says how an OpenSSH server lets people log in:
password, keyboard-interactive, public key, host-based and GSSAPI authentication on or off,
`AuthenticationMethods` as lists of methods (empty for `any`), `PermitRootLogin` and
`UsePAM` (None where sshd has no PAM), from `sshd -T` as root -- the same fixed command the
hardening reads. With PAM on, it also names the modules of sshd's PAM `auth` stack
with each control and its arguments, read from `/etc/pam.d` in one listing as the
login user and followed through `@include`, `include` and `substack`. Which of them is a
second factor is the consumer's call. `Match` blocks are not read. `OpenSshServerMixin`
carries it; a driver can also mix it in alone and supply `run_command`.
`SudoHistoryMixin` (`get_history_sudo_lines`, `remove_history_sudo_lines`) deals with what
a driver left behind when it typed `echo '<password>' | sudo -S -p "" …` into an interactive
shell: that line, password and all, in the login user's shell history, and files its partial
sudo wrote as the login user. The driver names the marker its lines carry
(`SUDO_HISTORY_MARKER`) and those files (`SUDO_ROOT_FILES`). The getter counts the lines per
history file with `grep -c`, as the login user; the fix deletes them with `sed -i` and
overwrites the backup with `shred -u`, then gives the files back to root (regular files the
login user owns, never a symbolic link). The password is never read, sent or written. A
driver mixes it in and supplies `run_command`.
**Update readers raise when they cannot read.** `get_available_updates` returns an empty
list only when nothing is pending; netOrk keeps "pending since" per package, and an empty
list for "don't know" would reset it.
@@ -252,6 +332,65 @@ trip (`list-unit-files` plus one `systemctl show` over every loaded unit) instea
`is-enabled` and a `show` per unit. A host without systemd raises `SystemdUnavailable`,
a `NotImplementedError`, so a driver can fall back to another init system.
`ScheduledJobsMixin` (`get_scheduled_jobs`, `manage_scheduled_job`, `put_scheduled_job`,
`delete_scheduled_job`) is what a host runs on its own schedule — cron lines, run-parts
scripts, anacron, systemd timers, `periodic`, a platform's own jobs, Windows tasks — as one
shape, `ScheduledJobDict`. Every job says who owns its definition (`managed_by`: package,
admin, platform, application) and what the driver can do to it (`actions`): every job a
driver can switch takes `enable`, `disable` and `run`; only a job an application wrote
through `put_scheduled_job` takes `edit` and `delete`. Every write names the `digest` the
caller saw and raises `ScheduledJobChanged` when the host's definition is no longer that.
`PosixScheduledJobsMixin` implements it over `run_command`: one framed report as root,
a cron line switched off behind `#napalm-disabled: `, a timer by `systemctl`, an
application's job as a timer where systemd runs, else in `/etc/cron.d` or the user's
crontab. A driver sets the paths its host keeps jobs in. `napalm_device_types.cron` parses
cron expressions, gives their next runs and writes them as `OnCalendar` lines.
`TimeSyncMixin` (`get_time_sync`, `configure_time_sync`) is how a device keeps its clock:
the NTP servers it is configured with, in its order, its time zone as an IANA name, whether
it is synchronised now and which client does it (`TimeSyncDict`). The servers it got by
DHCP (option 42) and uses are apart, in `dhcp_ntp_servers` (`[]` for none, left out where
the driver cannot tell): DHCP counts, a device that gets the site's time giver that way is
aligned. `get_time_sync` is declared only; `configure_time_sync(ntp_servers=None,
timezone=None)` is a fix template -- `None` leaves a value alone, the device ends up
configured with exactly the servers given, in that order, its DHCP-provided servers left
alone, and a `verify` step reads it back. A zone that is not in the IANA database, a
malformed server, an empty list or a value outside the driver's `time_sync_writes` is
refused before anything is written. A driver states in `time_sync_writes` which of
`ntp_servers` and `timezone` it can write (a BMC that runs on UTC only: `ntp_servers`).
`napalm_device_types.posix_tz` turns an IANA name into the POSIX rule libc wants in `TZ`
(OpenWrt) and that rule into a UTC offset and summer-time transitions (a switch), from the
`tzdata` package. `PosixTimeSyncMixin` implements it over `run_command`: one reading as the
login user (`timedatectl show`, the active client: chrony's configuration and `chronyc -n
sources`, `timedatectl show-timesync`, `ntp.conf`, OpenNTPD's `ntpd.conf`); the zone by
`timedatectl set-timezone`; chrony's servers in a file of their own --
`/etc/chrony/sources.d/netork.sources` where chrony reads that directory, else a `confdir`
or an included `/etc/chrony.d/netork.conf` -- with every other `server`, `pool` and `peer`
line commented out behind `#napalm-time-sync: `, then `chronyc reload sources` or a
restart; systemd-timesyncd's in `/etc/systemd/timesyncd.conf.d/netork.conf` (`NTP=` reset,
`FallbackNTP=` emptied), then a restart. What DHCP brings is read apart and kept: chrony's
DHCP `sourcedir` (`CHRONY_DHCP_DIRS`, `/run/chrony-dhcp`) is never commented out -- one
that 4.35.0 commented out comes back on the next write -- and systemd-timesyncd's
`LinkNTPServers` stay. ntpd and OpenNTPD are read, not written; what DHCP gives them is
unknown.
`ConfigRevisionsMixin` (`get_config_revisions`, `get_config_revision`,
`revert_config_revision`) is a device's own configuration history: a copy of the
configuration for every save, which the device can go back to. An application restores a
configuration there by reverting to the revision that holds it -- on OPNsense the only way,
since its API takes no uploaded config.xml (NetOrk/netork#2511). Listing (newest first, the
first in place now: id, UTC time, description, user, size as `ConfigRevisionDict`) and
reading one revision are declared only. `revert_config_revision(revision_id)` is a
template: it refuses an id that is not in the history (`UnknownConfigRevision`, a
`ValueError`) before anything changes, reads the revision -- before the revert, which may
rotate it out -- lets the driver revert (`_revert_to_config_revision`), and checks that the
newest revision afterwards holds the revision's configuration (`ConfigRevertFailed`
otherwise: a device may say it reverted when it did not). It returns that newest revision.
`config_revision_content(text)` is what of a revision is its configuration; a driver whose
device stamps every save (who, when) strips the stamp there, and an application that looks
for the revision holding a configuration compares through it too. A driver that can list
but not revert sets `revert_config_revision = None`.
### Access channels: why there is no container model here
`CommandChannelMixin` declares a driver's public channel to its host:
@@ -305,6 +444,26 @@ caller's; `ssh_host_keys.HostKeyCheck` is how every SSH driver verifies against
A caller that passes neither key gets first use on every connection, as before; a driver
that ignores both still connects. Old and new versions on either side keep working.
### TLS certificates: trust on first use
The same for a driver's HTTPS API. Devices ship self-signed certificates, so a CA check
fails on all of them and turning verification off trusts whatever answers;
`tls_certificates.CertificateCheck` pins the certificate by its fingerprint instead
(NAPALM/napalm-hpe-aruba-procurve#17). No CA is consulted.
- **In:** `optional_args["tls_fingerprints"]` = `{"443": "SHA256:AB:CD:…"}`, by port: the
SHA-256 of the DER certificate as `openssl x509 -fingerprint -sha256` prints it (prefix,
colons and case optional). With one, any other certificate is refused *before* a request
carries credentials and the connection raises `TlsCertificateMismatch` (a
`ConnectionException` naming both fingerprints). Without one, the first certificate is
accepted. A driver does not fall back to another transport on a mismatch.
- **Out:** `optional_args["tls_certificate_observer"]`, a callable, gets one dict per
connection: `{"host", "port", "seen": {"fingerprint"}, "expected", "accepted"}`.
- **requests:** `check = CertificateCheck(optional_args, host, port)`;
`check.verify_server(timeout)` before the login, `check.pin(session)` so every later
connection of the session must present that certificate, the login inside
`with check.refusing():`, then `check.observe()`.
## Design principle: generic vs. device-specific logic
When adding behavior to a device-type base class, split it along one line: **would
@@ -351,7 +510,10 @@ The same split applies to `DhcpServerMixin`: `get_dhcp_reservations`/
`apply_dhcp_reservation`/`commit_dhcp_reservations` are abstract (Kea REST on
OPNsense, dnsmasq/odhcpd UCI on OpenWrt), while `diff_dhcp_reservations` and
`apply_dhcp_reservationset` are concrete — matching by normalised MAC and the
apply-then-commit orchestration are identical for every DHCP server.
apply-then-commit orchestration are identical for every DHCP server. The apply writes
its changes so an address is released before another reservation takes it (an update
that frees an address first, adds last); a swap of addresses has no such order and is
reported as one where the server refuses it.
A new driver (FortiGate, pfSense, …) gets `diff_firewall_rules`/
`apply_firewall_ruleset` for free the moment it implements the three abstract
@@ -388,7 +550,7 @@ detection vs. presentation there.
pip install napalm-device-types
```
Requires Python ≥ 3.10 (tested up to 3.13) and NAPALM ≥ 4.0.
Requires Python ≥ 3.12 (tested up to 3.13) and NAPALM ≥ 4.0.
## Available base classes
@@ -404,6 +566,7 @@ Requires Python ≥ 3.10 (tested up to 3.13) and NAPALM ≥ 4.0.
| `PhoneDriver` | IP telephony endpoints | Yealink |
| `MediaDriver` | Speakers, streamers, media renderers | Sonos, Chromecast |
| `ManagementControllerDriver` | Server baseboard management controllers | Supermicro BMC, Dell iDRAC, HPE iLO, Lenovo XCC |
| `BackupServerDriver` | Backup servers: datastores, backup groups, GC/verify/prune/sync/tape jobs | Proxmox Backup Server |
Mixins that come with the classes above: `PingSweepMixin` (subnet sweeps, in
every driver through `DeviceTypeDriver`) and `DhcpServerMixin` (static DHCP
@@ -501,6 +664,14 @@ class ProxmoxDriver(HypervisorDriver):
def create_vm_snapshot(self, name, snapshot, description="", include_memory=False): ...
def get_network_targets(self):
# return List[NetworkTargetDict]: bridges and vnets a new VM's NIC may
# attach to; a bridge may name its physical NICs in `uplinks`, one group
# per way out: [["eno1", "eno2"]] for a bond, [["eno1"]] for a single NIC,
# no internal or VLAN ports -- leave the key out when they cannot be
# determined
...
def get_vm_cpu_types(self):
# optional — return List[VMCpuTypeDict]: the CPU models a new VM may get
# on this node, each with its cpuinfo flags and whether the node can run
@@ -516,7 +687,10 @@ any other. A QEMU-based driver takes `provisioning.QEMU_GUEST_AGENTS`
(Linux, FreeBSD, OpenBSD). `provisioning.network_config()` writes the
network-config v2 every cloud-init flavour reads (FreeBSD's nuageinit reads
no other), and `provisioning.split_compression()` says how a packed image is
unpacked; both are the same for every hypervisor.
unpacked; both are the same for every hypervisor. A creation that fails once
the VM exists raises `VMProvisionIncomplete`, a `RuntimeError` that names the
half-built VM (`vmid`, `node`) for the caller to `destroy_vm`, with the failure
as its cause.
### OS / Linux
@@ -540,10 +714,6 @@ class LinuxDriver(OSDriver):
def get_processes(self):
# return List[ProcessDict]
...
def get_cron_jobs(self):
# return List[CronJobDict]
...
```
### Storage / NAS
+245
View File
@@ -0,0 +1,245 @@
# Structure
Status: agreed plan. Nothing here is implemented yet; the migration at the end says in
which order it will be.
## Principle
**napalm-device-types is vendor-, OS-, protocol- and product-agnostic.** It holds the
contracts every driver implements and the generic logic that works unchanged for every
driver of a role. It knows no shell, no paramiko, no package manager, no container CLI,
no SNMP OID, no cloud-init and no application that uses the drivers.
That rules out three kinds of code that are here today:
- **How a kind of host is read.** Shell commands and their parsers for POSIX hosts move to
a new package, **napalm-unix**, which napalm-linux and napalm-bsd depend on.
- **What an application does with a device.** Polling a device over SNMP, sweeping a range
with pings, scoring discovery evidence, choosing what a provisioned guest installs, and
deciding what counts as a problem belong to the application, not to the drivers.
- **Product naming.** Files, markers and messages a driver leaves on a device carry the
driver's name or none, never the name of the application that called it.
```
napalm-device-types roles, capabilities, templates, models, reconcilers <- every driver
^
napalm-unix POSIX host readers, SSH exec transport, privilege <- napalm-linux, napalm-bsd
^
napalm-linux, napalm-bsd connection, platform getters, platform steps
application SNMP polling, ping sweeps, discovery scoring, guest
profiles, warnings, and the values every fix is given
```
## What stays in napalm-device-types
| Kind | Rule | Modules |
|---|---|---|
| **Role** | Declares only, under `TYPE_CHECKING`. No runtime method. | `access_point`, `switch`, `firewall`, `hypervisor`, `os`, `storage`, `phone`, `media`, `residential_gateway`, `management_controller`, `backup_server` |
| **Capability** | Declares only. | `packages`, `services`, `updates`, `nat_vpn`, `mac_acl`, `config_lifecycle`, `interface_filter`, `host_reboot` |
| **Template** | A concrete method that runs a fixed sequence of abstract steps. A driver implements the steps, or overrides the whole method when its device does it in one go. All values come from the caller. | the fix routines below |
| **Contract types** | Interfaces and data shapes other packages implement or fill. | `CommandChannelMixin`, `CommandResult`, `ByteStream`; the declaration of `ContainerEngineMixin`; the reader mixins' public method declarations; the SNMP profile type; `models` |
| **Reconciler** | Concrete and vendor-neutral, without I/O of its own. | `firewall_rules` and `dhcp` diff/apply, `_matching`, `_dhcp_order`, `lag` |
| **Introspection** | Concrete, without I/O. | `roles`; `base` (`FingerprintRule`, `PortSpec`, the fingerprint attributes) |
| **Driver facts** | Class attributes that describe the device or how it is reached, not what an application does with it. | `USES_SSH`, `FULL_UPGRADE`, `platform`, `SNMP_HEALTH_PROFILE` |
### Fix routines are templates
A change the application wants made on a device, for example "run an SNMP agent that
answers us", is one template per purpose. The template fixes the order of the steps; the
driver knows how each step is done on its device; the caller supplies every value.
```python
class SnmpAgentMixin:
def configure_snmp_agent(self, settings: SnmpAgentSettingsDict) -> FixResultDict:
steps = [
self._install_snmp_agent,
self._write_snmp_agent_config,
self._enable_snmp_agent,
self._verify_snmp_agent,
]
... # runs them in order, stops at the first failure
# declared only; napalm-unix implements them for net-snmp
def _install_snmp_agent(self) -> FixStepDict: ...
def _write_snmp_agent_config(self, settings: SnmpAgentSettingsDict) -> FixStepDict: ...
def _enable_snmp_agent(self) -> FixStepDict: ...
def _verify_snmp_agent(self, settings: SnmpAgentSettingsDict) -> FixStepDict: ...
```
napalm-unix implements the four steps for net-snmp, for Linux and the BSDs alike. A REST
driver such as OPNsense overrides `configure_snmp_agent` as a whole, because its device
takes the whole configuration in one call. The community, the allowed sources, contact
and location are the caller's.
The same shape replaces every action in today's `run_device_action`:
| Today | Template |
|---|---|
| `fix_snmp` | `configure_snmp_agent(settings)` |
| `fix_apt_proxy` | `configure_package_proxy(url)` |
| `fix_docker_permissions` | `grant_container_engine_access(user)` |
| `install_lldpd`, `install_auc` | `ensure_package(name)` |
The mixin is mixed in by the driver, or by the platform package, not by a role base. A
driver that inherits a template it cannot run on its device, as a NAS driver built on the
Linux driver inherits the package proxy, opts out by setting the method to `None`. So a
caller asks `callable(getattr(driver, "configure_snmp_agent", None))`, never `hasattr`.
The caller verifies the result from its own side where it can; the SNMP agent is asked
by the application's SNMP client, not by a probe the driver hand-builds.
### SNMP: drivers declare data, the application polls
The drivers do not collect over SNMP. A driver whose device answers the standard UCD-MIB
and IF-MIB declares nothing. A driver whose device publishes the same values under
proprietary OIDs declares them as data, in a class attribute of a type defined here:
```python
class ProcurveDriver(SwitchDriver):
SNMP_HEALTH_PROFILE: ClassVar[SnmpHealthProfile] = {
"uptime": {"oids": ["1.3.6.1.4.1.11.2.14.11.1.2.4.1.5.1"], "format": "ticks"},
"cpu": {"oids": ["1.3.6.1.4.1.11.2.14.11.5.1.9.6.1.0", "1.3.6.1.4.1.11.2.3.7.8.6.1.0"]},
"memory": {"total": "...", "free": "...", "unit": "bytes"},
}
```
One OID per value is not enough for the drivers there are. The profile type has to say:
- **Fallbacks:** an ordered list of OIDs; the first that answers wins (ProCurve's CPU).
- **Tables:** an OID to walk and how to fold its rows: first, average or maximum (TP-Link
reports CPU and memory per stack unit).
- **Text values:** a pattern that takes the number out of a string (Netgear reports CPU
as `"5 Secs ( 3.2%)"`).
- **Memory:** total with used or free, or a percentage only, and the unit.
- **Uptime:** which OID, standard ones included (`sysUpTime` where a device has no
`hrSystemUptime`), and its format.
- **Interfaces:** a pattern of names to skip, and whether the TX-error counter counts
drops. Today the second is `True` on every firewall, because OPNsense counts drops
there; it becomes OPNsense's profile entry.
Polling, the standard OIDs, the counter arithmetic (64-bit counters first, 32-bit as the
fallback) and the defaults are the application's.
### Warnings are facts
A driver reports facts: the proxy a host is configured with, a DSL line's SNR, the
firmware version it runs and the one it is offered. It does not compare them with a
setting of the application or with a threshold, and it does not name a warning.
The same goes for lookups a device does not make itself: what firmware a vendor
currently publishes is the application's catalogue, not a getter.
### Getters read, and say when they cannot
- A getter changes nothing on the device. It does not start a daemon, rewrite its
configuration or install a package on the way to an answer.
- A read that fails raises. An empty list means "there is none", never "could not ask";
a sync that deletes and re-adds on top of an empty answer would otherwise duplicate
everything.
- A write reports failure from the device's answer or exit status, never `success: True`
by default.
### The application uses public methods only
The application never calls a driver's private methods (`_get`, `_post`,
`_send_command`, `_action_*`). What it needs from a device becomes a declared method of
the driver.
## What moves to napalm-unix
| Module | Knows |
|---|---|
| `host_status` | apt, dnf, dpkg, systemd timers, `df` |
| `package_managers` | rpm, dnf and yum history, apk's and opkg's databases, `/proc/<pid>/comm` |
| `kernel` | `/proc`, `/lib/modules`, gzip, base64 |
| `listening` | `ss`, `netstat`, `/proc/<pid>/cgroup` |
| `systemd` | `systemctl` |
| `virtual_ips` | vip-manager, keepalived, awk |
| `system_identity` | SMBIOS via sysfs and `dmidecode` |
| `package_updates` | apt and dnf output |
| `ssh_server` | OpenSSH (`sshd -T`, `sshd -t`, `sshd_config`); which algorithms to drop stays the caller's |
| `terminal` | ANSI escape codes in shell output |
| `channel`: `run_on_transport`, `open_stream_on_transport`, `ParamikoExecStream` | paramiko exec channels |
| `container_engine`: `ContainerEngineConnection`, engine probing | the Docker CLI, `system dial-stdio` |
It also takes what napalm-linux and napalm-bsd each carry today:
- **Privilege:** one `sudo -S` / `sudo -n` / root layer, including the rule that a
privileged command never gets stdin while a sudo password is set, and one way to write a
file as root (stage with `mktemp`, then `install`). Two copies of this have already
drifted apart.
- **Parsers:** `/etc/passwd` with `/etc/group`, crontab lines, `ps` with `lstart`.
- **Fix steps:** net-snmp, the package proxy, container engine group membership.
The FreeBSD part of `host_status` (`freebsd-version -k` against `-r`) moves to napalm-bsd.
In napalm-unix every reader follows one pattern:
- **Shape:** `FOO_COMMAND`, `parse_foo(output)`, and `FooMixin.get_foo()`.
- **Reaching the host:** the mixin uses `run_command(command, privileged=...)` from
`CommandChannelMixin`. The six per-reader `_run_*` hooks, with four different signatures,
go away.
- **Framing:** one frame helper writes and parses the `BEGIN`/`END` markers for every reader.
- **Errors:** one exception for "cannot read" and one for "not on this host". A read that
fails or is cut short raises; it never reads as "nothing there".
- **Names on the host:** temporary files, markers and drop-ins carry `napalm` or the
driver's name. A drop-in written under an older name is still recognised and replaced.
Layout: `napalm_unix/{posix,linux,freebsd,openssh,transport}/`.
## What moves to the application
| Today | Why it is not a driver's |
|---|---|
| `health_metrics`, `_ucd_metrics` | Polling over SNMP is something the application does to a device; the driver is not even connected while it runs. |
| `ping_sweep` | Looping over addresses, capping the targets and reporting progress is a scan. A device with a faster batch keeps it as a capability of its own. |
| `rules_max_weight` | Scoring is how one application ranks drivers. The rules themselves stay with the driver. |
| `provisioning` (`QEMU_GUEST_AGENTS`, cloud-init network-config, decompressors) | Which agent a guest gets and how its network is configured is a guest profile. The hypervisor driver is handed the packages and commands. |
| `TYPE_LABEL`, `REBOOT_SETTLE_SECONDS` | A display name and a polling delay. |
| firmware catalogues (napalm-netgear's `netgear_plus`) | What a vendor publishes is not something the device reports. |
| `run_device_action` | Replaced by the templates above. |
`OSDriver.ROLE` stops defaulting to `"linux"`. Each OS driver states its own role.
## Rules, enforced by tests in this package
1. Roles and capabilities own no runtime method; a template runs only declared steps.
2. No module imports `paramiko`, `socket`, `subprocess` or `asyncio`, and none defines a
`*_COMMAND`.
3. No module, string or default names an application.
4. Role methods return TypedDicts, not `Any`.
Each rule is added with the step that makes it pass.
## Migration
This package cannot keep re-exports pointing at napalm-unix: the contract package would
then depend on its implementation. So a moved name is first deprecated here and then
removed, and nothing breaks in between.
| Step | Change |
|---|---|
| 1 | The application moves to 4.9 and the drivers that need it; napalm-bsd stops passing stdin to a privileged command while a sudo password is set. |
| 2 | This document; the application's architecture docs and this README follow it. |
| 3 | `SnmpHealthProfile`. The application polls on its own; the four switch and phone drivers declare their OIDs as data. `HealthMetricsMixin` is deprecated. |
| 4 | The fix templates. napalm-linux and napalm-bsd implement the steps; the application calls the templates with its values. `run_device_action` is deprecated. The other drivers follow, one issue each. |
| 5 | Ping sweep, scoring, provisioning, `TYPE_LABEL` and `REBOOT_SETTLE_SECONDS` move to the application and are deprecated here. Drivers report facts instead of warnings. |
| 6 | napalm-unix, with the moved code and its tests, the unified hooks and the frame helper. napalm-linux and napalm-bsd import from it; the moved names are deprecated here. |
| 7 | 5.0 removes everything deprecated. It also splits `models.py` by area, types the remaining `Dict[str, Any]` role returns, declares what drivers already offer undeclared (`get_networks`, `search_packages`, `first_boot_pending`, the base-system update entry), and makes `install_package` declare the result drivers return. |
| 8 | The README shrinks to the concepts; the reader details move with the readers. |
## Proposals this replaces
Several open issues proposed adding shared driver code here. Under this structure:
| Issue | Goes to |
|---|---|
| #8 netmiko session | not here: a CLI-driver helper package, or left in the drivers |
| #9 config mode for CLI switches | not here: a CLI-driver helper package, or left in the drivers |
| #10 LLDP rows | the projection of rows into NAPALM's two shapes stays here; the `lldpctl` parser goes to napalm-unix |
| #11 CLI text helpers | not here; the natural port sort, which `lag` already has, may become public |
| #12 ping | the reply builder stays here; the BusyBox and iputils parser goes to napalm-unix |
| #13 Linux userland | napalm-unix; the snmpd part becomes the SNMP template's steps |
| #14 `run_device_action` | replaced by the fix templates |
| #22 hooks onto `run_command` | step 6 |
+225 -5
View File
@@ -22,6 +22,7 @@ driver do X".
Role bases -- what a device *is*:
* :class:`~napalm_device_types.access_point.AccessPointDriver`
* :class:`~napalm_device_types.backup_server.BackupServerDriver`
* :class:`~napalm_device_types.switch.SwitchDriver`
* :class:`~napalm_device_types.firewall.FirewallDriver`
* :class:`~napalm_device_types.hypervisor.HypervisorDriver`
@@ -37,6 +38,7 @@ instead of being restated on every role that happens to need it:
* :class:`~napalm_device_types.channel.CommandChannelMixin`
* :class:`~napalm_device_types.config_lifecycle.ConfigLifecycleMixin`
* :class:`~napalm_device_types.config_revisions.ConfigRevisionsMixin`
* :class:`~napalm_device_types.config_sync.ConfigSyncMixin`
* :class:`~napalm_device_types.container_engine.ContainerEngineMixin`
* :class:`~napalm_device_types.dhcp.DhcpServerMixin`
@@ -44,20 +46,31 @@ instead of being restated on every role that happens to need it:
* :class:`~napalm_device_types.health_metrics.HealthMetricsMixin`
* :class:`~napalm_device_types.host_status.HostStatusMixin`
* :class:`~napalm_device_types.host_reboot.HostRebootMixin`
* :class:`~napalm_device_types.host_shutdown.HostShutdownMixin`
* :class:`~napalm_device_types.interface_filter.InterfaceFilterMixin`
* :class:`~napalm_device_types.kernel.KernelFactsMixin`
* :class:`~napalm_device_types.listening.ListeningSocketsMixin`
* :class:`~napalm_device_types.local_accounts.LocalAccountsMixin`
* :class:`~napalm_device_types.log_sources.LogSourcesMixin`
* :class:`~napalm_device_types.host_firewall.HostFirewallMixin`
* :class:`~napalm_device_types.mac_acl.MacAclMixin`
* :class:`~napalm_device_types.nat_vpn.NatVpnMixin`
* :class:`~napalm_device_types.packages.PackageManagementMixin`
* :class:`~napalm_device_types.ping_sweep.PingSweepMixin`
* :class:`~napalm_device_types.running_backups.RunningBackupsMixin`
* :class:`~napalm_device_types.service_clusters.ServiceClustersMixin`
* :class:`~napalm_device_types.service_packages.ServicePackagesMixin`
* :class:`~napalm_device_types.ssh_server_auth.SshServerAuthMixin`
* :class:`~napalm_device_types.sudo_history.SudoHistoryMixin`
* :class:`~napalm_device_types.scheduled_jobs.ScheduledJobsMixin`
* :class:`~napalm_device_types.scheduled_jobs_posix.PosixScheduledJobsMixin`
* :class:`~napalm_device_types.services.ServiceControlMixin`
* :class:`~napalm_device_types.systemd.SystemdServicesMixin`
* :class:`~napalm_device_types.time_sync.TimeSyncMixin`
* :class:`~napalm_device_types.time_sync_posix.PosixTimeSyncMixin`
* :class:`~napalm_device_types.updates.UpdateMixin`
* :class:`~napalm_device_types.virtual_ips.VirtualIpsMixin`
* :class:`~napalm_device_types.vm_stop_request.VmStopRequestMixin`
Introspection -- :func:`~napalm_device_types.roles.roles_of`,
:func:`~napalm_device_types.roles.role_keys_of` and
@@ -68,10 +81,28 @@ from napalm_device_types.access_point import AccessPointDriver
from napalm_device_types.apt_proxy import (
APT_PROXY_COMMAND,
AptProxyMixin,
https_proxy_for,
parse_apt_fetch,
parse_apt_proxy,
same_proxy_url,
)
from napalm_device_types.arp import ARP_AGE_UNKNOWN
from napalm_device_types.backup_server import BackupServerDriver
from napalm_device_types.backups import (
BACKUP_JOB_KINDS,
BackupDatastoreDict,
BackupGroupDict,
BackupJobKind,
BackupRemoteDict,
BackupRunState,
BackupServerJobDict,
BackupServerRefDict,
BackupSnapshotDict,
BackupSnapshotRefDict,
BackupTaskDict,
BackupTaskStatusDict,
BackupVerifyState,
)
from napalm_device_types.base import DeviceTypeDriver, FingerprintRule, PortSpec, rules_max_weight
from napalm_device_types.channel import (
ByteStream,
@@ -81,20 +112,36 @@ from napalm_device_types.channel import (
open_stream_on_transport,
run_on_transport,
)
from napalm_device_types.clusters import (
BackupDict,
BackupJobDict,
ClusterDict,
ClusterNetworkDict,
ClusterNodeDict,
ClusterStorageDict,
HAResourceDict,
)
from napalm_device_types.config_lifecycle import ConfigLifecycleMixin
from napalm_device_types.config_revisions import (
ConfigRevertFailed,
ConfigRevisionDict,
ConfigRevisionsMixin,
UnknownConfigRevision,
)
from napalm_device_types.config_sync import ConfigSyncMixin
from napalm_device_types.container_engine import (
ContainerEngineConnection,
ContainerEngineMixin,
ContainerEngineUnavailable,
)
from napalm_device_types.dhcp import DhcpServerMixin, normalize_cidr, normalize_mac
from napalm_device_types.dhcp import DhcpApplyError, DhcpServerMixin, normalize_cidr, normalize_mac
from napalm_device_types.dns_forward import DnsForwardMixin
from napalm_device_types.firewall import FirewallDriver
from napalm_device_types.firewall_rules import FirewallRuleMixin
from napalm_device_types.firmware import FirmwareOfferMixin
from napalm_device_types.fixes import (
EnsurePackageMixin,
LldpAgentMixin,
PackageProxyMixin,
SnmpAgentMixin,
run_fix_steps,
@@ -106,6 +153,7 @@ from napalm_device_types.host_firewall import (
parse_host_firewall,
)
from napalm_device_types.host_reboot import HostRebootMixin
from napalm_device_types.host_shutdown import HostShutdownMixin
from napalm_device_types.host_status import HOST_STATUS_COMMAND, HostStatusMixin, parse_host_status
from napalm_device_types.hypervisor import HypervisorDriver
from napalm_device_types.interface_filter import InterfaceFilterMixin
@@ -118,6 +166,33 @@ from napalm_device_types.listening import (
ListeningSocketsUnavailable,
parse_listening_sockets,
)
from napalm_device_types.lldp import (
LldpNeighborDict,
LldpNeighborRow,
lldp_neighbors,
lldp_neighbors_detail,
)
from napalm_device_types.lldpctl import LLDPCTL_KEYVALUE_COMMAND, parse_lldpctl_keyvalue
from napalm_device_types.local_accounts import (
LocalAccountChangeDict,
LocalAccountDict,
LocalAccountsMixin,
)
from napalm_device_types.log_forwarding import (
FileInputDict,
ForwardRuleDict,
LogForwardingDict,
parse_rsyslog,
)
from napalm_device_types.log_sources import (
LOG_SOURCES_COMMAND,
LogSourceDict,
LogSourcesDict,
LogSourcesMixin,
LogSourcesUnavailable,
parse_log_forwarding,
parse_log_sources,
)
from napalm_device_types.mac_acl import MacAclMixin
from napalm_device_types.management_controller import ManagementControllerDriver
from napalm_device_types.media import MediaDriver
@@ -142,10 +217,6 @@ from napalm_device_types.models import (
PowerAction,
PowerStateDict,
SnmpAgentSettingsDict,
SnmpHealthProfile,
SnmpInterfacesSpec,
SnmpMemorySpec,
SnmpValueSpec,
UpdateProgressDict,
)
from napalm_device_types.nat_vpn import NatVpnMixin
@@ -153,15 +224,44 @@ from napalm_device_types.os import OSDriver
from napalm_device_types.ospf import OspfMixin
from napalm_device_types.package_updates import (
APT_UPGRADABLE_COMMAND,
DNF_INSTALLED_CVES_COMMAND,
DNF_SECURITY_COMMAND,
parse_apt_upgradable,
parse_dnf_installed_cves,
parse_dnf_security,
)
from napalm_device_types.packages import PackageManagementMixin
from napalm_device_types.phone import PhoneDriver
from napalm_device_types.ping_sweep import PingSweepMixin, driver_supports_ping
from napalm_device_types.posix_tz import (
DstTransitionDict,
PosixTzDict,
UnknownTimezone,
is_iana_zone,
is_posix_tz,
parse_posix_tz,
posix_tz,
)
from napalm_device_types.provisioning import VMProvisionIncomplete
from napalm_device_types.residential_gateway import ResidentialGatewayDriver
from napalm_device_types.roles import primary_role_of, role_keys_of, roles_of
from napalm_device_types.running_backups import (
RUNNING_BACKUP_KINDS,
RunningBackupDict,
RunningBackupKind,
RunningBackupsMixin,
)
from napalm_device_types.scheduled_jobs import (
ScheduledJobChanged,
ScheduledJobDict,
ScheduledJobRefused,
ScheduledJobResultDict,
ScheduledJobsMixin,
ScheduledJobSpecDict,
ScheduledJobsReadingDict,
job_digest,
)
from napalm_device_types.scheduled_jobs_posix import PosixScheduledJobsMixin
from napalm_device_types.service_clusters import (
SERVICE_CLUSTERS_COMMAND,
ServiceClustersMixin,
@@ -174,6 +274,12 @@ from napalm_device_types.service_packages import (
parse_service_packages,
)
from napalm_device_types.services import ServiceControlMixin
from napalm_device_types.snmp_health_profile import (
SnmpHealthProfile,
SnmpInterfacesSpec,
SnmpMemorySpec,
SnmpValueSpec,
)
from napalm_device_types.ssh_host_keys import (
KNOWN_KEYS_ARG as SSH_HOST_KEYS_ARG,
)
@@ -193,7 +299,21 @@ from napalm_device_types.ssh_server import (
SshHardeningResult,
plan_ssh_hardening,
)
from napalm_device_types.ssh_server_auth import (
SSHD_EFFECTIVE_COMMAND,
PamAuthEntryDict,
SshServerAuthDict,
SshServerAuthMixin,
pam_auth_stack,
parse_sshd_auth,
)
from napalm_device_types.storage import StorageDriver
from napalm_device_types.sudo_history import (
HistoryCleanupDict,
HistorySudoLinesDict,
SudoHistoryCleanupDict,
SudoHistoryMixin,
)
from napalm_device_types.switch import SwitchDriver
from napalm_device_types.system_identity import (
SYSTEM_IDENTITY_COMMAND,
@@ -208,14 +328,44 @@ from napalm_device_types.systemd import (
parse_systemd_services,
)
from napalm_device_types.terminal import strip_terminal_codes
from napalm_device_types.time_sync import (
TIME_SYNC_VALUES,
TimeSyncDict,
TimeSyncMixin,
check_ntp_servers,
)
from napalm_device_types.time_sync_posix import PosixTimeSyncMixin, parse_time_sync
from napalm_device_types.updates import UpdateMixin
from napalm_device_types.virtual_ips import (
VIRTUAL_IPS_COMMAND,
VirtualIpsMixin,
parse_virtual_ips,
)
from napalm_device_types.vm_migration import MigrationPreflightDict
from napalm_device_types.vm_stop_request import VmStopRequestMixin
__all__ = [
"BACKUP_JOB_KINDS",
"BackupDatastoreDict",
"BackupGroupDict",
"BackupJobKind",
"BackupRemoteDict",
"BackupRunState",
"BackupServerJobDict",
"BackupServerRefDict",
"BackupSnapshotDict",
"BackupSnapshotRefDict",
"BackupTaskDict",
"BackupTaskStatusDict",
"BackupVerifyState",
"BackupDict",
"BackupJobDict",
"ClusterDict",
"ClusterNetworkDict",
"ClusterNodeDict",
"ClusterStorageDict",
"HAResourceDict",
"MigrationPreflightDict",
"HypervisorHostDict",
"DnsForwardDict",
"DnsForwardMixin",
@@ -231,9 +381,11 @@ __all__ = [
"FixResultDict",
"FixStepDict",
"FixStepOutcomeDict",
"LldpAgentMixin",
"PackageProxyMixin",
"APT_PROXY_COMMAND",
"AptProxyMixin",
"https_proxy_for",
"parse_apt_fetch",
"parse_apt_proxy",
"same_proxy_url",
@@ -245,6 +397,7 @@ __all__ = [
"SnmpMemorySpec",
"SnmpValueSpec",
"AccessPointDriver",
"BackupServerDriver",
"ByteStream",
"CommandChannelMixin",
"CommandResult",
@@ -255,11 +408,16 @@ __all__ = [
"open_stream_on_transport",
"run_on_transport",
"ConfigLifecycleMixin",
"ConfigRevisionsMixin",
"ConfigRevisionDict",
"ConfigRevertFailed",
"UnknownConfigRevision",
"ConfigSyncDict",
"ConfigSyncMixin",
"DhcpHaDict",
"DhcpHaPeerDict",
"DeviceTypeDriver",
"DhcpApplyError",
"DhcpServerMixin",
"FingerprintRule",
"rules_max_weight",
@@ -269,10 +427,12 @@ __all__ = [
"HOST_STATUS_COMMAND",
"HostStatusMixin",
"HostRebootMixin",
"HostShutdownMixin",
"HypervisorDriver",
"InterfaceFilterMixin",
"MacAclMixin",
"ManagementControllerDriver",
"ARP_AGE_UNKNOWN",
"POWER_ACTIONS",
"PowerAction",
"PowerStateDict",
@@ -285,8 +445,10 @@ __all__ = [
"OSDriver",
"PackageManagementMixin",
"APT_UPGRADABLE_COMMAND",
"DNF_INSTALLED_CVES_COMMAND",
"DNF_SECURITY_COMMAND",
"parse_apt_upgradable",
"parse_dnf_installed_cves",
"parse_dnf_security",
"parse_host_status",
"strip_terminal_codes",
@@ -302,12 +464,41 @@ __all__ = [
"parse_host_firewall",
"ListeningSocketsMixin",
"ListeningSocketsUnavailable",
"LOG_SOURCES_COMMAND",
"FileInputDict",
"ForwardRuleDict",
"LogForwardingDict",
"parse_log_forwarding",
"parse_rsyslog",
"LogSourceDict",
"LogSourcesDict",
"LogSourcesMixin",
"LogSourcesUnavailable",
"parse_log_sources",
"parse_listening_sockets",
"LLDPCTL_KEYVALUE_COMMAND",
"LldpNeighborDict",
"LldpNeighborRow",
"lldp_neighbors",
"lldp_neighbors_detail",
"parse_lldpctl_keyvalue",
"LocalAccountsMixin",
"LocalAccountDict",
"LocalAccountChangeDict",
"PhoneDriver",
"PingSweepMixin",
"PortSpec",
"ResidentialGatewayDriver",
"ServiceControlMixin",
"PosixScheduledJobsMixin",
"ScheduledJobChanged",
"ScheduledJobDict",
"ScheduledJobRefused",
"ScheduledJobResultDict",
"ScheduledJobSpecDict",
"ScheduledJobsMixin",
"ScheduledJobsReadingDict",
"job_digest",
"StorageDriver",
"SwitchDriver",
"SYSTEMD_SERVICES_COMMAND",
@@ -317,6 +508,25 @@ __all__ = [
"UpdateMixin",
"VIRTUAL_IPS_COMMAND",
"VirtualIpsMixin",
"VmStopRequestMixin",
"RUNNING_BACKUP_KINDS",
"RunningBackupDict",
"RunningBackupKind",
"RunningBackupsMixin",
"TIME_SYNC_VALUES",
"TimeSyncDict",
"TimeSyncMixin",
"check_ntp_servers",
"PosixTimeSyncMixin",
"parse_time_sync",
"DstTransitionDict",
"PosixTzDict",
"UnknownTimezone",
"is_iana_zone",
"is_posix_tz",
"parse_posix_tz",
"posix_tz",
"VMProvisionIncomplete",
"SERVICE_CLUSTERS_COMMAND",
"ServiceClustersMixin",
"parse_service_clusters",
@@ -326,10 +536,20 @@ __all__ = [
"is_placeholder_identity",
"parse_system_identity",
"OpenSshServerMixin",
"PamAuthEntryDict",
"SshServerAuthDict",
"SshServerAuthMixin",
"SSHD_EFFECTIVE_COMMAND",
"pam_auth_stack",
"parse_sshd_auth",
"SSH_ALGORITHM_CLASSES",
"SshHardeningPlan",
"SshHardeningResult",
"plan_ssh_hardening",
"SudoHistoryMixin",
"HistorySudoLinesDict",
"HistoryCleanupDict",
"SudoHistoryCleanupDict",
"SSH_HOST_KEYS_ARG",
"SSH_HOST_KEY_OBSERVER_ARG",
"HostKeyCheck",
+189
View File
@@ -0,0 +1,189 @@
"""How chrony is given exactly the time sources a caller names.
The sources go to one file of their own (:func:`plan_chrony` picks it):
1. the sources file in a ``sourcedir`` the main configuration reads (Debian's
``/etc/chrony/sources.d``): ``chronyc reload sources`` takes it without a restart;
2. else ``netork.conf`` -- the include file's name -- in the first ``confdir``;
3. else the include file, where an ``include`` pattern already takes it (SUSE's
``/etc/chrony.d/*.conf``);
4. else the include file, with an ``include`` line appended to the main configuration
(RHEL's ``/etc/chrony.conf``).
Every other configured source stops counting: a ``server``, ``pool`` or ``peer``
line in any other file chrony reads is commented out behind :data:`COMMENT_MARK`.
``sourcedir`` lines stay. The DHCP client's directory (``/run/chrony-dhcp``) and its
files are left alone: DHCP option 42 counts (NetOrk/netork#2380), and the client
rewrites those files on every lease anyway.
4.35.0 commented out every ``sourcedir`` other than the sources file's, the DHCP
client's among them, which left a host deaf to DHCP. A write gives a DHCP client's
``sourcedir`` back where it finds one behind :data:`COMMENT_MARK`; any other one it
took out stays out, its sources would be commented out now anyway.
chronyd reads its main configuration only when it starts: a change there means a
restart.
"""
from __future__ import annotations
import posixpath
import re
import shlex
from dataclasses import dataclass, field
from fnmatch import fnmatchcase
from napalm_device_types._time_sync_reading import (
CHRONY_DHCP_DIRS,
SOURCE_LINE,
ConfigFile,
in_dhcp_dir,
)
#: What a source line taken out of chrony's configuration starts with.
COMMENT_MARK = "#napalm-time-sync: "
#: The comment above an ``include`` line appended to the main configuration.
INCLUDE_COMMENT = "# napalm-time-sync: the time sources"
_DIRECTORY_LINE = re.compile(r"\s*(sourcedir|confdir|include)\s+(.+?)\s*$")
#: A ``sourcedir`` line 4.35.0 commented out.
_MARKED_SOURCEDIR = re.compile(rf"{re.escape(COMMENT_MARK)}sourcedir\s+(.+?)\s*$")
@dataclass
class ChronyPlan:
"""What :func:`plan_chrony` decided."""
target: str # the file the sources go to
content: str
comment_out: dict[str, list[int]] = field(default_factory=dict) # path: line numbers
restore: dict[str, list[int]] = field(default_factory=dict) # path: marked sourcedir lines
include_in: str | None = None # the main configuration an include of target is appended to
reload: bool = False # True: ``chronyc reload sources`` takes the change
def source_line(server: str) -> str:
"""The line a server is written as: a pool's name as a ``pool``, anything else a ``server``."""
directive = "pool" if ".pool." in f".{server.lower()}." else "server"
return f"{directive} {server} iburst"
def _directives(main: ConfigFile) -> list[tuple[int, str, list[str]]]:
found = []
for number, line in enumerate(main.lines, start=1):
match = _DIRECTORY_LINE.match(line)
if match:
found.append((number, match[1], match[2].split()))
return found
def _arguments(directives: list[tuple[int, str, list[str]]], kind: str) -> list[str]:
return [arg for _, k, args in directives if k == kind for arg in args]
def _target(main: ConfigFile, sources_file: str, include_file: str) -> tuple[str, str | None, bool]:
"""The file the sources go to, the main configuration to include it from, and
whether it is a sources file."""
directives = _directives(main)
if posixpath.dirname(sources_file) in _arguments(directives, "sourcedir"):
return sources_file, None, True
confdirs = _arguments(directives, "confdir")
if confdirs:
return posixpath.join(confdirs[0], posixpath.basename(include_file)), None, False
if any(fnmatchcase(include_file, p) for p in _arguments(directives, "include")):
return include_file, None, False
return include_file, main.path, False
def _dhcp_sourcedirs_to_restore(main: ConfigFile, dhcp_dirs: tuple[str, ...]) -> list[int]:
"""The lines of *main* where 4.35.0 commented out a DHCP client's ``sourcedir``."""
dhcp = {posixpath.normpath(d) for d in dhcp_dirs}
return [
number
for number, line in enumerate(main.lines, start=1)
if (marked := _MARKED_SOURCEDIR.match(line))
and all(posixpath.normpath(arg) in dhcp for arg in marked[1].split())
]
def plan_chrony(
files: list[ConfigFile],
servers: list[str],
*,
sources_file: str,
include_file: str,
dhcp_dirs: tuple[str, ...] = CHRONY_DHCP_DIRS,
) -> ChronyPlan:
"""Where *servers* go, what else is commented out, and which DHCP ``sourcedir``
(one of *dhcp_dirs*) comes back.
:raises RuntimeError: when the reading found no chrony configuration.
"""
main = next((f for f in files if f.kind == "main"), None)
if main is None:
raise RuntimeError("chrony runs, but no chrony configuration was found to write to")
target, include_in, is_sources = _target(main, sources_file, include_file)
comment_out = _comment_out(files, target, dhcp_dirs)
restore = _dhcp_sourcedirs_to_restore(main, dhcp_dirs)
return ChronyPlan(
target=target,
content="".join(f"{source_line(s)}\n" for s in servers),
comment_out=comment_out,
restore={main.path: restore} if restore else {},
include_in=include_in,
reload=(
is_sources
and include_in is None
and not restore
and _only_sources_files(files, comment_out)
),
)
def _only_sources_files(files: list[ConfigFile], changed: dict[str, list[int]]) -> bool:
"""Whether every file that changes is one ``chronyc reload sources`` reads again."""
return all(f.kind == "sourcedir" for f in files if f.path in changed)
def _comment_out(
files: list[ConfigFile], target: str, dhcp_dirs: tuple[str, ...]
) -> dict[str, list[int]]:
"""Per file, the lines that take a source; the DHCP client's files are left alone."""
found: dict[str, list[int]] = {}
for f in files:
if f.path == target or in_dhcp_dir(f, dhcp_dirs):
continue
numbers = [n for n, line in enumerate(f.lines, start=1) if SOURCE_LINE.match(line)]
if numbers:
found[f.path] = numbers
return found
def comment_out_command(path: str, numbers: list[int]) -> str:
"""``sed`` that comments out those lines of *path* that still hold a source."""
edits = " ".join(
"-e "
+ shlex.quote(f"{n}{{/^[[:space:]]*(server|pool|peer)[[:space:]]/s/^/{COMMENT_MARK}/;}}")
for n in numbers
)
return f"sed -i -E {edits} {shlex.quote(path)}"
def restore_command(path: str, numbers: list[int]) -> str:
"""``sed`` that takes :data:`COMMENT_MARK` off those lines of *path* that are still a
``sourcedir`` behind it. The mark holds nothing a regular expression or ``sed``'s
``s///`` would read as special."""
edits = " ".join(
"-e " + shlex.quote(f"{n}{{s/^{COMMENT_MARK}(sourcedir[[:space:]])/\\1/;}}")
for n in numbers
)
return f"sed -i -E {edits} {shlex.quote(path)}"
def include_command(main: str, target: str) -> str:
"""Append an ``include`` of *target* to *main*, on a line of its own."""
lines = " ".join(shlex.quote(line) for line in (INCLUDE_COMMENT, f"include {target}"))
return (
f'{{ [ -z "$(tail -c 1 -- {shlex.quote(main)})" ] || echo; '
f"printf '%s\\n' {lines}; }} >> {shlex.quote(main)}"
)
+222
View File
@@ -0,0 +1,222 @@
"""Crontabs and anacrontabs as text: the jobs in them, and changing one line of them.
Pure: no I/O. :mod:`napalm_device_types.scheduled_jobs_posix` reads the files and
writes them back; this module says what is in them and what the new text is.
**A line's key survives switching it off.** A job switched off is the same line
behind the marker :data:`DISABLED_MARKER`; the key is built from the line without
it, so the job keeps its key and the marker is all that changes. Two equal lines
in one file get ``:2``, ``:3`` after the first.
**A job an application wrote** is the line after ``# napalm-scheduled-job: <name>``
and is keyed ``job:<name>``, whatever its schedule.
**Everything else is left as it was**: comments, blank lines, variables, lines
cron would not take.
"""
from __future__ import annotations
import hashlib
import re
from dataclasses import dataclass, field
from shlex import quote
DISABLED_MARKER = "#napalm-disabled: "
JOB_MARKER = "# napalm-scheduled-job: "
_VARIABLE = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)$")
_ANACRON_PERIOD = re.compile(r"^(\d+|@(daily|weekly|monthly|yearly|annually))$")
@dataclass(frozen=True)
class CronLine:
"""One job in a crontab: where it is, what it says, and what was set before it."""
index: int # the line's number in the text, from 0
key: str
schedule: str
run_as: str
command: str
enabled: bool
env: dict[str, str] = field(default_factory=dict)
name: str = "" # the name an application gave it
description: str = ""
def _unquote(value: str) -> str:
value = value.strip()
if len(value) >= 2 and value[0] == value[-1] and value[0] in "'\"":
return value[1:-1]
return value
def _digest(text: str) -> str:
return hashlib.sha256(" ".join(text.split()).encode()).hexdigest()[:8]
def _split(content: str, *, system: bool, user: str) -> tuple[str, str, str] | None:
"""(schedule, run_as, command) of a cron line; None for one cron would not take."""
fields = 1 if content.startswith("@") else 5
owner = 1 if system else 0
parts = content.split(None, fields + owner)
if len(parts) < fields + owner + 1:
return None
schedule = " ".join(parts[:fields])
run_as = parts[fields] if system else user
return schedule, run_as, parts[-1]
def _anacron_split(content: str) -> tuple[str, str, str] | None:
"""(period, job identifier, command) of an anacrontab line."""
parts = content.split(None, 3)
if len(parts) < 4 or not _ANACRON_PERIOD.match(parts[0]) or not parts[1].isdigit():
return None
return parts[0], parts[2], parts[3]
@dataclass
class _Scan:
"""What is set while a text is read line by line."""
env: dict[str, str] = field(default_factory=dict)
seen: dict[str, int] = field(default_factory=dict)
name: str = ""
description: str = ""
def key(self, base: str) -> str:
n = self.seen.get(base, 0) + 1
self.seen[base] = n
return base if n == 1 else f"{base}:{n}"
def take_name(self) -> tuple[str, str]:
named = self.name, self.description
self.name, self.description = "", ""
return named
def _content(raw: str) -> tuple[str, bool] | None:
"""The job text of *raw* and whether it is on; None for a comment or a blank."""
line = raw.strip()
if line.startswith(DISABLED_MARKER.strip()):
return line[len(DISABLED_MARKER.strip()) :].strip(), False
if not line or line.startswith("#"):
return None
return line, True
def _remember(scan: _Scan, raw: str) -> bool:
"""Take a variable or an application's marker from *raw*; whether it was one."""
line = raw.strip()
if line.startswith(JOB_MARKER.strip()):
name, _, description = line[len(JOB_MARKER.strip()) :].strip().partition(" ")
scan.name, scan.description = name, description.strip()
return True
variable = _VARIABLE.match(line)
if variable:
scan.env[variable[1]] = _unquote(variable[2])
return True
return False
def parse_crontab(path: str, text: str, *, system: bool, user: str = "") -> list[CronLine]:
"""The jobs of the crontab *text* at *path*.
*system*: a sixth field names the user (``/etc/crontab``, ``/etc/cron.d``);
otherwise every job runs as *user*.
"""
scan, jobs = _Scan(), []
for index, raw in enumerate(text.splitlines()):
if _remember(scan, raw):
continue
found = _content(raw)
split = _split(found[0], system=system, user=user) if found else None
if found is None or split is None:
continue
content, enabled = found
name, description = scan.take_name()
key = f"job:{name}" if name else scan.key(f"cron:{path}:{_digest(content)}")
schedule, run_as, command = split
jobs.append(
CronLine(
index, key, schedule, run_as, command, enabled, dict(scan.env), name, description
)
)
return jobs
def parse_anacrontab(text: str) -> list[CronLine]:
"""The jobs of an anacrontab: period, delay, identifier, command; all run as root."""
scan, jobs = _Scan(), []
for index, raw in enumerate(text.splitlines()):
if _remember(scan, raw):
continue
found = _content(raw)
split = _anacron_split(found[0]) if found else None
if found is None or split is None:
continue
period, identifier, command = split
key = scan.key(f"anacron:{identifier}")
jobs.append(CronLine(index, key, period, "root", command, found[1], dict(scan.env)))
return jobs
def _replace(text: str, index: int, lines: list[str]) -> str:
rows = text.splitlines()
rows[index : index + 1] = lines
return "\n".join(rows) + "\n"
def switched(text: str, line: CronLine, *, enabled: bool) -> str:
"""*text* with *line* switched on or off: the marker added or taken away, nothing else."""
raw = text.splitlines()[line.index]
found = _content(raw)
if found is None: # pragma: no cover - a CronLine always points at a job
raise ValueError(f"line {line.index} is no job")
content = found[0]
return _replace(text, line.index, [content if enabled else DISABLED_MARKER + content])
def marked_lines(name: str, description: str, line: str, *, enabled: bool) -> list[str]:
"""An application's job as crontab lines: its marker, then the job, maybe switched off."""
marker = f"{JOB_MARKER}{name} {description}".rstrip()
return [marker, line if enabled else DISABLED_MARKER + line]
def with_job(text: str, jobs: list[CronLine], name: str, lines: list[str]) -> str:
"""*text* with the application's job *name* as *lines*: replaced, or added at the end."""
current = next((j for j in jobs if j.name == name), None)
if current is None:
body = text if not text or text.endswith("\n") else text + "\n"
return body + "\n".join(lines) + "\n"
rows = text.splitlines()
rows[current.index - 1 : current.index + 1] = lines
return "\n".join(rows) + "\n"
def without_job(text: str, jobs: list[CronLine], name: str) -> str:
"""*text* without the application's job *name* and its marker."""
current = next(j for j in jobs if j.name == name)
rows = text.splitlines()
del rows[current.index - 1 : current.index + 1]
return "\n".join(rows) + ("\n" if rows else "")
def _unescape(part: str) -> str:
return part.replace("\\%", "%")
def run_script(command: str, env: dict[str, str]) -> str:
"""*command* as ``sh -c`` takes it, the way cron would run it.
The crontab's variables are exported first. The first ``%`` that is not
escaped ends the command; what follows is its stdin, with every further
``%`` a newline, and ``\\%`` is a ``%``.
"""
parts = re.split(r"(?<!\\)%", command)
exports = "".join(f"export {k}={quote(v)}; " for k, v in env.items() if k != "MAILTO")
head = _unescape(parts[0])
if len(parts) == 1:
return exports + head
stdin = "\n".join(_unescape(p) for p in parts[1:])
return f"{exports}printf '%s\\n' {quote(stdin)} | {{ {head}\n}}"
+116
View File
@@ -0,0 +1,116 @@
"""The order an applied reservation set is written in (#106).
A DHCP server such as Kea keeps every address unique at each write, not only at the
commit: a reservation that takes an address another one still holds is refused as a
duplicate. So the changes are ordered for an address to be released before it is taken:
* An update that moves a reservation off its address goes before the change that takes
that address -- along a chain, from its free end.
* Adds come after every update: they release nothing, and an update may free their
address.
A *swap* -- two reservations trading addresses, or a longer rotation -- has no such
order. Its changes are still tried, in the order of the desired set: a server that checks
only at the commit takes them. A server that refuses them refuses each with its own
message, and the progress line says it was a swap. Nothing is moved to a temporary
address: the contract knows no free one (pools, leases and reservations nobody manages
all hold addresses), and a second write that failed would leave the reservation on it.
Without I/O: a reconciler's part, see ``docs/STRUCTURE.md``.
"""
from __future__ import annotations
import ipaddress
from dataclasses import dataclass
from napalm_device_types.models import DhcpReservationDict, DhcpReservationDiffDict
@dataclass(frozen=True)
class ReservationChange:
"""One write of an applied set, in the order it is made."""
reservation: DhcpReservationDict
#: The live reservation's id for an update, None for an add.
uuid: str | None
#: The changed fields of an update (``"ip, hostname"``), empty for an add.
fields: str = ""
#: The other reservations' MACs of the swap this change is part of, if any.
swap: tuple[str, ...] = ()
def _address(ip: str | None) -> str:
"""*ip* in one spelling, so ``192.0.2.010``-style differences cannot hide a match."""
text = (ip or "").strip()
try:
return str(ipaddress.ip_address(text))
except ValueError:
return text.lower()
def _waits_on(updates: list[tuple[str, str]]) -> list[int | None]:
"""For each ``(old, new)`` address pair, the update whose release it waits for."""
released = {old: i for i, (old, new) in enumerate(updates) if old and old != new}
return [released.get(new) if new != old else None for old, new in updates]
def _cycles(waits_on: list[int | None]) -> dict[int, list[int]]:
"""Each update that is part of a cycle, with the cycle's members in order."""
members: dict[int, list[int]] = {}
for start in range(len(waits_on)):
path: list[int] = []
node: int | None = start
while node is not None and node not in path and node not in members:
path.append(node)
node = waits_on[node]
if node is not None and node in path:
cycle = sorted(path[path.index(node) :])
members.update(dict.fromkeys(cycle, cycle))
return members
def _update_order(waits_on: list[int | None], cycles: dict[int, list[int]]) -> list[int]:
"""Every update after the one it waits on; stable otherwise. Where only cycles are
left, the first of them goes as it is, and so on."""
order: list[int] = []
done: set[int] = set()
pending = list(range(len(waits_on)))
while pending:
ready = [i for i in pending if waits_on[i] is None or waits_on[i] in done]
if not ready:
ready = [next(i for i in pending if i in cycles)]
order += ready
done.update(ready)
pending = [i for i in pending if i not in done]
return order
def order_reservation_changes(
diff: DhcpReservationDiffDict, live: list[DhcpReservationDict]
) -> list[ReservationChange]:
"""The changes of *diff* in the order they are written; see the module.
:param diff: What ``diff_dhcp_reservations`` found.
:param live: The reservations it compared against, for the addresses updates release.
"""
held = {r.get("uuid"): _address(r.get("ip")) for r in live if r.get("uuid")}
updates = diff["update"]
waits_on = _waits_on(
[(held.get(e["uuid"], ""), _address(e["reservation"].get("ip"))) for e in updates]
)
cycles = _cycles(waits_on)
def swap(i: int) -> tuple[str, ...]:
return tuple(updates[j]["reservation"]["mac"] for j in cycles.get(i, []) if j != i)
ordered = [
ReservationChange(
updates[i]["reservation"],
updates[i]["uuid"],
", ".join(updates[i]["changed_fields"]),
swap(i),
)
for i in _update_order(waits_on, cycles)
]
return ordered + [ReservationChange(r, None) for r in diff["add"]]
+4 -5
View File
@@ -8,12 +8,9 @@ name a single entry, and guessing would overwrite one the caller never meant.
from __future__ import annotations
from collections.abc import Callable, Iterable
from typing import TypeVar
T = TypeVar("T")
def index_live(entries: Iterable[T], key: Callable[[T], str]) -> tuple[dict[str, T], set[str]]:
def index_live[T](entries: Iterable[T], key: Callable[[T], str]) -> tuple[dict[str, T], set[str]]:
"""Live entries by key, and the keys more than one of them holds.
Entries without a key are left out: they are not the caller's to match.
@@ -32,7 +29,9 @@ def index_live(entries: Iterable[T], key: Callable[[T], str]) -> tuple[dict[str,
return by_key, shared
def check_desired(entries: list[T], key: Callable[[T], str], shared: set[str], what: str) -> None:
def check_desired[T](
entries: list[T], key: Callable[[T], str], shared: set[str], what: str
) -> None:
"""``ValueError`` for a desired entry with no key, a key given twice, or a
key that names more than one live entry."""
seen: set[str] = set()
+417
View File
@@ -0,0 +1,417 @@
"""The scheduled-jobs report of a POSIX host: the command that prints it, and its parse.
:mod:`napalm_device_types.scheduled_jobs_posix` sends :func:`reading_command` and
hands what came back to :func:`parse_reading`. Pure: no I/O.
**One framed report, as root.** Every crontab, cron.d file, anacrontab and job
script is printed behind a header line and with ``|`` before each of its lines,
so no line of a file can pass for a header; run-parts scripts by their execute
bit and checksum; systemd's timers and the units they start by ``systemctl show``;
then the time zone and which package owns each definition file. A report whose
end marker is missing raises: a list cut short must never read as jobs that went
away.
"""
from __future__ import annotations
import re
from dataclasses import dataclass, field
from datetime import UTC, datetime
from shlex import quote
from napalm_device_types._crontab import JOB_MARKER, CronLine, parse_anacrontab, parse_crontab
from napalm_device_types.scheduled_jobs import DELETE, EDIT, job_digest
from napalm_device_types.terminal import strip_terminal_codes
_BEGIN = "SJ_BEGIN"
_END = "SJ_END"
SCHEDULE_HEADER = "# schedule: "
TIMER_PROPERTIES = (
"Id,Unit,Description,TimersCalendar,TimersMonotonic,NextElapseUSecRealtime,"
"LastTriggerUSec,UnitFileState,ActiveState,FragmentPath"
)
UNIT_PROPERTIES = "Id,ExecStart,User,Result,FragmentPath"
#: The names run-parts runs (Debian's ``--lsbsysinit``-less default).
_RUN_PARTS_NAME = re.compile(r"^[A-Za-z0-9_-]+$")
_SECTION = re.compile(r"^\[([a-z-]+)(?: (.*))?\]$")
#: Reads definition paths on stdin, prints ``<package>: <path>`` for each a package owns.
OWNERS_SH = (
"if command -v dpkg >/dev/null 2>&1; then xargs dpkg -S 2>/dev/null; "
"elif command -v rpm >/dev/null 2>&1; then while read -r p; do "
'o=$(rpm -qf --qf \'%{NAME}\' -- "$p" 2>/dev/null) && printf \'%s: %s\\n\' "$o" "$p"; done; '
"elif command -v apk >/dev/null 2>&1; then while read -r p; do "
'o=$(apk info -qW -- "$p" 2>/dev/null) && [ -n "$o" ] && printf \'%s: %s\\n\' "$o" "$p"; done; '
"else cat >/dev/null; fi"
)
#: Prints the host's time zone: an IANA name, or a path ending in ``zoneinfo/<name>``.
TIME_ZONE_SH = "cat /etc/timezone 2>/dev/null || readlink /etc/localtime 2>/dev/null"
@dataclass(frozen=True)
class Paths:
"""Where a host keeps its scheduled jobs."""
system_files: tuple[str, ...]
cron_d_dirs: tuple[str, ...]
spool_dirs: tuple[str, ...]
run_parts: tuple[tuple[str, str], ...] # (period, directory)
anacrontab: str
script_dir: str
systemd_marker: str = "/run/systemd/system"
owners_sh: str = OWNERS_SH
time_zone_sh: str = TIME_ZONE_SH
@dataclass
class Reading:
"""What one report said, beyond the jobs: what writes need to know."""
jobs: list[dict[str, object]] = field(default_factory=list)
time_zone: str | None = None
systemd: bool = False
#: per key: the unit a timer starts, the crontab line a cron job is
units: dict[str, str] = field(default_factory=dict)
lines: dict[str, CronLine] = field(default_factory=dict)
def _q(path: str) -> str:
return quote(path)
def _files(section: str, pattern: str) -> str:
return (
f'for f in {pattern}; do [ -f "$f" ] || continue; printf \'[%s %s]\\n\' {section} "$f"; '
# GNU sed keeps a missing last newline; without one of its own the next
# header would run on into the file's last line.
'sed \'s/^/|/\' "$f"; [ -n "$(tail -c 1 "$f")" ] && echo; P="$P\n$f"; done; '
)
def _run_parts(period: str, directory: str) -> str:
return (
f'for f in {_q(directory)}/*; do [ -f "$f" ] || continue; '
'[ -x "$f" ] && x=1 || x=0; c=$(cksum < "$f" | cut -d" " -f1); '
f'printf \'[run-parts %s %s %s %s]\\n\' {period} "$x" "$c" "$f"; P="$P\n$f"; done; '
)
def _timers(paths: Paths) -> str:
show = f"systemctl show --no-pager -p {TIMER_PROPERTIES}"
units = f"systemctl show --no-pager -p {UNIT_PROPERTIES}"
return (
f"if [ -d {_q(paths.systemd_marker)} ]; then echo '[systemd]'; "
"t=$(systemctl list-unit-files --type=timer --no-legend --no-pager 2>/dev/null "
"| awk '$1 !~ /@\\./ {print $1}'); "
'if [ -n "$t" ]; then '
f"s=$({show} --timestamp=unix $t 2>/dev/null || {show} $t 2>/dev/null); "
"echo '[timers]'; printf '%s\\n' \"$s\" | sed 's/^/|/'; "
"u=$(printf '%s\\n' \"$s\" | sed -n 's/^Unit=//p'); "
f"echo '[units]'; {units} $u 2>/dev/null | sed 's/^/|/'; "
"P=\"$P\n$(printf '%s\\n' \"$s\" | sed -n 's/^FragmentPath=//p')\"; fi; fi; "
)
def reading_command(paths: Paths) -> str:
"""One POSIX ``sh`` script that prints every scheduled job of the host; run it as root."""
parts = ["printf '%s%s\\n' SJ_ BEGIN; P=''; "]
parts += [_files("system", _q(f)) for f in paths.system_files]
parts += [_files("system", f"{_q(d)}/*") for d in paths.cron_d_dirs]
parts += [_files("user", f"{_q(d)}/*") for d in paths.spool_dirs]
parts.append(_files("anacron", _q(paths.anacrontab)))
parts += [_run_parts(period, d) for period, d in paths.run_parts]
parts.append(_files("script", f"{_q(paths.script_dir)}/*"))
parts.append(_timers(paths))
parts.append(
f"echo '[tz]'; {{ {paths.time_zone_sh}; }}; "
f"echo '[owners]'; printf '%s\\n' \"$P\" | sed '/^$/d' | {{ {paths.owners_sh}; }}; "
"printf '%s%s\\n' SJ_ END"
)
return "".join(parts)
# -- parse ------------------------------------------------------------------------------------
@dataclass
class _Section:
name: str
args: str
lines: list[str] = field(default_factory=list)
def _sections(output: str) -> list[_Section]:
text = strip_terminal_codes(output)
begin, end = text.rfind(_BEGIN), text.rfind(_END)
if begin < 0 or end < begin:
raise ValueError("the scheduled-jobs report is cut short (no end marker)")
sections: list[_Section] = []
for line in text[begin + len(_BEGIN) : end].splitlines():
header = _SECTION.match(line)
if header:
sections.append(_Section(header[1], header[2] or ""))
elif sections:
sections[-1].lines.append(line[1:] if line.startswith("|") else line)
return sections
def _owned(lines: list[str]) -> set[str]:
owned: set[str] = set()
for line in lines:
package, sep, path = line.partition(": ")
if sep and path.startswith("/") and not package.startswith("diversion"):
owned.add(path.strip())
return owned
def _time_zone(lines: list[str]) -> str | None:
value = next((ln.strip() for ln in lines if ln.strip()), "")
if "zoneinfo/" in value:
value = value.split("zoneinfo/", 1)[1]
return value or None
def _blocks(lines: list[str]) -> list[dict[str, list[str]]]:
blocks: list[dict[str, list[str]]] = [{}]
for line in lines:
if not line.strip():
blocks.append({})
continue
key, sep, value = line.partition("=")
if sep:
blocks[-1].setdefault(key, []).append(value)
return [b for b in blocks if b.get("Id")]
def _one(block: dict[str, list[str]], key: str) -> str:
return (block.get(key) or [""])[0].strip()
def _timestamp(value: str) -> str | None:
"""systemd's ``@<seconds>`` (or a UTC time) as ISO 8601; None when it never was."""
value = value.strip()
if value.startswith("@") and value[1:].isdigit() and int(value[1:]) > 0:
return datetime.fromtimestamp(int(value[1:]), UTC).isoformat()
match = re.search(r"(\d{4}-\d\d-\d\d \d\d:\d\d:\d\d) UTC$", value)
if match:
return datetime.strptime(match[1], "%Y-%m-%d %H:%M:%S").replace(tzinfo=UTC).isoformat()
return None
def _timer_entries(block: dict[str, list[str]], key: str) -> list[str]:
"""``{ OnCalendar=daily ; next_elapse=… }`` → ``OnCalendar=daily``, one per entry.
``systemctl show`` names a monotonic setting by its property (``OnUnitActiveUSec``);
it is given as the unit file says it (``OnUnitActiveSec``).
"""
entries = (v.strip(" {}").split(" ; ")[0].strip() for v in block.get(key, []) if v.strip())
return [re.sub(r"^(On\w+)USec=", r"\1Sec=", e) for e in entries]
def _argv(exec_start: list[str]) -> str:
commands = []
for value in exec_start:
match = re.search(r"argv\[\]=(.*?) ; ", value)
if match:
commands.append(match[1].strip())
return "; ".join(commands)
@dataclass
class _Script:
name: str
description: str
schedule: str
command: str
def _script(path: str, lines: list[str]) -> _Script | None:
"""An application's job script: its marker, schedule and command; None for any other file."""
marker = next((ln for ln in lines if ln.startswith(JOB_MARKER)), None)
schedule = next((ln for ln in lines if ln.startswith(SCHEDULE_HEADER)), None)
if marker is None or schedule is None:
return None
name, _, description = marker[len(JOB_MARKER) :].partition(" ")
body = [ln for ln in lines if ln not in (marker, schedule) and not ln.startswith("#!")]
return _Script(
name, description.strip(), schedule[len(SCHEDULE_HEADER) :], "\n".join(body).strip()
)
def _job(
*,
key: str,
kind: str,
source: str,
run_as: str,
schedule: str,
schedule_kind: str,
command: str,
enabled: bool,
managed_by: str,
actions: list[str],
content: str = "",
**extra: object,
) -> dict[str, object]:
job: dict[str, object] = {
"key": key,
"kind": kind,
"source": source,
"run_as": run_as,
"schedule": schedule,
"schedule_kind": schedule_kind,
"command": command,
"enabled": enabled,
"managed_by": managed_by,
"actions": actions,
}
job["digest"] = job_digest(job, content=content)
job.update({k: v for k, v in extra.items() if v not in (None, "")})
return job
_SWITCH = ["enable", "disable", "run"]
_OWN = [*_SWITCH, EDIT, DELETE]
def _cron_jobs(section: _Section, owned: set[str], reading: Reading) -> None:
path = section.args
text = "\n".join(section.lines)
if section.name == "anacron":
found = parse_anacrontab(text)
else:
user = path.rsplit("/", 1)[-1]
found = parse_crontab(path, text, system=section.name == "system", user=user)
kind = "anacron" if section.name == "anacron" else "cron"
for line in found:
own = bool(line.name)
reading.lines[line.key] = line
reading.jobs.append(
_job(
key=line.key,
kind=kind,
source=path,
run_as=line.run_as,
schedule=line.schedule,
schedule_kind="period" if kind == "anacron" else "cron",
command=line.command,
enabled=line.enabled,
managed_by="application" if own else ("package" if path in owned else "admin"),
actions=_OWN if own else _SWITCH,
name=line.name,
description=line.description,
)
)
def _run_parts_job(section: _Section, owned: set[str], reading: Reading) -> None:
period, executable, checksum, path = section.args.split(" ", 3)
if not _RUN_PARTS_NAME.match(path.rsplit("/", 1)[-1]):
return
reading.jobs.append(
_job(
key=f"run-parts:{path}",
kind="run_parts",
source=path,
run_as="root",
schedule=period,
schedule_kind="period",
command=path,
enabled=executable == "1",
managed_by="package" if path in owned else "admin",
actions=_SWITCH,
content=checksum,
)
)
def _timer_job(
timer: dict[str, list[str]],
unit: dict[str, list[str]],
scripts: dict[str, _Script],
owned: set[str],
) -> dict[str, object]:
fragment = _one(timer, "FragmentPath")
calendar = _timer_entries(timer, "TimersCalendar")
schedule = "; ".join(calendar + _timer_entries(timer, "TimersMonotonic"))
command = _argv(unit.get("ExecStart", []))
script = scripts.get(command.split(" ", 1)[0])
last_run = _timestamp(_one(timer, "LastTriggerUSec"))
switchable = _one(timer, "UnitFileState") not in ("static", "generated", "transient")
common = {
"kind": "systemd_timer",
"source": fragment,
"run_as": _one(unit, "User") or "root",
"enabled": _one(timer, "ActiveState") == "active",
"last_run": last_run,
"last_result": _one(unit, "Result") or None if last_run else None,
"next_run": _timestamp(_one(timer, "NextElapseUSecRealtime")),
}
if script is not None:
return _job(
key=f"job:{script.name}",
schedule=script.schedule,
schedule_kind="cron",
command=script.command,
managed_by="application",
actions=_OWN,
name=script.name,
description=script.description,
**common, # type: ignore[arg-type]
)
packaged = fragment in owned or fragment.startswith(("/usr/lib/", "/lib/"))
return _job(
key=f"timer:{_one(timer, 'Id')}",
schedule=schedule,
schedule_kind="calendar" if calendar else "monotonic",
command=command,
managed_by="package" if packaged else "admin",
actions=_SWITCH if switchable else ["run"],
description=_one(timer, "Description"),
**common, # type: ignore[arg-type]
)
def _timer_jobs(
sections: dict[str, _Section], scripts: dict[str, _Script], owned: set[str], reading: Reading
) -> None:
timers = _blocks(sections["timers"].lines) if "timers" in sections else []
units = (
{_one(b, "Id"): b for b in _blocks(sections["units"].lines)} if "units" in sections else {}
)
for timer in timers:
unit = units.get(_one(timer, "Unit"), {})
job = _timer_job(timer, unit, scripts, owned)
reading.units[str(job["key"])] = _one(timer, "Unit")
reading.jobs.append(job)
def parse_reading(output: str) -> Reading:
"""The jobs, time zone and write context of a report from :func:`reading_command`.
:raises ValueError: when the report is cut short.
"""
sections = _sections(output)
named = {s.name: s for s in sections if s.name in ("timers", "units", "tz", "owners")}
owned = _owned(named["owners"].lines) if "owners" in named else set()
reading = Reading(
time_zone=_time_zone(named["tz"].lines) if "tz" in named else None,
systemd=any(s.name == "systemd" for s in sections),
)
for section in sections:
if section.name in ("system", "user", "anacron"):
_cron_jobs(section, owned, reading)
elif section.name == "run-parts":
_run_parts_job(section, owned, reading)
_timer_jobs(named, _scripts(sections), owned, reading)
return reading
def _scripts(sections: list[_Section]) -> dict[str, _Script]:
"""An application's job scripts, by path."""
scripts = ((s.args, _script(s.args, s.lines)) for s in sections if s.name == "script")
return {path: script for path, script in scripts if script is not None}
+369
View File
@@ -0,0 +1,369 @@
"""The reading behind :class:`~napalm_device_types.time_sync_posix.PosixTimeSyncMixin`.
One command, as the login user, prints framed sections (``@@ts <name> ...``):
* ``timedatectl`` -- ``timedatectl show``: the zone, and whether the kernel clock is
synchronised;
* ``zone`` -- ``/etc/timezone`` or the target of ``/etc/localtime``, for a host
without timedatectl;
* ``active`` -- one ``<unit> <daemon> <state>`` line per time sync service;
* ``timesync`` -- ``timedatectl show-timesync``: systemd-timesyncd's servers;
* ``chronyc`` -- ``chronyc -n sources``: whether chronyd has selected a source;
* ``chrony <line> <kind> <path>`` -- chrony's main configuration (line 0, kind
``main``), then each file an ``include``, ``confdir`` or ``sourcedir`` on that
line of it reads;
* ``ntpd <path>``, ``openntpd <path>`` -- their configuration;
* ``end`` -- the reading is complete.
What DHCP brought is read apart from the configured servers: chrony's files in a
DHCP client's ``sourcedir`` (:data:`CHRONY_DHCP_DIRS`), systemd-timesyncd's
``LinkNTPServers``.
"""
from __future__ import annotations
import posixpath
import re
import shlex
from dataclasses import dataclass, field
from napalm_device_types.time_sync import TimeSyncDict
_MARK = "@@ts "
#: ``<unit> <daemon>``: the services that keep a host's clock; the first active one counts.
TIME_SYNC_UNITS: tuple[tuple[str, str], ...] = (
("chrony", "chrony"),
("chronyd", "chrony"),
("systemd-timesyncd", "systemd-timesyncd"),
("ntpsec", "ntpd"),
("ntp", "ntpd"),
("ntpd", "ntpd"),
("openntpd", "openntpd"),
)
#: The ``sourcedir`` a DHCP client writes chrony's sources to, one file per interface
#: and lease: the NetworkManager dispatcher and dhclient hook of Debian, Ubuntu and
#: Red Hat. What chrony reads from there is DHCP's (option 42), not configuration.
CHRONY_DHCP_DIRS: tuple[str, ...] = ("/run/chrony-dhcp", "/var/run/chrony-dhcp")
#: A source line of chrony's or ntpd's configuration: the directive and what follows.
SOURCE_LINE = re.compile(r"\s*(server|pool|peer)\s+(.*)")
_OPENNTPD_SOURCE = re.compile(r"\s*(servers?)\s+(\S+)")
_ZONE_NAME = re.compile(r"[A-Za-z0-9_+-]+(?:/[A-Za-z0-9_+-]+)*")
#: ntpd's reference clocks are addresses in 127.127.0.0/16, not servers.
_REFCLOCK = "127.127."
# The files a chrony configuration reads, each after a marker with the line that names it.
_CHRONY_FOLLOW = """\
awk '$1=="include"||$1=="confdir"||$1=="sourcedir"{for(i=2;i<=NF;i++)print NR, $1, $i}' "$c" |
while read -r n k p; do
case $k in include) set -- $p ;; confdir) set -- "$p"/*.conf ;; *) set -- "$p"/*.sources ;; esac
for f in "$@"; do
[ -f "$f" ] && { printf '@@ts chrony %s %s %s\\n' "$n" "$k" "$f"; cat -- "$f"; echo; }
done
done"""
@dataclass(frozen=True)
class ReadPaths:
"""Where a host keeps what the reading reads."""
units: tuple[tuple[str, str], ...]
chrony_configs: tuple[str, ...]
ntpd_configs: tuple[str, ...]
openntpd_configs: tuple[str, ...]
time_zone_sh: str
@dataclass
class ConfigFile:
"""One configuration file as the reading printed it."""
path: str
kind: str # "main", "include", "confdir", "sourcedir"; "ntpd", "openntpd"
at: int # the line of the main configuration that names it; 0 for that file itself
lines: list[str] = field(default_factory=list)
@dataclass
class Reading:
"""Everything one reading said."""
timedatectl: dict[str, str] = field(default_factory=dict)
zone_files: list[str] = field(default_factory=list)
active: list[tuple[str, str]] = field(default_factory=list) # (unit, daemon), active ones
timesync: dict[str, str] = field(default_factory=dict)
chronyc: str = ""
chrony: list[ConfigFile] = field(default_factory=list)
ntpd: ConfigFile | None = None
openntpd: ConfigFile | None = None
@property
def daemon(self) -> str | None:
return self.active[0][1] if self.active else None
@property
def unit(self) -> str | None:
return self.active[0][0] if self.active else None
def _first_file(kind: str, paths: tuple[str, ...]) -> str:
candidates = " ".join(shlex.quote(p) for p in paths)
return (
f'for c in {candidates}; do [ -f "$c" ] && '
f'{{ printf \'@@ts {kind} %s\\n\' "$c"; cat -- "$c"; echo; break; }}; done'
)
def reading_command(paths: ReadPaths) -> str:
"""The command that prints the reading; ``sh`` runs it, whatever the login shell."""
active = "\n".join(
f"printf '%s %s %s\\n' {shlex.quote(unit)} {shlex.quote(daemon)} "
f'"$(systemctl is-active {shlex.quote(unit + ".service")} 2>/dev/null)"'
for unit, daemon in paths.units
)
chrony = " ".join(shlex.quote(p) for p in paths.chrony_configs)
script = "\n".join(
[
"echo '@@ts timedatectl'",
"timedatectl show 2>/dev/null",
"echo '@@ts zone'",
f"{{ {paths.time_zone_sh}; }}",
"echo '@@ts active'",
active,
"echo '@@ts timesync'",
"timedatectl show-timesync 2>/dev/null",
"echo '@@ts chronyc'",
"chronyc -n sources 2>/dev/null",
f'for c in {chrony}; do [ -f "$c" ] || continue',
'printf \'@@ts chrony 0 main %s\\n\' "$c"; cat -- "$c"; echo',
_CHRONY_FOLLOW,
"break; done",
_first_file("ntpd", paths.ntpd_configs),
_first_file("openntpd", paths.openntpd_configs),
"echo '@@ts end'",
]
)
return f"sh -c {shlex.quote(script)}"
def parse_key_values(text: str) -> dict[str, str]:
"""``Key=value`` lines (``timedatectl show``, ``show-timesync``)."""
values: dict[str, str] = {}
for line in text.splitlines():
key, sep, value = line.partition("=")
if sep and key.strip():
values[key.strip()] = value.strip()
return values
def _sections(text: str) -> list[tuple[list[str], list[str]]]:
sections: list[tuple[list[str], list[str]]] = []
for line in text.splitlines():
if line.startswith(_MARK):
sections.append((line[len(_MARK) :].split(maxsplit=3), []))
elif sections:
sections[-1][1].append(line)
return sections
def _active(lines: list[str]) -> list[tuple[str, str]]:
rows = [line.split() for line in lines]
return [(r[0], r[1]) for r in rows if len(r) == 3 and r[2] == "active"]
def _take(reading: Reading, header: list[str], lines: list[str]) -> None:
name, text = header[0], "\n".join(lines)
if name == "chrony" and len(header) == 4:
reading.chrony.append(ConfigFile(header[3], header[2], int(header[1]), lines))
elif name in ("ntpd", "openntpd") and len(header) == 2:
setattr(reading, name, ConfigFile(header[1], name, 0, lines))
else:
simple = {
"timedatectl": ("timedatectl", parse_key_values(text)),
"zone": ("zone_files", lines),
"active": ("active", _active(lines)),
"timesync": ("timesync", parse_key_values(text)),
"chronyc": ("chronyc", text),
}
if name in simple:
setattr(reading, *simple[name])
def parse_reading(text: str) -> Reading:
"""The reading's sections. :raises ValueError: for a reading cut short."""
sections = _sections(text)
if not sections or sections[-1][0] != ["end"]:
raise ValueError("the time sync reading was cut short")
reading = Reading()
for header, lines in sections:
_take(reading, header, lines)
return reading
def _source_name(rest: str) -> str | None:
"""The server in what follows ``server``/``pool``/``peer``: ntpd puts ``-4`` first."""
return next((word for word in rest.split() if not word.startswith("-")), None)
def in_dhcp_dir(f: ConfigFile, dhcp_dirs: tuple[str, ...]) -> bool:
"""Whether *f* is a sources file a DHCP client wrote to one of *dhcp_dirs*."""
if f.kind != "sourcedir":
return False
directory = posixpath.normpath(posixpath.dirname(f.path))
return directory in {posixpath.normpath(d) for d in dhcp_dirs}
def _named_on(files: list[ConfigFile], number: int) -> list[ConfigFile]:
"""The files the main configuration's line *number* reads."""
return [f for f in files if f.at == number and f.kind != "main"]
def _read_order(files: list[ConfigFile], dhcp_dirs: tuple[str, ...]) -> list[tuple[bool, str]]:
"""Every line chrony reads, in its order -- each file in place of the line naming
it -- and whether a DHCP client wrote it."""
main = next((f for f in files if f.kind == "main"), None)
if main is None:
return []
found: list[tuple[bool, str]] = []
for number, line in enumerate(main.lines, start=1):
found.append((False, line))
for included in _named_on(files, number):
dhcp = in_dhcp_dir(included, dhcp_dirs)
found.extend((dhcp, text) for text in included.lines)
return found
def chrony_servers(files: list[ConfigFile], dhcp_dirs: tuple[str, ...] = ()) -> list[str]:
"""chrony's configured sources in the order it reads them; not those in *dhcp_dirs*."""
return _sources([line for dhcp, line in _read_order(files, dhcp_dirs) if not dhcp])
def chrony_dhcp_servers(files: list[ConfigFile], dhcp_dirs: tuple[str, ...]) -> list[str]:
"""The sources chrony reads from the DHCP client's files in *dhcp_dirs*, in its order.
A ``sourcedir`` chrony does not read (commented out) prints no files: nothing then.
"""
return _sources([line for dhcp, line in _read_order(files, dhcp_dirs) if dhcp])
def _sources(lines: list[str]) -> list[str]:
names = []
for line in lines:
match = SOURCE_LINE.match(line)
if match and (name := _source_name(match[2])):
names.append(name)
return names
def ntpd_servers(lines: list[str]) -> list[str]:
"""ntpd's servers, pools and peers; reference clocks are not servers."""
return [name for name in _sources(lines) if not name.startswith(_REFCLOCK)]
def openntpd_servers(lines: list[str]) -> list[str]:
"""OpenNTPD's ``server`` and ``servers``; ``constraint`` and ``sensor`` are no sources."""
return [m[2] for line in lines if (m := _OPENNTPD_SOURCE.match(line))]
def _listed(values: dict[str, str], *keys: str) -> list[str]:
found: list[str] = []
for key in keys:
for server in values.get(key, "").split():
if server not in found:
found.append(server)
return found
def timesyncd_servers(values: dict[str, str]) -> list[str]:
"""systemd-timesyncd's configured servers, runtime then system; the fallback only
while it has no other server, a link's included. A link's servers are not among
them: :func:`timesyncd_link_servers`."""
configured = _listed(values, "RuntimeNTPServers", "SystemNTPServers")
if configured or timesyncd_link_servers(values):
return configured
return _listed(values, "FallbackNTPServers")
def timesyncd_link_servers(values: dict[str, str]) -> list[str]:
"""The servers network links gave systemd-timesyncd: systemd-networkd's, from a DHCP
lease (option 42) or a ``.network`` file's ``NTP=`` -- the reading cannot tell those
apart and counts them as DHCP's."""
return _listed(values, "LinkNTPServers")
def chrony_synchronized(sources: str) -> bool | None:
"""Whether chronyd has selected a source (``^*``); None for no answer from chronyc."""
rows = [line for line in sources.splitlines() if line[:1] in ("^", "=", "#")]
if not rows:
return None
return any(line[1:2] == "*" for line in rows)
def _zone_name(value: str) -> str | None:
value = value.strip()
if "zoneinfo/" in value:
value = value.split("zoneinfo/", 1)[1]
for prefix in ("posix/", "right/"):
value = value.removeprefix(prefix)
return value if _ZONE_NAME.fullmatch(value) else None
def _time_zone(reading: Reading) -> str | None:
named = reading.timedatectl.get("Timezone", "")
if named and named != "n/a":
return _zone_name(named)
return next((z for line in reading.zone_files if (z := _zone_name(line))), None)
def _servers(reading: Reading, dhcp_dirs: tuple[str, ...]) -> list[str]:
if reading.daemon == "chrony":
return chrony_servers(reading.chrony, dhcp_dirs)
if reading.daemon == "systemd-timesyncd":
return timesyncd_servers(reading.timesync)
if reading.daemon == "ntpd" and reading.ntpd is not None:
return ntpd_servers(reading.ntpd.lines)
if reading.daemon == "openntpd" and reading.openntpd is not None:
return openntpd_servers(reading.openntpd.lines)
return []
def _dhcp_servers(reading: Reading, dhcp_dirs: tuple[str, ...]) -> list[str] | None:
"""What DHCP brought: chrony's and timesyncd's; None where ntpd or OpenNTPD runs,
whose DHCP hooks keep a configuration of their own that is not read."""
if reading.daemon == "chrony":
return chrony_dhcp_servers(reading.chrony, dhcp_dirs)
if reading.daemon == "systemd-timesyncd":
return timesyncd_link_servers(reading.timesync)
if reading.daemon is None:
return []
return None
def _synchronized(reading: Reading) -> bool | None:
if reading.daemon == "chrony":
chrony = chrony_synchronized(reading.chronyc)
if chrony is not None:
return chrony
return {"yes": True, "no": False}.get(reading.timedatectl.get("NTPSynchronized", ""))
def time_sync_of(reading: Reading, dhcp_dirs: tuple[str, ...] = CHRONY_DHCP_DIRS) -> TimeSyncDict:
"""What a reading says, as ``get_time_sync`` returns it; *dhcp_dirs* are the DHCP
client's ``sourcedir`` of chrony."""
return {
"ntp_servers": _servers(reading, dhcp_dirs),
"timezone": _time_zone(reading),
"synchronized": _synchronized(reading),
"daemon": reading.daemon,
"dhcp_ntp_servers": _dhcp_servers(reading, dhcp_dirs),
}
def parse_time_sync(text: str, dhcp_dirs: tuple[str, ...] = CHRONY_DHCP_DIRS) -> TimeSyncDict:
"""The reading's output as ``get_time_sync`` returns it.
:raises ValueError: for a reading cut short.
"""
return time_sync_of(parse_reading(text), dhcp_dirs)
+254 -47
View File
@@ -12,12 +12,18 @@ Debian installer writes its proxy into ``apt.conf``, Ubuntu's into
(NetOrk/netork#1397). ``DIRECT`` for the proxy, or for one host
(``Acquire::http::Proxy::<host>``), means no proxy there.
**https** goes through the http proxy too. apt's https method asks
``Acquire::https::Proxy`` first and ``Acquire::http::Proxy`` after it, for one
host and then for all (NetOrk/netork#1811). An http proxy alone therefore takes
every https source as well, and a cache that does not tunnel them answers 403.
**Where it is set** is every line of apt's files that sets the http proxy, in the
order apt reads them. A line of the form ``Acquire::http::Proxy "…";`` can be
commented out; a setting inside a block (``Acquire::http { Proxy "…"; };``) is
named but left alone. apt skips a file in ``apt.conf.d`` whose name has other
characters than letters, digits, ``_``, ``-`` and ``.``, or an extension other
than ``.conf`` (``00proxy.bak``, ``*.dpkg-old``); so is it here.
order apt reads them; the https proxy's lines are named apart. A line of the form
``Acquire::http::Proxy "…";`` can be commented out; a setting inside a block
(``Acquire::http { Proxy "…"; };``) is named but left alone. apt skips a file in
``apt.conf.d`` whose name has other characters than letters, digits, ``_``, ``-``
and ``.``, or an extension other than ``.conf`` (``00proxy.bak``, ``*.dpkg-old``);
so is it here.
**Whether a proxy answers** is a fetch with apt's own downloader,
``apt-helper download-file``, which is on every apt host and needs no root:
@@ -29,13 +35,15 @@ page is the right one, the caller says (``expect``).
out every other line that sets a different one, reads the result back from
``apt-config``, and fetches the ``InRelease`` of one http source through it. The
last step is what proves it: apt reached a mirror through the proxy it is now set
to. The source chosen is one apt does not fetch ``DIRECT``.
to. The source chosen is one apt does not fetch ``DIRECT``. Then it fetches one
https source the same way; where the proxy refuses it, https goes ``DIRECT``.
"""
from __future__ import annotations
import re
import shlex
from collections.abc import Callable
from typing import TYPE_CHECKING
from urllib.parse import urlsplit
@@ -47,7 +55,7 @@ from napalm_device_types.terminal import strip_terminal_codes
class AptProxySettingDict(TypedDict):
"""One line of apt's configuration that sets the http proxy (``AptProxyMixin``)."""
"""One line of apt's configuration that sets a proxy (``AptProxyMixin``)."""
file: str
line: int # 1-based
@@ -68,6 +76,14 @@ class AptProxyReportDict(TypedDict):
auto_detect: str | None
#: Every line that sets it, in the order apt reads them; the last one wins.
settings: list[AptProxySettingDict]
#: The proxy apt takes for https: ``Acquire::https::Proxy``, else the http one;
#: None for none or ``DIRECT``.
https_proxy: str | None
#: The per-host proxies https takes: ``Acquire::https::Proxy::<host>``, else
#: ``Acquire::http::Proxy::<host>``.
https_per_host: dict[str, str]
#: Every line that sets ``Acquire::https::Proxy``, in the order apt reads them.
https_settings: list[AptProxySettingDict]
#: The base URIs of the host's sources over http and over https.
http_sources: list[str]
https_sources: list[str]
@@ -85,6 +101,15 @@ class ProxyFetchDict(TypedDict):
error: str
class HttpsSourceFetchDict(ProxyFetchDict):
"""One https source fetched the way apt fetches it (``check_https_source``)."""
#: What was fetched: the source's ``InRelease``.
url: str
#: The proxy apt took for it.
proxy: str
_BEGIN = "APROXY_BEGIN"
_END = "APROXY_END"
_APT_HELPER = "/usr/lib/apt/apt-helper"
@@ -98,7 +123,7 @@ PROXY_URL = re.compile(r"^https?://[A-Za-z0-9.\-\[\]:@%_~/]+$")
#: them early.
APT_PROXY_COMMAND = (
"( export LC_ALL=C; printf '%s%s\\n' APROXY_ BEGIN; "
"echo '[dump]'; apt-config dump 2>/dev/null | grep -i '^Acquire::http::Proxy'; "
"echo '[dump]'; apt-config dump 2>/dev/null | grep -i -E '^Acquire::https?::Proxy'; "
'eval "$(apt-config shell ap_parts Dir::Etc::parts/d ap_main Dir::Etc::main/f 2>/dev/null)"; '
"echo '[parts]'; for f in \"${ap_parts:-/etc/apt/apt.conf.d/}\"*; do "
'[ -f "$f" ] && grep -Hin proxy "$f"; done; '
@@ -108,12 +133,15 @@ APT_PROXY_COMMAND = (
"printf '%s%s\\n' APROXY_ END ) 2>/dev/null | cat"
)
_DUMP = re.compile(r'^Acquire::http::Proxy(::\S+|-Auto-Detect)?\s+"([^"]*)"', re.I)
_DUMP = re.compile(r'^Acquire::(https?)::Proxy(::\S+|-Auto-Detect)?\s+"([^"]*)"', re.I)
_GREP = re.compile(r"^(/[^:]+):(\d+):(.*)$")
_FLAT = re.compile(r'^\s*Acquire::http::Proxy\s+"([^"]*)"\s*;', re.I)
_NOT_HTTP_PROXY = re.compile(
r"^\s*(//|#)|Acquire::(https|ftp|socks)\b|::Proxy::|Proxy-Auto-Detect", re.I
)
_FLAT_HTTPS = re.compile(r'^\s*Acquire::https::Proxy\s+"([^"]*)"\s*;', re.I)
#: A line that sets the https proxy inside a block, all on the line itself.
_HTTPS_BLOCK = re.compile(r'^\s*Acquire::https\s*\{[^}]*\bProxy\s+"([^"]*)"', re.I)
_IN_BLOCK = re.compile(r'\bProxy\s+"([^"]*)"', re.I)
#: The names apt reads in ``apt.conf.d``: no extension, or ``.conf``.
_PART_NAME = re.compile(r"^(?:[A-Za-z0-9_\-]+|[A-Za-z0-9_.\-]+\.conf)$")
@@ -138,13 +166,22 @@ def _sections(output: str) -> dict[str, list[str]]:
return sections
def _setting(line: str, main: bool) -> AptProxySettingDict | None:
def _grep_line(line: str, main: bool) -> tuple[str, int, str] | None:
"""File, line number and text of one line grep printed; None for a file apt skips."""
found = _GREP.match(line)
if not found:
return None
path, number, text = found.groups()
if not main and not _PART_NAME.match(path.rsplit("/", 1)[-1]):
return None
return path, int(number), text
def _setting(line: str, main: bool) -> AptProxySettingDict | None:
found = _grep_line(line, main)
if not found:
return None
path, number, text = found
flat = _FLAT.match(text)
if flat:
return {"file": path, "line": int(number), "value": flat.group(1), "editable": True}
@@ -156,22 +193,81 @@ def _setting(line: str, main: bool) -> AptProxySettingDict | None:
return {"file": path, "line": int(number), "value": nested.group(1), "editable": False}
def _dump(lines: list[str]) -> tuple[str | None, dict[str, str], str | None]:
"""The proxy, the per-host proxies and the auto-detect program in ``apt-config dump``."""
proxy: str | None = None
per_host: dict[str, str] = {}
auto_detect: str | None = None
def _https_setting(line: str, main: bool) -> AptProxySettingDict | None:
found = _grep_line(line, main)
if not found:
return None
path, number, text = found
flat = _FLAT_HTTPS.match(text)
if flat:
return {"file": path, "line": number, "value": flat.group(1), "editable": True}
block = _HTTPS_BLOCK.match(text)
if block:
return {"file": path, "line": number, "value": block.group(1), "editable": False}
return None
class _Dump(TypedDict):
proxy: str | None
per_host: dict[str, str]
auto_detect: str | None
#: ``Acquire::https::Proxy`` as written; None when unset or empty.
https: str | None
https_per_host: dict[str, str]
def _dump(lines: list[str]) -> _Dump:
"""The proxies, the per-host proxies and the auto-detect program in ``apt-config dump``."""
dump: _Dump = {
"proxy": None,
"per_host": {},
"auto_detect": None,
"https": None,
"https_per_host": {},
}
for found in (_DUMP.match(line.strip()) for line in lines):
if not found:
continue
suffix, value = found.groups()
if not suffix:
proxy = None if value.lower() in _NO_PROXY else value
elif suffix.lower() == "-auto-detect":
auto_detect = value or None
else:
per_host[suffix[2:]] = value
return proxy, per_host, auto_detect
if found:
_take(dump, *found.groups())
return dump
def _take(dump: _Dump, scheme: str, suffix: str | None, value: str) -> None:
"""One line of the dump into *dump*; a later one of the same key wins, as in apt."""
https = scheme.lower() == "https"
if not suffix:
_take_for_all(dump, https, value)
elif suffix.lower() == "-auto-detect":
if not https:
dump["auto_detect"] = value or None
elif value:
dump["https_per_host" if https else "per_host"][suffix[2:]] = value
def _take_for_all(dump: _Dump, https: bool, value: str) -> None:
"""The proxy for every host: https keeps an empty value as unset, apt's fallback."""
if https:
dump["https"] = value or None
else:
dump["proxy"] = None if value.lower() in _NO_PROXY else value
def _https_proxy(dump: _Dump) -> str | None:
"""apt's https method takes its own setting and falls back to the http one."""
if dump["https"] is None:
return dump["proxy"]
return None if dump["https"].lower() in _NO_PROXY else dump["https"]
def https_proxy_for(report: AptProxyReportDict, host: str) -> str | None:
"""The proxy apt takes for an https source on *host*; None for ``DIRECT`` or none.
The host's own setting first, https before http, then the one for all hosts.
"""
for per_host in (report["https_per_host"], report["per_host"]):
value = next((v for h, v in per_host.items() if h.lower() == host.lower()), None)
if value:
return None if value.lower() in _NO_PROXY else value
return report["https_proxy"]
def parse_apt_proxy(output: str) -> AptProxyReportDict:
@@ -180,24 +276,36 @@ def parse_apt_proxy(output: str) -> AptProxyReportDict:
:raises ValueError: if the output carried no intact report.
"""
sections = _sections(output)
proxy, per_host, auto_detect = _dump(sections.get("dump", []))
settings = [
s
dump = _dump(sections.get("dump", []))
lines = [
(main, line)
for main, key in ((False, "parts"), (True, "main"))
for s in (_setting(line, main) for line in sections.get(key, []))
if s
for line in sections.get(key, [])
]
sources = [line.strip() for line in sections.get("sources", [])]
return {
"proxy": proxy,
"per_host": per_host,
"auto_detect": auto_detect,
"settings": settings,
"http_sources": [s for s in sources if s.startswith("http://")],
"https_sources": [s for s in sources if s.startswith("https://")],
"proxy": dump["proxy"],
"per_host": dump["per_host"],
"auto_detect": dump["auto_detect"],
"settings": _settings(lines, _setting),
"https_proxy": _https_proxy(dump),
"https_per_host": {**dump["per_host"], **dump["https_per_host"]},
"https_settings": _settings(lines, _https_setting),
"http_sources": _of_scheme(sources, "http"),
"https_sources": _of_scheme(sources, "https"),
}
def _settings(
lines: list[tuple[bool, str]], parse: Callable[[str, bool], AptProxySettingDict | None]
) -> list[AptProxySettingDict]:
return [s for s in (parse(line, main) for main, line in lines) if s]
def _of_scheme(sources: list[str], scheme: str) -> list[str]:
return [s for s in sources if s.startswith(f"{scheme}://")]
_DEFAULT_PORT = {"http": 80, "https": 443}
@@ -225,14 +333,17 @@ def apt_fetch_command(url: str, *, direct: bool = False, expect: str = "") -> st
*direct* goes past any proxy apt is set to, to *url*'s host itself; without it
the fetch takes apt's own way. *expect* is text the page should contain.
:raises ValueError: if *url* is not an http URL a shell word can carry.
:raises ValueError: if *url* is not an http(s) URL a shell word can carry.
"""
if not PROXY_URL.match(url) or not url.startswith("http://"):
if not PROXY_URL.match(url):
raise ValueError(f"not an http URL: {url!r}")
options = ["-o Acquire::Retries=0", "-o Acquire::http::Timeout=10"]
host = urlsplit(url).hostname or ""
if direct and host and ":" not in host:
# *host* comes from a URL PROXY_URL let through: no quote in it.
# *host* comes from a URL PROXY_URL let through: no quote in it. For https
# the host's https setting comes first, so both are set.
if url.startswith("https://"):
options.append(f"-o 'Acquire::https::Proxy::{host}=DIRECT'")
options.append(f"-o 'Acquire::http::Proxy::{host}=DIRECT'")
match = f"echo '[match]'; grep -c -F -- {shlex.quote(expect)} \"$t\"; " if expect else ""
return (
@@ -272,13 +383,16 @@ def parse_apt_fetch(output: str) -> ProxyFetchDict:
return {"reachable": answered, "expected": False if answered else None, "error": error}
#: sed's edit of one line: comment it out, if it still sets the http proxy.
_COMMENT_OUT = "-e '{line}{{/^[[:space:]]*Acquire::http::Proxy[[:space:]]/Is|^|// |}}'"
#: sed's edit of one line: comment it out, if it still sets the proxy of *scheme*.
_COMMENT_OUT = "-e '{line}{{/^[[:space:]]*Acquire::{scheme}::Proxy[[:space:]]/Is|^|// |}}'"
_SCHEMES = ("http", "https")
def comment_out_command(path: str, lines: list[int]) -> str:
"""The root command that comments out *lines* of *path* where they set the http proxy."""
edits = " ".join(_COMMENT_OUT.format(line=int(n)) for n in lines)
def comment_out_command(path: str, lines: list[int], scheme: str = "http") -> str:
"""The root command that comments out *lines* of *path* where they set *scheme*'s proxy."""
if scheme not in _SCHEMES:
raise ValueError(f"not a scheme apt proxies: {scheme!r}")
edits = " ".join(_COMMENT_OUT.format(line=int(n), scheme=scheme) for n in lines)
return f"sed -i {edits} {shlex.quote(path)}"
@@ -333,6 +447,8 @@ class AptProxyMixin(PackageProxyMixin):
* per_host ({host: proxy or "DIRECT"})
* auto_detect (string or None): ``Acquire::http::Proxy-Auto-Detect``
* settings (list of {file, line, value, editable}), in apt's reading order
* https_proxy (string or None): the proxy https takes, the http one if unset
* https_per_host ({host: proxy or "DIRECT"}), https_settings (as settings)
* http_sources, https_sources (lists of the sources' base URIs)
:raises ValueError: if the host's output carried no intact report.
@@ -360,8 +476,30 @@ class AptProxyMixin(PackageProxyMixin):
self._run_apt_command(apt_fetch_command(url, direct=True, expect=expect))
)
def check_https_source(self) -> HttpsSourceFetchDict | None:
"""Fetch one https source's ``InRelease`` the way apt does, through its proxy.
None when no https source goes through a proxy: there are none, or https
goes ``DIRECT``.
:raises ValueError: if the host's output carried no intact report.
:raises NotImplementedError: on a host without apt.
"""
if not self._is_apt_host():
raise NotImplementedError("not an apt host")
report = self._report()
found = _https_source(report, lambda proxy: proxy is not None)
if found is None:
return None
target, proxy = found
fetched = parse_apt_fetch(self._run_apt_command(apt_fetch_command(target)))
return {**fetched, "url": target, "proxy": proxy}
def configure_package_proxy(self, url: str) -> FixResultDict:
"""apt fetches through *url*, and is shown to; ``""`` removes :attr:`APT_PROXY_FILE`."""
"""apt fetches through *url*, and is shown to; ``""`` removes :attr:`APT_PROXY_FILE`.
https goes through *url* too, unless *url* refuses it: then ``DIRECT``.
"""
if not url:
return run_fix_steps([("configure", self._remove_package_proxy)])
return run_fix_steps(
@@ -370,6 +508,7 @@ class AptProxyMixin(PackageProxyMixin):
("conflicts", lambda: self._comment_out_other_proxies(url)),
("verify", lambda: self._verify_package_proxy(url)),
("fetch", lambda: self._fetch_through_package_proxy(url)),
("https", lambda: self._https_through_package_proxy(url)),
]
)
@@ -383,12 +522,15 @@ class AptProxyMixin(PackageProxyMixin):
f"rm -f {shlex.quote(self.APT_PROXY_FILE)}", "proxy removed"
)
def _write_package_proxy(self, url: str) -> FixStepOutcomeDict:
def _write_package_proxy(self, url: str, https_direct: bool = False) -> FixStepOutcomeDict:
if not self._is_apt_host():
return self._not_apt()
if not PROXY_URL.match(url):
raise ValueError(f"not a proxy URL: {url!r}")
return self._write_apt_file(self.APT_PROXY_FILE, f'Acquire::http::Proxy "{url}";\n')
content = f'Acquire::http::Proxy "{url}";\n'
if https_direct:
content += 'Acquire::https::Proxy "DIRECT";\n'
return self._write_apt_file(self.APT_PROXY_FILE, content)
def _report(self) -> AptProxyReportDict:
# The steps after "configure" run only once it passed, on an apt host.
@@ -442,16 +584,81 @@ class AptProxyMixin(PackageProxyMixin):
return {"ok": True, "output": f"fetched {target} through {url}"}
return {"ok": False, "output": f"{target}: {fetched['error']}"}
def _https_through_package_proxy(self, url: str) -> FixStepOutcomeDict:
"""An https source through *url*; where *url* refuses it, https goes ``DIRECT``."""
found = _https_source(self._report(), lambda proxy: same_proxy_url(proxy, url))
if found is None:
return {"ok": True, "output": f"no https source goes through {url}; not tried"}
target = found[0]
fetched = parse_apt_fetch(self._run_apt_command(apt_fetch_command(target)))
# An HTTP error came from the mirror, through the tunnel: the cache let it by.
if fetched["reachable"]:
return {"ok": True, "output": f"fetched {target} through {url}"}
refused = f"{url} refused https ({fetched['error']})"
outcome = self._https_direct(url)
if not outcome["ok"]:
return {"ok": False, "output": f"{refused}; {outcome['output']}"}
direct = parse_apt_fetch(self._run_apt_command(apt_fetch_command(target)))
if not direct["reachable"]:
return {"ok": False, "output": f"{refused}; direct too: {target}: {direct['error']}"}
named = f"; {outcome['output']}" if outcome["output"] else ""
return {
"ok": True,
"output": f"{refused}{named}; https sources go direct now: fetched {target}",
}
def _https_direct(self, url: str) -> FixStepOutcomeDict:
"""Write https ``DIRECT`` beside *url*, and comment out the later lines to *url*."""
written = self._write_package_proxy(url, https_direct=True)
if not written["ok"]:
return written
later = self._later_https_lines(self._report(), url)
for path, numbers in _editable_lines(later).items():
outcome = self._run_apt_root_command(
comment_out_command(path, numbers, scheme="https"), f"commented out in {path}"
)
if not outcome["ok"]:
return outcome
report = self._report()
if same_proxy_url(report["https_proxy"], url):
still = [f"{s['file']}:{s['line']}" for s in self._later_https_lines(report, url)]
where = f", set in {', '.join(still)}" if still else ""
return {"ok": False, "output": f"https still goes through {url}{where}"}
return {"ok": True, "output": "; ".join(map(_described, later))}
def _later_https_lines(self, report: AptProxyReportDict, url: str) -> list[AptProxySettingDict]:
"""The lines outside :attr:`APT_PROXY_FILE` that send https to *url*."""
return [
s
for s in report["https_settings"]
if s["file"] != self.APT_PROXY_FILE and same_proxy_url(s["value"], url)
]
def _https_source(
report: AptProxyReportDict, through: Callable[[str | None], bool]
) -> tuple[str, str] | None:
"""The ``InRelease`` of the first https source whose proxy *through* accepts, and that proxy."""
for source in report["https_sources"]:
if "/dists/" not in source:
continue
proxy = https_proxy_for(report, urlsplit(source).hostname or "")
if through(proxy):
return source.rstrip("/") + "/InRelease", proxy or ""
return None
__all__ = [
"APT_PROXY_COMMAND",
"AptProxyMixin",
"AptProxyReportDict",
"AptProxySettingDict",
"HttpsSourceFetchDict",
"ProxyFetchDict",
"PROXY_URL",
"apt_fetch_command",
"comment_out_command",
"https_proxy_for",
"parse_apt_fetch",
"parse_apt_proxy",
"same_proxy_url",
+13
View File
@@ -0,0 +1,13 @@
"""The ``age`` of an entry in NAPALM's neighbour tables when the device does not say.
NAPALM requires an ``age`` on every entry of ``get_arp_table`` and
``get_ipv6_neighbors_table``: seconds since the device last confirmed that
neighbour. Many devices do not report one. ``0.0`` means "confirmed just now"
and must never stand in for "not reported": a caller ranking sightings by
freshness takes it at its word (NetOrk/netork#1327). An expiry countdown is not
an age either. The value is NAPALM's own; its IOS driver reports ``-1.0``.
"""
from typing import Final
ARP_AGE_UNKNOWN: Final[float] = -1.0
+238
View File
@@ -0,0 +1,238 @@
"""
Abstract base class for backup server drivers.
Usage::
from napalm_device_types import BackupServerDriver, StorageDriver
class ProxmoxBackupDriver(BackupServerDriver, StorageDriver):
def get_backup_datastores(self):
...
"""
from typing import TYPE_CHECKING
from napalm_device_types.base import DeviceTypeDriver
if TYPE_CHECKING: # pragma: no cover - for type checkers only
from napalm_device_types.backups import (
BackupDatastoreDict,
BackupGroupDict,
BackupJobKind,
BackupRemoteDict,
BackupServerJobDict,
BackupSnapshotDict,
BackupSnapshotRefDict,
BackupTaskDict,
BackupTaskStatusDict,
)
class BackupServerDriver(DeviceTypeDriver):
"""
Abstract intermediate driver for backup servers (e.g. Proxmox Backup
Server): the device hypervisors and hosts write their backups to.
A backup server keeps its backups in datastores, one group per source (a
VM, a container, a host) and one snapshot per backup run, and keeps them
healthy with scheduled jobs: garbage collection, verify, prune, sync with
other servers and tape. None of that is a storage pool, a volume or a
replication job, so it is not modelled through ``StorageDriver``. A server
that also manages its own disks and pools lists ``StorageDriver`` as a
second role, after this one.
The actions start a task on the server and return at once with its id;
:meth:`get_backup_task` follows it.
"""
#: Stable key netOrk exposes as ``device_class``. The order in which a
#: driver lists its role bases is the ranking; see
#: :func:`napalm_device_types.roles.primary_role_of`.
ROLE: str = "backup_server"
TYPE_LABEL: str = "Backup Server"
if TYPE_CHECKING:
# ------------------------------------------------------------------
# Read
# ------------------------------------------------------------------
def get_backup_datastores(self) -> list[BackupDatastoreDict]:
"""
Returns the server's datastores with their usage.
Example::
[
{
"name": "store1",
"path": "/mnt/datastore/store1",
"backend": "local",
"total": 4000787030016,
"used": 1529008357376,
"available": 2471778672640,
"estimated_full": 1798761600,
"maintenance": None,
"comment": None,
},
]
:raises Exception: when the server could not be read; never an
empty list for a failed read.
"""
...
def get_backup_groups(self, datastore: str = "") -> list[BackupGroupDict]:
"""
Returns the backup groups of every namespace of *datastore*, or of
every datastore when it is empty, each with its newest snapshot
only. :meth:`get_backup_snapshots` lists all of a group's.
Example::
[
{
"datastore": "store1",
"namespace": "",
"type": "vm",
"id": "101",
"owner": "backup@pbs",
"count": 14,
"newest": {
"time": 1793660400,
"size": 34359738368,
"verify": "ok",
"protected": False,
"comment": "web-01",
},
"oldest_time": 1792537200,
"comment": None,
},
]
:raises Exception: when the server could not be read.
"""
...
def get_backup_snapshots(
self, datastore: str, namespace: str, group_type: str, group_id: str
) -> list[BackupSnapshotDict]:
"""
Returns every snapshot of one group, newest first.
:param namespace: ``""`` for the datastore's root.
:raises ValueError: when the datastore or the group does not exist.
"""
...
def get_backup_jobs(self) -> list[BackupServerJobDict]:
"""
Returns the server's scheduled jobs of every kind, garbage
collection included (one per datastore, ``id`` = the datastore).
Example::
[
{
"id": "store1",
"kind": "gc",
"datastore": "store1",
"namespace": None,
"schedule": "daily",
"enabled": True,
"last_run": 1793664000,
"last_state": "ok",
"last_error": None,
"next_run": 1793750400,
"remote": None,
"direction": None,
"comment": None,
},
{
"id": "offsite",
"kind": "sync",
"datastore": "store1",
"namespace": "",
"schedule": "sat 03:00",
"enabled": True,
"last_run": 1793502000,
"last_state": "error",
"last_error": "connection refused",
"next_run": 1794106800,
"remote": "pbs-offsite",
"direction": "push",
"comment": None,
},
]
:raises Exception: when the server could not be read.
"""
...
def get_backup_remotes(self) -> list[BackupRemoteDict]:
"""
Returns the other backup servers this one syncs with, without their
secrets.
:raises Exception: when the server could not be read.
"""
...
# ------------------------------------------------------------------
# Actions
# ------------------------------------------------------------------
def start_garbage_collection(self, datastore: str) -> BackupTaskDict:
"""
Starts garbage collection on *datastore*.
:raises ValueError: when the datastore does not exist.
:raises RuntimeError: when the server refuses (e.g. one is running).
"""
...
def run_backup_job(self, kind: BackupJobKind, job_id: str) -> BackupTaskDict:
"""
Runs a scheduled job now, outside its schedule.
:param kind: a :data:`~napalm_device_types.backups.BackupJobKind`;
``"gc"`` runs the datastore named *job_id*'s garbage collection.
:raises ValueError: when there is no such job.
:raises RuntimeError: when the server refuses.
"""
...
def verify_backup_snapshot(self, snapshot: BackupSnapshotRefDict) -> BackupTaskDict:
"""
Verifies one snapshot now.
:raises ValueError: when the snapshot does not exist.
"""
...
def set_backup_snapshot_protected(
self, snapshot: BackupSnapshotRefDict, protected: bool
) -> None:
"""
Protects a snapshot from pruning and deletion, or lifts that.
:raises ValueError: when the snapshot does not exist.
"""
...
def delete_backup_snapshot(self, snapshot: BackupSnapshotRefDict) -> None:
"""
Deletes one snapshot. The data is gone once garbage collection has
run; there is no undo.
:raises ValueError: when the snapshot does not exist.
:raises RuntimeError: when it is protected.
"""
...
def get_backup_task(self, task: str) -> BackupTaskStatusDict:
"""
Returns the state and the log so far of a task an action started.
:raises ValueError: when the server does not know the task.
"""
...
+119
View File
@@ -0,0 +1,119 @@
"""The shapes of a backup server (``BackupServerDriver``).
Apart from ``models.py``, which is over the file size limit. A value the
platform does not report is None; it is never guessed.
"""
from typing import Literal, TypedDict, get_args
#: What a backup server's jobs do. Garbage collection is one of them: it has a
#: schedule and a last run like the others.
BackupJobKind = Literal["gc", "verify", "prune", "sync", "tape"]
BACKUP_JOB_KINDS: tuple[str, ...] = get_args(BackupJobKind)
#: How a job's or a task's run ended; "running" while it has not.
BackupRunState = Literal["ok", "warning", "error", "running"]
#: The outcome of the last verification of a snapshot.
BackupVerifyState = Literal["ok", "failed"]
class BackupDatastoreDict(TypedDict):
"""A datastore: where a backup server keeps its backups."""
name: str
path: str | None # local path; for an object-store backend the local cache, if any
backend: str # "local", "s3", "removable", ... as the platform names it
total: int | None # bytes
used: int | None # bytes
available: int | None # bytes
estimated_full: int | None # unix seconds the platform expects it to be full
maintenance: str | None # "read-only", "offline", ...; None when in service
comment: str | None
class BackupSnapshotDict(TypedDict):
"""One backup of a group: the state of its source at one time."""
time: int # unix seconds the backup was taken
size: int | None # bytes
verify: BackupVerifyState | None # None: never verified
protected: bool # exempt from pruning and deletion
comment: str | None
class BackupGroupDict(TypedDict):
"""The backups of one source (a VM, a container, a host) in one namespace."""
datastore: str
namespace: str # "" for the datastore's root
type: str # "vm", "ct", "host"
id: str # the guest's id ("101") or the host's name
owner: str | None # the account that writes the group's backups
count: int # snapshots in the group
newest: BackupSnapshotDict | None # None for a group without a snapshot
oldest_time: int | None # unix seconds of the oldest snapshot
comment: str | None
class BackupSnapshotRefDict(TypedDict):
"""Names one snapshot, for an action on it."""
datastore: str
namespace: str # "" for the datastore's root
group_type: str # BackupGroupDict.type
group_id: str # BackupGroupDict.id
time: int # BackupSnapshotDict.time
class BackupServerJobDict(TypedDict):
"""A scheduled job of a backup server."""
id: str # for "gc", the datastore's name
kind: BackupJobKind
datastore: str | None
namespace: str | None
schedule: str | None # as the platform writes it: "daily", "sat 02:00", "*:0/30"
enabled: bool
last_run: int | None # unix seconds the last run ended (or started, while running)
last_state: BackupRunState | None # None: never ran
last_error: str | None
next_run: int | None # unix seconds
remote: str | None # a sync job's BackupRemoteDict.name
direction: Literal["pull", "push"] | None # a sync job's
comment: str | None
class BackupRemoteDict(TypedDict):
"""Another backup server this one syncs with. Never its secret."""
name: str
host: str
port: int | None
fingerprint: str | None # of the remote's TLS certificate
auth_id: str | None # the account the sync logs in as
comment: str | None
class BackupTaskDict(TypedDict):
"""A task an action started on the server, to follow with ``get_backup_task``."""
task: str # the platform's task id
class BackupTaskStatusDict(TypedDict):
task: str
running: bool
state: BackupRunState | None # "running" while it runs
error: str | None
log: list[str] # the task's log lines so far
class BackupServerRefDict(TypedDict):
"""Which backup server and datastore a hypervisor's backup storage writes to."""
server: str # host name or address, as the hypervisor has it configured
port: int | None
datastore: str
namespace: str | None
fingerprint: str | None # of the server's TLS certificate, as the hypervisor pins it
+8 -1
View File
@@ -15,6 +15,7 @@ from napalm.base import NetworkDriver
from napalm_device_types.channel import CommandChannelMixin
from napalm_device_types.host_reboot import HostRebootMixin
from napalm_device_types.host_shutdown import HostShutdownMixin
from napalm_device_types.ping_sweep import PingSweepMixin
@@ -74,7 +75,13 @@ class PortSpec(NamedTuple):
mandatory: bool = False
class DeviceTypeDriver(PingSweepMixin, HostRebootMixin, CommandChannelMixin, NetworkDriver): # type: ignore[misc]
class DeviceTypeDriver(
PingSweepMixin,
HostRebootMixin,
HostShutdownMixin,
CommandChannelMixin,
NetworkDriver, # type: ignore[misc]
):
"""Common base for all netOrk device-type drivers.
Sits between napalm.base.NetworkDriver and the type-specific abstract
+104
View File
@@ -0,0 +1,104 @@
"""The shapes of a hypervisor's clusters (``HypervisorDriver.get_clusters()``).
Apart from ``models.py``, which is over the file size limit.
"""
from typing import NotRequired, TypedDict
from napalm_device_types.backups import BackupServerRefDict
class ClusterNodeDict(TypedDict):
"""One host of a cluster, as its cluster sees it. A value the platform does not
report is None; it is never guessed."""
name: str # the node's name in its cluster, as get_vms() reports it in ``node``
status: str # "online", "offline" or "unknown"
local: bool # the node this connection talks to; False on a manager (vCenter)
maintenance: bool | None # in maintenance mode
cpu_threads: int | None # logical CPUs
cpu_cores: int | None # physical cores
cpu_model: str | None
cpu_flags: list[str] | None # cpuinfo flags; None where the platform does not say
cpu_usage: float | None # 0.0–1.0
memory_total: int | None # bytes
memory_used: int | None # bytes
votes: NotRequired[int | None] # quorum votes of this node; None standalone/unreadable
class ClusterStorageDict(TypedDict):
"""A storage of a cluster and the nodes that see it."""
name: str # usable as create_vm_from_cloud_init(storage=...) on each of ``nodes``
type: str # backend type: "dir", "lvmthin", "zfspool", "nfs", "rbd", "vmfs", ...
shared: bool # the same storage on every node in ``nodes``, so a VM on it can move
content: list[str] # what it holds: "images", "iso", "snippets", ...
nodes: list[str] # the nodes it is available on
total: int | None # bytes; None where the platform does not report it
used: int | None # bytes
# A backup storage that writes to a backup server: which one. Left out for every
# other storage; None where the platform does not say.
backup_server: NotRequired[BackupServerRefDict | None]
class ClusterNetworkDict(TypedDict):
"""A network a VM's NIC may attach to, and the nodes that carry it."""
name: str # usable as NICConfigDict.bridge on each of ``nodes``
kind: str # "bridge", "vnet" or "portgroup", as in NetworkTargetDict
vlan_aware: bool # True if a NICConfigDict.vlan_tag may be set on top of it
fixed_vlan_tag: NotRequired[int | None] # vnet/portgroup: the VLAN baked into it
nodes: list[str]
class HAResourceDict(TypedDict):
"""A guest the cluster's HA manager looks after."""
vmid: int
kind: str # "qemu" or "lxc"
requested: str # state HA is asked to keep: "started", "stopped", "disabled", "ignored"
status: str | None # what the HA manager reports now: "started", "fence", "migrate", ...
node: str | None # the node the HA manager places it on now
group: NotRequired[str | None] # HA group (Proxmox VE 8)
rules: list[str] # ids of the HA rules that name it (Proxmox VE 9); [] where there are none
class BackupJobDict(TypedDict):
"""A scheduled backup job of the cluster."""
id: str
enabled: bool
schedule: str | None # as the platform writes it: "sat 02:00", "daily", "mon..fri 21:00"
storage: str | None
mode: str | None # "snapshot", "suspend", "stop"
all: bool # every guest, minus ``exclude``
vmids: list[int] # the guests it names; a pool job: the pool's members now; [] with all
exclude: list[int]
pool: str | None
node: str | None # the one node it runs on; None for every node
class BackupDict(TypedDict):
"""The backups of one guest on one storage."""
vmid: int
storage: str
time: int # unix seconds of the newest backup of this guest on this storage
size: int | None # bytes of that newest backup
count: int # backups of this guest on this storage
class ClusterDict(TypedDict):
"""A cluster the connection can see. A standalone host is a cluster of one,
with no name."""
id: str | None # stable across connections where the platform has one; None otherwise
name: str | None # the cluster's name; None for a standalone host
quorate: bool | None # None where the platform has no quorum (standalone, vSphere)
nodes: list[ClusterNodeDict]
storages: list[ClusterStorageDict]
networks: list[ClusterNetworkDict]
# None: could not be read; []: read, nothing there.
ha_resources: NotRequired[list[HAResourceDict] | None]
backup_jobs: NotRequired[list[BackupJobDict] | None]
backups: NotRequired[list[BackupDict] | None]
+140
View File
@@ -0,0 +1,140 @@
"""A device's own configuration history: list it, read a revision, revert to one.
Some devices keep their earlier configurations themselves: a copy of the
configuration for every save, which the device can go back to. An application
restores a configuration there by reverting to the revision that holds it -- on a
device whose API takes no uploaded configuration (OPNsense), the only way
(NetOrk/netork#2511). :class:`ConfigRevisionsMixin` is the contract:
* ``get_config_revisions()`` and ``get_config_revision(revision_id)`` -- declared
only, so ``callable(getattr(driver, "get_config_revisions", None))`` is the
truthful answer to "does this device keep a history";
* ``revert_config_revision(revision_id)`` -- a template: it refuses an id that is not
in the history, reads the revision, lets the driver revert
(``_revert_to_config_revision``), and reads the configuration in place afterwards
to check that it holds the revision's. A device may answer that it reverted when it
did not; the check does not take its word;
* ``config_revision_content(text)`` -- what of a revision is its configuration. Every
save may stamp a revision anew (OPNsense writes who saved it and when into
config.xml), so two revisions with the same configuration differ as text. A driver
whose device does that strips the stamp here; by default the whole text counts. An
application that looks for the revision holding a configuration compares through
it too.
A driver mixes it in when its device keeps a history; no role base carries it. One
that can list the history but not revert to it sets ``revert_config_revision`` to
``None``.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, TypedDict
class ConfigRevisionDict(TypedDict):
"""One revision of the device's configuration (``get_config_revisions``)."""
id: str # what get_config_revision and revert_config_revision take, as the device names it
time: str | None # when it was saved, UTC ISO 8601; None when the device does not say
description: str | None # what made the change, as the device records it; None: none
user: str | None # who saved it, as the device names them; None when it names nobody
size: int | None # its size in bytes; None when not given
class UnknownConfigRevision(ValueError):
"""The device's configuration history has no revision of that id."""
def __init__(self, revision_id: object) -> None:
super().__init__(f"no configuration revision {revision_id!r} in the device's history")
class ConfigRevertFailed(RuntimeError):
"""The device refused the revert, or its configuration afterwards is not the revision's."""
class ConfigRevisionsMixin:
"""List, read and revert to the device's configuration revisions; see the module."""
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
def get_config_revisions(self) -> list[ConfigRevisionDict]:
"""The device's configuration history, newest first.
The first revision is the configuration in place now.
Example::
[
{
"id": "config-1794603885.1239.xml",
"time": "2026-11-13T21:04:45+00:00",
"description": "/api/firewall/filter/apply made changes",
"user": "apiuser@192.0.2.10",
"size": 58213,
},
]
:raises Exception: when the history could not be read; never an empty
list for a failed read.
"""
...
def get_config_revision(self, revision_id: str) -> str:
"""The configuration of revision *revision_id*, as the device keeps it.
:raises UnknownConfigRevision: for an id that is not in the history.
:raises Exception: when the device could not be read.
"""
...
# The step a driver implements:
def _revert_to_config_revision(self, revision_id: str) -> None:
"""Make revision *revision_id* the device's configuration; return once it is
in place, as the newest revision of the history.
:raises UnknownConfigRevision: when the device no longer has it.
:raises ConfigRevertFailed: when the device refuses.
"""
...
def config_revision_content(self, text: str) -> str:
"""What of a revision's *text* is its configuration: the whole text by default.
A driver whose device stamps every save into the configuration (who, when)
returns the text without that stamp, so that two revisions with the same
configuration compare equal.
"""
return text
def revert_config_revision(self, revision_id: str) -> ConfigRevisionDict:
"""Make revision *revision_id* the device's configuration again.
Returns the revision in place afterwards: the newest of the history, which
holds the configuration of *revision_id* (on most devices a new revision the
revert saved). What the device does beyond that -- reload its services,
restart, nothing -- is the driver's to document.
:raises UnknownConfigRevision: for an id that is not in the history now;
nothing has changed then.
:raises ConfigRevertFailed: when the device refuses, or when the configuration
in place afterwards is not the revision's.
:raises Exception: when the device could not be reached.
"""
if revision_id not in {r["id"] for r in self.get_config_revisions()}:
raise UnknownConfigRevision(revision_id)
# Read before the revert: saving it may rotate the revision out of the history.
wanted = self.config_revision_content(self.get_config_revision(revision_id))
self._revert_to_config_revision(revision_id)
revisions = self.get_config_revisions()
if not revisions:
raise ConfigRevertFailed(
f"the device has no configuration after reverting to {revision_id!r}"
)
now = revisions[0]
if self.config_revision_content(self.get_config_revision(now["id"])) != wanted:
raise ConfigRevertFailed(
f"the configuration in place ({now['id']!r}) is not revision {revision_id!r}"
" after the revert"
)
return now
+203
View File
@@ -0,0 +1,203 @@
"""Cron expressions: what they mean, when they fire next, and as systemd ``OnCalendar``.
A scheduled job is given as a cron expression on every host, and a host with
systemd runs it as a timer. So the expression has to mean the same in three
places -- on the host's cron, in a preview of the next runs, and in the
``OnCalendar`` lines a timer is written with -- and the meaning lives here once.
**What cron means is Vixie cron's.** Five fields (minute, hour, day of month,
month, day of week), each ``*``, a value, a range ``a-b``, a step ``*/n`` or
``a-b/n``, or a list of those; month and weekday names in any case; ``7`` is
Sunday as well as ``0``; the ``@hourly`` … ``@yearly`` aliases. When both day
fields are restricted -- neither starts with ``*`` -- a day matches if *either*
does; otherwise both must (``*/2`` counts as starting with ``*``).
**``OnCalendar`` by expansion.** Every field is written as its values, so a range
with a step needs no systemd syntax of its own, and a timer fires on exactly the
minutes cron would. The *either* rule is two ``OnCalendar`` lines, which systemd
ORs. ``@reboot`` has no calendar; a timer takes it as ``OnBootSec``.
**Next runs are in the host's time zone.** A local time the clock skips when
summer time starts is left out; cron implementations disagree on it, and a
preview that promised it would promise too much.
"""
from __future__ import annotations
from datetime import date, datetime, timedelta
from typing import NamedTuple
from zoneinfo import ZoneInfo
REBOOT = "@reboot"
ALIASES = {
"@yearly": "0 0 1 1 *",
"@annually": "0 0 1 1 *",
"@monthly": "0 0 1 * *",
"@weekly": "0 0 * * 0",
"@daily": "0 0 * * *",
"@midnight": "0 0 * * *",
"@hourly": "0 * * * *",
}
_MONTHS = ("jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep", "oct", "nov", "dec")
_WEEKDAYS = ("sun", "mon", "tue", "wed", "thu", "fri", "sat")
_SYSTEMD_WEEKDAYS = ("Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat")
#: (lowest, highest, names) per field; a name stands for its index plus the lowest.
_FIELDS = (
(0, 59, ()),
(0, 23, ()),
(1, 31, ()),
(1, 12, _MONTHS),
(0, 7, _WEEKDAYS),
)
#: How far ahead a next run is looked for: a 29 February comes within eight years.
_HORIZON_DAYS = 8 * 366
class CronSchedule(NamedTuple):
"""An expression's values per field, and whether a day matches on either day field."""
minutes: frozenset[int]
hours: frozenset[int]
days: frozenset[int]
months: frozenset[int]
weekdays: frozenset[int] # 0 = Sunday
either_day: bool
def is_reboot(expr: str) -> bool:
"""Whether *expr* is ``@reboot``: once at boot, no calendar."""
return expr.strip().lower() == REBOOT
def _value(token: str, low: int, names: tuple[str, ...]) -> int:
if token.isdigit():
return int(token)
if token.lower() in names:
return names.index(token.lower()) + low
raise ValueError(f"not a value: {token!r}")
def _part(part: str, low: int, high: int, names: tuple[str, ...]) -> set[int]:
span, _, step_text = part.partition("/")
step = 1
if step_text:
if not step_text.isdigit() or int(step_text) == 0:
raise ValueError(f"not a step: {part!r}")
step = int(step_text)
if span == "*":
first, last = low, high
elif "-" in span:
a, _, b = span.partition("-")
first, last = _value(a, low, names), _value(b, low, names)
else:
first = _value(span, low, names)
last = high if step_text else first
if not low <= first <= last <= high:
raise ValueError(f"out of range {low}-{high}: {part!r}")
return set(range(first, last + 1, step))
def _field(text: str, low: int, high: int, names: tuple[str, ...]) -> frozenset[int]:
values: set[int] = set()
for part in text.split(","):
if not part:
raise ValueError(f"empty list item in {text!r}")
values |= _part(part, low, high, names)
return frozenset(values)
def parse_cron(expr: str) -> CronSchedule:
"""The values *expr* stands for.
:raises ValueError: for anything Vixie cron would not take, and for
``@reboot``, which names no time.
"""
text = expr.strip()
if is_reboot(text):
raise ValueError("@reboot names no time; it runs once at boot")
if text.startswith("@"):
if text.lower() not in ALIASES:
raise ValueError(f"unknown alias {text!r}")
text = ALIASES[text.lower()]
fields = text.split()
if len(fields) != len(_FIELDS):
raise ValueError(f"a cron expression has five fields, not {len(fields)}: {expr!r}")
minutes, hours, days, months, weekdays = (
_field(f, low, high, names) for f, (low, high, names) in zip(fields, _FIELDS, strict=True)
)
if 7 in weekdays:
weekdays = (weekdays - {7}) | {0}
either = not fields[2].startswith("*") and not fields[4].startswith("*")
return CronSchedule(minutes, hours, days, months, frozenset(weekdays), either)
def _day_matches(s: CronSchedule, day: date) -> bool:
if day.month not in s.months:
return False
on_day = day.day in s.days
on_weekday = (day.isoweekday() % 7) in s.weekdays
return (on_day or on_weekday) if s.either_day else (on_day and on_weekday)
def _local(day: date, hour: int, minute: int, tz: ZoneInfo) -> datetime | None:
"""*day* at *hour*:*minute* in *tz*; None when the clock skips that time."""
naive = datetime(day.year, day.month, day.day, hour, minute)
aware = naive.replace(tzinfo=tz)
if aware.astimezone(ZoneInfo("UTC")).astimezone(tz).replace(tzinfo=None) != naive:
return None
return aware
def next_runs(expr: str, after: datetime, tz: ZoneInfo, count: int = 5) -> list[datetime]:
"""The next *count* times *expr* fires after *after*, in *tz*.
``@reboot`` and a day that never comes (31 February) give none.
:raises ValueError: for an expression :func:`parse_cron` refuses.
"""
if is_reboot(expr):
return []
s = parse_cron(expr)
start = after.astimezone(tz)
runs: list[datetime] = []
for offset in range(_HORIZON_DAYS):
day = start.date() + timedelta(days=offset)
if not _day_matches(s, day):
continue
for hour in sorted(s.hours):
for minute in sorted(s.minutes):
at = _local(day, hour, minute, tz)
if at is None or at <= after:
continue
runs.append(at)
if len(runs) == count:
return runs
return runs
def _values(values: frozenset[int], low: int, high: int) -> str:
if values == frozenset(range(low, high + 1)):
return "*"
return ",".join(f"{v:02d}" for v in sorted(values))
def to_on_calendar(expr: str) -> list[str]:
"""``OnCalendar`` lines that fire exactly when *expr* does.
:raises ValueError: for an expression :func:`parse_cron` refuses, ``@reboot``
among them -- a timer takes that as ``OnBootSec``.
"""
s = parse_cron(expr)
time = f"{_values(s.hours, 0, 23)}:{_values(s.minutes, 0, 59)}:00"
months = _values(s.months, 1, 12)
days = _values(s.days, 1, 31)
weekdays = ",".join(_SYSTEMD_WEEKDAYS[d] for d in (1, 2, 3, 4, 5, 6, 0) if d in s.weekdays)
every_weekday = s.weekdays == frozenset(range(7))
if s.either_day:
return [f"*-{months}-{days} {time}", f"{weekdays} *-{months}-* {time}"]
prefix = "" if every_weekday else f"{weekdays} "
return [f"{prefix}*-{months}-{days} {time}"]
+156 -61
View File
@@ -26,6 +26,7 @@ import re
from collections.abc import Iterator
from typing import TYPE_CHECKING, Any
from napalm_device_types._dhcp_order import ReservationChange, order_reservation_changes
from napalm_device_types._matching import check_desired, index_live
from napalm_device_types.models import (
DhcpHaDict,
@@ -57,6 +58,27 @@ _DHCP_SUBNET_COMPARE_FIELDS = (
_HEX_ONLY = re.compile(r"[^0-9a-f]")
#: Added to a refused change that is part of an address swap (#106).
_SWAP_NOTE = (
"an address swap with {}: no order frees an address first; "
"move one of them to a free address in a run of its own"
)
class DhcpApplyError(RuntimeError):
"""Some changes of an applied reservation or subnet set were refused by the device.
Raised after the rest were written and committed: ``failed`` of ``attempted``
changes did not go through, and each was named in a ``[failed]`` progress line.
``kind`` is what was applied: ``"reservation"`` or ``"subnet"``.
"""
def __init__(self, failed: int, attempted: int, kind: str = "reservation") -> None:
self.failed = failed
self.attempted = attempted
self.kind = kind
super().__init__(f"{failed} of {attempted} DHCP {kind} change(s) were refused")
def normalize_mac(mac: str | None) -> str:
"""Reduces a MAC address to lowercase colon-separated form.
@@ -106,6 +128,43 @@ def _subnet_field_differs(field: str, live: DhcpSubnetDict, desired: DhcpSubnetD
return any(live_options.get(option) != value for option, value in desired_options.items())
def _reservation_key(reservation: DhcpReservationDict) -> str:
return normalize_mac(reservation.get("mac"))
def _diff_reservations(
live: list[DhcpReservationDict], desired: list[DhcpReservationDict]
) -> DhcpReservationDiffDict:
"""``diff_dhcp_reservations`` against reservations already read."""
live_by_mac, shared = index_live(live, _reservation_key)
check_desired(desired, _reservation_key, shared, "MAC")
add: list[DhcpReservationDict] = []
update: list[DhcpReservationUpdateDict] = []
for desired_reservation in desired:
current = live_by_mac.get(_reservation_key(desired_reservation))
if current is None:
add.append(desired_reservation)
continue
changed_fields = [
field
for field in _DHCP_RESERVATION_COMPARE_FIELDS
if current.get(field) != desired_reservation.get(field)
]
if changed_fields:
update.append(
{
"uuid": current["uuid"],
"reservation": desired_reservation,
"changed_fields": changed_fields,
}
)
return {"add": add, "update": update}
class DhcpServerMixin:
"""Mixin providing DHCP reservation and subnet read/diff/apply.
@@ -258,36 +317,7 @@ class DhcpServerMixin:
:raises ValueError: When a desired reservation cannot name a single live one.
"""
def key(reservation: DhcpReservationDict) -> str:
return normalize_mac(reservation.get("mac"))
live_by_mac, shared = index_live(self.get_dhcp_reservations(), key)
check_desired(desired, key, shared, "MAC")
add: list[DhcpReservationDict] = []
update: list[DhcpReservationUpdateDict] = []
for desired_reservation in desired:
live = live_by_mac.get(key(desired_reservation))
if live is None:
add.append(desired_reservation)
continue
changed_fields = [
field
for field in _DHCP_RESERVATION_COMPARE_FIELDS
if live.get(field) != desired_reservation.get(field)
]
if changed_fields:
update.append(
{
"uuid": live["uuid"],
"reservation": desired_reservation,
"changed_fields": changed_fields,
}
)
return {"add": add, "update": update}
return _diff_reservations(self.get_dhcp_reservations(), desired)
def apply_dhcp_reservationset(self, desired: list[DhcpReservationDict]) -> Iterator[str]:
"""
@@ -298,29 +328,65 @@ class DhcpServerMixin:
committing reloads the DHCP daemon and drops in-flight requests, which
is too high a price for a no-op run.
The changes are written so an address is released before it is taken; a
swap of addresses has no such order and is reported as one (``_dhcp_order``).
A change the device refuses (say, an address in a subnet its DHCP server
does not manage) does not stop the others: it gets a ``[failed]`` line,
the rest are written and committed, and ``DhcpApplyError`` is raised at
the end. Any non-empty diff commits, even one whose every write was
refused, so that writes an earlier, interrupted run saved take effect.
:param desired: The complete desired reservation set.
:yields: Progress lines, one per applied add/update, plus a final
commit line.
:yields: Progress lines, one per applied, updated or refused change,
plus a final commit line.
:raises DhcpApplyError: after the commit, when a change was refused.
"""
diff = self.diff_dhcp_reservations(desired)
for reservation in diff["add"]:
self.apply_dhcp_reservation(reservation)
yield f"[add] {reservation['mac']} -> {reservation['ip']}"
for entry in diff["update"]:
self.apply_dhcp_reservation(entry["reservation"], uuid=entry["uuid"])
fields = ", ".join(entry["changed_fields"])
yield (
f"[update] {entry['reservation']['mac']} -> {entry['reservation']['ip']} ({fields})"
)
if not diff["add"] and not diff["update"]:
live = self.get_dhcp_reservations()
diff = _diff_reservations(live, desired)
attempted = len(diff["add"]) + len(diff["update"])
if not attempted:
yield "[commit] no changes"
return
added = updated = refused = 0
for change in order_reservation_changes(diff, live):
line, ok = self._write_change(change)
refused += not ok
added += ok and change.uuid is None
updated += ok and change.uuid is not None
yield line
# Committed even when every write was refused: a run that stopped before its
# commit left saved writes behind, which the device reports as configured, so
# no later diff would ever commit them.
self.commit_dhcp_reservations()
yield (f"[commit] applied {len(diff['add'])} add(s), {len(diff['update'])} update(s)")
yield f"[commit] applied {added} add(s), {updated} update(s)"
if refused:
raise DhcpApplyError(refused, attempted)
def _write_change(self, change: ReservationChange) -> tuple[str, bool]:
"""Write one change of an ordered set; its progress line, and whether it went in."""
tag = "[add]" if change.uuid is None else "[update]"
line, ok = self._write_reservation(change.reservation, change.uuid, tag)
if ok and change.fields:
line = f"{line} ({change.fields})"
elif not ok and change.swap:
line = f"{line} -- {_SWAP_NOTE.format(', '.join(change.swap))}"
return line, ok
def _write_reservation(
self, reservation: DhcpReservationDict, uuid: str | None, tag: str
) -> tuple[str, bool]:
"""Write one reservation; its progress line, and whether the device took it."""
target = f"{reservation['mac']} -> {reservation['ip']}"
try:
self.apply_dhcp_reservation(reservation, uuid=uuid)
except Exception as exc:
return f"[failed] {target}: {exc}", False
return f"{tag} {target}", True
def diff_dhcp_subnets(self, desired: list[DhcpSubnetDict]) -> DhcpSubnetDiffDict:
"""
@@ -391,24 +457,53 @@ class DhcpServerMixin:
As with ``apply_dhcp_reservationset``, an empty diff does **not**
commit: reconfiguring the DHCP daemon is too expensive for a no-op.
A subnet the device refuses (say, one that overlaps a subnet it serves)
does not stop the others: it gets a ``[failed]`` line, the rest are
written and committed, and ``DhcpApplyError`` is raised at the end. Any
non-empty diff commits, even one whose every write was refused, so that
writes an earlier, interrupted run saved take effect (#113).
:param desired: The complete desired subnet set.
:yields: Progress lines, one per applied add/update, plus a final
commit line.
:yields: Progress lines, one per applied, updated or refused change,
plus a final commit line.
:raises DhcpApplyError: after the commit, when a change was refused.
"""
diff = self.diff_dhcp_subnets(desired)
for subnet in diff["add"]:
self.apply_dhcp_subnet(subnet)
yield f"[add] {subnet['subnet']}"
for entry in diff["update"]:
self.apply_dhcp_subnet(entry["subnet"], uuid=entry["uuid"])
fields = ", ".join(entry["changed_fields"])
yield f"[update] {entry['subnet']['subnet']} ({fields})"
if not diff["add"] and not diff["update"]:
attempted = len(diff["add"]) + len(diff["update"])
if not attempted:
yield "[commit] no changes"
return
changes: list[tuple[DhcpSubnetDict, str | None, str]] = [
(subnet, None, "") for subnet in diff["add"]
]
changes += [
(e["subnet"], e["uuid"], ", ".join(e["changed_fields"])) for e in diff["update"]
]
added = updated = refused = 0
for subnet, uuid, fields in changes:
line, ok = self._write_subnet(subnet, uuid, fields)
refused += not ok
added += ok and uuid is None
updated += ok and uuid is not None
yield line
# Committed even when every write was refused; see apply_dhcp_reservationset.
self.commit_dhcp_subnets()
yield (f"[commit] applied {len(diff['add'])} add(s), {len(diff['update'])} update(s)")
yield f"[commit] applied {added} add(s), {updated} update(s)"
if refused:
raise DhcpApplyError(refused, attempted, "subnet")
def _write_subnet(
self, subnet: DhcpSubnetDict, uuid: str | None, fields: str
) -> tuple[str, bool]:
"""Write one subnet; its progress line, and whether the device took it."""
try:
self.apply_dhcp_subnet(subnet, uuid=uuid)
except Exception as exc:
return f"[failed] {subnet['subnet']}: {exc}", False
if uuid is None:
return f"[add] {subnet['subnet']}", True
return f"[update] {subnet['subnet']} ({fields})", True
+62 -1
View File
@@ -22,7 +22,7 @@ They replace ``run_device_action`` (docs/STRUCTURE.md):
from __future__ import annotations
from collections.abc import Callable, Sequence
from typing import TYPE_CHECKING, Any
from typing import TYPE_CHECKING, Any, ClassVar
from napalm_device_types.models import (
FixResultDict,
@@ -133,6 +133,66 @@ class EnsurePackageMixin:
def install_package(self, name: str, version: str = "") -> None: ...
class LldpAgentMixin:
"""Run an LLDP agent on the device, on the interfaces the caller names.
The steps: install the agent, write the interfaces it runs on, enable it at boot
and restart it so it reads them. Installing and starting are generic, over
``ensure_package`` and ``manage_service`` with ``LLDP_AGENT_PACKAGE`` and
``LLDP_AGENT_SERVICE``; the driver writes the configuration. Which interfaces is
the caller's choice -- an agent on every interface also talks to the uplink of a
WAN port. Whether neighbours appear is the caller's to read with
``get_lldp_neighbors``, after the agent's first advertisement interval.
"""
#: The package that brings the agent, as ``ensure_package`` takes it.
LLDP_AGENT_PACKAGE: ClassVar[str] = "lldpd"
#: The agent's service, as ``manage_service`` takes it.
LLDP_AGENT_SERVICE: ClassVar[str] = "lldpd"
def configure_lldp_agent(self, interfaces: Sequence[str]) -> FixResultDict:
"""Install, configure for *interfaces*, enable and restart the agent.
:raises ValueError: for no interface, before anything changes.
"""
names = [str(name) for name in interfaces]
if not names:
raise ValueError("name at least one interface for the LLDP agent")
return run_fix_steps(
[
("install", self._install_lldp_agent),
("configure", lambda: self._write_lldp_agent_config(names)),
("start", self._start_lldp_agent),
]
)
def _install_lldp_agent(self) -> FixStepOutcomeDict:
result = self.ensure_package(self.LLDP_AGENT_PACKAGE)
return {"ok": bool(result["success"]), "output": str(result.get("output", ""))}
def _start_lldp_agent(self) -> FixStepOutcomeDict:
service = self.LLDP_AGENT_SERVICE
for action in ("enable", "restart"):
result = self.manage_service(service, action)
if not result.get("success"):
output = str(result.get("output", "")).strip()
return {"ok": False, "output": f"{action} {service}: {output}".rstrip(": ")}
return {"ok": True, "output": f"{service} enabled and restarted"}
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
def ensure_package(self, name: str) -> FixResultDict: ...
def manage_service(self, name: str, action: str) -> dict[str, Any]: ...
# The step a driver implements:
def _write_lldp_agent_config(self, interfaces: list[str]) -> FixStepOutcomeDict:
"""Have the agent send and listen on exactly *interfaces* (names as the
device's interface getters report them), replacing the list it had."""
...
class PackageProxyMixin:
"""Point the device's package manager at a caching proxy."""
@@ -152,6 +212,7 @@ class PackageProxyMixin:
__all__ = [
"EnsurePackageMixin",
"FixStep",
"LldpAgentMixin",
"PackageProxyMixin",
"SnmpAgentMixin",
"run_fix_steps",
+2 -2
View File
@@ -2,8 +2,8 @@
**Deprecated** (docs/STRUCTURE.md): drivers do not collect over SNMP. The
application polls; a driver whose device publishes values under its own OIDs
declares them as ``SNMP_HEALTH_PROFILE`` (:class:`~napalm_device_types.models.SnmpHealthProfile`).
This mixin goes in 5.0.
declares them as ``SNMP_HEALTH_PROFILE``
(:class:`~napalm_device_types.snmp_health_profile.SnmpHealthProfile`). This mixin goes in 5.0.
The collection itself is generic: CPU, memory, uptime, load and per-interface
counters come from the same standard OIDs on any UCD-MIB/IF-MIB capable device.
+32
View File
@@ -0,0 +1,32 @@
"""Shutting the device itself down cleanly.
Declared under ``if TYPE_CHECKING``: a contract, not a placeholder. Only a
driver that can shut its device down through the device's own means defines
``shutdown_host``, so ``hasattr(driver, "shutdown_host")`` tells a caller
whether to offer it. There is no forced variant here, by design: cutting the
power is the BMC's business, not a driver's.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
class HostShutdownMixin:
if TYPE_CHECKING:
def shutdown_host(self) -> None:
"""
Shuts down the device this driver is connected to, cleanly.
Returns once the device has accepted the request; the session is
usually gone right after. Whether the device really went off is
the caller's business, e.g. through its BMC.
A driver that manages other machines shuts down its *own* host,
never one of them: a hypervisor node shuts down itself, and its
guests are stopped the way the node stops them on shutdown.
:raises RuntimeError: If the device refuses the request.
"""
...
+67 -13
View File
@@ -43,7 +43,9 @@ exits quietly every day (NetOrk/netork#738). While dpkg, apt, aptitude or
unattended-upgrade is running, packages are halfway on purpose -- also between
two of apt's dpkg calls -- so the answer is then ``None``, as on a host without
dpkg. unattended-upgrade is told from Ubuntu's idle shutdown helper by its
command line (:data:`BUSY_CHECK`).
command line (:data:`BUSY_CHECK`). On a host without dpkg, dnf or yum, apk and opkg
are read the same way (:mod:`napalm_device_types.package_managers`,
NetOrk/netork#851), and ``package_manager`` names the one the answer is about.
**Filesystems** are how full ``/``, ``/boot`` and ``/var`` are, from POSIX
``df -Pk``: where an upgrade unpacks a kernel, builds its initrd and keeps the
@@ -58,6 +60,7 @@ too full is not decided here.
from __future__ import annotations
import re
from functools import cmp_to_key
from typing import TYPE_CHECKING
from napalm_device_types.models import (
@@ -65,6 +68,11 @@ from napalm_device_types.models import (
HostStatusDict,
InterruptedPackageDict,
)
from napalm_device_types.package_managers import (
PACKAGE_MANAGERS_COMMAND,
PackageManagerReading,
parse_package_managers,
)
from napalm_device_types.terminal import strip_terminal_codes
_BEGIN = "HSTAT_BEGIN"
@@ -117,6 +125,7 @@ HOST_STATUS_COMMAND = (
f"echo '[dpkg-journal]'; ls -1 {_DPKG_JOURNAL} 2>/dev/null | head -n 20; "
"echo '[dpkg-audit]'; dpkg-query -W -f='${db:Status-Abbrev} ${Package}\\n' 2>/dev/null "
"| awk 'substr($0, 2, 1) ~ /[HUFWt]/ || substr($0, 3, 1) == \"R\"' | head -n 50; fi; "
f"{PACKAGE_MANAGERS_COMMAND}"
f"echo '[filesystems]'; df -Pk {_FILESYSTEMS}; "
f"echo '[timers]'; for u in {_APT_TIMER} {' '.join(_DNF_TIMERS)}; do "
'printf \'%s %s\\n\' "$u" "$(systemctl is-enabled "$u" 2>/dev/null)"; done; '
@@ -163,9 +172,39 @@ def _sections(output: str) -> dict[str, list[str]]:
return sections
def _version_key(version: str) -> tuple[object, ...]:
"""Natural order: 6.8.0-142 after 6.8.0-87, 7.0.14 after 7.0.2."""
return tuple(int(part) if part.isdigit() else part for part in re.split(r"(\d+)", version))
#: A version's segments as rpm compares them: runs of digits, runs of letters, and
#: ``~``; whatever else stands between them only separates.
_VERSION_SEGMENT = re.compile(r"~|\d+|[A-Za-z]+")
def _segment_cmp(x: str, y: str) -> int:
"""One segment against another, as :func:`_version_cmp` takes them; 0 when equal."""
if "~" in (x, y):
return (y == "~") - (x == "~")
if x.isdigit() and y.isdigit():
return (int(x) > int(y)) - (int(x) < int(y))
if x.isdigit() or y.isdigit():
return 1 if x.isdigit() else -1
return (x > y) - (x < y)
def _version_cmp(a: str, b: str) -> int:
"""rpm's ``rpmvercmp``: below 0 when ``a`` is older, above when newer.
Numbers compare as numbers (6.8.0-142 after 6.8.0-87), letters as text, and a
number is newer than letters in its place: ``112.5.1.el9uek`` after
``112.5.el9uek`` (#109). A version that has more segments where the other
ends is the newer one, unless its next is ``~``, which comes before anything:
``6.13~rc5`` before ``6.13``, as dpkg sorts it too.
"""
left, right = _VERSION_SEGMENT.findall(a), _VERSION_SEGMENT.findall(b)
for x, y in zip(left, right, strict=False):
if order := _segment_cmp(x, y):
return order
if len(left) == len(right):
return 0
longer, sign = (left, 1) if len(left) > len(right) else (right, -1)
return -sign if longer[min(len(left), len(right))] == "~" else sign
def kernel_reboot_pending(running: str, installed: list[str]) -> str | None:
@@ -180,8 +219,9 @@ def kernel_reboot_pending(running: str, installed: list[str]) -> str | None:
same = [k for k in installed if _kernel_flavour(k)[1] == flavour]
if not same:
return None
newest = max(same, key=lambda k: _version_key(_kernel_flavour(k)[0]))
if _version_key(_kernel_flavour(newest)[0]) > _version_key(version):
order = cmp_to_key(_version_cmp)
newest = max(same, key=lambda k: order(_kernel_flavour(k)[0]))
if order(_kernel_flavour(newest)[0]) > order(version):
return newest
return None
@@ -287,12 +327,24 @@ def _halfway(line: str) -> InterruptedPackageDict | None:
return {"name": name, "state": state} if name and state else None
def _dpkg(sections: dict[str, list[str]]) -> tuple[bool | None, list[InterruptedPackageDict]]:
if "dpkg-audit" not in sections or "dpkg-busy" in sections:
return None, []
def _dpkg(sections: dict[str, list[str]]) -> PackageManagerReading | None:
if "dpkg-audit" not in sections:
return None
if "dpkg-busy" in sections:
return PackageManagerReading("dpkg", None, [])
packages = [p for p in map(_halfway, sections["dpkg-audit"]) if p is not None]
journal = any(_JOURNAL_ENTRY.fullmatch(f) for f in sections.get("dpkg-journal") or [])
return bool(packages) or journal, packages
return PackageManagerReading("dpkg", bool(packages) or journal, packages)
def _package_manager(sections: dict[str, list[str]]) -> PackageManagerReading:
"""The reading that says most: one left halfway, then one that is not, then a busy one."""
readings = [r for r in (_dpkg(sections), *parse_package_managers(sections)) if r is not None]
for interrupted in (True, False):
found = next((r for r in readings if r.interrupted is interrupted), None)
if found is not None:
return found
return readings[0] if readings else PackageManagerReading(None, None, [])
def _filesystems(sections: dict[str, list[str]]) -> list[FilesystemUsageDict] | None:
@@ -318,15 +370,16 @@ def parse_host_status(output: str) -> HostStatusDict:
"""
sections = _sections(output)
required, reason = _reboot(sections)
interrupted, packages = _dpkg(sections)
manager = _package_manager(sections)
return {
"reboot_required": required,
"reboot_reason": reason,
"auto_updates": _auto_updates(sections),
"auto_updates_last_success": _last_success(sections),
"auto_updates_interval": _interval(sections),
"package_manager_interrupted": interrupted,
"interrupted_packages": packages,
"package_manager": manager.manager,
"package_manager_interrupted": manager.interrupted,
"interrupted_packages": manager.packages,
"filesystems": _filesystems(sections),
}
@@ -356,6 +409,7 @@ class HostStatusMixin:
* auto_updates (bool or None)
* auto_updates_last_success (Unix time or None)
* auto_updates_interval (seconds or None)
* package_manager (dpkg, dnf, yum, apk, opkg, or None)
* package_manager_interrupted (bool or None)
* interrupted_packages (list of {name, state})
* filesystems (list of {mountpoint, total, used, available} in bytes, or None)
+76 -14
View File
@@ -13,6 +13,7 @@ Usage::
from typing import TYPE_CHECKING, Any
from napalm_device_types.base import DeviceTypeDriver
from napalm_device_types.clusters import ClusterDict
from napalm_device_types.health_metrics import HealthMetricsMixin
from napalm_device_types.models import (
HypervisorHostDict,
@@ -30,6 +31,7 @@ from napalm_device_types.models import (
)
from napalm_device_types.packages import PackageManagementMixin
from napalm_device_types.provisioning import QEMU_GUEST_AGENTS, GuestAgents
from napalm_device_types.vm_migration import MigrationPreflightDict
class HypervisorDriver(PackageManagementMixin, HealthMetricsMixin, DeviceTypeDriver):
@@ -464,21 +466,18 @@ class HypervisorDriver(PackageManagementMixin, HealthMetricsMixin, DeviceTypeDri
storage: str | None = None,
cpu_type: str | None = None,
guest_os: str = "linux",
node: str | None = None,
download_timeout: int = 300,
timeout: int = 180,
) -> VMProvisionResultDict:
"""
Create a new virtual machine from a cloud image via Cloud-Init.
Downloads the cloud image (qcow2/raw) directly on the hypervisor if not
already cached there, creates a new VM shell, imports the image as its
root disk, configures virtual network interfaces, and injects Cloud-Init
configuration via a storage snippet or similar mechanism. The resulting
VM is left in a running state.
Implementations should cache downloaded images by URL/filename on the
hypervisor so repeated provisioning from the same image does not
re-download it every time.
Downloads the cloud image (qcow2/raw) directly on the hypervisor unless it is
cached there already (implementations cache by URL/filename), creates a new VM
shell, imports the image as its root disk, configures virtual network interfaces,
and injects Cloud-Init configuration via a storage snippet or similar mechanism.
The resulting VM is left in a running state.
Args:
name (string) - new VM display name
@@ -497,10 +496,8 @@ class HypervisorDriver(PackageManagementMixin, HealthMetricsMixin, DeviceTypeDri
image_checksum (string | None) - expected checksum of the downloaded
image (e.g. "sha256:<hex>"). If given, verified after download;
mismatch raises RuntimeError. If None, no verification is performed.
ssh_public_keys (list[str] | None) - SSH public keys to inject into guest.
If None or empty, no SSH key injection is performed.
disk_resize_gb (int | None) - resize root disk to this size in GB.
If None, disk remains the downloaded image's native size. Default None.
ssh_public_keys (list[str] | None) - SSH public keys to inject (None: none).
disk_resize_gb (int | None) - resize root disk to this size in GB (None: keep).
storage (string | None) - name of the storage pool to place the root
disk on (a name returned by ``get_image_storages()``). If None,
the driver auto-detects the first enabled, node-available storage
@@ -519,6 +516,9 @@ class HypervisorDriver(PackageManagementMixin, HealthMetricsMixin, DeviceTypeDri
network-config itself (``provisioning.network_config``). A
guest_os the driver does not list raises ValueError before
anything is created. Default ``"linux"``.
node (string | None) - a node ``name`` from ``get_clusters()`` to create
the VM on; None leaves it to the driver. One that only creates on its
own node raises ValueError for another, before creating anything.
download_timeout (int) - maximum seconds to wait for the image download
(skipped entirely if already cached on the hypervisor). Default 300.
timeout (int) - maximum seconds to wait for the remaining provisioning
@@ -531,6 +531,8 @@ class HypervisorDriver(PackageManagementMixin, HealthMetricsMixin, DeviceTypeDri
Raises:
RuntimeError - if provisioning fails (download failure, checksum
mismatch, storage unavailable, invalid config, timeout, etc.)
VMProvisionIncomplete - a RuntimeError for a failure after the VM was created,
with that VM's ``vmid`` and ``node`` for ``destroy_vm`` and the failure as cause.
"""
...
@@ -613,7 +615,9 @@ class HypervisorDriver(PackageManagementMixin, HealthMetricsMixin, DeviceTypeDri
List[NetworkTargetDict] - each entry's ``vlan_aware`` flag tells the
caller whether a ``NICConfigDict.vlan_tag`` may additionally be set
for a NIC using that target (see ``NetworkTargetDict`` for the
per-kind rules).
per-kind rules). ``uplinks``, where the driver can tell, names
the physical NICs a bridge leaves the host through, grouped:
a bond's slaves in one group, a single NIC in its own.
"""
...
@@ -635,6 +639,64 @@ class HypervisorDriver(PackageManagementMixin, HealthMetricsMixin, DeviceTypeDri
"""
...
def get_clusters(self) -> list[ClusterDict]:
"""
List the clusters this connection can see, each with its nodes, its
storages and its networks.
A cluster member (a Proxmox node) reports its one cluster, the whole of
it: every node, offline ones included with ``status`` ``"offline"``, and
``local`` set on the node it is. A manager (vCenter) reports each
cluster it runs, and each standalone host it runs as a cluster of one.
A standalone host reports itself as a cluster of one, ``name`` None.
The VMs ``get_vms()`` lists name their node in ``node``; that name is
the node's ``name`` here. A storage or network lists the nodes it is
available on, so a caller can tell where a VM may go. A value the
platform does not report is None, never a guess.
Facts only: which node is a good place for a VM is the caller's to
decide.
Returns:
List[ClusterDict]
"""
...
def get_vm_migration_preflight(self, name: str) -> MigrationPreflightDict:
"""Which nodes of its cluster the guest could move to, and what blocks it
(see ``vm_migration``).
:raises ValueError: If no VM with the given name/ID exists.
"""
...
def migrate_vm(
self,
name: str,
target_node: str,
online: bool = True,
target_storage: str | None = None,
timeout: int = 3600,
) -> None:
"""Move the guest to another node of its cluster; block until the
platform's task ended. ``online`` keeps a running guest running.
:raises ValueError: Unknown VM, or a target that is not another
member of the cluster.
:raises RuntimeError: The migration failed, with the platform's cause.
"""
...
def set_node_maintenance(self, node: str, enabled: bool) -> None:
"""Switch HA maintenance mode of a cluster node; HA guests leave it
while it is on.
:raises ValueError: If ``node`` is not a member of the cluster.
:raises RuntimeError: If the platform refuses or cannot be asked.
"""
...
def get_hypervisor_hosts(self) -> list[HypervisorHostDict]:
"""The hosts this hypervisor or its manager runs, and how each stands:
connected, in maintenance, with configuration issues, under a central
+97
View File
@@ -0,0 +1,97 @@
"""LLDP neighbours as rows, and the two shapes NAPALM asks for.
A driver reads its neighbours one row per neighbour -- from ``lldpctl``
(:func:`~napalm_device_types.lldpctl.parse_lldpctl_keyvalue`), an API or a CLI
-- and ``get_lldp_neighbors`` and ``get_lldp_neighbors_detail`` are the same
projection of those rows for every driver. A port with several neighbours (a
phone and the PC behind it, an unmanaged switch) is several rows, and stays
several entries in both shapes.
"""
from __future__ import annotations
import re
from collections.abc import Iterable
from typing import NotRequired, TypedDict
from napalm.base.models import LLDPNeighborDetailDict
#: A chassis id of type ``mac`` that really is a MAC address.
_MAC_RE = re.compile(r"^([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}$")
class LldpNeighborRow(TypedDict):
"""One LLDP neighbour, as the local port heard it.
``chassis_id_type`` and ``port_id_type`` name what the id is: ``mac``,
``ip``, ``ifname``, ``ifalias``, ``local`` or ``unhandled`` (an id type
lldpd does not decode), ``""`` when the neighbour sent no id.
``capabilities`` and ``enabled_capabilities`` use NAPALM's names
(``bridge``, ``router``, ``wlan-access-point``, ``telephone``, ...).
"""
local_port: str
chassis_id: str
chassis_id_type: str
port_id: str
port_id_type: str
port_description: str
system_name: str
system_description: str
management_addresses: list[str]
capabilities: list[str]
enabled_capabilities: list[str]
class LldpNeighborDict(TypedDict):
"""NAPALM's ``get_lldp_neighbors`` entry (``hostname``, ``port``), plus the chassis MAC.
``mac`` (lower case) is there only when the neighbour names its chassis by a
MAC address; an application can match the neighbour by it, where a name
may be ambiguous.
"""
hostname: str
port: str
mac: NotRequired[str]
def lldp_neighbors(rows: Iterable[LldpNeighborRow]) -> dict[str, list[LldpNeighborDict]]:
"""``get_lldp_neighbors``: per local port, every neighbour's name, port and chassis MAC."""
neighbors: dict[str, list[LldpNeighborDict]] = {}
for row in rows:
entry: LldpNeighborDict = {"hostname": row["system_name"], "port": row["port_id"]}
if row["chassis_id_type"] == "mac" and _MAC_RE.match(row["chassis_id"]):
entry["mac"] = row["chassis_id"].lower()
neighbors.setdefault(row["local_port"], []).append(entry)
return neighbors
def lldp_neighbors_detail(
rows: Iterable[LldpNeighborRow], interface: str = ""
) -> dict[str, list[LLDPNeighborDetailDict]]:
"""``get_lldp_neighbors_detail``: per local port, every neighbour in full.
With ``interface``, that port only. ``parent_interface`` stays empty: a row
does not know which aggregate its port belongs to.
"""
details: dict[str, list[LLDPNeighborDetailDict]] = {}
for row in rows:
if interface and row["local_port"] != interface:
continue
details.setdefault(row["local_port"], []).append(
{
"parent_interface": "",
"remote_port": row["port_id"],
"remote_port_description": row["port_description"],
"remote_chassis_id": row["chassis_id"],
"remote_system_name": row["system_name"],
"remote_system_description": row["system_description"],
"remote_system_capab": list(row["capabilities"]),
"remote_system_enable_capab": list(row["enabled_capabilities"]),
}
)
return details
__all__ = ["LldpNeighborDict", "LldpNeighborRow", "lldp_neighbors", "lldp_neighbors_detail"]
+204
View File
@@ -0,0 +1,204 @@
"""``lldpctl -f keyvalue``: lldpd's neighbours, one row each.
Every driver for a host that runs lldpd reads its neighbours the same way, so
the command and its parse live here once and a driver only carries the command
across. lldpd's keyvalue writer (``src/client/kv_writer.c``) prints one
``key=value`` line per field, the key being the field's path joined by dots
(a newline inside a value becomes a space)::
lldp.eth0.1.via=LLDP
lldp.eth0.1.chassis.mac=00:11:22:33:44:01
lldp.eth0.1.chassis.Tel.enabled=on
lldp.eth0.1.port.local=gi1/0/7
**A neighbour is a block, not a port.** Every neighbour on a port prints the
same keys, so grouping the lines by interface merges them. A neighbour opens
with ``via`` (after the local interface's ``alias``, when it has one); the next
``via`` on the same port is the next neighbour.
**An interface name may hold dots** (``eth0.1``, ``br-lan.10``). It is
everything between ``lldp.`` and the neighbour's first field, never the second
dotted part of the key.
**An id is printed under its type**, ``chassis.<type>=`` and ``port.<type>=``,
with lldpd's names for the type (``mac``, ``ip``, ``ifname``, ``ifalias``,
``local``, ``unhandled``); there is no ``chassis.id`` key.
**A failed read raises.** lldpctl exits non-zero when it cannot reach lldpd;
:data:`LLDPCTL_KEYVALUE_COMMAND` brings that status and lldpctl's message back,
and the parse raises on it instead of reading it as no neighbours. Output that
carries no exit status (a driver that runs plain ``lldpctl -f keyvalue``) raises
when it holds text but not a single neighbour line, as a shell's ``lldpctl: not
found`` does.
"""
from __future__ import annotations
import re
from napalm_device_types.lldp import LldpNeighborRow
_RC = "__LLDPCTL_RC"
#: One line, POSIX ``sh``, read-only. Without arguments lldpctl shows the
#: neighbours in detail; its error text and exit status come back with them.
LLDPCTL_KEYVALUE_COMMAND = f"lldpctl -f keyvalue 2>&1; echo {_RC}=$?"
_RC_RE = re.compile(rf"^{_RC}=(\d+)$")
_PREFIX = "lldp."
#: The fields a neighbour opens with: the local interface's alias, then ``via``.
_OPENERS = frozenset({"alias", "via"})
#: The first part after the interface name of every key lldpd prints for a
#: neighbour (``src/client/display.c``, ``display_interface``).
_CATEGORIES = _OPENERS | {
"rid",
"age",
"status",
"chassis",
"port",
"ttl",
"vlan",
"ppvid",
"pi",
"lldp-med",
"unknown-tlvs",
}
#: lldpd's id types (``src/lib/atoms/chassis.c``, ``port.c``), in the order read.
_CHASSIS_ID_TYPES = ("mac", "local", "ip", "ifname", "ifalias", "unhandled")
_PORT_ID_TYPES = ("ifname", "mac", "local", "ip", "ifalias", "unhandled")
#: lldpd's capability symbols in NAPALM's names; one NAPALM lacks is lowercased.
_CAPABILITIES = {
"Other": "other",
"Repeater": "repeater",
"Bridge": "bridge",
"Router": "router",
"Wlan": "wlan-access-point",
"Tel": "telephone",
"Docsis": "docsis-cable-device",
"Station": "station",
}
_SWITCH = {"on": True, "off": False}
_Fields = dict[str, list[str]]
def parse_lldpctl_keyvalue(output: str) -> list[LldpNeighborRow]:
"""One row per neighbour, in the order lldpctl listed them.
:raises ValueError: when lldpctl failed -- a non-zero exit status, or text
without a single neighbour line and no status at all.
"""
return [_row(port, fields) for port, fields in _neighbours(_pairs(output))]
def _pairs(output: str) -> list[tuple[str, str]]:
"""``(key without "lldp.", value)`` of every neighbour line; raises on a failed run."""
pairs: list[tuple[str, str]] = []
other: list[str] = []
status: int | None = None
for line in output.splitlines():
line = line.strip()
key, sep, value = line.partition("=")
if _RC_RE.match(line):
status = int(value)
elif sep and key.startswith(_PREFIX):
pairs.append((key[len(_PREFIX) :], value))
elif line:
other.append(line)
_raise_on_failure(status, other, bool(pairs))
return pairs
def _raise_on_failure(status: int | None, other: list[str], listed: bool) -> None:
"""A non-zero status, or text without a status and without a neighbour line."""
if status or (status is None and other and not listed):
said = " ".join(other)[:200] or f"exit status {status}"
raise ValueError(f"lldpctl did not list the neighbours: {said}")
def _neighbours(pairs: list[tuple[str, str]]) -> list[tuple[str, _Fields]]:
"""Split the lines into neighbours: ``(local interface, {field: [values]})``."""
neighbours: list[tuple[str, _Fields]] = []
port: str | None = None
fields: _Fields = {}
for key, value in pairs:
split = _split(key, port)
if split is None:
continue
name, field = split
if name != port or field == "alias" or (field == "via" and "via" in fields):
port, fields = name, {}
neighbours.append((port, fields))
fields.setdefault(field, []).append(value)
return neighbours
def _split(key: str, port: str | None) -> tuple[str, str] | None:
"""``(interface, field)`` of a key, or None when it is no neighbour field.
``via`` and ``alias`` end their key, so they name the interface exactly;
the keys after them continue that interface. Only a key that does neither
ends its interface name at the first known field.
"""
head, _, last = key.rpartition(".")
if head and last in _OPENERS:
return head, last
if port is not None and key.startswith(port + "."):
field = key[len(port) + 1 :]
if field.split(".", 1)[0] in _CATEGORIES:
return port, field
parts = key.split(".")
for i in range(1, len(parts)):
if parts[i] in _CATEGORIES:
return ".".join(parts[:i]), ".".join(parts[i:])
return None
def _row(port: str, fields: _Fields) -> LldpNeighborRow:
chassis_type, chassis_id = _id(fields, "chassis", _CHASSIS_ID_TYPES)
port_type, port_id = _id(fields, "port", _PORT_ID_TYPES)
capabilities, enabled = _capabilities(fields)
return {
"local_port": port,
"chassis_id": chassis_id,
"chassis_id_type": chassis_type,
"port_id": port_id,
"port_id_type": port_type,
"port_description": _first(fields, "port.descr"),
"system_name": _first(fields, "chassis.name"),
"system_description": _first(fields, "chassis.descr"),
"management_addresses": list(fields.get("chassis.mgmt-ip", [])),
"capabilities": capabilities,
"enabled_capabilities": enabled,
}
def _first(fields: _Fields, field: str) -> str:
return fields.get(field, [""])[0]
def _id(fields: _Fields, category: str, types: tuple[str, ...]) -> tuple[str, str]:
"""``(type, id)`` of the chassis or the port; ``("", "")`` when none was sent."""
for kind in types:
if f"{category}.{kind}" in fields:
return kind, _first(fields, f"{category}.{kind}")
return "", ""
def _capabilities(fields: _Fields) -> tuple[list[str], list[str]]:
"""``chassis.<Symbol>.enabled=on|off``: the capabilities offered, and those enabled."""
offered: list[str] = []
enabled: list[str] = []
for field, values in fields.items():
parts = field.split(".")
switch = _SWITCH.get(values[0])
if len(parts) != 3 or parts[0] != "chassis" or parts[2] != "enabled" or switch is None:
continue
name = _CAPABILITIES.get(parts[1], parts[1].lower())
offered.append(name)
if switch:
enabled.append(name)
return offered, enabled
__all__ = ["LLDPCTL_KEYVALUE_COMMAND", "parse_lldpctl_keyvalue"]
+943
View File
@@ -0,0 +1,943 @@
"""Personal login accounts on a host: reading them, adding one, locking one again.
A person who is to work on a host gets an account of their own: they log in
with their public key and, where they are to administer the host, may use
sudo with their own password. When they leave, the account is locked, not
deleted: their key goes, sudo goes, the login is refused, and the home stays.
**What the person changes follows them** (NetOrk/netork#2013). A key they
replace goes from the account as the new one comes in (``replaces``); a key
they delete goes without a new one (``remove_local_account_keys``). Either way
only the keys named go: one the person added on the host stays, and the
account stays open. A password they remove goes from the account too
(``clear_password``): it logs in with a key only and has no sudo, as the sudo
rule asks for that password.
:class:`LocalAccountsMixin` holds what is the same on every POSIX host -- the
checks, reading an account back, the order of the steps, the key file and the
sudo rule. A driver supplies the commands that differ between operating
systems: creating, opening and locking an account, reading its lock, and
taking it out of a group.
**Every command root runs is one fixed command**, with no shell syntax of its
own -- ``useradd ...``, ``install -m 440 ...``, ``rm -f -- <path>`` -- so a
sudoers allowlist of commands can name each one. What goes into a file is
staged first, without privilege; root never reads stdin, where a sudo that
does not prompt would leave the password.
**Nothing root runs follows the person's files.** The key files live in a home
the person owns, where a symbolic link may point anywhere, so they are read and
written as that person (``sudo -n -u <name>``). Only the existence of a key
file is asked as root.
**The sudo rule is a file of its own** in the drop-in directory sudo includes,
``<name> ALL=(ALL) ALL``: the person's own password, as on a new VM. A group
alone is not enough, as some distributions grant ``%wheel`` and others
``%sudo``. ``visudo`` checks the rule before it is installed, and ``sudo -l``
afterwards shows whether it took effect. A rule elsewhere -- cloud-init's
``90-cloud-init-users`` -- is not edited; it is reported.
**The login user's own account and system accounts are never touched**:
locking the first would lock the caller out.
**netOrk's own account** (``add_service_account``) is made the same way,
through the account it logs in with today: a key and no password, and a rule
of its own -- ``<name> ALL=(ALL) NOPASSWD: ALL`` in ``85-service-<name>`` --
so it gains root without a password that could end up anywhere. Once netOrk
logs in with it, the person's account it used before is one like any other.
"""
from __future__ import annotations
import re
import shlex
from collections.abc import Iterable, Sequence
from typing import TYPE_CHECKING, ClassVar
from typing_extensions import TypedDict
from napalm_device_types.fixes import FixStep, run_fix_steps
from napalm_device_types.models import FixResultDict, FixStepDict, FixStepOutcomeDict
if TYPE_CHECKING: # pragma: no cover - for type checkers only
from napalm_device_types.channel import CommandResult
#: A login name every POSIX host takes as it is: lowercase, a letter or an
#: underscore first, at most 32 characters. It is also safe in a root command.
ACCOUNT_NAME_RE = re.compile(r"[a-z_][a-z0-9_.-]{0,31}")
#: The crypt(3) prefixes of each password scheme a driver may take.
PASSWORD_SCHEMES: dict[str, tuple[str, ...]] = {
"sha512": ("$6$",),
"bcrypt": ("$2b$", "$2a$", "$2y$"),
}
_CRYPT_RE = re.compile(r"\$[0-9a-z]+\$[./0-9A-Za-z$]+")
#: The public key types OpenSSH takes in a key file.
_KEY_TYPE_RE = re.compile(
r"(ssh-ed25519|ssh-rsa|ecdsa-sha2-nistp(256|384|521)"
r"|sk-ssh-ed25519@openssh\.com|sk-ecdsa-sha2-nistp256@openssh\.com)"
)
_KEY_BLOB_RE = re.compile(r"[A-Za-z0-9+/]+={0,3}")
#: Before the rule's file name; after cloud-init's ``90-`` it would win, so before it.
SUDOERS_FILE_PREFIX = "80-account-"
#: Before the file name of netOrk's own account's rule; after a person's, before cloud-init's.
SERVICE_SUDOERS_FILE_PREFIX = "85-service-"
#: A line of ``sudo -l`` that lets the account run every command as root.
_RUN_ALL_RE = re.compile(r"^\s*\(ALL(\s*:\s*ALL)?\)\s+([A-Z_]+:\s*)*ALL\s*$", re.MULTILINE)
#: The same, without a password.
_RUN_ALL_NOPASSWD_RE = re.compile(
r"^\s*\(ALL(\s*:\s*ALL)?\)\s+([A-Z_]+:\s*)*NOPASSWD:\s*([A-Z_]+:\s*)*ALL\s*$",
re.MULTILINE,
)
#: Login shells that refuse every login: an account with one belongs to a service.
_NO_LOGIN_SHELLS = frozenset({"nologin", "false"})
#: Stages stdin as a file anyone may read -- the person reads the key file
#: from it -- and prints its path. The content is a public key or a sudo rule.
_STAGE = 't=$(mktemp) && chmod 644 "$t" && cat > "$t" && printf %s "$t"'
class LocalAccountDict(TypedDict):
"""A person's account as the host has it (``get_local_accounts``)."""
name: str
uid: int
home: str
shell: str
#: Primary group first.
groups: list[str]
#: No login, with a key or a password.
locked: bool
#: ``sudo -l`` lets it run every command as root.
admin: bool
#: The lines of its key files; None when they could not be read (a locked
#: account cannot run a command any more).
authorized_keys: list[str] | None
class LocalAccountChangeDict(FixResultDict):
"""The result of ``add_local_account`` and ``lock_local_account``."""
#: The account read back after the change; None when it is not there or
#: could not be read.
account: LocalAccountDict | None
def sudoers_rule(name: str) -> str:
"""The rule that lets *name* run everything as root, with its own password."""
return f"{name} ALL=(ALL) ALL\n"
def sudoers_rule_path(directory: str, name: str) -> str:
"""Where *name*'s rule lives in *directory*.
sudo skips a file whose name holds a dot, so a dot becomes ``_2e``, and an
underscore ``__``: no two names share a file.
"""
return f"{directory}/{SUDOERS_FILE_PREFIX}{_rule_file_name(name)}"
def service_sudoers_rule(name: str) -> str:
"""The rule that lets netOrk's own account *name* run everything as root, no password."""
return f"{name} ALL=(ALL) NOPASSWD: ALL\n"
def service_sudoers_rule_path(directory: str, name: str) -> str:
"""Where the rule of netOrk's own account *name* lives in *directory*."""
return f"{directory}/{SERVICE_SUDOERS_FILE_PREFIX}{_rule_file_name(name)}"
def _rule_file_name(name: str) -> str:
return name.replace("_", "__").replace(".", "_2e")
def authorized_keys_paths(home: str) -> tuple[str, str]:
"""The key files sshd reads in *home* unless configured otherwise."""
return f"{home}/.ssh/authorized_keys", f"{home}/.ssh/authorized_keys2"
def check_account_name(name: str) -> None:
""":raises ValueError: when *name* is no login name every host takes."""
if not ACCOUNT_NAME_RE.fullmatch(name):
raise ValueError(f"not an account name: {name!r}")
def _key_blob(line: str) -> str | None:
"""The base64 part of a key file line, options and comment aside."""
words = line.split()
for i, word in enumerate(words[:-1]):
if _KEY_TYPE_RE.fullmatch(word):
return words[i + 1]
return None
def check_public_key(key: str) -> None:
""":raises ValueError: when *key* is not one OpenSSH public key on one line."""
words = key.split()
if (
"\n" in key.strip()
or len(words) < 2
or not _KEY_TYPE_RE.fullmatch(words[0])
or not _KEY_BLOB_RE.fullmatch(words[1])
):
raise ValueError("not a public key: expected '<type> <base64> [comment]' on one line")
def _checked_keys(keys: Sequence[str]) -> list[str]:
"""*keys*, each one stripped.
:raises ValueError: when *keys* is one string, or one of them no public key.
"""
if isinstance(keys, str):
raise ValueError("expected a list of public keys, not one string")
for key in keys:
check_public_key(key)
return [key.strip() for key in keys]
def _command(argv: Sequence[str]) -> str:
return shlex.join(argv)
def _as(name: str, *argv: str) -> str:
"""*argv* run as *name*, from root: nothing it does reaches beyond that account."""
return _command(["sudo", "-n", "-u", name, *argv])
def _said(result: CommandResult) -> str:
return (result.stderr or result.stdout).strip() or f"exit {result.exit_code}"
class LocalAccountsMixin:
"""Reads, adds and locks personal accounts; a driver supplies the OS's commands.
Mixed in by the drivers that can, so ``callable(getattr(driver,
"add_local_account", None))`` says whether a driver can.
"""
#: The login shell of a new account.
ACCOUNT_SHELL: ClassVar[str] = "/bin/sh"
#: The password scheme the host's accounts take, a key of :data:`PASSWORD_SCHEMES`.
ACCOUNT_PASSWORD_SCHEME: ClassVar[str] = "sha512"
#: Groups that make an account an administrator; a person without sudo leaves them.
ACCOUNT_ADMIN_GROUPS: ClassVar[tuple[str, ...]] = ("wheel",)
#: The lowest uid of a person's account; below are the system's.
ACCOUNT_MIN_UID: ClassVar[int] = 1000
#: The drop-in directory sudo includes.
SUDOERS_DIR: ClassVar[str] = "/etc/sudoers.d"
#: The group of root's files.
ROOT_GROUP: ClassVar[str] = "root"
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
def run_command(
self,
command: str,
*,
privileged: bool = False,
timeout: float = 60,
stdin: bytes | None = None,
) -> CommandResult: ...
# -- what a driver supplies ------------------------------------------------------
#
# ``_account_*`` are the driver's: the five below it must supply, the two after
# them (``_account_create_with_uid``, ``_account_clear_password``) it may, and
# the key store (``_account_key_files``, ``_account_read_keys``,
# ``_account_add_key``, ``_account_drop_keys``, ``_account_remove_keys``) it may
# replace. ``_accounts_*`` are this mixin's own.
def _account_create(self, name: str, password_hash: str | None) -> list[list[str]]:
"""The commands that create *name* with a home and :attr:`ACCOUNT_SHELL`.
Without *password_hash* the account takes no password, only a key.
"""
raise NotImplementedError
def _account_open(
self, name: str, password_hash: str | None, *, locked: bool
) -> list[list[str]]:
"""The commands that let *name* log in again, with *password_hash* if given."""
raise NotImplementedError
def _account_lock(self, name: str, *, locked: bool) -> list[list[str]]:
"""The commands that refuse *name* every login, a key's too; none when *locked*."""
raise NotImplementedError
def _account_locked(self, name: str, passwd: list[str]) -> bool:
"""Whether *name* is locked; *passwd* is its ``getent passwd`` line, split.
:raises RuntimeError: when that cannot be read.
"""
raise NotImplementedError
def _account_leave_group(self, name: str, group: str, groups: list[str]) -> list[str]:
"""The command that takes *name* out of *group*; *groups* are all of its own."""
raise NotImplementedError
def _account_create_with_uid(self, name: str, uid: int) -> list[list[str]]:
"""The commands that create *name* as :meth:`_account_create` does, without a
password, and with the uid *uid*.
Only netOrk's own account asks for one (``add_service_account(uid=)``).
"""
raise NotImplementedError
def _account_clear_password(self, name: str) -> list[list[str]]:
"""The commands that take *name*'s password away; its key login stays as it is.
Not an empty password -- PAM's ``nullok`` lets anyone in without one --
and not a lock or an expiry, which sshd or PAM apply to a key too.
Only ``add_local_account(clear_password=True)`` and
``remove_local_account_keys(clear_password=True)`` ask for it.
"""
raise NotImplementedError
# -- reading ---------------------------------------------------------------------
def get_local_accounts(self, names: Iterable[str]) -> list[LocalAccountDict]:
"""The accounts of *names* the host has, in that order; a missing one is left out.
:raises ValueError: for a name that is no account name.
:raises RuntimeError: when an account cannot be read.
"""
wanted = list(names)
for name in wanted:
check_account_name(name)
found = (self._accounts_read(name) for name in wanted)
return [account for account in found if account is not None]
def _accounts_passwd(self, name: str) -> list[str] | None:
result = self.run_command(_command(["getent", "passwd", name]), timeout=30)
if result.exit_code == 2:
return None
fields = result.stdout.strip().split(":")
if result.exit_code != 0 or len(fields) < 7 or fields[0] != name:
raise RuntimeError(f"cannot read the account {name}: {_said(result)}")
return fields
def _accounts_read(self, name: str) -> LocalAccountDict | None:
passwd = self._accounts_passwd(name)
if passwd is None:
return None
groups = self.run_command(_command(["id", "-Gn", name]), timeout=30)
if groups.exit_code != 0:
raise RuntimeError(f"cannot read the groups of {name}: {_said(groups)}")
home = passwd[5]
return {
"name": name,
"uid": int(passwd[2]),
"home": home,
"shell": passwd[6],
"groups": groups.stdout.split(),
"locked": self._account_locked(name, passwd),
"admin": self._accounts_admin(name),
"authorized_keys": self._account_read_keys(name, home),
}
def _accounts_admin(self, name: str) -> bool:
result = self.run_command(_command(["sudo", "-l", "-U", name]), privileged=True, timeout=30)
if result.exit_code != 0:
raise RuntimeError(f"cannot ask sudo about {name}: {_said(result)}")
return bool(_RUN_ALL_RE.search(result.stdout))
def _account_key_files(self, name: str, home: str) -> tuple[str, ...]:
"""Every file sshd reads *name*'s keys from; a host that adds one of its own says so."""
return authorized_keys_paths(home)
def _account_read_keys(self, name: str, home: str) -> list[str] | None:
"""The lines of :meth:`_account_key_files`, read as *name*; None when it cannot."""
lines: list[str] = []
for path in self._account_key_files(name, home):
there = self.run_command(_command(["test", "-e", path]), privileged=True, timeout=30)
if there.exit_code != 0:
continue
result = self.run_command(_as(name, "cat", "--", path), privileged=True, timeout=30)
if result.exit_code != 0:
return None
lines += [ln for ln in result.stdout.splitlines() if ln.strip()]
return lines
# -- changing ----------------------------------------------------------------------
def add_local_account(
self,
name: str,
public_key: str,
*,
password_hash: str | None = None,
admin: bool = False,
clear_password: bool = False,
replaces: Sequence[str] = (),
) -> LocalAccountChangeDict:
"""Give *name* an account that logs in with *public_key*, or open it again.
An account that is there is opened again and keeps its other keys; a
new one gets a home. *password_hash* (crypt(3), in
:attr:`ACCOUNT_PASSWORD_SCHEME`) becomes its password; *admin* gives it
the sudo rule, which asks for that password -- so an administrator
needs one. Without *admin*, the rule and :attr:`ACCOUNT_ADMIN_GROUPS`
are taken away. Without *password_hash* the account keeps the password
it has, unless *clear_password* takes it away: the account then logs
in with a key only, and can have no sudo.
*replaces* are keys *public_key* takes the place of -- the person's
old ones. They go from the account, matched by their base64 whatever
a line's options or comment; every other key stays.
Steps: ``account``, then ``sudo``, ``password`` (with
*clear_password*), ``keys`` and ``old keys`` (with *replaces*), each of
which runs when another failed: an old key goes even when the new one
could not be added.
:raises ValueError: for a name, key or hash no host takes, a replaced
key that is none, *clear_password* with a *password_hash* or with
*admin*, the login user's own account, or a system account.
"""
check_account_name(name)
check_public_key(public_key)
new = _key_blob(public_key)
old = [key for key in _checked_keys(replaces) if _key_blob(key) != new]
self._accounts_check_password(password_hash, admin=admin, clear=clear_password)
existing = self._accounts_changeable(name)
first = run_fix_steps(
[("account", lambda: self._accounts_open_or_create(name, existing, password_hash))]
)
parts = [first]
if first["success"]:
steps = self._accounts_after_open(name, public_key, admin, clear_password, old)
parts += [run_fix_steps([step]) for step in steps]
return self._accounts_result(name, parts)
def _accounts_after_open(
self, name: str, public_key: str, admin: bool, clear_password: bool, old: list[str]
) -> list[FixStep]:
"""The steps of ``add_local_account`` once the account is there, each on its own."""
steps: list[FixStep] = [("sudo", lambda: self._accounts_set_sudo(name, admin=admin))]
if clear_password:
steps.append(("password", lambda: self._accounts_clear_password(name)))
steps.append(("keys", lambda: self._accounts_add_key(name, public_key)))
if old:
steps.append(("old keys", lambda: self._accounts_drop_keys(name, old)))
return steps
def remove_local_account_keys(
self, name: str, keys: Sequence[str], *, clear_password: bool = False
) -> LocalAccountChangeDict:
"""Take *keys* off *name*'s account, for a person who deleted them.
Only those keys go, matched by their base64 whatever a line's options
or comment; the other keys stay, and the account stays open -- to get
a new key once there is one. A locked account has none left to
remove: the lock took them all. Without *clear_password* the password
and sudo stay too; with it the password goes, and so do the sudo rule
and :attr:`ACCOUNT_ADMIN_GROUPS`, as sudo asks for that password -- for
a person whose password went with their key.
Steps: ``keys``, then ``password`` and ``sudo`` (with *clear_password*),
each of which runs when another failed.
:raises ValueError: for a name no host takes, no keys or one that is
none, the login user's own account, or a system account.
"""
check_account_name(name)
wanted = _checked_keys(keys)
if not wanted:
raise ValueError("no public key to remove")
if self._accounts_changeable(name) is None:
step: FixStepDict = {"step": "keys", "ok": False, "output": f"no account {name}"}
return {**_joined([step]), "account": None}
steps: list[FixStep] = [("keys", lambda: self._accounts_drop_keys(name, wanted))]
if clear_password:
steps.append(("password", lambda: self._accounts_clear_password(name)))
steps.append(("sudo", lambda: self._accounts_set_sudo(name, admin=False)))
return self._accounts_result(name, [run_fix_steps([step]) for step in steps])
def lock_local_account(self, name: str) -> LocalAccountChangeDict:
"""Lock *name*: its keys go, every login is refused, sudo goes; its home stays.
Steps ``keys``, ``lock`` and ``sudo``, each run when one before failed:
the lock matters most. A sudo rule outside the drop-in directory is
reported, not edited -- with the account locked it can no longer be used.
:raises ValueError: for a name no host takes, the login user's own
account, or a system account.
"""
check_account_name(name)
if self._accounts_changeable(name) is None:
step: FixStepDict = {"step": "account", "ok": False, "output": f"no account {name}"}
return {**_joined([step]), "account": None}
parts = [
run_fix_steps([("keys", lambda: self._accounts_remove_keys(name))]),
run_fix_steps([("lock", lambda: self._accounts_lock(name))]),
run_fix_steps(
[("sudo", lambda: self._accounts_set_sudo(name, admin=False, locked=True))]
),
]
return self._accounts_result(name, parts)
def add_service_account(
self,
name: str,
public_key: str,
*,
uid: int | None = None,
replaces: Sequence[str] = (),
) -> LocalAccountChangeDict:
"""Make *name* netOrk's own account: it logs in with *public_key*, and gains
root without a password.
Created through the account the driver logs in with now, or opened
again when it is there; its other keys stay. It takes no password. Run
again with the account the driver already logs in with, it only adds
what is missing.
A new account gets *uid* when one is given -- the same on every host --
once no other account has it; an account that is there keeps its own,
as a uid is never changed under files that carry it.
*replaces* are netOrk's keys *public_key* takes the place of: they go
from the account once *public_key* is in it, matched as in
:meth:`add_local_account`, and never *public_key* itself. Call it so
through a login with *public_key*, tried before: netOrk keeps its way
in (NetOrk/netork#2462).
Steps as in :meth:`add_local_account`: ``account``, then ``sudo`` and
``keys``, and ``old keys`` with *replaces* when ``keys`` went. The new
login itself is for the caller to try.
An account that is there and cannot log in (``nologin``, ``false``) is
another service's -- a satellite's ``netork``, say -- and is left alone.
:raises ValueError: for a name or key no host takes, a replaced key that
is none, root, a uid of 0 or less, or another service's account.
:raises RuntimeError: when the login user cannot be told.
"""
check_public_key(public_key)
old = [key for key in _checked_keys(replaces) if _key_blob(key) != _key_blob(public_key)]
login, existing = self._accounts_service_target(name, uid)
first = self._accounts_service_account_step(name, login, existing, uid)
parts = [first]
if first["success"]:
parts += self._accounts_service_keys(name, public_key, old)
return self._accounts_result(name, parts)
def _accounts_service_account_step(
self, name: str, login: str, existing: list[str] | None, uid: int | None
) -> FixResultDict:
"""The ``account`` step of netOrk's own account; nothing to do when the driver
logs in as *name* already."""
if name == login:
step: FixStepDict = {
"step": "account",
"ok": True,
"output": f"this driver logs in as {name} already",
}
return _joined([step])
return run_fix_steps(
[("account", lambda: self._accounts_service_account(name, existing, uid))]
)
def _accounts_service_keys(
self, name: str, public_key: str, old: list[str]
) -> list[FixResultDict]:
"""``sudo`` and ``keys`` of netOrk's own account, then ``old keys`` once the key is in."""
parts = [
run_fix_steps([("sudo", lambda: self._accounts_grant_service_sudo(name))]),
run_fix_steps([("keys", lambda: self._accounts_add_key(name, public_key))]),
]
if old and parts[-1]["success"]:
parts.append(run_fix_steps([("old keys", lambda: self._accounts_drop_keys(name, old))]))
return parts
def _accounts_service_target(self, name: str, uid: int | None) -> tuple[str, list[str] | None]:
"""The login user and *name*'s passwd line, once *name* may become netOrk's.
:raises ValueError: as :meth:`add_service_account` says.
"""
check_account_name(name)
if uid is not None and uid <= 0:
raise ValueError(f"not a uid for netOrk's own account: {uid}")
login = self._accounts_login()
existing = self._accounts_passwd(name)
if existing is not None and int(existing[2]) == 0:
raise ValueError(f"{name} is root")
if existing is not None and existing[6].rsplit("/", 1)[-1] in _NO_LOGIN_SHELLS:
raise ValueError(
f"{name} is there and cannot log in ({existing[6]}): another service's account"
)
return login, existing
def _accounts_service_account(
self, name: str, passwd: list[str] | None, uid: int | None
) -> FixStepOutcomeDict:
"""netOrk's own account opened or made, a new one with *uid* when given."""
if uid is None:
return self._accounts_open_or_create(name, passwd, None)
if passwd is not None:
outcome = self._accounts_open_or_create(name, passwd, None)
if outcome["ok"] and int(passwd[2]) != uid:
outcome["output"] += f"; it keeps its uid {passwd[2]}, not {uid}"
return outcome
owner = self._accounts_passwd_of_uid(uid)
if owner is not None:
return {"ok": False, "output": f"uid {uid} is the account {owner}'s"}
try:
create = self._account_create_with_uid(name, uid)
except NotImplementedError:
return {"ok": False, "output": f"this driver cannot give an account the uid {uid}"}
failed = self._accounts_run(create)
if failed is not None:
return {"ok": False, "output": _said(failed)}
return {"ok": True, "output": f"account {name} created, uid {uid}"}
def _accounts_passwd_of_uid(self, uid: int) -> str | None:
"""The name of the account with *uid*, or None."""
result = self.run_command(_command(["getent", "passwd", str(uid)]), timeout=30)
if result.exit_code == 2:
return None
if result.exit_code != 0:
raise RuntimeError(f"cannot ask who has uid {uid}: {_said(result)}")
return result.stdout.split(":", 1)[0].strip() or None
def _accounts_check_password(
self, password_hash: str | None, *, admin: bool, clear: bool
) -> None:
""":raises ValueError: for a hash the host does not take, or a password to
clear that is to be set, or that sudo would ask for."""
if clear and password_hash is not None:
raise ValueError(
"a password to clear cannot be set too: clear_password or password_hash"
)
if clear and admin:
raise ValueError("an account without a password cannot use sudo: its rule asks for it")
if password_hash is not None:
self._accounts_check_hash(password_hash)
def _accounts_check_hash(self, password_hash: str) -> None:
prefixes = PASSWORD_SCHEMES[self.ACCOUNT_PASSWORD_SCHEME]
if not _CRYPT_RE.fullmatch(password_hash) or not password_hash.startswith(prefixes):
raise ValueError(f"not a {self.ACCOUNT_PASSWORD_SCHEME} password hash")
def _accounts_changeable(self, name: str) -> list[str] | None:
"""*name*'s passwd line, None when there is none.
:raises ValueError: for the login user's own account or a system account.
:raises RuntimeError: when the login user cannot be told.
"""
if name == self._accounts_login():
raise ValueError(f"{name} is the account this driver logs in with")
passwd = self._accounts_passwd(name)
if passwd is not None and int(passwd[2]) < self.ACCOUNT_MIN_UID:
raise ValueError(f"{name} is a system account (uid {passwd[2]})")
return passwd
def _accounts_login(self) -> str:
""":raises RuntimeError: when the login user cannot be told."""
login = self.run_command("id -un", timeout=30).stdout.strip()
if not login:
raise RuntimeError("cannot tell which account this driver logs in with")
return login
def _accounts_result(self, name: str, parts: list[FixResultDict]) -> LocalAccountChangeDict:
steps = [s for part in parts for s in part["steps"]]
result = _joined(steps)
account: LocalAccountDict | None = None
try:
account = self._accounts_read(name)
except RuntimeError as exc:
result["output"] += f"\n[read back] {exc}"
return {**result, "account": account}
def _accounts_root(self, argv: Sequence[str]) -> CommandResult:
return self.run_command(_command(argv), privileged=True, timeout=60)
def _accounts_run(self, commands: list[list[str]]) -> CommandResult | None:
"""Run *commands* as root in order; the first that failed, or None."""
for argv in commands:
result = self._accounts_root(argv)
if result.exit_code != 0:
return result
return None
def _accounts_open_or_create(
self, name: str, passwd: list[str] | None, password_hash: str | None
) -> FixStepOutcomeDict:
if passwd is None:
failed = self._accounts_run(self._account_create(name, password_hash))
done = f"account {name} created"
else:
locked = self._account_locked(name, passwd)
failed = self._accounts_run(self._account_open(name, password_hash, locked=locked))
done = f"account {name} opened again" if locked else f"account {name} is there"
if failed is not None:
return {"ok": False, "output": _said(failed)}
return {"ok": True, "output": done + (", password set" if password_hash else "")}
def _accounts_clear_password(self, name: str) -> FixStepOutcomeDict:
try:
commands = self._account_clear_password(name)
except NotImplementedError:
return {"ok": False, "output": "this driver cannot remove a password"}
failed = self._accounts_run(commands)
if failed is not None:
return {"ok": False, "output": _said(failed)}
return {"ok": True, "output": f"{name} has no password: it logs in with a key only"}
def _accounts_lock(self, name: str) -> FixStepOutcomeDict:
passwd = self._accounts_passwd(name)
if passwd is None:
return {"ok": False, "output": f"no account {name}"}
locked = self._account_locked(name, passwd)
failed = self._accounts_run(self._account_lock(name, locked=locked))
if failed is not None:
return {"ok": False, "output": _said(failed)}
if not self._account_locked(name, passwd):
return {"ok": False, "output": f"{name} is still not locked"}
return {"ok": True, "output": f"{name} was locked already" if locked else f"{name} locked"}
# -- sudo --------------------------------------------------------------------------
def _accounts_set_sudo(
self, name: str, *, admin: bool, locked: bool = False
) -> FixStepOutcomeDict:
path = sudoers_rule_path(self.SUDOERS_DIR, name)
if admin:
return self._accounts_grant_sudo(name, path)
said: list[str] = []
removed = self._accounts_root(["rm", "-f", "--", path])
if removed.exit_code != 0:
return {"ok": False, "output": f"{path}: {_said(removed)}"}
groups = self.run_command(_command(["id", "-Gn", name]), timeout=30).stdout.split()
for group in self.ACCOUNT_ADMIN_GROUPS:
if group not in groups[1:]:
continue
left = self._accounts_root(self._account_leave_group(name, group, groups))
if left.exit_code != 0:
return {"ok": False, "output": f"cannot take {name} out of {group}: {_said(left)}"}
groups.remove(group)
said.append(f"{name} left {group}")
if self._accounts_admin(name):
rest = f"another rule than {path} still lets {name} run everything as root"
if locked:
return {"ok": True, "output": f"{rest}; the account is locked, so it cannot use it"}
return {"ok": False, "output": rest}
said.append(f"{name} has no sudo")
return {"ok": True, "output": "\n".join(said)}
def _accounts_grant_sudo(self, name: str, path: str) -> FixStepOutcomeDict:
failed = self._accounts_install_rule(sudoers_rule(name), path)
if failed is not None:
return failed
if not self._accounts_admin(name):
return {
"ok": False,
"output": f"{path} is written, but sudo does not read it: "
f"does sudoers include {self.SUDOERS_DIR}?",
}
return {"ok": True, "output": f"{name} may use sudo with its password ({path})"}
def _accounts_grant_service_sudo(self, name: str) -> FixStepOutcomeDict:
path = service_sudoers_rule_path(self.SUDOERS_DIR, name)
failed = self._accounts_install_rule(service_sudoers_rule(name), path)
if failed is not None:
return failed
result = self.run_command(_command(["sudo", "-l", "-U", name]), privileged=True, timeout=30)
if result.exit_code != 0:
return {"ok": False, "output": f"cannot ask sudo about {name}: {_said(result)}"}
if not _RUN_ALL_NOPASSWD_RE.search(result.stdout):
return {
"ok": False,
"output": f"{path} is written, but sudo does not read it: "
f"does sudoers include {self.SUDOERS_DIR}?",
}
return {"ok": True, "output": f"{name} may use sudo without a password ({path})"}
def _accounts_install_rule(self, rule: str, path: str) -> FixStepOutcomeDict | None:
"""Check *rule* with visudo and install it at *path*; the failure, or None."""
staged = self._accounts_stage(rule)
try:
checked = self._accounts_root(["visudo", "-cf", staged])
if checked.exit_code != 0:
return {"ok": False, "output": f"visudo refuses the rule: {_said(checked)}"}
installed = self._accounts_root(
["install", "-m", "440", "-o", "root", "-g", self.ROOT_GROUP, "--", staged, path]
)
if installed.exit_code != 0:
return {"ok": False, "output": f"{path}: {_said(installed)}"}
finally:
self._accounts_unstage(staged)
return None
# -- keys --------------------------------------------------------------------------
def _accounts_stage(self, content: str) -> str:
"""*content* in a fresh file of the login user's; its path.
:raises RuntimeError: when it cannot be staged.
"""
result = self.run_command(_STAGE, timeout=30, stdin=content.encode())
path = result.stdout.strip().splitlines()[-1:] or [""]
if result.exit_code != 0 or not path[0].startswith("/"):
raise RuntimeError(f"cannot stage a file: {_said(result)}")
return path[0]
def _accounts_unstage(self, path: str) -> None:
self.run_command(_command(["rm", "-f", "--", path]), timeout=30)
def _accounts_home(self, name: str) -> str | None:
passwd = self._accounts_passwd(name)
return passwd[5] if passwd is not None else None
def _accounts_add_key(self, name: str, public_key: str) -> FixStepOutcomeDict:
home = self._accounts_home(name)
if home is None:
return {"ok": False, "output": f"no account {name}"}
return self._account_add_key(name, home, public_key)
def _accounts_remove_keys(self, name: str) -> FixStepOutcomeDict:
home = self._accounts_home(name)
if home is None:
return {"ok": False, "output": f"no account {name}"}
return self._account_remove_keys(name, home)
def _account_add_key(self, name: str, home: str, public_key: str) -> FixStepOutcomeDict:
"""Add *public_key* to *name*'s first key file, as *name*; the others stay.
A driver whose host keeps keys elsewhere overrides this, with
:meth:`_account_read_keys` and :meth:`_account_remove_keys`.
"""
path = authorized_keys_paths(home)[0]
current = self._account_read_keys(name, home)
if current is None:
return {"ok": False, "output": f"cannot read the key files of {name}"}
blob = _key_blob(public_key)
if any(_key_blob(line) == blob for line in current):
return {"ok": True, "output": f"the key is already in {path}"}
mine = self._accounts_key_file(name, path)
if mine is None:
return {"ok": False, "output": f"cannot read {path}"}
failed = self._accounts_write_key_file(name, home, path, [*mine, public_key.strip()])
if failed is not None:
return {"ok": False, "output": f"{path}: {_said(failed)}"}
return {"ok": True, "output": f"key added to {path}"}
def _accounts_write_key_file(
self, name: str, home: str, path: str, lines: list[str]
) -> CommandResult | None:
"""*lines* as *name*'s key file *path*, written as *name*; None, or what failed.
Without lines the file goes: nothing is staged from an empty stdin.
"""
if not lines:
return self._accounts_run_as(name, [["rm", "-f", "--", path]])
staged = self._accounts_stage("".join(f"{line}\n" for line in lines))
try:
return self._accounts_run_as(
name,
[
["mkdir", "-p", "-m", "700", "--", f"{home}/.ssh"],
["install", "-m", "600", "--", staged, path],
],
)
finally:
self._accounts_unstage(staged)
def _accounts_drop_from(
self, name: str, home: str, path: str, blobs: set[str | None]
) -> FixStepOutcomeDict:
"""The keys of *blobs* out of *name*'s key file *path*; no output when none was in it."""
lines = self._accounts_key_file(name, path)
if lines is None:
return {"ok": False, "output": f"cannot read {path}"}
kept = [line for line in lines if _key_blob(line) not in blobs]
if len(kept) == len(lines):
return {"ok": True, "output": ""}
failed = self._accounts_write_key_file(name, home, path, kept)
if failed is not None:
return {"ok": False, "output": f"{path}: {_said(failed)}"}
return {"ok": True, "output": f"{len(lines) - len(kept)} removed from {path}"}
def _accounts_drop_keys(self, name: str, keys: list[str]) -> FixStepOutcomeDict:
home = self._accounts_home(name)
if home is None:
return {"ok": False, "output": f"no account {name}"}
return self._account_drop_keys(name, home, keys)
def _account_drop_keys(self, name: str, home: str, keys: list[str]) -> FixStepOutcomeDict:
"""Remove exactly *keys* from *name*'s key files in *home*, as *name*.
A key is matched by its base64, whatever a line's options or comment;
every other line stays. Afterwards none of *keys* may be left in
:meth:`_account_key_files`. A driver whose host keeps keys elsewhere
overrides this, with :meth:`_account_add_key`.
"""
blobs = {_key_blob(key) for key in keys}
said: list[str] = []
for path in authorized_keys_paths(home):
outcome = self._accounts_drop_from(name, home, path, blobs)
if not outcome["ok"]:
return outcome
said += [outcome["output"]] if outcome["output"] else []
left = self._account_read_keys(name, home)
if left is None:
return {"ok": False, "output": f"cannot read the key files of {name}"}
if any(_key_blob(line) in blobs for line in left):
return {"ok": False, "output": f"{name} still has a key that was to go"}
return {"ok": True, "output": "\n".join(said) or f"none of the keys is in {home}/.ssh"}
def _accounts_key_file(self, name: str, path: str) -> list[str] | None:
there = self.run_command(_command(["test", "-e", path]), privileged=True, timeout=30)
if there.exit_code != 0:
return []
result = self.run_command(_as(name, "cat", "--", path), privileged=True, timeout=30)
if result.exit_code != 0:
return None
return [line for line in result.stdout.splitlines() if line.strip()]
def _accounts_run_as(self, name: str, commands: list[list[str]]) -> CommandResult | None:
for argv in commands:
result = self.run_command(_as(name, *argv), privileged=True, timeout=60)
if result.exit_code != 0:
return result
return None
def _account_remove_keys(self, name: str, home: str) -> FixStepOutcomeDict:
"""Remove *name*'s key files in *home*, as *name*; then no key may be left."""
paths = authorized_keys_paths(home)
failed = self._accounts_run_as(name, [["rm", "-f", "--", *paths]])
if failed is not None:
return {"ok": False, "output": f"cannot remove the keys of {name}: {_said(failed)}"}
if self._account_read_keys(name, home):
return {"ok": False, "output": f"{name} still has keys"}
return {"ok": True, "output": f"keys of {name} removed"}
def _joined(steps: list[FixStepDict]) -> FixResultDict:
return {
"success": bool(steps) and all(s["ok"] for s in steps),
"steps": steps,
"output": "\n".join(f"[{s['step']}] {s['output']}".rstrip() for s in steps),
}
__all__ = [
"ACCOUNT_NAME_RE",
"PASSWORD_SCHEMES",
"SERVICE_SUDOERS_FILE_PREFIX",
"SUDOERS_FILE_PREFIX",
"LocalAccountChangeDict",
"LocalAccountDict",
"LocalAccountsMixin",
"authorized_keys_paths",
"check_account_name",
"check_public_key",
"service_sudoers_rule",
"service_sudoers_rule_path",
"sudoers_rule",
"sudoers_rule_path",
]
+353
View File
@@ -0,0 +1,353 @@
"""Where a host's logs go beyond it: what rsyslog forwards, which files it reads
in, and whether it gets the journal (NetOrk/netork#2053).
A log source (:mod:`napalm_device_types.log_sources`) says who writes a log;
this says who reads it on. It is read in the same command, so a reading of
the logs a host writes also says which of them leave it.
**rsyslog's configuration, read as text.** Both of its syntaxes are read:
the legacy one (``authpriv.* @@logs:514``, ``& @host``, ``:programname, isequal,
"sshd" @host``, ``$InputFileName``) and RainerScript (``action(type="omfwd" …)``,
``input(type="imfile" …)``, ``ruleset(…) { … }``, ``if … then``). A rule is
*complete* when it takes every message of its ruleset -- ``*.*`` with no
condition around it. Anything narrower is partial, and keeps its selector or
condition for the reader to see: which facility a unit logs to is not
computed (decided in #2053).
**The journal reaches rsyslog** through ``imjournal``, or through ``imuxsock``
when journald forwards to syslog (``ForwardToSyslog``). With no setting of its
own a host runs on its distribution's default, which ``journald.conf`` ships
commented out: ``#ForwardToSyslog=yes``.
"""
from __future__ import annotations
import re
from typing import TypedDict
class ForwardRuleDict(TypedDict):
"""One rule sending messages to another host."""
protocol: str # "udp", "tcp" or "relp"
target: str # the host, as configured
port: int | None # None: the protocol's default
selector: str # "*.*", "authpriv.*", or the condition around it
complete: bool # every message of its ruleset goes
ruleset: str | None # None: the default ruleset, which gets the journal and syslog
#: The template its messages are sent in; None: rsyslog's default, RFC 3164,
#: which carries the sender's local time without a zone (NetOrk/netork#493).
template: str | None
class FileInputDict(TypedDict):
"""A file rsyslog reads in (``imfile``), and the ruleset it hands it to."""
path: str # as configured: may hold a glob
ruleset: str | None
class RsyslogDict(TypedDict):
rules: list[ForwardRuleDict]
file_inputs: list[FileInputDict]
imjournal: bool
imuxsock: bool
class LogForwardingDict(TypedDict):
"""Where a host's logs go beyond it.
``journal`` is None when it cannot be told whether the journal reaches
rsyslog: ``imuxsock`` is loaded but ``journald.conf`` says nothing.
"""
rsyslog: bool # rsyslogd runs
journal: bool | None # the journal reaches rsyslog
rules: list[ForwardRuleDict]
file_inputs: list[FileInputDict]
#: Selectors that take every message.
_EVERYTHING = {"*.*", "*.debug"}
_PARAM_RE = re.compile(r'([A-Za-z][\w.]*)\s*=\s*"([^"]*)"')
_CALL_RE = re.compile(
r"^(?P<before>.*?)\b(?P<name>action|input|module|ruleset)\s*\((?P<args>.*)\)\s*;?$", re.S
)
_IF_RE = re.compile(r"^(?:else\s+)?if\s+(?P<cond>.*?)\s+then\b\s*(?P<rest>.*)$", re.S)
_PROPERTY_RE = re.compile(r'^(?P<sel>:[^,]+,\s*!?[\w-]+,\s*"(?:[^"\\]|\\.)*")\s+(?P<action>.+)$')
_LEGACY_RE = re.compile(r"^(?P<sel>\S+)\s+(?P<action>.+)$")
_FORWARD_RE = re.compile(
r"^(?:(?P<at>@@?)(?:\([^)]*\))?|:omrelp:)"
r"(?:\[(?P<v6>[^\]]+)\]|(?P<host>[^:;\s]+))(?::(?P<port>\d+))?"
)
_FORWARD_TYPES = {"omfwd", "omrelp"}
_TRUE = {"yes", "true", "1", "on"}
def _uncomment(text: str) -> str:
"""*text* without its comments: ``#`` to the end of the line, outside quotes."""
out: list[str] = []
quoted = commented = False
for char in text:
if char == "\n":
quoted = commented = False
elif commented:
continue
elif char == '"':
quoted = not quoted
elif char == "#" and not quoted:
commented = True
continue
out.append(char)
return "".join(out)
class _Pieces:
"""Cuts a configuration into statements, ``{`` and ``}``: a line runs on
while a bracket is open, and a brace outside one stands as a piece of its own."""
def __init__(self) -> None:
self.pieces: list[str] = []
self.current: list[str] = []
self.depth = 0
self.quoted = False
def close(self) -> None:
piece = "".join(self.current).strip()
if piece:
self.pieces.append(piece)
self.current.clear()
def bracket(self, char: str) -> None:
if char in "([":
self.depth += 1
elif char in ")]":
self.depth = max(self.depth - 1, 0)
def feed(self, char: str) -> None:
if char == '"':
self.quoted = not self.quoted
elif not self.quoted:
self.bracket(char)
if self.depth == 0 and char in "{}\n":
self.close()
if char != "\n":
self.pieces.append(char)
return
self.current.append(char)
def _pieces(text: str) -> list[str]:
"""Statements, ``{`` and ``}`` in order (:class:`_Pieces`)."""
cutter = _Pieces()
for char in text:
cutter.feed(char)
cutter.close()
return cutter.pieces
def _params(args: str) -> dict[str, str]:
return {key.lower(): value for key, value in _PARAM_RE.findall(args)}
def _port(value: str | None) -> int | None:
return int(value) if value and value.isdigit() else None
class _Reader:
"""Walks the pieces of a configuration, keeping the blocks it is in."""
def __init__(self) -> None:
self.parsed: RsyslogDict = {
"rules": [],
"file_inputs": [],
"imjournal": False,
"imuxsock": False,
}
self.stack: list[tuple[str, str | None]] = []
self.pending: tuple[str, str | None] | None = None
self.legacy_ruleset: str | None = None
self.last_selector = "*.*"
#: ``$ActionForwardDefaultTemplate``, for the legacy rules after it.
self.legacy_template: str | None = None
#: ``module(load="builtin:omfwd" template=…)``, for every omfwd action.
self.module_template: str | None = None
def ruleset(self) -> str | None:
names = [value for kind, value in self.stack if kind == "ruleset"]
return names[-1] if names else self.legacy_ruleset
def condition(self, once: str | None) -> str | None:
conditions = [value for kind, value in self.stack if kind == "if"]
return once or (conditions[-1] if conditions else None)
def forward(
self,
protocol: str,
target: str,
port: int | None,
selector: str,
once: str | None,
template: str | None,
) -> None:
condition = self.condition(once)
self.parsed["rules"].append(
{
"protocol": protocol,
"target": target,
"port": port,
"selector": condition or selector,
"complete": condition is None and selector in _EVERYTHING,
"ruleset": self.ruleset(),
"template": template,
}
)
def piece(self, piece: str) -> None:
if piece == "{":
self.stack.append(self.pending or ("block", None))
self.pending = None
return
if piece == "}":
if self.stack:
self.stack.pop()
return
if piece == "else":
self.pending = ("if", "else")
return
# A condition whose action follows on its own: the action is under it.
once = self.pending[1] if self.pending and self.pending[0] == "if" else None
self.pending = None
match = _IF_RE.match(piece)
if match:
if match["rest"]:
self.statement(match["rest"], once=match["cond"])
else:
self.pending = ("if", match["cond"])
return
self.statement(piece, once=once)
def statement(self, text: str, once: str | None) -> None:
call = _CALL_RE.match(text)
if call:
self.call(call["name"], _params(call["args"]), call["before"].strip(), once)
elif text.startswith("$"):
self.directive(text)
else:
self.legacy(text, once)
def call(self, name: str, params: dict[str, str], before: str, once: str | None) -> None:
if name == "module":
self.module(params.get("load", ""))
if params.get("load", "").removeprefix("builtin:") == "omfwd":
self.module_template = params.get("template") or self.module_template
elif name == "ruleset":
self.pending = ("ruleset", params.get("name"))
elif name == "input" and params.get("type") == "imfile" and params.get("file"):
self.parsed["file_inputs"].append(
{"path": params["file"], "ruleset": params.get("ruleset")}
)
elif name == "action":
self.action(params, before, once)
def action(self, params: dict[str, str], before: str, once: str | None) -> None:
selector = self.last_selector if before == "&" else (before or "*.*")
self.last_selector = selector
kind = params.get("type", "")
if kind in _FORWARD_TYPES and params.get("target"):
protocol = "relp" if kind == "omrelp" else params.get("protocol", "udp").lower()
template = params.get("template") or (self.module_template if kind == "omfwd" else None)
self.forward(
protocol, params["target"], _port(params.get("port")), selector, once, template
)
def module(self, name: str) -> None:
if name == "imjournal":
self.parsed["imjournal"] = True
elif name == "imuxsock":
self.parsed["imuxsock"] = True
def directive(self, text: str) -> None:
name, _, value = text.partition(" ")
name, value = name.lower(), value.strip()
if name == "$modload":
self.module(value)
elif name == "$inputfilename" and value:
self.parsed["file_inputs"].append({"path": value, "ruleset": self.legacy_ruleset})
elif name == "$ruleset":
self.legacy_ruleset = None if value == "RSYSLOG_DefaultRuleset" else value
elif name == "$actionforwarddefaulttemplate":
self.legacy_template = value or None
def legacy(self, text: str, once: str | None) -> None:
match = _PROPERTY_RE.match(text) or _LEGACY_RE.match(text)
if not match:
return
selector = self.last_selector if match["sel"] == "&" else match["sel"]
self.last_selector = selector
action = match["action"].strip()
forward = _FORWARD_RE.match(action)
if forward:
protocol = {"@": "udp", "@@": "tcp"}.get(forward["at"] or "", "relp")
self.forward(
protocol,
forward["v6"] or forward["host"],
_port(forward["port"]),
selector,
once,
self.legacy_template_of(action[forward.end() :], protocol),
)
def legacy_template_of(self, rest: str, protocol: str) -> str | None:
"""The template after a legacy rule's target (``;Name``), else the default
set before it; RELP takes no forwarding default."""
_, semicolon, named = rest.partition(";")
if semicolon and named.strip():
return named.strip()
return self.legacy_template if protocol != "relp" else None
def parse_rsyslog(text: str) -> RsyslogDict:
"""rsyslog's configuration -- ``rsyslog.conf`` and ``rsyslog.d`` as one text --
read for its forwarding rules, its file inputs, and the inputs that take
the journal."""
reader = _Reader()
for piece in _pieces(_uncomment(text)):
reader.piece(piece)
return reader.parsed
def journal_to_syslog(lines: list[str]) -> bool | None:
"""Whether journald forwards to syslog, from its ``ForwardToSyslog`` lines in
the order journald reads them: the last setting wins; with none, the
default the distribution ships commented out; None when nothing says."""
setting = shipped = None
for line in lines:
commented = line.lstrip().startswith("#")
_, _, value = line.lstrip("# \t").partition("=")
if commented:
shipped = value.strip().lower() in _TRUE
else:
setting = value.strip().lower() in _TRUE
return setting if setting is not None else shipped
def log_forwarding(
running: bool, rsyslog: RsyslogDict, journald_lines: list[str]
) -> LogForwardingDict:
"""Where a host's logs go: *rsyslog* is its configuration, read whether it runs or not."""
if not running:
return {"rsyslog": False, "journal": False, "rules": [], "file_inputs": []}
journal: bool | None
if rsyslog["imjournal"]:
journal = True
elif rsyslog["imuxsock"]:
journal = journal_to_syslog(journald_lines)
else:
journal = False
return {
"rsyslog": True,
"journal": journal,
"rules": rsyslog["rules"],
"file_inputs": rsyslog["file_inputs"],
}
+365
View File
@@ -0,0 +1,365 @@
"""Log sources: which logs a host writes, and who writes each of them.
Before anything can watch a host's logs -- a person, CrowdSec, a syslog
forwarder -- it has to know which logs there are. That is read the same way on
every Linux host, so the command and its parse live here once, and a driver
only carries the command across (NetOrk/netork#2049).
**Three kinds.** journald names every unit it holds entries for
(``journalctl -F _SYSTEMD_UNIT``); only services count, not login sessions.
``/proc/<pid>/fd`` shows the files running processes hold open, a log being a
``*.log`` / ``*.log.N`` file or anything under ``/var/log``; ``stat -L`` on the
descriptor gives its size and last write, also for a file inside a container,
whose path is the container's own. The engine's own file of a container's
output (``/var/lib/docker/containers/<id>/…``) is that container's stream, not
a file of ``dockerd``.
**Who writes it.** Only a process holding the file open for writing writes
it: the access mode in ``/proc/<pid>/fdinfo/<fd>``'s ``flags`` is ``O_WRONLY`` or
``O_RDWR``. A reader -- CrowdSec, Promtail, rsyslog's ``imfile``, a ``tail -F``
-- holds it too, and is no producer (NetOrk/netork#2092). ``/proc/<pid>/cgroup``
says which systemd service or container the first writer runs in, as for
listening sockets (:mod:`napalm_device_types.listening`). A file only readers
hold is listed without a producer; one only readers in a container hold is left
out, its path being the reader's view of a mount. journald's own files, files
already deleted and a database's write-ahead log (LevelDB's ``000004.log``) are
left out.
**Where they go** is read in the same round trip
(:mod:`napalm_device_types.log_forwarding`, NetOrk/netork#2053): whether
rsyslog runs, its configuration -- each line behind ``| ``, so that none reads
as a section -- and journald's ``ForwardToSyslog`` lines in the order journald
reads them.
**One round trip, root, and without it.** As for listening sockets the script
goes as one ``sh -c`` argument. Without root only the login user's processes
show their files; the command then runs again unprivileged, and the reading
says it is not ``attributed``. A report whose end is missing raises: a list cut
short must never read as logs that went away, and so must a host whose
``find`` cannot list open files (busybox): it raises
:class:`LogSourcesUnavailable`.
"""
from __future__ import annotations
import re
from datetime import UTC, datetime
from shlex import quote
from typing import TYPE_CHECKING, TypedDict
from napalm_device_types.listening import _cgroup_paths, _container, _sections, _unit
from napalm_device_types.log_forwarding import LogForwardingDict, log_forwarding, parse_rsyslog
from napalm_device_types.terminal import strip_terminal_codes
class LogSourceDict(TypedDict):
"""A log the host writes: a unit's journal, a file a process holds open, or a
container's stream as the engine keeps it.
``path`` is as its writer sees it: a file a container writes is named inside
that container. Whether anything reads the log is the consumer's question.
"""
kind: str # "journald", "file" or "container"
path: str | None # the file; None for a journal
unit: str | None # the journal's unit, or the file writer's service; without ".service"
container_id: str | None # the full container ID: the writer's, or the stream's
process: str | None # the first process writing the file; None when only readers hold it
size: int | None # bytes; None for a journal, or when the file could not be read
modified: str | None # the last write, ISO 8601 UTC; None as for size
class LogSourcesDict(TypedDict):
"""What ``LogSourcesMixin.get_log_sources`` read.
``attributed`` is False when the reading ran without root: only the login
user's own processes show their files then, so a missing log means "not
told", not "not there". ``journald`` is False on a host without systemd's
journal, where no unit has one.
"""
attributed: bool
journald: bool
sources: list[LogSourceDict]
forwarding: LogForwardingDict | None # where the logs go beyond the host
_BEGIN = "LOGS_BEGIN"
_END = "LOGS_END"
#: One line, POSIX ``sh`` with GNU ``find`` (``-lname``, ``-printf``), read-only.
#: The frame markers are printed in two halves so that a transport which echoes
#: the command does not show them early. ``-printf '%p %l'`` puts the descriptor
#: before its target, so a target with a space in it stays whole.
LOG_SOURCES_COMMAND = (
"PATH=$PATH:/usr/sbin:/sbin; "
"printf '%s%s\\n' LOGS_ BEGIN; "
"if [ -d /run/systemd/system ] && command -v journalctl >/dev/null 2>&1; then "
"j=journalctl; command -v timeout >/dev/null 2>&1 && j='timeout 30 journalctl'; "
"echo '[journald]'; $j -F _SYSTEMD_UNIT 2>/dev/null; fi; "
"if find / -maxdepth 0 -lname x -printf '' >/dev/null 2>&1; then "
"o=$(find /proc/[0-9]*/fd -mindepth 1 -maxdepth 1 -type l "
"\\( -lname '*.log' -o -lname '*.log.[0-9]*' -o -lname '/var/log/*' \\) "
"-printf '%p %l\\n' 2>/dev/null); "
"echo '[open]'; printf '%s\\n' \"$o\"; "
"echo '[stat]'; printf '%s\\n' \"$o\" | cut -d' ' -f1 "
"| xargs -r stat -L -c '%n %s %Y' 2>/dev/null; "
"echo '[flags]'; printf '%s\\n' \"$o\" | cut -d' ' -f1 | sed 's|/fd/|/fdinfo/|' "
"| xargs -r grep -H '^flags:' 2>/dev/null; "
"pids=$(printf '%s\\n' \"$o\" | cut -d/ -f3 | sort -un); "
"echo '[cgroups]'; for p in $pids; do sed \"s|^|$p |\" /proc/$p/cgroup 2>/dev/null; done; "
"echo '[comm]'; for p in $pids; do c=$(cat /proc/$p/comm 2>/dev/null) && echo \"$p $c\"; done; "
"else echo '[no-find]'; fi; "
"echo '[rsyslog]'; { pidof rsyslogd || pgrep -x rsyslogd; } >/dev/null 2>&1 && echo running; "
"echo '[rsyslog-conf]'; cat /etc/rsyslog.conf /etc/rsyslog.d/*.conf 2>/dev/null "
"| sed 's/^/| /'; "
"echo '[journald-conf]'; cat /etc/systemd/journald.conf "
"/usr/lib/systemd/journald.conf.d/*.conf /run/systemd/journald.conf.d/*.conf "
"/etc/systemd/journald.conf.d/*.conf 2>/dev/null "
"| grep -i '^[#[:space:]]*ForwardToSyslog'; "
"printf '%s%s\\n' LOGS_ END"
)
#: ``/proc/900/fd/5`` -- the process and its descriptor.
_FD_RE = re.compile(r"^/proc/(\d+)/fd/\d+$")
#: ``/proc/900/fdinfo/5:flags:<tab>0102001`` -- the descriptor's open flags, octal.
_FLAGS_RE = re.compile(r"^/proc/(\d+)/fdinfo/(\d+):flags:\s*([0-7]+)$")
#: The access mode bits of the open flags: 0 is O_RDONLY, 1 O_WRONLY, 2 O_RDWR.
_ACCESS_MODE = 0o3
#: The engine's file of a container's output: json-file and local drivers.
_STREAM_RE = re.compile(r"^/var/lib/docker/containers/([0-9a-f]{64})/")
#: Files that are no log to read: journald's store, a file already deleted, and
#: the write-ahead log LevelDB and RocksDB name ``000004.log``.
_NOT_A_SOURCE = re.compile(r"^/var/log/journal/|\.journal~?$| \(deleted\)$|/\d{6}\.log$")
_SERVICE_SUFFIX = ".service"
_NO_FIND = "no-find"
class LogSourcesUnavailable(NotImplementedError):
"""The host's ``find`` cannot list open files (busybox); nothing to read, nothing to retry."""
def _frame(output: str) -> list[str]:
lines = [line.strip() for line in strip_terminal_codes(output).splitlines()]
try:
start = lines.index(_BEGIN)
end = lines.index(_END, start)
except ValueError:
raise ValueError("no intact log source report in the output") from None
return [line for line in lines[start + 1 : end] if line]
def _stats(lines: list[str]) -> dict[str, tuple[int, str]]:
"""Each descriptor's size and last write: ``/proc/900/fd/5 2048 1762250200``."""
stats: dict[str, tuple[int, str]] = {}
for line in lines:
fd, _, rest = line.partition(" ")
size, _, mtime = rest.partition(" ")
if size.isdigit() and mtime.isdigit():
stats[fd] = (int(size), datetime.fromtimestamp(int(mtime), UTC).isoformat())
return stats
def _names(lines: list[str]) -> dict[int, str]:
names: dict[int, str] = {}
for line in lines:
pid, _, name = line.partition(" ")
if pid.isdigit() and name:
names[int(pid)] = name
return names
def _journals(units: list[str]) -> list[LogSourceDict]:
services = {u[: -len(_SERVICE_SUFFIX)] for u in units if u.endswith(_SERVICE_SUFFIX)}
return [
{
"kind": "journald",
"path": None,
"unit": unit,
"container_id": None,
"process": None,
"size": None,
"modified": None,
}
for unit in sorted(services)
]
def _held(lines: list[str]) -> list[tuple[int, str, str]]:
"""(pid, descriptor, target) for every log file a process holds, by pid."""
held = []
for line in lines:
fd, _, target = line.partition(" ")
match = _FD_RE.match(fd)
if match and target.startswith("/") and not _NOT_A_SOURCE.search(target):
held.append((int(match.group(1)), fd, target))
return sorted(held)
def _writes(lines: list[str]) -> dict[str, bool]:
"""Whether each descriptor is open for writing, by its ``/proc/<pid>/fd/<fd>``."""
writes: dict[str, bool] = {}
for line in lines:
match = _FLAGS_RE.match(line)
if match:
pid, fd, flags = match.groups()
writes[f"/proc/{pid}/fd/{fd}"] = int(flags, 8) & _ACCESS_MODE != 0
return writes
_Key = tuple[str, str | None, str]
def _holders(lines: list[str], paths: dict[int, str]) -> dict[_Key, list[tuple[int, str]]]:
"""The (pid, descriptor) pairs holding each log, by kind, container and path."""
holders: dict[_Key, list[tuple[int, str]]] = {}
for pid, fd, target in _held(lines):
stream = _STREAM_RE.match(target)
kind, container = (
("container", stream.group(1)) if stream else ("file", _container(paths.get(pid)))
)
holders.setdefault((kind, container, target), []).append((pid, fd))
return holders
def _writer(held: list[tuple[int, str]], writes: dict[str, bool] | None) -> int | None:
"""The first process writing; the first holding for a report without flags,
as an older command printed it."""
if writes is None:
return held[0][0]
return next((pid for pid, fd in held if writes.get(fd)), None)
def _producer(
kind: str, writer: int | None, paths: dict[int, str], names: dict[int, str]
) -> tuple[str | None, str | None]:
"""The writer's service and name; a stream's service is its container."""
if writer is None:
return None, None
return (_unit(paths.get(writer)) if kind == "file" else None), names.get(writer)
def _files(sections: dict[str, list[str]]) -> list[LogSourceDict]:
stats = _stats(sections.get("stat", []))
paths = _cgroup_paths(sections.get("cgroups", []))
names = _names(sections.get("comm", []))
writes = _writes(sections["flags"]) if "flags" in sections else None
found: list[LogSourceDict] = []
for (kind, container, target), held in _holders(sections.get("open", []), paths).items():
writer = _writer(held, writes)
if writer is None and kind == "file" and container is not None:
continue # a reader's view of a mount, inside its container
unit, process = _producer(kind, writer, paths, names)
size, modified = next((stats[fd] for _, fd in held if fd in stats), (None, None))
found.append(
{
"kind": kind,
"path": target,
"unit": unit,
"container_id": container,
"process": process,
"size": size,
"modified": modified,
}
)
return found
def _order(source: LogSourceDict) -> tuple[str, str, str]:
return (source["kind"], source["container_id"] or "", source["path"] or source["unit"] or "")
def parse_log_sources(output: str) -> tuple[bool, list[LogSourceDict]]:
"""Parse what :data:`LOG_SOURCES_COMMAND` printed: whether the host has a
journal, and every source, sorted by kind, container and name.
:raises LogSourcesUnavailable: when the host has no GNU ``find``.
:raises ValueError: when the output carries no intact report.
"""
sections = _sections(_frame(output))
if _NO_FIND in sections:
raise LogSourcesUnavailable("the host's find cannot list open files (no GNU find)")
journald = "journald" in sections
sources = _journals(sections.get("journald", [])) + _files(sections)
return journald, sorted(sources, key=_order)
def _reading(output: str, *, attributed: bool) -> LogSourcesDict:
journald, sources = parse_log_sources(output)
forwarding = parse_log_forwarding(output)
return {
"attributed": attributed,
"journald": journald,
"sources": sources,
"forwarding": forwarding,
}
def parse_log_forwarding(output: str) -> LogForwardingDict | None:
"""Parse where the logs go from what :data:`LOG_SOURCES_COMMAND` printed;
None for a report without the sections, as an older command printed it.
:raises ValueError: when the output carries no intact report.
"""
sections = _sections(_frame(output))
if "rsyslog" not in sections:
return None
config = "\n".join(
line[2:] if line.startswith("| ") else line[1:] for line in sections.get("rsyslog-conf", [])
)
running = "running" in sections["rsyslog"]
return log_forwarding(running, parse_rsyslog(config), sections.get("journald-conf", []))
class LogSourcesMixin:
"""Adds :meth:`get_log_sources` to a driver that can run a command on a Linux host.
The template form (README, "Function classes"): the command and its parse
are the same everywhere, so they are concrete here, and a driver supplies
only :meth:`_run_log_sources_command` -- how a command reaches its host, and
how it gains root there. Mixed in by the drivers that can, so
``hasattr(driver, "get_log_sources")`` stays truthful.
"""
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
def _run_log_sources_command(self, command: str, *, privileged: bool) -> str:
"""Run *command* on the host and return what it printed; as root
when *privileged*. The command is a single ``sh -c`` invocation."""
...
def get_log_sources(self) -> LogSourcesDict:
"""
Returns every log the host writes, with the service or container writing it.
* attributed (bool) - read as root, so every process's files are listed
* journald (bool) - the host keeps systemd's journal
* sources (list) - see :class:`~napalm_device_types.models.LogSourceDict`
* forwarding (dict) - where the logs go beyond the host, see
:class:`~napalm_device_types.log_forwarding.LogForwardingDict`
Example::
{
"attributed": True,
"journald": True,
"sources": [
{"kind": "journald", "path": None, "unit": "nginx",
"container_id": None, "process": None, "size": None, "modified": None},
{"kind": "file", "path": "/var/log/nginx/access.log", "unit": "nginx",
"container_id": None, "process": "nginx", "size": 2048,
"modified": "2026-11-04T09:56:40+00:00"},
],
}
:raises LogSourcesUnavailable: if the host has no GNU ``find``.
:raises ValueError: if neither reading carried an intact report.
"""
command = f"sh -c {quote(LOG_SOURCES_COMMAND)}"
try:
output = self._run_log_sources_command(command, privileged=True)
return _reading(output, attributed=True)
except ValueError:
pass
output = self._run_log_sources_command(command, privileged=False)
return _reading(output, attributed=False)
+18 -68
View File
@@ -5,9 +5,15 @@ These supplement the types defined in napalm.base.models and are used by the
abstract device-type driver classes in this package.
"""
from typing import Literal, get_args
from typing import Literal, NotRequired, get_args
from typing_extensions import NotRequired, TypedDict
from typing_extensions import TypedDict
# The SNMP health profile shapes live in snmp_health_profile.py; re-exported here, where they were.
from napalm_device_types.snmp_health_profile import SnmpHealthProfile as SnmpHealthProfile
from napalm_device_types.snmp_health_profile import SnmpInterfacesSpec as SnmpInterfacesSpec
from napalm_device_types.snmp_health_profile import SnmpMemorySpec as SnmpMemorySpec
from napalm_device_types.snmp_health_profile import SnmpValueSpec as SnmpValueSpec
# ---------------------------------------------------------------------------
# Common (shared across device types)
@@ -79,8 +85,8 @@ class InterruptedPackageDict(TypedDict):
"""A package an interrupted package manager run left halfway."""
name: str
#: dpkg's word for it: "half-installed", "unpacked", "half-configured",
#: "triggers-awaited", "triggers-pending", or "reinstall-required".
#: The manager's word for it: dpkg's and opkg's ("unpacked", "half-configured", ...),
#: rpm's "duplicate", apk's "broken-files" and "broken-script" (``package_managers``).
state: str
@@ -105,9 +111,9 @@ class HostStatusDict(TypedDict):
reboot_reason: str | None
#: True when the host installs updates on its own (unattended-upgrades, dnf-automatic).
auto_updates: bool | None
#: True when a package manager run was cut off and has to be finished
#: (``dpkg --configure -a``) before anything else installs; None when it
#: cannot tell, or a package manager is running right now.
#: The package manager read (dpkg, dnf, yum, apk, opkg; None: none) and whether a run
#: of it was cut off, left halfway (None: cannot tell, or one is running right now).
package_manager: NotRequired[str | None]
package_manager_interrupted: NotRequired[bool | None]
#: The packages that run left halfway; empty when only its journal says so.
interrupted_packages: NotRequired[list[InterruptedPackageDict]]
@@ -1032,15 +1038,6 @@ class ProcessDict(TypedDict):
command: str # full command line
class CronJobDict(TypedDict):
"""A scheduled cron task."""
user: str # owner of the crontab entry
schedule: str # five-field cron expression, e.g. "0 * * * *"
command: str # shell command to execute
description: NotRequired[str] # optional inline comment
class ApplyUpdatesResultDict(TypedDict):
"""Result of an ``apply_updates()`` call."""
@@ -1230,53 +1227,6 @@ class SnmpAgentSettingsDict(TypedDict):
listen: NotRequired[str] # the address the agent binds to; absent: every address
class SnmpValueSpec(TypedDict, total=False):
"""Where an SNMP agent publishes one value, inside a ``SnmpHealthProfile``.
Either ``oids``, scalars asked in order (the first non-empty answer wins), or
``table``, a column walked and folded to one value by ``fold``. ``pattern`` is a
regular expression whose first group is the number, for an agent that answers
with text.
"""
oids: list[str]
table: str
fold: Literal["first", "average", "max"]
pattern: str
class SnmpMemorySpec(TypedDict, total=False):
"""Memory: ``total`` with ``used`` or ``free``, or ``percent`` alone."""
total: SnmpValueSpec
used: SnmpValueSpec
free: SnmpValueSpec
percent: SnmpValueSpec
unit: Literal["KiB", "bytes"] # of total, used and free; KiB, as UCD-MIB counts, by default
class SnmpInterfacesSpec(TypedDict, total=False):
"""How the IF-MIB counters of a device are to be read."""
skip: str # a pattern of interface names whose counters are noise, not signal
tx_errors_are_drops: bool # the device counts dropped packets as TX errors (OPNsense)
class SnmpHealthProfile(TypedDict, total=False):
"""Where a device's SNMP agent publishes its health values.
A driver declares it as ``SNMP_HEALTH_PROFILE`` when its device does not answer
the standard UCD-MIB and IF-MIB; the application polls and computes. What a
profile does not name is read the standard way. Uptime is in TimeTicks, CPU in
percent busy.
"""
uptime: SnmpValueSpec
cpu: SnmpValueSpec
memory: SnmpMemorySpec
interfaces: SnmpInterfacesSpec
class NICConfigDict(TypedDict):
"""Network interface configuration for VM provisioning."""
@@ -1310,17 +1260,17 @@ class NetworkTargetDict(TypedDict):
Distinguishes real bridges (Linux or OVS) from SDN network segments (vnets),
and tells the caller whether a separate ``vlan_tag`` may be applied on top:
- Linux bridge: vlan_aware reflects the bridge's own ``bridge_vlan_aware`` flag.
- OVS bridge: always vlan_aware (OVS bridges tag per-port regardless of a
dedicated "VLAN aware" setting).
- SDN vnet: never vlan_aware — the VLAN is already fixed by the vnet's zone/tag,
so a NIC attached to it must not also carry a ``vlan_tag``. That fixed VLAN
is surfaced via ``fixed_vlan_tag`` instead, for display purposes.
- OVS bridge: always vlan_aware (OVS tags per port, with no "VLAN aware" setting).
- SDN vnet: never vlan_aware — its zone/tag fixes the VLAN (``fixed_vlan_tag``).
``uplinks``: the physical NICs it leaves the host through, one group per way out
(a bond's slaves together, a single NIC alone); no internal, VLAN or virtual ports.
"""
name: str # Bridge, vnet or port group name, usable directly as NICConfigDict.bridge
kind: str # "bridge", "vnet" or "portgroup" (VMware: VLAN fixed like a vnet's)
vlan_aware: bool # True if a NICConfigDict.vlan_tag may be set on top of this target
fixed_vlan_tag: NotRequired[int | None] # vnet only: the VLAN ID already baked into it
uplinks: NotRequired[list[list[str]]] # NIC groups it leaves through; absent = unknown
class StorageTargetDict(TypedDict):
-29
View File
@@ -15,7 +15,6 @@ from typing import TYPE_CHECKING
from napalm_device_types.base import DeviceTypeDriver
from napalm_device_types.health_metrics import HealthMetricsMixin
from napalm_device_types.models import (
CronJobDict,
DeviceActionResultDict,
ProcessDict,
SNMPConfigDict,
@@ -131,34 +130,6 @@ class OSDriver(
"""
...
# ------------------------------------------------------------------
# Cron jobs
# ------------------------------------------------------------------
def get_cron_jobs(self) -> list[CronJobDict]:
"""
Returns scheduled cron tasks from all user crontabs and ``/etc/cron.d``.
Each entry contains:
* user (string) - owner of the crontab entry
* schedule (string) - five-field cron expression (e.g. ``"0 * * * *"``)
* command (string) - shell command to execute
* description (string, optional) - inline comment text if present
Example::
[
{
"user": "root",
"schedule": "0 4 * * *",
"command": "/usr/local/bin/backup.sh",
"description": "nightly backup",
},
]
"""
...
# ------------------------------------------------------------------
# SNMP
# ------------------------------------------------------------------
+207
View File
@@ -0,0 +1,207 @@
"""Whether dnf or yum, apk or opkg was left halfway: the host status's answer for
a host without dpkg (NetOrk/netork#851).
Read the same way as dpkg's state (:mod:`napalm_device_types.host_status`): in the
one host status command, without root, and changing nothing. Each manager is asked
only where it exists, and while one of its processes runs -- from
``/proc/<pid>/comm``, which every Linux has and BusyBox's ``ps`` cannot narrow to --
its packages are halfway on purpose and the answer is unknown.
**dnf and yum** (rpm). A transaction cut off after rpm installed the new version
of a package and before it erased the old one leaves both in the rpm database:
what ``dnf check --duplicates`` and ``package-cleanup --dupes`` report. Packages
dnf installs side by side on purpose do not count: its default ``installonlypkgs``
(``kernel``, ``installonlypkg(kernel)`` and the rest), plus any the host adds in
``dnf.conf``, asked of rpm by what they provide; and ``gpg-pubkey``, one entry
per imported key. The defaults are the ones the host's libdnf was built with, asked
of its Python binding, as a distribution may add its own: Oracle Linux adds
``kernel-uek`` and its parts, whose packages provide nothing of upstream's list.
Upstream's list is the floor where libdnf cannot be asked (dnf 5, yum). Also
interrupted is a transaction that never ended: dnf 4's history (libdnf's ``trans``
table) holds its row with ``state`` 0 until ``finish()`` writes 1 (done) or 2
(error), so the newest row at 0 is one cut off;
yum leaves a ``transaction-all.*`` journal in ``/var/lib/yum``, which
``yum-complete-transaction`` reads (one renamed ``.disabled`` is given up). The
history is read with Python's sqlite3 module in read-only mode, as a stock EL host
has no ``sqlite3`` program and a ``dnf`` command loads every plugin and may reach
for its repositories. dnf 5's history lives elsewhere and is not read; its
duplicates are.
**apk** (Alpine, OpenWrt from 25.12). apk marks an installed package whose files
or scripts failed with ``f:`` in ``/lib/apk/db/installed`` -- ``f`` broken files,
``s`` a broken script; ``x`` (xattrs) and ``S`` (the checksum kind) are no fault --
and those are exactly the packages ``apk fix`` without arguments picks. A run
killed outright leaves no such mark; its next run usually repeats what it left.
**opkg** (OpenWrt before 25.12). ``/usr/lib/opkg/status`` holds
``Status: <want> <flags> <state>`` per package. A run cut off, or a postinst that
failed, leaves ``unpacked`` (or ``half-configured``, ``half-installed``), and the
``reinstreq`` flag asks for a reinstall; ``opkg configure`` finishes every unpacked
package.
What to do about it is not decided here.
"""
from __future__ import annotations
from typing import NamedTuple
from napalm_device_types.models import InterruptedPackageDict
#: dnf's default ``installonlypkgs``: names and what such packages provide.
_INSTALLONLY = (
"kernel",
"kernel-PAE",
"installonlypkg(kernel)",
"installonlypkg(kernel-module)",
"installonlypkg(vm)",
"multiversion(kernel)",
)
#: Where a host adds its own installonly packages, dnf's then yum's.
_DNF_CONF = "/etc/dnf/dnf.conf /etc/yum.conf"
DNF_HISTORY = "/var/lib/dnf/history.sqlite"
YUM_DIR = "/var/lib/yum"
APK_INSTALLED = "/lib/apk/db/installed"
OPKG_STATUS = "/usr/lib/opkg/status"
#: manager -> the process names that leave its packages halfway while they run.
_RUNNING = {
"rpm": frozenset({"rpm", "dnf", "dnf5", "yum", "dnf-automatic", "packagekitd"}),
"apk": frozenset({"apk"}),
"opkg": frozenset({"opkg"}),
}
_UNFINISHED = "0"
_YUM_JOURNAL = "transaction-all."
_YUM_GIVEN_UP = ".disabled"
_APK_FLAGS = (("f", "broken-files"), ("s", "broken-script"))
_OPKG_HALFWAY = frozenset({"unpacked", "half-configured", "half-installed"})
_OPKG_REINSTALL = "reinstreq"
#: The newest transaction in dnf's history as ``<id> <state>``; nothing when unreadable.
#: One literal, with :data:`DNF_HISTORY` spelt out: a fixed query, nothing put into it.
_DNF_HISTORY_PY = (
"import sqlite3; "
"c = sqlite3.connect('file:/var/lib/dnf/history.sqlite?mode=ro', uri=True); "
"r = c.execute('SELECT id, state FROM trans ORDER BY id DESC LIMIT 1').fetchone(); "
"r and print(*r)"
)
#: The ``installonlypkgs`` the host's libdnf (dnf 4) was built with, space-separated;
#: nothing where there is no libdnf. ``ConfigMain`` is held in a name: an option
#: taken from a temporary one outlives it, and reading it then crashes Python.
_LIBDNF_INSTALLONLY_PY = (
"import libdnf.conf; c = libdnf.conf.ConfigMain(); "
"print(c.installonlypkgs().getValueString().replace(',', ' '))"
)
_RUNNING_PART = (
"echo '[pm-running]'; cat /proc/[0-9]*/comm 2>/dev/null | grep -xE '"
+ "|".join(sorted(set().union(*_RUNNING.values())))
+ "' | sort -u; "
)
_RPM_PART = (
"if command -v rpm >/dev/null 2>&1 && { command -v dnf || command -v yum; } >/dev/null 2>&1; "
"then echo '[rpm-frontend]'; { command -v dnf || command -v yum; } | sed 's#.*/##'; "
"rpmpy=; for p in /usr/libexec/platform-python python3; do "
"if command -v $p >/dev/null 2>&1; then rpmpy=$p; break; fi; done; "
f"echo '[rpm-installonly]'; io=\"{' '.join(_INSTALLONLY)} "
f'$([ -n "$rpmpy" ] && $rpmpy -c "{_LIBDNF_INSTALLONLY_PY}" 2>/dev/null) '
f"$(sed -n 's/^[[:space:]]*installonlypkgs[[:space:]]*=//p' {_DNF_CONF} 2>/dev/null "
"| tr ',' ' ')\"; rpm -q --whatprovides $io --qf '%{NAME}\\n' 2>/dev/null "
"| grep -v ' ' | sort -u; "
"echo '[rpm-dupes]'; rpm -qa --qf '%{NAME} %{ARCH}\\n' 2>/dev/null "
"| grep -v '^gpg-pubkey ' | sort | uniq -d | head -n 50; "
f"echo '[yum-journal]'; ls -1 {YUM_DIR} 2>/dev/null | grep '^transaction-all\\.' "
"| head -n 20; "
"echo '[dnf-history]'; "
f'[ -n "$rpmpy" ] && $rpmpy -c "{_DNF_HISTORY_PY}" 2>/dev/null; fi; '
)
_APK_PART = (
f"if command -v apk >/dev/null 2>&1 && [ -r {APK_INSTALLED} ]; then echo '[apk-audit]'; "
"awk '/^P:/ {p = substr($0, 3)} /^f:/ {f = substr($0, 3)} "
'/^$/ {if (f ~ /[fs]/) print f, p; p = ""; f = ""} END {if (f ~ /[fs]/) print f, p}\' '
f"{APK_INSTALLED} | head -n 50; fi; "
)
_OPKG_PART = (
f"if command -v opkg >/dev/null 2>&1 && [ -r {OPKG_STATUS} ]; then echo '[opkg-audit]'; "
"awk '/^Package:/ {p = $2} /^Status:/ && "
"($4 ~ /^(unpacked|half-configured|half-installed)$/ || $3 ~ /reinstreq/) "
f"{{print $3, $4, p}}' {OPKG_STATUS} | head -n 50; fi; "
)
#: The part of the host status command that reads them; POSIX ``sh``, BusyBox too.
PACKAGE_MANAGERS_COMMAND = _RUNNING_PART + _RPM_PART + _APK_PART + _OPKG_PART
class PackageManagerReading(NamedTuple):
"""One package manager's state: halfway or not, None while it is running."""
manager: str | None
interrupted: bool | None
packages: list[InterruptedPackageDict]
def _unfinished_transaction(sections: dict[str, list[str]]) -> bool:
newest = (sections.get("dnf-history") or [""])[0].split()
if len(newest) == 2 and newest[1] == _UNFINISHED:
return True
return any(
name.startswith(_YUM_JOURNAL) and not name.endswith(_YUM_GIVEN_UP)
for name in sections.get("yum-journal") or []
)
def _rpm(sections: dict[str, list[str]], running: set[str]) -> PackageManagerReading | None:
if "rpm-dupes" not in sections:
return None
manager = (sections.get("rpm-frontend") or ["dnf"])[0]
if running & _RUNNING["rpm"]:
return PackageManagerReading(manager, None, [])
installonly = set(sections.get("rpm-installonly") or [])
names = dict.fromkeys(line.split()[0] for line in sections["rpm-dupes"])
packages: list[InterruptedPackageDict] = [
{"name": name, "state": "duplicate"} for name in names if name not in installonly
]
return PackageManagerReading(
manager, bool(packages) or _unfinished_transaction(sections), packages
)
def _apk(sections: dict[str, list[str]], running: set[str]) -> PackageManagerReading | None:
if "apk-audit" not in sections:
return None
if running & _RUNNING["apk"]:
return PackageManagerReading("apk", None, [])
packages: list[InterruptedPackageDict] = []
for line in sections["apk-audit"]:
flags, _, name = line.partition(" ")
name = name.strip()
packages += [
{"name": name, "state": state} for flag, state in _APK_FLAGS if name and flag in flags
]
return PackageManagerReading("apk", bool(packages), packages)
def _opkg_package(line: str) -> InterruptedPackageDict | None:
parts = line.split()
if len(parts) != 3:
return None
flags, state, name = parts
if _OPKG_REINSTALL in flags.split(","):
return {"name": name, "state": "reinstall-required"}
return {"name": name, "state": state} if state in _OPKG_HALFWAY else None
def _opkg(sections: dict[str, list[str]], running: set[str]) -> PackageManagerReading | None:
if "opkg-audit" not in sections:
return None
if running & _RUNNING["opkg"]:
return PackageManagerReading("opkg", None, [])
packages = [p for p in map(_opkg_package, sections["opkg-audit"]) if p is not None]
return PackageManagerReading("opkg", bool(packages), packages)
def parse_package_managers(sections: dict[str, list[str]]) -> list[PackageManagerReading]:
"""What :data:`PACKAGE_MANAGERS_COMMAND` found, one reading per manager it asked."""
running = set(sections.get("pm-running") or [])
readings = (read(sections, running) for read in (_rpm, _apk, _opkg))
return [reading for reading in readings if reading is not None]
+36 -2
View File
@@ -28,9 +28,24 @@ APT_UPGRADABLE_COMMAND = "{ LC_ALL=C apt list --upgradable 2>/dev/null; echo __A
#: Read-only. Lists the packages that a pending security advisory covers. A
#: failed dnf prints nothing, the same as no advisories, so its exit status is
#: printed inside the group, as apt's is.
#: printed inside the group, as apt's is. It reads the cached metadata only, as
#: apt reads its local index: an expired cache otherwise makes dnf download the
#: repositories' metadata first, which can take minutes (#76).
DNF_SECURITY_COMMAND = (
"{ LC_ALL=C dnf updateinfo list --security --quiet 2>/dev/null; echo __DNF_RC=$?; } | cat"
"{ LC_ALL=C dnf --cacheonly updateinfo list --security --quiet 2>/dev/null;"
" echo __DNF_RC=$?; } | cat"
)
#: Read-only. Every advisory whose fix the installed packages carry, one line per
#: CVE it names (``CVE SEVERITY/Sec. NEVRA``): the distribution's own word that a
#: backport is in, where netOrk's knowledge base holds no records of the release
#: (netOrk #2394). dnf 4 only: dnf 5 reads ``--with-cve`` as a filter, and
#: Fedora's advisories reference bugs rather than CVEs. From the cache, as
#: :data:`DNF_SECURITY_COMMAND` reads it; the output is long (90 000 lines on an
#: Oracle Linux 9 host), so a caller reads it seldom.
DNF_INSTALLED_CVES_COMMAND = (
"{ LC_ALL=C dnf --cacheonly updateinfo list --installed --with-cve --quiet 2>/dev/null;"
" echo __DNF_RC=$?; } | cat"
)
# openssl/noble-updates,noble-security 3.0.13-0ubuntu3.6 amd64 [upgradable from: 3.0.13-0ubuntu3.5]
@@ -38,6 +53,7 @@ _APT_LINE = re.compile(r"^(\S+)/(\S+)\s+(\S+)\s+\S+\s+\[upgradable from:\s+(\S+)
_SECURITY_SUITE = "-security"
_APT_STATUS = re.compile(r"^__APT_RC=(\d+)\s*$", re.MULTILINE)
_DNF_STATUS = re.compile(r"^__DNF_RC=(\d+)\s*$", re.MULTILINE)
_CVE_ID = re.compile(r"^CVE-\d{4}-\d{4,}$")
def _joined_lines(output: str) -> list[str]:
@@ -115,3 +131,21 @@ def parse_dnf_security(output: str) -> set[str]:
if len(parts) >= 3 and parts[1].endswith("/Sec."):
names.add(nevra_name(parts[-1]))
return names
def parse_dnf_installed_cves(output: str) -> dict[str, list[str]]:
"""The CVEs the installed advisories close, per package name, each once and sorted.
Parses :data:`DNF_INSTALLED_CVES_COMMAND`'s ``CVE SEVERITY/Sec. NEVRA`` lines;
a line that names no CVE (a key prompt, an advisory id) is no statement.
:raises ValueError: when dnf failed or its exit status never arrived -- "could
not read" must not look like "no advisory installed".
"""
by_name: dict[str, set[str]] = {}
listing = _listing(output, _DNF_STATUS, "dnf updateinfo list --installed --with-cve")
for line in listing.splitlines():
parts = line.split()
if len(parts) >= 3 and _CVE_ID.match(parts[0]):
by_name.setdefault(nevra_name(parts[-1]), set()).add(parts[0])
return {name: sorted(cves) for name, cves in by_name.items()}
+176
View File
@@ -0,0 +1,176 @@
"""A time zone as a device takes it: an IANA name, a POSIX TZ rule, or the rule's parts.
An application names a zone the IANA way (``Europe/Berlin``). Not every device
can: OpenWrt hands ``system.timezone`` to libc as ``TZ``, which wants the rule
itself (``CET-1CEST,M3.5.0,M10.5.0/3``), and a switch wants a UTC offset and the
summer-time rule. A driver translates with these helpers.
The rule is the footer of the zone's TZif file (version 2 and later): the line
after the last transition that says how the zone goes on. OpenWrt's LuCI table is
built from the same footers. The files come from the ``tzdata`` package, a
declared dependency, so every driver reads the same database whatever its host
ships.
"""
from __future__ import annotations
import re
from functools import lru_cache
from importlib.resources import files
from typing import TypedDict
#: An IANA name: letters, digits, ``_+-`` in ``/``-separated parts. No dots, so
#: nothing reaches outside the database.
_ZONE_NAME = re.compile(r"[A-Za-z0-9_+-]+(?:/[A-Za-z0-9_+-]+)*")
_ABBR = r"(?:[A-Za-z]{3,}|<[A-Za-z0-9+-]{3,}>)"
_OFFSET = r"[+-]?\d{1,2}(?::\d{1,2}){0,2}"
_DATE = r"(?:J\d{1,3}|\d{1,3}|M\d{1,2}\.\d\.\d)(?:/[+-]?\d{1,3}(?::\d{1,2}){0,2})?"
#: ``std offset [dst [offset] [,start,end]]`` (POSIX.1, 8.3), and a bare ``UTC`` or
#: ``GMT``, which OpenWrt itself writes by default.
_POSIX_TZ = re.compile(
rf"(?:UTC|GMT)|{_ABBR}{_OFFSET}(?:{_ABBR}(?:{_OFFSET})?(?:,{_DATE},{_DATE})?)?"
)
_PARTS = re.compile(
rf"(?P<std>{_ABBR})(?P<offset>{_OFFSET})"
rf"(?:(?P<dst>{_ABBR})(?P<dst_offset>{_OFFSET})?"
r"(?:,(?P<start>[^,]+),(?P<end>[^,]+))?)?"
)
_TRANSITION = re.compile(r"M(\d{1,2})\.([1-5])\.([0-6])(?:/([+-]?\d{1,3}(?::\d{1,2}){0,2}))?")
#: When a rule names no time of day for a transition: 02:00 local time.
_DEFAULT_TRANSITION_TIME = 7200
class DstTransitionDict(TypedDict):
"""When summer time starts or ends: a weekday of a week of a month, and a time."""
month: int # 1-12
week: int # 1-5; 5 is the month's last such weekday
weekday: int # 0 = Sunday ... 6 = Saturday
time: int # seconds after local midnight, in the time then in force; may be < 0 or > 24h
class PosixTzDict(TypedDict):
"""A POSIX TZ rule in parts (:func:`parse_posix_tz`)."""
std_name: str # "CET"; a quoted name without its <> ("-03")
utc_offset: int # seconds east of UTC in standard time: CET 3600, EST -18000
dst_name: str | None # "CEST"; None for a zone without summer time
dst_offset: int | None # seconds east of UTC in summer time; None without it
dst_start: DstTransitionDict | None # None without summer time or without a rule for it
dst_end: DstTransitionDict | None
class UnknownTimezone(ValueError):
"""A zone that is not in the zone database (nor, where one is taken, a POSIX TZ rule)."""
def __init__(self, zone: str) -> None:
super().__init__(zone)
self.zone = zone
def __str__(self) -> str:
return (
f"unknown time zone {self.zone!r}: not a name of the IANA time zone database"
" such as Europe/Berlin"
)
def is_posix_tz(value: str) -> bool:
"""Whether *value* reads as a POSIX TZ string -- the form libc takes in ``TZ``."""
return bool(_POSIX_TZ.fullmatch(value))
@lru_cache(maxsize=512)
def _footer(zone: str) -> str | None:
"""The POSIX rule at the end of *zone*'s TZif file; None for no such zone."""
if not _ZONE_NAME.fullmatch(zone):
return None
tzfile = files("tzdata").joinpath("zoneinfo", *zone.split("/"))
if not tzfile.is_file():
return None
data = tzfile.read_bytes()
if not data.startswith(b"TZif") or data[4:5] not in (b"2", b"3", b"4") or data[-1:] != b"\n":
return None
footer = data[:-1].rsplit(b"\n", 1)[-1].decode("ascii", "replace")
return footer if is_posix_tz(footer) else None
def is_iana_zone(zone: str) -> bool:
"""Whether *zone* names a zone of the database (``Europe/Berlin``, ``UTC``)."""
return _footer(zone) is not None
def posix_tz(zone: str) -> str | None:
"""The POSIX TZ string for *zone*: an IANA name's rule, a POSIX string as it is.
None when *zone* is neither -- a device given it would run on UTC.
"""
return _footer(zone) or (zone if is_posix_tz(zone) else None)
def _seconds(value: str) -> int:
"""``[+-]hh[:mm[:ss]]`` in seconds."""
sign = -1 if value.startswith("-") else 1
parts = [int(p) for p in value.lstrip("+-").split(":")]
hours, minutes, seconds = (parts + [0, 0])[:3]
return sign * (hours * 3600 + minutes * 60 + seconds)
def _transition(value: str) -> DstTransitionDict | None:
match = _TRANSITION.fullmatch(value)
if match is None:
return None
month, week, weekday, time = match.groups()
return {
"month": int(month),
"week": int(week),
"weekday": int(weekday),
"time": _DEFAULT_TRANSITION_TIME if time is None else _seconds(time),
}
def parse_posix_tz(rule: str) -> PosixTzDict | None:
"""*rule* in parts; None for what is not a rule, or one with Julian-day transitions.
POSIX counts offsets west of UTC (``CET-1``); the parts count them east, as a
device's "UTC+01:00" does. Summer time without an offset of its own is one hour
ahead of standard time.
"""
if rule in ("UTC", "GMT"):
rule += "0"
match = _PARTS.fullmatch(rule)
if match is None or not is_posix_tz(rule):
return None
utc_offset = -_seconds(match["offset"])
parts: PosixTzDict = {
"std_name": match["std"].strip("<>"),
"utc_offset": utc_offset,
"dst_name": None,
"dst_offset": None,
"dst_start": None,
"dst_end": None,
}
if match["dst"] is None:
return parts
parts["dst_name"] = match["dst"].strip("<>")
parts["dst_offset"] = (
utc_offset + 3600 if match["dst_offset"] is None else -_seconds(match["dst_offset"])
)
if match["start"] is not None:
start, end = _transition(match["start"]), _transition(match["end"])
if start is None or end is None:
return None
parts["dst_start"], parts["dst_end"] = start, end
return parts
__all__ = [
"DstTransitionDict",
"PosixTzDict",
"UnknownTimezone",
"is_iana_zone",
"is_posix_tz",
"parse_posix_tz",
"posix_tz",
]
+27 -2
View File
@@ -3,8 +3,9 @@ What provisioning a VM from a cloud image needs to know about the guest.
The same for every hypervisor: which agent a guest runs so a QEMU-based
hypervisor can read its IP, the network-config cloud-init and FreeBSD's
nuageinit both read, and how a packed cloud image is unpacked. How a
hypervisor attaches, boots and talks to the VM stays in its driver.
nuageinit both read, how a packed cloud image is unpacked, and how a failure
names the VM it left behind. How a hypervisor attaches, boots and talks to
the VM stays in its driver.
"""
from __future__ import annotations
@@ -37,6 +38,30 @@ _DECOMPRESSORS = {
}
class VMProvisionIncomplete(RuntimeError):
"""``create_vm_from_cloud_init`` failed after the VM was created.
The VM is still on the hypervisor, half built: ``vmid`` and ``node`` name it
as ``VMProvisionResultDict`` would have, so the caller can remove it with
``destroy_vm``. The failure itself is the ``__cause__`` (``raise ... from``),
and the message is its own. A failure before the VM exists raises as before.
"""
def __init__(self, message: str, *, vmid: str, node: str) -> None:
super().__init__(message)
self.vmid = vmid
self.node = node
def __reduce__(self) -> tuple[Any, ...]:
# The keyword-only arguments are not in ``args``, which is all that
# pickle passes back by default.
return (_vm_provision_incomplete, (str(self), self.vmid, self.node))
def _vm_provision_incomplete(message: str, vmid: str, node: str) -> VMProvisionIncomplete:
return VMProvisionIncomplete(message, vmid=vmid, node=node)
def network_config(nics: list[tuple[str, bool]]) -> dict[str, Any] | None:
"""Network-config v2: DHCP on every ``(mac, dhcp)`` NIC that asks for it.
+69
View File
@@ -0,0 +1,69 @@
"""The backups running on a device right now.
A restart aborts every backup that runs on the device it restarts: a
hypervisor's backup of its guests, a backup server's backup, sync, verify or
garbage collection, a host's own file-level backup (restic, borg and the like).
A guest's restart breaks the backup its hypervisor is taking of it. So an
application that restarts a device asks first, right before the restart.
Declared under ``if TYPE_CHECKING``: a driver mixes this in when it can tell;
``hasattr`` tells a caller whether it did. No role base carries it, and a driver
without it cannot tell -- which is not the same as "nothing runs".
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Literal, TypedDict, get_args
#: What a running backup does: the backup itself, or one of a backup server's
#: jobs (``napalm_device_types.backups.BackupJobKind``).
RunningBackupKind = Literal["backup", "sync", "verify", "gc", "prune", "tape"]
RUNNING_BACKUP_KINDS: tuple[str, ...] = get_args(RunningBackupKind)
class RunningBackupDict(TypedDict):
"""One backup running on the device (``get_running_backups``)."""
kind: RunningBackupKind
job: str | None # the job it runs for, as the platform names it; None: started by hand
task: str | None # this run's id on the platform: a task id, a unit, a process id
guests: list[str] # the guests (get_vms' ids) it is backing up now; [] for the host's own
since: int | None # unix seconds it started; None when the platform does not say
class RunningBackupsMixin:
if TYPE_CHECKING:
def get_running_backups(self) -> list[RunningBackupDict]:
"""
Returns the backups running on the device now; ``[]`` when none is.
Every entry runs on this device: restarting the device aborts it.
A hypervisor names in ``guests`` the guests a backup is taking right
now -- a guest that is locked for its backup counts, with or without
a task the platform still shows -- so the application can tell a
guest's own restart not to break it.
Example::
[
{
"kind": "backup",
"job": "backup-3f2a9c1e-77d1",
"task": "UPID:pve1:0001F2A3:00C4B1D2:6731A0C0:vzdump::root@pam:",
"guests": ["101"],
"since": 1793660400,
},
{
"kind": "gc",
"job": "store1",
"task": "UPID:localhost:000004D2:...:garbage_collection:store1:root@pam:",
"guests": [],
"since": 1793664000,
},
]
:raises Exception: when the device could not be read; never an
empty list for a failed read.
"""
...
+196
View File
@@ -0,0 +1,196 @@
"""Scheduled jobs: what a host runs on its own schedule, and switching it on, off and now.
A host runs work on its own: cron lines, run-parts scripts, anacron, systemd
timers, FreeBSD's ``periodic``, a firewall's or a NAS's own scheduled jobs, the
Windows Task Scheduler. :class:`ScheduledJobsMixin` reads all of them as one
shape, :class:`~napalm_device_types.models.ScheduledJobDict`, and acts on them.
Declared under ``if TYPE_CHECKING`` like a role: nothing exists at runtime until
a driver implements it. :class:`~napalm_device_types.scheduled_jobs_posix.PosixScheduledJobsMixin`
is the implementation for a host reached through the command channel.
**Who owns a definition** is on every job (``managed_by``): ``package`` (installed
by one), ``admin`` (written on the host), ``platform`` (generated by the device's
own configuration -- OPNsense, OpenMediaVault, QTS, Windows), ``application`` (written
through :meth:`put_scheduled_job`, under the ``name`` the application gave it).
What the application makes of that is the application's.
**What may be done is on every job too** (``actions``): every job a driver can
switch takes ``enable``, ``disable`` and ``run``; only an ``application`` job takes
``edit`` and ``delete``. A driver never edits or removes a job it did not write.
**Never over a change it did not see.** Every write names the ``digest`` the caller
saw. The driver reads the job again right before it writes and raises
:class:`ScheduledJobChanged` when the definition differs.
"""
from __future__ import annotations
import hashlib
import json
import re
from typing import TYPE_CHECKING, Any, NotRequired, TypedDict
from napalm_device_types.cron import is_reboot, parse_cron
#: What :meth:`ScheduledJobsMixin.manage_scheduled_job` takes.
ACTIONS = ("enable", "disable", "run")
#: What a job's ``actions`` may list beyond :data:`ACTIONS`.
EDIT = "edit"
DELETE = "delete"
KINDS = ("cron", "run_parts", "anacron", "systemd_timer", "periodic", "task")
MANAGED_BY = ("package", "admin", "platform", "application")
#: Seconds a run now may take on the host before ``timeout`` stops it.
RUN_TIMEOUT = 300
#: A job's name becomes a file and a unit name: lower case, digits, dashes.
_NAME = re.compile(r"[a-z0-9][a-z0-9-]{0,62}")
#: A user name as useradd and the BSDs take it.
_USER = re.compile(r"[A-Za-z_][A-Za-z0-9_.-]{0,31}\$?")
#: The parts of a job that are its definition; what the host says of its runs is not.
_DEFINITION = ("kind", "source", "run_as", "schedule", "command", "enabled")
class ScheduledJobDict(TypedDict):
"""A job the host runs on its own schedule (``get_scheduled_jobs``).
A cron line, a run-parts script, an anacron job, a systemd timer, a FreeBSD
``periodic`` script, a platform's own job, a Windows task -- one shape for all
of them (``napalm_device_types.scheduled_jobs``).
"""
key: str # stable name on this host: "timer:apt-daily.timer", "cron:/etc/crontab:1a2b3c4d"
kind: str # cron | run_parts | anacron | systemd_timer | periodic | task
source: str # the file, unit file or configuration it is defined in
run_as: str # the user it runs as
schedule: str # as the host has it: "*/5 * * * *", "OnCalendar=daily", "daily"
schedule_kind: str # cron | calendar | monotonic | period | triggers
command: str # as the host has it
enabled: bool
managed_by: str # package | admin | platform | application
digest: str # job_digest(): the definition as it is now
actions: list[str] # what the driver can do to this job: enable, disable, run, edit, delete
name: NotRequired[str] # the name an application gave a job it wrote (put_scheduled_job)
description: NotRequired[str]
last_run: NotRequired[str | None] # ISO 8601, UTC; where the host knows it
last_result: NotRequired[str | None] # "success", or what the host says went wrong
next_run: NotRequired[str | None] # ISO 8601, UTC; where the host knows it
class ScheduledJobsReadingDict(TypedDict):
"""Every scheduled job of a host, and the time zone its schedules are in."""
jobs: list[ScheduledJobDict]
time_zone: str | None # IANA name ("Europe/Berlin"); None when the host does not say
class ScheduledJobSpecDict(TypedDict):
"""A job an application wants on the host (``put_scheduled_job``)."""
name: str # the job's name on the host, chosen by the application: [a-z0-9][a-z0-9-]*
schedule: str # a cron expression (napalm_device_types.cron), "@reboot" included
run_as: str
command: str # one or more lines of POSIX sh; a platform may take its own (OPNsense: configd)
enabled: bool
description: NotRequired[str]
class ScheduledJobResultDict(TypedDict):
"""What an action on a scheduled job did."""
success: bool
output: str # what the host said; for a run, what the job printed
class ScheduledJobChanged(Exception): # noqa: N818 - a state, as the callers word it
"""The job's definition differs from the digest the caller saw: nothing was written."""
class ScheduledJobRefused(ValueError):
"""The job cannot take what was asked of it, or the request itself is not valid."""
def job_digest(job: dict[str, Any], *, content: str = "") -> str:
"""The definition of *job* in 16 hex characters.
Over kind, source, run_as, schedule, command and enabled -- so enabling a job
changes it -- and *content*: what the definition points at that the job's
fields do not show, such as a run-parts script's checksum.
"""
parts = [job.get(field) for field in _DEFINITION] + [content]
return hashlib.sha256(json.dumps(parts).encode()).hexdigest()[:16]
def check_job_name(name: str) -> None:
""":raises ScheduledJobRefused: for a name a file or a unit could not carry safely."""
if not _NAME.fullmatch(name or ""):
raise ScheduledJobRefused(f"not a job name: {name!r} (a-z, 0-9 and -, at most 63)")
def check_spec(spec: ScheduledJobSpecDict) -> None:
""":raises ScheduledJobRefused: for a spec no host could be given as it is."""
check_job_name(spec["name"])
if not _USER.fullmatch(spec.get("run_as") or ""):
raise ScheduledJobRefused(f"not a user name: {spec.get('run_as')!r}")
if not (spec.get("command") or "").strip():
raise ScheduledJobRefused("a job needs a command")
if "\n" in (spec.get("description") or ""):
raise ScheduledJobRefused("a description is one line")
if not is_reboot(spec["schedule"]):
try:
parse_cron(spec["schedule"])
except ValueError as exc:
raise ScheduledJobRefused(str(exc)) from exc
class ScheduledJobsMixin:
"""Declares the scheduled-jobs contract; a driver that can implements it.
A verb a driver cannot do at all is ``None`` on the driver, as ``manage_service``
is; a verb one job cannot take is missing from that job's ``actions``.
"""
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
def get_scheduled_jobs(self) -> ScheduledJobsReadingDict:
"""Every scheduled job of the host.
:raises ValueError: when the reading came back cut short -- never a
shorter list.
"""
...
def manage_scheduled_job(
self, key: str, action: str, *, digest: str
) -> ScheduledJobResultDict:
"""Enable, disable or run the job *key* now.
:raises ScheduledJobChanged: when the job is no longer as *digest* says.
:raises ScheduledJobRefused: when the job does not take *action*.
:raises LookupError: when the host has no job *key*.
"""
...
def put_scheduled_job(
self, spec: ScheduledJobSpecDict, *, digest: str | None = None
) -> ScheduledJobResultDict:
"""Create the job ``spec["name"]`` (*digest* None) or change it.
:raises ScheduledJobChanged: when the job is no longer as *digest*
says, or when a job of that name is there although *digest* is None.
:raises ScheduledJobRefused: for a spec :func:`check_spec` refuses, or
one the host cannot take.
"""
...
def delete_scheduled_job(self, key: str, *, digest: str) -> ScheduledJobResultDict:
"""Delete the job *key*, which an application wrote.
:raises ScheduledJobChanged: when the job is no longer as *digest* says.
:raises ScheduledJobRefused: when the job is not one an application wrote.
:raises LookupError: when the host has no job *key*.
"""
...
+493
View File
@@ -0,0 +1,493 @@
"""Scheduled jobs on a POSIX host, over the command channel: cron and systemd timers.
:class:`PosixScheduledJobsMixin` implements
:class:`~napalm_device_types.scheduled_jobs.ScheduledJobsMixin` for a driver that has
:meth:`~napalm_device_types.channel.CommandChannelMixin.run_command`. The driver
mixes it in and sets the paths its host keeps jobs in; the defaults are Debian's.
**Reading** is one framed report as root (:mod:`napalm_device_types._scheduled_reading`).
**Switching** a cron or anacron line comments it out behind ``#napalm-disabled: ``
and takes exactly that marker away again; a run-parts script loses or gets its
execute bit; a timer is ``systemctl disable --now`` / ``enable --now``. A timer that
cannot be switched (``static``) only runs.
**Writing** stages the new text as the login user and installs it as root with one
fixed command: ``crontab -u <user> <file>`` for a user's crontab, ``install`` with
the file's own mode and owner for a system file. Root never reads stdin. Before any
write the job is read again and compared with the digest the caller saw.
**An application's own job** (:meth:`put_scheduled_job`) is a timer on a host with
systemd -- ``<name>.timer`` and ``<name>.service`` in :attr:`UNIT_DIR`, the command
in :attr:`JOB_SCRIPT_DIR` ``/<name>`` behind a marker and its cron expression --
else a file in :attr:`OWN_CRON_D_DIR`, else marked lines in the user's crontab.
**Running now** is bounded by :data:`~napalm_device_types.scheduled_jobs.RUN_TIMEOUT`:
a timer's unit by ``systemctl start`` and its journal since then, a cron line's
command as its user the way cron runs it (``_crontab.run_script``).
"""
from __future__ import annotations
import fnmatch
import shlex
from collections.abc import Sequence
from typing import TYPE_CHECKING, ClassVar
from napalm_device_types._crontab import (
JOB_MARKER,
CronLine,
marked_lines,
parse_anacrontab,
parse_crontab,
run_script,
switched,
with_job,
without_job,
)
from napalm_device_types._scheduled_reading import (
OWNERS_SH,
SCHEDULE_HEADER,
TIME_ZONE_SH,
Paths,
Reading,
parse_reading,
reading_command,
)
from napalm_device_types.cron import is_reboot, to_on_calendar
from napalm_device_types.scheduled_jobs import (
ACTIONS,
RUN_TIMEOUT,
ScheduledJobChanged,
ScheduledJobRefused,
ScheduledJobResultDict,
ScheduledJobsMixin,
ScheduledJobSpecDict,
ScheduledJobsReadingDict,
check_spec,
)
if TYPE_CHECKING:
from napalm_device_types.channel import CommandResult
_STAGE = 't=$(mktemp) && chmod 644 "$t" && cat > "$t" && printf %s "$t"'
#: What a run keeps of the job's output.
_OUTPUT_LIMIT = 64 * 1024
_READ_TIMEOUT = 120
def _command(argv: Sequence[str]) -> str:
return shlex.join(argv)
def _said(result: CommandResult) -> str:
return "\n".join(p for p in (result.stdout.strip(), result.stderr.strip()) if p)[
-_OUTPUT_LIMIT:
]
def _result(result: CommandResult, done: str) -> ScheduledJobResultDict:
if result.exit_code != 0:
return {"success": False, "output": _said(result) or f"exit {result.exit_code}"}
return {"success": True, "output": _said(result) or done}
class PosixScheduledJobsMixin(ScheduledJobsMixin):
"""Scheduled jobs of a host reached through ``run_command``; the paths are the driver's."""
CRON_SYSTEM_FILES: ClassVar[tuple[str, ...]] = ("/etc/crontab",)
CRON_D_DIRS: ClassVar[tuple[str, ...]] = ("/etc/cron.d",)
CRON_SPOOL_DIRS: ClassVar[tuple[str, ...]] = (
"/var/spool/cron/crontabs",
"/var/spool/cron",
"/etc/crontabs",
)
RUN_PARTS_DIRS: ClassVar[tuple[tuple[str, str], ...]] = (
("hourly", "/etc/cron.hourly"),
("daily", "/etc/cron.daily"),
("weekly", "/etc/cron.weekly"),
("monthly", "/etc/cron.monthly"),
)
ANACRONTAB: ClassVar[str] = "/etc/anacrontab"
JOB_SCRIPT_DIR: ClassVar[str] = "/usr/local/libexec/scheduled-jobs"
UNIT_DIR: ClassVar[str] = "/etc/systemd/system"
#: Where a host without systemd takes an application's job as a file, if the directory
#: is there; else, or None, the user's crontab.
OWN_CRON_D_DIR: ClassVar[str | None] = "/etc/cron.d"
ROOT_GROUP: ClassVar[str] = "root"
#: Prints "<octal mode> <owner> <group>" of the path appended.
STAT_ARGV: ClassVar[tuple[str, ...]] = ("stat", "-c", "%a %U %G", "--")
#: ``install`` and how it makes missing directories (FreeBSD's ``-D`` is a destdir).
INSTALL: ClassVar[tuple[str, ...]] = ("install", "-D")
#: ``sh`` that reads definition paths on stdin and prints ``<package>: <path>``.
OWNERS_SH: ClassVar[str] = OWNERS_SH
#: ``sh`` that prints the time zone: an IANA name, or a path ending in ``zoneinfo/<name>``.
TIME_ZONE_SH: ClassVar[str] = TIME_ZONE_SH
#: ``fnmatch`` patterns on a job's ``source``: what the platform generates is ``platform``.
PLATFORM_SOURCES: ClassVar[tuple[str, ...]] = ()
#: Run as root after a user's crontab was written; for a host that keeps it elsewhere too.
CRONTAB_AFTER_WRITE: ClassVar[tuple[str, ...] | None] = None
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
def run_command(
self,
command: str,
*,
privileged: bool = False,
timeout: float = 60,
stdin: bytes | None = None,
) -> CommandResult: ...
# -- reading ---------------------------------------------------------------------------------
def _sj_paths(self) -> Paths:
return Paths(
system_files=self.CRON_SYSTEM_FILES,
cron_d_dirs=self.CRON_D_DIRS,
spool_dirs=self.CRON_SPOOL_DIRS,
run_parts=self.RUN_PARTS_DIRS,
anacrontab=self.ANACRONTAB,
script_dir=self.JOB_SCRIPT_DIR,
owners_sh=self.OWNERS_SH,
time_zone_sh=self.TIME_ZONE_SH,
)
def _sj_read(self) -> Reading:
result = self.run_command(
reading_command(self._sj_paths()), privileged=True, timeout=_READ_TIMEOUT
)
reading = parse_reading(result.stdout)
for job in reading.jobs:
source = str(job["source"])
if job["managed_by"] != "application" and any(
fnmatch.fnmatchcase(source, p) for p in self.PLATFORM_SOURCES
):
job["managed_by"] = "platform"
return reading
def get_scheduled_jobs(self) -> ScheduledJobsReadingDict:
"""Every scheduled job of the host, in one round trip.
:raises ValueError: when the report came back cut short.
"""
reading = self._sj_read()
return {"jobs": reading.jobs, "time_zone": reading.time_zone} # type: ignore[typeddict-item]
def _sj_job(self, reading: Reading, key: str, digest: str) -> dict[str, object]:
job = next((j for j in reading.jobs if j["key"] == key), None)
if job is None:
raise LookupError(f"no scheduled job {key}")
if job["digest"] != digest:
raise ScheduledJobChanged(f"{key} changed on the host since it was read")
return job
# -- commands --------------------------------------------------------------------------------
def _sj_root(self, argv: Sequence[str], timeout: float = 60) -> CommandResult:
return self.run_command(_command(argv), privileged=True, timeout=timeout)
def _sj_checked(self, argv: Sequence[str]) -> CommandResult:
result = self._sj_root(argv)
if result.exit_code != 0:
raise RuntimeError(f"{_command(argv)}: {_said(result) or result.exit_code}")
return result
def _sj_stage(self, content: str) -> str:
result = self.run_command(_STAGE, timeout=30, stdin=content.encode())
path = result.stdout.strip().splitlines()[-1:] or [""]
if result.exit_code != 0 or not path[0].startswith("/"):
raise RuntimeError(f"cannot stage a file: {_said(result)}")
return path[0]
def _sj_install(self, content: str, path: str, mode: str, owner: str, group: str) -> None:
staged = self._sj_stage(content)
try:
self._sj_checked(
[*self.INSTALL, "-m", mode, "-o", owner, "-g", group, "--", staged, path]
)
finally:
self.run_command(_command(["rm", "-f", "--", staged]), timeout=30)
# -- crontab files ---------------------------------------------------------------------------
def _sj_spool_user(self, path: str) -> str | None:
"""The user whose crontab *path* is; None for a system file."""
directory, _, user = path.rpartition("/")
return user if directory in self.CRON_SPOOL_DIRS else None
def _sj_crontab(self, user: str) -> str:
result = self._sj_root(["crontab", "-u", user, "-l"])
if result.exit_code != 0:
if "no crontab" in (result.stderr + result.stdout).lower():
return ""
raise RuntimeError(f"cannot read the crontab of {user}: {_said(result)}")
return result.stdout
def _sj_text(self, path: str) -> str:
user = self._sj_spool_user(path)
if user is not None:
return self._sj_crontab(user)
return self._sj_checked(["cat", "--", path]).stdout
def _sj_write_crontab(self, user: str, text: str) -> None:
staged = self._sj_stage(text)
try:
self._sj_checked(["crontab", "-u", user, staged])
finally:
self.run_command(_command(["rm", "-f", "--", staged]), timeout=30)
if self.CRONTAB_AFTER_WRITE:
self._sj_checked(self.CRONTAB_AFTER_WRITE)
def _sj_write(self, path: str, text: str) -> None:
user = self._sj_spool_user(path)
if user is not None:
self._sj_write_crontab(user, text)
return
# Through a symbolic link to the file it names, with that file's mode: the
# link's own mode is 777, and cron ignores a file anyone may write.
real = self._sj_checked(["readlink", "-f", "--", path]).stdout.strip() or path
mode, owner, group = self._sj_checked([*self.STAT_ARGV, real]).stdout.split()[:3]
self._sj_install(text, real, mode, owner, group)
def _sj_line(self, job: dict[str, object], text: str) -> CronLine:
"""The job's line in *text* as it is now, read again for the write."""
path = str(job["source"])
if job["kind"] == "anacron":
found = parse_anacrontab(text)
else:
user = self._sj_spool_user(path)
found = parse_crontab(path, text, system=user is None, user=user or "")
line = next((ln for ln in found if ln.key == job["key"]), None)
if line is None:
raise ScheduledJobChanged(f"{job['key']} is no longer in {path}")
return line
# -- switching and running -------------------------------------------------------------------
def _sj_timer_unit(self, job: dict[str, object]) -> str:
key = str(job["key"])
return f"{job['name']}.timer" if key.startswith("job:") else key.removeprefix("timer:")
def _sj_switch(self, job: dict[str, object], enabled: bool) -> ScheduledJobResultDict:
word = "enabled" if enabled else "disabled"
if job["kind"] == "systemd_timer":
verb = "enable" if enabled else "disable"
return _result(
self._sj_root(["systemctl", verb, "--now", "--", self._sj_timer_unit(job)]), word
)
if job["kind"] == "run_parts":
return _result(
self._sj_root(["chmod", "+x" if enabled else "-x", "--", str(job["source"])]), word
)
path = str(job["source"])
text = self._sj_text(path)
self._sj_write(path, switched(text, self._sj_line(job, text), enabled=enabled))
return {"success": True, "output": f"{job['key']} {word}"}
def _sj_as(self, user: str, script: str) -> list[str]:
"""*script* run by ``sh`` as *user*, from root."""
if user == "root":
return ["sh", "-c", script]
return ["su", "-s", "/bin/sh", user, "-c", script]
def _sj_run(self, job: dict[str, object], reading: Reading) -> ScheduledJobResultDict:
bound = ["timeout", str(RUN_TIMEOUT)]
wait = RUN_TIMEOUT + 30
if job["kind"] == "systemd_timer":
unit = reading.units[str(job["key"])]
since = self.run_command("date +%s", timeout=30).stdout.strip()
started = self._sj_root([*bound, "systemctl", "start", "--", unit], timeout=wait)
journal = self._sj_root(
[
"journalctl",
"-u",
unit,
"--since",
f"@{since}",
"--no-pager",
"-o",
"cat",
"-n",
"200",
]
)
output = "\n".join(p for p in (_said(started), journal.stdout.strip()) if p)
return {"success": started.exit_code == 0, "output": output or f"{unit} ran"}
if job["kind"] == "run_parts":
return _result(self._sj_root([*bound, str(job["source"])], timeout=wait), "ran")
path = str(job["source"])
line = self._sj_line(job, self._sj_text(path))
script = run_script(line.command, line.env)
return _result(
self._sj_root([*bound, *self._sj_as(line.run_as, script)], timeout=wait), "ran"
)
def manage_scheduled_job(self, key: str, action: str, *, digest: str) -> ScheduledJobResultDict:
"""Enable, disable or run the job *key* now; see :class:`ScheduledJobsMixin`."""
if action not in ACTIONS:
raise ScheduledJobRefused(f"unknown action {action!r}; allowed: {list(ACTIONS)}")
reading = self._sj_read()
job = self._sj_job(reading, key, digest)
if action not in job["actions"]: # type: ignore[operator]
raise ScheduledJobRefused(f"{key} cannot be told to {action} on this host")
if action == "run":
return self._sj_run(job, reading)
return self._sj_switch(job, action == "enable")
# -- an application's own jobs ---------------------------------------------------------------
def _sj_own(
self, reading: Reading, spec: ScheduledJobSpecDict, digest: str | None
) -> dict[str, object] | None:
current = next((j for j in reading.jobs if j["key"] == f"job:{spec['name']}"), None)
if digest is None:
if current is not None:
raise ScheduledJobChanged(f"a job {spec['name']} is there already")
return None
if current is None:
raise LookupError(f"no scheduled job job:{spec['name']}")
if current["digest"] != digest:
raise ScheduledJobChanged(f"job:{spec['name']} changed on the host since it was read")
return current
def _sj_group(self, user: str) -> str:
result = self.run_command(_command(["id", "-gn", "--", user]), timeout=30)
if result.exit_code != 0:
raise ScheduledJobRefused(f"the host has no user {user}")
return result.stdout.strip()
def _sj_calendar(self, schedule: str) -> list[str]:
if is_reboot(schedule):
return ["OnBootSec=1min"]
lines = to_on_calendar(schedule)
for line in lines:
checked = self.run_command(
_command(["systemd-analyze", "calendar", "--", line]), timeout=30
)
if checked.exit_code != 0:
raise ScheduledJobRefused(f"systemd does not take {line!r}: {_said(checked)}")
return [f"OnCalendar={line}" for line in lines]
def _sj_put_timer(self, spec: ScheduledJobSpecDict) -> None:
name, description = spec["name"], spec.get("description") or spec["name"]
script = f"{self.JOB_SCRIPT_DIR}/{name}"
unit_description = description.replace("%", "%%")
timer = "\n".join(
[
"[Unit]",
f"Description={unit_description}",
"",
"[Timer]",
*self._sj_calendar(spec["schedule"]),
f"Unit={name}.service",
"",
"[Install]",
"WantedBy=timers.target",
"",
]
)
service = "\n".join(
[
"[Unit]",
f"Description={unit_description}",
"",
"[Service]",
"Type=oneshot",
f"User={spec['run_as']}",
f"ExecStart={script}",
"",
]
)
body = f"#!/bin/sh\n{JOB_MARKER}{name} {spec.get('description') or ''}".rstrip()
body += f"\n{SCHEDULE_HEADER}{spec['schedule']}\n{spec['command'].strip()}\n"
self._sj_install(body, script, "750", "root", self._sj_group(spec["run_as"]))
self._sj_install(service, f"{self.UNIT_DIR}/{name}.service", "644", "root", self.ROOT_GROUP)
self._sj_install(timer, f"{self.UNIT_DIR}/{name}.timer", "644", "root", self.ROOT_GROUP)
self._sj_checked(["systemctl", "daemon-reload"])
verb = "enable" if spec["enabled"] else "disable"
self._sj_checked(["systemctl", verb, "--now", "--", f"{name}.timer"])
def _sj_cron_text(self, spec: ScheduledJobSpecDict, *, system: bool) -> list[str]:
if "\n" in spec["command"].strip():
raise ScheduledJobRefused("a cron line holds one line of command")
owner = f" {spec['run_as']}" if system else ""
line = f"{spec['schedule']}{owner} {spec['command'].strip()}"
return marked_lines(
spec["name"], spec.get("description") or "", line, enabled=spec["enabled"]
)
def _sj_cron_d(self) -> str | None:
""":attr:`OWN_CRON_D_DIR` where the host has it; BusyBox crond reads none."""
directory = self.OWN_CRON_D_DIR
if directory is None:
return None
found = self.run_command(_command(["test", "-d", directory]), timeout=30)
return directory if found.exit_code == 0 else None
def _sj_put_cron(self, spec: ScheduledJobSpecDict, current: dict[str, object] | None) -> None:
if self._sj_cron_d() is not None:
self._sj_group(spec["run_as"])
text = "\n".join(self._sj_cron_text(spec, system=True)) + "\n"
self._sj_install(
text, f"{self.OWN_CRON_D_DIR}/{spec['name']}", "644", "root", self.ROOT_GROUP
)
return
if current is not None and current["run_as"] != spec["run_as"]:
self._sj_delete_cron(current)
user = spec["run_as"]
text = self._sj_crontab(user)
jobs = parse_crontab("", text, system=False, user=user)
self._sj_write_crontab(
user, with_job(text, jobs, spec["name"], self._sj_cron_text(spec, system=False))
)
def put_scheduled_job(
self, spec: ScheduledJobSpecDict, *, digest: str | None = None
) -> ScheduledJobResultDict:
"""Create or change the application's own job; see :class:`ScheduledJobsMixin`."""
check_spec(spec)
reading = self._sj_read()
current = self._sj_own(reading, spec, digest)
if reading.systemd:
self._sj_put_timer(spec)
else:
self._sj_put_cron(spec, current)
return {
"success": True,
"output": f"job:{spec['name']} {'changed' if current else 'created'}",
}
def _sj_delete_cron(self, job: dict[str, object]) -> None:
path = str(job["source"])
if self._sj_spool_user(path) is None:
self._sj_checked(["rm", "-f", "--", path])
return
text = self._sj_text(path)
jobs = parse_crontab(path, text, system=False, user=str(job["run_as"]))
self._sj_write(path, without_job(text, jobs, str(job["name"])))
def delete_scheduled_job(self, key: str, *, digest: str) -> ScheduledJobResultDict:
"""Delete the application's job *key*; see :class:`ScheduledJobsMixin`."""
job = self._sj_job(self._sj_read(), key, digest)
if job["managed_by"] != "application":
raise ScheduledJobRefused(
f"{key} was not written by an application; it is only switched"
)
name = str(job["name"])
if job["kind"] != "systemd_timer":
self._sj_delete_cron(job)
return {"success": True, "output": f"{key} deleted"}
self._sj_root(["systemctl", "disable", "--now", "--", f"{name}.timer"])
self._sj_checked(
[
"rm",
"-f",
"--",
f"{self.UNIT_DIR}/{name}.timer",
f"{self.UNIT_DIR}/{name}.service",
f"{self.JOB_SCRIPT_DIR}/{name}",
]
)
self._sj_checked(["systemctl", "daemon-reload"])
return {"success": True, "output": f"{key} deleted"}
@@ -0,0 +1,60 @@
"""The shape a driver states its device's SNMP health OIDs in (``SNMP_HEALTH_PROFILE``).
Apart from ``models.py``, which is over the file size limit; ``models`` still exports them.
"""
from typing import Literal
from typing_extensions import TypedDict
class SnmpValueSpec(TypedDict, total=False):
"""Where an SNMP agent publishes one value, inside a ``SnmpHealthProfile``.
Either ``oids``, scalars asked in order (the first non-empty answer wins), or
``table``, a column walked and folded to one value by ``fold``. ``pattern`` is a
regular expression whose first group is the number, for an agent that answers
with text.
``unknown`` says the agent answers this value wrongly, or not at all, where it
should: the application reports no value -- neither the standard reading nor a
default. It stands alone, without ``oids`` or ``table``.
"""
oids: list[str]
table: str
fold: Literal["first", "average", "max"]
pattern: str
unknown: bool
class SnmpMemorySpec(TypedDict, total=False):
"""Memory: ``total`` with ``used`` or ``free``, or ``percent`` alone."""
total: SnmpValueSpec
used: SnmpValueSpec
free: SnmpValueSpec
percent: SnmpValueSpec
unit: Literal["KiB", "bytes"] # of total, used and free; KiB, as UCD-MIB counts, by default
class SnmpInterfacesSpec(TypedDict, total=False):
"""How the IF-MIB counters of a device are to be read."""
skip: str # a pattern of interface names whose counters are noise, not signal
tx_errors_are_drops: bool # the device counts dropped packets as TX errors (OPNsense)
class SnmpHealthProfile(TypedDict, total=False):
"""Where a device's SNMP agent publishes its health values.
A driver declares it as ``SNMP_HEALTH_PROFILE`` when its device does not answer
the standard UCD-MIB and IF-MIB; the application polls and computes. What a
profile does not name is read the standard way, and what it marks ``unknown`` is
not read at all. Uptime is in TimeTicks, CPU in percent busy.
"""
uptime: SnmpValueSpec
cpu: SnmpValueSpec
memory: SnmpMemorySpec
interfaces: SnmpInterfacesSpec
+57 -2
View File
@@ -27,6 +27,11 @@ A caller that passes neither keeps working as before (first use, every time),
and a driver that ignores both still connects for a caller that passes them:
old and new versions on either side do not break each other.
A connection sshd turns away before its banner -- OpenSSH 9.8+ does that to a
source it has just penalised -- is tried again after a pause
(:func:`retry_refused`, :data:`REFUSED_RETRY_WAITS`); :meth:`HostKeyCheck.connect`
does so itself, a netmiko driver wraps its ``ConnectHandler`` in it.
:class:`HostKeyCheck` is the one implementation, for a paramiko ``SSHClient``
(:meth:`~HostKeyCheck.connect`) and for netmiko (:meth:`~HostKeyCheck.netmiko`,
then :meth:`~HostKeyCheck.observe`). Like :mod:`napalm_device_types.channel` it
@@ -40,7 +45,8 @@ import hashlib
import logging
import os
import tempfile
from collections.abc import Iterator, Mapping
import time
from collections.abc import Callable, Iterator, Mapping
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Any
@@ -54,6 +60,13 @@ KNOWN_KEYS_ARG = "ssh_host_keys"
#: The ``optional_args`` key that carries the callable every seen key goes to.
OBSERVER_ARG = "ssh_host_key_observer"
#: The waits before each new attempt when sshd turns a connection away before its
#: banner. OpenSSH 9.8+ does that to every connection from a source it has just
#: penalised (``PerSourcePenalties``) -- one whose connections did not
#: authenticate, a scanner's probes -- for seconds to minutes. 26 s in all
#: outlasts the 16 s penalty seen on a new Ubuntu 26.04 host (NetOrk/netork#1800).
REFUSED_RETRY_WAITS = (1.0, 3.0, 7.0, 15.0)
#: The netmiko arguments this module decides; a caller's own values give way.
_NETMIKO_HOST_KEY_ARGS = ("ssh_strict", "system_host_keys", "alt_host_keys", "alt_key_file")
@@ -185,6 +198,48 @@ def _refused_key(exc: BaseException) -> Any | None:
return None
def refused_before_banner(exc: BaseException) -> bool:
"""Whether *exc*, or what it wraps, is sshd closing the connection before its banner.
paramiko says "Error reading SSH protocol banner" (netmiko and drivers wrap
it), with the connection's EOF or reset underneath. A failed login, a
silent host and a closed port are something else: waiting does not end them.
"""
seen: set[int] = set()
stack: list[BaseException | None] = [exc]
while stack:
current = stack.pop()
if current is None or id(current) in seen:
continue
seen.add(id(current))
if "protocol banner" in str(current):
return True
stack.extend([current.__cause__, current.__context__])
return False
def retry_refused[T](
attempt: Callable[[], T],
*,
waits: tuple[float, ...] = REFUSED_RETRY_WAITS,
sleep: Callable[[float], None] | None = None,
) -> T:
"""*attempt*, again after each of *waits* while sshd turns it away before its banner.
Any other error, and the last refusal, go to the caller unchanged.
"""
pause = sleep or time.sleep
for wait in waits:
try:
return attempt()
except Exception as exc:
if not refused_before_banner(exc):
raise
logger.info("SSH connection turned away before its banner; again in %s s", wait)
pause(wait)
return attempt()
class HostKeyCheck:
"""How one driver connection verifies its host's key.
@@ -220,7 +275,7 @@ class HostKeyCheck:
"""``client.connect()`` to this host with *kwargs*, verified and reported."""
self.prepare(client)
with self.refusing():
client.connect(hostname=self.host, port=self.port, **kwargs)
retry_refused(lambda: client.connect(hostname=self.host, port=self.port, **kwargs))
return self.observe(client.get_transport())
# ------------------------------------------------------------------
+305 -51
View File
@@ -15,7 +15,9 @@ accident.
``/etc/ssh/sshd_config.d/*.conf`` -- Debian and Ubuntu do, at the top -- the
lists go to :data:`DROPIN_PATH`, which sorts first and so wins. sshd_config
stays as the distribution shipped it, and removing the file undoes the change.
Only without such an ``Include`` is sshd_config itself rewritten, after a copy.
Only without such an ``Include`` is sshd_config itself rewritten, after a copy --
and not at all on a platform that generates it
(:attr:`OpenSshServerMixin.SSHD_CONFIG_GENERATED`).
**Nobody is locked out, by three checks.**
@@ -30,10 +32,19 @@ Only without such an ``Include`` is sshd_config itself rewritten, after a copy.
3. After the reload, *verify* -- the caller's -- opens a new session. The
running one survives a reload, so if the new one cannot get in, this one
puts the old state back and reloads again.
**The managing client is exempt from per-source penalties** where sshd has them
(OpenSSH 9.8+, :meth:`OpenSshServerMixin.exempt_from_ssh_penalties`). sshd
penalises a source whose connections do not authenticate -- a scanner's probes
-- and then drops every connection from it, the managing session's included.
The address is the one the host sees (``$SSH_CONNECTION``); it joins the
host's own ``PerSourcePenaltyExemptList`` in :data:`PENALTY_DROPIN_PATH`, with
the same three checks. An older sshd is not touched: it refuses the directive.
"""
from __future__ import annotations
import ipaddress
import time
from collections.abc import Callable, Iterable, Mapping, Sequence
from dataclasses import dataclass, field
@@ -42,6 +53,8 @@ from typing import (
TYPE_CHECKING,
)
from napalm_device_types.ssh_server_auth import SSHD_EFFECTIVE_COMMAND, SshServerAuthMixin
if TYPE_CHECKING: # pragma: no cover - for type checkers only
from napalm_device_types.channel import CommandResult
@@ -57,6 +70,9 @@ MAIN_CONFIG = "/etc/ssh/sshd_config"
DROPIN_DIR = "/etc/ssh/sshd_config.d"
#: ``00-`` so it sorts before every other drop-in: sshd keeps the first value it reads.
DROPIN_PATH = f"{DROPIN_DIR}/00-netork-ssh-hardening.conf"
#: Where :meth:`OpenSshServerMixin.exempt_from_ssh_penalties` names the managing
#: client; ``00-`` like the hardening drop-in, since sshd keeps the first value.
PENALTY_DROPIN_PATH = f"{DROPIN_DIR}/00-netork-penalty-exempt.conf"
#: What marks a line this module disabled in sshd_config.
DISABLED_PREFIX = "# netork: "
@@ -65,6 +81,11 @@ _HEADER = (
"# taken out of what this server offers. Delete this file to undo it.\n"
)
_PENALTY_HEADER = (
"# Written by netOrk: the address netOrk manages this host from, exempt from\n"
"# sshd's per-source penalties (PerSourcePenalties). Delete this file to undo it.\n"
)
#: Host key type, as ``ssh-keygen -l`` prints it -> the algorithms that key serves.
#: Certificate algorithms are left out: they also need a HostCertificate.
_KEY_TYPE_ALGORITHMS: dict[str, tuple[str, ...]] = {
@@ -302,6 +323,56 @@ def render_main_config(text: str, keep: Mapping[str, Sequence[str]], kinds: Iter
return render_dropin(keep, wanted) + "\n".join(out) + ("\n" if body.endswith("\n") else "")
def parse_penalty_exemptions(text: str) -> list[str] | None:
"""The ``persourcepenaltyexemptlist`` from ``sshd -T`` output; None without penalties.
Per-source penalties came with OpenSSH 9.8. An older sshd prints neither
line and refuses ``PerSourcePenaltyExemptList`` outright -- it does not
start with it in its configuration -- so None means: write nothing.
"""
penalties = False
exempt: list[str] = []
for raw in text.splitlines():
parts = raw.strip().split(None, 1)
if not parts:
continue
word = parts[0].lower()
if word == "persourcepenalties":
penalties = True
elif word == "persourcepenaltyexemptlist" and len(parts) == 2:
value = parts[1].strip()
exempt = [] if value.lower() == "none" else [a for a in value.split(",") if a]
return exempt if penalties else None
def client_address(ssh_connection: str) -> str | None:
"""The client's address in ``$SSH_CONNECTION`` (``client port server port``), or None."""
parts = ssh_connection.split()
if len(parts) != 4:
return None
try:
return str(ipaddress.ip_address(parts[0]))
except ValueError:
return None
def is_exempt(address: str, exempt: Iterable[str]) -> bool:
"""Whether *address* is one of *exempt*'s addresses or inside one of its networks."""
ip = ipaddress.ip_address(address)
for entry in exempt:
try:
if ip in ipaddress.ip_network(entry, strict=False):
return True
except ValueError:
continue
return False
def render_penalty_dropin(exempt: Sequence[str]) -> str:
"""The drop-in stating the full exempt list."""
return _PENALTY_HEADER + f"PerSourcePenaltyExemptList {','.join(exempt)}\n"
@dataclass
class SshHardeningResult:
"""What :meth:`OpenSshServerMixin.harden_ssh_server` did.
@@ -319,7 +390,39 @@ class SshHardeningResult:
message: str = ""
class OpenSshServerMixin:
@dataclass
class SshPenaltyExemption:
"""What :meth:`OpenSshServerMixin.exempt_from_ssh_penalties` did.
*address* is the one the host sees the managing client at. *applied* is
True only when sshd uses the exemption and a new session got in;
*rolled_back* says the host was changed and then put back.
"""
address: str | None = None
applied: bool = False
rolled_back: bool = False
path: str | None = None
message: str = ""
#: Why an algorithm stays, by :class:`Refusal` reason.
_WHYS = {
"last": "it is the last one",
"client": "netOrk could no longer connect",
"host_keys_unknown": "the host keys could not be read",
}
def _unchanged(
result: SshPenaltyExemption, message: str, say: Callable[[str], None]
) -> SshPenaltyExemption:
result.message = message
say(message)
return result
class OpenSshServerMixin(SshServerAuthMixin):
"""Adds :meth:`harden_ssh_server` to a driver that can run commands as root on an OpenSSH host.
Mixed in by the drivers whose host runs a stock OpenSSH, not by a base class,
@@ -327,9 +430,19 @@ class OpenSshServerMixin:
whose platform writes sshd_config itself (an appliance that regenerates it
from its own database) must not inherit this, or must opt out with
``harden_ssh_server = None``: its file would be overwritten on the next
apply.
apply. Where that generated file still includes the drop-in directory, the
driver sets :attr:`SSHD_CONFIG_GENERATED` instead and keeps the hardening.
It carries :class:`~napalm_device_types.ssh_server_auth.SshServerAuthMixin`:
a host whose sshd can be hardened can also say how it lets people log in.
"""
#: True where the platform writes sshd_config from its own database
#: (OpenMediaVault): only the drop-in directory is written, and a
#: sshd_config that does not include it is left alone, since the platform
#: would write over a change to it on its next apply.
SSHD_CONFIG_GENERATED: bool = False
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
def run_command(
@@ -405,7 +518,7 @@ class OpenSshServerMixin:
return parse_sshd_effective(self._sshd_effective_text())
def _sshd_effective_text(self) -> str:
result = self._ssh_run("PATH=$PATH:/usr/sbin:/sbin sshd -T")
result = self._ssh_run(SSHD_EFFECTIVE_COMMAND)
if result.exit_code != 0:
raise RuntimeError(f"sshd -T failed: {(result.stderr or result.stdout).strip()}")
return result.stdout
@@ -453,24 +566,7 @@ class OpenSshServerMixin:
:raises RuntimeError: if the current state cannot be read.
"""
say = log or (lambda _line: None)
say("Reading the algorithms sshd offers (sshd -T)")
sshd_t = self._sshd_effective_text()
host_keys: set[str] | None = None
if remove.get("hostkey"):
say("Reading which host keys the server has (ssh-keygen -l)")
host_keys = self._ssh_host_key_algorithms(sshd_t)
plan = plan_ssh_hardening(
parse_sshd_effective(sshd_t), remove, client=client, host_keys=host_keys
)
whys = {
"last": "it is the last one",
"client": "netOrk could no longer connect",
"host_keys_unknown": "the host keys could not be read",
}
for r in plan.refused:
last_key = r.reason == "last" and r.kind == "hostkey"
why = "it is the last one the host has a key for" if last_key else whys[r.reason]
say(f"Keeping {r.algorithm}: {why}")
plan = self._plan_hardening(remove, client, say)
kinds = plan.changed_classes
result = SshHardeningResult(plan=plan)
if not kinds:
@@ -483,8 +579,57 @@ class OpenSshServerMixin:
main = self._ssh_read(MAIN_CONFIG)
if main is None:
raise RuntimeError(f"cannot read {MAIN_CONFIG}")
if self.SSHD_CONFIG_GENERATED and not has_dropin_include(main):
result.message = (
f"Not changed: {MAIN_CONFIG} does not include {DROPIN_DIR}, and this"
" platform writes it from its own settings."
)
say(result.message)
return result
restore = self._write_change(plan, main, result, say)
return self._apply_or_restore(plan, kinds, result, restore, verify, say)
return self._apply_or_restore(
result,
restore,
verify,
say,
ignored=lambda: self._algorithms_ignored(plan, result.path),
done="The server no longer offers the selected algorithms.",
)
def _plan_hardening(
self,
remove: Mapping[str, Iterable[str]],
client: Mapping[str, Iterable[str]] | None,
say: Callable[[str], None],
) -> SshHardeningPlan:
"""Read what sshd offers and which host keys it has; plan, and say what stays."""
say("Reading the algorithms sshd offers (sshd -T)")
sshd_t = self._sshd_effective_text()
host_keys: set[str] | None = None
if remove.get("hostkey"):
say("Reading which host keys the server has (ssh-keygen -l)")
host_keys = self._ssh_host_key_algorithms(sshd_t)
plan = plan_ssh_hardening(
parse_sshd_effective(sshd_t), remove, client=client, host_keys=host_keys
)
for r in plan.refused:
last_key = r.reason == "last" and r.kind == "hostkey"
why = "it is the last one the host has a key for" if last_key else _WHYS[r.reason]
say(f"Keeping {r.algorithm}: {why}")
return plan
def _algorithms_ignored(self, plan: SshHardeningPlan, path: str | None) -> str | None:
"""Why sshd does not use the new lists; None when it does."""
now = self.get_ssh_server_algorithms()
names = [
SSH_ALGORITHM_CLASSES[k][1] for k in plan.changed_classes if now[k] != plan.keep[k]
]
if not names:
return None
return (
f"sshd does not use the new {', '.join(names)}: a setting read before"
f" {path} decides them"
)
def _write_change(
self,
@@ -520,43 +665,33 @@ class OpenSshServerMixin:
return restore_main
def _apply_or_restore(
def _apply_or_restore[R: (SshHardeningResult, SshPenaltyExemption)](
self,
plan: SshHardeningPlan,
kinds: list[str],
result: SshHardeningResult,
result: R,
restore: Callable[[], None],
verify: Callable[[], bool] | None,
say: Callable[[str], None],
) -> SshHardeningResult:
def back(message: str, reload: bool) -> SshHardeningResult:
say(message)
say("Putting the previous configuration back")
try:
restore()
if reload:
self._reload_ssh_server()
except Exception as exc: # noqa: BLE001 - report it, the caller must look
result.message = f"{message}; putting the previous configuration back failed: {exc}"
say(result.message)
return result
result.rolled_back = True
result.message = message
return result
*,
ignored: Callable[[], str | None],
done: str,
) -> R:
"""Check what was written, reload, verify; put the old state back on any failure.
*ignored* reads ``sshd -T`` after ``sshd -t`` and says why sshd does not
use the change, or None when it does; *done* is the message of success.
"""
def back(message: str, reload: bool) -> R:
return self._put_back(result, restore, say, message, reload=reload)
try:
say("Checking the configuration (sshd -t)")
complaint = self._sshd_accepts()
if complaint:
return back(f"sshd rejected the new configuration: {complaint}", reload=False)
now = self.get_ssh_server_algorithms()
ignored = [SSH_ALGORITHM_CLASSES[k][1] for k in kinds if now[k] != plan.keep[k]]
if ignored:
return back(
f"sshd does not use the new {', '.join(ignored)}: a setting read before"
f" {result.path} decides them",
reload=False,
)
why = ignored()
if why:
return back(why, reload=False)
say("Reloading sshd")
self._reload_ssh_server()
if verify is not None:
@@ -566,6 +701,125 @@ class OpenSshServerMixin:
except Exception as exc: # noqa: BLE001 - whatever went wrong, the host goes back
return back(f"The change failed: {exc}", reload=True)
result.applied = True
result.message = "The server no longer offers the selected algorithms."
result.message = done
say(result.message)
return result
def _put_back[R: (SshHardeningResult, SshPenaltyExemption)](
self,
result: R,
restore: Callable[[], None],
say: Callable[[str], None],
message: str,
*,
reload: bool,
) -> R:
"""Say *message*, put the previous state back, and say whether that worked."""
say(message)
say("Putting the previous configuration back")
try:
restore()
if reload:
self._reload_ssh_server()
except Exception as exc: # noqa: BLE001 - report it, the caller must look
result.message = f"{message}; putting the previous configuration back failed: {exc}"
say(result.message)
return result
result.rolled_back = True
result.message = message
return result
# ── the penalty exemption ───────────────────────────────────────────
def exempt_from_ssh_penalties(
self,
*,
verify: Callable[[], bool] | None = None,
log: Callable[[str], None] | None = None,
) -> SshPenaltyExemption:
"""Exempt the address this session comes from from sshd's per-source penalties.
OpenSSH 9.8+ penalises a source whose connections do not authenticate --
a scanner's probes -- and then drops every connection from it, the
managing client's included (NetOrk/netork#1800). The address is the
one the host sees (``$SSH_CONNECTION``), so NAT and relays do not
matter. The host's own exemptions stay: the new list is the one sshd
uses now plus this address.
Only into the drop-in directory: a sshd_config that does not include it
is left alone, and the result says so. An sshd before 9.8 is not
touched -- it would not start with the directive. Checked like
:meth:`harden_ssh_server`: ``sshd -t``, ``sshd -T`` shows the address,
reload, then *verify*; on any failure the previous state goes back.
:raises RuntimeError: if the session names no client address, or
``sshd -T`` or sshd_config cannot be read.
"""
say = log or (lambda _line: None)
address = self._session_client_address()
result = SshPenaltyExemption(address=address)
say(f"netOrk connects from {address}")
say("Reading sshd's per-source penalties (sshd -T)")
exempt = parse_penalty_exemptions(self._sshd_effective_text())
reason = self._penalty_exemption_unchanged(address, exempt)
if reason is not None or exempt is None:
return _unchanged(result, reason or "", say)
restore = self._write_penalty_dropin([*exempt, address])
result.path = PENALTY_DROPIN_PATH
say(f"Wrote {PENALTY_DROPIN_PATH}")
return self._apply_or_restore(
result,
restore,
verify,
say,
ignored=lambda: self._exemption_ignored(address),
done=f"{address} is exempt from sshd's per-source penalties.",
)
def _session_client_address(self) -> str:
"""The address the host sees this session come from (``$SSH_CONNECTION``)."""
session = self.run_command('printf %s "$SSH_CONNECTION"', privileged=False, timeout=30)
address = client_address(session.stdout)
if address is None:
raise RuntimeError("the session names no client address ($SSH_CONNECTION)")
return address
def _penalty_exemption_unchanged(self, address: str, exempt: list[str] | None) -> str | None:
"""Why the exemption is left as it is; None when it is to be written."""
if exempt is None:
return "Nothing to change: this sshd has no per-source penalties (before OpenSSH 9.8)."
if is_exempt(address, exempt):
return f"Nothing to change: {address} is already exempt."
main = self._ssh_read(MAIN_CONFIG)
if main is None:
raise RuntimeError(f"cannot read {MAIN_CONFIG}")
if not has_dropin_include(main):
return (
f"Not changed: {MAIN_CONFIG} does not include {DROPIN_DIR},"
" where the exemption would go."
)
return None
def _exemption_ignored(self, address: str) -> str | None:
"""Why sshd does not use the new exemption; None when it does."""
if is_exempt(address, parse_penalty_exemptions(self._sshd_effective_text()) or []):
return None
return (
"sshd does not use the new PerSourcePenaltyExemptList: a setting read before"
f" {PENALTY_DROPIN_PATH} decides it"
)
def _write_penalty_dropin(self, exempt: Sequence[str]) -> Callable[[], None]:
"""Write the exemption list; return what puts the previous drop-in back."""
previous = self._ssh_read(PENALTY_DROPIN_PATH)
self._ssh_run(f"mkdir -p -- {quote(DROPIN_DIR)}")
self._ssh_write(PENALTY_DROPIN_PATH, render_penalty_dropin(exempt))
def restore() -> None:
if previous is None:
self._ssh_remove(PENALTY_DROPIN_PATH)
else:
self._ssh_write(PENALTY_DROPIN_PATH, previous)
return restore
+281
View File
@@ -0,0 +1,281 @@
"""How an OpenSSH server lets people log in: the effective settings and sshd's PAM stack.
Whether a host takes a password, only a key, or a key *and* a second factor is
the same question on every host that runs OpenSSH, so the reading lives here,
and a driver only mixes in :class:`SshServerAuthMixin` and supplies
``run_command``.
**What sshd uses, not what a file says.** ``sshd -T`` prints the settings after
sshd's defaults, ``sshd_config``, every ``Include`` and first-match-wins -- the
same reading :class:`~napalm_device_types.ssh_server.OpenSshServerMixin` plans
its hardening on, by the same fixed command (:data:`SSHD_EFFECTIVE_COMMAND`). It
needs root: without it sshd cannot load its host keys and exits.
**PAM, as far as it can be read.** With ``UsePAM yes``, keyboard-interactive
asks whatever PAM's ``auth`` stack for sshd asks: a password from
``pam_unix.so``, a one-time code from an OTP module, or both. The stack is read
from ``/etc/pam.d`` in one listing as the login user (the files are
world-readable), and followed through ``@include``, ``include`` and
``substack``. Each module is reported with its control and arguments as
written; which of them is a second factor, and whether ``nullok`` lets a user
without one through, is the consumer's call.
Not read: ``Match`` blocks (``sshd -T`` without ``-C`` prints the global
settings), and what a PAM module's own configuration asks for.
"""
from __future__ import annotations
import re
from collections.abc import Mapping
from typing import TYPE_CHECKING, TypedDict
if TYPE_CHECKING: # pragma: no cover - for type checkers only
from napalm_device_types.channel import CommandResult
#: sshd's effective configuration. ``sshd`` lives in ``/usr/sbin``, outside a login
#: user's ``PATH``. One fixed command, run as root.
SSHD_EFFECTIVE_COMMAND = "PATH=$PATH:/usr/sbin:/sbin sshd -T"
PAM_DIR = "/etc/pam.d"
#: Every file in ``/etc/pam.d``, each after a ``==> path <==`` header (GNU, BSD and
#: busybox ``tail`` alike). One fixed command, as the login user.
PAM_FILES_COMMAND = f"tail -n +1 -- {PAM_DIR}/*"
#: The PAM service sshd runs as.
PAM_SERVICE = "sshd"
_TIMEOUT = 30
_HEADER_RE = re.compile(rf"^==> {re.escape(PAM_DIR)}/(?P<name>[^/]+) <==$")
#: A service name an include may name: a file in ``/etc/pam.d``, never a path out of it.
_SERVICE_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
_INCLUDE_CONTROLS = ("include", "substack")
class PamAuthEntryDict(TypedDict):
"""One line of a PAM ``auth`` stack: ``auth required pam_unix.so nullok``."""
#: The module's file name, without its directory: ``pam_unix.so``.
module: str
#: ``required``, ``requisite``, ``sufficient``, ``optional``, or a bracketed
#: ``[value=action ...]`` as written.
control: str
args: list[str]
class SshServerAuthDict(TypedDict):
"""How an OpenSSH server lets people log in, as ``get_ssh_server_auth`` read it.
The global settings ``sshd -T`` prints. A ``Match`` block can change them for
some users or addresses; ``sshd -T`` without ``-C`` does not show that.
``authentication_methods`` is ``AuthenticationMethods``: each inner list is one
way in, every method of it required in turn (``publickey,keyboard-interactive``
-> ``["publickey", "keyboard-interactive"]``); empty for ``any``, where any one
enabled method is enough. A method keeps its submethod
(``keyboard-interactive:pam``).
``use_pam`` is None where sshd has no PAM (OpenBSD). ``pam_auth_stack`` is the
``auth`` stack PAM runs for sshd, includes and substacks followed; None when
PAM is off or its files gave nothing for sshd.
"""
password_authentication: bool
#: ``KbdInteractiveAuthentication``; before OpenSSH 8.7, either it or
#: ``ChallengeResponseAuthentication``.
kbd_interactive_authentication: bool
pubkey_authentication: bool
hostbased_authentication: bool
#: False also where sshd was built without GSSAPI.
gssapi_authentication: bool
authentication_methods: list[list[str]]
#: ``yes``, ``prohibit-password`` (``without-password``), ``forced-commands-only`` or ``no``.
permit_root_login: str
use_pam: bool | None
pam_auth_stack: list[PamAuthEntryDict] | None
def _flag(settings: Mapping[str, str], keyword: str) -> bool:
return settings.get(keyword, "no") == "yes"
def _settings(text: str) -> dict[str, str]:
"""``keyword value`` per line, the first one of each keyword."""
found: dict[str, str] = {}
for raw in text.splitlines():
parts = raw.strip().split(None, 1)
if len(parts) == 2:
found.setdefault(parts[0].lower(), parts[1].strip())
return found
def _methods(value: str) -> list[list[str]]:
if value == "any":
return []
return [[m for m in alternative.split(",") if m] for alternative in value.split()]
def parse_sshd_auth(text: str) -> SshServerAuthDict:
"""The authentication settings from ``sshd -T`` output; ``pam_auth_stack`` left None.
:raises ValueError: if a setting every OpenSSH prints is missing -- not sshd -T
output, or cut short.
"""
settings = _settings(text)
missing = [
k
for k in ("passwordauthentication", "pubkeyauthentication", "permitrootlogin")
if k not in settings
]
if missing:
raise ValueError(f"sshd -T printed no {', '.join(missing)}")
use_pam = settings.get("usepam")
return {
"password_authentication": _flag(settings, "passwordauthentication"),
"kbd_interactive_authentication": _flag(settings, "kbdinteractiveauthentication")
or _flag(settings, "challengeresponseauthentication"),
"pubkey_authentication": _flag(settings, "pubkeyauthentication"),
"hostbased_authentication": _flag(settings, "hostbasedauthentication"),
"gssapi_authentication": _flag(settings, "gssapiauthentication"),
"authentication_methods": _methods(settings.get("authenticationmethods", "any")),
"permit_root_login": settings["permitrootlogin"],
"use_pam": None if use_pam is None else use_pam == "yes",
"pam_auth_stack": None,
}
def _pam_tokens(line: str) -> list[str]:
"""A PAM line's tokens, a bracketed control (``[success=1 default=ignore]``) as one."""
tokens: list[str] = []
for part in line.split():
if tokens and tokens[-1].startswith("[") and not tokens[-1].endswith("]"):
tokens[-1] += " " + part
else:
tokens.append(part)
return tokens
def _pam_entry(line: str) -> PamAuthEntryDict | None:
tokens = _pam_tokens(line.split("#", 1)[0])
if len(tokens) >= 2 and tokens[0] == "@include":
tokens = ["auth", "include", tokens[1]]
if len(tokens) < 3 or tokens[0].lstrip("-") != "auth":
return None
control, target, args = tokens[1], tokens[2], tokens[3:]
if control in _INCLUDE_CONTROLS:
if not _SERVICE_RE.match(target):
return None
return {"module": target, "control": control, "args": []}
return {"module": target.rsplit("/", 1)[-1], "control": control, "args": args}
def parse_pam_auth(text: str) -> list[PamAuthEntryDict]:
"""One PAM file's ``auth`` stack, in order.
A module by its file name (``pam_unix.so``), with its control and arguments.
An include is an entry whose control is ``include`` or ``substack`` and whose
module is the service it names: ``@include common-auth`` brings in a whole
file, ``auth include`` and ``auth substack`` its ``auth`` lines; both are
followed by :func:`pam_auth_stack`. An include that names a path rather than
a service is left out.
"""
return [entry for line in text.splitlines() if (entry := _pam_entry(line.strip()))]
def parse_pam_files(text: str) -> dict[str, str]:
""":data:`PAM_FILES_COMMAND`'s output, by file name."""
files: dict[str, list[str]] = {}
current: list[str] | None = None
for line in text.splitlines():
header = _HEADER_RE.match(line.strip())
if header:
current = files.setdefault(header["name"], [])
elif current is not None:
current.append(line)
return {name: "\n".join(lines).strip("\n") + "\n" for name, lines in files.items()}
def pam_auth_stack(
files: Mapping[str, str], service: str = PAM_SERVICE
) -> list[PamAuthEntryDict] | None:
"""*service*'s ``auth`` modules, includes and substacks followed, in stack order.
A file included twice -- or in a loop -- counts once; an include of a file
that is not there adds nothing, as with PAM. None without a file for *service*.
"""
if service not in files:
return None
stack: list[PamAuthEntryDict] = []
seen: set[str] = set()
def walk(name: str) -> None:
if name in seen or name not in files:
return
seen.add(name)
for entry in parse_pam_auth(files[name]):
if entry["control"] in _INCLUDE_CONTROLS:
walk(entry["module"])
else:
stack.append(entry)
walk(service)
return stack
class SshServerAuthMixin:
"""Adds :meth:`get_ssh_server_auth` to a driver that can run commands on an OpenSSH host.
Mixed in by the drivers whose host runs a stock OpenSSH --
:class:`~napalm_device_types.ssh_server.OpenSshServerMixin` carries it -- so
``hasattr(driver, "get_ssh_server_auth")`` stays a truthful answer. Reading,
not writing: a platform that generates sshd_config itself can carry it too.
"""
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
def run_command(
self,
command: str,
*,
privileged: bool = False,
timeout: float = 60,
stdin: bytes | None = None,
) -> CommandResult: ...
def get_ssh_server_auth(self) -> SshServerAuthDict:
"""
Returns how sshd lets people log in, from ``sshd -T`` and PAM's sshd stack.
Example::
{
"password_authentication": False,
"kbd_interactive_authentication": True,
"pubkey_authentication": True,
"hostbased_authentication": False,
"gssapi_authentication": False,
"authentication_methods": [["publickey", "keyboard-interactive"]],
"permit_root_login": "no",
"use_pam": True,
"pam_auth_stack": [
{"module": "pam_google_authenticator.so", "control": "required", "args": []}
],
}
:raises RuntimeError: if ``sshd -T`` fails -- no OpenSSH, or no root.
"""
result = self.run_command(SSHD_EFFECTIVE_COMMAND, privileged=True, timeout=_TIMEOUT)
if result.exit_code != 0:
raise RuntimeError(f"sshd -T failed: {(result.stderr or result.stdout).strip()}")
auth = parse_sshd_auth(result.stdout)
if auth["use_pam"]:
auth["pam_auth_stack"] = self._pam_auth_stack()
return auth
def _pam_auth_stack(self) -> list[PamAuthEntryDict] | None:
"""sshd's PAM auth stack; None when ``/etc/pam.d`` gave nothing for sshd.
``tail`` goes on past a file it cannot open and exits 1, so what it did
print is used whatever the exit status.
"""
result = self.run_command(PAM_FILES_COMMAND, privileged=False, timeout=_TIMEOUT)
return pam_auth_stack(parse_pam_files(result.stdout))
+312
View File
@@ -0,0 +1,312 @@
"""The sudo password an earlier driver left in shell histories, and taking it back.
A driver that typed ``echo '<password>' | sudo -S -p "" <command>`` into an
interactive shell left that line, password and all, in the login user's shell
history, once per command -- and where only the part before the first ``&&``
ran as root, it left files root should own to the login user. A driver that has
stopped doing so still leaves both behind on every host it ever reached.
:class:`SudoHistoryMixin` reads the first as a fact and takes both back. The
commands are the same on every POSIX host, so they live here once; a driver
mixes it in, supplies ``run_command`` and says what it once left behind:
:attr:`~SudoHistoryMixin.SUDO_HISTORY_MARKER` and
:attr:`~SudoHistoryMixin.SUDO_ROOT_FILES`.
**The password never travels.** Lines are counted with ``grep -c`` and deleted
with ``sed``, both matching the marker only, as the login user who owns the
history. Nothing of a line's content is read back, and nothing goes to stdin.
**What is deleted is gone from the disk too, as far as a file can say.** ``sed
-i`` with a backup suffix renames the old file to the backup and writes a new
one; the backup -- the old file's blocks -- is overwritten with ``shred -u``,
or removed with ``rm`` where there is no ``shred``. Backups, snapshots and an
audit log of the host may still hold the password: only rotating it ends that.
**A file given back to root** must exist as a regular file, not be a symbolic
link, and be owned by the login user. ``chown -h`` never follows a link that
appears in between.
"""
from __future__ import annotations
from collections.abc import Callable
from shlex import quote
from typing import TYPE_CHECKING, ClassVar
from typing_extensions import TypedDict
from napalm_device_types.fixes import run_fix_steps
from napalm_device_types.models import FixResultDict, FixStepDict, FixStepOutcomeDict
if TYPE_CHECKING: # pragma: no cover - for type checkers only
from napalm_device_types.channel import CommandResult
#: The shell history files looked at, in the login user's home.
HISTORY_FILES = (".bash_history", ".zsh_history", ".ash_history", ".sh_history")
#: The suffix of the moment's backup ``sed -i`` makes; it carries no application's name.
HISTORY_BACKUP_SUFFIX = ".napalm-bak"
_COUNT = 'LC_ALL=C grep -c -F -e "$m" -- "$p"'
class HistorySudoLinesDict(TypedDict):
"""A history file holding lines with the sudo password (``get_history_sudo_lines``)."""
#: The file, as ``~/<name>``.
file: str
lines: int
class HistoryCleanupDict(TypedDict):
"""What one history file held before the cleanup and holds after it."""
file: str
before: int
after: int
class SudoHistoryCleanupDict(FixResultDict):
"""The result of ``remove_history_sudo_lines``: the fix, and what it changed."""
#: Every history file there is, with its lines before and after.
files: list[HistoryCleanupDict]
#: How many lines were deleted, over all files.
removed: int
#: The files given back to root.
owners_restored: list[str]
def _bre(text: str) -> str:
"""*text* as a POSIX basic regular expression matching it literally, for ``sed``."""
return "".join("\\" + c if c in "\\.[]*^$/" else c for c in text)
def _history_loop(body: str) -> str:
names = " ".join(HISTORY_FILES)
return f'for f in {names}; do p="$HOME/$f"; [ -f "$p" ] || continue; {body}; done'
def count_script(marker: str) -> str:
"""Print ``<count> <name>`` per history file there is; exit 2 when one cannot be read."""
return f"m={quote(marker)}; " + _history_loop(
f'n=$({_COUNT}); [ $? -le 1 ] || exit 2; printf "%s %s\\n" "$n" "$f"'
)
def cleanup_script(marker: str) -> str:
"""Delete the lines holding *marker* from every history file there is.
Prints per file: ``done <before> <after> <name>``, ``link <count> <name>``
(left alone), ``unreadable <name>`` or ``failed <before> <name>``.
"""
backup = f'"$p"{quote(HISTORY_BACKUP_SUFFIX)}'
return f"m={quote(marker)}; r={quote(_bre(marker))}; " + _history_loop(
f"b=$({_COUNT}); "
'[ $? -le 1 ] || { printf "unreadable %s\\n" "$f"; continue; }; '
'if [ -L "$p" ]; then printf "link %s %s\\n" "$b" "$f"; continue; fi; '
'if [ "$b" -gt 0 ]; then '
f'LC_ALL=C sed -i{quote(HISTORY_BACKUP_SUFFIX)} -e "/$r/d" "$p"; s=$?; '
# A sed that renamed the file away and wrote no new one: the backup is the history.
f'[ -e "$p" ] || mv -f {backup} "$p"; '
f"[ -e {backup} ] && {{ shred -u {backup} 2>/dev/null || rm -f {backup}; }}; "
'[ $s -eq 0 ] || { printf "failed %s %s\\n" "$b" "$f"; continue; }; '
"fi; "
f'a=$({_COUNT}); printf "done %s %s %s\\n" "$b" "$a" "$f"'
)
def owners_script(user: str, paths: tuple[str, ...]) -> str:
"""As root: give each of *paths* back to root:root where *user* owns it.
Prints ``fixed <path>`` or ``failed <path>`` per file it changed or tried to.
"""
return (
f"u={quote(user)}; for p in {' '.join(quote(p) for p in paths)}; do "
'[ -f "$p" ] && [ ! -L "$p" ] || continue; '
'[ -n "$(find "$p" -prune -user "$u" 2>/dev/null)" ] || continue; '
'if chown -h 0:0 "$p"; then printf "fixed %s\\n" "$p"; '
'else printf "failed %s\\n" "$p"; fi; done'
)
def _home(name: str) -> str:
return f"~/{name}"
def _lines(n: int) -> str:
return "1 line" if n == 1 else f"{n} lines"
def _done(rest: list[str]) -> tuple[bool, str, HistoryCleanupDict | None]:
before, after, name = int(rest[0]), int(rest[1]), _home(rest[2])
removed = before - after
text = (
f"{name}: {_lines(removed)} removed ({before} before, {after} after)"
if before
else f"{name}: no such line"
)
return after == 0, text, {"file": name, "before": before, "after": after}
def _link(rest: list[str]) -> tuple[bool, str, HistoryCleanupDict | None]:
count = int(rest[0])
return count == 0, f"{_home(rest[1])}: a symbolic link, left alone ({_lines(count)})", None
def _unreadable(rest: list[str]) -> tuple[bool, str, HistoryCleanupDict | None]:
return False, f"{_home(rest[0])}: cannot be read", None
def _failed(rest: list[str]) -> tuple[bool, str, HistoryCleanupDict | None]:
kept = _lines(int(rest[0]))
return False, f"{_home(rest[1])}: could not be rewritten ({kept} kept)", None
#: Per kind of line :func:`cleanup_script` prints: how many words follow, and what it says.
_CLEANUP_LINES: dict[
str, tuple[int, Callable[[list[str]], tuple[bool, str, HistoryCleanupDict | None]]]
] = {
"done": (3, _done),
"link": (2, _link),
"unreadable": (1, _unreadable),
"failed": (2, _failed),
}
class SudoHistoryMixin:
"""Reads and removes the sudo password an earlier driver left in shell histories.
Mixed in by the drivers that once typed it into a shell, not by a base
class, so ``callable(getattr(driver, "get_history_sudo_lines", None))``
stays a truthful answer.
"""
#: What every line carries that typed the sudo password into the shell, as a
#: fixed string -- ``"| sudo -S -p "`` for ``echo '<password>' | sudo -S -p ""
#: <command>``. None for a driver that never did: nothing is looked for.
SUDO_HISTORY_MARKER: ClassVar[str | None] = None
#: Files an earlier version wrote as the login user where root should have.
SUDO_ROOT_FILES: ClassVar[tuple[str, ...]] = ()
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
def run_command(
self,
command: str,
*,
privileged: bool = False,
timeout: float = 60,
stdin: bytes | None = None,
) -> CommandResult: ...
def get_history_sudo_lines(self) -> list[HistorySudoLinesDict]:
"""The login user's history files that hold lines with the sudo password, and how many.
A fact: counted with ``grep -c`` as the login user; no line is read.
An empty list when no file holds one, or the driver never typed it.
:raises RuntimeError: when a history file cannot be read.
"""
marker = self.SUDO_HISTORY_MARKER
if not marker:
return []
result = self.run_command(count_script(marker), timeout=30)
if result.exit_code != 0:
raise RuntimeError(
f"cannot count the shell history: {(result.stderr or result.stdout).strip()}"
)
found: list[HistorySudoLinesDict] = []
for line in result.stdout.splitlines():
count, _, name = line.strip().partition(" ")
if count.isdigit() and int(count) and name:
found.append({"file": _home(name), "lines": int(count)})
return found
def remove_history_sudo_lines(self) -> SudoHistoryCleanupDict:
"""Delete those lines, as the login user, and give :attr:`SUDO_ROOT_FILES` back to root.
Two steps, ``history`` and ``owners``; the second runs when the first
failed, too. Succeeds when no history file holds such a line any more
and every file the login user owned is root's again.
"""
files: list[HistoryCleanupDict] = []
restored: list[str] = []
parts = [
run_fix_steps([("history", lambda: self._clean_histories(files))]),
run_fix_steps([("owners", lambda: self._restore_owners(restored))]),
]
steps: list[FixStepDict] = [s for part in parts for s in part["steps"]]
return {
"success": all(part["success"] for part in parts),
"steps": steps,
"output": "\n".join(f"[{s['step']}] {s['output']}".rstrip() for s in steps),
"files": files,
"removed": sum(f["before"] - f["after"] for f in files),
"owners_restored": restored,
}
def _clean_histories(self, files: list[HistoryCleanupDict]) -> FixStepOutcomeDict:
marker = self.SUDO_HISTORY_MARKER
if not marker:
return {"ok": True, "output": "this driver never typed the sudo password"}
result = self.run_command(cleanup_script(marker), timeout=60)
said: list[str] = []
ok = result.exit_code == 0
for line in result.stdout.splitlines():
ok = self._history_line(line.split(), files, said) and ok
if result.exit_code != 0:
said.append((result.stderr or result.stdout).strip())
return {"ok": ok, "output": "\n".join(said) or "no shell history"}
@staticmethod
def _history_line(words: list[str], files: list[HistoryCleanupDict], said: list[str]) -> bool:
"""Take one line of :func:`cleanup_script` into *files* and *said*; whether it is clean."""
kind, rest = (words[0], words[1:]) if words else ("", [])
read = _CLEANUP_LINES.get(kind)
if read is None or len(rest) != read[0]:
return True
clean, text, entry = read[1](rest)
said.append(text)
if entry is not None:
files.append(entry)
return clean
def _restore_owners(self, restored: list[str]) -> FixStepOutcomeDict:
paths = tuple(self.SUDO_ROOT_FILES)
if not paths:
return {"ok": True, "output": "no file to check"}
who = self.run_command("id -u; id -un", timeout=30).stdout.split()
if len(who) < 2:
return {"ok": False, "output": "cannot tell who the login user is"}
uid, user = who[0], who[1]
if uid == "0":
return {"ok": True, "output": "logged in as root: every file it wrote is root's"}
result = self.run_command(owners_script(user, paths), privileged=True, timeout=30)
said: list[str] = []
ok = result.exit_code == 0
for line in result.stdout.splitlines():
kind, _, path = line.strip().partition(" ")
if kind == "fixed":
restored.append(path)
said.append(f"{path}: owned by root again")
elif kind == "failed":
ok = False
said.append(f"{path}: could not be given back to root")
if result.exit_code != 0:
said.append((result.stderr or result.stdout).strip())
return {"ok": ok, "output": "\n".join(said) or f"no file owned by {user}"}
__all__ = [
"HISTORY_BACKUP_SUFFIX",
"HISTORY_FILES",
"HistoryCleanupDict",
"HistorySudoLinesDict",
"SudoHistoryCleanupDict",
"SudoHistoryMixin",
"cleanup_script",
"count_script",
"owners_script",
]
+193
View File
@@ -0,0 +1,193 @@
"""A device's time sources and time zone: read them, and set them to what the caller says.
An application keeps one list of NTP servers and a time zone per site, and aligns a
device that differs. :class:`TimeSyncMixin` is the contract for that:
* ``get_time_sync()`` -- declared only, so ``callable(getattr(driver,
"get_time_sync", None))`` is the truthful answer to "can this driver tell";
* ``configure_time_sync(ntp_servers=None, timezone=None)`` -- a fix template
(:mod:`napalm_device_types.fixes`): ``None`` leaves that value as it is, and the
device ends up configured with exactly the servers given, in their order. A driver
implements the steps; one whose device takes the change in a single call
overrides the method;
* ``time_sync_writes`` -- which of the two values the driver can write. A BMC that
runs on UTC only writes ``{"ntp_servers"}``; the application never asks it for a
zone.
A driver that inherits the mixin but cannot run it sets ``get_time_sync`` (and
``configure_time_sync``) to ``None``.
**Servers from DHCP count** (NetOrk/netork#2380). A device that takes its time
servers from a DHCP lease (option 42) gets them from whoever serves the subnet --
usually the site's time giver. ``ntp_servers`` are only the servers configured on
the device; ``dhcp_ntp_servers`` are those the device got by DHCP and uses. A write
replaces the configured servers and leaves DHCP's alone: it never switches the use of
DHCP-provided servers off, and the ``verify`` step does not count them.
A zone is an IANA name on the wire. A device that wants a POSIX rule or an offset
with a summer-time rule gets it from :mod:`napalm_device_types.posix_tz`.
"""
# No ``from __future__ import annotations`` here: TypedDict tells a NotRequired key
# only from an evaluated annotation (``TimeSyncDict.__optional_keys__``).
import ipaddress
import re
from collections.abc import Sequence
from typing import TYPE_CHECKING, ClassVar, NotRequired, TypedDict
from napalm_device_types.fixes import FixStep, run_fix_steps
from napalm_device_types.models import FixResultDict, FixStepOutcomeDict
from napalm_device_types.posix_tz import UnknownTimezone, is_iana_zone
#: The values a driver can write; ``time_sync_writes`` is a subset.
TIME_SYNC_VALUES: frozenset[str] = frozenset({"ntp_servers", "timezone"})
_HOST_NAME = re.compile(
r"(?=.{1,253}$)[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?"
r"(?:\.[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?)*\.?"
)
class TimeSyncDict(TypedDict):
"""How a device keeps its clock (``get_time_sync``)."""
ntp_servers: list[str] # configured time sources, in the device's order; [] = none configured
timezone: str | None # IANA name ("Europe/Berlin"); None = not readable or no named zone
synchronized: bool | None # the clock is synchronised to a source right now; None = unknown
daemon: str | None # "chrony", "systemd-timesyncd", "ntpd", "openntpd", ...; None = own client
# The servers the device got by DHCP (option 42) and uses, in its order; never in
# ntp_servers. [] = none; absent or None = the driver cannot tell.
dhcp_ntp_servers: NotRequired[list[str] | None]
def _is_server(name: str) -> bool:
try:
ipaddress.ip_address(name)
except ValueError:
return bool(_HOST_NAME.fullmatch(name))
return True
def check_ntp_servers(servers: Sequence[str]) -> list[str]:
"""*servers* as a driver writes them: stripped, in their order, each once.
:raises ValueError: for no server at all, or one that is not a host name or
an IP address -- before anything reaches a configuration file.
:raises TypeError: for a single string instead of a list of them.
"""
if isinstance(servers, str):
raise TypeError("ntp_servers is a list of servers, not one string")
kept: list[str] = []
for server in (str(s).strip() for s in servers):
if not server:
continue
if not _is_server(server):
raise ValueError(f"{server!r} is not a host name or address of an NTP server")
if server.lower() not in (k.lower() for k in kept):
kept.append(server)
if not kept:
raise ValueError("name at least one NTP server; None leaves the servers as they are")
return kept
class TimeSyncMixin:
"""Read and set a device's time sources and time zone; see the module."""
#: Which of the two values this driver can write: a subset of {"ntp_servers", "timezone"}.
#: A BMC that only runs on UTC writes {"ntp_servers"}; an application never asks it for a
#: zone. Declared only: a driver that names nothing writes nothing.
time_sync_writes: ClassVar[frozenset[str]]
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
def get_time_sync(self) -> TimeSyncDict:
"""How the device keeps its clock now.
``ntp_servers`` are the sources its configuration names, in its order
(a pool by its name, not the addresses it resolved to);
``dhcp_ntp_servers`` those a DHCP lease gave it and it uses, apart from
them -- left out where the driver cannot tell. ``timezone``
is an IANA name, or None where the device keeps an offset and a rule
instead of a name. ``daemon`` names the client that keeps the clock on
a host that has a choice; None for a device's own.
Example::
{
"ntp_servers": ["ntp1.example.org", "10.0.0.1"],
"timezone": "Europe/Berlin",
"synchronized": True,
"daemon": "chrony",
"dhcp_ntp_servers": ["192.168.1.1"],
}
:raises Exception: when the device could not be read.
"""
...
# The steps a driver implements:
def _write_ntp_servers(self, servers: list[str]) -> FixStepOutcomeDict:
"""Leave the device configured with exactly *servers*, in this order:
sources it had configured elsewhere stop counting (commented out,
replaced). Servers it gets by DHCP stay, and so does its use of them."""
...
def _write_time_zone(self, zone: str) -> FixStepOutcomeDict:
"""Set the device's zone to *zone*, an IANA name of the database."""
...
def configure_time_sync(
self, ntp_servers: Sequence[str] | None = None, timezone: str | None = None
) -> FixResultDict:
"""Give the device exactly *ntp_servers*, in order, and *timezone*; None leaves one alone.
What the device gets by DHCP is not touched: it keeps using those servers too.
The steps: ``ntp_servers``, ``timezone``, then ``verify``, which reads the
device again.
:raises UnknownTimezone: for a *timezone* that is not an IANA name of the
database, before anything is written.
:raises ValueError: for a value this driver does not write
(:attr:`time_sync_writes`), for no server or a malformed one, or for
nothing asked -- before anything is written.
"""
servers = None if ntp_servers is None else check_ntp_servers(ntp_servers)
if timezone is not None and not is_iana_zone(timezone):
raise UnknownTimezone(timezone)
asked = {name for name, v in (("ntp_servers", servers), ("timezone", timezone)) if v}
if not asked:
raise ValueError("nothing to configure: name ntp_servers, timezone or both")
refused = asked - getattr(self, "time_sync_writes", frozenset())
if refused:
raise ValueError(f"this driver does not write {', '.join(sorted(refused))}")
steps: list[FixStep] = []
if servers is not None:
steps.append(("ntp_servers", lambda: self._write_ntp_servers(servers)))
if timezone is not None:
steps.append(("timezone", lambda: self._write_time_zone(timezone)))
steps.append(("verify", lambda: self._verify_time_sync(servers, timezone)))
return run_fix_steps(steps)
def _verify_time_sync(self, servers: list[str] | None, zone: str | None) -> FixStepOutcomeDict:
"""Read the device again: does it now have what was written?
Servers compare without case; the servers DHCP brought are not among
them. A zone is compared where the device reads one back by name.
"""
now = self.get_time_sync()
wrong: list[str] = []
if servers is not None and [s.lower() for s in now["ntp_servers"]] != [
s.lower() for s in servers
]:
wrong.append(f"the device has the servers {', '.join(now['ntp_servers']) or 'none'}")
if zone is not None and now["timezone"] is not None and now["timezone"] != zone:
wrong.append(f"the device is in the time zone {now['timezone']}")
if wrong:
return {"ok": False, "output": "; ".join(wrong)}
return {"ok": True, "output": "the device reads back what was written"}
__all__ = ["TIME_SYNC_VALUES", "TimeSyncDict", "TimeSyncMixin", "check_ntp_servers"]
+219
View File
@@ -0,0 +1,219 @@
"""Time sync on a POSIX host, over the command channel: chrony, systemd-timesyncd, timedatectl.
:class:`PosixTimeSyncMixin` implements
:class:`~napalm_device_types.time_sync.TimeSyncMixin` for a driver that has
:meth:`~napalm_device_types.channel.CommandChannelMixin.run_command`. The defaults
are Debian's paths; a driver sets its host's.
**Reading** is one round trip as the login user
(:mod:`napalm_device_types._time_sync_reading`): ``timedatectl show`` for the zone
(``/etc/timezone`` or ``/etc/localtime``'s target without it), which time sync
service is active, and that client's configuration -- chrony's files as chronyd
reads them, ``timedatectl show-timesync`` for systemd-timesyncd, ``ntp.conf``,
OpenNTPD's ``ntpd.conf``. Synchronised is chronyd's own word where it runs (a
selected source in ``chronyc -n sources``), else the kernel's (``NTPSynchronized``).
What DHCP brought (option 42) is read apart, as ``dhcp_ntp_servers``, and never as a
configured server: chrony's sources from the DHCP client's ``sourcedir``
(:attr:`~PosixTimeSyncMixin.CHRONY_DHCP_DIRS`, the files as chronyd reads them),
systemd-timesyncd's ``LinkNTPServers``. Where ntpd or OpenNTPD runs it is unknown.
**Writing** runs as root through ``run_command(privileged=True)``; root never reads
stdin -- a file is staged by the login user and installed by root.
* The zone: ``timedatectl set-timezone``.
* chrony: the servers go to a file of their own and every other configured source is
commented out (:mod:`napalm_device_types._chrony`); the DHCP client's ``sourcedir``
stays, and one 4.35.0 commented out comes back. Then ``chronyc reload sources``, or
a restart of chronyd where its main configuration changed or it cannot reload.
* systemd-timesyncd: :attr:`~PosixTimeSyncMixin.TIMESYNCD_DROPIN` resets ``NTP=``,
sets the servers and empties ``FallbackNTP=``, then the service restarts. Servers a
network link brings (DHCP through systemd-networkd) stay and count.
* ntpd, OpenNTPD, or no client: the servers are read, not written -- the step fails
and says so.
"""
from __future__ import annotations
import shlex
from collections.abc import Sequence
from typing import TYPE_CHECKING, ClassVar
from napalm_device_types._chrony import (
COMMENT_MARK,
comment_out_command,
include_command,
plan_chrony,
restore_command,
)
from napalm_device_types._scheduled_reading import TIME_ZONE_SH
from napalm_device_types._time_sync_reading import (
CHRONY_DHCP_DIRS,
TIME_SYNC_UNITS,
Reading,
ReadPaths,
chrony_synchronized,
parse_key_values,
parse_reading,
parse_time_sync,
reading_command,
time_sync_of,
timesyncd_link_servers,
timesyncd_servers,
)
from napalm_device_types.models import FixStepOutcomeDict
from napalm_device_types.time_sync import TIME_SYNC_VALUES, TimeSyncDict, TimeSyncMixin
if TYPE_CHECKING:
from napalm_device_types.channel import CommandResult
_STAGE = 't=$(mktemp) && chmod 644 "$t" && cat > "$t" && printf %s "$t"'
_WRITERS = "chrony and systemd-timesyncd"
#: The head of the systemd-timesyncd drop-in.
TIMESYNCD_HEADER = (
"# The time sources, exactly these: written by napalm-device-types'\n"
"# PosixTimeSyncMixin and replaced on every change.\n"
)
def _said(result: CommandResult) -> str:
return "\n".join(p for p in (result.stdout.strip(), result.stderr.strip()) if p)
def timesyncd_dropin(servers: Sequence[str]) -> str:
"""The drop-in that leaves systemd-timesyncd with exactly *servers*."""
return f"{TIMESYNCD_HEADER}[Time]\nNTP=\nNTP={' '.join(servers)}\nFallbackNTP=\n"
class PosixTimeSyncMixin(TimeSyncMixin):
"""Time sync of a host reached through ``run_command``; the paths are the driver's."""
time_sync_writes: ClassVar[frozenset[str]] = TIME_SYNC_VALUES
#: ``(unit, daemon)``: the services that keep the clock; the first active one counts.
TIME_SYNC_UNITS: ClassVar[tuple[tuple[str, str], ...]] = TIME_SYNC_UNITS
#: chrony's main configuration: the first that exists.
CHRONY_CONFIGS: ClassVar[tuple[str, ...]] = ("/etc/chrony/chrony.conf", "/etc/chrony.conf")
#: The servers' file where the main configuration has a ``sourcedir`` of its directory.
CHRONY_SOURCES_FILE: ClassVar[str] = "/etc/chrony/sources.d/netork.sources"
#: The servers' file otherwise (its name also in a ``confdir``); included if need be.
CHRONY_INCLUDE_FILE: ClassVar[str] = "/etc/chrony.d/netork.conf"
#: The ``sourcedir`` a DHCP client writes chrony's sources to: read as
#: ``dhcp_ntp_servers``, never commented out.
CHRONY_DHCP_DIRS: ClassVar[tuple[str, ...]] = CHRONY_DHCP_DIRS
TIMESYNCD_DROPIN: ClassVar[str] = "/etc/systemd/timesyncd.conf.d/netork.conf"
NTPD_CONFIGS: ClassVar[tuple[str, ...]] = ("/etc/ntpsec/ntp.conf", "/etc/ntp.conf")
OPENNTPD_CONFIGS: ClassVar[tuple[str, ...]] = ("/etc/openntpd/ntpd.conf", "/etc/ntpd.conf")
#: ``sh`` that prints the zone without timedatectl: a name, or a path ending in it.
TIME_ZONE_SH: ClassVar[str] = TIME_ZONE_SH
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
def run_command(
self,
command: str,
*,
privileged: bool = False,
timeout: float = 60,
stdin: bytes | None = None,
) -> CommandResult: ...
# -- reading ---------------------------------------------------------------------------------
def _ts_read(self) -> Reading:
paths = ReadPaths(
units=self.TIME_SYNC_UNITS,
chrony_configs=self.CHRONY_CONFIGS,
ntpd_configs=self.NTPD_CONFIGS,
openntpd_configs=self.OPENNTPD_CONFIGS,
time_zone_sh=self.TIME_ZONE_SH,
)
return parse_reading(self.run_command(reading_command(paths), timeout=60).stdout)
def get_time_sync(self) -> TimeSyncDict:
"""The host's time sources, zone and client, in one round trip as the login user.
:raises ValueError: when the reading came back cut short.
"""
return time_sync_of(self._ts_read(), self.CHRONY_DHCP_DIRS)
# -- writing ---------------------------------------------------------------------------------
def _ts_root(self, command: str) -> CommandResult:
return self.run_command(command, privileged=True, timeout=60)
def _ts_checked(self, command: str) -> str:
result = self._ts_root(command)
if result.exit_code != 0:
raise RuntimeError(f"{command}: {_said(result) or f'exit {result.exit_code}'}")
return _said(result)
def _ts_install(self, path: str, content: str) -> None:
"""*content* to *path* as root, staged by the login user: root reads no stdin."""
staged = self.run_command(_STAGE, timeout=30, stdin=content.encode())
tmp = (staged.stdout.strip().splitlines() or [""])[-1]
if staged.exit_code != 0 or not tmp.startswith("/"):
raise RuntimeError(f"cannot stage {path}: {_said(staged)}")
try:
self._ts_checked(shlex.join(["install", "-D", "-m", "644", "--", tmp, path]))
finally:
self.run_command(shlex.join(["rm", "-f", "--", tmp]), timeout=30)
def _write_time_zone(self, zone: str) -> FixStepOutcomeDict:
result = self._ts_root(shlex.join(["timedatectl", "set-timezone", zone]))
if result.exit_code != 0:
return {"ok": False, "output": _said(result) or f"exit {result.exit_code}"}
return {"ok": True, "output": f"time zone set to {zone}"}
def _write_ntp_servers(self, servers: list[str]) -> FixStepOutcomeDict:
reading = self._ts_read()
if reading.daemon == "chrony":
return self._ts_chrony(reading, servers)
if reading.daemon == "systemd-timesyncd":
return self._ts_timesyncd(servers)
runs = f"this host runs {reading.daemon}" if reading.daemon else "none runs here"
return {"ok": False, "output": f"time sources are written for {_WRITERS}; {runs}"}
def _ts_timesyncd(self, servers: list[str]) -> FixStepOutcomeDict:
self._ts_install(self.TIMESYNCD_DROPIN, timesyncd_dropin(servers))
self._ts_checked("systemctl restart systemd-timesyncd.service")
return {"ok": True, "output": f"wrote {self.TIMESYNCD_DROPIN}; restarted systemd-timesyncd"}
def _ts_chrony(self, reading: Reading, servers: list[str]) -> FixStepOutcomeDict:
plan = plan_chrony(
reading.chrony,
servers,
sources_file=self.CHRONY_SOURCES_FILE,
include_file=self.CHRONY_INCLUDE_FILE,
dhcp_dirs=self.CHRONY_DHCP_DIRS,
)
said = [f"wrote {plan.target}"]
self._ts_install(plan.target, plan.content)
for path, numbers in plan.restore.items():
self._ts_checked(restore_command(path, numbers))
said.append(f"gave DHCP's sourcedir back in {path}:{','.join(map(str, numbers))}")
for path, numbers in plan.comment_out.items():
self._ts_checked(comment_out_command(path, numbers))
said.append(f"commented out {path}:{','.join(map(str, numbers))}")
if plan.include_in is not None:
self._ts_checked(include_command(plan.include_in, plan.target))
said.append(f"included it from {plan.include_in}")
if not (plan.reload and self._ts_root("chronyc reload sources").exit_code == 0):
self._ts_checked(shlex.join(["systemctl", "restart", f"{reading.unit}.service"]))
said.append(f"restarted {reading.unit}")
else:
said.append("reloaded the sources")
return {"ok": True, "output": "; ".join(said)}
__all__ = [
"COMMENT_MARK",
"PosixTimeSyncMixin",
"chrony_synchronized",
"parse_key_values",
"parse_time_sync",
"timesyncd_dropin",
"timesyncd_link_servers",
"timesyncd_servers",
]
+285
View File
@@ -0,0 +1,285 @@
"""TLS certificates: trust on first use by fingerprint, a changed one refused before any login.
Network devices ship self-signed certificates no CA vouches for. A driver that
then turns verification off logs in with the device's credentials at whatever
answers on its address (NetOrk/netork#1167, NAPALM/napalm-hpe-aruba-procurve#17).
The same answer as for SSH host keys (:mod:`napalm_device_types.ssh_host_keys`):
the layer above keeps the store of the certificates it trusts, a driver verifies.
* It receives the fingerprints to expect in ``optional_args["tls_fingerprints"]``
(:data:`KNOWN_FINGERPRINTS_ARG`), by the port they were seen on:
``{"443": "SHA256:AB:CD:…"}``, the SHA-256 of the DER certificate as
``openssl x509 -fingerprint -sha256`` prints it (the ``SHA256:`` prefix,
colons and case are optional, :func:`normalise_fingerprint`). With a
fingerprint, any other certificate is refused before a password leaves the
caller; no CA is consulted. Without one, the first certificate is accepted:
trust on first use.
* It reports every certificate it sees through
``optional_args["tls_certificate_observer"]`` (:data:`OBSERVER_ARG`), a
callable that receives one dict per connection::
{"host": str, "port": int,
"seen": {"fingerprint"},
"expected": {"fingerprint"} | None,
"accepted": bool}
``accepted`` is False only for a refused certificate; the connection then
raises :class:`TlsCertificateMismatch`. A failing observer is logged and
ignored.
A caller that passes neither keeps working as before (first use, every time),
and a driver that ignores both still connects for a caller that passes them.
:class:`CertificateCheck` is the one implementation. :meth:`~CertificateCheck.verify_server`
takes the certificate from a handshake of its own, before any request;
:meth:`~CertificateCheck.pin` makes every later connection of a ``requests``
session present that same certificate (urllib3's ``assert_fingerprint``), so
another host cannot step in between the check and the login;
:meth:`~CertificateCheck.refusing` turns such a refusal into
:class:`TlsCertificateMismatch`; :meth:`~CertificateCheck.observe` reports the
certificate once the login stands.
"""
from __future__ import annotations
import hashlib
import logging
import re
import socket
import ssl
from collections.abc import Callable, Iterator, Mapping
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Any
from napalm.base.exceptions import ConnectionException
logger = logging.getLogger(__name__)
#: The ``optional_args`` key that carries the fingerprints to expect, by port.
KNOWN_FINGERPRINTS_ARG = "tls_fingerprints"
#: The ``optional_args`` key that carries the callable every seen certificate goes to.
OBSERVER_ARG = "tls_certificate_observer"
_PREFIX = "SHA256:"
_SHA256_HEX = re.compile(r"^[0-9A-F]{64}$")
#: What urllib3 says when a connection presents another certificate than the pinned one
#: (``got "…"``; 1.26 writes the digest as ``b'…'``, escaped once requests wraps it).
_URLLIB3_MISMATCH = re.compile(r"Fingerprints did not match.*?got \"(?:b\\?')?([0-9a-fA-F]{64})")
def fingerprint(der: bytes) -> str:
"""``SHA256:AB:CD:…``: the SHA-256 of a DER certificate, as openssl prints it."""
return _format(hashlib.sha256(der).hexdigest())
def _format(hex_digest: str) -> str:
upper = hex_digest.upper()
return _PREFIX + ":".join(upper[i : i + 2] for i in range(0, len(upper), 2))
def normalise_fingerprint(value: Any) -> str:
"""*value* as :func:`fingerprint` writes it; ``SHA256:``, colons, spaces and case optional."""
if not isinstance(value, str):
raise ValueError(f"not a SHA-256 certificate fingerprint: {value!r}")
text = value.strip()
if text[: len(_PREFIX)].upper() == _PREFIX:
text = text[len(_PREFIX) :]
digits = re.sub(r"[\s:]", "", text).upper()
if not _SHA256_HEX.match(digits):
raise ValueError(f"not a SHA-256 certificate fingerprint: {value!r}")
return _format(digits)
def _hex(fingerprint_: str) -> str:
"""The bare lower-case hex urllib3's ``assert_fingerprint`` takes."""
return fingerprint_.removeprefix(_PREFIX).replace(":", "").lower()
def fetch_certificate(
host: str,
port: int,
timeout: float,
*,
connect: Callable[..., socket.socket] = socket.create_connection,
) -> bytes:
"""The DER certificate *host* presents on *port*, from a handshake that sends nothing else.
No CA and no host name are checked: the certificate is judged by its fingerprint.
"""
context = ssl.create_default_context()
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
with connect((host, port), timeout=timeout) as sock:
with context.wrap_socket(sock, server_hostname=host) as tls:
der = tls.getpeercert(binary_form=True)
if not der: # pragma: no cover - a TLS server always presents one
raise ConnectionException(f"{host}:{port} presented no TLS certificate")
return der
@dataclass(frozen=True)
class CertificateObservation:
"""The certificate one connection saw, against the one it was to expect."""
host: str
port: int
#: The fingerprint of the certificate presented.
seen: str
#: The fingerprint it was to present; None on first use.
expected: str | None
@property
def accepted(self) -> bool:
"""True on first use and for the expected certificate; False for a refused one."""
return self.expected is None or self.expected == self.seen
def as_dict(self) -> dict[str, Any]:
return {
"host": self.host,
"port": self.port,
"seen": {"fingerprint": self.seen},
"expected": {"fingerprint": self.expected} if self.expected else None,
"accepted": self.accepted,
}
class TlsCertificateMismatch(ConnectionException): # type: ignore[misc]
"""The device presented another TLS certificate than the one it is known by.
Raised before any request that carries credentials: nothing was sent to it.
"""
def __init__(self, observation: CertificateObservation) -> None:
self.observation = observation
super().__init__(
f"TLS certificate mismatch at {observation.host}:{observation.port}: it presented "
f"{observation.seen}, the trusted certificate is {observation.expected or 'none'}. "
"Nothing was sent to it."
)
def _expected(known: Any, port: int) -> str | None:
"""The fingerprint in *known* (``{port: fingerprint}``, port as text or number) for *port*."""
if not known:
return None
if not isinstance(known, Mapping):
raise ValueError(f"TLS fingerprints must be given by port, not as {type(known).__name__}")
value = known.get(str(port)) or known.get(port)
return normalise_fingerprint(value) if value else None
def _refused_fingerprint(exc: BaseException) -> str | None:
"""The fingerprint urllib3 saw when it refused a pinned connection, in *exc*'s chain."""
seen: set[int] = set()
stack: list[BaseException | None] = [exc]
while stack:
current = stack.pop()
if current is None or id(current) in seen:
continue
seen.add(id(current))
match = _URLLIB3_MISMATCH.search(str(current))
if match:
return _format(match.group(1))
stack.extend([current.__cause__, current.__context__])
return None
class CertificateCheck:
"""How one driver connection verifies its host's TLS certificate.
*optional_args* are the driver's (both keys optional), *host* and *port* the
address the connection goes to.
"""
def __init__(self, optional_args: Mapping[str, Any] | None, host: str, port: int) -> None:
args = optional_args or {}
self.host = host
self.port = int(port)
self.expected = _expected(args.get(KNOWN_FINGERPRINTS_ARG), self.port)
self._observer = args.get(OBSERVER_ARG)
self._verified: CertificateObservation | None = None
def verify_server(
self,
timeout: float,
*,
connect: Callable[..., socket.socket] = socket.create_connection,
) -> CertificateObservation:
"""Take the certificate from a handshake of its own and :meth:`verify` it."""
return self.verify(fetch_certificate(self.host, self.port, timeout, connect=connect))
def verify(self, der: bytes) -> CertificateObservation:
"""Judge the DER certificate *der*: a refused one is reported and raises.
An accepted one is reported by :meth:`observe`, once the login stands.
"""
observation = CertificateObservation(self.host, self.port, fingerprint(der), self.expected)
if not observation.accepted:
self._report(observation)
raise TlsCertificateMismatch(observation)
self._verified = observation
return observation
def pin(self, session: Any) -> None:
"""Every HTTPS connection of the ``requests`` *session* must present the verified one."""
verified = self._require_verified()
session.mount("https://", _pinned_adapter(_hex(verified.seen)))
@contextmanager
def refusing(self) -> Iterator[None]:
"""Around requests on a pinned session: a refused certificate becomes a mismatch."""
try:
yield
except Exception as exc:
offered = _refused_fingerprint(exc)
if offered is None:
raise
trusted = self.expected or self._require_verified().seen
observation = CertificateObservation(self.host, self.port, offered, trusted)
self._report(observation)
raise TlsCertificateMismatch(observation) from exc
def observe(self) -> CertificateObservation:
"""After the login: the verified certificate, reported."""
verified = self._require_verified()
self._report(verified)
return verified
def _require_verified(self) -> CertificateObservation:
if self._verified is None:
raise RuntimeError("verify the certificate first (verify_server or verify)")
return self._verified
def _report(self, observation: CertificateObservation) -> None:
if not callable(self._observer):
return
try:
self._observer(observation.as_dict())
except Exception: # the store is the caller's; a connection does not hang on it
logger.warning(
"Could not report the TLS certificate of %s:%s",
self.host,
self.port,
exc_info=True,
)
def _pinned_adapter(hex_digest: str) -> Any:
"""A ``requests`` adapter whose connections, direct or through a proxy, must match *hex_digest*.
``requests`` is imported here: it comes with napalm, and a module that only
verifies a fingerprint does not need it loaded.
"""
from requests.adapters import HTTPAdapter
class _PinnedAdapter(HTTPAdapter): # type: ignore[misc,unused-ignore]
def init_poolmanager(self, *args: Any, **kwargs: Any) -> None:
kwargs["assert_fingerprint"] = hex_digest
super().init_poolmanager(*args, **kwargs)
def proxy_manager_for(self, proxy: str, **kwargs: Any) -> Any:
kwargs["assert_fingerprint"] = hex_digest
return super().proxy_manager_for(proxy, **kwargs)
return _PinnedAdapter()
+22
View File
@@ -50,6 +50,28 @@ class UpdateMixin:
"""
...
def get_installed_advisory_cves(self) -> dict[str, list[str]]:
"""
The CVEs the host's installed security advisories close, per package
name: the distribution's own word that a fix is in the installed build,
although its upstream version did not move (a backport).
netOrk asks for it only where its knowledge base holds no records of
the host's release (netOrk #2394), since the answer can be long. A
package manager whose advisories name no CVEs has nothing to say here.
**An empty dict means no installed advisory names a CVE.** A reader
that cannot read raises, as :meth:`get_available_updates` does.
``NotImplementedError`` where the host's package manager keeps no such
advisories (:data:`~napalm_device_types.package_updates.DNF_INSTALLED_CVES_COMMAND`
is dnf 4's).
Example::
{"glibc-common": ["CVE-2023-4911"], "curl": ["CVE-2023-38545"]}
"""
...
def refresh_available_updates(self) -> dict[str, Any]:
"""
Refreshes the host's package index, so that :meth:`get_available_updates`
+26
View File
@@ -0,0 +1,26 @@
"""Moving a guest to another node of its cluster (``HypervisorDriver.migrate_vm``).
``get_vm_migration_preflight`` reports what the platform says before a move:
which nodes would take the guest, which refuse it and why, and what the guest
holds on its current node. ``migrate_vm`` then moves it and blocks until the
platform's task ended; a failure raises with the platform's own cause.
``set_node_maintenance`` switches a node's HA maintenance mode, which makes the
HA manager move its HA guests away while it is on.
Facts only: which node to pick, and whether a local disk is worth copying, is
the caller's to decide.
"""
from typing import TypedDict
class MigrationPreflightDict(TypedDict):
"""Where a guest could move to, and what stands in the way."""
running: bool
allowed_nodes: list[str] # nodes the platform would move it to
refused_nodes: dict[str, str] # node -> the platform's reason, in plain words
# volume ids on storage the target does not share; they block nothing by
# themselves, the migration copies them
local_disks: list[str]
local_resources: list[str] # e.g. "hostpci0", "usb0": they block the migration
+29
View File
@@ -0,0 +1,29 @@
"""Asking for a VM's clean shutdown without waiting for it.
``HypervisorDriver.stop_vm`` blocks until the VM is stopped, within a limit
of the driver's own. A caller that keeps its waits as timers asks with
``begin_stop_vm`` and reads ``get_vm_status`` until the VM says it stopped.
Declared under ``if TYPE_CHECKING``: a driver mixes this in when its
hypervisor can take the request on its own; ``hasattr`` tells a caller
whether it did. No role base carries it.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
class VmStopRequestMixin:
if TYPE_CHECKING:
def begin_stop_vm(self, name: str) -> None:
"""
Asks the hypervisor to shut a virtual machine down cleanly (ACPI)
and returns as soon as the request was accepted.
:param name: VM name or its ``vmid``.
:raises ValueError: If no VM with the given name/ID exists.
:raises RuntimeError: If the hypervisor refuses the request.
"""
...
+7 -7
View File
@@ -4,10 +4,10 @@ build-backend = "setuptools.build_meta"
[project]
name = "napalm-device-types"
version = "4.22.0"
version = "4.38.0"
description = "Abstract device-type base classes for NAPALM drivers"
readme = "README.md"
requires-python = ">=3.10"
requires-python = ">=3.12"
license = "Apache-2.0"
authors = [
{ name = "Christian Manivong", email = "christian@manivong.de" },
@@ -30,8 +30,6 @@ classifiers = [
"Intended Audience :: System Administrators",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Topic :: System :: Networking",
@@ -39,8 +37,10 @@ classifiers = [
"Typing :: Typed",
]
dependencies = [
"napalm>=4.0",
"napalm>=5.1", # older releases import pkg_resources, which a 3.12 venv lacks
"requests>=2.25.0",
"typing-extensions>=4.0",
"tzdata>=2024.1", # posix_tz reads the zones' TZif files from it
]
[project.urls]
@@ -65,7 +65,7 @@ napalm_device_types = ["py.typed"]
[tool.ruff]
line-length = 100
target-version = "py310"
target-version = "py312"
[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B"]
@@ -75,7 +75,7 @@ select = ["E", "F", "I", "UP", "B"]
"tests/**" = ["E501"]
[tool.mypy]
python_version = "3.10"
python_version = "3.12"
strict = true
ignore_missing_imports = true
+1 -3
View File
@@ -20,12 +20,10 @@
},
"function_lines": {
"napalm_device_types/_ucd_metrics.py::collect_ucd_metrics": 119,
"napalm_device_types/dhcp.py::DhcpServerMixin.diff_dhcp_reservations": 54,
"napalm_device_types/dhcp.py::DhcpServerMixin.diff_dhcp_subnets": 60,
"napalm_device_types/firewall_rules.py::FirewallRuleMixin.diff_firewall_rules": 56,
"napalm_device_types/hypervisor.py::HypervisorDriver.create_vm_from_cloud_init": 84,
"napalm_device_types/hypervisor.py::HypervisorDriver.get_vm_config": 77,
"napalm_device_types/ssh_server.py::OpenSshServerMixin.harden_ssh_server": 52,
"napalm_device_types/ssh_server.py::plan_ssh_hardening": 60,
"napalm_device_types/storage.py::StorageDriver.get_volumes": 52,
"napalm_device_types/switch.py::SwitchDriver.get_dot1x_ports": 51,
@@ -33,7 +31,7 @@
},
"mypy_relaxed": [],
"python_file_lines": {
"napalm_device_types/models.py": 1504
"napalm_device_types/models.py": 1454
},
"untyped_public": []
}
+306 -13
View File
@@ -20,6 +20,7 @@ from napalm_device_types.apt_proxy import (
AptProxyMixin,
apt_fetch_command,
comment_out_command,
https_proxy_for,
parse_apt_fetch,
parse_apt_proxy,
same_proxy_url,
@@ -141,7 +142,7 @@ class TestWhereItIsSet:
class TestSources:
def test_https_sources_bypass_an_http_proxy(self):
def test_sources_by_scheme(self):
report = parse_apt_proxy(
_report(
sources=[
@@ -155,6 +156,102 @@ class TestSources:
assert report["https_sources"] == ["https://download.docker.com/linux/ubuntu/dists/noble/"]
class TestHttpsProxy:
"""apt's https method asks ``Acquire::https::Proxy`` and then ``Acquire::http::Proxy``:
an http proxy alone takes the https sources too (NetOrk/netork#1811). Measured with
apt 2.8.3's apt-helper against a dead port."""
def test_the_http_proxy_alone_takes_https_too(self):
report = parse_apt_proxy(_report(dump=['Acquire::http::Proxy "http://c:3142";']))
assert report["https_proxy"] == "http://c:3142"
def test_an_https_proxy_of_its_own_wins(self):
report = parse_apt_proxy(
_report(
dump=[
'Acquire::http::Proxy "http://c:3142";',
'Acquire::https::Proxy "http://other:3128";',
]
)
)
assert (report["proxy"], report["https_proxy"]) == ("http://c:3142", "http://other:3128")
@pytest.mark.parametrize("value", ["DIRECT", "direct"])
def test_direct_https_is_none(self, value):
report = parse_apt_proxy(
_report(
dump=['Acquire::http::Proxy "http://c:3142";', f'Acquire::https::Proxy "{value}";']
)
)
assert (report["proxy"], report["https_proxy"]) == ("http://c:3142", None)
def test_an_empty_https_proxy_is_not_set(self):
"""apt takes an empty value for unset and goes on to the http one."""
report = parse_apt_proxy(
_report(dump=['Acquire::http::Proxy "http://c:3142";', 'Acquire::https::Proxy "";'])
)
assert report["https_proxy"] == "http://c:3142"
def test_per_host_https_then_http(self):
report = parse_apt_proxy(
_report(
dump=[
'Acquire::http::Proxy::a.example "DIRECT";',
'Acquire::http::Proxy::b.example "DIRECT";',
'Acquire::https::Proxy::b.example "http://c:3142";',
]
)
)
assert report["https_per_host"] == {"a.example": "DIRECT", "b.example": "http://c:3142"}
assert report["per_host"] == {"a.example": "DIRECT", "b.example": "DIRECT"}
def test_where_the_https_proxy_is_set(self):
report = parse_apt_proxy(
_report(
parts=[
f'{_D}00proxy:1:Acquire::http::Proxy "http://c:3142";',
f'{_D}90curtin-aptproxy:2:Acquire::https::Proxy "http://c:3142";',
f'{_D}91x:1:Acquire::https {{ Proxy "http://n:1"; }};',
f'{_D}92x:1:// Acquire::https::Proxy "http://c:3142";',
f'{_D}93x:1:Acquire::https::Proxy::h.example "DIRECT";',
]
)
)
assert report["https_settings"] == [
{
"file": f"{_D}90curtin-aptproxy",
"line": 2,
"value": "http://c:3142",
"editable": True,
},
{"file": f"{_D}91x", "line": 1, "value": "http://n:1", "editable": False},
]
assert [s["file"] for s in report["settings"]] == [f"{_D}00proxy"]
@pytest.mark.parametrize(
"host, proxy",
[
("deb.example", "http://c:3142"),
("direct.example", None),
("Other.Example", "http://o:1"),
],
)
def test_the_proxy_for_one_host(self, host, proxy):
report = parse_apt_proxy(
_report(
dump=[
'Acquire::http::Proxy "http://c:3142";',
'Acquire::https::Proxy::direct.example "DIRECT";',
'Acquire::http::Proxy::other.example "http://o:1";',
]
)
)
assert https_proxy_for(report, host) == proxy
class TestSameProxyUrl:
@pytest.mark.parametrize(
"a, b",
@@ -199,6 +296,12 @@ E: Download Failed"""
_FETCHED = """Get:1 http://172.22.8.50:3142/acng-report.html
Fetched 14.0 kB in 0s (0 B/s)"""
#: apt-cacher-ng without a PassThroughPattern, as apt 2.8.3 tells it (NetOrk/netork#1811).
_CONNECT_DENIED = """Err:1 https://download.docker.com/linux/ubuntu/dists/noble/InRelease
Invalid response from proxy: HTTP/1.0 403 CONNECT denied (ask the admin to allow HTTPS tunnels) [IP: 10.40.100.10 3142]
E: Failed to fetch https://download.docker.com/linux/ubuntu/dists/noble/InRelease Invalid response from proxy: HTTP/1.0 403 CONNECT denied (ask the admin to allow HTTPS tunnels) [IP: 10.40.100.10 3142]
E: Download Failed"""
def _fetch(output, rc, match=None):
lines = ["APROXY_BEGIN", "[fetch]", *output.splitlines(), "[rc]", str(rc)]
@@ -243,6 +346,12 @@ class TestFetch:
assert result["expected"] is False
assert result["error"] == "503 Host not found [IP: 172.22.8.50 3142]"
def test_a_refused_tunnel(self):
result = parse_apt_fetch(_fetch(_CONNECT_DENIED, 100))
assert result["reachable"] is False
assert "403 CONNECT denied" in result["error"]
def test_without_apt_helper_nothing_could_be_tried(self):
with pytest.raises(ValueError, match="apt-helper"):
parse_apt_fetch(_fetch("sh: 1: /usr/lib/apt/apt-helper: not found", 127))
@@ -265,6 +374,21 @@ class TestFetchCommand:
command = apt_fetch_command("http://archive.ubuntu.com/ubuntu/dists/noble/InRelease")
assert "DIRECT" not in command
def test_an_https_source_through_apt_s_own_proxy(self):
command = apt_fetch_command(
"https://download.docker.com/linux/ubuntu/dists/noble/InRelease"
)
assert "DIRECT" not in command
assert "download-file https://download.docker.com/" in command
def test_past_the_proxy_to_an_https_host(self):
"""For https the host's own https setting comes first; the http one alone
would lose to an https proxy for that host."""
command = apt_fetch_command("https://h.example/x", direct=True)
assert "'Acquire::https::Proxy::h.example=DIRECT'" in command
assert "'Acquire::http::Proxy::h.example=DIRECT'" in command
@pytest.mark.parametrize("url", ["http://c:3142/x'; rm -rf /", "ftp://c/x", "c:3142"])
def test_a_url_that_would_break_the_command_is_refused(self, url):
with pytest.raises(ValueError):
@@ -275,6 +399,7 @@ def test_the_commands_are_valid_posix_sh():
for command in (
APT_PROXY_COMMAND,
apt_fetch_command("http://c:3142/x", direct=True, expect="y"),
apt_fetch_command("https://h.example/dists/x/InRelease"),
):
subprocess.run(["sh", "-n", "-c", command], check=True)
@@ -283,16 +408,28 @@ def test_the_commands_are_valid_posix_sh():
def test_the_report_command_on_this_host():
output = subprocess.run(["sh", "-c", APT_PROXY_COMMAND], capture_output=True, text=True).stdout
report = parse_apt_proxy(output)
assert set(report) >= {"proxy", "settings", "https_sources"}
assert set(report) >= {"proxy", "https_proxy", "settings", "https_settings", "https_sources"}
_DOCKER = "https://download.docker.com/linux/ubuntu/dists/noble/"
class _Host(AptProxyMixin):
"""A Debian host: its apt files, and what apt-config would make of them."""
"""A Debian host: its apt files, and what apt-config would make of them.
def __init__(self, files=None, fetch_rc=0, apt=True):
*https_sources* are its sources over https. *tunnel* is whether the proxy lets
them through; *direct_rc* is how a fetch of one goes that takes no proxy.
"""
def __init__(
self, files=None, fetch_rc=0, apt=True, https_sources=(), tunnel=True, direct_rc=0
):
self.files = dict(files or {})
self.fetch_rc = fetch_rc
self.apt = apt
self.https_sources = list(https_sources)
self.tunnel = tunnel
self.direct_rc = direct_rc
self.root = []
self.fetched = []
@@ -309,26 +446,36 @@ class _Host(AptProxyMixin):
out.append((path, n, text))
return out
def _effective(self):
def _effective(self, scheme="http"):
value = None
for _path, _n, text in self._settings():
m = re.match(r'\s*Acquire::http::Proxy\s+"([^"]*)"', text, re.I)
m = re.match(rf'\s*Acquire::{scheme}::Proxy\s+"([^"]*)"', text, re.I)
if m:
value = m.group(1)
return value
def _run_apt_command(self, command):
if command == APT_PROXY_COMMAND:
value = self._effective()
value, https = self._effective(), self._effective("https")
dump = [f'Acquire::http::Proxy "{value}";'] if value else []
dump += [f'Acquire::https::Proxy "{https}";'] if https else []
return _report(
dump=[f'Acquire::http::Proxy "{value}";'] if value else [],
dump=dump,
parts=[f"{p}:{n}:{t}" for p, n, t in self._settings() if p.startswith(_D)],
main=[f"{p}:{n}:{t}" for p, n, t in self._settings() if not p.startswith(_D)],
sources=["http://deb.debian.org/debian/dists/trixie/"],
sources=["http://deb.debian.org/debian/dists/trixie/", *self.https_sources],
)
self.fetched.append(command)
if re.search(r"download-file '?https://", command):
return self._fetch_https(command)
return _fetch(_FETCHED if self.fetch_rc == 0 else _REFUSED, self.fetch_rc, match=1)
def _fetch_https(self, command):
https = self._effective("https") or self._effective()
if "DIRECT" in command or not https or https.upper() == "DIRECT":
return _fetch(_FETCHED if self.direct_rc == 0 else _UNRESOLVED, self.direct_rc)
return _fetch(_FETCHED, 0) if self.tunnel else _fetch(_CONNECT_DENIED, 100)
def _write_apt_file(self, path, content):
self.files[path] = content
return {"ok": True, "output": f"wrote {path}"}
@@ -346,6 +493,41 @@ class _Host(AptProxyMixin):
return {"ok": True, "output": done}
class TestCheckHttpsSource:
def test_one_source_the_way_apt_fetches_it(self):
host = _Host(
{f"{_D}00proxy": 'Acquire::http::Proxy "http://c:3142";\n'},
https_sources=[_DOCKER],
tunnel=False,
)
result = host.check_https_source()
assert result["url"] == f"{_DOCKER}InRelease"
assert result["proxy"] == "http://c:3142"
assert result["reachable"] is False and "403 CONNECT denied" in result["error"]
assert "DIRECT" not in host.fetched[0]
def test_none_without_an_https_source(self):
host = _Host({f"{_D}00proxy": 'Acquire::http::Proxy "http://c:3142";\n'})
assert host.check_https_source() is None
def test_none_where_https_goes_direct(self):
host = _Host(
{
f"{_D}00proxy": (
'Acquire::http::Proxy "http://c:3142";\nAcquire::https::Proxy "DIRECT";\n'
)
},
https_sources=[_DOCKER],
)
assert host.check_https_source() is None
def test_not_on_a_host_without_apt(self):
with pytest.raises(NotImplementedError):
_Host(apt=False).check_https_source()
class TestTheMixin:
def test_it_is_a_package_proxy(self):
assert issubclass(AptProxyMixin, PackageProxyMixin)
@@ -371,6 +553,10 @@ class TestTheMixin:
assert "Proxy::c=DIRECT" in host.fetched[0]
def _step(result, name):
return next(s for s in result["steps"] if s["step"] == name)
class TestConfigurePackageProxy:
def test_a_fresh_host(self):
host = _Host()
@@ -378,7 +564,14 @@ class TestConfigurePackageProxy:
result = host.configure_package_proxy("http://c:3142")
assert result["success"] is True
assert [s["step"] for s in result["steps"]] == ["configure", "conflicts", "verify", "fetch"]
assert [s["step"] for s in result["steps"]] == [
"configure",
"conflicts",
"verify",
"fetch",
"https",
]
assert "no https source" in result["steps"][-1]["output"]
assert host.files[f"{_D}00proxy"] == 'Acquire::http::Proxy "http://c:3142";\n'
assert "InRelease" in host.fetched[0] and "DIRECT" not in host.fetched[0]
@@ -410,7 +603,7 @@ class TestConfigurePackageProxy:
def test_a_setting_it_cannot_comment_out_fails_verification(self):
host = _Host({"/etc/apt/apt.conf": 'Acquire::http { Proxy "http://older:8080"; };\n'})
host._effective = lambda: "http://older:8080"
host._effective = lambda scheme="http": "http://older:8080"
result = host.configure_package_proxy("http://c:3142")
@@ -428,6 +621,100 @@ class TestConfigurePackageProxy:
assert result["steps"][-1]["step"] == "fetch"
assert "Connection refused" in result["steps"][-1]["output"]
def test_an_https_source_the_cache_tunnels_stays_with_it(self):
host = _Host(https_sources=[_DOCKER])
result = host.configure_package_proxy("http://c:3142")
assert result["success"] is True
assert host.files[f"{_D}00proxy"] == 'Acquire::http::Proxy "http://c:3142";\n'
assert result["steps"][-1]["output"] == (
f"fetched {_DOCKER}InRelease through http://c:3142"
)
def test_an_https_source_the_cache_refuses_goes_direct(self):
"""NetOrk/netork#1811: apt-cacher-ng without a PassThroughPattern answers
403 to every https source apt sends it."""
host = _Host(https_sources=[_DOCKER], tunnel=False)
result = host.configure_package_proxy("http://c:3142")
assert result["success"] is True
assert host.files[f"{_D}00proxy"] == (
'Acquire::http::Proxy "http://c:3142";\nAcquire::https::Proxy "DIRECT";\n'
)
output = result["steps"][-1]["output"]
assert "http://c:3142 refused https" in output and "403 CONNECT denied" in output
assert f"https sources go direct now: fetched {_DOCKER}InRelease" in output
def test_a_later_https_line_to_the_cache_is_commented_out(self):
"""Ubuntu's installer writes the https proxy too; it would win over 00proxy."""
curtin = f"{_D}90curtin-aptproxy"
host = _Host(
{
curtin: (
'Acquire::http::Proxy "http://c:3142";\n'
'Acquire::https::Proxy "http://c:3142";\n'
)
},
https_sources=[_DOCKER],
tunnel=False,
)
result = host.configure_package_proxy("http://c:3142")
assert result["success"] is True
assert host.files[curtin] == (
'Acquire::http::Proxy "http://c:3142";\n// Acquire::https::Proxy "http://c:3142";\n'
)
assert f"{curtin}:2 (http://c:3142) commented out" in result["steps"][-1]["output"]
def test_an_https_proxy_of_its_own_is_not_the_fix_s(self):
host = _Host(
{f"{_D}01https": 'Acquire::https::Proxy "http://corp:3128";\n'},
https_sources=[_DOCKER],
tunnel=False,
)
result = host.configure_package_proxy("http://c:3142")
assert result["success"] is True
assert "no https source goes through http://c:3142" in result["steps"][-1]["output"]
assert f"{_D}00proxy" in host.files and "https" not in host.files[f"{_D}00proxy"]
def test_an_http_error_through_the_tunnel_is_a_tunnel(self):
"""The mirror answered, so the cache let the source through."""
host = _Host(https_sources=[_DOCKER])
host._fetch_https = lambda command: _fetch(_HTTP_ERROR, 100)
result = host.configure_package_proxy("http://c:3142")
assert result["success"] is True
assert "https" not in host.files[f"{_D}00proxy"]
def test_refused_and_no_way_direct_fails(self):
host = _Host(https_sources=[_DOCKER], tunnel=False, direct_rc=100)
result = host.configure_package_proxy("http://c:3142")
assert result["success"] is False
assert result["steps"][-1]["step"] == "https"
assert "Could not resolve" in result["steps"][-1]["output"]
def test_a_later_https_line_it_cannot_comment_out_fails(self):
host = _Host(
{"/etc/apt/apt.conf": 'Acquire::https { Proxy "http://c:3142"; };\n'},
https_sources=[_DOCKER],
tunnel=False,
)
host._effective = lambda scheme="http": "http://c:3142"
result = host.configure_package_proxy("http://c:3142")
assert result["success"] is False
assert result["steps"][-1]["step"] == "https"
assert "/etc/apt/apt.conf:1" in result["steps"][-1]["output"]
def test_without_an_http_source_the_fetch_says_so(self):
host = _Host()
real = host._run_apt_command
@@ -440,7 +727,7 @@ class TestConfigurePackageProxy:
result = host.configure_package_proxy("http://c:3142")
assert result["success"] is True
assert "no http source" in result["steps"][-1]["output"]
assert "no http source" in _step(result, "fetch")["output"]
def test_a_direct_source_is_not_proof(self):
host = _Host()
@@ -451,7 +738,7 @@ class TestConfigurePackageProxy:
result = host.configure_package_proxy("http://c:3142")
assert "no http source" in result["steps"][-1]["output"]
assert "no http source" in _step(result, "fetch")["output"]
@pytest.mark.parametrize("url", ['http://x"; Acquire::x "y', "ftp://c", "c:3142"])
def test_a_url_that_would_break_the_file_is_refused(self, url):
@@ -531,3 +818,9 @@ def test_the_comment_out_command():
"sed -i -e '2{/^[[:space:]]*Acquire::http::Proxy[[:space:]]/Is|^|// |}' "
"-e '4{/^[[:space:]]*Acquire::http::Proxy[[:space:]]/Is|^|// |}' /etc/apt/apt.conf"
)
def test_the_comment_out_command_for_https():
assert comment_out_command("/etc/apt/apt.conf", [3], scheme="https") == (
"sed -i -e '3{/^[[:space:]]*Acquire::https::Proxy[[:space:]]/Is|^|// |}' /etc/apt/apt.conf"
)
+27
View File
@@ -0,0 +1,27 @@
"""An ARP or neighbour entry whose age the device does not report says so.
NAPALM's ``get_arp_table`` and ``get_ipv6_neighbors_table`` require an ``age``
on every entry, and a driver that could not read one used to fill in ``0.0``.
To a caller that is the freshest sighting there is: netOrk took a neighbour the
device had not confirmed for hours as seen this second (NetOrk/netork#1327).
Unknown is ``-1.0``, as in NAPALM's own IOS driver; ``0.0`` means "just now".
"""
from __future__ import annotations
from napalm_device_types import ARP_AGE_UNKNOWN
from napalm_device_types.arp import ARP_AGE_UNKNOWN as FROM_MODULE
def test_an_unknown_age_is_minus_one():
assert ARP_AGE_UNKNOWN == -1.0
assert isinstance(ARP_AGE_UNKNOWN, float)
def test_the_package_exports_the_modules_constant():
assert ARP_AGE_UNKNOWN is FROM_MODULE
def test_no_real_age_can_be_mistaken_for_unknown():
# An age is seconds since the entry was last confirmed: never negative.
assert ARP_AGE_UNKNOWN < 0
+234
View File
@@ -0,0 +1,234 @@
"""A backup server is a role of its own (NAPALM/napalm-device-types#82).
What makes a device a backup server -- datastores, the backup groups in them,
the jobs that keep them (garbage collection, verify, prune, sync, tape) and the
remotes it syncs with -- has no counterpart in ``StorageDriver``. A backup
server that also manages its own disks lists ``StorageDriver`` as a second role.
The declarations sit under ``TYPE_CHECKING`` (see test_role_contracts), so the
signatures are read from the source rather than from the class.
"""
from __future__ import annotations
import ast
import inspect
from typing import NotRequired, get_args, get_type_hints
import pytest
import napalm_device_types.backup_server as backup_server_module
from napalm_device_types import (
BACKUP_JOB_KINDS,
BackupDatastoreDict,
BackupGroupDict,
BackupRemoteDict,
BackupServerDriver,
BackupServerJobDict,
BackupServerRefDict,
BackupSnapshotDict,
BackupSnapshotRefDict,
BackupTaskDict,
BackupTaskStatusDict,
ClusterStorageDict,
OSDriver,
StorageDriver,
)
from napalm_device_types.backups import BackupJobKind, BackupRunState, BackupVerifyState
from napalm_device_types.roles import primary_role_of, role_keys_of
def _declared(name: str) -> ast.FunctionDef:
tree = ast.parse(inspect.getsource(backup_server_module))
for node in ast.walk(tree):
if isinstance(node, ast.FunctionDef) and node.name == name:
return node
raise AssertionError(f"BackupServerDriver does not declare {name}()")
def _signature(name: str) -> tuple[list[str], str]:
fn = _declared(name)
params = ast.unparse(fn.args).split(", ")[1:]
return params, ast.unparse(fn.returns) if fn.returns else ""
class TestRole:
def test_role_key_and_label(self):
assert vars(BackupServerDriver)["ROLE"] == "backup_server"
assert BackupServerDriver.TYPE_LABEL == "Backup Server"
def test_backup_server_first_storage_second(self):
"""A PBS manages its disks and ZFS pools too; that part is storage."""
class Pbs(BackupServerDriver, StorageDriver, OSDriver):
pass
assert role_keys_of(Pbs) == ["backup_server", "storage", "linux"]
assert primary_role_of(Pbs) == "backup_server"
class TestDeclarations:
@pytest.mark.parametrize(
("name", "params", "returns"),
[
("get_backup_datastores", [], "list[BackupDatastoreDict]"),
("get_backup_groups", ["datastore: str=''"], "list[BackupGroupDict]"),
(
"get_backup_snapshots",
["datastore: str", "namespace: str", "group_type: str", "group_id: str"],
"list[BackupSnapshotDict]",
),
("get_backup_jobs", [], "list[BackupServerJobDict]"),
("get_backup_remotes", [], "list[BackupRemoteDict]"),
("start_garbage_collection", ["datastore: str"], "BackupTaskDict"),
("run_backup_job", ["kind: BackupJobKind", "job_id: str"], "BackupTaskDict"),
("verify_backup_snapshot", ["snapshot: BackupSnapshotRefDict"], "BackupTaskDict"),
(
"set_backup_snapshot_protected",
["snapshot: BackupSnapshotRefDict", "protected: bool"],
"None",
),
("delete_backup_snapshot", ["snapshot: BackupSnapshotRefDict"], "None"),
("get_backup_task", ["task: str"], "BackupTaskStatusDict"),
],
)
def test_declared_with_its_signature(self, name, params, returns):
got_params, got_returns = _signature(name)
assert [p.replace(" ", "") for p in got_params] == [p.replace(" ", "") for p in params]
assert got_returns == returns
@pytest.mark.parametrize(
"name",
[
"get_backup_datastores",
"get_backup_groups",
"get_backup_snapshots",
"get_backup_jobs",
"get_backup_remotes",
"start_garbage_collection",
"run_backup_job",
"verify_backup_snapshot",
"set_backup_snapshot_protected",
"delete_backup_snapshot",
"get_backup_task",
],
)
def test_absent_until_a_driver_implements_it(self, name):
assert not hasattr(BackupServerDriver, name)
def test_every_declaration_has_a_docstring(self):
tree = ast.parse(inspect.getsource(backup_server_module))
missing = [
n.name
for n in ast.walk(tree)
if isinstance(n, ast.FunctionDef) and not ast.get_docstring(n)
]
assert missing == []
class TestShapes:
"""Neutral names: nothing here is Proxmox's. A value the platform does not
report is None; it is never guessed."""
def test_datastore(self):
assert set(get_type_hints(BackupDatastoreDict)) == {
"name",
"path",
"backend",
"total",
"used",
"available",
"estimated_full",
"maintenance",
"comment",
}
def test_group_carries_its_newest_snapshot_not_all_of_them(self):
assert set(get_type_hints(BackupGroupDict)) == {
"datastore",
"namespace",
"type",
"id",
"owner",
"count",
"newest",
"oldest_time",
"comment",
}
assert get_type_hints(BackupGroupDict)["newest"] == BackupSnapshotDict | None
def test_snapshot(self):
assert set(get_type_hints(BackupSnapshotDict)) == {
"time",
"size",
"verify",
"protected",
"comment",
}
assert set(get_args(BackupVerifyState)) == {"ok", "failed"}
def test_snapshot_reference_names_one_snapshot(self):
assert set(get_type_hints(BackupSnapshotRefDict)) == {
"datastore",
"namespace",
"group_type",
"group_id",
"time",
}
def test_job_kinds_include_garbage_collection(self):
"""GC is a job with a schedule and a last run like the others, so it is
listed with them rather than restated on the datastore."""
assert set(get_args(BackupJobKind)) == {"gc", "verify", "prune", "sync", "tape"}
assert set(BACKUP_JOB_KINDS) == set(get_args(BackupJobKind))
def test_job(self):
assert set(get_type_hints(BackupServerJobDict)) == {
"id",
"kind",
"datastore",
"namespace",
"schedule",
"enabled",
"last_run",
"last_state",
"last_error",
"next_run",
"remote",
"direction",
"comment",
}
assert set(get_args(BackupRunState)) == {"ok", "warning", "error", "running"}
def test_remote_never_carries_its_secret(self):
fields = set(get_type_hints(BackupRemoteDict))
assert fields == {"name", "host", "port", "fingerprint", "auth_id", "comment"}
assert not fields & {"password", "secret", "token"}
def test_task(self):
assert set(get_type_hints(BackupTaskDict)) == {"task"}
assert set(get_type_hints(BackupTaskStatusDict)) == {
"task",
"running",
"state",
"error",
"log",
}
class TestClusterStorageNamesItsBackupServer:
"""A hypervisor's backup storage says which server and datastore it points
at, so an application can tie the server's groups to the cluster's guests."""
def test_backup_server_reference_is_optional_on_a_storage(self):
hints = get_type_hints(ClusterStorageDict, include_extras=True)
assert hints["backup_server"] == NotRequired[BackupServerRefDict | None]
def test_reference(self):
assert set(get_type_hints(BackupServerRefDict)) == {
"server",
"port",
"datastore",
"namespace",
"fingerprint",
}
+173
View File
@@ -0,0 +1,173 @@
"""A hypervisor says which cluster it belongs to, and what that cluster holds.
A Proxmox node is one member of a cluster and can read the whole of it; a vCenter
manages several clusters and standalone hosts at once; a standalone host is a
cluster of one. One getter serves all three: ``get_clusters()`` lists every
cluster the connection can see, each with its nodes, its storages and its
networks, and which nodes see which. What a caller does with that (placement,
headroom, identity) is the caller's.
A new VM can be placed on a named node (``create_vm_from_cloud_init(node=…)``),
so a manager that runs several hosts takes the caller's choice instead of making
its own.
The declarations sit under ``TYPE_CHECKING`` (see test_role_contracts), so the
signatures are read from the source rather than from the class.
"""
from __future__ import annotations
import ast
import inspect
from typing import NotRequired, get_type_hints
import napalm_device_types.hypervisor as hypervisor_module
from napalm_device_types import (
BackupDict,
BackupJobDict,
BackupServerRefDict,
ClusterDict,
ClusterNetworkDict,
ClusterNodeDict,
ClusterStorageDict,
HAResourceDict,
HypervisorDriver,
)
def _declared(name: str) -> ast.FunctionDef:
tree = ast.parse(inspect.getsource(hypervisor_module))
for node in ast.walk(tree):
if isinstance(node, ast.FunctionDef) and node.name == name:
return node
raise AssertionError(f"HypervisorDriver does not declare {name}()")
class TestClusterListing:
def test_is_declared(self):
fn = _declared("get_clusters")
assert ast.unparse(fn.returns) == "list[ClusterDict]"
def test_absent_until_a_driver_implements_it(self):
"""netOrk probes capabilities with hasattr; a hypervisor that cannot
describe its cluster must not seem to."""
assert not hasattr(HypervisorDriver, "get_clusters")
class TestClusterShape:
def test_cluster(self):
assert get_type_hints(ClusterDict, include_extras=True) == {
"id": str | None,
"name": str | None,
"quorate": bool | None,
"nodes": list[ClusterNodeDict],
"storages": list[ClusterStorageDict],
"networks": list[ClusterNetworkDict],
"ha_resources": NotRequired[list[HAResourceDict] | None],
"backup_jobs": NotRequired[list[BackupJobDict] | None],
"backups": NotRequired[list[BackupDict] | None],
}
def test_cluster_keys_added_later_stay_optional(self):
"""A driver written before HA and backups were part of the report stays valid."""
assert ClusterDict.__required_keys__ == frozenset(
{"id", "name", "quorate", "nodes", "storages", "networks"}
)
def test_node(self):
hints = get_type_hints(ClusterNodeDict, include_extras=True)
assert hints == {
"name": str,
"status": str,
"local": bool,
"maintenance": bool | None,
"cpu_threads": int | None,
"cpu_cores": int | None,
"cpu_model": str | None,
"cpu_flags": list[str] | None,
"cpu_usage": float | None,
"memory_total": int | None,
"memory_used": int | None,
"votes": NotRequired[int | None],
}
assert "votes" not in ClusterNodeDict.__required_keys__
def test_storage(self):
hints = get_type_hints(ClusterStorageDict, include_extras=True)
assert hints == {
"name": str,
"type": str,
"shared": bool,
"content": list[str],
"nodes": list[str],
"total": int | None,
"used": int | None,
"backup_server": NotRequired[BackupServerRefDict | None],
}
# A driver written before backup servers were part of the report stays valid.
assert "backup_server" not in ClusterStorageDict.__required_keys__
def test_network(self):
hints = get_type_hints(ClusterNetworkDict, include_extras=True)
assert hints == {
"name": str,
"kind": str,
"vlan_aware": bool,
"fixed_vlan_tag": NotRequired[int | None],
"nodes": list[str],
}
assert ClusterNetworkDict.__required_keys__ == frozenset(
{"name", "kind", "vlan_aware", "nodes"}
)
class TestHAAndBackupShapes:
def test_ha_resource(self):
hints = get_type_hints(HAResourceDict, include_extras=True)
assert hints == {
"vmid": int,
"kind": str,
"requested": str,
"status": str | None,
"node": str | None,
"group": NotRequired[str | None],
"rules": list[str],
}
assert HAResourceDict.__required_keys__ == frozenset(
{"vmid", "kind", "requested", "status", "node", "rules"}
)
def test_backup_job(self):
assert get_type_hints(BackupJobDict) == {
"id": str,
"enabled": bool,
"schedule": str | None,
"storage": str | None,
"mode": str | None,
"all": bool,
"vmids": list[int],
"exclude": list[int],
"pool": str | None,
"node": str | None,
}
def test_backup(self):
assert get_type_hints(BackupDict) == {
"vmid": int,
"storage": str,
"time": int,
"size": int | None,
"count": int,
}
class TestCreateVmTakesANode:
def test_node_is_an_optional_keyword(self):
fn = _declared("create_vm_from_cloud_init")
kwonly = {
arg.arg: default
for arg, default in zip(fn.args.kwonlyargs, fn.args.kw_defaults, strict=True)
}
assert "node" in kwonly
default = kwonly["node"]
assert isinstance(default, ast.Constant) and default.value is None
+172
View File
@@ -0,0 +1,172 @@
"""ConfigRevisionsMixin: a device's own configuration history (NetOrk/netork#2511).
Some devices keep their earlier configurations themselves. An application restores a
configuration there by reverting to the revision that holds it -- on OPNsense the only
way, since no API takes an uploaded config.xml. Listing and reading are declared only,
so ``callable(getattr(driver, "get_config_revisions", None))`` is the truthful answer;
``revert_config_revision`` is a template: it refuses an id that is not in the history,
lets the driver revert, and checks that the configuration in place now is the
revision's.
"""
from __future__ import annotations
import inspect
from typing import get_type_hints
import pytest
from napalm_device_types import (
ConfigRevertFailed,
ConfigRevisionDict,
ConfigRevisionsMixin,
DeviceTypeDriver,
FirewallDriver,
OSDriver,
SwitchDriver,
UnknownConfigRevision,
)
def test_the_shape():
assert get_type_hints(ConfigRevisionDict) == {
"id": str,
"time": str | None,
"description": str | None,
"user": str | None,
"size": int | None,
}
def test_no_role_demands_it():
for role in (FirewallDriver, OSDriver, SwitchDriver, DeviceTypeDriver):
assert not issubclass(role, ConfigRevisionsMixin)
def test_listing_and_reading_are_declared_not_implemented():
assert not hasattr(ConfigRevisionsMixin, "get_config_revisions")
assert not hasattr(ConfigRevisionsMixin, "get_config_revision")
source = inspect.getsource(ConfigRevisionsMixin)
assert "def get_config_revisions(self) -> list[ConfigRevisionDict]" in source
assert "def get_config_revision(self, revision_id: str) -> str" in source
assert "def _revert_to_config_revision(self, revision_id: str) -> None" in source
def test_an_unknown_revision_is_a_value_error_that_names_it():
error = UnknownConfigRevision("r9")
assert isinstance(error, ValueError)
assert "'r9'" in str(error)
def test_a_failed_revert_is_a_runtime_error():
assert issubclass(ConfigRevertFailed, RuntimeError)
def _revision(revision_id: str, time: str) -> ConfigRevisionDict:
return {"id": revision_id, "time": time, "description": None, "user": None, "size": None}
class _Device(ConfigRevisionsMixin):
"""A device that keeps its history in memory, newest first, and saves a revert as a
new newest revision with a stamp of its own on the first line."""
def __init__(self) -> None:
self.history = [
(_revision("r3", "2026-11-13T21:00:00+00:00"), "#saved 3\nb\n"),
(_revision("r2", "2026-11-12T21:00:00+00:00"), "#saved 2\na\n"),
(_revision("r1", "2026-11-11T21:00:00+00:00"), "#saved 1\na\n"),
]
self.calls: list[str] = []
def get_config_revisions(self) -> list[ConfigRevisionDict]:
self.calls.append("list")
return [revision for revision, _ in self.history]
def get_config_revision(self, revision_id: str) -> str:
self.calls.append(f"read {revision_id}")
for revision, text in self.history:
if revision["id"] == revision_id:
return text
raise UnknownConfigRevision(revision_id)
def _revert_to_config_revision(self, revision_id: str) -> None:
self.calls.append(f"revert {revision_id}")
body = self.get_config_revision(revision_id).split("\n", 1)[1]
self.history.insert(0, (_revision("r4", "2026-11-14T08:00:00+00:00"), f"#saved 4\n{body}"))
def config_revision_content(self, text: str) -> str:
return text.split("\n", 1)[1]
class TestRevertConfigRevision:
def test_reverts_and_answers_with_the_revision_now_in_place(self):
device = _Device()
now = device.revert_config_revision("r1")
assert now["id"] == "r4"
assert device.calls == ["list", "read r1", "revert r1", "read r1", "list", "read r4"]
def test_the_revision_is_read_before_the_revert_which_may_rotate_it_out(self):
class Rotating(_Device):
def _revert_to_config_revision(self, revision_id: str) -> None:
super()._revert_to_config_revision(revision_id)
self.history.pop() # the oldest, r1, drops out
assert Rotating().revert_config_revision("r1")["id"] == "r4"
def test_a_revision_not_in_the_history_is_refused_before_anything_changes(self):
device = _Device()
with pytest.raises(UnknownConfigRevision, match="r9"):
device.revert_config_revision("r9")
assert device.calls == ["list"]
def test_a_revert_the_device_did_not_take_fails(self):
# OPNsense answers "reverted" even when it fell back to the previous file.
class Deaf(_Device):
def _revert_to_config_revision(self, revision_id: str) -> None:
self.calls.append(f"revert {revision_id}")
with pytest.raises(ConfigRevertFailed, match="r1"):
Deaf().revert_config_revision("r1")
def test_a_history_that_is_empty_afterwards_fails(self):
class Forgetful(_Device):
def _revert_to_config_revision(self, revision_id: str) -> None:
self.history.clear()
with pytest.raises(ConfigRevertFailed):
Forgetful().revert_config_revision("r1")
def test_what_a_save_stamps_anew_does_not_count(self):
# r2 and r1 differ only in their stamp: reverting to r2 leaves r1's content.
device = _Device()
assert device.revert_config_revision("r2")["id"] == "r4"
def test_a_refusal_of_the_device_reaches_the_caller(self):
class Refusing(_Device):
def _revert_to_config_revision(self, revision_id: str) -> None:
raise ConfigRevertFailed("read-only user")
with pytest.raises(ConfigRevertFailed, match="read-only"):
Refusing().revert_config_revision("r1")
class TestConfigRevisionContent:
def test_by_default_the_whole_text(self):
class Plain(ConfigRevisionsMixin):
pass
assert Plain().config_revision_content("a\nb\n") == "a\nb\n"
def test_without_a_stamp_every_save_differs(self):
class Plain(_Device):
def config_revision_content(self, text: str) -> str:
return ConfigRevisionsMixin.config_revision_content(self, text)
with pytest.raises(ConfigRevertFailed):
Plain().revert_config_revision("r1")
+141
View File
@@ -0,0 +1,141 @@
"""Cron expressions: what they mean, when they fire next, and as systemd ``OnCalendar``.
One module answers all three, because a job netOrk writes as a timer must fire
exactly when the cron expression the user gave says, and the preview must show
the same runs the host will make (NetOrk/netork#1933).
"""
from __future__ import annotations
from datetime import UTC, datetime
from zoneinfo import ZoneInfo
import pytest
from napalm_device_types.cron import is_reboot, next_runs, parse_cron, to_on_calendar
BERLIN = ZoneInfo("Europe/Berlin")
class TestParse:
def test_every_field_is_expanded_to_its_values(self):
s = parse_cron("*/15 6-8 1,15 * *")
assert s.minutes == frozenset({0, 15, 30, 45})
assert s.hours == frozenset({6, 7, 8})
assert s.days == frozenset({1, 15})
assert s.months == frozenset(range(1, 13))
assert s.weekdays == frozenset(range(7))
def test_a_range_takes_a_step(self):
assert parse_cron("1-30/10 * * * *").minutes == frozenset({1, 11, 21})
def test_names_are_months_and_weekdays_in_any_case(self):
s = parse_cron("0 0 * Jan,MAR mon-Fri")
assert s.months == frozenset({1, 3})
assert s.weekdays == frozenset({1, 2, 3, 4, 5})
def test_seven_is_sunday_too(self):
assert parse_cron("0 0 * * 7").weekdays == frozenset({0})
@pytest.mark.parametrize(
("alias", "expanded"),
[
("@hourly", "0 * * * *"),
("@daily", "0 0 * * *"),
("@midnight", "0 0 * * *"),
("@weekly", "0 0 * * 0"),
("@monthly", "0 0 1 * *"),
("@yearly", "0 0 1 1 *"),
("@annually", "0 0 1 1 *"),
],
)
def test_an_alias_is_its_expression(self, alias, expanded):
assert parse_cron(alias) == parse_cron(expanded)
def test_reboot_is_no_calendar(self):
assert is_reboot("@reboot")
assert not is_reboot("0 0 * * *")
with pytest.raises(ValueError, match="reboot"):
parse_cron("@reboot")
@pytest.mark.parametrize(
"bad",
[
"",
"* * * *",
"* * * * * *",
"60 * * * *",
"* 24 * * *",
"* * 0 * *",
"* * * 13 *",
"* * * * 8",
"*/0 * * * *",
"5-1 * * * *",
"a * * * *",
"@often",
"1,,2 * * * *",
],
)
def test_what_cron_would_not_take_is_refused(self, bad):
with pytest.raises(ValueError):
parse_cron(bad)
def test_day_of_month_and_weekday_are_or_only_when_both_are_restricted(self):
assert parse_cron("0 0 1 * 1").either_day
assert not parse_cron("0 0 */2 * 1").either_day
assert not parse_cron("0 0 1 * *").either_day
class TestNextRuns:
def test_runs_after_the_instant_in_the_hosts_time(self):
after = datetime(2026, 11, 3, 5, 50, tzinfo=UTC) # 06:50 in Berlin
runs = next_runs("0 7 * * *", after, BERLIN, count=2)
assert runs == [
datetime(2026, 11, 3, 7, 0, tzinfo=BERLIN),
datetime(2026, 11, 4, 7, 0, tzinfo=BERLIN),
]
def test_the_instant_itself_is_not_a_next_run(self):
after = datetime(2026, 11, 3, 6, 0, tzinfo=UTC)
assert next_runs("0 6 * * *", after, ZoneInfo("UTC"), count=1)[0].day == 4
def test_either_day_fires_on_both(self):
after = datetime(2026, 11, 30, 12, 0, tzinfo=UTC) # a Monday
runs = next_runs("0 0 1 * 1", after, ZoneInfo("UTC"), count=3)
assert [r.day for r in runs] == [1, 7, 14]
def test_a_day_that_never_comes_gives_no_runs(self):
assert next_runs("0 0 31 2 *", datetime(2026, 1, 1, tzinfo=UTC), BERLIN) == []
def test_a_time_the_clock_skips_is_left_out(self):
# 2026-03-29 02:30 does not exist in Berlin.
after = datetime(2026, 3, 28, 12, 0, tzinfo=UTC)
runs = next_runs("30 2 * * *", after, BERLIN, count=2)
assert [r.day for r in runs] == [30, 31]
def test_reboot_has_no_next_run(self):
assert next_runs("@reboot", datetime(2026, 1, 1, tzinfo=UTC), BERLIN) == []
class TestOnCalendar:
def test_every_field_as_its_values(self):
assert to_on_calendar("*/15 6 * * *") == ["*-*-* 06:00,15,30,45:00"]
def test_a_restricted_weekday_leads(self):
assert to_on_calendar("30 4 * * 1-5") == ["Mon,Tue,Wed,Thu,Fri *-*-* 04:30:00"]
def test_months_and_days(self):
assert to_on_calendar("0 0 1,15 1,7 *") == ["*-01,07-01,15 00:00:00"]
def test_either_day_is_two_lines(self):
assert to_on_calendar("0 0 1 * 0") == ["*-*-01 00:00:00", "Sun *-*-* 00:00:00"]
def test_both_days_with_a_star_are_one_line(self):
assert to_on_calendar("0 0 */10 * 6") == ["Sat *-*-01,11,21,31 00:00:00"]
def test_an_alias(self):
assert to_on_calendar("@daily") == ["*-*-* 00:00:00"]
def test_reboot_is_refused(self):
with pytest.raises(ValueError, match="reboot"):
to_on_calendar("@reboot")
+145
View File
@@ -0,0 +1,145 @@
"""Crontabs as text: the jobs in them, and changing one line (NetOrk/netork#1933)."""
from __future__ import annotations
import subprocess
import pytest
from napalm_device_types._crontab import (
DISABLED_MARKER,
marked_lines,
parse_anacrontab,
parse_crontab,
run_script,
switched,
with_job,
without_job,
)
SYSTEM = """\
# /etc/crontab: system-wide crontab
SHELL=/bin/sh
PATH="/usr/local/sbin:/usr/bin:/bin"
17 * * * * root cd / && run-parts --report /etc/cron.hourly
@reboot root /usr/local/bin/warm-cache # keeps the # in the command
#napalm-disabled: 0 4 * * * backup /usr/local/bin/backup
garbage line
0 4 * * * backup /usr/local/bin/backup
"""
USER = """\
MAILTO=ops@example.org
*/5 * * * * /usr/bin/php /var/www/cron.php
# napalm-scheduled-job: netork-cleanup tidy the cache
0 3 * * * find /tmp -mtime +7 -delete
"""
class TestParse:
def test_a_system_crontab_names_the_user(self):
jobs = parse_crontab("/etc/crontab", SYSTEM, system=True)
first = jobs[0]
assert (first.schedule, first.run_as) == ("17 * * * *", "root")
assert first.command == "cd / && run-parts --report /etc/cron.hourly"
assert first.env == {"SHELL": "/bin/sh", "PATH": "/usr/local/sbin:/usr/bin:/bin"}
assert first.key.startswith("cron:/etc/crontab:")
def test_an_alias_and_a_hash_inside_the_command(self):
reboot = parse_crontab("/etc/crontab", SYSTEM, system=True)[1]
assert reboot.schedule == "@reboot"
assert reboot.command == "/usr/local/bin/warm-cache # keeps the # in the command"
def test_a_line_switched_off_is_a_job_that_is_off(self):
jobs = parse_crontab("/etc/crontab", SYSTEM, system=True)
off = jobs[2]
assert not off.enabled
assert off.run_as == "backup"
def test_equal_lines_get_their_own_keys_and_switching_off_keeps_the_key(self):
jobs = parse_crontab("/etc/crontab", SYSTEM, system=True)
assert jobs[3].key == jobs[2].key + ":2"
assert len(jobs) == 4 # the garbage line is left out
def test_a_user_crontab_runs_as_its_user(self):
jobs = parse_crontab("/var/spool/cron/crontabs/www", USER, system=False, user="www-data")
assert jobs[0].run_as == "www-data"
assert jobs[0].env == {"MAILTO": "ops@example.org"}
def test_an_applications_job_is_keyed_by_its_name(self):
job = parse_crontab("/x", USER, system=False, user="root")[1]
assert (job.key, job.name, job.description) == (
"job:netork-cleanup",
"netork-cleanup",
"tidy the cache",
)
def test_an_anacrontab(self):
text = "SHELL=/bin/sh\n1\t5\tcron.daily\trun-parts --report /etc/cron.daily\n@monthly 15 cron.monthly run-parts /etc/cron.monthly\nbad line here\n"
jobs = parse_anacrontab(text)
assert [(j.key, j.schedule, j.run_as) for j in jobs] == [
("anacron:cron.daily", "1", "root"),
("anacron:cron.monthly", "@monthly", "root"),
]
assert jobs[0].command == "run-parts --report /etc/cron.daily"
class TestEdit:
def test_switching_off_and_on_again_gives_the_text_back(self):
jobs = parse_crontab("/etc/crontab", SYSTEM, system=True)
off = switched(SYSTEM, jobs[0], enabled=False)
assert DISABLED_MARKER + "17 *\t* * *\troot" in off
again = parse_crontab("/etc/crontab", off, system=True)
assert again[0].key == jobs[0].key and not again[0].enabled
assert switched(off, again[0], enabled=True) == SYSTEM.replace(
"17 *\t* * *\troot\tcd", "17 *\t* * *\troot\tcd"
)
def test_an_applications_job_is_added_replaced_and_removed(self):
lines = marked_lines("netork-x", "a job", "0 1 * * * /bin/true", enabled=True)
added = with_job("A=1\n", [], "netork-x", lines)
assert added == "A=1\n# napalm-scheduled-job: netork-x a job\n0 1 * * * /bin/true\n"
jobs = parse_crontab("/x", added, system=False, user="root")
changed = with_job(
added,
jobs,
"netork-x",
marked_lines("netork-x", "", "0 2 * * * /bin/true", enabled=False),
)
assert (
changed
== "A=1\n# napalm-scheduled-job: netork-x\n#napalm-disabled: 0 2 * * * /bin/true\n"
)
jobs = parse_crontab("/x", changed, system=False, user="root")
assert without_job(changed, jobs, "netork-x") == "A=1\n"
def test_adding_to_a_text_without_a_last_newline(self):
assert with_job("A=1", [], "n", ["# m", "x"]) == "A=1\n# m\nx\n"
def test_removing_the_only_job_leaves_nothing(self):
text = "# napalm-scheduled-job: n\n0 1 * * * x\n"
jobs = parse_crontab("/x", text, system=False, user="root")
assert without_job(text, jobs, "n") == ""
class TestRunScript:
def _run(self, script: str) -> str:
return subprocess.run(
["sh", "-c", script], capture_output=True, text=True, check=True
).stdout
def test_the_crontabs_variables_are_set(self):
assert (
self._run(run_script('echo "$GREETING"', {"GREETING": "hi there", "MAILTO": "x"}))
== "hi there\n"
)
def test_a_percent_starts_stdin_and_further_ones_are_newlines(self):
assert self._run(run_script("cat%one%two", {})) == "one\ntwo\n"
def test_an_escaped_percent_is_a_percent(self):
assert self._run(run_script("date -d @0 -u +\\%Y", {})) == "1970\n"
@pytest.mark.parametrize("command", ["echo plain", "echo 'quoted'"])
def test_without_percent_the_command_is_as_it_is(self, command):
assert run_script(command, {}) == command
+240 -1
View File
@@ -14,7 +14,7 @@ from typing import Any
import pytest
from napalm_device_types import DhcpServerMixin, FirewallDriver, ResidentialGatewayDriver
from napalm_device_types.dhcp import normalize_mac
from napalm_device_types.dhcp import DhcpApplyError, normalize_mac
from napalm_device_types.models import DhcpReservationDict
@@ -209,6 +209,245 @@ class TestApplyReservationSet:
assert lines == ["[commit] no changes"]
class _RefusingDhcpServer(_FakeDhcpServer):
"""A server that refuses the reservations for the IPs in *refused*, as Kea does
for one whose subnet it does not manage."""
def __init__(self, live: list[DhcpReservationDict] | None = None, *, refused: set[str]) -> None:
super().__init__(live)
self.refused = refused
def apply_dhcp_reservation(
self, reservation: DhcpReservationDict, *, uuid: str | None = None
) -> dict[str, Any]:
if reservation["ip"] in self.refused:
raise ValueError(f"No Kea-managed subnet contains {reservation['ip']}")
return super().apply_dhcp_reservation(reservation, uuid=uuid)
def _three() -> list[DhcpReservationDict]:
return [
_reservation(mac="aa:bb:cc:dd:ee:01", ip="10.10.20.50"),
_reservation(mac="aa:bb:cc:dd:ee:02", ip="10.10.99.5", subnet=""),
_reservation(mac="aa:bb:cc:dd:ee:03", ip="10.10.20.52"),
]
class TestApplyWithARefusedReservation:
"""One reservation the server refuses must not hold back the others, and what was
written must take effect: a server that saved a write and was never committed shows
it as configured, so the next apply finds nothing to do and never commits it."""
def test_the_others_are_still_written(self):
driver = _RefusingDhcpServer(refused={"10.10.99.5"})
with pytest.raises(DhcpApplyError):
list(driver.apply_dhcp_reservationset(_three()))
assert [entry["reservation"]["ip"] for entry in driver.applied] == [
"10.10.20.50",
"10.10.20.52",
]
def test_what_was_written_is_committed(self):
driver = _RefusingDhcpServer(refused={"10.10.99.5"})
with pytest.raises(DhcpApplyError):
list(driver.apply_dhcp_reservationset(_three()))
assert driver.commits == 1
def test_a_refused_set_still_commits(self):
# An earlier apply that stopped before its commit left saved writes behind; a
# run whose every write is refused commits too, so those take effect.
driver = _RefusingDhcpServer(refused={"10.10.99.5"})
with pytest.raises(DhcpApplyError):
list(driver.apply_dhcp_reservationset([_three()[1]]))
assert driver.commits == 1
def test_progress_names_the_refused_one_and_why(self):
driver = _RefusingDhcpServer(refused={"10.10.99.5"})
lines: list[str] = []
with pytest.raises(DhcpApplyError) as raised:
for line in driver.apply_dhcp_reservationset(_three()):
lines.append(line)
failed = [line for line in lines if line.startswith("[failed]")]
assert failed == [
"[failed] aa:bb:cc:dd:ee:02 -> 10.10.99.5: No Kea-managed subnet contains 10.10.99.5"
]
assert lines[-1] == "[commit] applied 2 add(s), 0 update(s)"
assert "1 of 3" in str(raised.value)
def test_a_refused_update_is_reported_like_an_add(self):
driver = _RefusingDhcpServer(
[_reservation(uuid="u1", ip="10.10.20.9")], refused={"10.10.20.50"}
)
with pytest.raises(DhcpApplyError, match="1 of 1"):
list(driver.apply_dhcp_reservationset([_reservation()]))
assert driver.applied == []
assert driver.commits == 1
def test_the_error_is_a_runtime_error(self):
# Callers that catch RuntimeError for a failed apply keep working.
assert issubclass(DhcpApplyError, RuntimeError)
#: What Kea answers through OPNsense's API for an address another reservation holds.
_KEA_DUPLICATE = (
"{'result': 'failed', 'validations': {'reservation.ip_address': 'Duplicate entry exists.'}}"
)
class _KeaLikeServer(_FakeDhcpServer):
"""A server that keeps each address unique at every write, as Kea does: a write
taking an address another reservation still holds is refused."""
def apply_dhcp_reservation(
self, reservation: DhcpReservationDict, *, uuid: str | None = None
) -> dict[str, Any]:
if any(r["ip"] == reservation["ip"] and r["uuid"] != uuid for r in self.live):
raise RuntimeError(_KEA_DUPLICATE)
stored = {**reservation, "uuid": uuid or f"new-{len(self.live)}"}
self.live = [r for r in self.live if r["uuid"] != uuid] + [stored] # type: ignore[list-item]
return super().apply_dhcp_reservation(reservation, uuid=uuid)
def ip_of(self, mac: str) -> str:
return next(r["ip"] for r in self.live if r["mac"] == mac)
_A, _B, _C, _N = (f"02:00:00:00:00:0{i}" for i in "abcd")
def _at(mac: str, ip: str, uuid: str = "") -> DhcpReservationDict:
return _reservation(mac=mac, ip=ip, uuid=uuid, subnet="192.0.2.0/24")
class TestApplyOrderReleasesBeforeTaking:
"""An address is released before another change takes it (#106)."""
def test_an_update_that_frees_an_address_goes_first(self):
# The issue: B takes A's address, A moves on; desired lists B first.
driver = _KeaLikeServer([_at(_A, "192.0.2.10", "ua"), _at(_B, "192.0.2.11", "ub")])
lines = list(
driver.apply_dhcp_reservationset([_at(_B, "192.0.2.10"), _at(_A, "192.0.2.20")])
)
assert driver.ip_of(_A) == "192.0.2.20"
assert driver.ip_of(_B) == "192.0.2.10"
assert lines == [
f"[update] {_A} -> 192.0.2.20 (ip)",
f"[update] {_B} -> 192.0.2.10 (ip)",
"[commit] applied 0 add(s), 2 update(s)",
]
def test_a_chain_is_written_from_its_free_end(self):
live = [
_at(_A, "192.0.2.10", "ua"),
_at(_B, "192.0.2.11", "ub"),
_at(_C, "192.0.2.12", "uc"),
]
driver = _KeaLikeServer(live)
list(
driver.apply_dhcp_reservationset(
[_at(_A, "192.0.2.11"), _at(_B, "192.0.2.12"), _at(_C, "192.0.2.13")]
)
)
assert [e["reservation"]["mac"] for e in driver.applied] == [_C, _B, _A]
assert [driver.ip_of(m) for m in (_A, _B, _C)] == ["192.0.2.11", "192.0.2.12", "192.0.2.13"]
def test_an_add_comes_after_the_update_that_frees_its_address(self):
driver = _KeaLikeServer([_at(_A, "192.0.2.10", "ua")])
lines = list(
driver.apply_dhcp_reservationset([_at(_N, "192.0.2.10"), _at(_A, "192.0.2.20")])
)
assert driver.ip_of(_N) == "192.0.2.10"
assert driver.ip_of(_A) == "192.0.2.20"
assert lines[-1] == "[commit] applied 1 add(s), 1 update(s)"
def test_addresses_match_however_they_are_spelled(self):
driver = _FakeDhcpServer([_at(_A, "2001:DB8::10", "ua"), _at(_B, "2001:DB8::11", "ub")])
list(driver.apply_dhcp_reservationset([_at(_B, "2001:db8::10"), _at(_A, "2001:db8::20")]))
assert [e["reservation"]["mac"] for e in driver.applied] == [_A, _B]
def test_an_update_that_keeps_its_address_frees_nothing(self):
# A hostname change is no release: the add for A's address stays refused.
driver = _KeaLikeServer([_at(_A, "192.0.2.10", "ua")])
with pytest.raises(DhcpApplyError, match="1 of 2"):
list(
driver.apply_dhcp_reservationset(
[_at(_N, "192.0.2.10"), {**_at(_A, "192.0.2.10"), "hostname": "renamed"}]
)
)
assert driver.ip_of(_A) == "192.0.2.10"
class TestApplyOrderWithASwap:
"""Two reservations trading addresses: no order frees one first. Each is still
tried -- a server that checks only at the commit takes them -- and a refusal says it
was a swap. Nothing is moved to a temporary address: the contract has no way to know
a free one, and a second write that failed would leave the reservation there."""
def _swap(self) -> list[DhcpReservationDict]:
return [_at(_A, "192.0.2.11"), _at(_B, "192.0.2.10"), _at(_N, "192.0.2.30")]
def _live(self) -> list[DhcpReservationDict]:
return [_at(_A, "192.0.2.10", "ua"), _at(_B, "192.0.2.11", "ub")]
def test_a_refused_swap_is_reported_as_one_and_the_rest_is_written(self):
driver = _KeaLikeServer(self._live())
lines: list[str] = []
with pytest.raises(DhcpApplyError, match="2 of 3"):
for line in driver.apply_dhcp_reservationset(self._swap()):
lines.append(line)
failed = [line for line in lines if line.startswith("[failed]")]
assert len(failed) == 2
assert all("Duplicate entry exists" in line for line in failed)
assert f"swap with {_B}" in failed[0] and f"swap with {_A}" in failed[1]
assert driver.ip_of(_A) == "192.0.2.10" and driver.ip_of(_B) == "192.0.2.11"
assert driver.ip_of(_N) == "192.0.2.30"
assert driver.commits == 1
def test_a_server_that_takes_the_swap_gets_it(self):
driver = _FakeDhcpServer(self._live())
lines = list(driver.apply_dhcp_reservationset(self._swap()))
assert len(driver.applied) == 3
assert not any("swap" in line for line in lines)
def test_a_rotation_of_three_is_a_swap_too(self):
driver = _KeaLikeServer([*self._live(), _at(_C, "192.0.2.12", "uc")])
lines: list[str] = []
with pytest.raises(DhcpApplyError, match="3 of 3"):
for line in driver.apply_dhcp_reservationset(
[_at(_A, "192.0.2.11"), _at(_B, "192.0.2.12"), _at(_C, "192.0.2.10")]
):
lines.append(line)
by_mac = {line.split()[1]: line for line in lines if line.startswith("[failed]")}
assert f"swap with {_B}, {_C}" in by_mac[_A]
assert f"swap with {_A}, {_C}" in by_mac[_B]
assert f"swap with {_A}, {_B}" in by_mac[_C]
class TestKeysThatCannotNameOneReservation:
def test_a_live_reservation_without_mac_is_never_matched(self):
driver = _FakeDhcpServer([_reservation(uuid="b", mac="")])
+90
View File
@@ -16,6 +16,7 @@ from typing import Any
import pytest
from napalm_device_types import DhcpServerMixin
from napalm_device_types.dhcp import DhcpApplyError
from napalm_device_types.models import DhcpSubnetDict
@@ -235,3 +236,92 @@ def test_calling_a_missing_device_method_fails_loudly() -> None:
def test_a_subnet_that_cannot_name_one_raises(desired):
with pytest.raises(ValueError):
_FakeDhcpServer([]).diff_dhcp_subnets(desired)
# ── apply: a refused subnet does not hold back the others (#113) ──────────────
class _RefusingDhcpServer(_FakeDhcpServer):
"""A server that refuses the subnets in *refused*, as Kea refuses one that
overlaps a subnet it already serves."""
def __init__(self, live: list[DhcpSubnetDict] | None = None, *, refused: set[str]) -> None:
super().__init__(live)
self.refused = refused
def apply_dhcp_subnet(
self, subnet: DhcpSubnetDict, *, uuid: str | None = None
) -> dict[str, Any]:
if subnet["subnet"] in self.refused:
raise RuntimeError(f"{subnet['subnet']} overlaps a subnet the server has")
return super().apply_dhcp_subnet(subnet, uuid=uuid)
def _three_subnets() -> list[DhcpSubnetDict]:
return [
_subnet(subnet="192.0.2.0/26"),
_subnet(subnet="192.0.2.64/26"),
_subnet(subnet="192.0.2.128/26"),
]
def test_a_refused_subnet_does_not_stop_the_others() -> None:
fake = _RefusingDhcpServer(refused={"192.0.2.64/26"})
with pytest.raises(DhcpApplyError):
list(fake.apply_dhcp_subnetset(_three_subnets()))
assert [e["subnet"]["subnet"] for e in fake.applied] == ["192.0.2.0/26", "192.0.2.128/26"]
def test_what_was_written_is_committed() -> None:
fake = _RefusingDhcpServer(refused={"192.0.2.64/26"})
with pytest.raises(DhcpApplyError):
list(fake.apply_dhcp_subnetset(_three_subnets()))
assert fake.commits == 1
def test_a_set_whose_every_write_is_refused_still_commits() -> None:
# A run that stopped before its commit left saved writes behind, which the device
# reports as configured: no later diff would commit them.
fake = _RefusingDhcpServer(refused={"192.0.2.64/26"})
with pytest.raises(DhcpApplyError):
list(fake.apply_dhcp_subnetset([_three_subnets()[1]]))
assert fake.commits == 1
def test_progress_names_the_refused_subnet_and_why() -> None:
fake = _RefusingDhcpServer(
[_subnet(uuid="dev-1", subnet="192.0.2.128/26")], refused={"192.0.2.64/26"}
)
lines: list[str] = []
with pytest.raises(DhcpApplyError) as raised:
for line in fake.apply_dhcp_subnetset(
[*_three_subnets()[:2], _subnet(subnet="192.0.2.128/26", description="Office")]
):
lines.append(line)
assert lines == [
"[add] 192.0.2.0/26",
"[failed] 192.0.2.64/26: 192.0.2.64/26 overlaps a subnet the server has",
"[update] 192.0.2.128/26 (description)",
"[commit] applied 1 add(s), 1 update(s)",
]
assert (raised.value.failed, raised.value.attempted) == (1, 3)
assert str(raised.value) == "1 of 3 DHCP subnet change(s) were refused"
def test_a_refused_update_is_counted_like_an_add() -> None:
live = _subnet(uuid="dev-1", subnet="192.0.2.0/24")
fake = _RefusingDhcpServer([live], refused={"192.0.2.0/24"})
with pytest.raises(DhcpApplyError, match="1 of 1"):
list(fake.apply_dhcp_subnetset([_subnet(subnet="192.0.2.0/24", description="Office")]))
assert fake.applied == []
assert fake.commits == 1
+1 -2
View File
@@ -30,8 +30,7 @@ def test_a_missing_license_may_say_what_the_device_withholds_without_it():
# A Supermicro X10 with only SFT-OOB-LIC answers Redfish but keeps its drives and
# firmware inventory until DCMS is activated: partly licensed, not unlicensed.
hints = get_type_hints(LicenseDict)
# Python 3.10 keeps NotRequired[...] around the type; later versions strip it.
assert "list[str]" in str(hints["withholds"])
assert hints["withholds"] == list[str]
assert "withholds" not in LicenseDict.__required_keys__
+83 -1
View File
@@ -11,9 +11,11 @@ import pytest
from napalm_device_types import (
AccessPointDriver,
BackupServerDriver,
EnsurePackageMixin,
FirewallDriver,
HypervisorDriver,
LldpAgentMixin,
ManagementControllerDriver,
MediaDriver,
OSDriver,
@@ -27,7 +29,7 @@ from napalm_device_types import (
)
ROLE_BASES = [
AccessPointDriver, FirewallDriver, HypervisorDriver, ManagementControllerDriver, MediaDriver,
AccessPointDriver, BackupServerDriver, FirewallDriver, HypervisorDriver, ManagementControllerDriver, MediaDriver,
OSDriver, PhoneDriver, ResidentialGatewayDriver, StorageDriver, SwitchDriver,
] # fmt: skip
@@ -169,11 +171,91 @@ class TestEnsurePackage:
assert "no candidate" in result["output"]
class _Lldp(LldpAgentMixin):
"""The generic steps over ensure_package and manage_service; the driver's configure."""
def __init__(self, refuse: str = ""):
self.calls: list = []
self.refuse = refuse
def ensure_package(self, name):
self.calls.append(("ensure_package", name))
ok = self.refuse != "install"
return {"success": ok, "steps": [], "output": "installed" if ok else "E: no candidate"}
def _write_lldp_agent_config(self, interfaces):
self.calls.append(("configure", interfaces))
return {"ok": self.refuse != "configure", "output": "interfaces set"}
def manage_service(self, name, action):
self.calls.append((action, name))
ok = self.refuse != action
return {"success": ok, "output": "" if ok else f"cannot {action}"}
class TestLldpAgent:
def test_install_configure_enable_and_restart_on_the_caller_s_interfaces(self):
agent = _Lldp()
result = agent.configure_lldp_agent(["br-lan"])
assert result["success"] is True
assert agent.calls == [
("ensure_package", "lldpd"),
("configure", ["br-lan"]),
("enable", "lldpd"),
("restart", "lldpd"),
]
assert [s["step"] for s in result["steps"]] == ["install", "configure", "start"]
def test_a_failed_install_configures_nothing(self):
agent = _Lldp(refuse="install")
result = agent.configure_lldp_agent(["br-lan"])
assert result["success"] is False
assert agent.calls == [("ensure_package", "lldpd")]
assert "no candidate" in result["output"]
def test_an_agent_that_will_not_start_at_boot_is_not_restarted(self):
agent = _Lldp(refuse="enable")
result = agent.configure_lldp_agent(["br-lan"])
assert result["success"] is False
assert ("restart", "lldpd") not in agent.calls
assert "cannot enable" in result["output"]
def test_a_failed_restart_fails_the_fix(self):
result = _Lldp(refuse="restart").configure_lldp_agent(["br-lan"])
assert result["success"] is False
assert result["steps"][-1]["step"] == "start"
def test_the_package_and_service_names_are_the_driver_s(self):
agent = _Lldp()
agent.LLDP_AGENT_PACKAGE = "lldpd-full"
agent.LLDP_AGENT_SERVICE = "lldpd.service"
agent.configure_lldp_agent(["eth0"])
assert ("ensure_package", "lldpd-full") in agent.calls
assert ("restart", "lldpd.service") in agent.calls
def test_no_interface_is_refused_before_anything_changes(self):
agent = _Lldp()
with pytest.raises(ValueError):
agent.configure_lldp_agent([])
assert agent.calls == []
@pytest.mark.parametrize("base", ROLE_BASES, ids=lambda b: b.__name__)
@pytest.mark.parametrize(
"method",
[
"configure_snmp_agent",
"configure_lldp_agent",
"ensure_package",
"configure_package_proxy",
"grant_container_engine_access",
+27
View File
@@ -0,0 +1,27 @@
"""shutdown_host: the contract for shutting the device itself down cleanly.
netOrk's graph workflows switch a hypervisor off when it has nothing to do
(NetOrk/netork#1636): through its own API, never by pulling the power. Like
``reboot_host`` it is declared, not implemented, so ``hasattr`` tells netOrk
whether a driver can.
"""
from __future__ import annotations
import inspect
from napalm_device_types import DeviceTypeDriver, HostShutdownMixin
def test_every_device_type_driver_carries_the_declaration():
assert issubclass(DeviceTypeDriver, HostShutdownMixin)
def test_declared_not_implemented():
assert not hasattr(DeviceTypeDriver, "shutdown_host")
def test_declaration_documents_the_contract():
source = inspect.getsource(HostShutdownMixin)
assert "def shutdown_host(self) -> None" in source
assert "RuntimeError" in source
+434
View File
@@ -13,6 +13,8 @@ from __future__ import annotations
import os
import re
import shutil
import sqlite3
import subprocess
import pytest
@@ -25,6 +27,12 @@ from napalm_device_types.host_status import (
kernel_reboot_pending,
parse_host_status,
)
from napalm_device_types.package_managers import (
APK_INSTALLED,
DNF_HISTORY,
OPKG_STATUS,
PACKAGE_MANAGERS_COMMAND,
)
def _wire(
@@ -146,6 +154,24 @@ class TestKernelRebootPending:
"5.14.0-503.11.1.el9_5.x86_64",
),
("6.9.7-200.fc40.x86_64", ["6.9.7-200.fc40.x86_64"], None),
# Oracle UEK: 112.5.1 is newer than 112.5, as rpm counts -- a number after
# 5 beats the letters ``el`` (#109, build-arm-01).
(
"6.12.0-208.112.5.1.el9uek.aarch64",
[
"6.12.0-207.111.5.6.el9uek.aarch64",
"6.12.0-208.112.5.el9uek.aarch64",
"6.12.0-208.112.5.1.el9uek.aarch64",
],
None,
),
(
"6.12.0-208.112.5.el9uek.aarch64",
["6.12.0-208.112.5.el9uek.aarch64", "6.12.0-208.112.5.1.el9uek.aarch64"],
"6.12.0-208.112.5.1.el9uek.aarch64",
),
# A release candidate (``~``) comes before its release, in rpm and in dpkg.
("6.13~rc5-1-amd64", ["6.13~rc5-1-amd64", "6.13-1-amd64"], "6.13-1-amd64"),
],
)
def test_the_newer_kernel_of_the_running_flavour(self, running, installed, newer):
@@ -570,3 +596,411 @@ class TestFilesystems:
mountpoints = [fs["mountpoint"] for fs in parse_host_status(out.stdout)["filesystems"]]
assert "/" in mountpoints
def _other(*sections: tuple[str, tuple[str, ...]], running: tuple[str, ...] = ()) -> str:
"""A report from a host without dpkg: the given sections before ``[timers]``."""
lines = ["[pm-running]", *running]
for name, body in sections:
lines += [f"[{name}]", *body]
return _wire().replace("[timers]", "\n".join([*lines, "[timers]"]))
class TestInterruptedRpm:
"""dnf and yum hosts: a transaction rpm was cut off in leaves the old and the
new version of a package both installed (``dnf check --duplicates``,
``package-cleanup --dupes``), and dnf's history records a transaction that
never ended. netOrk only reports it: there is no one repair that is safe on
every host (NetOrk/netork#851, decided 2026-11-07).
The outputs are shaped after the documented formats -- ``rpm -qa --qf``,
libdnf's ``trans`` table (``state`` 0 = UNKNOWN until ``finish()`` sets DONE
1 or ERROR 2), yum's ``transaction-all.*`` journal that
``yum-complete-transaction`` reads. Checked against Oracle Linux 9 (build-arm-01,
2026-11-12): its UEK kernels are installonly only by libdnf's built-in defaults.
"""
INSTALLONLY = ("kernel-core", "kernel-modules", "kernel-uek-core")
#: An upgrade of glibc and openssl-libs cut off after the new versions were in.
CUT_OFF = ("glibc x86_64", "kernel-uek-core x86_64", "openssl-libs x86_64")
def _rpm(self, dupes=(), history=("42 1",), journal=(), frontend="dnf", running=()):
return _other(
("rpm-frontend", (frontend,)),
("rpm-installonly", self.INSTALLONLY),
("rpm-dupes", dupes),
("yum-journal", journal),
("dnf-history", history),
running=running,
)
def test_a_clean_rpm_database_is_not_interrupted(self):
status = parse_host_status(self._rpm())
assert status["package_manager_interrupted"] is False
assert status["interrupted_packages"] == []
assert status["package_manager"] == "dnf"
def test_two_versions_of_a_package_left_by_a_cut_off_transaction(self):
status = parse_host_status(self._rpm(dupes=self.CUT_OFF))
assert status["package_manager_interrupted"] is True
assert status["interrupted_packages"] == [
{"name": "glibc", "state": "duplicate"},
{"name": "openssl-libs", "state": "duplicate"},
]
def test_kernels_are_installed_side_by_side_on_purpose(self):
"""dnf's installonly packages: several versions are the rule, not a fault."""
status = parse_host_status(self._rpm(dupes=("kernel-uek-core x86_64",)))
assert status["package_manager_interrupted"] is False
def test_a_package_duplicated_in_two_arches_is_named_once(self):
status = parse_host_status(self._rpm(dupes=("glibc i686", "glibc x86_64")))
assert status["interrupted_packages"] == [{"name": "glibc", "state": "duplicate"}]
def test_the_last_transaction_never_finished(self):
status = parse_host_status(self._rpm(history=("43 0",)))
assert status["package_manager_interrupted"] is True
assert status["interrupted_packages"] == []
@pytest.mark.parametrize("history", [("44 1",), ("45 2",), ()])
def test_a_finished_or_unreadable_history_says_nothing(self, history):
"""A failed transaction (ERROR) did end; an unreadable history is no answer."""
assert parse_host_status(self._rpm(history=history))["package_manager_interrupted"] is False
def test_yum_left_its_transaction_journal(self):
journal = ("transaction-all.2026-11-01.10:12.33", "transaction-done.2026-11-01.10:12.33")
status = parse_host_status(self._rpm(frontend="yum", history=(), journal=journal))
assert status["package_manager_interrupted"] is True
assert status["package_manager"] == "yum"
def test_a_journal_yum_gave_up_on_is_none(self):
"""yum-complete-transaction skips a journal it renamed ``.disabled``."""
journal = ("transaction-all.2026-11-01.10:12.33.disabled",)
status = parse_host_status(self._rpm(frontend="yum", history=(), journal=journal))
assert status["package_manager_interrupted"] is False
@pytest.mark.parametrize("name", ["rpm", "dnf", "dnf5", "yum", "dnf-automatic", "packagekitd"])
def test_a_running_transaction_makes_it_unknown(self, name):
status = parse_host_status(self._rpm(dupes=self.CUT_OFF, running=(name,)))
assert status["package_manager_interrupted"] is None
assert status["interrupted_packages"] == []
assert status["package_manager"] == "dnf"
def test_another_managers_run_does_not(self):
status = parse_host_status(self._rpm(dupes=self.CUT_OFF, running=("opkg",)))
assert status["package_manager_interrupted"] is True
def test_the_command_asks_rpm_only_where_dnf_or_yum_exists(self):
assert "command -v rpm" in HOST_STATUS_COMMAND
assert "rpm -qa --qf" in HOST_STATUS_COMMAND
assert f"file:{DNF_HISTORY}?mode=ro" in HOST_STATUS_COMMAND
def test_it_never_repairs(self):
for word in ("distro-sync", "--cleandupes", "remove", "yum-complete-transaction", "rpm -e"):
assert word not in HOST_STATUS_COMMAND
class TestInterruptedApk:
"""Alpine and OpenWrt 25.12: apk marks a package whose files or scripts failed
in ``/lib/apk/db/installed`` (``f:`` with ``f`` broken files, ``s`` a broken
script; ``x`` xattrs and ``S`` the checksum kind are no fault), and ``apk fix``
with no arguments is what picks exactly those (apk-tools ``app_fix.c``).
netOrk only reports it (NetOrk/netork#851). Shaped after apk-tools'
``database.c``; to be checked against a real OpenWrt 25.12 host."""
def _apk(self, audit=(), running=()):
return _other(("apk-audit", audit), running=running)
def test_a_clean_database_is_not_interrupted(self):
status = parse_host_status(self._apk())
assert status["package_manager_interrupted"] is False
assert status["package_manager"] == "apk"
@pytest.mark.parametrize(
("flags", "states"),
[
("s", ["broken-script"]),
("f", ["broken-files"]),
("fsS", ["broken-files", "broken-script"]),
],
)
def test_a_package_apk_marked_broken(self, flags, states):
status = parse_host_status(self._apk(audit=(f"{flags} busybox",)))
assert status["package_manager_interrupted"] is True
assert status["interrupted_packages"] == [
{"name": "busybox", "state": state} for state in states
]
def test_a_running_apk_makes_it_unknown(self):
status = parse_host_status(self._apk(audit=("s busybox",), running=("apk",)))
assert status["package_manager_interrupted"] is None
def test_the_command_reads_the_installed_database(self):
assert "/lib/apk/db/installed" in HOST_STATUS_COMMAND
assert "apk fix" not in HOST_STATUS_COMMAND
class TestInterruptedOpkg:
"""OpenWrt before 25.12: opkg writes ``Status: <want> <flags> <state>`` per
package to ``/usr/lib/opkg/status``; a package whose run was cut off or whose
postinst failed stays ``unpacked``, and ``opkg configure`` finishes every
unpacked package (opkg-lede ``opkg_cmd.c``, ``pkg.c``). netOrk's update run
does that first (NetOrk/netork#851). To be checked against a real router."""
def _opkg(self, audit=(), running=()):
return _other(("opkg-audit", audit), running=running)
def test_a_clean_status_file_is_not_interrupted(self):
status = parse_host_status(self._opkg())
assert status["package_manager_interrupted"] is False
assert status["package_manager"] == "opkg"
@pytest.mark.parametrize(
("line", "state"),
[
("user unpacked luci-app-sqm", "unpacked"),
("ok half-configured luci-app-sqm", "half-configured"),
("ok half-installed luci-app-sqm", "half-installed"),
("reinstreq,user installed luci-app-sqm", "reinstall-required"),
],
)
def test_every_state_an_interrupted_run_leaves(self, line, state):
status = parse_host_status(self._opkg(audit=(line,)))
assert status["package_manager_interrupted"] is True
assert status["interrupted_packages"] == [{"name": "luci-app-sqm", "state": state}]
def test_a_running_opkg_makes_it_unknown(self):
status = parse_host_status(self._opkg(audit=("ok unpacked x",), running=("opkg",)))
assert status["package_manager_interrupted"] is None
def test_the_command_reads_the_status_file_and_never_configures(self):
assert "/usr/lib/opkg/status" in HOST_STATUS_COMMAND
assert "opkg configure" not in HOST_STATUS_COMMAND
class TestPackageManager:
def test_dpkg_says_so(self):
assert parse_host_status(_wire(dpkg=()))["package_manager"] == "dpkg"
def test_a_busy_dpkg_still_says_which(self):
status = parse_host_status(_wire(dpkg=(), busy=True))
assert (status["package_manager"], status["package_manager_interrupted"]) == (
"dpkg",
None,
)
def test_no_package_manager_read_is_none(self):
status = parse_host_status(_wire())
assert status["package_manager"] is None
assert status["package_manager_interrupted"] is None
#: An rpm that answers the two queries the command asks, as on an EL host whose
#: glibc upgrade was cut off; and a dnf that only has to exist.
_RPM_STUB = """#!/bin/sh
case "$*" in
*--whatprovides*) printf '%s\\n' kernel-core kernel-core 'no package provides kernel-PAE' ;;
-qa*) printf '%s\\n' 'glibc x86_64' 'kernel-core x86_64' 'gpg-pubkey (none)' \\
'glibc x86_64' 'kernel-core x86_64' 'gpg-pubkey (none)' 'bash x86_64' ;;
esac
"""
#: An rpm on an Oracle Linux host: its UEK kernel ``vendor-kernel`` provides nothing
#: of dnf's upstream defaults and sits in the database three times, like glibc twice.
_RPM_VENDOR_KERNEL_STUB = """#!/bin/sh
case "$*" in
*--whatprovides*) for a; do [ "$a" = vendor-kernel ] && echo vendor-kernel; done
echo 'no package provides kernel-PAE' ;;
-qa*) printf '%s\\n' 'vendor-kernel aarch64' 'vendor-kernel aarch64' 'vendor-kernel aarch64' \\
'glibc aarch64' 'glibc aarch64' ;;
esac
"""
#: libdnf's Python binding, as far as the command asks it: the installonlypkgs a
#: distribution built into its libdnf -- here one more than upstream's.
_LIBDNF_CONF = """class _Option:
def getValueString(self):
return "kernel, installonlypkg(kernel), vendor-kernel"
class ConfigMain:
def installonlypkgs(self):
return _Option()
"""
#: /lib/apk/db/installed shaped after apk-tools' database.c: busybox's script failed.
_APK_DB = """C:Q1abc=
P:alpine-baselayout
V:3.6.8-r1
A:x86_64
F:etc
R:fstab
C:Q1def=
P:busybox
V:1.37.0-r12
A:x86_64
f:s
F:bin
R:busybox
C:Q1ghi=
P:musl
V:1.2.5-r9
f:S
"""
#: /usr/lib/opkg/status as opkg writes it: luci-app-sqm's postinst never ran.
_OPKG_STATUS = """Package: libc
Version: 1.2.4-4
Depends: libgcc1
Status: install hold installed
Architecture: mipsel_24kc
Installed-Time: 1717000000
Package: luci-app-sqm
Version: 1.5.3-r1
Depends: libc, sqm-scripts
Status: install user unpacked
Architecture: all
Installed-Time: 1730000000
"""
def _stubs(tmp_path, *names: str, rpm: bool = False) -> dict:
bin_dir = tmp_path / "bin"
bin_dir.mkdir()
for name in names:
stub = bin_dir / name
stub.write_text(_RPM_STUB if name == "rpm" else "#!/bin/sh\nexit 0\n")
stub.chmod(0o755)
return {**os.environ, "PATH": f"{bin_dir}:/usr/bin:/bin"}
def _run(command: str, env: dict, shell: tuple[str, ...]) -> str:
report = f"printf '%s%s\\n' HSTAT_ BEGIN; {command} printf '%s%s\\n' HSTAT_ END"
program = shutil.which(shell[0]) or shell[0]
return subprocess.run(
[program, *shell[1:], report], capture_output=True, text=True, env=env, timeout=30
).stdout
_SHELLS = [("sh", "-c")] + ([("busybox", "sh", "-c")] if shutil.which("busybox") else [])
def _skip_busybox_rpm(shell: tuple[str, ...]) -> None:
"""A BusyBox built with its own ``rpm`` applet runs that before any stand-in."""
if shell[0] == "busybox":
applets = subprocess.run(["busybox", "--list"], capture_output=True, text=True).stdout
if "rpm" in applets.split():
pytest.skip("this busybox has an rpm applet of its own")
@pytest.mark.parametrize("shell", _SHELLS)
class TestThePackageManagerCommandOnARealShell:
"""The command part run for real, with stand-in programs and files."""
def test_rpm_duplicates_without_the_kernel_and_the_keys(self, tmp_path, shell):
_skip_busybox_rpm(shell)
env = _stubs(tmp_path, "rpm", "dnf")
out = _run(PACKAGE_MANAGERS_COMMAND, env, shell)
status = parse_host_status(out)
assert status["package_manager"] == "dnf"
assert status["interrupted_packages"] == [{"name": "glibc", "state": "duplicate"}]
def test_the_installonly_packages_libdnf_was_built_with(self, tmp_path, shell):
"""Oracle builds kernel-uek into libdnf's defaults; its UEK packages provide
nothing of upstream's, so only libdnf itself tells (#107)."""
_skip_busybox_rpm(shell)
env = _stubs(tmp_path, "rpm", "dnf")
(tmp_path / "bin" / "rpm").write_text(_RPM_VENDOR_KERNEL_STUB)
libdnf = tmp_path / "py" / "libdnf"
libdnf.mkdir(parents=True)
(libdnf / "__init__.py").write_text("")
(libdnf / "conf.py").write_text(_LIBDNF_CONF)
env["PYTHONPATH"] = str(tmp_path / "py")
out = _run(PACKAGE_MANAGERS_COMMAND, env, shell)
status = parse_host_status(out)
assert status["interrupted_packages"] == [{"name": "glibc", "state": "duplicate"}]
def test_dnfs_last_transaction_that_never_ended(self, tmp_path, shell):
_skip_busybox_rpm(shell)
history = tmp_path / "history.sqlite"
with sqlite3.connect(history) as db:
db.execute("CREATE TABLE trans (id INTEGER PRIMARY KEY, state INTEGER NOT NULL)")
db.executemany("INSERT INTO trans VALUES (?, ?)", [(41, 1), (42, 0)])
env = _stubs(tmp_path, "rpm", "dnf")
out = _run(PACKAGE_MANAGERS_COMMAND.replace(DNF_HISTORY, str(history)), env, shell)
assert "[dnf-history]\n42 0\n" in out
def test_apk_names_the_package_it_marked(self, tmp_path, shell):
installed = tmp_path / "installed"
installed.write_text(_APK_DB)
env = _stubs(tmp_path, "apk")
out = _run(PACKAGE_MANAGERS_COMMAND.replace(APK_INSTALLED, str(installed)), env, shell)
status = parse_host_status(out)
assert status["package_manager"] == "apk"
assert status["interrupted_packages"] == [{"name": "busybox", "state": "broken-script"}]
def test_opkg_names_the_unpacked_package(self, tmp_path, shell):
status_file = tmp_path / "status"
status_file.write_text(_OPKG_STATUS)
env = _stubs(tmp_path, "opkg")
out = _run(PACKAGE_MANAGERS_COMMAND.replace(OPKG_STATUS, str(status_file)), env, shell)
status = parse_host_status(out)
assert status["package_manager"] == "opkg"
assert status["interrupted_packages"] == [{"name": "luci-app-sqm", "state": "unpacked"}]
def test_a_host_with_none_of_them_reads_none(self, tmp_path, shell):
env = _stubs(tmp_path)
env["PATH"] = str(tmp_path / "bin")
out = _run(PACKAGE_MANAGERS_COMMAND, env, shell)
assert parse_host_status(out)["package_manager"] is None
def test_the_running_processes_come_from_proc(self, tmp_path, shell):
"""The test runner's own process is there under its own name; no manager is."""
out = _run(PACKAGE_MANAGERS_COMMAND, _stubs(tmp_path), shell)
assert "[pm-running]\n" in out
class TestTheWholeCommand:
def test_it_runs_on_busybox(self):
"""OpenWrt's and Alpine's shell: still one intact report."""
if not shutil.which("busybox"):
pytest.skip("no busybox here")
out = subprocess.run(
["busybox", "sh", "-c", HOST_STATUS_COMMAND], capture_output=True, text=True, timeout=30
).stdout
assert parse_host_status(out)["filesystems"]
def test_the_package_managers_are_part_of_it(self):
assert PACKAGE_MANAGERS_COMMAND in HOST_STATUS_COMMAND
+127
View File
@@ -0,0 +1,127 @@
"""LLDP rows projected into NAPALM's ``get_lldp_neighbors`` and ``get_lldp_neighbors_detail``.
A driver reads its neighbours into :class:`LldpNeighborRow` (from ``lldpctl``,
an API, a CLI) and the two NAPALM shapes come from those rows the same way for
every driver (NAPALM/napalm-device-types#10).
"""
from __future__ import annotations
from typing import get_type_hints
from napalm_device_types import LldpNeighborRow, lldp_neighbors, lldp_neighbors_detail
def _row(local_port: str, name: str, **fields: object) -> LldpNeighborRow:
row: LldpNeighborRow = {
"local_port": local_port,
"chassis_id": "",
"chassis_id_type": "",
"port_id": "",
"port_id_type": "",
"port_description": "",
"system_name": name,
"system_description": "",
"management_addresses": [],
"capabilities": [],
"enabled_capabilities": [],
}
row.update(fields) # type: ignore[typeddict-item]
return row
PHONE = _row(
"eth0.1",
"desk-phone",
chassis_id="00:11:22:33:44:AB",
chassis_id_type="mac",
port_id="00:11:22:33:44:01",
port_id_type="mac",
port_description="WAN",
system_description="SIP phone",
capabilities=["bridge", "telephone"],
enabled_capabilities=["telephone"],
)
PC = _row("eth0.1", "desk-pc", port_id="eno1", port_id_type="ifname")
SWITCH = _row("eth1", "access-sw", port_id="gi1/0/7", port_id_type="local")
def test_a_row_names_its_fields():
assert set(get_type_hints(LldpNeighborRow)) == {
"local_port",
"chassis_id",
"chassis_id_type",
"port_id",
"port_id_type",
"port_description",
"system_name",
"system_description",
"management_addresses",
"capabilities",
"enabled_capabilities",
}
class TestNeighbors:
def test_every_neighbour_of_a_port_is_listed(self):
assert lldp_neighbors([PHONE, PC, SWITCH]) == {
"eth0.1": [
{"hostname": "desk-phone", "port": "00:11:22:33:44:01", "mac": "00:11:22:33:44:ab"},
{"hostname": "desk-pc", "port": "eno1"},
],
"eth1": [{"hostname": "access-sw", "port": "gi1/0/7"}],
}
def test_no_rows_is_no_neighbour(self):
assert lldp_neighbors([]) == {}
def test_a_chassis_mac_is_reported_in_lower_case(self):
# netOrk matches a neighbour by this MAC first (NAPALM/napalm-openwrt#31).
assert lldp_neighbors([PHONE])["eth0.1"][0]["mac"] == "00:11:22:33:44:ab"
def test_a_chassis_named_otherwise_has_no_mac(self):
switch = _row("eth1", "sw", chassis_id="00:11:22:33:44:55", chassis_id_type="local")
assert "mac" not in lldp_neighbors([switch])["eth1"][0]
def test_a_mac_id_that_is_no_mac_address_is_left_out(self):
odd = _row("eth1", "sw", chassis_id="00:11", chassis_id_type="mac")
assert "mac" not in lldp_neighbors([odd])["eth1"][0]
class TestNeighborsDetail:
def test_every_field_napalm_names(self):
assert lldp_neighbors_detail([PHONE]) == {
"eth0.1": [
{
"parent_interface": "",
"remote_port": "00:11:22:33:44:01",
"remote_port_description": "WAN",
"remote_chassis_id": "00:11:22:33:44:AB",
"remote_system_name": "desk-phone",
"remote_system_description": "SIP phone",
"remote_system_capab": ["bridge", "telephone"],
"remote_system_enable_capab": ["telephone"],
}
]
}
def test_every_neighbour_of_a_port_is_listed(self):
detail = lldp_neighbors_detail([PHONE, PC, SWITCH])
assert [n["remote_system_name"] for n in detail["eth0.1"]] == ["desk-phone", "desk-pc"]
assert [n["remote_system_name"] for n in detail["eth1"]] == ["access-sw"]
def test_one_interface_only(self):
assert list(lldp_neighbors_detail([PHONE, PC, SWITCH], "eth1")) == ["eth1"]
def test_an_interface_without_neighbours_is_empty(self):
assert lldp_neighbors_detail([PHONE], "eth9") == {}
def test_the_lists_are_the_rows_own_copies(self):
detail = lldp_neighbors_detail([PHONE])
detail["eth0.1"][0]["remote_system_capab"].append("router")
assert PHONE["capabilities"] == ["bridge", "telephone"]
+308
View File
@@ -0,0 +1,308 @@
"""``lldpctl -f keyvalue``: one row per LLDP neighbour, for every driver that reads lldpd.
napalm-linux, napalm-openwrt and napalm-proxmox each parsed this output on their
own, and each copy grouped the lines by interface, so two neighbours on one port
merged (NAPALM/napalm-openwrt#9, NAPALM/napalm-linux#58), split ``eth0.1`` at its
dot (NAPALM/napalm-openwrt#26) and looked for ``chassis.id.value``, a key lldpd
never prints. The parse lives here once now.
"""
from __future__ import annotations
import pytest
from napalm_device_types import LLDPCTL_KEYVALUE_COMMAND, parse_lldpctl_keyvalue
# The recording napalm-openwrt's tests have used since the driver was written
# (tests/unit/test_driver.py, ``LLDPCTL_KV``): one neighbour on eth0. Its two
# ``chassis.cap.*`` lines are not lldpd's format (lldpd prints one
# ``chassis.<Capability>.enabled=on|off`` line per capability), so they yield none.
OPENWRT_ONE_NEIGHBOUR = """\
lldp.eth0.via=LLDP
lldp.eth0.rid=1
lldp.eth0.age=0 day, 01:23:45
lldp.eth0.chassis.mac=00:aa:bb:cc:dd:ee
lldp.eth0.chassis.name=core-router
lldp.eth0.chassis.descr=RouterOS 7.x
lldp.eth0.chassis.mgmt-ip=10.0.0.1
lldp.eth0.chassis.cap.available=Router, Bridge
lldp.eth0.chassis.cap.enabled=Router
lldp.eth0.port.ifname=ether1
lldp.eth0.port.descr=uplink
"""
# Laid out the way lldpd 1.0's keyvalue writer prints ``show neighbors details``
# (src/client/display.c ``display_interface``, ``display_chassis``,
# ``display_port``, ``display_vlans``, ``display_med``; src/client/kv_writer.c):
# a neighbour opens with the local interface's optional ``alias`` and ``via``,
# an id is printed under its type (``chassis.mac``, ``port.local``), a
# capability as ``chassis.<Symbol>.enabled``, a management address once per
# address. Not recorded from a device.
#
# On ``eth0.1`` (a VLAN interface, its name holds a dot) a phone and a PC behind
# an unmanaged switch; on ``eth1`` a switch that names its port by a local id and
# itself by an address.
TWO_ON_ONE_DOTTED_PORT = """\
lldp.eth0.1.alias=desk
lldp.eth0.1.via=LLDP
lldp.eth0.1.rid=1
lldp.eth0.1.age=0 day, 00:10:00
lldp.eth0.1.chassis.mac=00:11:22:33:44:01
lldp.eth0.1.chassis.name=desk-phone
lldp.eth0.1.chassis.descr=SIP phone, firmware a=1
lldp.eth0.1.chassis.mgmt-ip=10.0.0.21
lldp.eth0.1.chassis.mgmt-iface=2
lldp.eth0.1.chassis.mgmt-ip=fe80::211:22ff:fe33:4401
lldp.eth0.1.chassis.mgmt-iface=2
lldp.eth0.1.chassis.Bridge.enabled=off
lldp.eth0.1.chassis.Tel.enabled=on
lldp.eth0.1.port.mac=00:11:22:33:44:01
lldp.eth0.1.port.descr=WAN
lldp.eth0.1.port.ttl=120
lldp.eth0.1.port.auto-negotiation.supported=yes
lldp.eth0.1.port.auto-negotiation.enabled=yes
lldp.eth0.1.port.auto-negotiation.100Base-TX.hd=yes
lldp.eth0.1.port.auto-negotiation.100Base-TX.fd=yes
lldp.eth0.1.port.auto-negotiation.current=100BaseTXFD - 2 pair category 5 UTP, full duplex mode
lldp.eth0.1.vlan.vlan-id=10
lldp.eth0.1.vlan.pvid=yes
lldp.eth0.1.vlan=voice
lldp.eth0.1.lldp-med.device-type=Communication Device Endpoint (Class III)
lldp.eth0.1.lldp-med.Capabilities.available=yes
lldp.eth0.1.lldp-med.inventory.software=2.1.4
lldp.eth0.1.alias=desk
lldp.eth0.1.via=LLDP
lldp.eth0.1.rid=2
lldp.eth0.1.age=0 day, 00:09:00
lldp.eth0.1.chassis.mac=00:11:22:33:44:02
lldp.eth0.1.chassis.name=desk-pc
lldp.eth0.1.chassis.descr=Debian GNU/Linux 12 (bookworm) Linux 6.1.0-25-amd64
lldp.eth0.1.chassis.Bridge.enabled=off
lldp.eth0.1.chassis.Router.enabled=off
lldp.eth0.1.chassis.Wlan.enabled=off
lldp.eth0.1.chassis.Station.enabled=on
lldp.eth0.1.port.ifname=eno1
lldp.eth0.1.port.ttl=120
lldp.eth0.1.unknown-tlvs.unknown-tlv.oui=00,80,C2
lldp.eth0.1.unknown-tlvs.unknown-tlv.subtype=9
lldp.eth0.1.unknown-tlvs.unknown-tlv.len=1
lldp.eth0.1.unknown-tlvs.unknown-tlv=01
lldp.eth1.via=LLDP
lldp.eth1.rid=3
lldp.eth1.age=1 day, 02:00:00
lldp.eth1.chassis.ip=192.0.2.7
lldp.eth1.chassis.name=access-sw
lldp.eth1.chassis.Bridge.enabled=on
lldp.eth1.chassis.Router.enabled=off
lldp.eth1.chassis.CVLAN.enabled=on
lldp.eth1.port.local=gi1/0/7
lldp.eth1.port.descr=GigabitEthernet1/0/7
lldp.eth1.ppvid.supported=no
lldp.eth1.ppvid.enabled=no
lldp.eth1.pi=88cc
"""
def _rows(output: str = TWO_ON_ONE_DOTTED_PORT) -> list[dict[str, object]]:
return [dict(row) for row in parse_lldpctl_keyvalue(output)]
class TestOneRowPerNeighbour:
def test_the_openwrt_recording_is_one_neighbour_with_its_fields(self):
assert _rows(OPENWRT_ONE_NEIGHBOUR) == [
{
"local_port": "eth0",
"chassis_id": "00:aa:bb:cc:dd:ee",
"chassis_id_type": "mac",
"port_id": "ether1",
"port_id_type": "ifname",
"port_description": "uplink",
"system_name": "core-router",
"system_description": "RouterOS 7.x",
"management_addresses": ["10.0.0.1"],
"capabilities": [],
"enabled_capabilities": [],
}
]
def test_two_neighbours_on_one_port_stay_apart(self):
rows = _rows()
assert [(r["local_port"], r["system_name"]) for r in rows] == [
("eth0.1", "desk-phone"),
("eth0.1", "desk-pc"),
("eth1", "access-sw"),
]
def test_a_neighbour_keeps_only_its_own_fields(self):
pc = _rows()[1]
assert pc["chassis_id"] == "00:11:22:33:44:02"
assert pc["port_description"] == ""
assert pc["management_addresses"] == []
def test_neighbours_without_alias_are_split_at_via(self):
output = OPENWRT_ONE_NEIGHBOUR + OPENWRT_ONE_NEIGHBOUR.replace("core-router", "edge")
assert [r["system_name"] for r in _rows(output)] == ["core-router", "edge"]
class TestInterfaceName:
def test_a_dotted_interface_name_survives(self):
assert {r["local_port"] for r in _rows()} == {"eth0.1", "eth1"}
def test_an_interface_named_like_a_field_survives(self):
output = (
"lldp.vlan.100.via=LLDP\n"
"lldp.vlan.100.chassis.mac=00:aa:bb:cc:dd:01\n"
"lldp.vlan.100.port.ifname=port1\n"
)
(row,) = _rows(output)
assert (row["local_port"], row["chassis_id"], row["port_id"]) == (
"vlan.100",
"00:aa:bb:cc:dd:01",
"port1",
)
def test_without_via_the_name_ends_at_the_first_field(self):
output = "lldp.br-lan.10.chassis.name=sw\nlldp.br-lan.10.port.ifname=1\n"
(row,) = _rows(output)
assert (row["local_port"], row["system_name"], row["port_id"]) == ("br-lan.10", "sw", "1")
class TestIds:
def test_a_port_given_by_its_mac_is_read(self):
phone = _rows()[0]
assert (phone["port_id"], phone["port_id_type"]) == ("00:11:22:33:44:01", "mac")
def test_a_port_given_by_a_local_id_is_read(self):
switch = _rows()[2]
assert (switch["port_id"], switch["port_id_type"]) == ("gi1/0/7", "local")
def test_a_chassis_given_by_its_address_is_read(self):
switch = _rows()[2]
assert (switch["chassis_id"], switch["chassis_id_type"]) == ("192.0.2.7", "ip")
@pytest.mark.parametrize("kind", ["local", "ifname", "ifalias", "unhandled"])
def test_every_id_type_lldpd_names_is_read(self, kind):
output = f"lldp.eth0.via=LLDP\nlldp.eth0.chassis.{kind}=c1\nlldp.eth0.port.{kind}=p1\n"
(row,) = _rows(output)
assert (row["chassis_id"], row["chassis_id_type"]) == ("c1", kind)
assert (row["port_id"], row["port_id_type"]) == ("p1", kind)
def test_a_neighbour_without_ids_has_empty_ones(self):
(row,) = _rows("lldp.eth0.via=CDPv2\nlldp.eth0.chassis.name=sw\n")
assert (row["chassis_id"], row["chassis_id_type"]) == ("", "")
assert (row["port_id"], row["port_id_type"]) == ("", "")
class TestOtherFields:
def test_every_management_address_is_kept(self):
phone = _rows()[0]
assert phone["management_addresses"] == ["10.0.0.21", "fe80::211:22ff:fe33:4401"]
def test_a_value_may_hold_an_equals_sign(self):
assert _rows()[0]["system_description"] == "SIP phone, firmware a=1"
def test_capabilities_in_napalm_names_and_which_are_enabled(self):
phone, pc, switch = _rows()
assert phone["capabilities"] == ["bridge", "telephone"]
assert phone["enabled_capabilities"] == ["telephone"]
assert pc["capabilities"] == ["bridge", "router", "wlan-access-point", "station"]
assert pc["enabled_capabilities"] == ["station"]
assert switch["capabilities"] == ["bridge", "router", "cvlan"]
assert switch["enabled_capabilities"] == ["bridge", "cvlan"]
def test_every_napalm_capability_name(self):
symbols = ["Other", "Repeater", "Bridge", "Router", "Wlan", "Tel", "Docsis", "Station"]
output = "lldp.eth0.via=LLDP\n" + "".join(
f"lldp.eth0.chassis.{s}.enabled=on\n" for s in symbols
)
(row,) = _rows(output)
assert row["capabilities"] == [
"other",
"repeater",
"bridge",
"router",
"wlan-access-point",
"telephone",
"docsis-cable-device",
"station",
]
assert row["enabled_capabilities"] == row["capabilities"]
def test_a_capability_line_needs_on_or_off(self):
(row,) = _rows(OPENWRT_ONE_NEIGHBOUR)
assert row["capabilities"] == []
class TestReadOrFailure:
"""A failed read raises; ``[]`` only when lldpctl ran and listed nobody."""
def test_no_output_is_no_neighbour(self):
assert parse_lldpctl_keyvalue("") == []
def test_a_zero_exit_status_without_neighbours_is_no_neighbour(self):
assert parse_lldpctl_keyvalue("__LLDPCTL_RC=0\n") == []
def test_a_missing_lldpctl_raises(self):
with pytest.raises(ValueError, match="lldpctl: not found"):
parse_lldpctl_keyvalue("sh: lldpctl: not found\n")
def test_a_non_zero_exit_status_raises_with_what_lldpctl_said(self):
output = (
"[WARN/control] unable to connect to socket /var/run/lldpd.socket: "
"No such file or directory\n"
"__LLDPCTL_RC=1\n"
)
with pytest.raises(ValueError, match="unable to connect to socket"):
parse_lldpctl_keyvalue(output)
def test_a_non_zero_exit_status_without_a_message_names_the_status(self):
with pytest.raises(ValueError, match="exit status 127"):
parse_lldpctl_keyvalue("__LLDPCTL_RC=127\n")
def test_with_status_zero_a_warning_beside_the_neighbours_is_ignored(self):
output = (
"lldpctl -f keyvalue 2>&1; echo __LLDPCTL_RC=$?\n"
"[WARN/lldpctl] something lldpd wants to say\n"
+ OPENWRT_ONE_NEIGHBOUR
+ "__LLDPCTL_RC=0\n"
)
assert [r["system_name"] for r in _rows(output)] == ["core-router"]
def test_a_line_outside_lldp_is_no_neighbour(self):
output = "lldpd.config.tx-delay=30\n" + OPENWRT_ONE_NEIGHBOUR
assert len(_rows(output)) == 1
def test_a_key_without_a_known_field_is_skipped(self):
output = "lldp.summary=1\nlldp.eth9.future-field=x\n" + OPENWRT_ONE_NEIGHBOUR
assert [r["local_port"] for r in _rows(output)] == ["eth0"]
class TestCommand:
def test_the_command_reads_the_neighbours_in_keyvalue_form(self):
assert LLDPCTL_KEYVALUE_COMMAND.startswith("lldpctl -f keyvalue ")
def test_the_command_brings_back_the_error_text_and_the_exit_status(self):
assert "2>&1" in LLDPCTL_KEYVALUE_COMMAND
assert LLDPCTL_KEYVALUE_COMMAND.endswith("echo __LLDPCTL_RC=$?")
File diff suppressed because it is too large. Load diff
+189
View File
@@ -0,0 +1,189 @@
"""Where a host's logs go beyond it: rsyslog's forwarding, its file inputs, and
whether it gets the journal (NetOrk/netork#2053)."""
from __future__ import annotations
import pytest
from napalm_device_types.log_forwarding import journal_to_syslog, parse_rsyslog
DEBIAN = """\
module(load="imuxsock")
module(load="imklog")
$FileOwner root
$IncludeConfig /etc/rsyslog.d/*.conf
*.*;auth,authpriv.none\t\t-/var/log/syslog
auth,authpriv.*\t\t\t/var/log/auth.log
*.emerg\t\t\t\t:omusrmsg:*
"""
GRAYLOG = "*.* @graylog.example.net:1514;RSYSLOG_SyslogProtocol23Format\n"
def _rules(text: str) -> list[dict]:
return parse_rsyslog(text)["rules"]
class TestRules:
def test_a_debian_host_forwards_nothing(self):
assert _rules(DEBIAN) == []
def test_netorks_own_graylog_rule_forwards_everything_over_udp(self):
assert _rules(DEBIAN + GRAYLOG) == [
{
"protocol": "udp",
"target": "graylog.example.net",
"port": 1514,
"selector": "*.*",
"complete": True,
"ruleset": None,
"template": "RSYSLOG_SyslogProtocol23Format",
},
]
def test_a_rule_without_a_template_sends_rsyslogs_default(self):
"""None: RFC 3164, the sender's local time without a zone (NetOrk/netork#493)."""
(rule,) = _rules("*.* @graylog.example.net:514\n")
assert rule["template"] is None
def test_an_actions_template_parameter(self):
(rule,) = _rules(
'action(type="omfwd" target="10.0.0.7" Template="RSYSLOG_ForwardFormat")\n'
)
assert rule["template"] == "RSYSLOG_ForwardFormat"
def test_the_legacy_default_template_holds_for_the_rules_after_it(self):
rules = _rules(
"*.* @a\n$ActionForwardDefaultTemplate RSYSLOG_SyslogProtocol23Format\n"
"*.* @@b:514\n*.* @c;RSYSLOG_TraditionalForwardFormat\n"
)
assert [(r["target"], r["template"]) for r in rules] == [
("a", None),
("b", "RSYSLOG_SyslogProtocol23Format"),
("c", "RSYSLOG_TraditionalForwardFormat"),
]
def test_the_forward_modules_template_holds_for_its_actions(self):
text = (
'module(load="builtin:omfwd" template="RSYSLOG_SyslogProtocol23Format")\n'
'action(type="omfwd" target="10.0.0.7")\n'
)
(rule,) = _rules(text)
assert rule["template"] == "RSYSLOG_SyslogProtocol23Format"
def test_two_ats_are_tcp_and_a_missing_port_stays_unknown(self):
(rule,) = _rules("*.* @@logs.lan\n")
assert (rule["protocol"], rule["target"], rule["port"]) == ("tcp", "logs.lan", None)
def test_an_ipv6_target_keeps_its_brackets_off(self):
(rule,) = _rules("*.* @[2001:db8::5]:514\n")
assert (rule["target"], rule["port"]) == ("2001:db8::5", 514)
def test_a_narrower_selector_is_partial_and_says_which(self):
(rule,) = _rules("authpriv.* @10.0.0.5\n")
assert (rule["selector"], rule["complete"]) == ("authpriv.*", False)
def test_excluding_a_facility_is_partial_too(self):
(rule,) = _rules("*.*;mail.none @10.0.0.5\n")
assert rule["complete"] is False
def test_an_ampersand_line_forwards_what_the_line_before_selected(self):
rules = _rules("kern.* /var/log/kern.log\n& @10.0.0.5:514\n")
assert [(r["selector"], r["target"]) for r in rules] == [("kern.*", "10.0.0.5")]
def test_relp_is_its_own_protocol(self):
(rule,) = _rules("*.* :omrelp:10.0.0.6:2514\n")
assert (rule["protocol"], rule["target"], rule["port"]) == ("relp", "10.0.0.6", 2514)
def test_an_action_over_several_lines_is_read_whole(self):
text = '*.* action(type="omfwd"\n target="10.0.0.7" port="6514"\n protocol="tcp")\n'
(rule,) = _rules(text)
assert (rule["protocol"], rule["target"], rule["port"], rule["complete"]) == (
"tcp",
"10.0.0.7",
6514,
True,
)
def test_an_action_on_its_own_takes_every_message(self):
(rule,) = _rules('action(type="omfwd" Target="10.0.0.7")\n')
assert (rule["selector"], rule["complete"], rule["protocol"]) == ("*.*", True, "udp")
def test_an_action_under_a_condition_is_partial_and_names_it(self):
(rule,) = _rules(
'if $programname == \'nginx\' then action(type="omfwd" target="10.0.0.8")\n'
)
assert rule["complete"] is False
assert rule["selector"] == "$programname == 'nginx'"
def test_a_condition_whose_action_is_on_the_next_line_still_holds(self):
(rule,) = _rules('if $msg contains "x" then\n action(type="omfwd" target="10.0.0.14")\n')
assert (rule["selector"], rule["complete"]) == ('$msg contains "x"', False)
def test_a_block_under_a_condition_makes_each_of_its_actions_partial(self):
text = 'if $syslogseverity <= 4 then {\n action(type="omfwd" target="a")\n action(type="omrelp" target="b")\n}\n*.* @c\n'
rules = _rules(text)
assert [(r["target"], r["complete"]) for r in rules] == [
("a", False),
("b", False),
("c", True),
]
def test_a_property_filter_is_a_partial_selector(self):
(rule,) = _rules(':programname, isequal, "sshd" @10.0.0.9\n')
assert rule["complete"] is False
assert rule["selector"] == ':programname, isequal, "sshd"'
def test_an_action_in_a_ruleset_belongs_to_that_ruleset(self):
text = 'ruleset(name="remote") {\n action(type="omfwd" target="10.0.0.10" port="514")\n}\n'
(rule,) = _rules(text)
assert (rule["ruleset"], rule["complete"]) == ("remote", True)
def test_comments_are_no_rules(self):
assert _rules("# *.* @10.0.0.11\n*.* /var/log/all.log # @10.0.0.12\n") == []
def test_a_hash_inside_quotes_is_no_comment(self):
(rule,) = _rules('action(type="omfwd" target="10.0.0.13" template="a#b")\n')
assert rule["target"] == "10.0.0.13"
def test_a_local_action_is_no_forwarding(self):
assert _rules('action(type="omfile" file="/var/log/x.log")\n*.* /var/log/y.log\n') == []
class TestInputs:
def test_an_imfile_input_names_its_file_and_ruleset(self):
text = 'input(type="imfile" File="/var/log/nginx/*.log" Tag="nginx" Ruleset="remote")\n'
assert parse_rsyslog(text)["file_inputs"] == [
{"path": "/var/log/nginx/*.log", "ruleset": "remote"}
]
def test_a_legacy_imfile_input_goes_to_the_default_ruleset(self):
text = "$ModLoad imfile\n$InputFileName /var/log/app.log\n$InputFileTag app:\n$InputRunFileMonitor\n"
assert parse_rsyslog(text)["file_inputs"] == [{"path": "/var/log/app.log", "ruleset": None}]
class TestModules:
def test_imjournal_reads_the_journal(self):
assert parse_rsyslog('module(load="imjournal")\n')["imjournal"] is True
def test_legacy_modload_counts(self):
modules = parse_rsyslog("$ModLoad imuxsock\n")
assert (modules["imuxsock"], modules["imjournal"]) == (True, False)
class TestJournalToSyslog:
"""``ForwardToSyslog``: the last setting wins; with none, the shipped default
the distribution prints commented out in its ``journald.conf``."""
@pytest.mark.parametrize(
("lines", "expected"),
[
(["#ForwardToSyslog=yes"], True),
(["#ForwardToSyslog=no"], False),
(["#ForwardToSyslog=yes", "ForwardToSyslog=no"], False),
(["ForwardToSyslog=no", "ForwardToSyslog = yes"], True),
([], None),
],
)
def test_the_last_setting_wins_else_the_shipped_default(self, lines, expected):
assert journal_to_syslog(lines) is expected
+394
View File
@@ -0,0 +1,394 @@
"""get_log_sources: which logs a host writes, and who writes each of them.
Before anything can watch a host's logs -- a person in a log window, CrowdSec,
a syslog forwarder -- something has to say which logs there are. Reading that
is the same on every Linux host: journald names the units it holds entries for,
``/proc/<pid>/fd`` shows the log files running processes have open, and
``/proc/<pid>/cgroup`` says which systemd service or container a writer runs in.
So all of it lives here once, and a driver only carries the command across
(NetOrk/netork#2049).
"""
from __future__ import annotations
import shutil
import subprocess
import pytest
from napalm_device_types.log_sources import (
LOG_SOURCES_COMMAND,
LogSourcesMixin,
LogSourcesUnavailable,
parse_log_forwarding,
parse_log_sources,
)
CID = "4f1c" + "0" * 60
OTHER = "9a2b" + "1" * 60
JOURNALD = """nginx.service
ssh.service
postgresql@16-main.service
session-3.scope
init.scope
"""
OPEN = f"""/proc/610/fd/7 /var/log/syslog
/proc/610/fd/8 /var/log/auth.log
/proc/620/fd/9 /var/log/journal/0123abcd/system.journal
/proc/900/fd/5 /var/log/nginx/access.log
/proc/900/fd/6 /var/log/nginx/error.log
/proc/901/fd/5 /var/log/nginx/access.log
/proc/700/fd/4 /var/log/old.log (deleted)
/proc/1100/fd/30 /var/lib/docker/containers/{CID}/{CID}-json.log
/proc/1100/fd/31 /var/lib/docker/containers/{OTHER}/local-logs/container.log
/proc/1400/fd/3 /usr/src/paperless/data/log/paperless.log
/proc/1500/fd/3 /srv/app/log with space.log
/proc/1500/fd/4 /srv/app/Local Storage/leveldb/000004.log
/proc/1500/fd/5 /var/lib/rocksdb/012345.log
/proc/300/fd/11 /var/log/nginx/access.log
/proc/300/fd/12 /mnt/docroot/data/log/paperless.log
/proc/300/fd/13 socket:[48213]
/proc/300/fd/14 pipe:[48214]
/proc/1600/fd/6 /var/log/host/syslog
"""
#: Each descriptor's ``flags`` from ``/proc/<pid>/fdinfo/<fd>``, octal: the access
#: mode (``& 3``) is 1 or 2 for a writer -- 0102001 is O_WRONLY|O_APPEND|O_LARGEFILE
#: -- and 0 for a reader (0100000, O_RDONLY|O_LARGEFILE). CrowdSec (300) reads
#: nginx's log and a log on a mount; a promtail in a container (1600) reads the
#: host's syslog as its own /var/log/host/syslog. 1500 opens its log O_RDWR.
FLAGS = """/proc/610/fdinfo/7:flags:\t0102001
/proc/610/fdinfo/8:flags:\t0102001
/proc/620/fdinfo/9:flags:\t02100002
/proc/900/fdinfo/5:flags:\t0102001
/proc/900/fdinfo/6:flags:\t0102001
/proc/901/fdinfo/5:flags:\t0102001
/proc/700/fdinfo/4:flags:\t0102001
/proc/1100/fdinfo/30:flags:\t0102001
/proc/1100/fdinfo/31:flags:\t0102001
/proc/1400/fdinfo/3:flags:\t0102001
/proc/1500/fdinfo/3:flags:\t0100002
/proc/1500/fdinfo/4:flags:\t0100002
/proc/1500/fdinfo/5:flags:\t0100002
/proc/300/fdinfo/11:flags:\t0100000
/proc/300/fdinfo/12:flags:\t0100000
/proc/300/fdinfo/13:flags:\t02004002
/proc/300/fdinfo/14:flags:\t02004001
/proc/1600/fdinfo/6:flags:\t0100000
"""
STAT = """/proc/610/fd/7 99000 1762250000
/proc/610/fd/8 4100 1762250100
/proc/900/fd/5 2048 1762250200
/proc/900/fd/6 512 1762250300
/proc/901/fd/5 2048 1762250200
/proc/1100/fd/30 7000 1762250400
/proc/1100/fd/31 300 1762250500
/proc/1400/fd/3 0 1762250600
/proc/300/fd/11 2048 1762250200
/proc/300/fd/12 81920 1762250700
"""
CGROUPS = f"""610 0::/system.slice/rsyslog.service
900 0::/system.slice/nginx.service
901 0::/system.slice/nginx.service
1100 0::/system.slice/docker.service
1400 0::/system.slice/docker-{CID}.scope
1500 0::/user.slice/user-1000.slice/session-3.scope
300 0::/system.slice/crowdsec.service
1600 0::/system.slice/docker-{OTHER}.scope
"""
COMM = """610 rsyslogd
900 nginx
901 nginx
1100 dockerd
1400 python3
1500 app
300 crowdsec
1600 promtail
"""
def _wire(
*,
journald: str | None = JOURNALD,
open_: str = OPEN,
stat: str = STAT,
cgroups: str = CGROUPS,
comm: str = COMM,
flags: str | None = FLAGS,
noise: str = "",
forwarding: str = "",
) -> str:
"""The report as the command prints it, framed."""
parts = [noise, "LOGS_BEGIN\n"]
if journald is not None:
parts += ["[journald]\n", journald]
parts += ["[open]\n", open_, "[stat]\n", stat, "[cgroups]\n", cgroups, "[comm]\n", comm]
if flags is not None:
parts += ["[flags]\n", flags]
parts += [forwarding, "LOGS_END\n"]
return "".join(parts)
def _sources(output: str | None = None) -> list:
return parse_log_sources(_wire() if output is None else output)[1]
def _by_key(sources: list) -> dict:
return {(s["kind"], s["container_id"], s["path"] or s["unit"]): s for s in sources}
class TestParse:
def test_journald_names_each_service_unit_once_without_its_suffix(self) -> None:
journald, sources = parse_log_sources(_wire())
units = [s["unit"] for s in sources if s["kind"] == "journald"]
assert units == ["nginx", "postgresql@16-main", "ssh"]
assert journald is True
def test_a_host_without_journald_says_so(self) -> None:
journald, sources = parse_log_sources(_wire(journald=None))
assert journald is False
assert not [s for s in sources if s["kind"] == "journald"]
def test_an_open_file_names_its_writer_size_and_last_write(self) -> None:
sources = _by_key(_sources())
access = sources[("file", None, "/var/log/nginx/access.log")]
assert access == {
"kind": "file",
"path": "/var/log/nginx/access.log",
"unit": "nginx",
"container_id": None,
"process": "nginx",
"size": 2048,
"modified": "2025-11-04T09:56:40+00:00",
}
def test_a_file_two_processes_hold_is_one_source(self) -> None:
paths = [s["path"] for s in _sources() if s["kind"] == "file"]
assert paths.count("/var/log/nginx/access.log") == 1
def test_a_file_without_a_suffix_under_var_log_counts(self) -> None:
sources = _by_key(_sources())
assert sources[("file", None, "/var/log/syslog")]["unit"] == "rsyslog"
def test_a_file_a_container_writes_carries_the_container_and_its_own_path(self) -> None:
sources = _by_key(_sources())
paperless = sources[("file", CID, "/usr/src/paperless/data/log/paperless.log")]
assert paperless["unit"] is None
assert paperless["process"] == "python3"
assert paperless["size"] == 0
def test_the_engines_own_log_file_is_the_containers_stream(self) -> None:
sources = _by_key(_sources())
json_file = sources[("container", CID, f"/var/lib/docker/containers/{CID}/{CID}-json.log")]
assert json_file["process"] == "dockerd"
assert json_file["size"] == 7000
local = sources[
("container", OTHER, f"/var/lib/docker/containers/{OTHER}/local-logs/container.log")
]
assert local["modified"] == "2025-11-04T10:01:40+00:00"
assert not [s for s in _sources() if "docker" in (s["unit"] or "")]
def test_journal_files_and_deleted_files_are_not_sources(self) -> None:
paths = [s["path"] or "" for s in _sources()]
assert not [p for p in paths if "/var/log/journal/" in p or p.startswith("/var/log/old")]
def test_a_databases_write_ahead_log_is_not_a_source(self) -> None:
# LevelDB and RocksDB name theirs 000004.log; it holds records, not lines.
paths = [s["path"] or "" for s in _sources()]
assert not [p for p in paths if p.endswith(("000004.log", "012345.log"))]
def test_a_path_with_a_space_survives(self) -> None:
sources = _by_key(_sources())
spaced = sources[("file", None, "/srv/app/log with space.log")]
# The writer runs in a login session, not in a service; stat did not answer.
assert spaced["unit"] is None
assert spaced["process"] == "app"
assert spaced["size"] is None
assert spaced["modified"] is None
def test_sources_come_sorted_by_kind_then_container_then_name(self) -> None:
sources = _sources()
order = [(s["kind"], s["container_id"] or "", s["path"] or s["unit"]) for s in sources]
assert order == sorted(order)
def test_noise_before_the_frame_is_ignored(self) -> None:
assert _sources(_wire(noise="Last login: today\r\n\x1b[0m"))
def test_a_host_without_gnu_find_raises_rather_than_reading_as_no_files(self) -> None:
# busybox find has neither -lname nor -printf: no files would read as none.
with pytest.raises(LogSourcesUnavailable, match="find"):
parse_log_sources("LOGS_BEGIN\n[journald]\nnginx.service\n[no-find]\nLOGS_END\n")
def test_a_report_cut_short_raises_rather_than_reading_as_fewer_logs(self) -> None:
cut = _wire().replace("LOGS_END\n", "")
with pytest.raises(ValueError, match="log source report"):
parse_log_sources(cut)
class TestWriters:
"""A file is attributed to the process writing it, not to one reading it
(NetOrk/netork#2092): CrowdSec, Promtail or a ``tail -F`` holding a log is
no producer of it."""
def test_a_reader_with_the_lower_pid_does_not_take_the_writers_file(self) -> None:
access = _by_key(_sources())[("file", None, "/var/log/nginx/access.log")]
assert (access["unit"], access["process"]) == ("nginx", "nginx")
def test_a_file_only_readers_hold_is_listed_without_a_producer(self) -> None:
paperless = _by_key(_sources())[("file", None, "/mnt/docroot/data/log/paperless.log")]
assert paperless == {
"kind": "file",
"path": "/mnt/docroot/data/log/paperless.log",
"unit": None,
"container_id": None,
"process": None,
"size": 81920,
"modified": "2025-11-04T10:05:00+00:00",
}
def test_a_file_open_for_reading_and_writing_is_written(self) -> None:
spaced = _by_key(_sources())[("file", None, "/srv/app/log with space.log")]
assert spaced["process"] == "app"
def test_a_file_only_a_reader_in_a_container_holds_is_left_out(self) -> None:
# Its path is the reader's view of a mount: it names nothing on the host,
# and the container does not write it.
assert not [s for s in _sources() if s["path"] == "/var/log/host/syslog"]
def test_sockets_and_pipes_are_no_sources(self) -> None:
assert not [s for s in _sources() if not (s["path"] or "/").startswith("/")]
def test_a_deleted_file_stays_out_though_it_is_written(self) -> None:
assert not [s for s in _sources() if (s["path"] or "").startswith("/var/log/old")]
def test_a_holder_whose_flags_did_not_answer_is_no_writer(self) -> None:
# The process ended between the listing and the read of its fdinfo.
flags = FLAGS.replace("/proc/900/fdinfo/5:flags:\t0102001\n", "")
flags = flags.replace("/proc/901/fdinfo/5:flags:\t0102001\n", "")
access = _by_key(_sources(_wire(flags=flags)))[("file", None, "/var/log/nginx/access.log")]
assert (access["unit"], access["process"]) == (None, None)
def test_a_report_without_flags_names_the_first_holder_as_before(self) -> None:
# An older command printed no [flags]: who writes cannot be told apart.
access = _by_key(_sources(_wire(flags=None)))[("file", None, "/var/log/nginx/access.log")]
assert (access["unit"], access["process"]) == ("crowdsec", "crowdsec")
#: rsyslog running, forwarding to Graylog, its config prefixed so that no line reads as a section.
FORWARDING = """[rsyslog]
running
[rsyslog-conf]
| module(load="imuxsock")
| [comment-looking line]
|
| *.* @graylog.lan:1514;RSYSLOG_SyslogProtocol23Format
| input(type="imfile" File="/var/log/nginx/*.log" Tag="nginx")
[journald-conf]
#ForwardToSyslog=yes
"""
class TestForwarding:
"""Where the logs go beyond the host, read in the same command (NetOrk/netork#2053)."""
def test_rsyslog_forwarding_and_the_journal_reaching_it(self) -> None:
forwarding = parse_log_forwarding(_wire(forwarding=FORWARDING))
assert forwarding is not None
assert forwarding["rsyslog"] is True
assert forwarding["journal"] is True
assert [(r["target"], r["port"], r["complete"]) for r in forwarding["rules"]] == [
("graylog.lan", 1514, True)
]
assert forwarding["file_inputs"] == [{"path": "/var/log/nginx/*.log", "ruleset": None}]
def test_a_config_line_in_brackets_is_no_section(self) -> None:
forwarding = parse_log_forwarding(_wire(forwarding=FORWARDING))
assert forwarding is not None and forwarding["rules"]
def test_an_rsyslog_that_does_not_run_forwards_nothing(self) -> None:
stopped = FORWARDING.replace("running\n", "")
forwarding = parse_log_forwarding(_wire(forwarding=stopped))
assert forwarding == {"rsyslog": False, "journal": False, "rules": [], "file_inputs": []}
def test_a_report_without_the_sections_says_nothing(self) -> None:
assert parse_log_forwarding(_wire()) is None
def test_the_reading_carries_it(self) -> None:
reading = _Host({True: _wire(forwarding=FORWARDING)}).get_log_sources()
assert reading["forwarding"] is not None
assert reading["forwarding"]["rules"][0]["target"] == "graylog.lan"
class _Host(LogSourcesMixin):
"""Carries the command the way a driver does; *answers* per privilege."""
def __init__(self, answers: dict[bool, str]) -> None:
self.answers = answers
self.calls: list[tuple[str, bool]] = []
def _run_log_sources_command(self, command: str, *, privileged: bool) -> str:
self.calls.append((command, privileged))
return self.answers[privileged]
class TestMixin:
def test_it_reads_as_root_in_one_sh_call(self) -> None:
host = _Host({True: _wire()})
reading = host.get_log_sources()
assert reading["attributed"] is True
command, privileged = host.calls[0]
assert privileged is True
assert command.startswith("sh -c ")
assert len(host.calls) == 1
def test_without_root_it_reads_again_and_says_so(self) -> None:
host = _Host({True: "sudo: a password is required\n", False: _wire()})
reading = host.get_log_sources()
assert reading["attributed"] is False
assert [privileged for _, privileged in host.calls] == [True, False]
assert reading["sources"]
def test_a_host_without_gnu_find_is_not_read_again_unprivileged(self) -> None:
host = _Host({True: "LOGS_BEGIN\n[no-find]\nLOGS_END\n"})
with pytest.raises(LogSourcesUnavailable):
host.get_log_sources()
assert len(host.calls) == 1
def test_a_host_that_answers_neither_way_raises(self) -> None:
host = _Host({True: "", False: "garbage"})
with pytest.raises(ValueError):
host.get_log_sources()
class TestCommand:
def test_it_is_valid_posix_sh(self) -> None:
assert subprocess.run(["sh", "-n", "-c", LOG_SOURCES_COMMAND]).returncode == 0
@pytest.mark.skipif(shutil.which("find") is None, reason="needs find on this host")
def test_here_a_file_held_for_writing_has_a_producer_and_one_held_for_reading_none(
self, tmp_path
) -> None:
written, read = tmp_path / "written.log", tmp_path / "read.log"
read.write_text("one line\n")
with written.open("a") as w, read.open() as r:
out = subprocess.run(
["sh", "-c", LOG_SOURCES_COMMAND], capture_output=True, text=True, timeout=60
).stdout
assert w.writable() and r.readable()
sources = {s["path"]: s for s in _sources(out)}
assert sources[str(written)]["process"] is not None
assert sources[str(read)]["process"] is None
@pytest.mark.skipif(shutil.which("find") is None, reason="needs find on this host")
def test_it_runs_here_and_parses(self, tmp_path) -> None:
log = tmp_path / "held.log"
log.write_text("one line\n")
with log.open() as held:
out = subprocess.run(
["sh", "-c", LOG_SOURCES_COMMAND], capture_output=True, text=True, timeout=60
).stdout
assert held.readable()
# This test's own process holds the file, so it must be among the sources.
mine = [s for s in _sources(out) if s["path"] == str(log)]
assert mine and mine[0]["size"] == len("one line\n")
# Whether rsyslog runs here or not, the report says where logs go.
assert parse_log_forwarding(out) is not None
+25
View File
@@ -0,0 +1,25 @@
"""A network target can say which physical NICs it leaves the host through.
A VM on a VLAN-aware bridge without a VLAN tag only gets an address when the
switch port behind the bridge carries an untagged network. To look that port up,
the caller needs the bridge's physical NICs: their LLDP neighbours name the
switch and the port (NetOrk/netork#1325). They come in groups, one per way out
of the bridge: a bond's slaves together, a single NIC alone. A bond has no
neighbour of its own, and its slaves are one link, so one slave's neighbour
speaks for the others when they report none.
"""
from __future__ import annotations
from typing import get_type_hints
from napalm_device_types.models import NetworkTargetDict
def test_uplinks_is_an_optional_list_of_nic_groups():
assert "uplinks" in NetworkTargetDict.__optional_keys__
assert get_type_hints(NetworkTargetDict)["uplinks"] == list[list[str]]
def test_contract_core_stays_required():
assert {"name", "kind", "vlan_aware"} <= NetworkTargetDict.__required_keys__
+86
View File
@@ -18,9 +18,11 @@ import pytest
from napalm_device_types.package_updates import (
APT_UPGRADABLE_COMMAND,
DNF_INSTALLED_CVES_COMMAND,
DNF_SECURITY_COMMAND,
nevra_name,
parse_apt_upgradable,
parse_dnf_installed_cves,
parse_dnf_security,
)
@@ -176,6 +178,18 @@ __DNF_RC=0
with pytest.raises(ValueError, match="exit status 1"):
parse_dnf_security(out)
def test_the_command_reads_only_the_cached_metadata(self, tmp_path):
"""An expired cache made dnf download the repositories' metadata inside the
poll, which outlasted its whole budget (#76)."""
args = tmp_path / "args"
dnf = tmp_path / "dnf"
dnf.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$@\" > {args}\n")
dnf.chmod(0o755)
env = {"PATH": f"{tmp_path}:{os.environ.get('PATH', '/usr/bin:/bin')}"}
subprocess.run(["sh", "-c", DNF_SECURITY_COMMAND], capture_output=True, env=env)
assert "--cacheonly" in args.read_text().split()
@pytest.mark.parametrize(
("nevra", "name"),
[
@@ -186,3 +200,75 @@ __DNF_RC=0
)
def test_the_name_of_a_nevra(self, nevra, name):
assert nevra_name(nevra) == name
class TestDnfInstalledCves:
"""The CVEs the host's installed advisories close, per package (netOrk #2394).
Where netOrk's knowledge base holds no records of a release, the host's own
advisories are the distribution's word that a backport is in. Lines from real
hosts: Oracle Linux 9.8 (dnf 4.14, aarch64) and Amazon Linux 2023 (dnf 4.14).
"""
ORACLE = """\
CVE-2024-6501 Low/Sec. NetworkManager-1:1.48.10-2.0.1.el9_5.aarch64
CVE-2024-3661 Moderate/Sec. NetworkManager-1:1.48.10-5.0.1.el9_5.aarch64
CVE-2023-38545 Important/Sec. curl-7.76.1-23.el9_2.4.aarch64
CVE-2023-38545 Important/Sec. curl-7.76.1-26.el9_3.2.aarch64
CVE-2023-4911 Important/Sec. glibc-common-2.34-60.0.3.el9.aarch64
CVE-2023-4911 Important/Sec. glibc-common-2.34-60.0.3.el9_2.7.aarch64
__DNF_RC=0
"""
AMAZON = """\
CVE-2022-3715 Medium/Sec. bash-5.2.15-1.amzn2023.0.2.x86_64
CVE-2022-23491 Important/Sec. ca-certificates-2021.2.50-1.0.amzn2023.0.6.noarch
CVE-2023-32803 Important/Sec. ca-certificates-2023.2.60-1.0.amzn2023.0.2.noarch
__DNF_RC=0
"""
def test_cves_per_package_once_each_and_sorted(self):
assert parse_dnf_installed_cves(self.ORACLE) == {
"NetworkManager": ["CVE-2024-3661", "CVE-2024-6501"],
"curl": ["CVE-2023-38545"],
"glibc-common": ["CVE-2023-4911"],
}
def test_amazon_linux(self):
assert parse_dnf_installed_cves(self.AMAZON) == {
"bash": ["CVE-2022-3715"],
"ca-certificates": ["CVE-2022-23491", "CVE-2023-32803"],
}
def test_lines_that_name_no_cve_are_left_out(self):
"""A GPG prompt a repository without its key prints, an advisory id."""
out = (
"Importing GPG key 0x6E93CD0C:\n"
' Userid : "https://packagecloud.io/crowdsec/crowdsec"\n'
"ELSA-2023-5453 Important/Sec. glibc-2.34-60.0.3.el9_2.7.aarch64\n"
"__DNF_RC=0\n"
)
assert parse_dnf_installed_cves(out) == {}
def test_a_failed_dnf_raises_instead_of_reading_as_no_advisories(self):
with pytest.raises(ValueError, match="exit status 1"):
parse_dnf_installed_cves("__DNF_RC=1\n")
def test_output_cut_short_raises(self):
with pytest.raises(ValueError, match="no exit status"):
parse_dnf_installed_cves(self.ORACLE.replace("__DNF_RC=0\n", ""))
def test_the_command_lists_installed_advisories_with_their_cves_from_the_cache(self, tmp_path):
args = tmp_path / "args"
dnf = tmp_path / "dnf"
dnf.write_text(f"#!/bin/sh\nprintf '%s\\n' \"$@\" > {args}\n")
dnf.chmod(0o755)
env = {"PATH": f"{tmp_path}:{os.environ.get('PATH', '/usr/bin:/bin')}"}
out = subprocess.run(
["sh", "-c", DNF_INSTALLED_CVES_COMMAND], capture_output=True, text=True, env=env
).stdout
assert {"--cacheonly", "updateinfo", "list", "--installed", "--with-cve"} <= set(
args.read_text().split()
)
assert out == "__DNF_RC=0\n"
+161
View File
@@ -0,0 +1,161 @@
"""A time zone as a device takes it: an IANA name, a POSIX TZ rule, or its parts.
The wire carries IANA names (``Europe/Berlin``). OpenWrt hands ``TZ`` to libc and
wants the rule itself (``CET-1CEST,M3.5.0,M10.5.0/3``); a switch wants a UTC
offset and the summer-time rule. The rule is the footer of the zone's TZif file
from the ``tzdata`` package (moved here from netOrk's ``core/posix_tz.py`` for
NetOrk/netork#2361).
"""
from __future__ import annotations
from importlib.resources import files
import pytest
from napalm_device_types import (
PosixTzDict,
UnknownTimezone,
is_iana_zone,
is_posix_tz,
parse_posix_tz,
posix_tz,
)
BERLIN = "CET-1CEST,M3.5.0,M10.5.0/3"
def _zones() -> list[str]:
return files("tzdata").joinpath("zones").read_text().split()
@pytest.mark.parametrize(
("zone", "rule"),
[
("Europe/Berlin", BERLIN),
("America/New_York", "EST5EDT,M3.2.0,M11.1.0"),
("America/Sao_Paulo", "<-03>3"),
("Australia/Lord_Howe", "<+1030>-10:30<+11>-11,M10.1.0,M4.1.0"),
("Asia/Kolkata", "IST-5:30"),
("UTC", "UTC0"),
],
)
def test_an_iana_name_becomes_its_posix_rule(zone, rule):
assert posix_tz(zone) == rule
assert is_iana_zone(zone)
@pytest.mark.parametrize("rule", [BERLIN, "<+03>-3", "EST5EDT,M3.2.0/2,M11.1.0/2", "UTC0"])
def test_a_posix_string_is_taken_as_it_is(rule):
assert posix_tz(rule) == rule
assert not is_iana_zone(rule)
@pytest.mark.parametrize(
"value",
["Mars/Olympus_Mons", "Europe", "../../etc/passwd", "/etc/localtime", "Europe/../UTC", ""],
)
def test_neither_a_zone_nor_a_rule_has_none(value):
assert posix_tz(value) is None
assert not is_iana_zone(value)
@pytest.mark.parametrize("value", ["UTC", "GMT0", BERLIN, "<-03>3", "IST-5:30"])
def test_posix_strings(value):
assert is_posix_tz(value)
@pytest.mark.parametrize("value", ["Europe/Berlin", "America/Argentina/Buenos_Aires", "", "CET-"])
def test_not_posix_strings(value):
assert not is_posix_tz(value)
def test_every_zone_of_the_database_has_a_rule_the_check_accepts():
# A caller tells an IANA name from a rule with is_posix_tz; it must never
# take the database's own rule for a name.
zones = _zones()
missing = [z for z in zones if not is_posix_tz(posix_tz(z) or "")]
assert len(zones) > 500
assert missing == []
def test_the_refusal_names_the_zone_in_plain_english():
exc = UnknownTimezone("Mars/Olympus_Mons")
assert isinstance(exc, ValueError)
assert exc.zone == "Mars/Olympus_Mons"
assert "Mars/Olympus_Mons" in str(exc)
assert "Europe/Berlin" in str(exc)
class TestTheRuleInParts:
"""What a switch takes: the standard offset and, where there is one, the summer-time rule."""
def test_berlin(self):
assert parse_posix_tz(BERLIN) == PosixTzDict(
std_name="CET",
utc_offset=3600,
dst_name="CEST",
dst_offset=7200,
dst_start={"month": 3, "week": 5, "weekday": 0, "time": 7200},
dst_end={"month": 10, "week": 5, "weekday": 0, "time": 10800},
)
def test_utc_has_no_summer_time(self):
assert parse_posix_tz("UTC0") == {
"std_name": "UTC",
"utc_offset": 0,
"dst_name": None,
"dst_offset": None,
"dst_start": None,
"dst_end": None,
}
@pytest.mark.parametrize("bare", ["UTC", "GMT"])
def test_the_bare_names_openwrt_writes(self, bare):
parts = parse_posix_tz(bare)
assert parts is not None
assert (parts["std_name"], parts["utc_offset"], parts["dst_name"]) == (bare, 0, None)
def test_sao_paulo_has_given_up_summer_time(self):
parts = parse_posix_tz(posix_tz("America/Sao_Paulo") or "")
assert parts is not None
assert (parts["std_name"], parts["utc_offset"], parts["dst_offset"]) == (
"-03",
-10800,
None,
)
def test_lord_howe_moves_by_half_an_hour(self):
parts = parse_posix_tz(posix_tz("Australia/Lord_Howe") or "")
assert parts is not None
assert (parts["utc_offset"], parts["dst_offset"]) == (37800, 39600)
assert parts["dst_start"] == {"month": 10, "week": 1, "weekday": 0, "time": 7200}
assert parts["dst_end"] == {"month": 4, "week": 1, "weekday": 0, "time": 7200}
def test_new_york_is_west_of_utc_and_gains_an_hour_by_default(self):
parts = parse_posix_tz("EST5EDT,M3.2.0,M11.1.0")
assert parts is not None
assert (parts["utc_offset"], parts["dst_offset"]) == (-18000, -14400)
def test_a_transition_time_may_lie_outside_the_day(self):
# Greenland's rule: a day's -1:00 and 0:00 (RFC 8536 3.3.1 allows -167..167 hours).
parts = parse_posix_tz("<-02>2<-01>,M3.5.0/-1,M10.5.0/0")
assert parts is not None
assert parts["dst_start"] == {"month": 3, "week": 5, "weekday": 0, "time": -3600}
assert parts["dst_end"] == {"month": 10, "week": 5, "weekday": 0, "time": 0}
def test_offsets_with_minutes_and_seconds(self):
parts = parse_posix_tz("<+1245>-12:45<+1345>,M9.5.0/2:45,M4.1.0/3:45")
assert parts is not None
assert (parts["utc_offset"], parts["dst_offset"]) == (45900, 49500)
assert parts["dst_start"] == {"month": 9, "week": 5, "weekday": 0, "time": 9900}
@pytest.mark.parametrize("rule", ["Europe/Berlin", "", "CET-", "CET-1CEST,J60,J300"])
def test_what_it_cannot_take_apart_is_none(self, rule):
# Julian-day rules are POSIX, but no zone of the database uses them.
assert parse_posix_tz(rule) is None
def test_every_zone_of_the_database_comes_apart(self):
broken = [z for z in _zones() if parse_posix_tz(posix_tz(z) or "") is None]
assert broken == []
+4
View File
@@ -18,6 +18,7 @@ import pytest
from napalm_device_types import (
AccessPointDriver,
BackupServerDriver,
DeviceTypeDriver,
FirewallDriver,
HypervisorDriver,
@@ -33,6 +34,7 @@ from napalm_device_types.roles import primary_role_of, roles_of
ROLE_BASES = [
AccessPointDriver,
BackupServerDriver,
FirewallDriver,
HypervisorDriver,
ManagementControllerDriver,
@@ -75,6 +77,8 @@ class TestDeclaredMethodsDoNotExistAtRuntime:
@pytest.mark.parametrize(
("base", "method"),
[
(BackupServerDriver, "get_backup_datastores"),
(BackupServerDriver, "run_backup_job"),
(StorageDriver, "get_disks"),
(StorageDriver, "get_volumes"),
(HypervisorDriver, "get_vms"),
+51
View File
@@ -0,0 +1,51 @@
"""get_running_backups: the backups running on a device right now.
An application that restarts a device asks first, right before the restart: a
restart aborts a backup that runs on the device -- a hypervisor's vzdump, a
backup server's backup, sync, verify or garbage collection, a host's own restic
or borg run -- and a guest's restart breaks the backup its hypervisor is taking
of it (NetOrk/netork#2318). A driver mixes it in when it can tell; no role
demands it, and a driver without it cannot tell.
"""
from __future__ import annotations
import inspect
from typing import get_type_hints
from napalm_device_types import (
RUNNING_BACKUP_KINDS,
BackupServerDriver,
DeviceTypeDriver,
HypervisorDriver,
OSDriver,
RunningBackupDict,
RunningBackupsMixin,
)
def test_no_role_demands_it():
for role in (BackupServerDriver, HypervisorDriver, OSDriver, DeviceTypeDriver):
assert not issubclass(role, RunningBackupsMixin)
def test_declared_not_implemented():
"""hasattr is the caller's probe: only a driver that can tell answers True."""
assert not hasattr(RunningBackupsMixin, "get_running_backups")
def test_declaration_documents_the_contract():
source = inspect.getsource(RunningBackupsMixin)
assert "def get_running_backups(self) -> list[RunningBackupDict]" in source
assert "empty list for a failed read" in source
def test_the_kinds_cover_a_backup_servers_jobs_and_the_backup_itself():
assert set(RUNNING_BACKUP_KINDS) == {"backup", "sync", "verify", "gc", "prune", "tape"}
def test_the_shape():
hints = get_type_hints(RunningBackupDict)
assert set(hints) == {"kind", "job", "task", "guests", "since"}
assert hints["guests"] == list[str]
assert hints["since"] == int | None
+108
View File
@@ -0,0 +1,108 @@
"""The scheduled-jobs contract: its digest, the names and specs it takes (NetOrk/netork#1933)."""
from __future__ import annotations
import pytest
from napalm_device_types import ScheduledJobsMixin
from napalm_device_types.scheduled_jobs import (
ACTIONS,
ScheduledJobChanged,
ScheduledJobRefused,
check_job_name,
check_spec,
job_digest,
)
JOB = {
"kind": "cron",
"source": "/etc/crontab",
"run_as": "root",
"schedule": "*/5 * * * *",
"command": "/usr/local/bin/backup",
"enabled": True,
}
class TestDigest:
def test_sixteen_hex_characters(self):
digest = job_digest(JOB)
assert len(digest) == 16
int(digest, 16)
def test_the_same_definition_has_the_same_digest(self):
assert job_digest(JOB) == job_digest(dict(JOB))
@pytest.mark.parametrize(
"field, value",
[
("schedule", "0 * * * *"),
("command", "/usr/local/bin/other"),
("run_as", "nobody"),
("enabled", False),
("source", "/etc/cron.d/backup"),
("kind", "anacron"),
],
)
def test_every_part_of_the_definition_changes_it(self, field, value):
assert job_digest({**JOB, field: value}) != job_digest(JOB)
def test_content_the_definition_points_at_changes_it(self):
assert job_digest(JOB, content="12345 678") != job_digest(JOB)
def test_what_the_host_says_of_its_runs_does_not(self):
assert job_digest({**JOB, "last_run": "2026-11-03T06:00:00Z"}) == job_digest(JOB)
class TestNames:
@pytest.mark.parametrize("name", ["backup", "netork-backup-db", "a", "x1"])
def test_a_good_name(self, name):
check_job_name(name)
@pytest.mark.parametrize("name", ["", "-x", "Backup", "a b", "a/b", "a.timer", "a" * 64])
def test_a_name_a_file_or_unit_could_not_carry_safely(self, name):
with pytest.raises(ScheduledJobRefused):
check_job_name(name)
class TestSpec:
SPEC = {
"name": "backup",
"schedule": "0 3 * * *",
"run_as": "root",
"command": "/usr/local/bin/backup",
"enabled": True,
}
def test_a_good_spec(self):
check_spec(self.SPEC)
check_spec({**self.SPEC, "schedule": "@reboot"})
@pytest.mark.parametrize(
"field, value",
[
("schedule", "every day"),
("run_as", "root; rm"),
("run_as", ""),
("command", ""),
("command", " "),
("description", "two\nlines"),
],
)
def test_a_spec_that_cannot_be_written(self, field, value):
with pytest.raises(ScheduledJobRefused):
check_spec({**self.SPEC, field: value})
class TestContract:
def test_the_mixin_declares_nothing_at_runtime(self):
driver = ScheduledJobsMixin()
for method in ("get_scheduled_jobs", "manage_scheduled_job", "put_scheduled_job"):
assert not hasattr(driver, method)
def test_the_actions(self):
assert ACTIONS == ("enable", "disable", "run")
def test_a_refusal_is_a_value_error_and_a_change_is_not(self):
assert issubclass(ScheduledJobRefused, ValueError)
assert not issubclass(ScheduledJobChanged, ValueError)
+540
View File
@@ -0,0 +1,540 @@
"""Scheduled jobs on a POSIX host: the report as root, switching, running, an application's jobs.
The reading command is run by a real ``sh`` over a directory tree that stands in
for the host, with ``systemctl``, ``crontab``, ``dpkg``, ``install`` and
``journalctl`` replaced by scripts that keep a log -- so the shell itself is what
is tested, not a hand-written copy of its output (NetOrk/netork#1933).
"""
from __future__ import annotations
import os
import subprocess
from pathlib import Path
import pytest
from napalm_device_types import PosixScheduledJobsMixin
from napalm_device_types._scheduled_reading import parse_reading
from napalm_device_types.channel import CommandResult
from napalm_device_types.scheduled_jobs import ScheduledJobChanged, ScheduledJobRefused
TIMERS = """\
Id=apt-daily.timer
Unit=apt-daily.service
Description=Daily apt download activities
TimersCalendar={ OnCalendar=*-*-* 06,18:00:00 ; next_elapse=@1762192800 }
NextElapseUSecRealtime=@1762192800
LastTriggerUSec=@1762149600
UnitFileState=enabled
ActiveState=active
FragmentPath=/usr/lib/systemd/system/apt-daily.timer
Id=fstrim.timer
Unit=fstrim.service
Description=Discard unused blocks once a week
TimersMonotonic={ OnUnitActiveUSec=1w ; next_elapse=n/a }
NextElapseUSecRealtime=n/a
LastTriggerUSec=n/a
UnitFileState=static
ActiveState=inactive
FragmentPath=/etc/systemd/system/fstrim.timer
Id=backup.timer
Unit=backup.service
Description=netork backup
TimersCalendar={ OnCalendar=*-*-* 03:00:00 ; next_elapse=n/a }
NextElapseUSecRealtime=Tue 2026-11-04 03:00:00 UTC
LastTriggerUSec=n/a
UnitFileState=enabled
ActiveState=active
FragmentPath=/etc/systemd/system/backup.timer
"""
UNITS = """\
Id=apt-daily.service
ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily update ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
User=
Result=success
FragmentPath=/usr/lib/systemd/system/apt-daily.service
Id=fstrim.service
ExecStart={ path=/sbin/fstrim ; argv[]=/sbin/fstrim --listed-in /etc/fstab ; ignore_errors=no }
Result=success
FragmentPath=/usr/lib/systemd/system/fstrim.service
Id=backup.service
ExecStart={ path=SCRIPTS/backup ; argv[]=SCRIPTS/backup ; ignore_errors=no }
User=backup
Result=exit-code
FragmentPath=/etc/systemd/system/backup.service
"""
SYSTEMCTL = """#!/bin/sh
echo "systemctl $*" >> "$LOG"
case "$1" in
list-unit-files) printf 'apt-daily.timer enabled enabled\\nfstrim.timer static -\\nbackup.timer enabled enabled\\ngetty@.timer static -\\n' ;;
show) case "$*" in *Unit,Description*) sed "s#SCRIPTS#$SCRIPTS#g" "$FIX/timers" ;; *) sed "s#SCRIPTS#$SCRIPTS#g" "$FIX/units" ;; esac ;;
esac
"""
CRONTAB = """#!/bin/sh
echo "crontab $*" >> "$LOG"
[ "$1" = -u ] || exit 2
if [ "$3" = -l ]; then
[ -f "$SPOOL/$2" ] || { echo "no crontab for $2" >&2; exit 1; }
cat "$SPOOL/$2"
else
cp "$3" "$SPOOL/$2"
fi
"""
DPKG = """#!/bin/sh
for p in "$@"; do case " $OWNED " in *" $p "*) echo "cron: $p" ;; esac; done
"""
INSTALL = """#!/bin/sh
echo "install $*" >> "$LOG"
args=""
while [ $# -gt 0 ]; do case "$1" in -o|-g) shift 2 ;; *) args="$args '$1'"; shift ;; esac; done
eval "/usr/bin/install $args"
"""
ECHO_LOG = """#!/bin/sh
echo "$(basename "$0") $*" >> "$LOG"
[ "$(basename "$0")" = journalctl ] && echo "backup ran fine"
exit 0
"""
class Host(PosixScheduledJobsMixin):
"""A host in a directory; every command is run by sh, root or not."""
def __init__(self, root: Path, *, systemd: bool = True, own_cron_d: bool = True) -> None:
self.root = root
etc = root / "etc"
self.CRON_SYSTEM_FILES = (str(etc / "crontab"),) # type: ignore[misc]
self.CRON_D_DIRS = (str(etc / "cron.d"),) # type: ignore[misc]
self.CRON_SPOOL_DIRS = (str(root / "spool"),) # type: ignore[misc]
self.RUN_PARTS_DIRS = (("daily", str(etc / "cron.daily")),) # type: ignore[misc]
self.ANACRONTAB = str(etc / "anacrontab") # type: ignore[misc]
self.JOB_SCRIPT_DIR = str(root / "scripts") # type: ignore[misc]
self.UNIT_DIR = str(root / "units") # type: ignore[misc]
self.OWN_CRON_D_DIR = str(etc / "cron.d") if own_cron_d else None # type: ignore[misc]
self.ROOT_GROUP = subprocess.run(
["id", "-gn"], capture_output=True, text=True
).stdout.strip() # type: ignore[misc]
self.systemd_dir = root / ("run-systemd" if systemd else "no-systemd")
self.commands: list[str] = []
def _sj_paths(self): # type: ignore[no-untyped-def]
paths = super()._sj_paths()
from dataclasses import replace
return replace(paths, systemd_marker=str(self.systemd_dir))
def run_command(self, command, *, privileged=False, timeout=60, stdin=None): # type: ignore[no-untyped-def]
self.commands.append(command)
env = {
**os.environ,
"PATH": f"{self.root / 'bin'}:{os.environ['PATH']}",
"LOG": str(self.root / "log"),
"SPOOL": str(self.root / "spool"),
"FIX": str(self.root / "fix"),
"SCRIPTS": str(self.root / "scripts"),
"OWNED": str(self.root / "etc" / "crontab"),
}
done = subprocess.run(
["sh", "-c", command], input=stdin, capture_output=True, env=env, timeout=30
)
return CommandResult(done.stdout.decode(), done.stderr.decode(), done.returncode)
def log(self) -> str:
log = self.root / "log"
return log.read_text() if log.exists() else ""
def _tool(root: Path, name: str, body: str) -> None:
path = root / "bin" / name
path.write_text(body)
path.chmod(0o755)
@pytest.fixture
def root(tmp_path: Path) -> Path:
for d in (
"etc/cron.d",
"etc/cron.daily",
"spool",
"scripts",
"units",
"bin",
"fix",
"run-systemd",
):
(tmp_path / d).mkdir(parents=True)
(tmp_path / "etc/crontab").write_text(
"SHELL=/bin/sh\nOUT="
+ str(tmp_path / "out")
+ "\n17 * * * * root cd / && run-parts /etc/cron.hourly\n"
'*/5 * * * * root echo "$OUT" > ' + str(tmp_path / "ran") + "\n"
)
(tmp_path / "etc/cron.d/certbot").write_text("0 */12 * * * root certbot -q renew\n")
(tmp_path / "etc/anacrontab").write_text("1 5 cron.daily run-parts /etc/cron.daily\n")
(tmp_path / "spool/www-data").write_text("*/5 * * * * php /var/www/cron.php\n")
for name, mode in (("logrotate", 0o755), ("off", 0o644), ("x.dpkg-old", 0o755)):
script = tmp_path / "etc/cron.daily" / name
script.write_text("#!/bin/sh\necho rotated\n")
script.chmod(mode)
(tmp_path / "scripts/backup").write_text(
"#!/bin/sh\n# napalm-scheduled-job: backup nightly backup\n# schedule: 0 3 * * *\n/usr/local/bin/backup --all\n"
)
(tmp_path / "fix/timers").write_text(TIMERS)
(tmp_path / "fix/units").write_text(UNITS)
for name, body in (
("systemctl", SYSTEMCTL),
("crontab", CRONTAB),
("dpkg", DPKG),
("install", INSTALL),
("journalctl", ECHO_LOG),
("systemd-analyze", ECHO_LOG),
):
_tool(tmp_path, name, body)
return tmp_path
def _jobs(host: Host) -> dict[str, dict]:
return {j["key"]: j for j in host.get_scheduled_jobs()["jobs"]}
class TestReading:
def test_every_source_is_read(self, root: Path):
jobs = _jobs(Host(root))
kinds = sorted({j["kind"] for j in jobs.values()})
assert kinds == ["anacron", "cron", "run_parts", "systemd_timer"]
users = {j["run_as"] for j in jobs.values() if j["kind"] == "cron"}
assert users == {"root", "www-data"}
def test_a_package_owns_what_it_installed(self, root: Path):
jobs = _jobs(Host(root))
crontab = [j for j in jobs.values() if j["source"] == str(root / "etc/crontab")]
assert {j["managed_by"] for j in crontab} == {"package"}
certbot = next(j for j in jobs.values() if j["source"].endswith("certbot"))
assert certbot["managed_by"] == "admin"
def test_run_parts_by_execute_bit_and_the_names_run_parts_runs(self, root: Path):
jobs = _jobs(Host(root))
daily = root / "etc/cron.daily"
assert jobs[f"run-parts:{daily}/logrotate"]["enabled"]
assert not jobs[f"run-parts:{daily}/off"]["enabled"]
assert f"run-parts:{daily}/x.dpkg-old" not in jobs
def test_a_changed_script_changes_its_digest(self, root: Path):
host = Host(root)
key = f"run-parts:{root}/etc/cron.daily/logrotate"
before = _jobs(host)[key]["digest"]
(root / "etc/cron.daily/logrotate").write_text("#!/bin/sh\necho something else\n")
assert _jobs(host)[key]["digest"] != before
def test_timers_with_their_unit_and_runs(self, root: Path):
apt = _jobs(Host(root))["timer:apt-daily.timer"]
assert apt["schedule"] == "OnCalendar=*-*-* 06,18:00:00"
assert apt["schedule_kind"] == "calendar"
assert apt["command"] == "/usr/lib/apt/apt.systemd.daily update"
assert (apt["run_as"], apt["managed_by"], apt["enabled"]) == ("root", "package", True)
assert apt["last_run"] == "2025-11-03T06:00:00+00:00"
assert apt["next_run"] == "2025-11-03T18:00:00+00:00"
assert apt["last_result"] == "success"
assert apt["actions"] == ["enable", "disable", "run"]
def test_a_static_timer_only_runs(self, root: Path):
fstrim = _jobs(Host(root))["timer:fstrim.timer"]
assert fstrim["schedule"] == "OnUnitActiveSec=1w"
assert fstrim["schedule_kind"] == "monotonic"
assert fstrim["actions"] == ["run"]
assert fstrim.get("last_run") is None
assert fstrim["managed_by"] == "admin"
def test_an_applications_timer_by_its_script(self, root: Path):
backup = _jobs(Host(root))["job:backup"]
assert backup["managed_by"] == "application"
assert (backup["name"], backup["description"]) == ("backup", "nightly backup")
assert (backup["schedule"], backup["schedule_kind"]) == ("0 3 * * *", "cron")
assert backup["command"] == "/usr/local/bin/backup --all"
assert backup["run_as"] == "backup"
assert "edit" in backup["actions"] and "delete" in backup["actions"]
assert backup["next_run"] == "2026-11-04T03:00:00+00:00"
assert "last_result" not in backup
def test_a_file_without_a_last_newline_does_not_swallow_the_next(self, root: Path):
(root / "etc/cron.d/certbot").write_text("0 */12 * * * root certbot -q renew")
jobs = _jobs(Host(root))
assert any(j["source"].endswith("certbot") for j in jobs.values())
assert any(j["source"].endswith("spool/www-data") for j in jobs.values())
def test_without_systemd_no_timers(self, root: Path):
jobs = _jobs(Host(root, systemd=False))
assert not [j for j in jobs.values() if j["kind"] == "systemd_timer"]
def test_a_report_cut_short_raises(self, root: Path):
host = Host(root)
full = host.run_command
host.run_command = lambda c, **kw: full(c, **kw)._replace(
stdout=full(c, **kw).stdout.split("SJ_END")[0][:-3]
) # type: ignore[method-assign]
with pytest.raises(ValueError, match="cut short"):
host.get_scheduled_jobs()
class TestDriverSettings:
def test_what_the_platform_generates(self, root: Path):
host = Host(root)
host.PLATFORM_SOURCES = ("*/cron.d/cert*",) # type: ignore[misc]
certbot = next(j for j in _jobs(host).values() if j["source"].endswith("certbot"))
assert certbot["managed_by"] == "platform"
def test_the_time_zone_and_owners_commands_are_the_drivers(self, root: Path):
host = Host(root)
host.TIME_ZONE_SH = "echo America/New_York" # type: ignore[misc]
host.OWNERS_SH = "cat >/dev/null" # type: ignore[misc]
reading = host.get_scheduled_jobs()
assert reading["time_zone"] == "America/New_York"
assert {j["managed_by"] for j in reading["jobs"] if j["kind"] == "cron"} == {"admin"}
class TestParseReading:
def test_the_time_zone_from_either_file(self):
out = "SJ_BEGIN\n[tz]\n/usr/share/zoneinfo/Europe/Berlin\nSJ_END\n"
assert parse_reading(out).time_zone == "Europe/Berlin"
assert parse_reading("SJ_BEGIN\n[tz]\nEtc/UTC\nSJ_END").time_zone == "Etc/UTC"
assert parse_reading("SJ_BEGIN\n[tz]\nSJ_END").time_zone is None
def test_owners_skip_diversions(self):
out = "SJ_BEGIN\n[owners]\ndiversion by x from: /etc/crontab\ncron, cron-daemon: /etc/cron.d/x\nSJ_END"
reading = parse_reading(out + "")
assert reading.jobs == []
def test_a_time_not_in_utc_is_left_unknown(self):
from napalm_device_types._scheduled_reading import _timestamp
assert _timestamp("Tue 2026-11-04 03:00:00 CET") is None
assert _timestamp("@0") is None
class TestSwitching:
def _cron(self, host: Host, path: str) -> dict:
return next(j for j in _jobs(host).values() if j["source"] == path and j["kind"] == "cron")
def test_a_system_line_switched_off_and_on(self, root: Path):
host = Host(root)
path = str(root / "etc/cron.d/certbot")
job = self._cron(host, path)
mode = os.stat(path).st_mode
assert host.manage_scheduled_job(job["key"], "disable", digest=job["digest"])["success"]
assert (
root / "etc/cron.d/certbot"
).read_text() == "#napalm-disabled: 0 */12 * * * root certbot -q renew\n"
off = _jobs(host)[job["key"]]
assert not off["enabled"]
host.manage_scheduled_job(job["key"], "enable", digest=off["digest"])
assert (root / "etc/cron.d/certbot").read_text() == "0 */12 * * * root certbot -q renew\n"
assert f"install -D -m {mode & 0o777:o}" in host.log()
assert os.stat(path).st_mode == mode
def test_a_linked_file_is_written_through_its_link(self, root: Path):
real = root / "etc/vzdump.cron"
real.write_text("0 1 * * * root vzdump --all\n")
real.chmod(0o640)
link = root / "etc/cron.d/vzdump"
link.symlink_to(real)
host = Host(root)
job = self._cron(host, str(link))
host.manage_scheduled_job(job["key"], "disable", digest=job["digest"])
assert link.is_symlink()
assert real.read_text() == "#napalm-disabled: 0 1 * * * root vzdump --all\n"
assert real.stat().st_mode & 0o777 == 0o640
def test_a_users_line_goes_through_crontab(self, root: Path):
host = Host(root)
job = self._cron(host, str(root / "spool/www-data"))
host.manage_scheduled_job(job["key"], "disable", digest=job["digest"])
assert (root / "spool/www-data").read_text().startswith("#napalm-disabled: */5")
assert "crontab -u www-data /" in host.log()
def test_a_change_it_did_not_see_is_refused(self, root: Path):
host = Host(root)
job = self._cron(host, str(root / "etc/cron.d/certbot"))
with pytest.raises(ScheduledJobChanged):
host.manage_scheduled_job(job["key"], "disable", digest="0" * 16)
def test_an_unknown_job_and_an_unknown_action(self, root: Path):
host = Host(root)
with pytest.raises(LookupError):
host.manage_scheduled_job("cron:/nowhere:1", "disable", digest="x")
with pytest.raises(ScheduledJobRefused):
host.manage_scheduled_job("cron:/nowhere:1", "explode", digest="x")
def test_a_static_timer_is_not_disabled(self, root: Path):
host = Host(root)
fstrim = _jobs(host)["timer:fstrim.timer"]
with pytest.raises(ScheduledJobRefused):
host.manage_scheduled_job("timer:fstrim.timer", "disable", digest=fstrim["digest"])
def test_a_timer_and_a_run_parts_script(self, root: Path):
host = Host(root)
jobs = _jobs(host)
apt = jobs["timer:apt-daily.timer"]
host.manage_scheduled_job(apt["key"], "disable", digest=apt["digest"])
assert "systemctl disable --now -- apt-daily.timer" in host.log()
key = f"run-parts:{root}/etc/cron.daily/off"
host.manage_scheduled_job(key, "enable", digest=jobs[key]["digest"])
assert os.access(root / "etc/cron.daily/off", os.X_OK)
def test_an_applications_timer_is_switched_by_its_name(self, root: Path):
host = Host(root)
backup = _jobs(host)["job:backup"]
host.manage_scheduled_job("job:backup", "disable", digest=backup["digest"])
assert "systemctl disable --now -- backup.timer" in host.log()
class TestRunning:
def test_a_cron_line_runs_as_cron_would(self, root: Path):
host = Host(root)
job = next(j for j in _jobs(host).values() if "ran" in j["command"])
result = host.manage_scheduled_job(job["key"], "run", digest=job["digest"])
assert result["success"]
assert (root / "ran").read_text() == f"{root / 'out'}\n"
assert "timeout 300 sh -c" in host.commands[-1]
def test_a_timer_runs_its_unit_and_says_what_its_journal_said(self, root: Path):
host = Host(root)
backup = _jobs(host)["job:backup"]
result = host.manage_scheduled_job("job:backup", "run", digest=backup["digest"])
assert result == {"success": True, "output": "backup ran fine"}
assert "systemctl start -- backup.service" in host.log()
def test_a_run_parts_script_runs(self, root: Path):
host = Host(root)
key = f"run-parts:{root}/etc/cron.daily/logrotate"
result = host.manage_scheduled_job(key, "run", digest=_jobs(host)[key]["digest"])
assert result == {"success": True, "output": "rotated"}
def test_another_user_runs_through_su(self):
assert Host(Path("/x"))._sj_as("www-data", "true") == [
"su",
"-s",
"/bin/sh",
"www-data",
"-c",
"true",
]
SPEC = {
"name": "netork-tidy",
"schedule": "*/30 1-2 * * *",
"run_as": "root",
"command": "find /tmp -mtime +7 -delete",
"enabled": True,
"description": "tidy %tmp",
}
class TestApplicationJobs:
def test_a_timer_on_a_host_with_systemd(self, root: Path):
host = Host(root)
assert host.put_scheduled_job(SPEC)["output"] == "job:netork-tidy created"
timer = (root / "units/netork-tidy.timer").read_text()
assert "OnCalendar=*-*-* 01,02:00,30:00" in timer
assert "Description=tidy %%tmp" in timer
service = (root / "units/netork-tidy.service").read_text()
assert f"ExecStart={root}/scripts/netork-tidy" in service and "User=root" in service
script = (root / "scripts/netork-tidy").read_text()
assert script.splitlines()[1:] == [
"# napalm-scheduled-job: netork-tidy tidy %tmp",
"# schedule: */30 1-2 * * *",
"find /tmp -mtime +7 -delete",
]
assert "systemctl daemon-reload" in host.log()
assert "systemctl enable --now -- netork-tidy.timer" in host.log()
def test_reboot_is_a_boot_timer(self, root: Path):
Host(root).put_scheduled_job({**SPEC, "schedule": "@reboot", "enabled": False})
assert "OnBootSec=1min" in (root / "units/netork-tidy.timer").read_text()
def test_creating_one_that_is_there_and_changing_one_that_is_not(self, root: Path):
host = Host(root)
with pytest.raises(ScheduledJobChanged):
host.put_scheduled_job({**SPEC, "name": "backup"})
with pytest.raises(LookupError):
host.put_scheduled_job(SPEC, digest="x")
with pytest.raises(ScheduledJobChanged):
host.put_scheduled_job({**SPEC, "name": "backup"}, digest="x")
def test_an_unknown_user_is_refused(self, root: Path):
with pytest.raises(ScheduledJobRefused, match="no user"):
Host(root).put_scheduled_job({**SPEC, "run_as": "no-such-user-here"})
def test_a_timer_deleted(self, root: Path):
host = Host(root)
backup = _jobs(host)["job:backup"]
host.delete_scheduled_job("job:backup", digest=backup["digest"])
assert not (root / "scripts/backup").exists()
assert "systemctl disable --now -- backup.timer" in host.log()
def test_only_an_applications_job_is_deleted(self, root: Path):
host = Host(root)
apt = _jobs(host)["timer:apt-daily.timer"]
with pytest.raises(ScheduledJobRefused):
host.delete_scheduled_job(apt["key"], digest=apt["digest"])
def test_a_cron_d_file_without_systemd(self, root: Path):
host = Host(root, systemd=False)
host.put_scheduled_job(SPEC)
assert (root / "etc/cron.d/netork-tidy").read_text() == (
"# napalm-scheduled-job: netork-tidy tidy %tmp\n*/30 1-2 * * * root find /tmp -mtime +7 -delete\n"
)
job = _jobs(host)["job:netork-tidy"]
assert job["managed_by"] == "application" and job["kind"] == "cron"
host.delete_scheduled_job("job:netork-tidy", digest=job["digest"])
assert not (root / "etc/cron.d/netork-tidy").exists()
def test_marked_lines_in_the_users_crontab_on_a_busybox_host(self, root: Path):
host = Host(root, systemd=False, own_cron_d=False)
host.put_scheduled_job({**SPEC, "run_as": "www-data"})
text = (root / "spool/www-data").read_text()
assert text.endswith(
"# napalm-scheduled-job: netork-tidy tidy %tmp\n*/30 1-2 * * * find /tmp -mtime +7 -delete\n"
)
job = _jobs(host)["job:netork-tidy"]
host.put_scheduled_job(
{**SPEC, "run_as": "www-data", "enabled": False}, digest=job["digest"]
)
assert "#napalm-disabled: */30" in (root / "spool/www-data").read_text()
job = _jobs(host)["job:netork-tidy"]
host.put_scheduled_job({**SPEC, "run_as": "nobody"}, digest=job["digest"])
assert "netork-tidy" not in (root / "spool/www-data").read_text()
assert "netork-tidy" in (root / "spool/nobody").read_text()
job = _jobs(host)["job:netork-tidy"]
host.delete_scheduled_job("job:netork-tidy", digest=job["digest"])
assert (root / "spool/nobody").read_text() == ""
def test_a_cron_d_that_is_not_there_is_the_crontab(self, root: Path):
host = Host(root, systemd=False)
host.OWN_CRON_D_DIR = str(root / "etc/no-cron.d") # type: ignore[misc]
host.put_scheduled_job(SPEC)
assert "netork-tidy" in (root / "spool/root").read_text()
def test_a_cron_line_holds_one_line(self, root: Path):
with pytest.raises(ScheduledJobRefused, match="one line"):
Host(root, systemd=False).put_scheduled_job({**SPEC, "command": "a\nb"})
def test_a_calendar_systemd_refuses(self, root: Path):
_tool(root, "systemd-analyze", "#!/bin/sh\necho 'Failed to parse' >&2\nexit 1\n")
with pytest.raises(ScheduledJobRefused, match="systemd does not take"):
Host(root).put_scheduled_job(SPEC)
def test_a_failed_install_says_so(self, root: Path):
_tool(root, "install", "#!/bin/sh\necho 'read-only file system' >&2\nexit 1\n")
with pytest.raises(RuntimeError, match="read-only"):
Host(root).put_scheduled_job(SPEC)
+350
View File
@@ -0,0 +1,350 @@
"""netOrk's own account on a host: created through the account it logs in with today.
The fake host of :mod:`tests.test_local_accounts`, taught the rule that lets
an account run everything as root without a password.
"""
from __future__ import annotations
import shlex
import pytest
from napalm_device_types.channel import CommandResult
from napalm_device_types.local_accounts import (
service_sudoers_rule,
service_sudoers_rule_path,
sudoers_rule_path,
)
from tests.test_local_accounts import (
OWN_KEY,
SHELL_SYNTAX,
_Account,
_Driver,
_Host,
_root_commands,
)
NETORK_KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINetork netork@central"
class _ServiceHost(_Host):
"""The fake host, whose sudo also takes the rule without a password."""
def is_passwordless(self, name: str) -> bool:
rule = self.files.get(service_sudoers_rule_path(_Driver.SUDOERS_DIR, name))
return self.includes_dropins and rule is not None
def _sudo(self, args: list[str], privileged: bool) -> CommandResult:
name = args[2]
if not self.is_passwordless(name):
return super()._sudo(args, privileged)
out = f"User {name} may run the following commands on host:\n (ALL) NOPASSWD: ALL\n"
return CommandResult(out, "", 0)
def _visudo(self, args: list[str], privileged: bool) -> CommandResult:
content = self.files[args[1]]
if content.endswith(" ALL=(ALL) NOPASSWD: ALL\n") and "\n" not in content[:-1]:
return CommandResult("parsed OK", "", 0)
return super()._visudo(args, privileged)
@pytest.fixture
def host() -> _ServiceHost:
return _ServiceHost()
@pytest.fixture
def driver(host: _ServiceHost) -> _Driver:
return _Driver(host)
def test_the_rule_runs_everything_as_root_without_a_password() -> None:
assert service_sudoers_rule("netork") == "netork ALL=(ALL) NOPASSWD: ALL\n"
def test_the_rule_has_a_file_of_its_own_no_personal_rule_shares() -> None:
path = service_sudoers_rule_path("/etc/sudoers.d", "net.ork_1")
assert path == "/etc/sudoers.d/85-service-net_2eork__1"
assert path != sudoers_rule_path("/etc/sudoers.d", "net.ork_1")
def test_a_new_service_account_gets_the_key_and_sudo_without_a_password(
host: _ServiceHost, driver: _Driver
) -> None:
result = driver.add_service_account("netork", NETORK_KEY)
assert result["success"], result["output"]
assert [s["step"] for s in result["steps"]] == ["account", "sudo", "keys"]
assert host.accounts["netork"].password is None
assert host.files["/home/netork/.ssh/authorized_keys"] == NETORK_KEY + "\n"
assert host.is_passwordless("netork")
account = result["account"]
assert account is not None and account["admin"]
def test_every_command_root_runs_is_one_fixed_command(host: _ServiceHost, driver: _Driver) -> None:
driver.add_service_account("netork", NETORK_KEY)
for command in _root_commands(host):
assert not any(s in command for s in SHELL_SYNTAX), command
def test_an_existing_account_keeps_its_keys_and_gets_the_rule(
host: _ServiceHost, driver: _Driver
) -> None:
host.accounts["netork"] = _Account(1001, "/home/netork", groups=["netork"])
host.files["/home/netork/.ssh/authorized_keys"] = OWN_KEY + "\n"
result = driver.add_service_account("netork", NETORK_KEY)
assert result["success"], result["output"]
assert host.files["/home/netork/.ssh/authorized_keys"] == f"{OWN_KEY}\n{NETORK_KEY}\n"
assert host.is_passwordless("netork")
def test_an_existing_system_account_may_become_netorks(host: _ServiceHost, driver: _Driver) -> None:
host.accounts["netork"] = _Account(998, "/var/lib/netork", groups=["netork"])
result = driver.add_service_account("netork", NETORK_KEY)
assert result["success"], result["output"]
def test_a_locked_account_is_opened_again(host: _ServiceHost, driver: _Driver) -> None:
host.accounts["netork"] = _Account(1001, "/home/netork", groups=["netork"], locked=True)
result = driver.add_service_account("netork", NETORK_KEY)
assert result["success"], result["output"]
assert not host.accounts["netork"].locked
def test_the_account_netork_logs_in_with_already_only_gets_what_it_lacks(
host: _ServiceHost, driver: _Driver
) -> None:
host.login = "netork"
host.accounts["netork"] = _Account(1001, "/home/netork", groups=["netork"])
host.files["/home/netork/.ssh/authorized_keys"] = NETORK_KEY + "\n"
result = driver.add_service_account("netork", NETORK_KEY)
assert result["success"], result["output"]
assert not any(c.startswith("openuser") for c in _root_commands(host))
assert "already" in result["output"]
def test_root_is_never_netorks_account(host: _ServiceHost, driver: _Driver) -> None:
host.accounts["toor"] = _Account(0, "/root", groups=["wheel"])
with pytest.raises(ValueError, match="root"):
driver.add_service_account("toor", NETORK_KEY)
assert not _root_commands(host)
@pytest.mark.parametrize("name", ["Netork", "net ork", "-n", ""])
def test_a_name_no_host_takes_is_refused(driver: _Driver, name: str) -> None:
with pytest.raises(ValueError):
driver.add_service_account(name, NETORK_KEY)
def test_a_key_that_is_none_is_refused(driver: _Driver) -> None:
with pytest.raises(ValueError):
driver.add_service_account("netork", "not a key")
def test_a_rule_sudo_does_not_read_fails_the_step(host: _ServiceHost, driver: _Driver) -> None:
host.includes_dropins = False
result = driver.add_service_account("netork", NETORK_KEY)
assert not result["success"]
sudo = next(s for s in result["steps"] if s["step"] == "sudo")
assert not sudo["ok"] and "does not read it" in sudo["output"]
def test_a_rule_with_a_password_is_not_enough(host: _ServiceHost, driver: _Driver) -> None:
host.other_rules.add("netork")
host.includes_dropins = False
result = driver.add_service_account("netork", NETORK_KEY)
assert not result["success"]
def test_a_failed_create_stops_before_sudo_and_keys(host: _ServiceHost, driver: _Driver) -> None:
host.failing.add("mkuser")
result = driver.add_service_account("netork", NETORK_KEY)
assert not result["success"]
assert [s["step"] for s in result["steps"]] == ["account"]
assert result["account"] is None
def test_the_staged_files_are_removed_again(host: _ServiceHost, driver: _Driver) -> None:
driver.add_service_account("netork", NETORK_KEY)
assert not [p for p in host.files if p.startswith("/tmp/")]
def test_a_personal_account_of_the_same_name_keeps_its_own_rule_apart(
host: _ServiceHost, driver: _Driver
) -> None:
driver.add_service_account("netork", NETORK_KEY)
rules = [c for c in _root_commands(host) if c.startswith("install")]
assert [shlex.split(c)[-1] for c in rules] == [
service_sudoers_rule_path(_Driver.SUDOERS_DIR, "netork")
]
# -- a fixed uid (infrastructure BAUSTANDARD A) ------------------------------------------
class _UidDriver(_Driver):
def _account_create_with_uid(self, name: str, uid: int) -> list[list[str]]:
return [["mkuseruid", name, str(uid)]]
def _mkuseruid(host: _ServiceHost):
def handler(args: list[str], privileged: bool) -> CommandResult:
assert privileged
name, uid = args[0], int(args[1])
host.accounts[name] = _Account(uid, f"/home/{name}", groups=[name])
return CommandResult("", "", 0)
return handler
def _getent_uid(host: _ServiceHost, original):
def handler(args: list[str], privileged: bool) -> CommandResult:
if args[1].isdigit():
owner = next((n for n, a in host.accounts.items() if a.uid == int(args[1])), None)
if owner is None:
return CommandResult("", "", 2)
a = host.accounts[owner]
return CommandResult(f"{owner}:*:{a.uid}:{a.uid}::{a.home}:{a.shell}\n", "", 0)
return original(args, privileged)
return handler
@pytest.fixture
def uid_driver(host: _ServiceHost) -> _UidDriver:
host._mkuseruid = _mkuseruid(host) # type: ignore[attr-defined]
host._getent = _getent_uid(host, host._getent) # type: ignore[method-assign]
return _UidDriver(host)
def test_a_new_account_gets_the_uid_asked_for(host: _ServiceHost, uid_driver: _UidDriver) -> None:
result = uid_driver.add_service_account("netork", NETORK_KEY, uid=900)
assert result["success"], result["output"]
assert host.accounts["netork"].uid == 900
account = result["account"]
assert account is not None and account["uid"] == 900
def test_a_uid_another_account_has_stops_before_anything_is_made(
host: _ServiceHost, uid_driver: _UidDriver
) -> None:
host.accounts["fwupd-refresh"] = _Account(900, "/run/fwupd")
result = uid_driver.add_service_account("netork", NETORK_KEY, uid=900)
assert not result["success"]
assert "fwupd-refresh" in result["output"]
assert "netork" not in host.accounts
def test_an_existing_account_keeps_its_uid_and_says_so(
host: _ServiceHost, uid_driver: _UidDriver
) -> None:
host.accounts["netork"] = _Account(1001, "/home/netork", groups=["netork"])
result = uid_driver.add_service_account("netork", NETORK_KEY, uid=900)
assert result["success"], result["output"]
assert host.accounts["netork"].uid == 1001
assert "uid 1001" in result["output"]
def test_a_uid_root_has_is_refused(uid_driver: _UidDriver) -> None:
with pytest.raises(ValueError):
uid_driver.add_service_account("netork", NETORK_KEY, uid=0)
def test_a_driver_that_cannot_set_a_uid_says_so(host: _ServiceHost, driver: _Driver) -> None:
result = driver.add_service_account("netork", NETORK_KEY, uid=900)
assert not result["success"]
assert "uid" in result["output"]
# -- another service's account of the same name (bw-01: the satellite's) ------------------
@pytest.mark.parametrize(
"shell", ["/usr/sbin/nologin", "/sbin/nologin", "/bin/false", "/usr/bin/false"]
)
def test_an_account_that_cannot_log_in_is_another_services_and_left_alone(
host: _ServiceHost, driver: _Driver, shell: str
) -> None:
host.accounts["netork"] = _Account(
999, "/home/netork", shell=shell, groups=["netork", "docker"]
)
with pytest.raises(ValueError, match="cannot log in"):
driver.add_service_account("netork", NETORK_KEY)
assert not _root_commands(host)
assert service_sudoers_rule_path(_Driver.SUDOERS_DIR, "netork") not in host.files
OLD_KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOldNetork netork@central"
def test_a_replaced_key_goes_once_the_new_one_is_in(host: _ServiceHost, driver: _Driver) -> None:
host.login = "netork"
host.accounts["netork"] = _Account(900, "/home/netork", groups=["netork"])
host.files["/home/netork/.ssh/authorized_keys"] = f"{OWN_KEY}\n{OLD_KEY}\n"
result = driver.add_service_account("netork", NETORK_KEY, replaces=[OLD_KEY])
assert result["success"], result["output"]
assert [s["step"] for s in result["steps"]] == ["account", "sudo", "keys", "old keys"]
assert host.files["/home/netork/.ssh/authorized_keys"] == f"{OWN_KEY}\n{NETORK_KEY}\n"
def test_the_new_key_is_never_one_that_goes(host: _ServiceHost, driver: _Driver) -> None:
host.login = "netork"
host.accounts["netork"] = _Account(900, "/home/netork", groups=["netork"])
host.files["/home/netork/.ssh/authorized_keys"] = f"{NETORK_KEY}\n"
result = driver.add_service_account(
"netork", NETORK_KEY, replaces=[f"{NETORK_KEY.split()[0]} {NETORK_KEY.split()[1]}"]
)
assert result["success"], result["output"]
assert [s["step"] for s in result["steps"]] == ["account", "sudo", "keys"]
assert host.files["/home/netork/.ssh/authorized_keys"] == f"{NETORK_KEY}\n"
def test_a_replaced_key_stays_while_the_new_one_is_not_in(
host: _ServiceHost, driver: _Driver
) -> None:
host.login = "netork"
host.accounts["netork"] = _Account(900, "/home/netork", groups=["netork"])
host.files["/home/netork/.ssh/authorized_keys"] = f"{OLD_KEY}\n"
host.failing.add("as install")
result = driver.add_service_account("netork", NETORK_KEY, replaces=[OLD_KEY])
assert not result["success"]
assert "old keys" not in [s["step"] for s in result["steps"]]
assert host.files["/home/netork/.ssh/authorized_keys"] == f"{OLD_KEY}\n"
def test_replaced_keys_are_a_list_not_one_string(driver: _Driver) -> None:
with pytest.raises(ValueError, match="list"):
driver.add_service_account("netork", NETORK_KEY, replaces=OLD_KEY)
+19 -1
View File
@@ -22,7 +22,7 @@ from napalm_device_types import (
def test_a_profile_says_where_each_value_lives():
assert set(get_type_hints(SnmpHealthProfile)) == {"uptime", "cpu", "memory", "interfaces"}
assert set(get_type_hints(SnmpValueSpec)) == {"oids", "table", "fold", "pattern"}
assert set(get_type_hints(SnmpValueSpec)) == {"oids", "table", "fold", "pattern", "unknown"}
assert set(get_type_hints(SnmpMemorySpec)) == {"total", "used", "free", "percent", "unit"}
assert set(get_type_hints(SnmpInterfacesSpec)) == {"skip", "tx_errors_are_drops"}
@@ -42,6 +42,24 @@ def test_a_driver_declares_its_profile_as_a_class_attribute():
assert Driver.SNMP_HEALTH_PROFILE["cpu"]["oids"][0] == "1.3.6.1.4.1.9.1"
def test_a_value_the_agent_gets_wrong_is_declared_unknown():
"""Neither the standard reading nor a default: the application reports none."""
assert get_type_hints(SnmpValueSpec)["unknown"] is bool
class Driver:
SNMP_HEALTH_PROFILE: ClassVar[SnmpHealthProfile] = {"cpu": {"unknown": True}}
assert Driver.SNMP_HEALTH_PROFILE["cpu"].get("unknown") is True
def test_the_shapes_live_in_their_own_module_and_models_still_exports_them():
from napalm_device_types import models, snmp_health_profile
for name in ("SnmpHealthProfile", "SnmpValueSpec", "SnmpMemorySpec", "SnmpInterfacesSpec"):
assert getattr(models, name) is getattr(snmp_health_profile, name), name
assert getattr(snmp_health_profile, name).__module__ == snmp_health_profile.__name__
def test_the_collector_in_the_base_package_warns_that_it_is_deprecated():
async def get(oid):
return None
+250
View File
@@ -0,0 +1,250 @@
"""Exempting the managing client from sshd's per-source penalties (NetOrk/netork#1800).
OpenSSH 9.8 penalises a source whose connections do not authenticate and then
drops every connection from it. A security scanner's probes earn the managing
client exactly that, and its next session -- a poll, an Ansible run -- is
turned away. ``exempt_from_ssh_penalties`` puts the address the host sees the
client at into ``PerSourcePenaltyExemptList``, where that directive exists.
The real outputs this is held to were taken from Debian 13 (OpenSSH 10.0p2) and
Debian 12 (OpenSSH 9.2p1, which knows no penalties and rejects the directive).
"""
from __future__ import annotations
import re
import pytest
from napalm_device_types.channel import CommandResult
from napalm_device_types.ssh_server import (
DROPIN_DIR,
MAIN_CONFIG,
PENALTY_DROPIN_PATH,
OpenSshServerMixin,
client_address,
is_exempt,
parse_penalty_exemptions,
)
#: ``sshd -T | grep -i penalt`` on Debian 13, OpenSSH 10.0p2, as installed.
DEBIAN_13 = (
"persourcepenaltyexemptlist none\n"
"persourcepenalties crash:90 authfail:5 noauth:1 grace-exceeded:10 refuseconnection:10"
" max:600 min:15 max-sources4:65536 max-sources6:65536 overflow:permissive"
" overflow6:permissive\n"
)
# ── reading ──────────────────────────────────────────────────────────────
def test_a_new_sshd_has_penalties_and_no_exemption():
assert parse_penalty_exemptions(DEBIAN_13) == []
def test_the_exemptions_sshd_uses_are_read():
text = DEBIAN_13.replace("none", "10.40.100.10,192.0.2.0/24")
assert parse_penalty_exemptions(text) == ["10.40.100.10", "192.0.2.0/24"]
def test_an_sshd_before_9_8_has_no_penalties():
"""It prints neither line -- and refuses the directive, so nothing may be written."""
assert parse_penalty_exemptions("port 22\nmacs hmac-sha2-256\n") is None
@pytest.mark.parametrize(
("value", "address"),
[
("10.40.100.10 41866 10.40.100.118 22", "10.40.100.10"),
("fd00::10 41866 fd00::118 22", "fd00::10"),
("", None),
("not-an-address 1 2 3", None),
("10.40.100.10 41866", None),
],
)
def test_the_client_address_comes_from_ssh_connection(value, address):
assert client_address(value) == address
def test_an_address_inside_an_exempt_network_is_exempt():
assert is_exempt("10.40.100.10", ["192.0.2.0/24", "10.40.100.0/24"])
assert not is_exempt("10.40.100.10", ["10.40.100.11", "bogus"])
# ── the mixin, against a fake host ───────────────────────────────────────
class PenaltyHost(OpenSshServerMixin):
"""An sshd that reads the main config and the drop-in directory, first value wins."""
def __init__(
self,
*,
main: str = f"Include {DROPIN_DIR}/*.conf\nPort 22\n",
penalties: bool = True,
client: str = "10.40.100.10 41866 10.40.100.118 22",
) -> None:
self.files: dict[str, str] = {MAIN_CONFIG: main}
self.penalties = penalties
self.client = client
self.syntax_ok = True
self.reloads = 0
self.ran: list[str] = []
def _config_lines(self) -> list[str]:
lines: list[str] = []
for line in self.files[MAIN_CONFIG].splitlines():
if re.match(r"\s*include\s", line, re.I):
for path in sorted(p for p in self.files if p.startswith(DROPIN_DIR + "/")):
lines += self.files[path].splitlines()
elif re.match(r"\s*match\s", line, re.I):
break
else:
lines.append(line)
return lines
def exempt(self) -> str:
for line in self._config_lines():
parts = line.split()
if len(parts) == 2 and parts[0].lower() == "persourcepenaltyexemptlist":
return parts[1]
return "none"
def run_command(self, command, *, privileged=False, timeout=60, stdin=None):
self.ran.append(command)
if "SSH_CONNECTION" in command:
assert not privileged, "it is the login session's, not sudo's"
return CommandResult(self.client, "", 0)
if "mktemp" in command:
path = f"/tmp/netork-ssh.{len(self.files)}"
self.files[path] = (stdin or b"").decode()
return CommandResult(path, "", 0)
assert privileged, command
m = re.search(r"install -m \"\$m\" -- '?([^' ]+)'? '?([^' ]+)'?", command)
if m:
target = re.search(r"mv -f -- '?[^' ]+'? '?([^' ;]+)'?", command).group(1)
self.files[target] = self.files.pop(m.group(1))
return CommandResult("", "", 0)
if "sshd -T" in command:
if not self.penalties:
return CommandResult("port 22\n", "", 0)
return CommandResult(DEBIAN_13.replace("none", self.exempt()), "", 0)
if "sshd -t" in command:
return CommandResult(
"", "" if self.syntax_ok else "bad line 1", 0 if self.syntax_ok else 255
)
m = re.match(r"cat -- '?([^' ]+)'?$", command)
if m:
path = m.group(1)
if path in self.files:
return CommandResult(self.files[path], "", 0)
return CommandResult("", "No such file", 1)
m = re.match(r"rm -f -- '?([^' ]+)'?$", command)
if m:
self.files.pop(m.group(1), None)
return CommandResult("", "", 0)
if "reload" in command or "HUP" in command:
self.reloads += 1
return CommandResult("", "", 0)
if command.startswith("mkdir"):
return CommandResult("", "", 0)
raise AssertionError(f"unexpected command: {command}")
def test_the_client_address_is_exempted_in_a_dropin():
host = PenaltyHost()
lines: list[str] = []
result = host.exempt_from_ssh_penalties(log=lines.append)
assert result.applied and not result.rolled_back
assert result.address == "10.40.100.10"
assert result.path == PENALTY_DROPIN_PATH
assert host.exempt() == "10.40.100.10"
assert host.reloads == 1
assert lines
def test_the_hosts_own_exemptions_stay():
host = PenaltyHost(
main=f"Include {DROPIN_DIR}/*.conf\nPerSourcePenaltyExemptList 192.0.2.0/24\n"
)
result = host.exempt_from_ssh_penalties()
assert result.applied
assert host.exempt() == "192.0.2.0/24,10.40.100.10"
def test_nothing_is_written_to_an_sshd_without_penalties():
"""OpenSSH before 9.8 does not start with the directive in its config."""
host = PenaltyHost(penalties=False)
result = host.exempt_from_ssh_penalties()
assert not result.applied and not result.rolled_back
assert PENALTY_DROPIN_PATH not in host.files
assert host.reloads == 0
assert "9.8" in result.message
def test_nothing_changes_when_the_address_is_already_exempt():
host = PenaltyHost(
main=f"Include {DROPIN_DIR}/*.conf\nPerSourcePenaltyExemptList 10.40.100.0/24\n"
)
result = host.exempt_from_ssh_penalties()
assert not result.applied
assert PENALTY_DROPIN_PATH not in host.files
assert "already" in result.message
def test_an_sshd_config_without_the_include_is_left_alone():
host = PenaltyHost(main="Port 22\n")
result = host.exempt_from_ssh_penalties()
assert not result.applied
assert host.files[MAIN_CONFIG] == "Port 22\n"
assert DROPIN_DIR in result.message
def test_a_rejected_config_is_taken_back():
host = PenaltyHost()
host.syntax_ok = False
result = host.exempt_from_ssh_penalties()
assert result.rolled_back and not result.applied
assert PENALTY_DROPIN_PATH not in host.files
def test_an_exemption_sshd_does_not_use_is_taken_back():
"""A drop-in read before ours sets the directive; ours is valid and ignored."""
host = PenaltyHost()
host.files[f"{DROPIN_DIR}/00-aaa.conf"] = "PerSourcePenaltyExemptList 192.0.2.1\n"
result = host.exempt_from_ssh_penalties()
assert result.rolled_back
assert PENALTY_DROPIN_PATH not in host.files
def test_it_is_taken_back_when_a_new_session_cannot_get_in():
host = PenaltyHost()
result = host.exempt_from_ssh_penalties(verify=lambda: False)
assert result.rolled_back
assert PENALTY_DROPIN_PATH not in host.files
assert host.reloads == 2
def test_a_session_without_a_client_address_raises():
host = PenaltyHost(client="")
with pytest.raises(RuntimeError):
host.exempt_from_ssh_penalties()
+168
View File
@@ -0,0 +1,168 @@
"""A connection sshd turns away before the SSH banner is tried again (NetOrk/netork#1800).
OpenSSH 9.8+ drops every connection from a source it has just penalised
(``PerSourcePenalties``): it accepts the TCP connection and closes it at once,
and paramiko reports "Error reading SSH protocol banner". The penalty lasts
seconds to minutes, so a connection that waits a little gets in.
"""
from __future__ import annotations
import socket
import threading
import paramiko
import pytest
from napalm.base.exceptions import ConnectionException
from napalm_device_types import ssh_host_keys
from napalm_device_types.ssh_host_keys import (
REFUSED_RETRY_WAITS,
HostKeyCheck,
refused_before_banner,
retry_refused,
)
def _banner_error() -> paramiko.SSHException:
try:
try:
raise EOFError()
except EOFError as eof:
raise paramiko.SSHException("Error reading SSH protocol banner") from eof
except paramiko.SSHException as exc:
return exc
class TestWhatCountsAsRefused:
def test_paramikos_banner_error(self):
assert refused_before_banner(_banner_error())
def test_wrapped_by_a_driver(self):
try:
try:
raise _banner_error()
except paramiko.SSHException as exc:
raise ConnectionException(f"cannot connect: {exc}") from exc
except ConnectionException as wrapped:
assert refused_before_banner(wrapped)
@pytest.mark.parametrize(
"exc",
[
paramiko.AuthenticationException("Authentication failed."),
TimeoutError("timed out"),
ConnectionRefusedError(111, "Connection refused"),
ValueError("anything else"),
],
)
def test_not_a_failed_login_a_silent_host_or_a_closed_port(self, exc):
"""Those do not end by waiting: retrying them only delays the error."""
assert not refused_before_banner(exc)
class TestRetry:
def test_it_waits_and_tries_again(self):
slept: list[float] = []
attempts = iter([_banner_error(), _banner_error(), "connected"])
def attempt():
value = next(attempts)
if isinstance(value, BaseException):
raise value
return value
assert retry_refused(attempt, sleep=slept.append) == "connected"
assert slept == list(REFUSED_RETRY_WAITS[:2])
def test_it_gives_up_with_the_last_error(self):
slept: list[float] = []
def attempt():
raise _banner_error()
with pytest.raises(paramiko.SSHException):
retry_refused(attempt, sleep=slept.append)
assert slept == list(REFUSED_RETRY_WAITS)
def test_other_errors_are_not_retried(self):
slept: list[float] = []
def attempt():
raise paramiko.AuthenticationException("Authentication failed.")
with pytest.raises(paramiko.AuthenticationException):
retry_refused(attempt, sleep=slept.append)
assert slept == []
def test_the_waits_outlast_the_penalty_seen(self):
"""16 s on a new Ubuntu 26.04 VM on .50."""
assert sum(REFUSED_RETRY_WAITS) > 16
class _PenalisingServer:
"""A TCP server that closes the first *drops* connections at once, as a penalised
source's are, and then speaks SSH with any password."""
def __init__(self, drops: int) -> None:
self.drops = drops
self.listener = socket.create_server(("127.0.0.1", 0))
self.port = self.listener.getsockname()[1]
self.key = paramiko.ECDSAKey.generate()
self._thread = threading.Thread(target=self._serve, daemon=True)
self._thread.start()
def _serve(self) -> None:
while True:
try:
conn, _ = self.listener.accept()
except OSError:
return
if self.drops:
self.drops -= 1
conn.close()
continue
transport = paramiko.Transport(conn)
transport.add_server_key(self.key)
class _Any(paramiko.ServerInterface):
def get_allowed_auths(self, username):
return "password"
def check_auth_password(self, username, password):
return paramiko.AUTH_SUCCESSFUL
try:
transport.start_server(server=_Any())
except Exception:
pass
self.transport = transport
return
def close(self) -> None:
self.listener.close()
transport = getattr(self, "transport", None)
if transport:
transport.close()
def test_host_key_check_connects_once_the_penalty_is_over(monkeypatch):
server = _PenalisingServer(drops=2)
slept: list[float] = []
monkeypatch.setattr(ssh_host_keys.time, "sleep", slept.append)
client = paramiko.SSHClient()
try:
HostKeyCheck({}, "127.0.0.1", server.port).connect(
client,
username="admin",
password="s3cret",
look_for_keys=False,
allow_agent=False,
timeout=5,
banner_timeout=5,
)
assert client.get_transport().is_authenticated()
assert slept == list(REFUSED_RETRY_WAITS[:2])
finally:
client.close()
server.close()
+27
View File
@@ -14,6 +14,7 @@ import pytest
from napalm_device_types.channel import CommandResult
from napalm_device_types.ssh_server import (
DROPIN_DIR,
DROPIN_PATH,
MAIN_CONFIG,
OpenSshServerMixin,
@@ -401,6 +402,32 @@ def test_mixin_falls_back_to_sshd_config_without_an_include():
assert host._effective()["mac"] == ["hmac-sha2-256"]
class GeneratedConfigHost(FakeHost):
"""A platform that writes sshd_config from its own database (OpenMediaVault)."""
SSHD_CONFIG_GENERATED = True
def test_generated_sshd_config_still_gets_the_dropin():
host = GeneratedConfigHost()
result = host.harden_ssh_server({"mac": ["hmac-sha1"]})
assert result.applied
assert result.path == DROPIN_PATH
assert host.files[MAIN_CONFIG] == "Include /etc/ssh/sshd_config.d/*.conf\nPort 22\n"
def test_generated_sshd_config_without_an_include_is_left_alone():
"""The platform would write its own file over the change on its next apply."""
host = GeneratedConfigHost(main="Port 22\nMACs hmac-sha1,hmac-sha2-256\n")
result = host.harden_ssh_server({"mac": ["hmac-sha1"]})
assert not result.applied and not result.rolled_back
assert result.path is None
assert host.files[MAIN_CONFIG] == "Port 22\nMACs hmac-sha1,hmac-sha2-256\n"
assert list(host.files) == [MAIN_CONFIG]
assert host.reloads == 0
assert DROPIN_DIR in result.message
def test_mixin_restores_when_sshd_rejects_the_config():
host = FakeHost()
host.syntax_ok = False
+353
View File
@@ -0,0 +1,353 @@
"""How an OpenSSH server lets people in: the effective settings and sshd's PAM auth stack.
The parsers are pure. The getter runs against a fake host that answers ``sshd -T``
and the dump of ``/etc/pam.d`` from a dict, so the root read, the PAM includes
and a failed read are covered without a machine.
The ``sshd -T`` texts are abridged from what OpenSSH 9.2p1 on Debian 12 prints
(the keywords and their order as sshd writes them; the lines that say nothing
about authentication are left out). OpenSSH before 8.7 also prints
``challengeresponseauthentication``; OpenBSD's sshd has no ``usepam``.
"""
from __future__ import annotations
import pytest
from napalm_device_types.channel import CommandResult
from napalm_device_types.ssh_server import OpenSshServerMixin
from napalm_device_types.ssh_server_auth import (
PAM_FILES_COMMAND,
SSHD_EFFECTIVE_COMMAND,
SshServerAuthMixin,
pam_auth_stack,
parse_pam_auth,
parse_pam_files,
parse_sshd_auth,
)
#: Debian 12 as installed: password login on, PAM on, no AuthenticationMethods.
SSHD_T_DEBIAN = """\
port 22
addressfamily any
listenaddress [::]:22
listenaddress 0.0.0.0:22
usepam yes
logingracetime 120
x11displayoffset 10
maxauthtries 6
maxsessions 10
clientaliveinterval 0
clientalivecountmax 3
permitrootlogin without-password
ignorerhosts yes
ignoreuserknownhosts no
hostbasedauthentication no
hostbasedusesnamefrompacketonly no
pubkeyauthentication yes
kerberosauthentication no
kerberosorlocalpasswd yes
kerberosticketcleanup yes
gssapiauthentication no
gssapicleanupcredentials yes
gssapikeyexchange no
gssapistrictacceptorcheck yes
gssapistorecredentialsonrotate no
passwordauthentication yes
kbdinteractiveauthentication no
printmotd no
x11forwarding yes
authenticationmethods any
kexalgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256
"""
#: Key only: password and keyboard-interactive off.
SSHD_T_KEY_ONLY = SSHD_T_DEBIAN.replace(
"passwordauthentication yes", "passwordauthentication no"
).replace("permitrootlogin without-password", "permitrootlogin no")
#: A key and then an OTP through PAM, for everyone.
SSHD_T_KEY_AND_OTP = SSHD_T_KEY_ONLY.replace(
"kbdinteractiveauthentication no", "kbdinteractiveauthentication yes"
).replace("authenticationmethods any", "authenticationmethods publickey,keyboard-interactive")
#: OpenSSH 8.4p1 (Debian 11): the old keyword beside the new one.
SSHD_T_OLD = SSHD_T_KEY_ONLY.replace(
"kbdinteractiveauthentication no",
"kbdinteractiveauthentication yes\nchallengeresponseauthentication no",
)
#: OpenBSD 7.6: no usepam line.
SSHD_T_OPENBSD = SSHD_T_KEY_ONLY.replace("usepam yes\n", "")
#: Debian 12's /etc/pam.d/sshd, abridged: the auth stack comes from common-auth.
PAM_SSHD_DEBIAN = """\
# PAM configuration for the Secure Shell service
# Standard Un*x authentication.
@include common-auth
# Disallow non-root logins when /etc/nologin exists.
account required pam_nologin.so
# Standard Un*x authorization.
@include common-account
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so close
@include common-session
@include common-password
"""
PAM_COMMON_AUTH = """\
auth\t[success=1 default=ignore]\tpam_unix.so nullok
auth\trequisite\t\t\tpam_deny.so
auth\trequired\t\t\tpam_permit.so
"""
PAM_COMMON_ACCOUNT = "account [success=1 new_authtok_reqd=done default=ignore] pam_unix.so\n"
#: The usual way to add a TOTP: a line of its own in sshd's stack.
PAM_SSHD_OTP = PAM_SSHD_DEBIAN.replace(
"@include common-auth",
"#@include common-auth\nauth required pam_google_authenticator.so nullok",
)
#: Red Hat's form: ``auth substack password-auth``, a module by its full path.
PAM_SSHD_RHEL = """\
#%PAM-1.0
auth substack password-auth
auth include postlogin
-auth optional /usr/lib64/security/pam_reauthorize.so prepare
account required pam_sepermit.so
"""
PAM_PASSWORD_AUTH = "auth required pam_env.so\nauth sufficient pam_unix.so try_first_pass nullok\n"
PAM_POSTLOGIN = "session optional pam_umask.so silent\n"
# ── parse_sshd_auth ──────────────────────────────────────────────────────
def test_parse_reads_the_settings_that_decide_how_people_log_in():
auth = parse_sshd_auth(SSHD_T_DEBIAN)
assert auth == {
"password_authentication": True,
"kbd_interactive_authentication": False,
"pubkey_authentication": True,
"hostbased_authentication": False,
"gssapi_authentication": False,
"authentication_methods": [],
"permit_root_login": "without-password",
"use_pam": True,
"pam_auth_stack": None,
}
def test_parse_splits_authentication_methods_into_lists_of_methods():
auth = parse_sshd_auth(
SSHD_T_KEY_ONLY.replace(
"authenticationmethods any",
"authenticationmethods publickey,keyboard-interactive:pam publickey,password",
)
)
assert auth["authentication_methods"] == [
["publickey", "keyboard-interactive:pam"],
["publickey", "password"],
]
def test_parse_counts_the_old_challenge_response_keyword_as_keyboard_interactive():
"""Before 8.7 either keyword could switch PAM's prompts on: either one is enough."""
assert parse_sshd_auth(SSHD_T_OLD)["kbd_interactive_authentication"] is True
old_only = SSHD_T_KEY_ONLY.replace(
"kbdinteractiveauthentication no",
"kbdinteractiveauthentication no\nchallengeresponseauthentication yes",
)
assert parse_sshd_auth(old_only)["kbd_interactive_authentication"] is True
def test_parse_says_none_for_usepam_where_sshd_has_no_pam():
assert parse_sshd_auth(SSHD_T_OPENBSD)["use_pam"] is None
def test_parse_takes_a_method_sshd_was_built_without_as_off():
"""sshd built without GSSAPI prints no gssapiauthentication line: that method cannot be used."""
text = SSHD_T_DEBIAN.replace("gssapiauthentication no\n", "")
assert parse_sshd_auth(text)["gssapi_authentication"] is False
def test_parse_refuses_output_that_is_not_sshd_t():
with pytest.raises(ValueError, match="passwordauthentication"):
parse_sshd_auth("sshd: no hostkeys available -- exiting.\n")
# ── PAM ──────────────────────────────────────────────────────────────────
def _pairs(entries) -> list[tuple[str, str]]:
return [(e["control"], e["module"]) for e in entries]
def _modules(stack) -> list[str]:
return [e["module"] for e in stack]
def test_pam_auth_lists_modules_and_includes_in_stack_order():
# Every @include: which of them hold auth lines is for the included file to say.
assert _pairs(parse_pam_auth(PAM_SSHD_DEBIAN)) == [
("include", "common-auth"),
("include", "common-account"),
("include", "common-session"),
("include", "common-password"),
]
def test_pam_auth_reads_bracketed_controls_paths_and_optional_lines():
assert _pairs(parse_pam_auth(PAM_SSHD_RHEL)) == [
("substack", "password-auth"),
("include", "postlogin"),
("optional", "pam_reauthorize.so"),
]
assert parse_pam_auth(PAM_COMMON_AUTH) == [
{"module": "pam_unix.so", "control": "[success=1 default=ignore]", "args": ["nullok"]},
{"module": "pam_deny.so", "control": "requisite", "args": []},
{"module": "pam_permit.so", "control": "required", "args": []},
]
def test_pam_auth_keeps_a_modules_arguments():
"""``nullok`` lets a user without an OTP secret through: the consumer has to see it."""
assert parse_pam_auth(PAM_SSHD_OTP)[0] == {
"module": "pam_google_authenticator.so",
"control": "required",
"args": ["nullok"],
}
def test_pam_auth_skips_comments_and_other_types():
assert _pairs(parse_pam_auth(PAM_SSHD_OTP)) == [
("required", "pam_google_authenticator.so"),
("include", "common-account"),
("include", "common-session"),
("include", "common-password"),
]
def test_pam_auth_ignores_an_include_that_is_not_a_service_name():
assert parse_pam_auth("auth include ../../etc/shadow\n@include\nauth required\n") == []
def _dump(files: dict[str, str]) -> str:
"""What ``tail -n +1`` prints for several files: a header before each."""
return "\n".join(f"==> /etc/pam.d/{name} <==\n{text}" for name, text in files.items())
def test_pam_files_splits_the_dump_by_file():
files = {"sshd": PAM_SSHD_DEBIAN, "common-auth": PAM_COMMON_AUTH}
assert parse_pam_files(_dump(files)) == files
def test_auth_stack_follows_includes_and_substacks_in_stack_order():
files = {
"sshd": PAM_SSHD_RHEL + "auth include password-auth\n",
"password-auth": PAM_PASSWORD_AUTH,
"postlogin": PAM_POSTLOGIN,
}
assert _modules(pam_auth_stack(files)) == ["pam_env.so", "pam_unix.so", "pam_reauthorize.so"]
def test_auth_stack_stops_at_an_include_loop_and_skips_a_missing_file():
files = {
"sshd": "@include a\n@include gone\n",
"a": "@include sshd\nauth required pam_unix.so\n",
}
assert _modules(pam_auth_stack(files)) == ["pam_unix.so"]
def test_auth_stack_is_none_without_a_file_for_the_service():
assert pam_auth_stack({"login": PAM_COMMON_AUTH}) is None
# ── the getter ───────────────────────────────────────────────────────────
class _Host(SshServerAuthMixin):
"""Answers ``sshd -T`` (root only) and the dump of /etc/pam.d."""
def __init__(self, sshd_t: str | None, pam: dict[str, str] | None = None) -> None:
self.sshd_t = sshd_t
self.pam = pam
self.calls: list[tuple[str, bool]] = []
def run_command(
self,
command: str,
*,
privileged: bool = False,
timeout: float = 60,
stdin: bytes | None = None,
) -> CommandResult:
self.calls.append((command, privileged))
if command == SSHD_EFFECTIVE_COMMAND:
if self.sshd_t is None or not privileged:
return CommandResult("", "sshd: no hostkeys available -- exiting.", 1)
return CommandResult(self.sshd_t, "", 0)
assert command == PAM_FILES_COMMAND, command
if self.pam is None:
return CommandResult("", "tail: cannot open '/etc/pam.d/*' for reading", 1)
return CommandResult(_dump(self.pam), "", 0)
def test_getter_reads_sshd_t_as_root_and_pam_without():
host = _Host(
SSHD_T_DEBIAN,
{
"sshd": PAM_SSHD_DEBIAN,
"common-auth": PAM_COMMON_AUTH,
"common-account": PAM_COMMON_ACCOUNT,
},
)
auth = host.get_ssh_server_auth()
assert auth["password_authentication"] is True
assert _modules(auth["pam_auth_stack"]) == ["pam_unix.so", "pam_deny.so", "pam_permit.so"]
# sshd -T once, as root; PAM's files once, as the login user.
assert host.calls == [(SSHD_EFFECTIVE_COMMAND, True), (PAM_FILES_COMMAND, False)]
def test_getter_names_an_otp_module_in_sshds_own_stack():
host = _Host(SSHD_T_KEY_AND_OTP, {"sshd": PAM_SSHD_OTP})
stack = host.get_ssh_server_auth()["pam_auth_stack"]
assert _modules(stack) == ["pam_google_authenticator.so"]
def test_getter_keeps_what_it_read_when_one_pam_file_is_unreadable():
"""tail goes on past a file it cannot open and exits 1: the rest is still read."""
class _Partial(_Host):
def run_command(self, command: str, **kwargs) -> CommandResult: # type: ignore[override]
result = super().run_command(command, **kwargs)
if command == PAM_FILES_COMMAND:
return CommandResult(result.stdout, "tail: cannot open 'x'", 1)
return result
host = _Partial(SSHD_T_DEBIAN, {"sshd": "auth required pam_unix.so\n"})
assert _modules(host.get_ssh_server_auth()["pam_auth_stack"]) == ["pam_unix.so"]
def test_getter_leaves_pam_unread_without_usepam():
host = _Host(SSHD_T_OPENBSD)
auth = host.get_ssh_server_auth()
assert auth["pam_auth_stack"] is None
assert host.calls == [(SSHD_EFFECTIVE_COMMAND, True)]
def test_getter_says_none_when_pam_cannot_be_read():
assert _Host(SSHD_T_DEBIAN).get_ssh_server_auth()["pam_auth_stack"] is None
def test_getter_raises_when_sshd_t_fails():
with pytest.raises(RuntimeError, match="no hostkeys"):
_Host(None).get_ssh_server_auth()
def test_the_hardening_mixin_carries_the_reader():
"""Every driver that can harden sshd can also say how it lets people in."""
assert issubclass(OpenSshServerMixin, SshServerAuthMixin)
+395
View File
@@ -0,0 +1,395 @@
"""The sudo password an earlier driver left in shell histories, and taking it back.
The scripts are run for real, with ``sh``, against a home directory in a temp
directory: what they count, delete and report is what a host would see. Root's
part -- giving files back to root -- runs with a ``chown`` that only records,
so the test needs no root.
"""
from __future__ import annotations
import getpass
import os
import subprocess
from pathlib import Path
import pytest
from napalm_device_types.channel import CommandResult
from napalm_device_types.sudo_history import HISTORY_BACKUP_SUFFIX, SudoHistoryMixin
MARKER = "| sudo -S -p "
LEAKED = "echo 'hunter2' | sudo -S -p \"\" apt-get update\n"
OTHER = "ls -la\n"
class _Host(SudoHistoryMixin):
"""A driver whose host is this machine, with *home* as the login user's home."""
SUDO_HISTORY_MARKER = MARKER
def __init__(self, home: Path, bin_dir: Path, *, root: bool = False) -> None:
self.home = home
self.bin_dir = bin_dir
self.root = root
#: Who ``id -un`` says the login user is; None for this machine's user name.
self.user: str | None = None
self.ran: list[tuple[str, bool]] = []
def run_command(
self,
command: str,
*,
privileged: bool = False,
timeout: float = 60,
stdin: bytes | None = None,
) -> CommandResult:
# The password goes ahead of a privileged command's stdin: there must be none.
assert not (privileged and stdin is not None)
self.ran.append((command, privileged))
env = {
**os.environ,
"HOME": str(self.home),
"PATH": f"{self.bin_dir}:{os.environ['PATH']}",
}
if command.strip() == "id -u; id -un" and self.root:
return CommandResult("0\nroot\n", "", 0)
if command.strip() == "id -u; id -un":
# Never this machine's uid: CI runs as root, which the owner step skips.
return CommandResult(f"4242\n{self.user or getpass.getuser()}\n", "", 0)
done = subprocess.run( # noqa: S603 - the test's own scripts
["sh", "-c", command], capture_output=True, text=True, env=env, timeout=timeout
)
return CommandResult(done.stdout, done.stderr, done.returncode)
@pytest.fixture
def bin_dir(tmp_path: Path) -> Path:
"""A ``chown`` that records its arguments instead of changing anything."""
d = tmp_path / "bin"
d.mkdir()
chown = d / "chown"
chown.write_text(f'#!/bin/sh\nprintf "%s\\n" "$*" >> {tmp_path / "chowned"}\n')
chown.chmod(0o755)
return d
@pytest.fixture
def home(tmp_path: Path) -> Path:
h = tmp_path / "home"
h.mkdir()
return h
def _chowned(tmp_path: Path) -> list[str]:
log = tmp_path / "chowned"
return log.read_text().splitlines() if log.exists() else []
class TestCounting:
def test_counts_the_lines_per_history_file_without_reading_them_out(self, home, bin_dir):
(home / ".bash_history").write_text(OTHER + LEAKED * 3)
(home / ".zsh_history").write_text(": 1700000000:0;" + LEAKED)
(home / ".sh_history").write_text(OTHER)
host = _Host(home, bin_dir)
found = host.get_history_sudo_lines()
assert found == [
{"file": "~/.bash_history", "lines": 3},
{"file": "~/.zsh_history", "lines": 1},
]
# Counted as the login user, and nothing of the content came back.
assert all(not privileged for _cmd, privileged in host.ran)
assert "hunter2" not in repr(found)
def test_no_history_no_lines(self, home, bin_dir):
assert _Host(home, bin_dir).get_history_sudo_lines() == []
def test_the_pattern_is_a_fixed_string(self, home, bin_dir):
# "-S" is no regex trouble, but "|" would be an alternation in an ERE.
(home / ".bash_history").write_text("sudo -S -p\n| sudo\n")
assert _Host(home, bin_dir).get_history_sudo_lines() == []
def test_a_driver_that_never_typed_the_password_counts_nothing(self, home, bin_dir):
(home / ".bash_history").write_text(LEAKED)
host = _Host(home, bin_dir)
host.SUDO_HISTORY_MARKER = None
assert host.get_history_sudo_lines() == []
assert host.ran == []
def test_an_unreadable_history_raises_rather_than_reporting_none(self, home, bin_dir):
if os.geteuid() == 0:
pytest.skip("root reads every file")
history = home / ".bash_history"
history.write_text(LEAKED)
history.chmod(0)
try:
with pytest.raises(RuntimeError):
_Host(home, bin_dir).get_history_sudo_lines()
finally:
history.chmod(0o600)
class TestRemoving:
def test_deletes_exactly_those_lines_and_says_how_many(self, home, bin_dir):
(home / ".bash_history").write_text(OTHER + LEAKED + "cd /etc\n" + LEAKED)
(home / ".ash_history").write_text(LEAKED)
host = _Host(home, bin_dir)
result = host.remove_history_sudo_lines()
assert result["success"] is True
assert (home / ".bash_history").read_text() == OTHER + "cd /etc\n"
assert (home / ".ash_history").read_text() == ""
assert result["removed"] == 3
assert result["files"] == [
{"file": "~/.bash_history", "before": 2, "after": 0},
{"file": "~/.ash_history", "before": 1, "after": 0},
]
assert "~/.bash_history: 2 lines removed (2 before, 0 after)" in result["output"]
def test_no_backup_with_the_password_is_left_behind(self, home, bin_dir):
(home / ".bash_history").write_text(LEAKED)
_Host(home, bin_dir).remove_history_sudo_lines()
assert sorted(p.name for p in home.iterdir()) == [".bash_history"]
assert not (home / f".bash_history{HISTORY_BACKUP_SUFFIX}").exists()
def test_the_history_runs_as_the_login_user(self, home, bin_dir):
(home / ".bash_history").write_text(LEAKED)
host = _Host(home, bin_dir)
host.remove_history_sudo_lines()
history_scripts = [(c, p) for c, p in host.ran if ".bash_history" in c]
assert history_scripts and all(not p for _c, p in history_scripts)
def test_a_clean_history_is_left_as_it_is(self, home, bin_dir):
history = home / ".bash_history"
history.write_text(OTHER)
before = history.stat().st_ino
result = _Host(home, bin_dir).remove_history_sudo_lines()
assert result["success"] is True
assert result["removed"] == 0
assert result["files"] == [{"file": "~/.bash_history", "before": 0, "after": 0}]
assert history.stat().st_ino == before # not rewritten
def test_a_history_that_is_a_link_is_left_alone_and_the_fix_fails(
self, home, bin_dir, tmp_path
):
target = tmp_path / "elsewhere"
target.write_text(LEAKED)
(home / ".bash_history").symlink_to(target)
result = _Host(home, bin_dir).remove_history_sudo_lines()
assert result["success"] is False
assert target.read_text() == LEAKED
assert (home / ".bash_history").is_symlink()
assert "~/.bash_history: a symbolic link, left alone (1 line)" in result["output"]
def test_the_lines_of_a_driver_that_never_typed_the_password_are_not_looked_for(
self, home, bin_dir
):
host = _Host(home, bin_dir)
host.SUDO_HISTORY_MARKER = None
result = host.remove_history_sudo_lines()
assert result["success"] is True
assert result["removed"] == 0
def test_an_unreadable_history_fails_the_step(self, home, bin_dir):
if os.geteuid() == 0:
pytest.skip("root reads every file")
history = home / ".bash_history"
history.write_text(LEAKED)
history.chmod(0)
try:
result = _Host(home, bin_dir).remove_history_sudo_lines()
finally:
history.chmod(0o600)
assert result["success"] is False
assert "~/.bash_history: cannot be read" in result["output"]
def test_a_sed_that_fails_keeps_the_lines_and_leaves_no_backup(self, home, bin_dir):
(bin_dir / "sed").write_text(
'#!/bin/sh\nfor a; do f="$a"; done\ncp "$f" "$f.napalm-bak"\nexit 4\n'
)
(bin_dir / "sed").chmod(0o755)
(home / ".bash_history").write_text(LEAKED * 2)
result = _Host(home, bin_dir).remove_history_sudo_lines()
assert result["success"] is False
assert "~/.bash_history: could not be rewritten (2 lines kept)" in result["output"]
assert sorted(p.name for p in home.iterdir()) == [".bash_history"]
def test_a_sed_that_lost_the_file_gets_it_back(self, home, bin_dir):
(bin_dir / "sed").write_text(
'#!/bin/sh\nfor a; do f="$a"; done\nmv "$f" "$f.napalm-bak"\nexit 4\n'
)
(bin_dir / "sed").chmod(0o755)
(home / ".bash_history").write_text(OTHER + LEAKED)
result = _Host(home, bin_dir).remove_history_sudo_lines()
assert result["success"] is False
assert (home / ".bash_history").read_text() == OTHER + LEAKED
assert sorted(p.name for p in home.iterdir()) == [".bash_history"]
def test_a_script_that_dies_says_why(self, home, bin_dir):
host = _Host(home, bin_dir)
host.run_command = lambda command, **kw: CommandResult("", "sh: killed", 137) # type: ignore[method-assign]
result = host.remove_history_sudo_lines()
assert result["success"] is False
assert "sh: killed" in result["output"]
def test_a_line_it_does_not_know_is_ignored(self):
files: list = []
said: list[str] = []
assert SudoHistoryMixin._history_line(["what", "ever"], files, said) is True
assert SudoHistoryMixin._history_line([], files, said) is True
assert files == [] and said == []
class TestOwners:
def _host(self, home, bin_dir, files):
host = _Host(home, bin_dir)
host.SUDO_ROOT_FILES = tuple(str(f) for f in files)
return host
def test_a_file_the_login_user_owns_goes_back_to_root(self, home, bin_dir, tmp_path):
conf = tmp_path / "snmpd.conf"
conf.write_text("x")
result = self._host(home, bin_dir, [conf]).remove_history_sudo_lines()
assert _chowned(tmp_path) == [f"-h 0:0 {conf}"]
assert result["owners_restored"] == [str(conf)]
assert result["success"] is True
def test_only_as_root_and_without_stdin(self, home, bin_dir, tmp_path):
conf = tmp_path / "snmpd.conf"
conf.write_text("x")
host = self._host(home, bin_dir, [conf])
host.remove_history_sudo_lines()
privileged = [c for c, p in host.ran if p]
assert len(privileged) == 1 and "chown" in privileged[0]
# The login user's name is quoted into root's script, never taken from stdin.
assert getpass.getuser() in privileged[0]
def test_a_link_or_a_missing_file_is_not_touched(self, home, bin_dir, tmp_path):
real = tmp_path / "real"
real.write_text("x")
link = tmp_path / "00proxy"
link.symlink_to(real)
result = self._host(home, bin_dir, [link, tmp_path / "missing"]).remove_history_sudo_lines()
assert _chowned(tmp_path) == []
assert result["owners_restored"] == []
assert result["success"] is True
def test_a_file_someone_else_owns_is_not_touched(self, home, bin_dir, tmp_path):
conf = tmp_path / "snmpd.conf"
conf.write_text("x")
host = self._host(home, bin_dir, [conf])
host.user = "nobody-here-4242"
result = host.remove_history_sudo_lines()
assert _chowned(tmp_path) == []
assert result["owners_restored"] == []
def test_a_root_login_has_nothing_to_give_back(self, home, bin_dir, tmp_path):
conf = tmp_path / "snmpd.conf"
conf.write_text("x")
host = self._host(home, bin_dir, [conf])
host.root = True
result = host.remove_history_sudo_lines()
assert _chowned(tmp_path) == []
assert result["success"] is True
def test_the_owners_are_set_even_when_the_history_could_not_be_cleaned(
self, home, bin_dir, tmp_path
):
target = tmp_path / "elsewhere"
target.write_text(LEAKED)
(home / ".bash_history").symlink_to(target)
conf = tmp_path / "snmpd.conf"
conf.write_text("x")
result = self._host(home, bin_dir, [conf]).remove_history_sudo_lines()
assert result["success"] is False
assert [s["step"] for s in result["steps"]] == ["history", "owners"]
assert result["owners_restored"] == [str(conf)]
def test_without_knowing_the_login_user_nothing_changes_owner(self, home, bin_dir, tmp_path):
conf = tmp_path / "snmpd.conf"
conf.write_text("x")
host = self._host(home, bin_dir, [conf])
real = host.run_command
def run(command, **kw):
if command.startswith("id "):
return CommandResult("", "id: not found", 127)
return real(command, **kw)
host.run_command = run # type: ignore[method-assign]
result = host.remove_history_sudo_lines()
assert result["success"] is False
assert "cannot tell who the login user is" in result["output"]
assert _chowned(tmp_path) == []
def test_a_sudo_that_refuses_fails_the_step(self, home, bin_dir, tmp_path):
conf = tmp_path / "snmpd.conf"
conf.write_text("x")
host = self._host(home, bin_dir, [conf])
real = host.run_command
def run(command, *, privileged=False, **kw):
if privileged:
return CommandResult("", "sudo: a password is required", 1)
return real(command, privileged=privileged, **kw)
host.run_command = run # type: ignore[method-assign]
result = host.remove_history_sudo_lines()
assert result["success"] is False
assert "sudo: a password is required" in result["output"]
def test_a_refused_chown_fails_the_step(self, home, bin_dir, tmp_path):
(bin_dir / "chown").write_text("#!/bin/sh\necho 'Operation not permitted' >&2\nexit 1\n")
conf = tmp_path / "snmpd.conf"
conf.write_text("x")
result = self._host(home, bin_dir, [conf]).remove_history_sudo_lines()
assert result["success"] is False
assert result["owners_restored"] == []
assert f"{conf}: could not be given back to root" in result["output"]
def test_exported_from_the_package():
import napalm_device_types
assert napalm_device_types.SudoHistoryMixin is SudoHistoryMixin
+245
View File
@@ -0,0 +1,245 @@
"""TimeSyncMixin: a device's time sources and time zone, read and set (NetOrk/netork#2361).
The application keeps one list of NTP servers and a zone per site, and aligns a
device that differs. ``get_time_sync`` is declared only, so ``callable(getattr(
driver, "get_time_sync", None))`` is the truthful answer; ``configure_time_sync``
is a template of steps like every fix, and an API driver overrides it whole.
"""
from __future__ import annotations
import inspect
from typing import ClassVar, get_type_hints
import pytest
from napalm_device_types import (
TIME_SYNC_VALUES,
AccessPointDriver,
DeviceTypeDriver,
FirewallDriver,
OSDriver,
SwitchDriver,
TimeSyncDict,
TimeSyncMixin,
UnknownTimezone,
check_ntp_servers,
)
def test_the_shape():
hints = get_type_hints(TimeSyncDict)
assert hints == {
"ntp_servers": list[str],
"timezone": str | None,
"synchronized": bool | None,
"daemon": str | None,
"dhcp_ntp_servers": list[str] | None,
}
def test_the_dhcp_servers_are_optional():
# A driver that cannot tell what DHCP brought leaves the key out (NetOrk/netork#2380).
assert TimeSyncDict.__required_keys__ == {"ntp_servers", "timezone", "synchronized", "daemon"}
assert TimeSyncDict.__optional_keys__ == {"dhcp_ntp_servers"}
def test_the_two_values_a_driver_may_write():
assert frozenset({"ntp_servers", "timezone"}) == TIME_SYNC_VALUES
def test_no_role_demands_it():
for role in (AccessPointDriver, FirewallDriver, OSDriver, SwitchDriver, DeviceTypeDriver):
assert not issubclass(role, TimeSyncMixin)
def test_the_reading_is_declared_not_implemented():
assert not hasattr(TimeSyncMixin, "get_time_sync")
source = inspect.getsource(TimeSyncMixin)
assert "def get_time_sync(self) -> TimeSyncDict" in source
assert "time_sync_writes: ClassVar[frozenset[str]]" in source
def test_the_template_takes_both_values_or_none_of_them():
signature = inspect.signature(TimeSyncMixin.configure_time_sync)
assert list(signature.parameters) == ["self", "ntp_servers", "timezone"]
assert signature.parameters["ntp_servers"].default is None
assert signature.parameters["timezone"].default is None
class _Device(TimeSyncMixin):
"""A device that keeps its values in memory and reads back what it was given."""
time_sync_writes: ClassVar[frozenset[str]] = frozenset({"ntp_servers", "timezone"})
def __init__(self, *, keeps_zone_name: bool = True, fail: str = "") -> None:
self.state: TimeSyncDict = {
"ntp_servers": ["0.pool.ntp.org"],
"timezone": "UTC",
"synchronized": True,
"daemon": None,
}
self.keeps_zone_name = keeps_zone_name
self.fail = fail
self.calls: list[tuple[str, object]] = []
def get_time_sync(self) -> TimeSyncDict:
state = dict(self.state)
if not self.keeps_zone_name:
state["timezone"] = None
return state # type: ignore[return-value]
def _write_ntp_servers(self, servers):
self.calls.append(("ntp_servers", servers))
if self.fail == "ntp_servers":
return {"ok": False, "output": "refused"}
self.state["ntp_servers"] = list(servers)
return {"ok": True, "output": "written"}
def _write_time_zone(self, zone):
self.calls.append(("timezone", zone))
self.state["timezone"] = zone
return {"ok": True, "output": f"zone {zone}"}
class TestConfigureTimeSync:
def test_both_values_then_a_check(self):
device = _Device()
result = device.configure_time_sync(["ntp1.example.org", "10.0.0.1"], "Europe/Berlin")
assert result["success"] is True
assert [s["step"] for s in result["steps"]] == ["ntp_servers", "timezone", "verify"]
assert device.calls == [
("ntp_servers", ["ntp1.example.org", "10.0.0.1"]),
("timezone", "Europe/Berlin"),
]
def test_none_leaves_a_value_alone(self):
device = _Device()
result = device.configure_time_sync(timezone="Europe/Berlin")
assert result["success"] is True
assert device.calls == [("timezone", "Europe/Berlin")]
assert device.state["ntp_servers"] == ["0.pool.ntp.org"]
def test_servers_alone(self):
device = _Device()
assert device.configure_time_sync(["ntp1.example.org"])["success"] is True
assert device.calls == [("ntp_servers", ["ntp1.example.org"])]
def test_an_unknown_zone_is_refused_before_anything_is_written(self):
device = _Device()
with pytest.raises(UnknownTimezone, match="Mars/Olympus_Mons"):
device.configure_time_sync(["ntp1.example.org"], "Mars/Olympus_Mons")
assert device.calls == []
def test_a_posix_rule_is_not_a_zone_on_the_wire(self):
with pytest.raises(UnknownTimezone):
_Device().configure_time_sync(timezone="CET-1CEST,M3.5.0,M10.5.0/3")
def test_a_value_the_driver_cannot_write_is_refused_before_anything_is_written(self):
class Bmc(_Device):
time_sync_writes = frozenset({"ntp_servers"})
device = Bmc()
with pytest.raises(ValueError, match="timezone"):
device.configure_time_sync(["ntp1.example.org"], "Europe/Berlin")
assert device.calls == []
def test_a_driver_that_names_nothing_it_writes_writes_nothing(self):
class Silent(TimeSyncMixin):
"""Mixes the contract in and never says what it writes."""
with pytest.raises(ValueError, match="ntp_servers"):
Silent().configure_time_sync(["ntp1.example.org"])
def test_nothing_asked_is_a_mistake(self):
with pytest.raises(ValueError, match="nothing"):
_Device().configure_time_sync()
def test_a_failed_step_stops_the_rest(self):
device = _Device(fail="ntp_servers")
result = device.configure_time_sync(["ntp1.example.org"], "Europe/Berlin")
assert result["success"] is False
assert [s["step"] for s in result["steps"]] == ["ntp_servers"]
assert device.calls == [("ntp_servers", ["ntp1.example.org"])]
def test_the_check_reads_back_what_was_written(self):
class Stubborn(_Device):
def _write_ntp_servers(self, servers):
self.calls.append(("ntp_servers", servers))
return {"ok": True, "output": "written"} # and nothing changed
result = Stubborn().configure_time_sync(["ntp1.example.org"])
assert result["success"] is False
verify = result["steps"][-1]
assert verify["step"] == "verify"
assert "0.pool.ntp.org" in verify["output"]
def test_the_check_names_the_zone_the_device_kept(self):
class Elsewhere(_Device):
def _write_time_zone(self, zone):
return {"ok": True, "output": "set"} # and the device stayed on UTC
result = Elsewhere().configure_time_sync(timezone="Europe/Berlin")
assert result["success"] is False
assert result["steps"][-1]["output"] == "the device is in the time zone UTC"
def test_servers_compare_without_case(self):
class Lower(_Device):
def get_time_sync(self):
state = super().get_time_sync()
state["ntp_servers"] = [s.lower() for s in state["ntp_servers"]]
return state
assert Lower().configure_time_sync(["NTP1.Example.org"])["success"] is True
def test_servers_from_dhcp_do_not_fail_the_check(self):
# DHCP option 42 counts: what the device got by DHCP stays and is no mistake.
device = _Device()
device.state["dhcp_ntp_servers"] = ["192.168.1.1"]
result = device.configure_time_sync(["ntp1.example.org"])
assert result["success"] is True, result["output"]
def test_a_device_without_a_zone_name_is_not_checked_for_one(self):
# A switch keeps an offset and a summer-time rule, not a name: it reads None.
result = _Device(keeps_zone_name=False).configure_time_sync(timezone="Europe/Berlin")
assert result["success"] is True
class TestCheckNtpServers:
def test_in_order_stripped_each_once(self):
assert check_ntp_servers([" ntp1.example.org", "10.0.0.1", "NTP1.example.org", "::1"]) == [
"ntp1.example.org",
"10.0.0.1",
"::1",
]
def test_none_at_all_is_refused(self):
# A device without a source drifts; leaving the value alone is None, not [].
with pytest.raises(ValueError, match="at least one"):
check_ntp_servers([])
with pytest.raises(ValueError, match="at least one"):
check_ntp_servers([" "])
@pytest.mark.parametrize(
"bad",
[
"ntp1.example.org iburst",
"-ntp.example.org",
"ntp1.example.org\nserver evil",
"ntp;reboot",
"$(id)",
"a" * 254,
],
)
def test_only_a_host_name_or_an_address(self, bad):
with pytest.raises(ValueError, match="not a host name or address"):
check_ntp_servers([bad])
def test_a_string_is_not_a_list_of_servers(self):
with pytest.raises(TypeError):
check_ntp_servers("ntp1.example.org") # type: ignore[arg-type]
+656
View File
@@ -0,0 +1,656 @@
"""Time sync on a POSIX host, over the command channel: chrony, systemd-timesyncd, timedatectl.
The reading is run by a real ``sh`` over a directory tree that stands in for the
host, with ``timedatectl``, ``systemctl`` and ``chronyc`` replaced by scripts that
keep a log -- so the shell itself is tested, not a hand-written copy of its output.
``sed`` and ``install`` are the real ones, so the files the writes leave are what a
host would get (NetOrk/netork#2361). Servers that DHCP brings are read apart and left
alone by a write (NetOrk/netork#2380).
"""
from __future__ import annotations
import os
import subprocess
from pathlib import Path
import pytest
from napalm_device_types import PosixTimeSyncMixin, UnknownTimezone, parse_time_sync
from napalm_device_types.channel import CommandResult
from napalm_device_types.time_sync_posix import (
chrony_synchronized,
parse_key_values,
timesyncd_link_servers,
timesyncd_servers,
)
# What the tools print, as recorded.
TIMEDATECTL = """\
Timezone=Europe/Berlin
LocalRTC=no
CanNTP=yes
NTP=yes
NTPSynchronized=yes
TimeUSec=Wed 2026-11-11 14:23:16 CET
RTCTimeUSec=Wed 2026-11-11 13:23:16 CET
"""
#: Ubuntu 24.04 without any server configured: only the compiled-in fallback.
TIMESYNC_FALLBACK = """\
FallbackNTPServers=ntp.ubuntu.com
ServerName=ntp.ubuntu.com
ServerAddress=91.189.91.157
RootDistanceMaxUSec=5s
PollIntervalMinUSec=32s
PollIntervalMaxUSec=34min 8s
PollIntervalUSec=34min 8s
NTPMessage={ Leap=0, Version=4, Mode=4, Stratum=2, Precision=-24, RootDelay=42.190ms, RootDispersion=1.190ms, Reference=84A36001, Ignored=no, PacketCount=119, Jitter=7.481ms }
Frequency=455471
"""
TIMESYNC_CONFIGURED = """\
LinkNTPServers=192.168.1.1
SystemNTPServers=ntp1.example.org ntp2.example.org
FallbackNTPServers=0.debian.pool.ntp.org 1.debian.pool.ntp.org
ServerName=ntp1.example.org
"""
CHRONYC_SYNCED = """\
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^- 162.159.200.123 3 6 377 41 +814us[ +814us] +/- 14ms
^* 192.168.1.1 2 6 377 40 -45us[ -63us] +/- 10ms
"""
CHRONYC_UNSYNCED = """\
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? 162.159.200.123 0 6 0 - +0ns[ +0ns] +/- 0ns
"""
#: Debian 12's chrony.conf, comments shortened, its directories under the test's root.
DEBIAN_CHRONY_CONF = """\
# Welcome to the chrony configuration file. See chrony.conf(5) for more
# information about usable directives.
# Include configuration files found in /etc/chrony/conf.d.
confdir {root}/etc/chrony/conf.d
# Use Debian vendor zone.
pool 2.debian.pool.ntp.org iburst
# Use time sources from DHCP.
sourcedir {root}/run/chrony-dhcp
# Use NTP sources found in /etc/chrony/sources.d.
sourcedir {root}/etc/chrony/sources.d
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
rtcsync
makestep 1 3
leapsectz right/UTC
"""
TIMEDATECTL_TOOL = """#!/bin/sh
echo "timedatectl $*" >> "$LOG"
case "$1" in
show) cat "$FIX/timedatectl" ;;
show-timesync) [ -f "$FIX/timesync" ] && cat "$FIX/timesync" ;;
set-timezone)
[ "$2" = "$REFUSE_ZONE" ] && { echo "Failed to set time zone: Invalid or not installed time zone '$2'" >&2; exit 1; }
sed -i "s#^Timezone=.*#Timezone=$2#" "$FIX/timedatectl" ;;
esac
"""
SYSTEMCTL_TOOL = """#!/bin/sh
echo "systemctl $*" >> "$LOG"
case "$1" in
is-active)
u=${2%.service}
case " $ACTIVE " in *" $u "*) echo active ;; *) echo inactive; exit 3 ;; esac ;;
restart)
[ -n "$FAIL_RESTART" ] && { echo "Job for $2 failed." >&2; exit 1; }
if [ "$2" = systemd-timesyncd.service ]; then
servers=$(sed -n 's/^NTP=\\(..*\\)$/\\1/p' "$DROPIN")
printf 'SystemNTPServers=%s\\n' "$servers" > "$FIX/timesync"
fi ;;
esac
"""
CHRONYC_TOOL = """#!/bin/sh
echo "chronyc $*" >> "$LOG"
case "$*" in
"-n sources") [ -f "$FIX/chronyc" ] && cat "$FIX/chronyc" ;;
"reload sources") exit "${RELOAD_EXIT:-0}" ;;
esac
"""
class Host(PosixTimeSyncMixin):
"""A host in a directory; every command is run by sh, root or not."""
def __init__(self, root: Path, *, active: str = "chrony") -> None:
self.root = root
etc = root / "etc"
self.CHRONY_CONFIGS = (str(etc / "chrony/chrony.conf"), str(etc / "chrony.conf")) # type: ignore[misc]
self.CHRONY_SOURCES_FILE = str(etc / "chrony/sources.d/netork.sources") # type: ignore[misc]
self.CHRONY_INCLUDE_FILE = str(etc / "chrony.d/netork.conf") # type: ignore[misc]
self.CHRONY_DHCP_DIRS = (str(root / "run/chrony-dhcp"),) # type: ignore[misc]
self.TIMESYNCD_DROPIN = str(etc / "systemd/timesyncd.conf.d/netork.conf") # type: ignore[misc]
self.NTPD_CONFIGS = (str(etc / "ntpsec/ntp.conf"), str(etc / "ntp.conf")) # type: ignore[misc]
self.OPENNTPD_CONFIGS = (str(etc / "openntpd/ntpd.conf"),) # type: ignore[misc]
self.TIME_ZONE_SH = ( # type: ignore[misc]
f"cat {etc}/timezone 2>/dev/null || readlink {etc}/localtime 2>/dev/null"
)
self.env = {
"PATH": f"{root / 'bin'}:{os.environ['PATH']}",
"LOG": str(root / "log"),
"FIX": str(root / "fix"),
"ACTIVE": active,
"DROPIN": self.TIMESYNCD_DROPIN,
}
self.commands: list[tuple[str, bool]] = []
def run_command(self, command, *, privileged=False, timeout=60, stdin=None): # type: ignore[no-untyped-def]
self.commands.append((command, privileged))
done = subprocess.run(
["sh", "-c", command],
input=stdin,
capture_output=True,
env={**os.environ, **self.env},
timeout=30,
)
return CommandResult(done.stdout.decode(), done.stderr.decode(), done.returncode)
def log(self) -> list[str]:
log = self.root / "log"
return log.read_text().splitlines() if log.exists() else []
def privileged(self) -> list[str]:
return [c for c, p in self.commands if p]
def _tool(root: Path, name: str, body: str) -> None:
path = root / "bin" / name
path.write_text(body)
path.chmod(0o755)
def _write(path: Path, text: str) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text)
@pytest.fixture
def root(tmp_path: Path) -> Path:
for d in ("bin", "fix", "etc"):
(tmp_path / d).mkdir()
(tmp_path / "fix/timedatectl").write_text(TIMEDATECTL)
for name, body in (
("timedatectl", TIMEDATECTL_TOOL),
("systemctl", SYSTEMCTL_TOOL),
("chronyc", CHRONYC_TOOL),
):
_tool(tmp_path, name, body)
return tmp_path
@pytest.fixture
def debian(root: Path) -> Path:
"""chrony as Debian 12 ships it, with a source in each place it reads."""
_write(root / "etc/chrony/chrony.conf", DEBIAN_CHRONY_CONF.format(root=root))
_write(root / "etc/chrony/conf.d/local.conf", "server ntp-conf.example.org iburst\n")
_write(root / "run/chrony-dhcp/eth0.sources", "server 192.168.1.1 iburst\n")
_write(root / "etc/chrony/sources.d/other.sources", "server ntp-other.example.org\n")
(root / "fix/chronyc").write_text(CHRONYC_SYNCED)
return root
def _no_timedatectl(root: Path) -> None:
"""A host without systemd: the command is not there (the build host's must not answer)."""
_tool(root, "timedatectl", '#!/bin/sh\necho "sh: timedatectl: not found" >&2\nexit 127\n')
def _text(path: Path) -> str:
return path.read_text()
class TestParsing:
def test_key_values(self):
values = parse_key_values(TIMEDATECTL)
assert values["Timezone"] == "Europe/Berlin"
assert values["NTPSynchronized"] == "yes"
def test_timesyncd_uses_the_fallback_only_when_nothing_else_is_known(self):
assert timesyncd_servers(parse_key_values(TIMESYNC_FALLBACK)) == ["ntp.ubuntu.com"]
# runtime, then system: the configured ones; a link's servers are DHCP's.
assert timesyncd_servers(parse_key_values(TIMESYNC_CONFIGURED)) == [
"ntp1.example.org",
"ntp2.example.org",
]
assert timesyncd_servers({}) == []
def test_timesyncd_does_not_fall_back_while_a_link_brings_servers(self):
values = {"LinkNTPServers": "192.168.1.1", "FallbackNTPServers": "ntp.ubuntu.com"}
assert timesyncd_servers(values) == []
assert timesyncd_link_servers(values) == ["192.168.1.1"]
def test_timesyncd_link_servers(self):
assert timesyncd_link_servers(parse_key_values(TIMESYNC_CONFIGURED)) == ["192.168.1.1"]
assert timesyncd_link_servers({"LinkNTPServers": "10.0.0.1 10.0.0.1 10.0.0.2"}) == [
"10.0.0.1",
"10.0.0.2",
]
assert timesyncd_link_servers(parse_key_values(TIMESYNC_FALLBACK)) == []
def test_chrony_is_synchronised_while_it_has_selected_a_source(self):
assert chrony_synchronized(CHRONYC_SYNCED) is True
assert chrony_synchronized(CHRONYC_UNSYNCED) is False
assert chrony_synchronized("") is None
def test_chrony_without_a_main_configuration_has_no_sources(self):
reading = parse_time_sync("@@ts active\nchrony chrony active\n@@ts end\n")
assert (reading["ntp_servers"], reading["dhcp_ntp_servers"]) == ([], [])
def test_the_dhcp_directory_is_debian_s_and_red_hat_s_by_default(self):
reading = parse_time_sync(
"@@ts active\nchrony chrony active\n"
"@@ts chrony 0 main /etc/chrony/chrony.conf\nsourcedir /run/chrony-dhcp\n"
"@@ts chrony 1 sourcedir /run/chrony-dhcp/eth0.sources\nserver 10.0.0.1 iburst\n"
"@@ts end\n"
)
assert (reading["ntp_servers"], reading["dhcp_ntp_servers"]) == ([], ["10.0.0.1"])
def test_a_reading_cut_short_raises(self):
with pytest.raises(ValueError, match="cut short"):
parse_time_sync("@@ts timedatectl\nTimezone=UTC\n")
class TestReading:
def test_chrony_names_its_sources_in_the_order_it_reads_them(self, debian: Path):
host = Host(debian)
assert host.get_time_sync() == {
"ntp_servers": [
"ntp-conf.example.org",
"2.debian.pool.ntp.org",
"ntp-other.example.org",
],
"timezone": "Europe/Berlin",
"synchronized": True,
"daemon": "chrony",
"dhcp_ntp_servers": ["192.168.1.1"],
}
def test_dhcp_servers_in_the_order_chrony_reads_them(self, debian: Path):
_write(debian / "run/chrony-dhcp/wlan0.sources", "server 10.9.0.1 iburst\n")
_write(debian / "run/chrony-dhcp/eth0.sources", "server 192.168.1.1\npool p.example\n")
assert Host(debian).get_time_sync()["dhcp_ntp_servers"] == [
"192.168.1.1",
"p.example",
"10.9.0.1",
]
def test_a_dhcp_directory_chrony_does_not_read_brings_nothing(self, debian: Path):
# What 4.35.0 left behind: DHCP's sourcedir commented out.
conf = debian / "etc/chrony/chrony.conf"
conf.write_text(_text(conf).replace("\nsourcedir ", "\n#napalm-time-sync: sourcedir ", 1))
reading = Host(debian).get_time_sync()
assert reading["dhcp_ntp_servers"] == []
assert "192.168.1.1" not in reading["ntp_servers"]
def test_without_a_dhcp_directory_dhcp_brings_nothing(self, root: Path):
_write(root / "etc/chrony.conf", "pool 2.rhel.pool.ntp.org iburst\n")
assert Host(root, active="chronyd").get_time_sync()["dhcp_ntp_servers"] == []
def test_the_reading_is_one_round_trip_without_root(self, debian: Path):
host = Host(debian)
host.get_time_sync()
assert [p for _, p in host.commands] == [False]
def test_chrony_says_whether_it_is_synchronised_not_the_kernel(self, debian: Path):
(debian / "fix/chronyc").write_text(CHRONYC_UNSYNCED)
assert Host(debian).get_time_sync()["synchronized"] is False
def test_without_an_answer_from_chronyc_the_kernel_says(self, debian: Path):
(debian / "fix/chronyc").unlink()
assert Host(debian).get_time_sync()["synchronized"] is True
def test_commented_and_other_directives_are_no_sources(self, root: Path):
_write(
root / "etc/chrony.conf",
"#pool 2.rhel.pool.ntp.org iburst\n! server old.example.org\n"
" server ntp1.example.org iburst\nrefclock PPS /dev/pps0\npeer peer.example.org\n",
)
reading = Host(root, active="chronyd").get_time_sync()
assert reading["ntp_servers"] == ["ntp1.example.org", "peer.example.org"]
assert reading["daemon"] == "chrony"
def test_an_include_pattern_is_followed(self, root: Path):
_write(root / "etc/chrony.conf", f"include {root}/etc/chrony.d/*.conf\nmakestep 1 3\n")
_write(root / "etc/chrony.d/pool.conf", "pool suse.pool.ntp.org iburst\n")
assert Host(root).get_time_sync()["ntp_servers"] == ["suse.pool.ntp.org"]
def test_timesyncd(self, root: Path):
(root / "fix/timesync").write_text(TIMESYNC_CONFIGURED)
reading = Host(root, active="systemd-timesyncd").get_time_sync()
assert reading["daemon"] == "systemd-timesyncd"
assert reading["ntp_servers"] == ["ntp1.example.org", "ntp2.example.org"]
assert reading["dhcp_ntp_servers"] == ["192.168.1.1"]
assert reading["synchronized"] is True
def test_ntpd(self, root: Path):
_write(
root / "etc/ntpsec/ntp.conf",
"driftfile /var/lib/ntpsec/ntp.drift\npool 0.debian.pool.ntp.org iburst\n"
"server 127.127.1.0\nserver ntp1.example.org iburst\nrestrict default kod\n",
)
reading = Host(root, active="ntpsec").get_time_sync()
assert reading["daemon"] == "ntpd"
assert reading["ntp_servers"] == ["0.debian.pool.ntp.org", "ntp1.example.org"]
# Debian's DHCP hook gives ntpd a configuration of its own: not read, so unknown.
assert reading["dhcp_ntp_servers"] is None
def test_openntpd(self, root: Path):
_write(
root / "etc/openntpd/ntpd.conf",
'servers pool.ntp.org\nserver ntp1.example.org\nconstraints from "https://www.google.com"\n',
)
reading = Host(root, active="openntpd").get_time_sync()
assert reading["daemon"] == "openntpd"
assert reading["ntp_servers"] == ["pool.ntp.org", "ntp1.example.org"]
assert reading["dhcp_ntp_servers"] is None
def test_no_client_runs(self, root: Path):
(root / "fix/timedatectl").write_text(
TIMEDATECTL.replace("NTPSynchronized=yes", "NTPSynchronized=no")
)
assert Host(root, active="").get_time_sync() == {
"ntp_servers": [],
"timezone": "Europe/Berlin",
"synchronized": False,
"daemon": None,
"dhcp_ntp_servers": [],
}
def test_without_timedatectl_the_zone_comes_from_the_files(self, root: Path):
_no_timedatectl(root)
(root / "etc/localtime").symlink_to("/usr/share/zoneinfo/America/Sao_Paulo")
reading = Host(root, active="").get_time_sync()
assert reading["timezone"] == "America/Sao_Paulo"
assert reading["synchronized"] is None
def test_etc_timezone_wins_over_the_link(self, root: Path):
_no_timedatectl(root)
(root / "etc/timezone").write_text("Australia/Lord_Howe\n")
assert Host(root, active="").get_time_sync()["timezone"] == "Australia/Lord_Howe"
def test_a_copied_localtime_has_no_name(self, root: Path):
_no_timedatectl(root)
assert Host(root, active="").get_time_sync()["timezone"] is None
class TestChronyServers:
def test_exactly_these_servers(self, debian: Path):
host = Host(debian)
result = host.configure_time_sync(["ntp1.example.org", "10.0.0.1", "2.de.pool.ntp.org"])
assert result["success"] is True, result["output"]
assert _text(debian / "etc/chrony/sources.d/netork.sources") == (
"server ntp1.example.org iburst\nserver 10.0.0.1 iburst\npool 2.de.pool.ntp.org iburst\n"
)
conf = _text(debian / "etc/chrony/chrony.conf")
assert "\n#napalm-time-sync: pool 2.debian.pool.ntp.org iburst\n" in conf
# DHCP option 42 counts: chrony keeps reading what DHCP brings.
assert f"\nsourcedir {debian}/run/chrony-dhcp\n" in conf
assert f"\nsourcedir {debian}/etc/chrony/sources.d\n" in conf
assert f"\nconfdir {debian}/etc/chrony/conf.d\n" in conf
assert _text(debian / "etc/chrony/conf.d/local.conf") == (
"#napalm-time-sync: server ntp-conf.example.org iburst\n"
)
assert _text(debian / "etc/chrony/sources.d/other.sources") == (
"#napalm-time-sync: server ntp-other.example.org\n"
)
# DHCP's own file is left alone: the DHCP client rewrites it on every lease.
assert _text(debian / "run/chrony-dhcp/eth0.sources") == "server 192.168.1.1 iburst\n"
# The main configuration changed: chronyd reads it only when it starts.
assert "systemctl restart chrony.service" in host.log()
reading = host.get_time_sync()
assert reading["ntp_servers"] == ["ntp1.example.org", "10.0.0.1", "2.de.pool.ntp.org"]
assert reading["dhcp_ntp_servers"] == ["192.168.1.1"]
def test_what_4_35_0_took_from_dhcp_is_given_back(self, debian: Path):
conf = debian / "etc/chrony/chrony.conf"
Host(debian).configure_time_sync(["ntp1.example.org"])
# A host 4.35.0 aligned: DHCP's sourcedir commented out behind the mark.
dhcp = f"sourcedir {debian}/run/chrony-dhcp"
conf.write_text(_text(conf).replace(f"\n{dhcp}\n", f"\n#napalm-time-sync: {dhcp}\n"))
assert Host(debian).get_time_sync()["dhcp_ntp_servers"] == []
(debian / "log").unlink()
host = Host(debian)
result = host.configure_time_sync(["ntp1.example.org"])
assert result["success"] is True, result["output"]
assert f"\n{dhcp}\n" in _text(conf)
assert "#napalm-time-sync: sourcedir" not in _text(conf)
# The main configuration changed again: a restart, not a reload.
assert "systemctl restart chrony.service" in host.log()
assert host.get_time_sync()["dhcp_ntp_servers"] == ["192.168.1.1"]
def test_a_foreign_sourcedir_4_35_0_took_out_stays_out(self, debian: Path):
conf = debian / "etc/chrony/chrony.conf"
foreign = f"#napalm-time-sync: sourcedir {debian}/etc/chrony/extra.d"
conf.write_text(_text(conf) + foreign + "\n")
_write(debian / "etc/chrony/extra.d/x.sources", "server ntp-extra.example.org\n")
assert Host(debian).configure_time_sync(["ntp1.example.org"])["success"] is True
assert f"\n{foreign}\n" in _text(conf)
def test_a_foreign_sourcedir_stays_and_its_sources_stop_counting(self, debian: Path):
conf = debian / "etc/chrony/chrony.conf"
conf.write_text(_text(conf) + f"sourcedir {debian}/etc/chrony/extra.d\n")
_write(debian / "etc/chrony/extra.d/x.sources", "server ntp-extra.example.org\n")
host = Host(debian)
result = host.configure_time_sync(["ntp1.example.org"])
assert result["success"] is True, result["output"]
assert f"\nsourcedir {debian}/etc/chrony/extra.d\n" in _text(conf)
assert _text(debian / "etc/chrony/extra.d/x.sources") == (
"#napalm-time-sync: server ntp-extra.example.org\n"
)
assert host.get_time_sync()["ntp_servers"] == ["ntp1.example.org"]
def test_dhcp_is_kept_where_the_servers_go_to_a_confdir(self, root: Path):
_write(
root / "etc/chrony/chrony.conf",
f"confdir {root}/etc/chrony/conf.d\npool a.pool.ntp.org\n"
f"sourcedir {root}/run/chrony-dhcp\n",
)
_write(root / "run/chrony-dhcp/eth0.sources", "server 192.168.1.1 iburst\n")
host = Host(root)
assert host.configure_time_sync(["ntp1.example.org"])["success"] is True
assert f"\nsourcedir {root}/run/chrony-dhcp\n" in _text(root / "etc/chrony/chrony.conf")
reading = host.get_time_sync()
assert (reading["ntp_servers"], reading["dhcp_ntp_servers"]) == (
["ntp1.example.org"],
["192.168.1.1"],
)
def test_a_second_run_only_reloads_the_sources(self, debian: Path):
Host(debian).configure_time_sync(["ntp1.example.org"])
conf = _text(debian / "etc/chrony/chrony.conf")
(debian / "log").unlink()
host = Host(debian)
result = host.configure_time_sync(["ntp2.example.org", "ntp1.example.org"])
assert result["success"] is True, result["output"]
assert _text(debian / "etc/chrony/chrony.conf") == conf
assert "chronyc reload sources" in host.log()
assert not any(line.startswith("systemctl restart") for line in host.log())
assert _text(debian / "etc/chrony/sources.d/netork.sources") == (
"server ntp2.example.org iburst\nserver ntp1.example.org iburst\n"
)
def test_a_chronyd_that_cannot_reload_sources_is_restarted(self, debian: Path):
Host(debian).configure_time_sync(["ntp1.example.org"])
host = Host(debian)
host.env["RELOAD_EXIT"] = "1"
assert host.configure_time_sync(["ntp2.example.org"])["success"] is True
log = host.log()
reload = log.index("chronyc reload sources")
assert log[reload + 1] == "systemctl restart chrony.service"
def test_writes_are_root_s_and_root_reads_no_stdin(self, debian: Path):
host = Host(debian)
host.configure_time_sync(["ntp1.example.org"])
staged = [c for c, p in host.commands if not p and "mktemp" in c]
assert len(staged) == 1
assert host.privileged()
assert all("mktemp" not in c for c in host.privileged())
def test_rhel_without_a_directory_gets_an_include(self, root: Path):
_write(
root / "etc/chrony.conf",
"pool 2.rhel.pool.ntp.org iburst\ndriftfile /var/lib/chrony/drift\n",
)
host = Host(root, active="chronyd")
result = host.configure_time_sync(["ntp1.example.org"])
assert result["success"] is True, result["output"]
include = root / "etc/chrony.d/netork.conf"
assert _text(include) == "server ntp1.example.org iburst\n"
assert _text(root / "etc/chrony.conf") == (
"#napalm-time-sync: pool 2.rhel.pool.ntp.org iburst\n"
"driftfile /var/lib/chrony/drift\n"
f"# napalm-time-sync: the time sources\ninclude {include}\n"
)
assert "systemctl restart chronyd.service" in host.log()
def test_an_include_that_already_takes_the_file_is_used(self, root: Path):
_write(root / "etc/chrony.conf", f"include {root}/etc/chrony.d/*.conf\n")
_write(root / "etc/chrony.d/pool.conf", "pool suse.pool.ntp.org iburst\n")
host = Host(root)
assert host.configure_time_sync(["ntp1.example.org"])["success"] is True
assert _text(root / "etc/chrony.conf") == f"include {root}/etc/chrony.d/*.conf\n"
assert (
_text(root / "etc/chrony.d/pool.conf")
== "#napalm-time-sync: pool suse.pool.ntp.org iburst\n"
)
assert _text(root / "etc/chrony.d/netork.conf") == "server ntp1.example.org iburst\n"
def test_a_confdir_takes_the_file_where_sources_d_is_not_read(self, root: Path):
_write(
root / "etc/chrony/chrony.conf",
f"confdir {root}/etc/chrony/conf.d\npool a.pool.ntp.org\n",
)
host = Host(root)
assert host.configure_time_sync(["ntp1.example.org"])["success"] is True
assert _text(root / "etc/chrony/conf.d/netork.conf") == "server ntp1.example.org iburst\n"
assert not (root / "etc/chrony/sources.d/netork.sources").exists()
def test_a_failed_restart_fails_the_step(self, debian: Path):
host = Host(debian)
host.env["FAIL_RESTART"] = "1"
result = host.configure_time_sync(["ntp1.example.org"])
assert result["success"] is False
assert "Job for chrony.service failed." in result["steps"][0]["output"]
def test_a_file_that_cannot_be_staged_is_not_installed(self, debian: Path):
_tool(debian, "mktemp", "#!/bin/sh\necho 'mktemp: failed' >&2\nexit 1\n")
host = Host(debian)
result = host.configure_time_sync(["ntp1.example.org"])
assert result["success"] is False
assert "cannot stage" in result["steps"][0]["output"]
assert host.privileged() == []
def test_no_configuration_is_a_failed_step(self, root: Path):
result = Host(root).configure_time_sync(["ntp1.example.org"])
assert result["success"] is False
assert "chrony" in result["steps"][0]["output"]
class TestTimesyncdServers:
def test_a_drop_in_with_exactly_these_servers(self, root: Path):
(root / "fix/timesync").write_text(TIMESYNC_FALLBACK)
host = Host(root, active="systemd-timesyncd")
result = host.configure_time_sync(["ntp1.example.org", "10.0.0.1"])
assert result["success"] is True, result["output"]
assert _text(root / "etc/systemd/timesyncd.conf.d/netork.conf").endswith(
"[Time]\nNTP=\nNTP=ntp1.example.org 10.0.0.1\nFallbackNTP=\n"
)
assert "systemctl restart systemd-timesyncd.service" in host.log()
assert host.get_time_sync()["ntp_servers"] == ["ntp1.example.org", "10.0.0.1"]
def test_servers_from_dhcp_stay_and_count(self, root: Path):
(root / "fix/timesync").write_text(TIMESYNC_CONFIGURED)
tool = (root / "bin/systemctl").read_text()
(root / "bin/systemctl").write_text(
tool.replace(
"printf 'SystemNTPServers", "printf 'LinkNTPServers=192.168.1.1\\nSystemNTPServers"
)
)
host = Host(root, active="systemd-timesyncd")
result = host.configure_time_sync(["ntp1.example.org"])
assert result["success"] is True, result["output"]
reading = host.get_time_sync()
assert (reading["ntp_servers"], reading["dhcp_ntp_servers"]) == (
["ntp1.example.org"],
["192.168.1.1"],
)
class TestOtherClients:
@pytest.mark.parametrize("active", ["ntpsec", "openntpd", ""])
def test_only_chrony_and_timesyncd_are_written(self, root: Path, active: str):
host = Host(root, active=active)
result = host.configure_time_sync(["ntp1.example.org"])
assert result["success"] is False
assert "chrony and systemd-timesyncd" in result["steps"][0]["output"]
assert host.privileged() == []
class TestTimeZone:
def test_timedatectl_sets_it(self, root: Path):
host = Host(root, active="")
result = host.configure_time_sync(timezone="America/Sao_Paulo")
assert result["success"] is True, result["output"]
assert host.privileged() == ["timedatectl set-timezone America/Sao_Paulo"]
assert host.get_time_sync()["timezone"] == "America/Sao_Paulo"
def test_a_zone_the_host_does_not_know_fails_the_step(self, root: Path):
host = Host(root, active="")
host.env["REFUSE_ZONE"] = "Europe/Kyiv"
result = host.configure_time_sync(timezone="Europe/Kyiv")
assert result["success"] is False
assert "Invalid or not installed time zone" in result["steps"][0]["output"]
def test_an_unknown_zone_is_refused_before_anything_is_written(self, debian: Path):
host = Host(debian)
with pytest.raises(UnknownTimezone):
host.configure_time_sync(["ntp1.example.org"], "Mars/Olympus_Mons")
assert host.commands == []
def test_both_at_once(self, debian: Path):
host = Host(debian)
result = host.configure_time_sync(["ntp1.example.org"], "UTC")
assert result["success"] is True, result["output"]
assert [s["step"] for s in result["steps"]] == ["ntp_servers", "timezone", "verify"]
reading = host.get_time_sync()
assert (reading["ntp_servers"], reading["timezone"]) == (["ntp1.example.org"], "UTC")
def test_it_writes_both_values():
assert frozenset({"ntp_servers", "timezone"}) == PosixTimeSyncMixin.time_sync_writes
+285
View File
@@ -0,0 +1,285 @@
"""TLS certificates: trust on first use by fingerprint, a changed one refused before any login.
The layer above keeps the store (NetOrk/netork#1167, NAPALM/napalm-hpe-aruba-procurve#17).
A driver gets the fingerprint it should expect through ``optional_args["tls_fingerprints"]``
(by port) and reports every certificate it sees through
``optional_args["tls_certificate_observer"]``. The handshake tests run real TLS against an
in-process server on a socket pair: what matters is the certificate the server presents in
a real handshake, which a fake could not show.
"""
from __future__ import annotations
import datetime
import hashlib
import socket
import ssl
import threading
import pytest
import requests
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.x509.oid import NameOID
from napalm.base.exceptions import ConnectionException
from napalm_device_types.tls_certificates import (
KNOWN_FINGERPRINTS_ARG,
OBSERVER_ARG,
CertificateCheck,
TlsCertificateMismatch,
fetch_certificate,
fingerprint,
normalise_fingerprint,
)
DER = b"not really a certificate, but bytes are bytes"
DER_HEX = hashlib.sha256(DER).hexdigest().upper()
DER_FP = "SHA256:" + ":".join(DER_HEX[i : i + 2] for i in range(0, 64, 2))
OTHER_FP = "SHA256:" + ":".join(["AB"] * 32)
def _self_signed(tmp_path, name: str = "switch") -> tuple[bytes, str, str]:
"""A self-signed certificate, as a ProCurve ships one: (DER, cert file, key file)."""
key = ec.generate_private_key(ec.SECP256R1())
subject = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, name)])
now = datetime.datetime.now(datetime.UTC)
cert = (
x509.CertificateBuilder()
.subject_name(subject)
.issuer_name(subject)
.public_key(key.public_key())
.serial_number(x509.random_serial_number())
.not_valid_before(now - datetime.timedelta(days=1))
.not_valid_after(now + datetime.timedelta(days=1))
.sign(key, hashes.SHA256())
)
cert_file, key_file = tmp_path / f"{name}.crt", tmp_path / f"{name}.key"
cert_file.write_bytes(cert.public_bytes(serialization.Encoding.PEM))
key_file.write_bytes(
key.private_bytes(
serialization.Encoding.PEM,
serialization.PrivateFormat.PKCS8,
serialization.NoEncryption(),
)
)
return cert.public_bytes(serialization.Encoding.DER), str(cert_file), str(key_file)
class _TlsServer:
"""A TLS server on one end of a socket pair; it handshakes and then waits."""
def __init__(self, cert_file: str, key_file: str) -> None:
self.client_sock, server_sock = socket.socketpair()
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.load_cert_chain(cert_file, key_file)
self._server_sock = server_sock
self._context = context
self.received = b""
self._thread = threading.Thread(target=self._serve, daemon=True)
self._thread.start()
def _serve(self) -> None:
try:
tls = self._context.wrap_socket(self._server_sock, server_side=True)
self.received = tls.recv(1024)
except (OSError, ssl.SSLError): # the client hung up after the handshake
pass
def connect(self, address, timeout=None):
return self.client_sock
def close(self) -> None:
self.client_sock.close()
self._server_sock.close()
self._thread.join(timeout=5)
@pytest.fixture
def certificate(tmp_path):
return _self_signed(tmp_path)
@pytest.fixture
def server(certificate):
_der, cert_file, key_file = certificate
s = _TlsServer(cert_file, key_file)
yield s
s.close()
def _check(expected: str | None, seen: list, port: int = 443) -> CertificateCheck:
args: dict = {OBSERVER_ARG: seen.append}
if expected is not None:
args[KNOWN_FINGERPRINTS_ARG] = {str(port): expected}
return CertificateCheck(args, "10.0.0.5", port)
class TestFingerprint:
def test_is_sha256_of_the_der_certificate_in_colon_hex(self):
assert fingerprint(DER) == DER_FP
@pytest.mark.parametrize(
"value",
[
DER_FP,
DER_FP.lower(),
DER_HEX,
DER_HEX.lower(),
DER_FP.removeprefix("SHA256:"),
" ".join(DER_HEX[i : i + 2] for i in range(0, 64, 2)),
"sha256:" + DER_HEX,
],
)
def test_a_pin_may_be_written_as_openssl_or_a_browser_shows_it(self, value):
assert normalise_fingerprint(value) == DER_FP
@pytest.mark.parametrize(
"value", ["", "AB:CD", "SHA1:" + DER_HEX, "ZZ" * 32, DER_HEX + "AB", 42]
)
def test_anything_else_is_refused(self, value):
with pytest.raises(ValueError):
normalise_fingerprint(value)
class TestKnownFingerprints:
def test_the_pin_is_the_one_for_the_port_connected_to(self):
args = {KNOWN_FINGERPRINTS_ARG: {"443": DER_HEX}}
assert CertificateCheck(args, "10.0.0.5", 443).expected == DER_FP
assert CertificateCheck(args, "10.0.0.5", 8443).expected is None
def test_a_port_may_be_given_as_a_number(self):
args = {KNOWN_FINGERPRINTS_ARG: {8443: DER_FP}}
assert CertificateCheck(args, "10.0.0.5", 8443).expected == DER_FP
def test_a_malformed_mapping_is_refused(self):
with pytest.raises(ValueError):
CertificateCheck({KNOWN_FINGERPRINTS_ARG: [DER_FP]}, "10.0.0.5", 443)
def test_without_optional_args_it_is_first_use(self):
assert CertificateCheck(None, "10.0.0.5", 443).expected is None
class TestVerify:
def test_first_use_accepts_and_reports_only_once_observed(self):
seen: list = []
check = _check(None, seen)
observation = check.verify(DER)
assert observation.accepted is True
assert seen == []
check.observe()
assert seen == [
{
"host": "10.0.0.5",
"port": 443,
"seen": {"fingerprint": DER_FP},
"expected": None,
"accepted": True,
}
]
def test_the_pinned_certificate_is_accepted(self):
seen: list = []
check = _check(DER_HEX.lower(), seen)
assert check.verify(DER).accepted is True
check.observe()
assert seen[0]["expected"] == {"fingerprint": DER_FP}
assert seen[0]["accepted"] is True
def test_another_certificate_is_refused_reported_and_named(self):
seen: list = []
with pytest.raises(TlsCertificateMismatch) as caught:
_check(OTHER_FP, seen).verify(DER)
assert caught.value.observation.seen == DER_FP
assert caught.value.observation.expected == OTHER_FP
assert seen == [
{
"host": "10.0.0.5",
"port": 443,
"seen": {"fingerprint": DER_FP},
"expected": {"fingerprint": OTHER_FP},
"accepted": False,
}
]
def test_the_mismatch_is_a_connection_failure_that_names_both(self):
with pytest.raises(ConnectionException) as caught:
_check(OTHER_FP, []).verify(DER)
message = str(caught.value)
assert "10.0.0.5:443" in message
assert DER_FP in message
assert OTHER_FP in message
def test_a_failing_observer_does_not_break_the_connection(self):
def boom(_observation):
raise RuntimeError("store down")
check = CertificateCheck({OBSERVER_ARG: boom}, "10.0.0.5", 443)
check.verify(DER)
check.observe()
def test_observe_before_verify_is_a_mistake(self):
with pytest.raises(RuntimeError):
_check(None, []).observe()
class TestHandshake:
def test_fetch_returns_the_certificate_the_server_presents(self, server, certificate):
der, _cert, _key = certificate
assert fetch_certificate("10.0.0.5", 443, 5, connect=server.connect) == der
def test_verify_server_refuses_another_certificate_before_anything_is_sent(
self, server, tmp_path
):
other_der, _c, _k = _self_signed(tmp_path, "impostor")
seen: list = []
check = _check(fingerprint(other_der), seen)
with pytest.raises(TlsCertificateMismatch):
check.verify_server(5, connect=server.connect)
server.close()
assert server.received == b""
assert seen[0]["accepted"] is False
def test_verify_server_accepts_on_first_use(self, server, certificate):
der, _cert, _key = certificate
observation = _check(None, []).verify_server(5, connect=server.connect)
assert observation.seen == fingerprint(der)
class TestRequests:
def test_pin_makes_every_connection_present_the_verified_certificate(self):
check = _check(None, [])
check.verify(DER)
session = requests.Session()
check.pin(session)
adapter = session.get_adapter("https://10.0.0.5/rest/v7/")
assert adapter.poolmanager.connection_pool_kw["assert_fingerprint"] == DER_HEX.lower()
proxy = adapter.proxy_manager_for("http://proxy.example:3128")
assert proxy.connection_pool_kw["assert_fingerprint"] == DER_HEX.lower()
def test_pin_before_verify_is_a_mistake(self):
with pytest.raises(RuntimeError):
_check(None, []).pin(requests.Session())
def test_a_changed_certificate_on_a_pinned_session_is_a_mismatch(self):
from urllib3.util.ssl_ import assert_fingerprint
seen: list = []
check = _check(DER_FP, seen)
check.verify(DER)
impostor = b"another certificate"
with pytest.raises(TlsCertificateMismatch) as caught, check.refusing():
try:
assert_fingerprint(impostor, DER_HEX)
except Exception as exc:
raise requests.exceptions.SSLError(exc) from exc
assert caught.value.observation.seen == fingerprint(impostor)
assert caught.value.observation.expected == DER_FP
assert seen[-1]["accepted"] is False
def test_refusing_lets_other_errors_through(self):
check = _check(None, [])
check.verify(DER)
with pytest.raises(requests.exceptions.ConnectionError), check.refusing():
raise requests.exceptions.ConnectionError("no route to host")
+2
View File
@@ -4,6 +4,7 @@ import pytest
from napalm_device_types import (
AccessPointDriver,
BackupServerDriver,
FirewallDriver,
HypervisorDriver,
ManagementControllerDriver,
@@ -15,6 +16,7 @@ from napalm_device_types import (
BASE_CLASSES = [
(AccessPointDriver, "Access Point"),
(BackupServerDriver, "Backup Server"),
(FirewallDriver, "Firewall"),
(HypervisorDriver, "Hypervisor"),
(ManagementControllerDriver, "Management Controller"),
+65
View File
@@ -0,0 +1,65 @@
"""Moving a guest to another node of its cluster, and a node's HA maintenance mode.
``get_vm_migration_preflight`` says where a guest could go and what holds it
back, ``migrate_vm`` moves it and waits, ``set_node_maintenance`` switches a
node's HA maintenance mode. Declared only, under ``TYPE_CHECKING``: netOrk
probes each with ``hasattr`` and hides what a driver lacks (VMware has none of
them yet).
"""
from __future__ import annotations
import ast
import inspect
from typing import get_type_hints
import pytest
import napalm_device_types.hypervisor as hypervisor_module
from napalm_device_types import HypervisorDriver, MigrationPreflightDict
def _declared(name: str) -> ast.FunctionDef:
tree = ast.parse(inspect.getsource(hypervisor_module))
for node in ast.walk(tree):
if isinstance(node, ast.FunctionDef) and node.name == name:
return node
raise AssertionError(f"HypervisorDriver does not declare {name}()")
def _signature(name: str) -> str:
fn = _declared(name)
return f"({ast.unparse(fn.args)}) -> {ast.unparse(fn.returns) if fn.returns else None}"
class TestDeclarations:
@pytest.mark.parametrize(
"method", ["get_vm_migration_preflight", "migrate_vm", "set_node_maintenance"]
)
def test_absent_until_a_driver_implements_it(self, method):
assert not hasattr(HypervisorDriver, method)
def test_preflight(self):
assert _signature("get_vm_migration_preflight") == (
"(self, name: str) -> MigrationPreflightDict"
)
def test_migrate(self):
assert _signature("migrate_vm") == (
"(self, name: str, target_node: str, online: bool=True, "
"target_storage: str | None=None, timeout: int=3600) -> None"
)
def test_maintenance(self):
assert _signature("set_node_maintenance") == "(self, node: str, enabled: bool) -> None"
class TestPreflightShape:
def test_keys(self):
assert get_type_hints(MigrationPreflightDict) == {
"running": bool,
"allowed_nodes": list[str],
"refused_nodes": dict[str, str],
"local_disks": list[str],
"local_resources": list[str],
}
+53
View File
@@ -0,0 +1,53 @@
"""A provisioning that failed after the hypervisor gave the VM its id says
which VM it left behind, so the caller can remove it (NAPALM/napalm-proxmox#28)."""
from __future__ import annotations
import pickle
import pytest
from napalm_device_types import VMProvisionIncomplete
from napalm_device_types.provisioning import VMProvisionIncomplete as from_module
def _raise_from(cause: Exception) -> VMProvisionIncomplete:
try:
try:
raise cause
except Exception as exc:
raise VMProvisionIncomplete(str(exc), vmid="101", node="pve1") from exc
except VMProvisionIncomplete as caught:
return caught
def test_it_is_exported_from_the_package_root():
assert VMProvisionIncomplete is from_module
def test_it_names_the_vm_left_behind():
caught = _raise_from(RuntimeError("disk import failed"))
assert (caught.vmid, caught.node) == ("101", "pve1")
def test_it_keeps_the_failure_and_its_words():
cause = RuntimeError("disk import failed")
caught = _raise_from(cause)
assert caught.__cause__ is cause
assert str(caught) == "disk import failed"
def test_a_caller_catching_runtime_error_still_catches_it():
# The contract has create_vm_from_cloud_init raise RuntimeError on failure.
with pytest.raises(RuntimeError):
raise VMProvisionIncomplete("start failed", vmid="7", node="n")
def test_vmid_and_node_are_keyword_only():
with pytest.raises(TypeError):
VMProvisionIncomplete("failed", "101", "pve1") # type: ignore[misc]
def test_it_survives_pickling_as_a_task_result_would():
caught = pickle.loads(pickle.dumps(VMProvisionIncomplete("failed", vmid="101", node="pve1")))
assert (str(caught), caught.vmid, caught.node) == ("failed", "101", "pve1")
+28
View File
@@ -0,0 +1,28 @@
"""begin_stop_vm: ask for a VM's clean shutdown and return at once.
``stop_vm`` blocks until the VM is stopped, with a limit of the driver's own.
A caller that keeps its waits as timers (netOrk's graph workflows,
NetOrk/netork#1636) asks with ``begin_stop_vm`` and reads ``get_vm_status``
until the VM is stopped. A driver mixes it in when it can; no role demands it.
"""
from __future__ import annotations
import inspect
from napalm_device_types import HypervisorDriver, VmStopRequestMixin
def test_no_role_demands_it():
assert not issubclass(HypervisorDriver, VmStopRequestMixin)
def test_declared_not_implemented():
assert not hasattr(VmStopRequestMixin, "begin_stop_vm")
def test_declaration_documents_the_contract():
source = inspect.getsource(VmStopRequestMixin)
assert "def begin_stop_vm(self, name: str) -> None" in source
assert "ValueError" in source
assert "RuntimeError" in source