Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e729b935a4 | ||
|
|
f654076560 | ||
|
|
2dea517e89 | ||
|
|
1a7885d22e | ||
|
|
71588613fb | ||
|
|
93ba335798 | ||
|
|
eb1964c4b3 | ||
|
|
3622d0f825 | ||
|
|
caf7ef4cc8 | ||
|
|
3422b7d030 | ||
|
|
f1940aa918 | ||
|
|
5217c6e333 | ||
|
|
7051199589 | ||
|
|
7c0e4fffc5 | ||
|
|
cb0f1cc05c |
No files matched your search
@@ -7,6 +7,70 @@ recorded here; their history is the git log.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- `AptReleaseMixin.get_apt_source_releases()`: the host's release from `/etc/os-release`
|
||||
(`ID`, `ID_LIKE`, `VERSION_CODENAME`, `UBUNTU_CODENAME`) and every enabled source apt
|
||||
reads, one-line and deb822, with its file, line, types, URI, suites and components
|
||||
(NetOrk/netork#2614). `AptProxyMixin` brings it, so every apt driver has it.
|
||||
- `AptReleaseMixin.set_apt_source_release(path, release, codename)`: every suite of
|
||||
*release* in one of apt's source files moved to *codename* (`focal-security` →
|
||||
`noble-security`), once each new suite's `InRelease` or `Release` is fetched with apt's
|
||||
downloader; nothing is written where the vendor has none.
|
||||
- `parse_apt_releases`, `rewrite_release`, `suite_release`, `APT_RELEASES_COMMAND`.
|
||||
|
||||
## [4.41.0] - 2026-11-17
|
||||
|
||||
### Added
|
||||
- `FirmwareUpgradeMixin.start_firmware_upgrade(version)`: start installing the firmware a
|
||||
driver offers; the device runs it by itself and restarts at its end, and
|
||||
`UpdateMixin.get_update_progress` follows it until then (NetOrk/netork#2615, OpenWrt's
|
||||
sysupgrade through owut).
|
||||
- `FirmwareOfferDict.installable`: whether the driver can install `latest_version` itself
|
||||
now, False while the device lacks what that takes; and
|
||||
`FirmwareOfferDict.latest_major_version`, the newest of a later release line, shown and
|
||||
never crossed to by the driver.
|
||||
|
||||
## [4.40.0] - 2026-11-17
|
||||
|
||||
### Added
|
||||
- `ZabbixAgentMixin.configure_zabbix_agent(settings)`: a native Zabbix agent, for a host
|
||||
without a container engine (NetOrk/netork#2571). Install (`ensure_package` with
|
||||
`ZABBIX_AGENT_PACKAGE`, and where the host's own packages have none and
|
||||
`settings["release"]` names the server's release line, the driver's
|
||||
`_add_zabbix_repository(release)` and a second try), configure (the driver's
|
||||
`_write_zabbix_agent_config`), firewall (the driver's `_allow_zabbix_agent_sources`),
|
||||
start (`manage_service` enable and restart of `ZABBIX_AGENT_SERVICE`). Settings that do
|
||||
not hold raise `ValueError` before anything changes.
|
||||
- `ZabbixAgentSettingsDict`; `zabbix_agent_config(text, settings)`, the agent's
|
||||
configuration with `Server`, `ServerActive`, `ListenPort` and `Hostname` set and the rest
|
||||
kept; `zabbix_repository_command(os_release, release)`, the root command that adds
|
||||
Zabbix's repository for Debian, Ubuntu, Raspberry Pi OS, RHEL, CentOS, Oracle Linux,
|
||||
AlmaLinux and Rocky Linux, in the layout before and since 7.2;
|
||||
`check_zabbix_agent_settings(settings)`.
|
||||
- `fixes.enable_and_restart(manage_service, service)`, the start step `LldpAgentMixin` and
|
||||
`ZabbixAgentMixin` share.
|
||||
- Podman as a container engine (NetOrk/netork#818): `container_engines()` lists it with the
|
||||
API `podman-libpod`, and `open_container_engine("podman")` reaches it with `podman system
|
||||
dial-stdio` (Podman 4.0 and later). It is the host's rootful Podman, so the connection
|
||||
runs every call as root (`ContainerEngineConnection.privileged`, `ROOT_ENGINES`).
|
||||
- `apt_sources`: apt's https sources through the package cache, in the form apt-cacher-ng
|
||||
fetches over https itself and stores (`http://HTTPS///<host>/…`, with `AllowUserPorts=80
|
||||
443` on the cache). `cacheable_https_hosts(report)` names the hosts this applies to, and
|
||||
`rewrite_sources_command` / `revert_sources_command` are the root commands, for `.list`
|
||||
and deb822 `.sources` files. A host apt keeps credentials for in `auth.conf` stays on
|
||||
https, because apt finds them by the host's name and the cache is shared. The rewrite
|
||||
names such a host in the proxy file as `Package-Proxy::Not-Cached` (NetOrk/netork#2555).
|
||||
- `get_package_proxy_report()` reports those hosts as `not_cached`.
|
||||
|
||||
### Changed
|
||||
- `container_engines()` lists an engine only when its CLI names itself on `--version`:
|
||||
podman-docker's `docker` is podman, and is no longer listed as Docker.
|
||||
- `configure_package_proxy(url)` has a step `cache` before `https`. It rewrites the https
|
||||
sources it can, fetches one of them through *url*, and turns them all back to https
|
||||
where *url* cannot. The `fetch` step proves the proxy with a plain http source only.
|
||||
- `configure_package_proxy("")` turns the sources back to https (step `sources`) before it
|
||||
removes the proxy file. Without the cache, apt would look for a host called `HTTPS`.
|
||||
|
||||
## [4.39.0] - 2026-11-14
|
||||
|
||||
### Added
|
||||
|
||||
@@ -124,7 +124,10 @@ application wants made on a device runs as fixed steps the driver implements, wi
|
||||
the caller supplies: `SnmpAgentMixin.configure_snmp_agent(settings)` (install, configure,
|
||||
firewall, start), `LldpAgentMixin.configure_lldp_agent(interfaces)` (install, write the
|
||||
interfaces, enable and restart; the driver writes only the interfaces),
|
||||
`EnsurePackageMixin.ensure_package(name)`, `PackageProxyMixin.configure_package_proxy(url)`,
|
||||
`ZabbixAgentMixin.configure_zabbix_agent(settings)` (install from the host's packages or
|
||||
else Zabbix's repository, configure, firewall, enable and restart; the driver writes the
|
||||
configuration and opens the firewall), `EnsurePackageMixin.ensure_package(name)`,
|
||||
`PackageProxyMixin.configure_package_proxy(url)`,
|
||||
`TimeSyncMixin.configure_time_sync(ntp_servers, timezone)` and
|
||||
`ContainerEngineMixin.grant_container_engine_access(user)`. A driver whose device takes the
|
||||
change in one call overrides the template method. A driver mixes in the fixes its device
|
||||
@@ -172,6 +175,15 @@ wins (`get_package_proxy`, `get_package_proxy_report` with every line that sets
|
||||
different proxy, reads the result back and fetches one http source's `InRelease` through it.
|
||||
A driver supplies `_run_apt_command`, `_run_apt_root_command` and `_write_apt_file`.
|
||||
|
||||
`AptReleaseMixin` (`napalm_device_types.apt_releases`) comes with it: every source apt reads,
|
||||
with its file, line, URI and suites, and the host's release from `/etc/os-release`
|
||||
(`get_apt_source_releases`). A release upgrade leaves third-party sources on the old release,
|
||||
where they stop getting updates once the vendor stops building for it; whether a suite names
|
||||
another release than the host's is the caller's to judge (`suite_release` gives the word).
|
||||
`set_apt_source_release(path, release, codename)` moves every suite of *release* in one
|
||||
source file to *codename*, after fetching each new suite's `InRelease` with apt's downloader:
|
||||
where the vendor has none, the file stays as it is.
|
||||
|
||||
`HostStatusMixin` (`get_host_status`) is mixed in the same way: whether a Linux host needs
|
||||
a reboot to finish an update (`/var/run/reboot-required`, `needs-restarting -r`, or a newer
|
||||
kernel of the running flavour installed), whether it patches itself (unattended-upgrades,
|
||||
@@ -403,10 +415,11 @@ but not revert sets `revert_config_revision = None`.
|
||||
The mixin sits in `DeviceTypeDriver` and only declares, so `hasattr(driver, "run_command")` is true exactly where a driver implements it. For SSH drivers, `channel.run_on_transport` and `channel.open_stream_on_transport` are the implementation over a paramiko exec channel: no PTY, stderr kept apart, a real exit code. How a command gains root stays with the driver.
|
||||
|
||||
**`ContainerEngineMixin` is deliberately different from `SystemdServicesMixin`.** The systemd mixin owns its command and its parse; this one owns neither.
|
||||
- `container_engines()` says which engines the host offers (`[{"engine": "docker", "api": "docker-engine"}]`).
|
||||
- `container_engines()` says which engines the host offers (`[{"engine": "docker", "api": "docker-engine"}, {"engine": "podman", "api": "podman-libpod"}]`). An engine is listed when its CLI answers `--version` with its own name; podman-docker's `docker` says it is podman and is left out.
|
||||
- `open_container_engine(engine)` returns a `ContainerEngineConnection`:
|
||||
- `open_api()` is a stream to the engine's API (`docker system dial-stdio`);
|
||||
- `open_api()` is a stream to the engine's API (`docker system dial-stdio`, `podman system dial-stdio`);
|
||||
- `run_cli(args)` and `stream_cli(args)` run the engine's CLI with arguments the caller chooses. With `privileged=True` the driver runs it the way `run_command` gains root, for an engine that refuses the login user.
|
||||
- **Podman is reached as root** (`ROOT_ENGINES`, `connection.privileged`): the host's rootful Podman, the engine of the system's containers and of Quadlet's units, whose API `podman.socket` serves. A login user's rootless containers are that user's own and invisible to others. `podman system dial-stdio` exists from Podman 4.0 on and needs the socket listening (`systemctl enable --now podman.socket`) (NetOrk/netork#818).
|
||||
- The driver decides only *how* the engine is reached. Its hook `_container_engine_binary(engine)` returns, for QNAP, the Container Station path, and the caller never sees it.
|
||||
|
||||
**What runs on the engine, and what to do with it, is netOrk's** (NetOrk/netork#765): the container model, the Engine API requests, compose, updates. Above the connection everything is specific to the service, so this package abstracts the connection and nothing more.
|
||||
|
||||
@@ -78,8 +78,10 @@ The same shape replaces every action in today's `run_device_action`:
|
||||
|---|---|
|
||||
| `fix_snmp` | `configure_snmp_agent(settings)` |
|
||||
| `fix_apt_proxy` | `configure_package_proxy(url)` |
|
||||
| an apt source on another release than the host (NetOrk/netork#2614) | `set_apt_source_release(path, release, codename)` |
|
||||
| `fix_docker_permissions` | `grant_container_engine_access(user)` |
|
||||
| `install_lldpd`, `install_auc` | `ensure_package(name)` |
|
||||
| the Zabbix agent where no container engine runs (NetOrk/netork#2571) | `configure_zabbix_agent(settings)` |
|
||||
|
||||
The mixin is mixed in by the driver, or by the platform package, not by a role base. A
|
||||
driver that inherits a template it cannot run on its device, as a NAS driver built on the
|
||||
@@ -158,6 +160,7 @@ the driver.
|
||||
| `virtual_ips` | vip-manager, keepalived, awk |
|
||||
| `system_identity` | SMBIOS via sysfs and `dmidecode` |
|
||||
| `package_updates` | apt and dnf output |
|
||||
| `apt_releases` | apt's source files (one-line and deb822), `/etc/os-release`, `apt-helper` |
|
||||
| `ssh_server` | OpenSSH (`sshd -T`, `sshd -t`, `sshd_config`); which algorithms to drop stays the caller's |
|
||||
| `terminal` | ANSI escape codes in shell output |
|
||||
| `channel`: `run_on_transport`, `open_stream_on_transport`, `ParamikoExecStream` | paramiko exec channels |
|
||||
|
||||
@@ -86,6 +86,15 @@ from napalm_device_types.apt_proxy import (
|
||||
parse_apt_proxy,
|
||||
same_proxy_url,
|
||||
)
|
||||
from napalm_device_types.apt_releases import (
|
||||
APT_RELEASES_COMMAND,
|
||||
AptReleaseFixDict,
|
||||
AptReleaseMixin,
|
||||
AptReleasesDict,
|
||||
AptSourceDict,
|
||||
parse_apt_releases,
|
||||
suite_release,
|
||||
)
|
||||
from napalm_device_types.arp import ARP_AGE_UNKNOWN
|
||||
from napalm_device_types.backup_server import BackupServerDriver
|
||||
from napalm_device_types.backups import (
|
||||
@@ -138,12 +147,13 @@ from napalm_device_types.dhcp import DhcpApplyError, DhcpServerMixin, normalize_
|
||||
from napalm_device_types.dns_forward import DnsForwardMixin
|
||||
from napalm_device_types.firewall import FirewallDriver
|
||||
from napalm_device_types.firewall_rules import FirewallRuleMixin
|
||||
from napalm_device_types.firmware import FirmwareOfferMixin
|
||||
from napalm_device_types.firmware import FirmwareOfferDict, FirmwareOfferMixin, FirmwareUpgradeMixin
|
||||
from napalm_device_types.fixes import (
|
||||
EnsurePackageMixin,
|
||||
LldpAgentMixin,
|
||||
PackageProxyMixin,
|
||||
SnmpAgentMixin,
|
||||
ZabbixAgentMixin,
|
||||
run_fix_steps,
|
||||
)
|
||||
from napalm_device_types.health_metrics import HealthMetricsMixin
|
||||
@@ -204,7 +214,6 @@ from napalm_device_types.models import (
|
||||
DnsForwardDict,
|
||||
EventLogEntryDict,
|
||||
FirmwareDict,
|
||||
FirmwareOfferDict,
|
||||
FixResultDict,
|
||||
FixStepDict,
|
||||
FixStepOutcomeDict,
|
||||
@@ -343,6 +352,12 @@ from napalm_device_types.virtual_ips import (
|
||||
)
|
||||
from napalm_device_types.vm_migration import MigrationPreflightDict
|
||||
from napalm_device_types.vm_stop_request import VmStopRequestMixin
|
||||
from napalm_device_types.zabbix_agent import (
|
||||
ZabbixAgentSettingsDict,
|
||||
check_zabbix_agent_settings,
|
||||
zabbix_agent_config,
|
||||
zabbix_repository_command,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"BACKUP_JOB_KINDS",
|
||||
@@ -375,6 +390,7 @@ __all__ = [
|
||||
"OspfMixin",
|
||||
"UpdateProgressDict",
|
||||
"FirmwareOfferMixin",
|
||||
"FirmwareUpgradeMixin",
|
||||
"LicenseDict",
|
||||
"LicenseMixin",
|
||||
"EnsurePackageMixin",
|
||||
@@ -389,9 +405,21 @@ __all__ = [
|
||||
"parse_apt_fetch",
|
||||
"parse_apt_proxy",
|
||||
"same_proxy_url",
|
||||
"APT_RELEASES_COMMAND",
|
||||
"AptReleaseFixDict",
|
||||
"AptReleaseMixin",
|
||||
"AptReleasesDict",
|
||||
"AptSourceDict",
|
||||
"parse_apt_releases",
|
||||
"suite_release",
|
||||
"SnmpAgentMixin",
|
||||
"SnmpAgentSettingsDict",
|
||||
"run_fix_steps",
|
||||
"ZabbixAgentMixin",
|
||||
"ZabbixAgentSettingsDict",
|
||||
"check_zabbix_agent_settings",
|
||||
"zabbix_agent_config",
|
||||
"zabbix_repository_command",
|
||||
"SnmpHealthProfile",
|
||||
"SnmpInterfacesSpec",
|
||||
"SnmpMemorySpec",
|
||||
|
||||
@@ -35,8 +35,12 @@ page is the right one, the caller says (``expect``).
|
||||
out every other line that sets a different one, reads the result back from
|
||||
``apt-config``, and fetches the ``InRelease`` of one http source through it. The
|
||||
last step is what proves it: apt reached a mirror through the proxy it is now set
|
||||
to. The source chosen is one apt does not fetch ``DIRECT``. Then it fetches one
|
||||
https source the same way; where the proxy refuses it, https goes ``DIRECT``.
|
||||
to. The source chosen is one apt does not fetch ``DIRECT``. Then the https sources
|
||||
go through the proxy in the form it stores (:mod:`~napalm_device_types.apt_sources`),
|
||||
and one of them is fetched; where the proxy cannot, they go back to https. Last it
|
||||
fetches one https source left the same way; where the proxy refuses it, https goes
|
||||
``DIRECT``. Removing the proxy turns the sources back to https first: without the
|
||||
proxy apt cannot fetch the stored form.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -49,6 +53,15 @@ from urllib.parse import urlsplit
|
||||
|
||||
from typing_extensions import TypedDict
|
||||
|
||||
from napalm_device_types.apt_releases import AptReleaseMixin
|
||||
from napalm_device_types.apt_sources import (
|
||||
NOT_CACHED_KEY,
|
||||
VIA_CACHE,
|
||||
cacheable_https_hosts,
|
||||
is_host_name,
|
||||
revert_sources_command,
|
||||
rewrite_sources_command,
|
||||
)
|
||||
from napalm_device_types.fixes import PackageProxyMixin, run_fix_steps
|
||||
from napalm_device_types.models import FixResultDict, FixStepOutcomeDict
|
||||
from napalm_device_types.terminal import strip_terminal_codes
|
||||
@@ -87,6 +100,9 @@ class AptProxyReportDict(TypedDict):
|
||||
#: The base URIs of the host's sources over http and over https.
|
||||
http_sources: list[str]
|
||||
https_sources: list[str]
|
||||
#: The hosts the cache rewrite left on https, for their credentials
|
||||
#: (:data:`~napalm_device_types.apt_sources.NOT_CACHED_KEY`).
|
||||
not_cached: list[str]
|
||||
|
||||
|
||||
class ProxyFetchDict(TypedDict):
|
||||
@@ -123,7 +139,8 @@ PROXY_URL = re.compile(r"^https?://[A-Za-z0-9.\-\[\]:@%_~/]+$")
|
||||
#: them early.
|
||||
APT_PROXY_COMMAND = (
|
||||
"( export LC_ALL=C; printf '%s%s\\n' APROXY_ BEGIN; "
|
||||
"echo '[dump]'; apt-config dump 2>/dev/null | grep -i -E '^Acquire::https?::Proxy'; "
|
||||
"echo '[dump]'; apt-config dump 2>/dev/null "
|
||||
"| grep -i -E '^(Acquire::https?::Proxy|Package-Proxy::)'; "
|
||||
'eval "$(apt-config shell ap_parts Dir::Etc::parts/d ap_main Dir::Etc::main/f 2>/dev/null)"; '
|
||||
"echo '[parts]'; for f in \"${ap_parts:-/etc/apt/apt.conf.d/}\"*; do "
|
||||
'[ -f "$f" ] && grep -Hin proxy "$f"; done; '
|
||||
@@ -133,6 +150,7 @@ APT_PROXY_COMMAND = (
|
||||
"printf '%s%s\\n' APROXY_ END ) 2>/dev/null | cat"
|
||||
)
|
||||
|
||||
_NOT_CACHED = re.compile(rf'^{re.escape(NOT_CACHED_KEY)}::\s+"([^"]+)"', re.I)
|
||||
_DUMP = re.compile(r'^Acquire::(https?)::Proxy(::\S+|-Auto-Detect)?\s+"([^"]*)"', re.I)
|
||||
_GREP = re.compile(r"^(/[^:]+):(\d+):(.*)$")
|
||||
_FLAT = re.compile(r'^\s*Acquire::http::Proxy\s+"([^"]*)"\s*;', re.I)
|
||||
@@ -293,6 +311,7 @@ def parse_apt_proxy(output: str) -> AptProxyReportDict:
|
||||
"https_settings": _settings(lines, _https_setting),
|
||||
"http_sources": _of_scheme(sources, "http"),
|
||||
"https_sources": _of_scheme(sources, "https"),
|
||||
"not_cached": [m.group(1) for m in map(_NOT_CACHED.match, sections.get("dump", [])) if m],
|
||||
}
|
||||
|
||||
|
||||
@@ -409,7 +428,7 @@ def _described(s: AptProxySettingDict) -> str:
|
||||
return f"{s['file']}:{s['line']} ({s['value']}) {what}"
|
||||
|
||||
|
||||
class AptProxyMixin(PackageProxyMixin):
|
||||
class AptProxyMixin(AptReleaseMixin, PackageProxyMixin):
|
||||
""":class:`~napalm_device_types.fixes.PackageProxyMixin` for a Debian-family host.
|
||||
|
||||
The template form, like :class:`~napalm_device_types.host_status.HostStatusMixin`:
|
||||
@@ -498,16 +517,20 @@ class AptProxyMixin(PackageProxyMixin):
|
||||
def configure_package_proxy(self, url: str) -> FixResultDict:
|
||||
"""apt fetches through *url*, and is shown to; ``""`` removes :attr:`APT_PROXY_FILE`.
|
||||
|
||||
https goes through *url* too, unless *url* refuses it: then ``DIRECT``.
|
||||
The https sources go through *url* in the form it stores, where it can; the
|
||||
rest goes through *url* too, unless *url* refuses it: then ``DIRECT``.
|
||||
"""
|
||||
if not url:
|
||||
return run_fix_steps([("configure", self._remove_package_proxy)])
|
||||
return run_fix_steps(
|
||||
[("sources", self._revert_sources), ("configure", self._remove_package_proxy)]
|
||||
)
|
||||
return run_fix_steps(
|
||||
[
|
||||
("configure", lambda: self._write_package_proxy(url)),
|
||||
("conflicts", lambda: self._comment_out_other_proxies(url)),
|
||||
("verify", lambda: self._verify_package_proxy(url)),
|
||||
("fetch", lambda: self._fetch_through_package_proxy(url)),
|
||||
("cache", lambda: self._cache_https_sources(url)),
|
||||
("https", lambda: self._https_through_package_proxy(url)),
|
||||
]
|
||||
)
|
||||
@@ -515,6 +538,11 @@ class AptProxyMixin(PackageProxyMixin):
|
||||
def _not_apt(self) -> FixStepOutcomeDict:
|
||||
return {"ok": False, "output": "not an apt host"}
|
||||
|
||||
def _revert_sources(self) -> FixStepOutcomeDict:
|
||||
if not self._is_apt_host():
|
||||
return self._not_apt()
|
||||
return self._run_apt_root_command(revert_sources_command(), "sources back on https")
|
||||
|
||||
def _remove_package_proxy(self) -> FixStepOutcomeDict:
|
||||
if not self._is_apt_host():
|
||||
return self._not_apt()
|
||||
@@ -522,7 +550,9 @@ class AptProxyMixin(PackageProxyMixin):
|
||||
f"rm -f {shlex.quote(self.APT_PROXY_FILE)}", "proxy removed"
|
||||
)
|
||||
|
||||
def _write_package_proxy(self, url: str, https_direct: bool = False) -> FixStepOutcomeDict:
|
||||
def _write_package_proxy(
|
||||
self, url: str, https_direct: bool = False, not_cached: list[str] | None = None
|
||||
) -> FixStepOutcomeDict:
|
||||
if not self._is_apt_host():
|
||||
return self._not_apt()
|
||||
if not PROXY_URL.match(url):
|
||||
@@ -530,6 +560,10 @@ class AptProxyMixin(PackageProxyMixin):
|
||||
content = f'Acquire::http::Proxy "{url}";\n'
|
||||
if https_direct:
|
||||
content += 'Acquire::https::Proxy "DIRECT";\n'
|
||||
# Host names the report read from this file; one a line could not carry is left out.
|
||||
content += "".join(
|
||||
f'{NOT_CACHED_KEY}:: "{host}";\n' for host in not_cached or [] if is_host_name(host)
|
||||
)
|
||||
return self._write_apt_file(self.APT_PROXY_FILE, content)
|
||||
|
||||
def _report(self) -> AptProxyReportDict:
|
||||
@@ -566,16 +600,7 @@ class AptProxyMixin(PackageProxyMixin):
|
||||
}
|
||||
|
||||
def _fetch_through_package_proxy(self, url: str) -> FixStepOutcomeDict:
|
||||
report = self._report()
|
||||
direct = {h.lower() for h, v in report["per_host"].items() if v.lower() == "direct"}
|
||||
source = next(
|
||||
(
|
||||
s
|
||||
for s in report["http_sources"]
|
||||
if "/dists/" in s and (urlsplit(s).hostname or "").lower() not in direct
|
||||
),
|
||||
None,
|
||||
)
|
||||
source = _plain_http_source(self._report())
|
||||
if source is None:
|
||||
return {"ok": True, "output": "no http source to fetch through the proxy; not tried"}
|
||||
target = source.rstrip("/") + "/InRelease"
|
||||
@@ -584,6 +609,49 @@ class AptProxyMixin(PackageProxyMixin):
|
||||
return {"ok": True, "output": f"fetched {target} through {url}"}
|
||||
return {"ok": False, "output": f"{target}: {fetched['error']}"}
|
||||
|
||||
def _cache_https_sources(self, url: str) -> FixStepOutcomeDict:
|
||||
"""The https sources through *url* in the form it stores; back to https where it cannot."""
|
||||
hosts = cacheable_https_hosts(self._report())
|
||||
if not hosts:
|
||||
return {"ok": True, "output": "no https source to cache"}
|
||||
rewritten = self._run_apt_root_command(
|
||||
rewrite_sources_command(hosts, self.APT_PROXY_FILE), "sources rewritten"
|
||||
)
|
||||
if not rewritten["ok"]:
|
||||
return rewritten
|
||||
return self._fetch_cached(url, hosts, self._report())
|
||||
|
||||
def _fetch_cached(
|
||||
self, url: str, hosts: list[str], report: AptProxyReportDict
|
||||
) -> FixStepOutcomeDict:
|
||||
"""One rewritten source through *url*: it stays where *url* fetched it."""
|
||||
left = sorted(set(hosts) & set(report["not_cached"]))
|
||||
named = f"; left on https, apt has credentials for them: {', '.join(left)}" if left else ""
|
||||
target = next(
|
||||
(s.rstrip("/") + "/InRelease" for s in report["http_sources"] if _cached(s)), None
|
||||
)
|
||||
if target is None:
|
||||
return {"ok": True, "output": f"no https source to cache{named}"}
|
||||
fetched = parse_apt_fetch(self._run_apt_command(apt_fetch_command(target)))
|
||||
if fetched["reachable"] and not fetched["error"]:
|
||||
cached = ", ".join(h for h in hosts if h not in left)
|
||||
return {"ok": True, "output": f"https sources cached through {url}: {cached}{named}"}
|
||||
return self._back_to_https(url, f"{target}: {fetched['error']}", named)
|
||||
|
||||
def _back_to_https(self, url: str, why: str, named: str) -> FixStepOutcomeDict:
|
||||
"""*url* did not fetch the stored form: every source goes back to https."""
|
||||
back = self._run_apt_root_command(revert_sources_command(), "sources back on https")
|
||||
if not back["ok"]:
|
||||
return {
|
||||
"ok": False,
|
||||
"output": f"{why}; turning the sources back to https failed: {back['output']}",
|
||||
}
|
||||
return {
|
||||
"ok": True,
|
||||
"output": f"{url} does not fetch https sources itself ({why}); "
|
||||
f"they stay on https, not cached{named}",
|
||||
}
|
||||
|
||||
def _https_through_package_proxy(self, url: str) -> FixStepOutcomeDict:
|
||||
"""An https source through *url*; where *url* refuses it, https goes ``DIRECT``."""
|
||||
found = _https_source(self._report(), lambda proxy: same_proxy_url(proxy, url))
|
||||
@@ -609,7 +677,9 @@ class AptProxyMixin(PackageProxyMixin):
|
||||
|
||||
def _https_direct(self, url: str) -> FixStepOutcomeDict:
|
||||
"""Write https ``DIRECT`` beside *url*, and comment out the later lines to *url*."""
|
||||
written = self._write_package_proxy(url, https_direct=True)
|
||||
written = self._write_package_proxy(
|
||||
url, https_direct=True, not_cached=self._report()["not_cached"]
|
||||
)
|
||||
if not written["ok"]:
|
||||
return written
|
||||
later = self._later_https_lines(self._report(), url)
|
||||
@@ -635,6 +705,26 @@ class AptProxyMixin(PackageProxyMixin):
|
||||
]
|
||||
|
||||
|
||||
def _plain_http_source(report: AptProxyReportDict) -> str | None:
|
||||
"""A suite's http source the proxy takes, not in the cache's https form: its proof."""
|
||||
direct = {h.lower() for h, v in report["per_host"].items() if v.lower() == "direct"}
|
||||
return next(
|
||||
(
|
||||
s
|
||||
for s in report["http_sources"]
|
||||
if "/dists/" in s
|
||||
and not s.startswith(VIA_CACHE)
|
||||
and (urlsplit(s).hostname or "").lower() not in direct
|
||||
),
|
||||
None,
|
||||
)
|
||||
|
||||
|
||||
def _cached(source: str) -> bool:
|
||||
"""A source of a suite in the form the cache stores."""
|
||||
return source.startswith(VIA_CACHE) and "/dists/" in source
|
||||
|
||||
|
||||
def _https_source(
|
||||
report: AptProxyReportDict, through: Callable[[str | None], bool]
|
||||
) -> tuple[str, str] | None:
|
||||
|
||||
@@ -0,0 +1,486 @@
|
||||
"""apt's sources and the release each names, and moving one file's sources to the host's.
|
||||
|
||||
A release upgrade moves the distribution's own sources to the new release and
|
||||
leaves the third-party ones where they were -- or turns them off, and the person
|
||||
who turns them back on keeps the old name. A source on the old release still
|
||||
answers, so apt says nothing; its packages simply stop getting updates once the
|
||||
vendor stops building for that release. Docker's last build for Ubuntu 20.04
|
||||
(``focal``) is 28.1.1, and a noble host on ``focal`` never sees a newer one
|
||||
(NetOrk/netork#2614).
|
||||
|
||||
**What is read** (:meth:`AptReleaseMixin.get_apt_source_releases`): the host's
|
||||
``/etc/os-release`` (``ID``, ``ID_LIKE``, ``VERSION_CODENAME``,
|
||||
``UBUNTU_CODENAME``) and every source apt reads -- ``sources.list`` and the
|
||||
``*.list`` and ``*.sources`` files of ``sources.list.d``, wherever ``apt-config``
|
||||
says they are -- with its file, line, types, URI, suites and components.
|
||||
Commented-out lines and deb822 paragraphs with ``Enabled: no`` are not sources.
|
||||
Whether a suite names another release than the host's is not said here: which
|
||||
words are releases of which distribution is the caller's to know
|
||||
(:func:`suite_release` gives the word).
|
||||
|
||||
**Moving a file** (:meth:`AptReleaseMixin.set_apt_source_release`): every suite of
|
||||
*release* in one of apt's source files becomes the same suite of *codename*
|
||||
(``focal-security`` → ``noble-security``), on enabled lines only. Before anything is
|
||||
written, each new suite's ``InRelease`` (or ``Release``) is fetched from its URI with
|
||||
apt's own downloader and the proxy apt is set to; when one of them is not there,
|
||||
the vendor does not build for *codename* and the file stays as it is. The file is
|
||||
written whole, as root, mode 644. Loading the package index afterwards is the
|
||||
caller's (``refresh_available_updates``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import re
|
||||
import shlex
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from typing_extensions import TypedDict
|
||||
|
||||
from napalm_device_types.fixes import run_fix_steps
|
||||
from napalm_device_types.models import FixResultDict, FixStepOutcomeDict
|
||||
from napalm_device_types.terminal import strip_terminal_codes
|
||||
|
||||
_BEGIN = "ARELEASE_BEGIN"
|
||||
_END = "ARELEASE_END"
|
||||
|
||||
#: A release's codename: one lower-case word (``noble``, ``bookworm``).
|
||||
_CODENAME = re.compile(r"^[a-z][a-z0-9]*$")
|
||||
#: A suite that names a release: its codename, then perhaps a suffix (``-security``).
|
||||
_SUITE = re.compile(r"^([a-z][a-z0-9]*)(-[a-z0-9-]+)?$")
|
||||
#: One source line of a ``.list`` file, its comment cut off: types, options, URI, suite.
|
||||
_LIST_LINE = re.compile(r"^(\s*(deb|deb-src)\s+(?:\[[^\]]*\]\s+)?(\S+)\s+)(\S+)(.*)$")
|
||||
#: A deb822 field's first line.
|
||||
_FIELD = re.compile(r"^([A-Za-z][A-Za-z0-9-]*):(.*)$")
|
||||
_DISABLED = frozenset({"no", "false", "0"})
|
||||
|
||||
|
||||
class AptSourceDict(TypedDict):
|
||||
"""One source apt reads (``get_apt_source_releases``)."""
|
||||
|
||||
#: The file it is in.
|
||||
file: str
|
||||
#: The line of a ``.list`` file's entry; of a deb822 paragraph's ``Suites:`` field.
|
||||
line: int
|
||||
#: ``deb``, ``deb-src``; a deb822 paragraph may name both.
|
||||
types: list[str]
|
||||
#: Its URI, as written; a deb822 paragraph with several URIs is one source per URI.
|
||||
uri: str
|
||||
#: Its suites; a flat repository's path (``./``) is one too.
|
||||
suites: list[str]
|
||||
components: list[str]
|
||||
|
||||
|
||||
class AptReleasesDict(TypedDict):
|
||||
"""The host's release and its sources (``get_apt_source_releases``)."""
|
||||
|
||||
#: ``ID`` and ``ID_LIKE`` of ``/etc/os-release``; "" and [] when it says none.
|
||||
os_id: str
|
||||
os_like: list[str]
|
||||
#: ``VERSION_CODENAME``; a derivative's own (Linux Mint's ``virginia``).
|
||||
codename: str
|
||||
#: ``UBUNTU_CODENAME``: the Ubuntu release Ubuntu and its derivatives are built on.
|
||||
ubuntu_codename: str
|
||||
sources: list[AptSourceDict]
|
||||
|
||||
|
||||
class AptReleaseFixDict(FixResultDict):
|
||||
"""The result of ``set_apt_source_release``: the fix, and how many suites moved."""
|
||||
|
||||
rewritten: int
|
||||
|
||||
|
||||
def apt_releases_command(
|
||||
apt_config: str = "apt-config", os_release: str = "/etc/os-release"
|
||||
) -> str:
|
||||
"""The read-only command that prints the host's release and apt's source files.
|
||||
|
||||
One line, POSIX ``sh``, no privileges. Every line of a file is printed behind a
|
||||
``|``, so an empty line -- which ends a deb822 paragraph -- comes through.
|
||||
*apt_config* and *os_release* let a test point it elsewhere.
|
||||
"""
|
||||
return (
|
||||
f"( export LC_ALL=C; printf '%s%s\\n' ARELEASE_ BEGIN; echo '@@os'; "
|
||||
f'o={shlex.quote(os_release)}; [ -r "$o" ] || o=/usr/lib/os-release; '
|
||||
'[ -r "$o" ] && sed \'s/^/|/\' "$o"; '
|
||||
f'eval "$({apt_config} shell sl Dir::Etc::sourcelist/f sp Dir::Etc::sourceparts/d '
|
||||
'2>/dev/null)"; sl="${sl:-/etc/apt/sources.list}"; sp="${sp:-/etc/apt/sources.list.d/}"; '
|
||||
'for f in "$sl" "$sp"*.list "$sp"*.sources; do [ -f "$f" ] || continue; '
|
||||
'if [ -r "$f" ]; then printf \'@@file %s\\n\' "$f"; sed \'s/^/|/\' "$f"; '
|
||||
"else printf '@@unreadable %s\\n' \"$f\"; fi; done; "
|
||||
"printf '%s%s\\n' ARELEASE_ END ) 2>/dev/null | cat"
|
||||
)
|
||||
|
||||
|
||||
#: The command as a driver runs it.
|
||||
APT_RELEASES_COMMAND = apt_releases_command()
|
||||
|
||||
|
||||
def suite_release(suite: str) -> str | None:
|
||||
"""The word a suite starts with, ``noble`` of ``noble-security``; None for a path.
|
||||
|
||||
Whether that word is a release, and of which distribution, the caller knows:
|
||||
``stable`` and ``nodistro`` have the form too.
|
||||
"""
|
||||
found = _SUITE.match(suite)
|
||||
return found.group(1) if found else None
|
||||
|
||||
|
||||
def _codename(name: str) -> str:
|
||||
if not _CODENAME.match(name):
|
||||
raise ValueError(f"not a release's codename: {name!r}")
|
||||
return name
|
||||
|
||||
|
||||
# ── Reading ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _blocks(output: str) -> list[tuple[str, list[str]]]:
|
||||
"""``(header, lines)`` of each block between the frame markers, in order."""
|
||||
lines = strip_terminal_codes(output).splitlines()
|
||||
stripped = [line.strip() for line in lines]
|
||||
try:
|
||||
start = stripped.index(_BEGIN)
|
||||
end = stripped.index(_END, start)
|
||||
except ValueError:
|
||||
raise ValueError("no intact apt release report in the output") from None
|
||||
blocks: list[tuple[str, list[str]]] = []
|
||||
for line in lines[start + 1 : end]:
|
||||
if line.startswith("@@"):
|
||||
blocks.append((line[2:].strip(), []))
|
||||
elif line.startswith("|") and blocks:
|
||||
blocks[-1][1].append(line[1:].rstrip("\r"))
|
||||
return blocks
|
||||
|
||||
|
||||
def _os_release(lines: list[str]) -> dict[str, str]:
|
||||
values: dict[str, str] = {}
|
||||
for line in lines:
|
||||
key, sep, value = line.strip().partition("=")
|
||||
if sep and key:
|
||||
values[key] = value.strip().strip("'\"")
|
||||
return values
|
||||
|
||||
|
||||
def _code(line: str) -> str:
|
||||
"""*line* without its comment."""
|
||||
return line.split("#", 1)[0]
|
||||
|
||||
|
||||
def _list_sources(path: str, lines: list[str]) -> list[AptSourceDict]:
|
||||
found: list[AptSourceDict] = []
|
||||
for number, line in enumerate(lines, start=1):
|
||||
entry = _LIST_LINE.match(_code(line))
|
||||
if entry:
|
||||
_, kind, uri, suite, rest = entry.groups()
|
||||
found.append(
|
||||
{
|
||||
"file": path,
|
||||
"line": number,
|
||||
"types": [kind],
|
||||
"uri": uri,
|
||||
"suites": [suite],
|
||||
"components": rest.split(),
|
||||
}
|
||||
)
|
||||
return found
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Field:
|
||||
first: int # index of its first line
|
||||
lines: list[int] = field(default_factory=list) # indices of every line it spans
|
||||
value: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Paragraph:
|
||||
fields: dict[str, _Field] = field(default_factory=dict)
|
||||
|
||||
def value(self, name: str) -> str:
|
||||
found = self.fields.get(name)
|
||||
return found.value if found else ""
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return self.value("enabled").strip().lower() not in _DISABLED
|
||||
|
||||
|
||||
def _paragraphs(lines: list[str]) -> list[_Paragraph]:
|
||||
"""The deb822 paragraphs of *lines*; comments are skipped, an empty line ends one."""
|
||||
paragraphs: list[_Paragraph] = []
|
||||
current: _Paragraph | None = None
|
||||
last: _Field | None = None
|
||||
for index, line in enumerate(lines):
|
||||
if not line.strip():
|
||||
current, last = None, None
|
||||
continue
|
||||
if line.lstrip().startswith("#"):
|
||||
continue
|
||||
if line[0] in " \t":
|
||||
if last is not None:
|
||||
last.lines.append(index)
|
||||
last.value += " " + line.strip()
|
||||
continue
|
||||
found = _FIELD.match(line)
|
||||
if not found:
|
||||
continue
|
||||
if current is None:
|
||||
current = _Paragraph()
|
||||
paragraphs.append(current)
|
||||
last = _Field(first=index, lines=[index], value=found.group(2).strip())
|
||||
current.fields[found.group(1).lower()] = last
|
||||
return paragraphs
|
||||
|
||||
|
||||
def _deb822_sources(path: str, lines: list[str]) -> list[AptSourceDict]:
|
||||
found: list[AptSourceDict] = []
|
||||
for paragraph in _paragraphs(lines):
|
||||
suites = paragraph.fields.get("suites")
|
||||
if not paragraph.enabled or suites is None:
|
||||
continue
|
||||
for uri in paragraph.value("uris").split():
|
||||
found.append(
|
||||
{
|
||||
"file": path,
|
||||
"line": suites.first + 1,
|
||||
"types": paragraph.value("types").split(),
|
||||
"uri": uri,
|
||||
"suites": suites.value.split(),
|
||||
"components": paragraph.value("components").split(),
|
||||
}
|
||||
)
|
||||
return found
|
||||
|
||||
|
||||
def _is_deb822(path: str) -> bool:
|
||||
return path.endswith(".sources")
|
||||
|
||||
|
||||
def parse_apt_releases(output: str) -> AptReleasesDict:
|
||||
"""What :func:`apt_releases_command` printed, as :class:`AptReleasesDict`.
|
||||
|
||||
:raises ValueError: if the output carried no intact report.
|
||||
"""
|
||||
os_values: dict[str, str] = {}
|
||||
sources: list[AptSourceDict] = []
|
||||
for header, lines in _blocks(output):
|
||||
kind, _, path = header.partition(" ")
|
||||
if kind == "os":
|
||||
os_values = _os_release(lines)
|
||||
elif kind == "file" and path:
|
||||
read = _deb822_sources if _is_deb822(path) else _list_sources
|
||||
sources.extend(read(path, lines))
|
||||
return {
|
||||
"os_id": os_values.get("ID", "").lower(),
|
||||
"os_like": os_values.get("ID_LIKE", "").lower().split(),
|
||||
"codename": os_values.get("VERSION_CODENAME", "").lower(),
|
||||
"ubuntu_codename": os_values.get("UBUNTU_CODENAME", "").lower(),
|
||||
"sources": sources,
|
||||
}
|
||||
|
||||
|
||||
# ── Rewriting ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _moved(suite: str, release: str, codename: str) -> str:
|
||||
return codename + suite[len(release) :] if suite_release(suite) == release else suite
|
||||
|
||||
|
||||
def _rewrite_list_line(line: str, release: str, codename: str) -> tuple[str, int]:
|
||||
code = _code(line)
|
||||
entry = _LIST_LINE.match(code)
|
||||
if not entry:
|
||||
return line, 0
|
||||
suite = entry.group(4)
|
||||
new = _moved(suite, release, codename)
|
||||
if new == suite:
|
||||
return line, 0
|
||||
head = entry.group(1)
|
||||
return head + new + line[len(head) + len(suite) :], 1
|
||||
|
||||
|
||||
def _rewrite_tokens(text: str, release: str, codename: str) -> tuple[str, int]:
|
||||
changed = 0
|
||||
|
||||
def one(token: re.Match[str]) -> str:
|
||||
nonlocal changed
|
||||
new = _moved(token.group(), release, codename)
|
||||
changed += new != token.group()
|
||||
return new
|
||||
|
||||
return re.sub(r"\S+", one, text), changed
|
||||
|
||||
|
||||
def _rewrite_deb822(lines: list[str], release: str, codename: str) -> int:
|
||||
changed = 0
|
||||
for paragraph in _paragraphs(lines):
|
||||
suites = paragraph.fields.get("suites")
|
||||
if not paragraph.enabled or suites is None:
|
||||
continue
|
||||
for index in suites.lines:
|
||||
line = lines[index]
|
||||
cut = line.index(":") + 1 if index == suites.first else 0
|
||||
tail, count = _rewrite_tokens(line[cut:], release, codename)
|
||||
lines[index] = line[:cut] + tail
|
||||
changed += count
|
||||
return changed
|
||||
|
||||
|
||||
def rewrite_release(text: str, release: str, codename: str, *, deb822: bool) -> tuple[str, int]:
|
||||
"""*text*, a source file, with every suite of *release* on *codename*; and how many.
|
||||
|
||||
Only enabled sources: not a commented-out line, not a paragraph with
|
||||
``Enabled: no``. A URI or comment that names *release* stays as it is.
|
||||
|
||||
:raises ValueError: if *release* or *codename* is not a codename.
|
||||
"""
|
||||
release, codename = _codename(release), _codename(codename)
|
||||
lines = text.split("\n")
|
||||
if deb822:
|
||||
changed = _rewrite_deb822(lines, release, codename)
|
||||
else:
|
||||
changed = 0
|
||||
for index, line in enumerate(lines):
|
||||
lines[index], count = _rewrite_list_line(line, release, codename)
|
||||
changed += count
|
||||
return "\n".join(lines), changed
|
||||
|
||||
|
||||
# ── The mixin ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _release_urls(uri: str, suite: str) -> list[str]:
|
||||
base = f"{uri.rstrip('/')}/dists/{suite}/"
|
||||
return [base + "InRelease", base + "Release"]
|
||||
|
||||
|
||||
class AptReleaseMixin:
|
||||
"""Reads the release each apt source names; moves one file's sources to another.
|
||||
|
||||
The template form, like :class:`~napalm_device_types.apt_proxy.AptProxyMixin`,
|
||||
which brings it to every driver that has it: a driver supplies
|
||||
``_run_apt_command`` and ``_write_apt_file``, and ``_is_apt_host`` where it
|
||||
can be something else.
|
||||
"""
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
|
||||
|
||||
def _run_apt_command(self, command: str) -> str:
|
||||
"""Run *command* on the host with ``sh``, without privileges; what it printed."""
|
||||
...
|
||||
|
||||
def _write_apt_file(self, path: str, content: str) -> FixStepOutcomeDict:
|
||||
"""Replace *path* with *content*, as a new file owned by root, mode 644."""
|
||||
...
|
||||
|
||||
def _is_apt_host(self) -> bool:
|
||||
"""Whether apt is the host's package manager; a driver that can tell overrides it."""
|
||||
return True
|
||||
|
||||
def get_apt_source_releases(self) -> AptReleasesDict | None:
|
||||
"""The host's release and every source apt reads; None on a host without apt.
|
||||
|
||||
:raises ValueError: if the host's output carried no intact report.
|
||||
"""
|
||||
if not self._is_apt_host():
|
||||
return None
|
||||
return parse_apt_releases(self._run_apt_command(APT_RELEASES_COMMAND))
|
||||
|
||||
def set_apt_source_release(self, path: str, release: str, codename: str) -> AptReleaseFixDict:
|
||||
"""Every suite of *release* in *path* on *codename*, where the vendor serves it.
|
||||
|
||||
Three steps: ``read`` (*path* is one of apt's source files, and which of its
|
||||
sources name *release*), ``check`` (each new suite's ``InRelease`` or
|
||||
``Release`` is there) and ``rewrite``. Nothing is written when a step fails;
|
||||
nothing is fetched or written when no source of *path* names *release*.
|
||||
"""
|
||||
moving: list[AptSourceDict] = []
|
||||
rewritten = [0]
|
||||
result = run_fix_steps(
|
||||
[
|
||||
("read", lambda: self._sources_on(path, release, codename, moving)),
|
||||
("check", lambda: self._served(moving, release, codename)),
|
||||
("rewrite", lambda: self._rewrite(path, release, codename, moving, rewritten)),
|
||||
]
|
||||
)
|
||||
return {**result, "rewritten": rewritten[0]}
|
||||
|
||||
def _sources_on(
|
||||
self, path: str, release: str, codename: str, moving: list[AptSourceDict]
|
||||
) -> FixStepOutcomeDict:
|
||||
_codename(release)
|
||||
_codename(codename)
|
||||
found = self.get_apt_source_releases()
|
||||
if found is None:
|
||||
return {"ok": False, "output": "not an apt host"}
|
||||
in_file = [s for s in found["sources"] if s["file"] == path]
|
||||
if not in_file:
|
||||
return {"ok": False, "output": f"{path} is none of apt's source files"}
|
||||
moving.extend(s for s in in_file if any(suite_release(x) == release for x in s["suites"]))
|
||||
if not moving:
|
||||
return {"ok": True, "output": f"no source in {path} is on {release} any more"}
|
||||
named = "; ".join(f"{s['uri']} {' '.join(s['suites'])}" for s in moving)
|
||||
return {"ok": True, "output": f"on {release} in {path}: {named}"}
|
||||
|
||||
def _served(
|
||||
self, moving: list[AptSourceDict], release: str, codename: str
|
||||
) -> FixStepOutcomeDict:
|
||||
from napalm_device_types.apt_proxy import apt_fetch_command, parse_apt_fetch
|
||||
|
||||
wanted = sorted(
|
||||
{
|
||||
(s["uri"], _moved(suite, release, codename))
|
||||
for s in moving
|
||||
for suite in s["suites"]
|
||||
if suite_release(suite) == release
|
||||
}
|
||||
)
|
||||
said: list[str] = []
|
||||
for uri, suite in wanted:
|
||||
error = ""
|
||||
for url in _release_urls(uri, suite):
|
||||
fetched = parse_apt_fetch(self._run_apt_command(apt_fetch_command(url)))
|
||||
error = fetched["error"]
|
||||
if not error:
|
||||
said.append(f"{uri} serves {suite}")
|
||||
break
|
||||
else:
|
||||
return {"ok": False, "output": f"{uri} has no {suite}: {error}; nothing changed"}
|
||||
return {"ok": True, "output": "; ".join(said) or "nothing to check"}
|
||||
|
||||
def _rewrite(
|
||||
self,
|
||||
path: str,
|
||||
release: str,
|
||||
codename: str,
|
||||
moving: list[AptSourceDict],
|
||||
rewritten: list[int],
|
||||
) -> FixStepOutcomeDict:
|
||||
if not moving:
|
||||
return {"ok": True, "output": "nothing to rewrite"}
|
||||
encoded = "".join(self._run_apt_command(f"base64 {shlex.quote(path)}").split())
|
||||
text = base64.b64decode(encoded).decode()
|
||||
new, count = rewrite_release(text, release, codename, deb822=_is_deb822(path))
|
||||
if not count:
|
||||
return {"ok": True, "output": f"no source in {path} is on {release} any more"}
|
||||
written = self._write_apt_file(path, new)
|
||||
if not written["ok"]:
|
||||
return written
|
||||
rewritten[0] = count
|
||||
suites = "1 suite" if count == 1 else f"{count} suites"
|
||||
return {"ok": True, "output": f"{path}: {suites} moved from {release} to {codename}"}
|
||||
|
||||
|
||||
__all__ = [
|
||||
"APT_RELEASES_COMMAND",
|
||||
"AptReleaseFixDict",
|
||||
"AptReleaseMixin",
|
||||
"AptReleasesDict",
|
||||
"AptSourceDict",
|
||||
"apt_releases_command",
|
||||
"parse_apt_releases",
|
||||
"rewrite_release",
|
||||
"suite_release",
|
||||
]
|
||||
@@ -0,0 +1,172 @@
|
||||
"""apt's https sources through the package cache, so that the cache stores them.
|
||||
|
||||
A source fetched over https goes through apt-cacher-ng's tunnel, if at all: the
|
||||
TLS runs end to end and nothing of it is stored. The cache fetches
|
||||
``http://HTTPS///<host>/<path>`` from ``https://<host>/<path>`` itself, with the
|
||||
certificate verified, and stores it -- where it allows port 443
|
||||
(``AllowUserPorts=80 443``) (NetOrk/netork#2555). apt fetches that form through
|
||||
its http proxy, so it works only while the cache is that proxy: without one apt
|
||||
looks for a host called ``HTTPS``.
|
||||
|
||||
**Which hosts** (:func:`cacheable_https_hosts`): the hosts of the https sources,
|
||||
on port 443, while apt has an http proxy for every host and none of its own for
|
||||
that one. Not a host with a login in the URL, nor one whose credentials apt keeps
|
||||
in ``auth.conf``: apt finds those by the host's name, which the rewrite changes,
|
||||
and paid content does not belong in a cache other hosts share. Only root reads
|
||||
``auth.conf``, so the rewrite judges that, leaves such a host on https and names
|
||||
it in the proxy file as ``Package-Proxy::Not-Cached``, which apt ignores and
|
||||
``apt-config dump`` shows to the report.
|
||||
|
||||
**The rewrite** (:func:`rewrite_sources_command`) turns ``https://<host>`` into
|
||||
the cache form on the lines that name sources: ``deb`` and ``deb-src`` in a
|
||||
``.list`` file, ``URIs:`` in a deb822 ``.sources`` file. A file without the host
|
||||
is not touched. :func:`revert_sources_command` turns every source back.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import shlex
|
||||
from typing import TYPE_CHECKING
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover - the report's type, without an import cycle
|
||||
from napalm_device_types.apt_proxy import AptProxyReportDict
|
||||
|
||||
#: The prefix apt-cacher-ng fetches over https.
|
||||
VIA_CACHE = "http://HTTPS///"
|
||||
|
||||
#: The key in the proxy file that names the hosts left on https for their credentials.
|
||||
NOT_CACHED_KEY = "Package-Proxy::Not-Cached"
|
||||
|
||||
_LABEL = r"[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?"
|
||||
_HOST = re.compile(rf"^{_LABEL}(?:\.{_LABEL})*$")
|
||||
_NO_PROXY = frozenset({"", "direct"})
|
||||
|
||||
#: Where apt's sources and credentials are, as shell variables; apt's defaults
|
||||
#: when apt-config cannot say.
|
||||
_LOCATE = (
|
||||
'eval "$({apt_config} shell sl Dir::Etc::sourcelist/f sp Dir::Etc::sourceparts/d '
|
||||
'nr Dir::Etc::netrc/f np Dir::Etc::netrcparts/d 2>/dev/null)"; '
|
||||
'sl="${{sl:-/etc/apt/sources.list}}"; sp="${{sp:-/etc/apt/sources.list.d/}}"; '
|
||||
'nr="${{nr:-/etc/apt/auth.conf}}"; np="${{np:-/etc/apt/auth.conf.d/}}"; '
|
||||
)
|
||||
|
||||
#: The lines that name sources: ``deb``/``deb-src`` and deb822's ``URIs:``.
|
||||
_SOURCE_LINES = "/^[[:space:]]*(deb|deb-src|[Uu][Rr][Ii][Ss]:)/"
|
||||
|
||||
#: The hosts apt has credentials for: each word after ``machine``, without its
|
||||
#: scheme, port and path, in lower case.
|
||||
_CREDENTIAL_HOSTS = (
|
||||
'creds=$(cat "$nr" "$np"* 2>/dev/null | tr -s " \\t\\r" "\\n" '
|
||||
"| awk 'p {{ print; p = 0; next }} $0 == \"machine\" {{ p = 1 }}' "
|
||||
"| sed -e 's|^[A-Za-z+]*://||' -e 's|[:/].*||' | tr A-Z a-z); "
|
||||
)
|
||||
|
||||
_REWRITE = (
|
||||
"( " + _LOCATE + _CREDENTIAL_HOSTS + "for h in {hosts}; do "
|
||||
'l=$(printf "%s" "$h" | tr A-Z a-z); '
|
||||
'if printf "%s\\n" "$creds" | grep -qxF -- "$l"; then '
|
||||
'grep -qF -- "\\"$h\\"" {proxy_file} 2>/dev/null || '
|
||||
'printf \'{key}:: "%s";\\n\' "$h" >> {proxy_file} || exit 1; continue; fi; '
|
||||
"e=$(printf '%s' \"$h\" | sed 's/[.]/[.]/g'); "
|
||||
'for f in "$sl" "$sp"*.list "$sp"*.sources; do '
|
||||
'[ -f "$f" ] && grep -qF -- "https://$h" "$f" || continue; '
|
||||
'sed -i -E "{lines} s#([[:space:]:])https://$e([/[:space:]]|\\$)#\\1{via}$h\\2#g" "$f" '
|
||||
"|| exit 1; done; done )"
|
||||
)
|
||||
|
||||
_REVERT = (
|
||||
"( " + _LOCATE + 'for f in "$sl" "$sp"*.list "$sp"*.sources; do '
|
||||
'[ -f "$f" ] && grep -qF -- "{via}" "$f" || continue; '
|
||||
"sed -i -E '{lines} s#([[:space:]:]){via}#\\1https://#g' \"$f\" || exit 1; done )"
|
||||
)
|
||||
|
||||
|
||||
def cache_form(url: str) -> str:
|
||||
"""*url*, an https URL, in the form apt-cacher-ng fetches over https and stores.
|
||||
|
||||
:raises ValueError: if *url* is not an https URL.
|
||||
"""
|
||||
if not url.startswith("https://"):
|
||||
raise ValueError(f"not an https URL: {url!r}")
|
||||
return VIA_CACHE + url[len("https://") :]
|
||||
|
||||
|
||||
def _per_host_direct(report: AptProxyReportDict) -> set[str]:
|
||||
hosts = {**report["per_host"], **report["https_per_host"]}
|
||||
return {h.lower() for h, v in hosts.items() if v.lower() in _NO_PROXY}
|
||||
|
||||
|
||||
def _host_of(source: str) -> str | None:
|
||||
"""The host of an https source the cache can fetch: port 443, no login in the URL."""
|
||||
try:
|
||||
parts = urlsplit(source)
|
||||
port = parts.port
|
||||
except ValueError:
|
||||
return None
|
||||
if parts.username is not None or port not in (None, 443) or not parts.hostname:
|
||||
return None
|
||||
return parts.hostname
|
||||
|
||||
|
||||
def cacheable_https_hosts(report: AptProxyReportDict) -> list[str]:
|
||||
"""The hosts of the https sources the rewrite would send through the cache, sorted.
|
||||
|
||||
None while apt has no http proxy for every host: the cache form goes through it.
|
||||
Not a host apt fetches ``DIRECT``, nor one the rewrite left on https for its
|
||||
credentials (``not_cached``), nor one on another port or with a login in the URL.
|
||||
"""
|
||||
if not report["proxy"]:
|
||||
return []
|
||||
skip = _per_host_direct(report) | {h.lower() for h in report.get("not_cached", [])}
|
||||
hosts = {_host_of(s) for s in report["https_sources"]}
|
||||
return sorted(h for h in hosts if h and h.lower() not in skip)
|
||||
|
||||
|
||||
def is_host_name(host: str) -> bool:
|
||||
"""Whether *host* is a DNS name: letters, digits and hyphens between dots."""
|
||||
return bool(_HOST.match(host))
|
||||
|
||||
|
||||
def _checked(hosts: list[str]) -> str:
|
||||
for host in hosts:
|
||||
if not is_host_name(host):
|
||||
raise ValueError(f"not a host name: {host!r}")
|
||||
return " ".join(hosts)
|
||||
|
||||
|
||||
def rewrite_sources_command(
|
||||
hosts: list[str], proxy_file: str, apt_config: str = "apt-config"
|
||||
) -> str:
|
||||
"""The root command that sends the https sources of *hosts* through the cache.
|
||||
|
||||
A host apt has credentials for stays on https and is named in *proxy_file*.
|
||||
*apt_config* is how apt-config is called; a test points apt's paths elsewhere.
|
||||
|
||||
:raises ValueError: if one of *hosts* is not a host name.
|
||||
"""
|
||||
return _REWRITE.format(
|
||||
apt_config=apt_config,
|
||||
hosts=_checked(hosts),
|
||||
proxy_file=shlex.quote(proxy_file),
|
||||
key=NOT_CACHED_KEY,
|
||||
lines=_SOURCE_LINES,
|
||||
via=VIA_CACHE,
|
||||
)
|
||||
|
||||
|
||||
def revert_sources_command(apt_config: str = "apt-config") -> str:
|
||||
"""The root command that turns every source in the cache form back to https."""
|
||||
return _REVERT.format(apt_config=apt_config, lines=_SOURCE_LINES, via=VIA_CACHE)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"NOT_CACHED_KEY",
|
||||
"VIA_CACHE",
|
||||
"cache_form",
|
||||
"cacheable_https_hosts",
|
||||
"is_host_name",
|
||||
"revert_sources_command",
|
||||
"rewrite_sources_command",
|
||||
]
|
||||
@@ -23,6 +23,22 @@ Mixed in by a driver, not by a role base: a Windows host is an
|
||||
:class:`~napalm_device_types.os.OSDriver` too, and without a shell channel it
|
||||
has no ``dial-stdio`` to offer. ``hasattr(driver, "open_container_engine")``
|
||||
stays truthful.
|
||||
|
||||
**Which engine a CLI is.** An engine is listed when its CLI answers
|
||||
``--version`` with its own name: ``Docker version 29.0.1, build …``, ``podman
|
||||
version 5.2.2``. podman-docker installs a ``docker`` that is podman; listed as
|
||||
Docker it would be read twice, the second time as the login user's rootless
|
||||
Podman (NetOrk/netork#818).
|
||||
|
||||
**Podman as root.** The Podman this reaches is the host's *rootful* one: the
|
||||
engine of the system's containers, whose units Quadlet generates from
|
||||
``/etc/containers/systemd`` and whose API ``podman.socket`` serves at
|
||||
``/run/podman/podman.sock``. Rootless containers belong to the user who runs
|
||||
them, in that user's own storage, and no other login sees them. So the
|
||||
connection to Podman runs every call as root (``privileged``), the way the
|
||||
driver gains root for any command: root itself, or sudo. ``podman system
|
||||
dial-stdio`` exists from Podman 4.0 on; it reaches the API socket, which has
|
||||
to be listening (``systemctl enable --now podman.socket``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -34,8 +50,11 @@ from typing import TYPE_CHECKING, Any, cast
|
||||
from napalm_device_types.channel import ByteStream, CommandResult
|
||||
from napalm_device_types.models import ContainerEngineDict, FixResultDict
|
||||
|
||||
#: Engines this package knows how to reach, and the API each speaks.
|
||||
ENGINES: dict[str, str] = {"docker": "docker-engine"}
|
||||
#: Engines this package knows how to reach, and the API each speaks. Podman's
|
||||
#: is its own service, which serves Docker's API next to libpod's.
|
||||
ENGINES: dict[str, str] = {"docker": "docker-engine", "podman": "podman-libpod"}
|
||||
#: Engines reached as root: rootful Podman's socket is root's (see above).
|
||||
ROOT_ENGINES: frozenset[str] = frozenset({"podman"})
|
||||
|
||||
_PROBE_TIMEOUT = 15
|
||||
_CLI_TIMEOUT = 120
|
||||
@@ -52,17 +71,24 @@ class ContainerEngineConnection:
|
||||
arguments; binary, quoting and the channel are the driver's.
|
||||
"""
|
||||
|
||||
def __init__(self, driver: Any, engine: str, binary: str) -> None:
|
||||
def __init__(self, driver: Any, engine: str, binary: str, *, privileged: bool = False) -> None:
|
||||
self._driver = driver
|
||||
self._binary = binary
|
||||
self.engine = engine
|
||||
#: Every call runs as root: the engine is root's (rootful Podman).
|
||||
self.privileged = privileged
|
||||
|
||||
def _command(self, args: Sequence[str]) -> str:
|
||||
return shlex.join([self._binary, *args])
|
||||
|
||||
def open_api(self) -> ByteStream:
|
||||
"""A stream to the engine's API: HTTP/1.1 over ``system dial-stdio``."""
|
||||
return cast(ByteStream, self._driver.open_stream(self._command(["system", "dial-stdio"])))
|
||||
return cast(
|
||||
ByteStream,
|
||||
self._driver.open_stream(
|
||||
self._command(["system", "dial-stdio"]), privileged=self.privileged
|
||||
),
|
||||
)
|
||||
|
||||
def run_cli(
|
||||
self,
|
||||
@@ -80,7 +106,10 @@ class ContainerEngineConnection:
|
||||
return cast(
|
||||
CommandResult,
|
||||
self._driver.run_command(
|
||||
self._command(args), privileged=privileged, timeout=timeout, stdin=stdin
|
||||
self._command(args),
|
||||
privileged=privileged or self.privileged,
|
||||
timeout=timeout,
|
||||
stdin=stdin,
|
||||
),
|
||||
)
|
||||
|
||||
@@ -96,7 +125,8 @@ class ContainerEngineConnection:
|
||||
return cast(
|
||||
ByteStream,
|
||||
self._driver.open_stream(
|
||||
f"{command} 2>&1" if merge_stderr else command, privileged=privileged
|
||||
f"{command} 2>&1" if merge_stderr else command,
|
||||
privileged=privileged or self.privileged,
|
||||
),
|
||||
)
|
||||
|
||||
@@ -134,20 +164,32 @@ class ContainerEngineMixin:
|
||||
* engine (string) - e.g. "docker"
|
||||
* api (string) - what its API is, e.g. "docker-engine"
|
||||
|
||||
An engine is listed when its CLI exists on the host. Whether the login
|
||||
user may use it is decided by the first API call, which says
|
||||
"permission denied" in the stream's stderr.
|
||||
An engine is listed when its CLI exists on the host and names itself
|
||||
when asked for its version: podman-docker's ``docker`` says it is podman,
|
||||
and is left out. Whether the login user may use the engine is decided by
|
||||
the first API call, which says "permission denied" in the stream's stderr.
|
||||
"""
|
||||
found: list[ContainerEngineDict] = []
|
||||
for engine, api in ENGINES.items():
|
||||
probe = shlex.join(["command", "-v", self._container_engine_binary(engine)])
|
||||
result = self.run_command(probe, timeout=_PROBE_TIMEOUT)
|
||||
if result.exit_code == 0 and result.stdout.strip():
|
||||
found.append({"engine": engine, "api": api})
|
||||
return found
|
||||
return [
|
||||
{"engine": engine, "api": api}
|
||||
for engine, api in ENGINES.items()
|
||||
if self._answers_as(engine)
|
||||
]
|
||||
|
||||
def _answers_as(self, engine: str) -> bool:
|
||||
"""Whether *engine*'s CLI is on the host and says it is *engine*."""
|
||||
binary = shlex.quote(self._container_engine_binary(engine))
|
||||
probe = f"command -v {binary} >/dev/null 2>&1 && {binary} --version"
|
||||
result = self.run_command(probe, timeout=_PROBE_TIMEOUT)
|
||||
words = result.stdout.split()
|
||||
return result.exit_code == 0 and bool(words) and words[0].lower() == engine
|
||||
|
||||
def open_container_engine(self, engine: str) -> ContainerEngineConnection:
|
||||
"""The connection to *engine*; :class:`ContainerEngineUnavailable` if unknown."""
|
||||
if engine not in ENGINES:
|
||||
raise ContainerEngineUnavailable(f"no way to reach container engine {engine!r}")
|
||||
return ContainerEngineConnection(self, engine, self._container_engine_binary(engine))
|
||||
return ContainerEngineConnection(
|
||||
self,
|
||||
engine,
|
||||
self._container_engine_binary(engine),
|
||||
privileged=engine in ROOT_ENGINES,
|
||||
)
|
||||
@@ -1,23 +1,39 @@
|
||||
"""The firmware a vendor offers for a device.
|
||||
"""The firmware a vendor offers for a device, and installing it.
|
||||
|
||||
A fact for the caller (docs/STRUCTURE.md): a driver may know the newer firmware
|
||||
from the vendor's download catalogue or the device's own update check; whether
|
||||
running an older one is a problem is decided above the driver. Comparing the
|
||||
vendor's version strings is vendor knowledge, so the driver says itself whether
|
||||
the offer is newer. Declared only; a driver that can tell implements it.
|
||||
|
||||
Installing it is a capability of its own (``FirmwareUpgradeMixin``): when and
|
||||
whether is the caller's call, as with package updates; how is the driver's.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
from typing import TYPE_CHECKING, NotRequired, TypedDict
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from napalm_device_types.models import FirmwareOfferDict
|
||||
class FirmwareOfferDict(TypedDict):
|
||||
"""The firmware a device runs and the newest its vendor offers
|
||||
(``FirmwareOfferMixin.get_firmware_offer``). A fact, not a judgement."""
|
||||
|
||||
current_version: str
|
||||
latest_version: str
|
||||
newer: bool # the offer is newer than what runs, by the vendor's versioning
|
||||
download_url: NotRequired[str]
|
||||
# The driver can install latest_version itself (FirmwareUpgradeMixin) now; False
|
||||
# while the device lacks what that takes (OpenWrt without owut). Left out where
|
||||
# the driver never installs firmware on this device.
|
||||
installable: NotRequired[bool]
|
||||
# The newest of a later release line (OpenWrt 25.12 on a 24.10 device), which
|
||||
# latest_version does not cross to: shown, never installed by the driver.
|
||||
latest_major_version: NotRequired[str]
|
||||
|
||||
|
||||
class FirmwareOfferMixin:
|
||||
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
|
||||
|
||||
def get_firmware_offer(self) -> FirmwareOfferDict | None:
|
||||
def get_firmware_offer(self) -> "FirmwareOfferDict | None":
|
||||
"""The firmware running and the newest the vendor offers.
|
||||
|
||||
None when the driver cannot tell for this device (an unknown model,
|
||||
@@ -29,3 +45,23 @@ class FirmwareOfferMixin:
|
||||
"newer": True, "download_url": "https://.../GS308E_1.0.2.12.zip"}
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
class FirmwareUpgradeMixin:
|
||||
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
|
||||
|
||||
def start_firmware_upgrade(self, version: str) -> None:
|
||||
"""Starts installing *version*, an offer's ``latest_version`` that
|
||||
:meth:`FirmwareOfferMixin.get_firmware_offer` called ``installable``.
|
||||
|
||||
The device runs the upgrade by itself and restarts at its end, which
|
||||
ends the session; the configuration is kept. Returns once it has
|
||||
started. ``UpdateMixin.get_update_progress`` follows it until then:
|
||||
``"failed"`` when it stopped before the device restarted, which leaves
|
||||
the running firmware as it was.
|
||||
|
||||
:raises ValueError: *version* is not a version string the driver takes.
|
||||
:raises RuntimeError: the device cannot start it (what it takes is
|
||||
missing, an upgrade is already running).
|
||||
"""
|
||||
...
|
||||
@@ -16,6 +16,7 @@ They replace ``run_device_action`` (docs/STRUCTURE.md):
|
||||
``fix_apt_proxy`` :meth:`PackageProxyMixin.configure_package_proxy`
|
||||
``fix_docker_permissions`` ``ContainerEngineMixin.grant_container_engine_access``
|
||||
``install_lldpd`` and co. :meth:`EnsurePackageMixin.ensure_package`
|
||||
(new) :meth:`ZabbixAgentMixin.configure_zabbix_agent`
|
||||
========================== ===================================================
|
||||
"""
|
||||
|
||||
@@ -31,6 +32,7 @@ from napalm_device_types.models import (
|
||||
PackageDict,
|
||||
SnmpAgentSettingsDict,
|
||||
)
|
||||
from napalm_device_types.zabbix_agent import ZabbixAgentSettingsDict, check_zabbix_agent_settings
|
||||
|
||||
#: One step of a fix: its name and the call that does it.
|
||||
FixStep = tuple[str, Callable[[], FixStepOutcomeDict]]
|
||||
@@ -58,6 +60,23 @@ def run_fix_steps(steps: Sequence[FixStep]) -> FixResultDict:
|
||||
}
|
||||
|
||||
|
||||
def enable_and_restart(
|
||||
manage_service: Callable[[str, str], dict[str, Any]], service: str
|
||||
) -> FixStepOutcomeDict:
|
||||
"""Enable *service* at boot, then restart it so it reads its configuration; not
|
||||
restarted when it cannot be enabled."""
|
||||
for action in ("enable", "restart"):
|
||||
result = manage_service(service, action)
|
||||
if not result.get("success"):
|
||||
output = str(result.get("output", "")).strip()
|
||||
return {"ok": False, "output": f"{action} {service}: {output}".rstrip(": ")}
|
||||
return {"ok": True, "output": f"{service} enabled and restarted"}
|
||||
|
||||
|
||||
def _joined(*parts: object) -> str:
|
||||
return "\n".join(str(p).strip() for p in parts if p and str(p).strip())
|
||||
|
||||
|
||||
class SnmpAgentMixin:
|
||||
"""Run an SNMP agent on the device that answers the caller.
|
||||
|
||||
@@ -171,13 +190,7 @@ class LldpAgentMixin:
|
||||
return {"ok": bool(result["success"]), "output": str(result.get("output", ""))}
|
||||
|
||||
def _start_lldp_agent(self) -> FixStepOutcomeDict:
|
||||
service = self.LLDP_AGENT_SERVICE
|
||||
for action in ("enable", "restart"):
|
||||
result = self.manage_service(service, action)
|
||||
if not result.get("success"):
|
||||
output = str(result.get("output", "")).strip()
|
||||
return {"ok": False, "output": f"{action} {service}: {output}".rstrip(": ")}
|
||||
return {"ok": True, "output": f"{service} enabled and restarted"}
|
||||
return enable_and_restart(self.manage_service, self.LLDP_AGENT_SERVICE)
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
|
||||
|
||||
@@ -193,6 +206,110 @@ class LldpAgentMixin:
|
||||
...
|
||||
|
||||
|
||||
class ZabbixAgentMixin:
|
||||
"""Run a native Zabbix agent that answers the caller's server.
|
||||
|
||||
For a host without a container engine, where the agent cannot run as a
|
||||
container. The steps: load the package index where the driver can
|
||||
(``refresh_available_updates``), install the agent from the host's own packages -- and
|
||||
where those have none, from Zabbix's repository for ``settings["release"]``
|
||||
(:func:`~napalm_device_types.zabbix_agent.zabbix_repository_command`) --, write
|
||||
its configuration (:func:`~napalm_device_types.zabbix_agent.zabbix_agent_config`),
|
||||
let the allowed sources through the host's firewall, enable and restart it.
|
||||
Installing and starting are generic, over ``ensure_package`` and
|
||||
``manage_service`` with ``ZABBIX_AGENT_PACKAGE`` and ``ZABBIX_AGENT_SERVICE``.
|
||||
Whether Zabbix reaches it is the caller's to ask Zabbix.
|
||||
|
||||
The settings are checked before anything changes; ``ValueError`` names the one
|
||||
that does not hold. A driver that cannot add a repository sets
|
||||
``_add_zabbix_repository`` to ``None``: its hosts then take the distribution's
|
||||
package or none.
|
||||
"""
|
||||
|
||||
#: The package that brings the agent, as ``ensure_package`` takes it.
|
||||
ZABBIX_AGENT_PACKAGE: ClassVar[str] = "zabbix-agent"
|
||||
#: The agent's service, as ``manage_service`` takes it.
|
||||
ZABBIX_AGENT_SERVICE: ClassVar[str] = "zabbix-agent"
|
||||
|
||||
def configure_zabbix_agent(self, settings: ZabbixAgentSettingsDict) -> FixResultDict:
|
||||
"""Install, configure, open, enable and restart the agent with *settings*.
|
||||
|
||||
:raises ValueError: for settings the agent's configuration cannot take,
|
||||
before anything changes.
|
||||
"""
|
||||
check_zabbix_agent_settings(settings)
|
||||
return run_fix_steps(
|
||||
[
|
||||
("install", lambda: self._install_zabbix_agent(settings)),
|
||||
("configure", lambda: self._write_zabbix_agent_config(settings)),
|
||||
("firewall", lambda: self._allow_zabbix_agent_sources(settings)),
|
||||
("start", self._start_zabbix_agent),
|
||||
]
|
||||
)
|
||||
|
||||
def _install_zabbix_agent(self, settings: ZabbixAgentSettingsDict) -> FixStepOutcomeDict:
|
||||
index = self._load_package_index()
|
||||
first = self.ensure_package(self.ZABBIX_AGENT_PACKAGE)
|
||||
if first["success"]:
|
||||
return {"ok": True, "output": _joined(index, first.get("output"))}
|
||||
release = settings.get("release")
|
||||
add = getattr(self, "_add_zabbix_repository", None)
|
||||
if not release or not callable(add):
|
||||
return {"ok": False, "output": _joined(index, first.get("output"))}
|
||||
added = add(release)
|
||||
if not added["ok"]:
|
||||
return {"ok": False, "output": _joined(index, added.get("output"))}
|
||||
second = self.ensure_package(self.ZABBIX_AGENT_PACKAGE)
|
||||
return {
|
||||
"ok": bool(second["success"]),
|
||||
"output": _joined(index, added.get("output"), second.get("output")),
|
||||
}
|
||||
|
||||
def _load_package_index(self) -> str:
|
||||
"""Load the package index where the driver can (OpenWrt keeps it in RAM, gone
|
||||
after a restart); what went wrong, or "". The install tries either way."""
|
||||
refresh = getattr(self, "refresh_available_updates", None)
|
||||
if not callable(refresh):
|
||||
return ""
|
||||
loaded = refresh()
|
||||
if loaded.get("success"):
|
||||
return ""
|
||||
return f"package index not loaded: {str(loaded.get('output', '')).strip()}"
|
||||
|
||||
def _start_zabbix_agent(self) -> FixStepOutcomeDict:
|
||||
return enable_and_restart(self.manage_service, self.ZABBIX_AGENT_SERVICE)
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover - declared for type checkers only
|
||||
|
||||
def ensure_package(self, name: str) -> FixResultDict: ...
|
||||
|
||||
def manage_service(self, name: str, action: str) -> dict[str, Any]: ...
|
||||
|
||||
def refresh_available_updates(self) -> dict[str, Any]: ...
|
||||
|
||||
# The steps a driver implements:
|
||||
|
||||
def _add_zabbix_repository(self, release: str) -> FixStepOutcomeDict:
|
||||
"""Add Zabbix's repository for *release* and refresh the package index
|
||||
(``zabbix_repository_command`` builds the command for most hosts)."""
|
||||
...
|
||||
|
||||
def _write_zabbix_agent_config(
|
||||
self, settings: ZabbixAgentSettingsDict
|
||||
) -> FixStepOutcomeDict:
|
||||
"""Write *settings* into the agent's configuration file, keeping the rest of
|
||||
it (``zabbix_agent_config``), and whatever else the device needs before the
|
||||
agent may start."""
|
||||
...
|
||||
|
||||
def _allow_zabbix_agent_sources(
|
||||
self, settings: ZabbixAgentSettingsDict
|
||||
) -> FixStepOutcomeDict:
|
||||
"""Let ``settings["allowed_sources"]`` reach tcp/``listen_port``; ``ok`` with
|
||||
nothing to do where the host has no firewall of its own, or for []."""
|
||||
...
|
||||
|
||||
|
||||
class PackageProxyMixin:
|
||||
"""Point the device's package manager at a caching proxy."""
|
||||
|
||||
@@ -215,5 +332,7 @@ __all__ = [
|
||||
"LldpAgentMixin",
|
||||
"PackageProxyMixin",
|
||||
"SnmpAgentMixin",
|
||||
"ZabbixAgentMixin",
|
||||
"enable_and_restart",
|
||||
"run_fix_steps",
|
||||
]
|
||||
@@ -1172,16 +1172,6 @@ class OspfDict(TypedDict):
|
||||
interfaces: dict[str, OspfInterfaceDict] # by interface name
|
||||
|
||||
|
||||
class FirmwareOfferDict(TypedDict):
|
||||
"""The firmware a device runs and the newest its vendor offers
|
||||
(``FirmwareOfferMixin.get_firmware_offer``). A fact, not a judgement."""
|
||||
|
||||
current_version: str
|
||||
latest_version: str
|
||||
newer: bool # the offer is newer than what runs, by the vendor's versioning
|
||||
download_url: NotRequired[str]
|
||||
|
||||
|
||||
class HypervisorHostDict(TypedDict):
|
||||
"""One host of a hypervisor or cluster (``HypervisorDriver.get_hypervisor_hosts``)."""
|
||||
|
||||
@@ -1440,3 +1430,7 @@ class ManagementProtocolsDict(TypedDict):
|
||||
snmp: ProtocolStateDict
|
||||
kvm: ProtocolStateDict
|
||||
virtual_media: ProtocolStateDict
|
||||
|
||||
|
||||
# Importable from here as before 4.41; it lives with its contract now.
|
||||
from napalm_device_types.firmware import FirmwareOfferDict as FirmwareOfferDict # noqa: E402
|
||||
@@ -97,7 +97,8 @@ class UpdateMixin:
|
||||
"""
|
||||
Where an update the device runs by itself stands, for a caller that
|
||||
follows it: a firewall's firmware upgrade, which :meth:`apply_updates`
|
||||
only starts. ``state`` is ``"running"`` until it ends in ``"done"``
|
||||
only starts, or one that ``FirmwareUpgradeMixin.start_firmware_upgrade``
|
||||
started. ``state`` is ``"running"`` until it ends in ``"done"``
|
||||
or ``"failed"``; ``log`` is the device's whole log of it.
|
||||
|
||||
Declared only: a driver whose :meth:`apply_updates` returns when the
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
"""A native Zabbix agent: its configuration and Zabbix's package repository.
|
||||
|
||||
What :class:`~napalm_device_types.fixes.ZabbixAgentMixin` needs that is the same on
|
||||
every host, without I/O: which settings an agent may be given
|
||||
(:func:`check_zabbix_agent_settings`), the agent's configuration file with them
|
||||
(:func:`zabbix_agent_config`), and the command that adds Zabbix's repository where
|
||||
the host's own packages have no agent (:func:`zabbix_repository_command`). A driver
|
||||
reads and writes the file and runs the command.
|
||||
|
||||
**Which lines.** ``Server`` (who may ask, for passive checks), ``ServerActive``
|
||||
(where active checks go), ``ListenPort`` and, when the caller names one,
|
||||
``Hostname`` -- the host's technical name in Zabbix, which an active agent sends.
|
||||
Every other line stays as it was; a key the file does not set is added at its end.
|
||||
|
||||
**The repository.** Zabbix publishes a ``zabbix-release`` package per release line
|
||||
and distribution that adds its repository and key. Since 7.2 it lives under
|
||||
``<release>/release/<distribution>/``; up to 7.0 under ``<release>/<distribution>/``,
|
||||
with the rpm in an ``x86_64`` directory. The release line is the server's: an agent
|
||||
may be older than its server, not newer.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import shlex
|
||||
from typing import NotRequired, TypedDict
|
||||
|
||||
|
||||
class ZabbixAgentSettingsDict(TypedDict):
|
||||
"""What the caller wants a native Zabbix agent to be (``ZabbixAgentMixin``)."""
|
||||
|
||||
server: str # Server: the addresses that may ask (passive checks), comma-separated
|
||||
server_active: str # ServerActive: where active checks go, comma-separated; "" for none
|
||||
listen_port: int # ListenPort
|
||||
# addresses or CIDRs let through to the port; [] changes no firewall
|
||||
allowed_sources: list[str]
|
||||
# Hostname: the host's technical name in Zabbix; absent keeps the agent's
|
||||
hostname: NotRequired[str]
|
||||
# the server's release line ("7.4"): Zabbix's repository to fall back to where the
|
||||
# host's own packages have no agent; absent: no fallback
|
||||
release: NotRequired[str]
|
||||
|
||||
|
||||
#: Addresses, comma-separated, as ``Server`` and ``ServerActive`` take them: names,
|
||||
#: IPv4, ``[IPv6]`` and ports. Nothing a config line or a shell word breaks out of.
|
||||
_ADDRESSES = re.compile(r"^[A-Za-z0-9.:\[\]-]+(,[A-Za-z0-9.:\[\]-]+)*$")
|
||||
#: What Zabbix allows in a technical host name.
|
||||
_HOSTNAME = re.compile(r"^[A-Za-z0-9._ -]+$")
|
||||
#: A release line, as Zabbix's repository names it.
|
||||
_RELEASE = re.compile(r"^(\d+)\.(\d+)$")
|
||||
|
||||
_KEYS = ("Server", "ServerActive", "ListenPort", "Hostname")
|
||||
_LINE = re.compile(r"^(\s*)(" + "|".join(_KEYS) + r")\s*=\s*(.*?)\s*$")
|
||||
|
||||
REPOSITORY = "https://repo.zabbix.com/zabbix"
|
||||
#: The distributions Zabbix publishes packages for, by ``/etc/os-release``'s ``ID``:
|
||||
#: the name in the repository's path, and whether they take a deb.
|
||||
_DISTRIBUTIONS: dict[str, tuple[str, bool]] = {
|
||||
"ubuntu": ("ubuntu", True),
|
||||
"debian": ("debian", True),
|
||||
"raspbian": ("raspbian", True),
|
||||
"rhel": ("rhel", False),
|
||||
"centos": ("rhel", False),
|
||||
"ol": ("oracle", False),
|
||||
"almalinux": ("alma", False),
|
||||
"rocky": ("rocky", False),
|
||||
}
|
||||
|
||||
|
||||
def check_zabbix_agent_settings(settings: ZabbixAgentSettingsDict) -> None:
|
||||
"""Refuse what an agent's configuration or a shell line cannot take.
|
||||
|
||||
:raises ValueError: naming the setting.
|
||||
"""
|
||||
_check_addresses("Server", str(settings.get("server", "")), required=True)
|
||||
_check_addresses("ServerActive", str(settings.get("server_active", "")), required=False)
|
||||
hostname = settings.get("hostname")
|
||||
if hostname is not None and not _HOSTNAME.match(hostname):
|
||||
raise ValueError(f"not a Zabbix host name: {hostname!r}")
|
||||
port = settings.get("listen_port")
|
||||
if not isinstance(port, int) or isinstance(port, bool) or not 1 <= port <= 65535:
|
||||
raise ValueError(f"not a port: {port!r}")
|
||||
release = settings.get("release")
|
||||
if release is not None and not _RELEASE.match(release):
|
||||
raise ValueError(f"not a Zabbix release line (major.minor): {release!r}")
|
||||
|
||||
|
||||
def _check_addresses(key: str, value: str, *, required: bool) -> None:
|
||||
if (value or required) and not _ADDRESSES.match(value):
|
||||
raise ValueError(f"not addresses an agent's {key} takes: {value!r}")
|
||||
|
||||
|
||||
def _wanted(settings: ZabbixAgentSettingsDict) -> dict[str, str]:
|
||||
wanted = {
|
||||
"Server": settings["server"],
|
||||
"ServerActive": settings.get("server_active", ""),
|
||||
"ListenPort": str(settings["listen_port"]),
|
||||
}
|
||||
if settings.get("hostname"):
|
||||
wanted["Hostname"] = str(settings["hostname"])
|
||||
return wanted
|
||||
|
||||
|
||||
def zabbix_agent_config(text: str, settings: ZabbixAgentSettingsDict) -> tuple[str, list[str]]:
|
||||
"""*text*, an agent's configuration file, with *settings*; and what changed, one
|
||||
``"Key: old → new"`` each.
|
||||
|
||||
An empty ``server_active`` comments ``ServerActive`` out: the agent then runs
|
||||
passive checks only.
|
||||
"""
|
||||
wanted = _wanted(settings)
|
||||
lines: list[str] = []
|
||||
changes: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for line in text.splitlines(keepends=True):
|
||||
match = _LINE.match(line.rstrip("\r\n"))
|
||||
if not match or match.group(2) not in wanted:
|
||||
lines.append(line)
|
||||
continue
|
||||
seen.add(match.group(2))
|
||||
lines.append(_set_line(line, match, wanted, changes))
|
||||
_add_missing(
|
||||
lines, [key for key in _KEYS if wanted.get(key) and key not in seen], wanted, changes
|
||||
)
|
||||
return "".join(lines), changes
|
||||
|
||||
|
||||
def _add_missing(
|
||||
lines: list[str], keys: list[str], wanted: dict[str, str], changes: list[str]
|
||||
) -> None:
|
||||
"""*keys*, which the file does not set, at its end."""
|
||||
if keys and lines and not lines[-1].endswith("\n"):
|
||||
lines[-1] += "\n"
|
||||
for key in keys:
|
||||
lines.append(f"{key}={wanted[key]}\n")
|
||||
changes.append(f"{key}: (not set) → {wanted[key]}")
|
||||
|
||||
|
||||
def _set_line(line: str, match: re.Match[str], wanted: dict[str, str], changes: list[str]) -> str:
|
||||
"""*line*, which sets a key the caller wants, as the caller wants it."""
|
||||
indent, key, old = match.groups()
|
||||
end = "\n" if line.endswith("\n") else ""
|
||||
new = wanted[key]
|
||||
if not new:
|
||||
changes.append(f"{key}: {old} → (not set)")
|
||||
return f"{indent}# {key}={old}{end}"
|
||||
if old == new:
|
||||
return line
|
||||
changes.append(f"{key}: {old} → {new}")
|
||||
return f"{indent}{key}={new}{end}"
|
||||
|
||||
|
||||
def _os_release(text: str) -> dict[str, str]:
|
||||
fields = {}
|
||||
for line in text.splitlines():
|
||||
key, sep, value = line.partition("=")
|
||||
if sep:
|
||||
fields[key.strip()] = value.strip().strip("'\"")
|
||||
return fields
|
||||
|
||||
|
||||
def _distribution(fields: dict[str, str]) -> tuple[str, bool]:
|
||||
for candidate in [fields.get("ID", ""), *fields.get("ID_LIKE", "").split()]:
|
||||
if candidate in _DISTRIBUTIONS:
|
||||
return _DISTRIBUTIONS[candidate]
|
||||
raise ValueError(f"Zabbix publishes no repository for {fields.get('ID') or 'this system'}")
|
||||
|
||||
|
||||
def zabbix_repository_command(os_release: str, release: str) -> str:
|
||||
"""The root command that adds Zabbix's *release* repository to the host whose
|
||||
``/etc/os-release`` is *os_release*, and refreshes the package index.
|
||||
|
||||
:raises ValueError: for a release line that is not ``major.minor``, or a
|
||||
distribution Zabbix publishes nothing for.
|
||||
"""
|
||||
match = _RELEASE.match(release)
|
||||
if not match:
|
||||
raise ValueError(f"not a Zabbix release line (major.minor): {release!r}")
|
||||
fields = _os_release(os_release)
|
||||
name, deb = _distribution(fields)
|
||||
version = fields.get("VERSION_ID", "")
|
||||
new_layout = (int(match.group(1)), int(match.group(2))) >= (7, 2)
|
||||
base = f"{REPOSITORY}/{release}/" + ("release/" if new_layout else "") + name
|
||||
if deb:
|
||||
# Raspberry Pi OS's packages are named after the Debian release they follow.
|
||||
label = "debian" if name == "raspbian" else name
|
||||
filename = f"zabbix-release_latest_{release}+{label}{version}_all.deb"
|
||||
url = f"{base}/pool/main/z/zabbix-release/{filename}"
|
||||
package = shlex.quote(url)
|
||||
return (
|
||||
"t=$(mktemp -d) && "
|
||||
f'{{ curl -fsSL -o "$t/zabbix-release.deb" {package} '
|
||||
f'|| wget -qO "$t/zabbix-release.deb" {package}; }} && '
|
||||
'dpkg -i "$t/zabbix-release.deb" && apt-get update; '
|
||||
# The download's directory goes either way; the status is the install's.
|
||||
's=$?; rm -rf "$t"; [ "$s" -eq 0 ]'
|
||||
)
|
||||
major = version.split(".")[0]
|
||||
arch = "noarch" if new_layout else "x86_64"
|
||||
url = f"{base}/{major}/{arch}/zabbix-release-latest-{release}.el{major}.noarch.rpm"
|
||||
return f"dnf -y install {shlex.quote(url)}"
|
||||
|
||||
|
||||
__all__ = [
|
||||
"REPOSITORY",
|
||||
"ZabbixAgentSettingsDict",
|
||||
"check_zabbix_agent_settings",
|
||||
"zabbix_agent_config",
|
||||
"zabbix_repository_command",
|
||||
]
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "napalm-device-types"
|
||||
version = "4.39.0"
|
||||
version = "4.41.0"
|
||||
description = "Abstract device-type base classes for NAPALM drivers"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
@@ -31,7 +31,7 @@
|
||||
},
|
||||
"mypy_relaxed": [],
|
||||
"python_file_lines": {
|
||||
"napalm_device_types/models.py": 1442
|
||||
"napalm_device_types/models.py": 1436
|
||||
},
|
||||
"untyped_public": []
|
||||
}
|
||||
+195
-1
@@ -25,6 +25,7 @@ from napalm_device_types.apt_proxy import (
|
||||
parse_apt_proxy,
|
||||
same_proxy_url,
|
||||
)
|
||||
from napalm_device_types.apt_sources import NOT_CACHED_KEY, VIA_CACHE
|
||||
from napalm_device_types.fixes import PackageProxyMixin
|
||||
|
||||
|
||||
@@ -302,6 +303,13 @@ _CONNECT_DENIED = """Err:1 https://download.docker.com/linux/ubuntu/dists/noble/
|
||||
E: Failed to fetch https://download.docker.com/linux/ubuntu/dists/noble/InRelease Invalid response from proxy: HTTP/1.0 403 CONNECT denied (ask the admin to allow HTTPS tunnels) [IP: 10.40.100.10 3142]
|
||||
E: Download Failed"""
|
||||
|
||||
#: A cache that does not fetch the cache form itself: ``AllowUserPorts`` is 80 alone
|
||||
#: (apt-cacher-ng 3.7.5, NetOrk/netork#2555).
|
||||
_PROHIBITED_PORT = """Err:1 http://HTTPS///download.docker.com/linux/ubuntu/dists/noble/InRelease
|
||||
403 Configuration error (confusing proxy mode) or prohibited port (see AllowUserPorts) [IP: 127.0.0.1 39143]
|
||||
E: Failed to fetch http://HTTPS///download.docker.com/linux/ubuntu/dists/noble/InRelease 403 Configuration error (confusing proxy mode) or prohibited port (see AllowUserPorts) [IP: 127.0.0.1 39143]
|
||||
E: Download Failed"""
|
||||
|
||||
|
||||
def _fetch(output, rc, match=None):
|
||||
lines = ["APROXY_BEGIN", "[fetch]", *output.splitlines(), "[rc]", str(rc)]
|
||||
@@ -422,9 +430,21 @@ class _Host(AptProxyMixin):
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self, files=None, fetch_rc=0, apt=True, https_sources=(), tunnel=True, direct_rc=0
|
||||
self,
|
||||
files=None,
|
||||
fetch_rc=0,
|
||||
apt=True,
|
||||
https_sources=(),
|
||||
tunnel=True,
|
||||
direct_rc=0,
|
||||
cache_https=False,
|
||||
credentials=(),
|
||||
):
|
||||
self.files = dict(files or {})
|
||||
#: Whether the proxy fetches the cache form itself (AllowUserPorts 443), and
|
||||
#: the hosts apt keeps credentials for.
|
||||
self.cache_https = cache_https
|
||||
self.credentials = set(credentials)
|
||||
self.fetch_rc = fetch_rc
|
||||
self.apt = apt
|
||||
self.https_sources = list(https_sources)
|
||||
@@ -459,6 +479,12 @@ class _Host(AptProxyMixin):
|
||||
value, https = self._effective(), self._effective("https")
|
||||
dump = [f'Acquire::http::Proxy "{value}";'] if value else []
|
||||
dump += [f'Acquire::https::Proxy "{https}";'] if https else []
|
||||
dump += [
|
||||
line
|
||||
for path, text in self.files.items()
|
||||
for line in text.splitlines()
|
||||
if line.startswith(NOT_CACHED_KEY)
|
||||
]
|
||||
return _report(
|
||||
dump=dump,
|
||||
parts=[f"{p}:{n}:{t}" for p, n, t in self._settings() if p.startswith(_D)],
|
||||
@@ -466,6 +492,8 @@ class _Host(AptProxyMixin):
|
||||
sources=["http://deb.debian.org/debian/dists/trixie/", *self.https_sources],
|
||||
)
|
||||
self.fetched.append(command)
|
||||
if re.search(r"download-file '?http://HTTPS///", command):
|
||||
return _fetch(_FETCHED, 0) if self.cache_https else _fetch(_PROHIBITED_PORT, 100)
|
||||
if re.search(r"download-file '?https://", command):
|
||||
return self._fetch_https(command)
|
||||
return _fetch(_FETCHED if self.fetch_rc == 0 else _REFUSED, self.fetch_rc, match=1)
|
||||
@@ -482,6 +510,11 @@ class _Host(AptProxyMixin):
|
||||
|
||||
def _run_apt_root_command(self, command, done):
|
||||
self.root.append(command)
|
||||
if NOT_CACHED_KEY in command:
|
||||
return self._rewrite(re.search(r"for h in ([^;]+); do", command).group(1).split(), done)
|
||||
if "#\\1https://#g" in command:
|
||||
self.https_sources = [s.replace(VIA_CACHE, "https://") for s in self.https_sources]
|
||||
return {"ok": True, "output": done}
|
||||
path = command.rsplit(" ", 1)[-1].strip("'")
|
||||
for m in re.finditer(r"(\d+)\{/[^}]*\}", command):
|
||||
n = int(m.group(1))
|
||||
@@ -492,6 +525,19 @@ class _Host(AptProxyMixin):
|
||||
self.files.pop(command.split()[-1].strip("'"), None)
|
||||
return {"ok": True, "output": done}
|
||||
|
||||
def _rewrite(self, hosts, done):
|
||||
for host in hosts:
|
||||
if host in self.credentials:
|
||||
proxy = self.files.get(f"{_D}00proxy", "")
|
||||
self.files[f"{_D}00proxy"] = proxy + f'{NOT_CACHED_KEY}:: "{host}";\n'
|
||||
continue
|
||||
prefix = f"https://{host}/"
|
||||
self.https_sources = [
|
||||
VIA_CACHE + s[len("https://") :] if s.startswith(prefix) else s
|
||||
for s in self.https_sources
|
||||
]
|
||||
return {"ok": True, "output": done}
|
||||
|
||||
|
||||
class TestCheckHttpsSource:
|
||||
def test_one_source_the_way_apt_fetches_it(self):
|
||||
@@ -569,8 +615,10 @@ class TestConfigurePackageProxy:
|
||||
"conflicts",
|
||||
"verify",
|
||||
"fetch",
|
||||
"cache",
|
||||
"https",
|
||||
]
|
||||
assert "no https source" in _step(result, "cache")["output"]
|
||||
assert "no https source" in result["steps"][-1]["output"]
|
||||
assert host.files[f"{_D}00proxy"] == 'Acquire::http::Proxy "http://c:3142";\n'
|
||||
assert "InRelease" in host.fetched[0] and "DIRECT" not in host.fetched[0]
|
||||
@@ -766,6 +814,144 @@ class TestConfigurePackageProxy:
|
||||
assert host.files == {}
|
||||
|
||||
|
||||
class TestCacheHttpsSources:
|
||||
"""NetOrk/netork#2555: the cache stores an https source only in its own form."""
|
||||
|
||||
def test_a_cache_that_fetches_https_gets_them(self):
|
||||
host = _Host(https_sources=[_DOCKER], cache_https=True)
|
||||
|
||||
result = host.configure_package_proxy("http://c:3142")
|
||||
|
||||
assert result["success"] is True
|
||||
assert host.https_sources == [f"{VIA_CACHE}download.docker.com/linux/ubuntu/dists/noble/"]
|
||||
assert _step(result, "cache")["output"] == (
|
||||
"https sources cached through http://c:3142: download.docker.com"
|
||||
)
|
||||
assert any(VIA_CACHE in c for c in host.fetched)
|
||||
assert "no https source goes through" in _step(result, "https")["output"]
|
||||
|
||||
def test_a_cache_that_does_not_turns_them_back(self):
|
||||
host = _Host(https_sources=[_DOCKER])
|
||||
|
||||
result = host.configure_package_proxy("http://c:3142")
|
||||
|
||||
assert result["success"] is True
|
||||
assert host.https_sources == [_DOCKER]
|
||||
output = _step(result, "cache")["output"]
|
||||
assert "http://c:3142 does not fetch https sources itself" in output
|
||||
assert "prohibited port (see AllowUserPorts)" in output
|
||||
assert "they stay on https, not cached" in output
|
||||
assert _step(result, "https")["output"] == (
|
||||
f"fetched {_DOCKER}InRelease through http://c:3142"
|
||||
)
|
||||
|
||||
def test_a_host_with_credentials_stays_on_https_and_is_named(self):
|
||||
pve = "https://enterprise.proxmox.com/debian/pve/dists/trixie/"
|
||||
host = _Host(
|
||||
https_sources=[pve, _DOCKER],
|
||||
cache_https=True,
|
||||
credentials={"enterprise.proxmox.com"},
|
||||
)
|
||||
|
||||
result = host.configure_package_proxy("http://c:3142")
|
||||
|
||||
assert result["success"] is True
|
||||
assert pve in host.https_sources
|
||||
output = _step(result, "cache")["output"]
|
||||
assert "cached through http://c:3142: download.docker.com" in output
|
||||
assert "left on https, apt has credentials for them: enterprise.proxmox.com" in output
|
||||
assert host.get_package_proxy_report()["not_cached"] == ["enterprise.proxmox.com"]
|
||||
|
||||
def test_only_hosts_with_credentials(self):
|
||||
pve = "https://enterprise.proxmox.com/debian/pve/dists/trixie/"
|
||||
host = _Host(https_sources=[pve], cache_https=True, credentials={"enterprise.proxmox.com"})
|
||||
|
||||
result = host.configure_package_proxy("http://c:3142")
|
||||
|
||||
assert _step(result, "cache")["output"] == (
|
||||
"no https source to cache; "
|
||||
"left on https, apt has credentials for them: enterprise.proxmox.com"
|
||||
)
|
||||
|
||||
def test_the_https_direct_fallback_keeps_the_hosts_left_on_https(self):
|
||||
pve = "https://enterprise.proxmox.com/debian/pve/dists/trixie/"
|
||||
host = _Host(https_sources=[pve], tunnel=False, credentials={"enterprise.proxmox.com"})
|
||||
|
||||
result = host.configure_package_proxy("http://c:3142")
|
||||
|
||||
assert result["success"] is True
|
||||
assert host.files[f"{_D}00proxy"] == (
|
||||
'Acquire::http::Proxy "http://c:3142";\n'
|
||||
'Acquire::https::Proxy "DIRECT";\n'
|
||||
f'{NOT_CACHED_KEY}:: "enterprise.proxmox.com";\n'
|
||||
)
|
||||
|
||||
def test_a_rewrite_that_fails_stops_the_fix(self):
|
||||
host = _Host(https_sources=[_DOCKER], cache_https=True)
|
||||
real = host._run_apt_root_command
|
||||
|
||||
def denied(command, done):
|
||||
if NOT_CACHED_KEY in command:
|
||||
return {"ok": False, "output": "sed: denied"}
|
||||
return real(command, done)
|
||||
|
||||
host._run_apt_root_command = denied
|
||||
|
||||
result = host.configure_package_proxy("http://c:3142")
|
||||
|
||||
assert result["success"] is False
|
||||
assert result["steps"][-1] == {"step": "cache", "ok": False, "output": "sed: denied"}
|
||||
|
||||
def test_a_turn_back_that_fails_says_so(self):
|
||||
host = _Host(https_sources=[_DOCKER])
|
||||
real = host._run_apt_root_command
|
||||
|
||||
def denied(command, done):
|
||||
if "#\\1https://#g" in command:
|
||||
return {"ok": False, "output": "sed: denied"}
|
||||
return real(command, done)
|
||||
|
||||
host._run_apt_root_command = denied
|
||||
|
||||
result = host.configure_package_proxy("http://c:3142")
|
||||
|
||||
assert result["success"] is False
|
||||
output = result["steps"][-1]["output"]
|
||||
assert "turning the sources back to https failed: sed: denied" in output
|
||||
|
||||
def test_the_proof_of_the_proxy_is_a_plain_http_source(self):
|
||||
"""A source already in the cache form says nothing about plain http."""
|
||||
host = _Host(https_sources=[f"{VIA_CACHE}download.docker.com/linux/ubuntu/dists/noble/"])
|
||||
|
||||
host.configure_package_proxy("http://c:3142")
|
||||
|
||||
fetch = next(c for c in host.fetched if "InRelease" in c)
|
||||
assert "deb.debian.org" in fetch
|
||||
|
||||
def test_removing_the_proxy_turns_the_sources_back_first(self):
|
||||
host = _Host(
|
||||
{f"{_D}00proxy": 'Acquire::http::Proxy "http://c:3142";\n'},
|
||||
https_sources=[f"{VIA_CACHE}download.docker.com/linux/ubuntu/dists/noble/"],
|
||||
)
|
||||
|
||||
result = host.configure_package_proxy("")
|
||||
|
||||
assert result["success"] is True
|
||||
assert [s["step"] for s in result["steps"]] == ["sources", "configure"]
|
||||
assert host.https_sources == [_DOCKER]
|
||||
assert host.files == {}
|
||||
|
||||
def test_a_turn_back_that_fails_keeps_the_proxy(self):
|
||||
"""Without the cache apt looks for a host called HTTPS: the proxy stays."""
|
||||
host = _Host({f"{_D}00proxy": 'Acquire::http::Proxy "http://c:3142";\n'})
|
||||
host._run_apt_root_command = lambda command, done: {"ok": False, "output": "denied"}
|
||||
|
||||
result = host.configure_package_proxy("")
|
||||
|
||||
assert result["success"] is False
|
||||
assert f"{_D}00proxy" in host.files
|
||||
|
||||
|
||||
class TestWhatTheParserSkips:
|
||||
def test_a_line_that_is_no_grep_output(self):
|
||||
assert (
|
||||
@@ -824,3 +1010,11 @@ def test_the_comment_out_command_for_https():
|
||||
assert comment_out_command("/etc/apt/apt.conf", [3], scheme="https") == (
|
||||
"sed -i -e '3{/^[[:space:]]*Acquire::https::Proxy[[:space:]]/Is|^|// |}' /etc/apt/apt.conf"
|
||||
)
|
||||
|
||||
|
||||
def test_a_line_left_on_https_that_is_no_host_name_is_not_written_back():
|
||||
host = _Host()
|
||||
host._write_package_proxy("http://c:3142", not_cached=["ok.example", 'x"; Foo "1'])
|
||||
assert host.files[f"{_D}00proxy"] == (
|
||||
f'Acquire::http::Proxy "http://c:3142";\n{NOT_CACHED_KEY}:: "ok.example";\n'
|
||||
)
|
||||
@@ -0,0 +1,393 @@
|
||||
"""apt's sources and the release they name, and moving one file's sources to the host's.
|
||||
|
||||
A release upgrade keeps third-party sources on the old release: z2m-house, Ubuntu
|
||||
24.04 (noble), kept Docker's source on ``focal`` and never saw a Docker newer than
|
||||
the last focal build (NetOrk/netork#2614). The read command runs here for real, on
|
||||
files in a temporary directory that ``apt-config -o`` points apt's settings at.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
import pytest
|
||||
|
||||
from napalm_device_types.apt_proxy import AptProxyMixin
|
||||
from napalm_device_types.apt_releases import (
|
||||
AptReleaseMixin,
|
||||
apt_releases_command,
|
||||
parse_apt_releases,
|
||||
rewrite_release,
|
||||
suite_release,
|
||||
)
|
||||
|
||||
_APT_CONFIG = shutil.which("apt-config")
|
||||
needs_apt = pytest.mark.skipif(_APT_CONFIG is None, reason="apt-config is not on this host")
|
||||
|
||||
_NOBLE = (
|
||||
'PRETTY_NAME="Ubuntu 24.04.3 LTS"\nNAME="Ubuntu"\nVERSION_ID="24.04"\n'
|
||||
"VERSION_CODENAME=noble\nID=ubuntu\nID_LIKE=debian\nUBUNTU_CODENAME=noble\n"
|
||||
)
|
||||
_TRIXIE = 'PRETTY_NAME="Debian GNU/Linux 13 (trixie)"\nVERSION_CODENAME=trixie\nID=debian\n'
|
||||
_MINT = (
|
||||
'NAME="Linux Mint"\nVERSION_CODENAME=virginia\nID=linuxmint\nID_LIKE="ubuntu debian"\n'
|
||||
"UBUNTU_CODENAME=jammy\n"
|
||||
)
|
||||
|
||||
#: z2m-house's Docker source, as found: through the cache, on focal (#2614).
|
||||
_Z2M_DOCKER = "deb http://HTTPS///download.docker.com/linux/ubuntu focal stable\n"
|
||||
_Z2M_DOCKER_FILE = "archive_uri-https_download_docker_com_linux_ubuntu-noble.list"
|
||||
_UBUNTU_SOURCES = (
|
||||
"Types: deb\n"
|
||||
"URIs: http://archive.ubuntu.com/ubuntu/\n"
|
||||
"Suites: noble noble-updates noble-backports\n"
|
||||
"Components: main restricted universe multiverse\n"
|
||||
"Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n"
|
||||
"\n"
|
||||
"Types: deb\n"
|
||||
"URIs: http://security.ubuntu.com/ubuntu/\n"
|
||||
"Suites: noble-security\n"
|
||||
"Components: main restricted universe multiverse\n"
|
||||
)
|
||||
|
||||
|
||||
def _framed(os_release: str, files: dict[str, str]) -> str:
|
||||
"""What :func:`apt_releases_command` prints for *os_release* and *files*."""
|
||||
lines = ["ARELEASE_BEGIN", "@@os"]
|
||||
lines += [f"|{line}" for line in os_release.splitlines()]
|
||||
for path, text in files.items():
|
||||
lines.append(f"@@file {path}")
|
||||
lines += [f"|{line}" for line in text.splitlines()]
|
||||
lines.append("ARELEASE_END")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
class TestSuiteRelease:
|
||||
@pytest.mark.parametrize(
|
||||
("suite", "release"),
|
||||
[
|
||||
("noble", "noble"),
|
||||
("noble-security", "noble"),
|
||||
("noble-updates", "noble"),
|
||||
("focal-backports", "focal"),
|
||||
("bookworm-proposed-updates", "bookworm"),
|
||||
("trixie-security", "trixie"),
|
||||
],
|
||||
)
|
||||
def test_the_release_a_suite_names(self, suite, release):
|
||||
assert suite_release(suite) == release
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("suite", "word"),
|
||||
[("stable", "stable"), ("nodistro", "nodistro"), ("pve-no-subscription", "pve")],
|
||||
)
|
||||
def test_a_word_the_caller_judges(self, suite, word):
|
||||
# Which words are releases is the caller's to know: "stable" has the form too.
|
||||
assert suite_release(suite) == word
|
||||
|
||||
@pytest.mark.parametrize("suite", ["/", "./", "focal/", "Focal", "$(ARCH)", ""])
|
||||
def test_a_suite_of_another_form(self, suite):
|
||||
# A flat repository's path, or anything not a lower-case word, names none.
|
||||
assert suite_release(suite) is None
|
||||
|
||||
|
||||
class TestParse:
|
||||
def test_the_host_and_a_list_source(self):
|
||||
found = parse_apt_releases(
|
||||
_framed(_NOBLE, {f"/etc/apt/sources.list.d/{_Z2M_DOCKER_FILE}": _Z2M_DOCKER})
|
||||
)
|
||||
assert found["os_id"] == "ubuntu"
|
||||
assert found["os_like"] == ["debian"]
|
||||
assert found["codename"] == "noble"
|
||||
assert found["ubuntu_codename"] == "noble"
|
||||
assert found["sources"] == [
|
||||
{
|
||||
"file": f"/etc/apt/sources.list.d/{_Z2M_DOCKER_FILE}",
|
||||
"line": 1,
|
||||
"types": ["deb"],
|
||||
"uri": "http://HTTPS///download.docker.com/linux/ubuntu",
|
||||
"suites": ["focal"],
|
||||
"components": ["stable"],
|
||||
}
|
||||
]
|
||||
|
||||
def test_options_comments_and_disabled_lines(self):
|
||||
text = (
|
||||
"# deb http://old.example/ubuntu bionic main\n"
|
||||
"\n"
|
||||
"deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] "
|
||||
"https://download.docker.com/linux/ubuntu jammy stable # moved by hand\n"
|
||||
"deb-src [ arch=amd64 ] http://archive.ubuntu.com/ubuntu noble main\n"
|
||||
"deb https://repo.example/flat ./\n"
|
||||
)
|
||||
found = parse_apt_releases(_framed(_NOBLE, {"/etc/apt/sources.list": text}))
|
||||
assert [
|
||||
(s["line"], s["types"], s["uri"], s["suites"], s["components"])
|
||||
for s in found["sources"]
|
||||
] == [
|
||||
(3, ["deb"], "https://download.docker.com/linux/ubuntu", ["jammy"], ["stable"]),
|
||||
(4, ["deb-src"], "http://archive.ubuntu.com/ubuntu", ["noble"], ["main"]),
|
||||
(5, ["deb"], "https://repo.example/flat", ["./"], []),
|
||||
]
|
||||
|
||||
def test_deb822_paragraphs_one_per_uri(self):
|
||||
text = (
|
||||
"Types: deb deb-src\n"
|
||||
"URIs: http://a.example/ubuntu http://b.example/ubuntu\n"
|
||||
"Suites: focal\n"
|
||||
" focal-updates\n"
|
||||
"Components: main\n"
|
||||
"\n"
|
||||
"# a comment between paragraphs\n"
|
||||
"Enabled: no\n"
|
||||
"Types: deb\n"
|
||||
"URIs: http://c.example/ubuntu\n"
|
||||
"Suites: bionic\n"
|
||||
"Components: main\n"
|
||||
)
|
||||
found = parse_apt_releases(_framed(_NOBLE, {"/etc/apt/sources.list.d/x.sources": text}))
|
||||
assert [(s["line"], s["types"], s["uri"], s["suites"]) for s in found["sources"]] == [
|
||||
(3, ["deb", "deb-src"], "http://a.example/ubuntu", ["focal", "focal-updates"]),
|
||||
(3, ["deb", "deb-src"], "http://b.example/ubuntu", ["focal", "focal-updates"]),
|
||||
]
|
||||
|
||||
def test_debian_and_a_derivative(self):
|
||||
assert parse_apt_releases(_framed(_TRIXIE, {}))["codename"] == "trixie"
|
||||
mint = parse_apt_releases(_framed(_MINT, {}))
|
||||
assert (mint["os_id"], mint["os_like"], mint["codename"], mint["ubuntu_codename"]) == (
|
||||
"linuxmint",
|
||||
["ubuntu", "debian"],
|
||||
"virginia",
|
||||
"jammy",
|
||||
)
|
||||
|
||||
def test_without_os_release(self):
|
||||
found = parse_apt_releases("ARELEASE_BEGIN\n@@os\nARELEASE_END\n")
|
||||
assert (found["os_id"], found["codename"], found["ubuntu_codename"]) == ("", "", "")
|
||||
|
||||
def test_an_unreadable_file_is_left_out(self):
|
||||
out = _framed(_NOBLE, {}).replace(
|
||||
"ARELEASE_END", "@@unreadable /etc/apt/sources.list\nARELEASE_END"
|
||||
)
|
||||
assert parse_apt_releases(out)["sources"] == []
|
||||
|
||||
def test_no_intact_report(self):
|
||||
with pytest.raises(ValueError):
|
||||
parse_apt_releases("ARELEASE_BEGIN\n@@os\n")
|
||||
|
||||
|
||||
@needs_apt
|
||||
class TestTheCommand:
|
||||
def test_reads_list_and_sources_files_as_apt_names_them(self, tmp_path):
|
||||
(tmp_path / "sources.list.d").mkdir()
|
||||
(tmp_path / "sources.list").write_text("")
|
||||
(tmp_path / "sources.list.d" / _Z2M_DOCKER_FILE).write_text(_Z2M_DOCKER)
|
||||
(tmp_path / "sources.list.d" / "ubuntu.sources").write_text(_UBUNTU_SOURCES)
|
||||
(tmp_path / "sources.list.d" / "old.list.distUpgrade").write_text(_Z2M_DOCKER)
|
||||
(tmp_path / "os-release").write_text(_NOBLE)
|
||||
command = apt_releases_command(
|
||||
apt_config=(
|
||||
f"apt-config -o Dir::Etc::sourcelist={tmp_path}/sources.list "
|
||||
f"-o Dir::Etc::sourceparts={tmp_path}/sources.list.d/"
|
||||
),
|
||||
os_release=str(tmp_path / "os-release"),
|
||||
)
|
||||
done = subprocess.run(["sh", "-c", command], capture_output=True, text=True, check=False)
|
||||
found = parse_apt_releases(done.stdout)
|
||||
assert found["codename"] == "noble"
|
||||
assert [(s["file"].rsplit("/", 1)[-1], s["suites"]) for s in found["sources"]] == [
|
||||
(_Z2M_DOCKER_FILE, ["focal"]),
|
||||
("ubuntu.sources", ["noble", "noble-updates", "noble-backports"]),
|
||||
("ubuntu.sources", ["noble-security"]),
|
||||
]
|
||||
|
||||
|
||||
class TestRewriteRelease:
|
||||
def test_a_list_line(self):
|
||||
assert rewrite_release(_Z2M_DOCKER, "focal", "noble", deb822=False) == (
|
||||
"deb http://HTTPS///download.docker.com/linux/ubuntu noble stable\n",
|
||||
1,
|
||||
)
|
||||
|
||||
def test_suffixes_options_and_a_uri_that_names_the_release(self):
|
||||
text = (
|
||||
"deb [signed-by=/k.asc] http://repo.example/focal/ focal-security main # focal\n"
|
||||
"# deb http://repo.example/focal/ focal main\n"
|
||||
"deb http://repo.example/other noble main\n"
|
||||
"deb http://repo.example/flat ./\n"
|
||||
)
|
||||
assert rewrite_release(text, "focal", "noble", deb822=False) == (
|
||||
"deb [signed-by=/k.asc] http://repo.example/focal/ noble-security main # focal\n"
|
||||
"# deb http://repo.example/focal/ focal main\n"
|
||||
"deb http://repo.example/other noble main\n"
|
||||
"deb http://repo.example/flat ./\n",
|
||||
1,
|
||||
)
|
||||
|
||||
def test_a_release_that_is_only_a_prefix_is_not_it(self):
|
||||
text = "deb http://repo.example/x focalfossa main\n"
|
||||
assert rewrite_release(text, "focal", "noble", deb822=False) == (text, 0)
|
||||
|
||||
def test_deb822_suites_and_continuations_of_enabled_paragraphs(self):
|
||||
text = (
|
||||
"Types: deb\n"
|
||||
"URIs: http://a.example/focal\n"
|
||||
"Suites: focal focal-updates\n"
|
||||
" focal-backports\n"
|
||||
"Components: main\n"
|
||||
"\n"
|
||||
"Enabled: no\n"
|
||||
"Types: deb\n"
|
||||
"URIs: http://b.example/ubuntu\n"
|
||||
"Suites: focal\n"
|
||||
"Components: main\n"
|
||||
)
|
||||
new, changed = rewrite_release(text, "focal", "noble", deb822=True)
|
||||
assert changed == 3
|
||||
assert new == (
|
||||
"Types: deb\n"
|
||||
"URIs: http://a.example/focal\n"
|
||||
"Suites: noble noble-updates\n"
|
||||
" noble-backports\n"
|
||||
"Components: main\n"
|
||||
"\n"
|
||||
"Enabled: no\n"
|
||||
"Types: deb\n"
|
||||
"URIs: http://b.example/ubuntu\n"
|
||||
"Suites: focal\n"
|
||||
"Components: main\n"
|
||||
)
|
||||
|
||||
def test_the_text_without_a_final_newline_stays_so(self):
|
||||
assert rewrite_release("deb http://a/ focal main", "focal", "noble", deb822=False) == (
|
||||
"deb http://a/ noble main",
|
||||
1,
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize("bad", ["", "Noble", "no ble", "noble;", "../x"])
|
||||
def test_a_name_that_is_no_codename(self, bad):
|
||||
with pytest.raises(ValueError):
|
||||
rewrite_release(_Z2M_DOCKER, bad, "noble", deb822=False)
|
||||
with pytest.raises(ValueError):
|
||||
rewrite_release(_Z2M_DOCKER, "focal", bad, deb822=False)
|
||||
|
||||
|
||||
_OK = "[fetch]\nGet:1 x\n[rc]\n0\n"
|
||||
_NOT_FOUND = "[fetch]\nE: Failed to fetch {url} 404 Not Found [IP: 1.2.3.4 80]\n[rc]\n100\n"
|
||||
|
||||
|
||||
class _Host(AptReleaseMixin):
|
||||
"""An apt host whose files are a dict and whose fetches answer from a set."""
|
||||
|
||||
def __init__(self, files, os_release=_NOBLE, served=(), apt=True):
|
||||
self.files = dict(files)
|
||||
self.os_release = os_release
|
||||
self.served = set(served)
|
||||
self.apt = apt
|
||||
self.fetched: list[str] = []
|
||||
self.written: list[str] = []
|
||||
|
||||
def _is_apt_host(self):
|
||||
return self.apt
|
||||
|
||||
def _run_apt_command(self, command):
|
||||
if "ARELEASE_" in command:
|
||||
return _framed(self.os_release, self.files)
|
||||
if command.startswith("base64 "):
|
||||
path = command.split(" ", 1)[1].strip("'")
|
||||
return base64.b64encode(self.files[path].encode()).decode() + "\n"
|
||||
url = command.split("download-file ", 1)[1].split()[0].strip("'")
|
||||
self.fetched.append(url)
|
||||
body = _OK if url in self.served else _NOT_FOUND.format(url=url)
|
||||
return f"APROXY_BEGIN\n{body}APROXY_END\n"
|
||||
|
||||
def _write_apt_file(self, path, content):
|
||||
self.written.append(path)
|
||||
self.files[path] = content
|
||||
return {"ok": True, "output": f"wrote {path}"}
|
||||
|
||||
|
||||
_DOCKER_PATH = f"/etc/apt/sources.list.d/{_Z2M_DOCKER_FILE}"
|
||||
_DOCKER_NOBLE = "http://HTTPS///download.docker.com/linux/ubuntu/dists/noble/InRelease"
|
||||
|
||||
|
||||
class TestTheMixin:
|
||||
def test_reads_the_host(self):
|
||||
found = _Host({_DOCKER_PATH: _Z2M_DOCKER}).get_apt_source_releases()
|
||||
assert found is not None
|
||||
assert found["codename"] == "noble"
|
||||
assert found["sources"][0]["suites"] == ["focal"]
|
||||
|
||||
def test_none_without_apt(self):
|
||||
assert _Host({}, apt=False).get_apt_source_releases() is None
|
||||
|
||||
def test_moves_the_file_once_the_vendor_serves_the_release(self):
|
||||
host = _Host({_DOCKER_PATH: _Z2M_DOCKER}, served={_DOCKER_NOBLE})
|
||||
result = host.set_apt_source_release(_DOCKER_PATH, "focal", "noble")
|
||||
assert result["success"], result["output"]
|
||||
assert host.files[_DOCKER_PATH] == (
|
||||
"deb http://HTTPS///download.docker.com/linux/ubuntu noble stable\n"
|
||||
)
|
||||
assert host.fetched == [_DOCKER_NOBLE]
|
||||
assert result["rewritten"] == 1
|
||||
assert [s["step"] for s in result["steps"]] == ["read", "check", "rewrite"]
|
||||
|
||||
def test_the_release_file_when_there_is_no_inrelease(self):
|
||||
release = _DOCKER_NOBLE.replace("InRelease", "Release")
|
||||
host = _Host({_DOCKER_PATH: _Z2M_DOCKER}, served={release})
|
||||
assert host.set_apt_source_release(_DOCKER_PATH, "focal", "noble")["success"]
|
||||
assert host.fetched == [_DOCKER_NOBLE, release]
|
||||
|
||||
def test_refuses_where_the_vendor_has_no_such_release(self):
|
||||
host = _Host({_DOCKER_PATH: _Z2M_DOCKER})
|
||||
result = host.set_apt_source_release(_DOCKER_PATH, "focal", "noble")
|
||||
assert not result["success"]
|
||||
assert host.written == []
|
||||
assert "noble" in result["output"] and "404" in result["output"]
|
||||
assert result["rewritten"] == 0
|
||||
|
||||
def test_one_suite_missing_leaves_the_whole_file(self):
|
||||
text = "deb http://a.example/u focal main\ndeb http://a.example/u focal-security main\n"
|
||||
served = {"http://a.example/u/dists/noble/InRelease"}
|
||||
host = _Host({"/etc/apt/sources.list": text}, served=served)
|
||||
result = host.set_apt_source_release("/etc/apt/sources.list", "focal", "noble")
|
||||
assert not result["success"]
|
||||
assert host.files["/etc/apt/sources.list"] == text
|
||||
|
||||
def test_a_file_apt_does_not_read_is_never_written(self):
|
||||
host = _Host({_DOCKER_PATH: _Z2M_DOCKER}, served={_DOCKER_NOBLE})
|
||||
result = host.set_apt_source_release("/etc/shadow", "focal", "noble")
|
||||
assert not result["success"]
|
||||
assert host.written == []
|
||||
|
||||
def test_nothing_left_on_that_release(self):
|
||||
host = _Host({_DOCKER_PATH: _Z2M_DOCKER.replace("focal", "noble")})
|
||||
result = host.set_apt_source_release(_DOCKER_PATH, "focal", "noble")
|
||||
assert result["success"]
|
||||
assert result["rewritten"] == 0
|
||||
assert host.written == [] and host.fetched == []
|
||||
|
||||
def test_a_deb822_file(self):
|
||||
path = "/etc/apt/sources.list.d/docker.sources"
|
||||
text = "Types: deb\nURIs: https://download.docker.com/linux/ubuntu\nSuites: focal\nComponents: stable\n"
|
||||
served = {"https://download.docker.com/linux/ubuntu/dists/noble/InRelease"}
|
||||
host = _Host({path: text}, served=served)
|
||||
assert host.set_apt_source_release(path, "focal", "noble")["success"]
|
||||
assert "Suites: noble\n" in host.files[path]
|
||||
|
||||
def test_not_an_apt_host(self):
|
||||
result = _Host({}, apt=False).set_apt_source_release(_DOCKER_PATH, "focal", "noble")
|
||||
assert not result["success"]
|
||||
|
||||
def test_a_name_that_is_no_codename_changes_nothing(self):
|
||||
host = _Host({_DOCKER_PATH: _Z2M_DOCKER}, served={_DOCKER_NOBLE})
|
||||
result = host.set_apt_source_release(_DOCKER_PATH, "focal", "noble; rm -rf /")
|
||||
assert not result["success"]
|
||||
assert host.written == [] and host.fetched == []
|
||||
|
||||
|
||||
def test_every_apt_proxy_host_reads_its_releases():
|
||||
# Every driver with apt's proxy is an apt host: it reads its sources' releases too.
|
||||
assert issubclass(AptProxyMixin, AptReleaseMixin)
|
||||
@@ -0,0 +1,318 @@
|
||||
"""https sources through the cache: which can be, and the root commands that turn them.
|
||||
|
||||
apt-cacher-ng fetches ``http://HTTPS///<host>/<path>`` from ``https://<host>/<path>``
|
||||
and stores it, where it allows port 443 (``AllowUserPorts=80 443``); a source
|
||||
fetched over https itself goes through the cache's tunnel and is never stored
|
||||
(NetOrk/netork#2555). The commands run here for real, on files in a temporary
|
||||
directory that ``apt-config -o`` points apt's own settings at.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
import pytest
|
||||
|
||||
from napalm_device_types.apt_proxy import parse_apt_proxy
|
||||
from napalm_device_types.apt_sources import (
|
||||
VIA_CACHE,
|
||||
cache_form,
|
||||
cacheable_https_hosts,
|
||||
is_host_name,
|
||||
revert_sources_command,
|
||||
rewrite_sources_command,
|
||||
)
|
||||
|
||||
|
||||
def _report(**over):
|
||||
report = {
|
||||
"proxy": "http://c:3142",
|
||||
"per_host": {},
|
||||
"auto_detect": None,
|
||||
"settings": [],
|
||||
"https_proxy": "http://c:3142",
|
||||
"https_per_host": {},
|
||||
"https_settings": [],
|
||||
"http_sources": ["http://deb.debian.org/debian/dists/trixie/"],
|
||||
"https_sources": [],
|
||||
"not_cached": [],
|
||||
}
|
||||
report.update(over)
|
||||
return report
|
||||
|
||||
|
||||
class TestCacheForm:
|
||||
def test_an_https_url(self):
|
||||
assert cache_form("https://download.docker.com/linux/ubuntu/dists/noble/") == (
|
||||
"http://HTTPS///download.docker.com/linux/ubuntu/dists/noble/"
|
||||
)
|
||||
|
||||
def test_the_prefix(self):
|
||||
assert VIA_CACHE == "http://HTTPS///"
|
||||
|
||||
def test_not_an_https_url(self):
|
||||
with pytest.raises(ValueError):
|
||||
cache_form("http://deb.debian.org/debian/")
|
||||
|
||||
|
||||
class TestCacheableHttpsHosts:
|
||||
def test_each_host_once_sorted(self):
|
||||
report = _report(
|
||||
https_sources=[
|
||||
"https://packages.wazuh.com/4.x/apt/dists/stable/",
|
||||
"https://download.docker.com/linux/ubuntu/dists/noble/",
|
||||
"https://download.docker.com/linux/ubuntu/dists/noble-test/",
|
||||
]
|
||||
)
|
||||
assert cacheable_https_hosts(report) == ["download.docker.com", "packages.wazuh.com"]
|
||||
|
||||
def test_none_without_an_http_proxy(self):
|
||||
# The cache form goes through the http proxy: without one it goes nowhere.
|
||||
report = _report(proxy=None, https_sources=["https://h.example/dists/x/"])
|
||||
assert cacheable_https_hosts(report) == []
|
||||
|
||||
def test_a_host_apt_fetches_direct_stays(self):
|
||||
report = _report(
|
||||
per_host={"a.example": "DIRECT"},
|
||||
https_per_host={"B.example": "direct"},
|
||||
https_sources=[
|
||||
"https://a.example/dists/x/",
|
||||
"https://b.example/dists/x/",
|
||||
"https://c.example/dists/x/",
|
||||
],
|
||||
)
|
||||
assert cacheable_https_hosts(report) == ["c.example"]
|
||||
|
||||
def test_a_host_with_credentials_stays(self):
|
||||
report = _report(
|
||||
https_sources=["https://enterprise.proxmox.com/debian/pve/dists/trixie/"],
|
||||
not_cached=["enterprise.proxmox.com"],
|
||||
)
|
||||
assert cacheable_https_hosts(report) == []
|
||||
|
||||
def test_another_port_or_a_login_in_the_url_stays(self):
|
||||
# The cache allows port 443 alone, and a login in the URL is a credential too.
|
||||
report = _report(
|
||||
https_sources=[
|
||||
"https://h.example:8443/dists/x/",
|
||||
"https://u:p@i.example/dists/x/",
|
||||
"https://j.example:443/dists/x/",
|
||||
]
|
||||
)
|
||||
assert cacheable_https_hosts(report) == ["j.example"]
|
||||
|
||||
def test_a_report_from_before_not_cached(self):
|
||||
report = _report(https_sources=["https://h.example/dists/x/"])
|
||||
del report["not_cached"]
|
||||
assert cacheable_https_hosts(report) == ["h.example"]
|
||||
|
||||
|
||||
class TestNotCachedInTheReport:
|
||||
def test_the_dump_names_them(self):
|
||||
output = "\n".join(
|
||||
[
|
||||
"APROXY_BEGIN",
|
||||
"[dump]",
|
||||
'Acquire::http::Proxy "http://c:3142";',
|
||||
'Package-Proxy::Not-Cached "";',
|
||||
'Package-Proxy::Not-Cached:: "enterprise.proxmox.com";',
|
||||
'Package-Proxy::Not-Cached:: "esm.ubuntu.com";',
|
||||
"[sources]",
|
||||
"APROXY_END",
|
||||
]
|
||||
)
|
||||
assert parse_apt_proxy(output)["not_cached"] == [
|
||||
"enterprise.proxmox.com",
|
||||
"esm.ubuntu.com",
|
||||
]
|
||||
|
||||
def test_none(self):
|
||||
assert parse_apt_proxy("APROXY_BEGIN\n[dump]\nAPROXY_END")["not_cached"] == []
|
||||
|
||||
|
||||
def test_the_hosts_are_checked():
|
||||
with pytest.raises(ValueError):
|
||||
rewrite_sources_command(["h.example; rm -rf /"], "/etc/apt/apt.conf.d/00proxy")
|
||||
|
||||
|
||||
_APT_CONFIG = shutil.which("apt-config")
|
||||
needs_apt = pytest.mark.skipif(_APT_CONFIG is None, reason="apt-config is not on this host")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def etc(tmp_path):
|
||||
"""apt's sources, its credentials and the proxy file, under *tmp_path*."""
|
||||
(tmp_path / "sources.list.d").mkdir()
|
||||
(tmp_path / "auth.conf.d").mkdir()
|
||||
(tmp_path / "00proxy").write_text('Acquire::http::Proxy "http://c:3142";\n')
|
||||
return tmp_path
|
||||
|
||||
|
||||
def _apt_config(etc):
|
||||
return (
|
||||
f"apt-config -o Dir::Etc::sourcelist={etc}/sources.list "
|
||||
f"-o Dir::Etc::sourceparts={etc}/sources.list.d/ "
|
||||
f"-o Dir::Etc::netrc={etc}/auth.conf -o Dir::Etc::netrcparts={etc}/auth.conf.d/"
|
||||
)
|
||||
|
||||
|
||||
def _run(command):
|
||||
done = subprocess.run(["sh", "-c", command], capture_output=True, text=True, check=False)
|
||||
assert done.returncode == 0, done.stderr
|
||||
return done.stdout
|
||||
|
||||
|
||||
_DOCKER_LIST = (
|
||||
"deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] "
|
||||
"https://download.docker.com/linux/ubuntu noble stable\n"
|
||||
)
|
||||
_DOCKER_SOURCES = (
|
||||
"Types: deb\n"
|
||||
"URIs: https://download.docker.com/linux/ubuntu\n"
|
||||
"Suites: noble\n"
|
||||
"Components: stable\n"
|
||||
"Signed-By: /etc/apt/keyrings/docker.asc\n"
|
||||
)
|
||||
|
||||
|
||||
@needs_apt
|
||||
class TestRewrite:
|
||||
def test_a_one_line_source(self, etc):
|
||||
(etc / "sources.list.d" / "docker.list").write_text(_DOCKER_LIST)
|
||||
|
||||
_run(rewrite_sources_command(["download.docker.com"], f"{etc}/00proxy", _apt_config(etc)))
|
||||
|
||||
assert (etc / "sources.list.d" / "docker.list").read_text() == (
|
||||
"deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] "
|
||||
"http://HTTPS///download.docker.com/linux/ubuntu noble stable\n"
|
||||
)
|
||||
|
||||
def test_a_deb822_source(self, etc):
|
||||
(etc / "sources.list.d" / "docker.sources").write_text(_DOCKER_SOURCES)
|
||||
|
||||
_run(rewrite_sources_command(["download.docker.com"], f"{etc}/00proxy", _apt_config(etc)))
|
||||
|
||||
assert (etc / "sources.list.d" / "docker.sources").read_text() == (
|
||||
_DOCKER_SOURCES.replace("https://", "http://HTTPS///")
|
||||
)
|
||||
|
||||
def test_sources_list_itself_and_only_the_hosts_named(self, etc):
|
||||
(etc / "sources.list").write_text(
|
||||
"deb http://deb.debian.org/debian trixie main\n"
|
||||
"deb https://a.example/debian trixie main\n"
|
||||
"deb-src https://a.example/debian trixie main\n"
|
||||
"deb https://b.example/debian trixie main\n"
|
||||
"# deb https://a.example/old trixie main\n"
|
||||
)
|
||||
|
||||
_run(rewrite_sources_command(["a.example"], f"{etc}/00proxy", _apt_config(etc)))
|
||||
|
||||
assert (etc / "sources.list").read_text() == (
|
||||
"deb http://deb.debian.org/debian trixie main\n"
|
||||
"deb http://HTTPS///a.example/debian trixie main\n"
|
||||
"deb-src http://HTTPS///a.example/debian trixie main\n"
|
||||
"deb https://b.example/debian trixie main\n"
|
||||
"# deb https://a.example/old trixie main\n"
|
||||
)
|
||||
|
||||
def test_a_longer_name_or_another_port_stays(self, etc):
|
||||
text = (
|
||||
"deb https://a.example.org/debian trixie main\n"
|
||||
"deb https://a.example:8443/debian trixie main\n"
|
||||
)
|
||||
(etc / "sources.list").write_text(text)
|
||||
|
||||
_run(rewrite_sources_command(["a.example"], f"{etc}/00proxy", _apt_config(etc)))
|
||||
|
||||
assert (etc / "sources.list").read_text() == text
|
||||
|
||||
def test_a_host_with_credentials_stays_and_is_named(self, etc):
|
||||
(etc / "sources.list.d" / "pve.sources").write_text(
|
||||
"Types: deb\nURIs: https://enterprise.proxmox.com/debian/pve\nSuites: trixie\n"
|
||||
)
|
||||
(etc / "sources.list.d" / "docker.list").write_text(_DOCKER_LIST)
|
||||
(etc / "auth.conf.d" / "pve.conf").write_text(
|
||||
"machine https://Enterprise.Proxmox.com/debian/pve\nlogin key\npassword secret\n"
|
||||
)
|
||||
|
||||
_run(
|
||||
rewrite_sources_command(
|
||||
["enterprise.proxmox.com", "download.docker.com"],
|
||||
f"{etc}/00proxy",
|
||||
_apt_config(etc),
|
||||
)
|
||||
)
|
||||
|
||||
assert "https://enterprise.proxmox.com" in (etc / "sources.list.d/pve.sources").read_text()
|
||||
assert "HTTPS///download.docker.com" in (etc / "sources.list.d/docker.list").read_text()
|
||||
assert (etc / "00proxy").read_text() == (
|
||||
'Acquire::http::Proxy "http://c:3142";\n'
|
||||
'Package-Proxy::Not-Cached:: "enterprise.proxmox.com";\n'
|
||||
)
|
||||
assert "secret" not in (etc / "00proxy").read_text()
|
||||
|
||||
def test_credentials_in_auth_conf_on_one_line(self, etc):
|
||||
(etc / "sources.list").write_text("deb https://esm.ubuntu.com/apps/ubuntu noble main\n")
|
||||
(etc / "auth.conf").write_text("machine esm.ubuntu.com/apps/ubuntu/ login a password b\n")
|
||||
|
||||
_run(rewrite_sources_command(["esm.ubuntu.com"], f"{etc}/00proxy", _apt_config(etc)))
|
||||
|
||||
assert "https://esm.ubuntu.com" in (etc / "sources.list").read_text()
|
||||
assert "esm.ubuntu.com" in (etc / "00proxy").read_text()
|
||||
|
||||
def test_a_file_without_the_host_is_left_untouched(self, etc):
|
||||
other = etc / "sources.list.d" / "other.list"
|
||||
other.write_text("deb http://deb.debian.org/debian trixie main\n")
|
||||
before = other.stat().st_ino # sed -i writes a new file; grep first leaves it be
|
||||
|
||||
_run(rewrite_sources_command(["a.example"], f"{etc}/00proxy", _apt_config(etc)))
|
||||
|
||||
assert other.stat().st_ino == before
|
||||
|
||||
def test_twice_is_once(self, etc):
|
||||
(etc / "sources.list.d" / "docker.list").write_text(_DOCKER_LIST)
|
||||
command = rewrite_sources_command(
|
||||
["download.docker.com"], f"{etc}/00proxy", _apt_config(etc)
|
||||
)
|
||||
|
||||
_run(command)
|
||||
_run(command)
|
||||
|
||||
assert (etc / "sources.list.d" / "docker.list").read_text().count("HTTPS///") == 1
|
||||
|
||||
|
||||
@needs_apt
|
||||
class TestRevert:
|
||||
def test_every_source_goes_back_to_https(self, etc):
|
||||
(etc / "sources.list.d" / "docker.list").write_text(_DOCKER_LIST)
|
||||
(etc / "sources.list.d" / "docker.sources").write_text(_DOCKER_SOURCES)
|
||||
_run(rewrite_sources_command(["download.docker.com"], f"{etc}/00proxy", _apt_config(etc)))
|
||||
|
||||
_run(revert_sources_command(_apt_config(etc)))
|
||||
|
||||
assert (etc / "sources.list.d" / "docker.list").read_text() == _DOCKER_LIST
|
||||
assert (etc / "sources.list.d" / "docker.sources").read_text() == _DOCKER_SOURCES
|
||||
|
||||
def test_nothing_to_revert(self, etc):
|
||||
(etc / "sources.list").write_text("deb http://deb.debian.org/debian trixie main\n")
|
||||
|
||||
_run(revert_sources_command(_apt_config(etc)))
|
||||
|
||||
assert (
|
||||
etc / "sources.list"
|
||||
).read_text() == "deb http://deb.debian.org/debian trixie main\n"
|
||||
|
||||
|
||||
def test_the_commands_are_valid_posix_sh():
|
||||
for command in (
|
||||
rewrite_sources_command(["a.example"], "/etc/apt/apt.conf.d/00proxy"),
|
||||
revert_sources_command(),
|
||||
):
|
||||
assert subprocess.run(["sh", "-n", "-c", command], check=False).returncode == 0
|
||||
|
||||
|
||||
def test_a_host_name():
|
||||
assert is_host_name("download.docker.com")
|
||||
assert not is_host_name('h"; Acquire::x "y')
|
||||
assert not is_host_name("-h.example")
|
||||
@@ -19,30 +19,40 @@ from napalm_device_types.container_engine import (
|
||||
ContainerEngineUnavailable,
|
||||
)
|
||||
|
||||
#: What each engine's CLI says to ``--version``.
|
||||
VERSIONS = {
|
||||
"docker": "Docker version 29.0.1, build 1a2b3c4",
|
||||
"podman": "podman version 5.2.2",
|
||||
}
|
||||
|
||||
|
||||
class FakeDriver(ContainerEngineMixin):
|
||||
"""A driver with a channel: records what it is asked to run."""
|
||||
"""A driver with a channel: records what it is asked to run.
|
||||
|
||||
def __init__(self, present=("docker",), binary=None):
|
||||
*present* are the engines whose CLI is installed; *says* overrides what an
|
||||
engine's CLI answers to ``--version`` (podman-docker's ``docker`` is podman).
|
||||
"""
|
||||
|
||||
def __init__(self, present=("docker",), binary=None, says=None):
|
||||
self.present = set(present)
|
||||
self.binary = binary
|
||||
self.says = dict(says or {})
|
||||
self.commands = []
|
||||
self.streams = []
|
||||
|
||||
def run_command(self, command, *, privileged=False, timeout=60, stdin=None):
|
||||
self.commands.append((command, privileged, timeout, stdin))
|
||||
found = any(command == f"command -v {b}" for b in self._binaries())
|
||||
return CommandResult(
|
||||
stdout="/usr/bin/x\n" if found else "", stderr="", exit_code=0 if found else 1
|
||||
)
|
||||
for engine in self.present:
|
||||
b = self._container_engine_binary(engine)
|
||||
if command == f"command -v {b} >/dev/null 2>&1 && {b} --version":
|
||||
answer = self.says.get(engine, VERSIONS[engine])
|
||||
return CommandResult(stdout=answer + "\n", stderr="", exit_code=0)
|
||||
return CommandResult(stdout="", stderr="", exit_code=1)
|
||||
|
||||
def open_stream(self, command, *, privileged=False):
|
||||
self.streams.append((command, privileged))
|
||||
return object()
|
||||
|
||||
def _binaries(self):
|
||||
return {self._container_engine_binary(e) for e in self.present}
|
||||
|
||||
def _container_engine_binary(self, engine):
|
||||
return self.binary or super()._container_engine_binary(engine)
|
||||
|
||||
@@ -123,6 +133,60 @@ def test_an_unknown_engine_is_refused():
|
||||
FakeDriver().open_container_engine("rkt")
|
||||
|
||||
|
||||
class TestPodman:
|
||||
"""Podman, reached the way Docker is (NetOrk/netork#818).
|
||||
|
||||
Its API is Podman's own service, which also speaks Docker's: netOrk reads the
|
||||
host's *rootful* Podman, the engine of the system's containers (Quadlet units
|
||||
in ``/etc/containers/systemd``, ``podman.socket``). A login user's rootless
|
||||
containers are that user's own and invisible to anybody else, so the
|
||||
connection to Podman runs as root, the way the driver gains root for any command.
|
||||
"""
|
||||
|
||||
def test_a_host_with_podman_lists_it_with_its_api(self):
|
||||
driver = FakeDriver(present=("podman",))
|
||||
|
||||
assert driver.container_engines() == [{"engine": "podman", "api": "podman-libpod"}]
|
||||
|
||||
def test_a_host_with_both_lists_both(self):
|
||||
driver = FakeDriver(present=("docker", "podman"))
|
||||
|
||||
assert driver.container_engines() == [
|
||||
{"engine": "docker", "api": "docker-engine"},
|
||||
{"engine": "podman", "api": "podman-libpod"},
|
||||
]
|
||||
|
||||
def test_podman_docker_is_not_docker(self):
|
||||
"""podman-docker installs a ``docker`` that runs podman: listing it as
|
||||
Docker would read the login user's rootless Podman a second time."""
|
||||
driver = FakeDriver(present=("docker", "podman"), says={"docker": VERSIONS["podman"]})
|
||||
|
||||
assert driver.container_engines() == [{"engine": "podman", "api": "podman-libpod"}]
|
||||
|
||||
def test_the_api_stream_runs_as_root(self):
|
||||
driver = FakeDriver(present=("podman",))
|
||||
|
||||
driver.open_container_engine("podman").open_api()
|
||||
|
||||
assert driver.streams == [("podman system dial-stdio", True)]
|
||||
|
||||
def test_the_cli_runs_as_root_too(self):
|
||||
driver = FakeDriver(present=("podman",))
|
||||
conn = driver.open_container_engine("podman")
|
||||
|
||||
conn.run_cli(["pull", "nginx"])
|
||||
conn.stream_cli(["pull", "nginx"])
|
||||
|
||||
assert driver.commands[-1][:2] == ("podman pull nginx", True)
|
||||
assert driver.streams == [("podman pull nginx", True)]
|
||||
|
||||
def test_the_connection_says_it_runs_as_root(self):
|
||||
driver = FakeDriver(present=("docker", "podman"))
|
||||
|
||||
assert driver.open_container_engine("podman").privileged is True
|
||||
assert driver.open_container_engine("docker").privileged is False
|
||||
|
||||
|
||||
def test_the_connection_names_its_engine():
|
||||
conn = FakeDriver().open_container_engine("docker")
|
||||
|
||||
|
||||
@@ -56,8 +56,26 @@ def test_a_firmware_offer_says_what_runs_and_what_is_offered():
|
||||
from napalm_device_types import FirmwareOfferDict, FirmwareOfferMixin
|
||||
|
||||
hints = get_type_hints(FirmwareOfferDict)
|
||||
assert set(hints) == {"current_version", "latest_version", "newer", "download_url"}
|
||||
assert set(hints) == {
|
||||
"current_version",
|
||||
"latest_version",
|
||||
"newer",
|
||||
"download_url",
|
||||
"installable",
|
||||
"latest_major_version",
|
||||
}
|
||||
assert FirmwareOfferDict.__required_keys__ == frozenset(
|
||||
{"current_version", "latest_version", "newer"}
|
||||
)
|
||||
assert not hasattr(FirmwareOfferMixin, "get_firmware_offer")
|
||||
|
||||
|
||||
def test_a_firmware_upgrade_is_started_and_followed_like_a_device_run_update():
|
||||
"""A driver that can install the firmware it offers starts it and the device
|
||||
runs it by itself, restarting at its end (OpenWrt's sysupgrade, NetOrk/netork#2615);
|
||||
the caller follows it through ``get_update_progress``. Declared only, so
|
||||
``hasattr`` tells a caller whether the driver offers it."""
|
||||
from napalm_device_types import FirmwareUpgradeMixin, UpdateMixin
|
||||
|
||||
assert not hasattr(FirmwareUpgradeMixin, "start_firmware_upgrade")
|
||||
assert not hasattr(UpdateMixin, "get_update_progress")
|
||||
@@ -259,6 +259,7 @@ class TestLldpAgent:
|
||||
"ensure_package",
|
||||
"configure_package_proxy",
|
||||
"grant_container_engine_access",
|
||||
"configure_zabbix_agent",
|
||||
],
|
||||
)
|
||||
def test_no_role_base_offers_a_fix(base, method):
|
||||
|
||||
@@ -0,0 +1,360 @@
|
||||
"""A native Zabbix agent: its configuration, its repository and the fix template
|
||||
(``ZabbixAgentMixin``, NetOrk/netork#2571).
|
||||
|
||||
Where a host has no container engine the application installs the agent from the
|
||||
host's own packages, and from Zabbix's repository only where those have none.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from napalm_device_types import (
|
||||
ZabbixAgentMixin,
|
||||
check_zabbix_agent_settings,
|
||||
zabbix_agent_config,
|
||||
zabbix_repository_command,
|
||||
)
|
||||
|
||||
SETTINGS = {
|
||||
"server": "zabbix.example.com",
|
||||
"server_active": "zabbix.example.com",
|
||||
"listen_port": 10050,
|
||||
"allowed_sources": ["10.40.100.254"],
|
||||
"hostname": "ap-1og",
|
||||
"release": "7.4",
|
||||
}
|
||||
|
||||
SAMPLE = """# This is a configuration file for Zabbix agent daemon (Unix)
|
||||
LogFile=/tmp/zabbix_agentd.log
|
||||
|
||||
### Option: Server
|
||||
# Server=
|
||||
|
||||
Server=127.0.0.1
|
||||
ServerActive=127.0.0.1
|
||||
Hostname=Zabbix server
|
||||
# Include=/etc/zabbix_agentd.conf.d/*.conf
|
||||
"""
|
||||
|
||||
|
||||
class TestConfig:
|
||||
def test_the_lines_that_say_who_may_ask_get_the_caller_s_values(self):
|
||||
text, changes = zabbix_agent_config(SAMPLE, SETTINGS)
|
||||
|
||||
assert "\nServer=zabbix.example.com\n" in text
|
||||
assert "\nServerActive=zabbix.example.com\n" in text
|
||||
assert "\nHostname=ap-1og\n" in text
|
||||
assert "Server: 127.0.0.1 → zabbix.example.com" in changes
|
||||
assert "Hostname: Zabbix server → ap-1og" in changes
|
||||
|
||||
def test_comments_and_other_lines_stay_as_they_were(self):
|
||||
text, _ = zabbix_agent_config(SAMPLE, SETTINGS)
|
||||
|
||||
assert text.startswith("# This is a configuration file")
|
||||
assert "LogFile=/tmp/zabbix_agentd.log\n" in text
|
||||
assert "# Server=\n" in text
|
||||
assert text.endswith("# Include=/etc/zabbix_agentd.conf.d/*.conf\n" + "ListenPort=10050\n")
|
||||
|
||||
def test_a_key_the_file_does_not_set_is_added_at_the_end(self):
|
||||
text, changes = zabbix_agent_config(SAMPLE, SETTINGS)
|
||||
|
||||
assert text.endswith("ListenPort=10050\n")
|
||||
assert "ListenPort: (not set) → 10050" in changes
|
||||
|
||||
def test_a_file_that_already_says_it_changes_nothing(self):
|
||||
text, _ = zabbix_agent_config(SAMPLE, SETTINGS)
|
||||
|
||||
again, changes = zabbix_agent_config(text, SETTINGS)
|
||||
|
||||
assert again == text
|
||||
assert changes == []
|
||||
|
||||
def test_without_a_hostname_the_agent_keeps_its_own(self):
|
||||
settings = {k: v for k, v in SETTINGS.items() if k != "hostname"}
|
||||
|
||||
text, changes = zabbix_agent_config(SAMPLE, settings)
|
||||
|
||||
assert "\nHostname=Zabbix server\n" in text
|
||||
assert not any(c.startswith("Hostname") for c in changes)
|
||||
|
||||
def test_no_active_server_comments_the_line_out(self):
|
||||
text, changes = zabbix_agent_config(SAMPLE, {**SETTINGS, "server_active": ""})
|
||||
|
||||
assert "\n# ServerActive=127.0.0.1\n" in text
|
||||
assert "\nServerActive=" not in text
|
||||
assert "ServerActive: 127.0.0.1 → (not set)" in changes
|
||||
|
||||
def test_a_file_without_a_final_newline_gets_one_before_what_is_added(self):
|
||||
text, _ = zabbix_agent_config("Server=127.0.0.1", SETTINGS)
|
||||
|
||||
assert text.startswith("Server=zabbix.example.com\n")
|
||||
assert text.endswith("\n")
|
||||
|
||||
def test_every_line_of_a_key_set_twice_says_the_same(self):
|
||||
text, _ = zabbix_agent_config("Server=a\nServer=b\n", SETTINGS)
|
||||
|
||||
assert text.count("Server=zabbix.example.com\n") == 2
|
||||
|
||||
|
||||
class TestSettings:
|
||||
@pytest.mark.parametrize(
|
||||
"server",
|
||||
["", "zabbix.example.com\nUserParameter=x,id", "a b", "a;rm -rf /", "'a'"],
|
||||
)
|
||||
def test_a_server_that_is_not_addresses_is_refused(self, server):
|
||||
with pytest.raises(ValueError):
|
||||
check_zabbix_agent_settings({**SETTINGS, "server": server})
|
||||
|
||||
def test_several_servers_and_ports_pass(self):
|
||||
check_zabbix_agent_settings(
|
||||
{**SETTINGS, "server": "10.0.0.1,zabbix.example.com", "server_active": "[::1]:10051"}
|
||||
)
|
||||
|
||||
def test_a_hostname_with_a_newline_is_refused(self):
|
||||
with pytest.raises(ValueError):
|
||||
check_zabbix_agent_settings({**SETTINGS, "hostname": "ap\nServer=evil"})
|
||||
|
||||
@pytest.mark.parametrize("port", [0, 65536, "10050"])
|
||||
def test_a_listen_port_out_of_range_is_refused(self, port):
|
||||
with pytest.raises(ValueError):
|
||||
check_zabbix_agent_settings({**SETTINGS, "listen_port": port})
|
||||
|
||||
@pytest.mark.parametrize("release", ["7", "7.4.1", "latest", "7.4; id"])
|
||||
def test_a_release_that_is_not_major_dot_minor_is_refused(self, release):
|
||||
with pytest.raises(ValueError):
|
||||
check_zabbix_agent_settings({**SETTINGS, "release": release})
|
||||
|
||||
|
||||
def _os_release(**fields: str) -> str:
|
||||
return "".join(f'{key}="{value}"\n' for key, value in fields.items())
|
||||
|
||||
|
||||
class TestRepository:
|
||||
BASE = "https://repo.zabbix.com/zabbix"
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("os_release", "url"),
|
||||
[
|
||||
(
|
||||
_os_release(ID="ubuntu", VERSION_ID="24.04", ID_LIKE="debian"),
|
||||
"7.4/release/ubuntu/pool/main/z/zabbix-release/"
|
||||
"zabbix-release_latest_7.4+ubuntu24.04_all.deb",
|
||||
),
|
||||
(
|
||||
_os_release(ID="debian", VERSION_ID="12"),
|
||||
"7.4/release/debian/pool/main/z/zabbix-release/"
|
||||
"zabbix-release_latest_7.4+debian12_all.deb",
|
||||
),
|
||||
(
|
||||
_os_release(ID="ol", VERSION_ID="9.4", ID_LIKE="fedora"),
|
||||
"7.4/release/oracle/9/noarch/zabbix-release-latest-7.4.el9.noarch.rpm",
|
||||
),
|
||||
(
|
||||
_os_release(ID="rhel", VERSION_ID="10.0", ID_LIKE="fedora"),
|
||||
"7.4/release/rhel/10/noarch/zabbix-release-latest-7.4.el10.noarch.rpm",
|
||||
),
|
||||
(
|
||||
_os_release(ID="almalinux", VERSION_ID="9.5", ID_LIKE="rhel centos fedora"),
|
||||
"7.4/release/alma/9/noarch/zabbix-release-latest-7.4.el9.noarch.rpm",
|
||||
),
|
||||
(
|
||||
_os_release(ID="rocky", VERSION_ID="9.5", ID_LIKE="rhel centos fedora"),
|
||||
"7.4/release/rocky/9/noarch/zabbix-release-latest-7.4.el9.noarch.rpm",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_the_release_package_of_the_host_s_distribution(self, os_release, url):
|
||||
command = zabbix_repository_command(os_release, "7.4")
|
||||
|
||||
assert f"{self.BASE}/{url}" in command
|
||||
|
||||
def test_a_release_before_7_2_lives_in_the_old_layout(self):
|
||||
deb = zabbix_repository_command(_os_release(ID="ubuntu", VERSION_ID="24.04"), "7.0")
|
||||
rpm = zabbix_repository_command(_os_release(ID="ol", VERSION_ID="9.4"), "7.0")
|
||||
|
||||
assert f"{self.BASE}/7.0/ubuntu/pool/main/z/zabbix-release/" in deb
|
||||
assert f"{self.BASE}/7.0/oracle/9/x86_64/zabbix-release-latest-7.0.el9.noarch.rpm" in rpm
|
||||
|
||||
def test_a_deb_is_installed_with_dpkg_and_the_index_refreshed(self):
|
||||
command = zabbix_repository_command(_os_release(ID="debian", VERSION_ID="12"), "7.4")
|
||||
|
||||
assert "dpkg -i" in command
|
||||
assert "apt-get update" in command
|
||||
|
||||
def test_an_rpm_is_installed_with_the_package_manager(self):
|
||||
command = zabbix_repository_command(_os_release(ID="rocky", VERSION_ID="9.5"), "7.4")
|
||||
|
||||
assert "dnf -y install" in command
|
||||
|
||||
def test_a_derivative_falls_back_to_what_it_is_like(self):
|
||||
command = zabbix_repository_command(
|
||||
_os_release(ID="centos", VERSION_ID="9", ID_LIKE="rhel fedora"), "7.4"
|
||||
)
|
||||
|
||||
assert "/7.4/release/rhel/9/noarch/" in command
|
||||
|
||||
def test_a_distribution_zabbix_has_no_repository_for_is_refused(self):
|
||||
with pytest.raises(ValueError, match="openwrt"):
|
||||
zabbix_repository_command(_os_release(ID="openwrt", VERSION_ID="25.12.2"), "7.4")
|
||||
|
||||
def test_a_release_that_is_not_major_dot_minor_is_refused(self):
|
||||
with pytest.raises(ValueError):
|
||||
zabbix_repository_command(_os_release(ID="debian", VERSION_ID="12"), "7.4;id")
|
||||
|
||||
|
||||
class _Agent(ZabbixAgentMixin):
|
||||
"""The generic steps over ensure_package and manage_service; the driver's others."""
|
||||
|
||||
def __init__(self, refuse: frozenset[str] = frozenset(), repository: bool = True):
|
||||
self.calls: list = []
|
||||
self.refuse = refuse
|
||||
self.installs = 0
|
||||
if not repository:
|
||||
self._add_zabbix_repository = None # type: ignore[assignment]
|
||||
|
||||
def refresh_available_updates(self):
|
||||
self.calls.append(("refresh",))
|
||||
ok = "refresh" not in self.refuse
|
||||
return {"success": ok, "output": "" if ok else "Temporary failure resolving"}
|
||||
|
||||
def ensure_package(self, name):
|
||||
self.calls.append(("ensure_package", name))
|
||||
self.installs += 1
|
||||
# The distribution has no package; after the repository is added it has.
|
||||
ok = "distribution" not in self.refuse or self.installs > 1
|
||||
return {"success": ok, "steps": [], "output": "installed" if ok else "E: no candidate"}
|
||||
|
||||
def _add_zabbix_repository(self, release):
|
||||
self.calls.append(("repository", release))
|
||||
ok = "repository" not in self.refuse
|
||||
return {"ok": ok, "output": f"Zabbix {release} repository added" if ok else "404"}
|
||||
|
||||
def _write_zabbix_agent_config(self, settings):
|
||||
self.calls.append(("configure", settings["server"]))
|
||||
return {"ok": "configure" not in self.refuse, "output": "written"}
|
||||
|
||||
def _allow_zabbix_agent_sources(self, settings):
|
||||
self.calls.append(("firewall", settings["allowed_sources"]))
|
||||
return {"ok": "firewall" not in self.refuse, "output": "open"}
|
||||
|
||||
def manage_service(self, name, action):
|
||||
self.calls.append((action, name))
|
||||
ok = action not in self.refuse
|
||||
return {"success": ok, "output": "" if ok else f"cannot {action}"}
|
||||
|
||||
|
||||
class TestZabbixAgentMixin:
|
||||
def test_install_configure_open_enable_and_restart(self):
|
||||
agent = _Agent()
|
||||
|
||||
result = agent.configure_zabbix_agent(SETTINGS)
|
||||
|
||||
assert result["success"] is True
|
||||
assert agent.calls == [
|
||||
("refresh",),
|
||||
("ensure_package", "zabbix-agent"),
|
||||
("configure", "zabbix.example.com"),
|
||||
("firewall", ["10.40.100.254"]),
|
||||
("enable", "zabbix-agent"),
|
||||
("restart", "zabbix-agent"),
|
||||
]
|
||||
assert [s["step"] for s in result["steps"]] == ["install", "configure", "firewall", "start"]
|
||||
|
||||
def test_without_a_package_in_the_distribution_zabbix_s_repository_is_added(self):
|
||||
agent = _Agent(refuse=frozenset({"distribution"}))
|
||||
|
||||
result = agent.configure_zabbix_agent(SETTINGS)
|
||||
|
||||
assert result["success"] is True
|
||||
assert agent.calls[:4] == [
|
||||
("refresh",),
|
||||
("ensure_package", "zabbix-agent"),
|
||||
("repository", "7.4"),
|
||||
("ensure_package", "zabbix-agent"),
|
||||
]
|
||||
assert "Zabbix 7.4 repository added" in result["output"]
|
||||
|
||||
def test_without_a_release_there_is_no_fallback(self):
|
||||
settings = {k: v for k, v in SETTINGS.items() if k != "release"}
|
||||
agent = _Agent(refuse=frozenset({"distribution"}))
|
||||
|
||||
result = agent.configure_zabbix_agent(settings)
|
||||
|
||||
assert result["success"] is False
|
||||
assert ("repository", "7.4") not in agent.calls
|
||||
assert "no candidate" in result["output"]
|
||||
|
||||
def test_a_driver_that_cannot_add_the_repository_fails_with_the_distribution_s_answer(self):
|
||||
agent = _Agent(refuse=frozenset({"distribution"}), repository=False)
|
||||
|
||||
result = agent.configure_zabbix_agent(SETTINGS)
|
||||
|
||||
assert result["success"] is False
|
||||
assert "no candidate" in result["output"]
|
||||
|
||||
def test_a_repository_that_cannot_be_added_configures_nothing(self):
|
||||
agent = _Agent(refuse=frozenset({"distribution", "repository"}))
|
||||
|
||||
result = agent.configure_zabbix_agent(SETTINGS)
|
||||
|
||||
assert result["success"] is False
|
||||
assert [c[0] for c in agent.calls] == ["refresh", "ensure_package", "repository"]
|
||||
assert "404" in result["output"]
|
||||
|
||||
def test_the_package_index_is_loaded_first(self):
|
||||
agent = _Agent()
|
||||
|
||||
agent.configure_zabbix_agent(SETTINGS)
|
||||
|
||||
assert agent.calls[0] == ("refresh",)
|
||||
|
||||
def test_an_index_that_will_not_load_still_lets_the_install_try(self):
|
||||
agent = _Agent(refuse=frozenset({"refresh"}))
|
||||
|
||||
result = agent.configure_zabbix_agent(SETTINGS)
|
||||
|
||||
assert result["success"] is True
|
||||
assert ("ensure_package", "zabbix-agent") in agent.calls
|
||||
assert "Temporary failure resolving" in result["steps"][0]["output"]
|
||||
|
||||
def test_a_driver_without_an_index_to_load_installs_straight_away(self):
|
||||
agent = _Agent()
|
||||
agent.refresh_available_updates = None # type: ignore[assignment]
|
||||
|
||||
result = agent.configure_zabbix_agent(SETTINGS)
|
||||
|
||||
assert result["success"] is True
|
||||
assert agent.calls[0] == ("ensure_package", "zabbix-agent")
|
||||
|
||||
def test_a_closed_firewall_starts_nothing(self):
|
||||
agent = _Agent(refuse=frozenset({"firewall"}))
|
||||
|
||||
assert agent.configure_zabbix_agent(SETTINGS)["success"] is False
|
||||
assert ("restart", "zabbix-agent") not in agent.calls
|
||||
|
||||
def test_an_agent_that_will_not_start_at_boot_is_not_restarted(self):
|
||||
agent = _Agent(refuse=frozenset({"enable"}))
|
||||
|
||||
result = agent.configure_zabbix_agent(SETTINGS)
|
||||
|
||||
assert result["success"] is False
|
||||
assert ("restart", "zabbix-agent") not in agent.calls
|
||||
assert "cannot enable" in result["output"]
|
||||
|
||||
def test_the_package_and_service_names_are_the_driver_s(self):
|
||||
agent = _Agent()
|
||||
agent.ZABBIX_AGENT_PACKAGE = "zabbix-agentd"
|
||||
agent.ZABBIX_AGENT_SERVICE = "zabbix_agentd"
|
||||
|
||||
agent.configure_zabbix_agent(SETTINGS)
|
||||
|
||||
assert ("ensure_package", "zabbix-agentd") in agent.calls
|
||||
assert ("restart", "zabbix_agentd") in agent.calls
|
||||
|
||||
def test_settings_that_do_not_hold_are_refused_before_anything_changes(self):
|
||||
agent = _Agent()
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
agent.configure_zabbix_agent({**SETTINGS, "server": "a\nb"})
|
||||
assert agent.calls == []
|
||||
Reference in new issue
Block a user