diff --git a/docs/STRUCTURE.md b/docs/STRUCTURE.md new file mode 100644 index 0000000..45986e7 --- /dev/null +++ b/docs/STRUCTURE.md @@ -0,0 +1,241 @@ +# Structure + +Status: agreed plan. Nothing here is implemented yet; the migration at the end says in +which order it will be. + +## Principle + +**napalm-device-types is vendor-, OS-, protocol- and product-agnostic.** It holds the +contracts every driver implements and the generic logic that works unchanged for every +driver of a role. It knows no shell, no paramiko, no package manager, no container CLI, +no SNMP OID, no cloud-init and no application that uses the drivers. + +That rules out three kinds of code that are here today: + +- **How a kind of host is read.** Shell commands and their parsers for POSIX hosts move to + a new package, **napalm-unix**, which napalm-linux and napalm-bsd depend on. +- **What an application does with a device.** Polling a device over SNMP, sweeping a range + with pings, scoring discovery evidence, choosing what a provisioned guest installs, and + deciding what counts as a problem belong to the application, not to the drivers. +- **Product naming.** Files, markers and messages a driver leaves on a device carry the + driver's name or none, never the name of the application that called it. + +``` +napalm-device-types roles, capabilities, templates, models, reconcilers <- every driver + ^ +napalm-unix POSIX host readers, SSH exec transport, privilege <- napalm-linux, napalm-bsd + ^ +napalm-linux, napalm-bsd connection, platform getters, platform steps + +application SNMP polling, ping sweeps, discovery scoring, guest + profiles, warnings, and the values every fix is given +``` + +## What stays in napalm-device-types + +| Kind | Rule | Modules | +|---|---|---| +| **Role** | Declares only, under `TYPE_CHECKING`. No runtime method. | `access_point`, `switch`, `firewall`, `hypervisor`, `os`, `storage`, `phone`, `media`, `residential_gateway`, `management_controller` | +| **Capability** | Declares only. | `packages`, `services`, `updates`, `nat_vpn`, `mac_acl`, `config_lifecycle`, `interface_filter`, `host_reboot` | +| **Template** | A concrete method that runs a fixed sequence of abstract steps. A driver implements the steps, or overrides the whole method when its device does it in one go. All values come from the caller. | the fix routines below | +| **Contract types** | Interfaces and data shapes other packages implement or fill. | `CommandChannelMixin`, `CommandResult`, `ByteStream`; the declaration of `ContainerEngineMixin`; the reader mixins' public method declarations; the SNMP profile type; `models` | +| **Reconciler** | Concrete and vendor-neutral, without I/O of its own. | `firewall_rules` and `dhcp` diff/apply, `_matching`, `lag` | +| **Introspection** | Concrete, without I/O. | `roles`; `base` (`FingerprintRule`, `PortSpec`, the fingerprint attributes) | +| **Driver facts** | Class attributes that describe the device or how it is reached, not what an application does with it. | `USES_SSH`, `FULL_UPGRADE`, `platform`, `SNMP_HEALTH_PROFILE` | + +### Fix routines are templates + +A change the application wants made on a device, for example "run an SNMP agent that +answers us", is one template per purpose. The template fixes the order of the steps; the +driver knows how each step is done on its device; the caller supplies every value. + +```python +class SnmpAgentMixin: + def configure_snmp_agent(self, settings: SnmpAgentSettingsDict) -> FixResultDict: + steps = [self._install_snmp_agent, self._write_snmp_agent_config, + self._enable_snmp_agent, self._verify_snmp_agent] + ... # runs them in order, stops at the first failure + + # declared only; napalm-unix implements them for net-snmp + def _install_snmp_agent(self) -> FixStepDict: ... + def _write_snmp_agent_config(self, settings: SnmpAgentSettingsDict) -> FixStepDict: ... + def _enable_snmp_agent(self) -> FixStepDict: ... + def _verify_snmp_agent(self, settings: SnmpAgentSettingsDict) -> FixStepDict: ... +``` + +napalm-unix implements the four steps for net-snmp, for Linux and the BSDs alike. A REST +driver such as OPNsense overrides `configure_snmp_agent` as a whole, because its device +takes the whole configuration in one call. The community, the allowed sources, contact +and location are the caller's. + +The same shape replaces every action in today's `run_device_action`: + +| Today | Template | +|---|---| +| `fix_snmp` | `configure_snmp_agent(settings)` | +| `fix_apt_proxy` | `configure_package_proxy(url)` | +| `fix_docker_permissions` | `grant_container_engine_access(user)` | +| `install_lldpd`, `install_auc` | `ensure_package(name)` | + +The mixin is mixed in by the driver, or by the platform package, not by a role base. A +driver that inherits a template it cannot run on its device, as a NAS driver built on the +Linux driver inherits the package proxy, opts out by setting the method to `None`. So a +caller asks `callable(getattr(driver, "configure_snmp_agent", None))`, never `hasattr`. + +The caller verifies the result from its own side where it can; the SNMP agent is asked +by the application's SNMP client, not by a probe the driver hand-builds. + +### SNMP: drivers declare data, the application polls + +The drivers do not collect over SNMP. A driver whose device answers the standard UCD-MIB +and IF-MIB declares nothing. A driver whose device publishes the same values under +proprietary OIDs declares them as data, in a class attribute of a type defined here: + +```python +class ProcurveDriver(SwitchDriver): + SNMP_HEALTH_PROFILE: ClassVar[SnmpHealthProfile] = { + "uptime": {"oids": ["1.3.6.1.4.1.11.2.14.11.1.2.4.1.5.1"], "format": "ticks"}, + "cpu": {"oids": ["1.3.6.1.4.1.11.2.14.11.5.1.9.6.1.0", + "1.3.6.1.4.1.11.2.3.7.8.6.1.0"]}, + "memory": {"total": "...", "free": "...", "unit": "bytes"}, + } +``` + +One OID per value is not enough for the drivers there are. The profile type has to say: + +- **Fallbacks:** an ordered list of OIDs; the first that answers wins (ProCurve's CPU). +- **Tables:** an OID to walk and how to fold its rows: first, average or maximum (TP-Link + reports CPU and memory per stack unit). +- **Text values:** a pattern that takes the number out of a string (Netgear reports CPU + as `"5 Secs ( 3.2%)"`). +- **Memory:** total with used or free, or a percentage only, and the unit. +- **Uptime:** which OID, standard ones included (`sysUpTime` where a device has no + `hrSystemUptime`), and its format. +- **Interfaces:** a pattern of names to skip, and whether the TX-error counter counts + drops. Today the second is `True` on every firewall, because OPNsense counts drops + there; it becomes OPNsense's profile entry. + +Polling, the standard OIDs, the counter arithmetic (64-bit counters first, 32-bit as the +fallback) and the defaults are the application's. + +### Warnings are facts + +A driver reports facts: the proxy a host is configured with, a DSL line's SNR, the +firmware version it runs and the one it is offered. It does not compare them with a +setting of the application or with a threshold, and it does not name a warning. + +The same goes for lookups a device does not make itself: what firmware a vendor +currently publishes is the application's catalogue, not a getter. + +### Getters read, and say when they cannot + +- A getter changes nothing on the device. It does not start a daemon, rewrite its + configuration or install a package on the way to an answer. +- A read that fails raises. An empty list means "there is none", never "could not ask"; + a sync that deletes and re-adds on top of an empty answer would otherwise duplicate + everything. +- A write reports failure from the device's answer or exit status, never `success: True` + by default. + +### The application uses public methods only + +The application never calls a driver's private methods (`_get`, `_post`, +`_send_command`, `_action_*`). What it needs from a device becomes a declared method of +the driver. + +## What moves to napalm-unix + +| Module | Knows | +|---|---| +| `host_status` | apt, dnf, dpkg, systemd timers, `df` | +| `kernel` | `/proc`, `/lib/modules`, gzip, base64 | +| `listening` | `ss`, `netstat`, `/proc//cgroup` | +| `systemd` | `systemctl` | +| `virtual_ips` | vip-manager, keepalived, awk | +| `system_identity` | SMBIOS via sysfs and `dmidecode` | +| `package_updates` | apt and dnf output | +| `ssh_server` | OpenSSH (`sshd -T`, `sshd -t`, `sshd_config`); which algorithms to drop stays the caller's | +| `terminal` | ANSI escape codes in shell output | +| `channel`: `run_on_transport`, `open_stream_on_transport`, `ParamikoExecStream` | paramiko exec channels | +| `container_engine`: `ContainerEngineConnection`, engine probing | the Docker CLI, `system dial-stdio` | + +It also takes what napalm-linux and napalm-bsd each carry today: + +- **Privilege:** one `sudo -S` / `sudo -n` / root layer, including the rule that a + privileged command never gets stdin while a sudo password is set, and one way to write a + file as root (stage with `mktemp`, then `install`). Two copies of this have already + drifted apart. +- **Parsers:** `/etc/passwd` with `/etc/group`, crontab lines, `ps` with `lstart`. +- **Fix steps:** net-snmp, the package proxy, container engine group membership. + +The FreeBSD part of `host_status` (`freebsd-version -k` against `-r`) moves to napalm-bsd. + +In napalm-unix every reader follows one pattern: + +- **Shape:** `FOO_COMMAND`, `parse_foo(output)`, and `FooMixin.get_foo()`. +- **Reaching the host:** the mixin uses `run_command(command, privileged=...)` from + `CommandChannelMixin`. The six per-reader `_run_*` hooks, with four different signatures, + go away. +- **Framing:** one frame helper writes and parses the `BEGIN`/`END` markers for every reader. +- **Errors:** one exception for "cannot read" and one for "not on this host". A read that + fails or is cut short raises; it never reads as "nothing there". +- **Names on the host:** temporary files, markers and drop-ins carry `napalm` or the + driver's name. A drop-in written under an older name is still recognised and replaced. + +Layout: `napalm_unix/{posix,linux,freebsd,openssh,transport}/`. + +## What moves to the application + +| Today | Why it is not a driver's | +|---|---| +| `health_metrics`, `_ucd_metrics` | Polling over SNMP is something the application does to a device; the driver is not even connected while it runs. | +| `ping_sweep` | Looping over addresses, capping the targets and reporting progress is a scan. A device with a faster batch keeps it as a capability of its own. | +| `rules_max_weight` | Scoring is how one application ranks drivers. The rules themselves stay with the driver. | +| `provisioning` (`QEMU_GUEST_AGENTS`, cloud-init network-config, decompressors) | Which agent a guest gets and how its network is configured is a guest profile. The hypervisor driver is handed the packages and commands. | +| `TYPE_LABEL`, `REBOOT_SETTLE_SECONDS` | A display name and a polling delay. | +| firmware catalogues (napalm-netgear's `netgear_plus`) | What a vendor publishes is not something the device reports. | +| `run_device_action` | Replaced by the templates above. | + +`OSDriver.ROLE` stops defaulting to `"linux"`. Each OS driver states its own role. + +## Rules, enforced by tests in this package + +1. Roles and capabilities own no runtime method; a template runs only declared steps. +2. No module imports `paramiko`, `socket`, `subprocess` or `asyncio`, and none defines a + `*_COMMAND`. +3. No module, string or default names an application. +4. Role methods return TypedDicts, not `Any`. + +Each rule is added with the step that makes it pass. + +## Migration + +This package cannot keep re-exports pointing at napalm-unix: the contract package would +then depend on its implementation. So a moved name is first deprecated here and then +removed, and nothing breaks in between. + +| Step | Change | +|---|---| +| 1 | The application moves to 4.9 and the drivers that need it; napalm-bsd stops passing stdin to a privileged command while a sudo password is set. | +| 2 | This document; the application's architecture docs and this README follow it. | +| 3 | `SnmpHealthProfile`. The application polls on its own; the four switch and phone drivers declare their OIDs as data. `HealthMetricsMixin` is deprecated. | +| 4 | The fix templates. napalm-linux and napalm-bsd implement the steps; the application calls the templates with its values. `run_device_action` is deprecated. The other drivers follow, one issue each. | +| 5 | Ping sweep, scoring, provisioning, `TYPE_LABEL` and `REBOOT_SETTLE_SECONDS` move to the application and are deprecated here. Drivers report facts instead of warnings. | +| 6 | napalm-unix, with the moved code and its tests, the unified hooks and the frame helper. napalm-linux and napalm-bsd import from it; the moved names are deprecated here. | +| 7 | 5.0 removes everything deprecated. It also splits `models.py` by area, types the remaining `Dict[str, Any]` role returns, declares what drivers already offer undeclared (`get_networks`, `search_packages`, `first_boot_pending`, the base-system update entry), and makes `install_package` declare the result drivers return. | +| 8 | The README shrinks to the concepts; the reader details move with the readers. | + +## Proposals this replaces + +Several open issues proposed adding shared driver code here. Under this structure: + +| Issue | Goes to | +|---|---| +| #8 netmiko session | not here: a CLI-driver helper package, or left in the drivers | +| #9 config mode for CLI switches | not here: a CLI-driver helper package, or left in the drivers | +| #10 LLDP rows | the projection of rows into NAPALM's two shapes stays here; the `lldpctl` parser goes to napalm-unix | +| #11 CLI text helpers | not here; the natural port sort, which `lag` already has, may become public | +| #12 ping | the reply builder stays here; the BusyBox and iputils parser goes to napalm-unix | +| #13 Linux userland | napalm-unix; the snmpd part becomes the SNMP template's steps | +| #14 `run_device_action` | replaced by the fix templates | +| #22 hooks onto `run_command` | step 6 |