feat: the managing client and sshd's per-source penalties (NetOrk/netork#1800)
driver / standard (pull_request) Failing after 9s
driver / lint (pull_request) Successful in 22s
driver / types (pull_request) Successful in 1m14s
driver / test (3.12) (pull_request) Successful in 1m11s
driver / test (3.13) (pull_request) Successful in 1m7s
driver / lowest (pull_request) Successful in 33s
driver / build (pull_request) Successful in 30s
driver / security (pull_request) Successful in 1m4s
CI / driver (pull_request) Failing after 2m24s
driver / publish (pull_request) Skipped

OpenSSH 9.8+ penalises a source whose connections do not authenticate --
a scanner's probes -- and then drops every connection from it, the
managing client's included.

- OpenSshServerMixin.exempt_from_ssh_penalties puts the address the host
  sees the client at ($SSH_CONNECTION) into PerSourcePenaltyExemptList,
  next to the host's own exemptions, in its own drop-in. Only where sshd
  has the penalties (an older one refuses the directive) and includes the
  drop-in directory; checked with sshd -t, sshd -T and a new session, and
  taken back on any failure. The check-and-restore step of
  harden_ssh_server is shared for it.
- retry_refused tries a connection again after 1, 3, 7 and 15 s while
  sshd closes it before its banner; HostKeyCheck.connect uses it.
This commit is contained in:
christianmanivong committed 2026-11-01 00:24:11 +01:00
1 parent 888a7229dc
commit 5fffb3b886
5 files changed
+698 -18

No files matched your search

+57 -2
View File
@@ -27,6 +27,11 @@ A caller that passes neither keeps working as before (first use, every time),
and a driver that ignores both still connects for a caller that passes them:
old and new versions on either side do not break each other.
A connection sshd turns away before its banner -- OpenSSH 9.8+ does that to a
source it has just penalised -- is tried again after a pause
(:func:`retry_refused`, :data:`REFUSED_RETRY_WAITS`); :meth:`HostKeyCheck.connect`
does so itself, a netmiko driver wraps its ``ConnectHandler`` in it.
:class:`HostKeyCheck` is the one implementation, for a paramiko ``SSHClient``
(:meth:`~HostKeyCheck.connect`) and for netmiko (:meth:`~HostKeyCheck.netmiko`,
then :meth:`~HostKeyCheck.observe`). Like :mod:`napalm_device_types.channel` it
@@ -40,7 +45,8 @@ import hashlib
import logging
import os
import tempfile
from collections.abc import Iterator, Mapping
import time
from collections.abc import Callable, Iterator, Mapping
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Any
@@ -54,6 +60,13 @@ KNOWN_KEYS_ARG = "ssh_host_keys"
#: The ``optional_args`` key that carries the callable every seen key goes to.
OBSERVER_ARG = "ssh_host_key_observer"
#: The waits before each new attempt when sshd turns a connection away before its
#: banner. OpenSSH 9.8+ does that to every connection from a source it has just
#: penalised (``PerSourcePenalties``) -- one whose connections did not
#: authenticate, a scanner's probes -- for seconds to minutes. 26 s in all
#: outlasts the 16 s penalty seen on a new Ubuntu 26.04 host (NetOrk/netork#1800).
REFUSED_RETRY_WAITS = (1.0, 3.0, 7.0, 15.0)
#: The netmiko arguments this module decides; a caller's own values give way.
_NETMIKO_HOST_KEY_ARGS = ("ssh_strict", "system_host_keys", "alt_host_keys", "alt_key_file")
@@ -185,6 +198,48 @@ def _refused_key(exc: BaseException) -> Any | None:
return None
def refused_before_banner(exc: BaseException) -> bool:
"""Whether *exc*, or what it wraps, is sshd closing the connection before its banner.
paramiko says "Error reading SSH protocol banner" (netmiko and drivers wrap
it), with the connection's EOF or reset underneath. A failed login, a
silent host and a closed port are something else: waiting does not end them.
"""
seen: set[int] = set()
stack: list[BaseException | None] = [exc]
while stack:
current = stack.pop()
if current is None or id(current) in seen:
continue
seen.add(id(current))
if "protocol banner" in str(current):
return True
stack.extend([current.__cause__, current.__context__])
return False
def retry_refused[T](
attempt: Callable[[], T],
*,
waits: tuple[float, ...] = REFUSED_RETRY_WAITS,
sleep: Callable[[float], None] | None = None,
) -> T:
"""*attempt*, again after each of *waits* while sshd turns it away before its banner.
Any other error, and the last refusal, go to the caller unchanged.
"""
pause = sleep or time.sleep
for wait in waits:
try:
return attempt()
except Exception as exc:
if not refused_before_banner(exc):
raise
logger.info("SSH connection turned away before its banner; again in %s s", wait)
pause(wait)
return attempt()
class HostKeyCheck:
"""How one driver connection verifies its host's key.
@@ -220,7 +275,7 @@ class HostKeyCheck:
"""``client.connect()`` to this host with *kwargs*, verified and reported."""
self.prepare(client)
with self.refusing():
client.connect(hostname=self.host, port=self.port, **kwargs)
retry_refused(lambda: client.connect(hostname=self.host, port=self.port, **kwargs))
return self.observe(client.get_transport())
# ------------------------------------------------------------------
+211 -16
View File
@@ -30,10 +30,19 @@ Only without such an ``Include`` is sshd_config itself rewritten, after a copy.
3. After the reload, *verify* -- the caller's -- opens a new session. The
running one survives a reload, so if the new one cannot get in, this one
puts the old state back and reloads again.
**The managing client is exempt from per-source penalties** where sshd has them
(OpenSSH 9.8+, :meth:`OpenSshServerMixin.exempt_from_ssh_penalties`). sshd
penalises a source whose connections do not authenticate -- a scanner's probes
-- and then drops every connection from it, the managing session's included.
The address is the one the host sees (``$SSH_CONNECTION``); it joins the
host's own ``PerSourcePenaltyExemptList`` in :data:`PENALTY_DROPIN_PATH`, with
the same three checks. An older sshd is not touched: it refuses the directive.
"""
from __future__ import annotations
import ipaddress
import time
from collections.abc import Callable, Iterable, Mapping, Sequence
from dataclasses import dataclass, field
@@ -57,6 +66,9 @@ MAIN_CONFIG = "/etc/ssh/sshd_config"
DROPIN_DIR = "/etc/ssh/sshd_config.d"
#: ``00-`` so it sorts before every other drop-in: sshd keeps the first value it reads.
DROPIN_PATH = f"{DROPIN_DIR}/00-netork-ssh-hardening.conf"
#: Where :meth:`OpenSshServerMixin.exempt_from_ssh_penalties` names the managing
#: client; ``00-`` like the hardening drop-in, since sshd keeps the first value.
PENALTY_DROPIN_PATH = f"{DROPIN_DIR}/00-netork-penalty-exempt.conf"
#: What marks a line this module disabled in sshd_config.
DISABLED_PREFIX = "# netork: "
@@ -65,6 +77,11 @@ _HEADER = (
"# taken out of what this server offers. Delete this file to undo it.\n"
)
_PENALTY_HEADER = (
"# Written by netOrk: the address netOrk manages this host from, exempt from\n"
"# sshd's per-source penalties (PerSourcePenalties). Delete this file to undo it.\n"
)
#: Host key type, as ``ssh-keygen -l`` prints it -> the algorithms that key serves.
#: Certificate algorithms are left out: they also need a HostCertificate.
_KEY_TYPE_ALGORITHMS: dict[str, tuple[str, ...]] = {
@@ -302,6 +319,56 @@ def render_main_config(text: str, keep: Mapping[str, Sequence[str]], kinds: Iter
return render_dropin(keep, wanted) + "\n".join(out) + ("\n" if body.endswith("\n") else "")
def parse_penalty_exemptions(text: str) -> list[str] | None:
"""The ``persourcepenaltyexemptlist`` from ``sshd -T`` output; None without penalties.
Per-source penalties came with OpenSSH 9.8. An older sshd prints neither
line and refuses ``PerSourcePenaltyExemptList`` outright -- it does not
start with it in its configuration -- so None means: write nothing.
"""
penalties = False
exempt: list[str] = []
for raw in text.splitlines():
parts = raw.strip().split(None, 1)
if not parts:
continue
word = parts[0].lower()
if word == "persourcepenalties":
penalties = True
elif word == "persourcepenaltyexemptlist" and len(parts) == 2:
value = parts[1].strip()
exempt = [] if value.lower() == "none" else [a for a in value.split(",") if a]
return exempt if penalties else None
def client_address(ssh_connection: str) -> str | None:
"""The client's address in ``$SSH_CONNECTION`` (``client port server port``), or None."""
parts = ssh_connection.split()
if len(parts) != 4:
return None
try:
return str(ipaddress.ip_address(parts[0]))
except ValueError:
return None
def is_exempt(address: str, exempt: Iterable[str]) -> bool:
"""Whether *address* is one of *exempt*'s addresses or inside one of its networks."""
ip = ipaddress.ip_address(address)
for entry in exempt:
try:
if ip in ipaddress.ip_network(entry, strict=False):
return True
except ValueError:
continue
return False
def render_penalty_dropin(exempt: Sequence[str]) -> str:
"""The drop-in stating the full exempt list."""
return _PENALTY_HEADER + f"PerSourcePenaltyExemptList {','.join(exempt)}\n"
@dataclass
class SshHardeningResult:
"""What :meth:`OpenSshServerMixin.harden_ssh_server` did.
@@ -319,6 +386,30 @@ class SshHardeningResult:
message: str = ""
@dataclass
class SshPenaltyExemption:
"""What :meth:`OpenSshServerMixin.exempt_from_ssh_penalties` did.
*address* is the one the host sees the managing client at. *applied* is
True only when sshd uses the exemption and a new session got in;
*rolled_back* says the host was changed and then put back.
"""
address: str | None = None
applied: bool = False
rolled_back: bool = False
path: str | None = None
message: str = ""
def _unchanged(
result: SshPenaltyExemption, message: str, say: Callable[[str], None]
) -> SshPenaltyExemption:
result.message = message
say(message)
return result
class OpenSshServerMixin:
"""Adds :meth:`harden_ssh_server` to a driver that can run commands as root on an OpenSSH host.
@@ -484,7 +575,25 @@ class OpenSshServerMixin:
if main is None:
raise RuntimeError(f"cannot read {MAIN_CONFIG}")
restore = self._write_change(plan, main, result, say)
return self._apply_or_restore(plan, kinds, result, restore, verify, say)
def ignored() -> str | None:
now = self.get_ssh_server_algorithms()
names = [SSH_ALGORITHM_CLASSES[k][1] for k in kinds if now[k] != plan.keep[k]]
if not names:
return None
return (
f"sshd does not use the new {', '.join(names)}: a setting read before"
f" {result.path} decides them"
)
return self._apply_or_restore(
result,
restore,
verify,
say,
ignored=ignored,
done="The server no longer offers the selected algorithms.",
)
def _write_change(
self,
@@ -520,16 +629,23 @@ class OpenSshServerMixin:
return restore_main
def _apply_or_restore(
def _apply_or_restore[R: (SshHardeningResult, SshPenaltyExemption)](
self,
plan: SshHardeningPlan,
kinds: list[str],
result: SshHardeningResult,
result: R,
restore: Callable[[], None],
verify: Callable[[], bool] | None,
say: Callable[[str], None],
) -> SshHardeningResult:
def back(message: str, reload: bool) -> SshHardeningResult:
*,
ignored: Callable[[], str | None],
done: str,
) -> R:
"""Check what was written, reload, verify; put the old state back on any failure.
*ignored* reads ``sshd -T`` after ``sshd -t`` and says why sshd does not
use the change, or None when it does; *done* is the message of success.
"""
def back(message: str, reload: bool) -> R:
say(message)
say("Putting the previous configuration back")
try:
@@ -549,14 +665,9 @@ class OpenSshServerMixin:
complaint = self._sshd_accepts()
if complaint:
return back(f"sshd rejected the new configuration: {complaint}", reload=False)
now = self.get_ssh_server_algorithms()
ignored = [SSH_ALGORITHM_CLASSES[k][1] for k in kinds if now[k] != plan.keep[k]]
if ignored:
return back(
f"sshd does not use the new {', '.join(ignored)}: a setting read before"
f" {result.path} decides them",
reload=False,
)
why = ignored()
if why:
return back(why, reload=False)
say("Reloading sshd")
self._reload_ssh_server()
if verify is not None:
@@ -566,6 +677,90 @@ class OpenSshServerMixin:
except Exception as exc: # noqa: BLE001 - whatever went wrong, the host goes back
return back(f"The change failed: {exc}", reload=True)
result.applied = True
result.message = "The server no longer offers the selected algorithms."
result.message = done
say(result.message)
return result
# ── the penalty exemption ───────────────────────────────────────────
def exempt_from_ssh_penalties(
self,
*,
verify: Callable[[], bool] | None = None,
log: Callable[[str], None] | None = None,
) -> SshPenaltyExemption:
"""Exempt the address this session comes from from sshd's per-source penalties.
OpenSSH 9.8+ penalises a source whose connections do not authenticate --
a scanner's probes -- and then drops every connection from it, the
managing client's included (NetOrk/netork#1800). The address is the
one the host sees (``$SSH_CONNECTION``), so NAT and relays do not
matter. The host's own exemptions stay: the new list is the one sshd
uses now plus this address.
Only into the drop-in directory: a sshd_config that does not include it
is left alone, and the result says so. An sshd before 9.8 is not
touched -- it would not start with the directive. Checked like
:meth:`harden_ssh_server`: ``sshd -t``, ``sshd -T`` shows the address,
reload, then *verify*; on any failure the previous state goes back.
:raises RuntimeError: if the session names no client address, or
``sshd -T`` or sshd_config cannot be read.
"""
say = log or (lambda _line: None)
result = SshPenaltyExemption()
session = self.run_command('printf %s "$SSH_CONNECTION"', privileged=False, timeout=30)
address = client_address(session.stdout)
if address is None:
raise RuntimeError("the session names no client address ($SSH_CONNECTION)")
result.address = address
say(f"netOrk connects from {address}")
say("Reading sshd's per-source penalties (sshd -T)")
exempt = parse_penalty_exemptions(self._sshd_effective_text())
if exempt is None:
return _unchanged(
result,
"Nothing to change: this sshd has no per-source penalties (before OpenSSH 9.8).",
say,
)
if is_exempt(address, exempt):
return _unchanged(result, f"Nothing to change: {address} is already exempt.", say)
main = self._ssh_read(MAIN_CONFIG)
if main is None:
raise RuntimeError(f"cannot read {MAIN_CONFIG}")
if not has_dropin_include(main):
return _unchanged(
result,
f"Not changed: {MAIN_CONFIG} does not include {DROPIN_DIR},"
" where the exemption would go.",
say,
)
previous = self._ssh_read(PENALTY_DROPIN_PATH)
self._ssh_run(f"mkdir -p -- {quote(DROPIN_DIR)}")
self._ssh_write(PENALTY_DROPIN_PATH, render_penalty_dropin([*exempt, address]))
result.path = PENALTY_DROPIN_PATH
say(f"Wrote {PENALTY_DROPIN_PATH}")
def restore() -> None:
if previous is None:
self._ssh_remove(PENALTY_DROPIN_PATH)
else:
self._ssh_write(PENALTY_DROPIN_PATH, previous)
def ignored() -> str | None:
if is_exempt(address, parse_penalty_exemptions(self._sshd_effective_text()) or []):
return None
return (
"sshd does not use the new PerSourcePenaltyExemptList: a setting read before"
f" {PENALTY_DROPIN_PATH} decides it"
)
return self._apply_or_restore(
result,
restore,
verify,
say,
ignored=ignored,
done=f"{address} is exempt from sshd's per-source penalties.",
)