fix: remove only the images the deploy replaced, never every unused one
CI / check (pull_request) Successful in 19s
CI / check (pull_request) Successful in 19s
The deploy ended with docker image prune -af: every image no container used at that moment. On a host that runs other things too, that takes their rollback images; the KB's copy of this line did exactly that on compute-01 (NetOrk/kb-server#157). deploy.sh now records the image IDs netOrk's own containers use before the swap and, after the verification and the migration, hands them to remote/remove_replaced_images.sh. It removes each by its tags, or by its ID once a moving tag left it without one, and keeps any image a container still uses: a redeploy of the same tag replaces nothing.
This commit is contained in:
1 parent
e35d42a20b
commit
476afe3982
5 files changed
+254
-3
No files matched your search
@@ -126,7 +126,12 @@ started earlier pulls whatever image the registry held before, which is stale.
|
||||
10. Verifies that the containers run exactly the image that was pulled. If they don't, the
|
||||
deploy fails.
|
||||
11. Runs `alembic upgrade head` inside `netork-api`.
|
||||
12. Prunes unused images.
|
||||
12. Removes the images the swap replaced, and only those. Before the swap it records which
|
||||
images netOrk's own containers use; afterwards `remote/remove_replaced_images.sh`
|
||||
removes them unless a container still uses one, a redeploy of the same tag for
|
||||
example. Other projects' images on the host stay. This used to be
|
||||
`docker image prune -af`, which took every unused image, theirs included
|
||||
(NetOrk/kb-server#157).
|
||||
|
||||
A failing step stops that server's deploy with a non-zero exit, and the other servers
|
||||
carry on. The script exits non-zero if any server failed and names those servers.
|
||||
|
||||
@@ -429,6 +429,14 @@ REMOTE_ENV
|
||||
# netOrk was down until someone ran the deploy again, which then went through
|
||||
# (NetOrk/netork#586). The second attempt is that rerun. Should it fail as well,
|
||||
# the container states go to the log and the deploy fails as before.
|
||||
# The images netOrk's own containers use now, before the swap: after it, these
|
||||
# are what the deploy replaced, and the only images it removes (below).
|
||||
local REPLACED
|
||||
REPLACED=$(ssh -n "$SERVER" "cd ~/netork && \
|
||||
REGISTRY_HOST='${REGISTRY_HOST}' NETORK_VERSION='${SERVER_VERSION}' \
|
||||
docker compose -f docker-compose.yml -f docker-compose.registry.yml ps -aq ${SERVICES} \
|
||||
| xargs -r docker inspect --format '{{.Image}}' | sort -u" 2>/dev/null || true)
|
||||
|
||||
echo "[${SERVER}] Starting containers..."
|
||||
local RECREATE="cd ~/netork && \
|
||||
REGISTRY_HOST='${REGISTRY_HOST}' NETORK_VERSION='${SERVER_VERSION}' \
|
||||
@@ -527,8 +535,17 @@ REMOTE_ENV
|
||||
run_remote "$SERVER" "alembic upgrade head (containers are ALREADY swapped)" "" \
|
||||
"cd ~/netork && docker compose exec -T netork-api alembic upgrade head"
|
||||
|
||||
echo "[${SERVER}] Pruning unused images..."
|
||||
ssh -n "$SERVER" "docker image prune -af 2>&1 | tail -1" | sed "s/^/[${SERVER}] /"
|
||||
# The images the swap replaced, and nothing else. This was `docker image prune
|
||||
# -af`, which removes every image no container uses at that moment: on a host
|
||||
# that runs other things too, their rollback images (NetOrk/kb-server#157).
|
||||
# remote/remove_replaced_images.sh keeps any image a container still uses.
|
||||
if [[ -n "$REPLACED" ]]; then
|
||||
echo "[${SERVER}] Removing the images this deploy replaced..."
|
||||
local ids="${REPLACED//$'\n'/ }"
|
||||
# shellcheck disable=SC2029 # the image IDs are meant to expand here
|
||||
ssh "$SERVER" "bash -s -- ${ids}" < "${HERE}/remote/remove_replaced_images.sh" 2>&1 \
|
||||
| sed "s/^/[${SERVER}] /" || true
|
||||
fi
|
||||
|
||||
echo "[${SERVER}] Done."
|
||||
}
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
#!/bin/bash
|
||||
# remove_replaced_images.sh <image id>...: the images a deploy replaced, and
|
||||
# nothing else. deploy.sh records the IDs netOrk's own containers use before the
|
||||
# swap and runs this on the host after it.
|
||||
#
|
||||
# It used to be `docker image prune -af`, which removes every image no container
|
||||
# uses at that moment. On a host that runs other things too, that took their
|
||||
# rollback images (NetOrk/kb-server#157). An ID goes here only if no container --
|
||||
# netOrk's or anyone else's -- uses it: a redeploy of the same tag replaces
|
||||
# nothing. It goes by every tag it carries, or by its ID once a moving tag such as
|
||||
# latest-dev has left it without one.
|
||||
set -u
|
||||
|
||||
in_use=$(docker ps -aq | xargs -r docker inspect --format '{{.Image}}' 2>/dev/null | sort -u)
|
||||
|
||||
for id in "$@"; do
|
||||
if printf '%s\n' "$in_use" | grep -qx -- "$id"; then
|
||||
echo "kept ${id:7:12} (in use)"
|
||||
continue
|
||||
fi
|
||||
tags=$(docker image inspect --format '{{range .RepoTags}}{{.}} {{end}}' "$id" 2>/dev/null) || continue
|
||||
# shellcheck disable=SC2086 # the tags are separate words on purpose
|
||||
if docker image rm -- ${tags:-$id} >/dev/null 2>&1; then
|
||||
echo "removed ${tags:-$id}"
|
||||
else
|
||||
echo "kept ${id:7:12}"
|
||||
fi
|
||||
done
|
||||
exit 0
|
||||
@@ -0,0 +1,80 @@
|
||||
"""A deploy removes the images it replaced, and nothing else (NetOrk/kb-server#157).
|
||||
|
||||
It ended with `docker image prune -af`: every image no container used at that
|
||||
moment, on a host that may run other things too. Now it records the image IDs of
|
||||
netOrk's own containers before the swap and, once the swap is verified and the
|
||||
migration ran, hands them to remote/remove_replaced_images.sh.
|
||||
|
||||
The remote side is a fake `ssh` that records every command and plays a host whose
|
||||
netOrk containers run one image.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
DEPLOY = Path(__file__).resolve().parent.parent / "deploy.sh"
|
||||
|
||||
_FAKE_SSH = """#!/bin/sh
|
||||
for arg in "$@"; do cmd="$arg"; done
|
||||
case "$cmd" in *.deploy.lock*) exec sh -c "$cmd" ;; esac
|
||||
case "$cmd" in
|
||||
*"bash -s"*) cat > "{state}/stdin-$(date +%s%N)" ;;
|
||||
*) cat > /dev/null ;;
|
||||
esac
|
||||
printf '%s\\n----\\n' "$cmd" >> "{state}/commands"
|
||||
case "$cmd" in *"xargs -r docker inspect"*) echo sha256:old ;; esac
|
||||
exit 0
|
||||
"""
|
||||
|
||||
|
||||
def _deploy(tmp_path: Path) -> tuple[subprocess.CompletedProcess[str], list[str]]:
|
||||
bin_dir = tmp_path / "bin"
|
||||
bin_dir.mkdir()
|
||||
fake = bin_dir / "ssh"
|
||||
fake.write_text(_FAKE_SSH.replace("{state}", str(tmp_path)))
|
||||
fake.chmod(0o755)
|
||||
result = subprocess.run(
|
||||
["bash", str(DEPLOY), "testhost"],
|
||||
env={
|
||||
"PATH": f"{bin_dir}:{os.environ['PATH']}",
|
||||
"HOME": str(tmp_path),
|
||||
"DEPLOY_ENV_FILE": "/dev/null",
|
||||
"REGISTRY_HOST": "registry.example",
|
||||
"NETORK_VERSION": "latest-dev",
|
||||
},
|
||||
stdin=subprocess.DEVNULL,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=60,
|
||||
check=False,
|
||||
)
|
||||
commands = (tmp_path / "commands").read_text().split("\n----\n")
|
||||
return result, commands
|
||||
|
||||
|
||||
def _index(commands: list[str], needle: str) -> int:
|
||||
return next(i for i, cmd in enumerate(commands) if needle in cmd)
|
||||
|
||||
|
||||
def test_no_prune_of_everything(tmp_path: Path) -> None:
|
||||
result, commands = _deploy(tmp_path)
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert not [cmd for cmd in commands if "image prune -af" in cmd]
|
||||
|
||||
|
||||
def test_the_replaced_images_are_recorded_before_the_swap_and_removed_after(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
result, commands = _deploy(tmp_path)
|
||||
assert result.returncode == 0, result.stderr
|
||||
recorded = _index(commands, "xargs -r docker inspect")
|
||||
swap = _index(commands, "--force-recreate")
|
||||
migrate = _index(commands, "alembic upgrade head")
|
||||
removal = _index(commands, "sha256:old")
|
||||
assert recorded < swap < migrate < removal
|
||||
assert "bash -s" in commands[removal]
|
||||
sent = [p.read_text() for p in tmp_path.glob("stdin-*")]
|
||||
assert any("remove_replaced_images.sh" in text for text in sent), result.stdout
|
||||
@@ -0,0 +1,120 @@
|
||||
"""After a deploy, the images it replaced go, and nothing else: remote/remove_replaced_images.sh.
|
||||
|
||||
deploy.sh used to end with `docker image prune -af`, which removes every image no
|
||||
container uses at that moment. On a host that runs other things too, that took
|
||||
their rollback images (NetOrk/kb-server#157, the same line in the KB's script).
|
||||
|
||||
deploy.sh now records the image IDs netOrk's own containers use before the swap
|
||||
and hands them to this script afterwards. It removes those, by every tag they
|
||||
carry, unless a container -- netOrk's or anyone else's -- still uses them: a
|
||||
redeploy of the same tag replaces nothing.
|
||||
|
||||
The remote side is a fake `docker` that keeps images and containers in files.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT = Path(__file__).resolve().parent.parent / "remote" / "remove_replaced_images.sh"
|
||||
|
||||
_FAKE_DOCKER = r"""#!/bin/bash
|
||||
state=$FAKE_STATE
|
||||
printf '%s\n' "$*" >> "$state/calls"
|
||||
case "$1 $2" in
|
||||
"ps -aq") cut -d' ' -f1 "$state/containers" ;;
|
||||
"inspect --format")
|
||||
shift 3
|
||||
for c in "$@"; do grep "^$c " "$state/containers" | cut -d' ' -f2; done ;;
|
||||
"image inspect")
|
||||
id="${*: -1}"
|
||||
line=$(grep "^$id" "$state/images") || exit 1
|
||||
echo "${line#* }" ;;
|
||||
"image rm")
|
||||
shift 2
|
||||
[ "$1" = -- ] && shift
|
||||
for ref in "$@"; do
|
||||
if grep -q " $ref\b" "$state/containers" 2>/dev/null; then exit 1; fi
|
||||
sed -i "s# $ref##" "$state/images"
|
||||
done
|
||||
# An image without tags is gone.
|
||||
sed -i '/^[^ ]*$/d' "$state/images" ;;
|
||||
esac
|
||||
exit 0
|
||||
"""
|
||||
|
||||
|
||||
def _run(tmp_path: Path, images: list[str], containers: list[str], ids: list[str]):
|
||||
(tmp_path / "images").write_text("".join(f"{line}\n" for line in images))
|
||||
(tmp_path / "containers").write_text("".join(f"{line}\n" for line in containers))
|
||||
bin_dir = tmp_path / "bin"
|
||||
bin_dir.mkdir()
|
||||
(bin_dir / "docker").write_text(_FAKE_DOCKER)
|
||||
(bin_dir / "docker").chmod(0o755)
|
||||
result = subprocess.run(
|
||||
["bash", str(SCRIPT), *ids],
|
||||
env={"PATH": f"{bin_dir}:{os.environ['PATH']}", "FAKE_STATE": str(tmp_path)},
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
check=False,
|
||||
)
|
||||
left = (tmp_path / "images").read_text().split()
|
||||
return result, left
|
||||
|
||||
|
||||
def test_the_replaced_release_goes_by_every_tag(tmp_path: Path) -> None:
|
||||
result, left = _run(
|
||||
tmp_path,
|
||||
images=[
|
||||
"sha256:old r/netork/engine:0.41.0 r/netork/engine:main-aaa",
|
||||
"sha256:new r/netork/engine:0.42.0",
|
||||
],
|
||||
containers=["c1 sha256:new"],
|
||||
ids=["sha256:old"],
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "sha256:old" not in left
|
||||
assert "sha256:new" in left
|
||||
assert "removed" in result.stdout
|
||||
|
||||
|
||||
def test_an_image_still_in_use_stays(tmp_path: Path) -> None:
|
||||
# A redeploy of the same tag: the "old" image is the one running now.
|
||||
result, left = _run(
|
||||
tmp_path,
|
||||
images=["sha256:same r/netork/engine:latest-dev r/netork/engine:main-bbb"],
|
||||
containers=["c1 sha256:same"],
|
||||
ids=["sha256:same"],
|
||||
)
|
||||
assert result.returncode == 0
|
||||
assert left == ["sha256:same", "r/netork/engine:latest-dev", "r/netork/engine:main-bbb"]
|
||||
assert "kept" in result.stdout
|
||||
|
||||
|
||||
def test_another_projects_images_are_never_named(tmp_path: Path) -> None:
|
||||
result, left = _run(
|
||||
tmp_path,
|
||||
images=["sha256:old r/netork/engine:0.41.0", "sha256:theirs other/app:1.0"],
|
||||
containers=[],
|
||||
ids=["sha256:old"],
|
||||
)
|
||||
assert result.returncode == 0
|
||||
assert "sha256:theirs" in left
|
||||
assert "prune" not in (tmp_path / "calls").read_text()
|
||||
|
||||
|
||||
def test_an_untagged_leftover_goes_by_its_id(tmp_path: Path) -> None:
|
||||
# latest-dev moved on: the old image has no tag left.
|
||||
result, left = _run(
|
||||
tmp_path, images=["sha256:dangling"], containers=[], ids=["sha256:dangling"]
|
||||
)
|
||||
assert result.returncode == 0
|
||||
assert "sha256:dangling" not in left
|
||||
|
||||
|
||||
def test_nothing_to_remove_is_no_failure(tmp_path: Path) -> None:
|
||||
result, _ = _run(tmp_path, images=[], containers=[], ids=["sha256:gone"])
|
||||
assert result.returncode == 0
|
||||
Reference in new issue
Block a user