fix: remove only the images the deploy replaced, never every unused one
CI / check (pull_request) Successful in 19s
CI / check (pull_request) Successful in 19s
The deploy ended with docker image prune -af: every image no container used at that moment. On a host that runs other things too, that takes their rollback images; the KB's copy of this line did exactly that on compute-01 (NetOrk/kb-server#157). deploy.sh now records the image IDs netOrk's own containers use before the swap and, after the verification and the migration, hands them to remote/remove_replaced_images.sh. It removes each by its tags, or by its ID once a moving tag left it without one, and keeps any image a container still uses: a redeploy of the same tag replaces nothing.
This commit is contained in:
1 parent
e35d42a20b
commit
476afe3982
5 files changed
+254
-3
No files matched your search
@@ -429,6 +429,14 @@ REMOTE_ENV
|
||||
# netOrk was down until someone ran the deploy again, which then went through
|
||||
# (NetOrk/netork#586). The second attempt is that rerun. Should it fail as well,
|
||||
# the container states go to the log and the deploy fails as before.
|
||||
# The images netOrk's own containers use now, before the swap: after it, these
|
||||
# are what the deploy replaced, and the only images it removes (below).
|
||||
local REPLACED
|
||||
REPLACED=$(ssh -n "$SERVER" "cd ~/netork && \
|
||||
REGISTRY_HOST='${REGISTRY_HOST}' NETORK_VERSION='${SERVER_VERSION}' \
|
||||
docker compose -f docker-compose.yml -f docker-compose.registry.yml ps -aq ${SERVICES} \
|
||||
| xargs -r docker inspect --format '{{.Image}}' | sort -u" 2>/dev/null || true)
|
||||
|
||||
echo "[${SERVER}] Starting containers..."
|
||||
local RECREATE="cd ~/netork && \
|
||||
REGISTRY_HOST='${REGISTRY_HOST}' NETORK_VERSION='${SERVER_VERSION}' \
|
||||
@@ -527,8 +535,17 @@ REMOTE_ENV
|
||||
run_remote "$SERVER" "alembic upgrade head (containers are ALREADY swapped)" "" \
|
||||
"cd ~/netork && docker compose exec -T netork-api alembic upgrade head"
|
||||
|
||||
echo "[${SERVER}] Pruning unused images..."
|
||||
ssh -n "$SERVER" "docker image prune -af 2>&1 | tail -1" | sed "s/^/[${SERVER}] /"
|
||||
# The images the swap replaced, and nothing else. This was `docker image prune
|
||||
# -af`, which removes every image no container uses at that moment: on a host
|
||||
# that runs other things too, their rollback images (NetOrk/kb-server#157).
|
||||
# remote/remove_replaced_images.sh keeps any image a container still uses.
|
||||
if [[ -n "$REPLACED" ]]; then
|
||||
echo "[${SERVER}] Removing the images this deploy replaced..."
|
||||
local ids="${REPLACED//$'\n'/ }"
|
||||
# shellcheck disable=SC2029 # the image IDs are meant to expand here
|
||||
ssh "$SERVER" "bash -s -- ${ids}" < "${HERE}/remote/remove_replaced_images.sh" 2>&1 \
|
||||
| sed "s/^/[${SERVER}] /" || true
|
||||
fi
|
||||
|
||||
echo "[${SERVER}] Done."
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user