Commit Graph
25 Commits
Author SHA1 Message Date
Christian Manivong 476afe3982 fix: remove only the images the deploy replaced, never every unused one
CI / check (pull_request) Successful in 19s
The deploy ended with docker image prune -af: every image no container
used at that moment. On a host that runs other things too, that takes
their rollback images; the KB's copy of this line did exactly that on
compute-01 (NetOrk/kb-server#157).

deploy.sh now records the image IDs netOrk's own containers use before
the swap and, after the verification and the migration, hands them to
remote/remove_replaced_images.sh. It removes each by its tags, or by its
ID once a moving tag left it without one, and keeps any image a
container still uses: a redeploy of the same tag replaces nothing.
2026-10-23 12:06:08 +02:00
christianmanivong e35d42a20b Merge pull request 'fix: hand ssl_certs to netork before the swap (NetOrk/netork#1389)' (#15) from fix/ssl-certs-owner into main
CI / check (push) Successful in 17s
2026-10-22 13:26:12 +00:00
Christian Manivong 30da942a7a fix: hand ssl_certs to netork before the swap (NetOrk/netork#1389)
CI / check (pull_request) Successful in 18s
netork-ui ran nginx as root, and its entrypoint wrote the self-signed
certificate into ssl_certs as root, the key at 0600. From NetOrk/netork#1389
on, nginx runs as netork, the API's user, and cannot read that key, so the
UI would not start on any existing host.

Before the containers are recreated, one container of the engine image of
the version being deployed, as root with only CHOWN, runs
`chown -Rc netork:netork /app/ssl`. -c names only what it changed, so a
host handed over earlier sees nothing. A failure stops the deploy with the
old containers running. Older engine images carry the same netork user, so
deploying an older version stays possible.
2026-10-22 15:09:50 +02:00
christianmanivong e0ec8554e6 Merge pull request 'feat: the bundled database on every deploy, and --infra-db to move it (NetOrk/netork#1308)' (#14) from feat/infra-db into main
CI / check (push) Successful in 20s
2026-10-20 23:36:57 +00:00
Christian Manivong 1e87478f7f fix: a refused database move leaves the host as it was
CI / check (pull_request) Successful in 19s
On .50 a deploy refused by the database step had already copied the refused
version's compose files and written its NETORK_VERSION to .env: a hand-typed
`docker compose up -d` would have carried postgres back to the old volume.
The database step now runs before .env is written, the previous compose files
are kept as .prev while the new ones are fetched, and a refusal (exit 3) puts
them back. Checked on .50: checksums of both compose files and .env unchanged
after a refused deploy.
2026-10-21 00:44:15 +02:00
Christian Manivong 4d3a7eb1f4 fix: shellcheck clean; remote/*.sh is checked too
CI / check (pull_request) Successful in 47s
SC2029 at the database step is intended: mode, services and version expand on
the client side. CI now runs shellcheck on remote/database.sh as well.
2026-10-21 00:03:45 +02:00
Christian Manivong 6e3ecb8680 feat: the bundled database on every deploy, and --infra-db to move it (NetOrk/netork#1308)
CI / check (pull_request) Failing after 28s
remote/database.sh runs on the host (over ssh's stdin) after .env and before
broker and engine:

- no bundled database: nothing to do;
- the volume it already uses: plain `up -d --wait`, recreated only on a changed
  image or definition, so a new PostgreSQL minor release arrives with the next
  deploy;
- another volume (a new major version): a normal deploy stops there, nothing
  changed; with --infra-db the engine stops, the old database is counted and
  dumped (kept in ~/netork/backups), the new container comes up on the new
  volume, restore, ANALYZE, counts and Alembic revision must match. The old
  volume stays, and deploying the previous version with --infra-db switches
  back onto it without a restore.

Checked on .50 in reconcile mode: the running postgres:16 container stayed as
it was. README describes the steps and the way back.
2026-10-20 23:15:48 +02:00
christianmanivong 9513594f1a Merge pull request 'feat: reconcile the broker before the engine (NetOrk/netork#1308)' (#13) from feat/reconcile-broker into main
CI / check (push) Successful in 19s
2026-10-20 20:40:21 +00:00
Christian Manivong b93a258efc style: ruff format test_deploy_broker.py
CI / check (pull_request) Successful in 1m48s
2026-10-20 22:38:11 +02:00
Christian Manivong 7e4fff8025 feat: reconcile the broker before the engine (NetOrk/netork#1308)
CI / check (pull_request) Failing after 2m28s
redis was in no list: the engine is recreated with --no-deps, and the
infrastructure reconcile left databases out on purpose. A changed definition
-- Valkey in place of redis:7-alpine -- therefore never reached a host.

The broker is now brought up with a plain `up -d --wait` right before the
engine is recreated: recreated only when its definition changed, so an
unchanged broker keeps running and keeps its queue, and the engine comes up
against a healthy broker. If the broker does not get healthy within 120 s, the
deploy stops with the old engine still running.
2026-10-20 22:26:25 +02:00
christianmanivong c1b1eb07b9 Merge pull request 'fix: record the version in .env before starting the containers' (#11) from fix/env-before-up-6 into main
CI / check (push) Successful in 20s
2026-10-18 16:37:17 +00:00
Christian Manivong 3a15c311a0 fix: record the version in .env before starting the containers
CI / check (pull_request) Successful in 19s
The services read ~/netork/.env through env_file, so whatever it says
when `up -d` creates a container ends up in that container's
environment. The settings were recorded only after the swap, so every
container carried the previous deploy's NETORK_VERSION: the image was
right, `docker exec <api> env` was not.

Record them once the pull has brought the images onto the host and
before the first `up -d`. A failed pull still leaves .env alone. If the
swap fails afterwards, .env names the tag whose compose files and
images are already on the host, which is what a hand-typed
`docker compose up -d` there should start.

Also say which branches a feature-<branch> tag comes from: feature/<name>
and feat/<name> become feature-<name>, and fix/<name> now becomes
feature-fix-<name> (NetOrk/netork#208), because the registry keeps only
feature-* branch tags. The accepted forms do not change; the comment, the
refusal message and the README now say so, and a test holds that
feature-fix-<name> passes and a bare fix-<name> does not.

Closes #6
2026-10-18 18:26:24 +02:00
christianmanivong a7e3617291 Merge pull request 'feat: refuse a tag without an image for the host's platform' (#10) from feat/arm64-preflight into main
CI / check (push) Successful in 14s
2026-10-11 06:39:07 +00:00
Christian Manivong 4bf71b7e1c feat: refuse a tag without an image for the host's platform
CI / check (pull_request) Successful in 17s
From 0.34.0 on, netOrk's releases (X.Y.Z, latest) are published for
linux/amd64 and linux/arm64, so a Raspberry Pi with a 64-bit OS can be a
target (NetOrk/netork#934). The deploy itself needed no change for that:
every step runs docker on the target host, which pulls its own platform,
and the images on the host (postgres, redis, apt-cacher-ng, signal-api)
exist for arm64 as well.

What did not work is a tag without arm64: latest-dev, main-<sha>,
feature-<branch> and every release before 0.34.0. On a Pi those failed at
the first pull with "no matching manifest for linux/arm64/v8", which names
neither the cause nor a tag that would work.

Now, right after the registry login, the host's architecture (docker
version) is compared with the platforms the registry lists for the
engine image, and on UI servers the ui image (docker manifest inspect).
A definite mismatch stops the deploy before anything on the host changes,
and says which tags carry arm64. When the registry's answer can't be read,
the pull decides, as before.
2026-10-11 08:38:39 +02:00
christianmanivong 6b334f42e2 Merge pull request 'fix: report only the retired services a host actually had' (#9) from fix/retired-removal-truthful into main
CI / check (push) Successful in 14s
2026-10-08 13:29:20 +00:00
Christian Manivong 1d99e401b9 fix: report only the retired services a host actually had
CI / check (pull_request) Successful in 17s
`docker rm -f` exits 0 for a container that does not exist, so the
"Removing retired services" step printed "removed container
netork-registry-1" on every deploy of every host, long after the
container was gone. That read as something recreating the registry
each time, and it would hide the one deploy that really removed it.

Check with `docker inspect` first, so only an existing container is
removed and reported. The volume branch needed nothing: `docker volume
rm` already fails for a missing volume.

Closes #7
2026-10-08 15:22:48 +02:00
christianmanivong 1d8353046e Merge pull request 'docs: a target server needs at least 4 GB of RAM' (#8) from docs/minimum-ram into main
CI / check (push) Successful in 17s
2026-10-08 12:10:49 +00:00
Christian Manivong 9f57b3a272 docs: a target server needs at least 4 GB of RAM
CI / check (pull_request) Successful in 23s
The requirements named Docker and the .env but no size. netOrk's own
containers take about 2.4 GB on a test server with 74 devices, and at least
0.9 GB stayed free once netOrk#752 was fixed. Less than 4 GB leaves no room
for a large task. The breakdown per container lives in netOrk's
docs/TECHNICAL_DESIGN.md, "Worker memory".
2026-10-08 14:02:44 +02:00
christianmanivong 0effa93895 Merge pull request 'fix: remove the bundled registry from every host, and stop starting it' (#5) from fix/retire-registry into main
CI / check (push) Successful in 17s
2026-10-07 14:19:28 +00:00
Christian Manivong de46ab8a0f fix: remove the bundled registry from every host, and stop starting it
CI / check (pull_request) Successful in 16s
netOrk's docker-compose.yml shipped a registry:2 for the satellite image,
and this script started it on every deploy as infrastructure. Satellites
pull from registry.netork.io; on 172.22.8.50 the registry held one old
image, nothing had pulled from it in 30 days, and it accepted anonymous
pushes on port 5000 of every instance (NetOrk/netork#763).

- registry leaves INFRA_SERVICES.
- A new step removes services netOrk no longer ships: the container
  netork-registry-1, then the volume netork_registry_data. `docker
  compose up` never removes a container whose service left the compose
  file, so without this each host would keep it until someone removed it
  by hand. The step is idempotent and works with the old compose file as
  well as the new one, so it can go out before netOrk drops the service.
2026-10-07 16:17:23 +02:00
christianmanivong 9a93897cd2 Merge pull request 'fix: one deploy per host at a time, under a lock the host frees on its own' (#4) from fix/host-deploy-lock into main
CI / check (push) Successful in 26s
2026-10-07 04:43:24 +00:00
Christian Manivong ec93619f8f fix: one deploy per host at a time, under a lock the host frees on its own
CI / check (pull_request) Successful in 17s
Several sessions deploy to the same test server, and two runs used to
overlap. On 2026-10-05 two `up -d --force-recreate` runs recreated each
other's containers and the API was down for a minute (#1). On 2026-10-06
two runs renamed each other's *.new compose files and one broke off; with
two different tags, one tag's compose files could have started the
other's images (#3).

- Each deploy first takes flock on ~/netork/.deploy.lock on the host. An
  ssh session holds it: the remote side takes the lock on fd 9, reports
  LOCKED and waits on its stdin, so ending the session frees it, whether
  the deploy finished, failed, was interrupted or lost its connection.
  Checked over real ssh on .50, including a client killed with -9.
- A second deploy prints who holds the lock, since when and with which
  tag, and waits up to DEPLOY_LOCK_WAIT seconds (default 900); then it
  gives up without touching the host.
- A host without flock is deployed without the lock, with a warning.
- deploy_server is now the lock around deploy_steps, the old body.

Closes #1
Closes #3
2026-10-07 06:42:13 +02:00
christianmanivong 6444aa97f3 Merge pull request 'fix: a failed container recreate is tried once more before the deploy gives up' (#2) from fix/recreate-retry into main
CI / check (push) Successful in 12s
2026-10-05 20:03:55 +00:00
Christian Manivong 923ec4ce65 fix: a failed container recreate is tried once more before the deploy gives up
CI / check (pull_request) Successful in 13s
On 2026-10-05 a deploy to 172.22.8.50 failed in "Starting containers":
docker compose up -d --force-recreate recreates the services in parallel,
lost a container it had just renamed ("No such container: 02586df7…") and
stopped with every engine container Created and none running. The old API
was already gone, so netOrk was down for about two minutes, until the same
deploy was run again and went through cleanly.

The script now does that second run itself, after DEPLOY_RECREATE_RETRY_DELAY
seconds (5 by default). If the second attempt fails too, it prints the
container states (docker compose ps -a) and fails as before.

Tests run the script against a fake ssh that fails the recreate zero, one
or two times. README step 3 says what happens.

Refs NetOrk/netork#586
2026-10-05 21:58:36 +02:00
Christian Manivong a928d5eec3 Initial import of the netOrk deploy script
CI / check (push) Successful in 12s
2026-09-28 19:55:22 +02:00