feat: the bundled database on every deploy, and --infra-db to move it (NetOrk/netork#1308)
CI / check (pull_request) Failing after 28s

remote/database.sh runs on the host (over ssh's stdin) after .env and before
broker and engine:

- no bundled database: nothing to do;
- the volume it already uses: plain `up -d --wait`, recreated only on a changed
  image or definition, so a new PostgreSQL minor release arrives with the next
  deploy;
- another volume (a new major version): a normal deploy stops there, nothing
  changed; with --infra-db the engine stops, the old database is counted and
  dumped (kept in ~/netork/backups), the new container comes up on the new
  volume, restore, ANALYZE, counts and Alembic revision must match. The old
  volume stays, and deploying the previous version with --infra-db switches
  back onto it without a restore.

Checked on .50 in reconcile mode: the running postgres:16 container stayed as
it was. README describes the steps and the way back.
This commit is contained in:
Christian Manivong committed 2026-10-20 23:15:48 +02:00
1 parent 9513594f1a
commit 6e3ecb8680
5 files changed
+460 -7

No files matched your search

+35 -6
View File
@@ -61,6 +61,7 @@ elsewhere.
./deploy.sh 10.0.0.1 # one server
./deploy.sh 10.0.0.1 10.0.0.2 # several, in parallel
./deploy.sh --version=main-1a2b3c4 10.0.0.1 # one-off tag, this run only
./deploy.sh --infra-db 10.0.0.1 # also move the bundled database (see below)
```
Only tags netOrk's CI publishes are accepted:
@@ -97,24 +98,52 @@ started earlier pulls whatever image the registry held before, which is stale.
hand-typed `docker compose` on the host uses the same images. It does this before
any container starts, because the services read `.env` into their environment.
A failed pull leaves `.env` as it was.
5. Force-recreates the API, the workers, `netork-beat`, `flower` and, on UI servers,
5. Reconciles the bundled PostgreSQL, if the host has one (`remote/database.sh`,
over ssh's stdin). On the volume it already uses, it is recreated only if its image
or definition changed, so a new minor release arrives with the next deploy. If the
compose files name another volume (a new major version), a normal deploy stops
here before anything changed; see *Moving the bundled database*.
6. Reconciles the broker, `redis` (Valkey): recreated only if its definition changed,
and waited for until healthy, before the engine comes up against it. If it does
not get healthy within 120 s, the deploy stops with the old engine running.
7. Force-recreates the API, the workers, `netork-beat`, `flower` and, on UI servers,
`netork-ui`. If the recreate fails, it is tried once more after 5 s
(`DEPLOY_RECREATE_RETRY_DELAY`). Compose's parallel recreate can lose a container it
just renamed and leave the rest stopped. If the second attempt fails too, the
container states are printed and the deploy fails.
6. Reconciles `apt-cacher-ng` and `signal-api`: each is recreated only if its
definition changed. It never touches `postgres` or `redis`.
8. Reconciles `apt-cacher-ng` and `signal-api`: each is recreated only if its
definition changed.
It then removes services netOrk no longer ships, container and volume, where a host
still has them: today the bundled `registry` (NetOrk/netork#763). It reports only
what it actually removed; on a host that was cleaned earlier it says nothing.
7. Verifies that the containers run exactly the image that was pulled. If they don't, the
9. Verifies that the containers run exactly the image that was pulled. If they don't, the
deploy fails.
8. Runs `alembic upgrade head` inside `netork-api`.
9. Prunes unused images.
10. Runs `alembic upgrade head` inside `netork-api`.
11. Prunes unused images.
A failing step stops that server's deploy with a non-zero exit, and the other servers
carry on. The script exits non-zero if any server failed and names those servers.
## Moving the bundled database
A new PostgreSQL major version cannot read the old data directory, so netOrk names a
new volume for it, for example `postgres17_data` (NetOrk/netork#1308). Moving onto it
takes `--infra-db`, for one host at an announced time, because the engine is down
while the data moves.
1. Stops the engine.
2. Counts every table and dumps the database, using `pg_dump -Fc`. The dump stays
in `~/netork/backups/`.
3. Brings the new container up on the new volume, then restores and runs
`ANALYZE`.
4. Compares the counts and the Alembic revision. If either differs, the move fails
and says how to go back.
5. Leaves the old volume as it was.
**The way back:** deploy the previous netOrk version with `--infra-db`. Its compose
files name the old volume again, which still exists, so the container switches onto
it without a restore. The data is then as it was before the move.
## Operating notes
- **Never delete these Docker volumes:**