feat: the bundled database on every deploy, and --infra-db to move it (NetOrk/netork#1308) #14

Merged
christianmanivong merged 3 commits from feat/infra-db into main 2026-10-20 23:36:57 +00:00
Owner

Release 4 of NetOrk/netork#1308 is netOrk's own PostgreSQL 17 image. It moves the database to a new volume (postgres17_data), and this PR makes deploy.sh able to do that move.

remote/database.sh runs on the host after .env is written and before the broker and the engine. deploy.sh sends it over ssh's stdin.

Host state What happens
No bundled database (external PostgreSQL) Nothing.
Compose names the volume the container already uses up -d --no-deps --wait postgres. The container is recreated only when its image or definition changed, so a new PG minor release arrives with the next deploy. Christian decided this.
Compose names another volume (new major version) A normal deploy stops here with exit 3. Nothing has changed at that point, the old engine keeps running, and the message points at --infra-db.
Same, with --infra-db, target volume absent The move (steps below).
Same, with --infra-db, target volume present The way back: the engine stops and the container switches onto that volume, without a restore.

The move under --infra-db runs in this order:

  1. Stop the engine.
  2. Count rows per table and record the Alembic revision.
  3. Dump with pg_dump -Fc. The dump stays in ~/netork/backups.
  4. Bring PostgreSQL up on the new volume.
  5. Run pg_restore --exit-on-error, then ANALYZE.
  6. Compare counts and revision. If either differs, the move fails and the message explains the way back.

The old volume always stays.

Tests

  • tests/test_remote_database.py runs the script against a fake docker in all five states. It also covers count mismatch, revision mismatch and a failed restore.
  • tests/test_deploy_infra_db.py covers the wiring: reconcile by default, move with the flag, the database step before the broker, and exit 3 stops the deploy before the engine.

Checked on .50

  • Run in reconcile mode with today's compose files from main. The running postgres:16-alpine container stayed as it was (Running, no recreate).
  • Rehearsal of the move itself: live PG16 → throwaway PG17 container.
    • 102 s for dump and restore, 5 s for ANALYZE.
    • 122 tables, same Alembic revision, no invalid index.
    • The size fell from 2.6 GB to 755 MB.

README: the new steps 5 and 6, and a section on moving the bundled database. Step 6 also corrects the earlier text that said deploy.sh never touches redis, which stopped being true with #13.

Release 4 of NetOrk/netork#1308 is netOrk's own PostgreSQL 17 image. It moves the database to a new volume (`postgres17_data`), and this PR makes deploy.sh able to do that move. **`remote/database.sh`** runs on the host after `.env` is written and before the broker and the engine. deploy.sh sends it over ssh's stdin. | Host state | What happens | |---|---| | No bundled database (external PostgreSQL) | Nothing. | | Compose names the volume the container already uses | `up -d --no-deps --wait postgres`. The container is recreated only when its image or definition changed, so a new PG minor release arrives with the next deploy. Christian decided this. | | Compose names another volume (new major version) | A normal deploy stops here with exit 3. Nothing has changed at that point, the old engine keeps running, and the message points at `--infra-db`. | | Same, with `--infra-db`, target volume absent | The move (steps below). | | Same, with `--infra-db`, target volume present | The way back: the engine stops and the container switches onto that volume, without a restore. | The move under `--infra-db` runs in this order: 1. Stop the engine. 2. Count rows per table and record the Alembic revision. 3. Dump with `pg_dump -Fc`. The dump stays in `~/netork/backups`. 4. Bring PostgreSQL up on the new volume. 5. Run `pg_restore --exit-on-error`, then `ANALYZE`. 6. Compare counts and revision. If either differs, the move fails and the message explains the way back. The old volume always stays. **Tests** - `tests/test_remote_database.py` runs the script against a fake `docker` in all five states. It also covers count mismatch, revision mismatch and a failed restore. - `tests/test_deploy_infra_db.py` covers the wiring: `reconcile` by default, `move` with the flag, the database step before the broker, and exit 3 stops the deploy before the engine. **Checked on .50** - Run in `reconcile` mode with today's compose files from main. The running `postgres:16-alpine` container stayed as it was (`Running`, no recreate). - Rehearsal of the move itself: live PG16 → throwaway PG17 container. - 102 s for dump and restore, 5 s for ANALYZE. - 122 tables, same Alembic revision, no invalid index. - The size fell from 2.6 GB to 755 MB. README: the new steps 5 and 6, and a section on moving the bundled database. Step 6 also corrects the earlier text that said deploy.sh never touches `redis`, which stopped being true with #13.
christianmanivong added 1 commit 2026-10-20 21:15:49 +00:00
remote/database.sh runs on the host (over ssh's stdin) after .env and before
broker and engine:

- no bundled database: nothing to do;
- the volume it already uses: plain `up -d --wait`, recreated only on a changed
  image or definition, so a new PostgreSQL minor release arrives with the next
  deploy;
- another volume (a new major version): a normal deploy stops there, nothing
  changed; with --infra-db the engine stops, the old database is counted and
  dumped (kept in ~/netork/backups), the new container comes up on the new
  volume, restore, ANALYZE, counts and Alembic revision must match. The old
  volume stays, and deploying the previous version with --infra-db switches
  back onto it without a restore.

Checked on .50 in reconcile mode: the running postgres:16 container stayed as
it was. README describes the steps and the way back.
christianmanivong added 1 commit 2026-10-20 22:03:46 +00:00
fix: shellcheck clean; remote/*.sh is checked too
CI / check (pull_request) Successful in 47s
4d3a7eb1f4
SC2029 at the database step is intended: mode, services and version expand on
the client side. CI now runs shellcheck on remote/database.sh as well.
christianmanivong added 1 commit 2026-10-20 22:44:15 +00:00
fix: a refused database move leaves the host as it was
CI / check (pull_request) Successful in 19s
1e87478f7f
On .50 a deploy refused by the database step had already copied the refused
version's compose files and written its NETORK_VERSION to .env: a hand-typed
`docker compose up -d` would have carried postgres back to the old volume.
The database step now runs before .env is written, the previous compose files
are kept as .prev while the new ones are fetched, and a refusal (exit 3) puts
them back. Checked on .50: checksums of both compose files and .env unchanged
after a refused deploy.
christianmanivong merged commit e0ec8554e6 into main 2026-10-20 23:36:57 +00:00
christianmanivong deleted branch feat/infra-db 2026-10-20 23:36:57 +00:00
Sign in to join this conversation.