feat: the bundled database on every deploy, and --infra-db to move it (NetOrk/netork#1308)
CI / check (pull_request) Failing after 28s
CI / check (pull_request) Failing after 28s
remote/database.sh runs on the host (over ssh's stdin) after .env and before broker and engine: - no bundled database: nothing to do; - the volume it already uses: plain `up -d --wait`, recreated only on a changed image or definition, so a new PostgreSQL minor release arrives with the next deploy; - another volume (a new major version): a normal deploy stops there, nothing changed; with --infra-db the engine stops, the old database is counted and dumped (kept in ~/netork/backups), the new container comes up on the new volume, restore, ANALYZE, counts and Alembic revision must match. The old volume stays, and deploying the previous version with --infra-db switches back onto it without a restore. Checked on .50 in reconcile mode: the running postgres:16 container stayed as it was. README describes the steps and the way back.
This commit is contained in:
1 parent
9513594f1a
commit
6e3ecb8680
5 files changed
+460
-7
No files matched your search
@@ -61,6 +61,7 @@ elsewhere.
|
||||
./deploy.sh 10.0.0.1 # one server
|
||||
./deploy.sh 10.0.0.1 10.0.0.2 # several, in parallel
|
||||
./deploy.sh --version=main-1a2b3c4 10.0.0.1 # one-off tag, this run only
|
||||
./deploy.sh --infra-db 10.0.0.1 # also move the bundled database (see below)
|
||||
```
|
||||
|
||||
Only tags netOrk's CI publishes are accepted:
|
||||
@@ -97,24 +98,52 @@ started earlier pulls whatever image the registry held before, which is stale.
|
||||
hand-typed `docker compose` on the host uses the same images. It does this before
|
||||
any container starts, because the services read `.env` into their environment.
|
||||
A failed pull leaves `.env` as it was.
|
||||
5. Force-recreates the API, the workers, `netork-beat`, `flower` and, on UI servers,
|
||||
5. Reconciles the bundled PostgreSQL, if the host has one (`remote/database.sh`,
|
||||
over ssh's stdin). On the volume it already uses, it is recreated only if its image
|
||||
or definition changed, so a new minor release arrives with the next deploy. If the
|
||||
compose files name another volume (a new major version), a normal deploy stops
|
||||
here before anything changed; see *Moving the bundled database*.
|
||||
6. Reconciles the broker, `redis` (Valkey): recreated only if its definition changed,
|
||||
and waited for until healthy, before the engine comes up against it. If it does
|
||||
not get healthy within 120 s, the deploy stops with the old engine running.
|
||||
7. Force-recreates the API, the workers, `netork-beat`, `flower` and, on UI servers,
|
||||
`netork-ui`. If the recreate fails, it is tried once more after 5 s
|
||||
(`DEPLOY_RECREATE_RETRY_DELAY`). Compose's parallel recreate can lose a container it
|
||||
just renamed and leave the rest stopped. If the second attempt fails too, the
|
||||
container states are printed and the deploy fails.
|
||||
6. Reconciles `apt-cacher-ng` and `signal-api`: each is recreated only if its
|
||||
definition changed. It never touches `postgres` or `redis`.
|
||||
8. Reconciles `apt-cacher-ng` and `signal-api`: each is recreated only if its
|
||||
definition changed.
|
||||
It then removes services netOrk no longer ships, container and volume, where a host
|
||||
still has them: today the bundled `registry` (NetOrk/netork#763). It reports only
|
||||
what it actually removed; on a host that was cleaned earlier it says nothing.
|
||||
7. Verifies that the containers run exactly the image that was pulled. If they don't, the
|
||||
9. Verifies that the containers run exactly the image that was pulled. If they don't, the
|
||||
deploy fails.
|
||||
8. Runs `alembic upgrade head` inside `netork-api`.
|
||||
9. Prunes unused images.
|
||||
10. Runs `alembic upgrade head` inside `netork-api`.
|
||||
11. Prunes unused images.
|
||||
|
||||
A failing step stops that server's deploy with a non-zero exit, and the other servers
|
||||
carry on. The script exits non-zero if any server failed and names those servers.
|
||||
|
||||
## Moving the bundled database
|
||||
|
||||
A new PostgreSQL major version cannot read the old data directory, so netOrk names a
|
||||
new volume for it, for example `postgres17_data` (NetOrk/netork#1308). Moving onto it
|
||||
takes `--infra-db`, for one host at an announced time, because the engine is down
|
||||
while the data moves.
|
||||
|
||||
1. Stops the engine.
|
||||
2. Counts every table and dumps the database, using `pg_dump -Fc`. The dump stays
|
||||
in `~/netork/backups/`.
|
||||
3. Brings the new container up on the new volume, then restores and runs
|
||||
`ANALYZE`.
|
||||
4. Compares the counts and the Alembic revision. If either differs, the move fails
|
||||
and says how to go back.
|
||||
5. Leaves the old volume as it was.
|
||||
|
||||
**The way back:** deploy the previous netOrk version with `--infra-db`. Its compose
|
||||
files name the old volume again, which still exists, so the container switches onto
|
||||
it without a restore. The data is then as it was before the move.
|
||||
|
||||
## Operating notes
|
||||
|
||||
- **Never delete these Docker volumes:**
|
||||
|
||||
Reference in new issue
Block a user