CI / check (pull_request) Failing after 28s
remote/database.sh runs on the host (over ssh's stdin) after .env and before broker and engine: - no bundled database: nothing to do; - the volume it already uses: plain `up -d --wait`, recreated only on a changed image or definition, so a new PostgreSQL minor release arrives with the next deploy; - another volume (a new major version): a normal deploy stops there, nothing changed; with --infra-db the engine stops, the old database is counted and dumped (kept in ~/netork/backups), the new container comes up on the new volume, restore, ANALYZE, counts and Alembic revision must match. The old volume stays, and deploying the previous version with --infra-db switches back onto it without a restore. Checked on .50 in reconcile mode: the running postgres:16 container stayed as it was. README describes the steps and the way back.
174 lines
8.1 KiB
Markdown
174 lines
8.1 KiB
Markdown
# netOrk deploy
|
|
|
|
Deploys [netOrk](https://git.netork.io/NetOrk/netork) to one or more Docker hosts in
|
|
parallel, from pre-built images in a container registry.
|
|
|
|
The machine running the deploy needs this repository, `ssh`, and a `deploy.env`. It does
|
|
not need a netOrk checkout. Each server pulls the images itself. The compose files come
|
|
out of the engine image for the tag being deployed, so they always match the images they
|
|
start, and database migrations run from that same image.
|
|
|
|
## Requirements
|
|
|
|
**On the machine running the deploy:**
|
|
- bash
|
|
- `ssh` with key-based access to every target
|
|
|
|
**On every target server:**
|
|
- at least 4 GB of RAM. On a test server with 74 devices, netOrk's containers took about
|
|
2.4 GB, and at least 0.9 GB stayed free. A large task needs that room. The breakdown
|
|
per container is in netOrk's `docs/TECHNICAL_DESIGN.md`, "Worker memory".
|
|
- Docker with the compose plugin
|
|
- a directory `~/netork/` with netOrk's `.env`, which holds the database and application
|
|
settings the compose file reads (`env_file: .env`)
|
|
|
|
**Images:** `netork/engine` and `netork/ui` must be published in the registry under the
|
|
tag you deploy. The engine image must carry its compose files under `/app/deploy/`.
|
|
Tags built before that change are refused with a clear message.
|
|
|
|
**Raspberry Pi and other arm64 hosts:** a Raspberry Pi 3, 4, 5 or Zero 2 W with a 64-bit
|
|
OS can be a target. Nothing changes in `deploy.env`: each host pulls the variant for its
|
|
own platform. netOrk publishes arm64 images for releases only, from 0.34.0 on (`latest`,
|
|
`X.Y.Z`). `latest-dev`, `main-<sha>`, `feature-<branch>` and older releases are amd64 only,
|
|
and a deploy of such a tag to an arm64 host is refused before the host is touched. The
|
|
images on the host (`postgres`, `redis`, `apt-cacher-ng`, `signal-api`) exist for arm64
|
|
as well.
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
cp deploy.env.example deploy.env
|
|
$EDITOR deploy.env
|
|
```
|
|
|
|
`deploy.env` is gitignored. Point `DEPLOY_ENV_FILE` at another file to keep it
|
|
elsewhere.
|
|
|
|
| Variable | Meaning |
|
|
|---|---|
|
|
| `DEPLOY_SERVERS` | Space-separated default targets |
|
|
| `UI_SERVER` | Targets that also run `netork-ui` |
|
|
| `REGISTRY_HOST` | Registry to pull from; required |
|
|
| `NETORK_VERSION` | Tag to deploy (default `latest`) |
|
|
| `REGISTRY_USER`, `REGISTRY_PASSWORD` | Registry login used on every server |
|
|
| `REGISTRY_USER_<server>`, `REGISTRY_PASSWORD_<server>`, `NETORK_VERSION_<server>` | Per-server overrides. `<server>` has its dots replaced by underscores, e.g. `_10_0_0_2` |
|
|
| `DEPLOY_LOCK_WAIT` | Seconds to wait for another deploy to the same host (default 900) |
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
./deploy.sh # every server in DEPLOY_SERVERS
|
|
./deploy.sh 10.0.0.1 # one server
|
|
./deploy.sh 10.0.0.1 10.0.0.2 # several, in parallel
|
|
./deploy.sh --version=main-1a2b3c4 10.0.0.1 # one-off tag, this run only
|
|
./deploy.sh --infra-db 10.0.0.1 # also move the bundled database (see below)
|
|
```
|
|
|
|
Only tags netOrk's CI publishes are accepted:
|
|
- `latest`
|
|
- `latest-dev`
|
|
- `main-<sha>`
|
|
- `feature-<name>`, a branch build: `feature/<name>` and `feat/<name>` become
|
|
`feature-<name>`, and `fix/<name>` becomes `feature-fix-<name>`. Any further `/` in
|
|
the branch name becomes `-` as well.
|
|
- `X.Y.Z`
|
|
|
|
Anything else is refused before any server is touched.
|
|
|
|
If you deploy a branch build, wait until its CI run has published the images. A deploy
|
|
started earlier pulls whatever image the registry held before, which is stale.
|
|
|
|
## What a deploy does, per server
|
|
|
|
0. Takes the host's deploy lock, `flock` on `~/netork/.deploy.lock`, and holds it until
|
|
the deploy ends. A second deploy to the same host waits for it and says who holds
|
|
it, since when, and which tag they are deploying. It gives up after 15 minutes
|
|
(`DEPLOY_LOCK_WAIT`, in seconds) without touching the host. The lock belongs to an
|
|
ssh session, so a deploy that fails, is interrupted or loses its connection frees it
|
|
on its own. A host without `flock` is deployed without the lock, with a warning.
|
|
1. Logs in to the registry. The password travels over ssh's stdin, never on a command
|
|
line. Then it checks that the tag has an image for the host's platform (`docker
|
|
version`, `docker manifest inspect`): engine, and on UI servers ui. If not, for
|
|
example `latest-dev` on a Raspberry Pi, it stops here and says which tags would work.
|
|
If the registry's answer can't be read, the pull below decides.
|
|
2. Pulls the engine image and copies `docker-compose.yml` and
|
|
`docker-compose.registry.yml` out of it into `~/netork/`.
|
|
3. Pulls the netOrk images.
|
|
4. Records `REGISTRY_HOST` and `NETORK_VERSION` in `~/netork/.env`, so that a
|
|
hand-typed `docker compose` on the host uses the same images. It does this before
|
|
any container starts, because the services read `.env` into their environment.
|
|
A failed pull leaves `.env` as it was.
|
|
5. Reconciles the bundled PostgreSQL, if the host has one (`remote/database.sh`,
|
|
over ssh's stdin). On the volume it already uses, it is recreated only if its image
|
|
or definition changed, so a new minor release arrives with the next deploy. If the
|
|
compose files name another volume (a new major version), a normal deploy stops
|
|
here before anything changed; see *Moving the bundled database*.
|
|
6. Reconciles the broker, `redis` (Valkey): recreated only if its definition changed,
|
|
and waited for until healthy, before the engine comes up against it. If it does
|
|
not get healthy within 120 s, the deploy stops with the old engine running.
|
|
7. Force-recreates the API, the workers, `netork-beat`, `flower` and, on UI servers,
|
|
`netork-ui`. If the recreate fails, it is tried once more after 5 s
|
|
(`DEPLOY_RECREATE_RETRY_DELAY`). Compose's parallel recreate can lose a container it
|
|
just renamed and leave the rest stopped. If the second attempt fails too, the
|
|
container states are printed and the deploy fails.
|
|
8. Reconciles `apt-cacher-ng` and `signal-api`: each is recreated only if its
|
|
definition changed.
|
|
It then removes services netOrk no longer ships, container and volume, where a host
|
|
still has them: today the bundled `registry` (NetOrk/netork#763). It reports only
|
|
what it actually removed; on a host that was cleaned earlier it says nothing.
|
|
9. Verifies that the containers run exactly the image that was pulled. If they don't, the
|
|
deploy fails.
|
|
10. Runs `alembic upgrade head` inside `netork-api`.
|
|
11. Prunes unused images.
|
|
|
|
A failing step stops that server's deploy with a non-zero exit, and the other servers
|
|
carry on. The script exits non-zero if any server failed and names those servers.
|
|
|
|
## Moving the bundled database
|
|
|
|
A new PostgreSQL major version cannot read the old data directory, so netOrk names a
|
|
new volume for it, for example `postgres17_data` (NetOrk/netork#1308). Moving onto it
|
|
takes `--infra-db`, for one host at an announced time, because the engine is down
|
|
while the data moves.
|
|
|
|
1. Stops the engine.
|
|
2. Counts every table and dumps the database, using `pg_dump -Fc`. The dump stays
|
|
in `~/netork/backups/`.
|
|
3. Brings the new container up on the new volume, then restores and runs
|
|
`ANALYZE`.
|
|
4. Compares the counts and the Alembic revision. If either differs, the move fails
|
|
and says how to go back.
|
|
5. Leaves the old volume as it was.
|
|
|
|
**The way back:** deploy the previous netOrk version with `--infra-db`. Its compose
|
|
files name the old volume again, which still exists, so the container switches onto
|
|
it without a restore. The data is then as it was before the move.
|
|
|
|
## Operating notes
|
|
|
|
- **Never delete these Docker volumes:**
|
|
- `celerybeat_schedule` holds the scheduler state.
|
|
- `signal_cli_data` holds netOrk's Signal device link. Losing it means pairing again
|
|
by QR code.
|
|
- `netork-beat` always rolls out together with the workers. The script does this for
|
|
you; keep it that way if you deploy by hand.
|
|
- Always deploy with this script. Do not point a local Docker client at a remote host
|
|
(`DOCKER_HOST=ssh://…`): it resolves volume paths locally and breaks the remote
|
|
containers.
|
|
|
|
## Development
|
|
|
|
```bash
|
|
pip install pytest ruff shellcheck-py
|
|
shellcheck deploy.sh
|
|
ruff format --check . && ruff check .
|
|
pytest
|
|
```
|
|
|
|
The tests are static and behavioural checks of `deploy.sh`. None of them reach a real
|
|
host.
|
|
|
|
## License
|
|
|
|
MIT, see [LICENSE](LICENSE).
|