CI / check (pull_request) Successful in 18s
netork-ui ran nginx as root, and its entrypoint wrote the self-signed certificate into ssl_certs as root, the key at 0600. From NetOrk/netork#1389 on, nginx runs as netork, the API's user, and cannot read that key, so the UI would not start on any existing host. Before the containers are recreated, one container of the engine image of the version being deployed, as root with only CHOWN, runs `chown -Rc netork:netork /app/ssl`. -c names only what it changed, so a host handed over earlier sees nothing. A failure stops the deploy with the old containers running. Older engine images carry the same netork user, so deploying an older version stays possible.
181 lines
8.7 KiB
Markdown
181 lines
8.7 KiB
Markdown
# netOrk deploy
|
|
|
|
Deploys [netOrk](https://git.netork.io/NetOrk/netork) to one or more Docker hosts in
|
|
parallel, from pre-built images in a container registry.
|
|
|
|
The machine running the deploy needs this repository, `ssh`, and a `deploy.env`. It does
|
|
not need a netOrk checkout. Each server pulls the images itself. The compose files come
|
|
out of the engine image for the tag being deployed, so they always match the images they
|
|
start, and database migrations run from that same image.
|
|
|
|
## Requirements
|
|
|
|
**On the machine running the deploy:**
|
|
- bash
|
|
- `ssh` with key-based access to every target
|
|
|
|
**On every target server:**
|
|
- at least 4 GB of RAM. On a test server with 74 devices, netOrk's containers took about
|
|
2.4 GB, and at least 0.9 GB stayed free. A large task needs that room. The breakdown
|
|
per container is in netOrk's `docs/TECHNICAL_DESIGN.md`, "Worker memory".
|
|
- Docker with the compose plugin
|
|
- a directory `~/netork/` with netOrk's `.env`, which holds the database and application
|
|
settings the compose file reads (`env_file: .env`)
|
|
|
|
**Images:** `netork/engine` and `netork/ui` must be published in the registry under the
|
|
tag you deploy. The engine image must carry its compose files under `/app/deploy/`.
|
|
Tags built before that change are refused with a clear message.
|
|
|
|
**Raspberry Pi and other arm64 hosts:** a Raspberry Pi 3, 4, 5 or Zero 2 W with a 64-bit
|
|
OS can be a target. Nothing changes in `deploy.env`: each host pulls the variant for its
|
|
own platform. netOrk publishes arm64 images for releases only, from 0.34.0 on (`latest`,
|
|
`X.Y.Z`). `latest-dev`, `main-<sha>`, `feature-<branch>` and older releases are amd64 only,
|
|
and a deploy of such a tag to an arm64 host is refused before the host is touched. The
|
|
images on the host (`postgres`, `redis`, `apt-cacher-ng`, `signal-api`) exist for arm64
|
|
as well.
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
cp deploy.env.example deploy.env
|
|
$EDITOR deploy.env
|
|
```
|
|
|
|
`deploy.env` is gitignored. Point `DEPLOY_ENV_FILE` at another file to keep it
|
|
elsewhere.
|
|
|
|
| Variable | Meaning |
|
|
|---|---|
|
|
| `DEPLOY_SERVERS` | Space-separated default targets |
|
|
| `UI_SERVER` | Targets that also run `netork-ui` |
|
|
| `REGISTRY_HOST` | Registry to pull from; required |
|
|
| `NETORK_VERSION` | Tag to deploy (default `latest`) |
|
|
| `REGISTRY_USER`, `REGISTRY_PASSWORD` | Registry login used on every server |
|
|
| `REGISTRY_USER_<server>`, `REGISTRY_PASSWORD_<server>`, `NETORK_VERSION_<server>` | Per-server overrides. `<server>` has its dots replaced by underscores, e.g. `_10_0_0_2` |
|
|
| `DEPLOY_LOCK_WAIT` | Seconds to wait for another deploy to the same host (default 900) |
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
./deploy.sh # every server in DEPLOY_SERVERS
|
|
./deploy.sh 10.0.0.1 # one server
|
|
./deploy.sh 10.0.0.1 10.0.0.2 # several, in parallel
|
|
./deploy.sh --version=main-1a2b3c4 10.0.0.1 # one-off tag, this run only
|
|
./deploy.sh --infra-db 10.0.0.1 # also move the bundled database (see below)
|
|
```
|
|
|
|
Only tags netOrk's CI publishes are accepted:
|
|
- `latest`
|
|
- `latest-dev`
|
|
- `main-<sha>`
|
|
- `feature-<name>`, a branch build: `feature/<name>` and `feat/<name>` become
|
|
`feature-<name>`, and `fix/<name>` becomes `feature-fix-<name>`. Any further `/` in
|
|
the branch name becomes `-` as well.
|
|
- `X.Y.Z`
|
|
|
|
Anything else is refused before any server is touched.
|
|
|
|
If you deploy a branch build, wait until its CI run has published the images. A deploy
|
|
started earlier pulls whatever image the registry held before, which is stale.
|
|
|
|
## What a deploy does, per server
|
|
|
|
0. Takes the host's deploy lock, `flock` on `~/netork/.deploy.lock`, and holds it until
|
|
the deploy ends. A second deploy to the same host waits for it and says who holds
|
|
it, since when, and which tag they are deploying. It gives up after 15 minutes
|
|
(`DEPLOY_LOCK_WAIT`, in seconds) without touching the host. The lock belongs to an
|
|
ssh session, so a deploy that fails, is interrupted or loses its connection frees it
|
|
on its own. A host without `flock` is deployed without the lock, with a warning.
|
|
1. Logs in to the registry. The password travels over ssh's stdin, never on a command
|
|
line. Then it checks that the tag has an image for the host's platform (`docker
|
|
version`, `docker manifest inspect`): engine, and on UI servers ui. If not, for
|
|
example `latest-dev` on a Raspberry Pi, it stops here and says which tags would work.
|
|
If the registry's answer can't be read, the pull below decides.
|
|
2. Pulls the engine image and copies `docker-compose.yml` and
|
|
`docker-compose.registry.yml` out of it into `~/netork/`.
|
|
3. Pulls the netOrk images.
|
|
4. Records `REGISTRY_HOST` and `NETORK_VERSION` in `~/netork/.env`, so that a
|
|
hand-typed `docker compose` on the host uses the same images. It does this before
|
|
any container starts, because the services read `.env` into their environment.
|
|
A failed pull leaves `.env` as it was.
|
|
5. Reconciles the bundled PostgreSQL, if the host has one (`remote/database.sh`,
|
|
over ssh's stdin). On the volume it already uses, it is recreated only if its image
|
|
or definition changed, so a new minor release arrives with the next deploy. If the
|
|
compose files name another volume (a new major version), a normal deploy stops
|
|
here before anything changed: the host keeps its previous compose files and
|
|
`.env`, so a hand-typed `docker compose up -d` cannot move the database either.
|
|
See *Moving the bundled database*. This step runs before `.env` is written.
|
|
6. Reconciles the broker, `redis` (Valkey): recreated only if its definition changed,
|
|
and waited for until healthy, before the engine comes up against it. If it does
|
|
not get healthy within 120 s, the deploy stops with the old engine running.
|
|
7. Hands the `ssl_certs` volume to `netork`, the user the API and nginx run as
|
|
(NetOrk/netork#1389), in one container of the engine image run as root with only the
|
|
capability to chown. A volume written by the old root nginx would keep the UI from
|
|
starting. `chown -c` names only what it changed; on a host handed over earlier it
|
|
does nothing. If it fails, the deploy stops with the old containers running.
|
|
8. Force-recreates the API, the workers, `netork-beat`, `flower` and, on UI servers,
|
|
`netork-ui`. If the recreate fails, it is tried once more after 5 s
|
|
(`DEPLOY_RECREATE_RETRY_DELAY`). Compose's parallel recreate can lose a container it
|
|
just renamed and leave the rest stopped. If the second attempt fails too, the
|
|
container states are printed and the deploy fails.
|
|
9. Reconciles `apt-cacher-ng` and `signal-api`: each is recreated only if its
|
|
definition changed.
|
|
It then removes services netOrk no longer ships, container and volume, where a host
|
|
still has them: today the bundled `registry` (NetOrk/netork#763). It reports only
|
|
what it actually removed; on a host that was cleaned earlier it says nothing.
|
|
10. Verifies that the containers run exactly the image that was pulled. If they don't, the
|
|
deploy fails.
|
|
11. Runs `alembic upgrade head` inside `netork-api`.
|
|
12. Prunes unused images.
|
|
|
|
A failing step stops that server's deploy with a non-zero exit, and the other servers
|
|
carry on. The script exits non-zero if any server failed and names those servers.
|
|
|
|
## Moving the bundled database
|
|
|
|
A new PostgreSQL major version cannot read the old data directory, so netOrk names a
|
|
new volume for it, for example `postgres17_data` (NetOrk/netork#1308). Moving onto it
|
|
takes `--infra-db`, for one host at an announced time, because the engine is down
|
|
while the data moves.
|
|
|
|
1. Stops the engine.
|
|
2. Counts every table and dumps the database, using `pg_dump -Fc`. The dump stays
|
|
in `~/netork/backups/`.
|
|
3. Brings the new container up on the new volume, then restores and runs
|
|
`ANALYZE`.
|
|
4. Compares the counts and the Alembic revision. If either differs, the move fails
|
|
and says how to go back.
|
|
5. Leaves the old volume as it was.
|
|
|
|
**The way back:** deploy the previous netOrk version with `--infra-db`. Its compose
|
|
files name the old volume again, which still exists, so the container switches onto
|
|
it without a restore. The data is then as it was before the move.
|
|
|
|
## Operating notes
|
|
|
|
- **Never delete these Docker volumes:**
|
|
- `celerybeat_schedule` holds the scheduler state.
|
|
- `signal_cli_data` holds netOrk's Signal device link. Losing it means pairing again
|
|
by QR code.
|
|
- `netork-beat` always rolls out together with the workers. The script does this for
|
|
you; keep it that way if you deploy by hand.
|
|
- Always deploy with this script. Do not point a local Docker client at a remote host
|
|
(`DOCKER_HOST=ssh://…`): it resolves volume paths locally and breaks the remote
|
|
containers.
|
|
|
|
## Development
|
|
|
|
```bash
|
|
pip install pytest ruff shellcheck-py
|
|
shellcheck deploy.sh
|
|
ruff format --check . && ruff check .
|
|
pytest
|
|
```
|
|
|
|
The tests are static and behavioural checks of `deploy.sh`. None of them reach a real
|
|
host.
|
|
|
|
## License
|
|
|
|
MIT, see [LICENSE](LICENSE).
|