CI / check (pull_request) Successful in 19s
The services read ~/netork/.env through env_file, so whatever it says when `up -d` creates a container ends up in that container's environment. The settings were recorded only after the swap, so every container carried the previous deploy's NETORK_VERSION: the image was right, `docker exec <api> env` was not. Record them once the pull has brought the images onto the host and before the first `up -d`. A failed pull still leaves .env alone. If the swap fails afterwards, .env names the tag whose compose files and images are already on the host, which is what a hand-typed `docker compose up -d` there should start. Also say which branches a feature-<branch> tag comes from: feature/<name> and feat/<name> become feature-<name>, and fix/<name> now becomes feature-fix-<name> (NetOrk/netork#208), because the registry keeps only feature-* branch tags. The accepted forms do not change; the comment, the refusal message and the README now say so, and a test holds that feature-fix-<name> passes and a bare fix-<name> does not. Closes #6
145 lines
6.6 KiB
Markdown
145 lines
6.6 KiB
Markdown
# netOrk deploy
|
|
|
|
Deploys [netOrk](https://git.netork.io/NetOrk/netork) to one or more Docker hosts in
|
|
parallel, from pre-built images in a container registry.
|
|
|
|
The machine running the deploy needs this repository, `ssh`, and a `deploy.env`. It does
|
|
not need a netOrk checkout. Each server pulls the images itself. The compose files come
|
|
out of the engine image for the tag being deployed, so they always match the images they
|
|
start, and database migrations run from that same image.
|
|
|
|
## Requirements
|
|
|
|
**On the machine running the deploy:**
|
|
- bash
|
|
- `ssh` with key-based access to every target
|
|
|
|
**On every target server:**
|
|
- at least 4 GB of RAM. On a test server with 74 devices, netOrk's containers took about
|
|
2.4 GB, and at least 0.9 GB stayed free. A large task needs that room. The breakdown
|
|
per container is in netOrk's `docs/TECHNICAL_DESIGN.md`, "Worker memory".
|
|
- Docker with the compose plugin
|
|
- a directory `~/netork/` with netOrk's `.env`, which holds the database and application
|
|
settings the compose file reads (`env_file: .env`)
|
|
|
|
**Images:** `netork/engine` and `netork/ui` must be published in the registry under the
|
|
tag you deploy. The engine image must carry its compose files under `/app/deploy/`.
|
|
Tags built before that change are refused with a clear message.
|
|
|
|
**Raspberry Pi and other arm64 hosts:** a Raspberry Pi 3, 4, 5 or Zero 2 W with a 64-bit
|
|
OS can be a target. Nothing changes in `deploy.env`: each host pulls the variant for its
|
|
own platform. netOrk publishes arm64 images for releases only, from 0.34.0 on (`latest`,
|
|
`X.Y.Z`). `latest-dev`, `main-<sha>`, `feature-<branch>` and older releases are amd64 only,
|
|
and a deploy of such a tag to an arm64 host is refused before the host is touched. The
|
|
images on the host (`postgres`, `redis`, `apt-cacher-ng`, `signal-api`) exist for arm64
|
|
as well.
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
cp deploy.env.example deploy.env
|
|
$EDITOR deploy.env
|
|
```
|
|
|
|
`deploy.env` is gitignored. Point `DEPLOY_ENV_FILE` at another file to keep it
|
|
elsewhere.
|
|
|
|
| Variable | Meaning |
|
|
|---|---|
|
|
| `DEPLOY_SERVERS` | Space-separated default targets |
|
|
| `UI_SERVER` | Targets that also run `netork-ui` |
|
|
| `REGISTRY_HOST` | Registry to pull from; required |
|
|
| `NETORK_VERSION` | Tag to deploy (default `latest`) |
|
|
| `REGISTRY_USER`, `REGISTRY_PASSWORD` | Registry login used on every server |
|
|
| `REGISTRY_USER_<server>`, `REGISTRY_PASSWORD_<server>`, `NETORK_VERSION_<server>` | Per-server overrides. `<server>` has its dots replaced by underscores, e.g. `_10_0_0_2` |
|
|
| `DEPLOY_LOCK_WAIT` | Seconds to wait for another deploy to the same host (default 900) |
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
./deploy.sh # every server in DEPLOY_SERVERS
|
|
./deploy.sh 10.0.0.1 # one server
|
|
./deploy.sh 10.0.0.1 10.0.0.2 # several, in parallel
|
|
./deploy.sh --version=main-1a2b3c4 10.0.0.1 # one-off tag, this run only
|
|
```
|
|
|
|
Only tags netOrk's CI publishes are accepted:
|
|
- `latest`
|
|
- `latest-dev`
|
|
- `main-<sha>`
|
|
- `feature-<name>`, a branch build: `feature/<name>` and `feat/<name>` become
|
|
`feature-<name>`, and `fix/<name>` becomes `feature-fix-<name>`. Any further `/` in
|
|
the branch name becomes `-` as well.
|
|
- `X.Y.Z`
|
|
|
|
Anything else is refused before any server is touched.
|
|
|
|
If you deploy a branch build, wait until its CI run has published the images. A deploy
|
|
started earlier pulls whatever image the registry held before, which is stale.
|
|
|
|
## What a deploy does, per server
|
|
|
|
0. Takes the host's deploy lock, `flock` on `~/netork/.deploy.lock`, and holds it until
|
|
the deploy ends. A second deploy to the same host waits for it and says who holds
|
|
it, since when, and which tag they are deploying. It gives up after 15 minutes
|
|
(`DEPLOY_LOCK_WAIT`, in seconds) without touching the host. The lock belongs to an
|
|
ssh session, so a deploy that fails, is interrupted or loses its connection frees it
|
|
on its own. A host without `flock` is deployed without the lock, with a warning.
|
|
1. Logs in to the registry. The password travels over ssh's stdin, never on a command
|
|
line. Then it checks that the tag has an image for the host's platform (`docker
|
|
version`, `docker manifest inspect`): engine, and on UI servers ui. If not, for
|
|
example `latest-dev` on a Raspberry Pi, it stops here and says which tags would work.
|
|
If the registry's answer can't be read, the pull below decides.
|
|
2. Pulls the engine image and copies `docker-compose.yml` and
|
|
`docker-compose.registry.yml` out of it into `~/netork/`.
|
|
3. Pulls the netOrk images.
|
|
4. Records `REGISTRY_HOST` and `NETORK_VERSION` in `~/netork/.env`, so that a
|
|
hand-typed `docker compose` on the host uses the same images. It does this before
|
|
any container starts, because the services read `.env` into their environment.
|
|
A failed pull leaves `.env` as it was.
|
|
5. Force-recreates the API, the workers, `netork-beat`, `flower` and, on UI servers,
|
|
`netork-ui`. If the recreate fails, it is tried once more after 5 s
|
|
(`DEPLOY_RECREATE_RETRY_DELAY`). Compose's parallel recreate can lose a container it
|
|
just renamed and leave the rest stopped. If the second attempt fails too, the
|
|
container states are printed and the deploy fails.
|
|
6. Reconciles `apt-cacher-ng` and `signal-api`: each is recreated only if its
|
|
definition changed. It never touches `postgres` or `redis`.
|
|
It then removes services netOrk no longer ships, container and volume, where a host
|
|
still has them: today the bundled `registry` (NetOrk/netork#763). It reports only
|
|
what it actually removed; on a host that was cleaned earlier it says nothing.
|
|
7. Verifies that the containers run exactly the image that was pulled. If they don't, the
|
|
deploy fails.
|
|
8. Runs `alembic upgrade head` inside `netork-api`.
|
|
9. Prunes unused images.
|
|
|
|
A failing step stops that server's deploy with a non-zero exit, and the other servers
|
|
carry on. The script exits non-zero if any server failed and names those servers.
|
|
|
|
## Operating notes
|
|
|
|
- **Never delete these Docker volumes:**
|
|
- `celerybeat_schedule` holds the scheduler state.
|
|
- `signal_cli_data` holds netOrk's Signal device link. Losing it means pairing again
|
|
by QR code.
|
|
- `netork-beat` always rolls out together with the workers. The script does this for
|
|
you; keep it that way if you deploy by hand.
|
|
- Always deploy with this script. Do not point a local Docker client at a remote host
|
|
(`DOCKER_HOST=ssh://…`): it resolves volume paths locally and breaks the remote
|
|
containers.
|
|
|
|
## Development
|
|
|
|
```bash
|
|
pip install pytest ruff shellcheck-py
|
|
shellcheck deploy.sh
|
|
ruff format --check . && ruff check .
|
|
pytest
|
|
```
|
|
|
|
The tests are static and behavioural checks of `deploy.sh`. None of them reach a real
|
|
host.
|
|
|
|
## License
|
|
|
|
MIT, see [LICENSE](LICENSE).
|