docs: write the README for people installing netOrk themselves #12
No files matched your search
@@ -1,12 +1,12 @@
|
|||||||
# netOrk deploy
|
# netOrk deploy
|
||||||
|
|
||||||
Deploys [netOrk](https://git.netork.io/NetOrk/netork) to one or more Docker hosts in
|
Installs and updates [netOrk](https://git.netork.io/NetOrk/netork) on one or more of your
|
||||||
parallel, from pre-built images in a container registry.
|
own Docker hosts, from netOrk's published images.
|
||||||
|
|
||||||
The machine running the deploy needs this repository, `ssh`, and a `deploy.env`. It does
|
The machine you run the deploy from needs this repository, `ssh` and a `deploy.env`.
|
||||||
not need a netOrk checkout. Each server pulls the images itself. The compose files come
|
Each server pulls the images itself. The compose files come out of the image for the
|
||||||
out of the engine image for the tag being deployed, so they always match the images they
|
version being deployed, so they always match the images they start, and database
|
||||||
start, and database migrations run from that same image.
|
migrations run from that same image.
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
@@ -15,24 +15,16 @@ start, and database migrations run from that same image.
|
|||||||
- `ssh` with key-based access to every target
|
- `ssh` with key-based access to every target
|
||||||
|
|
||||||
**On every target server:**
|
**On every target server:**
|
||||||
- at least 4 GB of RAM. On a test server with 74 devices, netOrk's containers took about
|
- at least 4 GB of RAM. On a server with 74 devices, netOrk's containers took about
|
||||||
2.4 GB, and at least 0.9 GB stayed free. A large task needs that room. The breakdown
|
2.4 GB, and at least 0.9 GB stayed free. Larger tasks need that room.
|
||||||
per container is in netOrk's `docs/TECHNICAL_DESIGN.md`, "Worker memory".
|
|
||||||
- Docker with the compose plugin
|
- Docker with the compose plugin
|
||||||
- a directory `~/netork/` with netOrk's `.env`, which holds the database and application
|
- a directory `~/netork/` with netOrk's `.env`, which holds the database and application
|
||||||
settings the compose file reads (`env_file: .env`)
|
settings
|
||||||
|
|
||||||
**Images:** `netork/engine` and `netork/ui` must be published in the registry under the
|
|
||||||
tag you deploy. The engine image must carry its compose files under `/app/deploy/`.
|
|
||||||
Tags built before that change are refused with a clear message.
|
|
||||||
|
|
||||||
**Raspberry Pi and other arm64 hosts:** a Raspberry Pi 3, 4, 5 or Zero 2 W with a 64-bit
|
**Raspberry Pi and other arm64 hosts:** a Raspberry Pi 3, 4, 5 or Zero 2 W with a 64-bit
|
||||||
OS can be a target. Nothing changes in `deploy.env`: each host pulls the variant for its
|
OS can be a target, from netOrk 0.34.0 on. Nothing changes in `deploy.env`: each host
|
||||||
own platform. netOrk publishes arm64 images for releases only, from 0.34.0 on (`latest`,
|
pulls the variant for its own platform. A version that has no arm64 image is refused
|
||||||
`X.Y.Z`). `latest-dev`, `main-<sha>`, `feature-<branch>` and older releases are amd64 only,
|
before the host is touched.
|
||||||
and a deploy of such a tag to an arm64 host is refused before the host is touched. The
|
|
||||||
images on the host (`postgres`, `redis`, `apt-cacher-ng`, `signal-api`) exist for arm64
|
|
||||||
as well.
|
|
||||||
|
|
||||||
## Setup
|
## Setup
|
||||||
|
|
||||||
@@ -41,15 +33,14 @@ cp deploy.env.example deploy.env
|
|||||||
$EDITOR deploy.env
|
$EDITOR deploy.env
|
||||||
```
|
```
|
||||||
|
|
||||||
`deploy.env` is gitignored. Point `DEPLOY_ENV_FILE` at another file to keep it
|
Set `DEPLOY_ENV_FILE` to keep the file somewhere else.
|
||||||
elsewhere.
|
|
||||||
|
|
||||||
| Variable | Meaning |
|
| Variable | Meaning |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `DEPLOY_SERVERS` | Space-separated default targets |
|
| `DEPLOY_SERVERS` | Space-separated default targets |
|
||||||
| `UI_SERVER` | Targets that also run `netork-ui` |
|
| `UI_SERVER` | Targets that also run the web interface (`netork-ui`) |
|
||||||
| `REGISTRY_HOST` | Registry to pull from; required |
|
| `REGISTRY_HOST` | Registry to pull from; required |
|
||||||
| `NETORK_VERSION` | Tag to deploy (default `latest`) |
|
| `NETORK_VERSION` | Version to deploy (default `latest`) |
|
||||||
| `REGISTRY_USER`, `REGISTRY_PASSWORD` | Registry login used on every server |
|
| `REGISTRY_USER`, `REGISTRY_PASSWORD` | Registry login used on every server |
|
||||||
| `REGISTRY_USER_<server>`, `REGISTRY_PASSWORD_<server>`, `NETORK_VERSION_<server>` | Per-server overrides. `<server>` has its dots replaced by underscores, e.g. `_10_0_0_2` |
|
| `REGISTRY_USER_<server>`, `REGISTRY_PASSWORD_<server>`, `NETORK_VERSION_<server>` | Per-server overrides. `<server>` has its dots replaced by underscores, e.g. `_10_0_0_2` |
|
||||||
| `DEPLOY_LOCK_WAIT` | Seconds to wait for another deploy to the same host (default 900) |
|
| `DEPLOY_LOCK_WAIT` | Seconds to wait for another deploy to the same host (default 900) |
|
||||||
@@ -60,60 +51,37 @@ elsewhere.
|
|||||||
./deploy.sh # every server in DEPLOY_SERVERS
|
./deploy.sh # every server in DEPLOY_SERVERS
|
||||||
./deploy.sh 10.0.0.1 # one server
|
./deploy.sh 10.0.0.1 # one server
|
||||||
./deploy.sh 10.0.0.1 10.0.0.2 # several, in parallel
|
./deploy.sh 10.0.0.1 10.0.0.2 # several, in parallel
|
||||||
./deploy.sh --version=main-1a2b3c4 10.0.0.1 # one-off tag, this run only
|
./deploy.sh --version=0.37.0 10.0.0.1 # a specific version, this run only
|
||||||
```
|
```
|
||||||
|
|
||||||
Only tags netOrk's CI publishes are accepted:
|
`NETORK_VERSION` and `--version` take `latest` for the newest release or a release
|
||||||
- `latest`
|
number such as `0.37.0`.
|
||||||
- `latest-dev`
|
|
||||||
- `main-<sha>`
|
|
||||||
- `feature-<name>`, a branch build: `feature/<name>` and `feat/<name>` become
|
|
||||||
`feature-<name>`, and `fix/<name>` becomes `feature-fix-<name>`. Any further `/` in
|
|
||||||
the branch name becomes `-` as well.
|
|
||||||
- `X.Y.Z`
|
|
||||||
|
|
||||||
Anything else is refused before any server is touched.
|
The same command installs netOrk the first time and updates it later.
|
||||||
|
|
||||||
If you deploy a branch build, wait until its CI run has published the images. A deploy
|
|
||||||
started earlier pulls whatever image the registry held before, which is stale.
|
|
||||||
|
|
||||||
## What a deploy does, per server
|
## What a deploy does, per server
|
||||||
|
|
||||||
0. Takes the host's deploy lock, `flock` on `~/netork/.deploy.lock`, and holds it until
|
1. Makes sure only one deploy runs on the host at a time. A second deploy to the same
|
||||||
the deploy ends. A second deploy to the same host waits for it and says who holds
|
host waits, says who is deploying which version, and gives up after 15 minutes
|
||||||
it, since when, and which tag they are deploying. It gives up after 15 minutes
|
(`DEPLOY_LOCK_WAIT`) without touching the host.
|
||||||
(`DEPLOY_LOCK_WAIT`, in seconds) without touching the host. The lock belongs to an
|
2. Logs in to the registry. The password travels over ssh's stdin, never on a command
|
||||||
ssh session, so a deploy that fails, is interrupted or loses its connection frees it
|
line. Then it checks that the version has an image for the host's platform and stops
|
||||||
on its own. A host without `flock` is deployed without the lock, with a warning.
|
if it doesn't.
|
||||||
1. Logs in to the registry. The password travels over ssh's stdin, never on a command
|
3. Pulls the images and copies the matching compose files into `~/netork/`.
|
||||||
line. Then it checks that the tag has an image for the host's platform (`docker
|
4. Records the registry and version in `~/netork/.env`, so that a `docker compose`
|
||||||
version`, `docker manifest inspect`): engine, and on UI servers ui. If not, for
|
command typed on the host uses the same images. A failed pull leaves `.env` as it
|
||||||
example `latest-dev` on a Raspberry Pi, it stops here and says which tags would work.
|
was.
|
||||||
If the registry's answer can't be read, the pull below decides.
|
5. Restarts netOrk's containers on the new images. If that fails, it tries once more;
|
||||||
2. Pulls the engine image and copies `docker-compose.yml` and
|
if it fails again, it prints the container states and stops.
|
||||||
`docker-compose.registry.yml` out of it into `~/netork/`.
|
6. Updates the helper services (`apt-cacher-ng`, `signal-api`) only when their
|
||||||
3. Pulls the netOrk images.
|
definition changed, and removes services netOrk no longer ships. It never touches
|
||||||
4. Records `REGISTRY_HOST` and `NETORK_VERSION` in `~/netork/.env`, so that a
|
the database (`postgres`) or `redis`.
|
||||||
hand-typed `docker compose` on the host uses the same images. It does this before
|
7. Verifies that the containers run exactly the images that were pulled.
|
||||||
any container starts, because the services read `.env` into their environment.
|
8. Applies database migrations.
|
||||||
A failed pull leaves `.env` as it was.
|
9. Removes unused images.
|
||||||
5. Force-recreates the API, the workers, `netork-beat`, `flower` and, on UI servers,
|
|
||||||
`netork-ui`. If the recreate fails, it is tried once more after 5 s
|
|
||||||
(`DEPLOY_RECREATE_RETRY_DELAY`). Compose's parallel recreate can lose a container it
|
|
||||||
just renamed and leave the rest stopped. If the second attempt fails too, the
|
|
||||||
container states are printed and the deploy fails.
|
|
||||||
6. Reconciles `apt-cacher-ng` and `signal-api`: each is recreated only if its
|
|
||||||
definition changed. It never touches `postgres` or `redis`.
|
|
||||||
It then removes services netOrk no longer ships, container and volume, where a host
|
|
||||||
still has them: today the bundled `registry` (NetOrk/netork#763). It reports only
|
|
||||||
what it actually removed; on a host that was cleaned earlier it says nothing.
|
|
||||||
7. Verifies that the containers run exactly the image that was pulled. If they don't, the
|
|
||||||
deploy fails.
|
|
||||||
8. Runs `alembic upgrade head` inside `netork-api`.
|
|
||||||
9. Prunes unused images.
|
|
||||||
|
|
||||||
A failing step stops that server's deploy with a non-zero exit, and the other servers
|
If a step fails, that server's deploy stops and the other servers carry on. The script
|
||||||
carry on. The script exits non-zero if any server failed and names those servers.
|
exits non-zero if any server failed and names those servers.
|
||||||
|
|
||||||
## Operating notes
|
## Operating notes
|
||||||
|
|
||||||
@@ -121,24 +89,10 @@ carry on. The script exits non-zero if any server failed and names those servers
|
|||||||
- `celerybeat_schedule` holds the scheduler state.
|
- `celerybeat_schedule` holds the scheduler state.
|
||||||
- `signal_cli_data` holds netOrk's Signal device link. Losing it means pairing again
|
- `signal_cli_data` holds netOrk's Signal device link. Losing it means pairing again
|
||||||
by QR code.
|
by QR code.
|
||||||
- `netork-beat` always rolls out together with the workers. The script does this for
|
|
||||||
you; keep it that way if you deploy by hand.
|
|
||||||
- Always deploy with this script. Do not point a local Docker client at a remote host
|
- Always deploy with this script. Do not point a local Docker client at a remote host
|
||||||
(`DOCKER_HOST=ssh://…`): it resolves volume paths locally and breaks the remote
|
(`DOCKER_HOST=ssh://…`): it resolves volume paths locally and breaks the remote
|
||||||
containers.
|
containers.
|
||||||
|
|
||||||
## Development
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install pytest ruff shellcheck-py
|
|
||||||
shellcheck deploy.sh
|
|
||||||
ruff format --check . && ruff check .
|
|
||||||
pytest
|
|
||||||
```
|
|
||||||
|
|
||||||
The tests are static and behavioural checks of `deploy.sh`. None of them reach a real
|
|
||||||
host.
|
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
MIT, see [LICENSE](LICENSE).
|
MIT, see [LICENSE](LICENSE).
|
||||||
+2
-2
@@ -10,7 +10,7 @@ UI_SERVER="10.0.0.1"
|
|||||||
# Registry the servers pull netOrk's images from.
|
# Registry the servers pull netOrk's images from.
|
||||||
REGISTRY_HOST=registry.example.com
|
REGISTRY_HOST=registry.example.com
|
||||||
|
|
||||||
# Image tag to deploy: latest, latest-dev, main-<sha>, feature-<branch> or X.Y.Z.
|
# Version to deploy: latest for the newest release, or a release number such as 0.37.0.
|
||||||
NETORK_VERSION=latest
|
NETORK_VERSION=latest
|
||||||
|
|
||||||
# Registry credentials used on every server without its own entry below.
|
# Registry credentials used on every server without its own entry below.
|
||||||
@@ -21,4 +21,4 @@ REGISTRY_PASSWORD=
|
|||||||
# with dots replaced by underscores.
|
# with dots replaced by underscores.
|
||||||
REGISTRY_USER_10_0_0_2=
|
REGISTRY_USER_10_0_0_2=
|
||||||
REGISTRY_PASSWORD_10_0_0_2=
|
REGISTRY_PASSWORD_10_0_0_2=
|
||||||
NETORK_VERSION_10_0_0_2=latest-dev
|
NETORK_VERSION_10_0_0_2=0.37.0
|
||||||
Reference in new issue
Block a user