deploy.sh: two deploys to one host at once recreate each other's containers #1

Closed
opened 2026-10-05 05:45:07 +00:00 by christianmanivong · 1 comment
Owner

What happened (2026-10-05, 07:42 CEST, 172.22.8.50)

Two sessions ran deploy.sh 172.22.8.50 (both latest-dev) within the same minute. Each run's docker compose up -d --force-recreate stopped and renamed the containers the other run was creating:

  • Error when allocating new name: Conflict. The container name "/netork-netork-worker-1" is already in use by container "4e895ef6…"
  • leftover containers named <id>_netork-netork-api-1 / <id>_netork-netork-worker-1 in state Created
  • dependency failed to start: container netork-netork-api-1 exited (137) — killed by the other run's recreate, not by OOM (OOMKilled=false)

For about a minute the API and the workers were down, and the UI container sat in Created. It recovered only because one of the two runs happened to finish last and cleanly. Nothing in the tooling made that the outcome: a slightly different interleaving would have left the host half-recreated, with both runs reporting FAILED.

Why it happens

deploy.sh holds no lock. Nothing on the target host stops a second docker compose up while one is in flight, and nothing on the workstation does either. With several sessions deploying to the same test server, overlap is a matter of time.

Options

  • A lock on the target host, around the compose and migration steps: flock on a file under ~/netork, taken over ssh. It also covers deploys from different workstations. A second run should either wait with a visible "another deploy is running on since
  • A local lock alone (flock on the workstation) would not cover another machine, and the test server is the shared resource.

The migration step matters here too: two alembic upgrade head runs at once are exactly what this would serialise.

## What happened (2026-10-05, 07:42 CEST, 172.22.8.50) Two sessions ran `deploy.sh 172.22.8.50` (both `latest-dev`) within the same minute. Each run's `docker compose up -d --force-recreate` stopped and renamed the containers the other run was creating: - `Error when allocating new name: Conflict. The container name "/netork-netork-worker-1" is already in use by container "4e895ef6…"` - leftover containers named `<id>_netork-netork-api-1` / `<id>_netork-netork-worker-1` in state `Created` - `dependency failed to start: container netork-netork-api-1 exited (137)` — killed by the other run's recreate, not by OOM (`OOMKilled=false`) For about a minute the API and the workers were down, and the UI container sat in `Created`. It recovered only because one of the two runs happened to finish last and cleanly. Nothing in the tooling made that the outcome: a slightly different interleaving would have left the host half-recreated, with both runs reporting `FAILED`. ## Why it happens `deploy.sh` holds no lock. Nothing on the target host stops a second `docker compose up` while one is in flight, and nothing on the workstation does either. With several sessions deploying to the same test server, overlap is a matter of time. ## Options - **A lock on the target host**, around the compose and migration steps: `flock` on a file under `~/netork`, taken over ssh. It also covers deploys from different workstations. A second run should either wait with a visible "another deploy is running on <host> since <time>" or fail fast with that message. - A local lock alone (`flock` on the workstation) would not cover another machine, and the test server is the shared resource. The migration step matters here too: two `alembic upgrade head` runs at once are exactly what this would serialise.
christianmanivong added the
prio
P3
label 2026-10-05 05:45:07 +00:00
Author
Owner

Happened again on 2026-10-06, 172.22.8.50, while moving the host from feature-sidebar-regroup back to latest-dev after NetOrk/netork#717 merged.

  • Two sessions ran deploy.sh at once: one with --version=latest-dev, the other with the deploy.env pin, which is also latest-dev. A third session was polling for the host to leave its feature-* image so it could deploy next.
  • One run failed twice with dependency failed to start: Error response from daemon: No such container: 3513b3df…. After its second attempt it left netork-beat-1, netork-poll-worker-1, flower-1 and netork-ui-1 in Created.
  • The other run finished last and cleanly, so the host recovered on its own: every container on latest-dev, healthy, migrations at head.

Same mechanism as above. This time beat was also down for about a minute, not only the API and the workers. The more sessions queue up behind a feature deploy, the likelier the overlap: they all start the moment the host frees up.

Happened again on 2026-10-06, 172.22.8.50, while moving the host from `feature-sidebar-regroup` back to `latest-dev` after NetOrk/netork#717 merged. - Two sessions ran `deploy.sh` at once: one with `--version=latest-dev`, the other with the deploy.env pin, which is also `latest-dev`. A third session was polling for the host to leave its `feature-*` image so it could deploy next. - One run failed twice with `dependency failed to start: Error response from daemon: No such container: 3513b3df…`. After its second attempt it left `netork-beat-1`, `netork-poll-worker-1`, `flower-1` and `netork-ui-1` in `Created`. - The other run finished last and cleanly, so the host recovered on its own: every container on `latest-dev`, healthy, migrations at head. Same mechanism as above. This time beat was also down for about a minute, not only the API and the workers. The more sessions queue up behind a feature deploy, the likelier the overlap: they all start the moment the host frees up.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: NetOrk/deploy#1