Managing watchers
The watcher fleet from the admin side: monitoring health, deregistering watchers, approving manual uploads, and rolling releases across the fleet.
The Watchers area of the web app is the fleet view: every registered watcher, its health, and the controls for deregistering watchers and rolling out releases.
Monitoring watcher health
Each watcher reports a heartbeat every 60 seconds carrying its version, instrument, watch directory, upload mode, activity counters, and uptime. The fleet list surfaces:
- Effective status: online vs. offline, derived from the last heartbeat.
- Last heartbeat: a stale value is the first sign of a watcher that's stopped, lost connectivity, or crashed.
- Hostname and instrument assignment: which PC is watching which instrument.
- Events: the per-watcher log (
watcher_started,run_reported,file_uploaded,update_*, anderrorwith adetails.kinddiscriminator). Events are your primary remote-diagnosis tool.
For a guided diagnosis, the troubleshoot_instrument MCP prompt inspects an
instrument's status, watcher heartbeats, and recent runs; see the
MCP server reference.
Deregistering a watcher
Each instrument can have at most one active watcher. Deregister the old one when a PC is reimaged or replaced, or to clear the “instrument already has an active watcher” error a new install runs into.
- Web app: open the watcher under Watchers and click Deregister.
- API:
DELETE /api/v1/watchers/:watcherId(requireswatchers:write).
Deregistration is a soft-delete: the watcher's heartbeats, events, and reported runs stay visible under Watchers > Deregistered for auditing.
Approving manual uploads
For instruments configured in manual upload mode, the watcher reports runs but waits for server approval before uploading file bytes. The watcher polls the upload queue on each heartbeat tick and uploads the files the server has marked for upload.
A couple of safeguards keep the queue healthy:
- Changing an instrument's
watch_directoryreverts its pending upload requests todetectedso the queue drains immediately rather than erroring on stale paths. - A queued file that keeps failing (missing on disk or failing to upload) is retried on at most three heartbeat polls, then cancelled server-side so it leaves the queue instead of erroring every tick.
Releases and fleet updates
Lab PCs upgrade themselves toward the version you advertise per environment. You control that from Settings → Watchers. For how upgrades land on an individual PC, see Upgrading the watcher.
Cutting a new release
Releases are tag-driven and publish from a production commit.
Bump the version in watcher/pyproject.toml and merge through staging to
production.
Tag and push from production:
git checkout production && git pull
git tag watcher-v0.3.0 && git push origin watcher-v0.3.0Approve the pypi deployment under Actions → Publish watcher in GitHub.
Advertise the release: in Settings → Watchers, set Latest version to
the new tag. Each environment has its own database, so bump staging and
production separately to roll the fleet gradually.
Always tag → publish → verify → save. Saving a Latest version before the wheel
is live on PyPI triggers a wave of update_failed events across the fleet.
Release-config fields
| Field | Purpose |
|---|---|
| Latest version | Required. Blank = “no update info available”. |
| Minimum supported version | Reserved; not enforced server-side yet. |
| Release channel | Defaults to stable. Surfaced in self-update output. |
| Mandatory update | Skips the activity-window guard; see below. |
Mandatory updates
Toggling Mandatory update skips the activity-window guard and fires the upgrade on the next hourly check. The server still compares the running version to Latest version, so a correctly-pinned PC isn't forced.
Use sparingly: a forced upgrade mid-acquisition loses data. Reserve it for security fixes, wire-protocol breaks, or cases where leaving the bad version running is strictly worse than restarting in flight.
Rolling back
Rollback is another release: set Latest version back to the older tag for the affected environment(s), toggle Mandatory update on if you need it to bypass the activity-window guard, and wait for the next hourly tick. Toggle Mandatory update back off once the fleet converges. A rolled-back release is still on PyPI and reinstallable; it's no longer advertised.
Related pages
- Upgrading the watcher: the per-PC upgrade mechanics and troubleshooting.
- Managing instruments: confirming and deactivating instruments.
- API reference: the watcher endpoints.