# Adding an instrument (/docs/adding-an-instrument) Lab operators and admins This guide walks through adding a new lab instrument to Data Hub end to end: registering it on the instrument PC, having an admin activate it, and starting to upload runs. Registration is done by a lab operator; activation (Step 2) is an admin action. ## Prerequisites [#prerequisites] * **[uv](https://docs.astral.sh/uv/getting-started/installation/)** installed on the instrument PC: it handles Python for you, so no separate Python install is needed. * **A personal access token** (`dhub_…`). Ask an admin if you don't have one; see [Managing tokens](/docs/managing-tokens). * **Access to the instrument PC** where the files are generated. ## Step 1: Install the watcher and register the instrument [#step-1-install-the-watcher-and-register-the-instrument] Follow [Installing a watcher](/docs/installing-a-watcher). During `data-hub-watcher init`, choose **Register a new instrument** and provide: * **Instrument ID**: a kebab-case identifier (e.g., `bio-rad-cfx96`). This becomes the storage key prefix and the permanent identifier across the system. * **Display name**: a human-readable name (e.g., "Bio-Rad CFX96"). Defaults to a title-cased version of the ID. The instrument is created with status `pending`. ## Step 2: Activate the instrument (admin) [#step-2-activate-the-instrument-admin] An admin must confirm the instrument before the watcher can upload. Open the Data Hub web app and navigate to the **Instruments** page. Find the new instrument; it has a yellow `pending` badge. Click **Confirm** next to it. The status changes to `active`. See [Managing instruments](/docs/managing-instruments) for the full admin view. ## Step 3: Start watching [#step-3-start-watching] On the instrument PC: ```sh data-hub-watcher watch ``` The watcher detects new files, groups them into runs, uploads them, and reports everything to the API. Files become viewable in the web dashboard immediately after upload. ## What you get [#what-you-get] Once the instrument is active and the watcher is running: * The watcher uploads raw files to cloud storage. * Runs and files appear in the web dashboard. * Files are downloadable via pre-signed URLs. * Watcher health monitoring (heartbeats, events) works in the dashboard. * Slack notifications fire whenever a new run is created. Some instruments also support **automated preprocessing** (metadata extraction, image processing, and similar) that runs server-side as files arrive. Adding a new processor requires changes to the Data Hub codebase; see [Adding a new instrument](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/lambda.md#adding-a-new-instrument) in the data-hub developer docs. # API reference (/docs/api-reference) Developers and integrators The Data Hub API is served by the web application at `/api/v1/`. It's used by the watcher, the Lambda, Model Context Protocol (MCP) clients, and the web dashboard. Data Hub is self-hosted, so the base URL is your own deployment's host followed by `/api/v1`. The examples below use `https://datahub.example.com`; substitute your host. ## Authentication [#authentication] Two methods (see [Security & permissions](/docs/security) for the full model): * **Session cookies**: the web dashboard (Google OAuth). Session callers implicitly hold every scope. * **Bearer tokens**: `Authorization: Bearer dhub_…`, used by the watcher, Lambda, and MCP clients. ```sh curl https://datahub.example.com/api/v1/instruments \ -H "Authorization: Bearer dhub_your_token_here" ``` Token-authenticated requests are gated on [scopes](/docs/security#token-scopes): a request missing the route's required scope is rejected with `403 FORBIDDEN`. Some mutations additionally require the admin role; see [Admin-gated operations](/docs/security#admin-gated-operations). ## Endpoints [#endpoints] ### Instruments [#instruments] | Method | Path | Description | | ------- | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `GET` | `/instruments` | List all instruments. | | `POST` | `/instruments` | Create a new instrument. | | `GET` | `/instruments/:instrumentId` | Get instrument details (includes run and watcher counts). | | `PATCH` | `/instruments/:instrumentId` | Update an instrument (e.g. confirm/deactivate). Admin-only for session callers; `instruments:write` for bearer tokens. | ### Runs [#runs] | Method | Path | Description | | -------- | ------------------------------------------------ | --------------------------------- | | `GET` | `/instruments/:instrumentId/runs` | List runs for an instrument. | | `POST` | `/instruments/:instrumentId/runs` | Create a new run. | | `GET` | `/instruments/:instrumentId/runs/:runId` | Get run details. | | `PATCH` | `/instruments/:instrumentId/runs/:runId` | Update a run. | | `DELETE` | `/instruments/:instrumentId/runs/:runId` | Soft-delete a run. | | `POST` | `/instruments/:instrumentId/runs/:runId/restore` | Restore a soft-deleted run. | | `GET` | `/instrument-runs` | List runs across all instruments. | ### Comments [#comments] | Method | Path | Description | | -------- | ------------------------------------------------------------ | -------------------------------------------------- | | `GET` | `/instruments/:instrumentId/runs/:runId/comments` | List active comments on a run (oldest first). | | `POST` | `/instruments/:instrumentId/runs/:runId/comments` | Create a markdown comment (max 10,000 characters). | | `PATCH` | `/instruments/:instrumentId/runs/:runId/comments/:commentId` | Edit a comment (author only; sets `edited_at`). | | `DELETE` | `/instruments/:instrumentId/runs/:runId/comments/:commentId` | Soft-delete a comment (author only). | Cross-user edit/delete returns `403 FORBIDDEN`; mutating a comment whose parent run is soft-deleted returns `409 CONFLICT`. ### Files [#files] | Method | Path | Description | | ------- | ----------------------------------------------------------- | ---------------------------------- | | `POST` | `/instruments/:instrumentId/runs/:runId/files` | Create a file record. | | `GET` | `/files/:fileId` | Get file details. | | `PATCH` | `/files/:fileId` | Update file metadata. | | `GET` | `/files/:fileId/download` | Get a pre-signed S3 download URL. | | `POST` | `/instruments/:instrumentId/runs/:runId/request-upload` | Request file upload (manual mode). | | `POST` | `/instruments/:instrumentId/runs/:runId/request-upload-url` | Get a pre-signed S3 upload URL. | ### Watchers [#watchers] | Method | Path | Description | | -------- | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `GET` | `/watchers` | List all watchers. | | `POST` | `/watchers/register` | Register a watcher. Returns `409 CONFLICT` (with `error.details.existing_watcher_id`) if the instrument already has an active watcher. | | `GET` | `/watchers/:watcherId` | Get watcher details. | | `DELETE` | `/watchers/:watcherId` | Deregister (soft-delete) a watcher. | | `POST` | `/watchers/:watcherId/heartbeat` | Send a heartbeat. | | `GET` | `/watchers/:watcherId/heartbeats` | Get heartbeat history. | | `POST` | `/watchers/:watcherId/events` | Submit watcher events. | | `GET` | `/watchers/:watcherId/config` | Get the synced config YAML. | | `PUT` | `/watchers/:watcherId/config` | Push config YAML and checksum. | | `GET` | `/watchers/:watcherId/config-checksum` | Get the config checksum. | | `GET` | `/watchers/:watcherId/upload-queue` | Get the pending upload queue. | | `GET` | `/watchers/:watcherId/update-check` | Server-advertised release info (latest version, channel, mandatory flag). | ### Archive jobs [#archive-jobs] | Method | Path | Description | | ------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET` | `/instruments/:instrumentId/runs/:runId/download-archive` | Download the run archive. Cache hit → `302` to a presigned S3 URL (or `200 {status:"ready", download_url, size_bytes}` with `Accept: application/json`). Cache miss → `202 {status:"building", job_id}` and an async build; re-issue the same URL to poll. `?file_ids=1,2,3` narrows to a subset. | | `PATCH` | `/archive-jobs/:id` | Lambda callback marking a build `ready` or `failed`. | ### Tokens [#tokens] | Method | Path | Description | | -------- | ------------- | ----------------------------------------------------------------------------------------------- | | `GET` | `/tokens` | List personal access tokens (includes each token's `scopes`). | | `POST` | `/tokens` | Create a token. Admin-only, session-only; requires a non-empty `scopes` array; `*` is rejected. | | `DELETE` | `/tokens/:id` | Revoke a token. Admin-only; admins can revoke any user's token. | ### Users [#users] | Method | Path | Description | | ------- | ---------------- | ------------------------------------------------------------------------------------------ | | `GET` | `/users` | List workspace users with their admin flag. Admin-only, session-only. | | `PATCH` | `/users/:userId` | Toggle a user's `is_admin` flag. Admin-only, session-only; admins can't demote themselves. | ## MCP (Model Context Protocol) [#mcp-model-context-protocol] | Method | Path | Description | | ------------- | ------ | ------------------------------------------------ | | `GET`, `POST` | `/mcp` | MCP server endpoint (Streamable HTTP transport). | The MCP server exposes Data Hub to AI clients (Claude Desktop, Cursor, and any compliant client) using Bearer tokens only; session cookies aren't supported. Tools enforce the same scopes as their REST counterparts. Key tools include `list_instruments`, `search_runs`, `get_run`, `list_run_files`, `get_run_archive`, `get_file_download_url`, `reprocess_file`, `claim_run`/`unclaim_run`, `get_system_status`, and `list_watchers`. There are also resources (`datahub://instruments`, filter options) and prompts (`daily_summary`, `troubleshoot_instrument`, `compare_runs`). For configuration and the full tool/resource/prompt catalog, see the [MCP server reference](/docs/mcp-server), and [Managing tokens → With an MCP client](/docs/managing-tokens#with-an-mcp-client) for client setup. ## Error responses [#error-responses] Errors follow a consistent shape: ```json { "code": "NOT_FOUND", "message": "Instrument not found.", "details": null } ``` Scope failures use `code: "FORBIDDEN"` with a message naming the missing scope. Common codes: `VALIDATION_ERROR` (400), `UNAUTHORIZED` (401), `FORBIDDEN` (403), `NOT_FOUND` (404), `CONFLICT` (409). # Architecture (/docs/architecture) Engineers and operators Data Hub has four components that coordinate through S3, a REST API, and a shared Python library. This page is the conceptual map; the deployment mechanics live in the [first-time deployment guide](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/first-time-deployment.md) in the data-hub developer docs. ## System overview [#system-overview] ## Components [#components] | Component | Package | Description | | -------------- | ------------------ | -------------------------------------------------------------------------------------------------- | | Web app | `data-hub-web` | Next.js web application, REST API, and MCP server. Deployed on Vercel. | | Lambda | `data-hub-lambda` | AWS Lambda function triggered by S3 uploads. Runs instrument-specific processing. | | Watcher | `data-hub-watcher` | CLI agent on lab instrument PCs. Detects new files, uploads them to S3, reports status to the API. | | Shared library | `data-hub-shared` | Shared Python library: S3 utilities, instrument enums, test infrastructure. | ## Data flow [#data-flow] ### Automatic upload (auto mode) [#automatic-upload-auto-mode] 1. A lab instrument writes output files to a watched directory. 2. The **watcher** detects new stable files via filesystem events. 3. It groups files into runs and reports each run to the **API**. 4. It uploads raw files to **S3** at the key `{instrument_id}/{run_id}/{filename}`. 5. The S3 upload triggers the **Lambda**. 6. Lambda downloads the file and dispatches to the instrument's processor for preprocessing. 7. Lambda creates/updates the run and files via the **API**, which sends a **Slack** notification once per new run. 8. Users view the run in the **web dashboard**. ### Manual upload (manual mode) [#manual-upload-manual-mode] Steps 1–3 are the same, but the watcher doesn't upload immediately: 4. The server adds files to an upload queue. 5. On each heartbeat tick the watcher polls the queue and uploads requested files. 6. Steps 5–8 from auto mode follow. ## Key design decisions [#key-design-decisions] * **S3 is the integration boundary.** The watcher and Lambda never talk directly; S3 is the durable hand-off: the watcher writes, Lambda reads. * **API-driven coordination.** The watcher registers, syncs its YAML config, and heartbeats, which is what powers dashboard health and the upload queue. * **Pre-signed URLs from the API.** The web app mints pre-signed S3 upload/download URLs so bytes move directly to/from S3 without routing through the API. On Vercel the app assumes an AWS Identity and Access Management (IAM) role via OpenID Connect (OIDC), with no long-lived AWS keys. * **Lambda-built run archives.** “Download all” delegates to the Lambda, which zips files into a separate archives bucket via S3 multipart upload; the web app then 302s the browser to a short-lived presigned URL. Bytes never traverse Vercel. * **Shared library for contracts.** Instrument IDs, S3 utilities, and environment config live in `data-hub-shared` so Lambda and the watcher stay consistent. * **MCP for AI access.** The web app serves an MCP endpoint at `/api/v1/mcp` exposing read tools, resources, and prompts to AI clients via a personal access token. See the [API reference](/docs/api-reference#mcp-model-context-protocol). ## Branch and environment strategy [#branch-and-environment-strategy] | Branch | Purpose | | ------------ | ----------------------------------------------- | | `staging` | Pre-production. Pull requests merge here first. | | `production` | Live. Changes are promoted from `staging`. | Feature branches target `staging` via pull requests (PRs). Continuous integration (CI) runs on every PR and on merges to both branches; merges to `staging` and `production` deploy each component to its environment automatically. Staging and production are independent deployments with separate databases. # Watcher CLI (/docs/cli-reference) Lab operators The complete `data-hub-watcher` command surface. For task-oriented walkthroughs see [Installing a watcher](/docs/installing-a-watcher), [Run as a Windows service](/docs/windows-service), and [Upgrading the watcher](/docs/upgrading-the-watcher). ## Global flags [#global-flags] These apply to every command: | Flag | Description | | --------------- | -------------------------------------------------------------------------------- | | `--config PATH` | Override the config file path. Equivalent to the `DATA_HUB_CONFIG_PATH` env var. | | `--verbose` | Enable debug-level logging on stderr. | | `--version` | Print the installed watcher version and exit. | | `--help` | Show help for the command or subcommand. | ## `init` [#init] Interactive setup wizard. Prompts for environment, API key, instrument, watch directory, file patterns, run detection, stability period, and upload mode; then registers the watcher and writes `~/.data-hub/config.yaml` and `~/.data-hub/.env.`. | Flag | Description | | ------------ | ------------------------------------------------------------------------------------ | | `--show-key` | Echo the API key as you type/paste it (useful when a terminal mangles hidden input). | The key is validated (must start with `dhub_`) and normalized to strip zero-width and non-breaking-space characters that Windows clipboards inject. See [Installing a watcher → Setup](/docs/installing-a-watcher#setup). ## `watch` [#watch] Start the file-monitoring loop: detect stable files, group them into runs, report runs, upload (auto mode) or poll the queue (manual mode), and send heartbeats every 60 seconds. Press `Ctrl+C` to stop. | Flag | Description | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `--dry-run` | Validate config, check API reachability and instrument status, and preview which files would be picked up, without starting the monitor. | ## `upload` [#upload] One-shot upload outside the watch loop. ```sh data-hub-watcher upload --file /path/to/file.csv --run-id RUN001 # upload a specific file data-hub-watcher upload # process the server-side queue (manual mode) data-hub-watcher upload --dry-run # preview without uploading ``` | Flag | Description | | ------------- | ------------------------------ | | `--file PATH` | The file to upload. | | `--run-id ID` | The run to attach the file to. | | `--dry-run` | Preview without uploading. | ## `config` [#config] Manage the YAML config. Changes that affect the server are synced automatically. | Subcommand | Description | | ---------------------------- | ------------------------------------------------------------------- | | `config show` | Pretty-print the current config. | | `config validate` | Validate the config file offline. | | `config set-environment ENV` | Switch the active environment (`staging`, `production`, `preview`). | | `config edit` | Re-prompt each field with current values as defaults. | | `config open` | Open the config in your editor, then re-validate. | | `config path` | Print the resolved config file path. | `config set-environment` (and `config edit` when the environment changes) accept: | Flag | Description | | -------------------- | --------------------------------------------------------------------------------------- | | `--api-base-url URL` | Required when switching to `preview`. | | `--api-key KEY` | Otherwise read from the env file or prompted. | | `--show-key` | Echo the key as entered. | | `--no-register` | Fail instead of registering a new watcher if none is stored for the target environment. | See the [Configuration reference → Switching environments](/docs/configuration-reference#switching-environments). ## `service` (Windows only) [#service-windows-only] Requires the `windows-service` extra. Manage the watcher as a Windows service. | Subcommand | Description | | ------------------- | ---------------------------------------------- | | `service install` | Register the service. | | `service uninstall` | Remove the service. | | `service start` | Start the service. | | `service stop` | Stop the service. | | `service status` | Show service status. | | `service reinstall` | Stop, uninstall, install, and start in one go. | `service install` and `service reinstall` accept `--env-path PATH` to override which `.env` file the service loads. Full details in [Run as a Windows service](/docs/windows-service). ## `self-update` [#self-update] Check the API for a newer published version and upgrade in place. ```sh data-hub-watcher self-update # check + upgrade if needed data-hub-watcher self-update --check # report status only, no upgrade data-hub-watcher self-update --force # re-run the upgrade even if versions match ``` | Flag | Description | | --------- | ---------------------------------------------------------------------------- | | `--check` | Report status only; don't upgrade. | | `--force` | Re-run the upgrade subprocess even if the installed version already matches. | The upgrade flow depends on your install method; see [Upgrading the watcher](/docs/upgrading-the-watcher#manual-upgrade-self-update). # Concepts (/docs/concepts) A glossary of the nouns the rest of this documentation leans on. If you've landed mid-doc and a term is unfamiliar, it's probably defined here. ## Instrument [#instrument] A piece of lab equipment registered in Data Hub, identified by a kebab-case **instrument ID** (e.g. `akta-fplc`, `bio-rad-cfx96`). The ID is permanent (it becomes the storage key prefix and the identifier used across the system) and has a human-readable display name alongside it. An instrument has three states: | Status | Meaning | | ---------- | ---------------------------------------------------------------------------- | | `pending` | Registered but not yet confirmed by an admin. A watcher can't upload for it. | | `active` | Confirmed and accepting uploads. | | `inactive` | Deactivated; no longer accepting uploads. | See [Adding an instrument](/docs/adding-an-instrument) and [Managing instruments](/docs/managing-instruments). ## Watcher [#watcher] The `data-hub-watcher` CLI agent that runs on an instrument PC. It monitors a directory, groups new files into runs, uploads them to cloud storage, and reports status to the API. Each instrument can have at most one active watcher at a time. A watcher is registered with the API and carries one registration ID per environment. Deregistering a watcher is a soft-delete: its history stays visible for auditing. See [Managing watchers](/docs/managing-watchers). ## Run [#run] A logical grouping of files produced by one acquisition. The watcher assigns each file a **run ID** using its configured [run detection](#run-detection) method, then reports the run to the API. The first file for a run creates it; later files are added incrementally to the same run's manifest. ## Run detection [#run-detection] The rule that maps a file to a run ID. Two broad strategies: * **Prefix**: a regex extracts the run ID from the filename. The default `^([^_]+)` takes everything before the first underscore, so `RUN001_data.csv` → `RUN001`. * **Directory**: each subdirectory under the watch directory is its own run, so `RUN001/data.csv` → `RUN001`. The full set of presets (and the custom-regex option) is in the [Configuration reference](/docs/configuration-reference#run-detection). ## Stability period [#stability-period] How many seconds a file must stay unchanged (same size and modification time) before the watcher treats it as fully written and eligible for upload. The default is 5 seconds; raise it for instruments that write large files slowly. A file that keeps changing for more than 5 minutes is abandoned with a `stability_timeout` error event. ## Upload mode [#upload-mode] How files get from the instrument to cloud storage: * **auto**: files upload immediately after they're detected and stabilized. * **manual**: runs are reported to the server, but files aren't uploaded until approved through the server-side upload queue (polled on each heartbeat). Use this when uploads need human approval. ## Heartbeat [#heartbeat] A status ping the watcher sends to the API every 60 seconds. It carries the watcher version, instrument ID, watch directory, upload mode, activity counters, and uptime, and is what drives the **Last Heartbeat** and online/offline health shown on the dashboard. The same tick also runs the auto-updater and (in manual mode) polls the upload queue. ## Events [#events] A per-watcher log of lifecycle moments and errors: `watcher_started`, `run_reported`, `file_uploaded`, `update_succeeded`, and `error` (with a `details.kind` discriminator), among others. Events are batched and flushed on each heartbeat and are visible on the dashboard, making them the primary tool for diagnosing a watcher remotely. ## Environment [#environment] Which Data Hub deployment a watcher talks to: `staging`, `production`, or `preview` (a Vercel preview deployment, which needs an explicit API base URL). Staging and production are separate deployments with separate databases, so a single PC keeps an independent registration and local state per environment and can switch between them. ## Initial scan [#initial-scan] What the watcher does with files already sitting in the watch directory the first time an environment is entered. `production` uploads the existing backlog (`full`); `staging` and `preview` record it as a baseline and skip it (`new-only`) so test environments aren't flooded with history. See the [Configuration reference](/docs/configuration-reference#initial-scan). ## Personal access token [#personal-access-token] A `dhub_`-prefixed bearer token that authenticates the watcher (and other API clients) with the Data Hub API. Tokens carry permission **scopes** and are created and revoked by admins. See [Managing tokens](/docs/managing-tokens) and [Security & permissions](/docs/security). # Watcher configuration (/docs/configuration-reference) Lab operators The complete reference for the watcher's configuration: the `config.yaml` schema, where files live, the environment variables that influence it, and the behaviors (run detection, upload modes, initial scan) those fields drive. ## File locations [#file-locations] Everything the watcher persists lives under `~/.data-hub/`: | Path | Contents | | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `~/.data-hub/config.yaml` | The main config (below). Override with `--config` or `DATA_HUB_CONFIG_PATH`. | | `~/.data-hub/.env.` | The API key for an environment (e.g. `.env.staging`, `.env.production`, `.env.preview`). The legacy `~/.data-hub/.env` is still loaded, with the per-environment file taking precedence. | | `~/.data-hub/watcher-.db` | Per-environment SQLite state (uploaded files, runs, detected files, baseline). | | `~/.data-hub/watcher.log` | Rotating log (10 MB × 5 backups). On Windows: `C:\ProgramData\DataHubWatcher\watcher.log`. | | `~/.data-hub/upgrade-worker.log` | Windows uv-tool upgrade transcript. | ## Environment variables [#environment-variables] | Variable | Purpose | | ---------------------------- | --------------------------------------------------------------------------------------------- | | `DATA_HUB_API_KEY` | Supplies the API key, skipping the `init` prompt. Saved into the per-environment `.env` file. | | `DATA_HUB_CONFIG_PATH` | Override the config file path (same as `--config`). | | `DATA_HUB_WATCHER_LOG_LEVEL` | Set to `DEBUG` (in the service's `.env` file) to enable debug logging without a reinstall. | ## `config.yaml` [#configyaml] ```yaml version: 1 environment: production # "staging", "production", or "preview" api_base_url: null # required when environment is "preview" watcher_ids: # one registration id per environment production: initial_scan: null # null (default), "full", or "new-only" instrument: id: akta-fplc # kebab-case instrument ID watch_directory: /path/to/data file_patterns: - "*.csv" - "*.xlsx" enabled: true upload_mode: auto # "auto" or "manual" stability_period_seconds: 5 # 1–300 run_detection: pattern: '^([^/]+)/' # regex with one capture group (run ID) recursive: true ``` ### Field reference [#field-reference] | Field | Type | Description | | ------------------------------------- | -------------- | ---------------------------------------------------------------------------------- | | `version` | int | Config schema version. Currently `1`. | | `environment` | enum | `staging`, `production`, or `preview`. | | `api_base_url` | string \| null | Required only for `preview` (the deployment's `/api/v1` URL). | | `watcher_ids` | map | One watcher registration id per environment, assigned by the API. | | `initial_scan` | enum \| null | `null` (env default), `full`, or `new-only`. See [Initial scan](#initial-scan). | | `instrument.id` | string | Kebab-case instrument ID. Permanent; the storage key prefix. | | `instrument.watch_directory` | string | Absolute path the instrument writes to. | | `instrument.file_patterns` | string\[] | Glob patterns; only matching files are uploaded. | | `instrument.enabled` | bool | Whether the watcher processes this instrument. | | `instrument.upload_mode` | enum | `auto` or `manual`. See [Upload modes](#upload-modes). | | `instrument.stability_period_seconds` | int (1–300) | Seconds a file must stay unchanged before it's considered written. Default `5`. | | `instrument.run_detection.pattern` | regex | One-capture-group regex applied to each file's path relative to `watch_directory`. | | `instrument.run_detection.recursive` | bool | `true` watches subdirectories; `false` only the top level. | A config written by an older watcher used a single top-level `watcher_id`. It's migrated transparently on load: lifted into `watcher_ids` under the active environment and dropped on the next save. ## Run detection [#run-detection] The `pattern` regex is applied with `re.search` to each file's path relative to `watch_directory`, with backslashes normalized to `/`. Capture group 1 is the run ID, and the pattern must have exactly one capture group. In YAML, single-quote patterns so backslash sequences like `\d` don't need escaping. The `init` wizard offers these presets (or supply a custom regex): | Preset | Pattern | Recursive | Run ID is… | | ---------------------- | ------------------------------ | --------- | ----------------------------------------------------------------- | | Filename prefix | `^([^_]+)` | no | everything before the first `_`: `RUN001_data.csv` → `RUN001` | | Top subdirectory | `^([^/]+)/` | yes | the top-level folder: `RUN001/data.csv` → `RUN001` | | Deepest subdirectory | `([^/]+)/[^/]+$` | yes | the immediate parent folder: `plate-a/well-b/data.csv` → `well-b` | | Timestamp subdirectory | `(?:^\|/)(\d{8}_\d{6}_\d{3})/` | yes | a `YYYYMMDD_HHMMSS_fff` folder anywhere in the path | | Filename stem | `^(?:.+/)?([^/]+?)\.[^/.]+$` | no | the filename without its extension (each file is its own run) | ## Upload modes [#upload-modes] * **`auto`**: files upload to cloud storage immediately after run detection. * **`manual`**: runs are reported without uploading; the server decides which files to upload via a queue polled on each heartbeat. Useful when uploads need human approval. See [Managing watchers → Approving manual uploads](/docs/managing-watchers#approving-manual-uploads). ## Initial scan [#initial-scan] The first time an environment is entered (via `init` or `config set-environment`), `initial_scan` decides what happens to files already in the watch directory. It defaults by environment: * `production` → `full`: the existing backlog is uploaded (production is the source of truth). * `staging` / `preview` → `new-only`: the on-disk files are recorded as a baseline and skipped; only files created afterwards are uploaded. This is why a fresh `init` on `staging`/`preview` doesn't upload a PC's history. Set `initial_scan: full` to opt back in (e.g. to deliberately populate staging), or `initial_scan: new-only` on production to suppress its backlog. Upgrading an existing watcher is unaffected: the environment's local DB already carries history, so baseline seeding is skipped. ## Switching environments [#switching-environments] A single PC can move between `staging`, `production`, and `preview`. Because these are separate deployments with separate databases, each keeps its own registration in `watcher_ids` and its own credentials in `~/.data-hub/.env.`. ```sh # Switch to staging (reuses a stored registration, or registers one). data-hub-watcher config set-environment staging # Point at a preview deployment (the base URL is required). data-hub-watcher config set-environment preview \ --api-base-url https://data-hub-git-my-branch.vercel.app/api/v1 ``` A switch validates the target API, reuses (or registers) the watcher id for that environment, pushes the config, and leaves a breadcrumb event in the old environment. A running `watch` process or service keeps using the old environment until restarted: on Windows, `service stop && service start`; elsewhere, restart `watch`. # FAQ (/docs/faq) The short answers. Each links to the page with the full story. ## Can two watchers run on one instrument? [#can-two-watchers-run-on-one-instrument] No. Each instrument can have at most one active watcher at a time. A second registration fails with “instrument already has an active watcher” and prints the existing watcher's id. Deregister the old one first; see [Managing watchers → Deregistering a watcher](/docs/managing-watchers#deregistering-a-watcher). ## What happens if I revoke a token a watcher is using? [#what-happens-if-i-revoke-a-token-a-watcher-is-using] The watcher starts getting `401 Unauthorized` on its next heartbeat or API call. Create a new token, re-run `data-hub-watcher init` on the PC to enter it, and restart the watcher. See [Managing tokens → After revoking a token](/docs/managing-tokens#after-revoking-a-token). ## I'm not an admin, how do I get a token? [#im-not-an-admin-how-do-i-get-a-token] You can't mint tokens yourself; only admins can. Ask an admin to issue you one (it'll start with `dhub_`). Members can view the token audit list but not create or delete tokens. See [Security & permissions](/docs/security#roles). ## Why aren't my existing files uploading on staging? [#why-arent-my-existing-files-uploading-on-staging] By design. On `staging` and `preview`, the watcher records files already on disk as a baseline and uploads only files created after setup, so test environments aren't flooded with history. `production` uploads the backlog. Override with `initial_scan: full`; see the [Configuration reference](/docs/configuration-reference#initial-scan). ## Can one PC talk to both staging and production? [#can-one-pc-talk-to-both-staging-and-production] Yes. A PC keeps a separate registration, credentials, and local state per environment, and you switch with `data-hub-watcher config set-environment`. A running watcher keeps using the old environment until restarted. See [Switching environments](/docs/configuration-reference#switching-environments). ## How do I control which files get picked up? [#how-do-i-control-which-files-get-picked-up] Two settings: **file patterns** (globs like `*.csv` that select files) and **run detection** (how matched files are grouped into runs). Note glob matching is case-sensitive on some systems (`*.csv` ≠ `data.CSV`). See [Run detection](/docs/configuration-reference#run-detection) and [Troubleshooting → Files aren't being detected](/docs/troubleshooting#files-arent-being-detected). ## What's the difference between auto and manual upload mode? [#whats-the-difference-between-auto-and-manual-upload-mode] In **auto** mode files upload immediately after detection. In **manual** mode runs are reported but files wait for server approval through the upload queue (polled on each heartbeat). See [Upload modes](/docs/configuration-reference#upload-modes). ## How do I upload a file by hand? [#how-do-i-upload-a-file-by-hand] `data-hub-watcher upload --file /path/to/file.csv --run-id RUN001`. Add `--dry-run` to preview. See the [CLI reference](/docs/cli-reference#upload). ## Will an upgrade interrupt a run in progress? [#will-an-upgrade-interrupt-a-run-in-progress] Normally no: the auto-updater only upgrades during a quiet window when no files have uploaded and no run has been reported recently. The exception is a release an admin flags mandatory, which skips that guard. See [Upgrading the watcher](/docs/upgrading-the-watcher#background-auto-update). ## Does the watcher need AWS credentials? [#does-the-watcher-need-aws-credentials] No. The web app issues short-lived pre-signed S3 URLs, and the watcher uploads directly to those; it never holds AWS credentials. See [Architecture](/docs/architecture). ## Where are the logs? [#where-are-the-logs] `~/.data-hub/watcher.log` on macOS/Linux, `C:\ProgramData\DataHubWatcher\watcher.log` on Windows (the CLI and the service share one file). See [Troubleshooting → Where are the logs?](/docs/troubleshooting#where-are-the-logs). # Installing a watcher (/docs/installing-a-watcher) Lab operators This guide covers the core install-to-running path for the watcher on an instrument PC: install, configure, start, and stop. For everything around it, see the related pages: * [Run as a Windows service](/docs/windows-service) to keep it running across reboots. * [Upgrading the watcher](/docs/upgrading-the-watcher) to stay current. * [Troubleshooting](/docs/troubleshooting) when something doesn't work. * [CLI reference](/docs/cli-reference) and [Configuration reference](/docs/configuration-reference) for every command and config field. ## Prerequisites [#prerequisites] * **[uv](https://docs.astral.sh/uv/getting-started/installation/)**: the recommended Python package manager. Install it if you don't have it yet; it handles Python for you, so you don't need a separate Python install. * **A personal access token**: it starts with `dhub_`. If you're a workspace admin you can create one at **Settings > Access Tokens**; if not, ask an admin to issue you one (you can't mint tokens yourself). See [Managing tokens](/docs/managing-tokens). * **The watch directory**: the folder where the instrument writes its output files. * **File patterns**: the file extensions you want to upload (e.g., `*.csv`, `*.xlsx`, `*.tiff`). ## Installation [#installation] The watcher is published as a versioned package on PyPI. For use on lab PCs, install it directly from PyPI: ```sh uv tool install data-hub-watcher ``` This installs the `data-hub-watcher` CLI into an isolated venv managed by `uv`, on PATH for any shell. Every example below that says `data-hub-watcher …` runs the installed CLI directly. To run the watcher as a Windows service, install with the `windows-service` extra instead so the `pywin32` dependency is included: `uv tool install "data-hub-watcher[windows-service]"`. See [Run as a Windows service](/docs/windows-service). ## Setup [#setup] Run the interactive setup wizard: ```sh data-hub-watcher init ``` The wizard walks you through: 1. **Environment**: `staging` (testing), `production`, or `preview`. Choosing `preview` also prompts for the deployment's API base URL. 2. **API key**: paste your personal access token. It's saved to `~/.data-hub/.env.`, so each environment keeps its own key. (You can also set `DATA_HUB_API_KEY` before running `init` to skip this prompt.) 3. **Instrument**: select an existing instrument, or register a new one. New instruments start as `pending` and must be activated by an admin before the watcher can start; see [Adding an instrument](/docs/adding-an-instrument). 4. **Watch directory**: the absolute path to the folder the instrument writes to. 5. **File patterns**: comma-separated globs (e.g., `*.csv,*.xlsx`). Only matching files are uploaded. 6. **Run detection**: how files are grouped into runs (prefix or directory). See [Run detection](/docs/configuration-reference#run-detection). 7. **Stability period**: seconds a file must stay unchanged before it's considered fully written. Default 5. 8. **Upload mode**: `auto` (upload immediately) or `manual` (wait for server approval). See [Upload modes](/docs/configuration-reference#upload-modes). The wizard saves your config to `~/.data-hub/config.yaml`, the API key to `~/.data-hub/.env.`, and syncs the config to the server. **The existing backlog.** On `staging` and `preview`, the watcher does not upload files already in the watch directory when you first run `init`; it records them as a baseline and uploads only files created afterwards, so test environments aren't flooded with history. `production` uploads the backlog. To override, set `initial_scan`; see the [Configuration reference](/docs/configuration-reference#initial-scan). ## Starting the watcher [#starting-the-watcher] First, verify your setup with a dry run: ```sh data-hub-watcher watch --dry-run ``` This validates the config, checks that the API is reachable and the instrument is active, and previews what files would be uploaded, without starting the monitor. When you're ready: ```sh data-hub-watcher watch ``` The watcher will now: * Monitor the watch directory for new and modified files. * Wait for files to stabilize before processing them. * Group files into runs and report them to the API. * Upload files (in auto mode) or wait for server approval (in manual mode). * Send heartbeats every 60 seconds so the web dashboard shows watcher health. ## Stopping the watcher [#stopping-the-watcher] Press `Ctrl+C` in the terminal running `watch`. The watcher sends a final `stopped` heartbeat and exits cleanly. If you're running it as a Windows service, use `data-hub-watcher service stop` instead; see [Run as a Windows service](/docs/windows-service). ## Changing configuration later [#changing-configuration-later] Re-run the wizard with current values as defaults, open the YAML directly, or print the current config: ```sh data-hub-watcher config edit # re-prompt each field data-hub-watcher config open # open the YAML in your editor data-hub-watcher config show # print the current config ``` Changes are synced to the server automatically. The full field-by-field breakdown is in the [Configuration reference](/docs/configuration-reference). # Managing instruments (/docs/managing-instruments) Administrators When a lab operator registers a new instrument from the watcher, it appears in your workspace as `pending` and can't upload until you confirm it. This page covers the admin side; for the operator's end-to-end flow see [Adding an instrument](/docs/adding-an-instrument). ## Instrument states [#instrument-states] | Status | Meaning | | ---------- | ------------------------------------------------------------------------ | | `pending` | Registered but not yet confirmed. The watcher is blocked from uploading. | | `active` | Confirmed and accepting uploads. | | `inactive` | Deactivated; no longer accepting uploads. | ## Confirming a pending instrument [#confirming-a-pending-instrument] Open the Data Hub web app and go to the **Instruments** page. Find the instrument with the yellow `pending` badge. Click **Confirm**. The status changes to `active` and the operator's watcher can start uploading. Confirming is admin-only. Under the hood it's a `PATCH /api/v1/instruments/:id` that requires the admin role for session callers (watcher and Lambda automation authenticate with the `instruments:write` scope instead, so they're unaffected). See the [API reference](/docs/api-reference#instruments). ## Viewing an instrument [#viewing-an-instrument] The instrument detail view shows its run count and watcher counts (online / offline), so you can see at a glance whether an instrument is healthy and actively reporting. Watcher health comes from heartbeats; see [Managing watchers](/docs/managing-watchers). ## Deactivating an instrument [#deactivating-an-instrument] Set an instrument to `inactive` when it's decommissioned or temporarily out of service. Its historical runs and files remain accessible; new uploads are refused until it's reactivated. Instrument IDs are permanent: they're the storage key prefix and the system-wide identifier. Deactivate rather than trying to rename or delete an instrument that's been in use. ## Related pages [#related-pages] * [Managing watchers](/docs/managing-watchers): the watcher fleet, deregistration, and the upload queue. * [Managing tokens](/docs/managing-tokens): issue tokens to lab operators. * [Security & permissions](/docs/security): what admins vs. members can do. # Managing tokens (/docs/managing-tokens) Administrators and members Personal access tokens authenticate the watcher and other API clients with the Data Hub API. This guide covers creating, using, and revoking tokens. For the bigger picture on roles and scopes, see [Security & permissions](/docs/security). Creating and revoking tokens requires the workspace admin role. Regular members can view the workspace token audit list at **Settings > Access Tokens** but cannot mint or delete tokens. Ask an existing admin if you need a token issued. ## Creating a token [#creating-a-token] ### In the web app [#in-the-web-app] 1. Sign in to the Data Hub web app as an admin. 2. Go to **Settings > Access Tokens**. 3. Click **Create Token**. 4. Enter a descriptive name (e.g., “FPLC watcher – Lab 201”). 5. Optionally set an expiration date. 6. Click **Create**. The plaintext token is displayed once: copy it immediately. It cannot be retrieved again. The token starts with `dhub_` followed by a 64-character hex string. ### Via the API [#via-the-api] ```sh curl -X POST https://datahub.example.com/api/v1/tokens \ -H "Cookie: " \ -H "Content-Type: application/json" \ -d '{"name": "FPLC watcher", "expires_at": "2027-01-01T00:00:00Z"}' ``` The `expires_at` field is optional. If omitted, the token never expires. The response includes the plaintext token in the `token` field; this is the only time it's returned. ## Using a token [#using-a-token] ### With the watcher [#with-the-watcher] During `data-hub-watcher init`, paste the token when prompted for the API key. Alternatively, set it as an environment variable: ```sh export DATA_HUB_API_KEY=dhub_your_token_here data-hub-watcher init ``` The watcher stores the API key in its environment configuration and uses it for all subsequent API calls. ### With an MCP client [#with-an-mcp-client] Data Hub exposes a Model Context Protocol (MCP) server so AI assistants can query your data. Add it to your MCP client configuration, for example in Claude Desktop (`claude_desktop_config.json`) or Cursor (`.cursor/mcp.json`): ```json { "mcpServers": { "data-hub": { "url": "https://datahub.example.com/api/v1/mcp", "headers": { "Authorization": "Bearer dhub_your_token_here" } } } } ``` See the [MCP server reference](/docs/mcp-server) for the full tool, resource, and prompt catalog, plus client-specific installation steps. ### With the API directly [#with-the-api-directly] Pass the token in the `Authorization` header: ```sh curl https://datahub.example.com/api/v1/instruments \ -H "Authorization: Bearer dhub_your_token_here" ``` ## Viewing tokens [#viewing-tokens] Go to **Settings > Access Tokens** in the web app. The table shows: | Column | Description | | ------------- | -------------------------------------------------------------------- | | **Name** | The label you gave the token | | **Token** | The prefix only (e.g., `dhub_a1b2…`): the full token is never stored | | **Last used** | When the token was last used to authenticate an API request | | **Expires** | Expiration date, or "Never" | | **Created** | When the token was created | ## Revoking a token [#revoking-a-token] ### In the web app [#in-the-web-app-1] 1. Go to **Settings > Access Tokens**. 2. Click the delete button next to the token you want to revoke. 3. Confirm the deletion. The token is immediately invalidated. Any watcher or client using it will start receiving `401 Unauthorized` errors. ### Via the API [#via-the-api-1] ```sh curl -X DELETE https://datahub.example.com/api/v1/tokens/ \ -H "Cookie: " ``` ## Security notes [#security-notes] * Tokens are hashed with SHA-256 before storage. The plaintext is never persisted. * Token create and delete are admin-only operations. Admins can revoke any user's token to support off-boarding and incident response. * Use descriptive names so you can identify which watcher or client each token belongs to. * Set expiration dates for tokens used in temporary setups. * Revoke tokens immediately when a watcher is decommissioned or a token may have been exposed. ## After revoking a token [#after-revoking-a-token] If you revoke a token that a running watcher is using, the watcher will start failing on its next heartbeat or API call. To fix it: 1. Create a new token. 2. On the instrument PC, re-run the setup wizard: ```sh data-hub-watcher init ``` 3. Enter the new token when prompted. 4. Restart the watcher: ```sh data-hub-watcher watch ``` # Managing watchers (/docs/managing-watchers) Administrators The **Watchers** area of the web app is the fleet view: every registered watcher, its health, and the controls for deregistering watchers and rolling out releases. ## Monitoring watcher health [#monitoring-watcher-health] Each watcher reports a [heartbeat](/docs/concepts#heartbeat) every 60 seconds carrying its version, instrument, watch directory, upload mode, activity counters, and uptime. The fleet list surfaces: * **Effective status**: online vs. offline, derived from the last heartbeat. * **Last heartbeat**: a stale value is the first sign of a watcher that's stopped, lost connectivity, or crashed. * **Hostname and instrument assignment**: which PC is watching which instrument. * **Events**: the per-watcher log (`watcher_started`, `run_reported`, `file_uploaded`, `update_*`, and `error` with a `details.kind` discriminator). Events are your primary remote-diagnosis tool. For a guided diagnosis, the `troubleshoot_instrument` MCP prompt inspects an instrument's status, watcher heartbeats, and recent runs; see the [MCP server reference](/docs/mcp-server#prompts). ## Deregistering a watcher [#deregistering-a-watcher] Each instrument can have at most one active watcher. Deregister the old one when a PC is reimaged or replaced, or to clear the [“instrument already has an active watcher”](/docs/troubleshooting#instrument-already-has-an-active-watcher) error a new install runs into. * **Web app**: open the watcher under **Watchers** and click **Deregister**. * **API**: `DELETE /api/v1/watchers/:watcherId` (requires `watchers:write`). Deregistration is a soft-delete: the watcher's heartbeats, events, and reported runs stay visible under **Watchers > Deregistered** for auditing. ## Approving manual uploads [#approving-manual-uploads] For instruments configured in [manual upload mode](/docs/concepts#upload-mode), the watcher reports runs but waits for server approval before uploading file bytes. The watcher polls the upload queue on each heartbeat tick and uploads the files the server has marked for upload. A couple of safeguards keep the queue healthy: * Changing an instrument's `watch_directory` reverts its pending upload requests to `detected` so the queue drains immediately rather than erroring on stale paths. * A queued file that keeps failing (missing on disk or failing to upload) is retried on at most three heartbeat polls, then cancelled server-side so it leaves the queue instead of erroring every tick. ## Releases and fleet updates [#releases-and-fleet-updates] Lab PCs upgrade themselves toward the version you advertise per environment. You control that from **Settings → Watchers**. For how upgrades land on an individual PC, see [Upgrading the watcher](/docs/upgrading-the-watcher). ### Cutting a new release [#cutting-a-new-release] Releases are tag-driven and publish from a `production` commit. **Bump the version** in `watcher/pyproject.toml` and merge through `staging` to `production`. **Tag and push** from `production`: ```sh git checkout production && git pull git tag watcher-v0.3.0 && git push origin watcher-v0.3.0 ``` **Approve** the `pypi` deployment under **Actions → Publish watcher** in GitHub. **Advertise the release**: in **Settings → Watchers**, set **Latest version** to the new tag. Each environment has its own database, so bump `staging` and `production` separately to roll the fleet gradually. Always tag → publish → verify → save. Saving a **Latest version** before the wheel is live on PyPI triggers a wave of `update_failed` events across the fleet. ### Release-config fields [#release-config-fields] | Field | Purpose | | ----------------------------- | ------------------------------------------------------- | | **Latest version** | Required. Blank = “no update info available”. | | **Minimum supported version** | Reserved; not enforced server-side yet. | | **Release channel** | Defaults to `stable`. Surfaced in `self-update` output. | | **Mandatory update** | Skips the activity-window guard; see below. | ### Mandatory updates [#mandatory-updates] Toggling **Mandatory update** skips the [activity-window guard](/docs/upgrading-the-watcher#background-auto-update) and fires the upgrade on the next hourly check. The server still compares the running version to **Latest version**, so a correctly-pinned PC isn't forced. Use sparingly: a forced upgrade mid-acquisition loses data. Reserve it for security fixes, wire-protocol breaks, or cases where leaving the bad version running is strictly worse than restarting in flight. ### Rolling back [#rolling-back] Rollback is another release: set **Latest version** back to the older tag for the affected environment(s), toggle **Mandatory update** on if you need it to bypass the activity-window guard, and wait for the next hourly tick. Toggle **Mandatory update** back off once the fleet converges. A rolled-back release is still on PyPI and reinstallable; it's no longer advertised. ## Related pages [#related-pages] * [Upgrading the watcher](/docs/upgrading-the-watcher): the per-PC upgrade mechanics and troubleshooting. * [Managing instruments](/docs/managing-instruments): confirming and deactivating instruments. * [API reference](/docs/api-reference#watchers): the watcher endpoints. # MCP server (/docs/mcp-server) Developers and integrators The Data Hub MCP server exposes instruments, runs, files, and watcher data to AI clients (such as Claude Desktop and Cursor) through the [Model Context Protocol](https://modelcontextprotocol.io/). Clients can list instruments, search runs, fetch experimental results, generate download URLs, and re-trigger Lambda processing, all through a single Bearer-authenticated HTTP endpoint served by the web app. The server lives at `/api/v1/mcp` on the same Next.js deployment that serves the [REST API](/docs/api-reference) and uses the MCP Streamable HTTP transport. ## Authentication [#authentication] The MCP server accepts Bearer tokens only; session cookies are not supported. Create a personal access token in the web app at **Settings > Access Tokens**, then pass it in the `Authorization: Bearer ` header when configuring your client. See [Managing tokens](/docs/managing-tokens) for details on creating, using, and revoking tokens. ## Installation [#installation] ### Claude Desktop [#claude-desktop] Edit `claude_desktop_config.json` (macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`): ```json { "mcpServers": { "data-hub": { "url": "https://datahub.example.com/api/v1/mcp", "headers": { "Authorization": "Bearer dhub_your_token_here" } } } } ``` Restart Claude Desktop. The `data-hub` server should appear in the MCP panel and its tools become available in conversations. ### Cursor [#cursor] Edit `.cursor/mcp.json` in your project or `~/.cursor/mcp.json` globally: ```json { "mcpServers": { "data-hub": { "url": "https://datahub.example.com/api/v1/mcp", "headers": { "Authorization": "Bearer dhub_your_token_here" } } } } ``` Reload Cursor. Tools are invoked via the agent automatically when relevant. ### Other clients [#other-clients] The endpoint follows the MCP Streamable HTTP spec, so any compliant client works. Configure it with: * **URL**: `https://datahub.example.com/api/v1/mcp` * **Transport**: Streamable HTTP (`GET` for the SSE stream, `POST` for client messages) * **Auth header**: `Authorization: Bearer dhub_your_token_here` For local development, point at `http://localhost:3000/api/v1/mcp` instead. ## Tools [#tools] All tools return JSON encoded as a single text content block. Error cases set `isError: true` and return a plain-text message. ### Instruments [#instruments] | Tool | Description | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `list_instruments` | List all registered instruments with run counts, watcher status, and file patterns. Optionally filter by `status` (`pending`, `active`, `inactive`). | | `get_instrument` | Get full detail for an instrument by its kebab-case ID, including watcher online/offline counts. | ### Runs [#runs] | Tool | Description | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `search_runs` | Paginated search across runs with filtering, sorting, and date range. Supports plate-reader metadata filters (`wavelength`, `measurementMode`, `measurementType`) and attribution filtering via `ranBy` (a user id or the literal `"unattributed"`). | | `get_run` | Get a single run by its natural key (`instrumentId` + `runId`). | | `list_run_files` | List all files attached to a run, including processing status and metadata. Use `get_file_download_url` on processed CSV files to access experimental results. | | `get_run_archive` | Get a downloadable ZIP archive of all uploaded files for a run. Returns a 15-minute pre-signed S3 URL on a cache hit (clickable in a browser, no auth required), or a `building` job + `retryAfterSeconds` hint on a miss; call again after the suggested wait to poll. | Both `get_run` and `search_runs` responses embed an `attributions` array on each run, listing the users who have claimed it (user id, display name, initials, avatar URL). No separate read tool is needed to inspect who ran a run. ### Run attribution [#run-attribution] | Tool | Description | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `claim_run` | **Write tool.** Mark a run as performed by the authenticated user. Idempotent. Only self-attribution is supported: the user id comes from the session token, never from an argument, so you cannot claim a run on behalf of another user. | | `unclaim_run` | **Write tool.** Remove the authenticated user's attribution from a run. Idempotent. Annotated `destructiveHint: true` because removing attribution is user-visible across the dashboard and runs tables. | | `list_run_attributors` | List distinct users who have claimed at least one run on a given instrument. Use the returned `userId` values to construct a `search_runs` call with `ranBy=`. | ### Files [#files] | Tool | Description | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `get_file` | Get detailed metadata for a single file by numeric ID. | | `get_file_download_url` | Get a pre-signed S3 URL for downloading a file's raw contents. URLs expire after 15 minutes and can be fetched without additional authentication. | | `reprocess_file` | **Write tool.** Re-run the Lambda processing workflow for a `failed` or `completed` file. Transitions the file back to `processing`. Annotated `destructiveHint: true` so clients can warn before invoking. | ### Watchers and system status [#watchers-and-system-status] | Tool | Description | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `get_system_status` | Dashboard-level overview of all instruments, watcher health, and pending upload counts. | | `list_watchers` | List watcher agents with effective status, hostname, instrument assignment, and last heartbeat. Optionally filter by `instrumentId`. | | `get_watcher_heartbeats` | Recent heartbeat history for a watcher, useful for diagnosing connectivity gaps and error trends. Set the `hours` lookback (default 24, max 168). | ## Resources [#resources] Resources are reference context that clients can attach to prompts without an explicit tool call. | URI | Description | | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `datahub://instruments` | List of all instrument IDs, display names, and types. Useful as grounding context when constructing tool calls. | | `datahub://instruments/{instrumentId}/filter-options` | Available filter values for an instrument. Plate readers expose wavelengths and measurement modes/types; gel-doc instruments expose capture types, imaging modes, wavelengths, and colors. Helps build valid `search_runs` queries. | ## Prompts [#prompts] Prompts are scripted workflows the client surfaces to the user. Each prompt assembles a multi-step instruction that the model then executes using the tools above. | Prompt | Args | Description | | ------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | `daily_summary` | `date` (optional, YYYY-MM-DD) | Summarize all instrument activity for a given day: run counts, failures, and system health. | | `troubleshoot_instrument` | `instrumentId` | Diagnose connectivity or processing issues for an instrument by inspecting its status, watcher heartbeats, and recent runs. | | `compare_runs` | `instrumentId`, `runId1`, `runId2` | Compare two runs on the same instrument side by side, highlighting differences in conditions and outcomes. | ## Example usage [#example-usage] Once installed, ask your client questions like: * *“What instruments are active right now?”* → `list_instruments` with `status="active"` * *“Show me all SpectraMax runs from last Friday.”* → `search_runs` with an `instrumentId` and date range * *“The gel-doc in Lab 3 stopped uploading, what's wrong?”* → `troubleshoot_instrument` prompt, which inspects the watcher list and heartbeat history * *“Re-run processing for file 4217, we pushed a parser fix.”* → `reprocess_file`. Clients typically confirm the destructive action with the user first. * *“Claim run `2026-03-26_experiment` on the SpectraMax, I ran it this morning.”* → `claim_run`. To find runs you've already claimed, use `search_runs` with `ranBy` set to your user id (discoverable via `list_run_attributors`). ## Troubleshooting [#troubleshooting] ### `401 Unauthorized` [#401-unauthorized] The Bearer token is missing, mistyped, revoked, or expired. Verify the token at **Settings > Access Tokens** and re-issue if necessary. ### Tools don't appear in the client [#tools-dont-appear-in-the-client] * Confirm the server is listed under `mcpServers` in the client config. * Check for JSON syntax errors in the config file. * Restart the client after editing; most clients don't hot-reload MCP server definitions. * Hit `https://datahub.example.com/api/v1/mcp` with `curl -H "Authorization: Bearer "` to confirm the endpoint responds. ### `get_file_download_url` vs. `get_run_archive` [#get_file_download_url-vs-get_run_archive] * `get_file_download_url` returns a pre-signed S3 URL for a single file. Anyone with the link can fetch it for 15 minutes; no Data Hub credentials required on the follow-up request. * `get_run_archive` returns the same kind of pre-signed S3 URL, but for a multi-file ZIP archive of an entire run. On a cache miss the Lambda builds the archive asynchronously and the tool returns `{ status: "building", jobId, retryAfterSeconds }`; call the tool again after the suggested wait until you get back `{ status: "ready", downloadUrl }`. Like `get_file_download_url`, the URL itself carries the auth, so the resulting link is browser-clickable without the original Bearer token. # Overview (/docs/overview) Data Hub is a platform for automatically ingesting, processing, and visualizing data generated by scientific instruments in the lab. It's self-hosted, so you can deploy it on your own infrastructure and keep your data private. And it's open source, so you can fork it and customize it to your needs. Because Data Hub is self-hosted, your team runs the backend (a PostgreSQL database, the web app on Vercel, and AWS S3 plus Lambda) before anyone can sign in or install a watcher. The step-by-step setup guide lives in the [data-hub developer docs](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/first-time-deployment.md). ## How it works [#how-it-works] 1. A Python service called the **watcher** monitors a directory on the instrument PC for new files (e.g. `*.csv` files in the `C:\Data` directory with prefix `RUN-`). 2. Incoming files are grouped into **runs** based on your configured patterns (e.g. `Experiment--