Monitoring
Multi-layer health monitoring for the agent fleet with real-time alerts, automatic cleanup of stuck resources, and a fleet-wide health view on the Operations page.
Health Levels
Ordered by severity:
| Level | Meaning |
|---|---|
| healthy | All checks passing |
| degraded | Minor issues detected |
| unhealthy | Significant problems |
| critical | Immediate attention required |
| unknown | Unable to determine status |
Three Monitoring Layers
Docker layer — Container status, CPU/memory usage, restart count, OOM detection.
Network layer — Agent HTTP reachability with latency tracking.
Business layer — Runtime availability, context usage, error rates.
Heartbeat Liveness and Alert Cooldowns
Heartbeat Liveness: In addition to the periodic health-check loop, each running agent pushes a lightweight heartbeat to the backend every 5 seconds. See Agent Heartbeats below.
Alert Cooldowns: Repeated alerts for the same condition are throttled to prevent notification spam.
Health Tab (Operations Page)
Fleet health lives on the Health tab of the Operations page (/operations?tab=health). The tab is admin-only — non-admin users do not see it, and deep links to ?tab=health fall back to the default tab. The legacy /monitoring route redirects there.
The tab shows summary cards (Total Agents, Healthy, Degraded, Unhealthy, Critical), active alerts, and a per-agent health list with a status filter. The cards always add up to the total: an agent with no health reading counts as unknown. A reading older than three check intervals (90 seconds by default, stale_after_seconds on the API) also reads unknown, and its issues lead with its age and last status, plus monitoring is off when the loop is stopped, so old readings are never shown as healthy. Admins can trigger a fleet-wide check with Check All or a single-agent check from each row.
Enabling Monitoring
The periodic health-check loop is disabled by default. A status badge at the top of the Health tab shows the current state: "Monitoring Active" or "Monitoring Disabled", with an Enable monitoring / Disable monitoringbutton next to it (admin only) — no API call needed.
The same control is available via the API (admin only):
POST /api/monitoring/enable
POST /api/monitoring/disableThe choice is persisted, so it survives backend restarts — if monitoring was enabled, the loop resumes automatically on boot. The check interval and other options are configured via GET/PUT /api/monitoring/config, which also persists and applies the enabled flag.
Agent Heartbeats
Independently of the health-check loop, every running agent (on a current image) POSTs a small heartbeat to the backend every ~5 seconds, authenticated with its own agent-scoped MCP key. A backend watch loop acts on missed beats:
Each agent resolves to one of three heartbeat states:
| State | Meaning |
|---|---|
| alive | Beating normally |
| stale | Was beating, then stopped |
| unsupported | Agent runs an older image that never sent a beat — never treated as dead |
Heartbeat fields (heartbeat_state, heartbeat_alive, last_heartbeat_age_s, heartbeat_memory_mb, heartbeat_active_executions) surface on GET /api/monitoring/status for each agent. Heartbeats work even when the periodic monitoring loop is disabled.
Cleanup Service
A background service that automatically recovers stuck resources:
running row the agent does not know is orphaned only if no live backend dispatcher owns it either — an execution admitted and parked in the backend's agent-call queue, or accepted by the agent but not yet spawned, is left alone. Rows withheld this way are reported as dispatch_inflight_skipped in the cleanup report; they are not recoveries. A genuine orphan is marked failed with an error stating what was observed, and its slot is released.status='running' past its per-slot timeout is marked failed.activity_state='started' past the configured threshold is marked failed.trinity.instance) are candidates, so two Trinity stacks sharing one Docker daemon never remove each other's volumes. Volumes created before that label existed are never reclaimed automatically; unowned, unattached ones are named in one WARNING log line for a person to remove. Every removal is logged at WARNING, because it cannot be undone.failed immediately and their slots are released.failed the watchdog asks the agent for it. If the agent has it, the execution is closed with its real result and cost, and a successful response opens with a Recovered by the watchdog notice. The reply is also written into the chat thread that was waiting on it; a chat already open in the browser shows it after a reload. Agents on an older base image keep the previous behaviour.A close the cleanup service fabricates records no duration — duration_ms is NULL, not a number computed from started_at. Earlier versions wrote a made-up duration, and on PostgreSQL a row older than about 25 days overflowed the column and rolled back the whole sweep, so nothing stale was ever closed again while the cycle still logged "complete". If you upgrade an instance in that state, the restart's startup recovery closes those rows; no manual SQL is needed.
Retention Sweeps
The same cleanup service runs retention sweeps to keep the database lean. Setting any window to 0 disables that sweep — except backup_retention_days, where 0 is rejected (disable backups with DB_BACKUP_ENABLED=false instead).

| Sweep | Default | Setting |
|---|---|---|
schedule_executions.execution_log nulled past | 30 days | execution_log_retention_days |
Terminal schedule_executions rows deleted past | 90 days | execution_row_retention_days |
agent_health_checks rows deleted past | 7 days | health_check_retention_days |
| Agent soft-delete purged past | 180 days | agent_soft_delete_retention_days |
| Schedule soft-delete purged past | 30 days | schedule_soft_delete_retention_days |
| Agent reports deleted past | 90 days | agent_reports_retention_days |
| Terminal operator-queue rows deleted past | 90 days | operator_queue_retention_days |
| Terminal agent reminders (fired/cancelled/failed) deleted past | 90 days | agent_reminders_retention_days |
| Subscription headroom probe history deleted past | 30 days | subscription_headroom_retention_days |
| Subscription rate-limit / auth failure events deleted past | 30 days | subscription_failure_event_retention_days |
| Recorded metric points deleted past | 365 days | metrics_retention_days (see Declared Metrics) |
| Database backup files deleted past | 14 days | backup_retention_days (1–3650; the newest 3 are always kept) |
audit_log rows deleted past | 365 days | AUDIT_LOG_RETENTION_DAYS (floor 365, exempt) |
The agent soft-delete purge is special: it destroys the agent's data volumes, so it is a recovery window, not a log window — it is exempt from the community floor below.
The 5,000-row figure some tooling reports bounds each transaction, not each sweep. Most prunes drain the whole candidate set in one sweep; the real bound on destruction is the blast-radius guard below, not chunking. A daily VACUUM at 04:30 UTC reclaims freed pages.
Community Floor (Fresh Installs)
Fresh community installs are seeded with a 5-day minimum retention on the log windows. This applies to new installs only— it is never a retroactive change to an existing install, and the agent soft-delete window is exempt in every edition. Any admin can widen a window at any time.
Every Install Owns Its Windows
On every boot, Trinity writes an explicit row for any retention window that has none, at the value already in force — upgraded installs included, not just fresh ones. Nothing prunes differently the day this happens. The consequence is that retention is per-install configuration: a later change to the built-in defaults, in either direction, never silently changes how much data an existing install keeps. GET /api/settings/retention therefore reports every window with source db-row — except the metric knobs below when their environment variable is set, which report env.
Blast-Radius Guard and Admin Approval
A sweep that would delete more than a fixed safety threshold (1,000 rows) of a single table refuses to run, logs an error, and raises an operator-queue alarm instead of deleting. An admin must then approve it in Settings → Retention, which shows a Deletion awaiting your approval banner with an Approve deletion button per pending prune.
The approval is:
Agent-purge sweeps always require an acknowledgement because every one destroys data volumes.
Metric Points: Window and Daily Quota
Two knobs govern recorded metric points: the retention window metrics_retention_days (default 365) and the write quota metrics_daily_point_cap — how many points one agent may record per UTC day (default 100,000; 0 = unlimited). Unlike the other windows, both can come from the environment: METRICS_RETENTION_DAYS and METRICS_DAILY_POINT_CAP (left commented out in .env.example). A saved value wins over the variable, and the variable wins over the default. GET /api/settings/retention reports the quota under quotas.metrics_daily_point_cap with its value and source (db-row, env, or code-default).
The recorded-metric-points sweep uses a larger threshold (100,000 rows, one agent-day at the default daily cap), because a busy fleet ages more than 1,000 points out of the window every cycle. It still stops for approval when you narrow the window and a large backlog expires at once, and one approval covers the whole backlog even when it takes several cycles to drain. When the window comes from METRICS_RETENTION_DAYS, the approval is bound to that environment value — the same window the sweep uses.
Changing a Window
Settings → Retention shows every window with its source. In the Community edition the panel is read-only — it shows the windows and the approval banner, and editing windows from the panel requires an enterprise entitlement. Where the panel can save, a value supplied by an environment variable is shown with an env badge and is read-only; unset the variable and restart to edit it there.
The OSS write path for retention windows is PUT /api/settings/ops/config (admin). It type- and range-validates every value all-or-nothing (one bad value rejects the whole request with a 422) and writes an audit entry naming which windows moved. The generic settings endpoint refuses these keys and points you here.
Two things worth internalising, because the risk is counter-intuitive:
0, which disables the sweep and retains forever.1 is well-formed and in range, and no range check can distinguish it from an operator who genuinely wants a one-day window. Validation buys you a loud failure instead of a silent coercion; what actually stops an accidental mass deletion is the blast-radius guard above and the admin-plus-human gate on approving it.Reset to defaults (POST /api/settings/ops/reset) deliberately skipsretention windows — resetting other operator settings must never silently change how much of your data is kept. The reset is itself audit-logged.
Endpoints:
| Endpoint | Method | Description |
|---|---|---|
| /api/settings/retention | GET | Effective windows (per-window value + source), the metric point quota (quotas), edition, community-floor days, the read-only guard threshold, pending acknowledgements, sweeps the guard is blocking for another reason (blocked_sweeps), and the database backup status (backup) (admin) |
| /api/settings/ops/config | PUT | The single validated, audited write path for retention windows (admin) |
| /api/settings/retention/acknowledge | POST | Approve one over-threshold prune — body {key, window_days} (admin, human-only) |
Database Backups
Trinity backs up its own database every night at 03:30 UTC and before migrations at boot, into ~/trinity-data/backups/. Admins see the status on the Database backups card in Settings → Retention: a state badge (Healthy, Stale, Last backup failed, Skipped — not enough disk space, No backup yet or Off), the last successful backup, the recovery points, the location, the schedule and how long backups are kept. The backup block of GET /api/settings/retention reports whether backups are enabled, the schedule, the last result and its age, stale (no success for more than 3 days), and the files on disk. A failed or skipped backup, or a stale run of them, raises an operator-queue alert. Backups sit on the same disk as the database: they protect against corruption and mistakes, not disk loss. See Backup and Restore.
Real-Time Event Reliability
Trinity uses a Redis Streams-backed event bus for all WebSocket delivery. This is invisible during normal operation but has operator-visible behaviour in a few edge cases.
Reconnect Replay
When a browser tab reconnects after a brief disconnect (e.g., laptop sleep, flaky network), it automatically requests missed events using a ?last-event-id= cursor tracked in memory. Events are replayed from the Redis stream, so the collaboration dashboard, activity timeline, and operator queue resume without stale state.
resync_required Events
If the cursor is too far behind (>5,000 events missed) or the stream has been trimmed past the stored cursor, the server sends a {"type": "resync_required"} message. The frontend clears the cursor and refetches authoritative state via REST. Users see a brief refresh but no data loss.
The stream retains approximately the last 10,000 events (configurable via REDIS_STREAM_MAXLEN in .env).
Admin Stats Endpoint
For soak monitoring and diagnosing delivery issues:
GET /api/debug/event-bus-stats (admin-only)Returns counters since last backend restart:
| Field | What to check |
|---|---|
| publisher.events_published | Total events emitted |
| dispatcher.drops_queue_full | Events dropped due to slow clients |
| dispatcher.clients_evicted | Connections closed after 3 consecutive send failures |
| dispatcher.resyncs_sent | Forced full-state refreshes sent to clients |
| watchdog.cumulative_orphaned | Orphaned executions recovered by cleanup service |
Healthy baseline: drops_queue_full + clients_evicted + resyncs_sent should be < 0.1% of events_published. Non-zero cumulative_orphaned warrants investigation.
MCP Tools
Git Sync Health on Fleet Health
Each agent in the fleet health response carries a sync block with its git sync health: whether it is an agent or a deployment, whether auto-sync is on, how many commits it is ahead of and behind its own branch on GitHub, how many files are uncommitted, when its last push landed, how long it has been out of step, its state (green / yellow / red / unknown), a reason and a recommendation. Agents with no GitHub binding report sync: null. The response also totals the fleet in sync_summary (diverged, frozen, auto-sync off, dirty, red, yellow).
A red agent gets an extra entry in its issues list — for example sync: diverged 0 behind / 7 ahead for 26h — enable auto-sync. Sync health never changes an agent's health status or the healthy/degraded counts. The reason never includes the raw git error text.
Agents can query monitoring data through these MCP tools:
| Tool | Description |
|---|---|
| get_fleet_health() | Fleet-wide health summary, with each agent's git sync block and the fleet sync_summary |
| get_fleet_sync_audit() | The fleet git sync audit: ahead/behind, dirty files, last push, state and recommendation per agent |
| get_agent_health(name) | Individual agent health, with an informational metrics block for the agent's declared metrics |
| trigger_health_check() | Force an immediate health check |
API Endpoints
| Endpoint | Method | Description |
|---|---|---|
| /api/monitoring/status | GET | Fleet health summary (includes heartbeat_* fields and a per-agent git sync block) |
| /api/monitoring/agents/{name} | GET | Single-agent health detail. Carries an informational metrics block — how many metrics are declared and recorded, which are stale, have no cadence, or have no points, and the newest point time. It never changes the health status or the issues list, and is null if the metric store cannot be read |
| /api/monitoring/agents/{name}/check | POST | Force immediate health check |
| /api/monitoring/enable | POST | Start the health-check loop; persisted (admin) |
| /api/monitoring/disable | POST | Stop the health-check loop; persisted (admin) |
| /api/monitoring/config | GET/PUT | Monitoring configuration, including the enabled flag (admin) |
| /api/monitoring/check-all | POST | Trigger a fleet-wide health check (admin) |
| /api/monitoring/cleanup-status | GET | Cleanup service status (admin) |
| /api/monitoring/cleanup-trigger | POST | Force a cleanup run (admin) |
| /api/agents/{name}/heartbeat | POST | Agent liveness heartbeat (sent by the agent itself every ~5s) |