Skip to main content
Trinity
Updated Nov 17, 2026main@87e21ff
Architecture/Monitoring

Monitoring

Multi-layer health monitoring for the agent fleet with real-time alerts, automatic cleanup of stuck resources, and a fleet-wide health view on the Operations page.

Health Levels

Ordered by severity:

LevelMeaning
healthyAll checks passing
degradedMinor issues detected
unhealthySignificant problems
criticalImmediate attention required
unknownUnable to determine status

Three Monitoring Layers

1

Docker layer — Container status, CPU/memory usage, restart count, OOM detection.

2

Network layer — Agent HTTP reachability with latency tracking.

3

Business layer — Runtime availability, context usage, error rates.

Heartbeat Liveness and Alert Cooldowns

Heartbeat Liveness: In addition to the periodic health-check loop, each running agent pushes a lightweight heartbeat to the backend every 5 seconds. See Agent Heartbeats below.

Alert Cooldowns: Repeated alerts for the same condition are throttled to prevent notification spam.

Health Tab (Operations Page)

Fleet health lives on the Health tab of the Operations page (/operations?tab=health). The tab is admin-only — non-admin users do not see it, and deep links to ?tab=health fall back to the default tab. The legacy /monitoring route redirects there.

The tab shows summary cards (Total Agents, Healthy, Degraded, Unhealthy, Critical), active alerts, and a per-agent health list with a status filter. The cards always add up to the total: an agent with no health reading counts as unknown. A reading older than three check intervals (90 seconds by default, stale_after_seconds on the API) also reads unknown, and its issues lead with its age and last status, plus monitoring is off when the loop is stopped, so old readings are never shown as healthy. Admins can trigger a fleet-wide check with Check All or a single-agent check from each row.

•Active Alerts lists every alert its header counts. A long list scrolls inside the panel (about five rows tall, reachable from the keyboard) instead of growing the page.
•An agent whose transport circuit breaker is not closed carries a Breaker open or Breaker dormant badge on its row, with the number of failed connection attempts. Open means the agent is not answering and requests fail fast until a probe succeeds; dormant means the platform has stopped probing it after too many failed probes — run a health check on the agent to reset it. A closed breaker shows nothing.
•Real-time WebSocket updates push health state changes as they occur.
•Individual agent health is visible in both the agent header and the Agents listing page.

Enabling Monitoring

The periodic health-check loop is disabled by default. A status badge at the top of the Health tab shows the current state: "Monitoring Active" or "Monitoring Disabled", with an Enable monitoring / Disable monitoringbutton next to it (admin only) — no API call needed.

The same control is available via the API (admin only):

POST /api/monitoring/enable
POST /api/monitoring/disable

The choice is persisted, so it survives backend restarts — if monitoring was enabled, the loop resumes automatically on boot. The check interval and other options are configured via GET/PUT /api/monitoring/config, which also persists and applies the enabled flag.

Agent Heartbeats

Independently of the health-check loop, every running agent (on a current image) POSTs a small heartbeat to the backend every ~5 seconds, authenticated with its own agent-scoped MCP key. A backend watch loop acts on missed beats:

•After 3 consecutive missed beats, a soft, high-priority operator alert fires — exactly once per loss episode.
•When beats resume after a loss, a recovery notification fires.
•A missed beat never hard-marks an agent as down by itself — the alert is advisory, because a missed beat can also mean a transient network issue.

Each agent resolves to one of three heartbeat states:

StateMeaning
aliveBeating normally
staleWas beating, then stopped
unsupportedAgent runs an older image that never sent a beat — never treated as dead

Heartbeat fields (heartbeat_state, heartbeat_alive, last_heartbeat_age_s, heartbeat_memory_mb, heartbeat_active_executions) surface on GET /api/monitoring/status for each agent. Heartbeats work even when the periodic monitoring loop is disabled.

Cleanup Service

A background service that automatically recovers stuck resources:

•Orphaned executions — The watchdog asks each running agent which executions it knows about (running, just finished, or accepted but not yet started). A running row the agent does not know is orphaned only if no live backend dispatcher owns it either — an execution admitted and parked in the backend's agent-call queue, or accepted by the agent but not yet spawned, is left alone. Rows withheld this way are reported as dispatch_inflight_skipped in the cleanup report; they are not recoveries. A genuine orphan is marked failed with an error stating what was observed, and its slot is released.
•Stale executions — Any execution with status='running' past its per-slot timeout is marked failed.
•Stale activities — Any activity with activity_state='started' past the configured threshold is marked failed.
•Stale Redis slots — Orphaned slot reservations are released.
•Orphan agent volumes — A Docker data volume whose agent no longer exists (left behind by a crash mid-create, or skipped because it was in use at purge time) is removed once it is at least an hour old and has been seen unattached on three consecutive cycles (about 15 minutes), up to 100 per cycle. Only volumes this Trinity install labelled as its own (trinity.instance) are candidates, so two Trinity stacks sharing one Docker daemon never remove each other's volumes. Volumes created before that label existed are never reclaimed automatically; unowned, unattached ones are named in one WARNING log line for a person to remove. Every removal is logged at WARNING, because it cannot be undone.
•Run frequency — Every 5 minutes, plus a one-shot sweep on backend restart.
•Startup recovery — Orphaned executions (container down, not in process registry, and not owned by a dispatcher in another worker) are marked failed immediately and their slots are released.
•Results recovered after a backend restart — A turn the backend was waiting on when it restarted is not failed just because the waiting connection died. The agent keeps each finished turn's result for up to 6 hours (the newest 50), and before marking such a row failed the watchdog asks the agent for it. If the agent has it, the execution is closed with its real result and cost, and a successful response opens with a Recovered by the watchdog notice. The reply is also written into the chat thread that was waiting on it; a chat already open in the browser shows it after a reload. Agents on an older base image keep the previous behaviour.

A close the cleanup service fabricates records no duration — duration_ms is NULL, not a number computed from started_at. Earlier versions wrote a made-up duration, and on PostgreSQL a row older than about 25 days overflowed the column and rolled back the whole sweep, so nothing stale was ever closed again while the cycle still logged "complete". If you upgrade an instance in that state, the restart's startup recovery closes those rows; no manual SQL is needed.

Retention Sweeps

The same cleanup service runs retention sweeps to keep the database lean. Setting any window to 0 disables that sweep — except backup_retention_days, where 0 is rejected (disable backups with DB_BACKUP_ENABLED=false instead).

Settings → Retention — per-window day inputs for logs, executions, health checks, soft-deleted agents/schedules, and the audit log, with the pending-approvals banner above them
SweepDefaultSetting
schedule_executions.execution_log nulled past30 daysexecution_log_retention_days
Terminal schedule_executions rows deleted past90 daysexecution_row_retention_days
agent_health_checks rows deleted past7 dayshealth_check_retention_days
Agent soft-delete purged past180 daysagent_soft_delete_retention_days
Schedule soft-delete purged past30 daysschedule_soft_delete_retention_days
Agent reports deleted past90 daysagent_reports_retention_days
Terminal operator-queue rows deleted past90 daysoperator_queue_retention_days
Terminal agent reminders (fired/cancelled/failed) deleted past90 daysagent_reminders_retention_days
Subscription headroom probe history deleted past30 dayssubscription_headroom_retention_days
Subscription rate-limit / auth failure events deleted past30 dayssubscription_failure_event_retention_days
Recorded metric points deleted past365 daysmetrics_retention_days (see Declared Metrics)
Database backup files deleted past14 daysbackup_retention_days (1–3650; the newest 3 are always kept)
audit_log rows deleted past365 daysAUDIT_LOG_RETENTION_DAYS (floor 365, exempt)

The agent soft-delete purge is special: it destroys the agent's data volumes, so it is a recovery window, not a log window — it is exempt from the community floor below.

The 5,000-row figure some tooling reports bounds each transaction, not each sweep. Most prunes drain the whole candidate set in one sweep; the real bound on destruction is the blast-radius guard below, not chunking. A daily VACUUM at 04:30 UTC reclaims freed pages.

Community Floor (Fresh Installs)

Fresh community installs are seeded with a 5-day minimum retention on the log windows. This applies to new installs only— it is never a retroactive change to an existing install, and the agent soft-delete window is exempt in every edition. Any admin can widen a window at any time.

Every Install Owns Its Windows

On every boot, Trinity writes an explicit row for any retention window that has none, at the value already in force — upgraded installs included, not just fresh ones. Nothing prunes differently the day this happens. The consequence is that retention is per-install configuration: a later change to the built-in defaults, in either direction, never silently changes how much data an existing install keeps. GET /api/settings/retention therefore reports every window with source db-row — except the metric knobs below when their environment variable is set, which report env.

Blast-Radius Guard and Admin Approval

A sweep that would delete more than a fixed safety threshold (1,000 rows) of a single table refuses to run, logs an error, and raises an operator-queue alarm instead of deleting. An admin must then approve it in Settings → Retention, which shows a Deletion awaiting your approval banner with an Approve deletion button per pending prune.

The approval is:

•Admin- and human-only — agent-scoped keys cannot approve.
•Bound to the exact window in force — approving names the deletion it authorizes; a mismatched window is rejected.
•Single-use — the guard re-arms after the prune runs, so each over-threshold sweep needs its own approval.

Agent-purge sweeps always require an acknowledgement because every one destroys data volumes.

Metric Points: Window and Daily Quota

Two knobs govern recorded metric points: the retention window metrics_retention_days (default 365) and the write quota metrics_daily_point_cap — how many points one agent may record per UTC day (default 100,000; 0 = unlimited). Unlike the other windows, both can come from the environment: METRICS_RETENTION_DAYS and METRICS_DAILY_POINT_CAP (left commented out in .env.example). A saved value wins over the variable, and the variable wins over the default. GET /api/settings/retention reports the quota under quotas.metrics_daily_point_cap with its value and source (db-row, env, or code-default).

The recorded-metric-points sweep uses a larger threshold (100,000 rows, one agent-day at the default daily cap), because a busy fleet ages more than 1,000 points out of the window every cycle. It still stops for approval when you narrow the window and a large backlog expires at once, and one approval covers the whole backlog even when it takes several cycles to drain. When the window comes from METRICS_RETENTION_DAYS, the approval is bound to that environment value — the same window the sweep uses.

Changing a Window

Settings → Retention shows every window with its source. In the Community edition the panel is read-only — it shows the windows and the approval banner, and editing windows from the panel requires an enterprise entitlement. Where the panel can save, a value supplied by an environment variable is shown with an env badge and is read-only; unset the variable and restart to edit it there.

The OSS write path for retention windows is PUT /api/settings/ops/config (admin). It type- and range-validates every value all-or-nothing (one bad value rejects the whole request with a 422) and writes an audit entry naming which windows moved. The generic settings endpoint refuses these keys and points you here.

Two things worth internalising, because the risk is counter-intuitive:

•Garbage always failed safe — an unparseable value became 0, which disables the sweep and retains forever.
•A small valid number is the dangerous input. 1 is well-formed and in range, and no range check can distinguish it from an operator who genuinely wants a one-day window. Validation buys you a loud failure instead of a silent coercion; what actually stops an accidental mass deletion is the blast-radius guard above and the admin-plus-human gate on approving it.

Reset to defaults (POST /api/settings/ops/reset) deliberately skipsretention windows — resetting other operator settings must never silently change how much of your data is kept. The reset is itself audit-logged.

Endpoints:

EndpointMethodDescription
/api/settings/retentionGETEffective windows (per-window value + source), the metric point quota (quotas), edition, community-floor days, the read-only guard threshold, pending acknowledgements, sweeps the guard is blocking for another reason (blocked_sweeps), and the database backup status (backup) (admin)
/api/settings/ops/configPUTThe single validated, audited write path for retention windows (admin)
/api/settings/retention/acknowledgePOSTApprove one over-threshold prune — body {key, window_days} (admin, human-only)

Database Backups

Trinity backs up its own database every night at 03:30 UTC and before migrations at boot, into ~/trinity-data/backups/. Admins see the status on the Database backups card in Settings → Retention: a state badge (Healthy, Stale, Last backup failed, Skipped — not enough disk space, No backup yet or Off), the last successful backup, the recovery points, the location, the schedule and how long backups are kept. The backup block of GET /api/settings/retention reports whether backups are enabled, the schedule, the last result and its age, stale (no success for more than 3 days), and the files on disk. A failed or skipped backup, or a stale run of them, raises an operator-queue alert. Backups sit on the same disk as the database: they protect against corruption and mistakes, not disk loss. See Backup and Restore.

Real-Time Event Reliability

Trinity uses a Redis Streams-backed event bus for all WebSocket delivery. This is invisible during normal operation but has operator-visible behaviour in a few edge cases.

Reconnect Replay

When a browser tab reconnects after a brief disconnect (e.g., laptop sleep, flaky network), it automatically requests missed events using a ?last-event-id= cursor tracked in memory. Events are replayed from the Redis stream, so the collaboration dashboard, activity timeline, and operator queue resume without stale state.

resync_required Events

If the cursor is too far behind (>5,000 events missed) or the stream has been trimmed past the stored cursor, the server sends a {"type": "resync_required"} message. The frontend clears the cursor and refetches authoritative state via REST. Users see a brief refresh but no data loss.

The stream retains approximately the last 10,000 events (configurable via REDIS_STREAM_MAXLEN in .env).

Admin Stats Endpoint

For soak monitoring and diagnosing delivery issues:

GET /api/debug/event-bus-stats    (admin-only)

Returns counters since last backend restart:

FieldWhat to check
publisher.events_publishedTotal events emitted
dispatcher.drops_queue_fullEvents dropped due to slow clients
dispatcher.clients_evictedConnections closed after 3 consecutive send failures
dispatcher.resyncs_sentForced full-state refreshes sent to clients
watchdog.cumulative_orphanedOrphaned executions recovered by cleanup service

Healthy baseline: drops_queue_full + clients_evicted + resyncs_sent should be < 0.1% of events_published. Non-zero cumulative_orphaned warrants investigation.

MCP Tools

Git Sync Health on Fleet Health

Each agent in the fleet health response carries a sync block with its git sync health: whether it is an agent or a deployment, whether auto-sync is on, how many commits it is ahead of and behind its own branch on GitHub, how many files are uncommitted, when its last push landed, how long it has been out of step, its state (green / yellow / red / unknown), a reason and a recommendation. Agents with no GitHub binding report sync: null. The response also totals the fleet in sync_summary (diverged, frozen, auto-sync off, dirty, red, yellow).

A red agent gets an extra entry in its issues list — for example sync: diverged 0 behind / 7 ahead for 26h — enable auto-sync. Sync health never changes an agent's health status or the healthy/degraded counts. The reason never includes the raw git error text.

Agents can query monitoring data through these MCP tools:

ToolDescription
get_fleet_health()Fleet-wide health summary, with each agent's git sync block and the fleet sync_summary
get_fleet_sync_audit()The fleet git sync audit: ahead/behind, dirty files, last push, state and recommendation per agent
get_agent_health(name)Individual agent health, with an informational metrics block for the agent's declared metrics
trigger_health_check()Force an immediate health check

API Endpoints

EndpointMethodDescription
/api/monitoring/statusGETFleet health summary (includes heartbeat_* fields and a per-agent git sync block)
/api/monitoring/agents/{name}GETSingle-agent health detail. Carries an informational metrics block — how many metrics are declared and recorded, which are stale, have no cadence, or have no points, and the newest point time. It never changes the health status or the issues list, and is null if the metric store cannot be read
/api/monitoring/agents/{name}/checkPOSTForce immediate health check
/api/monitoring/enablePOSTStart the health-check loop; persisted (admin)
/api/monitoring/disablePOSTStop the health-check loop; persisted (admin)
/api/monitoring/configGET/PUTMonitoring configuration, including the enabled flag (admin)
/api/monitoring/check-allPOSTTrigger a fleet-wide health check (admin)
/api/monitoring/cleanup-statusGETCleanup service status (admin)
/api/monitoring/cleanup-triggerPOSTForce a cleanup run (admin)
/api/agents/{name}/heartbeatPOSTAgent liveness heartbeat (sent by the agent itself every ~5s)