System Overview
Trinity provides a real-time operational layer for monitoring, managing, and interacting with your agent fleet. The Dashboard, Operations page, and monitoring systems give operators full visibility into agent activity, health, and collaboration.
Architecture Diagram
Dashboard
The main Dashboard monitors all agents and their activities in real time, with three switchable view modes — Timeline, Grid, and List. Timeline is the default; the choice persists per browser. Header filters (type-to-filter with /, quick tag filter, owner filter, time range) apply to every mode, and Create Agent sits in the header. See the full Dashboard reference.
Grid View
A magnetic tile canvas holding two kinds of occupant on one lattice: agent tiles (one per agent, with avatar, runtime badge, live status chips, and inline Running and Autonomy toggles) and info tiles (fleet-level readouts such as Fleet summary and Recent failures).
- •Drag a tile to move it, or drop it onto another to swap positions — the layout is saved per browser
- •Tidy re-packs tiles compactly; Reset restores the default auto-generated layout
- •Pan by dragging the background and zoom with scroll or pinch; tile metrics hydrate lazily so large fleets stay responsive
- •An optional org overlay draws departments (
dept-*tags) as labelled zones and reporting lines (reports-to-*tags) as arrows between tiles
Timeline View (Default)
The default mode. Timeline shows execution boxes per agent, arranged chronologically.
- •Color-coded by trigger type: Manual (green), MCP (pink), Scheduled (purple), Agent-Triggered (cyan), Paid (yellow), Public (teal)
- •Each row shows the agent's completion rate, total cost, and parallel slot count
- •Live streaming: running executions show progress in real-time with a “Live” indicator
- •Time range filter: 1h, 6h, 24h, 7d, or custom
- •Active only toggle hides agents with no recent activity
- •Jump to Now snaps the view to the current time
Agent-to-agent collaboration is surfaced here — in the Timeline replay via the Agent-Triggered trigger type — rather than as a live node graph.
List View
The former standalone Agents page, folded into the Dashboard as a third mode (/agents now redirects here).
- •One row per agent: name, status, tags, runtime, read-only state, and last activity, with inline Run and Autonomy toggles
- •Sort by name, status, or activity; filter by name and status; select multiple rows for bulk tag operations
- •System agents pin to the top and hide the Run toggle
Tag Clouds and Activity Feed
Agents are grouped visually by tags on the Dashboard. Click a tag cloud to filter the view to that group.
A real-time WebSocket-driven activity stream shows agent collaborations, task starts/completions, schedule executions, and errors as they happen.
Dashboard API Endpoints
| Endpoint | Method | Description |
|---|---|---|
/api/agents | GET | List all agents |
/api/agents/context-stats | GET | Context and activity state for all agents |
/api/agents/autonomy-status | GET | Autonomy status for all agents |
/api/activities/timeline | GET | Cross-agent activity timeline (filterable) |
/api/executions/timeline | GET | Bucketed fleet rollups for building your own readouts |
/api/agents/{name}/tags | PUT | Set an agent's full tag list, including dept-* and reports-to-* org tags (rejected for agent-scoped keys) |
/api/telemetry/host | GET | Host CPU/memory/disk |
Monitoring
Multi-layer health monitoring for the agent fleet with real-time alerts, automatic cleanup of stuck resources, and a fleet-wide health view on the Operations page. In addition to the periodic health-check loop, each running agent pushes a lightweight heartbeat to the backend every 5 seconds.
Health Levels
Agent health is reported at five severity levels:
| Level | Meaning |
|---|---|
healthy | All checks passing |
degraded | Minor issues detected |
unhealthy | Significant problems |
critical | Immediate attention required |
unknown | Unable to determine status |
Three Monitoring Layers
Docker Layer — Container status, CPU/memory usage, restart count, OOM detection.
Network Layer — Agent HTTP reachability with latency tracking.
Business Layer — Runtime availability, context usage, error rates.
Alert Cooldowns: Repeated alerts for the same condition are throttled to prevent notification spam.
Health Tab (Operations Page)
Fleet health lives on the admin-only Health tab of the Operations page (/operations?tab=health). It shows summary cards, active alerts, and a per-agent health list with a status filter; admins can trigger a fleet-wide check with Check All. The periodic health-check loop is disabled by default and can be enabled via the API. Real-time WebSocket updates push health state changes as they occur. Individual agent health is visible in both the agent header and the Agents listing page.
Cleanup Service
A background service that automatically recovers stuck resources:
- •Stale executions — Any execution with
status='running'past its per-slot timeout is markedfailed - •Stale activities — Any activity with
activity_state='started'past the configured threshold is markedfailed - •Stale Redis slots — Orphaned slot reservations are released
- •Run frequency — Every 5 minutes, plus a one-shot sweep on backend restart
- •Startup recovery — Orphaned executions (container down, not in process registry) are marked
failedimmediately and their slots are released
Monitoring MCP Tools
| Tool | Description |
|---|---|
get_fleet_health() | Fleet-wide health summary |
get_agent_health(name) | Individual agent health |
trigger_health_check() | Force an immediate health check |
Monitoring API Endpoints
| Endpoint | Method | Description |
|---|---|---|
/api/monitoring/status | GET | Fleet health summary (includes heartbeat_* fields) |
/api/monitoring/enable | POST | Start the health-check loop; persisted (admin) |
/api/monitoring/disable | POST | Stop the health-check loop; persisted (admin) |
/api/monitoring/check-all | POST | Trigger a fleet-wide health check (admin) |
/api/monitoring/cleanup-status | GET | Cleanup service status (admin) |
/api/monitoring/cleanup-trigger | POST | Force a cleanup run (admin) |
Operations
The Operations page at /operations is the single fleet-operations surface: one tabbed view with Needs Response, Notifications, Health, Executions, and Resolved tabs. It replaces the former standalone Health, Operating Room, and Executions pages, and the navigation bar shows a single Operations entry with one unified badge of pending items.
Needs Response Tab
Shows items from agents' operator queues that are waiting on a human: questions, approval requests, and alerts.
- •Agents write to
~/.trinity/operator-queue.jsoninside their container - •A background sync service polls running agents every 5 seconds and persists items to the backend database
- •Operators respond to items directly; responses are written back to the originating agent
- •The first open item auto-expands when items arrive
- •WebSocket events:
operator_queue_new,operator_queue_responded,operator_queue_acknowledged,operator_queue_cleared
Notifications Tab
Consolidated view of agent notifications (replaces the former standalone Events page).
- •Filter by agent, type, priority, or status; optionally show dismissed items
- •Stats cards display pending, acknowledged, total, and per-agent counts
- •Bulk selection and bulk actions
- •Real-time updates via WebSocket
Health and Executions Tabs
The Health tab hosts fleet health monitoring (admin-only); the Executions tab lists all task runs across the fleet, and Operations → Reports shows the structured results agents publish (see Agent Reports). See the Monitoring and Executions sections above and the Operations page reference.
Resolved Tab
Terminal operator-queue items (responded, acknowledged, cancelled, expired). Responded items stay visible until the agent confirms delivery of the response.
Sync Service
Restart-resilient sync between agent containers and the backend database, with a manual refresh button on the operator tabs. Cancelled and expired statuses are written back into agent queue files, so agents stop waiting on cleared items.
Operator Queue API
| Endpoint | Method | Description |
|---|---|---|
/api/operator-queue | GET | List queue items |
/api/operator-queue/stats | GET | Queue statistics |
/api/operator-queue/bulk-cancel | POST | Cancel listed pending items |
/api/operator-queue/clear-resolved | POST | Hide terminal items |
/api/operator-queue/{id} | GET | Get single item |
/api/operator-queue/{id}/respond | POST | Submit response |
/api/operator-queue/{id}/cancel | POST | Cancel item |
/api/operator-queue/agents/{name} | GET | Items for a specific agent |
/api/notifications/dismiss-all | POST | Dismiss all pending + acknowledged notifications |
MCP Tool: send_notification(agent_name, message, priority) — sends a notification to the Operations page from within an agent.
Infrastructure Components
Backend (FastAPI)
:8000Python 3.11, FastAPI, Uvicorn
Central orchestrator with 40+ routers covering agents, chat, schedules, credentials, skills, processes, monitoring, and more. Manages agent containers via the Docker socket (mounted read-only). Broadcasts real-time events over WebSocket.
Frontend (Vue.js 3)
:80Vue.js 3, Tailwind CSS, Vite, Nginx
Single-page web dashboard for managing agents, viewing activity streams, monitoring schedules, and interacting with agents via chat. Connects to the backend API and WebSocket for live updates.
MCP Server
:8080Node.js, TypeScript, SSE transport
Model Context Protocol server exposing 116 tools across 28 modules for agent management, chat, schedules, skills, systems, tags, notifications, and monitoring. Enables Claude Code and other MCP clients to control Trinity programmatically.
Scheduler Service
:8001Python, APScheduler, Redis distributed locks
Dedicated single-instance service for cron-based agent task execution. Uses Redis distributed locking to prevent duplicate runs. Syncs schedules from the SQLite database and dispatches tasks to agents via the backend API.
Redis
:6379Redis 7 Alpine, AOF persistence
Stores encrypted credentials, distributed locks for the scheduler, execution queue state, and pub/sub events. Supports optional password authentication for production deployments.
SQLite
:N/ASQLite 3, /data/trinity.db
Primary data store for agents, users, schedules, activities, permissions, skills, tags, chat sessions, audit logs, and execution history. Stored on the trinity-data volume shared between backend and scheduler.
Vector (Log Aggregation)
:8686Timber Vector 0.43, Docker log source
Collects logs from all containers via the Docker socket. Writes structured NDJSON to the trinity-logs volume. The backend reads these logs for the activity stream and log viewer in the dashboard.
OTel Collector (Optional)
:4317 / 8889OpenTelemetry Collector Contrib 0.120
Receives OTLP metrics and traces from Claude Code agents running inside containers. Exports metrics in Prometheus format on port 8889. Enabled by setting OTEL_ENABLED=1.
Data Flow
A typical request flows through the system as follows:
Authentication — JWT tokens for browser sessions, MCP API keys for programmatic access. WebSocket connections require token authentication.
Execution Queue — Each agent processes one request at a time. Additional requests queue (up to 3). The slot service tracks capacity for the dashboard meter.
Agent Execution— The backend proxies chat to the agent's internal web server (running inside the container). Claude Code or Gemini CLI processes the request with full tool access.
Real-time Updates — Events broadcast over WebSocket to the dashboard and filtered to MCP clients based on agent access permissions.
Network Topology
All services run on a single Docker bridge network (trinity-agent-network, subnet 172.28.0.0/16). Agent containers are dynamically attached to this network when created.
Exposed Ports (Host)
| Port | Service | Notes |
|---|---|---|
| 80 | Frontend (Nginx) | Configurable via FRONTEND_PORT |
| 8000 | Backend (FastAPI) | REST API + WebSocket |
| 8080 | MCP Server | SSE transport, API key auth |
| 8001 | Scheduler | Health check endpoint only |
| 6379 | Redis | Optional password auth |
| 8686 | Vector | Health/API endpoint |
| 4317 | OTel Collector | gRPC OTLP receiver (optional) |
| 8889 | OTel Collector | Prometheus exporter (optional) |
| 2222+ | Agent SSH | Auto-assigned, one per agent |
Internal Communication
Services reference each other by container name on the Docker network. The MCP server reaches the backend at http://backend:8000. The scheduler connects to Redis at redis://redis:6379. Agent containers communicate with the backend using the internal API secret for authentication.
Storage
| Volume | Purpose | Mounted In |
|---|---|---|
| trinity-data | SQLite DB, archives | Backend, Scheduler |
| redis-data | Redis AOF persistence | Redis |
| trinity-logs | Vector NDJSON logs | Vector, Backend (ro) |
| trinity-archives | Compressed log archives | Backend |
| agent-configs | Agent configuration | Backend |
| agent-{name}-workspace | Per-agent persistent FS | Agent container |
SQLite (/data/trinity.db) stores all platform state: users, agents, permissions, schedules, activities, chat sessions, audit logs, skills, tags, and execution history.
Redis stores encrypted credentials, distributed scheduler locks, execution queue state, and pub/sub events. Configured with AOF persistence.
Agent Workspaces — Each agent gets a dedicated Docker volume (agent-{name}-workspace) mounted at /home/developer. This volume persists across container restarts and contains the agent's code, configuration, CLAUDE.md, and working files.
Host Mounts — The Docker socket is mounted read-only into the backend for container management. Agent templates, process templates, hooks, and process docs are mounted read-only from the host config directory.
Security Posture
All platform containers run with no-new-privileges and drop all Linux capabilities except what is required. The backend uses a tmpfs for /tmp with noexec. Agent containers can optionally run in restricted mode (default) or full-capabilities mode for packages that require apt-get.