Upgrading Trinity
Apply a new Trinity release to a running instance. The procedure keeps agent containers running throughout — only the platform services are rebuilt (source installs) or re-pulled (hosted installs) and restarted.
The one command
On the host, in the Trinity checkout:
./scripts/deploy/update.sh --check # preflight only — changes nothing
./scripts/deploy/update.sh # update to the newest release tag
./scripts/deploy/update.sh --to v1.0.0A target older than your checkout is refused, because an install on a release candidate or a branch is often ahead of the newest release. Pass --allow-downgrade if you mean it.
First update from a release without this command
Older checkouts do not contain update.sh. Run a copy from outside the checkout. Do not copy it into scripts/deploy/: the untracked file would block the checkout of the release that tracks it. Replace <release> with a release that contains update.sh:
git fetch --force --tags origin
mkdir -p /tmp/trinity-update
git show <release>:scripts/deploy/update.sh > /tmp/trinity-update/update.sh
git show <release>:scripts/deploy/update-db.py > /tmp/trinity-update/update-db.py
bash /tmp/trinity-update/update.sh --dir "$PWD" --to <release>After that the script is in the checkout. Rolling back to the older release removes it again, so the next update starts from this step.
(trinity update from the CLIruns the same script.) It works out how the install was deployed — dev, source-built production (with or without the enterprise overlay) or hosted — from the running backend's own compose labels, so it never runs the wrong file set. Then it:
- 1Preflights before touching anything: Docker reachable,
CREDENTIAL_ENCRYPTION_KEYset, no local edits to tracked files, enough free disk, the compose files render, and on an enterprise install the enterprise pin recorded by the target release is available. Any failure stops here with nothing changed. - 2Backs up the database through the running backend (SQLite: a verified copy at
backups/pre-migration-<timestamp>.db; PostgreSQL: a freshpg_dumpatbackups/pre-update-<timestamp>.dump), and records the schema position — which migrations the database has applied. - 3Checks out the target, and on an enterprise install the enterprise tree at the pin that release records.
- 4Compares published ports (address and port) with what is running. If a binding would move or disappear (for example because a
.envkey such asFRONTEND_PORTnow applies), it puts the checkout back and stops before any container changes — pass--accept-port-changeonce your proxy is updated. A binding that is only added prints a warning and the update continues. - 5Rebuilds and restarts the platform services (hosted: pulls via
start.sh --hosted). The agent base image is rebuilt only when the release changed it (--rebuild-baseforces it); agents pick it up on their next cold stop/start. - 6Verifies the published URL —
FRONTEND_URLfrom.env, or--url https://your-domain— not justlocalhost, retrying for up to two minutes while the frontend restarts (TRINITY_UPDATE_PUBLISHED_TIMEOUT): the page must be real HTML, its hashed UI bundle must load, and/api/auth/modemust answer through the same address. On an enterprise install it also confirms the enterprise modules registered.
If verification fails it prints ./scripts/deploy/update.sh --rollback (and offers it on a terminal). Rollback first checks whether a migration ran during the update:
No migration ran — the code, the enterprise tree and the hosted image tag return to the recorded version, are rebuilt and re-verified. The database is untouched. On a hosted install left on
latest, the tag to return to is read from the running image; if it cannot be, the update warns and the rollback refuses, so pinTRINITY_IMAGE_TAGbefore updating.A migration ran (or it cannot tell) — it refuses to roll back the code alone, because the older release cannot be assumed to run on the newer schema. It names the pre-update backup and prints the exact steps: stop
backendandscheduler, restore the backup (the commands are printed for SQLite or PostgreSQL), then run./scripts/deploy/update.sh --rollback --db-restored. Restoring discards every write made since the backup, so it never happens automatically.
--verify runs step 6 on its own at any time.
The manual procedure below is what the command automates; use it when you need to run a step by hand.
When to Run This
- •Source install (
docker-compose.ymlordocker-compose.prod.yml): after everygit pullthat includes changes to platform code (backend, frontend, MCP server, scheduler). Skip it for documentation-only changes. - •Hosted install (
start.sh --hosted, including the DigitalOcean 1-Click): whenever you moveTRINITY_IMAGE_TAGto a new release. If you left the tag atlatest, every re-run ofstart.sh --hostedis an upgrade — pin a version on any install you intend to keep.
Pre-flight Checklist
- Back up the database (Step 1 below — do this first, always). The platform also takes its own
pre-migration-<timestamp>.dbcopy at boot whenever a schema migration is about to run, but an explicit copy you can name is still the one to roll back to. - Confirm at least 2 GB disk free:
df -h / - Note the current version:
git rev-parse HEAD(source) or theTRINITY_IMAGE_TAGline in.env(hosted) - Check that no critical agent tasks are running:
docker ps --filter "label=trinity.platform=agent" - Confirm Docker is running:
docker info >/dev/null 2>&1 - Hosted: confirm the release is published — the
v0.9.1/0.9.1/0.9tags appear on GHCR for every release tag (a pre-release tag such asv0.9.5-rc1publishes its own tags but never moveslatest)
Step 1: Back Up the Database
The database is the critical state. Back it up before every upgrade.
Development (named volume)
docker run --rm \
-v trinity_trinity-data:/data \
-v ~/backups:/backup \
alpine cp /data/trinity.db /backup/trinity-$(date +%Y%m%d-%H%M%S).dbProduction / hosted note: On a production or hosted server the database lives in a bind-mount directory (TRINITY_DATA_PATH, default ./trinity-data in the checkout — e.g. /srv/trinity-data/, or /opt/trinity/trinity-data/ on a DigitalOcean Droplet), not in the named volume. Adjust accordingly:
cp /srv/trinity-data/trinity.db ~/backups/trinity-$(date +%Y%m%d-%H%M%S).dbA plain cp of a live database can be torn by a write in flight. For a guaranteed-consistent copy while the platform is running, use the online-backup form in Backup and Restore → Manual On-Demand Backup, or simply keep the newest automatic artifact from backups/ in the data directory.
Verify the backup is readable:
sqlite3 ~/backups/trinity-<timestamp>.db ".tables"
# Expected: a list of table names, no errorsPostgreSQL deployments (a postgresql:// DATABASE_URL — every hosted install that started fresh on the bundled database): back up with pg_dump instead of copying trinity.db, for example docker exec trinity-postgres pg_dump -U trinity -Fc trinity > ~/backups/trinity-pg-$(date +%Y%m%d-%H%M%S).dump — see Backup and Restore. PostgreSQL schema migrations run automatically on backend boot, same as SQLite.
Step 2: Pull Latest Changes
Source install:
git pull origin mainReview what changed:
git log --oneline -10
git diff HEAD~1 HEAD --statIf anything under docker/base-image/ appears in the diff, see Base Image Upgrade below.
Hosted install: pin the new release in .env (the last TRINITY_IMAGE_TAG line wins) and check out the matching tag, so the compose file and the mounted config/ move with the images:
echo 'TRINITY_IMAGE_TAG=v0.9.1' >> .env
git fetch --tags && git checkout v0.9.1start.sh reads the pin from .env; a value exported in the shell overrides it for that run only.
Step 3: Rebuild Platform Services
# Development
docker compose build --no-cache backend frontend mcp-server scheduler
# Production
docker compose -f docker-compose.prod.yml build --no-cache backend frontend mcp-server schedulerThe trinity-agent-base image is not rebuilt by this command. Rebuilding it means every agent has to be restarted to adopt it, so it is a separate step: rebuild it when anything under docker/base-image/ changed in the release, via ./scripts/deploy/build-base-image.sh. See Base Image Upgrade.
Hosted (pull-only) installs
There is nothing to build. One command re-pulls the four platform images and the agent base image, retags the base image locally, and brings the stack up:
./scripts/deploy/start.sh --hostedDo not substitute a bare docker compose -f docker-compose.hosted.yml pull: the agent base image is not a compose service, so that pull skips it and leaves every agent on the old runtime. start.sh --hosted also performs Step 4 for you — docker compose up -d recreates only the containers whose image changed and leaves the agent network in place — so continue at Step 5.
Caddy-fronted installs (DigitalOcean 1-Click, the installer script, AWS, Vultr). On a host that start.sh --provision prepared (it finds /etc/trinity/public-ip and an installed caddy service), the same re-run also brings Caddy in line with the new release — no --caddy-only step:
- •Before touching
.envor the stack it takes the provisioning lock the IP-refresh timer uses, so a refresh cannot rewrite the Caddy configuration or recreate a service mid-upgrade. If the web UI'sFRONTEND_BIND_ADDRESSis empty, it is set to127.0.0.1(only Caddy needs that port). - •Before pulling, it checks that the ingress network's range (
TRINITY_PROXY_PREFIX, default172.30.0.0/24) is free and stops — changing nothing — if another Docker network already uses it, naming that network. Set a free/24prefix in.envand re-run. - •After the backend is healthy and the web UI answers on its local port, it re-renders the Caddyfile. Caddy is pointed at the web container's PROXY-protocol listener (
127.0.0.1:${CADDY_PROXY_PORT:-18082}) only after a live check that the listener answers. A recreated web container answers on its ordinary port before this listener is ready, so the run waits up to 90 seconds for it. That listener is what carries each visitor's address to Trinity, so per-client sign-in limits and audit addresses see real clients. If the running web image is older than the checkout, Caddy keeps the ordinary listener and the run prints a warning that visitors' addresses are not yet passed through — pin the matchingTRINITY_IMAGE_TAGand re-run. - •The first switch restarts Caddy once, so open live views (execution logs, chat streams) reconnect. Later re-runs with an unchanged configuration leave Caddy running. A configuration Caddy rejects is never installed; the previous one keeps serving.
- •
/etc/caddy/Caddyfilebelongs to Trinity and is rewritten on every upgrade. Put your own sites in/etc/caddy/Caddyfile.d/<name>.caddy: the Caddyfile imports every*.caddyfile there, and Trinity never writes to that directory. If the Caddyfile was edited by hand since Trinity last wrote it (or, on an install from before this check, holds a site Trinity never renders, such as your own hostname), the upgrade does not replace it. It prints why, leaves the running configuration serving, and finishes the rest of the upgrade. To apply the new configuration, move your blocks intoCaddyfile.d/, then re-run with--caddy-overwrite; the old file is kept beside it asCaddyfile.bak-<timestamp>. To see the file before anything is written, runsudo ./scripts/deploy/start.sh --provision --cloud <digitalocean|vultr|aws> --caddy-only --dry-run(--dry-runworks only with--caddy-only; anywhere else it is refused rather than ignored). - •After adding or editing a file in
Caddyfile.d/, apply it withsudo ./scripts/deploy/start.sh --provision --cloud <digitalocean|vultr|aws> --caddy-only, which reloads Caddy when that directory changed (orsudo systemctl reload caddy). Keep the directory0755and its files0644: Caddy runs as thecaddyuser, and a file only root can read is refused before Caddy is touched.
--caddy-only remains available for deliberate reconfiguration (for example after changing PRIVATE_NETWORK_CIDRS); it applies the same listener check.
Upgrading an install that runs on SQLite. Trinity refuses to start until the data moves, and never comes up on an empty database. start.sh stops before pulling and prints the copy command; a directly-run compose stack shows the same guidance in docker compose logs backend, with the backend unhealthy. Follow docs/migrations/SQLITE_TO_POSTGRES.md. The copy never writes to your SQLite file. On a source-built install, build all four platform images (backend frontend mcp-server scheduler) before the copy: start.sh builds nothing, and a backend-only build leaves the scheduler, MCP server and frontend on the previous release while the stack still reports healthy. To roll the move back, get the previous release's images before you start it again (rebuild all four on a source install; restore the old TRINITY_IMAGE_TAG and pull on a hosted one), or the old compose file starts the newer backend image, which refuses SQLite. update.sh --rollback does that rebuild for you.
Upgrading a source-built install on a PostgreSQL container you started by hand. docker-compose.prod.yml now runs that service itself, so the file stops at a render error until you adopt it. The same guide covers it, and no data is copied.
On a Marketplace Droplet or a droplet made by the DigitalOcean install script, where the admin was created in the browser, .env carries a blank ADMIN_PASSWORD with ADMIN_PASSWORD_SOURCE=browser; the re-run honours that marker and never generates a password over the one you chose. Setting a real ADMIN_PASSWORD in .env before the re-run is the deliberate way to reset it.
Step 4: Restart Platform Services
Use docker compose restart, not down/up. docker compose down removes the trinity-agent-network, orphaning every running agent container — they keep running but lose their network.
# Development
docker compose restart backend frontend mcp-server scheduler
# Production
docker compose -f docker-compose.prod.yml restart backend frontend mcp-server schedulerServices restart in parallel. The backend typically takes 10–20 seconds to become healthy.
Step 5: Verify
Run the six-probe verification. All must pass before you declare the upgrade complete:
# 1. Backend
curl -s http://localhost:8000/health
# Expected: {"status":"healthy",...}
# 2. Scheduler
curl -s http://localhost:8001/health
# Expected: {"status":"healthy","active_schedules":N}
# 3. Frontend (HTTP 200)
curl -s -o /dev/null -w '%{http_code}' http://localhost
# Expected: 200
# 4. Redis
docker exec trinity-redis redis-cli ping
# Expected: PONG
# 5. MCP Server
curl -s http://localhost:8080/health
# Expected: 200 OK
# 6. Vector (log aggregation)
docker exec trinity-vector wget -q -O - http://localhost:8686/health
# Expected: non-empty responseProbe 2 as written needs the scheduler's port 8001 on the host, which no compose file publishes — run it from inside the container instead: docker exec trinity-scheduler curl -sf http://localhost:8001/health. If your web UI is remapped, probe 3 uses http://localhost:$FRONTEND_PORT. A 503 from probe 1 with a migrations block names a pending or failed migration — see Single Server → Database backend.
Confirm the new version is live. After the probes pass, check that the backend is actually running the build you just deployed:
curl -s http://localhost:8000/api/version
# Expected: {"version":"<version>","git_commit_short":"<sha>","git_branch":"...","build_date":"...","install_source":"..."}The git_commit_short, git_branch, git_commit_subject, and build_date fields report the code that is actually running, not merely what was baked in at build time — so a container running mounted source, or one that drifted from its image, is reported honestly rather than claiming the image's commit. If they read "unknown" on a source install, the image was built without the deploy script's build args — rebuild with scripts/deploy/start.sh to populate them. On a hosted install they describe the commit the release workflow built, which is what you want.
The response also carries edition (oss or enterprise) enterprise_features, and install_source (how the instance was provisioned — unknown on an ordinary install), so you can confirm after an upgrade that the entitlements you expect are actually registered.
The same metadata is visible in the UI via the version chip in the navigation bar (click it for the Build Info dialog) and in Settings → General → Build Info.
Note: JWT tokens are invalidated when the backend restarts. Users with active web UI sessions will need to log in again. MCP clients (Claude Code) will need to reconnect — run /mcp in your Claude Code session or restart the client.
Retention windows do not change on upgrade. Every install stores its retention windows (execution logs, health checks, soft-deleted agents, backups, …) as explicit per-install settings, written at the values already in force the first time a release boots without them. A new release therefore never widens or narrows what your instance keeps; change a window deliberately under Settings → Retention (see Monitoring).
Base Image Upgrade (if needed)
The base image bakes in more than its Dockerfile: the agent server (agent_server/), the guard hooks (hooks/) and startup.sh are all copied into it. Rebuild it whenever anything under docker/base-image/ changed in the release — scripts/deploy/update.sh (trinity update) does this for you:
./scripts/deploy/build-base-image.shOn a hosted install there is no separate step — start.sh --hosted pulls the release's base image and retags it as trinity-agent-base:latest on every run.
How agents pick it up. A rebuilt or re-pulled base image is adopted at an agent's next cold start— that is, the next time it goes from stopped to running. Trinity compares the running container's image against the current base image and recreates the container when they differ.
What this means in practice:
A running agent is never replaced out from under you. Drift is detected but not acted on until the agent is next started cold.
Restart the fleet (Operations) to roll the new image out: each agent goes through the normal lifecycle, so a drifted agent is recreated and an up-to-date one keeps its container.
The system agent (
trinity-system) follows the same rule and is deliberately never replaced while running. If its image goes stale you get an operator alert rather than a mid-operation container swap; it adopts the rebuild the next time it is started cold.Rebuilding the base image is what makes newly shipped agent-side changes available, and an agent receives them only once it has restarted onto the new image. Until then the agent accepts the new platform's requests and ignores what it does not understand, so an agent-side fix in the release — a security fix included — does not apply to it.
After every upgrade, restart the fleet(or stop and start each agent) so agents receive the release's agent-side changes.
Trinity tells you which agents are behind. Shortly after the backend starts, and every ten minutes after that, it compares each running agent with the installed base image and keeps one alert in the operator queue naming the agents still on an older image. The alert updates as agents are restarted and ends by itself when none are left. It never restarts an agent for you. Ephemeral agents are not listed.
The same alert also appears when the installed base image is itself older than the platform — the case where the update path did not rebuild it, so no agent looks out of date although all of them are. It names the fix: ./scripts/deploy/build-base-image.sh on a source install, ./scripts/deploy/start.sh --hosted on a hosted one, then a fleet restart. If the base image has no version label, or Docker cannot be read, nothing is reported as up to date: the backend log says the check could not run.
Agent restart policy (one time, after upgrading)
Agent containers are now created with Docker's unless-stopped restart policy, so a host reboot brings them back. Agents created before that release still carry Docker's default (no) and stay Exited after a reboot until they are recreated. The create and recreate paths are fixed, so this population can only shrink — but an active fleet heals itself only as agents happen to be recreated. Run this once after upgrading rather than waiting:
# Census — which agents are still on `no`
for c in $(docker ps -a --format '{{.Names}}' | grep '^agent-'); do
docker inspect -f '{{.Name}} {{.HostConfig.RestartPolicy.Name}}' "$c"
done
# One-shot fix — changes the policy without starting a stopped agent
docker ps -a --format '{{.Names}}' | grep '^agent-' | while read -r c; do
[ "$(docker inspect -f '{{.HostConfig.RestartPolicy.Name}}' "$c")" = "no" ] &&
docker update --restart unless-stopped "$c"
doneUse unless-stopped, never always: Trinity stops an agent through Docker's manual stop, which unless-stopped honours, so a quarantined or deliberately stopped agent stays down across a reboot. On the dev compose the backend, frontend and redis services adopt the same policy on the next docker compose up (which start.sh performs). Two consequences worth knowing: “restarting frequently” alerts for agents now fire (they never could before), and a docker compose down leaves every agent in a restart loop rather than merely orphaned — see Monitoring → Recovery Patterns. Full runbook: docs/migrations/AGENT_RESTART_POLICY_2026-09.md.
GitHub token out of agent remotes (automatic, then rotate)
Agents' git remotes no longer carry the GitHub token. Git asks a credential helper for it instead (see GitHub PAT Setup → How git gets the token). Existing agents need no manual step:
About 20 seconds after the backend starts, Trinity cleans the remotes of every running agent once. It cleans each agent again every time that agent starts.
Each pass installs the helper (including into agents on an older base image), checks that a token still resolves, and only then removes the token from the URL.
An agent whose only token was inside its remote URL gets that token moved to a private file the helper reads. If Trinity cannot place a replacement, it leaves the URL alone and files an operator-queue alert, A git remote still carries an embedded credential. Give that agent its own token on its Git tab and the next start finishes the job.
A pass that cannot read an agent's workspace — usually a workspace ownership or permissions problem — files a separate alert, Trinity could not check this agent's git remotes. Check that agent's remotes by hand.
Then rotate the platform GitHub token. Backups, log archives, and container logs from before the upgrade can still contain it. Rotation is mandatory if the cleanup reports a token in a committed .gitmodules file, because that token is already in the repository's history. Full runbook, including how to verify an agent: docs/migrations/GIT_REMOTE_TOKEN_SCRUB_2026-09.md.
Agent SSH ports move to loopback (on recreate)
Each agent's SSH port (2222–2262) is now published on 127.0.0.1 by default, not on every interface. The backend applies the binding when it creates or recreates an agent, so agents created before the upgrade keep the old all-interfaces binding until they are recreated. The SSH grant response says, for each agent, which host to use. To reach agents directly over a VPN instead, set AGENT_SSH_BIND_ADDRESS (and SSH_HOST) to that interface's IP before recreating them. See Hardening and the .env reference.
Rollback
If something goes wrong after the upgrade:
1. Stop platform services
# Development
docker compose stop backend frontend mcp-server scheduler
# Production
docker compose -f docker-compose.prod.yml stop backend frontend mcp-server scheduler
# Hosted
docker compose -f docker-compose.hosted.yml stop backend frontend mcp-server scheduler2. Restore the database backup
Remove any journal sidecars beside the target first — a leftover -wal/-shm/-journal file belongs to the old database and must not be replayed against the restored one:
# Development (named volume)
docker run --rm \
-v trinity_trinity-data:/data \
-v ~/backups:/backup \
alpine sh -c "rm -f /data/trinity.db-wal /data/trinity.db-shm /data/trinity.db-journal && \
cp /backup/trinity-<timestamp>.db /data/trinity.db"
# Production / hosted (bind mount — adjust path)
rm -f /srv/trinity-data/trinity.db-wal /srv/trinity-data/trinity.db-shm /srv/trinity-data/trinity.db-journal
cp ~/backups/trinity-<timestamp>.db /srv/trinity-data/trinity.db3. Check out the previous version
# Source install
git checkout <previous-sha>
# or
git checkout <previous-tag>
# Hosted install — the previous tag, in .env (last line wins) and in the checkout
echo 'TRINITY_IMAGE_TAG=v0.9.0' >> .env
git checkout v0.9.0A hosted install on the bundled PostgreSQL cannot roll back to v0.9.5 or earlier. Those releases' docker-compose.hosted.yml has no postgres service, so the backend would start without its database. The data stays in the trinity_postgres-data volume; roll forward again instead. An install that stayed on SQLite (DATABASE_URL= empty) rolls back normally. To restore a PostgreSQL backup, see Backup and Restore.
4. Rebuild and restart
# Source install
docker compose build --no-cache backend frontend mcp-server scheduler
docker compose restart backend frontend mcp-server scheduler
# Hosted install — re-pulls the previous images and starts the stack
./scripts/deploy/start.sh --hostedOn a Caddy-fronted install, rolling back to a release older than the one that added the PROXY-protocol listener leaves Caddy pointed at a port the older web image does not have, because older releases do not re-render Caddy on upgrade. Re-render it from the rolled-back checkout, which writes the ordinary listener: sudo ./scripts/deploy/start.sh --provision --cloud <digitalocean|aws|vultr> --caddy-only. If that checkout's start.sh has no --caddy-only, change each reverse_proxy in /etc/caddy/Caddyfile back to 127.0.0.1:8081 without a transport block, run caddy validate --config /etc/caddy/Caddyfile, then sudo systemctl restart caddy.
5. Run the six-probe verification to confirm rollback succeeded.
Automation
The Ops Agent's /update skill runs this procedure end to end: back up, pull, rebuild the four platform images, restart, verify, and flag a stale base image. On a hosted install (COMPOSE_FILE=docker-compose.hosted.yml) it builds nothing: it checks out the release tag matching the server's TRINITY_IMAGE_TAG and runs start.sh --hosted, so set the new tag in the server's .env first. It then reports which database the backend runs on, so you can confirm that did not change.