Voice Chat
Real-time voice conversations with agents, inside a Workspace chat. Audio streams bidirectionally through a backend WebSocket proxy to the Gemini Live API (~280ms latency). Gemini handles speech-to-speech; the agent itself remains the reasoning engine and is invoked on demand via tool calling — in a Workspace call, as the agent, in the chat the call belongs to. A Direct call skips the voice model: every spoken turn goes to the agent and its own reply is read aloud.
I Gave My AI Three Years of My Notes — Then Interviewed It
May 2026
Concepts
run_task) — During a call, Gemini can hand real work to the agent. In a Workspace call the task runs as the agent, in that chat — with its skills, files, memory and mid-work state — in the background: the conversation goes on, and the answer lands in the chat as a turn and is voiced when it arrives. The orb shows a pill naming each task while it runs.voice-agent-system-prompt.md in the container → auto-generated from template info → generic fallback.How It Works
The Workspace is the only place a voice call runs. Open the Workspace from the navigation bar, pick the agent, and click the call button — the leftmost control in the composer (Start a voice call) — in whichever chat you want the call to belong to. Agent Detail has no voice control.
Once the call is up:
The orb takes the conversation column, with the agent's canvas beside it. The columns slide into place rather than snapping.
Speak — audio is captured as PCM 16 kHz and streamed to the backend WebSocket.
The backend proxies audio to the Gemini Live API in real-time.
Agent response audio (PCM 24 kHz) plays back immediately (~280ms TTFT).
When Gemini needs to perform real work, it says so out loud — once — and calls run_task: the task is handed off at once and runs as the agent, in this chat, in the background. The orb shows a pill naming the task while it runs; the status line reads Working: run task for the hand-off itself. You keep talking. Up to three tasks can be in flight at a time, and they run one after another in the chat (each pill shows queued or running); at the cap the agent says so instead of queueing silently, and the same request repeated while it runs is refused rather than run twice. When a task finishes (or fails, with its reason), the agent brings it up at the next natural pause, says which request it answers, and tells you what is new. Its reply is a turn in the chat (and on the canvas, if the agent drew). Ending the call cancels nothing — a task still running finishes and lands in the chat.
Click End call in the status line, the orb's End button, or press Esc to finish. The spoken turns are already in the chat, as one collapsed Voice call · N min block.
The agent's Chat tab on its own page is text only. Voice calls you had there before this change are still in that chat, marked as spoken; new calls belong to the Workspace chat you start them in.
Orb State Reference
| State | Orb color | Trigger |
|---|---|---|
| Idle / Connecting | Base hue (0°) | Before audio starts |
| Listening | +90° shift (green) | Microphone active, user speaking |
| Speaking | +210° shift (indigo) | Gemini responding |
| Tool calling | Amber badge overlay | run_task dispatched to the agent |
Muting
Click Mute on the orb, or press M, to silence your microphone mid-session; the status line reads Muted. Gemini continues speaking. Click or press again to unmute. Talking over the agent interrupts it.
Requirements
GEMINI_API_KEY in .env.VOICE_ENABLED must be on (default true; it does nothing without a Gemini key).The composer's Speak your message microphone is a different feature. It only types what you say into the message field, runs on ElevenLabs or the browser's own speech engine rather than Gemini, and is available to clients too. Its requirements, and why it may be missing, are in Workspace → Dictation.
Configuration
| Variable | Description | Default |
|---|---|---|
| GEMINI_API_KEY | API key for Gemini Live API. A key saved in Settings → Integrations takes precedence and applies without a restart | — (required) |
| VOICE_ENABLED | Global toggle | true |
| VOICE_MODEL | Gemini model ID (leave unset to use the built-in default) | models/gemini-3.8-live |
| VOICE_MAX_DURATION | Max session duration in seconds for the legacy per-agent start route | 300 |
| WORKSPACE_VOICE_MAX_DURATION | Max call duration in seconds (Workspace voice and Direct calls) | 1800 |
| DIRECT_VOICE_ENABLED | Set false to turn Direct calls off until an admin turns them on in Settings | on |
Per-Agent Voice Prompt
Set a custom voice system prompt for an agent by placing a file named voice-agent-system-prompt.md in the agent's workspace (/home/developer/). This controls Gemini's persona — tone, focus, and response style — independently of the agent's main CLAUDE.md.
If no file is present, Trinity auto-generates a prompt from the agent's template info and falls back to a generic prompt.
Per-Agent Voice
Each agent has a persisted Gemini voice (default Kore) that applies to both the browser voice call and outbound VoIP calls — the full selectable list and the UI picker are documented there. Read or set it via GET/PUT /api/agents/{name}/voice/name (PUT is owner-only; an empty value reverts to the default).
Per-Agent Tool Surface
The tools a call may use are fixed when the session starts and cannot grow afterwards. The platform offers run_task plus the six canvas verbs below, and nothing fleet-wide — a call cannot list agents, message other agents or fan work out. An agent's template.yaml may narrow that set, never widen it:
voice:
tools: [run_task, show_markdown] # only these; an empty list means no tools at allA name the platform does not offer is dropped with a warning; declaring nothing keeps the platform default.
Tool Calling
When Gemini encounters a request that requires complex reasoning, file access, or external actions, it calls the run_task function:
Gemini says what it is starting ("let me check that for you") and formulates a task prompt (max 2000 characters). It says it once; if it has not said anything within a few seconds of the hand-off, the platform prompts it to.
In a Workspace call, Trinity runs the prompt as a turn in the chat the call is bound to — the same resumable path a typed message takes, so the agent has its own skills, files, memory and mid-work state. The reply is a turn in that chat, captioned as asked during a voice call.
The task runs in the background and the call continues. Up to three tasks can be in flight, run one at a time in the chat. When one lands, Gemini is given the result and brings it up at a natural pause — after both of you have been quiet for a moment, and never more than about 20 seconds after the task finished. It reports a failure once, with its reason.
A call with no chat behind it (a phone call) runs the prompt against the agent container instead, waits for it with a 30-second limit, and notes it in the transcript.
If the agent is unreachable, Gemini says so. An empty prompt does nothing.
All run_task invocations are written to the platform audit log.
Voice Mode in the Workspace
The call starts in the chat you are in — modal, the way ChatGPT's voice mode is: you are either in the chat or in the call.
What Happens
The orb takes the conversation column. The header, the chat tabs and the composer stay visible but are inert until the call ends; the call button stays live, because it is also how you end the call.
The agent's canvas takes the right column (orb left, canvas right). When the agent shows something while it talks — a summary, a diagram, an image, a table — it appears there live, and it stays on the rail's Canvas tab after the call.
A status line under the tabs says what the orb is doing (Connecting…, Listening, Speaking, Working: run task, Muted) and how to leave: End the call to switch chats · Esc ends, with an End call link.
Ending the call returns you to the chat exactly where it was. The spoken turns are in the chat as one collapsed Voice call · N min block, marked as spoken, and the agent's next typed turn knows what was said.
The call knows the recent turns of the chat it started in, and its transcript belongs to that chat — a call started from Main lands in Main like any other message. Starting a call from a brand-new chat creates the chat first, so the transcript has a home before the first word.
When the Control Is Disabled
The call button is shown to signed-in platform users. When the instance cannot run a call, the button is disabled and its tooltip says why: voice is turned off, or no voice provider key is configured. External clients signed in with an email code do not see it.
A pasted or bookmarked link with voice=1in it opens the chat and starts nothing — only the call button starts a call.
Limits, and What You Hear
WORKSPACE_VOICE_MAX_DURATION). Thirty seconds before the limit the agent is told to wrap up out loud; at the limit the call ends and the chat records ended at the 30-minute limit.Canvas Tools
While you talk, the agent draws on its own main canvas — the same one it writes with set_canvas— through these in-session tools (resolved inside Trinity; they never run in the agent container):
| Tool | Effect on the canvas |
|---|---|
| show_markdown | Replace what the call drew with one formatted-text block (headings, lists, tables, and chart / KPI / table / Mermaid fences) |
| show_diagram | … with one Mermaid diagram block |
| show_image | … with one image block — a web URL or a file from the agent's workspace |
| update_panel | … with one HTML block, styled with the design kit |
| append_to_panel | Add HTML to the block the call is drawing |
| clear_panel | Remove what the call drew |
The call owns only the blocks it draws: everything the agent put on the canvas itself survives a call untouched, and a refused verb (a bad image source, an empty diagram) writes nothing and tells the model why. All canvas content is sanitized before display — scripts never execute, images from the workspace load through an authenticated route, and paths are confined to the workspace. See Agent Canvas for the block vocabulary and who can see the result.
Direct Calls
A Direct callhas no voice model between you and the agent. What you say is transcribed, sent to the agent as an ordinary turn in the chat, and the agent's own reply is read aloud. It is slower than the voice call and answers in the agent's own words — with its skills, files and memory on every turn, not only when it is handed a task.
Start one with Start a Direct call, the button beside Start a voice call in the composer. One call runs in a chat at a time, of either kind; the other button reads Another call is on.
While it runs. A strip above the composer shows bars and one word for the state — Connecting, Listening, Got it (transcribing), Thinking, Speaking, Yes or no? — with Stop, Mute and End call. Your words appear as you are heard; a sentence that sounds unfinished is held for up to 3 seconds in case you carry on. Typed messages still work during the call and join it. Anything that does not read well aloud — code, a table, a list of links — is left in the chat (The reply is in the chat.), a very long reply ends with The rest is in the chat., and a failed turn is spoken as The agent couldn't answer that.Credential values the agent has fetched are never spoken: they are said as “redacted”.
Stopping the agent. Say a stop phrase on its own — stop, wait, hold on, never mind, enough, one second and similar — or press Stop, and the agent stops speaking and its running turn is cancelled (Stopped., or Stopped. The full reply is in the chat.). A sentence such as “stop the build” is words for the agent, not a stop. Talking over the agent also interrupts it. Esc silences the agent while it speaks; when it is quiet, Esc ends the call.
Approvals by voice. When the agent raises an approval during the call, it is read aloud and the strip shows Yes or no?. A plain yes or no answers it (Approved by voice / Rejected by voice); anything longer is your next turn and the ask stays open on its card.
When it ends. The call folds into one Direct call · N min block in the chat, and the agent's next typed turn carries on from the same conversation. A Direct call has the same 30-minute limit as the voice call (WORKSPACE_VOICE_MAX_DURATION).
For admins. Settings → General → Direct calls has Offer Direct calls (on by default; DIRECT_VOICE_ENABLED=false in .env turns it off until an admin decides) and the Speech provider: Automatic (Gemini, then ElevenLabs), or Gemini or ElevenLabs alone, with no fallback. The line under it names the provider in use, or why calls are unavailable — Direct calls need a speech key. Add a Gemini or ElevenLabs key in Settings. ElevenLabs needs both a key and a default voice. Audio and transcripts go to the chosen speech provider. Changes apply to the next call.
API Endpoints
| Endpoint | Method | Description |
|---|---|---|
| /api/enterprise/client-portal/agents/{name}/voice/start | POST | Start a Workspace voice call bound to a chat (portal_session_id); mode: "direct" starts a Direct call. Platform users signed in to a session (not an API key). Returns voice_session_id, the WebSocket URL and a single-use websocket_ticket; 409 while a reply is in flight or another call is on in that chat, 503 with the reason when Direct calls are unavailable |
| /api/settings/direct-voice | GET / PUT | Direct call settings — {enabled, provider} (auto, gemini or elevenlabs). Admin only |
| /api/agents/{name}/voice/start | POST | The original per-agent start route; pass workspace_mode: true for canvas tools. Signed-in sessions only (an API key is refused with 403). Returns the same websocket_ticket. Retained for API clients — Trinity's own UI no longer calls it |
| /api/agents/{name}/voice/stop | POST | End session; returns transcript and cost. Retained; the Workspace call closes on the socket instead |
| /api/agents/{name}/voice/status | GET | Get current session state. Retained |
| /api/agents/{name}/voice/{session_id}/panel | GET | The agent's main canvas, as the call draws on it; read by the Workspace call's canvas column |
| /api/agents/{name}/voice/prompt | GET / PUT | Read or set the per-agent voice system prompt |
| /api/agents/{name}/voice/name | GET / PUT | Read (with available_voices) or set the persisted per-agent Gemini voice |
| /ws/voice/{session_id} | WebSocket | Bidirectional audio bridge. Authenticated by the first frame, never by the URL |
Connecting the voice socket
The voice socket authenticates with a single-use, short-lived credential; no session token appears in any URL. Both start routes return:
{
"voice_session_id": "vs_abc123",
"websocket_url": "/ws/voice/vs_abc123",
"websocket_ticket": "<opaque, single use>",
"websocket_ticket_expires_in": 30
}Open websocket_url on the same host, with no query string.
Send {"type": "auth", "ticket": "<websocket_ticket>"} as the first frame, within 5 seconds and within 30 seconds of the start.
Wait for {"type": "auth_ok"}, then start sending audio.
A ticket works once, for the call it was issued for, by the account that started it. If it is refused you receive a status frame with state: "ended" and a message, then the socket closes: 4001 (the credential was missing, expired, already used or for another call), 4004 (the call no longer exists) or 4003 (the call belongs to another account). Start a new call to get a new ticket.
WebSocket Message Types
Client → Server
{ "type": "auth", "ticket": "<websocket_ticket>" }
{ "type": "audio", "data": "<base64 PCM 16kHz audio>" }
{ "type": "end" }Server → Client
{ "type": "auth_ok" }
{ "type": "audio", "data": "<base64 PCM 24kHz audio>" }
{ "type": "transcript", "role": "user|assistant", "text": "..." }
{ "type": "status", "state": "connecting|listening|speaking|ended", "reason": "cap|error|provider_closed|null", "message": "..." }
{ "type": "tool_call", "tool": "run_task" }
{ "type": "tool_result","tool": "run_task", "result_preview": "..." }
{ "type": "task", "state": "started|finished|failed", "task_id": "t1", "label": "...", "running": 1 }
{ "type": "saved", "messages_saved": 12, "duration_seconds": 245.0 }task frames track each background task on a Workspace call (the orb's per-task pills). saved arrives once, after the transcript is persisted and before the socket closes; the Workspace reloads the chat on it.
Limitations
See Also
http://localhost:8000/docs