Advanced Features
Voice chat, outbound VoIP telephony, image generation, agent avatars, and agent-defined dynamic dashboards.
Voice Chat
Real-time voice conversations with agents via Gemini 2.5 Flash Native Audio model (~280ms latency). Audio streams bidirectionally through a backend WebSocket proxy.
Open an agent's Chat tab.
Click the microphone button.
A voice overlay appears with status, mute, and end controls.
Speak — audio is captured as PCM 16kHz and streamed to the backend WebSocket.
The backend proxies audio to the Google Gemini Live API.
Agent response audio (PCM 24kHz) plays back in real-time.
Transcripts are auto-saved to the chat session with source="voice" markers.
Requirement: GEMINI_API_KEY configured on the platform.
Configuration
| Variable | Description |
|---|---|
| VOICE_ENABLED | Enable or disable voice chat |
| VOICE_MODEL | Gemini model to use for voice |
| VOICE_MAX_DURATION | Maximum voice session duration |
Voice API
| Endpoint | Method | Description |
|---|---|---|
| /api/agents/{name}/voice/start | POST | Start a voice session |
| /api/agents/{name}/voice/stop | POST | Stop a voice session |
| /api/agents/{name}/voice/status | GET | Get session status |
| /ws/voice/{session_id} | WebSocket | Bidirectional audio bridge |
An opt-in Workspace Mode (BETA, admin-gated) adds a live canvas the agent can draw on — diagrams, images, and formatted text — while you talk. See the full Voice Chat guide.
Outbound Phone Calls (VoIP)
Agents can place real outbound phone calls. The agent dials a number through Twilio and holds a live, interruptible spoken conversation powered by Gemini Live; after you hang up, the transcript flows back to the agent so it can act on what was discussed.
This is distinct from Voice Chat: voice chat is you talking to your agent in the browser; VoIP is the agent calling a phone number over the public telephone network. This release is outbound only— agents place calls; they do not answer incoming ones.
VOIP_ENABLED=true and a GEMINI_API_KEY; off by default.call_user; rate-limited and daily-capped.Full setup, API reference, and limitations in the VoIP Telephony guide.
Image Generation
Platform image generation via a two-step Gemini pipeline: prompt refinement then image generation.
Submit an image generation request via API.
Prompt Refinement — Gemini refines the user's prompt using best-practice templates for the use case.
Image Generation — Gemini generates the image from the refined prompt. Returned as base64 or URL.
Used internally for agent avatars and other platform features. API: POST /api/image/generate
Agent Avatars
AI-generated avatars for agents using reference images, emotion variants, and default generation.
API: GET /api/agents/{name}/avatar (serve) and POST /api/agents/{name}/avatar (generate/upload).
Dynamic Dashboards
Agent-defined dashboards via dashboard.yaml with 11 widget types, historical tracking, and sparkline charts.
Widget Types
11 supported types: metric, status, progress, table, list, chart, text, badge, countdown, link, image.
How It Works
The agent writes a dashboard.yaml file to its workspace.
The file defines widgets with type, title, value, and optional configuration.
Open the agent detail page and select the Dashboard tab to see the widgets.
Auto-refresh updates values as the agent modifies the YAML file.
Historical values are tracked automatically — sparklines appear for metrics with enough data points. Trend indicators show up/down/stable arrows with percentage change.
A Platform Metrics section appears at the bottom of every dashboard, auto-injected with Tasks 24h, Success Rate, Cost, and Health. This section is not controlled by the YAML file.
Agents control their dashboard entirely by writing to dashboard.yaml. No API call is needed — the file is read on each dashboard request. API: GET /api/agents/{name}/dashboard.