Intelligence Labs · What we measure
Nothing on this page comes from someone else's benchmark. Each row says how it was measured and on which date. Small samples are marked as small.
Voice AI · 26 Oct 2026
Four ways to run a spoken advisor, asked the same five questions; then three pipelines on ten questions on 1 Nov. OpenAI Realtime mini answered first both times (1.2 s, then 1.15 s) and is the only one under 1.5 s, so the app now runs on it at about ₹2.6 a speech-minute.
How: Same 5 recorded questions (3 English, 2 Hindi) streamed at real-time speed from our Mac; end of speech to first audio byte.
| Provider | First sound, median | p90 | Stops when you talk over it | ₹ a minute of speech | ₹ a clock minute | How | Measured |
|---|---|---|---|---|---|---|---|
| OpenAI Realtime mini | 1.2 s | 1.4 s | 0.31 s | ≈ ₹1.9 | ≈ ₹1.2 | ₹ from the token bill of a 5-turn session at ₹95.30/$ and OpenAI's price list (see the note below the table) | |
| OpenAI Realtime (full) | 1 s | 1.1 s | – | ≈ ₹8 | ≈ ₹5.3 | ₹ from the token bill at ₹95.30/$ and OpenAI's price list (see the note below the table) | |
| ElevenLabs Agents + GPT-4o mini | 1.3 s | 1.8 s | 0.51 s | ≈ ₹17 | ≈ ₹7.6 + the model | ₹ from ElevenLabs' price ($0.08 a clock minute, elevenlabs.io/pricing/agents, read 29 Oct 2026) at ₹95.30/$, plus the model | |
| ElevenLabs Agents + Claude Sonnet | 2.4 s | 3.3 s | 0.51 s | ≈ ₹17 | ≈ ₹7.6 + the model | ₹ from ElevenLabs' price ($0.08 a clock minute, elevenlabs.io/pricing/agents, read 29 Oct 2026) at ₹95.30/$, plus the model | |
| Our live build: Sarvam + Claude | 4.9 s | 6.4 s | ≤ 0.5 s (by design, not re-measured) | ≈ ₹3 | ≈ ₹2 | ₹ from our own bill (voice is 74 % of it) | |
| Simli face (old setup) | 9.5 s | 10.8 s | 0.84 s | – | – | ₹ left out: the voice and model prices behind the old figure could not be rebuilt from their price pages (see the note below the table) | |
| OpenAI Realtime mini, 10 turns (what /intelligence/app/ now runs) | 1.15 s | 1.26 s | 0.31 s (26 Oct, not re-measured) | ≈ ₹2.6 | ≈ ₹2.2 | Same 10 recorded questions (7 English, 3 Hindi, bulbul voices) streamed at real-time speed from our Mac over a WebSocket; end of speech to the first output_audio.delta; ₹ from the token bill of the 10-turn session (₹6.85 for 2.7 minutes of speech) at ₹95.30/$ and OpenAI's price list; clock minute = speech plus the measured pauses (scratch/intel-1/measure-openai.js) | |
| Sarvam hears → Claude Haiku → Sarvam voice | 3.25 s | 4.07 s | ≤ 0.5 s (by design, not re-measured) | ≈ ₹2.2 | ≈ ₹2.0 | Same 10 recorded questions, in-process from our Mac: Sarvam saarika (REST) hears the WAV → Claude Haiku streams → the first sentence (the live cutter) goes to Sarvam bulbul (REST); first sound = the first WAV byte back; the Daily Brief lookup raced against 1 s as the live function does; ₹ from the character and token counts at our price list, ₹95.30/$ (scratch/intel-1/measure-chain.js) | |
| Browser hears (on the phone, ₹0) → Claude Haiku → Sarvam voice | 2.84 s + the browser's hearing | 3.75 s + the browser's hearing | ≤ 0.5 s (by design, not re-measured) | ≈ ₹2.1 | ≈ ₹1.9 | The same run as the row above with the hearing left out: the words are known on the device, so first sound = Haiku + bulbul only; the browser's own speech-to-text was NOT measured — Google Chrome refused a fake-microphone file twice (no-speech, 16 and 48 kHz), so its delay is still to be read on a phone |
Outside prices, all read on 29 Oct 2026. OpenAI, developers.openai.com/api/docs/pricing: Realtime mini $10 per 1M audio tokens in and $20 out; gpt-realtime $32 in and $64 out. ElevenLabs, elevenlabs.io/pricing/agents: $0.08 a minute of agent call, the model billed on top. Simli, simli.com/pricing: Hobby plan $0.01 a minute. ElevenLabs voice, elevenlabs.io/pricing/api: Flash/Turbo $0.04 per 1,000 characters. Rate: ₹95.30 per $ (open.er-api.com, 29 Oct 2026). The 26 Oct run used ₹88/$, a rate we cannot source, so every ₹ above was recomputed at today's rate from the same token counts. The Simli row has no ₹ figure because the voice and model prices behind the old range could not be rebuilt from these pages.
Not proven yet: all of it was run from a Mac with recordings, not from a phone; a 4G hop adds about 0.3–0.5 s to every number. Simli's figure is one session on its free tier. Total cost of measuring: about ₹25.
1 Nov 2026, 10 turns each, the same 10 questions (INTEL-1). OpenAI Realtime mini again answered first, 1.15 s median, and is the only one under 1.5 s, so /intelligence/app/ now runs on it. The Sarvam + Claude Haiku chain spends its time in three places: the Daily Brief lookup (raced at 1 s), Haiku's first sentence (≈ 0.7–0.9 s) and Sarvam's first WAV byte (1.25 s median, REST — Sarvam's docs index lists no streaming voice endpoint on 1 Nov). Per speech-minute the three cost about the same (₹2.1–2.6); per turn the Realtime model was cheapest (₹0.69 against ₹1.4) because its answers ran 9 s and Haiku's 32 s. Cost of this round: about ₹24 (OpenAI ₹6.85, Sarvam ≈ ₹17 of which ₹3 were the recordings, Haiku ₹0.27). Still not from a phone.
Claude Haiku vs Sonnet · 29 Oct 2026
Real inputs from the last 7 days of our own work, the same input to both. Haiku costs about a twentieth as much a call. It met the bar on Kriyā's answers; on cite-gap drafts it scored higher, but on 7 inputs, so that one is not decided.
How: Same input to both models; Claude Sonnet grades each answer against the source, order hidden; 1–5 on facts, numbers, plain words, length.
| Job | Inputs | Haiku | Sonnet | Difference ± error | $ a call, Haiku / Sonnet | We use | How | Measured |
|---|---|---|---|---|---|---|---|---|
| Story write-ups | 20 | 3.84 | 4.19 | -0.35 ± 0.11 | $0.0003 / $0.0053 | Sonnet | 20 real inputs from the 7 days to 29 Oct, one try each | |
| Kriyā answers | 20 | 3.78 | 3.51 | +0.26 ± 0.18 | $0.0006 / $0.0125 | Haiku | 20 real inputs from the 7 days to 29 Oct, one try each | |
| Tool price extraction | 20 | 3.63 | 3.76 | -0.14 ± 0.18 | $0.0004 / $0.0068 | Sonnet | 20 real inputs from the 7 days to 29 Oct, one try each | |
| Cite-gap drafts | 7 | 3.57 | 3.32 | +0.25 ± 0.17 | $0.0013 / $0.0312 | Sonnet (not decided, few inputs) | 7 real inputs from the 7 days to 29 Oct, one try each |
Limits: Sonnet grades its own answers, which may favour it. 20 inputs a job (7 for cite-gap). One try each. Spend: $1.75 in 226 calls.
Daily video · since 21 Oct 2026
Each morning the run checks its own work before anything is posted: one reel per story, cut where the words are spoken. The gate started on 21 Oct; 7 of 9 days passed.
How: The gate verdict the morning run writes into video/<date>/manifest.json (every story has a reel, bounds from speech, carousel present).
| Week | Days checked | Days passed | Reels made | How | Measured |
|---|---|---|---|---|---|
| Week of 19 Oct | 5 | 5 of 5 | 45 of 45 | 5 manifests with a gate verdict | |
| Week of 26 Oct | 4 | 2 of 4 | 29 of 36 | 4 manifests with a gate verdict |
Red bars are days the gate was short. Days before 21 Oct have no gate record, so they are not counted.
The daily run · 3 Sept – 29 Oct 2026
The brief is written, then voiced; the episode file lands around 09:20 IST. The video build after it went from 47 minutes (week of 7 Sep) to about 21.
How: Episode ready = Last-Modified of the day's episode MP3 on Spaces, in IST; video build = gate time (or manifest time) minus that, rebuilds hours later left out.
| Week | Days | Episode ready, median (IST) | Video build, median | Days in the median | How | Measured |
|---|---|---|---|---|---|---|
| Week of 31 Aug | 4 | 09:22 | 83 min | 1 | 4 manifests and episode files | |
| Week of 7 Sept | 7 | 09:20 | 47 min | 5 | 7 manifests and episode files | |
| Week of 14 Sept | 7 | 09:20 | 21 min | 5 | 7 manifests and episode files | |
| Week of 21 Sept | 7 | 09:20 | 22 min | 6 | 7 manifests and episode files | |
| Week of 28 Sept | 7 | 09:20 | 24 min | 6 | 7 manifests and episode files | |
| Week of 5 Oct | 7 | 09:20 | 24 min | 6 | 7 manifests and episode files | |
| Week of 12 Oct | 7 | 09:20 | 24 min | 6 | 7 manifests and episode files | |
| Week of 19 Oct | 7 | 09:20 | 21 min | 6 | 7 manifests and episode files | |
| Week of 26 Oct | 4 | 09:20 | 21 min | 4 | 4 manifests and episode files |
What this does not show: how long writing the brief itself takes. That start time is not in a public file yet; only the finish (the episode file) is. The first week has one countable day.