The 800ms budget a spoken answer has to fit into

Tel-Agent has not answered a phone call yet. It is at Milestone 0 of 12, and Milestone 0 is exactly one thing: a number rings, a script answers, speaks, listens, replies, takes a message, and prints a transcript. Everything below is the design we are building toward and the arithmetic that constrains it — not a description of something you can dial today.

The arithmetic is the interesting part, because it is the thing that decides the architecture before anyone gets to have an opinion about it.

What 800 milliseconds actually is

The specification sets one hard number: under 800ms from the end of caller speech to the first audio out. Not to the complete answer — to the first sound. That figure is not chosen for elegance. It is roughly the gap after which a person on a phone line concludes the other end has not understood them, and starts repeating themselves.

You have felt this. It is the pause where you say "hello?" into a silence that turns out to have been someone thinking. On a human call you forgive it. From a machine you do not, because the silence is indistinguishable from a fault, and a caller who suspects a fault hangs up.

So 800ms is not a performance target in the usual sense. Missing it does not make the system slow. It makes the system read as broken, which is a different and much worse failure.

The budget, line by line

Here is the whole of it, as the spec allocates it:

Two things stand out once it is written down.

The first is that the model is not the expensive part. 250ms of 800 is under a third, and it is the only line item that most people think about. The other 550ms are spent on knowing when to start, agreeing what was said, making a noise, and moving bytes.

The second is endpointing — the largest single line, and the one with no technology to buy. Endpointing is the decision that a caller has finished their sentence rather than paused inside it. Wait too long and you have burned a quarter of the budget doing nothing. Cut it short and you interrupt people, which is worse than being slow, because being slow reads as thoughtful and interrupting reads as rude. There is no setting that is right for both a person reciting a policy number and a person deciding what they want.

That single line is most of why building an AI receptionist is harder than it looks from the outside. The language model was never the hard part.

Why the first version rents most of the budget

The design that hits 800ms first uses LiveKit Agents for the SIP side and turn-taking, a hosted speech-to-text service, a hosted model, and a hosted voice. That is a list of other people's servers, and it sits awkwardly beside what this project says it is for: your machine, your keys, nobody in the middle.

It is worth being straight about that rather than quiet.

The reason is the budget above. A hosted provider gives you 100ms speech-to-text because it has already solved streaming, warm capacity, and a network path that is short from wherever the caller is. Standing all of that up yourself, on one machine, is a real piece of work — and doing it first, before there is any evidence the whole loop closes, means spending weeks on latency for a call that has never been answered.

So Milestone 0 borrows the fast parts to prove the loop, and pays for it with a dependency it intends to shed. What makes that a plan rather than an excuse is what Milestone 1 is.

What Milestone 1 changes

Milestone 1 is not a feature. It is three interfaces: one speech-to-text, one language model, one text-to-speech, each behind a boundary the rest of the system does not see through. The hosted providers become the first implementation of those interfaces rather than the architecture itself.

The requirement that gives this away is a small one. Milestone 1 mandates cancel() on the text-to-speech interface — the ability to stop audio mid-syllable the instant the caller starts talking over it. That method exists for barge-in, but its presence in the interface rather than in one provider's client is the actual statement: the system is being built to swap what is underneath it, and anything that cannot be cancelled cannot be plugged in.

Once those three seams exist, "self-hosted" stops being a promise about the future and becomes a configuration. A self-hosted voice AI agent is not a different product from the one that calls out to a vendor; it is the same product with different implementations behind the same three interfaces, and a worse latency budget you have chosen on purpose because the recordings never leave your building.

That is the whole trade, stated plainly: hosted buys you milliseconds, and self-hosted buys you custody. The interfaces are what let you pick per deployment instead of picking once, in public, forever.

What this means if you are reading the code

Nothing in the repository answers a call yet, so there is not much to run. There is something to read, though, and the two things worth reading first are the latency table in the specification — it is the constraint every later decision gets measured against — and the milestone list, which is short enough to hold in your head and honest about the order.

The full architecture, the provider boundaries and the numbers above are in the specification. If you want the shorter version of where things stand, the first post on this blog says what exists today and what does not.

The project is AGPL-3.0 and built at Dpro GmbH. The way to know the day the first call is answered is to watch the repository — that is the only announcement there will be, because there is nothing to sign up for and nothing to buy.