Build a booking assistant: how to build a voice AI agent

What is the fastest way to build a voice AI agent for inbound calls?

The fastest way to learn how to build a voice AI agent for inbound calls in one afternoon is to pair a voice platform like Vapi or Retell AI with your own LLM backend and limit scope to a single workflow such as appointment booking.⁷¹²

Instead of chasing a perfect receptionist, you’ll ship a narrow, testable booking assistant that actually runs: it picks up real phone calls, speaks naturally, checks availability via your backend, and books or hands off.

---

How does a voice AI inbound agent actually work under the hood?

A production‑grade voice AI inbound agent is a small orchestration layer around four pieces: telephony, speech‑to‑text, an LLM brain, and text‑to‑speech, all wired into your workflow tools.⁷

Most serious stacks today look like this: phone lines via Twilio or RingCentral, STT from Deepgram or Azure, an LLM (OpenAI, Anthropic, or your own hosted model), and TTS from ElevenLabs or similar, glued together by a platform like Vapi or Retell AI.⁷¹²

Voice AI is not “one model that magically answers phones.” It is a real‑time pipeline:

• Caller speaks → audio captured by telephony.
• Audio sent to STT → text transcript.
• Text + conversation history → LLM decides what to say and what tools to call.
• LLM output → TTS → audio back to the caller.

Contact‑centre vendors layer on transcription, sentiment, summaries, and QA monitoring, but the core loop remains this STT → LLM → TTS pipeline.²¹⁰

Measured end‑to‑end latency for these stacks typically lands between 400 ms and 800 ms per turn, and once you creep over ~800 ms, call quality drops fast for inbound support.¹²

---

Vapi vs Retell AI: which should you use for a one‑afternoon build?

For a one‑afternoon build, use Retell AI if you want a fast, ops‑friendly inbound agent, and Vapi if you care more about deep control and custom tool calls.

Here’s the practical distinction based on 2026 comparisons and deployments:⁷¹²

Scenario	Retell AI	Vapi
Goal	Get a working phone agent today	Engineer‑controlled voice stack
Typical user	Ops, support, solo founder	Backend / platform engineer
Latency focus	Optimised for sub‑800 ms, latency‑critical support12	500–900 ms typical; requires tuning at scale12
Setup time
Minutes–hours with built‑in flows6	1–3 days realistic time‑to‑first‑call for competent engineer12
Stack wiring	Telephony, STT, LLM, TTS abstracted	You wire STT, LLM, TTS, telephony yourself712
Script iteration	Non‑engineers can edit prompts and flows directly612	Changes usually go through code deploys
Best for afternoon	Yes	Yes, but only if you’re happy with a rough prototype

For this tutorial, we’ll assume Retell AI for telephony + voice orchestration, and your own LLM API behind a simple HTTP endpoint.

---

What problem will this booking assistant actually solve?

A narrow voice AI booking assistant is designed to handle the 60–70% of inbound calls that follow a structured pattern: checking availability, collecting details, booking, and confirming.⁹¹⁴

Market data in 2026 is clear: voice AI agents are being rolled out for appointment booking, FAQs, and intake—not for complex, emotionally charged cases or disputes.⁹¹⁴

Done well, these inbound agents materially change unit economics. Case studies report:

• $0.40 per AI‑handled call vs $7–$12 for human agents, a ~90–95% cost reduction per interaction.³
• 82% of inbound calls fully resolved without human handoff in healthcare intake deployments.¹
• Booked‑while‑calling rate moving from 41% to 73%, and net‑new patient growth of 28% per month after rollout.¹

You are not trying to replace your team; you’re building a narrow agent that eats the repetitive calls and frees humans for edge cases.

---

What stack do you need for how to build a voice AI agent in one afternoon?

For an afternoon build, the minimal practical stack for how to build a voice AI agent for inbound calls is: a voice platform (Retell or Vapi), one LLM, one calendar/booking backend, and a single phone number.⁶⁷

You do not need a CCaaS platform or a complex CRM pipeline to ship v1. You need:

• Voice layer – Retell AI (or Vapi) to handle calls, STT, and TTS.⁷¹²
• LLM backend – a simple HTTP API that wraps your model of choice.
• Booking service – any scheduler or calendar API (Google Calendar, Calendly, in‑house).
• Phone number – provisioned via Retell’s telephony or Twilio, mapped to your agent.⁶¹²

Complex contact centres on Microsoft Teams layer in AI summaries, sentiment analysis, and quality monitoring, but that’s an optimisation path, not a prerequisite.²¹⁰

---

Step 1: Define the booking workflow before you touch any tools

The first step is to write down, in plain language, the exact booking flow your agent will own end‑to‑end.

Resist the urge to “make it smart.” A good v1 booking assistant does three things:

1. Greets the caller and identifies what they want.
2. Collects required booking data (name, phone, date/time, service type).
3. Checks availability and either books or cleanly hands off.

Use your own call transcripts or a quick chat with staff to list the 5–10 most common paths: new booking, reschedule, cancel, “quick question then booking.” Limit v1 scope to new bookings and reschedules.

This scope decision is where most builds fail; they try to cover every edge case, then never ship.

---

Step 2: Spin up your LLM booking backend

Your LLM backend is a single endpoint that receives the caller’s latest message plus conversation state, decides what to do, and calls your booking tools.

In practice:

• Expose a POST /voice-agent endpoint.
• Input: caller_id, current turn text, conversation history, and a small state object (e.g. booking_intent, collected_fields).
• Output: assistant_reply and optional actions (e.g. check_slots, create_booking).

You can run this behind any modern LLM. Many teams use OpenAI or Anthropic here, but 2026 contact‑centre guidance is increasingly pointing at custom LLMs so you can tune context, tools, and guardrails to your process.⁷¹⁰

Keep tools minimal: one function to fetch available slots and one to create a booking. Persist the conversation state in your own datastore; do not rely solely on the voice platform for business state.

---

Step 3: Configure the Retell (or Vapi) voice agent

Next, create the actual phone agent in your chosen voice platform and connect it to your backend.

In Retell‑style flows, this is typically:

1. Create a voice agent (single‑prompt or scripted).
2. Choose voice and language – pick a neutral, clear voice from the TTS catalogue (many use ElevenLabs on other stacks for this).¹⁷
3. Set the entry prompt that explains who the agent is, what it can do, and how it should behave.
4. Configure webhooks or HTTP tool calls so the agent can call your /voice-agent endpoint for business logic.

Your system prompt should include:

• Role: “You are a booking assistant for [business].”
• Scope: “You only handle new bookings and reschedules. For all other topics, transfer politely.”
• Data collection: fields needed, one at a time.
• Hand‑off rules: when to transfer to a human or send the caller to voicemail.

This is where you enforce humility: your agent should say “I’m not able to help with that; let me transfer you” instead of hallucinating policies.

---

Step 4: Connect a real phone number and test with live calls

Once the agent is configured, connect it to a real inbound phone number and make it your front door during a controlled window.

On Retell, you typically:

• Purchase or connect a phone number.
• Map that number to your voice agent.
• Deploy the current agent version to that line.⁶

Case studies suggest realistic time‑to‑first‑live‑call on Vapi‑style stacks is 1–3 days for a competent backend engineer, but Retell and similar platforms are explicitly designed to cut that path down to minutes‑hours for a simple prototype.¹²

Start with a short internal test: have 3–5 team members call in, try common scenarios, and capture their feedback. Only then point real customer traffic at it for a quiet half‑day.

---

How should you tune latency and call quality for v1?

You should set a simple latency budget and stick to it: aim for <800 ms end‑to‑end per turn, and treat anything above that as a bug.¹²

Voice practitioners running production stacks recommend breaking that down as:

• STT partials: <150 ms.
• LLM “think time”: <300 ms.
• TTS generation/playback: <200 ms.²

Platforms like Retell AI exist partly because raw STT+LLM+TTS stacks can drift to 500–900 ms and degrade above ~1,100 ms at high concurrency.¹²

Do at least three live tests:

• Talk over the agent to check barge‑in (the agent should stop speaking when you interrupt).
• Try bad phone lines to see how robust STT is.
• Test long pauses and silences so the agent doesn’t over‑apologise or hang up prematurely.

If you plan to move this into a real contact‑centre environment, you’ll eventually need proper QoS, bandwidth planning, and failover, as recommended in modern Microsoft Teams contact‑centre architectures.²

---

How do you measure whether this agent is worth keeping?

You measure success with a small set of call‑level metrics: resolution rate, booking conversion, cost per call, and complaint rate.

Recent deployments provide useful reference points:

• 82% of inbound calls fully resolved without human handoff for a healthcare intake agent.¹
• 12 minutes saved per call on CRM note entry, because reps no longer type up summaries.¹
• 99.4% pickup in week two after routing calls through an AI agent.¹

Combine those with cost data—$0.40 per AI call vs $7–$12 per human agent call—and you can quickly see if your booking assistant has a realistic payback period.³

One 2026 case study reported a median payback in 3.8 months for an intake‑style voice agent, driven by time savings and incremental bookings.¹

For your v1, keep it simple: track, week over week, the share of inbound calls where the agent successfully books or reschedules without human help, and the share that end in frustration.

---

What should you avoid when building your first voice AI agent?

For a first build, the main risks are over‑scoping, pretending AI can replace every human, and underestimating how much latency breaks the illusion of competence.

2026 contact‑centre analysis is explicit that voice AI is targeting the 60–70% of structured inbound calls, not multi‑system troubleshooting or emotionally charged complaints.⁹¹⁴

If you ask your booking assistant to handle billing disputes, complaints, and detailed technical support, you will hit every failure mode at once.

Instead:

• Start with a single workflow (book/reschedule).
• Give the agent explicit transfer rules for everything else.
• Iterate scripts weekly based on real transcripts, which Retell and similar platforms expose by default.⁶

You are building a workflow tool, not a colleague.

---

How do you extend this afternoon prototype into a serious contact-center asset?

You turn an afternoon prototype into a durable asset by layering standard contact‑centre capabilities around it: real‑time transcription, sentiment analysis, summaries, and automated QA.²

Teams running on Microsoft platforms commonly combine:

• AI virtual agents to answer and route calls.
• Real‑time transcription and sentiment analysis to monitor customer experience and catch issues early.²
• AI‑generated call summaries that feed CRM notes automatically.²
• Automated quality monitoring to track compliance and script adherence.²

Azure Communication Services and newer Azure.AI voice APIs provide programmable voice calling and custom voices that can integrate with your own LLM stack if you prefer to stay within a Teams‑based ecosystem.⁵⁸¹⁰

But you do not need any of that to prove the concept. In one afternoon, your job is simpler: show that a narrow voice AI booking assistant can pick up real calls, speak clearly, and reliably book appointments into your existing calendar.