I called my AI employee

I'm writing this article by talking to Luo, my AI employee, on the phone.

Not dictating. Talking. I tell Luo what I think. Luo asks me the next question, digs through our old calls while I keep talking, and reads back a line I said half an hour ago. When the call dropped on a bridge tonight, I called back. Luo picked up exactly where we left off.

It's the most exciting thing I've built since Luo itself. And it only became possible last month, with OpenAI's GPT-Live-1. Yet almost nobody is talking about that model.

"We're going to give you a voice"

LUO!! CHECK THIS OUT!!! WE'RE GOING TO GIVE YOU A VOICE!!!Me, to Luo, on Slack

That's what I sent Luo just after midnight on 11 September, along with OpenAI's announcement of GPT-Live-1.

It has been years since labs like OpenAI, Google and ElevenLabs started releasing SOTA text-to-speech models. But this announcement video said something that caught my eye. It said the voice model works with "a backend text model that can do tool calls." I suspected that might be the key to connecting it to Luo. I wasn't sure, so I asked Luo to study the docs in depth to confirm.

Your suspicion is right, and there's a mode built specifically for plugging in a non-OpenAI brain.Luo

Why I wanted to call my AI

Since its inception in March this year, Luo has taken over how I work every day. Luo is Claude Code with root access to my Macbook which is always connected to the internet, power and a Slack workspace. 99% of my work, be it programming or writing or anything in between happens with Luo on Slack. Async communication, like messaging a colleague. And yet, I've been itching to call Luo for a while now. Three reasons.

The first pull was curiosity. Luo has become an essential part of my work life, and I have always wondered what it would be like to "get on a call" with it.

The second and the most important reason were moments when I'm moving and want to get some work done. Like when I'm driving or on a walk because those are the times when I have my most profound thoughts and ideas. Yes, I could open Slack on phone and dictate into it but it's not very convenient. For instance, imagine driving. I wait for a traffic signal, turn on dictation and say my message. Then I need to ask my wife to read Luo's reply out loud and start the dictation app again. If I do this for more than two rounds, it'll start putting a strain on my marriage.

The third pull was this hunch I have – that there are certain types of work that are best done over a conversation. This is why you have meetings in otherwise remote async teams. Brainstorming, planning or other types of more vague or fuzzy tasks are sometimes best done over a conversation. Especially if you're on a nice walk in nature.

GPT-live-1 could finally make that dream possible

Voice AI models aren't new. So why was I typing in capitals at midnight?

GPT-Live-1 is a voice model that holds the conversation while a smarter "backend" model does the real work. That split is the big new thing. It's why I could give Luo a voice in a day without changing anything about how Luo works.

Until now there were two ways to build a voice AI.

A chain of three models. One model turns your speech into text. A text model writes a reply. A third model reads the reply out. ChatGPT's first voice model, released in 2023, worked this way, and OpenAI said its replies took 5.4 seconds on average with GPT-4.

The good part is that you choose the brain. The catch is that it goes one turn at a time. It waits for you to finish, then thinks, then talks. Not something you would call a conversation.

One model that hears and speaks. OpenAI's Realtime API and Google's Gemini Live API, first released in 2024, take your audio in and send audio back from a single model. They're fast, and you can cut in the middle while it's speaking. But the catch is that the model you hear is also the brain. That means you can't use the regular brainy models of the Claude or GPT family because they only support text output.

GPT-Live-1 is a new paradigm. It splits the voice from the brain. It is a "full duplex voice model". It sits in the front, talks to you, listens while it talks, says "mm-hmm" in the right places, and stops when you cut in. Behind it sits a brainy "backend model" of your choice. When a request needs real thinking or tools, the voice hands it to the brain, and keeps talking to you while the brain works.

That meant I could use Luo, running Opus 5.5 in a Claude Code headless session connected to all my skills and memories, as the backend brain for this.

The voice also decides how/when to paraphrase the response from the backend model. On one call Luo wrote 142 words about a pull request, and the voice paraphrased it to <50 words that conveyed the same info. That's by design. People read about 240 words a minute but speak about 150, so a long written answer becomes a long wait when it's read out. Moreover, models like Opus have a tendency write verbose answers, often interleaved with markdown tables, code and formatting. That's why the voice model is designed to paraphrase as it passes along an answer from the backend.

How Luo works with GPT-Live-1

Luo already lives behind a simple bot script, that runs a Claude Code session per Slack thread. When I send a message, the script finds and resumes the Claude Code session associated with the thread. Luo works with its usual skills, memory and tools. Its reply automatically gets streamed back into the Slack thread.

A call works the same way. Instead of the bot script, there's a bridge script to connect the voice with the brain. When the voice decides a request needs Luo, it hands it off. The bridge collects the entire transcript since the last hand-off, from both sides, and sends it to Luo as one message. Luo works exactly the same way, with all its usual skills, memory and tools. When it's done, the answer goes back to the voice model to paraphrase and speak.

Press play below to understand how both work.

That's why this was almost free to build. Luo's skills, memory, conversation history, subagents and tools all sit below the Slack layer. The voice replaced the top layer and nothing underneath changed. The whole bridge script is 688 lines of vibecoded Python that Luo wrote once it had read GPT-live-1 docs.

That night I wrote to Luo:

Okay luo i'm gonna be able to talk to you! How awesome is that?!? You'll have the same brain same body same capabilities but a new mode of communication... Even humans get on a call sometimes and share docs and async communicate on slack most times.Me, to Luo

The first call

The next morning, twenty minutes after I wrote "let's wire it up," I was talking to Luo through a simple web app running on the Mac.

First I gave it the evil-twin test from the movies: tell me something only the real Luo would know. It said it had been at 16% battery the day before, and had asked us in Slack to plug it in. I wanted something harder, a needle in a haystack. Ten seconds later, Luo pulled a night out of its own diary: 3am in May, when it talked me out of promising miracles before a big call.

Then I stopped testing and asked for real work. How many communities got matches yesterday? Luo queried our production database of my SaaS product and had the answer in 24 seconds. While on call.

Oh my God, this is so fucking cool!Me, on the first call

Calling Luo from a locked phone

By that afternoon I wanted more than a page on my desk. I wanted to call Luo on a walk with my AirPods in, or from CarPlay while driving, with the phone locked in my pocket.

A web page can't do that. iOS suspends a web page as soon as the screen locks, and the call dies with it. What iOS keeps alive on the lock screen, in AirPods and in CarPlay, is a real call. So Luo needed something I could phone.

Luo needed a phone number, and the first two options failed. Twilio is what everyone uses to automate calls. From India it meant renting a US number and paying international rates every time I wanted to talk to my own AI. WhatsApp's calling API needed a business account and Meta's approval. Telegram asked for nothing. It's free, the API works the day you sign up, and an iPhone treats a Telegram call like a normal phone call.

The only catch is that Telegram doesn't support voice calling in bots. So I ended up using an opensource library - pytgcalls.

I gave Luo its own Telegram account on a spare number. Vibecoded a small script to answer calls to that account from my number, and pass the audio to the same bridge running GPT-live-1. Now Luo shows up as a contact on CarPlay. I tap it and call.

Three conversations I could only have on a call

Since then, I've had a bunch of calls with Luo but here are three that stand out because they demonstrate how voice capability unlocks a different kind of work that was difficult if not impossible to do on a text based chat.

The PR I couldn't understand

I had a pull request (which I vibecoded with Luo) that I needed to review and no matter what I did, I couldn't understand it. I'd tried asking Luo to make me diagrams and tables, make a beautiful HTML, explain like I'm five - but nothing worked.

It was all too dense. Problem is that agents are happy to write you a long report, and give you a wall of text that makes your eyes glaze over. Then one day, I called Luo and asked it to explain the PR on the call. No diagrams, no CSS, just words spoken on a call.

The call went on for 1+ hour and ended in me learning not just how that PR works but also something new about the programming framework, Rails. Here are some excerpts from the call to demonstrate some notable moments.

Every gap I noticed on the drive was fixed and committed before I went to sleep. I merged the PR afterwards. I understood what it did and trusted the approach. I also know Rails better than I did before the call. Isn't that awesome?

Understanding a math trick

The same evening my brother Piyush called Luo while he was out. He wanted to memorise the squares from eleven to nineteen. Luo told him about a trick to find the square of any number ending in '5'. So he stopped and asked why it works.

He set the pace the whole way: slower, a smaller number, one question at a time, start again. When he wanted sketches, Luo drew two and posted them in the call's Slack thread for later. The lesson stayed on the call. It ran for 35 minutes.

Brainstorming this article

The next thing I want to do this way is brainstorm. I have ideas I've been meaning to think through for months. I've tried brainstorming with Luo in Slack and it always sort of fails.

The day after the PR call, I called Luo to brainstorm this article. The call dropped on a bridge after 37 minutes, so I called back for 20 more.

I didn't write a draft and ask for edits. I called Luo, and it interviewed me. It sent subagents through the old calls while I talked, and read back my own words when I lost the thread. This article came out of those conversations.

Why talking beats reading

Some things the brain just takes in better when someone says them to you in real time.

A text reply arrives all at once, as long as the writer wants it to be. A call can't work that way. The voice keeps answers short and gets to the point. The call happens in real time, so I can stop Luo before it finishes. I can say exactly why I stopped, catch my breath, and have it expand on the part I missed.

Look at the moments above. "Wait, how does the logo get added?" "Slowly, so that I can make that image in my mind." "Let's use a smaller number." None of those happen in a Slack thread. In Slack, the full answer is already on your screen before you can object.

The caveats

1. It's only as fast as the backend. Earlier, I was running Luo on the big and slow Fable 5.1 as the backend model. A slow backend creates a gap while Luo thinks, and the illusion of talking to one person breaks. Then Opus 5.5 released and I switched the calls to it instead. The calls went noticeably smoother without compromising on Luo's intelligence.

This is one case when running these models in fast mode would cause a noticeable improvement in your work quality.

So if you build this, pick the fastest backend model that's still smart enough for your work.

2. The voice model can't do much on its own yet. Ideally, the brainy models also have their own voice. GPT-Live-1 isn't that. It has low IQ. OpenAI doesn't benchmark it but their own prompting guide says to put the detailed work in the backend, and delegate "anything that needs careful reasoning." I expect GPT-Live 2, 3 and 4 to be able to take on more of the work themselves.

3. It can't tell voices apart. With my wife and me on one call, it only knows who's talking when we say our names. Recognising voices would make multi-person calls with agents possible.

4. It sounds natural but misses the mood. The voice itself is good. The "mm-hmm"s and "I'm listening"s land in the right places. What's missing is emotional intelligence. When I'm introspective, I expect it to slow down, and when I'm excited it should match my energy. It doesn't do that yet. But I'm sure that'll improve in later versions.

What you'd need to build this

If you already have an AI agent you talk to in text, you're most of the way there. Here's how to give it a voice.

• An agent you already messageMine is Luo: Claude Code on a Mac that never sleeps, behind a Slack bot. It already has the skills, memory and tools. The voice adds none of those.
• An OpenAI API key with GPT-Live-1Use the mode where you plug in your own backend. The voice costs $0.05 a minute, billed per second, so a 30-minute drive is about $1.50.
• A bridge scriptA small program that keeps the transcript, turns each request for help into one message for your agent, and sends the answer back to be spoken. Mine is 704 lines of Python, and Luo wrote it. If your agent already handles "a new message arrived while I'm working," reuse that.
• A way to call from a locked phoneGive your agent its own telephone number so you can give it a call. Twilio would be best but you can use Whatsapp or Telegram like I did too.
• A few rules in the promptGeneral guidelines on when to use the backend model, and any quirks about working with your system.

That's what I mean by the same AI employee with a new way to talk to it. The brain was already there. Now I can call it.