Project 02 / 04 · Voice AI · Python · Whisper
Alfred, a voice AI agent
A hands-free voice assistant. I say 'Alfred', ask in English or Urdu, and he answers out loud while Claude Code does the work.
- Voice AI
- Whisper
- Claude Code
- ElevenLabs
- Python
(01) Why I built it
Why I built it
Typing is the slowest part of working with AI. I wanted to talk to my office the way you talk to a person across the room, including in Urdu, and hear the answer back without touching the keyboard.
(02) How it works
How it works
- 01
Wake word, tuned on my voice
Alfred listens for his name with a small, fast English Whisper model running locally. I recorded my own voice samples to tune the wake check, including the ways Whisper mishears 'Alfred'.
- 02
Understanding the request
Once he's awake, a larger multilingual Whisper model transcribes the request, so I can speak English or Urdu. A short pause ends the sentence.
- 03
Thinking with Claude Code
The request goes to Claude Code running in my HQ folder, so Alfred has the same context, files and tools as the rest of the office.
- 04
Speaking back
Replies are spoken through ElevenLabs' fastest voice model, with a free neural voice as the fallback. After answering, he keeps listening for eight seconds so the conversation can flow. 'Thank you' ends it and 'Alfred, go to sleep' pauses him.
- 05
A face for the voice
The desktop app shows Alfred as an animated character with idle, listening, thinking, talking and sleeping states, next to a conversation panel where I can also type, attach files or paste screenshots.
Stack · Python · faster-whisper · sounddevice · ElevenLabs · Edge neural voices · Claude Code
(03) Screens
Screens
What it taught me
What it taught me
Latency decides whether a voice agent feels alive. Splitting the job into a small model for the wake word and a bigger one for the request, and choosing the fastest speech model, mattered more than any single feature.
Next project
An AI driver for a racing game