Build a little desk buddy you can talk to
Wire an ESP32, a microphone and a tiny screen to Claude, then just talk. This guide takes you from a bag of parts to a working Buddy, one checked step at a time.
- Time
- About 2 hours
- Experience
- None needed
- Made on
- A Mac, with Windows notes
- GND
- VCC
- SCL
- SDA
This is Buddy's real screen layout with sample replies. Pick a sentence to watch one turn from start to finish.
Contents
How Buddy works
Claude understands text, not sound, and an ESP32 is far too small to understand speech by itself. So the work is shared between your desk, your computer and the cloud.
- DeskYou speak. The INMP441 microphone turns your voice into a stream of numbers.
- DeskThe ESP32 listens all the time. When it notices a sentence start and then stop, it sends just that sentence over Wi-Fi to your computer.
- ComputerWhisper writes it down. Whisper is a free speech-recognition model that runs on your own computer, so your voice is not uploaded for this step.
- CloudClaude replies. Your computer sends the written sentence to Claude and gets a short answer back.
- ComputerEdge TTS gives the answer a voice. A free text-to-speech service turns the reply into spoken audio.
- DeskBuddy shows it and says it. The reply appears on the OLED screen. The voice comes out of your computer's speakers, or out of a speaker wired to the ESP32.
The program on your computer that does the middle steps is called the relay. Your Claude key lives inside the relay and never goes onto the ESP32. That also means your computer has to be on, awake and running the relay whenever you want to talk to Buddy.
What you need
| Part | What it does | Good to know |
|---|---|---|
| ESP32 DevKit, 38 pins | The brain. It has Wi-Fi built in. | A full-size breadboard fits it best, because this board is wide. |
| 0.96 inch OLED screen (SSD1306, I2C, 4 pins) | Buddy's face and its written replies. | Pin order differs between brands, so read the labels printed on yours. |
| INMP441 microphone module | Hears you. | Power it from 3V3 only. 5V can destroy it. |
| MAX98357A amplifier and an 8 Ω 0.5 W speaker | Plays Buddy's voice from the ESP32. | Optional on a Mac, where the relay can use your computer's speakers instead. |
| Full-size breadboard and jumper wires | Joins everything without soldering. | About 20 male-to-male wires, plus a few spares. |
| USB data cable | Powers the ESP32 and loads programs onto it. | Some cheap cables only charge. Check whether your ESP32 has a micro-USB or a USB-C socket. |
If your OLED, microphone or amplifier came with a strip of pins that is not soldered on, solder it on. Loose pins pressed into the holes seem fine for a minute and then fail in confusing ways.
On the computer side
- A Mac (Apple Silicon or Intel) or a Windows 10 or 11 PC.
- Wi-Fi on the 2.4 GHz band. The ESP32 cannot use 5 GHz, and your computer must be on the same network.
- About 2 GB of free disk space for the speech model and Python packages.
- A Claude API key. Step 4 shows two ways to get one.
Two safety rules
Moving wires on a powered board is the quickest way to damage a part.
Connect the INMP441 to the 3V3 pin, never to 5V or VIN. 5V can destroy it.
Step 1: Set up your computer
You will install four things: Python (it runs the relay), the project files, the Arduino IDE (it loads programs onto the ESP32) and the ESP32 add-on for the Arduino IDE. It takes about 20 minutes, most of it waiting for downloads.
Pick your computer and the steps below will match it.They follow the same flow as the Mac steps but were not run on a real PC. If one is wrong for you, the troubleshooting section near the end lists the usual fixes.
Install Python
- Go to python.org/downloads and download the Python 3.12 installer for macOS (a
.pkgfile). Versions 3.10 to 3.12 are the safest. - Open the file and click through the installer.
- When it finishes, a Finder window may open. If you see Install Certificates.command, double-click it and wait until it says it is done.
- Open Terminal: press Command + Space, type
Terminaland press Enter. Then run:
python3 --version- Go to python.org/downloads and download the Python 3.12 Windows installer (64-bit).
- Run it. On the first screen, tick Add python.exe to PATH before you click Install Now.
- Open PowerShell: press the Windows key, type
PowerShelland press Enter. Then run:
python --versionIt prints something like Python 3.12.x.
If a window offers to install developer tools instead, close it, quit Terminal, open it again and retry.
If it says python is not recognized, run the installer again and tick the PATH box, then open a new PowerShell window.
Unzip the project
Double-click the zip in your Downloads folder. A folder called ai_buddy_project appears. Drag it into your Documents folder.
Right-click the zip in your Downloads folder, choose Extract All, and extract it into your Documents folder. You should end up with a folder called ai_buddy_project.
Install the relay's packages
- In Terminal, type
cdfollowed by one space. Do not press Enter yet. - In Finder, open
ai_buddy_projectand drag therelayfolder into the Terminal window. Its path appears aftercd. Now press Enter. - Type
lsand press Enter. If you seerelay.pyin the list, you are in the right place. - Run this. It downloads a few hundred megabytes and takes 5 to 10 minutes:
python3 -m pip install -r requirements.txt- In File Explorer, open
ai_buddy_project, then therelayfolder. - Click the address bar at the top, type
powershelland press Enter. A PowerShell window opens already inside that folder. - Run this. It downloads a few hundred megabytes and takes 5 to 10 minutes:
python -m pip install -r requirements.txtThe last lines say Successfully installed ... and there is no red error text.
If the later step fails with an error about a missing DLL, install the Microsoft Visual C++ Redistributable and try again.
Install the Arduino IDE
Go to arduino.cc/en/software and download Arduino IDE 2.x for macOS. Choose Apple Silicon for an M1, M2, M3 or M4 Mac and Intel otherwise (Apple menu, About This Mac, tells you). Open the download and drag Arduino IDE into Applications. If your Mac refuses to open it, right-click the app and choose Open.
Go to arduino.cc/en/software and download Arduino IDE 2.x for Windows 10 and newer. Run the installer and accept the defaults.
Add ESP32 support to the Arduino IDE
- Mac Open the settings with Command + , (comma).
- Windows Open File, Preferences.
- Find the box called Additional boards manager URLs and paste this line into it:
https://espressif.github.io/arduino-esp32/package_esp32_index.json- Click OK. Open the Boards Manager (the icon in the left bar) and search for
esp32. - Find esp32 by Espressif Systems and click Install. Choose the latest 3.x version. It is a big download, so wait for it to finish.
Install the screen libraries
Open the Library Manager (also in the left bar), search for Adafruit SSD1306 and click Install. When a window asks about dependencies, click Install all.
Check the USB connection
Plug the ESP32 into your computer. In the Arduino IDE open Tools, Port.
A new entry appears, such as /dev/cu.usbserial-0001 or /dev/cu.SLAB_USBtoUART. Always pick the one that starts with cu.
A new entry appears, such as COM3. The number can be different on your PC.
If no new port appears, try a different cable first (it must carry data), then a different USB port. Next look at the small chip near the USB socket. If it says CP2102, install the CP210x driver from Silicon Labs. If it says CH340, install the CH340 driver. On a Mac you may need to allow the driver in System Settings, Privacy & Security, and restart.
Tools, Port shows a new port whenever the ESP32 is plugged in.
Step 2: Wire it up
Change wires only while the ESP32 has no power.
Every pin on the ESP32 has a label printed beside it. Some boards print D32, some 32, IO32 or GPIO32. They all mean the same pin. 3V3 is the 3.3 volt pin, VIN is the 5 volt pin, and there are several GND pins that all work the same.
A colour code that makes mistakes easy to spot
Use whichever wires you have. If you can, follow these colours, because then a red wire should only ever touch a power pin, and anything else that is red is a mistake you can see.
- 3.3 V power
- 5 V power
- Ground (GND)
- Clock
- Data
- Left/right select
The microphone and the amplifier talk to the ESP32 with a standard called I2S. It always uses the same three jobs: a clock wire that keeps time, a select wire that says left or right, and a data wire. The screen uses a simpler standard called I2C, with its own clock (SCL) and data (SDA) wires.
Make the power rails
Plug the ESP32 into the breadboard so its two rows of pins sit on either side of the middle gap. Then:
- Connect the ESP32 3V3 pin to the breadboard's red (+) rail.
- Connect an ESP32 GND pin to the breadboard's blue (−) rail.
Use the printed stripes on your breadboard as the guide, even if your ground wire is black.
Connect each part
The order of the pins differs between brands, so read the labels printed on your own modules before you plug anything in.
OLED screen0.96 inch, 4 pins
- GNDblue (−) rail
- VCCred (+) rail3V3
- SDAGPIO 32
- SCLGPIO 33
INMP441 microphone3V3 only, never 5V
- VDDred (+) rail3V3 only
- GNDblue (−) rail
- L/Rblue (−) railthis picks the left slot
- SCKGPIO 26
- WSGPIO 25
- SDGPIO 35data out
MAX98357A amplifieroptional on a Mac
- VINESP32 VIN pin5 V, not the 3V3 rail
- GNDblue (−) rail
- BCLKGPIO 14
- LRCGPIO 13
- DINGPIO 27
- GAINleave empty
- SDleave emptythis one means shutdown
On the microphone, SD means "serial data out". On the amplifier, SD means "shutdown". They are different things, so do not mix them up. Leave the amplifier's SD and GAIN pins empty.
Connect the two speaker wires to the amplifier's output terminals, the two screw terminals or solder pads marked + and −. With a single speaker it does not matter which wire goes where. If you plan to use your computer's speakers instead, you can skip the amplifier and speaker for now.
Check everything slowly
Now plug the USB cable back in.
Step 3: Test the hardware
Before Wi-Fi, your computer and Claude come into it, check that each part works on its own. A small test program in the project does this, and it will save you hours of guessing later.
- In the Arduino IDE choose File, Open and open
hardware_check/hardware_check.inofrom the project folder. - Choose Tools, Board, esp32, ESP32 Dev Module.
- Choose your ESP32's port under Tools, Port.
- Click the Upload button, the right-pointing arrow at the top left. The first time, it takes a few minutes to compile.
- If the bottom window shows
Connecting........for more than 10 seconds, press and hold the BOOT button on the ESP32 until the upload starts, then let go. - When it says
Done uploading, open Tools, Serial Monitor and set the speed at the bottom right to 115200.
- Screen: it says "Hardware check", then a "Mic level" bar.
- Speaker: a short beep at the start, then one every 5 seconds.
- Microphone: when you clap or talk, the bar jumps and the numbers in the Serial Monitor go up.
Do not move on until the screen and the microphone work. The speaker is optional, as you will see below.
If one of them is missing
| What you see | What to check |
|---|---|
| The screen stays blank | SDA on pin 32 and SCL on pin 33 (try swapping them), plus VCC and GND. If it is still blank, upload i2c_scan/i2c_scan.ino. It lists every I2C device it finds, and your screen should appear as 0x3C. Odd addresses such as 0x0D or 0x52 mean a wire is loose. |
| No beep | Amplifier VIN to the ESP32 VIN pin, GND, then BCLK 14, LRC 13 and DIN 27. Check that the speaker wires are tight in the terminals. |
| The mic bar never moves | Microphone VDD on 3V3, L/R on GND, then SCK 26, WS 25 and SD 35. If all of that is right, open the file, change MIC_SLOT = 0 to MIC_SLOT = 1 and upload again. |
| Upload fails with "port not found" | Use a data USB cable, and see the USB check in step 1. |
No amplifier, or it only makes noise?
You do not need the amplifier at all. On a Mac, the relay can play Buddy's voice through your computer's own speakers, while the microphone and screen stay on the ESP32. The project's Mac launcher already does this.
- Disconnect the amplifier's VIN wire so the module stays off and silent. Everything else can stay in place.
- While Buddy speaks, the screen shows its reply and Buddy ignores sound, so it does not hear itself.
- Keep the microphone away from your computer's speakers and keep the volume moderate.
Playing through the computer's speakers uses a Mac command, afplay. On Windows, Buddy's voice has to come from the amplifier and speaker. The screen still shows every reply either way.
Step 4: Get a Claude key
Buddy talks to Claude through an API key, a long secret password tied to your account and your balance. You only need one of the two options below. Both give you a key that starts with sk-.
Option A: OneProvider (what this build used)
OneProvider is a separate company that sells prepaid access to Claude. The project files already point at it, so this option needs no extra setup.
- Open app.oneprovider.dev and sign in.
- Check that your balance is there. If it shows zero, look for the option to redeem a code. Button names may differ slightly from what is written here.
- Choose Issue a key, name it
buddyand, if you can, set a spend limit such as $10. If the key ever leaks, nobody can spend more than that. - Copy the key. It is shown in full only once, so paste it into a notes window for now.
Option B: Anthropic's own API
- Create an account in the Anthropic Console, add a few dollars of credit and create an API key.
- Tell the relay to use Anthropic's address instead of OneProvider's. In
relay/start_relay.command(Mac), remove the#in front of the line that mentionsONEPROVIDER_BASE_URL. On Windows you will type the same setting in step 5.
This build was only run through OneProvider. The relay uses Anthropic's own software library, so the official API should work, but nobody has confirmed it on this exact setup yet. The setting names say ONEPROVIDER because that is what the build used. They work for either provider.
Never post it, put it in a screenshot or share it in a chat. If it ever leaks, delete it in your provider's dashboard and issue a new one. The key lives only on your computer. The ESP32 never sees it.
Test the key
Open a terminal in the relay folder, the same way you did when you installed the packages. Then set your key and run the test. Put your real key between the quotes.
export ONEPROVIDER_KEY="sk-paste-your-key-here"
python3 test_key.py$env:ONEPROVIDER_KEY="sk-paste-your-key-here"
python test_key.pyIf you chose Option B, run the line export ONEPROVIDER_BASE_URL=https://api.anthropic.com first (on Windows: $env:ONEPROVIDER_BASE_URL="https://api.anthropic.com").
You see Claude says: ... followed by a short greeting, and then KEY WORKS. With OneProvider you also see the balance left on the key.
A 401 means the key was copied wrongly, so copy it again without spaces. A 429 means the balance or the key's spend limit is used up. The key setting only lasts as long as that window stays open.
Step 5: Start the relay
The relay is the program on your computer that turns your voice into text, asks Claude, and turns the answer into speech. It has to be running whenever you talk to Buddy.
- Open Terminal and go to the
relayfolder: typecdand a space, drag the folder into the window, press Enter. - Run the launcher with
bash. This avoids the "Apple could not verify..." warning that macOS shows for any script that came out of a zip file.
bash start_relay.command- When it asks, paste your key and press Enter.
- The first time, it downloads the speech-recognition model, about half a gigabyte. Wait for it.
- If your Mac asks "Do you want the application Python to accept incoming network connections?", click Allow.
Prefer double-clicking? Click Done on the warning, open System Settings, Privacy & Security, find the message about start_relay.command and click Open Anyway. After that, double-click works. The file is plain text, so you can open it in TextEdit first and see that it only starts the relay.
- Open a PowerShell window inside the
relayfolder, the way you did in step 1. - Set your key and start the relay. If you chose Option B, add the base URL line from step 4 above the last line.
$env:ONEPROVIDER_KEY="sk-paste-your-key-here"
python -m uvicorn relay:app --host 0.0.0.0 --port 8000- The first time, it downloads the speech-recognition model, about half a gigabyte. Wait for it.
- If Windows asks whether to let Python through the firewall, allow it on private networks.
The window shows Application startup complete and Uvicorn running on http://0.0.0.0:8000. Leave the window open. Closing it stops Buddy.
The launcher uses caffeinate so your Mac does not fall asleep while the relay runs. Keep the lid open anyway, because closing it stops Buddy.
Windows will sleep on its own and stop Buddy. While you use Buddy, set Settings, System, Power, Screen and sleep to Never.
Find your computer's address
The ESP32 needs to know where on your home network to find the relay. Open a second terminal window and run:
ipconfig getifaddr en0It prints something like 192.168.1.23. Write it down. If it prints nothing, try en1 instead of en0, or read it from System Settings, Wi-Fi, Details.
ipconfigUnder Wireless LAN adapter Wi-Fi, find the line IPv4 Address. It looks like 192.168.1.23. Write it down.
Test it from your phone
Connect your phone to the same Wi-Fi and open http://192.168.1.23:8000/ in its browser, using your own address.
The page shows something like {"ok":true,"model":"claude-haiku-5-5", ...}.
If it does not load, the usual causes are a wrong address, a phone on a different network (guest Wi-Fi often blocks this), or a privacy setting. On newer macOS, open System Settings, Privacy & Security, Local Network and make sure Terminal is switched on. If your computer is on a phone hotspot, its address often starts with 172.20.10. and the hotspot must be set to the 2.4 GHz band (on an iPhone, switch on Maximize Compatibility).
Step 6: Put Buddy on the ESP32
This is the real Buddy program. Open ai_buddy/ai_buddy.ino in the Arduino IDE and find these lines near the top:
// ---------------- Settings ----------------
const char* WIFI_SSID = "YOUR_WIFI_NAME";
const char* WIFI_PASS = "YOUR_WIFI_PASSWORD";
const char* RELAY_HOST = "192.168.1.50"; // LAN IP of the PC running relay.py
const uint16_t RELAY_PORT = 8000;Replace the three values with your own and keep the quote marks:
WIFI_SSIDis your Wi-Fi name. It must be a 2.4 GHz network.WIFI_PASSis your Wi-Fi password. Capital letters count.RELAY_HOSTis the computer address you wrote down in step 5.
Check that Tools, Board is still ESP32 Dev Module and the port is right, then click Upload. Hold the BOOT button if it sticks on Connecting..... Afterwards open the Serial Monitor at 115200.
The Serial Monitor says WiFi OK, IP ... and then noise floor .... The screen shows "Listening to the room... stay quiet" for about a second, then two eyes.
Stay quiet during that first second. Buddy uses it to learn how noisy your room is, so it knows what counts as someone speaking.
Step 7: Talk to Buddy
Just talk, normally, about 30 to 50 centimetres from the microphone. There is no button to hold. Try:
- "Hello Buddy, what's your name?"
- "What is two plus two?"
- "Tell me a fun fact."
This is what happens on the screen, in this order:
- The eyes widen, so Buddy knows you are speaking.
- The screen says Thinking... while your sentence travels to the relay and Claude answers.
- The reply appears as text.
- You hear the reply, and the eyes come back when Buddy is ready to listen again.
Expect a few seconds between the end of your sentence and the start of the reply. It depends on your computer. Look at the relay window too: it prints heard: (what Whisper understood) and reply: (what Claude said). That window is the best place to see what is going on.
You asked something, saw the reply on the screen and heard it. That is the whole project working.
Good to know
- Buddy cannot hear you while it is talking. Wait for the eyes to come back.
- Tap the BOOT button on the ESP32 to mute the microphone. The eyes become lines. Tap again to unmute.
- Buddy answers anyone it hears, including the TV. Use the wake word in the next section if you want it to answer only to its name.
- If you change the Arduino program, click Upload again. If you change the relay files, close the relay window and start it again.
Every day after that
- On your computer, start the relay and paste your key.
- Power the ESP32 from any USB charger. After the first upload it only needs the relay.
- Wait for the eyes, then talk.
If you restart your router, your computer's address may change. If Buddy then says "Can't reach the relay", find the address again and update RELAY_HOST in the program.
How the code works
You do not need to read any code to build Buddy. If you want to know why it works, these are the pieces that matter. Every snippet below is copied straight from the project files you downloaded.
The pins live in one place
// ---------------- Pins ----------------
const int PIN_BTN = 0; // BOOT button: tap = mute / unmute
const int PIN_SDA = 32, PIN_SCL = 33; // OLED
const int MIC_SCK = 26, MIC_WS = 25, MIC_SD = 35; // INMP441
const int SPK_BCLK = 14, SPK_LRC = 13, SPK_DIN = 27; // MAX98357AEach name is a job and each number is a pin on the ESP32. They match the wiring tables in step 2: SCK, WS and SD for the microphone, BCLK, LRC and DIN for the amplifier. If you wired something differently, change the numbers here to match what you built, then upload again. GPIO 35 can only be an input, which is exactly what the microphone's data wire needs.
Always listening, without a button
Buddy reads the microphone about 30 times a second, in short chunks. For each chunk it asks one question: is this louder than the room normally is? These settings decide the answer.
// ---------------- Voice detection (tune these if it triggers too much or too little) ----------------
const int CHUNK_FRAMES = 512; // 32 ms of audio per chunk at 16 kHz
const int PREROLL_CHUNKS = 8; // keep 256 ms from before speech is detected so the first word isn't clipped
const int START_CHUNKS = 3; // loud chunks in a row needed to start (about 100 ms)
const uint32_t END_SILENCE_MS = 900; // this much quiet ends your sentence
const uint32_t MAX_RECORD_MS = 15000; // safety limit for one sentence
const float VAD_MIN_LEVEL = 60; // never trigger below this level
const float VAD_RATIO = 3.0; // trigger when level > background noise x this
const bool DEBUG_LEVELS = false; // true = print levels to the Serial Monitor to help tuningAnd this is the loop that applies them:
// Always listening: each chunk goes into the ring buffer and is checked for speech
float level = readChunk(preroll[prerollHead]);
if (level < 0) return;
prerollHead = (prerollHead + 1) % PREROLL_CHUNKS;
if (prerollCount < PREROLL_CHUNKS) prerollCount++;
float threshold = fmaxf(VAD_MIN_LEVEL, noiseFloor * VAD_RATIO);
if (level > threshold) {
loudRun++;
} else {
loudRun = 0;
noiseFloor = noiseFloor * 0.97f + level * 0.03f; // slowly follow the room's background noise
}
if (DEBUG_LEVELS && millis() - lastDebug > 500) {
lastDebug = millis();
Serial.printf("level %.0f noise %.0f trigger %.0f\n", level, noiseFloor, threshold);
}
if (loudRun >= START_CHUNKS) {
loudRun = 0;
converse(threshold);
// ...- The trigger level is the larger of a fixed minimum and three times the room's background noise. A fan or a busy street raises it automatically.
- Three loud chunks in a row, about a tenth of a second, count as the start of speech. One door slam does not.
- Every chunk also goes into a small ring buffer. When Buddy decides you have started talking, it still has the quarter of a second before that, so your first word is not clipped.
- While it is quiet, Buddy slowly follows the room's noise level. That is the line with
0.97fand0.03f.
Sending a sentence to the relay
Once speech starts, the ESP32 opens a connection to the relay and sends the saved quarter second, then streams live audio until you have been quiet for 900 milliseconds. It cannot know in advance how long you will talk, so it uses chunked transfer, a standard feature of HTTP: each piece announces its own length, and a final empty piece, the 0, means "that is all".
// 1) the audio from just before speech was detected, oldest first
int first = (prerollHead - prerollCount + PREROLL_CHUNKS) % PREROLL_CHUNKS;
for (int k = 0; k < prerollCount; k++) sendChunk(c, preroll[(first + k) % PREROLL_CHUNKS]);
prerollCount = 0;
// 2) keep streaming until you have been quiet for END_SILENCE_MS
uint32_t startMs = millis(), lastVoice = millis();
float endThreshold = threshold * 0.7f; // a bit lower than the start level so trailing words don't cut off
float level = 0;
while (c.connected() && millis() - startMs < MAX_RECORD_MS) {
level = readChunk(chunkBuf);
if (level < 0) continue;
sendChunk(c, chunkBuf);
if (level > endThreshold) lastVoice = millis();
else if (millis() - lastVoice > END_SILENCE_MS) break;
}
if (millis() - startMs >= MAX_RECORD_MS) noiseFloor = level; // constant loud noise: raise the bar
c.print("0\r\n\r\n");The relay: ears, brain and voice
The relay is a small Python web server (FastAPI) with one job: take a sentence of sound and return Buddy's answer. It does that in three moves.
Ears: Whisper turns sound into text
def transcribe(pcm: bytes) -> str:
pcm = pcm[: len(pcm) // 2 * 2]
audio = np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0
peak = float(np.abs(audio).max()) if audio.size else 0.0
if peak > 0:
audio = audio * min(0.9 / peak, 8.0) # gentle normalisation for the quiet INMP441
segments, _ = get_whisper().transcribe(audio, vad_filter=True, beam_size=1, language=WHISPER_LANG)
return " ".join(s.text.strip() for s in segments).strip()The audio is scaled up a little first, because the INMP441 is quiet. vad_filter=True makes Whisper skip silence, and beam_size=1 trades a little accuracy for speed.
Brain: Claude writes the reply
messages = list(history) + [{"role": "user", "content": heard}]
while messages and messages[0]["role"] != "user":
messages.pop(0)
try:
resp = await client.messages.create(model=MODEL, max_tokens=200, system=SYSTEM, messages=messages)
reply = "".join(b.text for b in resp.content if b.type == "text").strip() or "Hmm, I have nothing to say."history remembers the last six exchanges, so Buddy can follow a conversation. The short loop makes sure the list starts with something you said, which is a rule of the API. max_tokens=200 keeps answers short.
Voice: Edge TTS speaks it
async def tts_mp3(text: str) -> bytes:
mp3 = bytearray()
async for chunk in edge_tts.Communicate(text, VOICE).stream():
if chunk["type"] == "audio":
mp3.extend(chunk["data"])
return bytes(mp3)Edge TTS streams back the spoken reply as MP3 audio. The relay then either plays it on your Mac or converts it to the format the ESP32 speaker expects.
Buddy's personality is one paragraph
SYSTEM = (
"You are Buddy, a friendly little desk companion living inside an ESP32 with a tiny screen and speaker. "
"Your replies are spoken aloud, so answer in one or two short plain sentences. "
"No markdown, no lists, no emojis."The reply is spoken aloud, so the instructions ask for short, plain sentences. Without "no markdown, no lists, no emojis", the voice would read out every asterisk.
The face is two rounded rectangles
void drawFace(bool blink) {
display.clearDisplay();
if (blink) {
display.fillRect(30, 34, 24, 3, SSD1306_WHITE);
display.fillRect(74, 34, 24, 3, SSD1306_WHITE);
} else {
display.fillRoundRect(30, 14, 24, 30, 6, SSD1306_WHITE);
display.fillRoundRect(74, 14, 24, 30, 6, SSD1306_WHITE);
}
display.setTextColor(SSD1306_WHITE);
display.setTextSize(1);
display.setCursor(34, 55);
display.print("just talk :)");
display.display();
}The demo at the top of this page draws Buddy's face with exactly these numbers. Change them, upload, and Buddy gets new eyes.
Make it yours
Most changes are one line. Edit the file, save it, and then either click Upload again (for ai_buddy.ino) or restart the relay (for everything in the relay folder).
| You want to | Change this | In |
|---|---|---|
| Give Buddy a new personality | The text under SYSTEM = ( | relay.py |
| Have it answer only when it hears its name | Remove the # before export WAKE_WORD=buddy. After a reply you can keep chatting for 20 seconds without the name. On Windows set $env:WAKE_WORD="buddy". | start_relay.command |
| Get faster replies | WHISPER_MODEL=base, a smaller and slightly less accurate listener | start_relay.command |
| Use a different voice | TTS_VOICE. The default is en-IN-NeerjaNeural. Run edge-tts --list-voices to see the rest. | start_relay.command |
| Talk in Hindi | TTS_VOICE=hi-IN-SwaraNeural, WHISPER_LANG=hi, and ask for Hindi replies in SYSTEM. The screen font only shows English letters, so Hindi is spoken but not shown. | start_relay.command, relay.py |
| Use another Claude model | CLAUDE_MODEL. The default is claude-haiku-5-5. | start_relay.command |
| Choose where the voice comes out | SPEAKER=mac for your Mac's speakers, or SPEAKER=esp32 for the speaker wired to the board | start_relay.command |
| Change the volume | MAC_VOLUME for Mac speakers. VOLUME for the ESP32 speaker, and keep that at 0.5 or below. | start_relay.command |
The launcher lists the optional settings, switched off, so you only have to remove a #:
# Optional settings: remove the # at the start of a line to turn it on.
# export WAKE_WORD=buddy
# export WHISPER_MODEL=base
# export VOLUME=0.35
# Using Anthropic's own API instead of OneProvider? Remove the # on the next line.
# export ONEPROVIDER_BASE_URL=https://api.anthropic.comTune how it listens
If Buddy ignores you or jumps at every noise, set DEBUG_LEVELS to true in ai_buddy.ino and upload. The Serial Monitor then prints level, noise and trigger twice a second, so you can see what the microphone sees.
| What happens | Try this |
|---|---|
| It never reacts to you | Lower VAD_MIN_LEVEL to 30, or lower MIC_SHIFT to 13 or 12. |
| It reacts to background noise | Raise VAD_MIN_LEVEL or VAD_RATIO. |
| It cuts you off mid-sentence | Raise END_SILENCE_MS to 1200. |
| It waits too long after you finish | Lower END_SILENCE_MS to 700. |
| The microphone is silent | Set MIC_SLOT = 1, and check that the microphone's L/R pin is on GND. |
If something goes wrong
Nearly every problem in this build comes down to a loose wire, a wrong pin number or a window that was closed. Find your symptom below. If none of them fits, the last section of this page shows what to send when you ask for help.
Setting up your computer
Arduino says: ESP_I2S.h: No such file
Your ESP32 board package is too old. Open the Boards Manager and install version 3.x of "esp32 by Espressif Systems".
Mac: error: externally-managed-environment
You are using a Python installed by Homebrew. Install Python from python.org (step 1), open a new Terminal window and try again.
Mac: command not found: python3, or a developer tools pop-up
Install Python from python.org (step 1), then quit and reopen Terminal.
Windows: 'python' is not recognized
Run the Python installer again and tick Add python.exe to PATH. Then open a new PowerShell window.
The Arduino IDE does not show a port
Use a cable that carries data (many cheap ones only charge) and try another USB port. Check the chip near the USB socket: install the CP210x driver for a CP2102 or the CH340 driver for a CH340.
Upload sticks on Connecting........
Press and hold the BOOT button on the ESP32 until the upload starts, then let go.
Wiring and the hardware check
The screen stays blank
Check SDA on pin 32 and SCL on pin 33, then VCC and GND. Upload i2c_scan/i2c_scan.ino: your screen should appear as 0x3C. If you only see odd addresses such as 0x0D or 0x52, a wire is loose. Push every wire in firmly and scan again.
The speaker plays a steady tone or hiss
First check that BCLK, LRC and DIN are all firmly connected to pins 14, 13 and 27, and that the amplifier's VIN is on the ESP32 VIN pin. A floating or loose pin makes this noise. If it keeps happening with correct wiring, the amplifier module itself may be faulty. See "What went wrong" below, then switch to your computer's speakers.
There is no beep at all
Amplifier VIN to the ESP32 VIN pin, GND to the rail, then BCLK 14, LRC 13, DIN 27. Check that the speaker wires are tight in the terminals.
The microphone bar never moves
Microphone VDD on 3V3, L/R on GND, SCK 26, WS 25, SD 35. If all that is right, change MIC_SLOT = 0 to MIC_SLOT = 1 and upload again.
The key and the relay
Mac: Apple could not verify start_relay.command
Open Terminal, go to the relay folder (type cd and a space, drag the folder in, press Enter) and run bash start_relay.command. See step 5 for the other way.
The relay window mentions ONEPROVIDER_KEY
The key was not entered. Start the relay again and paste your key when it asks.
test_key.py says 401
The key was copied wrongly. Copy it again with no spaces.
test_key.py says 429, or Buddy says its balance has run out
The balance, or that key's spend limit, is used up. Top up or raise the limit in your provider's dashboard.
The phone cannot open the relay address
Check the address, that the phone is on the same Wi-Fi (guest networks often block this) and that the relay window is still open. On macOS, make sure Terminal is allowed under System Settings, Privacy & Security, Local Network. On Windows, allow Python through the firewall on private networks.
Talking to Buddy
The screen says WiFi failed
Check the Wi-Fi name and password, where capital letters count. The network must be 2.4 GHz.
The screen says it can't reach the relay
The relay window is closed, RELAY_HOST is wrong, the computer is asleep or the computer and the ESP32 are on different networks. Run the phone test from step 5.
Buddy worked, then stopped answering
Your computer probably went to sleep or the lid was closed. Wake it. The relay window should still be there.
Buddy never reacts to my voice
Set DEBUG_LEVELS = true, upload, and watch the Serial Monitor while you talk. If level never goes above trigger, lower VAD_MIN_LEVEL to 30 or MIC_SHIFT to 13.
Buddy reacts to every little noise
Raise VAD_MIN_LEVEL (try 120), or use the wake word.
Buddy cuts me off mid-sentence
Raise END_SILENCE_MS to 1200.
The relay window says heard: (nothing)
Speak a little louder or closer, or lower MIC_SHIFT to 13.
Buddy answers itself in a loop
Its voice is reaching the microphone. Lower the volume, move the microphone away from the speaker, or use the wake word.
Mac: the screen shows the reply but there is no sound
Check that the Mac's volume is up and not muted, and that start_relay.command contains export SPEAKER=mac. The relay window prints reply: ... when it works. If it says afplay was not found, you are not on a Mac.
Replies are slow
Use the smaller speech model: remove the # before export WHISPER_MODEL=base in start_relay.command.
What went wrong while this was built
The first version of anything rarely works. These are the real problems that came up while building this Buddy, and what fixed them. If you hit one, you are in good company.
The screen vanished
The OLED worked, then showed nothing. The I2C scan reported strange addresses,
0x0Dand0x52, instead of the screen's0x3C. Nothing was wrong with the code: a jumper wire had worked loose in the breadboard.Lesson: when a module disappears, re-seat every wire on it before you touch the code. The scan sketch in the project exists for exactly this.
The amplifier would not stop humming
With the speaker wired up, the amplifier played a steady tone no matter what the code sent it. Connecting BCLK and LRC alone made noise, and all three signal wires made a continuous tone. Several test programs, including one that sent only silence and one that used an older audio driver, gave the same result. The module was most likely faulty.
Lesson: test the speaker early, with the hardware check, and keep a plan B. Here that was playing Buddy's voice through the Mac's speakers, which needs no amplifier at all.
The diagram and the code disagreed
The first wiring diagram used pins 32 and 33 for the screen, 26 and 25 for the microphone clock and 14 and 13 for the amplifier. The first version of the code expected different pins, so nothing worked until one side was changed to match the other.
Lesson: the pin numbers sit in three lines at the top of the program. Write down what you actually wired, and make those lines say the same thing.
The Mac refused to run the launcher
Double-clicking
start_relay.commandshowed "Apple could not verify..." because the file came out of a zip. It is a plain text script, so running it withbash start_relay.commandfrom Terminal avoids the warning completely.The hotspot network looked different
On an iPhone hotspot, the Mac's address was
172.20.10.2instead of the usual192.168.x.x. That is normal. The ESP32 needs the 2.4 GHz band, so the hotspot's Maximize Compatibility switch had to be on.
Safety, cost and privacy
- Speaker: the small speaker is rated 0.5 W and the amplifier can deliver far more. Keep
VOLUMEat 0.5 or below. - Power: the microphone gets 3V3 only, and you unplug the USB cable before you move wires.
- Cost: with a small model such as the default, a short chat costs a tiny fraction of a cent. Check your provider's price list, and set a spend limit on the key so a mistake can never get expensive.
- What travels where: Buddy is always listening, but sound only goes from the ESP32 to your own computer, and only when it hears speech. The written text of your sentence goes to Claude through your provider. The text of Buddy's reply goes to Microsoft's Edge text-to-speech service so it can be spoken. Your voice recording is not sent to either.
- Mute button: tap BOOT on the ESP32 whenever you do not want to be heard.
- Third parties: OneProvider is a separate company, not Anthropic, and what it does with your text is its own policy. Do not tell Buddy anything you would not type into any online service.
Get the project files
Everything you need is in one zip: the test programs, the Buddy program for the ESP32 and the relay for your computer.
ai_buddy_project/
README.md short technical reference
hardware_check/ test your screen, speaker and microphone first
i2c_scan/ finds your screen if it does not show up
speaker_test/ a plain speaker test
ai_buddy/ the Buddy program for the ESP32
relay/ the program that runs on your computer
relay.py speech to text, Claude, text to speech
requirements.txt the Python packages it needs
test_key.py checks that your Claude key works
start_relay.command Mac launcher (asks for your key)Stuck? Send three things when you ask for help: the text on the Buddy screen, the text in the Serial Monitor and the text in the relay window. With those, most problems can be found in minutes. Ask for help.
Ideas for what to build next
- Switch on the wake word so Buddy only answers to its name.
- Give it a new voice or a second language.
- Draw a different face: more expressions for thinking, speaking and muted.
- Print or build a small case so it sits on your desk without a breadboard.
- Run it from a power bank and carry it around the house.
Built with the ESP32 Arduino core, the Adafruit SSD1306 and GFX libraries, faster-whisper, FastAPI, edge-tts and Claude.
This guide is independent. Claude is made by Anthropic, and OneProvider is a separate company. Neither has reviewed or endorsed this project.