🧠 AI Assistant — "Claudia"

A full-featured voice AI assistant that lives inside Reachy Mini. Speaks Russian, recognizes faces and voices, remembers facts, researches the web, and expresses emotions through robot movements.

Source: src/reachy_mini_ai/

How It Works

  Microphone                                            Speaker
     │                                                     ▲
     ▼                                                     │
┌─────────┐   wake word    ┌───────────┐  silence   ┌─────────────┐
│  Vosk   │ ─────────────> │ Recording │ ────────>  │ Processing  │
│ (idle)  │  "клавдия"     │ (chunks)  │  1.5s      │             │
└─────────┘                └───────────┘            │  ┌────────┐ │
     ▲                          ▲                   │  │Voice ID│ │
     │                          │                   │  │Face ID │ │
     │        interrupted       │                   │  │Context │ │
     │        (no wake word     │                   │  └───┬────┘ │
     │         needed)          │                   │      ▼      │
     │                          │                   │  GPT-4o     │
     │                     ┌────┴────┐              │  audio      │
     │                     │ User    │              │  streaming  │──> chunks ──> resample ──> play
     │                     │ speaks  │              │             │       ▲
     │                     │ over    │              └─────────────┘       │
     │                     │ robot   │◄──── interruption detected ───────┘
     │                     └────┬────┘
     │                          │
     └──────────────────────────┘
                                         ┌─────────────────────────┐
     In parallel (non-blocking):         │ Background Brain        │
                                         │ (GPT-4o-mini)           │
     transcript from conversation ──────>│  - remember_fact()      │
                                         │  - deep_research()      │
                                         │  - recall_person()      │
                                         └─────────────────────────┘
                                         ┌─────────────────────────┐
     transcript from conversation ──────>│ Emotion Engine          │
                                         │ (GPT-4o-mini)           │
                                         │  - play emotions        │
                                         │  - nod / shake / wiggle │
                                         └─────────────────────────┘

Pipeline Step-by-Step

1. Wake word — Vosk (Russian) listens for "клавдия" / "робот" / "ричи"
2. Recording — audio chunks collected until 1.5s silence or 30s max
3. Person ID — face (dlib 128-dim) + voice (Resemblyzer 256-dim) identify the speaker, load known facts from SQLite
4. GPT-4o audio streaming — audio sent to GPT-4o-audio-preview via OpenRouter. Response streams back as PCM16 chunks (24kHz), resampled to 16kHz and played immediately (~1s to first audio)
5. Interruption — during playback, mic is monitored via RMS energy VAD. If user speaks over the robot: playback stops, transcript of interrupted speech saved, state goes to LISTENING (no wake word needed)
6. Background brain — transcript sent to GPT-4o-mini which autonomously decides: save a fact? research something? recall a person? Runs in a separate thread pool, never blocks the conversation
7. Emotion engine — another GPT-4o-mini call picks 1-2 robot actions (emotions, nod, shake, wiggle) based on the conversation. Tracks interruption frequency: 2x = annoyed, 3x = angry, 4x = furious

Quick Start

export OPENROUTER_API_KEY="sk-or-..."
export TAVILY_API_KEY="tvly-..."  # optional, for web research

# Download Vosk Russian model
mkdir -p ~/.cache/vosk && cd ~/.cache/vosk
wget https://alphacephei.com/vosk/models/vosk-model-small-ru-0.22.zip
unzip vosk-model-small-ru-0.22.zip

# Run with simulator
reachy-mini-daemon --sim &
python -m reachy_mini_ai.app

Key Dependencies

Package	Purpose
openai	GPT-4o audio + GPT-4o-mini brain (via OpenRouter)
vosk	Russian wake word detection
face_recognition	Face identification (dlib)
resemblyzer	Speaker voice embeddings
tavily-python	Web research
scipy	Audio resampling (24kHz → 16kHz)

🤝 Community & Contributing

• Join the Community: Join Discord to share your moments with Reachy, build apps together, and get help.
• Found a bug? Open an issue on this repository.

License

This project is licensed under the Apache 2.0 License. See the LICENSE file for details. Hardware design files are licensed under Creative Commons BY-SA-NC.