# Gemini Live + OpenVT + OBS + stream.place integration plan

## Objective

Connect these pieces into one loop:

1. stream.place chat arrives
2. chat triggers an agent response
3. Gemini Live API generates the live reply / speech stream
4. OpenVT shows Haru as the avatar renderer
5. OBS captures OpenVT and audio
6. OBS streams to stream.place

## Current repo state

### Already present

- OpenVT local setup notes and launcher scripts
- working Haru/OpenVT path in `vendor/open-vt`
- simple chat runtime scaffold in `src/index.ts`
- local chat simulation in `scripts/simulate-chat.ts`
- architecture notes mentioning:
  - stream.place chat
  - LLM reply generation
  - TTS
  - avatar actions
  - WHIP/RTMP uplink

### Not yet present

- real stream.place chat adapter
- Gemini Live API client
- agent orchestration layer
- OpenVT control bridge from external runtime
- OBS automation / obs-websocket control
- actual stream.place ingest automation

## Minimal end-to-end architecture

```text
stream.place chat
  -> chat adapter
  -> agent runtime
  -> Gemini Live session
       -> text reply
       -> streaming audio reply
       -> optional emotion tags / cues
  -> avatar control bridge
       -> OpenVT expression / motion triggers
  -> OBS scene
       -> captures OpenVT window or virtual camera
       -> mixes Gemini audio output / mic
  -> stream.place ingest (WHIP or RTMP)
```

## Recommended practical split

### Process A: bot runtime

Responsible for:

- reading chat from stream.place
- deciding whether to respond
- calling the agent
- opening / maintaining Gemini Live session
- emitting:
  - reply text
  - audio stream or audio file
  - avatar cues like `emotion:happy`, `gesture:nod`, `talk:start`

### Process B: OpenVT renderer

Responsible for:

- rendering Haru
- microphone / lip sync path
- avatar expressions / motions
- outputting window / virtual camera feed

### Process C: OBS production

Responsible for:

- capturing OpenVT window or loopback webcam
- ingesting Gemini audio output
- mixing optional mic/music
- sending program output to stream.place

## Critical integration gaps

### 1. Gemini Live client

Need a client that can:

- send text/chat turns
- receive streaming text/audio
- optionally maintain conversational session state

Likely implementation location:

- `src/gemini-live.ts`

### 2. Chat adapter for stream.place

Need a real adapter replacing the current scaffold.

Likely implementation location:

- `src/streamplace-chat.ts`

Expected responsibilities:

- subscribe to chat events (likely Jetstream / ATProto feed path)
- normalize events into local `ChatMessage`
- send replies back if desired

### 3. OpenVT control bridge

OpenVT currently supports internal blueprint / tracker-driven motion, but this repo does not yet expose an external API for:

- toggling expressions
- triggering gestures / animations
- setting mouth/talk state from external runtime

Likely easiest bridge options:

1. small local websocket server inside OpenVT, or
2. local IPC / file / TCP bridge from external runtime into OpenVT tracker/action nodes

Likely implementation target:

- add a lightweight control node under `vendor/open-vt` or use a separate control process

### 4. OBS control

OBS can be kept manual at first, but for automation the likely path is:

- `obs-websocket`

It would allow:

- scene switching
- source muting/unmuting
- start/stop streaming

Likely implementation location:

- `src/obs-control.ts`

### 5. stream.place ingest

Docs in this repo already assume:

- WHIP if available
- otherwise RTMP

Likely first step:

- manual OBS config for stream.place ingest

## Smallest viable path

To minimize scope, do this in order:

### Phase 1: fake chat -> fake reply -> OpenVT + OBS manual

- use local simulated chat
- use deterministic local text reply
- use existing OpenVT Haru setup
- use OBS manual window capture
- no Gemini yet

### Phase 2: Gemini Live reply + audio

- replace fake reply with Gemini Live
- route Gemini audio to system audio or a known sink OBS can capture
- keep OBS manual

### Phase 3: external avatar control

- map reply affect / punctuation / tags to:
  - expression
  - gesture
  - talk state
- send those cues into OpenVT

### Phase 4: real stream.place chat + real stream.place ingest

- replace simulated chat with stream.place adapter
- optionally post replies back to chat

## Exact current blocker

The repo does not yet contain any real implementation for:

- Gemini Live API calls
- stream.place chat subscription
- OpenVT external control bridge
- OBS websocket integration

So the task is not “wire existing modules together”; it is “build the missing connectors around the current OpenVT renderer path.”

## Recommended next implementation step

Build the first real connector:

1. `src/gemini-live.ts`
2. `src/runtime.ts` orchestrator
3. extend `scripts/simulate-chat.ts` to call the runtime

This gives:

- simulated chat
- agent-triggered response
- Gemini Live text/audio integration

Then the next connector should be:

- OpenVT control bridge for avatar cues

