Headless VTuber architecture notes#

Recommended first build#

Use a split architecture:

1. Chat/runtime process on the headless machine
2. Avatar renderer as either:
   • a headless browser scene with sprite layers and viseme/expression animation, or
   • a Linux-native VTuber renderer if one exposes remote control hooks
3. ffmpeg composition and push to stream.place via WHIP or RTMP

Why not start with a full Live2D rig?#

Live2D-on-Linux headless is possible but likely slows iteration. First prove:

• incoming chat
• generated reply
• synthesized audio
• deterministic motion/expression events
• continuous video stream

Then swap the renderer.

Recommended prototype phases#

Phase 1#

• simulate stream.place chat locally
• generate reply text
• synthesize or stub audio
• emit avatar actions like talk:start, emotion:happy, head:nod
• render a simple animated scene to a test stream or file

Phase 2#

• connect real stream.place chat feed
• post replies back to chat
• map reply prosody/emotion to avatar events

Phase 3#

• replace simple avatar scene with a real VTuber rig
• add interrupt handling, queueing, moderation, and pacing

Candidate render paths#

Path A: Browser/canvas renderer#

Pros:

• easiest to automate from CLI
• easy websocket control
• easy capture with Chromium or ffmpeg from a virtual display
• good enough for a first expressive avatar

Cons:

• not a full Live2D rig

Path B: OpenVT or custom Godot scene#

Pros:

• closer to real VTuber motion
• Linux-native

Cons:

• more integration work
• remote-control API may need custom glue

Path C: OBS scene graph on another machine#

Pros:

• simplest if final stream production happens elsewhere

Cons:

• not fully headless/self-contained