The Open Source xAI Ani that's next level

3 months ago 4

[Join Discord Server] [Try it] [简体中文] [日本語]

Heavily inspired by Neuro-sama

Note

We got a whole dedicated organization @proj-airi for all the sub-project that born from Project AIRI, check it out!

RAG, memory system, embedded database, icons, Live2D utilities, and more!

Have you dreamed about having a cyber living being (cyber waifu / husbando, digital pet), or digital companion that could play with and talk to you?

With the power of modern large langauge models like ChatGPT, and famous Claude, asking a virtual being able to have role playing and chat with us is already easy enough for everyone. Platforms like Character.ai (a.k.a. c.ai) and JanitorAI, and local playgrounds like SillyTavern is already a well-enough solution for chat based, or visuali adventure game like experience.

But, what about the abilities to play games? And see what you are coding at? Chatting while playing games, watching videos, and capable of doing many other things.

Perhaps you know Neuro-sama already, she is currenly the best companion capable of playing games, chatting, and interacting with you and the participants (in VTuber community), some call this kind of being, "digital human" too. Sadly, it's not open sourced, you cannot interact with her after she went offline from live stream.

Therefore, this project, AIRI, offers another possibilty here: let you own yours digital life, cyber living, easily, anywhere, anytime.

DevLogs we posted & Recent updates

DreamLog 0x1 on June 16, 2025
DevLog @ 2025.06.08 on June 8, 2025
DevLog @ 2025.05.16 on May 16, 2025
...more on documentation site

What's so special for this project?

Unlike the other AI driven VTuber open source projects, アイリ VTuber was built with many support of Web technologies such as WebGPU, WebAudio, Web Workers, WebAssembly, WebSocket, etc. from the first day.

Tip

Worry about the performance drop since we are using Web related technologies?

Don't worry, while Web browser version meant to give a insight about how much we can push and do inside browsers, and webviews, we will never fully rely on this, the desktop version of AIRI is capable of using native NVIDIA CUDA and Apple Metal by default (thanks to HuggingFace & beloved candle project), without any complex dependency managements, considering the tradeoff, it was partially powered by Web technologies for graphics, layouts, animations, and the WIP plugin systems for everyone to integrate things.

This means that アイリ VTuber is capable to run on modern browsers and devices, and even on mobile devices (already done with PWA support), this brought a lot of possibilities for us (the developers) to build and extend the power of アイリ VTuber to the next level, while still left the flexibilities for users to enable features that requires TCP connections or other non-Web technologies such as connect to voice channel to Discord, or playing Minecraft, Factorio with you and your friends.

Note

We are still in the early stage of development where we are seeking out talented developers to join us and help us to make アイリ VTuber a reality.

It's ok if you are not familiar with Vue.js, TypeScript, and devtools that required for this project, you can join us as an artist, designer, or even help us to launch our first live stream.

Even you are a big fan of React or Svelte, even Solid, we welcome you, you can open a sub-directory to add features that you want to see in アイリ VTuber, or would like to experiment with.

Fields (and related projects) that we are looking for:

Live2D modeller
VRM modeller
VRChat avatar designer
Computer Vision
Reinforcement Learning
Speech Recognition
Speech Synthesis
ONNX Runtime
Transformers.js
vLLM
WebGPU
Three.js
WebXR (checkout the another project we have under @moeru-ai organization)

If you are interested in, why not introduce yourself here? Would like to join part of us to build AIRI?

Capable of

Brain
- Play Minecraft
- Play Factorio (WIP, but PoC and demo available)
- Chat in Telegram
- Chat in Discord
- Memory
  - Pure in-browser database support (DuckDB WASM | pglite)
  - Memory Alaya (WIP)
- Pure in-browser local (WebGPU) inference
Ears
- Audio input from browser
- Audio input from Discord
- Client side speech recognition
- Client side talking detection
Mouth
- ElevenLabs voice synthesis
Body
- VRM support
  - Control VRM model
- VRM model animations
  - Auto blink
  - Auto look at
  - Idle eye movement
- Live2D support
  - Control Live2D model
- Live2D model animations
  - Auto blink
  - Auto look at
  - Idle eye movement

For detailed instructions to develop this project, follow the CONTRIBUTING.md

Note

By default, pnpm dev will start the development server for the Stage Web (browser version), if you would like to try developing the desktop version, please make sure you read CONTRIBUTING.md to setup the environment correctly.

Stage Tamagotchi (Desktop version)

Please update the version in Cargo.toml after running the bumpp:

npx bumpp --no-commit --no-tag

Supported the following LLM API Providers (powered by xsai)

OpenRouter
vLLM
SGLang
Ollama
Google Gemini
OpenAI
- Azure OpenAI API (PR welcome)
Anthropic Claude
- AWS Claude (PR welcome)
DeepSeek
Qwen
xAI
Groq
Mistral
Cloudflare Workers AI
Together.ai
Fireworks.ai
Novita
Zhipu
SiliconFlow
Stepfun
Baichuan
Minimax
Moonshot AI
Player2
Tencent Cloud
Sparks (PR welcome)
Volcano Engine (PR welcome)

Sub-projects born from this project

Awesome AI VTuber: A curated list of AI VTubers and related projects
unspeech: Universal endpoint proxy server for /audio/transcriptions and /audio/speech, like LiteLLM but for any ASR and TTS
hfup: tools to help on deploying, bundling to HuggingFace Spaces
xsai-transformers: Experimental 🤗 Transformers.js provider for xsAI.
WebAI: Realtime Voice Chat: Full example of implementing ChatGPT's realtime voice from scratch with VAD + STT + LLM + TTS.
@proj-airi/drizzle-duckdb-wasm: Drizzle ORM driver for DuckDB WASM
@proj-airi/duckdb-wasm: Easy to use wrapper for @duckdb/duckdb-wasm
tauri-plugin-mcp: A Tauri plugin for interacting with MCP servers.
AIRI Factorio: Allow AIRI to play Factorio
Factorio RCON API: RESTful API wrapper for Factorio headless server console
autorio: Factorio automation library
tstl-plugin-reload-factorio-mod: Reload Factorio mod when developing
Velin: Use Vue SFC and Markdown to write easy to manage stateful prompts for LLM
demodel: Easily boost the speed of pulling your models and datasets from various of inference runtimes.
inventory: Centralized model catalog and default provider configurations backend service
MCP Launcher: Easy to use MCP builder & launcher for all possible MCP servers, just like Ollama for models!
🥺 SAD: Documentation and notes for self-host and browser running LLMs.

%%{ init: { 'flowchart': { 'curve': 'catmullRom' } } }%% flowchart TD Core("Core") Unspeech("unspeech") DBDriver("@proj-airi/drizzle-duckdb-wasm") MemoryDriver("[WIP] Memory Alaya") DB1("@proj-airi/duckdb-wasm") SVRT("@proj-airi/server-runtime") Memory("Memory") STT("STT") Stage("Stage") StageUI("@proj-airi/stage-ui") UI("@proj-airi/ui") subgraph AIRI DB1 --> DBDriver --> MemoryDriver --> Memory --> Core UI --> StageUI --> Stage --> Core Core --> STT Core --> SVRT end subgraph UI_Components UI --> StageUI UITransitions("@proj-airi/ui-transitions") --> StageUI UILoadingScreens("@proj-airi/ui-loading-screens") --> StageUI FontCJK("@proj-airi/font-cjkfonts-allseto") --> StageUI FontXiaolai("@proj-airi/font-xiaolai") --> StageUI end subgraph Apps Stage --> StageWeb("@proj-airi/stage-web") Stage --> StageTamagotchi("@proj-airi/stage-tamagotchi") Core --> RealtimeAudio("@proj-airi/realtime-audio") Core --> PromptEngineering("@proj-airi/playground-prompt-engineering") end subgraph Server_Components Core --> ServerSDK("@proj-airi/server-sdk") ServerShared("@proj-airi/server-shared") --> SVRT ServerShared --> ServerSDK end STT -->|Speaking| Unspeech SVRT -->|Playing Factorio| F_AGENT SVRT -->|Playing Minecraft| MC_AGENT subgraph Factorio_Agent F_AGENT("Factorio Agent") F_API("Factorio RCON API") factorio-server("factorio-server") F_MOD1("autorio") F_AGENT --> F_API -.-> factorio-server F_MOD1 -.-> factorio-server end subgraph Minecraft_Agent MC_AGENT("Minecraft Agent") Mineflayer("Mineflayer") minecraft-server("minecraft-server") MC_AGENT --> Mineflayer -.-> minecraft-server end XSAI("xsAI") --> Core XSAI --> F_AGENT XSAI --> MC_AGENT Core --> TauriMCP("@proj-airi/tauri-plugin-mcp") Memory_PGVector("@proj-airi/memory-pgvector") --> Memory style Core fill:#f9d4d4,stroke:#333,stroke-width:1px style AIRI fill:#fcf7f7,stroke:#333,stroke-width:1px style UI fill:#d4f9d4,stroke:#333,stroke-width:1px style Stage fill:#d4f9d4,stroke:#333,stroke-width:1px style UI_Components fill:#d4f9d4,stroke:#333,stroke-width:1px style Server_Components fill:#d4e6f9,stroke:#333,stroke-width:1px style Apps fill:#d4d4f9,stroke:#333,stroke-width:1px style Factorio_Agent fill:#f9d4f2,stroke:#333,stroke-width:1px style Minecraft_Agent fill:#f9d4f2,stroke:#333,stroke-width:1px style DBDriver fill:#f9f9d4,stroke:#333,stroke-width:1px style MemoryDriver fill:#f9f9d4,stroke:#333,stroke-width:1px style DB1 fill:#f9f9d4,stroke:#333,stroke-width:1px style Memory fill:#f9f9d4,stroke:#333,stroke-width:1px style Memory_PGVector fill:#f9f9d4,stroke:#333,stroke-width:1px