Dory — AI Gaming Companion for Minecraft

Let AI companions superpower your players' experience.
Integrate an AI game companion with voice, memory, and in-game agency — in just a few steps.

Built for Global AI Game Hack 2026

▶️ Submission Video - 1m 20s   |   🎬 Interactive Walkthrough - 7m 28sec

The submission video was produced using Higgsfield for AI-generated visuals, CapCut for editing, and real gameplay interactions captured live from Dory.

Dory is an open-source AI companion that lives inside your Minecraft world. Players talk to Dory with their voice, and she listens, thinks, remembers, and acts — collecting resources, crafting items, building structures, and holding a natural conversation the whole time.

Under the hood, Dory is a six-service system with a web frontend, persona creation tools, voice interaction, and in-game AI agents. The architecture cleanly separates how the player communicates (voice) from what happens in the game (bot actions), making it straightforward to swap out components, add new games, or integrate into existing projects.

• 🎨 Custom Persona Creation — Build unique AI companions through an interactive chat interface. Define personality, appearance, gaming style, and voice — all through natural conversation.
• 🌐 Web Application — Beautiful Next.js frontend with seamless mode transitions between Gatekeeper Chat, Persona Builder, and Gaming Hub. State machine architecture enables smooth handoffs between agents.
• 🎤 Voice Conversation — Talk naturally using your microphone. Dory listens (Deepgram STT), thinks (LLM), and speaks back (ElevenLabs TTS) in real time via LiveKit.
• 🎮 In-Game Actions — Follow players, collect resources, craft items, manage inventory, fight mobs, and navigate the world using 30+ tool-calling capabilities.
• 🧠 Multi-Step Planning — Complex requests like "gather wood, craft planks, and make me a crafting table" are automatically broken into a plan and executed step by step.
• 🏗️ AI Structure Generation — Say "build me a medieval castle" and watch it materialize block by block. An LLM generates JavaScript build code, a sandbox executes it, and blocks are placed progressively in the live world.
• 💾 Persistent Memory — Dory remembers your preferences, past conversations, and goals across sessions using MongoDB-backed episodic, semantic, and procedural memory.
• ⚡ Event-Driven Awareness — Game events (damage, player joins, task completion) are prioritized and forwarded to the voice agent. Dory reacts to critical events immediately — if she takes fatal damage, she'll tell you about it mid-sentence.

• Prerequisites
• Quick Start
  • 1. Clone and install
  • 2. Configure environment
  • 3. Start MongoDB
  • 4. Initialize Database
  • 5. Start everything
  • 6. Connect and play
• Architecture
  • Reasoning Engine
• Dory AI Web Application
  • Three-Screen Design
  • User Flow
  • State Machine
  • Mode Switching
• Capabilities
  • Voice Agent
  • Game Agent
  • Memory System
  • AI Structure Generation
• Testing Tools
• Project Structure
• Available Scripts
• Tech Stack
• API Reference
• Troubleshooting
• For Game Developers
• License

In case you have trouble setting up the project, you can always watch our Walkthrough Demo

git clone https://github.com/your-org/dory.git
cd dory
pnpm install

Each service has its own .env. Copy the examples:

cp services/game-agent/.env.example services/game-agent/.env
cp services/voice-agent/.env.example services/voice-agent/.env
cp services/gatekeeper-agent/.env.example services/gatekeeper-agent/.env
cp services/persona-builder-agent/.env.example services/persona-builder-agent/.env
cp apps/web/.env.example apps/web/.env

GAME_AGENT_PORT=3000

# Minecraft server
MINECRAFT_HOST=localhost
MINECRAFT_PORT=25565
MINECRAFT_AUTH_MODE=offline

# LLM — pick one provider: mistral, openai, or anthropic
LLM_PROVIDER=openai
OPENAI_API_KEY=sk-...

# MongoDB (memory system)
MONGODB_URI=mongodb://localhost:27017/dory

# AI Structure Builder (optional but recommended)
# Uses a separate, more capable model for generating build code
BUILDER_LLM_PROVIDER=openai
BUILDER_LLM_MODEL=gpt-4o

PORT=4001

# LiveKit (required)
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...

# Deepgram STT
DEEPGRAM_API_KEY=...

# ElevenLabs TTS
ELEVEN_API_KEY=...

# LLM for voice conversation (must support OpenAI-compatible function calling)
LLM_API_KEY=sk-...
LLM_MODEL=gpt-4o-mini

# Game Agent URL (A2A connection)
GAME_AGENT_URL=http://localhost:3000

Tip — Budget-friendly voice LLM: You can use Groq with Qwen for the voice agent's LLM at no cost:

LLM_API_KEY=gsk_...
LLM_BASE_URL=https://api.groq.com/openai/v1
LLM_MODEL=qwen/qwen3-32b

Avoid llama models on Groq — they don't call tools reliably.

PORT=4002

# Persona Builder Service URL
PERSONA_BUILDER_URL=http://localhost:4003

# Groq API Key (required for LLM)
GROQ_API_KEY=gsk_...

PORT=4003

# MongoDB Connection
MONGODB_URI=mongodb://localhost:27017/dory

# LLM - OpenAI (or OpenRouter)
OPENAI_API_KEY=sk-...
# Optional: Set to 'https://openrouter.ai/api/v1' for OpenRouter
# OPENAI_BASE_URL=https://openrouter.ai/api/v1

# Image Generation - Gemini
GEMINI_API_KEY=...

# Cloudflare R2 Storage
R2_ACCOUNT_ID=...
R2_ACCESS_KEY_ID=...
R2_SECRET_ACCESS_KEY=...
R2_BUCKET_NAME=personas
R2_PUBLIC_URL=https://your-r2-bucket.r2.dev

# ElevenLabs (optional, for voice matching)
# ELEVEN_API_KEY=...

All variables are optional — defaults work for local development:

# Optional: Override default agent URLs if needed
# NEXT_PUBLIC_GATEKEEPER_WS_URL=ws://localhost:4002/ws
# NEXT_PUBLIC_PERSONA_WS_URL=ws://localhost:4003/ws
# NEXT_PUBLIC_VOICE_AGENT_WS_URL=ws://localhost:4001/ws
# NEXT_PUBLIC_VOICE_AGENT_API_URL=http://localhost:4001
# NEXT_PUBLIC_LIVEKIT_URL=wss://your-project.livekit.cloud

Make sure Docker Desktop is running:

docker compose up -d

Verify it's running:

docker ps   # should show "dory-mongo" container

Memory is optional — the bot works without it, but you won't get session summaries or player profiles.

Push the Prisma schema for the Persona Builder Agent:

cd services/persona-builder-agent
npx prisma db push
cd ../..

pnpm dev

This builds the shared package, then starts all services (web, gatekeeper, persona-builder, voice, game) with hot-reload via Turborepo.

Or start services individually:

pnpm dev:web         # Web app only (port 3001)
pnpm dev:gatekeeper  # Gatekeeper agent only (port 4002)
pnpm dev:persona     # Persona builder agent only (port 4003)
pnpm dev:game        # Game agent only (port 3000)
pnpm dev:voice       # Voice agent only (port 4001)

1. Start your Minecraft server — Java Edition, offline mode recommended for local testing.

2. Open the web application at http://localhost:3001 in your browser. This is the primary way to use Dory — you'll land on the Gatekeeper Chat interface.

3. Using the web app:

   • Create a persona: Click "Create New Persona" or say "I want to create a persona" → Follow the interactive persona builder flow
   • Play with a persona: Click "Let's Play" or say "I want to play" → Select a persona → Gaming Hub opens with voice controls
   • Talk to Dory in Gaming Hub:
     • "Join the game" — connects the bot to Minecraft
     • "Follow me" — bot follows your player
     • "Collect some oak wood" — gathers resources
     • "Craft a crafting table" — crafts items
     • "Build a pillar where I'm looking" — places blocks at your crosshair
     • "Build me a medieval castle" — AI generates and places the structure block by block

4. Alternative interfaces:

   • Text console — open services/game-agent/test-console.html for a browser-based chat
   • WebSocket — connect to ws://localhost:3000/ws for a raw command interface
   • Memory dashboard — open services/game-agent/test-memory.html to inspect stored memories, summaries, and player profile in real time

Important for AI structure generation: The bot must have operator permissions in the Minecraft server. Run /op <bot_username> in the server console before asking Dory to generate structures.

Dory is a six-service system that orchestrates user interaction, persona creation, voice communication, and game control:

flowchart LR
    subgraph frontend["Web App - Port 3001"]
        NextJS["Next.js Frontend"]
    end
    subgraph gatekeeper["Gatekeeper - Port 4002"]
        GK["Stone Golem Agent"]
    end
    subgraph persona["Persona Builder - Port 4003"]
        PB["Persona Architect Agent"]
    end
    subgraph voice["Voice Agent - Port 4001"]
        VA["LiveKit Voice Pipeline"]
    end
    subgraph game["Game Agent - Port 3000"]
        GA["Minecraft Bot"]
    end

    NextJS -->|WebSocket| GK
    NextJS -->|WebSocket| PB
    NextJS -->|WebRTC/LiveKit| VA
    VA -->|HTTP/A2A| GA
    GA --> Minecraft
    PB --> MongoDB
    GA --> MongoDB

When a player makes a complex request, the game agent's reasoning engine decomposes it into a step-by-step plan and executes each step sequentially:

If a step fails (e.g., missing materials), the engine re-plans automatically — adapting the approach based on what actually happened:

The web application (apps/web) is a Next.js frontend that serves as the single entry point for users. It uses a state machine pattern (StateMachine + WebSocketManager) to manage seamless transitions between three application modes: GATEKEEPER, PERSONA_BUILDER, and GAMER_AGENT.

• Gatekeeper Chat (landing page): Expandable chat UI connected to the Gatekeeper Agent via WebSocket. Users land here and see a hero section with CTAs ("Create New Persona" / "Let's Play"). The Gatekeeper — a stone golem personality — guides users to either create personas or select existing ones to play games.

• Persona Builder: 3-column layout (avatar preview, trait cards, chat) connected to the Persona Builder Agent via WebSocket. Real-time persona updates via persona_update / operation_status messages. Users interactively build personas through a conversational flow: species → visual details → name → avatar generation → personality → gaming style → voice selection → save.

• Gaming Hub: 2-column layout (companion sidebar with voice controls, chat transcript) connected to the Voice Agent via LiveKit WebRTC. Supports both voice and text communication. The companion sidebar shows the active persona's avatar, voice controls (mic mute, companion mute), game status, and chat history.

1. User opens web app → Lands on Gatekeeper Chat (hero view with CTAs)
2. "I want to play" → Gatekeeper fetches popular personas → User picks one → Backend sends mode_change to GAMER_AGENT → Gaming Hub loads with LiveKit voice connection
3. "I want to create a persona" → Backend sends mode_change to PERSONA_BUILDER → Persona Builder chat interface → User creates persona through conversation → Persona saved → Option to play with new persona or return to Gatekeeper

The frontend StateMachine orchestrates mode transitions driven by mode_change WebSocket messages from the backend:

stateDiagram-v2
    [*] --> GATEKEEPER: User opens app
    GATEKEEPER --> PERSONA_BUILDER: "Create persona" -> mode_change
    GATEKEEPER --> GAMER_AGENT: "Play" -> select persona -> mode_change
    PERSONA_BUILDER --> GATEKEEPER: Back / persona saved -> mode_change
    PERSONA_BUILDER --> GAMER_AGENT: Persona saved -> play -> mode_change
    GAMER_AGENT --> GATEKEEPER: End session -> page reload

Each transition: (1) disconnects current WebSocket, (2) generates/reuses sessionId for the target mode, (3) connects to the new agent's WebSocket passing conversationSummary for context continuity, (4) the UI renders the corresponding screen (GatekeeperChat / PersonaBuilder / GamingHub).

The backend agents (Gatekeeper, Persona Builder) send mode_change WebSocket messages when they detect user intent. The frontend StateMachine handles:

• Disconnecting from the current agent's WebSocket
• Connecting to the new agent's WebSocket with the appropriate sessionId
• Passing conversationSummary (if available) to preserve context across mode transitions
• Updating the UI to render the correct screen component

This architecture enables seamless handoffs between agents while maintaining conversation context, creating a unified experience despite multiple backend services.

Dory builds a persistent profile of each player over time — remembering preferences, goals, and shared history across sessions.

The builder module generates Minecraft structures from natural language:

1. Player says "build me a house"
2. Voice agent forwards the command to the game agent
3. Game agent calculates a build position (in front of the player, snapped to ground)
4. A dedicated LLM generates JavaScript build code using safeSetBlock / safeFill helpers
5. Code executes in a Node.js vm sandbox — no world modifications, just a list of block placements
6. Blocks are placed progressively via /setblock commands with configurable delays
7. On completion, a critical event fires and Dory announces "Your structure is finished!"

Supports cancellation mid-build ("stop building"), hollow/walkable interiors, and validates all blocks against a comprehensive block ID list.

During development, several browser-based tools are available for testing without the voice pipeline:

dory/
├── package.json                # Root scripts (pnpm dev, build, etc.)
├── turbo.json                  # Turborepo task configuration
├── pnpm-workspace.yaml         # Workspace definition
├── docker-compose.yml          # MongoDB service
│
├── packages/
│   └── shared/                 # @dory/shared — types, logger, utilities
│       └── src/
│           ├── types/          # Session, Minecraft, Agent interfaces
│           └── utils/          # Logger, sleep, retry helpers
│
├── apps/
│   └── web/                    # @dory/web — Next.js frontend
│       └── src/
│           ├── components/     # UI components (buttons, dialogs, inputs)
│           ├── config/         # Agent URL configuration
│           ├── contexts/       # UnifiedAgentContext (state management)
│           ├── hooks/          # useLiveKitSession, useVoiceAgent
│           ├── pages/          # Next.js pages (_app, index)
│           ├── screens/         # Main screens (home with GatekeeperChat, PersonaBuilder, GamingHub)
│           ├── services/       # StateMachine, WebSocketManager, chat persistence
│           ├── theme/          # Styled-components theme (colors, animations)
│           └── types/          # Agent types (AppMode, WSMessage, etc.)
│
└── services/
    ├── gatekeeper-agent/       # @dory/gatekeeper-agent
    │   └── src/
    │       ├── agent/          # Gatekeeper agent logic + prompt
    │       ├── config/         # Environment configuration
    │       ├── services/       # Session management, WebSocket server
    │       └── tools/          # Gatekeeper tools (fetchPopularPersonas, changeMode)
    │
    ├── persona-builder-agent/   # @dory/persona-builder-agent
    │   └── src/
    │       ├── agent/          # Persona builder agent logic
    │       ├── config/         # Environment configuration
    │       ├── db/             # Prisma client
    │       ├── services/       # WebSocket server, persona operations
    │       ├── tools/          # Persona builder tools (savePersona, generateAvatar)
    │       └── types/          # Persona data types
    │
    ├── game-agent/             # @dory/game-agent
    │   └── src/
    │       ├── a2a/            # Agent card + A2A message handler
    │       ├── actions/        # Building, vision, movement, helpers
    │       ├── agent/          # Message handler + system prompt
    │       ├── bot/            # Mineflayer bot wrapper + session manager
    │       ├── builder/        # AI structure generation (LLM → sandbox → placer)
    │       ├── events/         # Event bus, Minecraft listener, A2A forwarder
    │       ├── llm/            # Multi-provider LLM client (OpenAI/Anthropic/Mistral)
    │       ├── memory/         # MongoDB memory system (episodic/semantic/procedural)
    │       ├── planning/       # Multi-step plan engine
    │       └── tools/          # Tool registry (30+ tools) + executor
    │
    └── voice-agent/            # @dory/voice-agent
        └── src/
            ├── agent/          # LiveKit conversational agent + persona prompt builder
            ├── clients/        # Persona client (fetches persona data from persona-builder)
            ├── events/         # Event store + fetcher (polls game events)
            ├── routes/          # Room token generation
            ├── services/       # Context service (memory sync)
            ├── tools/          # HTTP tools for game agent control
            └── utils/          # Logger

pnpm build:shared   # Rebuild shared package first
pnpm dev             # Then retry

• Verify your Minecraft server is running and set to offline mode (online-mode=false in server.properties)
• Default config is localhost:25565 — adjust in the session creation call if different
• Make sure the bot username isn't already online

• Verify DEEPGRAM_API_KEY and ELEVEN_API_KEY are set in services/voice-agent/.env
• Make sure your browser has granted microphone permissions
• Check the browser console for WebRTC errors

• Create a bot session first — say "join the game" or use the test console
• Verify game agent is running on port 3000
• Check GAME_AGENT_URL=http://localhost:3000 in voice agent .env

• Make sure the bot has operator permissions: run /op <bot_username> in the Minecraft server console
• If using GPT-5 or o-series models, the provider automatically uses max_completion_tokens instead of max_tokens
• Check that OPENAI_API_KEY is set (required even if your main LLM provider is Mistral/Anthropic, if you use OpenAI for the builder)

• Make sure Docker Desktop is running, then docker compose up -d
• Check with docker ps — you should see dory-mongo
• Verify MONGODB_URI=mongodb://localhost:27017/dory in services/game-agent/.env
• Memory is optional — the bot works without it, you just won't get session summaries or player profiles

• Known issue with Silero VAD native runtime during worker shutdown
• Usually harmless — the worker restarts automatically
• If it persists, restart with pnpm dev:voice

Dory's architecture is designed to be modular and extensible:

• Add new tools — Define a tool in tools/registry.ts, implement it in tools/executor.ts. The LLM discovers tools automatically via function calling.
• Swap LLM providers — Change LLM_PROVIDER in .env. OpenAI, Anthropic, and Mistral work out of the box. Add new providers by implementing the LLMProvider interface.
• Change the voice — Swap TTS_VOICE_ID in the voice agent config, or replace ElevenLabs with another TTS provider.
• Replace the game — The A2A protocol is game-agnostic. Replace the mineflayer bot with any game's API and the voice agent still works.
• Add memory types — Extend the memory system with new document types in memory/types.ts.

The A2A protocol between agents is simple HTTP JSON — no proprietary SDKs or complex integrations required.

MIT

Built for Global AI Game Hack 2026