Maya combines a local LLM, real-time facial emotion detection, text sentiment analysis, guided mental exercises, and long-term conversational memory (RAG) to provide empathetic, context-aware support — all without any data ever leaving the device.

"Technology should care for people, not exploit them." — Maya was built on this principle.

1. Overview
2. Key Features
3. System Architecture
4. Project Structure
5. Hardware Requirements
6. LLM Model Research & Benchmarking
7. Software Stack & Technology Choices
8. Complete Processing Pipeline
9. Guided Mental Exercises
10. Web Interface & UI Design
11. Installation — Raspberry Pi 5
12. Installation — Windows (Development)
13. Running the Application
14. Configuration Reference
15. Module Deep Dive
16. API Reference
17. Utility Scripts
18. Troubleshooting
19. Privacy & Security
20. Future Scope
21. License

Maya is an AI-powered wellbeing companion that listens, understands, and responds with empathy. It runs 100% offline on a Raspberry Pi 5, ensuring complete privacy. The system uses:

• A local LLM (Microsoft Phi-3 Mini via Ollama) for natural conversation
• VADER sentiment analysis to understand the emotional tone of text
• FER (Facial Expression Recognition) with a webcam for real-time facial emotion detection
• ChromaDB vector database for long-term conversational memory (RAG)
• An Emotion Engine that fuses text sentiment, facial emotion, and historical patterns into a unified mental state model
• A Guided Exercise System that detects stress and offers evidence-based 30-second exercises

The companion is named Maya and provides brief, warm, supportive responses tailored to the user's current emotional state.

┌─────────────────────────────────────────────────────────────────┐
│                         User Interfaces                         │
│  ┌──────────────────────┐    ┌───────────────────────────────┐  │
│  │   Terminal CLI        │    │   Flask Web App (port 5000)   │  │
│  │   (main.py)           │    │   (web_app.py + index.html)   │  │
│  └──────────┬───────────┘    └──────────────┬────────────────┘  │
└─────────────┼───────────────────────────────┼───────────────────┘
              │                               │
              ▼                               ▼
┌─────────────────────────────────────────────────────────────────┐
│                       AgentBrain (brain.py)                      │
│          Central orchestrator — coordinates all modules          │
│                                                                  │
│  ┌─────────────┐ ┌──────────────┐ ┌───────────┐ ┌────────────┐ │
│  │ LLMClient   │ │ Sentiment    │ │ Emotion   │ │ Conversa-  │ │
│  │ (llm.py)    │ │ Analyzer     │ │ Engine    │ │ tion       │ │
│  │             │ │ (sentiment.  │ │ (emotion. │ │ Memory     │ │
│  │ Ollama API  │ │  py)         │ │  py)      │ │ (memory.py)│ │
│  │ phi3:mini   │ │ VADER        │ │ Fusion    │ │ ChromaDB   │ │
│  └─────────────┘ └──────────────┘ └───────────┘ └────────────┘ │
└─────────────────────────────────────────────────────────────────┘
              │                                        │
              ▼                                        ▼
┌──────────────────────┐               ┌──────────────────────────┐
│   Ollama Server      │               │  Camera Module           │
│   (localhost:11434)  │               │  (camera.py)             │
│   LLM Inference      │               │  OpenCV + FER            │
└──────────────────────┘               └──────────────────────────┘

wellbeing_ai/
│
├── main.py                    # Terminal CLI entry point
├── web_app.py                 # Flask web application (REST API + SSE streaming)
├── benchmark_models.py        # LLM model benchmark script (tests all Ollama models)
├── requirements.txt           # Python dependencies with version pins
├── setup_rpi.sh               # Automated setup script for Raspberry Pi (Linux/Bash)
├── setup_rpi.bat              # Automated setup script for Windows development
├── patch_fer.py               # Patches FER library to fix moviepy import on RPi
├── reset_memory.py            # Utility: clear all stored conversations
├── view_memory.py             # Utility: view stored conversations
├── test_camera.py             # Camera & FER diagnostic test script
├── benchmark_results.json     # Full benchmark data (auto-generated)
│
├── agent/                     # Core AI agent modules
│   ├── __init__.py
│   ├── brain.py               # AgentBrain — central orchestrator (7-step pipeline)
│   ├── llm.py                 # LLMClient — Ollama REST API integration
│   ├── sentiment.py           # SentimentAnalyzer — VADER-based text sentiment
│   ├── emotion.py             # EmotionEngine — multimodal emotion fusion
│   ├── memory.py              # ConversationMemory — ChromaDB RAG store
│   └── exercises.py           # ExerciseManager — 7 guided mental exercises
│
├── config/                    # Configuration
│   ├── __init__.py
│   └── config.py              # All tuneable parameters (LLM, camera, paths, etc.)
│
├── interface/                 # Hardware abstraction layers
│   ├── __init__.py
│   ├── camera.py              # BaseCamera / WebcamCamera — webcam + FER
│   └── display.py             # BaseDisplay / TerminalDisplay — output rendering
│
├── templates/                 # Flask HTML templates
│   └── index.html             # Web chat interface (glassmorphism UI, ~1100 lines)
│
├── data/                      # Runtime data (auto-created, gitignored)
│   └── memory/                # ChromaDB persistent vector storage
│
└── .gitignore                 # Git ignore rules

• Windows, macOS, or Linux with Python 3.10+
• Webcam (for testing emotion detection)
• 8GB+ RAM recommended

Selecting the right LLM is critical for a wellbeing companion running on resource-constrained hardware. The model must:

1. Fit in memory — RPi 5 has 4–8GB RAM shared between OS, app, and model
2. Respond quickly — Users in emotional distress need timely responses (<30s)
3. Show empathy — Generic/robotic responses are harmful in a wellbeing context
4. Follow instructions — Must stay in character as Maya, keep responses brief, not hallucinate
5. Run offline — Must be available via Ollama for local inference

We benchmarked 10 locally available Ollama models spanning a wide range of sizes and architectures:

Test Environment: Settings identical to production config — temperature=0.3, max_tokens=60, num_ctx=1024, num_thread=4. Each model was tested against 5 diverse wellbeing conversation prompts (50 total inferences).

Test Prompts:

Quality Scoring (0–10 weighted):

Legend: TTFT = Time To First Token | Tok/s = Tokens Per Second

qwen2.5         ██████████████████░░ 9.13   ✗ Too large (4.4GB)
llama3.1        █████████████████░░░ 8.65   ✗ Too large (4.6GB)
survival-gemma2 ████████████████░░░░ 8.59   ✓ GOOD (1.6GB)
phi3:mini       ████████████████░░░░ 8.56   ✓ SELECTED (2.0GB)
tinyllama       ████████████████░░░░ 8.41   ✓ Fast but less empathetic
survival-gemma3 ████████████████░░░░ 8.38   ~ Slow first token
mistral         ████████████████░░░░ 8.35   ✗ Too large (4.1GB)
gemma:2b        ███████████████░░░░░ 8.11   ✓ Acceptable fallback
survival-gemma  ███████████████░░░░░ 7.96   ✓ Acceptable
my-survival     ██████████████░░░░░░ 7.45   ✗ Poor empathy

tinyllama       ██░░░░░░░░░░░░░░░░░░  2.66s ← Fastest
my-survival     ██░░░░░░░░░░░░░░░░░░  2.74s
survival-gemma2 ███░░░░░░░░░░░░░░░░░  3.30s
phi3:mini       ███░░░░░░░░░░░░░░░░░  3.68s ← SELECTED
mistral         ████░░░░░░░░░░░░░░░░  4.53s
qwen2.5         █████░░░░░░░░░░░░░░░  5.68s
survival-gemma3 ████████░░░░░░░░░░░░  7.96s
llama3.1        █████████░░░░░░░░░░░  8.54s ← Slowest

tinyllama       ██████████████████░░ 17.88 ← Fastest generation
survival-gemma  █████████████████░░░ 14.57
gemma:2b        ██████████████░░░░░░ 14.00
survival-gemma2 █████████████░░░░░░░ 13.16
phi3:mini       ███████████░░░░░░░░░ 11.18 ← SELECTED
mistral         ██████████░░░░░░░░░░ 10.18
qwen2.5         ████████░░░░░░░░░░░░  8.48
llama3.1        ██████░░░░░░░░░░░░░░  6.29 ← Slowest

phi3:mini scored a perfect 10.00/10 on the Anxiety prompt — the most critical category for a wellbeing companion.

phi3:mini (9.55/10): "I'm truly sorry to hear that you're going through a tough time. Remember, it's okay to feel sad sometimes, but I believe in your strength too. If there's anything specific on your mind or if you need someone to talk to, I'm here for you."

qwen2.5 (9.55/10): "I'm sorry to hear you're feeling this way. It's okay to feel sad sometimes, and it's important to give yourself lots of kindness during these times. Would you like to talk about what's been on your mind?"

llama3.1 (8.50/10): "Sweetheart, I'm so sorry you're struggling. Would you like me to guide you through a gentle breathing exercise or simply listen while you share what's on your mind?"

my-survival (7.75/10): "Can you please suggest some ways I can make myself feel better?" ⚠️ This model responds as the user instead of the assistant — poor instruction following.

phi3:mini (10.00/10 — PERFECT): "It sounds like you're feeling anxious, which is completely normal before an important test. Remember to take breaks and get some rest tonight—you got this!"

qwen2.5 (9.40/10): "I understand how stressful it can be before an important exam. Remember, you've studied hard, and you've got this. Take some deep breaths and try to get a good night's rest tonight. You'll do great!"

After benchmarking all 10 models, Microsoft Phi-3 Mini (phi3:mini) was selected as the default:

Key Insights:

1. Best quality-to-size ratio — Achieves quality comparable to 7B models at nearly half the size
2. Perfect score on anxiety prompts — 10/10 on the most critical wellbeing category
3. Fits comfortably in 4GB RAM — Leaves room for OS, Python, ChromaDB, TensorFlow, and FER
4. Excellent instruction following — Stays in character as Maya, keeps responses brief
5. Natural empathy — Uses phrases like "I'm truly sorry", "it's okay to feel", "I believe in your strength"

Note: Users with 8GB RPi 5 may try qwen2.5:latest (LLM_MODEL=qwen2.5 in config) for higher quality at the cost of longer inference.

source venv/bin/activate   # Linux/RPi
# OR
venv\Scripts\activate      # Windows

python benchmark_models.py

The script discovers all local Ollama models, runs 5 prompts against each, measures latency/quality, prints a comparison table, and saves results to benchmark_results.json.

Why Phi-3 Mini? — It is one of the smallest high-quality LLMs that can run on RPi 5 hardware with acceptable latency. It handles empathetic conversation well within tight token budgets.

When a user sends a message, the AgentBrain.process() method orchestrates this pipeline:

User Input (text)
     │
     ▼
┌─────────────────────────────────┐
│ 1. SENTIMENT ANALYSIS           │
│    SentimentAnalyzer.analyze()  │
│    VADER scores the text →      │
│    label: positive/negative/    │
│           neutral               │
│    compound: -1.0 to +1.0      │
│    intensity: 0.0 to 1.0       │
└────────────┬────────────────────┘
             │
             ▼
┌─────────────────────────────────┐
│ 2. MEMORY RETRIEVAL (RAG)       │
│    ConversationMemory.retrieve()│
│    Query ChromaDB with user     │
│    input → retrieve top-2 most  │
│    semantically similar past    │
│    conversations                │
└────────────┬────────────────────┘
             │
             ▼
┌─────────────────────────────────┐
│ 3. EMOTION ENGINE UPDATE        │
│    EmotionEngine.update()       │
│    Fuses:                       │
│    • Text sentiment (VADER)     │
│    • Facial emotion (FER/cam)   │
│    • Historical sentiment avg   │
│    • Memory sentiment patterns  │
│    Outputs: MentalState object  │
│    (dominant_emotion, trend,    │
│     historical_avg)             │
└────────────┬────────────────────┘
             │
             ▼
┌─────────────────────────────────┐
│ 4. LLM RESPONSE GENERATION     │
│    LLMClient.chat()             │
│    Builds system prompt with:   │
│    • Base persona (Maya)        │
│    • Current user mood          │
│    • Emotional trend guidance   │
│    • Retrieved memory context   │
│    Sends last 4 messages +      │
│    system prompt to Ollama      │
│    Streams response tokens      │
└────────────┬────────────────────┘
             │
             ▼
┌─────────────────────────────────┐
│ 5. MEMORY STORAGE               │
│    ConversationMemory.store()   │
│    Stores in ChromaDB:          │
│    • user_message               │
│    • assistant_response         │
│    • sentiment_label & score    │
│    • dominant emotion           │
│    • timestamp                  │
│    Embedded for future RAG      │
└─────────────────────────────────┘

The Emotion Engine maintains a sliding window (10 turns) of sentiment scores and emotion labels to compute:

1. Dominant Emotion Resolution — If a facial emotion is detected (not unknown/None), it takes priority. Otherwise, text sentiment is mapped: positive→happy, negative→sad, neutral→neutral.
2. Historical Sentiment Average — Running mean of compound scores over the window.
3. Emotional Trend — Compares the average sentiment of the first half vs second half of the window. Difference >0.15 = "improving", <-0.15 = "declining", else "stable".
4. Memory Pattern Adjustment — If >60% of retrieved memories have negative sentiment, the historical average is shifted down by 0.1 to increase concern.

Browser polls /api/camera/snapshot every 2.5s
     │
     ▼
WebcamCamera.capture_snapshot_with_overlay()
     │
     ├── cv2.VideoCapture.read() → raw BGR frame
     ├── FER.detect_emotions(RGB frame) → bounding boxes + emotion scores
     ├── Draw green bounding box + emotion label on frame
     ├── cv2.imencode('.jpg') → JPEG bytes
     │
     ▼
Response: { image: base64 JPEG, emotion: "happy" }
     │
     ▼
Browser updates camera feed image + emotion emoji/label

Maya actively monitors the user's emotional state and offers guided mental exercises when stress is detected.

Exercises are offered when any of these conditions are met:

┌─────────────────────────────────────────────────────────────┐
│                  STRESS DETECTION ENGINE                     │
│                                                              │
│  Condition 1: hist_avg < -0.3      → Persistent low mood    │
│  Condition 2: trend="declining"    → Getting worse           │
│               AND hist_avg < -0.15                           │
│  Condition 3: sentiment < -0.5     → Very negative now       │
│  Condition 4: emotion ∈ {sad,      → Stress emotion          │
│               angry, fear, disgust}                          │
│               AND sentiment < -0.2                           │
│                                                              │
│  ANY condition true → needs_exercise = True                  │
│  Cooldown: min 5 turns between offers                        │
└─────────────────────────────────────────────────────────────┘

Stress Detected → Exercise Offer (SSE event)
       │
       ▼
┌──────────────────────────┐
│  Exercise Selection Card │
│  ┌────┐ ┌────┐ ┌────┐   │
│  │🌬️ │ │🍃 │ │🌍 │   │  ← User picks one
│  │Box │ │Calm│ │5421│   │
│  └────┘ └────┘ └────┘   │
│         [Skip]           │
└──────────┬───────────────┘
           │
           ▼
┌──────────────────────────┐
│  Step-by-Step Guide      │
│  "Breathe IN slowly..."  │
│  ┌──────────────────┐    │
│  │   ⏱️ 4 seconds    │   │  ← Timer countdown
│  │   ████████░░░░    │   │
│  └──────────────────┘    │
│  [Next Step] [Skip]      │
└──────────────────────────┘

The web interface is a single-page application built with a glassmorphism design language.

• Raspberry Pi OS (64-bit / Bookworm recommended)
• Python 3.10 or higher
• Internet connection (for initial setup only)

# Clone the repository
git clone <repository-url> ~/wellbeing_ai
cd ~/wellbeing_ai

# Run the setup script
chmod +x setup_rpi.sh
./setup_rpi.sh

The script will:

1. Create a Python virtual environment
2. Install all pip dependencies
3. Create the data/memory/ directory
4. Install Ollama (if not already installed)
5. Pull the phi3:mini model

# 1. Install system dependencies
sudo apt update && sudo apt install -y python3 python3-pip python3-venv libatlas-base-dev

# 2. Create & activate virtual environment
python3 -m venv venv
source venv/bin/activate

# 3. Upgrade pip
pip install --upgrade pip

# 4. Install Python dependencies
pip install -r requirements.txt

# 5. Patch FER for RPi (fixes moviepy import error)
python patch_fer.py

# 6. Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# 7. Start Ollama and pull the model
ollama serve &
sleep 3
ollama pull phi3:mini

# 8. Create data directory
mkdir -p data/memory

# 9. Enable camera (if using RPi Camera Module)
sudo raspi-config
# Navigate to: Interface Options → Camera → Enable
# Reboot if prompted

REM Clone the repository
git clone <repository-url>
cd wellbeing_ai

REM Run the setup script
setup_rpi.bat

Or manually:

python -m venv venv
venv\Scripts\activate
pip install --upgrade pip
pip install -r requirements.txt

REM Install Ollama from https://ollama.com/download/windows
ollama pull phi3:mini

source venv/bin/activate   # Linux/RPi
# OR
venv\Scripts\activate      # Windows

python main.py

This launches an interactive terminal session where you type messages and Maya responds. Camera emotion detection samples every 3 turns (configurable).

source venv/bin/activate   # Linux/RPi

python web_app.py

Then open in a browser:

• On the Pi: http://localhost:5000
• From another device on LAN: http://<raspberry-pi-ip>:5000

The web interface provides:

• A chat window with streaming responses
• Live camera feed with emotion overlay (bounding boxes + labels)
• Real-time emotion emoji display
• System status indicators (LLM, Memory, Camera)
• Conversation reset button

All configuration lives in config/config.py. Key settings:

These settings can be overridden via environment variables:

• OLLAMA_BASE_URL
• LLM_MODEL
• CAMERA_ENABLED (set to "true" / "false")
• DISPLAY_MODE

The central orchestrator. Initializes all subsystems and exposes:

• check_systems() → dict of subsystem health checks
• process(user_input, face_emotion, stream) → runs the full 5-step pipeline (sentiment → memory retrieval → emotion update → LLM generation → memory storage)
• Maintains a rolling conversation history (last 4 messages sent to LLM)
• Builds a dynamic system prompt that includes Maya's persona, current user mood, emotional trend guidance, and retrieved memory context

Communicates with Ollama's REST API:

• is_available() → checks if Ollama is running and the model is loaded (via /api/tags)
• generate(prompt, system) → single-shot generation via /api/generate (streaming internally)
• chat(messages, stream_output) → chat-style generation via /api/chat. When stream_output=True, returns a generator that yields tokens one by one for SSE streaming.
• Handles connection errors, timeouts, and server unavailability gracefully with error messages.

Wraps VADER for conversational sentiment analysis:

• analyze(text) → returns SentimentResult(label, compound, intensity, scores)
• Labels: positive (compound ≥ 0.05), negative (compound ≤ -0.05), neutral
• Intensity = absolute value of compound score (0.0–1.0)
• Zero dependencies beyond vaderSentiment, instant CPU execution

Maintains session-level emotional state:

• update(sentiment, face_emotion, retrieved_memories) → returns MentalState
• Tracks a sliding window of 10 sentiment scores and emotion labels
• Resolves dominant emotion (face > text mapping)
• Computes emotional trend (improving / declining / stable)
• Adjusts for long-term memory patterns (negative memory ratio)

ChromaDB-backed long-term memory with RAG:

• store(MemoryEntry) → stores a conversation turn with full metadata
• retrieve(query, top_k) → semantic similarity search, returns list[RetrievedMemory]
• Uses cosine distance in HNSW index
• Each entry stores: user message, assistant response, sentiment label/score, emotion, timestamp
• Documents are formatted as "The user said: ...\nMaya (the AI assistant) responded: ..." for embedding (prevents role confusion)

Manages guided mental exercises for stress relief:

• ExerciseManager — tracks exercise state and cooldown periods
• should_offer_exercise(current_turn, cooldown_turns) → checks if enough time has passed since last offer
• get_random_exercise() → selects a random exercise, avoiding repetition
• mark_exercise_offered(turn) → records when an exercise was offered
• format_exercise_offer() → generates the opt-in offer message
• format_exercise_steps(exercise) → formats exercise steps into a single message
• Contains 7 pre-built exercises: Box Breathing, Calming Breath, 5-4-3-2-1 Grounding, Quick Gratitude, Body Scan, Present Moment, Tension Release
• All exercises complete in under 30 seconds

Hardware abstraction for camera + emotion detection:

• BaseCamera — abstract base class defining the interface
• WebcamCamera — implementation using OpenCV + FER
• capture_emotion() → capture frame, detect dominant emotion, return label or None
• capture_frame() → return raw OpenCV frame
• capture_snapshot_with_overlay() → capture frame, detect emotion, draw bounding box + label, return (JPEG bytes, emotion label)
• is_available() → check if camera is accessible
• release() → release camera resources
• FER initialization is optional — if TensorFlow/FER is unavailable, the camera still works for frame capture without emotion detection

Hardware abstraction for output rendering:

• BaseDisplay — abstract base class
• TerminalDisplay — rich terminal output with text wrapping
• show_message(sender, message) — formatted chat output
• show_welcome() — welcome banner
• show_emotion(emotion) — emoji-mapped emotion display
• show_status(status) — system status messages
• clear() — clear terminal screen (platform-aware: cls on Windows, clear on Linux)
• Designed to be replaceable with an E-Ink display implementation

REST API + SSE streaming server:

• GET / — serves the chat interface (templates/index.html)
• GET /api/status — returns system health (LLM, memory, camera)
• POST /api/chat — synchronous chat endpoint (returns full response)
• POST /api/chat_stream — SSE streaming chat endpoint (yields tokens)
• GET /api/camera/snapshot — returns base64 JPEG with emotion overlay
• GET /api/camera/emotion — returns detected emotion label only
• POST /api/reset — resets conversation history

Single-page application with:

• Glassmorphism UI — frosted glass panels with gradient background
• Chat panel — message bubbles with avatars, typing indicator, auto-scroll
• Emotion sidebar — live camera feed, emotion emoji display, status indicators
• SSE streaming — reads token-by-token from /api/chat_stream using ReadableStream
• Emotion polling — polls /api/camera/snapshot every 2.5 seconds for live emotion updates
• Responsive — adapts to mobile screens with stacked layout
• Font — Google Quicksand for a friendly, approachable feel

The Flask web application exposes the following REST API endpoints:

The /api/chat_stream endpoint emits these Server-Sent Events:

Comprehensive LLM benchmark that tests all locally installed Ollama models against wellbeing conversation prompts. Measures latency, TTFT, tokens/second, and empathy quality scores. Results saved to benchmark_results.json.

python benchmark_models.py

Diagnostic script that tests the full camera pipeline in 4 steps:

1. Webcam access (OpenCV)
2. FER library import
3. FER detector initialization
4. Live emotion detection with confidence scores

python test_camera.py

Patches the FER library's classes.py to make the moviepy import optional. This fixes the "No module named 'moviepy.editor'" error that occurs on Raspberry Pi since moviepy is not needed for emotion detection.

python patch_fer.py

Deletes all stored conversations from ChromaDB. Asks for confirmation before proceeding.

python reset_memory.py

Displays all stored conversations with timestamps, sentiment labels, emotions, and message previews.

python view_memory.py

[!] Ollama is not running or model not found.

Fix:

ollama serve &          # Start Ollama server
ollama pull phi3:mini   # Download the model

⚠️ Camera enabled but not available

Fix (RPi Camera Module):

sudo raspi-config       # Interface Options → Camera → Enable
sudo reboot

Fix (USB Webcam):

ls /dev/video*          # Check available camera devices
# If your camera is at /dev/video1, set CAMERA_INDEX=1 in config/config.py

No module named 'moviepy.editor'

Fix:

python patch_fer.py

This is expected on RPi 5 CPU. Responses may take 30–120 seconds. To improve:

• Reduce LLM_MAX_TOKENS in config/config.py
• Reduce LLM_NUM_CTX (smaller context = faster)
• Ensure no other heavy processes are running

If you see numpy-related errors:

pip install "numpy>=1.24.0,<2.0.0"
pip install --force-reinstall "tensorflow>=2.15.0,<2.18.0"

On some RPi OS versions, the system SQLite may be too old:

pip install pysqlite3-binary

• All processing happens locally on the Raspberry Pi. No data is sent to any external server.
• Ollama runs locally — the LLM never contacts the internet during inference.
• ChromaDB stores data locally in data/memory/ on the device's filesystem.
• Camera frames are processed in-memory and never saved to disk (only emotion labels are stored).
• The web interface binds to 0.0.0.0:5000 — it is accessible on the local network. For additional security, configure a firewall to restrict access to trusted devices only.
• No API keys are required. No accounts, no cloud services, no telemetry.

This project is intended for personal and educational use.