Skip to content

feat: dual-modal WASM browser pose estimation demo (ADR-058)#244

Open
ruvnet wants to merge 5 commits intomainfrom
feat/ruvector-wasm-browser-example
Open

feat: dual-modal WASM browser pose estimation demo (ADR-058)#244
ruvnet wants to merge 5 commits intomainfrom
feat/ruvector-wasm-browser-example

Conversation

@ruvnet
Copy link
Copy Markdown
Owner

@ruvnet ruvnet commented Mar 12, 2026

Summary

  • Dual-modal pose estimation demo combining live webcam video (CNN) with WiFi CSI in the browser
  • ADR-058 documents the architecture: ruvector-cnn-wasm for CNN inference, attention-weighted embedding fusion, three operating modes (Dual/Video-only/CSI-only)
  • Pre-built WASM package (~52KB) from ruvector-cnn-wasm with JS fallback for immediate deployment
  • 8 JS modules: CNN embedder, video capture, CSI simulator, fusion engine, pose decoder, canvas renderer
  • GitHub Pages deployment live alongside Observatory at /pose-fusion.html

Architecture

  • Conv2D 3×3 → BatchNorm → ReLU → GlobalAvgPool → Linear → L2 Normalize (mirrors ruvector-cnn)
  • Per-dimension attention fusion with dynamic confidence gating
  • 17 COCO keypoints decoded from fused 512-dim embedding
  • CSI pseudo-image encoding (amplitude/phase/temporal → 224×224)

Files Added

  • docs/adr/ADR-058-ruvector-wasm-browser-pose-example.md — Architecture decision record
  • ui/pose-fusion.html — Main demo page (dark theme, responsive)
  • ui/pose-fusion/js/ — 7 JS modules (main, cnn-embedder, video-capture, csi-simulator, fusion-engine, pose-decoder, canvas-renderer)
  • ui/pose-fusion/css/style.css — Full dark theme CSS
  • ui/pose-fusion/pkg/ — Pre-built WASM package
  • ui/pose-fusion/build.sh — WASM rebuild script
  • ui/index.html — Updated with Pose Fusion nav link

Test plan

  • Open pose-fusion.html locally — verify webcam prompt, skeleton rendering, CSI heatmap
  • Toggle between Dual/Video/CSI modes — verify each works independently
  • Check fusion confidence bars update in real-time
  • Verify WASM auto-detection (loads JS fallback if WASM unavailable)
  • Verify GitHub Pages deployment at https://ruvnet.github.io/RuView/pose-fusion.html

🤖 Generated with claude-flow

ruvnet added 5 commits March 12, 2026 14:26
Live webcam video + WiFi CSI fusion for real-time pose estimation.
Two parallel CNN pipelines (ruvector-cnn-wasm) with attention-weighted
fusion and dynamic confidence gating. Three modes: Dual, Video-only,
CSI-only. Includes pre-built WASM package (~52KB) for browser deployment.

- ADR-058: Dual-modal architecture design
- ui/pose-fusion.html: Main demo page with dark theme UI
- 7 JS modules: video-capture, csi-simulator, cnn-embedder, fusion-engine,
  pose-decoder, canvas-renderer, main orchestrator
- Pre-built ruvector-cnn-wasm WASM package for browser
- CSI heatmap, embedding space visualization, latency metrics
- WebSocket support for live ESP32 CSI data
- Navigation link added to main dashboard

Co-Authored-By: claude-flow <ruv@ruv.net>
- Pose decoder now uses per-cell motion grid to track actual arm/head
  positions — raising arms moves the skeleton's arms, head follows
  lateral movement
- Motion grid (10x8 cells) tracks intensity per body zone: head,
  left/right arm upper/mid, legs
- Through-wall mode: when person exits frame, CSI maintains presence
  with slow decay (~10s) and skeleton drifts in exit direction
- CSI simulator persists sensing after video loss, ghost pose renders
  with decreasing confidence
- Reduced temporal smoothing (0.45) for faster response to movement

Co-Authored-By: claude-flow <ruv@ruv.net>
- Remove fixed aspect-ratio and max-height from video panel so it
  fills the available viewport space without scrolling
- Grid uses 1fr row for content area, overflow:hidden on main grid
- Fix WASM path: resolve relative to JS module file using import.meta.url
  instead of hardcoded ./pkg/ which resolved incorrectly on gh-pages
- Responsive: mobile still gets aspect-ratio constraint

Co-Authored-By: claude-flow <ruv@ruv.net>
- Add auto-connect to local sensing server WebSocket (ws://localhost:8765)
- Demo shows "Live ESP32" when connected to real CSI data
- Add build_firmware.ps1 for native Windows ESP-IDF builds (no Docker)
- Add read_serial.ps1 for ESP32 serial monitor

Pipeline: ESP32 → UDP:5005 → sensing-server → WS:8765 → browser demo

Co-Authored-By: claude-flow <ruv@ruv.net>
…inks

- ADR-059: Documents end-to-end ESP32 → sensing server → browser pipeline
- README: Add dual-modal pose fusion demo link, update ADR count to 49
- References issue #245

Co-Authored-By: claude-flow <ruv@ruv.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant