Skip to content

rexbrahh/uwscrape

Repository files navigation

uwscrape

Advisory, source-backed academic-planning system for University of Waterloo students.

uwscrape is a course-first, anonymous, offline-indexed planning tool. It does not replace Quest, official advisors, or UWFlow — it is an exploratory layer on top of the public Waterloo Kuali catalog, with academic truth computed server-side from a release-gated SQLite index.

Important

Advisory only. This tool is not affiliated with the University of Waterloo. Always confirm degree requirements with Quest and an official academic advisor before making registration decisions.

What it does

Two primary modes (per ADR 0025):

  • Kanban — term-based course planning board (the editable student state).
  • Canva — full-screen universal course graph projection with the Kanban state layered as overlay.

A floating Advisory widget (plus matching backend query surface) answers:

  • "What am I missing for course X?"
  • "What am I missing for credential Y?"
  • "Which alternative paths satisfy this gap?"
  • "What does taking this course unlock?"

Every answer carries one of satisfied | not_satisfied | partial | unknown | conflict | not_applicable, plus an explanation tree and source references back to the Kuali catalog.

Architecture

Layer Stack
Scraper / index pipeline Go CLI (cmd/uwscrape) → published SQLite artifact (offline, never run at request time)
Backend runtime Go HTTP service (cmd/uwscrape-server) — fail-closed startup, /api/v1 envelopes, typed graph views, direct-evaluator academic engine
State store Separate SQLite, anonymous bearer tokens (≥256 bits), HMAC verifier server-side, hard delete
Frontend SvelteKit + TypeScript strict, Node adapter, Zod transport validation
Atlas renderer Three.js (WebGL baseline) + Rust/WASM layout worker + non-canvas fallback

See .planning/PROJECT.md and .planning/intel/decisions.md for the full 26-ADR set governing the project.

Status

Phase P7 — Course-First Product Correction Wave (in progress).

Phases P0–P6 (M0–M9 milestones) are complete. The current candidate index covers the Faculty of Mathematics plus engineering support subjects (~1,925 course listings, ~137 credentials, ~9,182 requirement conditions). See .planning/ROADMAP.md for the per-phase status.

A reviewed, publishable Math index artifact and a full bounded path solver are tracked as Phase P9 (Public-Announcement Readiness) blockers in .planning/STATE.mdnot yet started. This repo is currently suitable for internal demo and soft-beta use, not a public V1 announcement.

Scope caveat — course count vs. Waterloo's public claims

The local Math+engineering-support candidate counts ~1,925 course listings. Waterloo's public materials cite a higher Faculty-of-Mathematics figure (~500+ to close-to-600 Math-subject courses alone). The two numbers count different populations — the local index is the complete current-undergraduate Kuali catalog Faculty-of-Mathematics group plus support-subject closure used by Math/Engineering requirements, not the union of every Math course ever offered.

Full reconciliation: docs/qa/math-course-count-reconciliation.md.

Requirements

  • Go 1.25+
  • Node 22+ and pnpm 11+ (via corepack)
  • Rust stable toolchain with wasm32-unknown-unknown target
rustup target add wasm32-unknown-unknown
corepack enable

Build & test

# Offline-safe check suite: lint-docs + Go tests + Rust tests + WASM build + frontend check + vitest
make check

# Frontend smoke e2e (boots backend + frontend on ephemeral ports)
make test-e2e

# Per-milestone targets (M2 first/second half, M3 first/second half, ...).
make test-m4-second-half

CI runs the same gates on every push / PR — see .github/workflows/check.yml.

Run the demo

make dev-demo

Boots cmd/uwscrape-server plus the SvelteKit frontend against the most complete locally-available index (prefers .dev/published/math-engineering-support, then .dev/math, then the small fixture). Frontend defaults to http://127.0.0.1:4173, backend to 127.0.0.1:18081.

A reviewed publishable index is not yet bundled with the repo. To run against your own local index build, see Makefile targets parse-math-engineering-snapshot, build-math-engineering-index, and publish-math-engineering-support-index. The UWSCRAPE_ALLOW_LIVE_WATERLOO flag must be 1 to perform new fetches; default is 0.

Privacy posture

uwscrape is anonymous-only. There is no account, email, password, or student-number surface. State persists via opaque bearer tokens transported only via the Authorization header. The backend stores an HMAC verifier with a rotatable key version; tokens themselves are never logged. DELETE /api/v1/state/current is a hard delete and leaves only a minimal tombstone.

See SECURITY.md for the full threat model.

Repository layout

cmd/                  Go binaries (offline scraper + runtime server)
crates/               Rust workspace (atlas-layout-wasm)
internal/             Go packages (api, server, evaluate, solver, catalogstore, ...)
web/                  SvelteKit frontend
scripts/              Dev + e2e helpers
docs/                 ADRs, specs, QA acceptance notes, audits
.planning/            Project-management artifacts (PROJECT.md, ROADMAP.md, STATE.md, ...)
testdata/             Fixture inputs for the parse/build pipeline

Contributing

External contributions are not currently being accepted; the project is pre-V1 and the contract surface is still moving. The relevant decision records live under docs/ADRs/ and the open-work backlog is in .planning/POLISH-MAP.md.

License

MIT — see LICENSE. The advisory-only notice in LICENSE is binding.

About

uwaterloo course requirement trees turned into constraint satisfiability problems

Resources

License

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors