Desktopia

Desktopia turns a cloud GPU machine into a full Linux desktop that you operate from an ordinary web browser tab. A desktop application — by default 3D Slicer — renders with real hardware GPU acceleration on the server; its screen is encoded as H.264 video and streamed to a single web page, and your keyboard and mouse travel back the other way over the same connection. The result is a low-latency "desktop in a browser tab," in the spirit of noVNC but built on modern web-streaming technology instead of the decades-old VNC (Virtual Network Computing) protocol.

It is meant to run on a rented cloud GPU — it was developed against vast.ai, a marketplace for renting GPU machines by the hour — but nothing in the design is tied to a particular host.

There are two data paths: video going out to the browser, and input coming back to the desktop.

3D Slicer and other apps
  │  draw with hardware OpenGL
  ▼
Xwayland  ──hosts the X11 apps──►  headless Wayland compositor   (on the GPU, no monitor)
                                          │
                                          ▼
            GStreamer:  capture the compositor's screen  →  H.264 encode (NVENC, or software)
                                          │
                                          ▼
                          split each video frame into QUIC packets
                                          │
                   WebTransport over QUIC ═══ public internet ═══► browser
                                          │
                                          ▼
                 reassemble each frame  →  WebCodecs decoder  →  <canvas> on the page

• The desktop runs on a headless Wayland compositor — a display server that renders entirely on the GPU with no physical monitor attached (built on Smithay via gst-wayland-display). It reaches the GPU through a DRM render node, which needs no special display privileges and no VirtualGL (a fragile shim that older remote-3D setups rely on).
• Traditional X11 applications such as 3D Slicer and Chrome run under Xwayland as clients of that compositor, which gives them hardware OpenGL acceleration.
• GStreamer (a media-pipeline framework) captures the compositor's output and encodes it to H.264 using NVIDIA's hardware encoder, NVENC, when the host allows it, or the x264 software encoder otherwise. (AV1, a newer codec, is available on recent GPUs.)
• Each encoded frame is split into QUIC packets and pushed to the browser with the WebTransport API (which runs over QUIC and HTTP/3). Unlike an ordinary TCP connection, one lost packet does not stall everything queued behind it (head-of-line blocking).
• The browser reassembles each frame and decodes it with the WebCodecs API straight into an HTML <canvas>, so the page itself controls how much it buffers and how much latency it accepts.
• Surviving packet loss. The encoder uses intra-refresh: instead of periodically sending one large, expensive keyframe, it refreshes a slice of the picture every frame. A dropped packet then causes a small, brief smear that heals within a few frames rather than a full freeze. The page notices when a frame is missing and asks for a fresh keyframe only when it truly needs one.

Mouse and keyboard events from the canvas are sent back over a reliable WebTransport stream and replayed into the desktop as synthetic events (through the X11 XTEST extension), so the remote applications receive them as ordinary input.

The server presents a short-lived, self-signed TLS (Transport Layer Security) certificate, and the browser trusts it by matching its hash (the WebTransport serverCertificateHashes option). This avoids needing a domain name or a certificate authority for a machine that may only exist for an hour.

Cloud "desktop in a browser" offerings usually ship one of two stacks: noVNC (an HTML5 client for the old VNC protocol) or Selkies (a WebRTC-based desktop streamer). Desktopia is a third point in the design space, optimized for low interactive latency on GPU/3D workloads and for being small enough to modify.

The core idea is latency and control: a direct QUIC connection plus a hand-driven WebCodecs decoder removes WebRTC's negotiation step and its opaque smoothing buffer, and drops the relay/signaling servers entirely — which matters for a short-lived rented machine reached at a bare public address. On this path, an interactive CT volume render has run at 80–90 frames per second, full-frame, end-to-end in the browser.

Honest trade-offs. Desktopia is experimental and minimal where the others are mature: it falls back to CPU encoding on hosts that block NVENC, has no audio or clipboard yet, and requires a Chromium-based browser (the WebTransport and WebCodecs APIs are not in Safari and only partly in Firefox). noVNC wins on universal browser support; Selkies wins on polish (audio, adaptive bitrate, clipboard, years of testing). Reach for Desktopia when you want the lowest interactive latency for a GPU/3D workload and a pipeline small enough to change.

You need the vast.ai command-line tool and a Chromium-based browser (Chrome, Edge, Brave, …).

pip install vastai && vastai set api-key <YOUR_KEY>

make best          # pick a suitable single-GPU offer (an RTX 4090 is ideal)
make up-best       # rent it
make wl-setup      # build the Wayland compositor on the bare instance (one time)
make stream        # start the compositor, encoder, QUIC server, desktop, and 3D Slicer
make port          # print the public address (IP:PORT) the stream is reachable at

make stream also prints a CERT_SHA256_BASE64= line (the certificate hash). Paste that hash and the IP:PORT from make port into client/index.html, then open that page in a Chromium-based browser (served from localhost or over HTTPS) to connect to the desktop.

Alternatively, the prebuilt container image (built by CI, with the compositor and 3D Slicer already baked in) boots straight into the stream — rent an offer with make up-ghcr OFFER=<id> and skip make wl-setup.

The GPU machine must be launched with these options (the tooling applies them automatically):

-p 4433:4433/udp                      # QUIC is UDP — the /udp suffix is required
-e NVIDIA_DRIVER_CAPABILITIES=all     # must include graphics, display, video, compute
-e NVIDIA_VISIBLE_DEVICES=all

When you are finished, make down destroys the machine and stops billing.

• GPU: a single NVIDIA GPU; an Ada-generation card (e.g. RTX 4090) gives the best hardware encoding and also supports AV1.
• Browser: Chromium-based only — the WebTransport and WebCodecs APIs are not available in Safari and only partially in Firefox.
• Encoding: some hosts block NVENC, in which case the pipeline falls back to slower CPU (x264) encoding.
• Not yet implemented: audio and clipboard sharing.

• BACKENDS.md — running the image on other GPU backends (AWS, GCP, Azure, GPU‑rental clouds, local) and how to test each.
• SECURITY.md — certificate/key handling and the access model.
• SETUP.md — reproducing the container image from scratch.

Apache License 2.0 — see LICENSE.