ref:main

Phase 1a: qemu capture source (shm + DMABUF zero-copy, mode changes, cursor) #3

open Opened by cole.christensen@gmail.com

Phase 1a: qemu capture source, complete (shm + DMABUF zero-copy, mode changes, cursor)

Parent: #1. Depends on #2 (QemuSession, shm capture spike, E2E harness).

Requirements

  • REQ-CAP-002: capture shared-memory scanouts (finish what #2 started; all pixman formats QEMU emits)
  • REQ-CAP-003: zero-copy DMABUF scanouts into VAAPI and CUDA encoders
  • REQ-CAP-004: survive guest mode changes, Disable, and reboot
  • REQ-CAP-005: composite the guest cursor into the stream
  • REQ-CAP-006: selectable via capture = qemu; console selection
  • REQ-NFR-001 (draft): capture-to-packet latency; this phase removes the capture pacing delay and records DMABUF latency

Background (verified)

  • DMABUF scanouts need a GL-capable QEMU display: -device virtio-vga-gl -display dbus,gl=on[,rendernode=/dev/dri/renderD128], or vfio-pci ...,display=on for mdev/vGPU. QEMU sends ScanoutDMABUF(fd, w, h, stride, fourcc, modifier, y0_top) or, if our Listener lists the interface in Interfaces, ScanoutDMABUF2(ah fds, x, y, w, h, au offsets, au strides, num_planes, fourcc, backing_w, backing_h, modifier, y0_top), then UpdateDMABUF(x, y, w, h) on damage.
  • Sunshine already imports DMABUFs: egl::surface_descriptor_t (src/platform/linux/graphics.h:496: fds[4], fourcc, modifier, pitches[4], offsets[4]) and egl::import_source. See how kmsgrab.cpp builds descriptors and implements make_avcodec_encode_device() for vram (kmsgrab.cpp:1731-1745: va::make_avcodec_encode_device(..., true), vk::make_avcodec_encode_device_vram, cuda). Model the qemu vram path on the kmsgrab vram display class. Don’t invent a new import path.
  • With DMABUF, the GPU that QEMU renders on must be the one Sunshine encodes on (same render node). Pick the render node from rendernode= / adapter_name config and fail with a clear error on mismatch.
  • Host for DMABUF work: native Linux with a DRM render node (/dev/dri/renderD*). A GPU under WSL2 is not enough (checked 2026-09-12 on an RTX 4090 under WSL2): there’s no /dev/dri, and QEMU refuses -device virtio-vga-gl -display dbus,gl=on with “egl: no drm render node available”. WSL2 can still run the shm path with NVENC (CUDA build).
  • Capture pacing dominates shm latency (measured in #2, see JOURNAL.md on issue-2-phase0). qemu::display_t::capture() wakes on a fixed tick of one frame interval (platf::handle_pacing) and copies whatever damage arrived since. Damage lands uniformly in the interval, so it waits 0–16.7 ms at 60 fps. At 1080p60 the host latency was p50 11.1 ms / p95 20.4 ms with NVENC and p50 14.1 / p95 21.4 ms with software x264: pacing is ~8 ms p50 / ~16 ms p95 of that, and NVENC ~3 ms.

Tasks

  • Two display classes, like kmsgrab: qemu::display_ram_t (shm, mem_type_e::system) and qemu::display_vram_t (DMABUF → VAAPI/CUDA/Vulkan). platf::display() picks by hwdevice_type. If QEMU only sends shm while a vram encoder is requested, upload to GPU through the existing RAM→encode path rather than failing.
  • Add org.qemu.Display1.Listener.Unix.ScanoutDMABUF2 to the Listener’s Interfaces; handle both the DMABUF and DMABUF2 methods. Honor y0_top (flip), x/y offsets and backing size.
  • FD lifetime: dup/close the fds received in GUnixFDList correctly. A new Scanout* replaces the previous buffer; release the old EGL image only after the encoder is done with it.
  • Pixman formats: at minimum x8r8g8b8, a8r8g8b8, x8b8g8r8, a8b8g8r8, r5g6b5 (fallback VGA). Unsupported formats are converted with pixman or logged once and ignored.
  • Mode changes (REQ-CAP-004): a size change in Scanout*/ScanoutMap returns capture_e::reinit so the pipeline re-creates the encoder at the new size. Disable shows a black frame and keeps the session alive. The guest rebooting (Disable, then a firmware-size scanout, then an OS-size scanout) must not end the stream.
  • Cursor (REQ-CAP-005): store the CursorDefine ARGB image and hotspot and the MouseSet(x, y, on) state. When *cursor is true, blend onto frames: on the CPU for RAM, and with the existing cursor blend shader in graphics.cpp for vram (kmsgrab/wlgrab already draw cursors; reuse that).
  • Console selection (REQ-CAP-006): #2 already selects consoles with Sunshine’s existing output_name (id or label), and display_names() returns the graphical console ids; there is no qemu_console key. Finish it: multi-head guests pick one console per stream; add tests for label and id selection through output_name.
  • Damage-driven capture (REQ-NFR-001): replace the fixed-tick loop in qemu::display_t::capture() with a wait on the frame store (condition variable, notified by touch_locked()) that wakes as soon as new damage arrives, rate-limited so frames are never pushed more often than the client’s frame interval. Keep repeating the last frame at the client’s fps when there’s no damage, and never block the listener thread on the encoder. Target: shm + NVENC p95 ≤ 12 ms at 1080p60 (from 20.4 ms). Apply the same wakeup to the DMABUF display class.

Tests

  • Unit, with the fake QEMU from #2: each Scanout variant; Update damage math; format conversion; y0_top flip; cursor composite pixel checks; size change → reinit; Disable → black frame. The DMABUF unit tests can build DMABUFs with udmabuf (/dev/udmabuf) or GBM when available; GTEST_SKIP() with a clear message otherwise, and document which requirement the skip affects.
  • Unit, pacing: damage delivered mid-interval is pushed within a few ms, not at the next tick; bursts of damage still push at most one frame per client frame interval; no damage still repeats nothing (the encoder’s minimum FPS covers static screens).
  • E2E: extend the #2 harness with a gl=on variant (hardware-gated; skipped in CI without /dev/dri) and a reboot-during-stream test (QMP system_reset, then assert the stream keeps producing frames).
  • Record latency for REQ-NFR-001 with E2E_WIDTH=1920 E2E_HEIGHT=1080 E2E_FPS=60 E2E_FRAMES=600 and E2E_ENCODER=nvenc (or vaapi): the shm path before and after damage-driven capture, and the DMABUF path. Update REQ-NFR-001 with the targets and set it active.

Done when

All REQ-CAP-00x requirements are linked to passing tests, the E2E test (shm) is green, the DMABUF E2E is green on a native Linux GPU host with /dev/dri (note the GPU, driver and distro in the PR), the shm + hardware encoder p95 at 1080p60 is ≤ 12 ms, and anvil requirement status passes. PR Closes #3.

colechristensen cole.christensen@gmail.com commented 2026-09-12 23:09

Progress on phase-1 (52f749fe..1f64e75f), from WSL2 with an RTX 4090 (no /dev/dri):

  • Pixman formats: every packed 16/24/32 bpp format, r5g6b5 included, for Scanout/Update and ScanoutMap/UpdateMap.
  • Damage-driven capture: 1080p60, NVENC, shm map. p95 went from 17.8-21.8 ms to 4.4-5.4 ms on a quiet host, and up to 10 ms under load. p50 went from 11-14 ms to about 4 ms. The <= 12 ms target is met. REQ-NFR-001 description is updated and stays draft until DMABUF is measured.
  • REQ-CAP-004: Disable shows black, mode changes reinit. E2E_RESET_AFTER_FRAMES reboots the guest with QMP system_reset mid-stream. It passes on QEMU 11.1 and 8.2, with NVENC too. QEMU 8.2 never sends the text-mode scanout, so run it with E2E_RESET_MIN_DISPLAYS=1.
  • REQ-CAP-005: CPU cursor compositing is unit-tested. The vram path hands the cursor to the existing GL shader, which is untested.
  • REQ-CAP-006: output_name selects by id or label. Labels didn’t work through video::refresh_displays before.
  • REQ-CAP-003: ScanoutDMABUF/ScanoutDMABUF2/UpdateDMABUF, display_vram_t (kmsgrab-style), y_invert flip on the GL path, RAM fallback and the render node check are unit-tested with memfds. Still unverified: the EGL import test and E2E_GL=1 skip here. That needs native Linux with /dev/dri.

Details in JOURNAL.md. Not closing: the DMABUF E2E and DMABUF latency are still open.