ref:main

Phase 1c: keyboard, mouse, touch input to QEMU #5

open Opened by cole.christensen@gmail.com

Phase 1c: keyboard, mouse, touch input to QEMU

Parent: #1. Depends on #2 (QemuSession). Can run in parallel with #4.

Requirements

  • REQ-INP-001: keyboard input delivered to the guest
  • REQ-INP-002: mouse input delivered to the guest (relative, absolute, buttons, scroll)
  • REQ-INP-003: touch input delivered to the guest

Background (verified)

  • Sunshine: src/input.cpp decodes Moonlight input and calls platform free functions: platf::move_mouse, abs_mouse(touch_port, x, y), button_mouse(button, release), scroll(high_res_distance), hscroll, keyboard_update(modcode, release, flags), unicode, touch_update, gamepad_update, alloc_gamepad… On Linux and macOS these live in src/platform/virtualhid_input.cpp (~L1026-1100) and forward to libvirtualhid (host uinput). platf::input() is created at src/input.cpp:2277. keyboard_update modcodes are Windows virtual-key codes.
  • QEMU D-Bus (on /org/qemu/Display1/Console_N):
    • Keyboard.Press/Release(u keycode), where keycode is a QEMU qnum (xtkbd scancode with the high bit re-encoded), plus a Modifiers property (Scroll/Num/Caps lock state).
    • Mouse.Press/Release(u button) with Left=0 Middle=1 Right=2 WheelUp=3 WheelDown=4 Side=5 Extra=6; SetAbsPosition(u x, u y) in console pixels (error unless IsAbsolute); RelMotion(i dx, i dy) (error if IsAbsolute).
    • MultiTouch.SendEvent(u kind Begin=0/Update=1/End=2/Cancel=3, t slot, d x, d y), MaxSlots.
  • IsAbsolute depends on the guest’s devices: add -device virtio-tablet-pci or usb-tablet for absolute mode; PS/2 mouse is relative. It can change at runtime, so watch PropertiesChanged.

Tasks

  • Routing: when capture == qemu, platf::input() creates a context holding a qemu::input_t (bound to the same QemuSession and console as capture), and the free functions dispatch to it. The gamepad functions keep going to libvirtualhid (see #7). Keep the diff to virtualhid_input.cpp small and obvious for upstream review: a dispatch branch, not interleaved logic.
  • Keymap: keymap.{h,cpp} translates Windows VK → Linux KEY_* → QEMU qnum.
    • libvirtualhid or Sunshine already has a VK→Linux keycode table; find it and reuse it.
    • The Linux→qnum table must be generated or ported from QEMU’s keycodemapdb (ui/input-keymap.c uses qemu_input_map_linux_to_qcode, then qcode→qnum). keycodemapdb is GPL-2.0-or-later, so it’s compatible with GPL-3 Sunshine; record provenance in a README.
    • Handle extended keys (flags from Moonlight), left/right modifiers, and keys with no mapping (log once).
  • Mouse:
    • Absolute mode: convert Moonlight’s touch_port-relative float coordinates to console pixels (use the console Width/Height, honoring any capture offsets) → SetAbsPosition.
    • Relative mode: accumulate sub-pixel deltas → RelMotion.
    • If the client sends absolute input while the guest is relative, or the reverse, convert rather than erroring: absolute→relative needs the last known guest cursor position from MouseSet (#3), otherwise drop with a debug log.
    • Scroll: each 120 high-res units = one WheelUp/WheelDown press+release. Keep the remainder. Horizontal scroll has no D-Bus button; document it as unsupported.
    • Buttons: Moonlight 1..5 → Left, Middle, Right, Side, Extra.
  • Touch: map Moonlight touch events to MultiTouch events with slot allocation up to MaxSlots; if the console lacks MultiTouch, fall back to absolute mouse emulation for single touch.
  • Keyboard state: release all pressed keys and buttons when a client disconnects, so no key stays stuck in the guest.
  • All D-Bus calls are async on the GMainContext thread, preserving event order, so a slow bus never blocks Sunshine’s input thread.

Tests

  • Unit, with a fake QEMU recording calls: VK→qnum table coverage for the full 104-key layout plus media keys; mouse abs scaling at different console sizes; rel accumulation; scroll remainder math; IsAbsolute switching; touch slot allocation; release-all on disconnect.
  • E2E: a Linux test guest runs evtest or a tiny evdev logger on its serial console (read from QEMU -serial to a file). The test client injects “a”, Shift+“A”, mouse move to a point, and a click, and the test asserts the guest logged the matching events.

Done when

REQ-INP-001..003 are linked to passing tests, typing and mouse work manually from a real Moonlight client in both PS/2 (relative) and virtio-tablet (absolute) guests, and anvil requirement status passes. PR Closes #5.

colechristensen cole.christensen@gmail.com commented 2026-09-13 00:43

Progress on phase-1 (b5ae5cd6, 82c2d38e, cbc1a543):

  • Keyboard, mouse and touch with capture = qemu go to the captured console through Keyboard/Mouse/MultiTouch (dispatch branch in virtualhid_input.cpp; gamepads unchanged). VK -> KEY_* table ported from libvirtualhid and extended; KEY_* -> qnum from keycodemapdb (third-party/qemu-keycodemapdb, provenance README).
  • IsAbsolute tracking with abs<->rel conversion, scroll remainder, buttons 1..5, touch slots with mouse fallback, release-all on streaming stop, async ordered calls on a non-blocking GLib thread.
  • 27 unit tests (fake QEMU records input calls) linked to REQ-INP-001..003; E2E_INPUT=absolute|relative|touch passes on QEMU 11.1.1 and 8.2.2 with the Linux guest’s evdev log.
  • Open: manual check with a real Moonlight client (no GUI client on this host). Known limits: QEMU always exports MultiTouch, so VMs without virtio-multitouch-pci need native_pen_touch = disabled; with virtio-multitouch-pci QEMU sends mouse buttons to the touch device and drops wheel events. Details in JOURNAL.md.