[MS-RDPBCGR] Relative mouse input

[sasha0552]

Specification

Related: neutrinolabs/xorgxrdp#314

Fixes #2131

[matt335672]

Thanks for this @sasha0552

One (fairly significant) comment/question at this stage; The general approach you're taking (if I'm reading this right) is to send both relative and absolute events to xorgxrdp, and then within the module to generate an absolute position.

Wouldn't it be better to simply do this within xrdp before sending the event to the module? That way you'll get support for the VNC backend for free - at the moment this won't work with VNC without duplicating the code you're adding to xorgxrdp within vnc.c.

[sasha0552]

@matt335672

The general approach you're taking (if I'm reading this right) is to send both relative and absolute events to xorgxrdp, and then within the module to generate an absolute position.

No, neither xrdp nor xorgxrdp hold absolute position when using relative input - they just pass relative mouse movements directly to Xorg, and then, Xorg itself calculates the absolute position and passes the events with delta to the applications.

Applications that need relative input don't want absolute coordinates, they need only delta (by how many pixels the cursor has been moved, this value may even exceed the screen borders), so doing this on the xrdp side essentially implements the current behavior, but with the risk that the cursor may desynchronize.

The applications want this:

And about VNC - I don't think VNC supports relative input. I'm not sure how xrdp works with the VNC backend - does xrdp just translate RDP to VNC? If so, then relative input is not possible on the VNC backend.

[matt335672]

Thanks for the clarification - that makes more sense now.

Classic VNC doesn't support relative mouse events:-

https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#pointerevent

which from what you've said means either:-

1. [Eventually] we will need a way to disable advertising of INPUT_FLAG_MOUSE_RELATIVE in the server capability flags where the user knows they will be connecting to a VNC session.
2. We'll need to convert relative mouse events to absolute ones before sending them to VNC.

Also from what you've said, 2) runs the risk of de-synchronisation.

[jsorg71]

@sasha0552 I want to test this. What clients supports relative mouse? I know it's added to later version of FreeRDP but does mstsc or any other clients support this feature?

[sasha0552]

I don't think clients other than FreeRDP currently support this feature.

[Givlucas]

Any updates on this?

[matt335672]

Not yet @Givlucas

Is there a use-case you have that requires it? That would be useful information to have.

[Givlucas]

Yeah, certain games require relative mouse motion. Usually FPS but in my case Minecraft

[jsorg71]

I asked about this at RDP IO last week. The rdcore MS clients(non-mstsc) use this relative mouse input when the mouse pointer is set to nil.
I think when using
https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-rdpbcgr/4423dd94-be0f-4b6d-b9de-653c67ce4134
and sending SYSPTR_NULL.
Currently, xrdp does not use TS_SYSTEMPOINTERATTRIBUTE and sets a nil cursor by sending all zeros or all ones, can't remember. I'll make a PR to start using TS_SYSTEMPOINTERATTRIBUTE and we should see some clients start using relative input. Of course with the capability set.

[pnowack]

I asked about this at RDP IO last week.

Thank you for asking that at RDP IO. When the relative mouse motion handling was added to [MS-RDPBCGR] and [MS-RDPECI], this raised some question marks. Reason is, that diffs (or deltas) in relative mouse motion differ completely from absolute mouse motion (either the delta is way higher than the actual motion diff, or way lower), so the assumption was, that in order to properly support relative mouse motion was to get both relative and absolute motion events, apply the relative one first and update the mouse ptr position based upon the absolute mouse event again.
This is not perfect, because depending on the compositor implementation, this can cause a "jumping" or "jittering" cursor.

In any case, the client handling in FreeRDP needs to be adjusted. Will do some relative mouse motion tests with g-r-d then too to ensure the FreeRDP client handling works.

[jsorg71]

@sasha0552 I don't see the capability set in this PR. or did I miss it.

[Chimrod]

Hello,

How can I test this feature? I’ve applied your branche over the debian packages xrdp and xorgxrdp but I do not see any changes.

I’m using wfreerdp with the right options, and I can see this capability in the freerdp logs:

[18:52:19:896] [32268:000012fc] [DEBUG][com.freerdp.core.gcc] - [gcc_write_client_core_data]: Sending highColorDepth=HIGH_COLOR_24BPP, supportedColorDepths=RNS_UD_32BPP_SUPPORT|RNS_UD_24BPP_SUPPORT|RNS_UD_16BPP_SUPPORT|RNS_UD_15BPP_SUPPORT, earlyCapabilityFlags=RNS_UD_CS_WANT_32BPP_SESSION|RNS_UD_CS_RELATIVE_MOUSE_INPUT|RNS_UD_CS_VALID_CONNECTION_TYPE|RNS_UD_CS_SUPPORT_NETCHAR_AUTODETECT|RNS_UD_CS_SUPPORT_DYNVC_GFX_PROTOCOL|RNS_UD_CS_SUPPORT_DYNAMIC_TIME_ZONE|RNS_UD_CS_SUPPORT_HEARTBEAT_PDU|RNS_UD_CS_SUPPORT_SKIP_CHANNELJOIN|[0x00000fb2]

But I do not see relevant message from the xrdp side. Is there an easy way to test if the channel is used ?

[akallabeth]

@Chimrod well, you should try a client that has it implemented as well ;)
So, sdl-freerdp or xfreerdp

[Chimrod]

I’ve also tried with the windows binary sdl3-freerdp and thoses arguments but without any success.

./sdl3-freerdp.exe /f /gdi:hw /rfx /rfx-mode:video /gfx:RFX:on,progressive:on /video /size:1680x1050 /sound /mouse:relative:on,grab:off +mouse-relative +async-channels

That’s the point of my question, how can check if the whole chain is OK ?

[jsorg71]

According to the docs, in order for the server to trust the INPUT_FLAG_MOUSE_RELATIVE flag. The client must report 0x00080011 in the version field of TS_UD_CS_CORE.
I'm trying to find a client that reports this capability.
Mac client(Windows App) 11.1.3 report TS_UD_CS_CORE version 0x00080005 which was a bit disappointing.
MSTSC 10.0.19041 report TS_UD_CS_CORE version 0x0008000d.
I feel like we need to have an MS client to implement this correctly.

[akallabeth]

@jsorg71 mstsc from windows 11 24h2 reports 0x00080011 but no relative mouse event support flag...

[akallabeth]

@Chimrod did update FreeRDP with FreeRDP/FreeRDP#11384 in a few details.

as for your test, I did simply run sdl-freerdp /v:host /u:user /p:pwd /mouse:relative:on +mouse-relative and that works quite fine (if the remote sends a Pointer NULL mouse cursor) against freerdp-shadow-cli (check the referenced pull request for details)

[edit] discussing with @pnowack currently if it is possible to send both, absolute and relative events.
that is something that can not really be derived from the spec, so we would need a reference client to check that.