Support for mouse capturing games

[lkd70]

Is there any intent to support games which capture the mouse input? Or ones that simply track mouse movement, even if the mouse moves beyond the bounds of the screen (ARK, for example)

Windows 10 - Native install

When attempting to use this tool to move my mouse in a game (Using ARK, for example) the cursor doesn't move within the game, but does seem to move when not in game. This issue doesn't seem to occur in robot.js or any other software I've played with, but the dynamic movement support of this library is fantastic. Is this something this library could do to? Is this possible or is it outside the bounds of this project?

Thanks for this amazing work of art!

[svettwer]

Hi 👋

I think that nut.js might be detected as automation software from the anti cheat protection of the game. There are several functionalities which are not indented to be automated. E.g the login screen for windows forbids automated interaction using the OS API. I've had an idea a while ago where we could try to simulate a usb keyboard and a usb mouse with virtual USB devices, similar to VM integrations. But that would require a lot of research an POC work at first.

[s1hofmann]

Hi @lkd70,

nut.js relies on our N-API port of robotjs, so if it's doable with robotjs, it should™ be doable with nut.js as well.
We're currently using a different syscall to set the cursor position limited to the desktop area.
There's another approach which also synthesizes input events, which might do the trick.

Since we're not primarily focusing on game automation, I'm not 100% sure on this, but feel free to follow along here for updates and feedback!

Thanks for reaching out and your kind words!

[Reiss-Cashmore]

For anyone who really wants to do this. It is possible by modifying mouse.c inside of libnut. It requires you to roll libnut yourself then import that binary into a copy of the nut.js repo and then roll that yourself too. So not for the feint of heart.

Here is a crude implementation of SendInput as suggested by @s1hofmann which offers a very limited usage within mouse capture apps.

HERE BE DRAGONS Be aware this snippet fundamentally changes the way moveMouse works and breaks most of the existing mouse behavious of nut.js. It offers only a relative pixel movement of the mouse so you will lose ALL nut.js functionality around setting the mouse position to absolute pixel values on a screen.

***mouse.c***

void moveMouse(MMPoint point)
{
	// SetCursorPos ((int)point.x, (int)point.y);

        INPUT mouseInput;

	mouseInput.type = INPUT_MOUSE;
	mouseInput.mi.dx = (int)point.x;
	mouseInput.mi.dy = (int)point.y;
	mouseInput.mi.mouseData = 0;     
	mouseInput.mi.time = 0;
        mouseInput.mi.dwFlags = MOUSEEVENTF_MOVE;
        mouseInput.mi.dwExtraInfo = 0;
	SendInput(1, &mouseInput, sizeof(mouseInput));
}

This enables you to do this using your modified nut.js lib

async function testMouseCapturedMovement() {
    await mouse.move([new Point(100, 0)]); // Move the mouse 100 pixels to the right
    await mouse.move([new Point(0, 100)]); // Move the mouse 100 pixels down
    await mouse.move([new Point(0, -100)]); // Move the mouse 100 pixels back up
}

[s1hofmann]

Hi @Reiss-Cashmore 👋

That’s almost exactly what I had in mind!
I’d suggest changing

mouseInput.mi.dwFlags = MOUSEEVENTF_MOVE;

to

mouseInput.mi.dwFlags = MOUSEEVENTF_ABSOLUTE;

This treats x and y values as absolute coordinates, keeping it consistent with the current implementation and thus not breaking nut.js API.

See https://docs.microsoft.com/en-us/windows/win32/api/winuser/ns-winuser-mouseinput#remarks

I’d happily accept a PR for this change!
Thanks for your support!

[Reiss-Cashmore]

Hi @Reiss-Cashmore 👋

That’s almost exactly what I had in mind! I’d suggest changing

mouseInput.mi.dwFlags = MOUSEEVENTF_MOVE;

to

mouseInput.mi.dwFlags = MOUSEEVENTF_ABSOLUTE;

This treats x and y values as absolute coordinates, keeping it consistent with the current implementation and thus not breaking nut.js API.

See https://docs.microsoft.com/en-us/windows/win32/api/winuser/ns-winuser-mouseinput#remarks

I’d happily accept a PR for this change! Thanks for your support!

Hey, no worries I did a more technical response over here: nut-tree/libnut-core#26

I would be happy to do a PR if you feel the code is high enough quality. I wasn't sure, I am an absolute noob with C/C++ so could be doing something wrong. I think if you let me know whether you want me to modify moveMouse or create separate methods for relative / absolute movements