Related
I'm trying write an X11 program to monitor all mouse movements on the desktop. The program should be able to receive a notification whenever the mouse is moved by the human user, or moved programmatically via XWarpPointer() by a robotic application. I know it should be possible by setting a PointerMotionMask via XSelectInput() and monitor MotionNotify, but I'm having troubles receiving mouse events from all windows, not just one.
Initially, I just tried to receive pointer motion events from the root window, in the following demo.
#include <stdio.h>
#include <X11/Xlib.h>
int main(int argc, char **argv)
{
Display *display;
Window root_window;
XEvent event;
display = XOpenDisplay(0);
root_window = XRootWindow(display, 0);
XSelectInput(display, root_window, PointerMotionMask);
while (1) {
XNextEvent(display, &event);
switch(event.type) {
case MotionNotify:
printf("x: %d y: %d\n", event.xmotion.x, event.xmotion.y );
break;
}
}
return 0;
}
But it doesn't receive any events, unless the mouse pointer is on an empty desktop background. It's clear that merely receiving events from the root window won't work. Then I tried a workaround: first, set SubstructureNotifyMask on the root window to monitor all CreateNotify events to catch all newly created windows, then call XSelectInput() to enable the PointerMotionMask on these windows.
#include <stdio.h>
#include <X11/Xlib.h>
int main(int argc, char **argv)
{
Display *display;
Window root_window;
XEvent event;
display = XOpenDisplay(0);
root_window = XRootWindow(display, 0);
XSelectInput(display, root_window, SubstructureNotifyMask);
while (1) {
XNextEvent(display, &event);
switch(event.type) {
case CreateNotify:
XSelectInput(display, event.xcreatewindow.window, PointerMotionMask);
break;
case MotionNotify:
printf("x: %d y: %d\n", event.xmotion.x, event.xmotion.y);
break;
}
}
return 0;
}
This approach is more successful, I started to receive some mouse events from new windows. Unfortunately, it still doesn't work in all portions inside a window - for example, it cannot receive mouse events from the console area in terminal emulators, but can receive events when the mouse is located around the title bar. It appears that a window can create more subwindows, so the mouse events won't be recorded.
Then I tried another workaround - set both SubstructureNotifyMask and PointerMotionMask in CreateNotify, so when a window creates a child window, SubstructureNotifyMask ensures more CreateNotify events will be received in a recursive manner, so all child windows will get PointerMotionMask as well.
#include <stdio.h>
#include <X11/Xlib.h>
int main(int argc, char **argv)
{
Display *display;
Window root_window;
XEvent event;
display = XOpenDisplay(0);
root_window = XRootWindow(display, 0);
XSelectInput(display, root_window, SubstructureNotifyMask);
while (1) {
XNextEvent(display, &event);
switch(event.type) {
case CreateNotify:
XSelectInput(display, event.xcreatewindow.window, SubstructureNotifyMask | PointerMotionMask);
break;
case MotionNotify:
printf("x: %d y: %d\n", event.xmotion.x, event.xmotion.y);
break;
}
}
return 0;
}
It works a bit better than the second example, but it's not reliable:
X is fully asynchronous, is it possible that the child window got created before we had a chance to XSelectInput()?
Sometimes it just reports a BadWindow error and crashes.
X event handling becomes messy - if the program already handles a lot of different X events, enabling SubstructureNotifyMask recursively will make many unrelated events delivered to other handlers, and it's a pain to add extra code to discriminate between wanted and unwanted events.
So, how do I monitor mouse movement events in all windows on X11?
After doing some research, especially reading the source code of Xeyes (I always fell the demo is stupid, but it helps a lot here!), I found:
Calling XSelectInput() on all the windows and subwindows is a futile attempt, you have to set a mask on every single window and child window ever created, it's not a robust solution, and not recommended.
Instead, it's better to just continuously pulling the mouse pointer from the X server explicitly via XQueryPointer(), rather than asking the X server to push MotionEvent to us.
One naive solution is simply setting up a timer by XtAppAddTimeOut() and calling XQueryPointer() periodically, it works, and indeed, it was what Xeyes did in the past! But it unnecessarily wastes CPU time. Nowadays, the best practice is to take advantage of XInputExtention 2.0. The workflow is:
Initialize XInput v2.0
Enable various masks via XISetMask() and XIEventMask() to receive XI_RawMotion events (or XI_Motion, see notes below) from XIAllMasterDevices (or XIAllDevices).
When a XI_RawMotion (or XI_Motion) event has been received, call XQueryPointer().
XQueryPointer() returns:
Mouse coordinates with respect to the root window.
The active window under the mouse cursor, if any.
Perform a XTranslateCoordinates() if we want relative coordinates with respect to the active window under the mouse cursor.
Demo
Here's a demo (save as mouse.c, compile with gcc mouse.c -o mouse -lX11 -lXi). However, it cannot detect XWarpPointer(), see notes below.
#include <stdio.h>
#include <assert.h>
#include <X11/Xlib.h>
#include <X11/extensions/XInput2.h>
int main(int argc, char **argv)
{
Display *display;
Window root_window;
/* Initialize (FIXME: no error checking). */
display = XOpenDisplay(0);
root_window = XRootWindow(display, 0);
/* check XInput */
int xi_opcode, event, error;
if (!XQueryExtension(display, "XInputExtension", &xi_opcode, &event, &error)) {
fprintf(stderr, "Error: XInput extension is not supported!\n");
return 1;
}
/* Check XInput 2.0 */
int major = 2;
int minor = 0;
int retval = XIQueryVersion(display, &major, &minor);
if (retval != Success) {
fprintf(stderr, "Error: XInput 2.0 is not supported (ancient X11?)\n");
return 1;
}
/*
* Set mask to receive XI_RawMotion events. Because it's raw,
* XWarpPointer() events are not included, you can use XI_Motion
* instead.
*/
unsigned char mask_bytes[(XI_LASTEVENT + 7) / 8] = {0}; /* must be zeroed! */
XISetMask(mask_bytes, XI_RawMotion);
/* Set mask to receive events from all master devices */
XIEventMask evmasks[1];
/* You can use XIAllDevices for XWarpPointer() */
evmasks[0].deviceid = XIAllMasterDevices;
evmasks[0].mask_len = sizeof(mask_bytes);
evmasks[0].mask = mask_bytes;
XISelectEvents(display, root_window, evmasks, 1);
XEvent xevent;
while (1) {
XNextEvent(display, &xevent);
if (xevent.xcookie.type != GenericEvent || xevent.xcookie.extension != xi_opcode) {
/* not an XInput event */
continue;
}
XGetEventData(display, &xevent.xcookie);
if (xevent.xcookie.evtype != XI_RawMotion) {
/*
* Not an XI_RawMotion event (you may want to detect
* XI_Motion as well, see comments above).
*/
XFreeEventData(display, &xevent.xcookie);
continue;
}
XFreeEventData(display, &xevent.xcookie);
Window root_return, child_return;
int root_x_return, root_y_return;
int win_x_return, win_y_return;
unsigned int mask_return;
/*
* We need:
* child_return - the active window under the cursor
* win_{x,y}_return - pointer coordinate with respect to root window
*/
int retval = XQueryPointer(display, root_window, &root_return, &child_return,
&root_x_return, &root_y_return,
&win_x_return, &win_y_return,
&mask_return);
if (!retval) {
/* pointer is not in the same screen, ignore */
continue;
}
/* We used root window as its reference, so both should be the same */
assert(root_x_return == win_x_return);
assert(root_y_return == win_y_return);
printf("root: x %d y %d\n", root_x_return, root_y_return);
if (child_return) {
int local_x, local_y;
XTranslateCoordinates(display, root_window, child_return,
root_x_return, root_y_return,
&local_x, &local_y, &child_return);
printf("local: x %d y %d\n\n", local_x, local_y);
}
}
XCloseDisplay(display);
return 0;
}
Sample Output
root: x 631 y 334
local: x 140 y 251
root: x 628 y 338
local: x 137 y 255
root: x 619 y 343
local: x 128 y 260
XWarpPointer() Troubles
The demo above doesn't work if the pointer is moved via XWarpPointer() by a robotic application on newer systems after X.Org 1.10.4. This is intentional, see Bug 30068 on FreeDesktop.
In order to receive mouse events triggered by all mouse movements, including XWarpPointer(), change XI_RawMotion to XI_Motion, and change XIAllMasterDevices to XIAllDevices.
References
This demo lacks error checking and may contain bugs. If in doubts, please check the following authoritative references.
Tracking Cursor Position by Keith Packard, a real X expert, has been heavily involved in the development of X since the late 1980s, and responsible for many X extensions and technical papers.
Xeyes source code from X.Org.
I write dummy window for software KVM. Idea is when user switches to another machine, previous machine opens dummy window beyond the screen bounds, so emulates focus lost. Windows without focus are transparent on my system (awesome wm). If I just use XSetInputFocus(dpy, None, RevertToNone, CurrentTime) current window loses focus but doesn't become transparent. That is why I use dummy window.
Come to the point. My program saves focused window before opens dummy window. When dummy window loses focus program restores saved. So It needs to catch FocusOut event and then call XSetInputFocus function to restore focus. Problem is that xserver sends FocusOut notify twice. If I handle it only once program works incorrect.
Code is below.
#include <X11/Xlib.h>
#include <X11/Xatom.h>
int x = 100, y = 100, height = 200, width = 200;
typedef struct {
Window win;
int notify;
} wait_arg;
Bool xevent_handler(Display *dpy, XEvent *ev, XPointer arg) {
wait_arg *a = (wait_arg *) arg;
return (ev->type == a->notify) && (ev->xvisibility.window == a->win);
}
int main(int argc, char *argv[]) {
Display *dpy;
Window focused;
int revert_to;
Window win;
wait_arg arg;
XEvent ev;
dpy = XOpenDisplay(NULL);
// Save window which has focus now
XGetInputFocus(dpy, &focused, &revert_to);
int s = DefaultScreen(dpy);
win = XCreateSimpleWindow(dpy, RootWindow(dpy, s), 0, 0, height, width, 0,
CopyFromParent, CopyFromParent);
// Set DIALOG type for window
Atom type = XInternAtom(dpy, "_NET_WM_WINDOW_TYPE", False);
long value = XInternAtom(dpy, "_NET_WM_WINDOW_TYPE_DIALOG", False);
XChangeProperty(dpy, win, type, XA_ATOM, 32, PropModeReplace,
(unsigned char *) &value, 1);
// Set events to handle
XSelectInput(dpy, win, VisibilityChangeMask | FocusChangeMask);
// Draw window
XMapWindow(dpy, win);
// Wait until window stands visible, otherwise will get
// "Error of failed request: BadMatch"
arg.win = win;
arg.notify = VisibilityNotify;
XIfEvent(dpy, &ev, &xevent_handler, (XPointer) &arg);
XSetInputFocus(dpy, win, RevertToNone, CurrentTime);
XSync(dpy, False);
XMoveWindow(dpy, win, x, y);
// Wait until focus lost
arg.win = win;
arg.notify = FocusOut;
XIfEvent(dpy, &ev, &xevent_handler, (XPointer) &arg);
// Why I should handle this event twice?
XIfEvent(dpy, &ev, &xevent_handler, (XPointer) &arg);
// Restore focus
XSetInputFocus(dpy, focused, revert_to, CurrentTime);
XSync(dpy, False);
}
How to check incorrect behavior:
I comment second FocusOut handling. I use two monitors: left and right. I start program in terminal on right monitor. Program opens dummy window on left monitor. When dummy window loses focus another window on left monitor obtains it. Not terminal window on right monitor!
Why xserver sends FocusOut twice? Or it is bug/feature of my window manager?
Without your implementation it is really impossible to know exactly what the driver is for the behavior you are seeing, but the man pages do provide some insight into what may be happening.
This particular behavior may be the result of the last-focus-change time. From this man page:
...The XSetInputFocus() function changes the input focus and the last-focus-change time. It has no effect if the specified time is earlier than the current
last-focus-change time or is later than the current X server time.
I want to create 2 windows in linux that I'll later draw in from a separate thread. I currently have a non-deterministic bug where the second window that I create sometimes doesn't get created (no errors though).
Here is the code.
static void create_x_window(Display *display, Window *win, int width, int height)
{
int screen_num = DefaultScreen(display);
unsigned long background = WhitePixel(display, screen_num);
unsigned long border = BlackPixel(display, screen_num);
*win = XCreateSimpleWindow(display, DefaultRootWindow(display), /* display, parent */
0,0, /* x, y */
width, height, /* width, height */
2, border, /* border width & colour */
background); /* background colour */
XSelectInput(display, *win, ButtonPressMask|StructureNotifyMask);
XMapWindow(display, *win);
}
int main(void) {
XInitThreads(); // prevent threaded XIO errors
local_display = XOpenDisplay(":0.0");
Window self_win, remote_win;
XEvent self_event, remote_event;
create_x_window(local_display, &remote_win, 640,480);
// this line flushes buffer and blocks so that the window doesn't crash for a reason i dont know yet
XNextEvent(local_display, &remote_event);
create_x_window(local_display, &self_win, 320, 240);
// this line flushes buffer and blocks so that the window doesn't crash for a reason i dont know yet
XNextEvent(local_display, &self_event);
while (1) {
}
return 0;
}
I don't really care for capturing input in the windows, but I found a tutorial that had XSelectInput and XNextEvent (in an event loop) and I was having trouble making this work without either.
It's not a bug, it's a feature. You left out the event loop.
Although you cleverly called XNextEvent twice, the X protocol is asynchronous so the server may still be setting up the actual window while you call XNextEvent, so there is nothing to do.
Tutorial here.
This assumes the two sessions are on the same box in a multi user desktop environment. Here is my C code:
int main(int argc, char* argv[]) {
Display *d_remote; /* - Display of User you are shadowing - */
Display *d_local; /* - Your display */
XImage *img; /* - Used to hold the pixles of the shadowed user's display - */
Window w, pointer_root, pointer_child;
XEvent e;
int s;
int pointerX, pointerY, winX, winY;
unsigned int mask;
//Open your display, if you cant set to NULL
d_local = XOpenDisplay(NULL);
if(d_local == NULL) {
printf("Error opening 'local' display\n");
exit(1);
}
//Set the XAUTHORITY to teh argument provided. This is the XAUTHORITY for the remote display
setenv("XAUTHORITY", argv[2],1);
d_remote = XOpenDisplay(argv[1]);
if(d_remote == NULL) {
printf("Error opening 'remote' display %s\n", argv[1]);
exit(1);
}
//Get the size of the remote display
Window root_remote = RootWindow(d_remote,0);
XWindowAttributes attr;
XGetWindowAttributes(d_remote,root_remote,&attr);
img = XGetImage(d_remote,RootWindow(d_remote,0),0,0,attr.width,attr.height,XAllPlanes(),ZPixmap);
printf("attr.width = %d, attr.height=%d\n", attr.width, attr.height);
//Create the window on your display
s = DefaultScreen(d_local);
w = XCreateSimpleWindow(d_local, RootWindow(d_local, s), 10, 10, 100, 100, 1,
BlackPixel(d_local, s), WhitePixel(d_local, s));
XSelectInput(d_local, w, ExposureMask | KeyPressMask);
XMapWindow(d_local, w);
while (1) {
/* - Get the image from the remote display and push it to the local window - */
img = XGetImage(d_remote,RootWindow(d_remote,0),0,0,attr.width,attr.height,XAllPlanes(),ZPixmap);
XPutImage(d_local, w, DefaultGC(d_local,0), img, 10, 10, 0, 0, attr.width, attr.height);
XQueryPointer(d_remote, RootWindow(d_remote,0), &pointer_root, &pointer_child, &pointerX, &pointerY, &winX, &winY, &mask);
XFillRectangle(d_local, w, DefaultGC(d_local, s), pointerX-10, pointerY-10, 10, 10);
XFlush(d_local);
usleep(10);
}
XCloseDisplay(d_local);
XCloseDisplay(d_remote);
return 0;
}
I basically loop, get the raw image of the shadowed user's screen using XGetImage then dump that image into a Window. This is using massive amounts of CPU. I can see why considering that I am manipulating large images every centisecond.
Is there a more efficient way to do this? VNC essentially takes a capture of the user's screen and sends it over the wire (compressed of course) but does not use nearly as much CPU.
You can use the DAMAGE extension to get notified when the screen is updated and what part of the screen has changed, so you don't need to grab the whole screen every 10 microseconds if nothing changes (which is much more often than users can see changes if the display is running with a 60 Hz refresh rate), nor would you need to grab the entire screen if all that changed is a 10x10 pixel area where their clock is displayed.
I want to display an image in OpenCV in a full screen borderless window.
In other words, only the image pixels will appear, without menu, toolbar, or window background.
Using imshow() or cvShowImage() don't enable it:
The window grows to be full screen
in width but not in height. It misses few pixels.
I could not make it borderless even by changing settings of window
handler.
I think that the problem is rooted in cvNamedWindow() method which creates main WS_OVERLAPPED window, then creates a child and all functions like imshow() or cvGetWindowHandle() operate on the child.
Thus even windows command:
SetWindowLong((HWND)cvGetWindowHandle(winName), GWL_STYLE, WS_VISIBLE | WS_EX_TOPMOST | WS_POPUP);
Doesnt help, since the child cannot become borderless WS_POPUP. Someone got a workaround?
Maybe, showing opencv mat to window
without using opencv built in methods
Or some kind of windows trick
P.S. I tried the following code:
cvMoveWindow("AAA",0,0);
cvSetWindowProperty("AAA", CV_WINDOW_FULLSCREEN, CV_WINDOW_FULLSCREEN);
// Also I tried this:
HWND hwnd = (HWND)cvGetWindowHandle("AAA");
RECT windowRect;
windowRect.left = 0;
windowRect.top = 0;
windowRect.right = cxScreen; //Display resolution
windowRect.bottom = cyScreen; //Display resolution
AdjustWindowRect(&windowRect,WS_VISIBLE,false);
long p_OldWindowStyle = SetWindowLongPtr(hwnd,GWL_STYLE,WS_POPUP);
SetWindowPos(hwnd,HWND_TOP,0,0,windowRect.right,windowRect.bottom,SWP_FRAMECHANGED | SWP_SHOWWINDOW);
SetWindowLong(hwnd, GWL_STYLE, WS_VISIBLE | WS_EX_TOPMOST | WS_POPUP);
Have you issued cvShowImage() to display the window? Because it seems you are not doing it. Anyway, you might want to call the win32 API for this instead, so add a call to ShowWindow(hwnd, SW_SHOW); after SetWindowPos().
If your current call to SetWindowPos() doesn't do the trick, check this answer: Hide border of window, if i know a handle of this window
I recommend you doing your tests without calling cvSetWindowProperty() at first, just to make sure you can find a method that works.
Just a note, if you check modules/highgui/src/window_w32.cpp you can see how OpenCV creates windows on Windows.
EDIT:
The following code implements the tips I gave before and bypasses the problems the OP reported. The trick is NOT using cvGetWindowHandle() to retrieve the windows' handle and use directly win32 API for that: FindWindow()
IplImage* cv_img = cvLoadImage("test.jpg", CV_LOAD_IMAGE_UNCHANGED);
if(!cv_img)
{
printf("Failed cvLoadImage\n");
return -1;
}
cvNamedWindow("main_win", CV_WINDOW_AUTOSIZE);
cvMoveWindow("main_win", 0, 0);
cvSetWindowProperty("main_win", CV_WINDOW_FULLSCREEN, CV_WINDOW_FULLSCREEN);
cvShowImage("main_win", cv_img);
//HWND cv_hwnd = (HWND)cvGetWindowHandle("main_win");
//if (!cv_hwnd)
//{
// printf("Failed cvGetWindowHandle\n");
//}
//printf("cvGetWindowHandle returned %p\n", *cv_hwnd);
HWND win_handle = FindWindow(0, L"main_win");
if (!win_handle)
{
printf("Failed FindWindow\n");
}
SetWindowLong(win_handle, GWL_STYLE, GetWindowLong(win_handle, GWL_EXSTYLE) | WS_EX_TOPMOST);
ShowWindow(win_handle, SW_SHOW);
cvWaitKey(0);
cvReleaseImage(&cv_img);
cvDestroyWindow("main_win");
This code will make the window created by OpenCV borderless, but you still might have to tweak one thing or another to make this operation perfect. You'll see why. One idea is to resize the window and make it the size of the image.
EDIT:
Well, since you stated:
writing a demo might be very hard
I also decided to do this last part for you, since I'm such a nice guy =]
This is a small improvement of the code above:
HWND win_handle = FindWindow(0, L"main_win");
if (!win_handle)
{
printf("Failed FindWindow\n");
}
// Resize
unsigned int flags = (SWP_SHOWWINDOW | SWP_NOSIZE | SWP_NOMOVE | SWP_NOZORDER);
flags &= ~SWP_NOSIZE;
unsigned int x = 0;
unsigned int y = 0;
unsigned int w = cv_img->width;
unsigned int h = cv_img->height;
SetWindowPos(win_handle, HWND_NOTOPMOST, x, y, w, h, flags);
// Borderless
SetWindowLong(win_handle, GWL_STYLE, GetWindowLong(win_handle, GWL_EXSTYLE) | WS_EX_TOPMOST);
ShowWindow(win_handle, SW_SHOW);
And on my system it displays exactly what you asked on the question.