I am trying to make an application which will graphically repeat the mouse pointer, so I can ultimately make a mouse trail program, for Ubuntu 18.04 - and it seems, the way to do it is via X11/Xlib - although, these days I don't even know, as my machine says also wayland:
$ loginctl | while IFS= read line; do echo "$line"; if [[ $line == *"tty"* ]]; then sessnum=$(echo "$line" | awk '{print $1;}'); echo sessnum: $sessnum\; $(loginctl show-session $sessnum -p Type); fi; done
SESSION UID USER SEAT TTY
c1 121 gdm seat0 tty1
sessnum: c1; Type=wayland
2 1000 administrator seat0 tty2
sessnum: 2; Type=x11
2 sessions listed.
Regardless, I managed to put together an unholy assemblage of:
https://keithp.com/blogs/Cursor_tracking/ - which sets up the program for capturing raw mouse events, so the mouse pointer position can be extracted (and a redraw triggered) whenever the mouse pointer position changes
xosd.c (via https://github.com/AndreRenaud/XOSD) - I thought at first that On-Screen Display would have a special method to draw on top - but this sets up a topmost window, child of the root, where all drawing happens; and it also sets up event and timer thread
.... plus a ton of other code snippets (mostly from SO), which sort of does what I want (even if I don't really fully understand all of the layers and compositing that goes on in it). I posted this as a gist: xosd_track_cursor.c since it's 700+ lines (but can post it here if needed).
Here is how the application behaves (also see full-res imgur .mp4 video)
Basically, at start, the "OSD" topmost window is set up, and it's quite smaller than the desktop window - which helps us see the window border decorations around it (ultimately, I'd make this window the same size as the desktop).
At start, the desktop pixels at the location of this window have seemingly been copied as the window background.
Once the mouse pointer enters the OSD window, there is a draw of a circle, which becomes the mask for the OSD window (which again can be seen via the window border decorations) - and this circular window follows the mouse. Then, inside it, I draw a XFillRectangle to draw a lime rectangle, and then XPutImage to draw the pixels captured from the latest mouse pointer (the video doesn't show it, but also the copied cursor changed when the normal one does, say from left_ptr to bottom_side or xterm cursor bitmaps).
So far so good - but these are the problems, and questions:
All of the draws - both the lime rectangle and the mouse pointer copy - remain on the OSD window, and are not cleared upon redraw (which is quite obvious when the mouse pointer leaves the OSD window, so there is no masking). How can I erase these previous draws each time a new state is rendered?
When I click on window to change the focus, it is obvious (especially when the mouse pointer leaves the OSD window, so there is no masking) that the desktop "background" shown in the OSD window, shows the state when the program started. How can I capture the current state of the desktop background (that is, behind the OSD window), so I can use that for clearing the OSD window in the previous step?
(I thought I could hide the OSD window, then capture the desktop at the same location with XGetImage, maybe (?) - then show the window; but show always sends Expose event, which otherwise runs the expose function that does the redraw, and so I get a bunch of recursive calls hogging the application)
The mouse pointer copy is rendered with a black background - how can I make the drawing of mouse pointer copy transparent, where it is black now?
And, a sort of a bonus question (just curious here - obviously I'd rather not have the leftovers to begin with):
I first do XFillRectangle to draw a lime rectangle, then XPutImage to draw the pixels of the mouse pointer copy. I'd expect this to show the mouse cursor copy on top of the green pixels - and it is indeed so, while the OSD window is masked with the circle. But when the OSD window is shown in full, the leftovers make it seem as if the green pixels were drawn on top of the mouse cursor copy pixels. Why is this so?
Well, I think I got somewhere - the result is in the same gist, just different revision: gist: xosd_track_cursor.c (a31e9dff5); and it looks like this:
And so, to answer my questions:
How can I erase these previous draws each time a new state is rendered?
Cannot - not in the way the previous code was set up. It was set up as a override_redirect, meaning it would stay out of the management of any window manager. Furthermore, the default bit depth was 24, meaning that transparency was not supported, meaning that to grab the desktop "behind" (to use as "clear" background), we'd had to hide and then show the window, which used to cause recursion due to reaction to Expose events.
However, I saw in How to make an OpenGL rendering context with transparent background? that using an glXCreateContext might help - and it did. However, it turns out, it was not necessary - as soon as XMatchVisualInfo successfully returned a match for 32-bit depth for the OSD window (alpha transparency supported), then it was possible to define a "fully transparent" color, via XSetForeground, as 0x00000000 (as far as I see, that is 0xAARRGGBB format) - and use that to draw directly on the window with XFillRectangle -> that manages to clear the entire OSD window transparently.
The mouse pointer copy is rendered with a black background - how can I make the drawing of mouse pointer copy transparent, where it is black now?
Turns out, also this started working as soon as the window was created with XCreateWindow using settings from XMatchVisualInfo for 32-bit depth. By that, I mean that the result of XPutImage was such, that the transparent points in the cursor image were now "see-through"/transparent - whereas previously, the result of XPutImage showed black pixels at those locations.
But when the OSD window is shown in full, the leftovers make it seem as if the green pixels were drawn on top of the mouse cursor copy pixels. Why is this so?
Apparently, I didn't remember correctly what order the pixels were drawn in; when that demo capture was taken, indeed the mouse cursor pixels were copied first, and then the green pixels were copied on top. ( which now changes the question - how come the mouse cursor was visible in that capture, at all?! but now that the overall problem is solved, I'm not that curious :) )
Otherwise, few more notes on gist: xosd_track_cursor.c (a31e9dff5): since X11 has a client/server architecture, that means the user program can only queue requests to the server, and thus all of the drawing calls are asynchronous/non-blocking - and so, when we run, say, XFillRectangle and it exits, it does not mean that the drawing of pixels has been finished - just that the request has been sent to the queue, that ends up being sent to the server. Furthermore, in spite of commands like XFlush, XSync - there is never a guarantee that we can wait for a finished drawing operation; and there is no guarantee either that the server will honor any given request.
However, the less you try to do, the bigger the probability the X Server will honor the requests. So this version of the code actually makes a smallish window, 60x60 pixels, then sets it up so it is (centrally aligned) dragged by the motion of the mouse pointer. Then, the (main) mouse pointer is simply copied in this window at the same relative location.
Finally, there is a primitive attempt to do a mouse trail, by rendering two "ghost" copies of the mouse pointer, and have them be displaced by a history of mouse motion delta vector - the effect, as is visible on the gif, is not really amazing, but at least it's there as a "proof of concept", of sorts. Also, the window is setup at start as "click-through" using XShapeCombineRectangles - meaning the OSD window doesn't pick up/handle any mouse events (clicks) directly on it, instead everything is automatically passed to the windows below it, so the interaction remains the same, as if the program was not running at all.
(Note that to get the behavior of gist: xosd_track_cursor.c (a31e9dff5) shown in the gif, you should look up the defines DEBUGPRINT and MOUSE_TRAIL, and have them uncommented when you build)
During the development of my program I came across one drawing glitch that I was unable to solve. I am using GtkRevealer to show and hide a GtkInfoBar above another box holding a scrolled window with a GtkTextView (actually GtkSourceView) a lot like it is in gedit.
The whole thing is packed in a GtkPaned.
When I hide the infobar through the revealer, a black area in the text view appears and it disappears when a redrawing of the widget must occur (for instance when I click on the text view to place the cursor)
Additionally, this glitch does not appear if I enlarge the default resolution of the window, which for compatibility reasons is 640x480.
I understand that this may be hard to reproduce, but I am unaware who encountered the same problem. He maybe knows a workaround to this issue.
Also any idea is welcome.
I will draw exactly how the glitch looks on gedit (although it doesn't happen in gedit or it does but I cannot hit the correct resolution).
I tried to force a redrawing with gtk_widget_queue_draw() but nothing happens and it won't, because it has nothing to redraw.
I will try now to place the cursor automatically. It may work, but it is not functionally adequate.
EDIT:
Surprisingly it did not work. Grabbing the focus and placing the cursor
had no effect on the glitched visual behavior. I had to even make a new line
to fix it this time.
The fact that I have to resize the window with a value more than a specific point, may mean that the glitch may be due to alignment miscaulculations, but this doesn't explain why adding new line or a text mark in the gutter that colorizes the entire line also removes the black field.
Currently this bug has been encountered in another programs with a text view as well, but hasn't been reported as a bug yet.
Perhaps this has something to do with the fact that it is hard to reproduce.
I found that hiding the gtk text view with gtk_widget_hide() and then showing it with gtk_widget_show() works. The flicker of this visual glitch is not noticeable, perhaps due to GTK+ being event-based and waiting for the function to return to process changes.
I am working on a program that is supposed to simulate basic mouse input for a program while it stays in the background (meaning, I want to do other things in other windows with the actual mouse and keyboard while the target receives input). One thing I need to be able to do is move the mouse to a specific (x,y) point.
It seems to work for the most part, but in a certain region of the screen the message only works correctly some of the time. Other times, it moves to a consistent but wrong other point within the region. I am reading that sometimes it is relative, but if I spam the message repeatedly, it does seem to work consistently. Also, reading in Spy++, the messages that are sent by me actually moving my mouse are using what seem to be absolute coordinates.
My function is here:
void mouseMove(short x, short y) {
PostMessage(wnd, WM_MOUSEMOVE, 0, MAKELPARAM(x, y));
}
Preferably, I'd like for any (x,y) to be absolute so I can have the region treated as any other. But it would also be fine to set the position to a coordinate relative to the top left corner of the region.
I have tried just about all of the messages I've seen sent to the window in Spy++ before the WM_MOUSEMOVE but nothing is helping.
How might I approach this?
Edit, since details might not prove enough info:
I am sending these messages in this order:
WM_ACTIVATE 2 0
WM_MOUSEMOVE 0 MAKELPARAM(x,y)
WM_LBUTTONDOWN MK_LBUTTON MAKELPARAM(x,y)
WM_LBUTTONUP 0 MAKELPARAM(x,y)
I think what I initially thought (bad WM_SETCURSOR area) is wrong because Spy++ consistently shows HTCLIENT being used.
I have found a lot of people trying to do this (background input) and most seem to be told it's impossible. It's not impossible, and I think if someone can answer how to do this correctly once and for all it would be doing quite a few people a big favor.
Synthesizes keystrokes, mouse motions, and button clicks.
UINT WINAPI SendInput(
_In_ UINT nInputs,
_In_ LPINPUT pInputs,
_In_ int cbSize
);
PostMessage function. Places (posts) a message in the message queue
associated with the thread that created the specified window and
returns without waiting for the thread to process the message.
For PostMessage to work you need to be in the process as the target that you can achieve with SetWindowHook but the these links will do the jobs:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms646310(v=vs.85).aspx
https://msdn.microsoft.com/en-us/library/ms171542(v=vs.110).aspx
Let me first decribe the symptoms of the problem. I'll then give additional facts and explain my question.
Symptoms
I have written a custom Windows control. The control paints itself in response to a WM_PAINT message. It also uses tracking tooltips (ie the tracking feature of the TOOLTIPS_CLASS common control).
When I drag the mouse over the control, the tooltip nicely follows the mouse. The problem is that it leaves a grey streak in its wake. This streak takes an observable amount of time to repaint -- as you can see from the attached image, I was able to hit PRNTSCRN and take a screenshot of it before the control had time to repaint itself.
(What is even more peculiar is that the WM_PAINT handler doesn't appear to have run even once. But note that the code that causes the tooltip to track is in WM_MOUSEMOVE, and that is obviously being perfectly responsive.)
Facts
Please assume vanilla C using Win32 libraries.
The WM_PAINT handler is actually very fast. The control has a number of features that require repainting the entire client area, and this is imperceptible to the user.
Indeed, some features run animations that repaint the whole client area at 15-24fps.
It is also decently efficient and doesn't repaint much more than the update rectangle on any given repaint.
The WM_ERASEBKGND handler does nothing and simply returns 1.
I never erase the background, I simply paint over it.
The window has the following style bits set:
ws: WS_CHILD | WS_VISIBLE
ex: WS_EX_COMPOSITED
cs: CS_DBLCLKS
The parent window is a top-level window with the following style bits set:
ws: WS_TILEDWINDOW | WS_CLIPSIBLINGS | WS_VISIBLE
ex: WS_EX_WINDOWEDGE
cs: CS_REDRAW | CS_DBLCLKS
The control's window class background brush is GetStockObject(NULL_BRUSH).
The only other way I have found to cause the same kind of "trail" is by dragging another top-level window over my control. The area that is temporarily obscured by the dragged top-level window leaves the same trail.
Giving the control's window class the CS_SAVEBITS style doesn't seem to make any difference. I still get the same perceptible trail of slow repaints.
Questions
Why am I getting the grey at all, especially if I set CS_SAVEBITS?
What can I do to make the grey go away?
Should I call UpdateWindow() each time I move the tooltip?
But this doesn't solve the issue of other top-level windows being dragged over top of my control.
Help!
Adding the WS_CLIPCHILDREN style bit to the parent window made this issue go away.
For whatever reason, when a window is partly obscured and then revealed, the OS is very generous with the WM_ERASEBKGND messages and very stingy with the WM_PAINT messages. What was happening is that the parent's WM_ERASEBKGND handler was erasing over top of my control. Adding WS_CLIPCHILDREN causes the parent window to clip its erasing.
Funnily enough, this solution worked for my control, which simply ignores the WM_ERASEBKGND message, but didn't work for standard BUTTON controls with style BS_GROUPBOX style. I expect this is because of the same generous WM_ERASEBKGND policy. The standard button control probably dutifully erases its background in handling that message and then vainly waits around for a WM_PAINT message.
I've got a drawing area inside a scrolled window (with convenience viewport),
and this drawing area updates itself according to incoming motion-notify-events.
When I scroll the area though obviously no motion events are emitted,
but I wanted to work around this, and so tried to connect the drawing area's
"scroll-event" signal to the same motion-notify callback.
The problem I'm facing here is that this signal is emitted before the scrolled window
has update its viewport, so in the callback, I end up with pointer coordinates that were true just before scrolling, making it look like the drawing area is always "lagging a step behind" the actual pointer when scrolling while not moving the pointer itself.
I thought I could compensate for this by manually extracting the coordinates with gdk_window_get_pointer, but then I realized this cannot work as the pointer is technically still at its old position when the callback is commencing.
I also tried using g_signal_connect_after in hopes it would have the callback get called after the viewport was scrolled, but it didn't help.
My last hope would be to start a timer on scroll-events, and have the callback fire after a minimal amount of time, but this sounds realllly ugly, and I'd like to avoid that at any cost.
Any ideas as to how this could be realized?
Thanks in advance!
A solution would be to connect to the "value-changed" signal of the GtkScrolledWindow
adjustments.
Source: https://mail.gnome.org/archives/gtk-app-devel-list/2011-September/msg00014.html