I am trying programming an automated tester application using the new native Microsoft UI Automation interface 3.0 (in VC++ 2010, Win7). The Application Under Test (AUT) is a WPF application.
Almost everything works fine... I can install event handlers, navigate through the tree, search elements using various conditions and control the found elements using their patterns.
But yesterday I found a behaviour that leaves me despaired: The UIA tree of my AUT simply is not updated after switching its GUI main panel by clicking one of its main menu buttons.
After clicking the main menu button, I can see the new widgets in the AUT's GUI but the UIA tree still contains the controls that have been there before clicking the main menu button. The (outdated) UIA tree can be still completely read using search functions or using walker but of course it can not be written since the widgets do not exist anymore.
This exactly looks like there would be an outdated cache... however I do NOT use any caching UIA functions at all. None. Never. Nowhere.
I was not able to make the UIA tree update programatically... neither by calling any UIA functions nor by restarting the tester application nor by switching the AUT's GUI back and forth.
This does not happen every time. Sometimes after clicking the main button the tree seems properly up-to-date and everything works fine. However most of the runs it fails.
There is only one (mysterious) way that updates the UIA tree reliably: Using inspect.exe. When using inspect.exe tool for having a brief look into the AUT's UIA sub tree, the problem is suddenly gone and my tester application can access the actual, updated tree immediately! Of course the problem re-appears after restarting the AUT.
What does inspect.exe do to make the UIA tree (of another application!!!) update? How is it possible at all to access gone elements without using any caching? What did I miss?
I really need help.
Ok, some more findings:
UISpy.exe is able to refresh the UIA tree the same mysterious way inspect.exe does (this is especially strange because inspect.exe uses the same native interface as I do but UISpy.exe uses the .NET interface AFAIK). This means this is a kind of system wide and persistent UIA problem and not a pure native-UIA-problem.
The problem does not happend if I do not access the tree before switching the view. I.e. if my tester application does not access the AUT'S view before switching the view by clicking the main menu button, it sees the new widgets without problems. This strongly indicates some caching problem of the native UIA API - even if I have no idea how this could happen because I do not caching at all. Does someone know if there is some internal caching taking place?
I think this could be a API bug. However considering my current experiences with Microsoft Connect I am kind of lost with that showstopper :-(
Someone any idea?
I also tested the Snoop tool. Using Snoop does NOT heal the problem temporarily like Inspect and UISpy do. Regarding Inspect.exe, there is another detail... it is enough to collapse and expands the AUT's sub-tree to temporarily heal the problem.
Ok, update on this. UIA simply seems horribly buggy. I found that reading an element's content array and the length of that array updates the hidden caches. I also had a case where a subtree of a tab was not update but could be updated by switching to another tab and back (reading the content by pattern did NOT help in that case). Both cases were reproducable but I could not find any way to predict or prevent them. Also al lot of third party WPF components seem to be buggy. We finally gave up to use that API.
I know the question is quite old but I figured out what Inspect.exe does to refresh the UIA tree for other applications: look at the Options menu; there's an item labeled SPI_SCREENREADER flag which is checked by default.
Do the following in your code and you will probably get a fresh UIA tree:
SystemParametersInfo( SPI_SETSCREENREADER, TRUE, NULL, SPIF_UPDATEINIFILE | SPIF_SENDCHANGE);
PostMessage( HWND_BROADCAST, WM_WININICHANGE, SPI_SETSCREENREADER, 0);
Related
Returning user here. I have a requirement to migrate a system of OpenMotif 2.3 apps currently running under RHEL5, Gnome Desktop 2.x, to Gnome 3 under RHEL7. The problem is that the apps require modifying the window menu, adding items on a per-window basis and accelerators. Currently we are required by our customer not to migrate to GTK or any other widget set. Customer security requirements require using gnome-shell, gdm, et al, so xfce or anything that uses an external process for screen locking is unusable.
I've looked at the mess that is gnome-shell, and while I've found the JavaScript that assembles the window menu, it requires access to the window property holding the customized menu. I see no way to have JavaScript call back into C code to do this. (If all else fails, I could use xprop to dump the property and go from there.)
I'm thinking client-side decoration, with all of its horrors of having to manage windows yourself, might be the solution here. I can't seem to get CSD to kick in, though, without using GTK. I've tried applying the _GTK_FRAME_EXTENTS property, but gnome-shell ignores it. I've tried _NET_FRAME_EXTENTS with equal lack of success. The Extended Window Manager Hints documents say nothing about this. Google, of course, is ripe with comments about using GtkHeaderBar and discussion of avoiding CSD, neither of which apply here. Wayland-based items are equally unworkable.
I am quite familiar with modifying & extending the Motif widget set, having done that for a good chunk of my career here. I need to know how to tell the Window Manager to lay off without making the window override-redirect.
I'm trying to use MS UI Automation to test a WPF application, and am using the Inspect Object tool (inspect.exe) included with the Windows SDK to look up the AutomationId property on certain elements.
Inspect is behaving very strangely for me:
If I close all applications and start the WPF application and Inspect, inspect is able to see the AutomationId property for various UI elements. Elements which do not have an AutomationId simply show two quotation marks denoting an empty string ("").
After I perform a few actions in the WPF application, inspect.exe hangs and I have to kill it and restart it. Even though the machine's CPU and RAM utilization are around 50% or less, I've tried waiting several minutes--possibly close to 20 or 30 mins on a couple occasions--to no avail.
After restarting, inspect.exe can no longer find an AutomationId for any UI element, even those which did have them previously. What's more, the property is completely missing when hovering the mouse over the WPF application--it is no longer listed at all, not even with an empty string value.
If I move the mouse to another screen (specifically, to another computer, using Mouse Without Borders), the AutomationId property reappears with a value of "FormDot"
If I restart only inspect additional times while the WPF application is still running, inspect still behaves the same as after the first restart.
If I restart only the WPF app while inspect is still running, inspect still behaves the same as after its first restart.
If I close both inspect and the WPF app, then start inspect, then start the WPF app, everything works correctly for a while and inspect finds the AutomationId on a few elements in the WPF app...up until the point at which inspect hangs again.
I've tried running inspect both normally and as an administrator as suggested in https://stackoverflow.com/a/7833728/44737, and it behaves the same either way.
What, if anything, am I doing wrong? Am I just too impatient and do I need to wait a really long time instead of assuming inspect is hung? And why does inspect's behavior regarding AutomationId vary?
There are more than one version of Inspect.exe. The latest to my knowledge is the one dated from 2012 that says version 7.2.0.0 in the help/about dialog box.
The old one doesn't have a tree view on the left with all detected automation elements displayed in a tree, so it's easy to check you're using the right one.
The latest one works quite correctly, however, IMHO, the best tool so far to work with UI Automation is Visual UI Automation Verify. It's a .NET program, and he source is available here:
UI Automation Verify (UIA Verify) Test Automation Framework.
Note that although it's a .NET program, it doesn't use the standard .NET automation dlls (more on that here: What's the difference of UISpy.exe and Inspect.exe? (From Microsoft Windows SDK)).
About the AutomationId property, to clarify my initial comment to the question, I meant its usefulness depends on the program that you're trying to automate.
If you own it as a developer, it's clearly interesting. For example, if you're working with WPF, you can use the x:Uid property, it's clearly meant for UI automation. In the Winforms space, it's also quite useful because UI Automation will use the control's AccessibleName by default and revert to the Name as a fallback, for the AutomationId value.
But there are many apps that don't rely on .NET (browsers, native apps, etc.) Usually, for these apps, it's easier to use other properties.
I have been using inspect.exe for a while on a Microsoft Surface Studio PC (running Windows 10), and my experience is that inspect.exe will hang much more frequently (sometimes always) when Windows Updates are pending. When the updates are out of the way, inspect.exe is still somewhat slow, but much more stable.
Snoop allows you to look inside the application and change element properties.
Its a great asset for developers, but can be a security issue in some cases, like when we have users who like to look in places where they shouldn't be looking. Is there a way to do something to block applications like Snoop from "snooping" your application?
And if there is no way to block it, what do you recommend to do to minimize security risks?
Snoop is a utility that allows you browse visual tree of a wpf application and view and change properties. Its very useful when you are trying to debug something and have no idea what is going on. You can find more here.
Thank you.
There actually is a way to detect whether your application is being "snooped" by the snoop program. The solution I will give is not a silver bullet, and if someone really wants to snoop your application, they'd have to modify the snoop source code (it's an open source project).
What snoop actually does is it injects an assembly into your application, and the injected assembly recursively examines your applications visual tree starting at the root. In other words, snoop actually runs inside your application. That being said, the solution is to raise an event when the snoop assembly is injected into your application.
First, you need to subscribe to the assembly-loaded event somewhere in your application (preferably the begginging):
AppDomain.CurrentDomain.AssemblyLoad += new AssemblyLoadEventHandler(CurrentDomain_AssemblyLoad);
Then, you would implement the handler somewhat like this:
void CurrentDomain_AssemblyLoad(object sender, AssemblyLoadEventArgs args)
{
if (args.LoadedAssembly.FullName.StartsWith("ManagedInjector"))
MessageBox.Show("hey you, stop snooping");//and shut down your application.
}
You would probably need to enhance this solution for real silver-bullet solution, but at least this solution will definitely stop the current latest version of snoop being run as is (without the code modified). The better solution would be to check that no external assemblies are being injected into your application.
However, Kent is still right in the fact that a utility like Snoop should not cause any security vulnerabilities since security should not be implemented at the UI level. But at least this shows you how to prevent people from "snooping" your application.
By implementing security properly. If your "security" can be thwarted with a tool like Snoop, then you're doing it wrong.
Suppose there's a command that only certain users can execute. It sounds like the only place you're enforcing this is at the UI level (by disabling the corresponding button, for example). That being the case, you're right - I could easily use Snoop to enable the button and execute the command. But you should be enforcing the security constraints on your server, or perhaps in your command execution logic if you have no server. Basically, security should be implemented as close to the thing you're trying to protect as possible. Security at the UI level is merely for convenience of the user.
Great responses for a very good question,
I want to add an example of the password that should never be saved in your datacontext, as snoop check also the datacontext of the UI, so if you use the right control PasswordBox you will find that you can’t bind the password property, that means even if you snoop the application you can’t get the password as it is not saved in any property
But we found many people who try to create a work around (helpers, behaviors, usercontrol ...) to bind the password, but they forget that snoop can get it like that
So your server should never trust your client , every time it should check the right privilege (as we can sniff the client request using Fiddler for exp then rebuild a custome request to by pass client application security)
When hosting WPF user controls within a WinForms MDI app there is a drawing issue when you have multiple forms that overlap each other that causes very distinct visual artifacts. These artifacts are mostly visible after dragging one child form over another one that also hosts WPF content or by allowing the edges of the child form to be clipped by the main MDI parent when dragging it around. After the drag and drop of the child form is completed the artifacts stay around generally but I've found that setting focus to a different application's window and then refocusing back on to my application window that it is redrawn and all is good again until the child forms are moved once again. Please see the image below which demonstrates the problem.
Those at Microsoft insist that the WinForms MDI is already a sufficient solution for MDI and doesn't need reinventing in WPF although I find it hard to believe they tried creating a WPF app this way because of the obvious shortcomings.
UPDATE: A few extra notes that I left out is that if I create these Forms without setting the MdiParent they are created as regular forms and this issue doesn't happen. This issue seems unique to the WinForms MDI scenario. Also I've currently running on Windows 7 Enterprise and I'm aware the results may be quite different on Windows XP but I haven't been able to test this.
UPDATE: I've found a few other related resources on this issue that I thought I should share.
elementHost repaint problem in MDI
application
elementHost repaint problem in MDI application on Tech Archive
It appears that another workaround is to revert to software rendering as opposed to taking advantage of hardware acceleration. This was the suggestion by Marco Zhou on the MSDN Forums.
public partial class UserControl1 : UserControl
{
public UserControl1()
{
InitializeComponent();
this.Loaded += delegate
{
var source = PresentationSource.FromVisual(this);
var hwndTarget = source.CompositionTarget as HwndTarget;
if (hwndTarget != null)
{
hwndTarget.RenderMode = RenderMode.SoftwareOnly;
}
};
}
}
I've tested this and this solution seems to work very well and so far is the only solution that I've found for solving this problem within a FoxPro interop scenario which is very similar to the WinForms one I posted about originally. For now I'm planning to use my original Refresh on the MDI Parent solution for my WinForms project but then for my other native interop applications such as when my WPF controls are hosted in Visual FoxPro I'll use this solution. That is unless of course if a more elegant solution is discovered for either of the cases.
Also it's important to note that from what I'm aware software rendering is the only option on XP systems and normally Visual FoxPro nore WinForms normally take advantage of the same type of hardware acceleration that native WPF apps do on Vista OS and up. So using this option may not be as bad as it sounds when you do have to deal with interop. Currently I'm not aware of any related side effects when using this solution but if there are any those would have to be taken into serious consideration.
Well, I may have found a solution although it feels like a bit of a hack. It appears that if you call the Refresh method on the MDI parent whenver a child MDI Form is moved that the noted artifacts go away. Visually things appear a bit jittery when dragging a window but it seems much more acceptable than the example I showed in my original post.
private void Form1_Move(object sender, EventArgs e)
{
this.ParentForm.Refresh();
System.Diagnostics.Debug.WriteLine(string.Format("Form Moved to: ({0},{1})", this.Left, this.Top));
}
I've tried many combinations in the same vein such as refreshing just the child window that was being moved by calling methods such as Update(), Invalidate(), Refresh() and also I've tried these same methods on the MDI parent as well as Dispatcher.Invoke(DispatcherPriority.Render, ...) and InvalidateVisual() on my hosted WPF control but none of those other methods worked accept for calling Refresh() specifically on the MDI parent.
I realize that this probably isn't the optimal solution since I'm forcing the whole main application window to refresh every time a child window moves a few pixels but as for right now it's the only reasonable solution that I found that works. If anybody else has any alternative solutions or any improvements upon this I will gladly accept your answer instead.
Check video drivers and try disabling hardware acceleration. Most artifacts are caused by bad drivers, failing video card, or insufficient time to complete the refresh.
First troubleshooting step: Update video drivers. Obvious, I know.
I had similar issue, checking my video card settings (NVidia Control Panel) showed global setting set very high causing a longer refresh interval which may be aborted if taking too long. Setting my settings back to defaults resolved most of the issue. But I also run hashing programs which use the GPU intensely so this is likely the cause of my remaing artifact issue which is very seldom now and mostly shows its ugly face in Visual Studio.
Another troubleshooting step I ran across is to disable hardware acceleration for WPF, this can be done in 'HKEY_CURRENT_USER/SOFTWARE/Microsoft/Avalon.Graphics', or maybe an application can do it BUT this is only for troubleshooting; never set these within an application because it will disable for ALL WPF applications. I do not have this registry setting nor did I add it so I am not sure of the success with it, but many say this resolved their issue. Also note some applications have this option available, try disabling it if available.
Another troubleshooting step is to make sure the video card is a proper tier level for rendering. Any card that supports DX9 or greater should be sufficient, but other factors are involved (as is my case) so just because it is on the list does not mean it is adequate for your purpose.
Finally, you can use the Visual Profiler (part of Windows SDK), and other tools, to help determine what is going on more precisely with WPF lacking performance in relation to graphics ability.
Rendering Tier level notes and WPF Performance information --> http://msdn.microsoft.com/en-us/library/vstudio/ms742196(v=vs.90).aspx
Hope this helps someone.
--Ryan Strassburg
Your usercontrol or window loaded event ;
this.WindowState = System.Windows.WindowState.Minimized;
this.WindowState = System.Windows.WindowState.Normal;
it may seem bad solution. no need to hit your head against the wall.
A Turkish proverb says: the best code is the code is running :)
It seems like a common question on Google, but I couldn't find a satisfactory answer (unless the answer is 'no')
I would like to add menu items or show a custom menu when a user right-clicks on my Silverlight app.
The closest thing that I found catches the context menu in IE, but not in Firefox, and Chrome shows the context menu and then shows the custom event.
(The tutorial I mentioned was here
http://silverlight.net/blogs/msnow/archive/2008/07/01/tip-of-the-day-14-how-to-right-click-on-a-silverlight-application.aspx)
Here you have an implementation but it's CTRL+Left click, then you get a context menu (about the text, sorry it's in spanish):
http://www.albloguera.es/?p=29
Another option, is the one pointed out do it via javascript, quite chunky solution, and not sure if now it's working in the current 2.0 version :-(.
Good Luck
Braulio
There is no way to do this natively in Silverlight.
I believe that the Telerik control noted elsewhere uses the HtmlBridge to intercept the context click, then draw their own menu, however it does not appear to be working with Silverlight 2.0.40115 now, so maybe Microsoft have closed that gap.
Having no right click is a design decision based on security concerns, rather than a technical one. However I believe that the Microsoft team are considering allowing context menu's in future versions (perhaps even v3)
I've come to the conclusion that it seems it is possible based on my use of Telerik's context menu which works fine on IE, Chrome and Firefox.
Check it out, http://demos.telerik.com/silverlight/#ContextMenu/FirstLook