I don't want to go too much into detail about functionality, but our application's window resizes when a child control is added to or removed from it. During stress tests performed recently we noticed high peak working set of the application in Windows task manager. The child control was added and removed every 200 ms, and so the window became higher or smaller every so. We performed these tests on VMware Workstation Pro virtual machines (Windows Server 2012, Windows 10).
After at most a couple of hours the application would always report out of memory exceptions, and said window would freeze or disappear. We noticed that the working set was normal after the crash (below 100 MB), but peak working set was always over 1 GB (~1.1 up to ~1.6 GB) at that point.
To find out whether the issue was caused by a bug in our application or not, we decided to create a single-window WPF application with only a timer that changes main window's height every 200 ms. It turned out that this application had the same issue.
The thing is that I couldn't find information concerning similar issues, so I decided to post a question here. How would you explain such behaviour?
Let me also note that:
When the window was minimized, the issue would never occur.
It was hard to reproduce the issue on a host (not a VM), but we happened to observe it there as well (on Windows 8).
We also performed tests on a virtual computer (WMware ESX 5.5). The crash would take place much faster when nobody was looking :-). I mean we connected to a VM from vSphere Client and opened console to start our test app on the OS. Then we switched to another tab of vSphere Client to hide the console. When we returned to it later on (let's say in half an hour), we could observe memory usage history of our test app with a slowly rising slope that fell instantly at the moment the console was displayed.
I will be grateful for any information that could help me understand the reason of this issue. Thank you.
-- EDIT --
I started my test app in two instances on a VM (Windows Server 2012) - both compiled for Any CPU, but one of them had the Prefer 32-bit check box unchecked in Visual Studio. So one started in 32-bit mode and the other in 64-bit mode. The 32-bit version crashed in an hour or so, with peak working set below 150 MB. The 64-bit version kept running. I'm wondering if this might be the way to go...
The following is problem signature of the 32-bit application version:
Problem signature:
Problem Event Name: CLR20r3
Problem Signature 01: ResizeStressTest 32.exe
Problem Signature 02: 1.0.0.0
Problem Signature 03: 5762641e
Problem Signature 04: PresentationCore
Problem Signature 05: 4.0.30319.34209
Problem Signature 06: 5348b56b
Problem Signature 07: 283
Problem Signature 08: 13
Problem Signature 09: System.OutOfMemoryException
OS Version: 6.2.9200.2.0.0.272.7
Locale ID: 1031
Additional Information 1: 5861
Additional Information 2: 5861822e1919d7c014bbb064c64908b2
Additional Information 3: 5c1f
Additional Information 4: 5c1f554df00553b422a8baa03b19335a
Read our privacy statement online:
http://go.microsoft.com/fwlink/?linkid=190175
If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt
When I have to track down memory leaks like the one you're describing, I always turn to WinDBG
It's free and it's very effective. It has a very steep learning curve though. It will tell you how many instances of objects are in memory and lead you to what is holding onto their references
Related
Observed behaviour (everything here is on Windows 10):
I run Visual Studio (tried 13 and 15, both behave the same) logged into user A
After starting up, VS takes virtually no CPU time (<1%)
I log into user B, without signing out of A
VS imediately starts using A LOT of CPU time (~25% on my 4 cores with hyper threading)
I can go back and forth between A and B, and it goes back and forth between low and high CPU usage
This is all without any projects or files opened, though it also happens in that case.
I noticed this because I was originally investigating similar behaviour of a WPF application (after a user reported this issue).
While trying to isolate the problem, I found that even a completely new WPF project, with just a single empty window, behaves exactly the same (whether or not run through Visual Studio).
Through profiling and debugging I found that the app seems to spend a huge amount of time handling windows messages.
Specifically I found that it seems to be almost exclusively WM_PAINT messages (we are talking easily hundreds or thousands of messages per second - as many as the CPU can handle it seems).
No other programs I have running (chrome, skype, sublime text, ..) behaves this way.
Has anybody else seen this kind of behaviour?
And/or any ideas what could cause this, or how I could investigate this further?
Naturally, I cannot fix Visual Studio (unless the problem is with my setup somehow) but I hope there is something I can do about my WPF application.
As per Hans Passant's suggestion in the comments, I reported this problem to Microsoft here:
http://connect.microsoft.com/VisualStudio/feedback/details/2390593/wpf-apps-use-a-lot-of-cpu-time-when-logged-into-different-user
As it turns out this indeed seems to have been a bug in WPF, which is fixed in the current version of Windows 10 (probably specifically since the Anniversary Update (version 1607)).
Hence the solution: Make sure to update your OS.
We have a higher-end Win7-64 Dell precision workstation notebook with an i7, 8 gigs of ram, tons of hd space and running dedicated AMD graphics. The machine is about a month old. It was one of the highest-end we could get at the time.
What we're experiencing is when we run our WPF/SQL Server (local) app, it tends to hang and stall, sometimes completely crashing, but mostly just hanging until we force it to close. However, the exact same installer running in a VMware virtual machine running on that same machine runs flawlessly. Actually, the VM install runs better than a lot of native installs on other machines. It's very snappy with no hangs or hesitations at all. But again, same app, same installer running direct in the OS, and we're back to the issues above.
We've ran all Windows updates.... we've tried completely reinstalling everything... .NET frameworks, SQL Server, video drivers, even updated the BIOS and checked for rogue services but it still happens.
At first we thought it was Symantec AV's real-time protection because when we first shut that off, things started getting snappy again (and slowed down and froze when it auto-re-enabled itself furthering this hypothesis) but then it just started slowing down again, and more surprising, that same AV is running in the VM without issue! Checked the exceptions but there weren't any.
We even tried forcing WPF to run in software-render mode but again, nothing.
Now the odd thing is this only seems to be happening on this and a few other machines, but we can't seem to find anything in common except they're all running Win7 64-bit. As such, we have absolutely no idea where to start. And since most are hangs, not crashes, we can't even look at the crash reports.
So can anyone give us any idea what else we can look at? This is holding up us shipping a three-years-in-the-making major release of our software so to say this is a show-stopper would be an understatement. We've been stumped for about a month now and getting nowhere fast.
Found it!! Turns out there's a bug in .NET 4.0 regarding UI Automation and the changes MS introduced. Here's the info, and the fix! (Note: Even if you call MS, they will send you a link, but it's always a broken link. I managed to track this down manually.)
Note: Their article talks about a specific case that causes this behavior, but if you google around, you'll see tons of issues around hangs related to those DLLs. The latest is they're promising a fix in the .NET 4.5 runtime (from a MS post on this issue.)
Here's the KB article...
http://support.microsoft.com/kb/2484841/en-us
...and here is the actual hotfix.
http://archive.msdn.microsoft.com/KB2484841/Release/ProjectReleases.aspx?ReleaseId=5583
Apparently the VM didn't suffer from this. We weren't sure if the VM had the hotfix applied or not or if this only happens on non-virtualized machines. Still, this solved all of the issues and the app is now snappy again. (Man, was this fun to track down! Ugh!!)
First I'd like to head off the "wait a minute" questions.
Yes, I've read every thread with similar titles and subjects on SO and beyond.
Yes, I'm using the very latest version of SP1 and any potential hotfixes that might be related to this problem.
So, with specificity, here is my problem:
When I'm debugging Silverlight applications and it hits a breakpoint, everything works just fine as long as I use the mouse. I can continue, step, stop debugging, disable and enable breakpoints all just fine -- as long as I do so with the mouse.
But when I use the keyboard, Visual Studio freezes for a number of seconds, sometimes over a minute. This is any use of the keyboard. But after a short time, it starts to work just fine until that debugging session is over. But the next debugging session (regardless of whether I use the same instance of Casini or not) it will do exactly the same thing.
When I say any keyboard use, I mean it. F5 (Continue), Shift-F5 (Stop Debugging), F10 (Step Over), as well as typing in the Immediate window or even hitting the ALT key to highlight the menu.
For instance, if I try to type something in the Immediate window, it freezes for about 15 seconds and then shows the first key and repeats the pattern about 5 times until it releases and everything is just fine.
When I say "freezes" I'm not talking just Visual Studio but it hangs the entire operating system except for my ability to move the mouse. Even so, the mouse cursor remains what it was when it froze regardless of the program it's over, such as the text selector.
It only happens with Silverlight and it doesn't matter which version I use.
Here's what I'm running:
Windows 7 Professional 64-bit, all of the latest updates included, 12 GB RAM & 1.5 TB HD.
Visual Studio 2010 Ultimate with all of the latest updates.
All software is legit and I've already tried removing and/or disabling just about everything that can be removed or disabled, including extensions. I've used Process Explorer from Sysinternals to kill or stop everything I can. I've also tried it on a very simple Silverlight application with nothing more than one line of code that doesn't access anything.
EDIT: Forgot to mention that I also don't have it loading symbols beyond the "Just My Code" that it normally would and I've tried every debug setting that I can think of as well. And Intellitrace is also turned off (since I do have Ultimate), but that makes no difference either.
UPDATE (11/18): It gets more and more odd. I determined that the temporary freeze doesn't occur only when Visual Studio has focus. When VS (during the super-simple Silverlight app) goes into debug mode, any keyboard access AT ALL causes the same effect. SO, it appears that it has to (in some way) be related (at least in part) to my Windows setup, contrary to my assertion in the comments, but I cannot possibly fathom how. And if other programs are doing something when it freezes, the delay is longer (which makes a certain amount of sense with the symptoms). I'm still at a complete loss.
Anyone have any ideas or see this before? It's really frustrating.
I solved the problem.
It turned out to be the LastPass toolbar. It's a very handy thing and I've used it for years without issue. Yeah, yeah, toolbars are evil, I get it. :)
Disabling it in IE fixed the problem completely. I can still use it in FF (which is my main browser anyway except for working for which I use IE) and everything works fine.
Even so, it's bizarre to me that the toolbar would only have an effect when SL debugs. I didn't have an issue with ASP.Net debugging and I didn't any issues when the SL app was actually running, only when I hit a breakpoint AND hit a key within the first 15 seconds or so.
In any case, problem solved and thanks for your suggestions.
Try changing registry setting LowLevelHooksTimeout from 5000 to something low, like 300.
It can be at
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Policies\Microsoft\Windows\Control Panel\Desktop
or
HKEY_CURRENT_USER\Control Panel\Desktop
(first setting at Policy key seems to globally override user setting).
I am writing code using VS.Net 10 and SlimDX to render 3D content on a D3DImage. It works perfectly under 32 Bit Windows XP. However, after migrating to 64 bit Windows 7 (quad core and 4 GB Ram), the same code does not work any more. The symptom is that after rendering about 10 or 20 times, the D3DImage's IsFrontBufferAvailableChanged event is raised and the property of IsFrontBufferAvailable has a value of false.
I have checked everything I can think of, e.g. RenderCapability.Tier, calling SetBackBuffer, checking the device (in fact it is DeviceEx type) after the front buffer is lost, updating video card driver (nvidia 9500 GT 1G RAM), etc. None of them worked.
One thing that may contribute to the problem is that the image control which uses D3DImage as the source is not created on the GUI thread. I am doing to reduce the work load of the GUI thread to make the application more responsive. Of course, I have been using Dispatcher.Invoke to avoid threading problems. Again, it works perfectly in XP.
Any help is much appreciated. Thank you in advance.
I think there is a x64 version of the slimdx.dll.. if you are using the x32 version, that could be the problem.
I've been working on this problem on and off for about a month. I don't expect anybody to be able to give me a definitive answer. I'm just completely out of ideas at this point and could use anything.
The problem is that my app crashes on only some models of phones. I have an HTC surround and it runs fine. It has also been tested on a Samsung Focus and it works there to. It crashes pretty consistently on a HTC mozart. There are other phones it crashes on but I don't know what models they are. I don't have access to an HTC mozart so debugging has been very difficult.
I'm handling the application UnhandledException event and I have try catch around every background thread. The error handling code never runs.
What I know:
Sometimes it freezes and requires a press of the power button. At least once it required the battery taken out. Most of the time it freezes and then crashes.
Most of the time it crashes on the main menu, before everything displays.
It's not 100% consistent. Sometimes it works for a little bit, but never very long.
It's not because it's out of memory. Most of the time it crashes while using less then 8MB.
When all the exception handling didn't work I added debug logging. This slows things way down but at the same time the issue goes away.
These symptoms make it sound like a deadlock to me. Although I have gone over the code and there is no thread ever entering more then 1 lock at a time.
Any ideas on how I should track this down would be appreciated.
Edit: This is a WP7 version of my game. I've just been able to confirm with a volunteer with an HTC mozart that the simplest conversion of the code meant for to run on the web will crash on the phone. That's code has no networking, isolated storage, or sound.
I also should have mentioned that this has passed certification and was in the marketplace for a few days until I took it down because it got bad reviews (because it was unplayable for some people)
If the devices are locking up, then my suspicions would be to look at the areas where you are closest to the driver level/hardware, which (looking at your game) are:
the display
the sound
It could also be just about due to processing/CPU activity - but generally "User level" code shouldn't be able to lock up your phone - that functionality is reserved for kernel software.
The only way of really testing this is to get hold of a device where this "reliably crashes" (e.g. a Mozart) and to go through the process of disabling bits of functionality one by one.
If this is a Silverlight app, then I'd expect Microsoft to want to help - I'd contact them via AppHub and via their local Evangelist team - they'll have the means and the motivation to assist you.
I'm happy to assist on testing on a developer-unlocked HTC Trophy if that helps!
The only thing I can think of that's unique to the Mozart is that it has an 8MP camera and all others have a 5MP camera.
Are you using a CameraCaptureTask and expecting the returned image to be a certain size?
Other than that, what does the app do? what services and device functionality are you using?
Do you have any network access running on a background thread?
I had roughly the same problem when developing my WP7 app. As far as I can tell it isn't as much model bound as it is device bound. In fact I had my app deployed to 15+ Trophy's (my company gave all their employees one of those) and it would repeatedly crash on some of them all having the same firmware. Some of the feedback I got through the reviews seems to indicate that it also happens in the wild.
In my case the crash occurs primarily (only?) when the app is launched. There does seem to be a strong correlation between internet connectivity and crashing in that I can 'recreate' the crash by putting my phone in flight mode, unplugging the network cable from my computer and then deploying the app. In that case too it immediately crashes and no event or break point is ever raised.
My gut feeling tell me it might have something to do with the map control as that does tend to respond funky to poor connectivity when the application is loading (like displaying an error message that the card cant be loaded while simultaneously displaying the map)
Does your app also use the Map Control (in combination with a Pivot control perhaps?)
Some of the devices have issues where you need set things to "content" instead of "embedded resource" in your project.
Although i've more heard about that issue related to app startup time, as on some devices (HD7?) the load of the app was taking enough time that the app was never allowed to start, the OS thought it was taking too long and killed it.