Im trying to create a PNG sequencer class that will allow me to change an ImageBrush's ImageSource property via an animation.
The issue is that I have around 150 PNG files to load, and it really really affects performance when I have a few animations on the screen.
I have read a little about RenderTargetBitmap and also WriteableBitmap but Im not sure how to get a big performance boost, because I really do need it.
Im getting down to 6fps in some cases, which is obviously not acceptable.
In my Sequencer class, I just update a CurrentFrame DP that changes the ImageSource property of the ImageBrush.
Any ideas on how to increase the performance here?
Step 1 is loading all your images in ahead of time (preferably on a background thread). You should have your BitmapImage objects initialized with the CacheOption = BitmapCacheOption.OnLoad. You may already be doing this or it might not be the problem (the images cache by default).
However the rendering thread also needs to do some work when you change the image source. If you're not displaying at the source image size, that might be a problem, as by default the Image control uses the high quality Fant scaling algorithm. In that case you could get a performance increase by calling RenderOptions.SetBitmapScalingMode(uiImage, BitmapScalingMode.LowQuality); on your Image. Low quality scaling is orders of magnitude faster. However even after that there's still a bit of work involved. If you want to get the fastest animation possible, you can create an Image control for every frame, then overlap them all on the same place and change which one appears on top. You'll still take the hit on the render thread loading all the images in, but the actual animation should be quite snappy.
Related
So here's my setup:
Camera images coming in at 1920x1080 # 25 FPS
Writing image data to WriteableBitmap on UI thread (simple copy, no processing)
Two Image controls in two different windows on two different monitors has their Source property set to the WriteableBitmap
Some generic UI stuff goes over the camera images
This works great, uses about 4% CPU on an old laptop (8 logical processors). The video is as smooth as can be. However, my UI has some animations (stuff moving around). When the camera display is running, those animations gets choppy.
Right now, the camera image is in Gray8 format, so it will be converted (I guess when calling WritePixels?). I also tried forcing one of the animations to 25 FPS too, no change.
Where should I start to resolve this? Is the issue that I'm locking the bitmap for too long, or is there something else going on? From what I can see locking the bitmap will cause the render thread to block, so moving that to another thread seems pointless. And it does feel like that somewhat defeats the purpose of WriteableBitmap.
This is always going to be tricky because you're capturing at 25FPS whilst WPF tries to update at 60. It's difficult to offer any meaninful advice without seeing a testable project but I'd probably start by doing the updates in a CompositionTarget.Rendering handler.
I am trying to smoothly scroll some images across a window using DirectX11 via SharpDX in a WPF application.
A bit of background:
The images are signal returns, varying by time, that have been loaded from a file and loaded into D3D as a texture array, and the image itself is rendered as a series of textured quads (a series of adjoining rectangles in a very long horizontal line).
The viewport is set up so that the left and right edges represent time offsets, and the signal is displayed for the time period that falls between these times.
The DirectX implementation is very simple; the quad vertices are generated once, and a very small per-frame buffer contains a simple 2D world-view transform that updates scale and translation according to the current visible range/zoom etc.
As far as I can tell, the D3D implementation is not part of the problem I am having - it really is a very simple set up, and seems to be rendering extremely quickly (as it should), and although I do have some fancier stuff (streaming of textures from disk as required), I have disabled all of these and am running with very simple (small) textures whilst I try and resolve the issue I am having..
The problem:
Scrolling of the image is not smooth when the visible time range is animated. The image "jitters" much of the time, and frankly looks awful (when it is not jittering, it looks great).
The setup:
DirectX is rendered to a D3DImage, with a bit of work going on behind the scenes to make DX11 work - this code is taken from https://sharpdxwpf.codeplex.com/
There are multiple D3DImages (up to a total of 4), arranged in a grid (I have been testing with two, both of which contain signals for the same time period and are animated together).
These D3DImages are drawn by a DrawingVisual (which is hosted by a custom FrameworkElement) where they are used as the source of an ImageBrush. The frameworkelement object triggers a render as required, and the drawing visual handles the D3D render call and draws a rectangle to fill the control using the D3DImage brush.
The time range value is animated using a WPF DoubleAnimation. The visuals that are currently displayed are bound to this value via INotifyPropertyChanged and trigger a render (via InvalidateVisual) on each change.
DrawingVisual render code, triggered by change of "Position" value (start of visible time range):
// update scene per-frame buffer first, with world-view transform
using (DrawingContext dc = RenderOpen()
{
_scene.Renderer.Render(_draw_args);
_image.Invalidate();
dc.DrawRectangle(_brush, null, new Rect(viewer.RenderSize));
}
What I have tried:
I have tried a number of things (and searched a lot) to try and determine whether the issue is due to jittery render request timing, or if the issue is further into the render process.
Hooking into CompositionTarget.Rendering
Firstly driving the update of the position value via CompositionTarget.Rendering, and leaving the rest of the process as-is (i.e. elements react to the change of this value):
(needless to say, this is very much "test" code):
Stopwatch rsw;
long last_time = 0;
void play()
{
rsw = new Stopwatch();
last_time = 0;
rsw.Start();
CompositionTarget.Rendering += rendering;
}
void rendering(object sender, EventArgs e)
{
double update_distance = rsw.ElapsedMilliseconds - last_time;
Position += 10 * update_distance;
last_time = rsw.ElapsedMilliseconds;
}
Results - worse than the simple double animation.
Using a DispatcherTimer. Similar to above, using a timer in conjunction with a stopwatch to judge the elapsed time a little better. Worse results again, with the interesting side note that CPU usage dropped from about 7% (half a core) to 0.7%.
The original attempt, along with variants 1 & 2, all try to update a single value that then triggers the render of interested parties. Out of interest, I logged the time differences between render requests as they reached the Drawing visual - although the DisptacherTimer actually gave the most consistent results there (both WPF animation and CompositionTarget dropped the odd frame), the DisptacherTimer also gave the most jittery animation.
Updating the image, but not invalidating the visual. Updating the source of an ImageBrush updates the displayed image, without having to re-render the DrawingVisual. This method produced very jittery results.
CompositionTarget.Rendering in the framework element itself. This produced the best results of the lot, but still not perfect. Jitter would happen, then dissipate, only to return again. In this approach, the framework element that holds the DV hooks up to CompositionTarget.Rendering, and the visual queries the current position, which is being animated independently. I could almost live with this approach.
Attempting the wait until the D3D scene is rendered, before invalidating the image (no discernible improvement):
_scene.Renderer.Render(_draw_args);
_scene.Renderer.Device.ImmediateContext.End(q);
while (!(_scene.Renderer.Device.ImmediateContext.IsDataAvailable(q)))
Thread.Yield();
_image.Invalidate();
Observations:
I really don't think this is a performance issue as such. My dev machine has a good graphics card, 8 i7 cores etc, and this is a simple rendering operation.
This really seems like some sort of synchronisation issue between D3D and the WPF rendering, but I have no idea how to begin looking into this.
If I have two images animating in parallel, the jitter is much more pronounced on the first of the two (usually).
If I actively resize the window whilst animating, animation is perfectly smooth (despite the fact that a lot of extra work is being done, as the D3D context is being resized constantly).
EDIT
I've taken things back as far as possible to try and isolate the problem, and the issue seems to be fundamental to the way in which D3DImage is updated & rendered by WPF.
I have modified the WPFHost example in the SharpDX Samples solution so that the simple trigangle that is displayed is animated across the screen. This example is hosted in a DX10ImageSource that is rendered by a DPFCanvas on CompositionTarget.Rendering.
This example couldn't be more simple, and is about as "close to the metal" as you can get whilst rendering a D3DImage in WPF. A single triangle, translated across the screen by a value calculated from the time difference between renders. The stutter remains, coming and going, as if some sort of synchronisation issue. It baffles me, but essentially makes SharpDX unusable within WPF for any sort of smooth animations, which is extremely disappointing.
If anyone is interested in reproducing this problem, the SharpDX samples are available here: https://github.com/sharpdx/SharpDX-Samples
I made the following simple changes to the WPFHost example:
void IScene.Render()
{
...
EffectMatrixVariable wv = this.SimpleEffect.GetVariableBySemantic("WorldView").AsMatrix();
wv.SetMatrix(this.WorldViewMatrix);
...
}
void IScene.Update(TimeSpan sceneTime)
{
float x = (float)sceneTime.Milliseconds * 0.001f - 0.5f;
WorldViewMatrix = Matrix.Translation(x, 0, 0);
}
and in shader Simple.fx:
float4x4 WorldViewTransform : WorldView;
PS_IN VS( VS_IN input )
{
PS_IN output = (PS_IN)0;
output.pos = mul(input.pos, WorldViewTransform);
output.col = input.col * Overlay;
return output;
}
D3DImage is fundamentally broken.
If you don't need to overlay XAML elements on top of D3D or you do not need to resize too frequently the D3D surface, prefer to host a HWND/WinForm into your WPF apps and render to it directly using regular rendering loop. If you want a bit more details about the reasons, you can check this issue.
What is the best approach to render charts and then save them on a hard drive for further distribution using WPF?
I found a number of ways to accomplish this by using the following types:
DrawingVisual - creating a object of this type and then rendering graphics on its context;
Shape - deriving from the Shape class and then overriding its DefiningGeometry property where the actual rendering is happening;
PathFigure - adding LineSegment-s to an instance of this class and then adding this instance to a Canvas;
Adorner - deriving from it and then overriding its OnRender method;
WritableBitmap - rendering on it and then adding the bitmap to a Canvas.
Of course I'm going to write an app to test how fast each of these will be. But can anybody tell me:
whether am I on the right track?
are there any other means to do such rendering?
which one of them is the best in
terms of performance?
It all depends on your actual usage, in your case you mention saving on the hard drive for "further distribution" - I'm going to assume you are saving them as an image (jpg or png) and not as wpf objects (xaml).
You should consider if WPF is the right tool for the job, WPF is a UI framework and not a generic image processing library, it may be best to use something else entirely for generating images.
For a reasonable number of points your performance bottleneck will be encoding the image and saving it to disk - not actually rendering it - so you should choose the method that is easier for you to code.
All the articles about high performance WPF charts are a: about charts with 10,000 points and more (because that is where the performance problems are), b: about charts you display in your GUI (because otherwise you can use an image processing library to create the bitmap) and c: charts that change all the time (so they work nicely with data binding) - there's a reason why they don't talk about saving charts to disk.
For a very large number of points:
The fastest way to draw in WPF is to inherit from FrameworkElement (not Adorner) and override OnRender.
When the data changes often it is recommended to use multiple DrawingVisual objects because then you don't have to re-render everything when one value change - but this is not relevant for you since the image won't change after you save it anyway.
WritableBitmap is used for raw bitmap access, you use it when you decide to give up on all the nice layout and drawing WPF gives you because you can't take the overhead, if this is the case you should re-read my first point above.
So, to summarize, you are asking the wrong question :-) if you need to save images to disk than either the WPF rendering speed is not your bottleneck or you shouldn't be using WPF to begin with. If you do use WPF just pick whatever is easiest for you to code.
BTW: Adorners are used to display "floating" elements above the normal UI, you can use them for tooltip-like features but not for the main chart rendering (and you probably don't want them at all since your main usage is saving the image to disk), FrameworkElement is the base class you are looking for.
I've written a simple game-like app in WPF. The number of objects drawn is well within WPF capabilities - something like a few hundred ellipses and lines with simple fills. I have a DispatcherTimer to adjust the positions of the objects from time to time (1/60th of a second).
The code to compute the new positions can be quite intensive when there are lots of objects, and can fully load a processor. Whenever this occurs, WPF starts skipping frames, presumably trying to compensate for the "slowness" of my application.
What I would much rather happen is for all the frames to be drawn anyway, only slower. The dropped frames do not add any speed - because visual updates were pretty quick anyway.
Can I somehow force WPF to have my changes to the visuals be reflected on the screen regardless of whether WPF thinks it's a good idea?
Unfortunately I don't think there's anything you can do about this, although I will happily be corrected! WPF is designed to be an application creation framework, not a games library, so it priortises application performance and "usability" over framerate. This actually works very well when producing applications as it allows you to use quite rich animations and effects while maintaining perceived performance on lower end systems.
The only thing I think you might be able to try is push your movement code's Dispatcher priority down slightly to below Render (Loaded is the next one down) using something like:
this.Dispatcher.BeginInvoke(DispatcherPriority.Loaded, MoveMyStuff);
I don't have any kind of test harness to verify if that will help though.
This issue was fixed by using a Canvas with an OnRender override instead of creating and moving UIElements. This does mean that everything needs to be drawn by hand in OnRender, but it can now run at any FPS consistently, without skipping any frames.
I've created a WPF control (inheriting from FrameworkElement) that displays a tiled graphic that can be panned. Each tile is 256x256 pixels at 24bpp. I've overridden OnRender. There, I load any new tiles (as BitmapFrame), then draw all visible tiles using drawingContext.DrawImage.
Now, whenever there are more than a handful new tiles per render cycle, the framerate drops from 60fps to zero for about a second. This is not caused by loading the images (which takes in the order of milliseconds), nor by DrawImage (which takes no time at all, as it merely fills some intermediate render data structure).
My guess is that the render thread itself chokes whenever it gets a large number (~20) of new BitmapSource instances (that is, ones it had not already cached). Either it spends a lot of time converting them to some internal DirectX-compatible format or it might be a caching issue. It cannot be running out of video RAM; Perforator shows peaks at below 60MB, I have 256MB. Also, Perforator says all render targets are hardware-accelerated, so that can't be it, either.
Any insights would be appreciated!
Thanks in advance
Daniel
#RandomEngy:
BitmapScalingMode.LowQuality reduced the problem a little, but did not get rid of it. I am already loading tiles at the intended resolution. And it can't be the graphics driver, which is up-to-date (Nvidia).
I'm a little surprised to learn that scaling takes that much time. The way I understood it, a bitmap (regardless of its size) is just loaded as a Direct3D texture and then hardware-scaled. As a matter of fact, once the bitmap has been rendered for the first time, I can change its rotation and scale without any further freezes.
It's not just with a large number of images. Just one large image is enough to hold up rendering until it has been loaded in, and that can be quite noticable when your image dimensions start getting up in the thousands.
I do agree with you that it's probably the render thread: I did a test and the UI thread was still happily dispatching messages while this render delay was taking place from trying to display a fully pre-cached BitmapImage.
It must be doing some sort of conversion or preparation on the image, like you were speculating. I've tried to mitigate this in my app by "rendering" but hiding the image, then revealing it when I need to show it. However this is less than ideal because the rendering freezes happen anyway.
(Edit)
Some followup: After a discussion on the MS WPF alias I found what was causing the delays. On my Server 2008 machine it was a combination of old video drivers that don't support the new WDDM driver model and a delay for resizing the image.
If the source image size is different from the display size, that will delay the render thread before the image shows up. By default an image is set to the highest quality, but you can change the scaling options for rendering by calling RenderOptions.SetBitmapScalingMode(uiImage, BitmapScalingMode.LowQuality); . Once I did that, the mysterious freeze before displaying an image went away. An alternative, if you don't like the quality drop in scaling, is to load the BitmapImage with DecodePixelWidth/Height equal to the size it will be displayed at. Then if you load the BitmapImage on a background thread, you should have no delay in displaying it.
Also try these;
/* ivis is declared in XAML <Image x:Name="iVis" UseLayoutRounding="True" SnapsToDevicePixels="True" /> */
iVis.Stretch = Stretch.None;
RenderOptions.SetBitmapScalingMode(iVis, BitmapScalingMode.NearestNeighbor);
RenderOptions.SetEdgeMode(iVis, EdgeMode.Aliased);
VisualBitmapScalingMode = BitmapScalingMode.NearestNeighbor;
iVis.Source = **** your bitmap source ****
I was having some trouble with performance when using a huge amount of "A" channel color's, waiting until after the image had rendered to scale it worked much better for me.
Also, as you said your using a tiled graphic?
You would usually use a TileBrush to simply set as the Brush on your FrameworkElement. If you are animating them or adding new ones dynamically, you could generate your brushes then apply them to your object as you go manually too, be sure to Freeze them if you can. Also, VisualBitmapScalingMode is a property of any Visual.