Multiple thread image compositing - wpf

I'm using WPF imaging to composite text, 3d graphics and images together using a DrawingVisual. The result is then rendered into a RenderTargetBitmap and saved to disk as a jpeg. All of this happens in code with no visual window.
It's working well as long as I only try to do one thing at a time. However I'd like to see if I can speed things up by using multiple threads to do the rendering. Each thread uses an object that creates its own DrawingContext, DrawingVisual, etc. However there's clearly some shared state somewhere as I get spurious errors when I attempt to access them in parallel. Sometimes it's "the calling thread cannot access this object because another thread created it". Other times it's more evil looking NullReferenceExceptions bubbling up from the bowels of WPF when I'm adding, say, points to a 3D geometry.
Is there a way to ensure each thread stays off of each other with WPF? Or is shared state unavoidable?

Is it possible that you are accidentally using the same resources across the threads? Are there any lamda expressions or anonymous methods involved in your processing code?

Eureka. I was creating my objects within threadpool threads, which is apparently a no-no when working with WPF. Instead, all objects need to be created from dispatcher threads. Incidentally this also cleared up a horrific memory leak I hadn't discovered.
My original class definition looked like this:
public class Compositor
{
private int _width;
private int _height;
private DrawingVisual _drawingVisual;
private DrawingContext _drawingContext;
private bool _isReady = false;
public void Reset(int width, int height)
{
_width = width;
_height = height;
_drawingVisual = new DrawingVisual();
_drawingContext = _drawingVisual.RenderOpen();
_isReady = true;
}
// ... compositing methods below
}
To rectify, I had my class inherit from DispatcherObject, then used the Dispatcher property to instantiate my objects.
public class Compositor : DispatcherObject
{
private int _width;
private int _height;
private DrawingVisual _drawingVisual;
private DrawingContext _drawingContext;
private bool _isReady = false;
public void Reset(int width, int height)
{
Dispatcher.Invoke(
new Action(
() =>
{
_width = width;
_height = height;
_drawingVisual = new DrawingVisual();
_drawingContext = _drawingVisual.RenderOpen();
_isReady = true;
}));
}
// ... compositing methods below
}

From MSDN:
Objects that derive from DispatcherObject have thread affinity.
Objects that derive from Freezable are free-threaded when they are frozen. For more information, see the Freezable Objects Overview.
I have encountered similar problem when working with bitmaps, brushes and other classes that should be really used (not only created) on UI thread.
This is annoying because one would like to leverage powers of WPF rendering to parallel processing realm, but this seems to be impossible (except for some scenarios like updating WriteableBitmap with pointers).

Related

WPF rendering is too slow

I am experiencing a strange problem trying to use WPF to render a number of polylines (64 polylines about 400-500 vertices in each on a 2300x1024 Canvas). Polylines are updated every 50ms.
For some reason my application UI becomes very sluggish and almost unresponsive to user input.
I am using to following class to avoid updating the point collection while it is displayed:
class DoubleBufferPlot
{
/// <summary>
/// Double-buffered point collection
/// </summary>
private readonly PointCollection[] mLineBuffer =
{
new PointCollection(),
new PointCollection()
};
private int mWorkingBuffer; //index of the workign buffer (buffer being modified)
#region Properties
//Polyline displayed
public Polyline Display { get; private set; }
/// <summary>
/// index operator to access points
/// </summary>
/// <param name="aIndex">index</param>
/// <returns>Point at aIndex</returns>
public Point this[int aIndex]
{
get { return mLineBuffer[mWorkingBuffer][aIndex]; }
set { mLineBuffer[mWorkingBuffer][aIndex] = value; }
}
/// <summary>
/// Number of points in the working buffer
/// </summary>
public int WorkingPointCount
{
get { return mLineBuffer[mWorkingBuffer].Count; }
set
{
SetCollectionSize(mLineBuffer[mWorkingBuffer], value);
}
}
#endregion
public DoubleBufferPlot(int numPoints = 0)
{
Display = new Polyline {Points = mLineBuffer[1]};
if (numPoints > 0)
{
SetCollectionSize(mLineBuffer[0], numPoints);
SetCollectionSize(mLineBuffer[1], numPoints);
}
}
/// <summary>
/// Swap working and display buffer
/// </summary>
public void Swap()
{
Display.Points = mLineBuffer[mWorkingBuffer]; //display workign buffer
mWorkingBuffer = (mWorkingBuffer + 1) & 1; //swap
//adjust buffer size if needed
if (Display.Points.Count != mLineBuffer[mWorkingBuffer].Count)
{
SetCollectionSize(mLineBuffer[mWorkingBuffer], Display.Points.Count);
}
}
private static void SetCollectionSize(IList<Point> collection, int newSize)
{
while (collection.Count > newSize)
{
collection.RemoveAt(collection.Count - 1);
}
while (collection.Count < newSize)
{
collection.Add(new Point());
}
}
}
I update the working buffer offscreen and then call Swap() to have it displayed. All 64 polylines (DoubleBufferPlot.Display) are added to a Canvas as children.
I used Visual Studio Concurrency Analyzer tool to see what's going on and discovered that after each update the main thread spends 46ms performing some WPF-related tasks: System.Widnows.ContextLayoutManager.UpdateLayout() and System.Windows.Media.MediaContex.Render().
I also discovered that there is another thread that's running almost non-stop rendering
wpfgfx_v0400.dll!CPartitionThread::ThreadMain
...
wpfgfx_v0400.dll!CDrawingContext::Render
...
etc.
I read a number of articles on WPF including this: Can WPF render a line path with 300,000 points on it in a performance-sensitive environment?
and also this article http://msdn.microsoft.com/en-us/magazine/dd483292.aspx.
I am (or my company rather) trying to avoid DrawingVisual since the rest of the project uses WPF shapes API.
Any idea why this is so slow? I even tried disabling anti-aliasing (RenderOptions.SetEdgeMode(mCanvas, EdgeMode.Aliased)) but it did not help very much.
Why does layout update takes so long. Anyone who is an expert in WPF internals?
Thank you very much.
After trying different approaches including DrawingVisual it seems that drawing polylines with so many vertices is too inefficient.
I ended up implementing at approach where I draw polylines only when there 1 or fewer vertices per pixel. Otherwise I render manually to a WriteableBitmap object. This is surprisingly much more efficient.
The fastest way I've found to draw frequently updated geometry is to create a DrawingGroup "backingStore", output that backing store during OnRender(), and then update that backingStore when my data needs to update, by using backingStore.Open(). (see code below)
In my tests, this was more efficient than using WriteableBitmap or RenderTargetBitmap.
If your UI is becoming unresponsive, how are you triggering your redraw every 50ms? Is it possible some of the redraw is taking longer than 50ms and backing up the message-pump with redraw messages? One method to avoid this is to shut off your redraw timer during your redraw loop (or make it a one-shot timer), and only enable it at the end. Another method is to do your redraw during a CompositionTarget.Rendering event, which happens right before the WPF redraw.
DrawingGroup backingStore = new DrawingGroup();
protected override void OnRender(DrawingContext drawingContext) {
base.OnRender(drawingContext);
Render(); // put content into our backingStore
drawingContext.DrawDrawing(backingStore);
}
// I can call this anytime, and it'll update my visual drawing
// without ever triggering layout or OnRender()
private void Render() {
var drawingContext = backingStore.Open();
Render(drawingContext);
drawingContext.Close();
}

Threading with WPF Images (System.InvalidOperationException)

I'm using a thread to get an image from a website and shoot it back to the parent form (WPF) to display. I ran into an issue and have managed to debug it to this example:
public void Watch()
{
while (true)
{
Bitmap bmp = new Bitmap(1, 1);
BitmapImage bmpImg = new BitmapImage();
this.SetImage(bmp, bmpImg);
}
}
public delegate void SetImageCallback(Bitmap bmp, BitmapImage bmpImg);
private void SetImage(Bitmap bmp, BitmapImage bmpImg)
{
if (!this.imgVideo.Dispatcher.CheckAccess())
{
SetImageCallback del = new SetImageCallback(SetImage);
this.Dispatcher.Invoke(del, bmp, bmpImg);
}
else
{
Bitmap bitmap = bmp;
BitmapImage bitmapImage = bmpImg;
}
}
Keep in mind that Watch() runs on its own thread. If I use the bitmap object (which I can use with PictureBox in Window Forms) everything works great. That is, debugging this code, when I get to the line
Bitmap bitmap = bmp;
And inspect the variable bmp, everything is great and works as expected. HOWEVER, when I get to the next line
BitmapImage bitmapImage = bmpImg;
And inpsect the variable bmpImage, I get a ton of System.InvalidOperationException's. When this is in practice and gets assigned to a WPF Image object, it says that "The calling thread cannot access this object because a different thread owns it." Why am I running into this issue with WPF BitmapImages (which are required to set an ImageSource) but NOT in Windows Forms Bitmap objects (which can be used to set a PictureBox)? How do I fix this in WPF?
Most objects in WPF are of this category: they cannot be shared between different threads. However certain low-level resources such as brushes and bitmaps are derived from a special class called Freezable that if frozen can be shared between different threads. Of course once an object is frozen is can no longer be modified in any way. To freeze a freezable object simply call Freeze and this will prevent cross-thread exceptions.
Instead of
if (!this.imgVideo.Dispatcher.CheckAccess())
{
SetImageCallback del = new SetImageCallback(SetImage);
this.Dispatcher.Invoke(del, bmp, bmpImg);
}
try using :
if (!App.Current.Dispatcher.CheckAccess())
App.Current.Dispatcher.Invoke(DispatcherPriority.Normal, new Action<CustomObject>(SetImage),CustomeObjectInstance );
Here Cutom object will be a wrapper class wrapping
Bitmap bmp, BitmapImage bmpImg
Obviously, your SetImage signature will change to
SetImage(CutomObject custObj)
I have not tested the code but this may solve the issue.
Let us know if this works so that some poor soul can be benefitted from this post.
All the best!
Sid

wpf BackgroundWorker - Regarding updating UI

I use a browse for files dialog to allow a user to select multiple images. If a lot of images are selected, as expected it takes a bit. Below is an example of what I do with the selected images. I loop through the filepaths to images and create an instance of a user control, the user control has an Image control and a few other controls. I create the instance of this control then add it to a existing stackPanel created in the associating window xaml file. The example just below works fine, but I'm trying to understand BackGroundWorker better, I get the basics of how to set it up, with it's events, and pass back a value that could update a progress bar, but because my loop that takes up time below adds the usercontrol instance to an existing stackPanel, It won't work, being in a different thread. Is BackGroundWorker something that would work for an example like this? If so, what's the best way to update the ui (my stackpanel) that is outside the thread. I'm fairly new to wpf and have never used the BackGroundWorker besides testing having it just update progress with a int value, so I hope this question makes sense, if I'm way off target just let me know. Thanks for any thoughts.
Example of how I'm doing it now, which does work fine.
protected void myMethod(string[] fileNames) {
MyUserControl uc;
foreach (String imagePath in fileNames) {
uc = new MyUserControl();
uc.setImage(imagePath);
stackPanel.Children.Add(uc);
progressBar.Value = ++counter;
progressBar.Refresh();
}
}
below this class i have this so I can have the progressBar refresh:
public static class extensionRefresh {
private static Action EmptyDelegate = delegate() { };
public static void Refresh(this UIElement uiElement) {
uiElement.Dispatcher.Invoke(DispatcherPriority.Background, EmptyDelegate);
}
}
Check out this article on
Building more responsive apps with the Dispatcher
Now that you have a sense of how the Dispatcher works, you might be surprised to know that you will not find use for it in most cases. In Windows Forms 2.0, Microsoft introduced a class for non-UI thread handling to simplify the development model for user interface developers. This class is called the BackgroundWorker
In WPF, this model is extended with a DispatcherSynchronizationContext class. By using BackgroundWorker, the Dispatcher is being employed automatically to invoke cross-thread method calls. The good news is that since you are probably already familiar with this common pattern, you can continue using BackgroundWorker in your new WPF projects
Basically the approach is
BackgroundWorker _backgroundWorker = new BackgroundWorker();
// Set up the Background Worker Events
_backgroundWorker.DoWork += _backgroundWorker_DoWork;
_backgroundWorker.RunWorkerCompleted += _backgroundWorker_RunWorkerCompleted;
// Run the Background Worker
_backgroundWorker.RunWorkerAsync(5000);
// Worker Method
void _backgroundWorker_DoWork(object sender, DoWorkEventArgs e)
{
// Do something
}
// Completed Method
void _backgroundWorker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
{
// Doing UI stuff
if (e.Cancelled)
{
statusText.Text = "Cancelled";
}
else if (e.Error != null)
{
statusText.Text = "Exception Thrown";
}
else
{
statusText.Text = "Completed";
}
}
Using a BackgroundWorker alone won't solve your issue since elements created during the DoWork portion will still have originated from a non-UI thread. You must call Freeze on any objects you intend to use on another thread. However only certain UI objects will be freezable. You may have to load in the images as BitmapImages on the background thread, then create the rest of your user control on the UI thread. This may still accomplish your goals, since loading in the image is probably the most heavyweight operation.
Just remember to set BitmapImage.CacheOption to OnLoad, so it actually loads up the image when you create the object rather than waiting until it needs to be displayed.

WinForms Control.Scale Method

The code below shows a small WinForms app which includes a simple Control that draws a circle. I'm trying to understand the behavior of the Control.Scale method.
If I call the Scale method on the Control from Main, as shown in the code, it scales properly. But if I instead call Scale from Circle's constructor, no scaling occurs.
My puzzlement here no doubt indicates a gross misunderstanding on my part regarding what Scale is supposed to do. Can anyone enlighten me?
using System;
using System.Windows.Forms;
using System.Drawing;
class Program
{
[STAThread]
public static void Main()
{
var circle = new Circle(Color.Orange)
{
Size = new Size(23, 23),
Location = new Point(50, 50)
};
circle.Scale(new SizeF(3.0f, 3.0f)); // <-- scaling here works
var form = new Form();
form.Controls.Add(circle);
Application.Run(form);
}
}
class Circle : Control
{
public Circle(Color color)
{
ForeColor = color;
// Scale(new SizeF(3.0f, 3.0f)); // <-- scaling here DOESN'T work
}
protected override void OnPaint(PaintEventArgs e)
{
e.Graphics.FillEllipse(new SolidBrush(ForeColor), ClientRectangle);
}
}
The Scale() method isn't meant to do this. It is a helper method to implement the AutoScaleMode property. When your control is created by the form's InitializeComponent() method, scaling is suspended with SuspendLayout(). Which is why it has no effect in your constructor. The AutoScaleMode property value is applied when the form handle is created. Which cancels any scaling you applied.
I think you are looking for e.Graphics.ScaleTransform() in your OnPaint method. It doesn't scale the control, it scales the drawing. If you really did mean to scale the control then just change its Size property.

Ensuring WPF Window is inside screen bounds

I restore coordinates of Window on application startup. In good-old-Windows-Forms I used System.Windows.Forms.Screen collection. Is there anything similar in WPF world?
I did notice PrimaryScreen*, VirtualScreen* parameters in System.Windows.SystemParameters. However they left me hanging since it seems to be impossible to detect whether Window is inside bounds in cases when monitors are not same size.
System.Windows.Forms.Screen works perfectly well within WPF, so I think the designers of WPF saw no advantage in replacing it with a WPF-specific version.
You'll have to do a coordinate transformation of course. Here's an easy class to do the conversion:
public class ScreenBoundsConverter
{
private Matrix _transform;
public ScreenBoundsConverter(Visual visual)
{
_transform =
PresentationSource.FromVisual(visual).CompositionTarget.TransformFromDevice;
}
public Rect ConvertBounds(Rectangle bounds)
{
var result = new Rect(bounds.X, bounds.Y, bounds.Width, bounds.Height);
result.Transform(_transform);
return result;
}
}
Example usage:
var converter = new ScreenBoundsConverter(this);
foreach(var screen in System.Windows.Forms.Screen.AllScreens)
{
Rect bounds = converter.ConvertBounds(screen.Bounds);
...
}

Resources