I'm currently ingesting HD1080p video at 59.94 FPS from a camcorder via the HDMI input on the DeckLink 4K Extreme.
My goal is to replicate the incoming image in a WPF UI element. To accomplish this I'm using the DeckLink SDK in a C# WPF application.
In this program I've implemented the VideoInputFrameArrived callback. In this callback I'm copying the bytes from each frame into a WriteableBitmap which I've set as the source for an Image.
All this works as it should, and when I run the program, the Image is indeed updated in real time as the frames arrive.
My problem then, is that the only two supported Pixel Formats for the video input are 8BitYUV and 10BitYUV, neither of which can be natively displayed on computer monitors.
The WriteableBitmap can only take in various RGB, Black and White, and CMYK formats.
Here is what I've tried so far.
I've tried to convert each frame using the IDeckLinkVideoConversion::ConvertFrame()
Problem: ConvertFrame() requires a destination frame to be rendered on the DeckLink using IDeckLinkOutput::CreateVideoFrame(). As I currently understand it, the DeckLink cannot act as both an input (to capture the video feed) and an output (to render the destination frame).
I've set the incoming stream to 8BitYUV, and copied each frame into the WriteableBitmap with a format of BGR32.
Problem: As I mentioned earlier, this will display an image, but the color is incorrect and the picture is only half the width that it needs to be.
The reason for this is that the incoming stream of 8BitYUV is 16 bits/pixel, whereas the Bitmap expects 32 bits/pixel, and so the Bitmap treats each incoming MacroPixel (4 bytes) as one pixel instead of the 2 pixels it really is.
Currently I'm using a pixel shader to fix the color and a RenderTransform to scale the Image horizontally by a factor of 2 to "fix" the aspect ratio. The porblem is that I have half of the original resolution.
I don't believe this is a hardware limitation, because when I hook up another monitor to the HDMI output on the DeckLink, the incoming picture displays in full 1080p in perfect color. Would it be possible to capture that outgoing stream somewhere in memory?
TL;DR
What is the best way to convert 4:2:2 YUV (UYVY) into a RGB or CMYK pixel format in real time? (1080p # 59.94 FPS)
Preferably a hardware solution i.e. DeckLink or GPU.
You have several options here.
First of all, you can display UYVY directly. Most video adapters will accept UYVY data through DirectDraw, DirectShow, DirectX versions up to 9 APIs and you won't need a real time conversion for the video frames. Integrating this into WPF application might require some effort, and perhaps the most popular way is to utilize DirectShow through DirectShow.NET library and WPF Media Kit. On this way, however, you could also capture video using DeckLink's video capture DirectShow filter. You could connect all parts together faster, however you already capture using DeckLink SDK and this way you have more control and flexibility on the capture process so you might not want to get back to DirectShow.
Second option is to convert to RGB as you wanted. I don't think DeckLink can do it for you, and GPU based conversion definitely exists (the conversion formula is well known, simple and easy to parallelize), however is hardware dependent or otherwise not immediately available. Instead, Microsoft ships Color Converter DSP which can do the conversion (from 8 bits, not 10 though) in a very efficient way. The API is native, and you might need Media Foundation .NET to access it from your app. An alternate efficient software conversion can also be done using FFmpeg's libswscale (for managed app through respective wrappers).
I just did this with the decklink api because the card I have can act as both inputs and outputs. And the outputs do not need to be in playback mode to access this part of the api:
com_ptr<IDeckLinkOutput> m_deckLinkOutput;
if (SUCCEEDED(m_deckLink->QueryInterface(IID_IDeckLinkOutput, (void **)&m_deckLinkOutput)))
{
IDeckLinkMutableVideoFrame *pRGBFrame;
if (SUCCEEDED(m_deckLinkOutput->CreateVideoFrame(videoFrame->GetWidth(), videoFrame->GetHeight(), videoFrame->GetWidth() * 4, bmdFormat8BitBGRA, videoFrame->GetFlags(), &pRGBFrame)))
{
m_deckLinkVideoConversion->ConvertFrame(pFrame, pRGBFrame);
//use the rgbFrame
pRGBFrame->Release();
}
}
Related
I would like to convert png file(transparent icon) to WPF "path data".
Any idea how to get single wpf path data from .png icon.?
There is no direct way to do it. A PNG file is a raster format (i.e. it stores the colour of each pixel in the image). WPF Path Data is a vector format (i.e. it stores the image as geometric drawing instructions). If you are unsure what this means, see here for more info.
Vector can be converted to raster (at a set size), but raster cannot be obviously converted to vector (which is what you want).
The only way to try and convert raster to vector, with varying results, is to "trace" the raster image to guess what the equivalent geometric vector instructions might possibly be. The ability to trace a raster image accurately is directly proportional to its pixel dimensions and complexity of graphics. So tracing a [presumably] small icon might not be possible at all.
If I were faced with your problem, I would get as high a quality PNG image as possible, import it into Adobe Illustrator, use the Illustrator tracing tools to trace the image, and finally export the result of the trace to XAML (using something like http://www.mikeswanson.com/xamlexport/).
If you are looking for runtime tracing, this is something I have not come across. Given the massively varying tracing parameters which are required for different styles of raster images, I don't suppose it would even be realistically possible.
Good luck.
I am writing a simple 3D rendering engine.
The end result of my 3D processing is pixel data. Next I need to display it on the screen with GDI+.
I am using WinForms and Visual Basic. I am drawing directly on form's ClientRectangle.
I have some questions.
After I process a pixel, should I be writing pixel data to a buffer first, instead of sending each pixel to GDI+ individually?
- If so, how much of a screen should I buffer at one time? Full screen, half, quarter, eighth? I think there may be RAM usage / performance trade-offs here.
- What is the best data structure for the pixel buffer?
- Which GDI+ command do I use to render the pixel buffer (or the individual pixel)? Is it possible to avoid creating the bitmap as an intermediate step and send pixel data directly to screen?
Maximum screen size I anticipate is 1600x1200. RAM could be as low as 1GB.
Thanks.
Hope you can find some of those answers here
Write the data into a buffer of RGBA structs first. This will make it easy if, for example you want to render multiple "layers" and then composite those as well. It will also make it easy if you want to perform any deferred processing at some point. Once a full (tile?) render is complete, you can flush it to the output bitmap/file.
This depends on what resolutions you allow the user to render to. If you want to render gigapixel images, you will need to tile it at some reasonable size. I would recommend that the tile size be configurable and then you can set it at a reasonable default after testing.
I would recommend starting out with a simple RGBA buffer if you're not looking to perform any deferred shading.
If you are NOT performing tiled rendering/rendering images that can fit in memory, you can simply use Bitmap.LockBits and write the data that way. If you are using tiled rendering, you will need to either find a library that allows you to render a scanline at a time (and make that a "tile") or fix the file format you want to write TGA, PNG and seek/write directly to the file. Dumping the image as a RAW file and then using a command-line tool to convert it would also be another option.
Hope this helps!
I'm developing an application that shall receive images from a camera device and display them in a GTK window.
The camera delivers raw RGB images (3 bytes per pixel, no alpha channel, fixed size) at a varying frame rate (1-50 fps).
I've already done all that hardware stuff and now have a callback function that gets called with every new image captured by the camera.
What is the easyest but fast enough way to display those images in my window?
Here's what I already tried:
using gdk_draw_rgb_image() on a gtk drawing area: basically worked, but rendered so slow that the drawing processes overlapped and the application crashed after the first few frames, even at 1 fps capture rate.
allocating a GdkPixbuf for each new frame and calling gtk_image_set_from_pixbuf() on a gtk image widget: only displays the first frame, then I see no change in the window. May be a bug in my code, but don't know if that will be fast enough.
using Cairo (cairo_set_source_surface(), then cairo_paint()): seemed pretty fast, but the image looked striped, don't know if the image format is compatible.
Currently I'm thinking about trying something like gstreamer and treating those images like a video stream, but I'm not sure whether this is like an overkill for my simple mechanism.
Thanks in advance for any advice!
The entire GdkRGB API seems to be deprecated, so that's probably not the recommended way to solve this.
The same goes for the call to render a pixbuf. The documentation there points at Cairo, so the solution seems to be to continue investigating why your image looked incorrect when rendered by Cairo.
unwind is right, cairo is the way to go if you want something that will work in GTK2 and GTK3. As your samples are RGB without alpha, you should use the CAIRO_FORMAT_RGB24 format. Make sure the surface you paint is in that format. Also try to make sure that you're not constantly allocating/destroying the surface buffer if the input image keeps the same size.
As I said on the title.
I just want to know which is better between using image files and drawing vector shapes (or path).
I know that using vector is better for appearance but what about performance.
And if this depends on cases. Can anyone explain.
(This question may include WP7, Silverlight, WPF or even in general cases.)
Here is a general answer to compare pros/cons of Bitmap (what I think you mean by "image file") vs. Vector.
Bitmap-based images (gif, tiff, jpeg, png, bmp) are essentially the concept of mapping colours (and other data such as alpha layer) to a pixel grid. Different file formats offer variations of what is supported and levels of compression but this is the high-level concept. The complete map of pixels and data is stored in the file as a matrix/table.
Vector-based images, as you say, are path based. Instead of storing information by pixels, the file format will store geometric points and data.
The pros for bitmaps are:
They usually render faster than a vector. This is because there is minimal computation involved in presenting the image (just take the pixel map and display).
They handle "photographic" content better than a vector.
They are more portable than vector. GIF, JPEG, PNG, BMP are more standard than any vector format (where usually Adobe has the market)
The cons for bitmaps are:
They don't scale without degradation (pixelization)
Manipulation (i.e. resizing, blurring, lighting, etc) of a bitmap is more processor expensive than a vector
The files are usually much larger than vector-based files
The pros for vectors are:
Flexible for scaling and manipulation
Smaller file formats than vector
Ideal for print and animation (i.e. manipulating a shape to produce the animation effect)
The cons for vectors are:
Render time, depending on the complexity of the vector, can be longer
Portability most formats are highly proprietary
Work for "graphic" based images but not useful for photorealism
Hope this helps.
Jeremiah Morrill gave a great overview of WPF rendering that basically shows a vector will always be more expensive to render than an image. Basically an image gets treated as a directx texture...no matter the size, scaling or whatever, there is a set constant cost for rendering an image. As Jer's overview shows, even the simplest vector image takes a number of operations to render in WPF. The moral of the story is that when giving an option, go for the image instead of vector.
Based on our experience with Windows Phone 7 (Non-mango) apps, we find using Images instead of using drawing produces a far more responsiveness hence UX Performance for continuous animation in pages. (YMMV)
I would initially say that images render faster than vectors. The complexer the vector, more time it takes to render. The bigger the image, more time to render.
I'm going to speculate that (in Silverlight terms) most of the current video hardware is capable of directly handling the images rendering getting so a boost in the performance. I'm not sure if calculations for vectors can be done at video hardware level.
From the point of view of Windows Phone 7, you'll typically get faster rendering of images/bitmaps rather than paths/vectors. As a general rule for mobile development, due to the constrained resources on the device and the increased need to consider performance, if you can do something once, such as preparing an image, at design (or compile) time that definitely preferable to doing it multiple times on each client.
Be very careful of applying rules across platforms (WPF, Silverlight & WP7) as they are used for different things in different situations and are under different constraints. Things you have to consider on the phone may not be as much of an issue in a WPF app running on an high powered PC.
Using WPF, is it possible to access a video frame by frame.
Ideally, given a video file and time, one would get an ImageSource.
Is it possible ?
You can capture frames with this library : http://videorendererelement.codeplex.com/
System should have proper codecs for that particular video format (E.g. DivX).