Goal:
SceneKit Style Transfer with coreML
How:
Grabbing CVPixelBuffer from a SceneKit frame, like on WWDC video bellow:
https://developer.apple.com/videos/play/wwdc2020/10642/
Tried:
I tried accessing a SCN frame, but it doesn't seem to have access to CVPixelBuffer
Question:
How can I grab CVPixelBuffer from a SceneKit frame? Or is there a better way to do Style Transfer in SceneKit?
I found that there is a description like this.
Here I stylize offline the virtual object texture with a different style but I want to do even more.
I think it displays the virtual object texture through matel. I try to get the CVPixelBuffer through CIImage, and process it through arView.renderCallbacks.postProcess to get real-time processing effects, which is much more convenient than getting video processing, but how to process data through matel I am not sure, hope someone can provide more information.
You can refer to this speech video
https://developer.apple.com/videos/play/wwdc2021/10075/
Related
I'm developing an AR app for iOS that lets the user place a model in the physical world, using ARKit and SceneKit.
I've looked at this Apple sample code for inspiration. In the sample project, they use tracked raycasts to position 3D models in a scene. This is close to what I want, and what led me to assume I need to do the same to achieve the most accurate positioning.
However, when I use a tracked raycast to position my model, the model drifts around the scene a lot as ARKit updates the position of the raycast.
I get much more stable positioning when using a non-tracked raycast.
That makes me ask: what actually is the intended use case for a tracked raycast? Am I understanding this API wrong?
I've tried:
Positioning the model using an image anchor. This is very stable.
Positioning the model using a non-tracked raycast. This is about as stable as the image anchor.
Positioning the model using a tracked raycast. This drifts all over the scene.
I also understand what an AR raycast in general is for:
getting the intersection of a 2D point on the screen with the 3D geometries that ARKit is tracking.
As this post has explained already.
In Apple's example app you mentioned, raycasting is used to update the FocusSquare all the time. You don't really need it for placing your model. You can get a certain (real-world) position (using the FocusSquare) to place a model on that exact location. For this you can fetch static positon data from the FocusSquare at the moment you add your model on scene. I hope I understood corectly what you want.
I have a video playing of lines being drawn on the screen. Is it possible to create a pixel shader (for WPF) that turns newly colored pixels a certain color for N milliseconds?
That way, there can be some indication to the user to movement on the screen when the lines don't move often and the user isn't always looking at the screen.
You can use DirectShow. Its written in unmanaged code, so you need to use this wrapper DirectShow.NET in order to use it in your C# application which is running in managed environment (samples are included, even with EVR which stands for Enhanced video Renderer which means MUCH better video quality). And when you will be passing a control handle to wrapper method for setting the video output, you need a WinForms control, because only from them you can get your desired control handle. That WinForms control you can then host in your WPF application using the WindowsFormsHost control provided for such situations when you need to use some WinForms control(s) in a WPF application. Its just theory, so i dont know if its an ultimate solution for you.
BTW: The whole idea is based on fact, that DirectShow is just some query constructed from separated filters. Renderer is a filter (EVR, VMR-7, VMR-9). Sound player is a filter. And they are connected through their pins. Its like a diagram. Electronic schema or something like that. And you can put for example Grey scale filter in there. And voila, video output will be greyscale. There is a bunch of tutorials for that. And completed simple filters as well. Unfortunately, filters must be written in C++:(
PS: I never said its gonna be easy:D
I am working on a Chart Control,
I need to implement the Chart Snapshot feature for capturing a vector based image in high quality of Curves and Texts, in any requested size.
Can anyone suggest me any solution or a pointer to resolve this problem.
Any kind of help in deeply appreciated.
If you have to save your image in a vector format you can using tracing. Potrace is an open source bitmap-to-vector tracer library (but considers that bitmap tracing is imperfect). Also considers that wpf is linked closely with XAML, a vector graphics markup languages, so you may want convert raster graphics to XAML (though Charles Petzold you can't embed a bitmap in a XAML file)
If you need to draw a vector image you can use the Shape element that provides a base class for shape elements, such as Ellipse, Polygon, and Rectangle; and add the shape as Panel child. If you are dealing with thousands of shape I suggest to use the DrawingVisual class, a visual object that can be used to render vector graphics on the screen, and its RenderOpen method.
To zoom you have to work with transformations, in particular ScaleTrasnform and apply the transformation to your panel or to your shapes.
Hope this help.
You can take a look at XamlToy but I have not already try it.
http://xamltoys.codeplex.com/
I'm developing an application that shall receive images from a camera device and display them in a GTK window.
The camera delivers raw RGB images (3 bytes per pixel, no alpha channel, fixed size) at a varying frame rate (1-50 fps).
I've already done all that hardware stuff and now have a callback function that gets called with every new image captured by the camera.
What is the easyest but fast enough way to display those images in my window?
Here's what I already tried:
using gdk_draw_rgb_image() on a gtk drawing area: basically worked, but rendered so slow that the drawing processes overlapped and the application crashed after the first few frames, even at 1 fps capture rate.
allocating a GdkPixbuf for each new frame and calling gtk_image_set_from_pixbuf() on a gtk image widget: only displays the first frame, then I see no change in the window. May be a bug in my code, but don't know if that will be fast enough.
using Cairo (cairo_set_source_surface(), then cairo_paint()): seemed pretty fast, but the image looked striped, don't know if the image format is compatible.
Currently I'm thinking about trying something like gstreamer and treating those images like a video stream, but I'm not sure whether this is like an overkill for my simple mechanism.
Thanks in advance for any advice!
The entire GdkRGB API seems to be deprecated, so that's probably not the recommended way to solve this.
The same goes for the call to render a pixbuf. The documentation there points at Cairo, so the solution seems to be to continue investigating why your image looked incorrect when rendered by Cairo.
unwind is right, cairo is the way to go if you want something that will work in GTK2 and GTK3. As your samples are RGB without alpha, you should use the CAIRO_FORMAT_RGB24 format. Make sure the surface you paint is in that format. Also try to make sure that you're not constantly allocating/destroying the surface buffer if the input image keeps the same size.
What is the best approach to render charts and then save them on a hard drive for further distribution using WPF?
I found a number of ways to accomplish this by using the following types:
DrawingVisual - creating a object of this type and then rendering graphics on its context;
Shape - deriving from the Shape class and then overriding its DefiningGeometry property where the actual rendering is happening;
PathFigure - adding LineSegment-s to an instance of this class and then adding this instance to a Canvas;
Adorner - deriving from it and then overriding its OnRender method;
WritableBitmap - rendering on it and then adding the bitmap to a Canvas.
Of course I'm going to write an app to test how fast each of these will be. But can anybody tell me:
whether am I on the right track?
are there any other means to do such rendering?
which one of them is the best in
terms of performance?
It all depends on your actual usage, in your case you mention saving on the hard drive for "further distribution" - I'm going to assume you are saving them as an image (jpg or png) and not as wpf objects (xaml).
You should consider if WPF is the right tool for the job, WPF is a UI framework and not a generic image processing library, it may be best to use something else entirely for generating images.
For a reasonable number of points your performance bottleneck will be encoding the image and saving it to disk - not actually rendering it - so you should choose the method that is easier for you to code.
All the articles about high performance WPF charts are a: about charts with 10,000 points and more (because that is where the performance problems are), b: about charts you display in your GUI (because otherwise you can use an image processing library to create the bitmap) and c: charts that change all the time (so they work nicely with data binding) - there's a reason why they don't talk about saving charts to disk.
For a very large number of points:
The fastest way to draw in WPF is to inherit from FrameworkElement (not Adorner) and override OnRender.
When the data changes often it is recommended to use multiple DrawingVisual objects because then you don't have to re-render everything when one value change - but this is not relevant for you since the image won't change after you save it anyway.
WritableBitmap is used for raw bitmap access, you use it when you decide to give up on all the nice layout and drawing WPF gives you because you can't take the overhead, if this is the case you should re-read my first point above.
So, to summarize, you are asking the wrong question :-) if you need to save images to disk than either the WPF rendering speed is not your bottleneck or you shouldn't be using WPF to begin with. If you do use WPF just pick whatever is easiest for you to code.
BTW: Adorners are used to display "floating" elements above the normal UI, you can use them for tooltip-like features but not for the main chart rendering (and you probably don't want them at all since your main usage is saving the image to disk), FrameworkElement is the base class you are looking for.