I want to create a 3D scene with the SceneKit modeler and then read it into my Metal app. I see there is SceneKit and ModelIO API to do this but I am unclear on how the pieces fit together.
So, what I need is path from .scn file -> MDL Mesh -> geometry + texture. I am unclear on how I would sync my Metal shaders with materials created in the SceneKit modeler.
There's two main parts to what you're asking here: getting SceneKit data into ModelIO, and rendering ModelIO data with Metal.
To get SceneKit scenes into ModelIO, first use SceneKit API (SCNScene or SCNSceneSource to load the .scn file, then use ModelIO API to get the objects you want as meshes. You can create a MDLAsset from the entire scene using assetWithSCNScene:bufferAllocator: and then walk through the asset's object hierarchy in ModelIO to find the mesh you want, or walk through the node hierarchy in SceneKit to find the SCNNode or SCNGeometry you want and then get the mesh to ModelIO using objectWithSCNNode:bufferAllocator: or meshWithSCNGeometry:bufferAllocator:.
As for using ModelIO meshes in a Metal app, Apple has a sample code project that shows how to use ModelIO to load an OBJ mesh, use MetalKit to get the mesh data into Metal GPU buffers, and hook up the material information you get from ModelIO to shader variables for use in your own renderer.
You should be able to put the two of these together: where the sample code loads an OBJ to get a MDLAsset or MDLMesh, use the methods in (1) to get an asset or mesh out of a SceneKit file instead.
The SceneKit material model is of course much more complicated than the simple Phong shader used in the sample code. But the sample code does show how to iterate through a MDLMaterial's properties and set corresponding arguments in a Metal shader — if you create a more complicated shader, just follow the same steps to map the material properties to whatever your shader's inputs are.
Related
I’ve been stuck for days, and couldn’t figure it out.
I’m trying to put a small graphic (as in Team Badge, and player’s number at the back) onto a basketball jersey. I can’t use Texture as anything I specified will cover the whole jersey.
From this https://threejs.org/examples/#webgl_decals I believe decal is the way to go. What I don’t understand is that I need to shove a Mesh into the Constructor of DecalGeometry. What kind of mesh should I use? Not the mesh of the basketball jersey?
Is this the right step?
get the upload of PNG from browser
create DecalGeometry using an arbitrary Geometry (may be a square?)
specify some material (I want it to be transparent)
create a new mesh using the above DecalGeometry and material (I can’t figure out what values to use for the decal projector…)
load the PNG into a Texture
load the basketball jersey’s Mesh
merge the basketball jersey’s Mesh with the new Mesh containing the DecalGeometry
Also, how do I control the size of the graphic inside the DecalGeometry? As the graphic has color, I’m not sure if the basketball jersey’s color will affect the graphic and make the graphic look broken?
Thanks a lot!
I am rendering 3d furniture models in Autodesk Maya for an AR app. For showing a shadow of 3d model in scene file I need export of transparent PNG file from Autodesk Maya which I can use as diffuse image for shadow plain. I have attached example of the kind of file needed but I don't know how to generate the same in Autodesk Maya. Please help.Shadow file generated for chair model.
If you want to generate in Maya a UV texture containing shadows for ARKit / SceneKit models follow these steps (I should say that it's not easy if you're a beginner in Maya):
Create objects with appropriate shaders
Create UV map for your models in UV Editor
Activate Maya Software Renderer
Create lights with Raytraced Shadows
Don't forget turn Raytracing on in Render Options
Duplicate all 3D objects and lights
Select an object with cast shadows and a corresponding light
Go to Rendering module and choose Lighting/Shading - Transfer Maps... menu.
Setup all the properties for Shaded Output Map and its appropriate location and then press Bake.
Also you can output four additional passes in Maya Software Renderer:
Normal Map
Displace Map
Diffuse Map
Alpha Map
If you want just shadows, without diffuse texture, use white coloured Lambertt Shader in Maya.
I am trying to create a particle system in SceneKit. I use the
particleSystem.birthLocation = .volume
option on a SceneKit node. This is working fine for built-in SceneKit shapes like SCNBox or SCNCylinder but now I need to make more complex meshes in Blender for use as the emitter. As simple test I tried exporting the default Box you get in Blender. However, when I try the exported .dae file I get the following error in the console.
[SceneKit] Error: Cannot use volume generation on a generic mesh. fallbacking on surface
Even the surface emitter it creates is not correct and consists of 3 triangles.
I've tried various options in the Blender export but I can't get this to work. Anyone have any ideas?
I would like to place a hairstyle after facial landmark detection. I'm able to render 2D images properly. I would like to render 3D model. I thought of using SceneKit to render 3D model. I would like to know how Instagram, snapchat and other face filter apps are rendering 3D models. I observe SceneKit coordinate system is different from UIKit coordinate system. I have googled but couldn't find the conversion of coordinate system. Could anyone help. Thanks.
Look for worldUp and simdWorldUp instance properties to understand how ARKit constructs a scene coordinate system based on real-world device motion (also, you can inspect ARConfiguration.WorldAlignment enum).
Please, look at this SO post: Understand coordinate spaces in ARKit for complete info.
And remember, ARAnchor is your best friend when placing 3D object. Click here
for further details.
I have a Viewport3D object containing a scene of hierarchical Model3D objects (grouped in Model3DGroups).
I need to "attach" 2D objects to 3D elements in the scene.
I read some articles on 3D projection but I couldn't fully understand how to apply my WPF scene data to those solutions.
What is the proper way to do that? I also tried Petzold.Media3D library but I didn't understand how to use it...
Thanks
I think you are looking for the Viewport2DVisual3D class. It allows you to render any Visual2D into the 3D space.
Searching the web for more info on this class will give you examples and videos explaining how it works.
Anyway, if you are doing any WPF 3D you also need to know about the Helix Toolkit - an awesome (free) WPF 3D toolkit.
It has two classes added recently that might also be relevant to what you are doing:
BillboardVisual3D (quad that always faces camera) and
TextBillboardVisual3D (text that always faces camera).
There are useful if you want to make some 2D image that must always face towards the 3D camera.