I am a beginner at AR game development for both iOS and Android. I have the following questions:
What kinds of 3D model formats are supported by ARKit for iOS and ARCore for Android respectively? (I tried .dae and .obj are supported by ARkit, not yet test ARCore.)
Our 3D model vendor can only provide us FBX format. How can I convert it to the formats supported by ARKit and ARCore? I tried using 3D model converter, but the converted model has no texture.
Updated: January 22, 2023.
SceneKit
Apple SceneKit framework handles 3D models for ARKit and VR apps. SceneKit supports the following 3D assets with corresponding material files:
.dae (supports animations)
.obj (single-frame) with its texture and .mtl file
.abc (only single-frame supported)
.usdz (supports animations)
.scn (native SceneKit's format, supports animations)
RealityKit
Apple RealityKit framework also handles 3D models for ARKit, AR and VR apps. You can prototype a content for RealityKit in a standalone app called Reality Composer. RealityKit supports the following 3D assets:
.usdz (supports animations and dynamics)
.reality (supports animations and dynamics, optimized for a faster loading)
.rcproject (supports animations and dynamics)
Additionally you can use Terminal's usdzconvert command to generate .usdz from the following formats:
.obj
.glTF
.fbx
.abc
.usda
.usdc
.usd
And, of course, you can use Reality Converter app with its simple GUI.
Sceneform
Pity but since June 2020 Sceneform has been archived and no longer maintained by Google.
Google Sceneform handles 3D models for ARCore SDK. Sceneform supports the following 3D assets with their material dependencies:
.obj (with its .mtl dependency)
.glTF (animations not supported)
.fbx (with or without animations)
.sfa (ascii asset definition, deprecated in Sceneform 1.16)
.sfb (binary asset definition, deprecated in Sceneform 1.16)
SceneKit, RealityKit, Sceneform and Reality Composer support Physically Based Rendering.
ARKit and ARCore
But what's the role of ARKit and ARCore then?
These two AR modules don't care about importing and rendering of a 3D geometry. They are only responsible for tracking (world, image, face, geo, etc) and scene understanding (i.e. plane detection, hit-testing & raycasting, depth perception, light estimation, and geometry reconstruction).
ARKit doesn't care about model formats, because ARKit doesn't display 3D models (or anything else) itself.
ARKit provides information for use by a higher level rendering engine — Apple's SceneKit or SpriteKit, third-party Unreal or Unity plug-ins, or an engine you build yourself with Metal or OpenGL. The rendering engine is responsible for drawing the camera feed and overlaying 3D content to create AR illusions, and it uses the information ARKit provides in order to do so convincingly.
I don't know much about ARCore, but from all appearances it has the same role in the Android ecosystem — it's Unity, Unreal, or some other engine that handles the 3D models there, too.
So, questions like this are specific to whatever 3D engine you're using with ARKit/ARCore. SceneKit can handle DAE and OBJ directly, and a few more formats via Model I/O (see MDLAsset and SCNScene.init(mdlAsset:)). For Unreal, Unity, and whatever else you use with ARCore handle... see the documentation for those technologies.
ARcore itself doesn't come with any 3d model handling logic at this moment. To render a 3D model in any format, you need to parse the data and draw it using openGL. The sample app in the package demonstrates how this can be done for a simple 3D model i.e 1 OBJ and 1 texture file.
I not sure did you checked ARCore properly. The basic example delivered by Google is working on .obj format. ARCore is set of tools related with AR only. You can use 3D format whatever you want as long as you will be able to use it on Android. It is not related with ARCore
Question 2 is not related with ARCore and/or android or even arkit
Related
I have purchased the book programming a multiplayer fps book.
The problem is that I am not able to get the 3d's max scene to load. Is there a step to load a library of 3d's max to load the scene? What is the missing piece in loading the scene. I am not able to load the scene.
In order to load and render 3D mesh data, you need some kind of exporter to convert the 3D Studio MAX authoring data into a binary form suitable for rendering, and some sort of rendering engine capable of drawing that binary form.
For exporting the data, you can use an existing tool such as Open-Asset-Importer-Lib or write your own using something like Autodesk FBX.
There are of course numerous choices for rendering engines, or you can write your own from scratch.
Note that Direct3D 9 is extremely old at this point. DirectX 11 is the modern sweet spot for starting out programming Direct3D. See DirectX Tool Kit and DirectX SDK Samples Content Exporter.
The 3dsmax file format is propriatary. You just don't load those *.max files into anything but 3ds max. Like Chuck said above, you have to export your max file from 3ds max into a common, 3rd party file format like OBJ, FBX, etc...
There is probably a hundred 3D third-party file formats out there. We have no idea which 'game' you are trying to code up, and you have not provided any information, except asked all the wrong questions.
DirectX 11 is a graphics platform provided by microsoft. it is file format agnostic. It is up to you to learn it.
I've seen ARkit and I've seen the demo for animoji on the keynote, but I'm wondering if there is a way to create your own animoji (that will work within messages) within Xcode.
You can use ARKit to provide facial movement data to animate your own 3D models. In conjunction with an iMessage app, you should be able to export videos of animated characters similar to animojis.
Take a look at ARBlendShapeLocation (documentation) which provides high-level facial feature detection. You could track these features and use them to animate your models.
I'd also recommend watching the recent Apple developer video called "Face Tracking with ARKit" (link) which gives a good overview of the API's available.
When you're ready to jump right in, start with this face-tracking sample code (link) from Apple. (thanks #rickster)
Note that these features are only available on the new iPhone X.
I am trying to integrate the creation of objects in ARKit with
SceneKit and Metal. There are objects I can easily create with SeneKit and others with Metal, so I'd like to be able to use both side by side.
For example creating an ArKit project with the sceneKit template places a space ship as a demo through SceneKit Nodes. On the other hand using the Metal template creates a cube in ARKit with a tap using an anchor.
How would I be able to use both in the same project? Creating a cube through Metal and a spaceship with SceneKit, for example.
Another example, there is a MTKView that I want to wrap as an object or layer it on top of an object in SceneKit, and I also have other objects that I am creating with PBR in SceneKit, I want to have these side by side in ARKit. If possible.
Maybe with this? :
https://developer.apple.com/documentation/scenekit/scnprogram
I think this, if possible would be very useful for the growing ARKit community.
Thanks!
You should be able to use SceneKit and Metal content in your scene at the same time - you can use SCNSceneRendererDelegate for that by having access to the renderer and it's currentRenderCommandEncoder property (https://developer.apple.com/documentation/scenekit/scnscenerendererdelegate):
renderer(_:willRenderScene:atTime:) - for rendering Metal content before SceneKit's
renderer(_:didRenderScene:atTime:) - for rendering Metal content after SceneKit's
Or use SCNNodeRendererDelegate to replace the content of the node with Metal content (https://developer.apple.com/documentation/scenekit/scnnoderendererdelegate):
renderNode(_:renderer:arguments:)
How would I be able to use both in the same project?
You have to choose wether you want to build your app with Metal or SceneKit, both have they ups and downs.
I think there a two approaches you could take:
Use SceneKit. Since SceneKit uses Metal you can enhance and change shaders or the program. Maybe start reading the documentation about using Metal in SceneKit. Maybe you can do what you want to do in Metal while still using SceneKits abstractions where possible.
Use Metal. If using Metal in SceneKit as Apple intended it is not enough you have to build everything in Metal.
To load complex models you can use Apples ModelIO to load 3D content into metal.
I want to build an application for mobile (ios,android) and Desktop (Windows) or Web.
The application will look like this: a 3D object which the user can play with the camera perspective around it and some menus.
What I need is to manipulate a 3D Object like torus or tube. by manipulate I mean: change materials and edit the object like a polygon. of course I also need menus and communication to a server.
*Optional: I am not sure if I need to load 3D model from a file.
What I don't understand is should I look for all in one solution or combine a cross platform framework with other libraries? Are Game engine suitable for this task?
My options so far:
Use Three.js with PhoneGap and write in java script
Use OGRE and write in C++ with some cross-platform framework that allow me to write in C++.
I never developed for mobile and I wonder about those cross-platform frameworks: does the application's size is big? does the application runs well? which obstacles should i have comparing writing separate applications for each platform?
Thank you for your help
Might want to look at libgdx .
I am developing a webapp which the most challenging aspect is the 3D model section. It also contains other things such as drag and drop and a sliding bar with arrows each side which the user can go through to select different items.
I have been looking into WebGL but it seems IE doesn't support it without using a plugin. This isn't ideal so I was wondering what other options I have.
Flash? Silverlight? Anything else?
Silverlight is WAY too rare to be used for any meaningful development, and ajax is simply not stunning enuogh. That leaves us with Flash, which in my opinion, is the best of the three options. Flash Player 11 introduces the new Stage3D API which can create pretty stunning 3D graphics. There are also many AS3 libraries for 3D rendering. I prefer Away3D
Just found this, which looks promising.
http://iewebgl.com/
Edit: Disregard, seems the user has to download the executable for it to work.
at the moment flash is most popular plugin for 3D based web apps. which is cross browser, but no mobile browsers support :(
Many technologies and every one have his disatvantages