I am attempting to find a simple way in SceneKit to calculate the depth of a pixels in SceneKit and LiDAR data from
sceneView.session.currentFrame?.smoothedSceneDepth?.depthMap
Ideally I don't want to use metal shaders. I would prefer find a points in my currentFrame and their corresponding depth map, to get the depth of a points in SceneKit (ideally in world coordinates, not just local to that frustum at that point in time).
Fast performance isn't necessary as it won't be calculated at capture.
I am aware of the Apple project at link, however this is far too complex for my needs.
As a starting point, my code works like this:
guard let depthData = frame.sceneDepth else { return }
let camera = frame.camera
let depthPixelBuffer = depthData.depthMap
let depthHeight = CVPixelBufferGetHeight(depthPixelBuffer)
let depthWidth = CVPixelBufferGetWidth(depthPixelBuffer)
let resizeScale = CGFloat(depthWidth) / CGFloat(CVPixelBufferGetWidth(frame.capturedImage))
let resizedColorImage = frame.capturedImage.toCGImage(scale: resizeScale);
guard let colorData = resizedColorImage.pixelData() else {
fatalError()
}
var intrinsics = camera.intrinsics;
let referenceDimensions = camera.imageResolution;
let ratio = Float(referenceDimensions.width) / Float(depthWidth)
intrinsics.columns.0[0] /= ratio
intrinsics.columns.1[1] /= ratio
intrinsics.columns.2[0] /= ratio
intrinsics.columns.2[1] /= ratio
var points: [SCNVector3] = []
let depthValues = depthPixelBuffer.depthValues()
for vv in 0..<depthHeight {
for uu in 0..<depthWidth {
let z = -depthValues[uu + vv * depthWidth]
let x = Float32(uu) / Float32(depthWidth) * 2.0 - 1.0;
let y = 1.0 - Float32(vv) / Float32(depthHeight) * 2.0;
points.append(SCNVector3(x, y, z))
}
}
The resulting point cloud looks ok, but is severely bent on the Z-axis. I realize this code is also not adjusting for screen orientation either.
Cupertino kindly got back to me with this response on the forums at developer.apple.com:
The unprojection calculation itself is going to be identical, regardless of whether it is done CPU side or GPU side.
CPU side, the calculation would look something like this:
/// Returns a world space position given a point in the camera image, the eye space depth (sampled/read from the corresponding point in the depth image), the inverse camera intrinsics, and the inverse view matrix.
func worldPoint(cameraPoint: SIMD2<Float>, eyeDepth: Float, cameraIntrinsicsInversed: simd_float3x3, viewMatrixInversed: simd_float4x4) -> SIMD3<Float> {
let localPoint = cameraIntrinsicsInversed * simd_float3(cameraPoint, 1) * -eyeDepth
let worldPoint = viewMatrixInversed * simd_float4(localPoint, 1);
return (worldPoint / worldPoint.w)[SIMD3(0,1,2)];
}
Implemented, this looks like
for vv in 0..<depthHeight {
for uu in 0..<depthWidth {
let z = -depthValues[uu + vv * depthWidth]
let viewMatInverted = (sceneView.session.currentFrame?.camera.viewMatrix(for: UIApplication.shared.statusBarOrientation))!.inverse
let worldPoint = worldPoint(cameraPoint: SIMD2(Float(uu), Float(vv)), eyeDepth: z, cameraIntrinsicsInversed: intrinsics.inverse, viewMatrixInversed: viewMatInverted * rotateToARCamera )
points.append(SCNVector3(worldPoint))
}
}
The point cloud is pretty messy, needs confidence worked out, and there are gaps vertically where Int rounding has occurred, but it's a solid start. Missing functions come from the link to the Apple demo project in the question above.
Related
I’m trying to rotate and move a SCNNode. It is acting as if I’m still rotating in local space rather than world space. If I rotate the item and then apply force, it moves in the same direction as before, just pointing in a new direction, rather than ‘forward’ being redefined by the rotation. What am I doing wrong? Thanks.
func applyForce(to node: SCNNode) {
let newTransform = SCNMatrix4Translate(node.worldTransform, force.x, force.y, force.z)
node.setWorldTransform(newTransform)
}
func applyRotation(to node: SCNNode, radians: Float) {
let newTransform = SCNMatrix4Rotate(node.worldTransform, radians, 0, 1, 0)
node.setWorldTransform(newTransform)
}
I've found an answer. I perform the rotation in local space and translate the node in world space. The nil argument in simdConvertVector converts to world. This accomplishes my goal.
```
func applyForce(_ force: SCNVector3, to node: SCNNode) {
let force = SIMD3<Float>(force)
let translation = SIMD3<Float>(1, 1, 1) * force
let newPosition = node.simdConvertVector(translation, to: nil)
node.simdPosition += newPosition
}
func applyRotation(_ radians: Float, to node: SCNNode) {
let newTransform = SCNMatrix4Rotate(node.transform, radians, 0, 1, 0)
node.transform = newTransform
}
```
I have the code below, which works in swift 2.3. I am struggling to understand how to convert it to swift 3/4 - the issue it those is
Value of type 'Range<Int>' has no member 'map'
let grainSize = CGFloat(0.01)
let min = CGFloat(-3)
let max = CGFloat(3)
let range = Range<Int>(uncheckedBounds: (lower: Int(min/grainSize), upper: Int(max/grainSize)))
let lol = NSRange(range)
var points = range.map { step -> CGPoint in
let i = grainSize * CGFloat(step)
let x = x_base + (i * controller.width / 4.0)
let y = y_base + (m * equation(i))
if x_init == CGFloat.max { x_init = x }
return CGPointMake(x, y)
}
points.append(CGPointMake(x_init, y_base))
points.forEach { renderer.lineTo($0) }
I am wondering if someone can point me in the right direction for this - even to documentation regarding this, as I can't find anything about it in apple docs either =[
Range does not adopt Sequence, just create the range literally as CountableClosedRange
let range = Int(min/grainSize)...Int(max/grainSize)
this is the surface shader which I use to make a trail on floor surface.
#pragma arguments
uniform vec2 trailPoints[5];
uniform float count;
#pragma body
float trailRadius = 10.0;
float x = _surface.diffuseTexcoord.x;
float x100 = float(x * 100);
float y = _surface.diffuseTexcoord.y;
float y100 = float(y * 100);
for (int i = 0; i < int(count); i++) {
vec2 position = trailPoints[i];
if ((x100 > position.x - trailRadius && x100 < position.x + trailRadius) && (y100 > position.y - trailRadius && y100 < position.y + trailRadius)) {
_surface.diffuse.rgb = vec3(0.0, 10.0 ,0.0);
}
}
and this is the swift side code which I use to pass vector data to surface shader.
if let geometry = self.floorNode.geometry {
if let material = geometry.firstMaterial {
// this is the temporary data which I use to find the problem.
// this data will be dynamic later on.
let myValueArray:[float2] = [float2(x:80, y:80),float2(x:60, y:60),float2(x:40, y:40),float2(x:20, y:20),float2(x:0, y:0)]
// Passing array count to shader. There is no problem here.
var count = Float(myValueArray.count)
let countData = Data(buffer: UnsafeBufferPointer(start: &count, count: 1))
material.setValue(countData, forKey: "count")
// and here is the problem start.
// myValueArray converted to data with its size.
let valueArrayData = Data(buffer: UnsafeBufferPointer(start: myValueArray, count: myValueArray.count))
material.setValue(valueArrayData, forKey: "trailPoints")
}
}
When I build and run the project the following error occurred and no data passed to the "trailPoints" in shader.
Error: arguments trailPoints : mismatch between the NSData and the buffer size 40 != 8
When I change the array count to 1 while converting array to data,
let valueArrayData = Data(buffer: UnsafeBufferPointer(start: myValueArray, count: 1))
the errors dissapear but only the first member of the array will passing to shader.
so, the problem is,
how can I pass the all array members to the shader?
I think, the answer of this question this:
I recently realized, the OpenGl ES 2.0 only allow the following array definitions:
float myValue[3];
myValue[0] = 1.0;
myValue[1] = 2.0;
myValue[2] = 3.0;
But as far as I can tell, it is not possible to do this using the SCNShaderModifierEntryPoint with following way.
material.setValue (1.0, forKey: "myValue[0]")
material.setValue (2.0, forKey: "myValue[1]")
material.setValue (3.0, forKey: "myValue[2]")
And finally I found a way to pass the array to fragment shader with SCNProgram handleBinding method.
let myValue:[Float] = [1.0, 2.0, 3.0]
material.handleBinding(ofSymbol:"myValue", handler: { (programId:UInt32, location:UInt32, node:SCNNode?, renderer:SCNRenderer) in
for (index, v) in myValue.enumerated() {
var v1 = v
let aLoc = glGetUniformLocation(programId, String(format: "myValue[%i]", index))
glUniform1fv(GLint(aLoc), 1, &v1)
}
})
But, SCNProgram is completly rid off the default swift shader program and use yours.
The default shader program of swift is highly complex and do lots of things to your place.
default vertex shader of swift
default fragment shader of swift
So maybe its not a good idea to use SCNProgram for only pass the arrays to shader.
And one interesting thing, SCNProgram does not work on SCNFloor geometry.
So I have a program. And I am trying to simulate tons of moving particles with intricate moment logic that i would not want to have going on the CGP for many reasons. Of course I am then going to draw this all on the GPU.
Now originally I thought that when simulating TONS of particles that GPU delay would be a problem not the CPU. Unfortunately I am running 500 particles at a whopping 6fps :(.
I have tracked the latency down to how I send the vertices to the particle simulator. And not even the buffer creation, simply how I build the arrays. Basically I have arrays I clear every frame, and then go through for each particle in an array of particles and create arrays for each of them. And this leads to around 17500 append calls (with 500 particles). So I need a different way to do this because without building these arrays it runs at 60fps no cpu latency. Most of these append calls call a member of a struct.
Currently each particle is made based off of a class object. And it has things like position and color that are stored in structs. Would it be wroth my while to switch structs to arrays? Or perhaps I should switch everything to arrays? Obviously doing any of that would make things much harder to program. But would it be worth it?
A big problem is that I need each particle to be drawn as a capsule. Which I would make out of two dots and a thick line. Unfortunately OpenGL es 2.0 doesn't support thick lines so I have to draw it with two dots and two triangles :(. As you can see the function "calculateSquare" makes these two triangles based off to the two points. It is also very laggy, however it isn't the only problem, I will try to find a different way later.
What are your thoughts?
Note: According to xcode ram usage is only at 10 mb. However the cpu frame time is 141 ms.
Here is the code BTW:
func buildParticleArrays()
{
lineStrip = []
lineStripColors = []
lineStripsize = []
s_vertes = []
s_color = []
s_size = []
for cparticle in particles
{
let pp = cparticle.lastPosition
let np = cparticle.position
if (cparticle.frozen == true)
{
addPoint(cparticle.position, color: cparticle.color, size: cparticle.size)
}
else
{
let s = cparticle.size / 2.0
//Add point merely adds the data in array format
addPoint(cparticle.position, color: cparticle.color, size: cparticle.size)
addPoint(cparticle.lastPosition, color: cparticle.color, size: cparticle.size)
lineStrip += calculateSquare(pp, pp2: np, size: s)
for var i = 0; i < 6; i++
{
let rgb = hsvtorgb(cparticle.color)
lineStripColors.append(GLfloat(rgb.r))
lineStripColors.append(GLfloat(rgb.g))
lineStripColors.append(GLfloat(rgb.b))
lineStripColors.append(GLfloat(rgb.a))
lineStripsize.append(GLfloat(cparticle.size))
}
}
}
}
func addPoint(theObject: point, color: colorhsv, size: CGFloat)
{
let rgb = hsvtorgb(color)
s_vertes += [GLfloat(theObject.x), GLfloat(theObject.y), GLfloat(theObject.z)]
s_color += [GLfloat(rgb.r), GLfloat(rgb.g), GLfloat(rgb.b), GLfloat(rgb.a)]
s_size.append(GLfloat(size))
}
func calculateSquare(pp1: point, pp2: point, size: CGFloat) -> [GLfloat]
{
let p1 = pp1
var p2 = pp2
var s1 = point()
var s2 = point()
let center = CGPointMake((p1.x + p2.x) / 2.0, (p1.y + p2.y) / 2.0)
var angle:CGFloat = 0
if ((p1.x == p2.x) && (p1.y == p2.y))
{
//They are ontop of eachother
angle = CGFloat(M_PI) / 2.0
p2.x += 0.0001
p2.y += 0.0001
}
else
{
if(p1.x == p2.x)
{
//UH OH x axis is equal
if (p1.y < p2.y)
{
//RESULT: p1 is lower so should be first
s1 = p1
s2 = p2
}
else
{
//RESULT: p2 is lower and should be first
s1 = p2
s2 = p1
}
}
else
{
//We could be all good
if (p1.y == p2.y)
{
//Uh oh y axis is equal
if (p1.x < p2.x)
{
//RESULT: p1 is left so should be first
s1 = p1
s2 = p2
}
else
{
//RESULT: p2 is to the right so should be first
s1 = p2
s2 = p1
}
}
else
{
//Feuf everything is ok
if ((p1.x < p2.x) && (p1.y < p2.y)) //First point is left and below
{
//P1 should be first
s1 = p1
s2 = p2
}
else //First point is right and top
{
//P2 should be first
s1 = p2
s2 = p1
}
}
}
angle = angle2p(s1, p2: s2)
}
if (angle < 0)
{
angle += CGFloat(M_PI) * 2.0
}
let yh = size / 2.0
let distance = dist(p1, p2: p2)
let xh = distance / 2.0
let tl = rotateVector(CGPointMake(-xh, yh), angle: angle) + center
let tr = rotateVector(CGPointMake(xh, yh), angle: angle) + center
let bl = rotateVector(CGPointMake(-xh, -yh), angle: angle) + center
let br = rotateVector(CGPointMake(xh, -yh), angle: angle) + center
let c1:[GLfloat] = [GLfloat(bl.x), GLfloat(bl.y), 0]
let c2:[GLfloat] = [GLfloat(tl.x), GLfloat(tl.y), 0]
let c3:[GLfloat] = [GLfloat(br.x), GLfloat(br.y), 0]
let c4:[GLfloat] = [GLfloat(tr.x), GLfloat(tr.y), 0]
let part1 = c1 + c2 + c3
let part2 = c2 + c3 + c4
return part1 + part2
}
Do you really need all particles in system RAM? e.g. for some physics collision calculation in relation to other objects in the scene? Otherwise you could just create one particle, send it to the GPU and do the calculations in a GPU shader.
Ok so after hours of tweaking the code for small bits in efficiency I have it running 500 particles at a fps of 28 which looks pretty smooth! I still have some ways to go. The best piece of advice had to do with allocating memory instead appending it. That saved tons of problems.
Special thanks to #Darko, #Marcelo_Cantos for coming up with the ideas that would ultimately optimize my code!
Currently, I am using a convex hull algorithm to get the outer most points from a set of points randomly placed. What I aim to do is draw a polygon from the set of points returned by the convex hull however, when I try to draw the polygon it looks quite strange.
My question, how do I order the points so the polygon draws correctly?
Thanks.
EDIT:
Also, I have tried sorting using orderby(...).ThenBy(...) and I cant seem to get it working.
Have you tried the gift wrapping algorithm ( http://en.wikipedia.org/wiki/Gift_wrapping_algorithm)? This should return points in the correct order.
I had an issue where a random set of points were generated from which a wrapped elevation vector needed a base contour. Having read the link supplied by #user1149913 and found a sample of gift-wrapping a hull, the following is a sample of my implementation:
private static PointCollection CalculateContour (List<Point> points) {
// locate lower-leftmost point
int hull = 0;
int i;
for (i = 1 ; i < points.Count ; i++) {
if (ComparePoint(points[i], points[hull])) {
hull = i;
}
}
// wrap contour
var outIndices = new int[points.Count];
int endPt;
i = 0;
do {
outIndices[i++] = hull;
endPt = 0;
for (int j = 1 ; j < points.Count ; j++)
if (hull == endPt || IsLeft(points[hull], points[endPt], points[j]))
endPt = j;
hull = endPt;
} while (endPt != outIndices[0]);
// build countour points
var contourPoints = new PointCollection(points.Capacity);
int results = i;
for (i = 0 ; i < results ; i++)
contourPoints.Add(points[outIndices[i]]);
return contourPoints;
}
This is not a full solution but a guide in the right direction. I faced a very similar problem just recently and I found a reddit post with an answer (https://www.reddit.com/r/DnDBehindTheScreen/comments/8efeta/a_random_star_chart_generator/dxvlsyt/) suggesting to use Delaunay triangulation which basically returns a solution with all possible triangles made within the data points you have. Once you have all possible triangles, which by definition you know won't result on any overlapped lines, you can chose which lines you use which result on all nodes being connected.
I was coding my solution on python and fortunately there's lots of scientific libraries on python. I was working on a random sky chart generator which would draw constellations out of those stars. In order to get all possible triangles (and draw them, just for fun), before going into the algorithm to draw the actual constellations, all I had to do was this:
# 2D array of the coordinates of every star generated randomly before
points = list(points_dict.keys())
from scipy.spatial import Delaunay
tri = Delaunay(points)
# Draw the debug constellation with the full array of lines
debug_constellation = Constellation(quadrants = quadrants, name_display_style = config.constellation_name_display_style)
for star in available_stars:
debug_constellation.add_star(star)
for triangle in tri.simplices:
star_ids = []
for index in triangle:
star_ids.append(points_dict[points[index]].id)
debug_constellation.draw_segment(star_ids, is_closed = True)
# Code to generate the image follows below
You can see the full implementation here: fake_sky_chart_generator/fake_libs/constellation_algorithms/delaunay.py
This is the result: