Passing Metal buffer to SceneKit shader - scenekit

I'd like to use a Metal compute shader to calculate some positions that are then fed into a Metal shader. Sounds straight forward, but I'm having trouble getting my MTLBuffer data into the Metal based SCNProgram.
The compute kernel is as follows, in this contrived example it's taking in three 3D vectors (in both buffers).
kernel void doSimple(const device float3 *inVector [[ buffer(0) ]],
device float3 *outVector [[ buffer(1) ]],
uint id [[ thread_position_in_grid ]]) {
float yDisplacement = 0;
. . . .
outVector[id] = float3(
inVector[id].x,
inVector[id].y + yDisplacement,
inVector[id].z);
}
This kernel function is run each frame in the - renderer:willRenderScene:atTime: method of my SCNSceneRendererDelegate. There's two buffers, and they get switched after each frame.
Buffers are created as follows;
func setupBuffers() {
positions = [vector_float3(0,0,0), vector_float3(1,0,0), vector_float3(2,0,0)]
let bufferSize = sizeof(vector_float3) * positions.count
//copy same data into two different buffers for initialisation
buffer1 = device.newBufferWithBytes(&positions, length: bufferSize, options: .OptionCPUCacheModeDefault)
buffer2 = device.newBufferWithBytes(&positions, length: bufferSize, options: .OptionCPUCacheModeDefault)
}
And the compute shader is run using the following (in the willRenderScene func);
let computeCommandBuffer = commandQueue.commandBuffer()
let computeCommandEncoder = computeCommandBuffer.computeCommandEncoder()
computeCommandEncoder.setComputePipelineState(pipelineState)
computeCommandEncoder.setBuffer(buffer1, offset: 0, atIndex: 0)
computeCommandEncoder.setBuffer(buffer2, offset: 0, atIndex: 1)
computeCommandEncoder.dispatchThreadgroups(numThreadgroups, threadsPerThreadgroup: threadsPerGroup)
computeCommandEncoder.endEncoding()
computeCommandBuffer.commit()
computeCommandBuffer.waitUntilCompleted()
let bufferSize = positions.count*sizeof(vector_float3)
var data = NSData(bytesNoCopy: buffer2.contents(), length: bufferSize, freeWhenDone: false)
var resultArray = [vector_float3](count: positions.count, repeatedValue: vector_float3(0,0,0))
data.getBytes(&resultArray, length:bufferSize)
for outPos in resultArray {
print(outPos.x, ", ", outPos.y, ", ", outPos.z)
}
This works, and I can see my compute shader is updating the y coordinate for each vector in the array.
This scene consists of three spheres evenly spaced. The vertex shader simply takes the position calculated in the compute shader and adds it to each vertex position (well the y component anyway). I give each sphere an index, the vertex shader uses this index to pull the appropriate position out of my computed array.
The Metal vertex function is shown below, it's referenced by a SCNProgram and set to the material of each sphere.
vertex SimpleVertex simpleVertex(SimpleVertexInput in [[ stage_in ]],
constant SCNSceneBuffer& scn_frame [[buffer(0)]],
constant MyNodeBuffer& scn_node [[buffer(1)]],
constant MyPositions &myPos [[buffer(2)]],
constant uint &index [[buffer(3)]]
)
{
SimpleVertex vert;
float3 posOffset = myPos.positions[index];
float3 pos = float3(in.position.x,
in.position.y + posOffset.y,
in.position.z);
vert.position = scn_node.modelViewProjectionTransform * float4(pos,1.0);
return vert;
}
MyPositions is a simple struct containing an array of float3s.
struct MyPositions
{
float3 positions[3];
};
I have no problem passing data to the vertex shader using the setValue method of each sphere's material as shown below (also done in the willRenderScene method). Everything works as expected (the three spheres move upwards).
var i0:UInt32 = 0
let index0 = NSData(bytes: &i0, length: sizeof(UInt32))
sphere1Mat.setValue(index0, forKey: "index")
sphere1Mat.setValue(data, forKey: "myPos")
BUT this requires the data be copied from the GPU to CPU to GPU and is really something I'd rather avoid. So my question is... How do I pass a MTLBuffer to a SCNProgram?
Have tried the following in willRenderScene but get nothing but EXEC_BAD...
let renderCommandEncoder = renderer.currentRenderCommandEncoder!
renderCommandEncoder.setVertexBuffer(buffer2, offset: 0, atIndex: 2)
renderCommandEncoder.endEncoding()
Complete example is over on GitHub.
Thanks for reading, been struggling with this one. Workaround is to use a MTLTexture in place of a MTLBuffer as I've been able to pass these into an SCNProgram via the diffuse mat prop.

just switch the bindings of the buffers from step to step.
step1
computeCommandEncoder.setBuffer(buffer1, offset: 0, atIndex: 0)
computeCommandEncoder.setBuffer(buffer2, offset: 0, atIndex: 1)
step2
computeCommandEncoder.setBuffer(buffer1, offset: 0, atIndex: 1)
computeCommandEncoder.setBuffer(buffer2, offset: 0, atIndex: 0)
step3
computeCommandEncoder.setBuffer(buffer1, offset: 0, atIndex: 0)
computeCommandEncoder.setBuffer(buffer2, offset: 0, atIndex: 1)
and so on ...
the out buffer becomes the new in buffer and vice versa ...

Related

Swift Int array to MetalBuffer<Int>

I'm studying app development with swift and metal kit
I want to convert an int type array to Metalbuffer, because I have to cover that array data in Shader.h
Before operating the shader.h, the array's values have not problems.
But, in and after the shader.h, the buffer's that converted from array have some problems.
for example, I send [1, 2, 3, 4] to Metalbuffer and I call the renderEncoder.setVertexBuffer(Metalbuffer).
then, In the Shader.h it appears like [1, 0, 2, 0]
what's the problem???
this is my sample code
var int_array = Array(reapeating: 1, count: 100)
init(){
Buffers = MetalBuffer<Int>(device: device, array: int_array, index: kBuffers.rawValue, options: [])
}
func updateIntArray(){
for i in 0..<100 {
int_array[i] = i % 20
}
Buffers = MetalBuffer<Int>(device: device, array: int_array, index: kBuffers.rawValue, options: [])
(other codes about rendering)
renderEncoder.setVertexBuffer(Buffers)
}
the updateIntArray is called per every frame by view controller.
I printed the sizeof(Buffers[i]) after the shader call, it was 4!
How can I preserve the data without being inserted with 0?
Thank you!
There's a lot of code missing from your question, but I will show you how I pass an array as a parameter to my fragment shader.
I have a ShaderTypes.h file to create a custom value type that can be used with both Swift and MSL (Metal Shading Language). You will need a bridging header for this if you want to use it on the Swift side.
#ifndef ShaderTypes_h
#define ShaderTypes_h
#include <simd/simd.h>
typedef struct {
float intensity;
float time;
float width;
float height;
float originX;
float originY;
}KeyPointValues;
My goal is to pass an array of KeyPointValues as a parameter to my fragment shader. On the swift side I do something like this:
var keyPoints = [KeyPointValues()]
for index in 0...10 {
keyPoints.append(KeyPointValues())
}
var count = keyPoints.count
guard let keyPointBuffer = context.device.makeBuffer(bytes: keyPoints, length: MemoryLayout<KeyPointValues>.stride * count) else { return }
commandEncoder.setFragmentBuffer(keyPointBuffer, offset: 0, index: 0)
commandEncoder.setFragmentBytes(&count, length: MemoryLayout.size(ofValue: count), index: 1)
You need to pass in count as well because there is no keyPoints.count counterpart in MSL.
Then the fragment shader looks something like this:
#include <metal_stdlib>
#include <simd/simd.h>
#import "ShaderTypes.h"
using namespace metal;
fragment half4 rosyFilter(device const KeyPointValues *keyPoints [[buffer(0)]],
constant uint &count [[ buffer(1) ]]
) {
for(uint index = 0; index < count; index++) {
float intensity = keyPoints[index].intensity;
}
}
Hopefully this can get you started in the right direction. You are using an array of Int, so it should be easier as there is probably no need to define a custom struct to use between Swift and MSL.
I am also somewhat new to Metal so I'm not sure if this is the best way of doing things. I appreciate any feedback from people with more experience.

Rotating and moving SCNNode in world space

I’m trying to rotate and move a SCNNode. It is acting as if I’m still rotating in local space rather than world space. If I rotate the item and then apply force, it moves in the same direction as before, just pointing in a new direction, rather than ‘forward’ being redefined by the rotation. What am I doing wrong? Thanks.
func applyForce(to node: SCNNode) {
let newTransform = SCNMatrix4Translate(node.worldTransform, force.x, force.y, force.z)
node.setWorldTransform(newTransform)
}
func applyRotation(to node: SCNNode, radians: Float) {
let newTransform = SCNMatrix4Rotate(node.worldTransform, radians, 0, 1, 0)
node.setWorldTransform(newTransform)
}
I've found an answer. I perform the rotation in local space and translate the node in world space. The nil argument in simdConvertVector converts to world. This accomplishes my goal.
```
func applyForce(_ force: SCNVector3, to node: SCNNode) {
let force = SIMD3<Float>(force)
let translation = SIMD3<Float>(1, 1, 1) * force
let newPosition = node.simdConvertVector(translation, to: nil)
node.simdPosition += newPosition
}
func applyRotation(_ radians: Float, to node: SCNNode) {
let newTransform = SCNMatrix4Rotate(node.transform, radians, 0, 1, 0)
node.transform = newTransform
}
```

Dereference UnsafeMutablePointer<UnsafeMutableRawPointer>

I have a block that is passing data in that I'd like to convert to an array of array of floats -- e.g. [[0.1,0.2,0.3, 1.0], [0.3, 0.4, 0.5, 1.0], [0.5, 0.6, 0.7, 1.0]]. This data is passed to me in the form of data:UnsafeMutablePointer<UnsafeMutableRawPointer> (The inner arrays are RGBA values)
fwiw -- the block parameters are from SCNParticleEventBlock
How can I dereference data into a [[Float]]? Once I have the array containing the inner arrays, I can reference the inner array (colorArray) data with:
let rgba: UnsafeMutablePointer<Float> = UnsafeMutablePointer(mutating: colorArray)
let count = 4
for i in 0..<count {
print((rgba+i).pointee)
}
fwiw -- this is Apple's example Objective-C code for referencing the data (from SCNParticleSystem handle(_:forProperties:handler:) )
[system handleEvent:SCNParticleEventBirth
forProperties:#[SCNParticlePropertyColor]
withBlock:^(void **data, size_t *dataStride, uint32_t *indices , NSInteger count) {
for (NSInteger i = 0; i < count; ++i) {
float *color = (float *)((char *)data[0] + dataStride[0] * i);
if (rand() & 0x1) { // Switch the green and red color components.
color[0] = color[1];
color[1] = 0;
}
}
}];
You can actually subscript the typed UnsafeMutablePointer without having to create an UnsafeMutableBufferPointer, as in:
let colorsPointer:UnsafeMutableRawPointer = data[0] + dataStride[0] * i
let rgbaBuffer = colorsPointer.bindMemory(to: Float.self, capacity: dataStride[0])
if(arc4random_uniform(2) == 1) {
rgbaBuffer[0] = rgbaBuffer[1]
rgbaBuffer[1] = 0
}
Were you ever able to get your solution to work? It appears only a handful of SCNParticleProperties can be used within an SCNParticleEventBlock block.
Based on this answer, I've written the particle system handler function in swift as:
ps.handle(SCNParticleEvent.birth, forProperties [SCNParticleSystem.ParticleProperty.color]) {
(data:UnsafeMutablePointer<UnsafeMutableRawPointer>, dataStride:UnsafeMutablePointer<Int>, indicies:UnsafeMutablePointer<UInt32>?, count:Int) in
for i in 0..<count {
// get an UnsafeMutableRawPointer to the i-th rgba element in the data
let colorsPointer:UnsafeMutableRawPointer = data[0] + dataStride[0] * i
// convert the UnsafeMutableRawPointer to a typed pointer by binding it to a type:
let floatPtr = colorsPointer.bindMemory(to: Float.self, capacity: dataStride[0])
// convert that to a an UnsafeMutableBufferPointer
var rgbaBuffer = UnsafeMutableBufferPointer(start: floatPtr, count: dataStride[0])
// At this point, I could convert the buffer to an Array, but doing so copies the data into the array and any changes made in the array are not reflected in the original data. UnsafeMutableBufferPointer are subscriptable, nice.
//var rgbaArray = Array(rgbaBuffer)
// about half the time, mess with the red and green components
if(arc4random_uniform(2) == 1) {
rgbaBuffer[0] = rgbaBuffer[1]
rgbaBuffer[1] = 0
}
}
}
I'm really not certain if this is the most direct way to go about this and seems rather cumbersome compared to the objective-C code (see above question). I'm certainly open to other solutions and/or comments on this solution.

Scenekit giving buffer size error while passing array data to uniform array in openGL shader

this is the surface shader which I use to make a trail on floor surface.
#pragma arguments
uniform vec2 trailPoints[5];
uniform float count;
#pragma body
float trailRadius = 10.0;
float x = _surface.diffuseTexcoord.x;
float x100 = float(x * 100);
float y = _surface.diffuseTexcoord.y;
float y100 = float(y * 100);
for (int i = 0; i < int(count); i++) {
vec2 position = trailPoints[i];
if ((x100 > position.x - trailRadius && x100 < position.x + trailRadius) && (y100 > position.y - trailRadius && y100 < position.y + trailRadius)) {
_surface.diffuse.rgb = vec3(0.0, 10.0 ,0.0);
}
}
and this is the swift side code which I use to pass vector data to surface shader.
if let geometry = self.floorNode.geometry {
if let material = geometry.firstMaterial {
// this is the temporary data which I use to find the problem.
// this data will be dynamic later on.
let myValueArray:[float2] = [float2(x:80, y:80),float2(x:60, y:60),float2(x:40, y:40),float2(x:20, y:20),float2(x:0, y:0)]
// Passing array count to shader. There is no problem here.
var count = Float(myValueArray.count)
let countData = Data(buffer: UnsafeBufferPointer(start: &count, count: 1))
material.setValue(countData, forKey: "count")
// and here is the problem start.
// myValueArray converted to data with its size.
let valueArrayData = Data(buffer: UnsafeBufferPointer(start: myValueArray, count: myValueArray.count))
material.setValue(valueArrayData, forKey: "trailPoints")
}
}
When I build and run the project the following error occurred and no data passed to the "trailPoints" in shader.
Error: arguments trailPoints : mismatch between the NSData and the buffer size 40 != 8
When I change the array count to 1 while converting array to data,
let valueArrayData = Data(buffer: UnsafeBufferPointer(start: myValueArray, count: 1))
the errors dissapear but only the first member of the array will passing to shader.
so, the problem is,
how can I pass the all array members to the shader?
I think, the answer of this question this:
I recently realized, the OpenGl ES 2.0 only allow the following array definitions:
float myValue[3];
myValue[0] = 1.0;
myValue[1] = 2.0;
myValue[2] = 3.0;
But as far as I can tell, it is not possible to do this using the SCNShaderModifierEntryPoint with following way.
material.setValue (1.0, forKey: "myValue[0]")
material.setValue (2.0, forKey: "myValue[1]")
material.setValue (3.0, forKey: "myValue[2]")
And finally I found a way to pass the array to fragment shader with SCNProgram handleBinding method.
let myValue:[Float] = [1.0, 2.0, 3.0]
material.handleBinding(ofSymbol:"myValue", handler: { (programId:UInt32, location:UInt32, node:SCNNode?, renderer:SCNRenderer) in
for (index, v) in myValue.enumerated() {
var v1 = v
let aLoc = glGetUniformLocation(programId, String(format: "myValue[%i]", index))
glUniform1fv(GLint(aLoc), 1, &v1)
}
})
But, SCNProgram is completly rid off the default swift shader program and use yours.
The default shader program of swift is highly complex and do lots of things to your place.
default vertex shader of swift
default fragment shader of swift
So maybe its not a good idea to use SCNProgram for only pass the arrays to shader.
And one interesting thing, SCNProgram does not work on SCNFloor geometry.

Copy contents of Swift Array to Struct embedded Tuple

For communicating with a BLE characteristic, I have a Swift struct that looks like:
struct Packet {
var control1:UInt8 = 0
var control2:UInt8 = 0
var payload:(UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8) = (0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0)
init(control1:UInt8, control2:UInt8) {
self.control1 = control1
self.control2 = control2
}
}
I have payload defined as a tuple, because that seems to be the only way to have an array (of bytes in this case) of fixed size embedded in a Swift struct. Verbose, but whatever.
I have a big ol' source:[UInt8] that I want to pull swatches of into that Packet struct, so I can send them via BLE to the remote device. When I do:
var packet = Packet(control1: self.pageIndex, control2: sentenceIndex)
let offset = (Int(self.pageIndex) * self.pageSize) + (Int(sentenceIndex) * self.sentenceSize)
let limit = offset + self.sentenceSize
packet.payload = self.source[offset..<limit]
For the last line, I get the rather confusing error:
Cannot subscript a value of type '[UInt8]'
Cryptic I say, because it actually can. If I take the assignment to the packet.payload out, it has no problem subscripting the value.
What I'm really interested in at a higher level, is how one puts together a struct with a fixed size array of bytes, and then copies swatches of a large buffer into those. I would like to both understand the above, as well as know how to solve my problem.
UPDATE:
I ended up backing up a little, influenced by both answers below, and rethinking. My main driving force was that I wanted a simple/clever way to have convert a struct with an internal array to/from NSData, primary in BLE communications. What I ended up doing was:
struct Packet {
var pageIndex:UInt8 = 0
var sentenceIndex:UInt8 = 0
var payload:ArraySlice<UInt8> = []
var nsdata:NSData {
let bytes:[UInt8] = [self.pageIndex, self.sentenceIndex] + self.payload
return NSData(bytes: bytes, length: bytes.count)
}
}
Not the most efficient because I have to create the intermediate [UInt8] array, but I decided that a simple way to convert didn't exist, that I'd have to do things with as conversions or memcpy and friends.
I'm not sure which of the two below to mark as an answer, since both influenced what I ended up with.
There are two ugly/simple solutions:
To assign each member of the tuple separately:
var offset = ...
packet.payload = (source[offset++], source[offset++], ... , source[offset++])
To just copy the raw memory (recommended)
var values = Array(source[offset..<limit])
memcpy(&packet.payload, &values, sentenceSize)
Note that it's possible to create an array from a tuple:
func tupleToArray<T>(tuple: Any, t: T.Type) -> [T] {
return Mirror(reflecting: tuple).children.flatMap{ $0.value as? T }
}
tupleToArray((1, 2, 3, 4, 5), t: Int.self) // [1, 2, 3, 4, 5]
But the other way around doesn't work, as Swift's reflection is read-only.
Another much more complicated but more beautiful solution would be to use Dependent Types, which enables you to have arrays with compile-time known length. Check out this great blog post, in which he also mentions this post on the Apple Developer forums which is basically what you'd need:
let vector = 3.0 ⋮ 4.0 ⋮ 5.0 // [3.0, 4.0, 5.0]
vector[1] // 4.0
vector.count // 3
sizeofValue(vector) // 3 * 8 ( same size as a tuple with 3 elements)
First of all don't use tuples to create contiguous arrays of memory. Go ahead and use the [UInt8] type. I would recommend using a stride function to create your indices for you like this. You will have to handle the case of your data source not being a multiple of the Packet payload size.
struct Packet {
var control1: UInt8 = 0
var control2: UInt8 = 0
static let size = 16
var payload = [UInt8].init(count: Packet.size, repeatedValue: 0)
init(control1: UInt8, control2: UInt8) {
self.control1 = control1
self.control2 = control2
}
}
// random values between 0...255
let blob = (0..<(Packet.size * 3)).map{_ in UInt8(arc4random_uniform(UInt32(UInt8.max)))}
for index in 0.stride(through: blob.count - 1, by: Packet.size) {
var packet = Packet(control1: 4, control2: 5)
packet.payload[0..<Packet.size] = blob[index..<index + Packet.size]
print(packet.payload)
}
As far as the cannot subscript error, I encountered that too. I suspect that this has changed recently. I was able to eliminate the error by matching the packet indice slice with the data source slice.
UPDATE
A commenter correctly pointed out that Packet structure contained a reference to an Array and therefore did not meet the OP's need. While I was focused more on iterating through a large data source using stride, here is an alternative using an untyped [UInt8] for such a simple data structure.
// payload size in count of UInt8
let size = 16
// field offsets
let control1 = 0
let control2 = 1
let payload = 2..<(2 + size)
// random values between 0...255
let blob = (0..<size * 3).map{_ in UInt8(arc4random_uniform(UInt32(UInt8.max)))}
for index in 0.stride(through: blob.count - 1, by: size) {
var buffer = [UInt8](count: 2 + size, repeatedValue: 0)
buffer[control1] = 255
buffer[control2] = 0
buffer[payload] = blob[index..<index + size]
let data = NSData(bytesNoCopy: &buffer, length: buffer.count, freeWhenDone: false)
// send data
}

Resources