Hi there stackoverflowers. I'm implementing a wrapper for Secure Transport and I'm stuck on some of the C -> Swift syntax.
func sslReadCallback(connection: SSLConnectionRef,
data: UnsafeMutablePointer<Void>,
var dataLength: UnsafeMutablePointer<Int>) -> OSStatus
{
//let bytesRequested = dataLength.memory
let transportWrapper:SecureTransportWrapper = UnsafePointer(connection).memory
let bytesRead:NSData = transportWrapper.readFromConnectionFunc(transportWrapper.connection)
dataLength = UnsafeMutablePointer<Int>.alloc(1)
dataLength.initialize(bytesRead.length)
if (bytesRead.length == 0)
{
return OSStatus(errSSLClosedGraceful)
}
else
{
data.alloc(sizeof(bytesRead.length)) //<----compile error here
return noErr
}
}
I've marked the location of the compile error. I don't blame it for erring, I was kind of guessing here :P. I'm trying to copy the the NSData to the data:UnsafeMutablePointer. How do I do that?
Compile error:
/Users/*/SecureTransportWrapper.swift:108:9: Static member 'alloc' cannot be used on instance of type 'UnsafeMutablePointer' (aka 'UnsafeMutablePointer<()>')
Thanks a ton!
================
Update: here is the api doc for what the sslReadCallback is supposed to do:
connection: A connection reference.
data: On return, your callback should overwrite the memory at this location with the data read from the connection.
dataLength: On input, a pointer to an integer
representing the length of the data in bytes. On return, your callback
should overwrite that integer with the number of bytes actually
transferred.
Excerpt from here
OK, lets go through your code:
dataLength = UnsafeMutablePointer<Int>.alloc(1)
dataLength.initialize(bytesRead.length)
dataLength is a pointer you get passed in, it is where the caller of the function both gives you the size of the buffer and wants you to put the number of bytes you read. You don't need to alloc this, it is already allocated.
(Irrelevant for this example but: Also in alloc(N) and initialize(N) the N should be the same (it is the amount of memory being allocated, and then initialized))
I think what you want (Swift 3 uses pointee instead of memory) is this:
dataLength.memory = bytesRead.length
The C API says that you also get the size of the data buffer from this variable. data will be pre-allocated for this size.
Make sure the data you read fits (bytesRead.length <= dataLength.memory), then just do a
memcpy(data, bytesRead.bytes, bytesRead.length)
That's all.
Related
I have some medium large CSV files (about 140mb) and I'm trying to turn them into an array of structs.
I don't want to to load the hole file in the memory so I'm using a steam reader.
For each line I read the data, turn the line into my struct and append the struct to the array. Because there are more then 5_000_000 lines in total, I used reserveCapacity to get a better memory management.
var dataArray : [inputData] = []
dataArray.reserveCapacity(5_201_014)
Unfortunately, that doesn't help at all. There is no perforce difference.
The memory graph in the debug session rises up to 1.54 GB and then stays there.
Im wondering what im doing wrong, because I can't imaging that it takes 1.54Gb of RAM to store an array of structs from a file with an original size of 140mb.
I use the following code to create the array:
var dataArray : [inputData] = []
dataArray.reserveCapacity(5_201_014)
let stream = StreamReader(path: "pathToDocument")
defer { stream!.close() }
while let line = stream!.nextLine() {
if line.isHeader() {} else {
let array = line.components(separatedBy: ",")
dataArray.append(inputData(a: Float32(array[0])!, b: Float32(array[1])!, c: Float32(array[2])!))
}
}
I know that here are several CSV reader packages on GitHub but they are extremely slow.
Here a screenshot of the debug session:
Thanks for any advice.
I have Swift code which reads a binary file representing a sequence of UInt32 values like this:
let fileData = binaryFile.readData(ofLength: 44)
guard fileData.count > 0 else { break }
let headerData = fileData.withUnsafeBytes {
Array(UnsafeBufferPointer<UInt32>(start: $0, count: 11))
}
let polyCount = headerData[1].bigEndian
let polyFlags = headerData[2].bigEndian
I'd not used the program containing this code for a while, but when returning to it recently, it still works as expected, but now gives a deprecation warning:
"withUnsafeBytes is deprecated: use withUnsafeBytes<R>(_: (UnsafeRawBufferPointer) throws -> R) rethrows -> R instead"
I've searched for quite a long time for an un-deprecated way to do this without success. There are many examples across the web (including in stackoverflow) but almost all of them written before this deprecation into effect. Frankly, I've fried my brain hunting and trying suggestions! I am prepared to accept that I'm missing something totally obvious ("If it's getting complicated, you're doing it wrong."), but I work in an environment where I have no colleagues to ask .. except here.
Any guidance would be much appreciated.
RamsayCons's own solution is nice, but what about the performance? I think, it could be better if we reduce all unnecessary operation.
extension Data {
func toArray<T>(type: T.Type) -> [T] where T: ExpressibleByIntegerLiteral {
Array(unsafeUninitializedCapacity: self.count/MemoryLayout<T>.stride) { (buffer, i) in
i = copyBytes(to: buffer) / MemoryLayout<T>.stride
}
}
}
measured performance gain depends, but number of execution per second is at least doubled. Bigger the data, bigger advantage!
self.dataArray = data.withUnsafeBytes{ Array($0.bindMemory(to: UInt32.self))}
Well, letting some time pass restored brain function! I found an answer (in stackoverflow, of course):
round trip Swift number types to/from Data
some way into that question/answer is:
extension Data {
func toArray<T>(type: T.Type) -> [T] where T: ExpressibleByIntegerLiteral {
var array = Array<T>(repeating: 0, count: self.count/MemoryLayout<T>.stride)
_ = array.withUnsafeMutableBytes { copyBytes(to: $0) }
return array
}
}
The trick is that Arrays are not necessarily stored in contiguous memory so simply copying enough bytes in order to the destination doesn't do it. I hope this helps the next person hunting with a fried brain!!
I wrote a small application which records data from a sound card and stores the data in an array for later processing.
Whenever new data is available, portaudio executes the callback record. Within the callback I append the data to the array RecData.data.
The golang builtin function append adds as expected another element to the slice, but for whatever reason also overwrites all existing elements within the array with exactly the same data.
I have been trying to isolate the problem for more than two days, without success.
Here is a stripped down version of the code, which works and shows the problem:
package main
import (
"fmt"
"time"
// "reflect"
"github.com/gordonklaus/portaudio"
)
type RecData struct{
data [][][]float32
}
func main() {
var inputChs int = 1
var outputChs int = 0
var samplingRate float64 = 48000
var framesPerBuffer int = 3 //for test purpose that low. Would normally be 1024 or 2048
rec := RecData{make([][][]float32, 0, 1000)}
portaudio.Initialize()
stream, err := portaudio.OpenDefaultStream(inputChs, outputChs, samplingRate, framesPerBuffer, rec.record)
if err != nil {
fmt.Println(err)
}
defer stream.Close()
stream.Start()
for {
time.Sleep(time.Millisecond * 10)
}
}
// callback which gets called when new data is in the buffer
func (re *RecData)record(in [][]float32) {
fmt.Println("Received sound sample: ")
fmt.Println(in)
re.data = append(re.data, in)
fmt.Println("Content of RecData.data after adding received sound sample:")
fmt.Println(re.data, "\n")
time.Sleep(time.Millisecond * 500) //limit temporarily the amount of data read
// iterate over all recorded data and compare them
/*
for i, d := range re.data{
if reflect.DeepEqual(d, in){
fmt.Printf("Data at index %d is the same as the recorded one, but should not be!\n", i )
}
}*/
}
2. Update
This is the application output:
Received sound sample:
[[0.71575254 1.0734825 0.7444282]]
Content of RecData.data after adding received sound sample:
[[[0.71575254 1.0734825 0.7444282]]]
Received sound sample:
[[0.7555193 0.768355 0.6575008]]
Content of RecData.data after adding received sound sample:
[[[0.7555193 0.768355 0.6575008]] [[0.7555193 0.768355 0.6575008]]]
Received sound sample:
[[0.7247052 0.68471473 0.6843796]]
Content of RecData.data after adding received sound sample:
[[[0.7247052 0.68471473 0.6843796]] [[0.7247052 0.68471473 0.6843796]] [[0.7247052 0.68471473 0.6843796]]]
Received sound sample:
[[0.6996536 0.66283375 0.67252487]]
Content of RecData.data after adding received sound sample:
[[[0.6996536 0.66283375 0.67252487]] [[0.6996536 0.66283375 0.67252487]] [[0.6996536 0.66283375 0.67252487]] [[0.6996536 0.66283375 0.67252487]]]
.... etc ....
As we one can see, over time, the size of the slice is growing, but instead of just appending the data, the data in the array gets also overwritten.
This should not happen. portaudio provides in the callback a [][]float32 with the audio sample recorded from the sound card. As you can see they are always different.
As mentioned, the code above is a stripped down version of my application. Usually I would record lets say 5 seconds, and then perform a Fast Fourier Transformation (FFT) over the samples to calculate the spectrum. I left this part away since it has no impact on this particular problem.
I would very much appreciate any help. Maybe somebody can point me out what I'm doing wrong.
Thanks!
The buffer passed into the callback is reused by the portaudio package, so you are appending the same slice structure to your data slice. Each time the buffer allocated by portaudio overwrites the data, you see the results in every element of your data slice.
You will need to allocate new slices and make a copy of the data:
func (re *RecData) record(in [][]float32) {
buf := make([][]float32, len(in))
for i, v := range in {
buf[i] = append([]float32(nil), v...)
}
re.data = append(re.data, buf)
Example:
https://play.golang.org/p/cF57lQIZFU
I'm having trouble using reflection in Go to fetch data from a cache dynamically into various statically declared struct types:
func FetchFromCacheOrSomewhereElse(cacheKey string, returnType reflect.Type) (out interface {}, err error) {
fetchFromCache := reflect.New(returnType).Interface();
_, err=memcache.Gob.Get(*context, cacheKey, &fetchFromCache);
if (err==nil) {
out=reflect.ValueOf(fetchFromCache).Elem().Interface();
} else if (err==memcache.ErrCacheMiss) {
/* Fetch data manually... */
}
return out, err;
}
It seems that reflect won't translate this statically typed cache data back into a reflect value, and returns this error instead: gob: local interface type *interface {} can only be decoded from remote interface type; received concrete type ... :\
This data is saved elsewhere in the code to the cache without the need for reflect.
memcache.Gob.Get() which is Codec.Get() expects the "target" as a pointer, wrapped into an interface{}.
Your fetchFromCache is already just that: a pointer to a value of the specified type (returnType) wrapped in an interface{}. So you don't need to take its address when passing it to Gob.Get(): pass it as-is:
_, err=memcache.Gob.Get(*context, cacheKey, fetchFromCache)
For communicating with a BLE characteristic, I have a Swift struct that looks like:
struct Packet {
var control1:UInt8 = 0
var control2:UInt8 = 0
var payload:(UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8,UInt8) = (0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0)
init(control1:UInt8, control2:UInt8) {
self.control1 = control1
self.control2 = control2
}
}
I have payload defined as a tuple, because that seems to be the only way to have an array (of bytes in this case) of fixed size embedded in a Swift struct. Verbose, but whatever.
I have a big ol' source:[UInt8] that I want to pull swatches of into that Packet struct, so I can send them via BLE to the remote device. When I do:
var packet = Packet(control1: self.pageIndex, control2: sentenceIndex)
let offset = (Int(self.pageIndex) * self.pageSize) + (Int(sentenceIndex) * self.sentenceSize)
let limit = offset + self.sentenceSize
packet.payload = self.source[offset..<limit]
For the last line, I get the rather confusing error:
Cannot subscript a value of type '[UInt8]'
Cryptic I say, because it actually can. If I take the assignment to the packet.payload out, it has no problem subscripting the value.
What I'm really interested in at a higher level, is how one puts together a struct with a fixed size array of bytes, and then copies swatches of a large buffer into those. I would like to both understand the above, as well as know how to solve my problem.
UPDATE:
I ended up backing up a little, influenced by both answers below, and rethinking. My main driving force was that I wanted a simple/clever way to have convert a struct with an internal array to/from NSData, primary in BLE communications. What I ended up doing was:
struct Packet {
var pageIndex:UInt8 = 0
var sentenceIndex:UInt8 = 0
var payload:ArraySlice<UInt8> = []
var nsdata:NSData {
let bytes:[UInt8] = [self.pageIndex, self.sentenceIndex] + self.payload
return NSData(bytes: bytes, length: bytes.count)
}
}
Not the most efficient because I have to create the intermediate [UInt8] array, but I decided that a simple way to convert didn't exist, that I'd have to do things with as conversions or memcpy and friends.
I'm not sure which of the two below to mark as an answer, since both influenced what I ended up with.
There are two ugly/simple solutions:
To assign each member of the tuple separately:
var offset = ...
packet.payload = (source[offset++], source[offset++], ... , source[offset++])
To just copy the raw memory (recommended)
var values = Array(source[offset..<limit])
memcpy(&packet.payload, &values, sentenceSize)
Note that it's possible to create an array from a tuple:
func tupleToArray<T>(tuple: Any, t: T.Type) -> [T] {
return Mirror(reflecting: tuple).children.flatMap{ $0.value as? T }
}
tupleToArray((1, 2, 3, 4, 5), t: Int.self) // [1, 2, 3, 4, 5]
But the other way around doesn't work, as Swift's reflection is read-only.
Another much more complicated but more beautiful solution would be to use Dependent Types, which enables you to have arrays with compile-time known length. Check out this great blog post, in which he also mentions this post on the Apple Developer forums which is basically what you'd need:
let vector = 3.0 ⋮ 4.0 ⋮ 5.0 // [3.0, 4.0, 5.0]
vector[1] // 4.0
vector.count // 3
sizeofValue(vector) // 3 * 8 ( same size as a tuple with 3 elements)
First of all don't use tuples to create contiguous arrays of memory. Go ahead and use the [UInt8] type. I would recommend using a stride function to create your indices for you like this. You will have to handle the case of your data source not being a multiple of the Packet payload size.
struct Packet {
var control1: UInt8 = 0
var control2: UInt8 = 0
static let size = 16
var payload = [UInt8].init(count: Packet.size, repeatedValue: 0)
init(control1: UInt8, control2: UInt8) {
self.control1 = control1
self.control2 = control2
}
}
// random values between 0...255
let blob = (0..<(Packet.size * 3)).map{_ in UInt8(arc4random_uniform(UInt32(UInt8.max)))}
for index in 0.stride(through: blob.count - 1, by: Packet.size) {
var packet = Packet(control1: 4, control2: 5)
packet.payload[0..<Packet.size] = blob[index..<index + Packet.size]
print(packet.payload)
}
As far as the cannot subscript error, I encountered that too. I suspect that this has changed recently. I was able to eliminate the error by matching the packet indice slice with the data source slice.
UPDATE
A commenter correctly pointed out that Packet structure contained a reference to an Array and therefore did not meet the OP's need. While I was focused more on iterating through a large data source using stride, here is an alternative using an untyped [UInt8] for such a simple data structure.
// payload size in count of UInt8
let size = 16
// field offsets
let control1 = 0
let control2 = 1
let payload = 2..<(2 + size)
// random values between 0...255
let blob = (0..<size * 3).map{_ in UInt8(arc4random_uniform(UInt32(UInt8.max)))}
for index in 0.stride(through: blob.count - 1, by: size) {
var buffer = [UInt8](count: 2 + size, repeatedValue: 0)
buffer[control1] = 255
buffer[control2] = 0
buffer[payload] = blob[index..<index + size]
let data = NSData(bytesNoCopy: &buffer, length: buffer.count, freeWhenDone: false)
// send data
}