I'm trying to length prefix a payload that i'm streaming through a socket, and so far the only way i've been able to get it working is:
Uint8List payload = getPayloadSomehow();
final lBytes = ByteData(4)..setUint32(0, payload.length);
final prefix = lBytes.buffer.asUint8List();
final prefixedPayload = []..addAll(prefix)..addAll(payload);
Creating a ByteData and filling it with the length, and then extracting the buffer as a Uint8List feels very roundabout. But i haven't been able to find a cleaner way to do the conversion and prefixing.
I'd really appreciate it if someone could point me to a better solution, thanks.
How about:
var payload = getPayloadSomehow();
var prefixed = Uint8List(payload.length + 4);
prefixed.buffer.asUint32List(0, 1)[0] = payload.length;
prefixed.setRange(4, prefixed.length, payload);
Related
How to properly encode data with NanoPB when having several nested 'repeated' fields?
This is my schema:
message Report {
message SensorData {
required uint32 sensorid = 1;
required uint32 sample = 2;
}
message DeviceData {
required uint32 devid = 1;
repeated SensorData sensor_data = 2;
}
required uint32 reportnum = 1;
repeated DeviceData dev_data = 2;
}
I have already made a working version in which SensorData fields are embedded inside DeviceData message based on the server.c example from the NanoPB source. This way I have only one repeated field and everything works fine. However this way I have to repeat the 'devid' field for every sensorid and every 'sample', instead of giving it just one time and then loop through an array of SensorData messages. However I am struggling to encode this with NanoPB, the decoding part is in Python. Can someone give me an example how to properly encode data in this case?
For me the simplest way to do this was by statically defining the size of the array's using a nanopb options file. After that you can access each element just like an array.
report.dev_data[i].devid[j] = 1234;
report.dev_data[i].sensor_data[j] = 9876;
using the randdusing bluetoothle plugin for ionic app, need to read the advertisement.
The ble scan returns with Start Scan Success :
{"address":"14::30:c6:60:e8;9f","name":null,"rssi":-50,"advertisement":"AgEGG/9SVgIADSw5YTNlMTQAAAJlAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA;=","status":"scanResult"}
query: need to decipher this json data and convert this advertisement data into array containing hex values of advertisement data? The advertisement data seems to be base64 encoded. Please advice.
I made for this purpose a little helper function as shown below. The key is the $cordovaBluetoothLE.encodedStringToBytes as you can see in docs https://github.com/randdusing/ng-cordova-bluetoothle.
var encodedToByteString = function encodedToByteString(input) {
var val = $cordovaBluetoothLE.encodedStringToBytes(input);
var byteStr = "";
for (var i = 0; i < val.length; i++) {
var byte = val[i].toString(16);
if (byte.length == 1) byte = "0" + byte;
byteStr += byte;
}
return byteStr;
};
The same goes for the opposite operation - that is sending data. You first need to get your hex-string into an array of bytes and then encode it via $cordovaBluetoothLE.bytesToEncodedString(value).
I am receiving a Base64 encoded string that contains a word document (generated by thunderHead), that I decode with:
byte[] dataBytes = Convert.FromBase64String(data)
When I write directly this byteArray into a file:
File.WriteAllBytes("myFile.doc", dataByte);
The word document is properly written (é, è are shown properly), but it's not the case when I send this byteArray to the front (angularjs):
var response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new ByteArrayContent(dataByte);
response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/msword");(also tried "application/octet-stream")
response.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachement");
response.Content.Headers.ContentDisposition.Filename = "toto.doc"
return response;
I can't figure it out....some help would be appreciated thanks!
Solved my problem, as #peco pointed, the problem was on the front part, I was doin a $http.post(url, data).then ... and missed to add {responseType: 'arrayBuffer'} as a third parameter: $http.post(url, data, {responseType: 'arrayBuffer'}).then... Thanks.
JSON.stringify is obviously not space-efficient. What is the most elegant way to serialize and store a float32array using Node.js?
EDIT: People are closing the question for reasons such as being "opinion based" and "lack of an understanding of the problem". I seriously believe the first one was a missclick. For the second one, maybe this makes it more clear:
var fs = require("fs");
var len = 1000*1000*10;
var big_array = new Float32Array(len);
for (var i=0; i<len; ++i)
big_array[i] = Math.random();
// OBVIOUSLY NOT SPACE EFFICIENT \/
fs.writeFileSync("big_array.json",JSON.stringify(big_array));
It is not space efficient because you are representing numbers as strings, so an 8 bytes float will be using as much as ~20 utf8 chars, which is a waste. The question is: how to store the array in a space-efficient manner?
Finally I managed to write float32array to disk with nodejs and retrieve them on the browser, and I hope it will help you.
Write Float32Array to binary file in NodeJS
var fs = require('fs');
var wstream = fs.createWriteStream('data.dat');
var data = new Float32Array([1.1,2.2,3.3,4.4,5.5]);
//prepare the length of the buffer to 4 bytes per float
var buffer = new Buffer(data.length*4);
for(var i = 0; i < data.length; i++){
//write the float in Little-Endian and move the offset
buffer.writeFloatLE(data[i], i*4);
}
wstream.write(buffer);
wstream.end();
Read the file and convert it to a Float32Array on a Browser
var urlToFloatFile = 'data.dat';
var request = new XMLHttpRequest();
request.open('GET', urlToFloatFile, true);
//specify the response type as arraybuffer
request.responseType = 'arraybuffer';
request.onload = function (msg) {
var yourFloatData = new Float32Array(this.response);
console.log(yourFloatData);
};
request.send();
Thanks to #ben_a_adams from WebGL Dev List GGroup https://groups.google.com/forum/#!topic/webgl-dev-list/EbGUi_iSEx8 for the client side code
I've create a simple test to test roughly how much space a JSON serialization of a float array differs from a binary representation and the results are:
2.000.000 floating point values
7.8MB on a binary file
38.5MB on a JSON file
17.5 on a Gzipped JSON file
There is actually a much simpler version possible
let fs = require('fs')
let data = [150, 180]
fs.writeFileSync('mydata', new Buffer(new Uint32Array(data).buffer))
fs.readFile('mydata', (err, buf) => {
let restoredData = new Uint32Array(buf.buffer, buf.offset, buf.byteLength/4)
console.log(data[1])
console.log(restoredData[1])
});
Easy, clean way to do it:
const float32Array = new Float32Array([.69,.420])
const buffer = Buffer.from(float32Array.buffer)
fs.writeFileSync(filePath, buffer)
const loadedBuffer = fs.readFileSync(filePath)
const newFloat32Array = new Float32Array(loadedBuffer.buffer)
I believe you could use Meteor's EJSON:
http://docs.meteor.com/#ejson
https://npmjs.org/package/meteor-ejson
EJSON is an extension of JSON to support more types. It supports all
JSON-safe types, as well as:
Date (JavaScript Date)
Binary (JavaScript Uint8Array or the result of EJSON.newBinary)
User-defined types (see EJSON.addType. For example, Meteor.Collection.ObjectID is implemented this way.)
I'm trying to convert the byteArray of a Sound Object to an array with floats. The Sound Object plays back fine & at full length, but the float Array i get from it is cut off (but sounds correct), so i must be doing something wrong in the conversion:
var s:Sound = mySound;
s.play(); // plays fine
var bytes:ByteArray = new ByteArray();
bytes.endian = Endian.LITTLE_ENDIAN;
s.extract(bytes, s.bytesTotal, 0);
var leftChannel:Array = new Array();
var rightChannel:Array = new Array();
bytes.position = 0;
while (bytes.bytesAvailable)
{
leftChannel.push(bytes.readFloat());
rightChannel.push(bytes.readFloat());
}
and this is what i get:
The top two channels are the original Sound Object.
The lower two is the float Array Data. I aligned them so you can see that the beginning is cut off and obviously the length is incorrect.
Thanks for any answers...
ok there were two problems:
the mp3 file i was importing was somehow corrupt, that caused the beginning to be cut off
the length i defined to extract was not correct, to find the full sound length use
var numTotalSamples:Number = int(s.length * 44.1); //assuming 44.1kHz sample rate
then:
s.extract(bytes, numTotalSamples, 0);