I would like to write a serializer application that will encode any type of C data structure into MessagePack format. All examples I have seen show the encoding of known structures such as this using MPack:
// encode to memory buffer
char* data;
size_t size;
mpack_writer_t writer;
mpack_writer_init_growable(&writer, &data, &size);
// write the example on the msgpack homepage
mpack_start_map(&writer, 2);
mpack_write_cstr(&writer, "compact");
mpack_write_bool(&writer, true);
mpack_write_cstr(&writer, "schema");
mpack_write_uint(&writer, 0);
mpack_finish_map(&writer);
// finish writing
if (mpack_writer_destroy(&writer) != mpack_ok) {
fprintf(stderr, "An error occurred encoding the data!\n");
return;
}
// use the data
do_something_with_data(data, size);
free(data);
But what I would like is to be able to encode any C data structure. For example if I had the following:
struct My_Struct{
int number1;
float number2;
char array[6];
};
struct My_Struct ms = {10, 4.44, "Hello"};
programatically, how would I know that the first 4 bytes represents an int so that I could call an mpack_write_int function to start packing the int into a messagepack format?
Related
I currently have a string in C that I have coded like this:
#define MESSAGE_STRING "{\"deviceName\":string, \"Telemetry\":{\"voltage\":12,\"current\":0.12}}"
As I am sending it to a cloud service that processes it as a JSON message and receives this:
But I want to do this in an easier way as this could get a lot more complex when I am sending more complicated JSON messages. I was wondering if there was any easy way to create these JSON objects within C easier?
Cheers
Write your JSON in a text file, and replace every parameter with a %d, %f, %s placeholder.
Then read this file, and fill in the values with sprintf()
But pay attention to the order of your parameters.
json.txt:
{
"device_name": "%s",
"Telemetry": {
"voltage": %d,
"current": %f
}
}
main.c
#include <stdio.h>
int main(void)
{
char json_format[4096];
char message_string[4096];
FILE* fp;
// Read in json format
fp = fopen("json.txt", "r");
int len = fread(json_format, 1, 4096, fp);
json_format[len] = '\0';
fclose(fp);
// Fill in parameters
const char* device_name = "string";
int voltage = 12;
float current = 0.12;
sprintf(message_string, json_format, device_name, voltage, current);
printf("%s", message_string);
}
The code is only for understanding the concept.
It works, but don't use it, because it doesn't check for erorrs and the memory management is bad too.
In the following example, I am taking a struct that occupies 32 bytes in memory and writing that to a file and reading it back in -- i.e., serializing the data to a binary format:
#include <stdio.h>
typedef struct _Person {
char name[20];
int age;
double weight;
} Person;
int main(void)
{
Person tom = (Person) {.name="Tom", .age=20, .weight=125.0};
// write the struct to a binary file
FILE *fout = fopen("person.b", "wb");
fwrite(&tom, sizeof tom, 1, fout);
fclose(fout);
// read the binary data and set the person to that
Person unknown;
FILE *fin = fopen("person.b", "rb");
fread(&unknown, sizeof unknown, 1, fin);
fclose(fin);
// confirm all looks ok
printf("{name=%s, age=%d, weight=%f}", unknown.name, unknown.age, unknown.weight);
}
Note however that these are all values on the stack, and no pointers/indirection is involved. How might data be serialized to a file when, for example, multiple pointers can be involved, multiple variables may point to the same memory location, etc. Is this effectively what protocol buffers do?
Ok so you want a binary file. I used to do it this way long ago. It's fine. It just breaks when you move to another platform or bitness. I'm teaching the old way because it's a good place to start. Newer ways are popular now because they keep working when changing platforms or bitnesses.
When writing records to the file I would use structs like this:
typedef struct _Person {
char name[20];
int age;
double weight;
} Person;
typedef struct _Thing {
char name[20];
};
typedef struct _Owner {
int personId;
int thingId;
} Owner;
See how the Owner structure has Id members. These are just indices into the arrays of the other structures.
These can be written out to a file one after another, usually prefixed by a single integer written directly that says how many records of each kind. The reader just allocates an array of structs with malloc big enough to hold them. As we add more items in memory we resize the arrays with realloc. We can (and should) also mark for deleting (say by setting the first character of name to 0) and reusing the record later.
The writer looks something like this:
void writeall(FILE *h, Person *allPeople, int nPeople, Thing *allThings, int nThings, Owner *allOwners, int nOwners)
{
// Error checking omitted for brevity
fwrite(&nPeople, sizeof(nPeople), 1, h);
fwrite(allPeople, sizeof(*allPeople), nPeople, h);
fwrite(&nThings, sizeof(nThings), 1, h);
fwrite(allThings, sizeof(*allThings), nThings, h);
fwrite(&nOwners, sizeof(nOwners), 1, h);
fwrite(allOwners, sizeof(*allOwners), nOwners, h);
}
The reader in turn looks like this:
int writeall(FILE *h, Person **allPeople, int *nPeople, int *aPeople, Thing **allThings, int *nThings, int *aThings, Owner **allOwners, int *nOwners, int *aOwners)
{
*aPeople = 0; // Don't crash on bad read
*aThigns = 0;
*aOwners = 0;
*allPeople = NULL;
*allThings = NULL;
*allOwners = NULL;
if (1 != fread(nPeople, sizeof(*nPeople), 1, h)) return 0;
*allPeople = malloc(sizeof(**allPeople) * *nPeople);
if (!allPeople) return 0; // OOM
*aPeople = *nPeople;
if (*nPeople != fread(*allPeople, sizeof(**allPeople), nPeople, h)) return 0;
if (1 != fread(nThings, sizeof(*nThings), 1, h)) return 0;
*allThings = malloc(sizeof(**allThings) * *nThings);
if (!allThings) return 0; // OOM
*aThings = *nThings;
if (*nThings != fread(*allThings, sizeof(**allThings), nThings, h)) return 0;
if (1 != fread(nOwners, sizeof(*nOwners), 1, h)) return 0;
*allOwners = malloc(sizeof(**allOwners) * *nOwners);
if (!allOwners) return 0; // OOM
*aOwners = *nOwners;
if (*nOwners != fread(*allOwners, sizeof(**allOwners), nOwners, h)) return 0;
return 1;
}
There was an old technique for writing a heap arena directly to disk and reading it back again. I recommend never using it, and never storing pointers on disk. That way lies security nightmares.
When memory was cheap I would have talked about how to use block allocation and linked blocks to dynamically update the on-disk records partially; but now for problems you're going to encounter at this level I say don't bother, and just read the whole thing into RAM and write it back out again. Eventually you will learn databases, which handles that stuff for you.
Desired outcome: Send a struct of GPS data from C using a web server and have the data ( lat, long, etc.) display on a web page.
Using: Linux, libwebsockets (LWS) library for web server. C code is implemented as a standalone plugin in the LWSWS (web server) app.
Current understanding/work on project: Casting struct from plugin to a char array and trying to parse the array (using JSON) into the individual members of the original struct. Then display each variable on a web page.
I am new to Linux, LWS, HTML, JavaScript, and JSON. I can use JSON or Javascript to work with the variables, whichever makes more sense. I am not completely sure of my plan on either side (C/JS) of this part of the project.
What is the best way to transfer this information and convert for use on the web page?
Here is my struct:
struct per_session_data__gps_rcvr {
struct time_struct{
int month;
int day;
int year;
int hour;
int minute;
double second;
}time;
double latitude;
char lat_indicator;
double longitude;
char lon_indicator;
double heading;
int quality;
int satellites;
};
Sending struct out, cast to a char pointer:
// Write GPS data to GUI
n = lws_snprintf( (char *)p, sizeof(buf) - LWS_PRE, "%s", (char *)pss );
m = lws_write(wsi, p, n, LWS_WRITE_TEXT);
if (m < n) {
lwsl_err("ERROR %d writing to di socket\n", n);
return -1;
}
break;
The test code I am modifying appears to receive data in a variable called "msg". I am assuming I need to parse the data sent in the char array into chunks that correspond to the members of the original struct. Then I can display each piece of GPS data on the web page.
var socket_gps;
if (use_lws_meta)
socket_gps = lws_meta.new_ws("", "gps-rcvr-protocol");
else
socket_gps = new_ws(get_appropriate_ws_url(""), "gps-rcvr-protocol");
try {
socket_gps.onopen = function() {
document.getElementById("wsdi_statustd").style.backgroundColor = "#40ff40";
document.getElementById("wsdi_status").innerHTML =
" <b>websocket connection opened</b><br>" +
san(socket_gps.extensions);
}
socket_gps.onmessage =function got_packet(msg) {
document.getElementById(" **VARIABLE GOES HERE** ").textContent = msg.data + "\n";
}
socket_gps.onclose = function(){
document.getElementById("wsdi_statustd").style.backgroundColor = "#ff4040";
document.getElementById("wsdi_status").textContent = " websocket connection CLOSED ";
}
}
catch(exception) {
alert('<p>Error' + exception);
}
Please let me know if there is a better way to send the data out from C, and/or if I have correctly configured the data. In my research, it was mentioned that data could be sent as binary, but sending plain text messages and using JSON seemed like a better plan.
It looks like JSON can "stringify" info, but only if it's typed in or in JavaScript format already? I'm not sure. I have not seen any way to parse out a char array, a set number of characters at a time, and save those segments as variables on the HTML side. That is what I was hoping for.
Thank you for your suggestions!
Like this:
n = snprintf( (char *)p, sizeof(buf), "{\"NameofParam1\":\"%d\",\"NameofParam2\":\"%d\"}", param1, param2 );
That gives you your whole JSON string in one go (stored in buffer). Just use the appropriate flag (%s, %X, %c, %f, etc.) to capture your parameter.
hi i have used libpng to convert grayscale png image to raw image using c. in that lib the function png_init_io needs file pointer to read the png. but i pass the png image as buffer is there any other alternative functions to to read png image buffer to raw image. please help me
int read_png(char *file_name,int *outWidth,int *outHeight,unsigned char **outRaw) /* We need to open the file */
{
......
/* Set up the input control if you are using standard C streams */
png_init_io(png_ptr, fp);
......
}
instead i need this like
int read_png(unsigned char *pngbuff, int pngbuffleng, int *outWidth,int *outHeight,unsigned char **outRaw) /* We need to open the file */
{
}
From the manual of png_init_io, it is apparent that you can override the read function with png_set_read_fn.
Doing this, you can fool png_init_io into thinking that it is reading from a file, while in reality you're reading from a buffer:
struct fake_file
{
unsigned int *buf;
unsigned int size;
unsigned int cur;
};
static ... fake_read(FILE *fp, ...) /* see input and output from doc */
{
struct fake_file *f = (struct fake_file *)fp;
... /* read a chunk and update f->cur */
}
struct fake_file f = { .buf = pngBuff, .size = pngbuffleng, .cur = 0 };
/* override read function with fake_read */
png_init_io(png_ptr, (FILE *)&f);
Try to use png_set_read_fn instead of png_init_io.
I am very much stuck in the following issue. Any help is very much appreciated!
Basically I have a program wich contains an array of structs and I am getting a segmentation error when I call an external function. The error only happens when I have more than 170 items on the array being passed.
Nothing on the function is processed. The program stops exactly when accessing the function.
Is there a limit for the size of the parameters that are passed to external functions?
Main.c
struct ratingObj {
int uid;
int mid;
double rating;
};
void *FunctionLib; /* Handle to shared lib file */
void (*Function)(); /* Pointer to loaded routine */
const char *dlError; /* Pointer to error string */
int main( int argc, char * argv[]){
// ... some code ...
asprintf(&query, "select mid, rating "
"from %s "
"where uid=%d "
"order by rand()", itable, uid);
if (mysql_query(conn2, query)) {
fprintf(stderr, "%s\n", mysql_error(conn2));
exit(1);
}
res2 = mysql_store_result(conn2);
int movieCount = mysql_num_rows(res2);
// withhold is a variable that defines a percentage of the entries
// to be used for calculations (generally 20%)
int listSize = round((movieCount * ((double)withhold/100)));
struct ratingObj moviesToRate[listSize];
int mvCount = 0;
int count =0;
while ((row2 = mysql_fetch_row(res2)) != NULL){
if(count<(movieCount-listSize)){
// adds to another table
}else{
moviesToRate[mvCount].uid = uid;
moviesToRate[mvCount].mid = atoi(row2[0]);
moviesToRate[mvCount].rating = 0.0;
mvCount++;
}
count++;
}
// ... more code ...
FunctionLib = dlopen("library.so", RTLD_LAZY);
dlError = dlerror();
if( dlError ) exit(1);
Function = dlsym( FunctionLib, "getResults");
dlError = dlerror();
(*Function)( moviesToRate, listSize );
// .. more code
}
library.c
struct ratingObj {
int uid;
int mid;
double rating;
};
typedef struct ratingObj ratingObj;
void getResults(struct ratingObj *moviesToRate, int listSize);
void getResults(struct ratingObj *moviesToRate, int listSize){
// ... more code
}
You are likely blowing up the stack. Move the array to outside of the function, i.e. from auto to static land.
Another option is that the // ... more code - array gets populated... part is corrupting the stack.
Edit 0:
After you posted more code - you are using C99 variable sized array on the stack - Bad IdeaTM. Think what happens when your data set grows to thousands, or millions, of records. Switch to dynamic memory allocation, see malloc(3).
You don't show us what listsize is, but I suppose it is a variable and not a constant.
What you are using are variable length arrays, VLA. These are a bit dangerous if they are too large since they usually allocated on the stack.
To work around that you can allocate such a beast dynamically
struct ratingObj (*movies)[listSize] = malloc(sizeof(*movies));
// ...
free(movies);
You'd then have in mind though that movies then is a pointer to array, so you have to reference with one * more than before.
Another, more classical C version would be
struct ratingObj * movies = malloc(sizeof(*movies)*listsize);
// ...
free(movies);