I'm trying to save a binary blob of my structure as value in Redis (via hiredis) using SET. I set it with one execution of my program and try to retrieve it later in another run. For some reason, I'm not able to get the dynamically allocated string back.
Here's the simplified code fragments of what I'm doing. In the real implementation, my struct is more complex with strings, linked lists, sub structs.
typedef struct mytest {
int myInt;
char *myStr;
} mytest;
char *key = strdup("test_key");
int vsize = 0;
mytest t;
memset(&t, 0, sizeof(t));
t.myInt = 100;
vsize += sizeof(t.myInt);
t.myStr = strdup("test_string");
vsize += strlen(t.myStr);
redisReply *reply = 0;
// set value
reply = redisCommand(context, "SET %b %b", key, (size_t) strlen(key), &t, (size_t) vsize);
if (!reply)
return REDIS_ERR;
freeReplyObject(reply);
// get the value back
reply = redisCommand(context, "GET %b", key, (size_t) strlen(key));
struct mytest *retval = (struct mytest *) reply->str;
printf("GET %s: myInt = %d myStr = %s", retval->myInt, retval->myStr);
I have following questions:
What am I doing wrong here?
Is SET/GET the correct way to store complex data structure or should I use something else in Redis? I want to access the structure as single entity and not as separate fields.
In case of member linked lists, how should they be stored?
I couldn't find any examples that address my use case, so any examples would help.
Your code do not work properly because of variable to contains string myStr as a pointer to string rather than string itself:
reply = redisCommand(context, "SET %b %b", key, (size_t) strlen(key), &t, (size_t) vsize);
Try to work with this like that:
redisReply *reply = redisCommand(context, "SET %b %b", key, (size_t) strlen(key), t.myStr, t.myInt);
...
struct mytest;
memset(&mytest, 0, sizeof(mytest));
reply = redisCommand(context, "GET %b", key, (size_t) strlen(key));
mytest.myInt = strlen(reply->str);
mytest.myStr = strdup(reply->str);
freeReplyObject (reply);
Related
My question is like this
below is the structure which i have to populate and send it through socket
struct Mystruct
{
int numofarray1elements;
array1[50];
int numofarray2elements;
array2[25];
};
Here size of 1 array1 member is 1024 bytes i.e total size of array1 = 50*1024 = 51200 bytes
size of 1 member of array2 is say 500 so total size of array2 = 12500 bytes
whenever i use send api of socket(unix domain socket) i have to send 51200+12500 +4+4 = 63708 bytes
Problem is i have to send entire size of structure even if i have very less
numofarray1elements, and numofarray2elements
this leads to performance issue
where in almost cases my original data can be less than 10kb but i end up sending 63k everytime
i cannot keep dynamic arrays as its socket message
i have already otimized my data strutures , array1 must have max 50 elements
array2 must have max 25 elements.
now is there any way that i can send exact data which i have populated?
please provide some method if any
Thanks
Actually the way for doing it is to have messages of variable length. One of the methods is to use a single array with undefined size as the last element of the struct. Depending on the types of the messages it could be represented by messages or by bytes, e.g.
struct Mystruct
{
int numofarray1elements;
int numofarray2elements;
char array[];
};
The size of your struct can be calculated as the size of static fields plus sizes needed for actual payload as this:
int packetSize = (sizeof(struct Mystruct) + n1 * sizeof(el1) + n2 * sizeof(el2));
now you can use it to allocate the struct and send the packet in a single operation.
struct Mystruct *packet = malloc(packetSize);
// assign packet fields
...
write(fd, packet, packetSize);
Here is a simple example which emulates a version of write/read. It will work if writer and reader have the same endian order. It also assumes that the packet size is sent separately an is known to the reader.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
struct Mystruct {
int numofarray1elements;
int numofarray2elements;
char payload[];
};
struct Element1 {
int len;
char name[30];
};
struct Element2 {
char name[20];
int len;
};
// reader emulation
void readData(int packetSize, char *dataIn) {
union {
char data[packetSize];
struct Mystruct packet;
} dataUnion;
int i;
struct Element1 *e1 = NULL;
struct Element2 *e2 = NULL;
memcpy(dataUnion.data, dataIn, packetSize);
printf("Read data e1 size is %d, e2 size is %d\n",
dataUnion.packet.numofarray1elements, dataUnion.packet.numofarray2elements);
e1 = malloc(sizeof(struct Element1) * dataUnion.packet.numofarray1elements);
e2 = malloc(sizeof(struct Element2) * dataUnion.packet.numofarray2elements);
memcpy(e1, dataUnion.packet.payload, sizeof(struct Element1) * dataUnion.packet.numofarray1elements);
memcpy(e2, dataUnion.packet.payload + sizeof(struct Element1) * dataUnion.packet.numofarray1elements,
sizeof(struct Element2) * dataUnion.packet.numofarray2elements);
for (i = 0; i < dataUnion.packet.numofarray1elements; i++) {
printf("e1[%d].len = %d, name = %s\n", i, e1[i].len, e1[i].name);
}
for (i = 0; i < dataUnion.packet.numofarray2elements; i++) {
printf("e2[%d].len = %d, name = %s\n", i, e2[i].len, e2[i].name);
}
}
void main() {
struct Element1 e1[4];
struct Element2 e2[8];
int i;
int packetSize;
struct Mystruct *packet = NULL;
for (i = 0; i < 4; i++) {
sprintf(e1[i].name, "e1:%d", i);
e1[i].len = i;
}
for (i = 0; i < 8; i++) {
sprintf(e2[i].name, "e2:%d", i);
e2[i].len = i;
}
// emulated write data
packetSize = (sizeof(struct Mystruct) + sizeof(e1) + sizeof(e2));
packet = malloc(packetSize);
packet->numofarray1elements = 4;
packet->numofarray2elements = 8;
memcpy(packet->payload, &e1, sizeof(e1));
memcpy(packet->payload + sizeof e1, &e2, sizeof(e2));
// here you do write data, e.g. write(socFd, packet, packetSize);
// emulate read data
readData(packetSize, (char*)packet);
}
Instead of using structures, use TLV mechanism. So for your solution: you can use Type , , Count of Type, Length , Value.
Define types which is known at both receiver and sender side
Define your message structure as
Type taking 2 bytes, Count-of-type taking 2 or 4 bytes , Length taking 4 bytes and Value.
This is extensible as you can add any number of types in future as long as the type is known at both side. At receiver side if type is not known, they can ignore that TLV.
You should not send structs over network protocols (or store them in files, etc). You need to serialize them.
But to solve your problem, simply change your send code:
//this is what your code looks like, I assume:
write(sockFd, myStructVariable, sizeof(struct MyStruct));
to:
//be aware of writev(2) if you want to send these in one system call at once, or copy them into one buffer
write(sockFd, myStructVariable.numofarray1elements, sizeof(int));
write(sockFd, myStructVariable.array1, sizeof(MyArray1) * myStructVariable.numofarray1elements);
write(sockFd, myStructVariable.numofarray2elements, sizeof(int));
write(sockFd, myStructVariable.array2, sizeof(MyArray2) * myStructVariable.numofarray2elements);
or something similar
Then change your receive code:
read(sockFd, myStructVariable.numofarray1elements, sizeof(int));
read(sockFd, myStructVariable.array1, sizeof(MyArray1) * myStructVariable.numofarray1elements);
read(sockFd, myStructVariable.numofarray2elements, sizeof(int));
read(sockFd, myStructVariable.array2, sizeof(MyArray2) * myStructVariable.numofarray2elements);
of course, you could also leave the array and counts defined locally and send those instead. Make sure you check for errors, short reads, etc
Take a look at flatbuffers as well, this lets you write structures that come with serialization functions
Here, I explain my problem, I am a beginner on the ptrace function and I would like to succeed in recovering the hard information of a structure.
For example with this command, I will have strace -e trace = fstat ls
a line: fstat (3, {st_mode = ..., st_size = ...}
and I would like to successfully retrieve the contents of the structure (st_mode) and (st_size).
I try this but to no avail:
int buffer(unsigned long long addr, pid_t child, size_t size, void *buffer)
{
size_t byte = 0;
size_t data;
unsigned long tmp;
while (byte < size) {
tmp = ptrace(PTRACE_PEEKDATA, child, addr + byte);
if ((size - byte) / sizeof(tmp))
data = sizeof(tmp);
else
data = size % sizeof(tmp);
memcpy((void *)(buffer + byte), &tmp, data);
byte += data;
}
}
and in params :
struct stat stat_i;
buffer(addr, pid, sizeof(stat_i), &stat_i);
printf("%lu", stat_i.st_size); -> fake value :/
Thank'ks !
From the man page,
PTRACE_PEEKTEXT, PTRACE_PEEKDATA
Read a word at the address addr in the tracee's memory,
returning the word as the result of the ptrace() call. Linux
does not have separate text and data address spaces, so these
two requests are currently equivalent. (data is ignored; but
see NOTES.)
Thus you must understand that tmp would hold the actually value that was read.
Your checks are wrong - you should set errno = 0 before the call and then check if it has changed. If it has - you've got an error. If it hasn't - you can be assured that tmp has the word from the remote process.
Try something like this:
int buffer(unsigned long long addr, pid_t child, size_t size, void *buffer)
{
size_t byte = 0;
size_t data;
unsigned long tmp;
// support for word aligned sizes only
if (size % sizeof(long) != 0)
return -1;
long * buffer_int = (long*) buffer;
while (byte < size) {
errno = 0;
tmp = ptrace(PTRACE_PEEKDATA, child, addr + byte);
if (errno)
return -1;
buffer_int[byte / sizeof(long)] = tmp;
byte += sizeof(long);
}
}
I am implementing a way to transfer a set of data to a programmable dongle. The dongle is based on a smart card technology and can execute an arbitrary code inside. The input and output data is passed as a binary blocks that can be accessed via input and output pointers.
I would like to use an associative array to simplify the data processing code. Everything should work this way:
First the host application:
// Host application in C++
in_data["method"] = "calc_r";
in_data["id"] = 12;
in_data["loc_a"] = 56.19;
in_data["loc_l"] = 44.02;
processor->send(in_data);
Next the code inside the dongle:
// Some dongle function in C
char* method_name = assoc_get_string(in_data, "method");
int id = assoc_get_int(in_data, "id");
float loc_a = assoc_get_float(in_data, "loc_a");
float loc_l = assoc_get_float(in_data, "loc_l");
So my question is about the dongle part functionality. Is there C code or library to implement such an associative array behavior like the above?
Glib's hash table. implements a map interface or (associative array).
And it's most likely the most used hash table implementation for C.
GHashTable *table=g_hash_table_new(g_str_hash, g_str_equal);
/* put */
g_hash_table_insert(table,"SOME_KEY","SOME_VALUE");
/* get */
gchar *value = (gchar *) g_hash_table_lookup(table,"SOME_KEY");
My suspicion is that you would have to write your own. If I understand the architecture you are describing, then you will need to send the entire chunk of data in a single piece. If so, then most libraries will not work for that because they will most likely be allocating multiple pieces of memory, which would require multiple transfers (and an inside understanding of the structure). It would be similar to trying to use a library hash function and then sending its contents over the network on a socket just by passing the root pointer to the send function.
It would be possible to write some utilities of your own that manage a very simple associative array (or hash) in a single block of memory. If the amount of data is small, it could use a simple linear search for the entries and would be a fairly compact bit of code.
Try uthash, a header library implementing a hash table in C. It's small and fairly easy to use.
This is an old thread, but I thought this might still be useful for anyone out there looking for an implementation. It doesn't take too much code; I did mine in ~100 lines of without any extra library. I called it a dictionary since it parallels (sort of) the python datatype. Here is my code:
#include <stdlib.h>
#include <stdio.h>
#include <stdbool.h>
typedef struct hollow_list hollow_list;
struct hollow_list{
unsigned int size;
void *value;
bool *written;
hollow_list *children;
};
//Creates a hollow list and allocates all of the needed memory
hollow_list hollow_list_create(unsigned int size){
hollow_list output;
output = (hollow_list) {.size = size, .value = (void *) 0, .written = calloc(size, sizeof(bool)), .children = calloc(size, sizeof(hollow_list))};
return output;
}
//Frees all memory of associated with a hollow list and its children
void hollow_list_free(hollow_list *l, bool free_values){
int i;
for(i = 0; i < l->size; i++){
hollow_list_free(l->children + i, free_values);
}
if(free_values){
free(l->value);
}
free(l);
}
//Reads from the hollow list and returns a pointer to the item's data
void *hollow_list_read(hollow_list *l, unsigned int index){
if(index == 0){
return l->value;
}
unsigned int bit_checker;
bit_checker = 1<<(l->size - 1);
int i;
for(i = 0; i < l->size; i++){
if(bit_checker & index){
if(l->written[i] == true){
return hollow_list_read(l->children + i, bit_checker ^ index);
} else {
return (void *) 0;
}
}
bit_checker >>= 1;
}
}
//Writes to the hollow list, allocating memory only as it needs
void hollow_list_write(hollow_list *l, unsigned int index, void *value){
if(index == 0){
l->value = value;
} else {
unsigned int bit_checker;
bit_checker = 1<<(l->size - 1);
int i;
for(i = 0; i < l->size; i++){
if(bit_checker & index){
if(!l->written[i]){
l->children[i] = hollow_list_create(l->size - i - 1);
l->written[i] = true;
}
hollow_list_write(l->children + i, bit_checker ^ index, value);
break;
}
bit_checker >>= 1;
}
}
}
typedef struct dictionary dictionary;
struct dictionary{
void *value;
hollow_list *child;
};
dictionary dictionary_create(){
dictionary output;
output.child = malloc(sizeof(hollow_list));
*output.child = hollow_list_create(8);
output.value = (void *) 0;
return output;
}
void dictionary_write(dictionary *dict, char *index, unsigned int strlen, void *value){
void *hollow_list_value;
dictionary *new_dict;
int i;
for(i = 0; i < strlen; i++){
hollow_list_value = hollow_list_read(dict->child, (int) index[i]);
if(hollow_list_value == (void *) 0){
new_dict = malloc(sizeof(dictionary));
*new_dict = dictionary_create();
hollow_list_write(dict->child, (int) index[i], new_dict);
dict = new_dict;
} else {
dict = (dictionary *) hollow_list_value;
}
}
dict->value = value;
}
void *dictionary_read(dictionary *dict, char *index, unsigned int strlen){
void *hollow_list_value;
dictionary *new_dict;
int i;
for(i = 0; i < strlen; i++){
hollow_list_value = hollow_list_read(dict->child, (int) index[i]);
if(hollow_list_value == (void *) 0){
return hollow_list_value;
} else {
dict = (dictionary *) hollow_list_value;
}
}
return dict->value;
}
int main(){
char index0[] = "hello, this is a test";
char index1[] = "hello, this is also a test";
char index2[] = "hello world";
char index3[] = "hi there!";
char index4[] = "this is something";
char index5[] = "hi there";
int item0 = 0;
int item1 = 1;
int item2 = 2;
int item3 = 3;
int item4 = 4;
dictionary d;
d = dictionary_create();
dictionary_write(&d, index0, 21, &item0);
dictionary_write(&d, index1, 26, &item1);
dictionary_write(&d, index2, 11, &item2);
dictionary_write(&d, index3, 13, &item3);
dictionary_write(&d, index4, 17, &item4);
printf("%d\n", *((int *) dictionary_read(&d, index0, 21)));
printf("%d\n", *((int *) dictionary_read(&d, index1, 26)));
printf("%d\n", *((int *) dictionary_read(&d, index2, 11)));
printf("%d\n", *((int *) dictionary_read(&d, index3, 13)));
printf("%d\n", *((int *) dictionary_read(&d, index4, 17)));
printf("%d\n", ((int) dictionary_read(&d, index5, 8)));
}
Unfortunately you can't replicate the list[x] syntax, but this is the best alternative I have come up with.
Yes, but it will not work in the way you have specified. It will instead use a struct to store the data and functions that operate on that struct, giving you the result you want. See A Simple Associative Array Library In C. Example of use:
struct map_t *test;
test=map_create();
map_set(test,"One","Won");
map_set(test,"Two","Too");
map_set(test,"Four","Fore");
GLib's Hash Tables and Balanced Binary Trees might be what you're after.
Mark Wilkins gave you the right answer. If you want to send the data as a single chunk, you need to understand how C++ maps are represented in your architecture and write the access functions.
Anyway, if you decide to recreate the map on the dongle, I've written a small C library where you could write thinks like:
tbl_t in_data=NULL;
tblSetSS(in_data,"method","calc_r");
tblSetSN(in_data,"id",12);
tblSetSF(in_data,"loc_a",56.19);
tblSetSF(in_data,"loc_l",44.02);
and then:
char *method_name = tblGetP(in_data, "method");
int id = tblGetN(in_data, "id");
float loc_a = tblGetF(in_data, "loc_a");
float loc_l = tblGetF(in_data, "loc_l");
The hashtable is a variation of the Hopscotch hash, which is rather good on average, and you can have any mix of type for keys and data (i.e. you can use an entire table as a key).
The focus for that functions was on easing programming rather than pure speed and the code is not thoroughly tested but if you like the idea and want to expand on it, you can have a look at the code on googlecode.
(There are other things like variable length strings and a fast sttring pattern matching function but those might not be of interest in this case).
I'm implementing an iterator to go over the records from a Berkeley DB. However, it seems I need to set the DB_DBT_USERMEM flag before the call to cursor->get with DB_NEXT.
Doing it that way would make my iterator less cohesive and will have to implement multiple iterators for each data type I want to retrieve.
Is there a way to have a generic iterator that can traverse structures w/o pointers, and basic data types? Here's what I'm trying to achieve.
#include <stdio.h>
#include <string.h>
#include <db.h>
// let this function handle integers and use DB_DBT_USERMEM for memory alignment
void integer_items(DBT key, DBT data) {
int number = 0;
data.data = &number;
data.flags = DB_DBT_USERMEM;
data.ulen = sizeof(int);
printf("key is: %s, data is: %d\n", (char *) key.data,number);
}
// let this function handle pointer structs. No need for DB_DBT_USERMEM
void ptr_struct_items(DBT key, DBT data) {
// MY_STRUCT user;
// marshall struct...
// buffsize = sizeof(int) +(strlen(user.familiar_name) + strlen(user.surname) + 2);
// databuff = malloc(buffsize);
// memset(databuff, 0, buffsize);
// ...
// printf("key is: %s, data is: %d\n", (char *) key.data,number);
}
int iterator(DB *database, void(*function)(DBT key, DBT data)) {
DBT key, data;
DBC *cursor;
memset(&key, 0, sizeof(DBT));
memset(&data, 0, sizeof(DBT));
database->cursor(database, NULL, &cursor, 0);
while(cursor->c_get(cursor, &key, &data, DB_NEXT) == 0){
(*function)(key, data);
}
cursor->c_close(cursor);
return 0;
}
int main() {
DB_ENV *myEnv;
DB *dbp;
DBT key, data;
int r, v = 10;
char *k = "Test";
db_env_create(&myEnv, 0);
myEnv->open(myEnv, "./", DB_CREATE | DB_INIT_MPOOL, 0);
db_create(&dbp, myEnv, 0);
dbp->open(dbp, NULL, "test.db", NULL, DB_HASH, DB_CREATE, 0664);
memset(&key, 0, sizeof(key));
memset(&data, 0, sizeof(data));
key.data = k;
key.size = strlen(k) +1;
data.data = &v;
data.size = sizeof(int);
if((r=dbp->put(dbp, NULL, &key, &data, 0)!=0))
fprintf(stderr, "%s\n", db_strerror(r));
iterator(dbp, integer_items);
iterator(dbp, ptr_struct_items);
return 0;
}
You almost always want to use DB_DBT_USERMEM, if only to avoiding the malloc() from inside BDB for DB_DBT_MALLOC/REALLOC. When you use it, you must pass in your own memory large enough to hold the largest item in your database. This holds for the key DBT too, as you may want to use it there.
In your example, as the key and data are so small, I'd just put character arrays on the stack in your "iterator" function, and then initialize key and data after the call to memset(). What you've got above is wrong because you're setting USERMEM after the call to c_get().
Here's a reworked example that gives BDB 256 bytes to work with for key and data.
void integer_items(DBT key, DBT data) {
int number = 0;
if (data.size == sizeof number) {
number = *(int *)data.data;
printf("key is: %s, data is: %d\n", (char *) key.data, number);
}
}
int iterator(DB *database, void(*function)(DBT key, DBT data)) {
DBT key, data;
DBC *cursor;
char kmem[256];
char dmem[256];
memset(&key, 0, sizeof(DBT));
memset(&data, 0, sizeof(DBT));
key.flags = DB_DBT_USERMEM;
key.data = kmem;
key.ulen = sizeof kmem;
data.flags = DB_DBT_USERMEM;
data.data = dmem;
data.ulen = sizeof dmem;
database->cursor(database, NULL, &cursor, 0);
while(cursor->c_get(cursor, &key, &data, DB_NEXT) == 0){
(*function)(key, data);
}
cursor->c_close(cursor);
return 0;
}
To handle different structures inside your iterator, include the data type as part of the key somehow. For example, instead of a bare integer for the key, use a struct, and have the first character define which kind of type it is. Then, inside your iterator function, you can switch on that.
I'm having some difficulties extracting data from a buffer using memcpy.
First, I memcpy some variables into a buffer:
int l1_connect(const char* hostname, int port) {
// Variables to be stored in the buffer
char *msg = "Hi, I'm a message"; // strlen(msg) == 17
uint16_t sender_id = htons(1); // sizeof(sender_id) == 2
uint16_t packet_size = htons(sizeof(packet_size)+sizeof(sender_id)+strlen(msg)); // sizeof(packet_size) == 2
// Checking values
printf("l1_connect():\nsender_id: %d, packet_size: %d\n\n", ntohs(sender_id), ntohs(packet_size));
// sender_id == 1, packet_size == 21
// The buffer
char buf[100];
// Copying everything
memcpy(&buf, &sender_id, sizeof(sender_id));
memcpy(&buf+sizeof(sender_id), &packet_size, sizeof(packet_size));
memcpy(&buf+sizeof(sender_id)+sizeof(packet_size), &msg, strlen(msg));
// Passing buf to another function
int bytes_sent = l1_send(1, buf, sizeof(buf));
}
I then try to extract that data (checking, before sending over UDP socket):
int l1_send( int device, const char* buf, int length ) {
// Variables in which to store extracted data
uint16_t id = 0;
uint16_t size = 0;
char msg[50];
memcpy(&id, &buf, sizeof(id));
memcpy(&size, &buf+sizeof(id), sizeof(size));
int remaining = ntohs(size) - (sizeof(id) + sizeof(size));
printf("l1_send():\nremaining: %d\n", remaining); // -37041
// memcpy-ing with correct(?) offset
memcpy(&msg, &buf+sizeof(id)+sizeof(size), 50);
msg[49] = '\0';
printf("id: %d\n", ntohs(id)); // 8372
printf("size: %d\n", ntohs(size)); // 37045
printf("msg: %s\n", msg); // ��$_�
return 0; // For now
}
As you can see, the values aren't quite what I'm expecting. Can anyone tell me what I'm doing wrong?
Your pointer math is incorrect. You're using &buf where you should just be using buf. If this doesn't explain what is wrong, nothing else I can say will:
#include <stdio.h>
int main(int argc, char **argv)
{
char buff[100];
printf("buff : %p\nbuff+10 : %p\n&buff+10 : %p\n", buff, buff+10, &buff+10);
return 0;
}
Output (varies by platform, obviously)
buff : 0xbf87a8bc
buff+10 : 0xbf87a8c6
&buff+10 : 0xbf87aca4
See it live. The math you're doing is incrementing by type, which for &buf is a pointer to array of 100 chars; not a simple char address. Therefore, &buff + 10 (in my sample) says "give me the 10th array of 100 chars from where I am now.". The subsequent write is invoking undefined behavior as a consequence.
Valgrind is your buddy here, btw. It would have caught this in a heartbeat.
Update
May as well fill in the entire gambit while I'm here. This is also wrong in l1_send:
memcpy(&id, &buf, sizeof(id));
// this------^
and the subsequent other areas you're using it in that function. You're taking the address of a parameter pointer, not the value within it. I'm confident you need buf there as well.
Try this:
memcpy(buf, &sender_id, sizeof(sender_id));
memcpy(buf + sizeof(sender_id), &packet_size, sizeof(packet_size));
memcpy(buf + sizeof(sender_id) + sizeof(packet_size), msg, strlen(msg));
To help you understand what is wrong with your code, you can read this.
Related: Pointer math vs. Array index