So ,i implemented my own mpi library(a simplified version) and I need to send/receive between processes some data. MPI_Send looks like this(void *buf,int count,datatype data,etc...). SO this means i need to send count elements of type data(char,double or int) pointed at the address by buf. I need to send em through a message queue(mq). MPI_Recv takes about the same params. At the moment here is what i do in Send and Recv:
//Sender part of code
ret=mq_send(mq,buf,sizeof(buf),0);
if(ret < 0)
return MPI_ERR_IO;
//Receiver part of code
ret = mq_receive(mq, buf, MSGSIZE, NULL );
if(ret < 0)
return MPI_ERR_IO;
Right now i'm only receiving the first element of the array. How would i go about sending the whole thing ? Here is what i have in mind
//Sender part of pseudocode
for(i=0,count)
element=(cast to datatype)buf[i];
mq_send(mq,element,sizeof,0);
//Receiver part of pseudocode
//i receive the count number of elements prior to this message
for(i=0,count)
mq_receive(mq,local_variable,etc...)
somehow store them into my void *buf which i receive as an argument ??
If something isn't clear enough , i'll reply
You specify the amount of data you are putting in the queue with the third argument of mq_send. In your case it's:
ret=mq_send(mq,buf,sizeof(buf),0);
Assuming that buf is initialized somewhere along the lines of
float f[2];
void *buf = f;
Then the expression sizeof(buf) means: the size of the pointer called "buf". While it may work on some architectures, the proper way would be
ret=mq_send(mq,buf,sizeof(float) * <number of elements>, 0);
which means, the size of a float, multiplied by the amount of floats stored in the array.
In which case you would put the whole array in the queue. You would also avoid the iteration and use only a constant number of messages, instead of a linear number.
Related
I have built a Winsock2 server. Part of that program has a function that receives data from clients. Originally the receive function I built would peek at the incoming data and determine if there was any additional data to be read before allowing recv() to pull the data from the buffer. This worked fine in the beginning of the project but I am now working on improving performance.
Here is a portion of the code I've written to eliminate the use of peek:
unsigned char recv_buffer[4096];
unsigned char *pComplete_buffer = malloc(sizeof(recv_buffer) * sizeof(unsigned char*));
int offset = 0;
int i = 0;
...
for (i; i <= sizeof(recv_buffer); i++) {
if (recv_buffer[i] == NULL) {
break;
}
pComplete_buffer[offset] = recv_buffer[i];
offset++;
}
...
This code would work great but the problem is that NULL == 0. If the client happens to send a 0 this loop will break prematurely. I thought I would be clever and leave the data uninitialized to 0xcc and use that to determine the end of recv_buffer but it seems that clients sometimes send that as part of their data as well.
Question:
Is there a character I can initialize recv_buffer to and reliably break on?
If not, is there another way I can eliminate the use of peek?
The correct solution is to keep track of how many bytes you store in recv_buffer to begin with. sizeof() gives you the TOTAL POSSIBLE size of the buffer, but it does not tell you HOW MANY bytes actually contain valid data.
recv() tells you how many bytes it returns to you. When you recv() data into recv_buffer, use that return value to increment a variable you define to indicate the number of valid bytes in recv_buffer.
For example:
unsigned char recv_buffer[4096];
int num_read, recv_buffer_size = 0;
const int max_cbuffer_size = sizeof(recv_buffer) * sizeof(unsigned char*);
unsigned char *pComplete_buffer = malloc(max_cbuffer_size);
...
num_read = recv(..., recv_buffer, sizeof(recv_buffer), ...);
if (num_read <= 0) {
// error handling...
return;
}
recv_buffer_size = num_read;
...
int available = max_cbuffer_size - offset;
int num_to_copy = min(recv_buffer_size, available);
memcpy(pComplete_buffer + offset, recv_buffer, num_to_copy);
offset += num_to_copy;
memmove(recv_buffer, recv_buffer + num_to_copy, recv_buffer_size - num_to_copy);
recv_buffer_size -= num_to_copy;
...
Is there a character I can initialize recv_buffer to and reliably break on?
Nope. If the other side can send any character at any time, you'll have to examine them.
If you know the sender will never send two NULs in a row (\0\0), you could check for that. But then some day the sender will decide to do that.
If you can change the message structure, I'd send the message length first (as a byte, network-ordered short or int depending on your protocol). Then, after parsing that length, the receiver will know exactly how long to keep reading.
Also if you're using select, that will block until there's something to read or the socket closes (mostly -- read the docs).
I'm trying to read a message that contains 2 strings. This message contains 2 strings that could be anything, and it is sent over a socket.
Note that I'm using C in Ubuntu environment.
The format of the message is, in a single void* buffer:
[string1]\0[string2]\0
I figured I'd be able to separate them once they arrive, using the '\0' to figure out where to split them. I'm using a function to read just the string and it sort of works, but I keep getting complaints from Valgrind, and I don't understand why.
I'm going to use an example where only 1 string is read from the buffer, but I mention the strategy because I'm not able to just put the message into a char* buffer. I need the function to extract a string from more complex buffers.
It all starts like this:
void* buffer = malloc(msgSize * sizeof(char)); //the message size is properly calculated to include the '\0' at the end
char* instanceId = malloc(msgSize * sizeof(char));
if(recv(socket_desc, (void*) buffer, msgSize * sizeof(char), MSG_WAITALL) <= 0) {
log_error(logger, "Message failed.");
return;
}
bufferToString(buffer, &instanceId, 0);
bufferToString2(buffer, instanceId, 0);
I made several attempts to make bufferToString work, as you can see... Of course I don't invoke them all at the same time, but I want to share those lines in case I'm making a mistake there.
Attempt #Number 1: char by char
int bufferToString(void* buffer, char** string, int startPtr) {
//startPtr can be used to read strings that are in the middle of a buffer
char a;
int thisStringPtr = 0;
do {
a = *(char*) (buffer + startPtr);
(*string)[thisStringPtr] = a;
startPtr++;
thisStringPtr++;
} while (a != '\0');
return startPtr; //return end position to use for extracting more values later
}
This one complains:
==23047== Invalid read of size 1
==23047== at 0x403A27A: bufferToString (buffer.c:16)
==23047== by 0x804A0C2: handleHiloInstancia (coordinador.c:232)
==23047== by 0x8049C54: procesarConexion (coordinador.c:85)
==23047== by 0x4066294: start_thread (pthread_create.c:333)
==23047== by 0x41650AD: clone (clone.S:114)
==23047== Address 0x423bc8a is 0 bytes after a block of size 10 alloc'd
==23047== at 0x402C17C: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==23047== by 0x804A063: handleHiloInstancia (coordinador.c:225)
==23047== by 0x8049C54: procesarConexion (coordinador.c:85)
==23047== by 0x4066294: start_thread (pthread_create.c:333)
==23047== by 0x41650AD: clone (clone.S:114)
Line 16 of bufferToString is the first line inside the do statement.
Attempt 2: cast and copy
int bufferToString2(void* buffer, char* string, int startPtr) {
strcpy(string, (char*) (buffer + startPtr));
return (strlen(string) + 1)*sizeof(char);
}
With or without the +startPtr, this causes slightly different problems:
==23190== Invalid read of size 1
==23190== at 0x402F489: strcpy (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==23190== by 0x403A1E3: bufferToString2 (buffer.c:3)
==23190== by 0x804A0C1: handleHiloInstancia (coordinador.c:232)
==23190== by 0x8049C54: procesarConexion (coordinador.c:85)
==23190== by 0x4066294: start_thread (pthread_create.c:333)
==23190== by 0x41650AD: clone (clone.S:114)
==23190== Address 0x423bc8a is 0 bytes after a block of size 10 alloc'd
I tried a few other combinations (like using char** string and all the required modifications in bufferToString2), but I keep getting similar error messages. What am I not seeing?
UPDATE: How message is being sent:
int bufferSize;
void* buffer = serializePackage(HANDSHAKE_INSTANCE_ID ,instancia_config->nombre, &bufferSize );
printf("Buffer size: %i - Instancia Name = %s - Socket num: %i\n", bufferSize, instancia_config->nombre, socket_coordinador); //this shows right data
if (send(socket_coordinador,buffer,bufferSize, 0) <= 0) {
log_error(logger, "Could not send ID.");
endProcess(EXIT_FAILURE);
}
instancia_config->nombre is of type char*
void* serializePackage(int codigo,char * mensaje, int* tamanioPaquete){
int puntero = 0;
int length = strlen(mensaje);
int sizeOfPaquete = strlen(mensaje) * sizeof(char) + 1 + 2 * sizeof(int);
void * paquete = malloc(sizeOfPaquete);
memcpy((paquete + puntero) ,&codigo,sizeof(int));
puntero += sizeof(int);
memcpy((paquete + puntero),&length,sizeof(int));
puntero += sizeof(int);
memcpy((paquete + puntero),mensaje,length * sizeof(char) + 1);
*tamanioPaquete = sizeOfPaquete;
return paquete;
}
Do you have your src and dst the right way around?
The destination (or target if you prefer) of memory services in C is the first parameter, so:
strcpy(string, buffer)
will copy buffer into string. (https://www.tutorialspoint.com/c_standard_library/c_function_strcpy.htm)
But: bufferToString2 is being called with buffer as the first parameter (and it's the source in this case).
In the first case, as was pointed out, you can't do arithmetic on a void* because the math is trying to go to the Nth element if you say:
*(x + N)
and if x is 'void' it has no size, and therefore the Nth element isn't meaningful.
Since there is not an answer yet that eventually solved the problem ostensibly, and no further progress on this question, I will summarize what I partially mentioned in my comments to at least provide hints where to start looking for the problem:
A lot of information is, unfortunately, missing to seriously treat this question.
For example, which kind of socket do you use to send / receive your messages?
Is it a pipe, or a network socket - which kind of transport do you use then?
How do you ensure that you received the entire message?
You should at least check the return value of recv (which happens to be the number of received octets) and check with the expected length if there is one.
When having received your message,have the message dumped as is - if you are not familiar with using a debugger, a function like this would do as a starter:
void dump(const char* buffer, size_t length) {
for(size_t i = 0; i < length; ++i) {
printf("%x", buffer[i] & 0xff);
}
printf("\n");
}
and invoke it in your code after the ssize_t received = recv(...) like dump(buffer, received).
Furthermore, you don't provide the way you calculate msgSize in your first code snippet - how do you guarantee that the string in mensaje you pass over to serializePacket is not longer than msgSize ?
Then, in serializePacket, you create a buffer filled like this:
| codigo (int) | length (int) | mensaje ( = terminal zero) |
but what you read back at the other end is just a c string - you ought to read the 2 ints as well, ought you not?
Then, there is another problem with this bit of serialization: Even if you read back the 2 ints, you would have a piece of code that is entirely unportable and works only if both sender and receiver run on an architecture that represent ints precisely the same way and in the same bit order. if you run the sender on a 32 bit system and the receiver on a 64 bit system, you write 4 byte integers, while attempt to read back 8 byte integers. Better use
types with precisely defined width (like uint32_t from inttypes.h ) and explicitly convert to/from network byte order using e.g. ntohl(3) / htonl(3).
Lastly, I want to add some remarks to improve the quality of code, that might prevent a lot of potential errors:
Take as example your bufferToString:
You passed in string as char** - why?
Use appropriate data types - don't use a signed int where you deal with non-negative sizes - use the standard type size_t instead
Don't use void* as type unless really necessary - you loose a hell lot of compile time error checking. C automatically converts from void* in assignments. So declaring buffer as char* is highly suggested.
This function, e.g. can harshly be simplified:
size_t bufferToString(const char* buffer, char* string, size_t offset) {
for(size_t i = offset; 0 != buffer[i]; ++i) {
string[i] = buffer[i];
}
return ++i;
}
I can only emphazise that network code tends to be hard to debug - there are all kinds of external error sources you cannot really oversee.
Hopefully these hints help to track down the root cause of your problem.
I have a problem that I'm having difficulty solving.
I have a union that contains a buffer with a struct mapping the bits of the buffer. Something along the lines of (it is pragma packed of course):
union
uint32 buf[512]
struct
uint8_t pad[256];
uint32_t data[256];
the buf[] part is intended to be passed to the Linux spi driver as a Receive buffer. The issue I'm having is, depending on my transmits, the size of the padding I receive back is variable, and because of this it isn't straight forward to access using the union.
What i need to do is to be able to pass buf[] at a specific index to the spi driver, I.E the Rx buffer begins at buf[128] instead of buf[0]. This isn't always equal, so i have an equation that tells me where i need the start point to be which is &(buf[0]+padmax-padsize]) which should result in a value between buf[0] and buf[256]. However, the issue is the spi driver expects the argument of the transfer buffer to contain a pointer to a buffer, and passing it the straight address isn't giving me what i want.
I have also tried assigning a pointer to the address of the above equation and passing that to the rxbuffer part of the spi struct and it again doesn't give me what i want.
Is it possible to create an array that is a subset of another array, starting at a specified address of the outer array? I think this may solve my problem but I'm also afraid of the memory implications of that
The reason is most likely that you're calculating the address in 32-bit units (in units of buf elements), not bytes as you expect, based on the arithmetic.
Let's simplify the situation, and say the structure is just
#define MAX_PAD 256
#define MAX_DATA 256
struct spi_data {
uint8_t pad[MAX_PAD];
uint32_t data[MAX_DATA];
};
and that you want to implement a function similar to
size_t spi_recv(int fd, struct spi_data *ref, size_t pad, size_t data)
where
fd is the file descriptor to read() from
ref is a pointer to the struct spi_data to be used
pad is the number of padding entries filled at the end of the ref->pad[] array
data is the number of entries filled at the beginning of the ref->data[] array
the return value is the number of data entries received (completely filled)
Consider the following (argument checks for fd == -1, ref == NULL, pad > MAX_PAD, data > MAX_DATA omitted for simplicity):
size_t spi_recv(int fd, struct spi_data *ref, size_t pad, size_t data)
{
ssize_t n;
n = read(fd, &ref->pad[sizeof ref->pad / sizeof ref->pad[0] - pad],
pad * sizeof ref->pad[0] + data * sizeof ref->data[0]);
if (n == -1) {
/* Error; errno already set */
return 0;
} else
if (n < 0) {
/* Should never occur, but let's be paranoid */
errno = EIO;
return 0;
} else
if (n < pad * sizeof ref->pad[0]) {
/* Only partial padding received */
errno = 0;
return 0;
} else {
/* Zero or more data words received */
errno = 0;
return (n - pad * sizeof ref->pad[0]) / sizeof ref->data[0];
}
}
The pointer to the last pad elements of padding is
&ref->pad[sizeof ref->pad / sizeof ref->pad[0] - pad])
which is essentially equivalent to &(ref->pad[MAX_PAD - pad]), except that instead of the MAX_PAD macro, we use (sizeof ref->pad)/(sizeof ref->pad[0]) to evaluate the number of members declared for the ref->pad[] array. (This only works if ref->pad is an array; it does not work if it is a pointer.)
As usual, read() takes the number of bytes -- not elements of ref->pad or ref->data -- as a parameter, so we need to multiply the element counts by their respective element sizes in bytes; thus, the number of bytes in pad elements of padding and data elements of data is pad * sizeof ref->pad[0] + data * sizeof ref->data[0].
Since the function returns the number of complete data words received, the number of padding bytes must be subtracted from the return value, then divided by the data element type (integer division rounding down), to get the number of complete data words.
I don't think the above interface is optimal, however. I particularly dislike the possibility of the SPI transfer ending with a partial word; the above interface does not let the caller detect such a situation reliably.
If you use spidev, the ioctl() interface would be much better to use. For one, you could use SPI_IOC_MESSAGE(2) to read the padding and the data into separate buffers, or even SPI_IOC_MESSAGE(3) to write a command, followed by a read to the padding buffer and another to the data buffer. The Linux-Sunxi Wiki page has a pretty simple example of this kind of usage here, except that it uses single reads instead of reading padding into a separate buffer. However, it should be quite simple to extend the examples to do that.
I am trying to send a char array to a server and I am not able to send the whole thing. I discovered that I am not receiving that it is sending only the first four packets.
This is the data I want to send in one go.
char msg1[]={0xff,0xfd,0x03,0xff,0xfb,0x18,0xff,0xfb,0x1f,0xff,0xfb,0x20,0xff,0xfb,0x21,0xff,0xfb,0x22,0xff,0xfb,0x27,0xff,0xfd,0x05,0xff,0xfb,0x23};
This is the function I am using to check whether all the bytes have been send or not. and if not. send the rest.
int send_all(int s,char *buf,int *len)
{
int total=0; //total bytes sent
int byteslft= *len; //bytes left to be sent
int n; //value returned by send()
while(total<*len)
{
n=send(s,(buf+total),byteslft,0);
if(n== -1)
break;
total +=n; //total bytes sent will be in total
byteslft -=n; //total bytes left to be sent is stored here
}
*len= total;
return (n==-1)?-1:0;
}
And this how I am trying to send it
len=sizeof (*(arr+i)) ;
if((numbytes=send_all(sockfd,(*(arr+i)),&len))==-1)
{
perror("send");
continue;
}
I have done the connect(),socket(), etc. steps before this just to be clear.
And this is how I've initialized the array of the pointers to the msg array.
char *arr[]={msg1,msg2,msg3,msg4....};
When I debug my program, I see that it is only sending
ff fd 03 ff //message stops sending here
And that's why I'm not getting the required response from the server.
The expression *(arr + i) is not an actual array, but a pointer to an array. This means that the sizeof operation will return the size of the pointer (which is 32 bits, i.e. four bytes on a 32-bit operating system) and not what it points to.
You can only use sizeof to get the size of an array if you have the actual array, not on pointers.
While receiving the integer array, checking the bytes of data received is needed.
For example, when receiving an integer array with length 100:
int count = 0;
int msg[100];
while(count < 100 * sizeof(int)){
count += read(fd, msg + count / sizeof(int), 100 * sizeof(int) - count);
}
Is this a right way? Will read() return a value which is not a multiple of sizeof(int)?
If this is not correct, what the right way to receive an integer array?
On Linux you can use the MSG_WAITALL option for recv(), which makes the function wait for the full given length of incoming data.
In alternative (working on all platforms) you can also write a generic receive function that receives a given amount of bytes, like this one (which assumes the socket is not set as non-blocking; requires including <stdint.h>):
/// \brief Receives a block of data of the specified size
/// \param sk Socket for incoming data
/// \param data Pointer to input buffer
/// \param len Number of bytes to read
/// \return Number of bytes received (same as len) on success, -1 on failure
int block_recv(const int sk, uint8_t* data, unsigned int len)
{
int i, j = 0;
while (len > 0) {
i = recv(sk, (char*) data, len, 0);
if (i <= 0) {
return -1;
}
data += i;
len -= i;
j += i;
}
return j;
}
Then you can just call it to receive your integer buffer:
if (block_recv(fd, (uint8_t*) msg, sizeof(msg)) != sizeof(msg)) {
fprintf(stderr, "Error receiving integer buffer...\n");
// whatever error handling you need...
}
You are correct that read may not return all the data you requested, esp. if it is connected to a network socket. read will not necessarily return a value with multiple of sizeof(int). If you want to use this (manual) method of receiving data, I would probably recommend you count bytes instead of sizeof(int)s (which can be 4 or 8 depending on your system). Even easier than doing this is to use something like Protocol Buffers, which lets you define a data format for your packets and serialize/deserialize them quickly and easily. (Define a message that simply includes your integer array and let protobuf take care of everything else.)
You're right - there's no guarantee that read will return data whose size is a multiple of sizeof(int).
The minimum size you may receive is a char.
There are other issues such as endianness when receiving integers from across a network (which obviously also apply to sending integers across a network) that you should be aware of.
For these reasons, an easier solution is to use a char[] instead of int[] to store the message, and then copy it to an int[]. If you are concerned about efficiency, prove to yourself that this is a bottleneck (profile your code) before you worry about optimizing code.
Also, if you are sending and recving across a network, be aware that protocols like TCP are stream-based, i.e. they simply send streams of characters and you need to implement some way of detecting the end of a message and formatting it to your needs. Two common ways are to either send the length of the message as a header or to use a special character like '\n' to signal the end of the message. Also, since you are sending an array, you could use something like '|' to separate elements.
So a sample message can be: "1|100|239|23|\n"