In my code, the client will initiate a request to server, upon so, server will keep sending data to client until a special output is presented (say "END") then client will stop reading.
Here's some code (just an example):
/**Client**/
char req[] = "some-request";
write(socket,req,strlen(req));
while(1)
{
read(socket,readline,100);
strtok(readline, "\n");
if(strcmp(readline,"END") == 0){ //if end of sending
break;
}
/** HANDLE OUTPUT **/
}
/** Server **/
int data[] = {5532,127,332,65,723,8,912,2,421,126,8,3,2}
int i;
for(i = 0 ; i < sizeof(data); i++)
{
write(socket, data[i], sizeof(data[i]);
}
write(socket, "END", 3);
This code works fine, but due some context switching in processes, the server writes twice before the client reads, so the client reads two lines at once, say:
5532127
332
65723
8912
2421
1268
32END
As seen, sometimes 2 or more writes are grouped into 1 read, and as a result of course the END isn't handled as it is combined..
I tried using usleep() function, which works most of the time, but still isn't a solution.
So any idea how I make sure the server doesn't write to buffer if it is not empty?
You are using a stream oriented socket. The server can't know if client has already received the data or not, and there's not really any difference between your multiple writes in a loop, or just one write which sends the whole array.
You will need to use something packet oriented, or have a way to figure out what are your "packets". As you've figured out, you can't rely on timings, so you'll need to define some sort of protocol. Something simple would be to just read the ints*, and have a special value as end.
Btw. your code also has a couple of issue with number representation. You need to used a fixed length type (int can be 16 or 32-bit, so use something like uint32_t), and you need to be aware of endianess issues (should probably use htonl/ntohl or similar).
I can see some problems in your code.
You use a strtok(readline, "\n"); client side while sending raw int server side. It means that in one of the sent integers is 0x..10 or 0x10.. it will be changed to (resp.) 0x..00 or 0x00... Never use strxx functions on raw bytes !
Next, you are using stream oriented sockets. You can have no assurance that packets send will not be grouped or splitted. So you should collate everything in the receiver until you have 3 consecutive chars containing END. But you cannot use strxx function for that ...
Finally, you transfer raw integer, and wait for 3 characters END. But if you try to send {12, 17743, 17408, 13} (on a big-endian system) you will have a major problem : 17743 = 0x454F, and 17408 = 0x4400 => you have exactly "END" ! On a little endian, the magic values would be 20293 = 0x4F45 and 68 = 0x0044.
The rules are :
in you want to transer binary data, you should use a lenght + packet pattern because any pattern can happen in raw data. And a good practice is to convert all data longer than one byte to network order (htonx functions) to avoid endianness problems
if you want to use delimiters, you should only use character data, ensuring (via escaping or ...) that the end pattern cannot occur in data.
In you use case, you should simply tranfer textual representation, converting integer using sprintf, separating them with space, end terminating with \n.
Related
I need to send data from the PC to my STM32F3, so I decided to use a built-in USB in uC.
But now I have a problem - I want to send to stm32 big amount of data at once - I mean something like 200-500 Bytes.
When I send from PC with minicom packets which have less than 64 chart - everything is fine - callback CDC_Receive_FS(uint8_t* Buf, uint32_t *Len) occurs once - it enables UsbRxFlag, just to inform the running program that there is data available.
static int8_t CDC_Receive_FS(uint8_t* Buf, uint32_t *Len)
{
/* USER CODE BEGIN 6 */
USBD_CDC_SetRxBuffer(&hUsbDeviceFS, &Buf[0]);
USBD_CDC_ReceivePacket(&hUsbDeviceFS);
if( (Buf[0] == 'A') & (Buf[1] == 'T') ){
GPIOB->BSRR = (uint32_t)RX_Led_Pin;
UsbRxFlag = 1;
}
return (USBD_OK);
/* USER CODE END 6 */
}
But when I try to send more data (just long text from minicom ) to uC, something weird happens - sometimes uC doesn't react at all - sometimes it doesn't take into account some data.
How can I handle sending to STM32F3 more than 64Bytes over USB-CDC?
The maximum packet length for full-speed USB communication is 64 bytes. So the data will be transferred in chunks of 64 bytes and needs to be reassembled on the other end.
USB CDC is based on bulk transfer endpoints and implements a data stream (also known as pipe), not a message stream. It's basically a stream of bytes. So if you send 200 bytes, do not expect any indication of where the 200 bytes end. Such information is not transmitted.
Your code looks a bit suspicious:
You probably meant '&&' instead of '&' as pointed out by Reinstate Monica.
Unless you change buffers, USBD_CDC_SetRxBuffer only needs to be called once at initialization.
When CDC_Receive_FS is called, a data packet has already been received. Buf will point to the buffer you have specified with USBD_CDC_SetRxBuffer. Len provides the length of the packet. So the first thing you would do is process the received data. Once the data has been processed and the buffer can be reused again, you would call USBD_CDC_ReceivePacket to indicate that you are ready to receive the next packet. So move USBD_CDC_SetRxBuffer to another function (unless you want to use several buffers) and move USBD_CDC_ReceivePacket to the end of CDC_Receive_FS.
The incorrect order of the function calls could likely have led to the received data being overwritten while you are still processing it.
But the biggest issue is likely that you expect that the entire data is received in a single piece if you sent is as a single piece, or that it at least contains an indication of the end of the piece. That's not the case. You will have to implement this yourself.
If you are using a text protocol, you could buffer all incoming data until you detect a line feed. Then you know that you have a complete command and can execute it.
The following is a general purpose implementation for reading an arbitrary number of bytes: https://github.com/philrawlings/bluepill-usb-cdc-test.
The full code is a little too long to post here, but this essentially modifies usb_cdc_if.c to create a circular buffer and exposes additional functions (CDC_GetRxBufferBytesAvailable_FS(), CDC_ReadRxBuffer_FS() and CDC_FlushRxBuffer_FS()) which can be consumed from main.c. The readme.md text shown on the main page describes all the code changes required.
As mentioned by #Codo, you will need to either add termination characters to your source data, or include a "length" value (which itself would be a fixed number of bytes) at the beginning to then indicate how many bytes are in the data payload.
I'd like to write little file transfer program in C (I'm working on Linux).
I'm quite new to sockets programming, but I already managed to write little server and
client programs.
My question:
If I have something like this:
int read_bytes = read(socket_id, buffer, 4096);
While reading from the socket, how can I get the progress of reading?
For example I want to display every 100ms how many bytes have been transferred
so far. I'm pretty sure I have to use threads or other async functions here.
Is there a function so I can get the number of bytes read and the number of bytes to read?
Update 1:
Accumulating read()
int total_read_bytes=0, read_bytes;
char buffer[4096];
do {
read_bytes = read(socket_id, buffer, 4096);
total_read_bytes += read_bytes;
} while(read_bytes > 0); /* Read to end or read failed */
Isn't this very, very inefficient? (For example to transfer a 1MiB file)
Thank you
If you have control to the code of both the server and the client, you can let the sender to tell the receiver the size of the file before actually sending the file. For example, using the first 8 bytes in the message. That's the number of bytes to read.
By accumulating the read_bytes in your example, you can get number of bytes read
Each recv call will block until it's read some more, but it won't (by default / sans MSG_WAITALL) wait for the supplied buffer to be full. That means there's no need to reduce the buffer size in an effort to get more frequent updates to the total bytes read information, you can trivially keep a total of recv() return values that updates as packets arrive (or as fast as your app can process them from the OS buffers).
As you observe, if that total is being updated 100 times a second, you probably don't want to issue 100 screen updates, preferring to limit them to your 100ms min interval. You could use a timer/alarm to manage that, but then you have more edge cases to see if the timer is already pending etc.. It's probably simpler just to use two threads where one checks periodically if an update is needed:
thread1:
while ((n_bytes = recv(...)) > 0)
total_bytes += n_bytes;
set screen-updater-termination flag
thread2:
last_total_bytes = -1;
while the screen-updater-termination flag's not set
sleep 100ms
if last_total_bytes != total_bytes
update screen
last_total_bytes = total_bytes
Of course you'll need to use a mutex or atomic operations to coordinate some of these actions.
Is there a function so I can get... the number of bytes to read?
Not from TCP itself - it doesn't have a notion of message sizes, and the API just ensures an app receives the bytes in the same order they were sent. If you want to be able to display a "N bytes received so far, X more expected", then at the application protocol level, the sending side should prefix the data in a logic message with the size - commonly in either a fixed width binary format (ideally using htonl or similar to send and ntohl on the recv side to avoid endian issues), or in a textual representation that's either fixed with or separated from the data by a known sentinel character (e.g. perhaps a NUL, a space or newline).
I am working on a client-server project and need to implement a logic where I need to check whether I have received the last data over a TCP socket connection, before I proceed.
To make sure that I have received all the data , I am planning to pad a flag to the last packet sent.I had two options in mind as below and also related prob.
i. Use a struct as below and populate the vst_pad for the last packet sent and check the same on the recv side for its presence. The advantage over option two is that, I dont have to remove the flag from actual data before writing it to a file.Just check the first member of the struct
typedef struct
{
/* String holding padding for last packet when socket is changed */
char vst_pad[10];
/* Pointer to data being transmitted */
char *vst_data;
//unsigned char vst_data[1];
} st_packetData;
The problem is I have to serialize the struct on every send call. Also I am not sure whether I will receive the entire struct over TCP in one recv call and so have to add logic/overhead to check this every time. I have implemented this so far but figured it later that stream based TCP may not guarantee to recv entire struct in one call.
ii. Use function like strncat to add that flag at the end to the last data being sent.
The prob is I have to check on every receive call either using regex functions or function like strstr for the presence of that flag and if so have to remove it from the data.
This application is going to be used for large data transfers and hence want to add minimal overhead on every send/recv/read/write call. Would really appreciate to know if there is a better option then the above two or any other option to check the receipt of last packet. The program is multithreaded.
Edit: I do not know the total size of file I am going to send, but I am sending fixed amount of data. That is fgets read until the size specified -1 or until a new line is encountered.
Do you know the size of the data in advance, and is it a requirement that you implement a end of message flag?
Because I would simplify the design, add a 4-byte header (assuming you're not sending more than 4gb of data per message), that contains the expected size of the message.
Thus you parse out the first 4 bytes, calculate the size, then continue calling recv until you get that much data.
You'll need to handle the case where your recv call gets data from the next message, and obviously error handling.
Another issue not raised with your 10byte pad solution is what happens if the actual message contains 10 zero bytes--assuming you're padding it with zeros? You'd need to escape the 10bytes of zeros otherwise you may mistakenly truncate the message.
Using a fixed sized header and a known size value will alleviate this problem.
For a message (data packet) first send a short (in network order) of the size, followed by the data. This can be achieved in one write system call.
On the reception end, just read the short and convert back into host order (this will enable one to use different processors at a later state. You can then read the rest of the data.
In such cases, it's common to block up the data into chunks and provide a chunk header as well as a trailer. The header contains the length of the data in the chunk and so the peer knows when the trailer is expected - all it has to do is count rx bytes and then check for a valid trailer. The chunks allow large data transfers without huge buffers at both ends.
It's no great hassle to add a 'status' byte in the header that can identify the last chunk.
An alternative is to open another data connection, stream the entire serialization and then close this data connection, (like FTP does).
Could you make use of an open source network communication library written in C#? If so checkout networkComms.net.
If this is truely the last data sent by your application, use shutdown(socket, SHUT_WR); on the sender side.
This will set the FIN TCP flag, which signals that the sender->receiver stream is over. The receiver will know this because his recv() will return 0 (just like an EOF condition) when everything has been received. The receiver can still send data afterward, and the sender can still listen for them, but it cannot send more using this connection.
Our application is a C server (this problem is for the Windows port of said server) that communicates with a Windows Java client. In this particular instance we are sending data to the client, particularly, the message consists of a 7 byte header where the first 3 bytes all have a specific meaning (op type, flags, etc) and the last 4 bytes contain the size of the rest of the message. For some reason I absolutely can't figure out, the third byte in the header is somehow changing; if I put a break point on the send() I can see that the third byte is what I'm expecting (0xfe), but when I check in the client, that byte is set to 0. Every other byte is fine. A did some traffic capturing with WireShark and saw that the byte was 0 leaving the server, which I find even more baffling. The third byte is set via a define, ala:
#define GET_TOP_FRAME 0xfe
Some testing I did that further confuses the issue:
I changed the value from using the define to first 0x64, 0xff, 0xfd: all came across to the client.
I changed the value from using the define to using 0xfe itself: the value was zero at the client.
I changed the value of the define itself from 0xfe to 0xef: the value was zero at the client.
Nothing about this makes a lick of sense. The code goes through several levels of functions, but here is most of the core code:
int nbytes; /* network order bytes */
static int sendsize = 7;
unsigned char tbuffer[7];
tbuffer[0]= protocolByte;
tbuffer[1]= op;
tbuffer[2]= GET_TOP_FRAME;
nbytes = htonl(bytes);
memcpy((tbuffer+3), &nbytes, JAVA_INT);
send(fd, tbuffer, sendsize, 0);
Where fd is a previously put together socket, protocolByte, op, and bytes are previously set. It then sends the rest of the message with a very similar send command immediately after this one. As I mentioned, if I put a break point on that send function, the tbuffer contains exactly what I expect.
Anybody have any ideas here? I'm completely stumped; nothing about this makes sense to me. Thanks.
It might be that there's a simple bug somewhere in your system doing a simple buffer overflow or similar, but it's hard to tell where, given this little information. However, keep in mind:
TCP doesn't send messages - it's a stream. One send() call might take several recv() calls to receive. one recv call might receive partial "messages" your application have defined.
Are you checking the return value of send ? And the return values of recv ? send might send less bytes than you tell it to. recv might receive less bytes than you tell it to, or it might receive more data than one of your application "messages" if you've given it a large enough buffer.
Turns out something else was getting in the way: in addition to the C server, we have a Tomcat/Java server that sort of runs on top of everything; I'll be honest, I'm not involved in that piece of development so I don't understand it very well. As it turns out, we have some code that is shared between that middletier portion and our client; this shared code was taking that particular byte as it was leaving the server and, if it was set to 0xfe was setting it to an uninitialized value (hence the zero). Then, when it got to the client, it was wrong. That's why when I set the values to different things in the C code it was making it to the other side and also why the behavior seemed inconsistent; I was toggling values in the Java quite a bit to see what was going on.
Thanks for the pointers all, but as it turns out it was merely a case of me not understanding the path properly.
First of all, I've never worked with C before (mostly Java which is the reason you'll find me write some naive C code). I am writing a simple command interpreter in C. I have something like this:
//Initialization code
if (select(fdmax+1, &read_fds, NULL, NULL, NULL) == -1) {
perror("Select dead");
exit(EXIT_FAILURE);
}
....
....
//Loop through connections to see who has the data ready
//If the data is ready
if ((nbytes = recv(i, buf, sizeof(buf), 0)) > 0) {
//Do something with the message in the buffer
}
Now if I'm looking at something like a long paragraph of commands, it is obvious that a 256 byte buffer will not be able to get the entire command. For the time being, I'm using a 2056 byte buffer to get the entire command. But if I want to use the 256 byte buffer, how would I go about doing this? Do I keep track of which client gave me what data and append it to some buffer? I mean, use something like two dimensional arrays and such?
Yes, the usual approach is to have a buffer of "data I've received but not processed" for each client, large enough to hold the biggest protocol message.
You read into that buffer (always keeping track of how much data is currently in the buffer), and after each read, check to see if you have a complete message (or message(s), since you might get two at once!). If you do, you process the message, remove it from the buffer and shift any remaining data up to the start of the buffer.
Something roughly along the lines of:
for (i = 0; i < nclients; i++)
{
if (!FD_ISSET(client[i].fd, &read_fds))
continue;
nbytes = recv(client[i].fd, client[i].buf + client[i].bytes, sizeof(client[i].buf) - client[i].bytes, 0);
if (nbytes > 0)
{
client[i].bytes += nbytes;
while (check_for_message(client[i]))
{
size_t message_len;
message_len = process_message(client[i]);
client[i].bytes -= message_len;
memmove(client[i].buf, client[i].buf + message_len, client[i].bytes);
}
}
else
/* Handle client close or error */
}
By the way, you should check for errno == EINTR if select() returns -1, and just loop around again - that's not a fatal error.
I would keep a structure around for each client. Each structure contains a pointer to a buffer where the command is read in. Maybe you free the buffers when they're not used, or maybe you keep them around. The structure could also contain the client's fd in it as well. Then you just need one array (or list) of clients which you loop over.
The other reason you'd want to do this, besides the fact that 256 bytes might not be enough, is that recv doesn't always fill the buffer. Some of the data might still in transit over the network.
If you keep around buffers for each client, however, you can run into the "slowloris" attack, where a single client keeps sending little bits of data and takes up all your memory.
It can be a serious pain when you get tons of data like that over a network. There is a constant trade between allocating a huge array or multiple reads with data moves. You should consider getting a ready made linked list of buffers, then traverse the linked list as you read the buffers in each node of the linked list. That way it scales gracefully and you can quickly delete what you've processed. I think that's the best approach and it's also how boost asio implements buffered reads.
If you're dealing with multiple clients a common approach to to fork/exec for each connection. Your server would listen for incoming connections, and when one is made it would fork and and exec a child version of itself that would then handle the "command interpreter" portion of the problem.
This way you're letting the OS manage the client processes--that is, you don't have to have a data structure in your program to manage them. You will still need to clean up child processes in your server as they terminate.
As for managing the buffer...How much data do you expect before you post a response? You may need to be prepared to dynamically adjust the size of your buffer.