I need to send data from the PC to my STM32F3, so I decided to use a built-in USB in uC.
But now I have a problem - I want to send to stm32 big amount of data at once - I mean something like 200-500 Bytes.
When I send from PC with minicom packets which have less than 64 chart - everything is fine - callback CDC_Receive_FS(uint8_t* Buf, uint32_t *Len) occurs once - it enables UsbRxFlag, just to inform the running program that there is data available.
static int8_t CDC_Receive_FS(uint8_t* Buf, uint32_t *Len)
{
/* USER CODE BEGIN 6 */
USBD_CDC_SetRxBuffer(&hUsbDeviceFS, &Buf[0]);
USBD_CDC_ReceivePacket(&hUsbDeviceFS);
if( (Buf[0] == 'A') & (Buf[1] == 'T') ){
GPIOB->BSRR = (uint32_t)RX_Led_Pin;
UsbRxFlag = 1;
}
return (USBD_OK);
/* USER CODE END 6 */
}
But when I try to send more data (just long text from minicom ) to uC, something weird happens - sometimes uC doesn't react at all - sometimes it doesn't take into account some data.
How can I handle sending to STM32F3 more than 64Bytes over USB-CDC?
The maximum packet length for full-speed USB communication is 64 bytes. So the data will be transferred in chunks of 64 bytes and needs to be reassembled on the other end.
USB CDC is based on bulk transfer endpoints and implements a data stream (also known as pipe), not a message stream. It's basically a stream of bytes. So if you send 200 bytes, do not expect any indication of where the 200 bytes end. Such information is not transmitted.
Your code looks a bit suspicious:
You probably meant '&&' instead of '&' as pointed out by Reinstate Monica.
Unless you change buffers, USBD_CDC_SetRxBuffer only needs to be called once at initialization.
When CDC_Receive_FS is called, a data packet has already been received. Buf will point to the buffer you have specified with USBD_CDC_SetRxBuffer. Len provides the length of the packet. So the first thing you would do is process the received data. Once the data has been processed and the buffer can be reused again, you would call USBD_CDC_ReceivePacket to indicate that you are ready to receive the next packet. So move USBD_CDC_SetRxBuffer to another function (unless you want to use several buffers) and move USBD_CDC_ReceivePacket to the end of CDC_Receive_FS.
The incorrect order of the function calls could likely have led to the received data being overwritten while you are still processing it.
But the biggest issue is likely that you expect that the entire data is received in a single piece if you sent is as a single piece, or that it at least contains an indication of the end of the piece. That's not the case. You will have to implement this yourself.
If you are using a text protocol, you could buffer all incoming data until you detect a line feed. Then you know that you have a complete command and can execute it.
The following is a general purpose implementation for reading an arbitrary number of bytes: https://github.com/philrawlings/bluepill-usb-cdc-test.
The full code is a little too long to post here, but this essentially modifies usb_cdc_if.c to create a circular buffer and exposes additional functions (CDC_GetRxBufferBytesAvailable_FS(), CDC_ReadRxBuffer_FS() and CDC_FlushRxBuffer_FS()) which can be consumed from main.c. The readme.md text shown on the main page describes all the code changes required.
As mentioned by #Codo, you will need to either add termination characters to your source data, or include a "length" value (which itself would be a fixed number of bytes) at the beginning to then indicate how many bytes are in the data payload.
Related
I am getting from a sensor via UART communication every second 10 bytes. But I don't need all bytes, actually only certain bytes to work with. Now what I do is to save all bytes into an array and create two new uint8_t's and assign them the byte from the buffer array I need.
Is there a way to only receive and save the bytes I need in the first place instead of all 10?
uint8_t buffer[10];
HAL_UART_Receive_DMA(&huart4, (uint8_t*)buffer, 10)
void HAL_UART_RxCpltCallback(UART_HandleTypeDef *huart4)
{
uint8_t value1 = buffer[4];
uint8_t value2 = buffer[5];
.
.
.
}
In DMA mode you need to provide the full size buffer. There is no other way as reception is not controlled by the core (it is done ion the background) and the DMA controller signals end (ad if you want half and error conditions) of the transaction only
It is possible with just raw interrupt handling, without any DMA and (mostly) without fancy HAL functions.
You'll have to write manual UART interrupt handler for RXNE flag, which is set when UART receives signle character. Read it from DR register and decide - save or discard it. Of cource it's up to you now, to count all received bytes and detect "end of message" condition.
OR
If your code have nothing to do during receiving this message, use
HAL_UART_Receive
To read message byte-by-byte
I'd like to write little file transfer program in C (I'm working on Linux).
I'm quite new to sockets programming, but I already managed to write little server and
client programs.
My question:
If I have something like this:
int read_bytes = read(socket_id, buffer, 4096);
While reading from the socket, how can I get the progress of reading?
For example I want to display every 100ms how many bytes have been transferred
so far. I'm pretty sure I have to use threads or other async functions here.
Is there a function so I can get the number of bytes read and the number of bytes to read?
Update 1:
Accumulating read()
int total_read_bytes=0, read_bytes;
char buffer[4096];
do {
read_bytes = read(socket_id, buffer, 4096);
total_read_bytes += read_bytes;
} while(read_bytes > 0); /* Read to end or read failed */
Isn't this very, very inefficient? (For example to transfer a 1MiB file)
Thank you
If you have control to the code of both the server and the client, you can let the sender to tell the receiver the size of the file before actually sending the file. For example, using the first 8 bytes in the message. That's the number of bytes to read.
By accumulating the read_bytes in your example, you can get number of bytes read
Each recv call will block until it's read some more, but it won't (by default / sans MSG_WAITALL) wait for the supplied buffer to be full. That means there's no need to reduce the buffer size in an effort to get more frequent updates to the total bytes read information, you can trivially keep a total of recv() return values that updates as packets arrive (or as fast as your app can process them from the OS buffers).
As you observe, if that total is being updated 100 times a second, you probably don't want to issue 100 screen updates, preferring to limit them to your 100ms min interval. You could use a timer/alarm to manage that, but then you have more edge cases to see if the timer is already pending etc.. It's probably simpler just to use two threads where one checks periodically if an update is needed:
thread1:
while ((n_bytes = recv(...)) > 0)
total_bytes += n_bytes;
set screen-updater-termination flag
thread2:
last_total_bytes = -1;
while the screen-updater-termination flag's not set
sleep 100ms
if last_total_bytes != total_bytes
update screen
last_total_bytes = total_bytes
Of course you'll need to use a mutex or atomic operations to coordinate some of these actions.
Is there a function so I can get... the number of bytes to read?
Not from TCP itself - it doesn't have a notion of message sizes, and the API just ensures an app receives the bytes in the same order they were sent. If you want to be able to display a "N bytes received so far, X more expected", then at the application protocol level, the sending side should prefix the data in a logic message with the size - commonly in either a fixed width binary format (ideally using htonl or similar to send and ntohl on the recv side to avoid endian issues), or in a textual representation that's either fixed with or separated from the data by a known sentinel character (e.g. perhaps a NUL, a space or newline).
I'm trying to read and write a serial port in Linux (Ubuntu 12.04) where a microcontroller on the other end blasts 1 or 3 bytes whenever it finishes a certain task. I'm able to successfully read and write to the device, but the problem is my reads are a little 'dangerous' right now:
do
{
nbytes = read(fd, buffer, sizeof(buffer));
usleep(50000);
} while(nbytes == -1);
I.e. to simply monitor what the device is sending me, I poll the buffer every half second. If it's empty, it idles in this loop. If it receives something or errors, it kicks out. Some logic then processes the 1 or 3 packets and prints it to a terminal. A half second is usually a long enough window for something to fully appear in the buffer, but quick enough for a human who will eventually see it to not think it's slow.
'Usually' is the keyword. If I read the buffer in the middle of it blasting 3 bytes. I'll get a bad read; the buffer will have either 1 or 2 bytes in it and it'll get rejected in the packet processing (If I catch the first of a 3 byte packet, it won't be a purposefully-sent-one-byte value).
Solutions I've considered/tried:
I've thought of simply reading in one byte at a time and feeding in additional bytes if its part of a 3 byte transmission. However this creates some ugly loops (as read() only returns the number of bytes of only the most previous read) that I'd like to avoid if I can
I've tried to read 0 bytes (eg nbytes = read(fd, buffer, 0);) just to see how many bytes are currently in the buffer before I try to load it into my own buffer, but as I suspected it just returns 0.
It seems like a lot of my problems would be easily solved if I could peek into the contents of the port buffer before I load it into a buffer of my own. But read() is destructive up to the amount of bytes that you tell it to read.
How can I read from this buffer such that I don't do it in the middle of receiving a transmission, but do it fast enough to not appear slow to a user? My serial messenger is divided into a sender and receiver thread, so I don't have to worry about my program loop blocking somewhere and neglecting the other half.
Thanks for any help.
Fix your packet processing. I always end up using a state machine for instances like this, so that if I get a partial message, I remember (stateful) where I left off processing and can resume when the rest of the packet arrives.
Typically I have to verify a checksum at the end of the packet, before proceeding with other processing, so "where I left off processing" is always "waiting for checksum". But I store the partial packet, to be used when more data arrives.
Even though you can't peek into the driver buffer, you can load all those bytes into your own buffer (in C++ a deque is a good choice) and peek into that all you want.
You need to know how large the messages being sent are. There are a couple of ways to do that:
Prefix the message with the length of the message.
Have a message-terminator, a byte (or sequence of bytes) that can not be part of a message.
Use the "command" to calculate the length, i.e. when you read a command-byte you know how much data should follow, so read that amount.
The second method is best for cases when you can come out of sync, because then read until you get the message-terminator sequence and you're sure that the next bytes will be a new message.
You can of course combine these methods.
To poll a device, you should better use a multiplexing syscall like poll(2) which succeeds when some data is available for reading from that device. Notice that poll is multiplexing: you can poll several file descriptors at once, and poll will succeed as soon as one (any) file descriptor is readable with POLLIN (or writable, if so asked with POLLOUT, etc...).
Once poll succeeded for some fd which you POLLIN you can read(2) from that fd
Of course, you need to know the conventions used by the hardware device about its messages. Notice that a single read could get several messages, or only a part of one (or more). There is no way to prevent reading of partial messages (or "packets") - probably because your PC serial I/O is much faster than the serial I/O inside your microcontroller. You should bear with that, by knowing the conventions defining the messages (and if you can change the software inside the microcontroller, define an easy convention for that) and implementing the appropriate state machine and buffering, etc...
NB: There is also the older select(2) syscall for multiplexing, which has limitations related to the C10K problem. I recommend poll instead of select in new code.
I am working on a client-server project and need to implement a logic where I need to check whether I have received the last data over a TCP socket connection, before I proceed.
To make sure that I have received all the data , I am planning to pad a flag to the last packet sent.I had two options in mind as below and also related prob.
i. Use a struct as below and populate the vst_pad for the last packet sent and check the same on the recv side for its presence. The advantage over option two is that, I dont have to remove the flag from actual data before writing it to a file.Just check the first member of the struct
typedef struct
{
/* String holding padding for last packet when socket is changed */
char vst_pad[10];
/* Pointer to data being transmitted */
char *vst_data;
//unsigned char vst_data[1];
} st_packetData;
The problem is I have to serialize the struct on every send call. Also I am not sure whether I will receive the entire struct over TCP in one recv call and so have to add logic/overhead to check this every time. I have implemented this so far but figured it later that stream based TCP may not guarantee to recv entire struct in one call.
ii. Use function like strncat to add that flag at the end to the last data being sent.
The prob is I have to check on every receive call either using regex functions or function like strstr for the presence of that flag and if so have to remove it from the data.
This application is going to be used for large data transfers and hence want to add minimal overhead on every send/recv/read/write call. Would really appreciate to know if there is a better option then the above two or any other option to check the receipt of last packet. The program is multithreaded.
Edit: I do not know the total size of file I am going to send, but I am sending fixed amount of data. That is fgets read until the size specified -1 or until a new line is encountered.
Do you know the size of the data in advance, and is it a requirement that you implement a end of message flag?
Because I would simplify the design, add a 4-byte header (assuming you're not sending more than 4gb of data per message), that contains the expected size of the message.
Thus you parse out the first 4 bytes, calculate the size, then continue calling recv until you get that much data.
You'll need to handle the case where your recv call gets data from the next message, and obviously error handling.
Another issue not raised with your 10byte pad solution is what happens if the actual message contains 10 zero bytes--assuming you're padding it with zeros? You'd need to escape the 10bytes of zeros otherwise you may mistakenly truncate the message.
Using a fixed sized header and a known size value will alleviate this problem.
For a message (data packet) first send a short (in network order) of the size, followed by the data. This can be achieved in one write system call.
On the reception end, just read the short and convert back into host order (this will enable one to use different processors at a later state. You can then read the rest of the data.
In such cases, it's common to block up the data into chunks and provide a chunk header as well as a trailer. The header contains the length of the data in the chunk and so the peer knows when the trailer is expected - all it has to do is count rx bytes and then check for a valid trailer. The chunks allow large data transfers without huge buffers at both ends.
It's no great hassle to add a 'status' byte in the header that can identify the last chunk.
An alternative is to open another data connection, stream the entire serialization and then close this data connection, (like FTP does).
Could you make use of an open source network communication library written in C#? If so checkout networkComms.net.
If this is truely the last data sent by your application, use shutdown(socket, SHUT_WR); on the sender side.
This will set the FIN TCP flag, which signals that the sender->receiver stream is over. The receiver will know this because his recv() will return 0 (just like an EOF condition) when everything has been received. The receiver can still send data afterward, and the sender can still listen for them, but it cannot send more using this connection.
which is the beast approach to send packets that can be of different size using TCP sockets in C?
I wonder because we're trying to write a multiplayer games that needs a protocol which has many kinds of packets of different sizes.. according to recv documentation I can get how many bytes have been read but how should I manage to dispatch packets only when they are exaclty full?
Suppose that I have packets with a 5 bytes header that contains also the length of the payload.. should I use circular buffers or something like that to dispatch packets when ready and keep new partials?
Create a static variable which represents the packet header, this variable will be five bytes long. Create an associated integer which counts how many of those five bytes have yet been read. Create a second integer which counts how many bytes of the "content" have been read. Zero both those integers. Create an associated char * pointer which eventually will point to the received packet content.
As data arrives (e.g., select indicates so), read the five bytes of header. You may receive these bytes gradually, thus you need the first integer count variable. Account for the header bytes you have received here.
When you are done receiving the header, sanity check it. Are the size values possible to satisfy (e.g. not greater than 2^30)? If so, malloc a buffer of that size or that size plus the header. (If you want the header contiguous, allocate sufficient space, then memcpy it into your new buffer.)
Now, as data arrives, place it in your allocated buffer. Account for the received bytes in the second integer you created. When you have received all the bytes the header called for, then repeat all the above.
you can design a custom header for your packet transmission, which specifies packet length, indexing info (if packet fragmentation is implemented) and some hashing if you need.
some rough pseudocode as follows :
recv(socket, headerBuf, headerSize, MSG_WAITALL);
nPacketSize = headerBuf[16]; //sample
nByteRead = 0;
while (nByteRead != nPacketSize)
{
nByteToRead = nPacketSize - nByteRead;
nCurRead = recv(socket, someBuf, nByteToRead, MSG_PARTIAL);
nByteRead += nCurRead;
Sleep(0); // Yield processor
}
TCP is a stream based protocol, not a datagram one. That means that there isn't necessarily a one to one correspondence between the number and size of buffers passed to send and what comes out of receive. That means that you need to implement your own "protocol" on top of TCP.
Examples of such protocols include HTTP, where HTTP messages are delineated by two consecutive carriage return, line feed pairs. \r\n\r\n. The only concern with such delineation is to make sure that the pattern can't occur in the body of the message, either that or to make sure it is escaped. Other protocols create a header which contains the information necessary to correctly identify and read the next piece of information. I can't think of an application that does this off the top of my head. You could even go for a hybrid approach that combines the two.