File transfer with arduino - file

My final goal is to send a 30 KB file over XBEE to another arduino. But for now i am just trying to duplicate a 4KB file on SD connected to first arduino. First i tried to send the data one byte by one byte.it worked and file duplicated successfully . but i have to have a buffer and then send data on 64 byte packets to XBEE so i should be able to read and write file in 64 byte packets. this is what i have done:
#include <SD.h>
#include <SPI.h>
void setup() {
Serial.begin(115200);
while (!Serial) {
; // wait for serial port to connect. Needed for native USB port only
}
if (!SD.begin(4)) {
Serial.println("begin failed");
return;
}
File file = SD.open("student.jpg",FILE_READ);
File endFile = SD.open("cop.jpg",FILE_WRITE);
Serial.flush();
char buf[64];
if(file) {
while (file.position() < file.size())
{
while (file.read(buf, sizeof(buf)) == sizeof(buf)) // read chunk of 64bytes
{
Serial.println(((float)file.position()/(float)file.size())*100);//progress %
endFile.write(buf); // Send to xbee via serial
delay(50);
}
}
file.close();
}
}
void loop() {
}
It successfully finish its progress until 100% but when i open the SD on laptop the file is created but it shown as 0 KB file.
whats the problem?

You are not telling .write what's the length of your buffer, so it will think it's a null-terminated string (which it isn't).
Plus, the inner loop appears to be not only unnecessary but even harmful because it would skip the last chunk if it's less than 64 bytes.
Check this out:
while(file.position() < file.size()) {
// The docs tell me this should be file.readBytes... but then I wonder why file.read even compiled for you?
// So if readBytes doesn't work, go back to "read".
int bytesRead = file.readBytes(buf, sizeof(buf));
Serial.println(((float)file.position()/(float)file.size())*100);//progress %
// We have to specify the length! Otherwise it will stop when encountering a null byte...
endFile.write(buf, bytesRead); // Send to xbee via serial
delay(50);
}

Related

Linux on RPi debian, hidraw write() to USB device outputs a few junk characters to /dev/hidraw0 which if not cleared jam the device

We have a set of USB devices which we monitor using a RPi. The monitoring code polls the devices using hidraw direct interface about once a second. The protocol uses 64 byte packets to send commands and receive data and all responses are 64 bytes long at most.
The same scheme works fine under Windows using the Windows HID driver. On Linux however we use hidraw and find that the device interface gets jammed after a short time resulting in unsuccessful write{}s to the device.
After a lot of investigation I came across a recommendation to try to follow the communication between a host and an hidraw device using this in a terminal:
sudo cat /dev/hidraw0
As it turns out, running this command outputs 4-8 bytes of unreadable characters to the terminal every write() and unexpectedly it also clears the jam for hidraw0. All subsequent write()'s and read()'s to that device work flawlessly.
If that device is disconnected and then reconnected the jam condition returns shortly thereafter. I have single stepped the code and verified that the "junk" is output during the execution of the write().
I tried to add fsync() calls before and after the write() in hope to clear the buffers and avoid this issue but that did not help. The code for the write() and subsequent read() is standard as follows:
#define USB_PACKET 64
#define USB_WRDELAY 10 //ms
FILE* fd;
int errno, res;
char packet[USB_PACKET];
fd = 0;
/* Open the Device with non-blocking reads. */
fd = open("/dev/hidraw0", O_RDWR|O_NONBLOCK);
if (fd < 0) {
perror("Unable to open device");
return 0; // failure
}
memset(packet, 0x0, sizeof(packet));
packet[0] = 0x34; // command code - request for USB device status bytes
fsync();
res = write(fd, &packet, sizeof(packet));
fsync();
if (res < 0) {
printf("Error: %d in USB write()\n", errno);
close(fd);
return 0; // failure
} else {
usleep(1000*USB_WRDELAY ); // delay gives OS and device time to respond
res = read(fd, &packet, sizeof(packet));
if (res < 0) {
printf("Error: %d in USB read()\n", errno);
close(fd);
return 0; // failure
} else {
// good read, packet holds the response data
// process the device data
close(fd);
return 1; // OK
}
}
return 0; // failure
This is a sample of the gibberish we read on the terminal running the cat command for each executed write():
4n��#/5 �
I am not understanding where this junk comes from and how to get rid of it. I tried several things that did not work out such as adding a read() with a timeout before the write - hoping it is some data left from a previous incomplete read().
Also tried to write a smaller buffer as I need only send only a 2 byte command as well as adding a delay between the open() and write().
Unfortunately using the cat in the terminal interferes with the hot plug/unplug detection of the USB devices so it is not a solution we can use in deployment.
I'll appreciate any words of wisdom on this.

I can't receive more than 64 bytes on custom USB CDC class based STM32 device

currently I try to sent 720 bytes from Windows application to custom STM32 device (now for testing purposes I use Blue Pill - STM32F103xxx). Ah, I forgot to point that I am totally newbie into programming :). So on device side I have 1000 bytes buffers for receiving and sending (Thanks to STMCube for this). Testing device with terminal program ( packets < than 64 bytes) works. Then I rework one of Microsoft examples to be able to sent more data to device. Used device driver on Windows is "usbser.sys". In short my console program do following:
Calculate SINE weave (360) samples - 16 bytes size
Sent them to USB Device as 720 bytes (byte size protocol for COM port)
My problem is that no more than 64 bytes comes into device.
Somewhere I read that reason for this can be into built in Rx,Tx Windows buffers (64 bytes long by mention somewhere on internet) and for this into code below I insert:
SetupComm(hCom,1000,1000)
in hope that this will solve my troubles but nope. Below is "my" code, any ideas how I can fix this?
#include <windows.h>
#include <tchar.h>
#include <stdio.h>
#include <math.h>
#define PI 3.14159265
void PrintCommState(DCB dcb)
{
// Print some of the DCB structure values
_tprintf(TEXT("\nBaudRate = %d, ByteSize = %d, Parity = %d, StopBits = %d\n"),
dcb.BaudRate,
dcb.ByteSize,
dcb.Parity,
dcb.StopBits);
}
int _tmain(int argc, TCHAR* argv[])
{
DCB dcb;
HANDLE hCom;
BOOL fSuccess;
const TCHAR* pcCommPort = TEXT("COM3"); // Most systems have a COM1 port
unsigned __int8 aOutputBuffer[720];// Data that will sent to device
unsigned __int16 aCalculatedWave[360];// Data that will sent to device
int iCnt; // temp counter to use everywhere
for (iCnt = 0; iCnt < 360; iCnt = iCnt + 1)
{
aCalculatedWave[iCnt] = (unsigned short)(0xFFFF * sin(iCnt * PI / 180));
if (iCnt > 180) aCalculatedWave[iCnt] = 0 - aCalculatedWave[iCnt];
}
// 16 bit aCalculatedWaveto to 8 bit aOutputBuffer
for (int i = 0, j = 0; i < 720; i += 2, ++j)
{
aOutputBuffer[i] = aCalculatedWave[j] >> 8; // Hi byte
aOutputBuffer[i + 1] = aCalculatedWave[j] & 0xFF; // Lo byte
}
// Open a handle to the specified com port.
hCom = CreateFile(pcCommPort,
GENERIC_READ | GENERIC_WRITE,
0, // must be opened with exclusive-access
NULL, // default security attributes
OPEN_EXISTING, // must use OPEN_EXISTING
0, // not overlapped I/O
NULL); // hTemplate must be NULL for comm devices
if (hCom == INVALID_HANDLE_VALUE)
{
// Handle the error.
printf("CreateFile failed with error %d.\n", GetLastError());
return (1);
}
if (SetupComm(hCom,1000,1000) !=0)
printf("Windows In/Out serial buffers changed to 1000 bytes\n");
else
printf("Buffers not changed with error %d.\n", GetLastError());
// Initialize the DCB structure.
SecureZeroMemory(&dcb, sizeof(DCB));
dcb.DCBlength = sizeof(DCB);
// Build on the current configuration by first retrieving all current
// settings.
fSuccess = GetCommState(hCom, &dcb);
if (!fSuccess)
{
// Handle the error.
printf("GetCommState failed with error %d.\n", GetLastError());
return (2);
}
PrintCommState(dcb); // Output to console
// Fill in some DCB values and set the com state:
// 57,600 bps, 8 data bits, no parity, and 1 stop bit.
dcb.BaudRate = CBR_9600; // baud rate
dcb.ByteSize = 8; // data size, xmit and rcv
dcb.Parity = NOPARITY; // parity bit
dcb.StopBits = ONESTOPBIT; // stop bit
fSuccess = SetCommState(hCom, &dcb);
if (!fSuccess)
{
// Handle the error.
printf("SetCommState failed with error %d.\n", GetLastError());
return (3);
}
// Get the comm config again.
fSuccess = GetCommState(hCom, &dcb);
if (!fSuccess)
{
// Handle the error.
printf("GetCommState failed with error %d.\n", GetLastError());
return (2);
}
PrintCommState(dcb); // Output to console
_tprintf(TEXT("Serial port %s successfully reconfigured.\n"), pcCommPort);
if (WriteFile(hCom, aOutputBuffer, 720, NULL, 0) != 0)
_tprintf(TEXT("720 bytes successfully writed to Serial port %s \n"), pcCommPort);
else
_tprintf(TEXT("Fail on write 720 bytes to Serial port %s \n"), pcCommPort);
return (0);
}
USB bulk endpoints implement a stream-based protocol, i.e. an endless stream of bytes. This is in contrast to a message-based protocol. So USB bulk endpoints have no concept of messages, message start or end. This also applies to USB CDC as it is based on bulk endpoints.
At the lower USB level, the stream of bytes is split into packets of at most 64 bytes. As per USB full-speed standard, packets cannot be larger than 64 bytes.
If the host sends small chunks of data that are more than 1ms apart, they will be sent and received in separate packets and it looks as if USB is a message-based protocol. However, for chunks of more than 64 bytes, they are split into smaller packets. And if small chunks are sent with less than 1ms in-between, the host will merge them into bigger packets.
Your design seems to require that data is grouped, e.g. the group of 720 bytes mentioned in the question. If this is a requirement, the grouping must be implemented, e.g. by first sending the size of the group and then the data.
Since larger groups are split into chunks of 64 bytes and the receive callback is called for every packet, the packets must be joined until the full group is available.
Also note a few problems in your current code (see usbd_cdc_if.c, line 264):
USBD_CDC_SetRxBuffer(&hUsbDeviceFS, &Buf[0]);
USBD_CDC_ReceivePacket(&hUsbDeviceFS);
NewDataFromUsb = *Len;
USBD_CDC_SetRxBuffer sets the buffer for the next packet to be received. If you always use the same buffer – as in this case – it's not needed. The initial setup is sufficient. However, it could be used to set a new buffer if the current packet does not contain a full group.
Despite its name, USBD_CDC_ReceivePacket does not receive a packet. Instead, it gives the OK to receive the next package. It should only be called if the data in the buffer has been processed and the buffer is ready to receive the next packet. Your current implementation runs the risk that the buffer is overwritten before it is processed, in particular if you send a group of more than 64 bytes, which will likely result in a quick succession of packets.
Note that Windows hasn't been mentioned here. The Windows code seems to be okay. And changing to Winusb.sys will just make your life harder but not get you packets bigger than 64 bytes.

ESP32 Bluetooth Connection disconnects when finished with reading buffer

I'm programming an ESP32 to accept Bluetooth commands and send Bluetooth Data back to my phone using the Serial profile. For this I'm using the Arduino Espressif Bluetooth Serial Library. Whenever I send something to the ESP32 it processes it and then suddenly closes the Bluetooth Connection.
Up to know I have already tried various delays because I thought that maybe the processor was not keeping up with other stuff due to which it crashed.
However when monitoring using the Serial Connection via USB it still keeps sending status updates.
Other than that I could not really find a solution (also on the Internet).
As I'm pretty much a beginner I did not want to try and build my own Serial Bluetooth Library.
The ESP does not crash when sending it the data. It also keeps processing the data which was sent. I can see that as it sends the chars I sent it via Bluetooth using the Serial interface after having collected them.
The Connection cannot be rebuild after this incident, no matter how long I wait.
My Main function, containing the function call as well as a buffer to write the result to as I thought that maybe I was misusing that.
void loop() {
if (ESP_BT.available() > 0)
{
char *buffer = (char*) malloc(InputSize);
getCurrentMessage(ESP_BT, buffer, InputSize);
Serial.println(buffer);
strncpy(currentMessage, buffer, InputSize);
free(buffer);
}
if (millis() %2000 == 0){
Serial.println("Debug");
delay(1);
}
}
The function which gets called which should read the inputBuffer of the BluetoothSerial into my Buffer.
void getCurrentMessage(BluetoothSerial ESP_BT, char* receivedChars, int InputSize)
{
Serial.println("DEBUG: getCurrentMessageInit");
static byte ndx = 0;
char rc;
while (ESP_BT.available() > 0){
ESP_BT.println("DEBUG: Message Available");
Serial.println("DEBUG: Message Available");
rc = ESP_BT.read();
receivedChars[ndx] = rc;
ndx++;
delay(100);
if (ndx >= InputSize){
while(ESP_BT.available() > 0){
ESP_BT.read();
}
}
}
}
I would expect the Bluetooth Connection to keep working. This it does not do.
I also got the error Code "queue.c:1442 (xQueueGenericReceive)- assert failed!" When not using the delays and the ESP then rebooted.
This it does not do after I included the delays.
The problem was that I did not call my Bluetooth Object by reference.
Instead of giving my function the Bluetooth Object I should point to it:
void getCurrentMessage(BluetoothSerial* ESP_BT, char* receivedChars, int InputSize)
{
Serial.println("DEBUG: getCurrentMessageInit");
static byte ndx = 0;
char rc;
while (ESP_BT->available() > 0){
ESP_BT->println("DEBUG: Message Available");
Serial.println("DEBUG: Message Available");
rc = ESP_BT.read();
receivedChars[ndx] = rc;
ndx++;
delay(100);
if (ndx >= InputSize){
while(ESP_BT->available() > 0){
ESP_BT->read();
}
}
}
}

Unable to write the complete script onto a device on the serial port

The script file has over 6000 bytes which is copied into a buffer.The contents of the buffer are then written to the device connected to the serial port.However the write function only returns 4608 bytes whereas the buffer contains 6117 bytes.I'm unable to understand why this happens.
{
FILE *ptr;
long numbytes;
int i;
ptr=fopen("compass_script(1).4th","r");//Opening the script file
if(ptr==NULL)
return 1;
fseek(ptr,0,SEEK_END);
numbytes = ftell(ptr);//Number of bytes in the script
printf("number of bytes in the calibration script %ld\n",numbytes);
//Number of bytes in the script is 6117.
fseek(ptr,0,SEEK_SET);
char writebuffer[numbytes];//Creating a buffer to copy the file
if(writebuffer == NULL)
return 1;
int s=fread(writebuffer,sizeof(char),numbytes,ptr);
//Transferring contents into the buffer
perror("fread");
fclose(ptr);
fd = open("/dev/ttyUSB3",O_RDWR | O_NOCTTY | O_NONBLOCK);
//Opening serial port
speed_t baud=B115200;
struct termios serialset;//Setting a baud rate for communication
tcgetattr(fd,&serialset);
cfsetispeed(&serialset,baud);
cfsetospeed(&serialset,baud);
tcsetattr(fd,TCSANOW,&serialset);
long bytesw=0;
tcflush(fd,TCIFLUSH);
printf("\nnumbytes %ld",numbytes);
bytesw=write(fd,writebuffer,numbytes);
//Writing the script into the device connected to the serial port
printf("bytes written%ld\n",bytesw);//Only 4608 bytes are written
close (fd);
return 0;
}
Well, that's the specification. When you write to a file, your process normally is blocked until the whole data is written. And this means your process will run again only when all the data has been written to the disk buffers. This is not true for devices, as the device driver is the responsible of determining how much data is to be written in one pass. This means that, depending on the device driver, you'll get all data driven, only part of it, or even none at all. That simply depends on the device, and how the driver implements its control.
On the floor, device drivers normally have a limited amount of memory to fill buffers and are capable of a limited amount of data to be accepted. There are two policies here, the driver can block the process until more buffer space is available to process it, or it can return with a partial write only.
It's your program resposibility to accept a partial read and continue writing the rest of the buffer, or to pass back the problem to the client module and return only a partial write again. This approach is the most flexible one, and is the one implemented everywhere. Now you have a reason for your partial write, but the ball is on your roof, you have to decide what to do next.
Also, be careful, as you use long for the ftell() function call return value and int for the fwrite() function call... Although your amount of data is not huge and it's not probable that this values cannot be converted to long and int respectively, the return type of both calls is size_t and ssize_t resp. (like the speed_t type you use for the baudrate values) long can be 32bit and size_t a 64bit type.
The best thing you can do is to ensure the whole buffer is written by some code snippet like the next one:
char *p = buffer;
while (numbytes > 0) {
ssize_t n = write(fd, p, numbytes);
if (n < 0) {
perror("write");
/* driver signals some error */
return 1;
}
/* writing 0 bytes is weird, but possible, consider putting
* some code here to cope for that possibility. */
/* n >= 0 */
/* update pointer and numbytes */
p += n;
numbytes -= n;
}
/* if we get here, we have written all numbytes */

Ubuntu Socket programming : Packets are repackaged between TX and RX

I have 2 Ubuntu 14.04 PCs. One is used as a server and the other one is used as a client. The client setup a TCP connection to the server which sends some packets back. Here's the code on the server:
send(sd, pkt, pkt_len, MSG_NOSIGNAL);
The code on the client side is also very simple:
read(sd, buf, buf_size);
If the transmissions on the server is spaced out, I don't see any issue. However, if server is doing rapid transmissions, then thing looks ugly. Here's an example when the server is sending 8 packets back-to-back.
The server code shows the size of these 8 packets are: 752 (bytes), 713, 713, 713, 396, 398, 396, 396
tcpdump on the server captures 4 TX packets: 752 (bytes), 1398, 1398, 929
tcpdump on the client captures 3 RX packets: 752 (bytes), 2796, 929
The client code shows it receives only 2 packets with 3548 bytes and 929 bytes, respectively.
So you can see all the bytes sent by the server are received by the client. However, packets are combined at various points in the transmission path. I guess this is due to TSO, GSO, GRO, etc. However, shouldn't these optimizations re-assemble the packets back to the correct form when the packets are delivered to the receiving application?
How do I get around this issue?
TCP is carefully designed to not only permit but implement exactly what you're seeing. It is a byte-stream protocol. If you want messages you have to implement them yourself via a superimposed application protocol.
How do I get around this issue?
So you're using TCP (a byte-stream-oriented transport mechanism) but you'd like it to have message-oriented behavior. You can't change the way TCP works (it is, by design, allowed to transport bytes in whatever-sized groups it chooses to, as long as the bytes are all received and they are received in the same order). But you can add a layer on top of TCP to simulate packet-oriented behavior.
For example, say you wanted to simulate the transmission of a 1000-byte "packet". Your sending program could first send out a fixed-size (let's say, 4-byte) header that would tell the receiver how many bytes the "packet" will contain:
size_t myPacketSize = 1000; // or whatever the size of your packet is
uint32_t bePacketSize = htonl(myPacketSize); // convert native-endian to big-endian for cross-platform compatibility
if (send(sd, &bePacketSize, sizeof(bePacketSize), 0) != sizeof(bePacketSize))
{
perror("send(header)");
}
.... then right after that you'd send out the packet's payload data:
if (send(sd, packetDataPtr, myPacketSize, 0) != myPacketSize)
{
perror("send(body)");
}
The receiver would need to receive the header/size value, then allocate an array of that size and receive the payload data into it. Since this code has to handle the incoming data correctly no matter how many bytes are returned by each recv() call, it's a little more complex than the sending code:
void HandleReceivedPseudoPacket(const char * packetBytes, uint32_t packetSizeBytes)
{
// Your received-packet-handling code goes here
}
// Parses an incoming TCP stream of header+body data back into pseudo-packets for handling
void ReadPseudoPacketsFromTCPStreamForever(int sd)
{
uint32_t headerBuf; // we'll read each 4-byte header's bytes into here
uint32_t numValidHeaderBytes = 0; // how many bytes in (headerBuf) are currently valid
char * bodyBuf = NULL; // will be allocated as soon as we know how many bytes to allocate
uint32_t bodySize; // How many bytes (bodyBuf) points to
uint32_t numValidBodyBytes = 0; // how many bytes in (bodyBuf) are currently valid
while(1)
{
if (bodyBuf == NULL)
{
// We don't know the bodySize yet, so read in header bytes to find out
int32_t numBytesRead = recv(sd, ((char *)&headerBuf)+numValidHeaderBytes, sizeof(headerBuf)-numValidHeaderBytes, 0);
if (numBytesRead > 0)
{
numValidHeaderBytes += numBytesRead;
if (numValidHeaderBytes == sizeof(headerBuf))
{
// We've read the entire 4-byte header, so now we can allocate the body buffer
numValidBodyBytes = 0;
bodySize = ntohl(headerBuf); // convert from big-endian to the CPU's native-endian
bodyBuf = (char *) malloc(bodySize);
if (bodyBuf == NULL)
{
perror("malloc");
break;
}
}
}
else if (numBytesRead < 0)
{
perror("recv(header)");
break;
}
else
{
printf("TCP connection was closed while reading header bytes!\n");
break;
}
}
else
{
// If we got here, then we know the bodySize and now we need to read in the body bytes
int32_t numBytesRead = recv(sd, &bodyBuf[numValidBodyBytes], bodySize-numValidBodyBytes, 0);
if (numBytesRead > 0)
{
numValidBodyBytes += numBytesRead;
if (numValidBodyBytes == bodySize)
{
// At this point the pseudo-packet is fully received and ready to be handled
HandleReceivedPseudoPacket(bodyBuf, bodySize);
// Reset our state variables so we'll be ready to receive the next header
free(bodyBuf);
bodyBuf = NULL;
numValidHeaderBytes = 0;
}
}
else if (numBytesRead < 0)
{
perror("recv(body)");
break;
}
else
{
printf("TCP connection was closed while reading body bytes!\n");
break;
}
}
}
// Avoid memory leak if we exited the while loop in the middle of reading a psuedo-packet's body
if (bodyBuf) free(bodyBuf);
}

Resources