I made an app in C for Linux environment. This app works, but after
some time (5 hours more or less) its performance is erratic.
This is the background. I have some remote computers (400 more or
less) that send its data through sockets (my app) every 5 minutes. All
process is a loop and I need that works all day. The app first send
data (if exists) and then receive data (also, if exists). My app works
like a Server side with some arguments or it works like a Client side
with other arguments. The app working as Server mode, after the listening state,
only waits for client connections. Once that a client it's connected,
the App in Client mode sends a message to indicates that will be sent
data, then sends a new message with the size and name concatenated to
server side (always sends 64 bytes, so my app writes 64 bytes and in
the other side reads 64 bytes), then sends data (the file). The Server
reads the message of size and name of file to receive (of 64 bytes),
splits the message, storing the size in a variable and file name in
other, and then after received the file data. compares the size stored
in the variable with the size of data readed, if all is ok, then the
server sends a message and stores the data in a new file with the name
that was it before received. So on untill the last file will be sent
it.
When my app was in state of development, I noticed that after some
hours the messages they become corrupted, I mean that the messages are
incomplete or have more data, probably data of the next ones. So the
file data too. So, how can i send the file data through the same
socket?? Note: I have a buffer to send and receive messages, and
other buffer for send and receive file data.
And other question:
Original code:
int copydata(int readfd, int writefd, int offset, int bytes) {
char buffer[4096];
int crbytes = 0, cwbytes = 0, trbytes = 0, twbytes = 0;
if (offset) {
if (lseek(readfd, offset, SEEK_CUR) < 0) {
return -1;
}
}
while ((crbytes = read(readfd, buffer, bytes)) > 0) {
if (crbytes < 0) {
return -1;
}
trbytes += crbytes;
//printf("dbgmsg::Readed data <%dB> | Total readed data <%dB>\n", crbytes, trbytes);
while (crbytes > 0) {
cwbytes = write(writefd, buffer, crbytes);
if (cwbytes < 0) {
return -1;
}
twbytes += cwbytes;
crbytes -= cwbytes;
//printf("dbgmsg::Written data <%dB> | Total written data <%dB>\n", cwbytes, twbytes);
}
}
return twbytes;
}
This code is used to send and receive file data. The side that sends
data uses the file descriptor of the file that we want to send (to
read) <readfd> and writes on the file descriptor of the socket (to
write) <writefd>. The side that receives data, use <readfd> to read
from the socket file descriptor and <writefd> to write in the file
descriptor from the file where we want to write the data.
If are sent other messages before use this function, both, client and
server will be stuck in the end of the inner "while loop", in other
words, the client sends all file data, and the server receives all
file data (in this point, the received data is complete, how i know?
Because i can open the the received file). No errors, only one, "no message of
desired type". If i not sent messages before this function all works
fine.
To skip this little problem, i modify the code, passing a file size as
argument too and write between two whiles one if structure.
int copydata(int readfd, int filesz, int writefd, int offset, int bytes) {
char buffer[4096];
int crbytes = 0, cwbytes = 0, trbytes = 0, twbytes = 0;
if (offset) {
if (lseek(readfd, offset, SEEK_CUR) < 0) {
return -1;
}
}
while ((crbytes = read(readfd, buffer, bytes)) > 0) {
if (crbytes < 0) {
return -1;
}
trbytes += crbytes;
//printf("dbgmsg::Readed data <%dB> | Total readed data <%dB>\n", crbytes, trbytes);
while (crbytes > 0) {
cwbytes = write(writefd, buffer, crbytes);
if (cwbytes < 0) {
return -1;
}
twbytes += cwbytes;
crbytes -= cwbytes;
//printf("dbgmsg::Written data <%dB> | Total written data <%dB>\n", cwbytes, twbytes);
}
if (twbytes == filesz) { break; }
}
return twbytes;
}
Thanks for advance, and sorry for my english!!
If the program encounters an "incomplete" write(), your inner loop resends the beginning of the buffer.
while (crbytes > 0) {
cwbytes = write(writefd, buffer, crbytes);
if (cwbytes < 0) {
return -1;
}
twbytes += cwbytes;
crbytes -= cwbytes;
//printf("dbgmsg::Written data <%dB> | Total written data <%dB>\n", cwbytes, twbytes);
}
You could change it to something like:
for ( sentbytes=0; sentbytes < crbytes; sentbytes += cwbytes ) {
cwbytes = write(writefd, buffer+sentbytes, crbytes - sentbytes);
if (cwbytes < 0) { return -1; }
twbytes += cwbytes;
}
Related
Summary: I am working on a multithreaded application (both the proxy and cache are multithreaded). I am passing a request the client (client is a black box, cannot see the code) makes to the proxy, and the proxy connects with a cache to query for the file. The proxy and cache share a message queue for the requests, and a POSIX shared memory channel for the data. If the request path is not found in the cache, then I post to the shared memory channel a -1. If it is found, I send back a concatenated string (size of data chunk|data) thru the shared memory channel instead. The proxy determines the status from the shared memory channel and sends the response back to the client, followed by the data if it exists.
The problem: I send back the perfect amount of data and have no issues, client and proxy filesizes and chunks sent/received match. I can also print the data and see that it APPEARS to be the same in both the proxy and cache (the characters are really weird). But when I go to open the jpg which the client wrote from the data it received, it is corrupted. I have scoured this code for days, but cannot find the reason.
I have a handler function for a proxy I made. It is ran infinitely inside the server, and waits until there is a request to serve:
ssize_t proxy_worker_handler(custom_struct *ctx, const char *path, void *arg)
{
int segment_to_lock_into = *(int *)arg % GLOB_NSEGMENTS;
size_t filesize = 0;
mq_send(COMMAND_CHANNEL_ID, path, BUFSIZE, 1);
data_channel current_channel = GLOB_DATA_CHANNEL_SEGMENTS[segment_to_lock_into];
// Take proxy ownership of this segment for now
sem_wait(current_channel.proxy_sem);
// Wait until we can read, writer sends the signal
sem_wait(current_channel.reader_sem);
// Now we can read, see if file was found and send proper header
if (strncmp(current_channel.data, "-1", strlen("-1")) == 0)
{
gfs_sendheader(ctx, GF_FILE_NOT_FOUND, 0);
sem_post(current_channel.writer_sem);
}
else
{
// Get the filesize
filesize = (size_t)atoi(current_channel.data);
gfs_sendheader(ctx, GF_OK, filesize);
sem_post(current_channel.writer_sem);
// Now receive data
int curr_received = 0;
int delimiter, data_len;
int padding_size = 10;
char *data_str;
char data_len_str[padding_size];
while (curr_received < (int)filesize)
{
// Wait to be able to read
sem_wait(current_channel.reader_sem);
data_str = "";
memset(data_len_str, 0, padding_size);
// Break up the memory segment, to get size and string sent
data_str = strstr(current_channel.data, "|") + 1;
delimiter = (data_str - 1) - current_channel.data;
for (int i = 0; i < delimiter; i++)
data_len_str[i] = current_channel.data[i];
data_len = atoi(data_len_str);
printf("%s", data_str);
// Use the above info to send the data
curr_received += data_len;
gfs_send(ctx, data_str, (size_t)data_len);
sem_post(current_channel.writer_sem);
}
printf("%d of %zu\n", curr_received, filesize);
}
// Free up the segment for proxy workers waiting
sem_post(current_channel.proxy_sem);
return filesize;
}
I also have a cache worker, which is called infinitely as well but handles itself:
void *cache_worker(void *arg)
{
char *path;
int fd;
struct stat st;
char filesize[21];
size_t batch_read, start_point;
int segment_to_lock_into = *(int *)arg % GLOB_NSEGMENTS;
free(arg);
data_channel current_channel = GLOB_DATA_CHANNEL_SEGMENTS[segment_to_lock_into];
while (1)
{
// Receive the request path from the boss
pthread_mutex_lock(&queue_lock);
while (steque_isempty(&queue) == 1)
pthread_cond_wait(&queue_cond, &queue_lock);
path = steque_pop(&queue);
pthread_mutex_unlock(&queue_lock);
fd = simplecache_get(path);
sleep(cache_delay);
// Because strdup uses malloc
free(path);
// Take cache ownership of this segment for now
sem_wait(current_channel.cache_sem);
// Now, see if the file exists
if (fd == CACHE_FAILURE)
{
sem_wait(current_channel.writer_sem);
// Write and then increment the read semaphore, so the proxy knows to read it
memset(current_channel.data, 0, GLOB_SEGSIZE);
memcpy(current_channel.data, "-1", strlen("-1"));
sem_post(current_channel.reader_sem);
close(fd);
}
else
{
// Tell the reader the size of the file first
sem_wait(current_channel.writer_sem);
fstat(fd, &st);
snprintf(filesize, 21, "%ld", st.st_size);
memset(current_channel.data, 0, GLOB_SEGSIZE);
memcpy(current_channel.data, filesize, 21);
sem_post(current_channel.reader_sem);
// Open the file and send the data
int padding_size = 10;
char data[GLOB_SEGSIZE];
char to_send[GLOB_SEGSIZE];
batch_read = 0, start_point = 0;
while ((int)(batch_read = pread(fd, data, (size_t)(GLOB_SEGSIZE - padding_size), start_point)) > 0)
{
sem_wait(current_channel.writer_sem);
memset(to_send, 0, GLOB_SEGSIZE);
memset(current_channel.data, 0, GLOB_SEGSIZE);
snprintf(to_send, GLOB_SEGSIZE, "%zu|%s", batch_read, data);
memcpy(current_channel.data, to_send, (size_t)GLOB_SEGSIZE);
start_point += batch_read;
printf("%s", data);
sem_post(current_channel.reader_sem);
}
printf("%zu of %s\n", start_point, filesize);
}
// Free up the segment for cache workers waiting
sem_post(current_channel.cache_sem);
}
return NULL;
}
I'm trying to send a jpg image from a client process to a server via sockets. The image contains binary data so I want to do it on a low-level programming basis using reads and writes. I'm also sending the image data in iterations of 100 bytes.
This is the code I've done, which is not sending the image identically as I want:
CLIENT
void send_image(char *path, char *filename, int socket) {
int fd = open(path, O_RDONLY); //I open the file of the image.jpg
int n = 1;
while (n > 0) {
char img_data[100];
n = read(fd, img_data, 100); //sending 100 bytes of image each iteration till n=0 (end of file)
if (!n) break;
int sending = 1;
write(socket, &sending, sizeof(int)); //Tell the client the image still has data to send
write(socket, img_data, strlen(img_data));
usleep(250);
}
sending = 0; //Tell the server the image has been fully sent
write(socket, &sending, sizeof(int));
close(fd);
}
SERVER
void receiving_image(char *path) {
int receiving = 0;
int j=0;
char *image_data = NULL; //Variable to store all the image data
read(socket, &receiving, sizeof(int)); //Reads that the client is going to send an image
while (receiving) {
char data[100]; //Variable that stores partial data (100 bytes) of an image on each iteration
read(socket, data, 100);
image_data = realloc(image_data, (j + strlen(data)) * sizeof(char)); //Readjust the size of the main image data.
for (int i=0; i<(int) strlen(data); i++) {
image_data[j] = data[i]; //copy the partial data of the image to the main variable of the image
j++;
}
j = (int) strlen(image_data);
read(socket, &receiving, sizeof(int)); //Read if the image is still sending
}
image_to_directory(path, image_data); //Copy image to directory
}
This compiles and runs fine, but when I check the directory on the server side where the image has been stored, I can see it's not the same image as the client has sent (I confirmed via md5sum and hashes are not equal).
Is there something I am missing?
You shouldn't be using strlen for your binary data length calculations. It is intended only for terminated strings (thus the name). You also have highly-ill-advised naked calls to read/write, which is a recipe for disaster when sending data over sockets.
You never seem to be sending more than 100 bytes at a time, which is helpful in this case to develop a more solid protocol. Consider this:
First octet is a uint8_t byte count N, and will be in 0..100.
Following the byte count, N bytes are transferred.
Repeat 1-2 until no more bytes remain.
Notify the server of EOF by sending a single zero-octet
An example of this sender code is shown here.
void send_image(const char *path, int socket)
{
int fd = open(path, O_RDONLY); //I open the file of the image.jpg
if (fd == -1)
return;
ssize_t n = 0;
do
{
// note the first octet will prefix the length
uint8_t img_data[101];
n = read(fd, img_data+1, 100);
if (n > 0)
{
// you never know just how many bytes are going to
// be sent, so setup the frame, but then ensure even
// piecewise deliver can succeed.
img_data[0] = (uint8_t)n;
ssize_t sent = 0;
size_t pos = 0;
do
{
sent = write(socket, img_data+pos, (n+1)-pos);
if (sent < 0)
break;
pos += sent;
} while ( pos < (n+1) && sent > 0);
}
} while (n > 0);
uint8_t done = 0;
write(socket, &done, sizeof done); // not much we can do if this fails
close(fd);
}
I make no claims the above code will even compile, but the concept should be fairly obvious. That's it, however. Obviously there is more that could/should be done (checksums, restart options, etc.), but that's the basic premise.
The server side can do something similar, which I leave as an exercise for you. The point of all of this is to utilize the return values from your read/write calls. They're there for a reason. If you find yourself coding a "naked" read or write (where you don't gather the result of function and utilize it in some way), chances are you've done something horribly wrong.
I write program and it works fine, but i want to rewrite it using sendfile() and now i got stuck in a loop.
Server side:
send name = ok
send md5 checksum = ok
send size = ok
send file = ko
Client side:
recv name = ok
recv md5 cecksum = ok
recv size = ok
create dir and create file = ok
write data to created file = ko
P.S In previous version of program i stuck some time to, but it depend how much i use printf why? for e.x i add one line with printf program stuck, delete it, works fine.
UPDT: rewrite code client/server
client
/* Received file name */
int rc_byte = 0;
rc_byte = recv(fd, rx_tx_file->out_name, sizeof(rx_tx_file->out_name),0);
if (rc_byte < 0){
perror("Failed to receive file name: ");
exit(-1);
} else
printf("Recv out name %s\n", rx_tx_file->out_name);
//printf("file name rc %s\n", rx_tx_file->out_name);
trimm_path_name(rx_tx_file);
/* Received md5sum */
rc_byte = recv(fd, rx_tx_file->md5sum, sizeof(rx_tx_file->md5sum), 0);
if (rc_byte < 0) {
perror("Failed to receive check sum: ");
exit(-1);
} else
printf("recv md5s %s\n", rx_tx_file->md5sum);
/* Received file size */
rc_byte = recv(fd, &size, sizeof(size), 0);
if(rc_byte < 0) {
perror("Recevid size of file: ");
exit(-1);
}
printf("%d recv size\n", size);
to_read = size;
if (stat(dir, &st) == -1){
mkdir(dir, 0777);
}
send_data: (add func to server)
void send_data(int client_fd, m_file *rx_tx_file, int option, int size) {
int send_byte = 0;
int total_send = 0;
if (option == SEND_NAME) {
while (total_send < strlen(rx_tx_file->in_name)) {
send_byte = send(client_fd, rx_tx_file->in_name, sizeof(rx_tx_file->in_name),0);
if(send_byte == -1) {
perror("Failed to send file name to client: ");
exit(SEND_TO_CLIENT_ERROR);
}
total_send += send_byte;
}
}
else if (option == SEND_MD5) {
total_send = 0;
send_byte = 0;
while (total_send < strlen(rx_tx_file->md5sum)) {
send_byte = send(client_fd, rx_tx_file->md5sum, sizeof(rx_tx_file->md5sum),0);
if(send_byte == -1){
perror("Failed to send file md5sum to client: ");
exit(-1);
}
total_send += send_byte;
}
}
else if (option == SEND_SIZE) {
send_byte = send(client_fd, &size, sizeof(size),0);
if (send_byte == -1) {
perror("Failed to send size: ");
}
}
}
server:
client_fd = accept(server_fd, (struct sockaddr*) &client_addr, &length)
/*send name of file*/
send_data(client_fd, rx_tx_file, SEND_NAME, 0);
/*send md5 sum*/
take_check_sum(rx_tx_file,rx_tx_file->file_in, 0);
send_data(client_fd, rx_tx_file, SEND_MD5, 0);
/*send size of file*/
size = stats.st_size;
send_data(client_fd, rx_tx_file, SEND_SIZE, size);
remain_data = stats.st_size;
printf("File [%s] ready to send\ncheck sum [%s]\n", rx_tx_file->in_name,rx_tx_file->md5sum);
while (((send_byte = sendfile(client_fd, file_fd, &offset, size)) > 0) && (remain_data > 0))
{
remain_data -= send_byte;
printf("remain %d", remain_data);
}
printf("Succesfully");
Since i work with one client and pass file which should send on server side through command line args, i dont need to wait in while (client_fd = accpet) i just work with one connection and close server. Now its work good. But one question is open, how i should rewrite client side to recv data in a loop. I don't know which size i should recv and because of that i cant write right condition to my while loop. THX all for helping.
TCP is a stream. It has no message boundaries. Your code won't work because of that.
First, you send the name of the file:
send(client_fd, rx_tx_file->in_name, strlen(rx_tx_file->in_name)+1,0)
then you immediately send the md5 sum and then the file size:
send(client_fd, rx_tx_file->md5sum, strlen(rx_tx_file->md5sum)+1, 0)
send(client_fd, &size, sizeof(int),0)
Since the first two strings don't have a fixed number of bytes, it's quite likely that when you try to read the file size or md5 sum from the server you also read the size of the file and maybe even some of the file data.
First, stop trying to put as much of your send and read code as you can into the conditional clause of your if and while statements.
What exactly does
if (send(client_fd, rx_tx_file->md5sum, strlen(rx_tx_file->md5sum)+1, 0) == -1) {
perror("Failed to send file md5sum to client: ");
exit(-1);
}
gain you over
ssize_t bytes_sent = send(client_fd, rx_tx_file->md5sum, strlen(rx_tx_file->md5sum)+1, 0);
if ( bytes_sent < 0 )
{
perror("Failed to send file md5sum to client: ");
exit(-1);
}
Putting all that code into the if clause gains you nothing on the send. And what if strlen(rx_tx_file->md5sum)+1 is 87 and the send() call returns 15? That's a possible return value that your code can't handle because it stuffs everything into the if clause.
ssize_t bytes_sent = send(client_fd, rx_tx_file->md5sum, strlen(rx_tx_file->md5sum)+1, 0);
if ( bytes_sent < 0 )
{
perror("Failed to send file md5sum to client: ");
exit(-1);
}
else if ( bytes_sent < strlen(rx_tx_file->md5sum)+1 )
{
// partial send...
}
That's actually better coded as a loop.
You didn't post your receive code, but if it's in the same style you not only don't gain anything, by putting everything into the if clause you again can't do any decent error detection or correction.
If your file name recv code is similar to
char filename[1024];
if (recv(fd, &filename, sizeof(filename), 0) < 0) {
perror("Failed to read file name: ");
exit(-1);
}
you can't tell what you just received. How many bytes did you just receive? You may have received the file name. You may have received only part of the file name. You may have received the file name, the md5 sum, and some of the file contents itself.
You don't know what you received, and with your code you can't tell. If you zero out the file name and md5 receive buffers and only recv up to one byte less than the size of the buffer, you at least avoid undefined behavior. But if you don't zero out the buffer, or if you read up the the last byte of the buffer, you can also wind up without a nul-terminated string for your filename or md5 sum. And when you then try to treat it as a nul-terminated string you get undefined behavior.
And if you did get extra bytes in the recv calls you make before trying to read the file data, that explains why your code gets stuck - it already read some of the file contents before getting to the loop, so the loop will never see all the content - some of it is gone.
You should avoid using strlen here in your server:
if(send(client_fd, rx_tx_file->in_name, strlen(rx_tx_file->in_name)+1,0) == -1)
Rather just send fixed length string of size sizeof(rx_tx_file->out_name) as you expect in your client
If the filename is smaller just pad it with spaces to make it of length sizeof(rx_tx_file->out_name)
You should also put each receive call in while loop, and add checks that it actually received expected number of bytes, at times recv will just return partial data, you need to post another recv to receive rest of the expected data
I have a web service written in .net on a remote computer with IIS, I am trying to connect to it with a C program using socker to do a SOAP request.
My problem is that I have some probem receiving the data:
The receiving data loop does not work in a way or in another.
If I write:
nByte = 1;
while(nByte!=512)
{
nByte = recv(sockfd,buffer,512, 0);
if( nByte < 0 )
{
// check the error
}
if( nByte > 0)
{
// append buffer to received data
}
}
sometime does not return all data, if it run without debugger and breackpoints.
If I try: while(nByte!=0) at the end of data it stalls and go in error.
How is it supposed to be done?
Thanks,
Antonino
** EDIT **
I resolved my situation in another way, I check the returned value for soap xml end:
nByte = 1;
while(nByte!=0)
{
nByte = recv(sockfd,buffer,512, 0);
if( nByte < 0 )
{
// check the error
}
if( nByte > 0)
{
// append nByte buffer to received data
if( strstr("</soap:Envelope>", buffer) != NULL)
break;
}
}
It is very sad...
#define BUFFERSIZE 512
byte buffer[BUFFERSIZE];
int nByte = BUFFERSIZE;
int rByte;
while(nByte!=0)
{
rByte = recv(sockfd, &buffer[BUFFERSIZE-nByte], nByte, 0);
if( rByte < 0 )
{
// socket error
break;
}
if( rByte == 0)
{
// connection closed by remote side or network breakdown, buffer is incomplete
break;
}
if(rByte>nByte)
{
// impossible but you must check it: memory crash, system error
break;
}
nByte -= rByte; // rByte>0 all is ok
// if nByte==0 automatically end of loop, you read all
// if nByte >0 goto next recv, you need read more bytes, recv is prtialy in this case
}
//**EDIT**
if(nByte!=0) return false;
// TO DO - buffer complete
Where does it say it fills the buffer? Read the man image. It blocks until at least one byte of data can be transferred, then transfers whatever data has arrived.
I am trying to implement a web server in C. I am sending a response of a request using the send command in a loop.
buf_len = 1;
while (buf_len > 0)
{
buf_len = read(fh, out_buf, BUF_SIZE);
if (buf_len > 0)
{
send(client_s, out_buf, buf_len, 0);
}
}
Should I terminate the last character of by char out_buf[BUF_SIZE] with '\0'.
Also when I am trying to send a png file with the same code, Sometimes I receive the entire image and sometimes only half of it. It loads fully when I refresh the page.
For send function, see man page
RETURN VALUE
On success, these calls return the number of characters sent. On error, -1 is returned, and errno is set appropriately.
So it doesn't guarantee that buf_len bytes are sent out, you can check the return value.
As Hardy said above, send does not guarantee that buf_len bytes are sent out. So one of many ways to do it is
buf_len = 1;
while (buf_len > 0) {
buf_len = read(fh, out_buf, BUF_SIZE);
if(buf_len < 0) {
perror("read");
exit(1);
}
int sent = 0;
int count = buf_len;
while(count) {
int n = send(client_s, out_buf + sent, count, 0);
count -= n;
sent += n;
}
}
How is out_buf defined?