Below is code I created for epoll_wait on UNIX domain datagram sockets (note this is UNIX domain, not internet domain). Each of these C programs is by a NASM program -- the C object files are linked into the NASM executable.
I have no trouble with epoll_instance_create or add_to_epoll_fd_list (which adds each of the file descriptors). However, epoll_wait_next calls perror and returns "epoll_wait: Success", then loops through "for(i = 0; i < event_count; i++) { " as it should, but bytes_read is -1 on the line "bytes_read = read(events[i].data.fd, read_buffer, BUF_SIZE);"
The struct epoll_event epoll_events is defined as global so it can be passed by NASM when these programs are called.
The problem may be that I don't understand how to call read() in my context as the code comes from other code samples (this is the first time I've used epoll).
#define BUF_SIZE 750
#define SV_SOCK_PATH "/tmp/ud_ucase"
#define epoll_maxevents 100
struct epoll_event epoll_events;
int64_t epoll_instance_create() {
int epoll_fd = epoll_create1(0);
return (int64_t) epoll_fd;
}
int64_t add_to_epoll_fd_list(int epoll_fd, int this_fd) {
epoll_events.data.fd = epoll_fd;
epoll_events.events = EPOLLIN;
int res = epoll_ctl(epoll_fd, EPOLL_CTL_ADD, this_fd, &epoll_events);
perror("epoll_ctl");
if (res > 0)
return 1;
return 0;
}
int64_t epoll_wait_next(int epoll_fd, struct epoll_event * events){
int event_count, i;
ssize_t count;
ssize_t bytes_read;
int64_t read_buffer[750];
int64_t test_data;
event_count = epoll_wait(epoll_fd, events, epoll_maxevents, - 1);
perror("epoll_wait");
for(i = 0; i < event_count; i++) {
printf("Reading file descriptor '%d' -- ", events[i].data.fd);
bytes_read = read(events[i].data.fd, read_buffer, BUF_SIZE);
if (bytes_read > 0)
test_data = read_buffer[0];
printf("%zd bytes read.\n", bytes_read);
read_buffer[bytes_read] = '\0';
}
return 0;
}
The overall code (including the NASM code) is very large, and even a minimal version would be over 300 lines, so I have posted the relevant C programs above, which I think should be enough to spot the problem(s).
Thanks in advance for any help on understanding why bytes_read comes as -1; it should be 720 bytes.
Related
I'm currently trying to send raw binary data in the format of decimal to an external device over serial. I currently have the data in a buffer array but would like it in a structure like this:
struct packetData{
uint8_t sync1;
uint8_t sync2;
uint16_t messageId;
uint16_t dataWordCount;
uint16_t flags;
uint16_t checksum;
};
I'm also using 9600 baud, and have all the termios settings set using cfmakeraw and I'm currently writing using:
#include <stdio.h>
#include <stdint.h>
#include <unistd.h>
#include <fcntl.h>
#include <termios.h>
#include <string.h>
#include <errno.h>
#include <stdlib.h>
int flags = O_RDWR | O_NOCTTY | O_NDELAY;
fd = open(device, flags);
uint16_t buf_tx[BUFFER_SIZE] = {255,129,191,0,2057,0};
if(fd == -1){
printf("\n Failed to open port! ");
return -1;
}
tcgetattr(fd, &tty); //Get the current attributes of the Serial port
cfmakeraw(&tty);
cfsetispeed(&tty, B9600); //Set read speed as 9600 baud
cfsetospeed(&tty, B9600); //Set write speed as 9600 baud
if((tcsetattr(fd, TCSANOW, &tty)) != 0){
printf("Error! Can't set attributes.\n");
return -1;
}
else{
printf("Connection successful! \n");
}
while(x < 1000){
memset(buf_tx, 0, sizeof(buf_tx));
tcflush(fd, TCOFLUSH);
if(y < 5){
if(write(fd, buf_tx, 5) == -1){
printf("\n");
printf("Error>>: %s\n",strerror(errno));
y++;
}
}
tcflush(fd, TCIOFLUSH);
usleep(1000);
x++;
}
This code isnt the full code, just the setup/write parts so no need to worry about its syntax. if possible it would be nice not to have that buffer array and just use the struct directly, but I'll take what I can get.
It seems you have the serial port opening more or less in hand. I prefer to set the termios member components explicitly myself, but cfmakeraw() is perfectly fine too.
What you should consider, is having a separate function to send one or more of those structures at a time. For example,
int write_all(const int fd, const void *buf, const size_t len)
{
const char *data = buf;
size_t written = 0;
ssize_t n;
while (written < len) {
n = write(fd, data + written, len - written);
if (n > 0) {
written += n;
} else
if (n != -1) {
/* C library bug, should never occur */
errno = EIO;
return -1;
} else {
/* Error; n == -1, so errno is already set. */
return -1;
}
}
/* Success. */
return 0;
}
The function will return 0 if all data was successfully written, and -1 with errno set if an error occurs.
To send a struct packetData pkt; just use write_all(fd, &pkt, sizeof pkt).
To send a full array struct packetData pkts[5]; use write_all(fd, pkts, sizeof pkts).
To send n packets starting at pkts[i], use write_all(fd, pkts + i, n * sizeof pkts[0]).
However, you do not want to use tcflush(). It does not do what you think it does; it actually just discards data.
Instead, to ensure that the data you have written has been transmitted, you need to use tcdrain(fd).
I recommend against adding tcdrain(fd) at the end of write_all() function, because it blocks, pauses the program, until the data has been transmitted. This means that you should only use tcdrain() before you do something that requires the other end has received the transmission; for example before trying to read the response.
However, if this is a query-response interface, and you do intend to also read from the serial device, you should set tty.c_cc[VMIN] and tty.c_cc[VTIME] to reflect how you intend to use the interface. I prefer asynchronous full-duplex operation, but that requires select()/poll() handling. For half-duplex, with these exact structures only, you can use tty.c_cc[VMIN] = sizeof (struct packetData) with say tty.c_cc[VTIME] = 30, which causes read() to try and wait until a full structure is available, but at most 30 deciseconds (3.0 seconds). Something like tty.c_cc[VMIN] = 1; tty.c_cc[VTIME] = 1; is more common; that causes read() to return a short count (even 0!) if there is no additional data received within a decisecond (0.1 seconds). Then, the receive function could be along the following lines:
int read_all(const int fd, void *buf, const size_t len)
{
char *const ptr = buf;
size_t have = 0;
ssize_t n;
/* This function is to be used with half-duplex query-response protocol,
so make sure we have transmitted everything before trying to
receive a response. Also assumes c_cc[VTIME] is properly set for
both the first byte of the response, and interbyte response interval
in deciseconds. */
tcdrain(fd);
while (have < len) {
n = read(fd, ptr + have, len - have);
if (n > 0) {
have += n;
} else
if (n == 0) {
/* Timeout or disconnect */
errno = ETIMEDOUT;
return -1;
} else
if (n != -1) {
/* C library bug, should never occur */
errno = EIO;
return -1;
} else {
/* Read error; errno set by read(). */
return -1;
}
}
/* Success; no errors. */
return 0;
}
If this returns -1 with errno == ETIMEDOUT, the other side took too long to answer. There may be remainder of the late response in the buffer, which you can discard with tcflush(TCIFLUSH) (or with tcflush(TCIOFLUSH), which also discards any written data not yet transmitted). Synchronization in this case is a bit difficult, because the above read_all() function doesn't return how many bytes it received (and therefore how many bytes to discard of a partial structure).
Sometimes the interface used always returns the number of bytes, but also sets errno (to 0 if no error occurred, and a nonzero error constant otherwise). That would be better for a query-response interface read and write functions, but many programmers find this use case "odd", even though it is perfectly okay by POSIX.1 standard (which is the relevant standard here).
I am practicing the read and write system call, the below code is working fine with a while loop and also without them. could you please tell me what is the use of while loop here, is it necessary to add it while using read and write system calls. I am a beginner. Thanks.
#include <unistd.h>
#define BUF_SIZE 256
int main(int argc, char *argv[])
{
char buf[BUF_SIZE];
ssize_t rlen;
int i;
char from;
char to;
from = 'e';
to = 'a';
while (1) {
rlen = read(0, buf, sizeof(buf));
if (rlen == 0)
return 0;
for (i = 0; i < rlen; i++) {
if (buf[i] == from)
buf[i] = to;
}
write(1, buf, rlen);
}
return 0;
}
You usually need to use while loops (or some kind of loop in general) with read and write, because, as you should know from the manual page (man 2 read):
RETURN VALUE
On success, the number of bytes read is returned (zero indicates end
of file), and the file position is advanced by this number. It is
not an error if this number is smaller than the number of bytes
requested; this may happen for example because fewer bytes are
actually available right now (maybe because we were close to end-of-
file, or because we are reading from a pipe, or from a terminal), or
because read() was interrupted by a signal. See also NOTES.
Therefore, if you ever want to read more than 1 byte, you need to do this in a loop, because read can always process less than the requested amount.
Similarly, write can also process less than the requested size (see man 2 write):
RETURN VALUE
On success, the number of bytes written is returned (zero indicates nothing was written). It is not an error if this
number is smaller than the number of bytes requested; this may happen for example because the disk device was filled.
See also NOTES.
On error, -1 is returned, and errno is set appropriately.
The only difference here is that when write returns 0 it's not an error or an end of file indicator, you should just retry writing.
Your code is almost correct, in that it uses a loop to keep reading until there are no more bytes left to read (when read returns 0), but there are two problems:
You should check for errors after read (rlen < 0).
When you use write you should also add a loop there too, because as I just said, even write could process less than the requested amount of bytes.
A correct version of your code would be:
#include <stdio.h>
#include <unistd.h>
#define BUF_SIZE 256
int main(int argc, char *argv[])
{
char buf[BUF_SIZE];
ssize_t rlen, wlen, written;
char from, to;
int i;
from = 'e';
to = 'a';
while (1) {
rlen = read(0, buf, sizeof(buf));
if (rlen < 0) {
perror("read failed");
return 1;
} else if (rlen == 0) {
return 0;
}
for (i = 0; i < rlen; i++) {
if (buf[i] == from)
buf[i] = to;
}
for (written = 0; written < rlen; written += wlen) {
wlen = write(1, buf + written, rlen - written);
if (wlen < 0) {
perror("write failed");
return 1;
}
}
}
return 0;
}
I am sending integer array of size from one raspberry pi to another using POSIX sockets in C. I am writing an integer array of size 131072 from one pi, the return value from the write command shows that all 131072 values have been written. ret = write(socket, &array, sizeof(array)) Using the same method for receive ret = read(socket, &array, sizeof(array)) shows that all the sent values are not been read rather the number of values read correctly is also not constant but varies between 10000 to 20000.
I tried to use the read function inside a loop where I read integer in each loop iteration
for(int i =0; i<131072; i++){
ret = read(socket, &value, sizeof(value));
data[i] = value;}
I was able to receive all the values with no error or losses.
The underlying protocol (probably TCP and IP) splits the data into packets, which might arrive as individual data blocks on the receiving application. Theoretically, read() might receive every single byte individually (i.e. return 1 each time when called 2000 times). Your application needs to be able to work with that.
You might need to use code similar to this:
multi-rw.h
#ifndef JLSS_ID_MULTI_RW_H
#define JLSS_ID_MULTI_RW_H
#include <sys/types.h>
extern ssize_t multi_read(int fd, char *buffer, size_t nbytes);
extern ssize_t multi_write(int fd, const char *buffer, size_t nbytes);
#endif
multi-rw.c
It's a little unfortunate that the block of test code has to precede the active functions, but that's necessary (and test code is necessary — at least, I find it extremely helpful and reassuring). I suppose it could go into a separate header file (multi-rw-test.h or thereabouts) which would be conditionally included; it would be better for presentation on SO, but otherwise is just another file to worry about.
#include "multi-rw.h"
#include <unistd.h>
#ifdef TEST
#ifndef MAX_WRITE_SIZE
#define MAX_WRITE_SIZE 64
#endif
#ifndef MAX_READ_SIZE
#define MAX_READ_SIZE 64
#endif
static inline size_t min_size(size_t x, size_t y) { return (x < y) ? x : y; }
static inline ssize_t pseudo_read(int fd, char *buffer, size_t nbytes)
{
return read(fd, buffer, min_size(MAX_READ_SIZE, nbytes));
}
static inline ssize_t pseudo_write(int fd, const char *buffer, size_t nbytes)
{
return write(fd, buffer, min_size(MAX_READ_SIZE, nbytes));
}
#undef read
#undef write
#define read(fd, buffer, nbytes) pseudo_read(fd, buffer, nbytes)
#define write(fd, buffer, nbytes) pseudo_write(fd, buffer, nbytes)
#endif
ssize_t multi_read(int fd, char *buffer, size_t nbytes)
{
ssize_t nb = 0;
size_t nleft = nbytes;
ssize_t tbytes = 0;
while (nleft > 0 && (nb = read(fd, buffer, nleft)) > 0)
{
tbytes += nb;
buffer += nb;
nleft -= nb;
}
if (tbytes == 0)
tbytes = nb;
return tbytes;
}
ssize_t multi_write(int fd, const char *buffer, size_t nbytes)
{
ssize_t nb = 0;
size_t nleft = nbytes;
ssize_t tbytes = 0;
while (nleft > 0 && (nb = write(fd, buffer, nleft)) > 0)
{
tbytes += nb;
buffer += nb;
nleft -= nb;
}
if (tbytes == 0)
tbytes = nb;
return tbytes;
}
#ifdef TEST
#include "stderr.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(int argc, char **argv)
{
if (argc != 0)
err_setarg0(argv[0]);
char buffer[4096];
ssize_t ibytes;
while ((ibytes = multi_read(0, buffer, sizeof(buffer))) > 0)
{
ssize_t obytes;
if ((obytes = multi_write(1, buffer, ibytes)) != ibytes)
err_syserr("failed to write %lld bytes - only wrote %lld bytes\n",
(long long)ibytes, (long long)obytes);
}
if (ibytes < 0)
err_syserr("I/O error reading standard input: ");
return 0;
}
#endif
The test harness allows you to test the code reading from standard input and writing to standard output. You can configure the amount of data read via (for example) compilation command line options -DMAX_WRITE_SIZE=132 and -DMAX_READ_SIZE=103. You need to test it on files smaller than (a) 4096 bytes, and (b) smaller than the maximum read and write sizes, and also on files bigger than 4096 bytes. If you are motivated enough, you can upgrade the pseudo_read() and pseudo_write() functions to generate errors quasi-randomly, to see how the code handles such errors.
I am trying to send a long int number using sockets (from client to server).
Both the client and the server are x86 (and identical machines). The client writes 21 long int's to the socket and the server reads it out.
Here is a part of the code.
Server:
long int num = 0;
int ret;
for (int i = 0; i < 21; i++) {
if ((ret = read(sfd_client, (char *) &num, sizeof(num))) == -1) {
perror("read");
exit(1);
}
printf("number = %d = %ld ret = %d\n", i, num, ret);
}
Client:
for (int i = 0; i < 21; i++) {
if (write(sockfd_client, &temp[i], sizeof(long int)) == -1) {
exit(1);
}
}
I noticed that after calling the read the return value is exactly 8, which means that 8 bytes were read; yet the data is always 0. I don't understand what I am doing wrong. Also I was looking at various functions, and all cater to unsigned numbers but not signed (ntoh and hton family).
NOTE: Also I noticed that the first read() is 8 bytes, but the following ones are only 1 byte.
Whats the best way I can transmit all these numbers? (Also I noticed that the for loop refuses to quit if I have the read statement in there.)
Solution (The problem was the fact that the read was returning less bytes than required, This function solved it)
void read_long(int sockfd_client, char *num) {
unsigned int size = sizeof(long int);
int rlen = 0;
int ret;
while (rlen < size) {
if ((ret = read(sockfd_client, (num + rlen), size - rlen)) == -1) {
perror("read_long");
exit(1);
}
if (ret == 0) {
perror("socket closed before consumption");
exit(1);
}
rlen += ret;
}
}
[I'm going to repeat my comment as an answer, given that it turned out to be correct]
Note that calling read() with sizeof(num) returns up to sizeof(num) bytes. It might return fewer and it's your responsibility to accumulate them.
Similarly write() does not guarantee to write the requested number of bytes. You need to check the return value from write to see how many bytes were actually written, and then write the remaining bytes.
My friend and I have wrote a small download manager in C that splits the target file into several parts and downloads each part using a single posix thread. Everything seems to work fine, except that it is very slow compared to other download managers like wget (which as I know, does not split the file into several chunks). In every thread, we use a simple loop to download each part from a socket:
while ((nrecv = recv(sockfd, downbuf, sizeof(downbuf), 0)) > 0)
{
if ((nwrite = write(fd, downbuf, nrecv)) != nrecv)
die("write");
totalrw += nwrite;
}
/* ... */
I've tried with several different sizes for "downbuf", like 2014, 2048, 4096 and 8192, but with not much difference. It takes almost 45 seconds to download a 270 MB file, while wget downloads the same file in just 5 seconds. Both server and client are on the same host. Why is the difference so vast? Could you please tell me what trick wget uses?
This is how I make the request to the server:
sockfd = make_conn(website);
hdr.rq_buf = headerbuf; /* buffer to save response header */
hdr.rq_bufsize = sizeof(headerbuf);
hdr.rq_host = website;
hdr.rq_fpath = filepath; /* target file */
hdr.rq_flags = S_HEADFLAG; /* use head method at this moment
to get the total file size */
error = headerinit(hdr);
if (error)
{
die("headerinit()");
}
send(sockfd, headerbuf, strlen(headerbuf), 0); /* send the initial request */
recv(sockfd, respbuf, sizeof(respbuf), 0);
if (-1 == response_proc(respbuf, strlen(respbuf), &resp))
{
myperror("response_proc()");
exit(EXIT_FAILURE);
} /* process the header */
size_t sz = (size_t)strtol(resp.rs_content_length, NULL, 10);
divide(sz, chunks, numcons); /* divide the file into several parts */
for (int i = 0; i < numcons; i++)
{
/* populate data needed for threads */
args[i].t_hdr.rq_offset.c_start = chunks[i].c_start; /* where to start */
args[i].t_hdr.rq_offset.c_end = chunks[i].c_end; /* download up to this point */
args[i].t_hdr.rq_host = strdup(website);
args[i].t_hdr.rq_fpath = strdup(filepath);
snprintf(args[i].t_fname, BUFSIZ, "%sp%i", outfile, i);
args[i].t_order = i;
}
for (i = 0; i < numcons; i++)
{
if (0 != pthread_create(&threads[i], NULL, thread_main,
&args[i]))
{
die("pthread_create()");
}
}
for (i = 0; i < numcons; i++)
{
if (0 != pthread_join(threads[i], &thread_status))
{
die("pthread_join()");
}
}
http_request_header_t is defined as:
typedef struct {
void *rq_buf;
size_t rq_bufsize;
char *rq_host;
char *rq_fpath;
chunk_t rq_offset;
int rq_flags;
} http_request_header_t;
and http_response_header_t is defined as:
typedef struct {
#ifdef WITH_EXTRA_HEADERS
char *rs_version;
#endif
char *rs_status;
char *rs_date;
char *rs_server;
char *rs_last_modified;
char *rs_accept_ranges;
char *rs_content_length;
char *rs_connection;
char *rs_content_type;
} http_response_header_t;
This is the main routine that every thread use:
void *
thread_main(void *arg_orig)
{
thr_arg_t *arg = (thr_arg_t*)arg_orig;
int fd, sockfd;
http_response_header_t resp;
size_t totalrw = 0;
ssize_t nrecv;
char *line = malloc(BUFSIZ * sizeof(char));
char hdrbuf[BUFSIZ];
char respbuf[BUFSIZ];
mode_t mode = S_IRUSR | S_IWUSR | S_IRGRP;
ssize_t nwrite = 0;
void *downbuf = malloc(DOWNBUF * sizeof(char));
sockfd = make_conn(arg->t_hdr.rq_host);
fd = open(arg->t_fname, O_WRONLY | O_CREAT | O_TRUNC | O_EXCL, mode);
if (-1 == fd)
{
die("thread_open(): fd");
}
arg->t_hdr.rq_flags = S_OFFSET;
arg->t_hdr.rq_buf = hdrbuf;
arg->t_hdr.rq_bufsize = sizeof(hdrbuf);
headerinit(arg->t_hdr);
//printf("%s\n", arg->t_hdr.rq_buf);
sendn(sockfd, hdrbuf, strlen(hdrbuf), 0);
/* first, read the header */
while ((nrecv = readheader(sockfd, &line, BUFSIZ)) > 0)
{
strncpy(respbuf + nwrite, line, sizeof(respbuf) - nwrite);
nwrite += nrecv;
}
nwrite = 0;
//printf("\n\n%s\n\n", respbuf);
if (-1 == response_proc(respbuf, strlen(respbuf), &resp))
{
myperror("thread_response_proc()");
exit(EXIT_FAILURE);
}
if (strncmp(resp.rs_status, "416", 3) == 0)
{
fprintf(stderr, "Partial content is not supported by the server\n");
exit(EXIT_FAILURE);
}
/* now read the actual data */
while ((nrecv = recv(sockfd, downbuf, sizeof(downbuf), 0)) > 0)
{
if ((nwrite = write(fd, downbuf, nrecv)) != nrecv)
die("write");
totalrw += nwrite;
}
if(-1 == nrecv)
{
die("recv()");
}
close(sockfd);
close(fd);
idxwr(arg->t_fname, arg->t_order, totalrw);
return ((void*)0);
}
You haven't posted enough here, but usually the cause of an unexpected slowdown in TCP will be Nagle's algorithm. This is triggered when you write small chunks of data to a socket. These would be inefficient to put on the wire on their own so the TCP stack waits for the user program to add more data before it sends out the packet. Only if nothing is added for 'a while' does it actually send an incomplete packet.
This can be disabled, but as your aim is an efficient bulk transfer you probably shouldn't.