C socket - not all socket receive in select() - c

i've a problem wiht my multi-threads project: i create some threads, these must send and receive udp messages with each other, so each thread is listening on multiple sockets with an select(). The problem is: only last socket inserted in a fd_set variable, receive messages. I don't konw why
this is code:
initial part of thread:
int fdmax;
fd_set read_fd_set, service_fd_set;
/* allocate memory for receive msg */
msg=malloc(sizeof(char)*(SIZEBUF));
FD_ZERO(&read_fd_set);
FD_ZERO(&service_fd_set);
fdmax=0;
if (param->n_port != 0){
for (x=0; x<(param->n_port); x++){
/* take the port from array */
local_port_number = param->l_port_in[x];
/* create socket udp and bind on localhost */
socketfd=create_socket(local_port_number);
/* save socket_fd in my data struct */
param->sock_fd_local[x]=socketfd;
/* add socket in fd_set */
FD_SET(socketfd,&service_fd_set);
if (socketfd > (fdmax-1)){
fdmax=socketfd + 1;
}
secondo part of thread:
for(;;){
read_fd_set=service_fd_set;
ris=select(fdmax,&read_fd_set,NULL,NULL,NULL);
if(ris<0){
if (errno!=EINTR){
printf(_KRED "Error in select: errno different from EINTR \n" _KNRM);
}
}
if (ris>0){
for(p=0; p<fdmax; p++){
if((FD_ISSET(p,&read_fd_set))!=0){
for( x=1; x<=5; x++){
if( p == param->sock_fd_local[x]){
/* setup datagram to receive */
memset(&From, 0, sizeof(From));
Fromlen=sizeof(struct sockaddr_in);
/* RECVFROM */
msglen = recvfrom ( p, msg, (int)SIZEBUF, 0, (struct sockaddr*)&From, &Fromlen);
if (msglen<0){
...
}else{
sprintf((char*)string_remote_ip_address,"%s",inet_ntoa(From.sin_addr));
remote_port_number = ntohs(From.sin_port);
print_msg(...);
}
}
}
}
}
}
}
anyone can help me?
The entire project is here: https://github.com/bonfi/SpanningTreeUDP
(sorry for my english, i'm italian, the project is commented in italian)

I believe there are is at least one if not two issues. The first one is in the second part, in the recvfrom function, you are using socketfd but not setting it to the socket from which you want to receive data. The variable socketfd contains the last socket you created from the initial code. I suspect that is why you are only getting data from one thread which was the last socket created.
The other item which I don't quite understand is the loop after the if statement with teh FD_ISSET in the second part. I'm not sure why you are looping with x and then setting p or why you are doing 4 times.
I would propose the code in the second part should look like the follow starting at the FD_ISSET if statement.
if((FD_ISSET(p,&read_fd_set))!=0){
socketfd == param->sock_fd_local[p]){
/* setup datagram to receive */
memset(&From, 0, sizeof(From));
Fromlen=sizeof(struct sockaddr_in);
/* RECVFROM */
msglen = recvfrom ( socketfd, msg, (int)SIZEBUF, 0, (struct sockaddr*)&From, &Fromlen);
if (msglen<0){
...
}else{
sprintf((char*)string_remote_ip_address,"%s",inet_ntoa(From.sin_addr));
remote_port_number = ntohs(From.sin_port);
print_msg(...);
}
}
I removed the for loop with the x variable and I am using p to index the structure to set the socketfd for receiving data based on if the FD_ISSET returns true. That way you are setting the socketfd variable based on the file descriptor if it is set.

Related

TCP Server workers with kqueue

I recently did some testing with kernel events and I came up with the following:
Does it make sense to use a kernel event for accepting sockets? My testing showed that I was only able to handle one accept at once (even if the eventlist array is bigger)(Makes sense to me cause .ident == sockfd is only true for one socket).
I thought the use of kevent is mainly to read from multiple sockets at once. Is that true?
Is this how a TCP server is done with a kqueue implementation? :
Listening Thread (without kqueue)
Accepts new connections and adds FD to a worker kqueue.
QUESTION: Is this even possible? My testing showed yes, but is it guaranteed that the worker thread will be aware of the changes and is kevent really thread safe?
Worker thread (with kqueue)
Waits on reads on file descriptors added from the listening thread.
QUESTION: How many sockets at once would make sense to check for updates?
Thanks
This is not really an answer but I made a little server script with kqueue explaining the problem:
#include <stdio.h> // fprintf
#include <sys/event.h> // kqueue
#include <netdb.h> // addrinfo
#include <arpa/inet.h> // AF_INET
#include <sys/socket.h> // socket
#include <assert.h> // assert
#include <string.h> // bzero
#include <stdbool.h> // bool
#include <unistd.h> // close
int main(int argc, const char * argv[])
{
/* Initialize server socket */
struct addrinfo hints, *res;
int sockfd;
bzero(&hints, sizeof(hints));
hints.ai_family = AF_INET;
hints.ai_socktype = SOCK_STREAM;
assert(getaddrinfo("localhost", "9090", &hints, &res) == 0);
sockfd = socket(AF_INET, SOCK_STREAM, res->ai_protocol);
assert(sockfd > 0);
{
unsigned opt = 1;
assert(setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)) == 0);
#ifdef SO_REUSEPORT
assert(setsockopt(sockfd, SOL_SOCKET, SO_REUSEPORT, &opt, sizeof(opt)) == 0);
#endif
}
assert(bind(sockfd, res->ai_addr, res->ai_addrlen) == 0);
freeaddrinfo(res);
/* Start to listen */
(void)listen(sockfd, 5);
{
/* kevent set */
struct kevent kevSet;
/* events */
struct kevent events[20];
/* nevents */
unsigned nevents;
/* kq */
int kq;
/* buffer */
char buf[20];
/* length */
ssize_t readlen;
kevSet.data = 5; // backlog is set to 5
kevSet.fflags = 0;
kevSet.filter = EVFILT_READ;
kevSet.flags = EV_ADD;
kevSet.ident = sockfd;
kevSet.udata = NULL;
assert((kq = kqueue()) > 0);
/* Update kqueue */
assert(kevent(kq, &kevSet, 1, NULL, 0, NULL) == 0);
/* Enter loop */
while (true) {
/* Wait for events to happen */
nevents = kevent(kq, NULL, 0, events, 20, NULL);
assert(nevents >= 0);
fprintf(stderr, "Got %u events to handle...\n", nevents);
for (unsigned i = 0; i < nevents; ++i) {
struct kevent event = events[i];
int clientfd = (int)event.ident;
/* Handle disconnect */
if (event.flags & EV_EOF) {
/* Simply close socket */
close(clientfd);
fprintf(stderr, "A client has left the server...\n");
} else if (clientfd == sockfd) {
int nclientfd = accept(sockfd, NULL, NULL);
assert(nclientfd > 0);
/* Add to event list */
kevSet.data = 0;
kevSet.fflags = 0;
kevSet.filter = EVFILT_READ;
kevSet.flags = EV_ADD;
kevSet.ident = nclientfd;
kevSet.udata = NULL;
assert(kevent(kq, &kevSet, 1, NULL, 0, NULL) == 0);
fprintf(stderr, "A new client connected to the server...\n");
(void)write(nclientfd, "Welcome to this server!\n", 24);
} else if (event.flags & EVFILT_READ) {
/* sleep for "processing" time */
readlen = read(clientfd, buf, sizeof(buf));
buf[readlen - 1] = 0;
fprintf(stderr, "bytes %zu are available to read... %s \n", (size_t)event.data, buf);
sleep(4);
} else {
fprintf(stderr, "unknown event: %8.8X\n", event.flags);
}
}
}
}
return 0;
}
Every time a client sends something the server experiences a "lag" of 4 seconds. (I exaggerated a bit, but for testing quite reasonable). So how do get around to that problem? I see worker threads (pool) with own kqueue as possible solution, then no connection lag would occur. (each worker thread reads a certain "range" of file descriptors)
Normally, you use kqueue as an alternative to threads. If you're going to use threads, you can just set up a listening thread and a worker threadpool with one thread per accepted connection. That's a much simpler programming model.
In an event-driven framework, you would put both the listening socket and all the accepted sockets into the kqueue, and then handle events as they occur. When you accept a socket, you add it to the kqueue, and when a socket handler finishes it works, it could remove the socket from the kqueue. (The latter is not normally necessary because closing a fd automatically removes any associated events from any kqueue.)
Note that every event registered with a kqueue has a void* userdata, which can be used to identify the desired action when the event fires. So it's not necessary that every event queue have a unique event handler; in fact, it is common to have a variety of handlers. (For example, you might also want to handle a control channel set up through a named pipe.)
Hybrid event/thread models are certainly possible; otherwise, you cannot take advantage of multicore CPUs. One possible strategy is to use the event queue as a dispatcher in a producer-consumer model. The queue handler would directly handle events on the listening socket, accepting the connection and adding the accepted fd into the event queue. When a client connection event occurs, the event would be posted into the workqueue for later handling. It's also possible to have multiple workqueues, one per thread, and have the accepter guess which workqueue a new connection should be placed in, presumably on the basis of that thread's current load.

Knowing which file-descriptors are ready after select() call in C

I'm new to socket programming and I've been introduced to the select() system call. My question is, lets say I'm writing a server in C (which I am attempting to do) and I want to use the select() call in my implementation for practice. I'm trying to write a server that receives information from a client, so my approach is to use select(), followed by read() and just output the information.
According to the documentation I've read select() returns the number of file descriptors in the input set which are ready for i/o. My question is, how do know which file descriptors in the original set are the ones that are ready for i/o? I can't seem to find this in my searches or examples I've looked at for the past while.
Let's say my code looks like the below:
int main() {
/* Create socket/server variables */
int select_value;
int this_socket;
int maxfd;
struct sockadder_in address;
fd_set allset;
/* Bind the socket to a port */
main_socket = socket(AF_INET, SOCK_STREAM, 0);
if (main_socket < 0) {
perror("socket()");
exit(1);
}
Connect(main_socket, (struct sockaddr *)&address, sizeof(address));
/* Add the socket to the list of fds to be monitored */
FD_ZERO(&allset);
FD_SET(main_socket, &allset);
fd_set read_ready = allset;
fd_set write_ready = allset;
while (1) {
/* Listen for a connection */
/* Accept a connection */
select_value = Select(maxfd+1, &read_ready, &write_ready, NULL, NULL);
if (select_value == -1) {
perror("select()");
exit(1);
}
else if(select_value > 0) {
/* How to access i/o ready file descriptors
now that we know there are some available? */
}
}
}
One can do this using the FD_ISSET macro that is part of <sys/select.h>.
When your select unblocks and a file descriptor is ready, you can test all of your file descriptors using the FD_ISSET macro in a simple loop. This can be translated to the following sample :
for (i = 0; i < FD_SETSIZE; ++i) {
if (FD_ISSET (i, &read_fd_set)) {
if (i == bound_socket) {
// A new client is waiting to be accepted
new = accept(sock, (struct sockaddr *) &clientname, &size);
// ...
FD_SET (new, &active_fd_set);
}
else {
// There is something to be read on the file descriptor.
data = read_from_client_on(i);
}
}
}
Of course, this is just sample which is obviously lacking any error handling, which you should handle in your application.

recvfrom() error 10035 using non-blocking sockets

I am using ioctlsocket() function to make my socket non-blocking but when I call recvfrom(), I get the error 10035 (WSAEWOULDBLOCK).
u_long mode = 1;
ioctlsocket(newSocketIdentifier, FIONBIO, &mode);
while(1)
{
if((recv_len = recvfrom(newSocketIdentifier, receiveBuffer, sizeof(receiveBuffer), 0, (struct sockaddr *) &clientSocket, &clientSocketLength)) == SOCKET_ERROR)
{
char err[128];
itoa(WSAGetLastError(),err,10);
MessageBox( NULL,"Could not Receive Data",err,MB_ICONINFORMATION);
BREAK;
}
}
Can anybody explain why this happens? :(
This is normal if no data is available. The code is WSAEWOULDBLOCK (see this table) and means, that on a blocking port the function would have to sit and wait until it could be served.
while(1)
{
if((recv_len = recvfrom(newSocketIdentifier, receiveBuffer, sizeof(receiveBuffer), 0, (struct sockaddr *) &clientSocket, &clientSocketLength)) == SOCKET_ERROR)
{
int ierr= WSAGetLastError();
if (ierr==WSAEWOULDBLOCK) { // currently no data available
Sleep(50); // wait and try again
continue;
}
// Other errors
char err[128];
itoa(ierr,err,10);
MessageBox( NULL,"Could not Receive Data",err,MB_ICONINFORMATION);
break;
}
}
I cannot agree that this is "normal" like posted above.
In your call of recvfrom you will receive an error in recv_len. I recommend to check that value - it will be SOCKET_ERROR and by calling WSAGetLastErrorenter you will see error WSAEWOULDBLOCK.
I am not a (Windows) socket expert, but based on my tests, I cannot use combination of ioctlsocket and recvfrom for receiving data via UDP in non-blocking mode (I did the same thing like you in your example).
I am planning to use combination of select and recvfrom with minimum possible timeout (1us). I do not know any other possibly better solution now.
Note: you should check also the return value of ioctlsocket for possible error.
I will provide my code sample later today.
UPDATE (adding code as promised):
/* define list of sockets for function select(..) */
fd_set readfds;
/* define timeout for function select(..) */
TIMEVAL tv;
/* timeout: 1us */
tv.tv_usec = 1;
/* timeout: 0s */
tv.tv_sec = 0;
/* just 1 socket is used */
readfds.fd_count = 1;
readfds.fd_array[0] = receivingSocket;
/* determine the status of one or more sockets with timeout */
int selectReturnValue = select(0, &readfds, 0, 0, &tv);
/* check return value of the call of function select(..) */
switch (selectReturnValue)
{
/* select(..) function timeout */
case 0:
/* time limit expired */
break;
/* select(..) function error */
case SOCKET_ERROR:
/* check the error status for the last windows sockets operation */
selectError(WSAGetLastError());
break;
/* no timeout and no error */
default:
/* receive data from UDP */
resultOfrecvfrom = recvfrom(receivingSocket, receivingBuffer, sizeof(receivingBuffer), 0, (SOCKADDR *)&serverReceptionInfo, &serverReceptionInfoLength);
/* check result of call of recvfrom(..) */
switch (resultOfrecvfrom)
{
/* connection has been gracefully closed */
case 0:
/* socket was closed */
break;
/* socket error occurred during last call of socket operation */
case SOCKET_ERROR:
/* check the error status for the last Windows Sockets operation */
recvfromError(WSAGetLastError());
break;
/* resultOfrecvfrom amount of data received */
default:
/* ... add your code here */
break;
}
break;
}
I just copied crucial part of my code if more is needed, let me know in comments.

C - Using poll to multiplex between socket(s) and stdin - Server

I'm writing a client server application and I'm using poll to multiplex between several client sockets and stdin, where I can insert commands (example: stop the server). I believe the structure (the "logic") of my code is correct, however it's not behaving the way I expect it to:
struct pollfd pfd[NSERVER]; //defined as 10
pfd[0].fd = fileno(stdin);
pfd[0].events = POLLIN;
pfd[1].fd = socktfd; //server bind, listen socket
pfd[1].events = POLLIN;
struct sockaddr_storage remoteaddr; // client address
socklen_t addrlen;
char remoteIP[INET6_ADDRSTRLEN];
addrlen = sizeof remoteaddr;
char buf[1024]; // buffer
int pos=2;
while(poll(pfd,1,0) >= 0)
{
if(pfd[0].revents & POLLIN) { //stdin
//process input and perform command
}
if(pfd[1].revents & POLLIN) {
/* new connection */
int connsockfd = accept(socktfd, (struct sockaddr *)&remoteaddr,&addrlen);
pfd[pos].fd=connsockfd;
}
int i=2;
//Loop through the fd in pfd for events
while (i<=NSERVER)
{
if (pfd[i].revents & POLLIN) {
int c=recv(pfd[i].fd, buf, sizeof buf, 0);
if(c<=0) {
if (c==0)
{
/* Client closed socket */
close(pfd[i].fd);
}
}else
{//Client sent some data
c=send(pfd[i].fd,sbuff,z,0);
if (c<=0)
{
Error;
}
free(sbuff);
}
}
i++;
}
}
I've removed some code inside the recv and send to make the code easier to read.
It fails to behave (it just hangs, doesn't accept connections or reacts to input from stdin).
Note: I would prefer to use poll over select, so please don't point to select :-).
Thanks in advance for any assistance.
you should set every pfd[i].fd = -1, so they get ignored initially by poll().
poll(pfd, 1, 0) is wrong and should at least be poll(pfd, 2, 0) or even poll(pfd, NSERVER, 0).
while(i<=NSERVER) should be while(i<NSERVER)
Your program probably hangs, because you loop through the pfd array, which is not initialized and containes random values for .fd and .revents, so it wants to send() or recv() on some random FD which might block. Do if(pdf[i].fd < 0) {i++; continue;} in the i<NSERVER loop.
You also don't set pfd[pos].events = POLLIN on newly accepted sockets. Don't set POLLOUT unless you have something to send, because it will trigger almost every time.

Boss Worker Pthreads Web Server in C - Server crashes if more requests sent than number of threads

I'm writing a web server in C (which I suck with) using Pthreads (which I suck with even more) and I'm stuck at this point. The model for the server is boss-worker so the boss thread instantiates all worker threads at the beginning of the program. There is a global queue that stores the socket of the incoming connection(s). The boss thread is the one that adds all items (sockets) to the queue as the connections are accepted. All of the worker threads then wait for an item to be added to a global queue in order for them to take up the processing.
The server works fine as long as I connect to it less times than the number of worker threads that the server has. Because of that, I think that either something is wrong with my mutexes (maybe the signals are getting lost?) or the threads are being disabled after they run once (which would explain why if there are 8 threads, it can only parse the first 8 http requests).
Here is my global queue variable.
int queue[QUEUE_SIZE];
This is the main thread. It creates a queue struct (defined elsewhere) with methods enqueue, dequeue, empty, etc. When the server accepts a connection, it enqueues the socket that the incoming connection is on. The worker threads which were dispatched at the beginning are constantly checking this queue to see if any jobs have been added, and if there are jobs, then they dequeue the socket, connect to that port, and read/parse/write the incoming http request.
int main(int argc, char* argv[])
{
int hSocket, hServerSocket; /* handle to socket */
struct hostent* pHostInfo; /* holds info about a machine */
struct sockaddr_in Address; /* Internet socket address stuct */
int nAddressSize = sizeof(struct sockaddr_in);
int nHostPort;
int numThreads;
int i;
init(&head,&tail);
//**********************************************
//ALL OF THIS JUST SETS UP SERVER (ADDR STRUCT,PORT,HOST INFO, ETC)
if(argc < 3) {
printf("\nserver-usage port-num num-thread\n");
return 0;
}
else {
nHostPort=atoi(argv[1]);
numThreads=atoi(argv[2]);
}
printf("\nStarting server");
printf("\nMaking socket");
/* make a socket */
hServerSocket=socket(AF_INET,SOCK_STREAM,0);
if(hServerSocket == SOCKET_ERROR)
{
printf("\nCould not make a socket\n");
return 0;
}
/* fill address struct */
Address.sin_addr.s_addr = INADDR_ANY;
Address.sin_port = htons(nHostPort);
Address.sin_family = AF_INET;
printf("\nBinding to port %d\n",nHostPort);
/* bind to a port */
if(bind(hServerSocket,(struct sockaddr*)&Address,sizeof(Address)) == SOCKET_ERROR) {
printf("\nCould not connect to host\n");
return 0;
}
/* get port number */
getsockname(hServerSocket, (struct sockaddr *) &Address,(socklen_t *)&nAddressSize);
printf("Opened socket as fd (%d) on port (%d) for stream i/o\n",hServerSocket, ntohs(Address.sin_port));
printf("Server\n\
sin_family = %d\n\
sin_addr.s_addr = %d\n\
sin_port = %d\n"
, Address.sin_family
, Address.sin_addr.s_addr
, ntohs(Address.sin_port)
);
//Up to this point is boring server set up stuff. I need help below this.
//**********************************************
//instantiate all threads
pthread_t tid[numThreads];
for(i = 0; i < numThreads; i++) {
pthread_create(&tid[i],NULL,worker,NULL);
}
printf("\nMaking a listen queue of %d elements",QUEUE_SIZE);
/* establish listen queue */
if(listen(hServerSocket,QUEUE_SIZE) == SOCKET_ERROR) {
printf("\nCould not listen\n");
return 0;
}
while(1) {
pthread_mutex_lock(&mtx);
printf("\nWaiting for a connection");
while(!empty(head,tail)) {
pthread_cond_wait (&cond2, &mtx);
}
/* get the connected socket */
hSocket = accept(hServerSocket,(struct sockaddr*)&Address,(socklen_t *)&nAddressSize);
printf("\nGot a connection");
enqueue(queue,&tail,hSocket);
pthread_mutex_unlock(&mtx);
pthread_cond_signal(&cond); // wake worker thread
}
}
Here is the worker thread. This should be always running checking for new requests (by seeing if the queue is not empty). At the end of this method, it should be deferring back to the boss thread to wait for the next time it is needed.
void *worker(void *threadarg) {
pthread_mutex_lock(&mtx);
while(empty(head,tail)) {
pthread_cond_wait(&cond, &mtx);
}
int hSocket = dequeue(queue,&head);
unsigned nSendAmount, nRecvAmount;
char line[BUFFER_SIZE];
nRecvAmount = read(hSocket,line,sizeof line);
printf("\nReceived %s from client\n",line);
//***********************************************
//DO ALL HTTP PARSING (Removed for the sake of space; I can add it back if needed)
//***********************************************
nSendAmount = write(hSocket,allText,sizeof(allText));
if(nSendAmount != -1) {
totalBytesSent = totalBytesSent + nSendAmount;
}
printf("\nSending result: \"%s\" back to client\n",allText);
printf("\nClosing the socket");
/* close socket */
if(close(hSocket) == SOCKET_ERROR) {
printf("\nCould not close socket\n");
return 0;
}
pthread_mutex_unlock(&mtx);
pthread_cond_signal(&cond2);
}
Any help would be greatly appreciated. I can post more of the code if anyone needs it, just let me know. I'm not the best with OS stuff, especially in C, but I know the basics of mutexes, cond. variables, semaphores, etc. Like I said, I'll take all the help I can get. (Also, I'm not sure if I posted the code exactly right since this is my first question. Let me know if I should change the formatting at all to make it more readable.)
Thanks!
Time for a workers' revolution.
The work threads seem to be missing a while(true) loop. After the HTTP exchange and closing the socket, they should be looping back to wait on the queue for more sockets/requests.

Resources