All below is from man epoll page:
The function do_use_fd() uses the new
ready file descriptor
until EAGAIN is returned by either read(2) or write(2).
Code example for ET triggered :
for(;;) {
nfds = epoll_wait(kdpfd, events, maxevents, -1);
for(n = 0; n < nfds; ++n) {
if(events[n].data.fd == listener) {
client = accept(listener, (struct sockaddr *) &local,
&addrlen);
if(client < 0){
perror("accept");
continue;
}
setnonblocking(client);
ev.events = EPOLLIN | EPOLLET;
ev.events = EPOLLIN | EPOLLET;
ev.data.fd = client;
if (epoll_ctl(kdpfd, EPOLL_CTL_ADD, client, &ev) < 0) {
fprintf(stderr, "epoll set insertion error: fd=%d\n",
client);
return -1;
}
}
else
do_use_fd(events[n].data.fd);
}
}
So for read/write operation,we should do it by looping until a EAGAIN is received;but why it's not the case for accept?
IMO the above code will miss some requests when there're multiple client sockets waiting to be accepted, as it accepts only 1 client socket, we should also wrap it in a loop until EAGAIN received.
Or maybe is there something I'm missing?
Look at how the listener socket is added to the epollfd:
ev.events = EPOLLIN; // this is the crucial bit
ev.data.fd = listen_sock;
It's not added in edge-triggered, it's added in level-triggered. So no need for a loop until EAGAIN on that one.
Related
I have a server that has 2 ports with a socket for each port, each of them doing different thigs. I made this work with epoll but I have a question, can 2 different clients connect to each port and use the functionality offered by them at the same time? For example: client1 connects to port X and wants to read data and client2 connects at the same time and wants to write data at the port Y. Is this possible? Thanks!
// create the 2 sockets for each port
// bind(), listen() for each
// tried to do this 'parallelism' using epoll but after someone connects
// to a port that port takes over, the other can't be used
epollfd = epoll_create1(0);
if (epollfd == -1) {
perror("epoll_create1");
exit(EXIT_FAILURE);
}
ev.events = EPOLLIN;
ev.data.fd = sd;
if (epoll_ctl(epollfd, EPOLL_CTL_ADD, sd, &ev) == -1) {
perror("epoll_ctl: listen_sock");
exit(EXIT_FAILURE);
}
ev2.events = EPOLLIN;
ev2.data.fd = sd2;
if (epoll_ctl(epollfd, EPOLL_CTL_ADD, sd2, &ev2) == -1) {
perror("epoll_ctl: listen_sock");
exit(EXIT_FAILURE);
}
for (;;) {
nfds = epoll_wait(epollfd, events, MAX_EVENTS, -1);
if (nfds == -1) {
perror("epoll_wait");
exit(EXIT_FAILURE);
}
for (int n = 0; n < nfds; ++n) {
if (events[n].data.fd == sd2) {
// handle the clients at PORT2 where I just show data from
// database, I'm using processes to handle clients
}
else if (events[n].data.fd == sd) {
// handle the clients at PORT1 where they can update/add/delete
// from data base, I'm using processes to handle clients
}
Recently, I am study system call epoll. I already have basic concept about this topic, but I stuck in example given by manual.
/* Code to set up listening socket, 'listen_sock',
(socket(), bind(), listen()) omitted */
epollfd = epoll_create1(0);
if (epollfd == -1) {
perror("epoll_create1");
exit(EXIT_FAILURE);
}
ev.events = EPOLLIN;
ev.data.fd = listen_sock;
if (epoll_ctl(epollfd, EPOLL_CTL_ADD, listen_sock, &ev) == -1) {
perror("epoll_ctl: listen_sock");
exit(EXIT_FAILURE);
}
for (;;) {
nfds = epoll_wait(epollfd, events, MAX_EVENTS, -1);
if (nfds == -1) {
perror("epoll_wait");
exit(EXIT_FAILURE);
}
for (n = 0; n < nfds; ++n) {
if (events[n].data.fd == listen_sock) {
conn_sock = accept(listen_sock,
(struct sockaddr *) &addr, &addrlen);
if (conn_sock == -1) {
perror("accept");
exit(EXIT_FAILURE);
}
setnonblocking(conn_sock);
ev.events = EPOLLIN | EPOLLET;
ev.data.fd = conn_sock;
if (epoll_ctl(epollfd, EPOLL_CTL_ADD, conn_sock,
&ev) == -1) {
perror("epoll_ctl: conn_sock");
exit(EXIT_FAILURE);
}
} else {
do_use_fd(events[n].data.fd);
}
}
}
Following is my assumption:
Above is server code.
listen_sock is file descriptor return from socket() in server.
My question is:
What did file descriptor events[n].data.fd mean? Is server's fd or client's?
Why if events[n].data.fd == listen_sock , then we need to create a new connection?
You can look at the man page for epoll_wait for details on struct epoll_event and its data member and to help make sense of the code.
Any given events[n].data.fd refers to an fd that had some kind of event on it. It could either be the listening socket, or it could be a client socket if there are active client connections.
If the fd with activity is the listen_sock listening socket, this means that a new client is attempting a connection. At this point we accept the connection and add the new conn_sock fd to the set of epoll's fds.
I am building a server/client application. The server will need to deal with less than 1000+ clients. I currently have a simple EPOLL implementation to be able to receive information from the clients to the server. However, I will need to be able to send tasks to specific clients as well. My question is there a way to use EPOLL to identify which clients I need to send a task too (potentially using the EPOLLOUT flag) and send the message out. I've attached the snippet of what I currently have. If possible, how would I go about implementing this. It would be great to be able to have one epoll for all the sending an the receiving if it could work like that.
Thanks for any help/recommendations!
void epoll(int listening_port)
{
char buffer[500]; //buffer for message
int listen_sock = 0; //file descriptor (fd) for listening socket
int conn_sock = 0; //fd for connecting socket
int epollfd = 0; // fd for epoll
int nfds = 0; //number of fd's ready for i/o
int i = 0; //index to which file descriptor we are lookng at
int curr_fd = 0; //fd for socket we are currently looking at
bool loop = 1; //boolean value to help identify whether to keep in loop or not
socklen_t address_len;
struct sockaddr_in serv_addr;
struct epoll_event ev, events[EPOLL_MAX_EVENTS];
ssize_t result = 0;
bzero(buffer, sizeof(buffer));
bzero(&serv_addr, sizeof(serv_addr));
serv_addr.sin_family = AF_INET;
serv_addr.sin_port = listening_port;
serv_addr.sin_addr.s_addr = INADDR_ANY;
listen_sock = create_socket();
if(bind(listen_sock, SA &serv_addr, sizeof(serv_addr)) != 0)
{
perror("Bind failed");
}
else
{
printf("Bind successful\n");
}
set_socket_nonblocking(listen_sock);
listen_on_socket(listen_sock, SOMAXCONN); //specifying max connections in backlog
epollfd = initialize_epoll();
ev.events = EPOLLIN | EPOLLOUT | EPOLLET | EPOLLRDHUP;
ev.data.fd = listen_sock;
if(epoll_ctl(epollfd, EPOLL_CTL_ADD, listen_sock, &ev) == ERROR)
{
perror("Epoll_ctl: listen sock");
}
else
{
printf("Successfully added listen socket to epoll\n");
}
while (RUN_EPOLL)
{
nfds = epoll_wait(epollfd, events, EPOLL_MAX_EVENTS, 0); //waiting for incoming connection;
if(nfds == ERROR)
{
perror("EPOLL_Wait");
}
//printf("Finished waiting\i");
for(i = 0; i < nfds; ++i)
{
curr_fd = events[i].data.fd;
loop = true; //reset looping flag
//Notification from Listening Socket - Process Incoming Connections
if (curr_fd == listen_sock) {
while(loop)
{
conn_sock = accept(listen_sock, SA &serv_addr, &address_len); //accept incoming connection
printf("Accepted new incoming connection - socket fd: %d\n", conn_sock);
if (conn_sock > 0) //if successful set socket nonblocking and add it to epoll
{
set_socket_nonblocking(conn_sock);
ev.events = EPOLLIN | EPOLLOUT| EPOLLET | EPOLLRDHUP; //setting flags
ev.data.fd = conn_sock; //specify fd of new connection in event to follow
if (epoll_ctl(epollfd, EPOLL_CTL_ADD, conn_sock, &ev) == ERROR) //add fd to monitored fd's
{
perror("epoll_ctl: conn_sck");
}
else
{
printf("Added %d to monitor list\n", conn_sock);
}
}
else if (conn_sock == ERROR)
{
if ((errno == EAGAIN) || (errno == EWOULDBLOCK))
{
printf("All incoming connections processed\n");
loop = false;
}
else
{
perror("Accept remote socket");
loop = false;
}
}
}
}
else if(events[i].events & EPOLLRDHUP) //detecting if peer shutdown
{
printf("Detected socket peer shutdown. Closing now. \n");
if (epoll_ctl(epollfd, EPOLL_CTL_DEL, curr_fd, NULL) == ERROR) {
perror("epoll_ctl: conn_sck");
}
close_socket(curr_fd);
}
else if(events[i].events & EPOLLIN)
{
while(loop)
{
result = recv(curr_fd, buffer, sizeof(buffer), 0);
//printf("Length of incoming message is %d\i", result);
if(result > 0) //
{
printf("File Descriptor: %d. Message: %s\n", curr_fd, buffer);
bzero(buffer, sizeof(buffer));
}
else if(result == ERROR) //Message is completely sent
{
if(errno == EAGAIN)
{
send_message(curr_fd, "Success", strlen("Success"));
loop = false;
}
}
else if(result == 0)
{
//Removing the fd from the monitored descriptors in epoll
if (epoll_ctl(epollfd, EPOLL_CTL_DEL, curr_fd, NULL) == ERROR) {
perror("epoll_ctl: conn_sck");
}
close_socket(curr_fd); //Closing the fd
loop = false;
}
}
}
}
}
close_socket(listen_sock);
//need to develop way to gracefully close out of epoll
return;
}
I have some code that just tests if a port is open on a device, for that I made a little timeout socket function:
int timeout_socket(struct sockaddr *addr, int timeout_ms) {
int fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (fd < 0) return 0;
int on = 1;
setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &on, sizeof(int));
//Set the socket for non-blocking I/O
if (ioctl(fd, FIONBIO, (char *)&on) < 0) {
close(fd);
return 0;
}
int result = connect(fd, addr, sizeof(struct sockaddr_in));
if (result != 0 && errno != EINPROGRESS) {
close(fd);
return 0;
}
struct pollfd fds;
fds.fd = fd;
fds.events = POLLOUT;
//Poll for timeout_ms
while (1==1) {
int res = poll(&fds, 1, timeout_ms);
if (res == EINTR) continue;
if (fds.revents & POLLOUT || fds.revents & POLLIN) {
close(fd);
return 1;
}
break;
}
close(fd);
return 0;
}
The problem is that, when the target device (a Mac) is sleeping it wakes up just after the connect method runs, but despite the timeout_ms being something like 10000 (10secs) it just doesn't respond.
My possible fix is:
Connect to the device using a socket/connect
Close it
Open/Connect another socket
Poll for timeout_ms
Is this the only way? This behavior seems strange to me, but I have never used posix sockets with non-blocking before. Is this normal behavior?
I'm trying to figure out what is blocking my program. I'm running a server that uses POSIX threads. I have to for my computer programming lab. The main function listens for new connections. Once it accepts a connection, it creates a new thread by passing the FD to the thread. I'm able to successfully connect to the server using multiple telnet/client connections. I can send data to the server successfully once, but if I try sending again the server won't do anything.
Part of the main function
int active_thread = 0;
//The Running loop
while(running)
{
if(active_thread > NUMBTHREADS)
{
printf("Unable to accept client connection! Threads are all used up");
running = false;
}
else
{
if(FD_ISSET(sockfd, &readfds))
{
if((bindfd[active_thread] = accept(sockfd, (struct sockaddr *) &client_addr, &client_sock_size)) == -1)
{
fprintf(stderr, "Unable to accept client \n");
perror("What");
break;
}
activethreads[active_thread] = pthread_create( &threads[active_thread], NULL, server_handler, (void*) &bindfd[active_thread]);
//close(bindfd[active_thread]);
//pthread_join( threads[active_thread], NULL);
active_thread++;
//running = false;
}
}
}
close(sockfd);
return 0;
}
Part of the POSIX THREAD CODE
void *server_handler( void *sockfd)
{
int bindfd = *( (int *) sockfd);
char buffer[MESSAGELENGTH];
bool running = true;
printf("Thread was created successfully\n");
char intro[] = "Successfully Connected to server!\n";
struct pollfd pfd;
pfd.fd = bindfd;
pfd.events = POLLIN;
if ( (send(bindfd, intro, strlen(intro), 0)) < 0)
{
perror("Unable to send");
}
while(running){
char msg[] = "\nYou have the following options!\n1) Insert an integer: insert <integer>\n2) Remove An Integer: remove <integer>\n3) Get number of integers in list: get_count\n4) Get first integer: get_first\n5) Get last integer: get_last\n6) Quit program: quit\n ";
if ( (send(bindfd, msg, strlen(msg), 0)) < 0)
{
perror("Unable to send");
}
memset(&buffer, 0, MESSAGELENGTH);
if (recv(bindfd, buffer, MESSAGELENGTH, 0) > 0)
{
//SOme other code
}
}
I think its blocking at either the accept or recv. I've heard of select() and various other methods, but I'm having difficulty trying to implement them. Thanks!
The root cause of your issue appears to be that you are unconditionally executing close(sockfd); return 0; at the bottom of your while (running) loop, which means that the loop only ever executes once.
Additionally, you should not be using FD_ISSET() unless you are also using select(). Your main loop should look something more like:
int active_thread = 0;
while (active_thread < NUMBTHREADS)
{
if((bindfd[active_thread] = accept(sockfd, (struct sockaddr *) &client_addr, &client_sock_size)) == -1)
{
fprintf(stderr, "Unable to accept client \n");
perror("What");
break;
}
activethreads[active_thread] = pthread_create( &threads[active_thread], NULL, server_handler, (void*) &bindfd[active_thread]);
active_thread++;
}
if (active_thread >= NUMBTHREADS)
{
printf("Unable to accept client connection! Threads are all used up.\n");
}
running = false;
close(sockfd);
return 0;
By default network sockets are blocking. You need to set the O_NONBLOCK flag on the socket.
if(fcntl(fd, F_GETFL, &flags) < 0 ||
fcntl(fd, F_SETFL, flags | O_NONBLOCK) < 0)
perror("Failed to set socket as non-blocking");
Now, instead of blocking when there is no input (or buffer space to store output), the error EAGAIN (or EWOUDLBLOCK) is returned. Lastly, you will need to use select() or poll() when you have nothing else to do but wait on I/O. These calls will only wake the process when either there is input, room for output, or possibly a time-out period passes.
int maxFd;
fdset fds;
FD_ZERO(&fds);
FD_SET(listenFd, &fds);
FD_SET(sockFd1, &fds);
FD_SET(sockFd2, &fds);
maxFd = listenFd+1;
maxFd = sockFd1 > maxFd ? sockFd1+1 : maxFd;
maxFd = sockFd2 > maxFd ? sockFd2+1 : maxFd;
if(select(maxFd, &fds, &fds, &fds, NULL) < 0) {
perror("Failed on select()");
exit(1);
}
if(FD_ISSET(listenFd, &fds))
...
This example is not complete or neccessarily 100% correct, but should be a good start. Also, I tend to reserve using send*() and recv*() when dealing with SOCK_DGRAM sockets and just use read(), write() on SOCK_STREAM sockets.