I am really interested in event-driven programming in C especially with sockets so I am going to dedicate some time doing my researches.
Let's assume that I want to build a program with much File and Network I/O like a client/server app, basically, the first question is what is the philosophy behind this model. While in normal programming I would spawn new processes, how come a single process can actually serve many other requests. For example, there are some web-servers which can handle connections without creating threads or other processes, just one main process.
I know this is complicated but it's always nice to know how different solutions work.
You definitely must read the following: http://www.kegel.com/c10k.html. That page is the perfect overview of event-driven and asynchronous techniques.
However, a quick & dirty answer: event-driven is neither non-blocking, nor asynchronous.
Event-driven means, that the process will monitor its file descriptors (and sockets), and act only when some event occurs on some descriptor (events are: data received, error, became writeable, ...).
BSD sockets have the "select()" function. When called, the OS will monitor the descriptors, and return to the process as soon as some event on one of the descriptors occurs.
However, the website above has much better descriptions (and details about the different APIs).
"what is the philosophy behind this model"
Event driven means there is no "monitoring", but that the event itself initiates the action.
Usually this is initiated by an interrupt, which is a signal to the system from an external device, or (in the case of a software interrupt) an asynchronous process.
https://en.wikipedia.org/wiki/Interrupt
Further reading seems to be here:
https://docs.oracle.com/cd/E19455-01/806-1017/6jab5di2m/index.html#sockets-40
- "Interrupt-Driven Socket I/O"
Also http://cs.baylor.edu/~donahoo/practical/CSockets/textcode.html has some examples of Interrupt-Driven Sockets, as well as other socket programming examples.
Event driven programming is based on an event loop. The loop simply waits for a new event, dispatches code to handle the event, then loops back to wait for the next event. In the case of sockets, you're talking about "asynchronous network programming". This involves select() or some other option like Kqueue() to wait for the events in the event loop. Sockets would need to be set to non blocking, so that when you read() or write() your code won't wait for the I/O to complete.
Asynchronous network programming can be very complex, and tricky to get right. Check out a couple of introductions here and here. I strongly suggest using a library such as libevent or liboop to get this right.
That kind of TCP servers/clients can be implemented by using select(2) call and non-blocking sockets.
It is more tricky to use non-blocking sockets than blocking sockets.
Example:
connect call usually return -1 immediately and set errno EINPROGRESS when non-blocking socket are used. In this case you should use select to wait when connection is opened or failed. connect may also return 0. This can happen if you create connection to the local host.
This way you can serve other sockets, while one socket is opening a TCP connection.
It's actually very platform specific as to how that works.
If your running on a linux system it's really not to difficult though, you simply need to spawn a copy of your process using 'fork' something like the following would do the trick:
#include <sys/types.h>
#include <sys/socket.h>
#include <stdio.h>
#include <netinet.h>
#include <signal.h>
#include <unistd.h>
int main()
{
int server_sockfd, client_sockfd;
int server_len, client_len;
struct sockaddr_in server_address;
struct sockaddr_in client_address;
server_sockfd = socket(AF_INET, SOCK_STREAM, 0);
server_address.sin_family = AF_INET;
server_address.sin_addr.s_addr = htonl(INADDR_ANY);
server_Address.sin_port = htons(1234);
server_len = sizeof(server_address);
bind(server_sockfd, (struct sockaddr *)&server_address, server_len);
listen(server_sockfd, 5);
signal(SIGCHLD, SIG_IGN);
while(1)
{
char ch;
printf("Server Waiting\n");
client_len = sizeof(client_address);
client_sockfd = accept(server_sockfd, (struct sockaddr *)&client_address, &client_len)
// Here's where we do the forking, if you've forked already then this will be the child running, if not then your still the parent task.
if(fork() == 0)
{
// Do what ever the child needs to do with the connected client
read(client_sockfd, &ch, 1);
sleep(5); // just for show :-)
ch++;
write(client_sockfd, &ch, 1);
close(client_sockfd);
exit(0);
}
else
{
// Parent code here, close and loop for next connection
close(client_sockfd);
}
}
}
You may have to fiddle with that code a little I'm not near a Linux box at the moment to do a test compile, and I've pretty much typed it from memory.
Using fork however is the standard way to do this in C under a Linux / Unix based system.
Under windows it's a very different story, and one which I can't quite remember all the code needed (I'm way to used to coding in C# these days) but setting up the socket is pretty much the same except you need to use the 'Winsock' API for better compatibility.
You can (I believe anyway) still use standard berkley sockets under windows but it's full of pitfalls and holes, for windows winsock this is a good place to start:
http://tangentsoft.net/wskfaq/
As far as I'm aware too, if your using Winsock it has stuff in to help with the spawning and multi client, myself personally however, I usually just spin off a separate thread and copy the socket connection to that, then go back into the loop listening to my server.
Related
I often see sample codes for a server program using socket.
(For simplicity, here I don't check return values of functions such as socket() or bind() etc.)
int sockfd = 0, newsockfd = 0;
struct sockaddr_in address;
sockfd = socket(AF_INET, SOCK_STREAM, 0);
bind(sockfd, (struct sockaddr*)&address, sizeof(address));
listen(sockfd, 1);
newsockfd = accept(sockfd, (struct sockaddr*)NULL, NULL);
... do some communication ...
close(newsockfd);
Most of sample codes I found on the web have close(newsockfd) but don't have close(sockfd).
My question is whether it is really correct NOT to close sockfd.
If it's correct, I want to know why.
My understanding is that sockfd is one of the file descriptors and
it seems to have no reason to quit program without closing it.
More specifically, I'm wondering that not-closing-sockfd can cause the bind error (e.g. this socket is aready in use...) when the program works next time.
I really appreciate if you help me.
Thank you for your time.
Resources allocated by the application, like memory or open file descriptors (which includes sockets) will be automatically freed by modern OS if the program exits. Thus, if the server socket should be available throughout the whole program (in order to accept connections) it is fine to not explicitly close it but let the OS do this when the application exits.
You should always close sockfd when you stop the listening.
Two reasons why some developers do not care very much about not closing sockfd:
If your program quits without closing it, the OS will close sockfd for you
Most of the time, servers keep the listening open all the time (eg: for months)
However if your program launches a listening socket on some event, then closes it after a while, and loops, then you must close the sockfd, to prevent an error EADDRINUSE (Address already in use) at next iteration, and a memory leak.
Besides, the error EADDRINUSE (Address already in use) may occur on other circumstances that I do not detail here.
sockfd act as a server socket: it is used only to accept more and more incoming connections. You keep sockfd opened, bound and listening as long as you have to accept and handle new connections on newsockfd, wich hold the current connection on wich you are reading/writing from/to some peer program. When done with newsockfd you close it and, if requiref, accept a new one with accept() on sockfd. And so on.
I am writing a two daemon application - a client and a server. It is a very basic version of distributed shell. Clients are connecting to the server, and server issues a command, that is propagated to every client.
I dont know how to create the socket logic on server-side - I do some testing and for now I am accepting connections in an loop and for every incoming connection I fork a child to process the connection
while (1) {
clisockfd = accept(sockfd, (struct sockaddr *) &cliaddr, &clilen);
if (clisockfd < 0) {
log_err("err on opening client socket");
exit(EXIT_FAILURE);
}
/* create a new child to process the connection */
if((pid = fork()) < 0) {
log_err("err on forking, something is really broken!");
exit(EXIT_FAILURE);
}
if(!pid) {
/* here we are in forked process so we dont need the sockfd */
close(sockfd);
/* function that handles connection */
handle_connection(clisockfd);
exit(EXIT_FAILURE);
} else {
close(clisockfd);
}
}
However what I have now have some disadvantages - I can accept a connection, do something with it, and return to main process (forked process have to return, and then execution in main process is resumed). I would like to keep every socketfd somewhere(a list?) and be able to choose one of those (or all of them) and send to this socketfd a command that I want to issue on my client/s. I assume that I cant do it in traditional accept->fork->return to main process manner.
So it probably should looks like:
client connects -> server set up a new socketfd and saves it somewhere -> drops to shell where I can choose one of socket and send it a command -> somewhere in the whole process it also should wait for next incoming client connections - but where?
If someone could give me an idea what mechanisms should I use to create the logic that I need? Maybe it would be better to issue connection from server to client, not from client to server.
Regards,
Krzysztof
I assume that I cant do it in traditional accept->fork->return to main process manner.
You could but it will be hard to design/maintain.
The best solution is to use select() (POSIX), epoll() (Linux), kqueue() (BSD) or I/O Completion Ports (Windows) depending on your platform.
There is a good examples/explanations about select() in Beej's network programming guide.
I have the following piece of code:
SOCKET sock = open_socket(szListenHost, iListenPort);
if (sock > 0) {
SOCKET client;
struct sockaddr_in peeraddr;
T_socklen len = sizeof (struct sockaddr_in);
char buf[1024];
sin.dwFlags = STARTF_USESTDHANDLES | STARTF_USESHOWWINDOW;
sin.hStdInput = GetStdHandle(STD_INPUT_HANDLE);
sin.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
sin.hStdError = GetStdHandle(STD_ERROR_HANDLE);
sin.wShowWindow = SW_HIDE;
dwCreationFlags = CREATE_NO_WINDOW;
CreateProcess(NULL, buf, NULL, NULL, FALSE, dwCreationFlags,
NULL, NULL, &sin, &pin);
memset(&peeraddr, 0, sizeof (struct sockaddr_in));
client = accept(sock, (sockaddr*)&peeraddr, &len);
if (client > 0) {
rv = message_loop(client);
}
closesocket(sock);
}
As you can see, this is opening a TCP socket for interrogation reasons.
The situation is the following: my client application (who is opening those sockets) might need to open different TCP sockets simultaneously, which might cause problems.
In order to avoid those problems, I would like to ask whether the socket is already opened. If yes, then wait until the socket is freed again and then try again to open the socket.
I have understood that semaphores can be used for this intention, but I have no idea how to do this.
Can anybody help me?
Thanks
First I'd like to thank John Bollinger for your fast response. Unfortunately their seems to be as misunderstanding: I am not looking for a way to open one socket different times simultaneously, but I am looking for a way to be noticed when a socket becomes available. In fact, I would like to do the following: Instead of:
SOCKET sock = open_socket(szListenHost, iListenPort);
I could do this (very basically):
while (open_socket(szListenHost, iListenPort)) {sleep (1 second;)}
This however means that I would need to poll the socket constantly, creating quite some overhead. I have heard that semaphores could solve this issue, something like:
SOCKET sock = handle_semaphore(open_socket(szListenHost, iListenPort));
and the "handle_semaphore" would then be a system that automatically waits for the socket to be released, so that immediately my client process can open the socket, without the risk of being pushed behind. As you can see, it's all about rumours but I have no idea how to realise this. Does anybody know whether indeed semaphores can be used for this intention and if possible, give me some guidance on how to do this?
Thanks
Once opened, a socket cannot be reopened, even if it is closed. You can create a similar, new socket, though. Either way, it is difficult to reliably determine whether a previously-opened socket has been closed, except by closing it.
In any case, the usual paradigm does not require the kind of coordinating mechanism you ask about. Normally, one thread of one process would open the socket and have responsibility for accepting connections on it. If it is desired that the program be able to handle more than one connection at a time, then each time that thread accepts a new connection, it assigns that connection to be handled by another thread or process -- typically, but not necessarily, a newly-created one.
It is not usually necessary or desirable to open a new socket to receive additional connections at the same address and port. Usually you just use the same socket, without caring about the state of any connections already established via that socket. You could, perhaps, use a semaphore to coordinate multiple threads of the same process receiving connections from the same socket, but I would avoid that if I were you.
Meanwhile the situation has changed, and I now have been able to add semaphores to my socket related application. Generally this works but sometimes the application hangs.
After some debugging I have understood that the application hangs at the moment I launch following C command:
printf("Will it be accepted?\n");
fflush(stdout);
memset(&peeraddr, 0, sizeof (struct sockaddr_in));
client = accept(sock, (sockaddr*)&peeraddr, &len);
printf("It is accepted, the client is %d.\n",client);
=> I can see in my debug log "Will it be accepted?", but I don't see "It is accepted, ...".
I admit that I am quite violent while testing (sometimes I stop debugging sessions without giving the application to close the socket, ...), but you can imagine customers behaving in the same way, to the application needs to be sufficiently robust.
Does anybody know how I can avoid the "accept" command going into such an infinite loop?
Thanks
I'm new to linux programming and not entirely familiar with all the synchronization facilities so I'd like to ask more knowledgeable people how they might go about solving this problem.
I have a single thread that I would like to run through a loop. The stopping point in the loop will be a read operation on a socket. I want the read operation to block for some period of time and then timeout. However, I need a way unblock the thread from the read, if some event needs attention. The "event" could be any one of a number of different things so I need some way to tell the thread what cause the read to unblock.
I know that you can unblock a blocked read with a signal but I'm not sure how that's done.
See the select() system call.
This is especially useful for waiting for multiple file channels.
You can set timeout of socket operation. Example:
struct timeval timeout;
timeout.tv_sec = TIMEOUT_SEC;
timeout.tv_usec = TIMEOUT_MSEC;
setsockopt(sock_fd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof(timeout));
/* now receive msg */
recvmsg(sock_fd, &msg, 0);
When you want to make your socket blocking, do:
timeout.tv_sec = 0;
timeout.tv_usec = 0;
setsockopt(sock_fd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof(timeout));
epoll seems to be the way to go:
The epoll API performs a similar task to poll(2): monitoring multiple
file descriptors to see if I/O is possible on any of them. The epoll
API can be used either as an edge-triggered or a level-triggered inter‐
face and scales well to large numbers of watched file descriptors. The
following system calls are provided to create and manage an epoll
instance:
man epoll for more info. You might want to see "Example for Suggested Usage" section on manual.
See also epoll vs select
Sounds like you want to use select() as others have mentioned, but you also want a way to interrupt it when a "message" of some sort is available. A typical way of interrupting a select() is to use the self pipe trick. Basically you create a pipe() and also select() on the read file descriptor of the pipe. When a message arrives in the queue maintained by your program, write a byte to the pipe. This will cause your select call to return and you'll be able to check to see if your pipe is ready for reading. If it is then you know you have a message to process (whatever that is in your context), so you process it and then go back to select(). Better yet, you could have your pipe actually be your message queue. If you just use the pipe as a way to signal that messages are on your queue, make sure you actually read() the bytes out of your pipe each time through, or it will fill up eventually and block you from writing more notifications to it.
Although, as others have mentioned, why not just have one thread service your queue and do your writes to the socket, while another thread does the reads? Probably a lot simpler.
Perhaps these two libraries may be of use to you:
libev
libuv
They both use the event-driven paradigm on one or more threads (if so desired). Of course, you can implement your own event-driven framework using already mentioned APIs and conditional variables, but that might be more work than necessary.
I'm working on Linux system (Ubuntu 7.04 server with a 2.6.20 kernel).
I've got a program that has a thread (thread1) waiting on a select for a UDP socket to become readable.
I'm using the select (with my socket as the single readfd and the single exceptfd) instead of just calling recvfrom because I want a timeout.
From another thread, I shutdown and close the socket.
If I do this while thread1 is blocked in a recvfrom, then the recvfrom will terminate immediately.
If I do this while thread1 is blocked in a select with a timeout, then the select will NOT terminate immediately, but will eventually timeout properly.
Can anyone tell me why it is that the select doesn't exit as soon as the socket is closed? Isn't that an exception? I can see where it isn't readable (obviously), but it's closed, which seems to be to be exeptional.
Here's the opening of the socket (all error handling removed to keep things simple):
m_sockfd = socket(PF_INET, SOCK_DGRAM, 0);
struct sockaddr_in si_me;
memset((char *) &si_me, 0, sizeof(si_me));
si_me.sin_family = AF_INET;
si_me.sin_port = htons(port);
si_me.sin_addr.s_addr = htonl(INADDR_ANY);
if (bind(m_sockfd, (struct sockaddr *)(&si_me), sizeof(si_me)) < 0)
{
// deal with error
}
Here's the select statement that thread1 executes:
struct timeval to;
to.tv_sec = timeout_ms/1000;// just the seconds portion
to.tv_usec = (timeout_ms%1000)*1000;// just the milliseconds
// converted to microseconds
// watch our one fd for readability or
// exceptions.
fd_set readfds, exceptfds;
FD_ZERO(&readfds);
FD_SET(m_sockfd, &readfds);
FD_ZERO(&exceptfds);
FD_SET(m_sockfd, &exceptfds);
int nsel = select(m_sockfd+1, &readfds, NULL, &exceptfds, &to);
UPDATE: Obviously (as stated below), closing the socket isn't an exceptional condition (from select's point of view). I think what I need to know is: Why? And, Is that intentional?.
I REALLY want to understand the thinking behind this select behavior because it seems counter to my expectations. Thus, I obviously need to adjust my thinking on how the TCP stack works. Please explain it to me.
Maybe you should use something else to wake up the select. Maybe a pipe or something like that.
UDP is a connectionless protocol. Since there is no connection, none can be broken, so the consumer doesn't know that the producer will never send again.
You could make the producer send an "end of stream" message, and have the consumer terminate upon receiving it.
I think the most obvious solution is that being closed isn't considered an exceptional condition. I think the root of the problem is, that you're not really embracing the philosophy of select. Why on earth are you fiddling around with the socket in another thread, that sounds like a recipe for disaster.
Could you not send a signal (e.g. USR2) to the thread which would cause select() to return with EINTR?
Then in the signal handler set a flag telling it not to restart the select()?
That would remove the need for waiting on multiple file descriptors, and seems a lot cleaner than using a pipe to kill it.
I would say the difference is that recvfrom is actively trying to read a message from a single socket, where select is waiting for a message to arrive, possibly on multiple handles, and not necessarily socket handles.
Your code is fundamentally broken. Variations on this mistake are common and have caused serious bugs with massive security implications in the past. Here's what you're missing:
When you go to close the socket, there is simply no possible way you can know whether the other thread is blocked in select or about to block in select. For example, consider the following:
The thread goes to call select, but doesn't get scheduled.
You close the socket.
In a thread your code is unaware of (maybe it's part of the platform's internal memory management or logging internals) a library opens a socket and gets the same identifier as the socket you closed.
The thread now goes into select, but it's selecting on the socket opened by the library.
Disaster strikes.
You must not attempt to release a resource while another thread is, or might be, using it.