I recently did some testing with kernel events and I came up with the following:
Does it make sense to use a kernel event for accepting sockets? My testing showed that I was only able to handle one accept at once (even if the eventlist array is bigger)(Makes sense to me cause .ident == sockfd is only true for one socket).
I thought the use of kevent is mainly to read from multiple sockets at once. Is that true?
Is this how a TCP server is done with a kqueue implementation? :
Listening Thread (without kqueue)
Accepts new connections and adds FD to a worker kqueue.
QUESTION: Is this even possible? My testing showed yes, but is it guaranteed that the worker thread will be aware of the changes and is kevent really thread safe?
Worker thread (with kqueue)
Waits on reads on file descriptors added from the listening thread.
QUESTION: How many sockets at once would make sense to check for updates?
Thanks
This is not really an answer but I made a little server script with kqueue explaining the problem:
#include <stdio.h> // fprintf
#include <sys/event.h> // kqueue
#include <netdb.h> // addrinfo
#include <arpa/inet.h> // AF_INET
#include <sys/socket.h> // socket
#include <assert.h> // assert
#include <string.h> // bzero
#include <stdbool.h> // bool
#include <unistd.h> // close
int main(int argc, const char * argv[])
{
/* Initialize server socket */
struct addrinfo hints, *res;
int sockfd;
bzero(&hints, sizeof(hints));
hints.ai_family = AF_INET;
hints.ai_socktype = SOCK_STREAM;
assert(getaddrinfo("localhost", "9090", &hints, &res) == 0);
sockfd = socket(AF_INET, SOCK_STREAM, res->ai_protocol);
assert(sockfd > 0);
{
unsigned opt = 1;
assert(setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)) == 0);
#ifdef SO_REUSEPORT
assert(setsockopt(sockfd, SOL_SOCKET, SO_REUSEPORT, &opt, sizeof(opt)) == 0);
#endif
}
assert(bind(sockfd, res->ai_addr, res->ai_addrlen) == 0);
freeaddrinfo(res);
/* Start to listen */
(void)listen(sockfd, 5);
{
/* kevent set */
struct kevent kevSet;
/* events */
struct kevent events[20];
/* nevents */
unsigned nevents;
/* kq */
int kq;
/* buffer */
char buf[20];
/* length */
ssize_t readlen;
kevSet.data = 5; // backlog is set to 5
kevSet.fflags = 0;
kevSet.filter = EVFILT_READ;
kevSet.flags = EV_ADD;
kevSet.ident = sockfd;
kevSet.udata = NULL;
assert((kq = kqueue()) > 0);
/* Update kqueue */
assert(kevent(kq, &kevSet, 1, NULL, 0, NULL) == 0);
/* Enter loop */
while (true) {
/* Wait for events to happen */
nevents = kevent(kq, NULL, 0, events, 20, NULL);
assert(nevents >= 0);
fprintf(stderr, "Got %u events to handle...\n", nevents);
for (unsigned i = 0; i < nevents; ++i) {
struct kevent event = events[i];
int clientfd = (int)event.ident;
/* Handle disconnect */
if (event.flags & EV_EOF) {
/* Simply close socket */
close(clientfd);
fprintf(stderr, "A client has left the server...\n");
} else if (clientfd == sockfd) {
int nclientfd = accept(sockfd, NULL, NULL);
assert(nclientfd > 0);
/* Add to event list */
kevSet.data = 0;
kevSet.fflags = 0;
kevSet.filter = EVFILT_READ;
kevSet.flags = EV_ADD;
kevSet.ident = nclientfd;
kevSet.udata = NULL;
assert(kevent(kq, &kevSet, 1, NULL, 0, NULL) == 0);
fprintf(stderr, "A new client connected to the server...\n");
(void)write(nclientfd, "Welcome to this server!\n", 24);
} else if (event.flags & EVFILT_READ) {
/* sleep for "processing" time */
readlen = read(clientfd, buf, sizeof(buf));
buf[readlen - 1] = 0;
fprintf(stderr, "bytes %zu are available to read... %s \n", (size_t)event.data, buf);
sleep(4);
} else {
fprintf(stderr, "unknown event: %8.8X\n", event.flags);
}
}
}
}
return 0;
}
Every time a client sends something the server experiences a "lag" of 4 seconds. (I exaggerated a bit, but for testing quite reasonable). So how do get around to that problem? I see worker threads (pool) with own kqueue as possible solution, then no connection lag would occur. (each worker thread reads a certain "range" of file descriptors)
Normally, you use kqueue as an alternative to threads. If you're going to use threads, you can just set up a listening thread and a worker threadpool with one thread per accepted connection. That's a much simpler programming model.
In an event-driven framework, you would put both the listening socket and all the accepted sockets into the kqueue, and then handle events as they occur. When you accept a socket, you add it to the kqueue, and when a socket handler finishes it works, it could remove the socket from the kqueue. (The latter is not normally necessary because closing a fd automatically removes any associated events from any kqueue.)
Note that every event registered with a kqueue has a void* userdata, which can be used to identify the desired action when the event fires. So it's not necessary that every event queue have a unique event handler; in fact, it is common to have a variety of handlers. (For example, you might also want to handle a control channel set up through a named pipe.)
Hybrid event/thread models are certainly possible; otherwise, you cannot take advantage of multicore CPUs. One possible strategy is to use the event queue as a dispatcher in a producer-consumer model. The queue handler would directly handle events on the listening socket, accepting the connection and adding the accepted fd into the event queue. When a client connection event occurs, the event would be posted into the workqueue for later handling. It's also possible to have multiple workqueues, one per thread, and have the accepter guess which workqueue a new connection should be placed in, presumably on the basis of that thread's current load.
Related
This question is similar to How to call a function on a thread's creation and exit? but more specific. In another multi-process shared memory project I used a combination of an __attribute__((constructor)) labeled library init routine, lazy initialisation for each thread, and robust futexes to make sure resources weren't leaked in the shared memory even if a sys admin chose to SIGKILL one of the processes using it. However futexes within the APIs are way too heavyweight for my current project and even the few instructions to deke around some lazy initialisation is something I'd rather avoid. The library APIs will literally be called several trillion times over a few hundred threads across several processes (each API is only a couple hundred instructions.)
I am guessing the answer is no, but since I spent a couple hours looking for and not finding a definitive answer I thought I'd ask it here, then the next person looking for a simple answer will be able to find it more quickly.
My goal is pretty simple: perform some per-thread initialisation as threads are created in multiple processes asynchronously, and robustly perform some cleanup at some point when threads are destroyed asynchronously. Doesn't have to be immediately, it just has to happen eventually.
Some hypothetical ideas to engage critical thinking: a hypothetical pthread_atclone() called from an __attribute__((constructor)) labeled library init func would satisfy the first condition. And an extension to futex()es to add a semop-like operation with a per-thread futex_adj value that, if non-zero in do_exit(), causes FUTEX_OWNER_DIED to be set for the futex "semaphore" allowing cleanup the next time the futex is touched.
Well, first, you should document that library users should not asynchronously terminate threads in such a manner that they dont explictly release resources belonging to your library, (closing a handle, whatever), TBH, just terminating threads at all before process termination is a bad idea.
It's more difficult to detect if a whole process is SIGKILLed while it's using your lib. My current best guess is that all processes wishing to use your library have to log in first so that their pid can be added to a container. Using a thread started at your lib initialization, poll for pid's that have diappeared with kill(pid,0) and take any approriate cleanup. It's not very satisfactory, (I hate polling), but I don't see any alternatives that are not grossly messy:(
After research and experimentation I've come up with what seems to be current "best practice" as far as I can tell. If anyone knows any better, please comment!
For the first part, per-thread initialisation, I was not able to come up with any alternative to straightforward lazy initialisation. However, I did decide that it's slightly more efficient to move the branch to the caller so that pipelining in the new stack frame isn't immediately confronted with an effectively unnecessary branch. so instead of this:
__thread int tInf = 0;
void
threadDoSomething(void *data)
{
if (!tInf) {
_threadInitInfo(&tInf);
}
/*l
* do Something.
*/
}
This:
__thread int tInf = 0;
#define threadDoSomething(data) (((!tInf)?_threadInitInfo(&tInf):0), \
_threadDoSomething((data)))
void
_threadDoSomething(void *data)
{
/*l
* do Something.
*/
}
Comments on the (admittedly slight) usefulness of this welcome!
For the second part, robustly performing some cleanup when threads die no matter how asynchronously, I was not able to find any solution better than to have a reaping process epoll_wait() on a file descriptor for the read end of an open pipe passed to it via an SCM_RIGHTS control message in a sendmsg() call on an abstract UNIX domain socket address. Sounds complex, but it's not that bad, here's the client side:
/*m
* Client that registers a thread with a server who will do cleanup of a
* shared interprocess object even if the thread dies asynchronously.
*/
#include <sys/socket.h> // socket(), bind(), recvmsg()
#include <sys/syscall.h> // syscall()
#include <sys/un.h> // sockaddr_un
#include <stdint.h> // uint64_t
#include <fcntl.h> // O_CLOEXEC()
#include <malloc.h> // malloc()
#include <stdlib.h> // random()
#include <unistd.h> // close(), usleep()
#include <pthread.h> // pthread_create()
#include <tsteplsrv.h> // Our API.
char iovBuf[] = "SP1"; // 3 char buf to send client type
__thread pid_t cliTid = 0; // per-thread copy of self's Thread ID
/*f
* initClient() is called when we realise we need to lazily initialise
* our thread based on cliTid being zero.
*/
void *
initClient(void *ptr)
{
struct sockaddr_un svAddr;
struct msghdr msg;
struct iovec io;
struct cmsghdr *ctrMsg;
uint64_t ltid; // local 8-byte copy of the tid
int pfds[2], // two fds of our pipe
sfd; // socket fd
/*s
* This union is necessary to ensure that the buffer is aligned such that
* we can read cmsg_{len,level,type} from the cmsghdr without causing an
* alignment fault (SIGBUS.)
*/
union {
struct cmsghdr hdr;
char buf[CMSG_SPACE(sizeof(int))];
} ctrBuf;
pfds[0] = pfds[1] = sfd = -1;
/*l
* Get our Thread ID.
*/
ltid = (uint64_t)(cliTid = syscall(SYS_gettid));
/*l
* Set up an abstract unix domain socket address.
*/
svAddr.sun_family = AF_UNIX;
svAddr.sun_path[0] = '\0';
strcpy(&svAddr.sun_path[1], EPLS_SRV_ADDR);
/*l
* Set up a socket datagram send buffer.
*/
io.iov_base = iovBuf;
io.iov_len = sizeof(iovBuf);
msg.msg_iov = &io;
msg.msg_iovlen = 1;
msg.msg_control = ctrBuf.buf;
msg.msg_controllen = sizeof(ctrBuf);
msg.msg_name = (struct sockaddr *)&svAddr,
msg.msg_namelen = (&svAddr.sun_path[0] - (char *)&svAddr)
+ 1
+ sizeof(EPLS_SRV_ADDR);
/*l
* Set up the control message header to indicate we are sharing a file
* descriptor.
*/
ctrMsg = CMSG_FIRSTHDR(&msg);
ctrMsg->cmsg_len = CMSG_LEN(sizeof(int));
ctrMsg->cmsg_level = SOL_SOCKET;
ctrMsg->cmsg_type = SCM_RIGHTS;
/*l
* Create file descriptors with pipe().
*/
if (-1 == pipe(pfds)) {
printErrMsg("TID: %d pipe() failed", cliTid);
} else {
/*l
* Write our tid to the pipe.
*/
memmove(CMSG_DATA(ctrMsg), &pfds[0], sizeof(int));
if (-1 == write(pfds[1], <id, sizeof(uint64_t))) {
printErrMsg("TID: %d write() failed", cliTid);
} if (-1 == (sfd = socket(AF_UNIX, SOCK_DGRAM, 0))) {
printErrMsg("TID: %d socket() failed", cliTid);
} else if (-1 == sendmsg(sfd, &msg, 0)) {
printErrMsg("TID: %d sendmsg() failed", cliTid);
} else {
printVerbMsg("TID: %d sent write fd %d to server kept read fd %d",
cliTid,
pfds[0],
pfds[1]);
/*l
* Close the read end of the pipe, the server has it now.
*/
close(pfds[0]);
pfds[0] = -1;
}
}
if (-1 != pfds[1]) close(pfds[1]);
if (-1 != pfds[0]) close(pfds[0]);
if (-1 != sfd) close(sfd);
return (void *)0;
}
And the reaper's code:
/*m
* Abstract datagram socket listening for FD's from clients.
*/
#include <sys/socket.h> // socket(), bind(), recvmsg()
#include <sys/epoll.h> // epoll_{create,wait}()
#include <sys/un.h> // sockaddr_un
#include <malloc.h> // malloc()
#include <unistd.h> // close()
#include <tsteplsrv.h> // Our API.
/*s
* socket datagram structs for receiving structured messages used to transfer
* fds from our clients.
*/
struct msghdr msg = { 0 };
struct iovec io = { 0 };
char iovBuf[EPLS_MSG_LEN]; // 3 char buf to receive client type
/*s
* This union is necessary to ensure that the buffer is aligned such that
* we can read cmsg_{len,level,type} from the cmsghdr without causing an
* alignment fault (SIGBUS.)
*/
union {
struct cmsghdr hdr;
char buf[CMSG_SPACE(sizeof(int))];
} ctrBuf;
typedef struct _tidFd_t {
struct _tidFd_t *next;
pid_t tid;
int fd;
} tidFd_t;
tidFd_t *tidFdLst = (tidFd_t *)0;
/*f
* Perform some handshaking with a new client and add the file descriptor
* it shared with us to the epoll set.
*/
static void
welcomeClient(int efd, int cfd)
{
uint64_t tid;
tidFd_t *tfd;
struct epoll_event epEv;
tfd = (tidFd_t *)-1;
/*l
* The fd is a pipe and should be readable, and should contain the
* tid of the client.
*/
if (-1 != read(cfd, &tid, sizeof(tid)) && (tfd = malloc(sizeof(*tfd)))) {
tfd->fd = cfd;
tfd->tid = (pid_t)tid;
tfd->next = tidFdLst;
/*l
* Single threaded process, no race condition here.
*/
tidFdLst = tfd;
/*l
* Add the fd to the epoll() set so that we will be woken up with
* an error if the thread dies.
*/
epEv.events = EPOLLIN;
epEv.data.fd = cfd;
if (-1 == epoll_ctl(efd, EPOLL_CTL_ADD, cfd, &epEv)) {
printErrMsg("TID: %ld Could not register fd %d with epoll set",
tid,
cfd);
} else {
printVerbMsg("TID: %ld Registered fd %d with epoll set", tid, cfd);
}
/*l
* Couldn't allocate memory for the new client.
*/
} else if (!tfd) {
printErrMsg("Could not allocate memory for new client");
/*l
* Could not read from the eventfd() file descriptor.
*/
} else {
printErrMsg("Could not read from client file descriptor");
}
}
/*f
* Perform some handshaking with a new client and add the file descriptor
* it shared with us to the epoll set.
*/
static void
processClientEvent(int efd, struct epoll_event *epEv)
{
tidFd_t *tfd, **bLnk;
/*l
* Walk the list of per-tid fd structs.
*/
for (bLnk = &tidFdLst; (tfd = *bLnk); bLnk = &tfd->next)
if (tfd->fd == epEv->data.fd)
break;
if (!tfd) {
printErrMsg("client file descriptor %d not found on the tfd list!",
epEv->data.fd);
/*l
* If we received an EPOLLHUP on the fd, cleanup.
*/
} else if (epEv->events & EPOLLHUP) {
/*l
* Try to remove the tid's pipe fd from the epoll set.
*/
if (-1 == epoll_ctl(efd, EPOLL_CTL_DEL, epEv->data.fd, epEv)) {
printErrMsg("couldn't delete epoll for tid %d", tfd->tid);
/*l
* Do tid cleanup here.
*/
} else {
printVerbMsg("TID: %d closing fd: %d", tfd->tid, epEv->data.fd);
close(epEv->data.fd);
/*l
* Remove the per-tid struct from the list and free it.
*/
*bLnk = tfd->next;
free(tfd);
}
} else {
printVerbMsg("TID: %d Received unexpected epoll event %d",
tfd->tid,
epEv->events);
}
}
/*f
* Create and listen on a datagram socket for eventfd() file descriptors
* from clients.
*/
int
main(int argc, char *argv[])
{
struct sockaddr_un svAddr;
struct cmsghdr *ctrMsg;
struct epoll_event *epEv,
epEvs[EPLS_MAX_EPEVS];
int sfd, efd, cfd, nfds;
sfd = efd = -1;
/*l
* Set up an abstract unix domain socket address.
*/
svAddr.sun_family = AF_UNIX;
svAddr.sun_path[0] = '\0';
strcpy(&svAddr.sun_path[1], EPLS_SRV_ADDR);
/*l
* Set up a socket datagram receive buffer.
*/
io.iov_base = iovBuf; // 3-char buffer to ID client type
io.iov_len = sizeof(iovBuf);
msg.msg_name = (char *)0; // No need for the client addr
msg.msg_namelen = 0;
msg.msg_iov = &io; // single IO vector in the S/G array
msg.msg_iovlen = 1;
msg.msg_control = ctrBuf.buf; // Control message buffer
msg.msg_controllen = sizeof(ctrBuf);
/*l
* Set up an epoll event.
*/
epEv = &epEvs[0];
epEv->events = EPOLLIN;
/*l
* Create a socket to receive datagrams on and register the socket
* with our epoll event.
*/
if (-1 == (epEv->data.fd = sfd = socket(AF_UNIX, SOCK_DGRAM, 0))) {
printErrMsg("socket creation failed");
/*l
* Bind to the abstract address. The pointer math is to portably
* handle weird structure packing _just_in_case_.
*/
} else if (-1 == bind(sfd,
(struct sockaddr *)&svAddr,
(&svAddr.sun_path[0] - (char *)&svAddr)
+ 1
+ sizeof(EPLS_SRV_ADDR))) {
printErrMsg("could not bind address: %s", &svAddr.sun_path[1]);
/*l
* Create an epoll interface. Set CLOEXEC for tidiness in case a thread
* in the server fork()s and exec()s.
*/
} else if (-1 == (efd = epoll_create1(EPOLL_CLOEXEC))) {
printErrMsg("could not create epoll instance");
/*l
* Add our socket fd to the epoll instance.
*/
} else if (-1 == epoll_ctl(efd, EPOLL_CTL_ADD, sfd, epEv)) {
printErrMsg("could not add socket to epoll instance");
/*l
* Loop receiving events on our epoll instance.
*/
} else {
printVerbMsg("server listening on abstract address: %s",
&svAddr.sun_path[1]);
/*l
* Loop forever listening for events on the fds we are interested
* in.
*/
while (-1 != (nfds = epoll_wait(efd, epEvs, EPLS_MAX_EPEVS, -1))) {
/*l
* For each fd with an event, figure out what's up!
*/
do {
/*l
* Transform nfds from a count to an index.
*/
--nfds;
/*l
* If the fd with an event is the listening socket a client
* is trying to send us their eventfd() file descriptor.
*/
if (sfd == epEvs[nfds].data.fd) {
if (EPOLLIN != epEvs[nfds].events) {
printErrMsg("unexpected condition on socket: %d",
epEvs[nfds].events);
nfds = -1;
break;
}
/*l
* Reset the sizes of the receive buffers to their
* actual value; on return they will be set to the
* read value.
*/
io.iov_len = sizeof(iovBuf);
msg.msg_controllen = sizeof(ctrBuf);
/*l
* Receive the waiting message.
*/
if (-1 == recvmsg(sfd, &msg, MSG_CMSG_CLOEXEC)) {
printVerbMsg("failed datagram read on socket");
/*l
* Verify that the message's control buffer contains
* a file descriptor.
*/
} else if ( NULL != (ctrMsg = CMSG_FIRSTHDR(&msg))
&& CMSG_LEN(sizeof(int)) == ctrMsg->cmsg_len
&& SOL_SOCKET == ctrMsg->cmsg_level
&& SCM_RIGHTS == ctrMsg->cmsg_type) {
/*l
* Unpack the file descriptor.
*/
memmove(&cfd, CMSG_DATA(ctrMsg), sizeof(cfd));
printVerbMsg("Received fd %d from client type %c%c%c",
cfd,
((char *)msg.msg_iov->iov_base)[0],
((char *)msg.msg_iov->iov_base)[1],
((char *)msg.msg_iov->iov_base)[2]);
/*l
* Process the incoming file descriptor and add
* it to the epoll() list.
*/
welcomeClient(efd, cfd);
/*l
* Note but ignore incorrectly formed datagrams.
*/
} else {
printVerbMsg("could not extract file descriptor "
"from client's datagram");
}
/*l
* The epoll() event is on one of the file descriptors
* shared with a client, process it.
*/
} else {
processClientEvent(efd, &epEvs[nfds]);
}
} while (nfds);
/*l
* If something happened to our socket break the epoll_wait()
* loop.
*/
if (nfds)
break;
}
}
/*l
* An error occurred, cleanup.
*/
if (-1 != efd)
close(efd);
if (-1 != sfd)
close(sfd);
return -1;
}
At first I tried using eventfd() rather than pipe() but eventfd file descriptors represent objects not connections, so closing the fd in the client code did not produce an EPOLLHUP in the reaper. If anyone knows of a better alternative to pipe() for this, let me know!
For completeness here's the #defines used to construct the abstract address:
/*d
* server abstract address.
*/
#define EPLS_SRV_NAM "_abssSrv"
#define EPLS_SRV_VER "0.0.1"
#define EPLS_SRV_ADDR EPLS_SRV_NAM "." EPLS_SRV_NAM
#define EPLS_MSG_LEN 3
#define EPLS_MAX_EPEVS 32
That's it, hope this is useful for someone.
Here is the threaded-server code in C. My question is: do we need to set unused thread to NULL? In java, we need to set thread to NULL to let it return to thread pool.
I made the change to Martin Broadhurst's source code (see gray text as comment)
/*
* A threaded server
* by Martin Broadhurst (www.martinbroadhurst.com)
* Compile with -pthread
*/
#include <stdio.h>
#include <string.h> /* memset() */
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
#include <netdb.h>
#include <pthread.h>
#define PORT "32001" /* Port to listen on */
#define BACKLOG 10 /* Passed to listen() */
void *handle(void *pnewsock)
{
/* send(), recv(), close() */
return NULL;
}
int main(void)
{
int sock;
pthread_t thread;
struct addrinfo hints, *res;
int reuseaddr = 1; /* True */
/* Get the address info */
memset(&hints, 0, sizeof hints);
hints.ai_family = AF_INET;
hints.ai_socktype = SOCK_STREAM;
if (getaddrinfo(NULL, PORT, &hints, &res) != 0) {
perror("getaddrinfo");
return 1;
}
/* Create the socket */
sock = socket(res->ai_family, res->ai_socktype, res->ai_protocol);
if (sock == -1) {
perror("socket");
return 1;
}
/* Enable the socket to reuse the address */
if (setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &reuseaddr, sizeof(int)) == -1) {
perror("setsockopt");
return 1;
}
/* Bind to the address */
if (bind(sock, res->ai_addr, res->ai_addrlen) == -1) {
perror("bind");
return 0;
}
freeaddrinfo(res);
/* Listen */
if (listen(sock, BACKLOG) == -1) {
perror("listen");
return 0;
}
/* Main loop */
while (1) {
pthread_attr_t *attr; //<===I added this
size_t size = sizeof(struct sockaddr_in);
struct sockaddr_in their_addr;
int * ptr; //<===I added this
ptr = malloc(sizeof(int)); //<===I added this
ptr = accept(sock, (struct sockaddr*)&their_addr, &size);
if (newsock == -1) {
perror("accept");
}
else {
printf("Got a connection from %s on port %d\n",
inet_ntoa(their_addr.sin_addr), htons(their_addr.sin_port));
//I added the following "if" statement
if (pthread_attr_setdetachstate(attr, PTHREAD_CREATE_DETACHED) != 0){
fprintf(stderr, "Failed to set thread detached\n");
}
else {
//if (pthread_create(&thread, NULL, handle, &newsock) != 0) {
if (pthread_create(&thread, attr, handle, ptr) != 0 ) {
fprintf(stderr, "Failed to create thread\n");
}
}
}
}
close(sock);
return 0;
}
==========-==============
code is from here:
http://martinbroadhurst.com/source/threaded-server.c.html
No. Well, it's not 100% clear what Java construct it is you're thinking of (I bet there's a close method you can call instead of setting it to null and having the GC take care of it), but that's irrelevant because...
pthread_t is an integer (maybe) type, not a pointer, so it can't be set to NULL.
C is not garbage collected, so even if it were a pointer the thread would have no way of knowing or caring that you set it to null.
POSIX threads does not use a thread pool. The pthread_create function actually creates a brand-new OS-level thread, and returning from the handler actually exits it. (Well, not really. It still hangs around until you call pthread_join, since you didn't create it as a detached thread.)
What you should do is create the threads as detached threads, since your code right now is leaking joinable threads.
Also, using &newsock as the argument is dangerous, since it gets destroyed and recreated in every iteration of the main loop. This is a race condition that probably never showed up in the author's testing because under light load the main thread would be waiting for the next accept to return while it is accessed on the worker thread, and on most systems the same space will be used over and over for the variable.
You can use malloc to create a place to store the socket fd in (which you will need to free at the end of your handler function), or, if your platform allows this (most do), just cast the value to a pointer then cast it back out in the handler function.
Since C doesn't have objects, there is no object that represents the thread and so nothing to set to NULL. A detached thread will go away when it terminates. An undetached thread will go away when it's joined.
You have a pthread_t as thread id, there is no object, and it don't work in the way as java gc.
Thread terminate in 1 of following cases:
its start function return,
the thread call pthread_exit()
canceled by pthread_cancel()
any of threads in the process call exit(),
the main thread returns,
in this case, all threads in the process will terminate immediately,
I just wondered about how Instant Messengers and Online Games can accept and deliver messages so fast. (Network programming with sockets)
I read about that this is done with nonblocking sockets.
I tried blocking sockets with pthreads (each client gets its own thread) and nonblocking sockets with kqueue.Then I profiled both servers with a program which made 99 connections (each connection in one thread) and then writes some garbage to it (with a sleep of 1 second). When all threads are set up, I measured in the main thread how long it took to get a connection from the server (with wall clock time) (while "99 users" are writing to it).
threads (avg): 0.000350 // only small difference to kqueue
kqueue (avg): 0.000300 // and this is not even stable (client side)
The problem is, while testing with kqueue I got multiple times a SIGPIPE error (client-side). (With a little timeout usleep(50) this error was fixed). I think this is really bad because a server should be capable to handle thousands of connections. (Or is it my fault on the client side?) The crazy thing about this is the infamous pthread approach did just fine (with and without timeout).
So my question is: how can you build a stable socket server in C which can handle thousands of clients "asynchronously"? I only see the threads approach as a good thing, but this is considered bad practice.
Greetings
EDIT:
My test code:
double get_wall_time(){
struct timeval time;
if (gettimeofday(&time,NULL)){
// Handle error
return 0;
}
return (double)time.tv_sec + (double)time.tv_usec * .000001;
}
#define NTHREADS 100
volatile unsigned n_threads = 0;
volatile unsigned n_writes = 0;
pthread_mutex_t main_ready;
pthread_mutex_t stop_mtx;
volatile bool running = true;
void stop(void)
{
pthread_mutex_lock(&stop_mtx);
running = false;
pthread_mutex_unlock(&stop_mtx);
}
bool shouldRun(void)
{
bool copy;
pthread_mutex_lock(&stop_mtx);
copy = running;
pthread_mutex_unlock(&stop_mtx);
return copy;
}
#define TARGET_HOST "localhost"
#define TARGET_PORT "1336"
void *thread(void *args)
{
char tmp = 0x01;
if (__sync_add_and_fetch(&n_threads, 1) == NTHREADS) {
pthread_mutex_unlock(&main_ready);
fprintf(stderr, "All %u Threads are ready...\n", (unsigned)n_threads);
}
int fd = socket(res->ai_family, SOCK_STREAM, res->ai_protocol);
if (connect(fd, res->ai_addr, res->ai_addrlen) != 0) {
socket_close(fd);
fd = -1;
}
if (fd <= 0) {
fprintf(stderr, "socket_create failed\n");
}
if (write(fd, &tmp, 1) <= 0) {
fprintf(stderr, "pre-write failed\n");
}
do {
/* Write some garbage */
if (write(fd, &tmp, 1) <= 0) {
fprintf(stderr, "in-write failed\n");
break;
}
__sync_add_and_fetch(&n_writes, 1);
/* Wait some time */
usleep(500);
} while (shouldRun());
socket_close(fd);
return NULL;
}
int main(int argc, const char * argv[])
{
pthread_t threads[NTHREADS];
pthread_mutex_init(&main_ready, NULL);
pthread_mutex_lock(&main_ready);
pthread_mutex_init(&stop_mtx, NULL);
bzero((char *)&hint, sizeof(hint));
hint.ai_socktype = SOCK_STREAM;
hint.ai_family = AF_INET;
if (getaddrinfo(TARGET_HOST, TARGET_PORT, &hint, &res) != 0) {
return -1;
}
for (int i = 0; i < NTHREADS; ++i) {
pthread_create(&threads[i], NULL, thread, NULL);
}
/* wait for all threads to be set up */
pthread_mutex_lock(&main_ready);
fprintf(stderr, "Main thread is ready...\n");
{
double start, end;
int fd;
start = get_wall_time();
fd = socket(res->ai_family, SOCK_STREAM, res->ai_protocol);
if (connect(fd, res->ai_addr, res->ai_addrlen) != 0) {
socket_close(fd);
fd = -1;
}
end = get_wall_time();
if (fd > 0) {
fprintf(stderr, "Took %f ms\n", (end - start) * 1000);
socket_close(fd);
}
}
/* Stop all running threads */
stop();
/* Waiting for termination */
for (int i = 0; i < NTHREADS; ++i) {
pthread_join(threads[i], NULL);
}
fprintf(stderr, "Performed %u successfull writes\n", (unsigned)n_writes);
/* Lol.. */
freeaddrinfo(res);
return 0;
}
SIGPIPE comes when I try to connect to the kqueue server (after 10 connections are made, the server is "stuck"?). And when too many users are writing stuff, the server cannot open a new connection. (kqueue server code from http://eradman.com/posts/kqueue-tcp.html)
SIGPIPE means you're trying to write to a socket (or pipe) where the other end has already been closed (so noone will be able to read it). If you don't care about that, you can ignore SIGPIPE signals (call signal(SIGPIPE, SIG_IGN)) and the signals won't be a problem. Of course the write (or send) calls on the sockets will still be failing (with EPIPE), so you need to make you code robust enough to deal with that.
The reason that SIGPIPE normally kills the process is that its too easy to write programs that ignore errors on write/send calls and run amok using up 100% of CPU time otherwise. As long as you carefully always check for errors and deal with them, you can safely ignore SIGPIPEs
Or is it my fault?
It was your fault. TCP works. Most probably you didn't read all the data that was sent.
And when too many users are writing stuff, the server cannot open a new connection
Servers don't open connections. Clients open connections. Servers accept connections. If your server stops doing that, there something wrong with your accept loop. It should only do two things: accept a connection, and start a thread.
I'm writing a web server in C (which I suck with) using Pthreads (which I suck with even more) and I'm stuck at this point. The model for the server is boss-worker so the boss thread instantiates all worker threads at the beginning of the program. There is a global queue that stores the socket of the incoming connection(s). The boss thread is the one that adds all items (sockets) to the queue as the connections are accepted. All of the worker threads then wait for an item to be added to a global queue in order for them to take up the processing.
The server works fine as long as I connect to it less times than the number of worker threads that the server has. Because of that, I think that either something is wrong with my mutexes (maybe the signals are getting lost?) or the threads are being disabled after they run once (which would explain why if there are 8 threads, it can only parse the first 8 http requests).
Here is my global queue variable.
int queue[QUEUE_SIZE];
This is the main thread. It creates a queue struct (defined elsewhere) with methods enqueue, dequeue, empty, etc. When the server accepts a connection, it enqueues the socket that the incoming connection is on. The worker threads which were dispatched at the beginning are constantly checking this queue to see if any jobs have been added, and if there are jobs, then they dequeue the socket, connect to that port, and read/parse/write the incoming http request.
int main(int argc, char* argv[])
{
int hSocket, hServerSocket; /* handle to socket */
struct hostent* pHostInfo; /* holds info about a machine */
struct sockaddr_in Address; /* Internet socket address stuct */
int nAddressSize = sizeof(struct sockaddr_in);
int nHostPort;
int numThreads;
int i;
init(&head,&tail);
//**********************************************
//ALL OF THIS JUST SETS UP SERVER (ADDR STRUCT,PORT,HOST INFO, ETC)
if(argc < 3) {
printf("\nserver-usage port-num num-thread\n");
return 0;
}
else {
nHostPort=atoi(argv[1]);
numThreads=atoi(argv[2]);
}
printf("\nStarting server");
printf("\nMaking socket");
/* make a socket */
hServerSocket=socket(AF_INET,SOCK_STREAM,0);
if(hServerSocket == SOCKET_ERROR)
{
printf("\nCould not make a socket\n");
return 0;
}
/* fill address struct */
Address.sin_addr.s_addr = INADDR_ANY;
Address.sin_port = htons(nHostPort);
Address.sin_family = AF_INET;
printf("\nBinding to port %d\n",nHostPort);
/* bind to a port */
if(bind(hServerSocket,(struct sockaddr*)&Address,sizeof(Address)) == SOCKET_ERROR) {
printf("\nCould not connect to host\n");
return 0;
}
/* get port number */
getsockname(hServerSocket, (struct sockaddr *) &Address,(socklen_t *)&nAddressSize);
printf("Opened socket as fd (%d) on port (%d) for stream i/o\n",hServerSocket, ntohs(Address.sin_port));
printf("Server\n\
sin_family = %d\n\
sin_addr.s_addr = %d\n\
sin_port = %d\n"
, Address.sin_family
, Address.sin_addr.s_addr
, ntohs(Address.sin_port)
);
//Up to this point is boring server set up stuff. I need help below this.
//**********************************************
//instantiate all threads
pthread_t tid[numThreads];
for(i = 0; i < numThreads; i++) {
pthread_create(&tid[i],NULL,worker,NULL);
}
printf("\nMaking a listen queue of %d elements",QUEUE_SIZE);
/* establish listen queue */
if(listen(hServerSocket,QUEUE_SIZE) == SOCKET_ERROR) {
printf("\nCould not listen\n");
return 0;
}
while(1) {
pthread_mutex_lock(&mtx);
printf("\nWaiting for a connection");
while(!empty(head,tail)) {
pthread_cond_wait (&cond2, &mtx);
}
/* get the connected socket */
hSocket = accept(hServerSocket,(struct sockaddr*)&Address,(socklen_t *)&nAddressSize);
printf("\nGot a connection");
enqueue(queue,&tail,hSocket);
pthread_mutex_unlock(&mtx);
pthread_cond_signal(&cond); // wake worker thread
}
}
Here is the worker thread. This should be always running checking for new requests (by seeing if the queue is not empty). At the end of this method, it should be deferring back to the boss thread to wait for the next time it is needed.
void *worker(void *threadarg) {
pthread_mutex_lock(&mtx);
while(empty(head,tail)) {
pthread_cond_wait(&cond, &mtx);
}
int hSocket = dequeue(queue,&head);
unsigned nSendAmount, nRecvAmount;
char line[BUFFER_SIZE];
nRecvAmount = read(hSocket,line,sizeof line);
printf("\nReceived %s from client\n",line);
//***********************************************
//DO ALL HTTP PARSING (Removed for the sake of space; I can add it back if needed)
//***********************************************
nSendAmount = write(hSocket,allText,sizeof(allText));
if(nSendAmount != -1) {
totalBytesSent = totalBytesSent + nSendAmount;
}
printf("\nSending result: \"%s\" back to client\n",allText);
printf("\nClosing the socket");
/* close socket */
if(close(hSocket) == SOCKET_ERROR) {
printf("\nCould not close socket\n");
return 0;
}
pthread_mutex_unlock(&mtx);
pthread_cond_signal(&cond2);
}
Any help would be greatly appreciated. I can post more of the code if anyone needs it, just let me know. I'm not the best with OS stuff, especially in C, but I know the basics of mutexes, cond. variables, semaphores, etc. Like I said, I'll take all the help I can get. (Also, I'm not sure if I posted the code exactly right since this is my first question. Let me know if I should change the formatting at all to make it more readable.)
Thanks!
Time for a workers' revolution.
The work threads seem to be missing a while(true) loop. After the HTTP exchange and closing the socket, they should be looping back to wait on the queue for more sockets/requests.
I was trying to create an application that allows me to multicast my webcam feed over my LAN using a specific multicast address and using sendto() to just send the frame buffer. The application I am trying to build is pretty much the same as on this site
http://nashruddin.com/Streaming_OpenCV_Videos_Over_the_Network
and uses the same architecture.
Only instead of a TCP socket I use SOCK_DGRAM. The problem is that when I use the sendto() function from a different thread it tends to fail i.e it returns -1 and errno gets set to 90 (EMSGSIZE), this basically means that the packet formed is too large to be sent over the network.
But this happens even if I try to send a simple string (like "hello") to the same multicast address. This seems to work fine if the application is a single thread one. that is to say i just capture the image and multicast it all in the same thread. This is the code:
#include <netinet/in.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
#include "cv.h"
#include "highgui.h"
#define PORT 12345
#define GROUP "225.0.0.37"
CvCapture* capture;
IplImage* img0;
IplImage* img1;
int is_data_ready = 0;
int serversock, clientsock;
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
void* streamServer(void* arg);
void quit(char* msg, int retval);
int main(int argc, char** argv)
{
pthread_t thread_s;
int key;
if (argc == 2) {
capture = cvCaptureFromFile(argv[1]);
} else {
capture = cvCaptureFromCAM(0);
}
if (!capture) {
quit("cvCapture failed", 1);
}
img0 = cvQueryFrame(capture);
img1 = cvCreateImage(cvGetSize(img0), IPL_DEPTH_8U, 1);
cvZero(img1);
cvNamedWindow("stream_server", CV_WINDOW_AUTOSIZE);
/* print the width and height of the frame, needed by the client */
fprintf(stdout, "width: %d\nheight: %d\n\n", img0->width, img0->height);
fprintf(stdout, "Press 'q' to quit.\n\n");
/* run the streaming server as a separate thread */
if (pthread_create(&thread_s, NULL, streamServer, NULL)) {
quit("pthread_create failed.", 1);
}
while(key != 'q') {
/* get a frame from camera */
img0 = cvQueryFrame(capture);
if (!img0) break;
img0->origin = 0;
cvFlip(img0, img0, -1);
/**
* convert to grayscale
* note that the grayscaled image is the image to be sent to the client
* so we enclose it with pthread_mutex_lock to make it thread safe
*/
pthread_mutex_lock(&mutex);
cvCvtColor(img0, img1, CV_BGR2GRAY);
is_data_ready = 1;
pthread_mutex_unlock(&mutex);
/* also display the video here on server */
cvShowImage("stream_server", img0);
key = cvWaitKey(30);
}
/* user has pressed 'q', terminate the streaming server */
if (pthread_cancel(thread_s)) {
quit("pthread_cancel failed.", 1);
}
/* free memory */
cvDestroyWindow("stream_server");
quit(NULL, 0);
}
/**
* This is the streaming server, run as a separate thread
* This function waits for a client to connect, and send the grayscaled images
*/
void* streamServer(void* arg)
{
struct sockaddr_in server;
/* make this thread cancellable using pthread_cancel() */
pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL);
pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
/* open socket */
if ((serversock = socket(AF_INET, SOCK_DGRAM, 0)) == -1) {
quit("socket() failed", 1);
}
memset(&server,0,sizeof(server));
server.sin_family = AF_INET;
server.sin_port = htons(PORT);
server.sin_addr.s_addr = inet_addr(GROUP);
int opt = 1;
//if(setsockopt(serversock,SOL_SOCKET,SO_BROADCAST,&opt,sizeof(int))==-1){
// quit("setsockopt failed",0);
//}
// /* setup server's IP and port */
// memset(&server, 0, sizeof(server));
// server.sin_family = AF_INET;
// server.sin_port = htons(PORT);
// server.sin_addr.s_addr = INADDR_ANY;
//
// /* bind the socket */
// if (bind(serversock, (const void*)&server, sizeof(server)) == -1) {
// quit("bind() failed", 1);
// }
//
// /* wait for connection */
// if (listen(serversock, 10) == -1) {
// quit("listen() failed.", 1);
// }
//
// /* accept a client */
// if ((clientsock = accept(serversock, NULL, NULL)) == -1) {
// quit("accept() failed", 1);
// }
/* the size of the data to be sent */
int imgsize = img1->imageSize;
int bytes=0, i;
/* start sending images */
while(1)
{
/* send the grayscaled frame, thread safe */
pthread_mutex_lock(&mutex);
if (is_data_ready) {
// bytes = send(clientsock, img1->imageData, imgsize, 0);
is_data_ready = 0;
if((bytes = sendto(serversock,img1->imageData,imgsize,0,(struct sockaddr*)&server,sizeof(server)))==-1){
quit("sendto FAILED",1);
}
}
pthread_mutex_unlock(&mutex);
// /* if something went wrong, restart the connection */
// if (bytes != imgsize) {
// fprintf(stderr, "Connection closed.\n");
// close(clientsock);
//
// if ((clientsock = accept(serversock, NULL, NULL)) == -1) {
// quit("accept() failed", 1);
// }
// }
/* have we terminated yet? */
pthread_testcancel();
/* no, take a rest for a while */
usleep(1000);
}
}
/**
* this function provides a way to exit nicely from the system
*/
void quit(char* msg, int retval)
{
if (retval == 0) {
fprintf(stdout, (msg == NULL ? "" : msg));
fprintf(stdout, "\n");
} else {
fprintf(stderr, (msg == NULL ? "" : msg));
fprintf(stderr, "\n");
}
if (clientsock) close(clientsock);
if (serversock) close(serversock);
if (capture) cvReleaseCapture(&capture);
if (img1) cvReleaseImage(&img1);
pthread_mutex_destroy(&mutex);
exit(retval);
}
In the sendto() call, you reference imgsize which is initialized to img1->imageSize.
But I don't see where img1->imageSize is set and it appears that imgsize is never updated.
So first check that the imgsize value being passed to sendto() is correct.
Then check that it is not too large:
UDP/IP datagrams have a hard payload limit of 65,507 bytes. However, an IPv4 network is not required to support more than 548 bytes of payload. (576 is the minimum IPv4 MTU size, less 28 bytes of UDP/IP overhead). Most networks have an MTU of 1500, giving you a nominal payload of 1472 bytes.
Most networks allow you to exceed the MTU by breaking the datagram into IP fragments, which the receiving OS must reassemble. This is invisible to your application: recvfrom() either gets the whole reassembled packet or it gets nothing. But the odds of getting nothing go up with fragmentation because the loss of any fragment will cause the entire packet to be loss. In addition, some routers and operating systems have obscure security rules which will block some UDP patterns or fragments of certain sizes.
Finally, any given network may enforce a maximum datagram size even with fragmentation, and this is often much less than 65507 bytes.
Since you are dealing with a specific network, you will need to experiment to see how big you can reliably go.
UDP/IP at Wikipedia
IPv4 at Wikipedia
Are you absolutely sure that you don't try to send more than limit of UDP which is around 65500 bytes? From my experience you shouldn't even send more than Ethernet packet limit which is around 1500 bytes to keep best UDP reliability.
I think that right now you are trying to send much more data in a form of stream. UDP isn't a stream protocol and you can't replace TCP by it. But of course it is possible to use UDP to send video stream on multicast, but you need some protocol on top of UDP that will handle message size limit of UDP. In real world RTP protocol on top of UDP is used for such kind of task.