I'm still trying to implement inter-thread communication, and I don't understand how to make threads read a shared chunk of data when it is available. The problem is: I'm listening to a few sockets inside epoll in each thread, and if I receive information form one I need to inform other threads to read it.
I have tried eventfd with EFD_SEMAPHORE, but it doesn't suit because there's one thread that reads all data from it ahead of everybody. I cannot find a use for posix semaphores or condition variables because they can hardly be used with epoll.
Is there any way to implement such a synchronization easily?
UPDATE:
int k = epoll_wait(epollfd, n, events);
for (i = 0; i < k; i++) {
int fd = events[i].data.fd;
if (fd == eventfd) {
if (eventfd_read(eventfd, ..) != -1)
/* for each socket on this thread send latest_message to it */
}
else {
latest_message = read_message(fd);
eventfd_write(eventfd, 1)
}
}
this is the example of what I'm trying to do, assuming recvs are rare.
Actually, I thought about better idea, why do I have to bind sockets to thread when I can have a list of all sockets and just let the fastest threads send the messages, increasing something like last_reciever variable on message structure.
Related
I am trying to understand the usage of socket APIs (recv, send, select, close, etc) on parallel threads. That means using one socket file descriptor on two parallel threads. I have gone through this question. But still I am not able to find any standard doc which explains the usage of socket APIs in multiple thread. Even opengroup man page is not telling anything about this.
I also want to know whether below listed parallel thread usage scenarios are valid in POSIX socket APIs
1) Calling recv and send in two parallel threads
int main_thread() {
fd = do_connect(); //TCP or UDP
spawn_thread(recv_thread, fd);
spwan_thread(send_thread, fd);
...
}
int recv_thread(fd) {
while(1) {
recv(fd, ..)
...
}
}
int send_thread(fd) {
while(1) {
send(fd, ..)
...
}
}
2) Calling recv and send with select in two parallel threads
int recv_thread(fd) {
while(1) {
select(fd in readfd)
recv(fd, ..)
...
}
}
int send_thread(fd) {
while(1) {
select(fd in write)
send(fd, ..)
...
}
}
3) Calling recv and send with setsockopt, ioctl, fcntl in two paralle threads
int recv_thread(fd) {
int flag = 1
while(1) {
ioctl(fd, FIONBIO, &flag); //enable non block
recv(fd, ..)
flag = 0;
ioctl(fd, FIONBIO, &flag); //disable non block
...
}
}
int send_thread(fd) {
while(1) {
select(fd in write)
send(fd, ..)
...
}
}
Posix functions are thread-safe "by default":
2.9.1 Thread-Safety
All functions defined by this volume of POSIX.1-2008 shall be
thread-safe, except that the following functions need not be
thread-safe.
As many have already commented, you can safely call the mentioned calls from different threads.
Case "1" and "2" are quite typical (one thread receiving, one sending, each thread handling many connections with select()) for production code.
Case "3" is somehow odd, and probably source of troubles (it will work, the calls are valid, but it may be not straightforward to get the desired behaviour). Generally you either put the socket in non-blocking mode at the beginning and handle EAGAIN/EWOULDBLOCK errors in send()/recv() calls or blocking and use select()/pselect()/poll()/ppoll().
The sending thread in this case will randomly "find" the socket being in blocking or not-blocking mode: I wouldn't do that.
This question is a follow-up to a question which happened to be more complex than I had initially thought would be. In a program I'm writing the main thread takes care of GUI-driven data updates, a producer thread (with a number of sub-threads, because the producer task is "embarrassingly parallel") writes to the circular buffer, while the real-time consumer thread reads from it. Original platform of development was OSX/Darwin, but I'd like to make the code more portable, UNIX source compatible. Everything can easily be written in POSIX, except for the following OSX-specific GCD command for which I can't estimate a POSIX equivalent, if any. It launches the producer thread, from which its subthreads are being launched programmatically, depending on the number of available logical CPU cores:
void dproducer (bool on, int cpuNum, uData* data)
{
if (on == true)
{
data->state = starting;
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
producerSum(on, cpuNum, data);
});
}
return;
}
This is the block diagram of the program:
For clarity I'm also adding the producerSum code. It's an infinite loop whose execution can either wait for the consumer thread to do the work, or get interrupted by changing data->state, which has global scope:
void producerSum(bool on, int cpuNum, uData* data)
{
int rc;
pthread_t threads[cpuNum]; //subthreads
tData thread_args[cpuNum];
void* resulT;
static float frames [4096];
while(on){
memset(frames, 0, 4096*sizeof(float));
if( (fbuffW = (float**)calloc(cpuNum + 1, sizeof(float*)))!= NULL)
for (int i=0; i<cpuNum; ++i){
fbuffW[i] = (float*)calloc(data->frames, sizeof(float));
thread_args[i].tid = i; //ord. number of thread
thread_args[i].cpuCount = cpuNum; //counter increment step
thread_args[i].data = data;
rc = pthread_create(&threads[i], NULL, producerTN, (void *) &thread_args[i]);
if(rc != 0)printf("rc = %s\n",strerror(rc));
}
for (int i=0; i<cpuNum; ++i) rc = pthread_join(threads[i], &resulT);
//each subthread writes to fbuffW[i] and results get summed
for(UInt32 samp = 0; samp < data->frames; samp++)
for(int i = 0; i < cpuNum; i++)
frames[samp] += fbuffW[i][samp];
switch (data->state) { ... } //graceful interruption, resuming and termination mechanism
{ … } //simple code for copying frames[] to the circular buffer
pthread_cond_wait (&cond, &mutex);//wait for the real-time consumer
for(int i = 0; i < cpuNum; i++) free(fbuffW[i]); free(fbuffW);
} //end while(on)
return;
}
The syncing inside the producer thread is being successfully handled by pthread_create( ) and pthread_join( ), while necessary coordination between the producer and consumer threads is being successfully handled by a variable of pthread_mutex_t and a variable of pthread_cond_t (with corresponding locking, unlocking, broadcasting and waiting commands). uData is a program defined struct (or class instance). Any direction where to look at would help indeed.
Thanks for reading this post!
A dispatch queue is just what it sounds like: a queue, as in the standard FIFO list data structure. It holds tasks. Those tasks can be represented by Objective-C Blocks as in your code or by function pointers and context pointer values. You'll presumably need to avoid Blocks if you're aiming for cross-platform compatibility. In fact, since you only ever dispatch one task, your tasks can just encapsulate the parameters (on, cpuNum, and data) and not the code (the call to producerSum()).
The queues are serviced by threads from a thread pool. GCD manages the threads and pool. At least on OS X, there's integration with the OS such that the pool's size is governed by overall system load, which you won't be able to reproduce in a cross-platform manner.
Operations on a dispatch queue are thread-safe. This includes adding tasks to them and the worker threads removing tasks from them.
You're going to have to implement all of this. It's definitely possible, but it will be a bother. In many ways, the queues and the thread pool constitute a producer-consumer architecture. Basically, your GCD-based solution was a bit of a cheat because you just used a producer-consumer API to implement your producer-consumer design. Now, you're going to have to really implement a producer-consumer design without the crutch.
There's basically no more to it than the thread-creation and POSIX condition variables you're already using.
dispatch_async() is basically just locking the mutex for the queue of tasks, adding the task to the queue, signalling the condition variable, and unlocking the mutex. Each worker thread will just wait on the condition variable and, when it wakes, lock the mutex, pop a task off the queue if there's one, unlock the mutex, and run the task if it got one. You probably also need a mechanism to signal the worker thread that it's time to gracefully terminate.
my question is that i am using rawsocket passing high rate ( larger than 50kpps) traffic, two threads, one is to send ( read from buffer), another one is to receive ( write to buffer).
i have to use while(1) loop to make sure an infinite loop, and i cannot use usleep since then i will loose packet ( i have tried that)... now the cpu usage is 100% and i think i am buring my cpu...
here is the code:
while (1)
{
if (sendIndex == PACKET_COUNT_MAX)
{
sendIndex = 0;
}
else if (ringBuffer[sendIndex].drop == 0)
{if(sendtosocket (ringBuffer, sendIndex, rawout) < 0)
a++;
else
sendIndex++;}
else if (ringBuffer[sendIndex].drop == 1) {
ringBuffer[sendIndex].header.free = 1;
memset (ringBuffer[sendIndex].data, 0, sizeof (ringBuffer[sendIndex].data));
sendIndex++;
}
else
{
a++;
}
//nanosleep((struct timespec[]){{0, 5}}, NULL);
}
Thanks in advance!!!!!!!
Lisa
You need to pass the control over to the kernel. The command you may find useful is select. Check out the whole story on http://manpages.courier-mta.org/htmlman2/select.2.html. For more info, http://www.gnu.org/software/libc/manual/html_node/Waiting-for-I_002fO.html.
It's all about knowing you have nothing else to do except wait for input from the network. Or the file system. Or anything else that is a file descriptor (U*ix lingo). So, you let the kernel awake you once you've got something to process.
You can try
Increasing the receive buffer
Slowing down the sendto with a sleep, that should not lose packets
Use the MSG_WAITALL flag with recvfrom to make it a blocking read, and make sure the socket was not opened with SOCK_NONBLOCK or O_NONBLOCK
You need sane synchronization between your threads. This includes:
Using locks of some kind to ensure that a variable isn't read by one thread while it is, or might be, modified in another.
Using some kind of waiting scheme so that the sending thread can wait for there to be work for it do without spinning.
Check out pthread_mutex_lock and pthread_cond_wait (assuming you're using POSIX threads).
I am working on one project in which i need to read from 80 or more clients and then write their o/p into a file continuously and then read these new data for another task. My question is what should i use select or multithreading?
Also I tried to use multi threading using read/fgets and write/fputs call but as they are blocking calls and one operation can be performed at one time so it is not feasible. Any idea is much appreciated.
update 1: I have tried to implement the same using condition variable. I able to achieve this but it is writing and reading one at a time.When another client tried to write then it cannot able to write unless i quit from the 1st thread. I do not understand this. This should work now. What mistake i am doing?
Update 2: Thanks all .. I am able to succeeded to get this model implemented using mutex condition variable.
updated Code is as below:
**header file*******
char *mailbox ;
pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER ;
pthread_cond_t writer = PTHREAD_COND_INITIALIZER;
int main(int argc,char *argv[])
{
pthread_t t1 , t2;
pthread_attr_t attr;
int fd, sock , *newfd;
struct sockaddr_in cliaddr;
socklen_t clilen;
void *read_file();
void *update_file();
//making a server socket
if((fd=make_server(atoi(argv[1])))==-1)
oops("Unable to make server",1)
//detaching threads
pthread_attr_init(&attr);
pthread_attr_setdetachstate(&attr,PTHREAD_CREATE_DETACHED);
///opening thread for reading
pthread_create(&t2,&attr,read_file,NULL);
while(1)
{
clilen = sizeof(cliaddr);
//accepting request
sock=accept(fd,(struct sockaddr *)&cliaddr,&clilen);
//error comparison against failire of request and INT
if(sock==-1 && errno != EINTR)
oops("accept",2)
else if ( sock ==-1 && errno == EINTR)
oops("Pressed INT",3)
newfd = (int *)malloc(sizeof(int));
*newfd = sock;
//creating thread per request
pthread_create(&t1,&attr,update_file,(void *)newfd);
}
free(newfd);
return 0;
}
void *read_file(void *m)
{
pthread_mutex_lock(&lock);
while(1)
{
printf("Waiting for lock.\n");
pthread_cond_wait(&writer,&lock);
printf("I am reading here.\n");
printf("%s",mailbox);
mailbox = NULL ;
pthread_cond_signal(&writer);
}
}
void *update_file(int *m)
{
int sock = *m;
int fs ;
int nread;
char buffer[BUFSIZ] ;
if((fs=open("database.txt",O_RDWR))==-1)
oops("Unable to open file",4)
while(1)
{
pthread_mutex_lock(&lock);
write(1,"Waiting to get writer lock.\n",29);
if(mailbox != NULL)
pthread_cond_wait(&writer,&lock);
lseek(fs,0,SEEK_END);
printf("Reading from socket.\n");
nread=read(sock,buffer,BUFSIZ);
printf("Writing in file.\n");
write(fs,buffer,nread);
mailbox = buffer ;
pthread_cond_signal(&writer);
pthread_mutex_unlock(&lock);
}
close(fs);
}
I think for the the networking portion of things, either thread-per-client or multiplexed single-threaded would work fine.
As for the disk I/O, you are right that disk I/O operations are blocking operations, and if your data throughput is high enough (and/or your hard drive is slow enough), they can slow down your network operations if the disk I/O is done synchronously.
If that is an actual problem for you (and you should measure first to verify that it really is a problem; no point complicating things if you don't need to), the first thing I would try to ameliorate the problem would be to make your file's output-buffer larger by calling setbuffer. With a large enough buffer, it may be possible for the C runtime library to hide any latency caused by disk access.
If larger buffers aren't sufficient, the next thing I'd try is creating one or more threads dedicated to reading and/or writing data. That is, when your network thread wants to save data to disk, rather than calling fputs()/write() directly, it allocates a buffer containing the data it wants written, and passes that buffer to the IO-write thread via a (mutex-protected or lockless) FIFO queue. The I/O thread then pops that buffer out of the queue, writes the data to the disk, and frees the buffer. The I/O thread can afford to be occasionally slow in writing because no other threads are blocked waiting for the writes to complete. Threaded reading from disk is a little more complex, but basically the IO-read thread would fill up one or more buffers of in-memory data for the network thread to drain; and whenever the network thread drained some of the data out of the buffer, it would signal the IO-read thread to refill the buffer up to the top again. That way (ideally) there is always plenty of input-data already present in RAM whenever the network thread needs to send some to a client.
Note that the multithreaded method above is a bit tricky to get right, since it involves inter-thread synchronization and communication; so don't do it unless there isn't any simpler alternative that will suffice.
Either select/poll or multithreading is ok if you you program solves the problem.
I' guess your program would be io-bound as the number of clients grows up, as you have disk read/write frequently. So it would not speed up to have multiple threads doing the io operation. Polling may be a better choice then
You can set a socket that you get from accept to be non-blocking. Then it is easy to use select to find out when there is data, read the number of bytes that are available and process them.
With (only) 80 clients, I see no reason to expect any significant difference from using threads unless you get very different amounts of data from different clients.
As for threads, I have mutex and conditionals so I could manipulate them easily.
However, if I create two processes by fork(), how could I make them alternating?
Or, is there any way to create a "critical section" for processes?
I intended to make a program that prints "r" and "w" alternatively, here is the code.
#include <stdio.h>
#include <stdlib.h>
int pipe_1[2];
int flag = 0;
void r();
void w();
int main() {
pipe(pipe_1);
if(fork())
r();
else
w();
}
void r() {
int count = 0;
while(1) {
printf("%d \n", flag);
if (count == 10)
exit(0);
if(flag == 0) {
puts("r");
flag = 1;
count++;
while(flag == 1)
;
}
}
}
void w() {
while(1) {
if(flag == 1) {
puts("w");
flag = 0;
while(flag == 0)
;
}
}
}
The out put is only:
0
r
Then it seems to enter a infinite loop.
What's the problem?
And what's the right way to make alternating processes?
Thanks.
This may be overwhelming, but there are TONS of primitives you could use. See here for a list.
http://beej.us/guide/bgipc/output/html/singlepage/bgipc.html
Glancing at the list, just about all of those could be used. Some are more like traditional pthread synchronization primitives, others are higher-level, but can still be used for synchronization.
For example, you could just open a TCP socket between the two and send messages when it's the other side's turn. Maybe with an incrementing number.
Something perhaps more traditional would be semaphores:
http://beej.us/guide/bgipc/output/html/singlepage/bgipc.html#semaphores
Also, this assumes a modern unix-like platform. Windows is likely very different.
It looks like you have a pipe already, so you can use that to have each side send a message to the other after it's done its print. The other side would do a blocking read, then return when the message was sent, do it's print, send a message back, and go back to a blocking read.
They are separate processes, so each has it's own flag; r changing its doesn't affect w's.
In order for two processes to communicate with each other without sharing the same address space (like threads do), they must use Inter-Process Communication means (aka IPC). Some of the IPC mechanisms are: shared memory, semaphore, pipes, sockets, message queues and more. Most of the time, IPC mechanisms are operating system specific. However, many ideas are general enough so it is possible to come up with a portable implementations, which Boost project did as part of Boost.Interprocess library. What I think you should take a look at first is Synchronization Mechanisms section. Note, however, that this is a C++ library. I am not aware of any C library that is as good as Boost.
Hope it helps. Good Luck!