I have a client-server model. A multithreaded client sends a message to the server over the TCP sockets. The server is also multiple threaded with each request handled by a thread from the worker pool.
Now, the server must send back the message to the client via shared-memory IPC. For example:
multi threaded client --- GET /a.png --> server
|
|
one worker
|
|
\ /
puts the file descriptor into the shared memory
When worker thread adds the information into the shared memory, how do I make sure that it is read by the same client that requested it?
I feel clueless here as to how to proceed. Currently, I have created one segment of shared memory and there are 20 threads on the server and 10 threads on the client.
While you can use IPC between threads, it's generally not a good idea. Threads share all memory anyway since they are part of the same process and there are very efficient mechanisms for communications between threads.
It might just be easier to have the same thread handle a request all the way through. That way, you don't have to hand off a request from thread to thread. However, if you have a pool of requests that are being worked on, it often makes sense to have a thread be able to "put down" a request and then later be able to have that thread or a different thread "pick up" the request.
The easiest way to do this is to make all the information related to the request live in a single structure or object. Use standard thread synchronization tools (like mutexes) to control finding the object, taking ownership of it, and so on.
So when an I/O thread receives a request, it creates a new request object, acquires a mutex, and adds it to the global collection of requests the server is working on. Worker threads can check this global collection to see which requests need work or they can be explicitly dispatched by the thread that created the request.
Related
I'm trying to architecture the main event handling of a libuv-based application. The application is composed of one (or more) UDP receivers sharing a socket, whose job is to delegate processing incoming messages to a common worker pool.
As the protocol handled is stateful, all packets coming from any given server should always be directed to the same worker – this constraint seem to make using LibUV built-in worker pool impossible.
The workers should be able to send themselves packets.
As such, and as I am new to LibUV, I wanted to share with you the intended architecture, in order to get feedback and best practices about it.
– Each worker run their very own LibUV loop, allowing them to send directly packets over the network. Additionally, each worker has a dedicated concurrent queue for sending it messages.
– When a packet is received, its source address is hashed to select the corresponding worker from the pool.
– The receiver created a unique async handle on the receiver loop, to act as callback when processing has finished.
– The receiver notifies the worker with an async handle that a new message is available, which wakes up the worker, that starts to process all enqueued messages.
– The worker thread calls the async handle on the receiver queue, which will cause the receiver to return the buffer to pool and free all allocated resources (as such, the pool does not need to be thread-safe).
The main questions I have would be:
– What is the overhead of creating an async handle for each received message? Is it a good design?
– Is there any built-in way to send a message to another event loop?
– Would it be better to send outgoing packets using another loop, instead of doing it right from the worker loop?
Thanks.
I am writing an HTTP reverse-proxy in C using Libevent and I would like to implement multithreading to make use of all available CPU cores. I had a look at this example: http://roncemer.com/software-development/multi-threaded-libevent-server-example/
In this example it appears that one thread is used for the full duration of a connection, but for HTTP 1.1 I don't think this will be the most effective solution as connections are kept alive by default after each request so that they can be reused later. I have noticed that even one browser panel can open several connections to one server and keep them open until the tab is closed which would immediately exhaust the thread pool. For an HTTP 1.1 proxy there will be many open connections but only very few of them actively transferring data at a given moment.
So I was thinking of an alternative, to have one event base for all incoming connections and have the event callback functions delegate to worker threads. This way we could have many open connections and make use of a thread only when data arrives on a connection, returning it back to the pool once the data has been dealt with.
My question is: is this a suitable implementation of threads with Libevent?
Specifically – is there any need to have one event base per connection as in the example or is one for all connections sufficient?
Also – are there any other issues I should be aware of?
Currently the only problem I can see is with burstiness, when data is received in many small chunks triggering many read events per HTTP response which would lead to a lot of handing-off to worker threads. Would this be a problem? If it would be, then it could be somewhat negated using Libevent's watermarking, although I'm not sure how that works if a request arrives in two chunks and the second chunk is sufficiently small to leave the buffer size below the watermark. Would it then stay there until more data arrives?
Also, I would need to implement scheduling so that a chunk is only sent once the previous chunk has been fully sent.
The second problem I thought of is when the thread pool is exhausted, i.e. all threads are currently doing something, and another read event occurs – this would lead to the read event callback blocking. Does that matter? I thought of putting these into another queue, but surely that's exactly what happens internally in the event base. On the other hand, a second queue might be a good way to organise scheduling of the chunks without blocking worker threads.
I am working on a UDP server/client application.
I want my server to be able to handle 40 clients at a time. I have thought of creating 40 threads at server side, each thread handling one client. Clients are distinguished on the basis of IP addresses and there is one thread for each unique IP address.
Whenever a client sends some data to a server, the main thread extracts the IP address of the client and decides which thread will process this specific client. Is there a better way to achieve this functionality?
There are different approaches for scale able server application, one thread per client seems good if no of clients are not many, another most efficient approach to accomplish this task is to use thread pool. These threads are work as task base when ever you have any new task assign this task to free worker thread.
Take a look at this project, I think it is very helpful to start with: http://www.codeproject.com/Articles/16935/A-Chat-Application-Using-Asynchronous-UDP-sockets
With IPAddress.Any, we specify that the server should accept client
requests coming on any interface. To use any particular interface, we
can use IPAddress.Parse (“192.168.1.1”) instead of IPAddress.Any. The
Bind function then bounds the serverSocket to this IP address. The
epSender identifies the clients from where the data is coming.
With BeginReceiveFrom, we start receiving the data that will be sent
by the client. Note that we pass epSender as the last parameter of
BeginReceiveFrom, the AsyncCallback OnReceive gets this object via the
AsyncState property of IAsyncResult, and it then processes the client
requests (login, logout, and send message to the users). Please see
the code attached to understand the implementation of OnReceive.
A better way would be to use the Proactor pattern (take a look at Boost.Asio library), instead of creating thread per client. With such an approach your application would have much better scalability and performace (especially on platforms that have native async i/o)
Besides, with this technique the threading would be de-coupled from the concurrency, meaning that you don't necessarily have to mess with multi-threading with all its complications.
I am trying to implement a TCP server which is a part of a larger project. Basically the server should be able to maintain a TCP connection with any number of clients (a minimum of 32) and service any client that requests servicing. In our scenario the thing is that it will be assumed that once the client is connected to the server, it will never close the connection unless some sort of failure occurs (e-g the machine running the client breaks down ) and it will repeatedly request service from the server. Same is the case with all the other clients i-e each will maintain a connection with the server and perform transactions. so to sum up the server will be at the same time maintaining the connection with the clients while simultaneously serving each client as needed and should also have the ability to accept any other client connections that want to connect to the server.
Now I implemented the above functionality using the select() system call of the berkely socket API and it works fine when we have a small number of clients (say 10). But the server needs to be scaled to the highest possible level as we are implementing it on a 16 core machine. For that I looked through various multi threading design techniques e-g one thread per client etc and the best one in my opinion would be a thread pool design. Now As I was about to implement that I ran into some problems:
If I designate the main thread to accept any number of incoming connections and save each connections File descriptor in a data structure, and I have a pool of threads, how would I get the threads to poll that whether a particular client is requesting for service or not. The design is simple enough for scenarios in which client contacts the server and after getting the service it closes the connection so that we can pick a thread from a pool, service the client and then push it back into the pool for future connection handling. But when we have to service a set of clients that maintain a connection and request services intermittently, what would be the best approach to do this. All help will be much appreciated as I am really stuck in this.
Thanks.
Use pthreads, with one thread per CPU plus one extra thread.
The extra thread (the main thread) listens for new connections with the listen() system call, accepts the new connections with accept(), then determines which worker thread currently has the least number of connections, acquires a lock/mutex for that worker thread's "pending connections" FIFO queue, places the descriptor for the accepted connection onto the worker thread's "pending connections" FIFO queue, and sends a "check your queue" notification (e.g. using a pipe) to the worker thread.
The worker threads use "select()", and send/receive data to whatever connections they've accepted. If/when a worker thread receives a "check your queue" notification from the main thread it would acquire the lock/mutex for its "pending connections" FIFO queue and add any newly accepted connections to its "fd_set" list.
For 1024 connections and 16 CPUs; you might end up with one main thread waiting for new connections (but doing almost nothing as you wouldn't be expecting many new connections), and 16 worker threads handling an average of 64 connections each.
One thread per client is almost certainly the best design. Make sure you always have at least one thread blocked in accept waiting for a new connection - this means that after accept succeeds, you might need to create a new thread before proceeding if it was the last one. I've found semaphores to be a great primitive for keeping track of the need to spawn new listening threads.
This question is related to Many processes executed by one thread and Socket server or file server implementation using multiple threads: concept not clear.
I am still unclear about a few things. Now the client and server of a socket server or a file server need not be in different machines(ofcourse it can be too).
The requests the servers receive are from different PROCESSES but they are processed by threads(say one per process) and these task threads belong to a different process(the server process). What I am confused about is how can calls from different processes be processed by threads of a single process and these threads communicate using 'shared memory" architecture that is so "THREAD" and very unlike "PROCESSES"
Thanks
Some simple ground work. Your server process contains one or more threads to process requests from any number of client processes. The clients and server can be on either the same or different machines. Clients and server are "connected" by sockets which are used to send requests from the client to the server. The same socket will be used to provide a response to the client once the request has been processed. Each client will have a unique connection to the server.
There are many ways to implement a server as described above. One possibility is that the server has one thread which handles the sockets using select(). Lets call this the Main Thread. The server process will also have several threads that will be responsible for processing the requests and responding to the clients. Lets call these Worker Threads.
When the Main Thread receives a message from one of the its client's sockets, the Main Thread will take this request and hand it to one of the Worker Threads for processing. The Worker Thread will take that request and process it and then respond using the original socket.
This server model uses a producer/consumer model where the Main Thread is the producer (in that it takes a request from the socket and produces a piece of work requiring processing) and the consumers are the Worker Threads.
There a several challenges in implementing this type of server all of which are documented and discussed in various Data Structure and Algorithms text, not the least of which are:
How do the Main and Worker threads communicate?
How do I protect data shared by the various threads from simultaneous modification?
How do I select which Worker thread should process a request?
I hope this helps.