I want to design a TCP server which listens for x clients. X is small, say around 10 and its fixed. Clients may connect at any time. Once the connection is established (accept() successful) I spawn a thread for each client and handle them.
In this handle() function I want to send commands to client and receive the data accordingly.
Problem:
Once a command has been sent from the server the client responds by sending data continuously. How do I send a command back to the client to stop it? As with the current code I'm in a loop receiving data from the client.
I don't know how to send the command from server thread while receive is in progress, like should I need to have another thread (to send cmds) once the connection is established?
How to continuously receive data from clients and also send commands at the same time? Sending commands to each client based on user inputs. (Say user wants client1 to start sending data, then I have to send START to client1. And user wants to stop the client1 from sending, so I need to send STOP to client1 And if user wants data3 cmd to be sent to client 4 then send command DATA3 to client4 etc. How do identify the client in this case? Basically forming a small protocol**
The below code is working where I can listen on socket and client connects and sends data. I'm not sure of how to send user inputted commands to the right client (say client4) and also receive at same time.
If you want to really continously stream data and in parallel want to exchange commands you won't get around an addtional connection to establish the command channel. The alternativ would be some kind of multiplexing. Stream a chunk of data, check for commands, stream the next chunk, check for commands again ... - complicated and error prone as the stream is continously interupted ...
The stone old ftp protocol does something similar: http://en.wikipedia.org/wiki/Ftp and https://www.rfc-editor.org/rfc/rfc959 (see ascii art in chapter 2.3)
Presuming you want to have another thread initiate the request to send a command, you can accomplish what you want using standard asynchronous i/o, adding in another channel - a pipe - to receive commands from the other thread. Pseudocode:
Master thread:
while(1) {
newsocket = accept(listen socket)
pipefds = pipe()
new thread(Receiver, newsocket, pipefds.read)
}
Receiver thread:
while(1) {
readfds = [ pipefds.read, newsocket ]
poll( readfds ) // wait for there to be data on one of the fds
if (data ready on newsocket) {
read (newsocket)
process data
}
if (data ready on pipefds.read) {
read (pipefds.read)
send command
}
}
Commander thread:
write (pipefds.write, command)
The select in the main Receiver loop will wake up whenever there is data to read on the socket, OR if another thread has sent a command that needs to be sent to the remote connection.
Key syscalls to look up info on are poll (or select), and pipe.
WOO HOO! You've decided to dive into a pretty hairy subject, my friend.
Let me first rephrase your problem: your program can only wait for one thing at a time. If it's waiting on receive, it can't be waiting on send. So you just can't send and receive at the same time.
This is solved by multiplexing: waiting on multiple things.
Googling keywords: io, multiplexing, select, poll.
SO related question: read and write to same socket (TCP) using select
Another approach is to enter a nonblocking-read -> nonblocking-write -> sleep loop. This is obviously less than optimal, but may be enough for your case.
I've had some fun in the past designing my own bi-directional protocol for low level devices that can't communicate at the same time. One method you can use is mutual yielding: establish a mechanism for passing messages to and from the client and server. Stream any commands or messages you need to send, then yield the stream to the other side. The other side will then stream any messages or commands, then yield the stream to the original side. The only problem with this mechanism is that it's very laggy with high-ping connections, such as international internet connections.
This has been mentioned already, but I might as well rehash it. Computers have multiplexing built in to their networking hardware already, so you can do "concurrent" send/recv calls. Just run a pair of threads for each connection: a recv thread and a send thread. This should be the most robust solution.
Related
I am building a client/server protocol using libwebsockets library. In the poll thread, each callback on incoming packets triggers a certain method to either save data, respond accordingly, and so on. However, if there are other incoming packets that would trigger LWS_CALLBACK_CLIENT_RECEIVE, the processing of the previous task is left undone.
Is there a way to finish the tasks before dealing with the packets received? Should LWS_CALLBACK_CLIENT_RECEIVE start new threads to handle the processing of data?
This was a typical case of multi-threading: one thread receives the packets, fills a buffer and goes back to listening for incoming packets. Working threads will empty the buffer.
I am fairly new to networking concepts in C, and was wondering about following.
Say I have client and server.
On the client side, I have such code:
1. send(connfd, var1, var1Size);
2. read(connfd, &x, size1);
3. close(connfd);
The server also does one receive and one send, e.g.,
1. read(connfd, &var, size);
2. send(connfd, var1, varSize);
My question is following.
On the client side, after the client does send -- it takes some time
before the data arrives at server, before server reads it, and sends back
right?
So could not it happen that after client code runs send, then
it directly jumps to read, but by this time, the server has
not yet managed to prepare response and send it back --
so the read call on the client side (line 2), will receive nothing.
And connection will terminate right? (program will exit).
Is it how it may happen?
This really has nothing to do with C, it's about how networking protocols and I/O work.
The answer is that unless you go out of your way to make the I/O non-blocking, the send() and recv() calls are synchronous, i.e. they will block if necessary, to wait for available outgoing bandwidth or incoming data.
So the case you describe will typically not happen, the connection will not terminate.
Both send and read are blocking, meaning that the call will block until the command is finished.
For send this means that the call will not finish until the program has send the data (this not necessarily means that the data has arrived, but that the data was passed to a buffer (OS-handled) (depends on the protocol)).
for receive this means that the call will block until there is some data to receive.
So the client, after sending will block in the receive call until the server sends a response. And the server will block in the read call until the client sends the data. The only malfunction here is if the client calls send before the server is started listening on the socket.
I have a server-program which processes audio-data and passes it thru to the audio-drivers.
The server-program copies the audio-data and puts the copy in a named FIFO in a seconds thread.
If there is no client reading on the other side of the FIFO it does not matter, because it just blocks the FIFO-thread.
Now I would like to add a "control"-functionality like "increase volume, play faster etc." so the eventually connected client can control the server-program.
The important thing is: If the client eventually disconnects (through close() or abort) the server has detect this and should fall back into normal mode and forget all the commands from the client.
I have never used sockets until now, so I'm not sure what's the best way:
use the FIFO from server->client as it is and add a socket just for client->server communication?
use one socket to stream server->client and give commands from client->server (in byte-format?)
I would use "AF_UNIX, SOCK_STREAM" for the socket. Is #2 the better variant? And how can I determine if the client disconnected without a close()?
i vote option nr.2 and a possible solution for that is:
1-create socket[sock_stream....];
2-fork()[inherits the socket descriptor];
-father[use to read];
-son[use to write];
you can implement to detect a client disconnection when read() from socket descriptor returns 0bytes
I have to receive data from 15 different clients each of them sending on 5 different ports. totally 15 *5 sockets.
for each client port no is defined and fixed. example client 1 ,ports 3001 to 3005. client 2 ,ports 3051 to 3055 etc. They have one thing in common say first port (3001 , 3051) is used to send commands. other ports send some data.
After receiving the data i have to check for checksum. keep track of recvd packets, Re request the packet if lost and also have to write to files on hard disk.
Restriction I cannot change the above design and i cannot change from UDP to TCP.
The two methods i'm aware of after reading are
asynchronous multiplexing using select().
Thread per socket.
I tried the first one and i'm stuck at the point when i get the data. I'm able to receive data. I have some processing to do so i want to start a thread for each socket (or) for sockets to handle (say all first ports, all second, etc ..i.e.3001,3051 etc)
But here if client sends any data then FD_ISSET becomes true , so if i start a thread ,then it becomes thread for every message.
Question:
How to add thread code here, Say if i include pthread_create inside if(FD_ISSET .. ) then for every message that i receive i create a thread. But i wanted a thread per socket.
while(1)
{
int nready=0;
read_set = active_set;
if((nready = select(fdmax+1,&read_set,NULL,NULL,NULL)) == -1)
{
printf("Select Errpr\n");
perror("select");
exit(EXIT_FAILURE);
}
printf("number of ready desc=%d\n",nready);
for(index=1;index <= 15*5;index++)
{
if(FD_ISSET(sock_fd[index],&read_fd_set))
{
rc = recvfrom(sock_fd[index],clientmsgInfo,MSG_SIZE,0,
(struct sockaddr *)&client_sockaddr_in,
&sockaddr_in_length);
if(rc < 0)
printf("socket %d down\n",sock_fd[index]);
printf("Recieved packet from %s: %d\nData: %s\n\n", inet_ntoa(client_sockaddr_in.sin_addr), ntohs(client_sockaddr_in.sin_port), recv_client_message);
}
} //for
} //while
create the threads at the startup of program and divide them to handle data, commmands e.t.c.
how?
1. lets say you created 2 threads, one for data and another for the commands.
2. make them sleep in the thread handler or let them wait on a lock that the main thread
acquired, seems to be that mainthread got two locks one for each of them.
3. when any client data or command that got into the recvfrom at mainthread, depending on the
type of the buffer(data, commands), copy the buffer into the shared data by mainthread and
other threads and unlock the mutex.
4. at threads lock the mutex so that mainthread wont' corrupt the data and once processing is
done at the threads unlock and sleep.
The better one would be to have a queue, that fills up by main thread and can be accessed element wise by the other threads.
I assume that each client context is independent of the others, ie. one client socket group can be managed on its own, and the data pulled from the sockets can be processed alone.
You express two possibilities of handling the problem:
Asynchronous multiplexing: in this setting, the sockets are all managed by one single thread. This threads selects which socket must be read next, and pulls data out of it
Thread per socket: in this scenario, you have as many threads as there are sockets, or more probably group of sockets, ie. clients - this the interpretation I will build from.
In both cases, threads must keep ownership of their respective resources, meaning sockets. If you start moving sockets around between threads, you will make things more difficult that it needs to be.
Outside the work that needs to be done, you will need to handle thread management:
How do threads get started?
How and when are they stopped?
What are the error handling policies?
Your question doesn't cover these issues, but they might play a significant role in your final design.
Scenario (2) seems simpler: you have one main "template" (I use the word in a general meaning here) for handling a group of sockets using select on them, and in the same thread receive and process the data. It's quite straightforward to implement, with a struct to contain the context specific data (socket ports, pointer to function for packet processing), and a single function looping on select and process, plus perhaps some other checks for errors and thread life management.
Scenario (1) requires a different setup: one I/O thread reads all the packets and pass them on to specialized worker threads to do the processing. If processing error occurs, worker threads will have to generate the adhoc packet to be sent to the client, and pass it to the I/O thread for sending. You will need packet queues both ways to allow communication between I/O and workers, and have the I/O thread check the worker queues somehow for resend requests. So this solution is a bit more expensive in terms of developement, but reduce the I/O contention to one single point. It's also more flexible, in case some processing must be done against data coming from several clients, or if you want to chain up processing somehow. For instance, you could have instead one thread per client socket, and then one other thread per client group of socket further down the work pipeline, with each step of the pipeline interconnected by packet queue.
A blend of both solution can of course be implemented, with one IO thread per client, and pipelined worker threads.
The advantage of both outlined solutions is the fixed number of threads: no need to spawn and destroy threads on demand (although you could design a thread pool to handle that as well).
For a solution involving moving sockets between threads, the questions are:
When should these resources be passed on? What happens after a worker thread has read a packet? Should it return the socket to the IO thread, or risk a blocking read on the socket for the next packet? If it does a select to poll the socket for more packets, we fall in scenario (2), where each client will has its own I/O thread when there is network trafic from all of them, in which case what is the gain of the initial I/O thread doing the select?
If it passes the socket back, should the IO thread wait for all workers to give back their socket before initiating another select? If it waits, it takes the risk of making unserved client wait for packets already in the network buffers, inducing processing lag. If it does not wait, and return to select to avoid lag on unserved sockets, then the served ones will have to wait for the next wake up to see their sockets back in the select pool.
As you can see, the problem is difficult to handle. That's the reason why I recommend exclusive sockets ownership by threads as described in scenarii (1) and (2).
Your solution requires a fixed, relatively small, number of connections.
Create a help procedure that creates thread procedures that listen to each of the five ports and block on the recvfrom(), process the data, and block again. You can then call the helper 15 times to create the threads.
This avoids all polling, and allows Linux to schedule each thread when the I/O completes. No CPU used while waiting, and this can scale to somewhat larger solutions.
If you need to scale massively, why not use a single set of ports, and get the partner address from the client_sockaddr_in structure. If the processing takes a material amount of time, you could extend it by keeping a pool of threads available and assign a new one each time a message is received and continue processing the message thereafter, and adding the thread back to the pool after the response is sent.
In C, I understood that if we close a socket, it means the socket will be destroyed and can be re-used later.
How about shutdown? The description said it closes half of a duplex connection to that socket. But will that socket be destroyed like close system call?
This is explained in Beej's networking guide. shutdown is a flexible way to block communication in one or both directions. When the second parameter is SHUT_RDWR, it will block both sending and receiving (like close). However, close is the way to actually destroy a socket.
With shutdown, you will still be able to receive pending data the peer already sent (thanks to Joey Adams for noting this).
None of the existing answers tell people how shutdown and close works at the TCP protocol level, so it is worth to add this.
A standard TCP connection gets terminated by 4-way finalization:
Once a participant has no more data to send, it sends a FIN packet to the other
The other party returns an ACK for the FIN.
When the other party also finished data transfer, it sends another FIN packet
The initial participant returns an ACK and finalizes transfer.
However, there is another "emergent" way to close a TCP connection:
A participant sends an RST packet and abandons the connection
The other side receives an RST and then abandon the connection as well
In my test with Wireshark, with default socket options, shutdown sends a FIN packet to the other end but it is all it does. Until the other party send you the FIN packet you are still able to receive data. Once this happened, your Receive will get an 0 size result. So if you are the first one to shut down "send", you should close the socket once you finished receiving data.
On the other hand, if you call close whilst the connection is still active (the other side is still active and you may have unsent data in the system buffer as well), an RST packet will be sent to the other side. This is good for errors. For example, if you think the other party provided wrong data or it refused to provide data (DOS attack?), you can close the socket straight away.
My opinion of rules would be:
Consider shutdown before close when possible
If you finished receiving (0 size data received) before you decided to shutdown, close the connection after the last send (if any) finished.
If you want to close the connection normally, shutdown the connection (with SHUT_WR, and if you don't care about receiving data after this point, with SHUT_RD as well), and wait until you receive a 0 size data, and then close the socket.
In any case, if any other error occurred (timeout for example), simply close the socket.
Ideal implementations for SHUT_RD and SHUT_WR
The following haven't been tested, trust at your own risk. However, I believe this is a reasonable and practical way of doing things.
If the TCP stack receives a shutdown with SHUT_RD only, it shall mark this connection as no more data expected. Any pending and subsequent read requests (regardless whichever thread they are in) will then returned with zero sized result. However, the connection is still active and usable -- you can still receive OOB data, for example. Also, the OS will drop any data it receives for this connection. But that is all, no packages will be sent to the other side.
If the TCP stack receives a shutdown with SHUT_WR only, it shall mark this connection as no more data can be sent. All pending write requests will be finished, but subsequent write requests will fail. Furthermore, a FIN packet will be sent to another side to inform them we don't have more data to send.
There are some limitations with close() that can be avoided if one uses shutdown() instead.
close() will terminate both directions on a TCP connection. Sometimes you want to tell the other endpoint that you are finished with sending data, but still want to receive data.
close() decrements the descriptors reference count (maintained in file table entry and counts number of descriptors currently open that are referring to a file/socket) and does not close the socket/file if the descriptor is not 0. This means that if you are forking, the cleanup happens only after reference count drops to 0. With shutdown() one can initiate normal TCP close sequence ignoring the reference count.
Parameters are as follows:
int shutdown(int s, int how); // s is socket descriptor
int how can be:
SHUT_RD or 0
Further receives are disallowed
SHUT_WR or 1
Further sends are disallowed
SHUT_RDWR or 2
Further sends and receives are disallowed
This may be platform specific, I somehow doubt it, but anyway, the best explanation I've seen is here on this msdn page where they explain about shutdown, linger options, socket closure and general connection termination sequences.
In summary, use shutdown to send a shutdown sequence at the TCP level and use close to free up the resources used by the socket data structures in your process. If you haven't issued an explicit shutdown sequence by the time you call close then one is initiated for you.
I've also had success under linux using shutdown() from one pthread to force another pthread currently blocked in connect() to abort early.
Under other OSes (OSX at least), I found calling close() was enough to get connect() fail.
"shutdown() doesn't actually close the file descriptor—it just changes its usability. To free a socket descriptor, you need to use close()."1
Close
When you have finished using a socket, you can simply close its file descriptor with close; If there is still data waiting to be transmitted over the connection, normally close tries to complete this transmission. You can control this behavior using the SO_LINGER socket option to specify a timeout period; see Socket Options.
ShutDown
You can also shut down only reception or transmission on a connection by calling shutdown.
The shutdown function shuts down the connection of socket. Its argument how specifies what action to perform:
0
Stop receiving data for this socket. If further data arrives, reject it.
1
Stop trying to transmit data from this socket. Discard any data waiting to be sent. Stop looking for acknowledgement of data already sent; don’t retransmit it if it is lost.
2
Stop both reception and transmission.
The return value is 0 on success and -1 on failure.
in my test.
close will send fin packet and destroy fd immediately when socket is not shared with other processes
shutdown SHUT_RD, process can still recv data from the socket, but recv will return 0 if TCP buffer is empty.After peer send more data, recv will return data again.
shutdown SHUT_WR will send fin packet to indicate the Further sends are disallowed. the peer can recv data but it will recv 0 if its TCP buffer is empty
shutdown SHUT_RDWR (equal to use both SHUT_RD and SHUT_WR) will send rst packet if peer send more data.
linux: shutdown() causes listener thread select() to awake and produce error. shutdown(); close(); will lead to endless wait.
winsock: vice versa - shutdown() has no effect, while close() is successfully catched.