I am implementing a server that serves multiple clients and I need some server's variable to be in shared memory so a client actually sees what another client has eventually edited.
I tried looking around but I didn't understand if there is any way to achieve this by using fork() or I totally need to change the way I handle clients. In particular I don't know if I should implement piped processes or threads. Also, what is the simpler way?
This is my code after declaring int var in the main:
while(1) {
printf("Waiting connection...\n");
if ((connfd = accept(listenfd, (struct sockaddr *) NULL, NULL)) < 0) {
perror("Accept Error");
exit(1);
}
if ((pid = fork()) == 0) {
close (listenfd);
printf("Variable: %d\n", var); // var = 0
var = 1;
printf("Variable: %d\n", var); // var = 1
exit(0);
}
close(connfd);
}
When I connect with another client I see again var = 0 because the child generates a copy of the parent process.
Tried using static or declaring global variables outside of the main() but as I understood it has no effect.
fork does not duplicate variables but the entire address space (by definition of fork) of the invoking process.
You might want to use some shared memory, but then you should care about synchronization. Read shm_overview(7) and sem_overview(7) (you could share some memory using mmap(2) but you need to synchronize anyway).
Maybe you don't need to fork, but you just want to have several threads sharing the same address space. Read pthreads(7) and a good pthread tutorial. Then you'll also should care about synchronization, probably using mutexes.
You could also (and instead) use some other form of IPC. Read Advanced Linux Programming, consider using pipe(7)-s and have some event loop around a multiplexing syscall like poll(2)
In a server/client scenario, you might have some request and protocol to query (from inside clients) some state variable (inside the server).
PS. The main issue is not sharing data, it is synchronization and deadlock avoidance.
Related
I have a C program that forks a child process at some point in a loop. The child process waits for the parent process to finish its job (some numerical calculations). If things go wrong, the parent process aborts and the child process should continue from the state when it was forked and retry the calculation with some modifications. Otherwise, the parents keeps running, and the child process should be killed.
The communication between the parent and child process is through a memory mapped file, which only has 1 byte as a character that indicates the status of the parent process.
The memory map is done like this
char child_flag[]="W";
fp1 = fopen( "child_interface.dat","wb");
// the interface file has two bytes, but only one is meaningful to the program
fwrite(child_flag, 1, sizeof(child_flag), fp1);
fclose(fp1);
printf("child_interface.dat created\n");
if(mmap_child_flag() ==0) {
printf("memory map of parent-child interface successful.\n");
fflush(stdout);
}
The wait loop in the child process is like this
child_pid = fork();
if (child_pid ==0) { /* child process, wait for parent process to finish*/
mmap_child_flag();
while(child_file[0]=='W' ){ //Child waits
usleep(100000);
}
if(child_file[0]=='R'){ // run child process (as a new parent process)
child_file[0]='W';
goto label2;
}
if(child_file[0]=='K'){ //Kill child process
exit(0);
}
}
The problem is that the child process seems to get stuck in the sleep while loop, even when the parent process has set the status to 'K' (checked in the file that is memory mapped). This code has been run on several linux based super computers, and the behavior seems very inconsistent. On some platforms, it can run smoothly, but on some others, it constantly get stuck in the while loop. Sometimes, if I add some statements inside the while loop after the usleep call, it can then run just fine.
However, I'm not sure if the sleep while loop is the root cause of this problem. My guess is that because the process has almost nothing to do except to check a byte in the memory, the system let it sleep all the time and somehow "forget" to let it check the memory. Can such thing happen in the Linux system?
This the function that does the actual mapping
/* Memory map for parent-child processes interface */
int mmap_child_flag()
{
int fd_child;
struct stat st_child;
// open files
if ((fd_child = open("child_interface.dat", O_RDWR)) == -1){
perror("open child_interface.dat");
exit(1);
}
// stat
if (stat("child_interface.dat", &st_child) == -1){
perror("stat of child_interface.dat");
exit(1);
}
// map, child_file is global char array
child_file = mmap(0, st_child.st_size, PROT_WRITE, MAP_SHARED, fd_child, 0);
if (child_file == (char *)(-1)) {
perror("mmap child_interface.dat");
exit(1);
}
return 0;
}
The problem is that the child process seems to get stuck in the sleep while loop, even when the parent process has set the status to 'K' (checked in the file that is memory mapped).
There are several odd things about your program, with one of them being that you are using shared memory for this task at all. See below for a better approach.
Issues with the current approach
As to the question as it stands, however, you have a synchronization problem. The contents of the mapped memory are being changed outside the scope of the child process, but you've given it no reason to suspect that that might be the case. The compiler can therefore assume that if the wait loop condition is satisfied when it is first evaluated, then it will be satisfied on every subsequent evaluation, too.
For a more complicated interaction, you might need to set up a process-shared mutex or similar to guard access to the shared memory, but for this, it would probably be sufficient to declare child_file as a pointer to volatile char.
A better approach
You want the child to wait for a one- or maybe two-byte instruction from the parent. You presently do this by polling the contents of a shared memory segment, but that's complex to set up and use, as you discovered. It would be a lot easier to use a pipe to convey the needed information from parent to child:
setup: Declare an array. Call pipe().
child use: The child performs a blocking read() on the pipe.
parent use: write() the message to the pipe when ready, then close it. Or just close it.
Note that the pipe itself then provides adequate synchronization, and that there is no need for a wait loop. Note also that the child can detect the case that the parent dies without sending any message, which your shared memory approach does not support.
A shared memory region is good for sharing a lot of data, but it is a bad way to communicate between processes. The reason is that you can't get a notification that something has been changed, nor do you get a notification if the other user of the shared memory died.
To communicate between two processes, use pipe() if you need to create a one-way communication channel, or if you need bidirectional communication, use socketpair(). You can use poll() to wait for the other side to send some data. You will also get notified if the process on the other side terminated.
You were using a loop like this:
while(child_file[0]=='W' ){ //Child waits
usleep(100000);
}
This is bad, since you are wasting on average 50 ms of time that you could have spent doing something useful. Apart from that, there is also the problem that both the compiler and the CPU can sometimes change the order in which things are written to memory. If you have more data in child_file than just the flag at the start, then this might be an issue, unless you use atomics or explicit barriers.
I'm currently working on a project where I have a parent process that sets up a socketpair, forks and then uses this socketpair to communicate. The child, if it wants to open a file (or any other file descriptor based resource) should always go to the parent, request the resource and get the fd sent via the socketpair. Furthermore I want to prevent the child from opening any file descriptor by itself.
I stumbled over setrlimit which successfully prevents the child from opening new file descriptors, but it also seems to invalidate any file descriptors sent over the initial socket connection. Is there any method on Linux that allows a single process to open any file, send its file descriptor to other processes and lets them use them without allowing these other processes to open any file descriptor by themselves?
For my use case that can be any kernel configuration, system call, etc. as long as it can be applied after fork and as long as it applies to all file descriptors (not just files but also sockets, socketpairs, etc.).
What you have here is exactly the use case of seccomp.
Using seccomp, you can filter syscalls in different ways. What you want to do in this situation is, right after fork(), to install a seccomp filter that disallows the use of open(2), openat(2), socket(2) (and more).
To accomplish this, you can do the following:
First, create a seccomp context using seccomp_init(3) with the default behavior of SCMP_ACT_ALLOW.
Then add a rule to the context using seccomp_rule_add(3) for each syscall that you want to deny. You can use SCMP_ACT_KILL to kill the process if the syscall is attempted, SCMP_ACT_ERRNO(val) to make the syscall fail returning the specified errno value, or any other action value defined in the manual page.
Load the context using seccomp_load(3) to make it effective.
Before continuing, NOTE that a blacklist approach like this one is in general weaker than a whitelist approach. It allows any syscall that is not explicitly disallowed, and could result in a bypass of the filter. If you believe that the child process you want to execute could be maliciously trying to avoid the filter, or if you already know which syscalls will be needed by the children, a whitelist approach is better, and you should do the opposite of the above: create filter with the default action of SCMP_ACT_KILL and allow the needed syscalls with SCMP_ACT_ALLOW. In terms of code the difference is minimal (the whitelist is probably longer, but the steps are the same).
Here's an example of the above (I'm doing exit(-1) in case of error just for simplicity's sake):
#include <stdlib.h>
#include <seccomp.h>
static void secure(void) {
int err;
scmp_filter_ctx ctx;
int blacklist[] = {
SCMP_SYS(open),
SCMP_SYS(openat),
SCMP_SYS(creat),
SCMP_SYS(socket),
SCMP_SYS(open_by_handle_at),
// ... possibly more ...
};
// Create a new seccomp context, allowing every syscall by default.
ctx = seccomp_init(SCMP_ACT_ALLOW);
if (ctx == NULL)
exit(-1);
/* Now add a filter for each syscall that you want to disallow.
In this case, we'll use SCMP_ACT_KILL to kill the process if it
attempts to execute the specified syscall. */
for (unsigned i = 0; i < sizeof(blacklist) / sizeof(blacklist[0]); i++) {
err = seccomp_rule_add(ctx, SCMP_ACT_KILL, blacklist[i], 0);
if (err)
exit(-1);
}
// Load the context making it effective.
err = seccomp_load(ctx);
if (err)
exit(-1);
}
Now, in your program, you can call the above function to apply the seccomp filter right after the fork(), like this:
child_pid = fork();
if (child_pid == -1)
exit(-1);
if (child_pid == 0) {
secure();
// Child code here...
exit(0);
} else {
// Parent code here...
}
A few important notes on seccomp:
A seccomp filter, once applied, cannot be removed or altered by the process.
If fork(2) or clone(2) are allowed by the filter, any child processes will be constrained by the same filter.
If execve(2) is allowed, the existing filter will be preserved across a call to execve(2).
If the prctl(2) syscall is allowed, the process is able to apply further filters.
I started writing a c web server a while ago (windows 8), but I tried using only threads by myself, without using the select() option.
This is my main loop, and I'm opening each new thread like this:
uintptr_t new_thread;
while (client_sock = accept(server->sock, (struct sockaddr *)&client_info, &size))
{
if (client_sock <= 0) quit();
printf("\n[***] : Got a connection from localhost on port %d\n",ntohs(client_info.sin_port));
code = init_connection(client_sock);
if (code)
{
new_thread = _beginthread(handle_connection, 0, ID++, client_sock);
if (new_thread == -1)
{
fprintf(stderr, "Could not create thread for sending data: %d\n", GetLastError());
closesocket(client_sock);
quit();
}
}
else
{
debug("Failed to init connection");
closesocket(client_sock);
debug("Connection to client ended");
}
}
First of all, I would love to here if I can make this code better.
Testing this program by trying to enter the localhost from chrome, I see that no more data is sent (after recieving one http request).
My question is what would be the best way for the program to act then: close the thread and when another request will be made it will open a new one? if so, how do I close that thread? if not, when should I close that thread?
Normally, when implementing a server that forks separate processes, I would make the child process stay alive to serve predefined amount of requests (e.g. 100) and then kill itself. This is to reduce overhead created by forking and on the other hand recover from possible memory leaks or other problems in the process. Threads are lighter than processes, so it may make sense to close them faster.
I think you should compare the benefits and drawbacks. Measure the overhead of thread creation and closing compared to keeping them alive. In any case you must make sure that there is limit on the number threads you have alive at one time.
About the windows specifics on creating ans closing the thread you could go and add e.g. this response.
I'm trying to understand the different practices when it comes to socket programming and handling multiple connections.
In particular when a server needs to serve multiple clients.
I have looked at some code examples; where some use fd_set and others use a fork() system call.
Roughly:
FD_SET
//Variables
fd_set fds, readfds;
//bind(...)
//listen(...)
FD_ZERO(&fds);
FD_SET(request_socket, &fds);
while(1) {
readfds = fds;
if (select (FD_SETSIZE, &readfds, NULL, NULL, NULL) < 0)
//Something went wrong
//Service all sockets with input pending
for(i = 0; i < FD_SETSIZE; i++) {
if (FD_ISSET (i, &readfds)) {
if (i == request_socket) {
/* Connection request on original socket. */
int new;
size = sizeof (clientname);
new = accept (request_socket, (struct sockaddr *) &clientname, &size);
if (new < 0)
//Error
fprintf (stderr, "Server: connect from host %s, port %hd.\n", inet_ntoa (clientname.sin_addr), ntohs (clientname.sin_port));
FD_SET (new, &fds);
}
else {
/* Data arriving on an already-connected socket. */
if (read_from_client (i) < 0) { //handles queries
close (i);
FD_CLR (i, &fds);
}
}//end else
fork()
//bind()
//listen()
while(1) {
//Connection establishment
new_socket = accept(request_socket, (struct sockaddr *) &clientaddr, &client_addr_length);
if(new_socket < 0) {
error("Error on accepting");
}
if((pid = fork()) < 0) {
error("Error on fork");
}
if((pid = fork()) == 0) {
close(request_socket);
read_from_client(new_socket);
close(new_socket);
exit(0);
}
else {
close(new_socket);
}
}
My question is then: what is the difference between the two practices (fd_set and fork)? Is one more suitable than the other?
You would choose between one of the two approaches, select() or fork() based on the nature of the IO operations you have to do once you receive a connection from a client.
Many IO system calls are blocking. While a thread is blocked on IO performed for one client (e.g. connecting to a database or server, reading a file on disk, reading from the network, etc.), it cannot serve the other clients' requests. If you create a new process with fork(), then each process can block independently without impeding progress on the other connections. Although it may seem advantageous to start a process for each client, it has drawbacks: multiple processes are harder to coordinate, and consume more resources. There is no right or wrong approach, it is all about trade-offs.
You may read about "events vs threads" to understand the various tradeoffs to consider: See: Event Loop vs Multithread blocking IO
The select() system call approach (which you've called the FD_SET approach), would generally classify as a polling approach. Using this, a process can wait on multiple file descriptor events at once, sleep there, and be woken up when activity arises on at least one of the file descriptors specified in the FD_SET. You may read the man page on select for details (man 2 select). This will allow the server process to read from the multiple clients bit by bit (but still one at a time), as soon as new data arrives on any socket of interest.
Trying to call read() on a socket that has no data available would block -- select just makes sure you only do it on those that have data available. It is generally called in a loop so that the process comes back for the next piece of work. Writing the program in that style often forces one to handle requests iteratively, and carefully, because you want to avoid blocking in your single process.
fork() (man 2 fork) creates a child process. Child processes are created with a copy of the file descriptors open in the parent, which explains all the fd-closing business when the system call returns. Once you have a child process to take care of the client's socket, then you can write straightforward linear code with blocking calls without affecting the other connections (because those would be handled in parallel by other child processes of the server).
The main difference between the two practices is the number of processes used to handle multiple connections. With select, single process (in fact a single thread) can handle concurrent connections from multiple clients. When we use the fork based approach, a new process is created for every new connections. So if there are N concurrent client connections, there will be N processes to handle those connections.
When we use select, we don't need to worry about shared memory or synchronization as everything is happening within the same thread of execution.
On the other hand, when we use select, we need to be more careful while coding as the same thread of execution is going to handle multiple clients. In fork based approach, the child process has to handle only a single client so it tends to be bit easier to implement.
When we use fork based approach, we end up using more system resource as a result of creating more processes.
The choice of approach depends on the application - expected number of connections, the nature of connections (persistent or short duration), whether there is a need to share data among connection handlers etc.
As I am currently doing this project in only C, I've up untill this point only used my webserver as a single threaded application. However, I dont want that anymore! So I have the following code that handles my Work.
void BeginListen()
{
CreateSocket();
BindSocket();
ListenOnSocket();
while ( 1 )
{
ProcessConnections();
}
}
Now I've added fork(); before the start of ProcessConnection(); which helpes me allowing multiple connections! However, when I add code for daemoning the application found in this answer. I've encounted a little problem, using fork() will create a copy of my whole running app, which is the purpose of fork(). So, I'd like to solve this problem.
My ProcessConnection() looks like this
void ProcessConnections()
{
fork();
addr_size = sizeof(connector);
connecting_socket = accept(current_socket, (struct sockaddr *)&connector, &addr_size);
if ( connecting_socket < 0 )
{
perror("Accepting sockets");
exit(-1);
}
HandleCurrentConnection(connecting_socket);
DisposeCurrentConnection();
}
How would I do to simply just add a couple of lines above or after connecting=socket = accept... in order to make it accept more than one connection at the time? Can i use fork(); but when it comes down to DisposeCurrentConnection(); I want to kill that process and just have the parent-thread running.
I'm not a 100% sure what it is that you're trying to do, buy off the top of my head, I'd prefer to do the fork after the accept, and simply exit() when you're done. Keep in mind though, that you need to react to the SIGCHLD signal when the child process exits, otherwise you'll have a ton of zombie-processes hanging around, waiting to deliver their exit-status to the parent process. C-pseudo-code:
for (;;) {
connecting_socket = accept(server_socket);
if (connecting_socket < 0)
{
if (errno == EINTR)
continue;
else
{
// handle error
break;
}
}
if (! (child_pid = fork ()))
{
// child process, do work with connecting socket
exit (0);
}
else if (child_pid > 0)
{
// parent process, keep track of child_pid if necessary.
}
else
{
// fork failed, unable to service request, send 503 or equivalent.
}
}
The child_pid is needed to (as already mentioned) to kill the child-process, but also if you wish to use waitpid to collect the exit status.
Concerning the zombie-processes, if you're not interested in what happened to the process, you could install a signal hander for SIGCHLD and just loop on waitpid with -1 until it there are no more child-processes, like this
while (-1 != waitpid (-1, NULL, WNOHANG))
/* no loop body */ ;
The waitpid function will return the pid of the child that exited, so if you wish you can correlate this to some other information about the connection (if you did keep track of the pid). Keep in mind that accept will probably exit with errno set to EINTR, without a valid connection if a SIGCHLD is caught, so remember to check for this on accepts return.
EDIT:
Don't forget to check for error conditions, i.e. fork returns -1.
Talking about fork() and threads on unix is not strictly correct. Fork creates a whole new process, which has no shared address space with the parent.
I think you are trying to achieve a process-per-request model, much like a traditional unix web server such as NCSA httpd or Apache 1.x, or possibly build a multi-threaded server with shared global memory:
Process-per-request servers:
When you call fork(), the system creates a clone of the parent process, including file descriptiors. This means that you can accept the socket request and then fork. The child process has the socket request, which it can reply to and then terminate.
This is relatively efficient on unix, as the memory of the process is not physically copied - the pages are shared between the process. The system uses a mechanism called copy-on-write to make copies on a page-by-page basis when the child process writes to memory. Thus, the overhead of a process-per-request server on unix is not that great, and many systems use this architecture.
Better to use select() function which enables u to listen and connect from different
requests in one program.... It avoids blocking but forking creates a new address space
for the copy of the program which leads to memory inefficiency....
select(Max_descr, read_set, write_set, exception_set, time_out);
i.e u can
fd_set* time_out;
fd_set* read_set;
listen(1);
listen(2);
while(1)
{
if(select(20, read_set, NULL,NULL, timeout) >0)
{
accept(1);
accept(2); .....
pthread_create(func);....
}
else
}
Check the return value of fork(). If it is zero, you are the child process, and you can exit() after doing your work. If it is a positive number then it's the process ID of the newly created process. This can let you kill() the child processes if they are hanging around too long for some reason.
As per my comment, this server is not really multi-threaded, it is multi-process.
If you want a simple way to make it accept multiple connections (and you don't care too much about performance) then you can make it work with inetd. This leaves the work of spawning the processes and being a daemon to inetd, and you just need to write a program that handles and processes a single connection. edit: or if this is a programming exercise for you, you could grab the source of inetd and see how it does it
You can also do what you want to do without either threads or new processes, using select.
Here's an article that explains how to use select (pretty low overhead compared to fork or threads - here's an example of a lightweight web server written this way)
Also if you're not wedded to doing this in C, and C++ is OK, you might consider porting your code to use ACE. That is also a good place to look for design patterns of how to do this as I believe it supports pretty much any connection handling model and is very portable.