I am new to threads and processes.
I have code that works fine right now with forking the code into multiple processes. However each process needs to add to a global variable, but from what I read, each time the process forks, it takes a copy of the global, and adds them independently. Is there a way to join them, like you can with threads?
Different processes can communicate and exchange data via shared memory.
On linux, you can look:
man shm_overview
for attaching a memory segment on several processes
and
man sem_overview
for the semaphore library for controlling parallel access.
You should define a struct with two fields, one for your global and one for a semaphore. Then, before any forking occurs, create some shared memory in the parent process big enough to hold this struct and initialize one there. In the children, map in the shared memory so they can access the global. All processes, parent and children, should obey the rules of the semaphore when accessing the global.
To avoid unnecessary blocking which can hurt performance, try not to hold the semaphore too long. When reading the global, make a quick copy of it in a process and use that, rather than holding the semaphore for the entire time you are using its value. Likewise, when changing the global, prepare your changes ahead of time (before you grab the semaphore) and, once you have the semaphore, copy them in all at once. Sometimes your work depends on reading and writing the global without it changing in between being read and written. In this case, some blocking may be inevitable.
It is not clear what platform you are on, but all major PC and server platforms (Windows, Linux/Unix/Mac OS) have support for shared memory and semaphores. The APIs may be different, but the functionality you need is there.
Related
When a process forks, would the child process have the customized shared library (.so file) in its address space?
If so, is the address of the shared library be same or different from its parent process (due to ASLR) ?
Would the function running before the main function __attribute__ ((constructor)) constructor be executed again in all the child process? What about thread?
Yes, the child will retain the parent's mappings. Ordinarily, Linux's virtual memory system will actually share the page between the two processes, up until either one tries to write new data. At that point, a copy will be made and each process will have its own unique version - at a different physical address but retaining the same virtual address. This is referred to as "copy on write" and is a substantial efficiency and resources advantage over systems which cannot support this, particularly running code which forks frequently.
Address Space Layout Randomization (ASLR) can't apply for libraries or objects which are already allocated virtual addresses, as to do so would break any pointers held anywhere in the code - something that a system running non-managed code can't know enough about to account for.
Since all previously constructed objects already exist in memory, constructors are not called again just because of the fork. Any objects which need to be duplicated because they are being uniquely modified have this done invisibly by the VM system behind the scenes - they don't really know that they are being cloned, and you could very well end up having a pair of objects where part of the implementation continues to share a physical page with identical contents while another part has been invisibly bifurcated into distinct physical pages with differing contents for each process.
You also asked about threads, and that is an area where things get complicated. Normally, only the thread which called fork() will exist in live form in the child (though data belonging to the others will exist in shared mappings, since it can't be known what might be shared with the forked thread). If you need to try to fork a multithreaded program, you will need to see the documentation of your threading implementation. For the common pthreads implementation on Linux, particularly pay attention to pthread_atfork()
I'm working with semaphores in C , especifically to control the access to a shared memory zone in linux. but there is one thing that I can't understand.
I am using a mutex to control the access to a specific zone because i have 2 processes that must read/write from that zone. the thing is, when we use the fork() to create a new child process, the whole program is "copied" to another program as if they were two seperate programs right ? so, when i do V(mutex) in one process, how does the other one know he can't access ?
I know its a noob question but nobody could explain this to me until now.
After the fork neither process is going to know about the memory actions of the other because they are separate copies. You have to put your shared variables in shared memory, including mutexes and semaphores. Then all the processes are operating on the same resource.
For unrelated (i.e. non-forked) process there are usually system facilities (e.g. named semaphores) that each process can open based on a path name or similar method that each can use to find and use the resource.
You synchronisation objects must be placed in process shared memory, for example created with mmap (... MAP_ANONYMOUS ...). In addition, they must have the PTHREAD_PROCESS_SHARED attribute set, for example, by using pthread_mutexattr_setpshared.
See here:
Semaphores and Mutex for Thread and Process Synchronization
So mutex in practice is often used in threads, which makes sharing trivial. For processes however, mutex could be stored as a part of the shared mem.
For semaphores however, linux has built in library, which identifies global semaphores by keys. See below.
http://beej.us/guide/bgipc/output/html/multipage/semaphores.html
Or you can use other IPC to sync. Signals, for example.
Hope this helps.
How is the implementation of threads done in a system?
I know that child processes are created using the fork() call
and a thread is a light weight. How does the creation of a thread differ from that of a child process?
Threads are created using the clone() system call that can make a new process that shares memory space and some of the kernel control structures with its parent. These processes are called LWPs (light-weight processes) and are also known as kernel-level threads.
fork() creates a new process that initially shares memory with its parent but pages are copy-on-write, which means that separate memory pages are created when the content of the original one is altered. Thus both parent and child processes can no longer change each other's memory and effectively they run as separate processes. Also the newely forked child is a full-blown processes with its separate kernel control structures.
Each process has its own address space aka range of virtual addresses that the process can access. When a new process is forked a duplicate copy of all the resources involved has to be made. After the forking is complete the child and the parent have their own distinct address space and all the resources involved within it.Naturally, this is an performance intensive operation.
While all threads in the same process share the same address space, So when a new thread is spawned each thread only needs its own stack and there is no duplication of all resources as in case of processes.Hence spawning of an thread is considerably less performance intensive.
Ofcourse the two operations cannot and should not be compared because both provide essentially different features for different requirements.
Well it differs very much, first of all child process is in some way copy of parent program and have all variables duplicated, and you differ child from parent by its PID. Threads are like new programs , they run at the same time as main program (it looks like at the same time, due to slicing time of cpu by os ). Threads could use global variables in program, but they don't make duplicate as processes. So it`s much cheaper to use threads then new processes.
Well you've read the important parts, now here's something behind the curtains:
In current implementations(where current means the last few decades), the process memory isn't technically copied immediately upon forking. Read-only sections are just shared between the two processes (as they can't change anyway), as well as the read-only parts of shared libraries, of course. But most importantly, everything writeable is initially also just shared. However, it is shared in a write-protected manner, and as soon as you write to the child process memory (e.g. by incrementing a variable), a page fault is generated in the kernel, which only then causes the kernel to actually copy the respective page (where the modification then occurs).
This great optimization, which is called "copy on write", results in child processes usually not really consuming exactly as much (physical) memory as their parent processes. To the program developer (and user), however, it's completely transparent.
I am trying to work out a way to synchronize two processes which share data.
Basically I have two processes linked using shared memory. I need process A to set some data in the shared memory area, then process B to read that data and act on it.
The sequence of events I am looking to have is:
B blocks waiting for data available signal
A writes data
A signals data available
B reads data
B blocks waiting for data not available signal
A signals data not available
All goes back to the beginning.
In other terms, B would block until it got a "1" signal, get the data, then block again until that signal went to "0".
I have managed to emulate it OK using purely shared memory, but either I block using a while loop which consumes 100% of CPU time, or I use a while loop with a nanosleep in it which sometimes misses some of the signals.
I have tried using semaphores, but I can only find a way to wait for a zero, not for a one, and trying to use two semaphores just didn't work. I don't think semaphores are the way to go.
There will be numerous processes all accessing the same shared memory area, and all processes need to be notified when that shared memory has been modified.
It's basically trying to emulate a hardware data and control bus, where events are edge rather than level triggered. It's the transitions between states I am interested in, rather than the states themselves.
So, any ideas or thoughts?
Linux has its own eventfd(2) facility that you can incorporate into your normal poll/select loop. You can pass eventfd file descriptor from process to process through a UNIX socket the usual way, or just inherit it with fork(2).
Edit 0:
After re-reading the question I think one of your options is signals and process groups: start your "listening" processes under the same process group (setpgid(2)), then signal them all with negative pid argument to kill(2) or sigqueue(2). Again, Linux provides signalfd(2) for polling and avoiding slow signal trampolines.
If 2 processes are involved you can use a file , shared memory or even networking to pass the flag or signal. But if the processes are more, there may be some suitable solutions in modifying the kernel. There is one shared memory in your question, right ?! How the signals are passed now ?!
In linux, all POSIX control structures (mutex, conditions, read-write-locks, semaphores) have an option such that they also can be used between processes if they reside in shared memory. For the process that you describe a classic mutex/condition pair seem to fit the job well. Look into the man pages of the ..._init functions for these structures.
Linux has other proper utilities such as "futex" to handle this even more efficiently. But these are probably not the right tools to start with.
1 Single Reader & Single Writer
1 Single Reader & Single Writer
This can be implemented using semaphores.
In posix semaphore api, you have sem_wait() which will wait until value of the semaphore count is zero once it is incremented using sem_post from other process the wait will finish.
In this case you have to use 2 semaphores for synchronization.
process 1 (reader)
sem_wait(sem1);
.......
sem_post(sem2);
process 2(writer)
sem_wait(sem2);
.......
sem_post(sem1);
In this way you can achieve synchronization in shared memory.
my question is somewhat conceptual, how is parent process' data shared with child process created by a fork() call or with a thread created by pthread_create()
for example, are global variables directly passed into child process and if so, does modification on that variable made by child process effect value of it in parent process?
i appreciate partial and complete answers in advance, if i'm missing any existing resource, i'm sorry, i've done some search on google but couldn't find good results
thanks again for your time and answers
The semantics of fork() and pthread_create() are a little different.
fork() will create a new process, where the global variables will be separate between the parent and children. Most OS implementations will use copy-on-write semantics, meaning that both the parent and child process will use the same physical memory pages for all global variables until one of the processes attempts to edit the physical memory, at which point a copy of that page is made, so that now each process gets its own copy and does not see the other process's, so that the processes are isolated.
pthread_create() on the other hand, creates a new thread within the same process. The new thread will have a separate stack space from the other running threads of the same process, however the global variables and heap space are shared between all threads of the same process. This is why you often need a mutex to coordinate access to a shared piece of memory between multiple threads of the same process.
TL;DR version: with fork(), you don't see the other guy's changes; with pthread_create() you do.
A fork creates an almost exact copy of the calling process, including memory and file descriptors. Global variables are copied along with everything else, but they are not in any way linked to the parent process. Since file descriptors are also copied, parent and child can interact via these (as long as they're setup properly, usually via pipe or socketpair).
There's a big difference between processes created by fork and between threads created with pthread_create. Processes don't share global variables and should communicate through pipes, sockets, or other tools provided by the OS. A good solution is MPI - which is a message-passing library for inter-process communication.
Threads are quite different. A thread created with pthread_create shares all the global variables with its caller. Moreover, the caller can pass an arbitrary structure into the thread, and this structure will also be shared. This means that one should be extremely careful when programming with threads - such amounts of sharing are dangerous and error prone. The pthread API provides mutexes and conditions for robust synchronization between threads (although it still requires practice and expertise to implement correctly).