I have 2 programs (processes). One process writes to shared memory while the other reads it. So, my approach was like this:
Initially, value of shared memory is 0. So, process 1 writes only when value is 0. Now process 1 has written some other value to shm and "waits" till value becomes 0. Process 2 reads the shm and writes 0 to it. By wait, I mean in while(1) loop.
My question is if this approach fine or can I do better with some other approach in terms of CPU usage and memory usage?
Mentioned problem known as Process Synchronization problem and given logic is nothing but Busy Waiting approach of the problem which is very primary solution.
Read Producer-Consumer Problem which is similiar to given problem.
There are some better solutions to this than Busy Waiting like:
Spinlock, Semaphore etc.
You can get basic knowledge of all of this from here
Hope it will help!!
I think this is fine but the problem occurs when both the process write to a shared memory block.
At that time you could use a semaphore to synchronize the two process allowing one at a time to write to shared resource/memory block.
You can find regarding Semaphores Click [here](https://en.wikipedia.org/wiki/Semaphore_(programming)
Related
Almost every resource that I have looked up has talked about how to enforce mutual exclusion, or deal with the producer/consumer problem.
The problem is that I need to get certain threads to execute before other threads, but can't figure out how. I am trying to use semaphores, but don't really see how they can help in my case.
I have
a read thread,
N number of search threads, and
a write thread.
The read thread fills a buffer with data, then the search threads parse the data and outputs it to a different buffer, which the write thread then writes to a file.
Any idea as to how I would accomplish this?
I can post the code I have so far if anyone thinks that would help.
I think what you're looking for is a monitor
I would use a few condition variables.
You have read buffers. Probably two. If the search threads are slow you want read to wait rather than use all the memory on buffers.
So: A read_ready condition variable and a read_ready_mutex. Set to true when there is an open buffer.
Next: A search_ready condition variable and a search_ready_mutex. Set to true when there is a complete read buffer.
Next: A write_ready variable and a write_ready mutex. Set to true when there is work to do for the write thread.
Instead of true/false you could use integers of the number of buffers that are ready. As long as you verify the condition while the mutex is held and only modify the condition while the mutex is held, it will work.
[Too long for a comment]
Cutting this down to two assumptions:
Searchig cannot be done before reading has finished.
Writing cannot be done before searching has finished.
I conclude:
Do not use threads for reading and writing, but do it from the main thread.
Just do the the searching in parallel using threads.
Generally speaking, threads are used precisely when we don't care about the order of execution.
If you want to execute some statements S1, S2, ... , SN in that order, then you catenate them into a program that is run by a single thread: { S1; S2; ...; SN }.
The specific problem you describe can be solved with a synchronization primitive known as a barrier (implemented as the POSIX function pthread_barrier_wait).
A barrier is initialized with a number, the barrier count N. Threads which call the barrier wait operation are suspended, until N threads accumulate. Then they are are all released. One of the threads receives a return value which tells it that it is the "serial thread".
So for instance, suppose we have N threads doing this read, process-in-paralle, and write sequence. It goes like this (pseudocode):
i_am_serial = barrier.wait(); # at first, everyone waits at the barrier
if (i_am_serial) # serial thread does the reading, preparing the data
do_read_task();
barrier.wait(); # everyone rendezvous at the barrier again
do_paralallel_processing(); # everyone performs the processing on the data
i_am_serial = barrier.wait(); # rendezvous after parallel processing
if (i_am_serial)
do_write_report_task(); # serialized integration and reporting of results
I'm trying to implement a pipe using shared memory and semaphores (it may be that I need signals also, to complete my implementation)
I encountered the algorithmic problem of how to set the semaphores right.
Lets say I already allocated a piece of shared memory for the pipe buffer,
and a piece of shared memory for the pipe's info (such as how much bytes there are in the pipe, etc...)
I want to create mutual exclusion (only one reader/writer using the pipe at once)
If reader wants to read from an empty pipe, I should block him, till a writer writes something
Same thing like '2', but writer who writes to a full pipe
I tried to search for an answer but I didn't find any even though it seems like a common exercise...
I'm aware of a solution called "Bounded buffer problem" or "consumer producer problem"
which is implemented like this:
There are 3 semaphores:
mutex - initialized to 1
full - initialized to 0
empty - initialized to n (whilst n is the number of, lets say "bytes" I have in the pipe)
Consumer's code:
wait(full)
wait(mutex)
remove a byte from the pipe
signal(mutex)
signal(empty)
Producer's code:
wait(empty)
wait(mutex)
add a byte to the pipe
signal(mutex)
signal(full)
The problem in this solution (to use as a solution to my problem) is that in a given time, only one byte is read from the pipe, or write into it.
In my problem - Implementing a pipe, I don't know for sure how much bytes a writer will write. If he wants to write 'n' bytes, then he will write it only if there is a place in the pipe, and if not, he will write less then 'n' bytes...
That means that a writer must check how much free space there is in the pipe, before writing into it. This is a problem - because the writer will touch a critical section (the pipe's information) without mutual exclusion..
So I thought about putting this part inside the critical section, but then - if a writer wants to write and the pipe is full - how can I let only one reader inside, and then letting the writer to write more?
I've got confused...
Any help will be appreciated, Thanks!
There is no need to have so many mutexes or lock them for that amount of time. In single producer/consumer scenario, the producer never needs to worry about the free space reducing (it is the only one that can use up that space), and similarly for the consumer. Therefore your pseudocode should be:
Producer
while (lock_and_get_free_space() < bytes_to_write)
wait()
unlock()
write(bytes_to_write)
lock_and_update_free_space()
Consumer
while (lock_and_get_data() < bytes_to_read)
wait()
unlock()
read(bytes_to_read)
lock_and_update_free_space()
I am trying to work out a way to synchronize two processes which share data.
Basically I have two processes linked using shared memory. I need process A to set some data in the shared memory area, then process B to read that data and act on it.
The sequence of events I am looking to have is:
B blocks waiting for data available signal
A writes data
A signals data available
B reads data
B blocks waiting for data not available signal
A signals data not available
All goes back to the beginning.
In other terms, B would block until it got a "1" signal, get the data, then block again until that signal went to "0".
I have managed to emulate it OK using purely shared memory, but either I block using a while loop which consumes 100% of CPU time, or I use a while loop with a nanosleep in it which sometimes misses some of the signals.
I have tried using semaphores, but I can only find a way to wait for a zero, not for a one, and trying to use two semaphores just didn't work. I don't think semaphores are the way to go.
There will be numerous processes all accessing the same shared memory area, and all processes need to be notified when that shared memory has been modified.
It's basically trying to emulate a hardware data and control bus, where events are edge rather than level triggered. It's the transitions between states I am interested in, rather than the states themselves.
So, any ideas or thoughts?
Linux has its own eventfd(2) facility that you can incorporate into your normal poll/select loop. You can pass eventfd file descriptor from process to process through a UNIX socket the usual way, or just inherit it with fork(2).
Edit 0:
After re-reading the question I think one of your options is signals and process groups: start your "listening" processes under the same process group (setpgid(2)), then signal them all with negative pid argument to kill(2) or sigqueue(2). Again, Linux provides signalfd(2) for polling and avoiding slow signal trampolines.
If 2 processes are involved you can use a file , shared memory or even networking to pass the flag or signal. But if the processes are more, there may be some suitable solutions in modifying the kernel. There is one shared memory in your question, right ?! How the signals are passed now ?!
In linux, all POSIX control structures (mutex, conditions, read-write-locks, semaphores) have an option such that they also can be used between processes if they reside in shared memory. For the process that you describe a classic mutex/condition pair seem to fit the job well. Look into the man pages of the ..._init functions for these structures.
Linux has other proper utilities such as "futex" to handle this even more efficiently. But these are probably not the right tools to start with.
1 Single Reader & Single Writer
1 Single Reader & Single Writer
This can be implemented using semaphores.
In posix semaphore api, you have sem_wait() which will wait until value of the semaphore count is zero once it is incremented using sem_post from other process the wait will finish.
In this case you have to use 2 semaphores for synchronization.
process 1 (reader)
sem_wait(sem1);
.......
sem_post(sem2);
process 2(writer)
sem_wait(sem2);
.......
sem_post(sem1);
In this way you can achieve synchronization in shared memory.
I have a fixed size array (example: struct bucket[DATASIZE]) where at the very beginning I load information from a file. Since I am concerned about scalability and execution time, no dynamic array was used.
Each time I process half of the array I am free to replace those spots with more data from the file. I don't have a clear idea on how I would do that but I thought about pthreads to start 2 parallel tasks: one would be the actual data processing and the other one would make sure to fill out the array.
However, all the examples that I've seen on pthreads show that they are all working on the same task but concurrently. Is there a way to have them do separate things? Any ideas, thoughts?
You can definitely have threads doing different tasks. The pattern you're after is very common - it's called a Producer-Consumer arrangement.
What you are trying to do seems very similar to standard concurrent program called producer-consumer (look it up, you surely find an example in pthreads). This program has one fixed size buffer which is processed by consumer and filled by producer.
Yes, that's an excellent use for pthreads: it's one of the very things that pthreads was made for.
You might think about fork( )ing twice, once to create the process to do the data manipulation; and then a second fork( ) to create the process that fills in the blanks. Use a mutex to let each process protect the array from the other process and it will work fine.
Why would your array need a mutex? How would you set it up? When would each process need to acquire the mutex and when would it need to release the mutex?
-- pete
HI,
I have a program in which a master processes spawns N workers who will invert, each one, each row of an image, giving me an inverted image at the end. The program uses shared memory and posix semaphores, unnamed sems, more spefically and I use shmctl with IPC_RMID and sem_close and sem_destroy in the terminate() function.
However, when I run the program several times, sometimes it gives me a segmentation fault and is in the first shmget. I've already modified my shmmax value in the kernel, but I can't do the same to the shmall value, I don't know why.
Can someone please help me? Why this happens and why isn't it all the time? The code seems fine, gives me what I want, efficient and so...but sometimes I have to reboot Ubuntu to be able to run it again, even thought I'me freeing the resources.
Please enlighten me!
EDIT:
Here are the 3 files needed to run the code + the makefile:
http://pastebin.com/JqTkEkPv
http://pastebin.com/v7fQXyjs
http://pastebin.com/NbYFAGYq
http://pastebin.com/mbPg1QJm
You have to run it like this ./invert someimage.ppm outimage.ppm
(test with a small one for now please)
Here are some values that may be important:
$ipcs -lm
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 262144
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1
$ipcs -ls
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
EDIT: the seg fault was solved! I was allocating an **array in shared memory and that was a little bit odd.So, I've allocated segment for an *array only and voilà. If you want, check the new code and comment.
If all your sem_t POSIX semaphores are unnamed you should only use sem_init and sem_destroy on them and never sem_close.
Now that you posted your code we can say a bit more.
Without having read all in detail, I think the cleanup phase of your main looks suspicious. In fact it seems to me that all your worker processes will perform that cleanup phase too.
After the fork you should more clearly distinguish what main does and what the workers do. Alternatives:
Your main process could just
wait on the pids of the workers
and only then do the rest of
processing and cleanup.
All the worker processes could
return in main after the call to
worker.
Call exit at the end of the worker
function.
Edit after your code update:
I think still a better solution would be to do a classical wait for all the processes.
Now let's look into your worker process. In fact these never terminate, there is no break statement in the while (1) loop. I think what is happening is that once there is no more work to be done
the worker is stuck in
sem_wait(sem_remaining_lines)
your main process gets notified of
the termination
it destroys the sem_remaining_lines
the worker returns from sem_wait
and continues
since mutex3 is also already
destroyed (or maybe even unmapped) the wait on it returns
immediately
now it tries to access the data, and
depending on how far the main
process got on destruction the data
is mapped or not and the worker
crashes (or not)
As you can see you have many problems in there. What I would do to clean up this mess is
waitpid before destroy the shared
data
sem_trywait instead of the 1 in
while (1). But perhaps I didn't completely understand your control flow. In any case, give them a termination condition.
capture all returns from system
functions, in particular the sem_t
family. These can be interrupted by
IO, so you definitively must
check for EINTR on these.