I am making a C program that uses fork and execv to run other programs in parallel.
I can't seem to be able to time the duration of the execution of the program called by execv since the new process dies immediately after that program is done running. Another complication is not being able to use the parent process to wait for the child process to finish (I am using waitpid) because I need the parent process to do some other work instead of waiting for the child process to be finished.
So my question is: is there a way to measure the duration of the execv call without the use of an auxiliary fork, pthread or text file?
Thank you in advance
Your parent process knows when it issued the fork() system call. That's not exactly the moment that the execv'd process starts running, since the execv() system call takes some amount of time, but it's not totally unreasonable to include that time in the tally. If you accept that limitation, you can just record the start time as the time at which you called fork().
When the child terminates, the parent will receive a SIGCHLD signal. The default action for SIGCHLD is to ignore it, but you probably want to change that anyway. If you attach a signal handler to SIGCHLD, then in that signal handler you can call waitpid (with the WNOHANG option) until you've received all the child terminated notifications. For each notification, you record the notification time as the process's end time. (Again, if the system is under heavy load, the signal might lag from the termination, causing your time measure to be inaccurate. But most of the time, it will be accurate.)
Clearly, the parent needs to track more than one child process. So you'll need to use the child's PID to index these values.
Now you have a start time and an end time for each child process.
There's a small problem, though. You cannot attach the start time to the child process's PID until the fork() call returns to the parent. But it's entirely possible that the fork() call will return to the child, and that the child will call execv() and that the execv()'d process terminates all before the fork() call returned to the parent. (Honest. It happens.)
So it is possible for the SIGCHLD handler to receive a notification of the termination of a process whose start time has not yet been recorded.
This is easy to fix, but when you do so you need to take into account the fact that signal handlers cannot allocate memory. So if you're recording the start and end time information in dynamically allocated storage, you need to have allocated storage before the signal handler runs.
So the code will look something like this:
1. Allocate storage for a new process times table entry
(PID / start time / end time / status result). Set all
fields to 0 to indicate that the entry is available.
2. Recall the current time as start_time (a local variable,
not the table entry).
3. Fork()
4. (Still in the parent). Using an atomic compare-and-swap
(or equivalent), set the PID of the table entry created
in step 1 to the child's PID. If the entry was 0 (and is
now the PID) or if the entry was already the PID, then
continue to step 6.
5. If the entry has some other non-zero PID, find an empty entry
in the table and return to step 4.
6. Now record the start time in the table entry. If the table entry
already has an end time recorded, then the signal handler already
ran and you know how long it took and what its return status is.
(This is the case where the child terminated before you got to
step 4.) You can now report this information.
In the SIGCHLD signal handler, you need to do something like this:
For each successful call to waitpid():
1. Find the entry in the child process information table whose PID
corresponds to the PID returned by waitpid(). If you find one,
skip to step 4.
2. Find an empty entry in the child process information table.
Note that the signal handler cannot be interrupted by the main
program, so locking is not required here.
3. Claim that entry by setting its PID field to the PID returned by
waitpid() above.
4. Now that you have an entry, record the end time and return status
information in the table entry. If the table entry existed
previously, you need to put the entry on a notification queue
so that the main process can notify the user. (You cannot call
printf in a signal handler either.) If the table entry didn't
exist before, then the main process will notice by itself.
You might have to draw some diagrams to convince yourself that the above algorithm is correct and has no race conditions. Good luck.
Also, if you haven't done any of these things before, you'll want to do some reading :-)
waitpid(). Pay particular attention to the macros used to extract status information.
sigaction(). How to assign a handler function to a signal. If that's still greek to you, start with signal(7) or a relevant chapter in your Unix programming textbook.
Race conditions (from Wikipedia)
Compare and Swap (on Wikipedia). (Don't use their sample code; it doesn't work. GCC has a built-in extension which implements atomic compare and swap on any architecture which has a way of supporting it. I know that section is marked legacy and you should use the more complicated functions in the next section __atomic, but in this case the defaults are fine. But if you use __atomic_compare_exchange_n, kudos.)
Related
Well, I'm learning about processes using the C language, and I have seen that when you call the exit function a process is terminated and without waiting for it, it will become a zombie process. My question is, if the first process created when executing the program is a process itself, is there a 0S routine that wait for it after an exit() call, avoiding that it becomes a zombie process? I'm curious about it.
For Unix systems at least (and I expect Windows is similar), when the system boots, it creates one special first process. Every process after that is created by some existing process.
When you log into a windowed desktop interface, there is some desktop manager process (that has been created by the first process or one of its descendants) managing windows. When you start a program by clicking on it, that desktop manager or one of its children (maybe some file manager software) creates a process to run the program. When you start a program by executing a command in a terminal window, there is a command line shell process that is interpreting the things you type, and it creates a process to run the program.
So, in all cases, your user program has a parent process, either a command-line shell or some desktop software.
If a child process creates another child (even as the first instruction) then the parent also has to wait for it or it becomes a zombie.
Basically processes always become zombie until they are removed from the process table, the OS (via the process init) will handle and wait() for orphans (zombies without parents), it does that periodically so normally you won't have orphans running for very long.
On Linux, the top most (parent) process is init. This is the only process, which has no parent. Any other process (without any exception) do have a parent and hence is a child of another process.
See:
init
Section NOTES on wait
A child that terminates, but has not been waited for becomes a
"zombie". The kernel maintains a minimal set of information
about the zombie process (PID, termination status, resource usage
information) in order to allow the parent to later perform a wait
to obtain information about the child. As long as a zombie is
not removed from the system via a wait, it will consume a slot in
the kernel process table, and if this table fills, it will not be
possible to create further processes. If a parent process
terminates, then its "zombie" children (if any) are adopted by
init(1), ... init(1) automatically performs a wait to remove the
zombies.
The waiting works fine with pidfd_open and poll.
The problem I’m facing, after the process quits, apparently the poll() API removes the information about the now dead process, so the waitid with P_PIDFD argument fails at once saying code 22 “Invalid argument”
I don’t think I can afford launching a thread for every child process to sleep on the blocking waitpid, I have multiple processes, and another handles which aren’t processes I need to poll efficiently.
Any workarounds?
If it matters, I only need to support Linux 5.13.12 and newer running on ARM64 and ARMv7 CPUs.
The approximate sequence of kernel calls is following:
fork
In the child: setresuid, setresgid, execvpe
In the new child: printf, sleep, _exit
Meanwhile in the parent: pidfd_open, poll, once completed waitid with P_PIDFD first argument.
Expected result: waitid should give me the exit code of the child.
Actual result: it does nothing and sets errno to EINVAL
There is one crucial bit. From man waitid:
Applications shall specify at least one of the flags WEXITED, WSTOPPED, or WCONTINUED to be OR'ed in with the options argument.
I was passing was WNOHANG
And you want to pass WNOHAND | WEXITED ;)
You can use a single reaper thread, looping on waitpid(-1, &status, 0). Whenever it reaps a child process, it looks it up in the set of current child processes, handles possible notifications (semaphore or callback), and stores the exit status.
There is one notable situation that needs special consideration: the child process may exit before fork() returns in the parent process. This means it is possible for the reaper to see a child process exiting before the code that did the fork() manages to register the child process ID in any data structure. Thus, both the reaper and the fork() registering functions must be ready to look up or create the record in the data store keeping track of child processes; including calling the callback or posting the semaphore. It is not complicated at all, but unless you are used to thinking in asynchronous terms, it is easy to miss these corner cases.
Because wait(...)/waitpid(-1,...) returns immediately when there are no child processes to wait for (with -1 and errno set to ECHILD), the reaper thread should probably wait on a condition variable when there are no child processes to wait for, with the code that registers the child process ID signaling on that condition variable to minimize resource use in the no-child-processes case. (Also, do remember to minimize the reaper thread stack size, as it is unreasonably large (order of 8 MiB) by default, and wastes resources. I often use 2*PTHREAD_STACK_MIN, myself.)
Currently, I'm learning about processes on the UNIX system.
My issue is, I need to do something every time a background process terminates. That means that I can't use the typical functionality of waitpid because then the process won't be running in the background and it'll hang the program.
I'm also aware of the SIGCHLD signal which is sent whenever a child of the parent process is terminated however I'm not aware of how to get the process id of the said process which I will need.
What is the proper way to go about this in C? I've tried things such as WNOHANG option on waitpid however that of course only gets called once so I don't see how I could make that apply to my current situation.
waitpid because then the process won't be running in the background and it'll hang the program.
If the process won't be running in the backrgound, waitpid with the pid argument will exit immediately (assuming there are no pid clashes). And still, that's not true - just use WNOHANG...
however I'm not aware of how to get the process id of the said process which I will need. What is the proper way to go about this in C?
Use sigaction to register the signal handler and use the field si_pid from the second signal handler argument of type siginfo_t. From man sigaction:
SIGCHLD fills in si_pid, si_uid, si_status, si_utime, and si_stime,
providing information about the child. The si_pid field is the
process ID of the child
A working example that uses it is in the man 3p wait page under section Waiting for a Child Process in a Signal Handler for SIGCHLD.
What is the proper way to go about this in C?
The C standard is not aware of child processes and SIGCHLD signals. These are part of your operating system. In this case the behavior is standardized by POSIX.
I am just going to post pseudo code,
but my question is I have a loop like such
for(i<n){
createfork();
if(child)
/*
Exit so I can control exact amount of forks
without children creating more children
*/
exit
}
void createfork(){
fork
//execute other methods
}
Does my fork create a process do what it is suppose to do and exit then create another process and repeat? And if so what are some ways around this, to get the processes running concurrently?
Your pseudocode is correct as written and does not need to be modified.
The processes are already executing in parallel, all six of them or however many you spawn. As written, the parent process does not wait for the children to finish before spawning more children. It calls fork(), checks if (child) (which is skipped), then immediately proceeds to the next for loop iteration and forks again.
Notably, there's no wait() call. If the parent were to call wait() or waitpid() to wait for each child to finish then that would introduce the serialism you're trying to avoid. But there is no such call, so you're good.
When a process successfully performs a POSIX fork(), that process and the new child process are initially both eligible to run. In that sense, they will run concurrently until one or the other blocks. Whether there will be any periods of time when both are executing machine instructions (on different processing units) depends at least on details of hardware capabilities, OS scheduling, the work each process is performing, and what other processes there are in the system and what they are doing.
The parent certainly does not, in general, automatically wait for the child to terminate before it proceeds with its own work (there is a family of functions to make it wait when you want that), nor does the child process automatically wait for any kind of signal from the parent. If the next thing the parent does is fork another child, then that will under many circumstances result in the parent running concurrently with both (all) children, in the sense described above.
I cannot speak to specifics of the behavior of your pseudocode, because it's pseudocode.
When I call kill() on a process, it returns immediately, because it just send a signal. I have a code where I am checking some (foreign, not written nor modifiable by me) processes in a loop infinitely and if they exceed some limits (too much ram eaten etc) it kills them (and write to a syslog etc).
Problem is that when processes are heavily swapped, it takes many seconds to kill them, and because of that, my process executes the same check against same processes multiple times and attempts to send the signal many times to same process, and write this to syslog as well. (this is not done on purpose, it's just a side effect which I am trying to fix)
I don't care how many times it send a signal to process, but I do care how many times it writes to syslog. I could keep a list of PID's that were already sent the kill signal, but in theory, even if there is low probability, there could be another process spawned with same pid as previously killed one had, which might also be supposed to be killed and in this case, the log would be missing.
I don't know if there is unique identifier for any process, but I doubt so. How could I kill a process either synchronously, or keep track of processes that got signal and don't need to be logged again?
Even if you could do a "synchronous kill", you still have the race condition where you could kill the wrong process. It can happen whenever the process you want to kill exits by its own volition, or by third-party action, after you see it but before you kill it. During this interval, the PID could be assigned to a new process. There is basically no solution to this problem. PIDs are inherently a local resource that belongs to the parent of the identified process; use of the PID by any other process is a race condition.
If you have more control over the system (for example, controlling the parent of the processes you want to kill) then there may be special-case solutions. There might also be (Linux-specific) solutions based on using some mechanisms in /proc to avoid the race, though I'm not aware of any.
One other workaround may be to use ptrace on the target process as if you're going to debug it. This allows you to partially "steal" the parent role, avoiding invalidation of the PID while you're still using it and allowing you to get notification when the process terminates. You'd do something like:
Check the process info (e.g. from /proc) to determine that you want to kill it.
ptrace it, temporarily stopping it.
Re-check the process info to make sure you got the process you wanted to kill.
Resume the traced process.
kill it.
Wait (via waitpid) for notification that the process exited.
This will make the script wait for process termination.
kill $PID
while [ kill -0 $PID 2>/dev/null ]
do
sleep 1
done
kill -0 [pid] tests the existence of a process
The following solution works for most processes that aren't debuggers or processes being debugged in a debugger.
Use ptrace with argument PTRACE_ATTACH to attach to the process. This stops the process you want to kill. At this point, you should probably verify that you've attached to the right process.
Kill the target with SIGKILL. It's now gone.
I can't remember whether the process is now a zombie that you need to reap or whether you need to PTRACE_CONT it first. In either case, you'll eventually have to call waitpid to reap it, at which point you know it's dead.
If you are writing this in C you are sending the signal with the kill system call. Rather than repeatedly sending the terminating signal just send it once and then loop (or somehow periodically check) with kill(pid, 0); The zero value of signal will just tell you if the process is still alive and you can act appropriately. When it dies kill will return ESRCH.
when you spawn these processes, the classical waitpid(2) family can be used
when not used anywhere else, you can move the processes going to be killed into an own cgroup; there can be notifiers on these cgroups which get triggered when process is exiting.
to find out, whether process has been killed, you can chdir(2) into /proc/<pid> or open(2) this directory. After process termination, the status files there can not be accessed anymore. This method is racy (between your check and the action, the process can terminate and a new one with the same pid be spawned).