Related
Could someone tell me what this line does:
if(signal(SIGUSR1, handler) == (sighandler_t)-1)
It is a line I copied from an exercise, which made it work, but I don't really understand it. Could someone explain this to me? (It is actually the second part I don't understand: what is the value of (sighandler_t)-1?)
Thank you :)
edit: the sighandler_t comes from
typedef void (*sighandler_t)(int);
First of all, it is a bad style and probably non-portable code, (sighandler_t)-1 should be replaced with one of the predefined signal dispositions. On my system they are declared in next way
/* Fake signal functions. */
#define SIG_ERR ((__sighandler_t) -1) /* Error return. */
#define SIG_DFL ((__sighandler_t) 0) /* Default action. */
#define SIG_IGN ((__sighandler_t) 1) /* Ignore signal. */
Other systems may use another values, so assuming that your uses the same definitions, we get next code:
if(signal(SIGUSR1, handler) == SIG_ERR) {
/* got problem */
} else {
/* handler installed */
}
This code installs function handler as handler for signal SIGUSR1 and checks returned value to ensure that it was done successfully. handler must be declared as void handler(int signo);
(sighandler_t)-1 is the minus one digit, cast it into sighandler_t type. You must check to see if the signal call has failed.
Assume that we have a single thread program and we hope to capture the value of program counter (PC) when a predefined interrupt occurs (like a timer interrupt).
It seems easy as you know we just write a specific assembly code using a special keyword __asm__ and pop the value on the top of the stack after making a shift 4 byte.
What about Multithreaded programs ?
How can we get values of all threads from another thread which run in the same process? (It seems extremely incredible to get values from thread which run on a separate core in multi-core processors).
(in multithreaded programs, every thread has its stack and registers too).
I want to implement a saboteur thread.
in order to perform fault injection in the target multi-threaded program, the model of fault is SEU (single error upset) which means that an arbitrary bit in the program counter register modified randomly (bit-flip) causing to violate the right program sequence. therefore, control flow error (CFE) occurs.
Since our target program is a multi-threaded program, we have to perform fault injection on all threads' PC. This is the task of saboteur tread. It should be able to obtain threads' PC to perform fault injection.
assume we have this code,
main ()
{
foo
}
void foo()
{
__asm__{
pop "%eax"
pop "%ebx" // now ebx holds porgram counter value (for main thread)
// her code injection like 00000111 XOR ebx for example
push ...
push ...
};
}
If our program was a multithreaded program.
is it means that we have more than one stack?
when OS perform context switching, it means that the stack and registers of the thread that was running moved to some place in the memory. Does this mean that if we want to get the values of the program counter for those threads, we find them in memory? where? and is it possible during run-time?
When you install a signal handler using sigaction() with SA_SIGINFO in the flags, the second parameter the signal handler gets is a pointer to siginfo_t, and the third parameter is a pointer to an ucontext_t. In Linux, this structure contains, among other things, the set of register values when the kernel interrupted the thread, including program counter.
#define _POSIX_C_SOURCE 200809L
#define _GNU_SOURCE
#include <signal.h>
#include <ucontext.h>
#if defined(__x86_64__)
#define PROGCOUNTER(ctx) (((ucontext *)ctx)->uc_mcontext.greg[REG_RIP])
#elif defined(__i386__)
#define PROGCOUNTER(ctx) (((ucontext *)ctx)->uc_mcontext.greg[REG_EIP])
#else
#error Unsupported architecture.
#endif
void signal_handler(int signum, siginfo_t *info, void *context)
{
const size_t program_counter = PROGCOUNTER(context);
/* Do something ... */
}
As usual, printf() et al. are not async-signal safe, which means it is not safe to use them in a signal handler. If you wish to output the program counter to e.g. standard error, you should not use any of the standard I/O to print to stderr, and instead construct the string to be printed by hand, and use a loop to write() the contents of the string; for example,
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
static void wrerr(const char *p)
{
const int saved_errno = errno;
const char *q = p;
ssize_t n;
/* Nothing to print? */
if (!p || !*p)
return;
/* Find end of q. strlen() is not async-signal safe. */
while (*q) q++;
/* Write data from p to q. */
while (p < q) {
n = write(STDERR_FILENO, p, (size_t)(q - p));
if (n > 0)
p += n;
else
if (n != -1 || errno != EINTR)
break;
}
errno = saved_errno;
}
Note that you'll want to keep the value of errno unchanged in the signal handler, so that if interrupted after a failed library function, the interrupted thread still sees the correct errno value. (It's mostly a debugging issue, and "good form"; some idiots pooh-pooh this as "it does not happen often enough for me to worry about".)
Your program can examine the /proc/self/maps pseudofile (it is not a real file, but something that the kernel generates on the fly when the file is read) to see the memory regions used by the program, to determine whether the program was running a C library function (very common) or something else when the interrupt was delivered.
If you wish to interrupt a specific thread in a multi-threaded program, just use pthread_kill(). Otherwise the signal is delivered to one of the threads that has not blocked the signal, more or less at random.
Here is an example program, that is tested to in x86-64 (AMD64) and x86, when compiled with GCC-4.8.4 using -Wall -O2:
#define _POSIX_C_SOURCE 200809L
#define _GNU_SOURCE
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <signal.h>
#include <ucontext.h>
#include <time.h>
#include <stdio.h>
#if defined(__x86_64__)
#define PROGRAM_COUNTER(mctx) ((mctx).gregs[REG_RIP])
#define STACK_POINTER(mctx) ((mctx).gregs[REG_RSP])
#elif defined(__i386__)
#define PROGRAM_COUNTER(mctx) ((mctx).gregs[REG_EIP])
#define STACK_POINTER(mctx) ((mctx).gregs[REG_ESP])
#else
#error Unsupported hardware architecture.
#endif
#define MAX_SIGNALS 64
#define MCTX(ctx) (((ucontext_t *)ctx)->uc_mcontext)
static void wrerr(const char *p, const char *q)
{
while (p < q) {
ssize_t n = write(STDERR_FILENO, p, (size_t)(q - p));
if (n > 0)
p += n;
else
if (n != -1 || errno != EINTR)
break;
}
}
static const char hexc[16] = "0123456789abcdef";
static inline char *prehex(char *before, size_t value)
{
do {
*(--before) = hexc[value & 15];
value /= (size_t)16;
} while (value);
*(--before) = 'x';
*(--before) = '0';
return before;
}
static volatile sig_atomic_t done = 0;
static void handle_done(int signum)
{
done = signum;
}
static int install_done(const int signum)
{
struct sigaction act;
memset(&act, 0, sizeof act);
sigemptyset(&act.sa_mask);
act.sa_handler = handle_done;
act.sa_flags = 0;
if (sigaction(signum, &act, NULL) == -1)
return errno;
return 0;
}
static size_t jump_target[MAX_SIGNALS] = { 0 };
static size_t jump_stack[MAX_SIGNALS] = { 0 };
static void handle_jump(int signum, siginfo_t *info, void *context)
{
const int saved_errno = errno;
char buffer[128];
char *p = buffer + sizeof buffer;
*(--p) = '\n';
p = prehex(p, STACK_POINTER(MCTX(context)));
*(--p) = ' ';
*(--p) = 'k';
*(--p) = 'c';
*(--p) = 'a';
*(--p) = 't';
*(--p) = 's';
*(--p) = ' ';
*(--p) = ',';
p = prehex(p, PROGRAM_COUNTER(MCTX(context)));
*(--p) = ' ';
*(--p) = '#';
wrerr(p, buffer + sizeof buffer);
if (signum >= 0 && signum < MAX_SIGNALS) {
if (jump_target[signum])
PROGRAM_COUNTER(MCTX(context)) = jump_target[signum];
if (jump_stack[signum])
STACK_POINTER(MCTX(context)) = jump_stack[signum];
}
errno = saved_errno;
}
static int install_jump(const int signum, void *target, size_t stack)
{
struct sigaction act;
if (signum < 0 || signum >= MAX_SIGNALS)
return errno = EINVAL;
jump_target[signum] = (size_t)target;
jump_stack[signum] = (size_t)stack;
memset(&act, 0, sizeof act);
sigemptyset(&act.sa_mask);
act.sa_sigaction = handle_jump;
act.sa_flags = SA_SIGINFO;
if (sigaction(signum, &act, NULL) == -1)
return errno;
return 0;
}
int main(int argc, char *argv[])
{
const struct timespec sec = { .tv_sec = 1, .tv_nsec = 0L };
const int pid = (int)getpid();
ucontext_t ctx;
printf("Run\n");
printf("\tkill -KILL %d\n", pid);
printf("\tkill -TERM %d\n", pid);
printf("\tkill -HUP %d\n", pid);
printf("\tkill -INT %d\n", pid);
printf("or press Ctrl+C to stop this process, or\n");
printf("\tkill -USR1 %d\n", pid);
printf("\tkill -USR2 %d\n", pid);
printf("to send the respective signal to this process.\n");
fflush(stdout);
if (install_done(SIGTERM) ||
install_done(SIGHUP) ||
install_done(SIGINT) ) {
printf("Cannot install signal handlers: %s.\n", strerror(errno));
return EXIT_FAILURE;
}
getcontext(&ctx);
if (install_jump(SIGUSR1, &&usr1_target, STACK_POINTER(MCTX(&ctx))) ||
install_jump(SIGUSR2, &&usr2_target, STACK_POINTER(MCTX(&ctx))) ) {
printf("Cannot install signal handlers: %s.\n", strerror(errno));
return EXIT_FAILURE;
}
/* These are expressions that should evaluate to false, but the compiler
* should not be able to optimize them away. */
if (argv[0][1] == 'A') {
usr1_target:
fputs("USR1\n", stdout);
fflush(stdout);
}
if (argv[0][1] == 'B') {
usr2_target:
fputs("USR2\n", stdout);
fflush(stdout);
}
while (!done) {
putchar('.');
fflush(stdout);
nanosleep(&sec, NULL);
}
fputs("\nAll done.\n", stdout);
fflush(stdout);
return EXIT_SUCCESS;
}
If you save the above as example.c, you can compile it using
gcc -Wall -O2 example.c -o example
and run it
./example
Press Ctrl+C to exit the program. Copy the commands (for sending SIGUSR1 and SIGUSR2 signals), and run them from another window, and you'll see they modify the position for current execution. (The signals cause the program counter/instruction pointer to jump back, into an if clause that should never be executed otherwise.)
There are two sets of signal handlers. handle_done() just sets the done flag. handle_jump() outputs a message to standard error (using low-level I/O), and if specified, updates the program counter (instruction pointer) and stack pointer.
The stack pointer is the tricky part when creating an example program like this. It would be easy if we were satisfied with just crashing the program. However, an example is only useful if it works.
When we arbitrarily change the program counter/instruction pointer, and the interrupt was delivered when in a function call (most C library functions...), the return address is left on the stack. The kernel can deliver the interrupt at any point, so we cannot even assume that the interrupt was delivered when in a function call, either! So, to make sure the test program does not crash, I had to update the program counter/instruction pointer and stack pointer as a pair.
When a jump signal is received, the stack pointer is reset to a value I obtained using getcontext(). This is not guaranteed to be suitable for any jump location; it's just the best I could do for a minimal example. I definitely assume the jump labels are nearby, and not in subscopes where the compiler is likely to mess with the stack, mind you.
It is also important to keep in mind that because we are dealing with details left to the C compiler, we must conform to whatever binary code the compiler produces, not the other way around. For reliable manipulation of a process and its threads, ptrace() is a much better (and honestly, easier) interface. You just set up a parent process, and in the target traced child process, explicitly allow the tracing. I've shown examples here and here (both answers to the same question) on how to start, stop, and single-step individual threads in a target process. The hardest part is understanding the overall scheme, the concepts; the code itself is easier -- and much, much more robust than this signal-handler-context-manipulation way.
For self-introducing register errors (either to program counter/instruction pointer, or to any other register), with the assumption that most of the time that leads to the process crashing, this signal handler context manipulation should be sufficient.
No, it's not possible while a thread is executing. While a thread is executing, the current value of its program counter (EIP) is private to the CPU core it's running on. It's not available in memory anywhere.
It would be possible for an architecture to have special instructions to send inter-processor requests with queries about execution state, but x86 doesn't have this.
However, you can use ptrace system calls to do anything a debugger could; interrupt another thread and modify any of its state (general purpose registers, flags, program counter, etc. etc.) I can't give you an example, I just know that's the system call that debuggers use to modify the saved state of another thread / process. For example, this question asks about modifying another process's RIP using ptrace (for testing code-injection).
I'm not sure it's viable to ptrace one thread from another thread in the same process; your fault injector might work better as a separate process that interferes with the threads of another process.
Anyway, what will happen when you make a ptrace system call to modify something in another thread is that the CPU running your system call will send and inter-processor message to the kernel on the CPU running the other thread, which will interrupt that thread you want to mess with. Its state will be saved into memory by the kernel, where it can be modified by any CPU.
Once the other thread stops running, it isn't strongly associated with any CPU anymore. It will be cheaper to resume it on the CPU that already has hot caches for it, but that isn't guaranteed because that CPU could have started running any other thread once it was no longer busy running the thread you caused to be stopped.
Side note, not relevant to inter-thread fault injection:
Your C function for modifying EIP (foo()) is really ugly, BTW:
First of all, it's MSVC inline asm, so no Linux compiler will accept it (maybe icc?). Second, it only works with -fno-omit-frame-pointer, because it assumes that its inside a function that's pushed %ebp.
It would be so much easier to just write the whole function in asm. In 64bit non-inline asm, you'd just write:
global fault_inject_program_counter
fault_inject_program_counter:
xor qword [rsp], 0b00000111
ret
and assemble that file separately with NASM or YASM, and link the .o with code that calls it. (I'm assuming you'd prefer Intel syntax, since you used MSVC-style asm {} instead GNU C asm("pop ; ... ; "::: ); inline asm.)
an inline asm version might look like:
// this can't possibly work if inlined, or if compiled without `-fno-omit-frame-pointer
__attribute__((noinline)) void foo()
{
__asm__ volatile(
// "pop %eax\n\t"
// "pop %ebx\n\t" // now ebx holds the return address
// here code injection like 00000111 XOR ebx for example
// normal people would just write
"xorl $0b00000111, -4(%esp)\n\t"
// to modify the return value in-place, in a function with a frame pointer.
// push ...
// push ...
);
}
We dont want anything to be printed after user interrupt via CTRL-C. We have tried adding __fpurge as well fflush inside sigInt signal handler, but it is not working.
How can I clear buffered stdout values immediately? I have came across few similar thread but no where i could able to find a working solution .
Few additional info's:
Inside sigInt signal handler even after adding exit(0) , buffer content are getting printed but the processor is killed .
added exit(0) to narrow down the issue , i dont want to kill the processor
I know the above is expected behavior , not sure how to avoid it .
Consider this edited example -- edited; this one does not exit the process:
#define _POSIX_C_SOURCE 200809L /* For nanosleep() */
#include <unistd.h>
#include <stdlib.h>
#include <termios.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <signal.h>
#include <string.h>
#include <errno.h>
#include <time.h>
#include <stdio.h>
static void exit_handler(int signum)
{
int fd, result;
/* If the standard streams are connected to a tty,
* tell the kernel to discard already buffered data.
* (That is, in kernel buffers. Not C library buffers.)
*/
if (isatty(STDIN_FILENO))
tcflush(STDIN_FILENO, TCIOFLUSH);
if (isatty(STDOUT_FILENO))
tcflush(STDOUT_FILENO, TCIOFLUSH);
if (isatty(STDERR_FILENO))
tcflush(STDERR_FILENO, TCIOFLUSH);
/* Redirect standard streams to /dev/null,
* so that nothing further is output.
* This is a nasty thing to do, and a code analysis program
* may complain about this; it is suspicious behaviour.
*/
do {
fd = open("/dev/null", O_RDWR);
} while (fd == -1 && errno == EINTR);
if (fd != -1) {
if (fd != STDIN_FILENO)
do {
result = dup2(fd, STDIN_FILENO);
} while (result == -1 && (errno == EINTR || errno == EBUSY));
if (fd != STDOUT_FILENO)
do {
result = dup2(fd, STDOUT_FILENO);
} while (result == -1 && (errno == EINTR || errno == EBUSY));
if (fd != STDERR_FILENO)
do {
result = dup2(fd, STDERR_FILENO);
} while (result == -1 && (errno == EINTR || errno == EBUSY));
if (fd != STDIN_FILENO && fd != STDOUT_FILENO && fd != STDERR_FILENO)
close(fd);
}
}
static int install_exit_handler(const int signum)
{
struct sigaction act;
memset(&act, 0, sizeof act);
sigemptyset(&act.sa_mask);
act.sa_handler = exit_handler;
act.sa_flags = 0;
if (sigaction(signum, &act, NULL) == -1)
return errno;
return 0;
}
int main(void)
{
if (install_exit_handler(SIGINT)) {
fprintf(stderr, "Cannot install signal handler: %s.\n", strerror(errno));
return EXIT_FAILURE;
}
while (1) {
struct timespec t = { .tv_sec = 0, .tv_nsec = 200000000L };
printf("Output\n");
fflush(stdout);
nanosleep(&t, NULL);
}
/* Never reached. */
return EXIT_SUCCESS;
}
When the process receives a SIGINT signal, it will first flush whatever is in kernel terminal buffer, then redirect the standard streams to /dev/null (i.e., nowhere).
Note that you'll need to kill the process by sending it the TERM or KILL signal (i.e. killall ./yourprogname in another terminal).
When you are running the verbose process over a remote connection, quite a lot of information may be in flight at all times. Both the local machine and the remote machine running the process will have their socket buffers nearly full, so the latency may be much larger than ordinarily -- I've seen several second latencies in this case even on fast (GbE) local networks.
This means that propagating the signal from the local machine to the remote machine will take a measurable time; in worst cases on the order of seconds. Only then will the remote process stop outputting data. All pending data will still have to be transmitted from the remote machine to the local machine, and that may take quite a long time. (Typically, the bottleneck is the terminal itself; in most cases it is faster to minimize the terminal, so that it does not try to render any of the text it receives, only buffers it internally.)
This is why Ctrl+C does not, and cannot, stop remote output instantaneously.
In most cases, you'll be using an SSH connection to the remote machine. The protocol does not have a "purge" feature, either, that might help here. Many, myself included, have thought about it -- at least my sausage fingers have accidentally tabbed to the executable file instead of the similarly named output file, and not only gotten the terminal full of garbage, but the special characters in binary files sometimes set the terminal state (see e.g. xterm control sequences, ANSI escape codes) to something unrecoverable (i.e., Ctrl+Z followed by reset Enter does not reset the terminal back to a working state; if it did, kill -KILL %- ; fg would stop the errant command in Bash, and get you your terminal back), and you need to break the connection, which will also terminate all processes started from the same terminal running remotely in the background.
The solution here is to use a terminal multiplexer, like GNU screen, which allows you to connect to and disconnect from the remote machine, without interrupting an existing terminal connection. (To put it simply, screen is your terminal avatar on the remote machine.)
First up, a quote from the C11 standard, emphasis mine:
7.14.1.1 The signal function
5 If the signal occurs other than as the result of calling the abort or raise function, the behaviour is undefined if [...] the signal handler calls any function in the standard library other than the abort function, the _Exit function, the quick_exit function, or the signal function with the first argumentt equal to the signal number corresponding to the signal that caused the invocation of the handler.
This means calling fflush is undefined behaviour.
Looking at the functions you may call, abort and _Exit both leave the flushing of buffers implementation-defined, and quick_exit calls _Exit, so you are out of luck as far as far as the standard is concerned since I could not find the implementation's definition on their behaviour for Linux. (Surprise. Not.)
The only other "terminating" function, exit, does flush the buffers, and you may not call it from the handler in the first place.
So you have to look at Linux-specific functionality. The man page to _exit makes no statement on buffers. The close man page warns against closing file descriptors that may be in use by system calls from other threads, and states that "it is not common for a filesystem to flush the buffers when the stream is closed", meaning that it could happen (i.e. close not guaranteeing that unwritten buffer contents are actually discarded).
At this point, if I were you, I would ask myself "is this such a good idea after all"...
The problem is that neither Posix nor Linux library declares that fpurge nor __fpurge to be safe in a signal handler function. As explained by DevSolar, C language itsel does not declare many safe functions for standard library (at least _Exit, but Posix explicitely allows close and write. So, you can always close the underlying file descriptor which should be 1:
void handler(int sig) {
static char msg[] = "Interrupted";
write(2, msg, sizeof(msg) - 1); // carefully use stderr here
close(1); // foo is displayed if this line is commented out
_Exit(1);
}
int main() {
signal(SIGINT, handler);
printf("bar");
sleep(15);
return 0;
}
When I type Ctrl-C during the sleep it gives as expected:
$ ./foo
^CInterrupted with 2
$
The close system call should be enough, because as it closes the underlying file descriptor. So even if there are later attemps to flush stdout buffer, they will write on a closed file descriptor as as such have no effect at all. The downside is that stdout has been redirected, the program should store the new value of the underlying file descriptor in a global variable.
If you do kill(getpid(), SIGKILL); with in the signal handler (which is async-safe), you would get killed immediately by the OS (as you wanted to exit(0) anyway). Further output is not to be expected any more.
Only problem: you won't be able to clean up poperly, exit gracefully afterwards in the main thread. If you can afford that...
Program 1:
#include<stdio.h>
#include<signal.h>
void handler(int sig);
void main()
{
printf("PID: %d\n",getpid());
signal(SIGABRT,handler);
while(1){
printf("Hai\n");
sleep(1);
abort();
}
}
void handler(int sig)
{
printf("Signal handled\n");
}
Output 1:
$ ./a.out
PID: 32235
Hai
Signal handled
Aborted (core dumped)
$
As per the reference, the abort function works like raise(SIGABRT). So, the signal generated by abort() function is SIGABRT. So for that I created the above program.
In that program, SIGABRT signal is handled. After the execution of signal handler, it doesn't return to the main function from where it is called. Why does it not return to the main function after the handler is completed?
Program 2:
#include<stdio.h>
#include<signal.h>
void handler(int sig);
void main()
{
printf("PID: %d\n",getpid());
signal(SIGABRT,handler);
while(1){
printf("Hai\n");
sleep(1);
}
}
void handler(int sig)
{
printf("Signal handled\n");
}
Output 2:
$ ./a.out
PID: 32247
Hai
Hai
Hai
Signal handled
Hai
Signal handled
Hai
Hai
^C
$
Unlike program 1, program 2 executes as expected. In the above program, the signals are sent to the process via command line through the kill command as shown below.
$ kill -6 32247
$ kill -6 32247
So once the signal occurred, the handler function executed and then it returns to the main function. But it does not happen in program 1. Why does it behave like this? The abort function and SIGABRT are different?
See this piece of documentation from man 3 abort:
This results in the abnormal termination of the process unless the SIGABRT signal is caught and the signal handler does not return (see longjmp(3)).
And also this:
If the SIGABRT signal is ignored, or caught by a handler that returns, the abort() function will still terminate the process. It does this by restoring the default disposition for SIGABRT and then raising the signal for a second time.
So the only way you can prevent abort() from aborting your program is by longjmp()-ing from the signal handler.
Libc implements abort(). In their implementation, abort() checks to see if the process is still alive, because abort() is executing after the raise(SIGABRT). If it is, then it knows that the user has handled SIGABRT. According to the documentation, it doesn't matter, because the process will still exit:
You can see the exact implementation in the GLIBC source code (stdlib/abort.c):
/* Cause an abnormal program termination with core-dump. */
void
abort (void)
{
struct sigaction act;
sigset_t sigs;
/* First acquire the lock. */
__libc_lock_lock_recursive (lock);
/* Now it's for sure we are alone. But recursive calls are possible. */
/* Unlock SIGABRT. */
if (stage == 0)
{
++stage;
if (__sigemptyset (&sigs) == 0 &&
__sigaddset (&sigs, SIGABRT) == 0)
__sigprocmask (SIG_UNBLOCK, &sigs, (sigset_t *) NULL);
}
/* Flush all streams. We cannot close them now because the user
might have registered a handler for SIGABRT. */
if (stage == 1)
{
++stage;
fflush (NULL);
}
/* Send signal which possibly calls a user handler. */
if (stage == 2)
{
/* This stage is special: we must allow repeated calls of
`abort' when a user defined handler for SIGABRT is installed.
This is risky since the `raise' implementation might also
fail but I don't see another possibility. */
int save_stage = stage;
stage = 0;
__libc_lock_unlock_recursive (lock);
raise (SIGABRT);
__libc_lock_lock_recursive (lock);
stage = save_stage + 1;
}
/* There was a handler installed. Now remove it. */
if (stage == 3)
{
++stage;
memset (&act, '\0', sizeof (struct sigaction));
act.sa_handler = SIG_DFL;
__sigfillset (&act.sa_mask);
act.sa_flags = 0;
__sigaction (SIGABRT, &act, NULL);
}
/* Now close the streams which also flushes the output the user
defined handler might has produced. */
if (stage == 4)
{
++stage;
__fcloseall ();
}
/* Try again. */
if (stage == 5)
{
++stage;
raise (SIGABRT);
}
/* Now try to abort using the system specific command. */
if (stage == 6)
{
++stage;
ABORT_INSTRUCTION;
}
/* If we can't signal ourselves and the abort instruction failed, exit. */
if (stage == 7)
{
++stage;
_exit (127);
}
/* If even this fails try to use the provided instruction to crash
or otherwise make sure we never return. */
while (1)
/* Try for ever and ever. */
ABORT_INSTRUCTION;
}
The abort function sends the SIGABRT signal that's true, but it doesn't matter if you catch (or ignore) that signal, the abort function will still exit your process.
From the linked manual page:
RETURN VALUE
The abort() function never returns.
According to the standard it's not entirely specified what should happen if you handle SIGABRT:
The abort function causes abnormal program termination to occur,
unless the signal SIGABRT is being caught and the signal handler does
not return. Whether open streams with unwritten buffered data are
flushed, open streams are closed, or temporary files are removed is
implementation-defined. An implementation-defined form of the status
unsuccessful termination is returned to the host environment by means
of the function call raise(SIGABRT) .
However it's specified what should not happen:
The abort function does not return to its caller.
So the correct behavior is to ensure that an "abnormal termination" occurs. This ensured by the abort function doing it's very best to terminate the program abnormally, it does this by trying to terminate in various ways and if nothing seem to do the trick it enters an infinite loop (and at least ensure that it does not return to the caller).
They are not the same. The abort function calls raise(SIGABRT) twice. If you defined a handler for the SIGABRT, it will call your handler first and call the default one after that.
I have a large C/C++ program on a Suse linux system. We do automated testing of it with a bash script, which sends input to the program, and reads the output. It's mainly "black-box" testing, but some tests need to know a few internal details to determine if a test has passed.
One test in particular needs to know how times the program runs a certain function (which parses a particular response message). When that function runs it issues a log and increments a counter variable. The automated test currently determines the number of invocations by grepping in the log file for the log message, and counting the number of occurrences before and after the test. This isn't ideal, because the logs (syslog-ng) aren't guaranteed, and they're frequently turned off by configuration, because they're basically debug logs.
I'm looking for a better alternative. I can change the program to enhance the testability, but it shouldn't be heavy impact to normal operation. My first thought was, I could just read the counter after each test. Something like this:
gdb --pid=$PID --batch -ex "p numServerResponseX"
That's slow when it runs, but it's good because the program doesn't need to be changed at all. With a little work, I could probably write a ptrace command to do this a little more efficiently.
But I'm wondering if there isn't a simpler way to do this. Could I write the counter to shared memory (with shm_open / mmap), and then read /dev/shm in the bash script? Is there some simpler way I could setup the counter to make it easy to read, without making it slow to increment?
Edit:
Details: The test setup is like this:
testScript <-> sipp <-> programUnderTest <-> externalServer
The bash testScript injects sip messages with sipp, and it generally determines success or failure based on the completion code from sipp. But in certain tests it needs to know the number of responses the program received from the external server. The function "processServerResponseX" processes certain responses from the external server. During the testing there isn't much traffic running, so the function is only invoked perhaps 20 times over 10 seconds. When each test ends and we want to check the counter, there should be essentially no traffic. However during normal operation, it might be invoked hundreds of times a second. The function is roughly:
unsigned long int numServerResponseX;
int processServerResponseX(DMsg_t * dMsg, AppId id)
{
if (DEBUG_ENABLED)
{
syslog(priority, "%s received %d", __func__, (int) id);
}
myMutex->getLock();
numServerResponseX++;
doLockedStuff(dMsg, id);
myMutex->releaseLock();
return doOtherStuff(dMsg, id);
}
The script currently does:
grep processServerResponseX /var/log/logfile | wc -l
and compares the value before and after. My goal is to have this work even if DEBUG_ENABLED is false, and not have it be too slow. The program is multi-threaded, and it runs on an i86_64 smp machine, so adding any long blocking function would not be a good solution.
I would have that certain function "(which parses a particular response message)" write (probably using fopen then fprintf then fclose) some textual data somewhere.
That destination could be a FIFO (see fifo(7) ...) or a temporary file in a tmpfs file system (which is a RAM file system), maybe /run/
If your C++ program is big and complex enough, you could consider adding some probing facilities (some means for an external program to query about the internal state of your C++ program) e.g. a dedicated web service (using libonion in a separate thread), or some interface to systemd, or to D-bus, or some remote procedure call service like ONC/RPC, JSON-RPC, etc etc...
You might be interested by POCOlib. Perhaps its logging framework should interest you.
As you mentioned, you might use Posix shared memory & semaphores (see shm_overview(7) and sem_overview(7) ...).
Perhaps the Linux specific eventfd(2) is what you need.... (you could code a tiny C program to be invoked by your testing bash scripts....)
You could also try to change the command line (I forgot how to do that, maybe libproc or write to /proc/self/cmdline see proc(5)...). Then ps would show it.
I personally do usually use the methods Basile Starynkevitch outlined for this, but I wanted to bring up an alternative method using realtime signals.
I am not claiming this is the best solution, but it is simple to implement and has very little overhead. The main downside is that the size of the request and response are both limited to one int (or technically, anything representable by an int or by a void *).
Basically, you use a simple helper program to send a signal to the application. The signal has a payload of one int your application can examine, and based on it, the application responds by sending the same signal back to the originator, with an int of its own as payload.
If you don't need any locking, you can use a simple realtime signal handler. When it catches a signal, it examines the siginfo_t structure. If sent via sigqueue(), the request is in the si_value member of the siginfo_t structure. The handler answers to the originating process (si_pid member of the structure) using sigqueue(), with the response. This only requires about sixty lines of code to be added to your application. Here is an example application, app1.c:
#define _POSIX_C_SOURCE 200112L
#include <unistd.h>
#include <signal.h>
#include <errno.h>
#include <string.h>
#include <time.h>
#include <stdio.h>
#define INFO_SIGNAL (SIGRTMAX-1)
/* This is the counter we're interested in */
static int counter = 0;
static void responder(int signum, siginfo_t *info,
void *context __attribute__((unused)))
{
if (info && info->si_code == SI_QUEUE) {
union sigval value;
int response, saved_errno;
/* We need to save errno, to avoid interfering with
* the interrupted thread. */
saved_errno = errno;
/* Incoming signal value (int) determines
* what we respond back with. */
switch (info->si_value.sival_int) {
case 0: /* Request loop counter */
response = *(volatile int *)&counter;
break;
/* Other codes? */
default: /* Respond with -1. */
response = -1;
}
/* Respond back to signaler. */
value.sival_ptr = (void *)0L;
value.sival_int = response;
sigqueue(info->si_pid, signum, value);
/* Restore errno. This way the interrupted thread
* will not notice any change in errno. */
errno = saved_errno;
}
}
static int install_responder(const int signum)
{
struct sigaction act;
sigemptyset(&act.sa_mask);
act.sa_sigaction = responder;
act.sa_flags = SA_SIGINFO;
if (sigaction(signum, &act, NULL))
return errno;
else
return 0;
}
int main(void)
{
if (install_responder(INFO_SIGNAL)) {
fprintf(stderr, "Cannot install responder signal handler: %s.\n",
strerror(errno));
return 1;
}
fprintf(stderr, "PID = %d\n", (int)getpid());
fflush(stderr);
/* The application follows.
* This one just loops at 100 Hz, printing a dot
* about once per second or so. */
while (1) {
struct timespec t;
counter++;
if (!(counter % 100)) {
putchar('.');
fflush(stdout);
}
t.tv_sec = 0;
t.tv_nsec = 10000000; /* 10ms */
nanosleep(&t, NULL);
/* Note: Since we ignore the remainder
* from the nanosleep call, we
* may sleep much shorter periods
* when a signal is delivered. */
}
return 0;
}
The above responder responds to query 0 with the counter value, and with -1 to everything else. You can add other queries simply by adding a suitable case statement in responder().
Note that locking primitives (except for sem_post()) are not async-signal safe, and thus should not be used in a signal handler. So, the above code cannot implement any locking.
Signal delivery can interrupt a thread in a blocking call. In the above application, the nanosleep() call is usually interrupted by the signal delivery, causing the sleep to be cut short. (Similarly, read() and write() calls may return -1 with errno == EINTR, if they were interrupted by signal delivery.)
If that is a problem, or you are not sure if all your code handles errno == EINTR correctly, or your counters need locking, you can use separate thread dedicated for the signal handling instead.
The dedicated thread will sleep unless a signal is delivered, and only requires a very small stack, so it really does not consume any significant resources at run time.
The target signal is blocked in all threads, with the dedicated thread waiting in sigwaitinfo(). If it catches any signals, it processes them just like above -- except that since this is a thread and not a signal handler per se, you can freely use any locking etc., and do not need to limit yourself to async-signal safe functions.
This threaded approach is slightly longer, adding almost a hundred lines of code to your application. (The differences are contained in the responder() and install_responder() functions; even the code added to main() is exactly the same as in app1.c.)
Here is app2.c:
#define _POSIX_C_SOURCE 200112L
#include <signal.h>
#include <errno.h>
#include <pthread.h>
#include <string.h>
#include <time.h>
#include <stdio.h>
#define INFO_SIGNAL (SIGRTMAX-1)
/* This is the counter we're interested in */
static int counter = 0;
static void *responder(void *payload)
{
const int signum = (long)payload;
union sigval response;
sigset_t sigset;
siginfo_t info;
int result;
/* We wait on only one signal. */
sigemptyset(&sigset);
if (sigaddset(&sigset, signum))
return NULL;
/* Wait forever. This thread is automatically killed, when the
* main thread exits. */
while (1) {
result = sigwaitinfo(&sigset, &info);
if (result != signum) {
if (result != -1 || errno != EINTR)
return NULL;
/* A signal was delivered using *this* thread. */
continue;
}
/* We only respond to sigqueue()'d signals. */
if (info.si_code != SI_QUEUE)
continue;
/* Clear response. We don't leak stack data! */
memset(&response, 0, sizeof response);
/* Question? */
switch (info.si_value.sival_int) {
case 0: /* Counter */
response.sival_int = *(volatile int *)(&counter);
break;
default: /* Unknown; respond with -1. */
response.sival_int = -1;
}
/* Respond. */
sigqueue(info.si_pid, signum, response);
}
}
static int install_responder(const int signum)
{
pthread_t worker_id;
pthread_attr_t attrs;
sigset_t mask;
int retval;
/* Mask contains only signum. */
sigemptyset(&mask);
if (sigaddset(&mask, signum))
return errno;
/* Block signum, in all threads. */
if (sigprocmask(SIG_BLOCK, &mask, NULL))
return errno;
/* Start responder() thread with a small stack. */
pthread_attr_init(&attrs);
pthread_attr_setstacksize(&attrs, 32768);
retval = pthread_create(&worker_id, &attrs, responder,
(void *)(long)signum);
pthread_attr_destroy(&attrs);
return errno = retval;
}
int main(void)
{
if (install_responder(INFO_SIGNAL)) {
fprintf(stderr, "Cannot install responder signal handler: %s.\n",
strerror(errno));
return 1;
}
fprintf(stderr, "PID = %d\n", (int)getpid());
fflush(stderr);
while (1) {
struct timespec t;
counter++;
if (!(counter % 100)) {
putchar('.');
fflush(stdout);
}
t.tv_sec = 0;
t.tv_nsec = 10000000; /* 10ms */
nanosleep(&t, NULL);
}
return 0;
}
For both app1.c and app2.c the application itself is the same.
The only modifications needed to the application are making sure all the necessary header files get #included, adding responder() and install_responder(), and a call to install_responder() as early as possible in main().
(app1.c and app2.c only differ in responder() and install_responder(); and in that app2.c needs pthreads.)
Both app1.c and app2.c use the signal SIGRTMAX-1, which should be unused in most applications.
app2.c approach, also has a useful side-effect you might wish to use in general: if you use other signals in your application, but don't want them to interrupt blocking I/O calls et cetera -- perhaps you have a library that was written by a third party, and does not handle EINTR correctly, but you do need to use signals in your application --, you can simply block the signals after the install_responder() call in your application. The only thread, then, where the signals are not blocked is the responder thread, and the kernel will use tat to deliver the signals. Therefore, the only thread that will ever get interrupted by the signal delivery is the responder thread, more specifically sigwaitinfo() in responder(), and it ignores any interruptions. If you use for example async I/O or timers, or this is a heavy math or data processing application, this might be useful.
Both application implementations can be queried using a very simple query program, query.c:
#define _POSIX_C_SOURCE 200112L
#include <unistd.h>
#include <signal.h>
#include <string.h>
#include <errno.h>
#include <time.h>
#include <stdio.h>
int query(const pid_t process, const int signum,
const int question, int *const response)
{
sigset_t prevmask, waitset;
struct timespec timeout;
union sigval value;
siginfo_t info;
int result;
/* Value sent to the target process. */
value.sival_int = question;
/* Waitset contains only signum. */
sigemptyset(&waitset);
if (sigaddset(&waitset, signum))
return errno = EINVAL;
/* Block signum; save old mask into prevmask. */
if (sigprocmask(SIG_BLOCK, &waitset, &prevmask))
return errno;
/* Send the signal. */
if (sigqueue(process, signum, value)) {
const int saved_errno = errno;
sigprocmask(signum, &prevmask, NULL);
return errno = saved_errno;
}
while (1) {
/* Wait for a response within five seconds. */
timeout.tv_sec = 5;
timeout.tv_nsec = 0L;
/* Set si_code to an uninteresting value,
* just to be safe. */
info.si_code = SI_KERNEL;
result = sigtimedwait(&waitset, &info, &timeout);
if (result == -1) {
/* Some other signal delivered? */
if (errno == EINTR)
continue;
/* No response; fail. */
sigprocmask(SIG_SETMASK, &prevmask, NULL);
return errno = ETIMEDOUT;
}
/* Was this an interesting signal? */
if (result == signum && info.si_code == SI_QUEUE) {
if (response)
*response = info.si_value.sival_int;
/* Return success. */
sigprocmask(SIG_SETMASK, &prevmask, NULL);
return errno = 0;
}
}
}
int main(int argc, char *argv[])
{
pid_t pid;
int signum, question, response;
long value;
char dummy;
if (argc < 3 || argc > 4 ||
!strcmp(argv[1], "-h") || !strcmp(argv[1], "--help")) {
fprintf(stderr, "\n");
fprintf(stderr, "Usage: %s [ -h | --help ]\n", argv[0]);
fprintf(stderr, " %s PID SIGNAL [ QUERY ]\n", argv[0]);
fprintf(stderr, "\n");
return 1;
}
if (sscanf(argv[1], " %ld %c", &value, &dummy) != 1) {
fprintf(stderr, "%s: Invalid process ID.\n", argv[1]);
return 1;
}
pid = (pid_t)value;
if (pid < (pid_t)1 || value != (long)pid) {
fprintf(stderr, "%s: Invalid process ID.\n", argv[1]);
return 1;
}
if (sscanf(argv[2], "SIGRTMIN %ld %c", &value, &dummy) == 1)
signum = SIGRTMIN + (int)value;
else
if (sscanf(argv[2], "SIGRTMAX %ld %c", &value, &dummy) == 1)
signum = SIGRTMAX + (int)value;
else
if (sscanf(argv[2], " %ld %c", &value, &dummy) == 1)
signum = value;
else {
fprintf(stderr, "%s: Unknown signal.\n", argv[2]);
return 1;
}
if (signum < SIGRTMIN || signum > SIGRTMAX) {
fprintf(stderr, "%s: Not a realtime signal.\n", argv[2]);
return 1;
}
/* Clear the query union. */
if (argc > 3) {
if (sscanf(argv[3], " %d %c", &question, &dummy) != 1) {
fprintf(stderr, "%s: Invalid query.\n", argv[3]);
return 1;
}
} else
question = 0;
if (query(pid, signum, question, &response)) {
switch (errno) {
case EINVAL:
fprintf(stderr, "%s: Invalid signal.\n", argv[2]);
return 1;
case EPERM:
fprintf(stderr, "Signaling that process was not permitted.\n");
return 1;
case ESRCH:
fprintf(stderr, "No such process.\n");
return 1;
case ETIMEDOUT:
fprintf(stderr, "No response.\n");
return 1;
default:
fprintf(stderr, "Failed: %s.\n", strerror(errno));
return 1;
}
}
printf("%d\n", response);
return 0;
}
Note that I did not hardcode the signal number here; use SIGRTMAX-1 on the command line for app1.c and app2.c. (You can change it. query.c does understand SIGRTMIN+n too. You must use a realtime signal, SIGRTMIN+0 to SIGRTMAX-0, inclusive.)
You can compile all three programs using
gcc -Wall -O3 app1.c -o app1
gcc -Wall -O3 app2.c -lpthread -o app2
gcc -Wall -O3 query.c -o query
Both ./app1 and ./app2 print their PIDs, so you don't need to look for it. (You can find the PID using e.g. ps -o pid= -C app1 or ps -o pid= -C app2, though.)
If you run ./app1 or ./app2 in one shell (or both in separate shells), you can see them outputting the dots at about once per second. The counter increases every 1/100th of a second. (Press Ctrl+C to stop.)
If you run ./query PID SIGRTMAX-1 in another shell in the same directory on the same machine, you can see the counter value.
An example run on my machine:
A$ ./app1
PID = 28519
...........
B$ ./query 28519 SIGRTMAX-1
11387
C$ ./app2
PID = 28522
...
B$ ./query 28522 SIGRTMAX -1
371
As mentioned, the downside of this mechanism is that the response is limited to one int (or technically an int or a void *). There are ways around that, however, by also using some of the methods Basile Starynkevich outlined. Typically, the signal is then just a notification for the application that it should update the state stored in a file, shared memory segment, or wherever. I recommend using the dedicated thread approach for that, as it has very little overheads, and minimal impact on the application itself.
Any questions?
A hard-coded systemtap solution could look like:
% cat FOO.stp
global counts
probe process("/path/to/your/binary").function("CertainFunction") { counts[pid()] <<< 1 }
probe process("/path/to/your/binary").end { println ("pid %d count %sd", pid(), #count(counts[pid()]))
delete counts[pid()] }
# stap FOO.stp
pid 42323 count 112
pid 2123 count 0
... etc, until interrupted
Thanks for the responses. There is lots of good information in the other answers. However, here's what I did. First I tweaked the program to add a counter in a shm file:
struct StatsCounter {
char counterName[8];
unsigned long int counter;
};
StatsCounter * stats;
void initStatsCounter()
{
int fd = shm_open("TestStats", O_RDWR|O_CREAT, 0);
if (fd == -1)
{
syslog(priority, "%s:: Initialization Failed", __func__);
stats = (StatsCounter *) malloc(sizeof(StatsCounter));
}
else
{
// For now, just one StatsCounter is used, but it could become an array.
ftruncate(fd, sizeof(StatsCounter));
stats = (StatsCounter *) mmap(NULL, sizeof(StatsCounter),
PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
}
// Initialize names. Pad them to 7 chars (save room for \0).
snprintf(stats[0].counterName, sizeof(stats[0].counterName), "nRespX ");
stats[0].counter = 0;
}
And changed processServerResponseX to increment stats[0].counter in the locked section. Then I changed the script to parse the shm file with "hexdump":
hexdump /dev/shm/TestStats -e ' 1/8 "%s " 1/8 "%d\n"'
This will then show something like this:
nRespX 23
This way I can extend this later if I want to also look at response Y, ...
Not sure if there are mutual exclusion problems with hexdump if it accessed the file while it was being changed. But in my case, I don't think it matters, because the script only calls it before and after the test, it should not be in the middle of an update.