I was writing a project using MPI for a parallel programming course, and decided to name one of my functions connect(). But whenever I tried to mpirun the program (using recent versions of Open MPI on Linux and OS X), I would receive output from the connect() function, even if I had not called connect() from main(); also, some of the output from main() would not appear.
This is a simplified program with the issue:
#include <stdlib.h>
#include <stdio.h>
#include <mpi.h>
void connect(); //function name breaks mpi
int main(void) {
int comm_sz, my_rank;
MPI_Init(NULL, NULL);
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
printf("my_rank is %d\n", my_rank);
fflush(stdout);
MPI_Finalize();
return EXIT_SUCCESS;
}
void connect() {
printf("\nNot main! \n");
return;
}
and the output:
[me#host ~]$ mpicc bad.c -Wall
[me#host ~]$ mpirun -n 1 a.out
Not main!
--------------------------------------------------------------------------
orterun noticed that process rank 0 with PID 17245 on node host exited on signal 13 (Broken pipe).
--------------------------------------------------------------------------
I was about to ask on Stack Overflow what was wrong in the first place, until I discovered that renaming the function fixes it. So what I'm curious about now is why naming the function connect() prevents the program from running correctly. Could it also be an issue with mpirun/Open RTE?
Possible leads:
There's a connect() function in <sys/socket.h>, but I haven't yet found it mentioned in the MPI header files.
There's also a Connect() function (with an uppercase C) in "ompi/mpi/cxx/intracomm.h" which is indirectly included by <mpi.h>, but I thought case mattered in C/C++, and it looks like a method of a C++ class.
If I try executing the program like a normal one, it works when run on OS X, but not on Linux:
mac:~ me$ ./a.out
my_rank is 0
vs
[me#linux ~]$ ./a.out
Not main!
I would guess that one of the MPI functions you call is in turn calling the connect() system call. But since ELF executables have a flat namespace for symbols, your connect() is being called instead.
The problem doesn't happen on Mac OS because Mach-O libraries have a two-level namespace, so symbols in different libraries don't conflict with each other.
If you make your function static, that would probably avoid the problem as well.
Related
I am using a shared library with LD_PRELOAD, and it seems that I can't call some functions from the function set with -fini= ld option. I am running Linux Ubuntu 20.04 on a 64-bit machine.
Here is the SSCCE:
shared.sh:
#!/bin/bash
gcc -shared -fPIC -Wl,-init=init -Wl,-fini=fini shared.c -o shared.so
LD_PRELOAD=$PWD/shared.so whoami
shared.c:
#include <stdio.h>
#include <unistd.h>
void init() {
printf("%s\n", __func__);
fflush(stdout);
}
void fini() {
int printed;
printed = printf("%s\n", __func__);
if (printed < 0)
sleep(2);
fflush(stdout);
}
When I call ./shared.sh , I get
init
mark
and 2 second pause.
So it seems printf() fails in fini() but sleep() succeeds (errno values are not specified for printf, so I don't check it) Why and what kind of functions can I call from fini? ld manpage does not say anything about any restrictions.
The initialization functions of each dynamically linked component are executed in the order in which the components are loaded. In particular, if A depends on B but B does not depend on A, then B's initialization functions run before A's. The termination functions of each dynamically linked component are executed in the order in which the components are unloaded. In particular, if A depends on B but B does not depend on A, then B's initialization functions run after A's. Generally, termination functions run in reverse order from initialization functions, but I don't know if that's true in all cases (for example when there are circular dependencies). You can find the rules in the System V ABI specification which Linux and many other Unix variants follow. Note that the rules leave some cases unspecified; they might depend on the compiler and on the standard library (possibly on the kernel, but I think for this particular topic it doesn't matter).
A shared library loaded with LD_PRELOAD is loaded before the main executable, so its initialization functions run before the ones from libc and its termination functions run after the ones from libc. In particular, libc flushes standard streams and closes the file descriptors for the output streams. You can see this happening by tracing system calls:
$ strace env LD_PRELOAD=$PWD/shared.so whoami
…
write(1, "gilles\n", 6gilles
) = 6
close(1) = 0
close(2) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=2, tv_nsec=0}, 0x7ffc12bd2df0) = 0
exit_group(0) = ?
+++ exited with 0 +++
The call to clock_nanosleep is sleep(2). The calls to printf and fflush happen just before; since stdout has been closed, they do nothing and return -1. Check the return value or use a debugger to confirm this.
Contrast with what happens if shared.so is linked normally, rather than preloaded.
$ cat main.c
#include <stdio.h>
int main(void) {
puts("main");
return 0;
}
$ gcc -o main main.c -Wl,-rpath,. -Wl,--no-as-needed -L. -l:shared.so
$ ./main
init
main
fini
Here, since main loads shared.so, the shared library is initialized last and terminated first. So by the time the fini function in shared.so runs, libc hasn't run its termination functions and the standard streams are still available.
I need to use MUMPS in my project. I wanted to test the installation with the simple example of solving equations (below). The problem is, that this code works correctly in my personal PC, but it gives me this error every time I try to run it in my work computer:
*** The MPI_Comm_f2c() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[espreso-ws:3263] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed!
I've tried to look for possible solutions, but I've found only these questions:
error: The MPI_Send() function was called before MPI_INIT was invoked
http://www.open-mpi.org/community/lists/users/2012/05/19262.php
None of them provides any solution.
So now, I'm incredibly confused about this and I don't really know, what should I try to do. There IS MPI_Init() invoked before MPI_Comm_f2c() of course.
Do you know, how to solve this problem?
mumps_solve.c
/*
* file c_example.c
* This file is part of MUMPS 4.10.0
* To run: aprun -n 2 ./dsimpletest < input_simpletest_real
*/
/* Example program using the C interface to the
* double real arithmetic version of MUMPS, dmumps_c.
* We solve the system A x = RHS
*/
#include <stdio.h>
#include <string.h>
#include <mpi.h>
#include <dmumps_c.h>
#define JOB_INIT -1
#define JOB_END -2
#define USE_COMM_WORLD -987654
int main(int argc, char ** argv)
{
DMUMPS_STRUC_C id;
int n = 2;
int nz = 4;
int irn[] = {1,1,2,2};
int jcn[] = {1,2,1,2};
double a[4];
double rhs[2];
int myid, ierr;
ierr = MPI_Init(&argc, &argv);
ierr = MPI_Comm_rank(MPI_COMM_WORLD, &myid);
/* Define A and rhs */
rhs[0]=5.0;rhs[1]=2.0;
a[0]=3.0;a[1]=2.0;a[2]=1.0;a[3]=4.0;
/* Initialize a MUMPS instance. Use MPI_COMM_WORLD */
id.job=JOB_INIT;id.par=1; id.sym=0;id.comm_fortran=USE_COMM_WORLD;
dmumps_c(&id); // here the program crashes
/* Define the problem on the host */
if (myid == 0) {
id.n = n; id.nz =nz; id.irn=irn; id.jcn=jcn;
id.a = a; id.rhs = rhs;
}
#define ICNTL(I) icntl[(I)-1] /* macro s.t. indices match documentation */
/* No outputs */
id.ICNTL(1)=-1; id.ICNTL(2)=-1; id.ICNTL(3)=-1; id.ICNTL(4)=0;
/* Call the MUMPS package. */
id.job=6;
dmumps_c(&id);
id.job=JOB_END; dmumps_c(&id); /* Terminate instance */
if (myid == 0) {
printf("Solution is : (%8.2f %8.2f)\n", rhs[0],rhs[1]);
}
ierr = MPI_Finalize();
return 0;
}
Makefile
.DEFAULT_GOAL=all
BINARIES=mumps_solve
.PHONY=all
all: mumps_solve
mumps_solve: mumps_solve.o
mpicc mumps_solve.o -o mumps_solve -ldmumps
mumps_solve.o: mumps_solve.c
mpicc -c mumps_solve.c -o mumps_solve.o
.PHONY=clean
clean:
rm -f ${BINARIES} *.o
The problem could be hidden in the linking. I see that the actual call which raises the error is in a library against which you are linking.
In short: you maybe linked against multiple copies of MPI. One is initialized, one is not. The latter is called by your library and raises an error.
Solution in short: compile both the external library and your own code again, verifying that all compilation is done by the exact same physical copy of mpicc.
The library could be compiled and linked against a different copy of your MPI distribution than the copy against which you link your main library. If that is the case, there are multiple copies of the global variable which keeps track of the initialization state of MPI. Your call to MPI_Init() goes to the MPI copy which you linked against by invoking mpicc in your Makefile (located at which mpicc). If you call MPI_Initialized(int*) after MPI_Init(), you should find that it returns true.
If you had the chance to modify the source of the external library (mumps), and call MPI_Initialized(int*) just before the line that crashes, you should find that it returns false, even though you called MPI_Init(). This is a hint that the library and your binary are linked against different copies of your MPI distribution.
In my case (after two days of bumping my head against the wall), it was even a matter of different dynamically linked versions of MPI. I had the luck to be in control of the compilation of the library, and could hence modify the version of MPI it links against.
I have some application for which I need to write extension using shared library. In my shared library I need to use threads. And main application neither uses threads neither linked with threads library (libpthread.so, for example).
As first tests showed my library causes crashes of the main application. And if i use LD_PRELOAD hack crashes goes away:
LD_PRELOAD=/path/to/libpthread.so ./app
The only OS where i have no segfaults without LD_PRELOAD hack is OS X. On other it just crashes. I tested: Linux, FreeBSD, NetBSD.
My question is: is there a way to make my threaded shared library safe for non-threaded application without changing of the main application and LD_PRELOAD hacks?
To reproduce the problem i wrote simple example:
mylib.c
#include <pthread.h>
#include <assert.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
void *_thread(void *arg) {
int i;
struct addrinfo *res;
for (i=0; i<10000; i++) {
if (getaddrinfo("localhost", NULL, NULL, &res) == 0) {
if (res) freeaddrinfo(res);
}
}
pthread_mutex_lock(&mutex);
printf("Just another thread message!\n");
pthread_mutex_unlock(&mutex);
return NULL;
}
void make_thread() {
pthread_t tid[10];
int i, rc;
for (i=0; i<10; i++) {
rc = pthread_create(&tid[i], NULL, _thread, NULL);
assert(rc == 0);
}
void *rv;
for (i=0; i<10; i++) {
rc = pthread_join(tid[i], &rv);
assert(rc == 0);
}
}
main.c
#include <stdio.h>
#include <dlfcn.h>
int main() {
void *mylib_hdl;
void (*make_thread)();
mylib_hdl = dlopen("./libmy.so", RTLD_NOW);
if (mylib_hdl == NULL) {
printf("dlopen: %s\n", dlerror());
return 1;
}
make_thread = (void (*)()) dlsym(mylib_hdl, "make_thread");
if (make_thread == NULL) {
printf("dlsym: %s\n", dlerror());
return 1;
}
(*make_thread)();
return 0;
}
Makefile
all:
cc -pthread -fPIC -c mylib.c
cc -pthread -shared -o libmy.so mylib.o
cc -o main main.c -ldl
clean:
rm *.o *.so main
And all together: https://github.com/olegwtf/sandbox/tree/bbbf76fdefe4bacef8a0de7a2475995719ae0436/threaded-so-for-non-threaded-app
$ make
cc -pthread -fPIC -c mylib.c
cc -pthread -shared -o libmy.so mylib.o
cc -o main main.c -ldl
$ ./main
*** glibc detected *** ./main: double free or corruption (fasttop): 0x0000000001614c40 ***
Segmentation fault
$ ldd libmy.so | grep thr
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe7e2591000)
$ LD_PRELOAD=/lib/x86_64-linux-gnu/libpthread.so.0 ./main
Just another thread message!
Just another thread message!
Just another thread message!
Just another thread message!
Just another thread message!
Just another thread message!
Just another thread message!
Just another thread message!
Just another thread message!
Just another thread message!
My question is: is there a way to make my threaded shared library safe
for non-threaded application without changing of the main application
and LD_PRELOAD hacks?
No, those are the two ways you can make it work. With neither in place, your program is invalid.
dlopen is supposed to do the right thing, and to open all the libraries your own .so depends upon.
In fact, your code is working for me if I comment out the address lookup code that you placed inside your thread function. So loading the pthread library works perfectly.
And if I run the code including the lookup, valgrind shows me that the crash is below getaddrinfo.
So the problem is not that the libraries aren't loaded, somehow their initialization code is not executed or not in the right order.
gdb helped to understand what's goin on with this example.
After 3 tries gdb showed that app always crashed at rewind.c line 36 inside libc. Since tests were run on Debian 7, libc implementation is eglibc. And here you can see line 36 of rewind.c:
http://www.eglibc.org/cgi-bin/viewvc.cgi/branches/eglibc-2_13/libc/libio/rewind.c?annotate=12752
_IO_acquire_lock() is a macros and after grepping eglibc source I found 2 places where it is defined:
bits/stdio-lock.h line 49: http://www.eglibc.org/cgi-bin/viewvc.cgi/branches/eglibc-2_13/libc/bits/stdio-lock.h?annotate=12752
sysdeps/pthread/bits/stdio-lock.h line 91: http://www.eglibc.org/cgi-bin/viewvc.cgi/branches/eglibc-2_13/libc/nptl/sysdeps/pthread/bits/stdio-lock.h?annotate=12752
Comment for first says Generic version and for second NPTL version, where NTPL is Native POSIX Thread Library. So in few words first defines non-threaded implementation for this and several other macroses and second threaded implementation.
When our main application is not linked with pthreads it starts and loads this first non-threaded implementation of _IO_acquire_lock() and others macroses. Then it opens our threaded shared library and executes function from it. And this function uses already loaded and non thread safe version of _IO_acquire_lock(). However in fact should use threads compatible version defined by pthreads. This is where segfault occures.
This is how it works on Linux. On *BSD situation is even more sad. On FreeBSD your program will hang up immediately after your threaded library will try to create new thread. On NetBSD instead of hang up program will be terminated with SIGABRT.
So answering to the main question: is it possible to use threaded shared library from application not linked with pthreads?
In general -- no. And particularly this depends on libc implementation. For OS X, for example, this will work without any problems. For Linux this will work if you'll not use libc functions that uses such special macroses redefined by pthreads. But how to know which uses? Ok, you can make 1+1, this looks safe. On *BSD your program will crash or hang up immediately, no matter what your thread do.
I have a program on C that uses both MPI and OpenMP. In order to compile such program on Windows system I have downloaded and installed a gcc compiler provided by MinGW. Using this compiler I can compile and execute C programs with OpenMP using the key -fopenmp for gcc. Such programs run without problems. In order to compile and execute C programs with MPI I have downloaded and installed MPICH2. Now I can compile and run such programs without problems, specifying additional parameters for gcc, provided by MinGW. But when I want to compile and run a program that uses both OpenMP and MPI I have a problem. I specified both keys -fopenmp and keys for MPI program for gcc compiler. Compilator didn't give me any error. I tried to launch my program by mpiexec, provided by MPICH2. My program didn't want to work (It was a HelloWorld program and it didn't print anything to output). Please help me to compile and launch such programs correctly.
Here is my HelloWorld program, that doesn't produce any output.
#include <stdio.h>
#include <mpi.h>
int main(int argc, char ** argv)
{
int thnum, thtotal;
int pid, np;
MPI_Init(&argc,&argv);
MPI_Comm_rank(MPI_COMM_WORLD,&pid);
MPI_Comm_size(MPI_COMM_WORLD,&np);
printf("Sequental %d out of %d!\n",pid,np);
MPI_Barrier(MPI_COMM_WORLD);
#pragma omp parallel private(thnum,thtotal)
{
thnum = omp_get_thread_num();
thtotal = omp_get_num_threads();
printf("parallel: %d out of %d from proc %d out of %d\n",thnum,thtotal,pid,np);
}
MPI_Barrier(MPI_COMM_WORLD);
MPI_Finalize();
return 0;
}
You can use the mpicc compiler with the -openmp option. For example,
mpicc -openmp hello.c -o hello
This might not be the root cause of your problem, but the MPI standard mandates that threaded programs use MPI_Init_thread() instead of MPI_Init(). In your case there are no MPI calls from within the parallel region so threading level of MPI_THREAD_FUNNELED should suffice. You should replace the call to MPI_Init() with:
int provided;
MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &provided);
if (provided < MPI_THREAD_FUNNELED)
{
MPI_Abort(MPI_COMM_WORLD, 1);
return 1; // Usually not reached
}
Although some MPI libraries might not advertise threading support (provided as returned is MPI_THREAD_SINGLE) they still work fine with hybrid OpenMP/MPI codes if one does not make MPI calls from within parallel regions.
The OpenMP portion of your program might require #include <omp.h> :
parallel: 0 out of 2 from proc 0 out of 0
parallel: 1 out of 2 from proc 0 out of 0
I'll explain:
Let's say I'm interested in replacing the rand() function used by a certain application.
So I attach gdb to this process and make it load my custom shared library (which has a customized rand() function):
call (int) dlopen("path_to_library/asdf.so")
This would place the customized rand() function inside the process' memory. However, at this point the symbol rand will still point to the default rand() function. Is there a way to make gdb point the symbol to the new rand() function, forcing the process to use my version?
I must say I'm also not allowed to use the LD_PRELOAD (linux) nor DYLD_INSERT_LIBRARIES (mac os x) methods for this, because they allow code injection only in the beginning of the program execution.
The application that I would like to replace rand(), starts several threads and some of them start new processes, and I'm interested in injecting code on one of these new processes. As I mentioned above, GDB is great for this purpose because it allows code injection into a specific process.
I followed this post and this presentation and came up with the following set of gdb commands for OSX with x86-64 executable, which can be loaded with -x option when attaching to the process:
set $s = dyld_stub_rand
set $p = ($s+6+*(int*)($s+2))
call (void*)dlsym((void*)dlopen("myrand.dylib"), "my_rand")
set *(void**)$p = my_rand
c
The magic is in set $p = ... command. dyld_stub_rand is a 6-byte jump instruction. Jump offset is at dyld_stub_rand+2 (4 bytes). This is a $rip-relative jump, so add offset to what $rip would be at this point (right after the instruction, dyld_stub_rand+6).
This points to a symbol table entry, which should be either real rand or dynamic linker routine to load it (if it was never called). It is then replaced by my_rand.
Sometimes gdb will pick up dyld_stub_rand from libSystem or another shared library, if that happens, unload them first with remove-symbol-file before running other commands.
This question intrigued me, so I did a little research. What you are looking for is a 'dll injection'. You write a function to replace some library function, put it in a .so, and tell ld to preload your dll. I just tried it out and it worked great! I realize this doesn't really answer your question in relation to gdb, but I think it offers a viable workaround.
For a gdb-only solution, see my other solution.
// -*- compile-command: "gcc -Wall -ggdb -o test test.c"; -*-
// test.c
#include "stdio.h"
#include "stdlib.h"
int main(int argc, char** argv)
{
//should print a fairly random number...
printf("Super random number: %d\n", rand());
return 0;
}
/ -*- compile-command: "gcc -Wall -fPIC -shared my_rand.c -o my_rand.so"; -*-
//my_rand.c
int rand(void)
{
return 42;
}
compile both files, then run:
LD_PRELOAD="./my_rand.so" ./test
Super random number: 42
I have a new solution, based on the new original constraints. (I am not deleting my first answer, as others may find it useful.)
I have been doing a bunch of research, and I think it would work with a bit more fiddling.
In your .so rename your replacement rand function, e.g my_rand
Compile everything and load up gdb
Use info functions to find the address of rand in the symbol table
Use dlopen then dlsym to load the function into memory and get its address
call (int) dlopen("my_rand.so", 1) -> -val-
call (unsigned int) dlsym(-val-, "my_rand") -> my_rand_addr
-the tricky part- Find the hex code of a jumpq 0x*my_rand_addr* instruction
Use set {int}*rand_addr* = *my_rand_addr* to change symbol table instruction
Continue execution: now whenever rand is called, it will jump to my_rand instead
This is a bit complicated, and very round-about, but I'm pretty sure it would work. The only thing I haven't accomplished yet is creating the jumpq instruction code. Everything up until that point works fine.
I'm not sure how to do this in a running program, but perhaps LD_PRELOAD will work for you. If you set this environment variable to a list of shared objects, the runtime loader will load the shared object early in the process and allow the functions in it to take precedence over others.
LD_PRELOAD=path_to_library/asdf.so path/to/prog
You do have to do this before you start the process but you don't have to rebuild the program.
Several of the answers here and the code injection article you linked to in your answer cover chunks of what I consider the optimal gdb-oriented solution, but none of them pull it all together or cover all the points. The code-expression of the solution is a bit long, so here's a summary of the important steps:
Load the code to inject. Most of the answers posted here use what I consider the best approach -- call dlopen() in the inferior process to link in a shared library containing the injected code. In the article you linked to the author instead loaded a relocatable object file and hand-linked it against the inferior. This is quite frankly insane -- relocatable objects are not "ready-to-run" and include relocations even for internal references. And hand-linking is tedious and error-prone -- far simpler to let the real runtime dynamic linker do the work. This does mean getting libdl into the process in the first place, but there are many options for doing that.
Create a detour. Most of the answers posted here so far have involved locating the PLT entry for the function of interest, using that to find the matching GOT entry, then modifying the GOT entry to point to your injected function. This is fine up to a point, but certain linker features -- e.g., use of dlsym -- can circumvent the GOT and provide direct access to the function of interest. The only way to be certain of intercepting all calls to a particular function is overwrite the initial instructions of that function's code in-memory to create a "detour" redirecting execution to your injected function.
Create a trampoline (optional). Frequently when doing this sort of injection you'll want to call the original function whose invocation you are intercepting. The way to allow this with a function detour is to create a small code "trampoline" which includes the overwritten instructions of the original function then a jump to the remainder of the original. This can be complex, because any IP-relative instructions in the copied set need to be modified to account for their new addresses.
Automate it all. These steps can be tedious, even if doing some of the simpler solutions posted in other answers. The best way to ensure that the steps are done correctly every time with variable parameters (injecting different functions, etc) is to automate their execution. Starting with the 7.0 series, gdb has included the ability to write new commands in Python. This support can be used to implement a turn-key solution for injecting and detouring code in/to the inferior process.
Here's an example. I have the same a and b executables as before and an inject2.so created from the following code:
#include <unistd.h>
#include <stdio.h>
int (*rand__)(void) = NULL;
int
rand(void)
{
int result = rand__();
printf("rand invoked! result = %d\n", result);
return result % 47;
}
I can then place my Python detour command in detour.py and have the following gdb session:
(gdb) source detour.py
(gdb) exec-file a
(gdb) set follow-fork-mode child
(gdb) catch exec
Catchpoint 1 (exec)
(gdb) run
Starting program: /home/llasram/ws/detour/a
a: 1933263113
a: 831502921
[New process 8500]
b: 918844931
process 8500 is executing new program: /home/llasram/ws/detour/b
[Switching to process 8500]
Catchpoint 1 (exec'd /home/llasram/ws/detour/b), 0x00007ffff7ddfaf0 in _start ()
from /lib64/ld-linux-x86-64.so.2
(gdb) break main
Breakpoint 2 at 0x4005d0: file b.c, line 7.
(gdb) cont
Continuing.
Breakpoint 2, main (argc=1, argv=0x7fffffffdd68) at b.c:7
7 {
(gdb) detour libc.so.6:rand inject2.so:rand inject2.so:rand__
(gdb) cont
Continuing.
rand invoked! result = 392103444
b: 22
Program exited normally.
In the child process, I create a detour from the rand() function in libc.so.6 to the rand() function in inject2.so and store a pointer to a trampoline for the original rand() in the rand__ variable of inject2.so. And as expected, the injected code calls the original, displays the full result, and returns that result modulo 47.
Due to length, I'm just linking to a pastie containing the code for my detour command. This is a fairly superficial implementation (especially in terms of the trampoline generation), but it should work well in a large percentage of cases. I've tested it with gdb 7.2 (most recently released version) on Linux with both 32-bit and 64-bit executables. I haven't tested it on OS X, but any differences should be relatively minor.
For executables you can easily find the address where the function pointer is stored by using objdump. For example:
objdump -R /bin/bash | grep write
00000000006db558 R_X86_64_JUMP_SLOT fwrite
00000000006db5a0 R_X86_64_JUMP_SLOT write
Therefore, 0x6db5a0 is the adress of the pointer for write. If you change it, calls to write will be redirected to your chosen function. Loading new libraries in gdb and getting function pointers has been covered in earlier posts. The executable and every library have their own pointers. Replacing affects only the module whose pointer was changed.
For libraries, you need to find the base address of the library and add it to the address given by objdump. In Linux, /proc/<pid>/maps gives it out. I don't know whether position-independent executables with address randomization would work. maps-information might be unavailable in such cases.
As long as the function you want to replace is in a shared library, you can redirect calls to that function at runtime (during debugging) by poking at the PLT. Here is an article that might be helpful:
Shared library call redirection using ELF PLT infection
It's written from the standpoint of malware modifying a program, but a much easier procedure is adaptable to live use in the debugger. Basically you just need to find the function's entry in the PLT and overwrite the address with the address of the function you want to replace it with.
Googling for "PLT" along with terms like "ELF", "shared library", "dynamic linking", "PIC", etc. might find you more details on the subject.
You can still us LD_PRELOAD if you make the preloaded function understand the situations it's getting used in. Here is an example that will use the rand() as normal, except inside a forked process when it will always return 42. I use the dl routines to load the standard library's rand() function into a function pointer for use by the hijacked rand().
// -*- compile-command: "gcc -Wall -fPIC -shared my_rand.c -o my_rand.so -ldl"; -*-
//my_rand.c
#include <sys/types.h>
#include <unistd.h>
#include <dlfcn.h>
int pid = 0;
int (*real_rand)(void) = NULL;
void f(void) __attribute__ ((constructor));
void f(void) {
pid = getpid();
void* dl = dlopen("libc.so.6", RTLD_LAZY);
if(dl) {
real_rand = dlsym(dl, "rand");
}
}
int rand(void)
{
if(pid == getpid() && real_rand)
return real_rand();
else
return 42;
}
//test.c
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char** argv)
{
printf("Super random number: %d\n", rand());
if(fork()) {
printf("original process rand: %d\n", rand());
} else {
printf("forked process rand: %d\n", rand());
}
return 0;
}
jdizzle#pudding:~$ ./test
Super random number: 1804289383
original process rand: 846930886
forked process rand: 846930886
jdizzle#pudding:~$ LD_PRELOAD="/lib/ld-linux.so.2 ./my_rand.so" ./test
Super random number: 1804289383
original process rand: 846930886
forked process rand: 42
I found this tutorial incredibly useful, and so far its the only way I managed to achieve what I was looking with GDB: Code Injection into Running Linux Application: http://www.codeproject.com/KB/DLL/code_injection.aspx
There is also a good Q&A on code injection for Mac here: http://www.mikeash.com/pyblog/friday-qa-2009-01-30-code-injection.html
I frequently use code injection as a method of mocking for automated testing of C code. If that's the sort of situation you're in -- if your use of GDB is simply because you're not interested in the parent processes, and not because you want to interactively select the processes which are of interest -- then you can still use LD_PRELOAD to achieve your solution. Your injected code just needs to determine whether it is in the parent or child processes. There are several ways you could do this, but on Linux, since your child processes exec(), the simplest is probably to look at the active executable image.
I produced two executables, one named a and the other b. Executable a prints the result of calling rand() twice, then fork()s and exec()s b twice. Executable b print the result of calling rand() once. I use LD_PRELOAD to inject the result of compiling the following code into the executables:
// -*- compile-command: "gcc -D_GNU_SOURCE=1 -Wall -std=gnu99 -O2 -pipe -fPIC -shared -o inject.so inject.c"; -*-
#include <sys/types.h>
#include <unistd.h>
#include <limits.h>
#include <stdio.h>
#include <dlfcn.h>
#define constructor __attribute__((__constructor__))
typedef int (*rand_t)(void);
typedef enum {
UNKNOWN,
PARENT,
CHILD
} state_t;
state_t state = UNKNOWN;
rand_t rand__ = NULL;
state_t
determine_state(void)
{
pid_t pid = getpid();
char linkpath[PATH_MAX] = { 0, };
char exepath[PATH_MAX] = { 0, };
ssize_t exesz = 0;
snprintf(linkpath, PATH_MAX, "/proc/%d/exe", pid);
exesz = readlink(linkpath, exepath, PATH_MAX);
if (exesz < 0)
return UNKNOWN;
switch (exepath[exesz - 1]) {
case 'a':
return PARENT;
case 'b':
return CHILD;
}
return UNKNOWN;
}
int
rand(void)
{
if (state == CHILD)
return 47;
return rand__();
}
constructor static void
inject_init(void)
{
rand__ = dlsym(RTLD_NEXT, "rand");
state = determine_state();
}
The result of running a with and without injection:
$ ./a
a: 644034683
a: 2011954203
b: 375870504
b: 1222326746
$ LD_PRELOAD=$PWD/inject.so ./a
a: 1023059566
a: 986551064
b: 47
b: 47
I'll post a gdb-oriented solution later.