I have a compiled source code executable which has redefined malloc() with a custom function CustMalloc() using macro substitution.
As seen from the below backtrace, the compiled source code executable is supposed to be dynamically linked with libMRegAccess.so and libusb-1.0.so.0 shared objects. Both these shared objects were independently compiled and they have no means of knowing the above macro substitution during their compilation.
Now, bsd-asprintf.c I presume is a linux source file and the call to asprintf() and vasprintf() are called from the libusb-1.0.so.0 shared object when executing, which in turn tries to call malloc().
I'm unable to understand why the custom function CustMalloc() is getting called instead of the actual malloc().
FYI, the semaphore that is required in this backtrace is not yet created and hence the crash. The expectation is that the CustMalloc() should not be invoked in this code flow as the call is being made from an independently built shared object.
Program received signal SIGSEGV, Segmentation fault.
[Switching to LWP 1269]
0x0000007fb7f896cc in __new_sem_wait_fast () from /lib//libpthread.so.0
(gdb) bt
#0 0x0000007fb7f896cc in __new_sem_wait_fast () from /lib//libpthread.so.0
#1 0x0000007fb7f898fc in sem_wait##GLIBC_2.17 () from /lib//libpthread.so.0
#2 0x0000000001b09000 in SemTake (SemId=0x0) at <compiled_source_code.c>
#3 0x0000000000d6cffc in ContextLock () at <compiled_source_code.c>
#4 0x0000000000d993e4 in CustMalloc (size=128) at <compiled_source_code.c>
#5 0x0000000001c88a2c in vasprintf (str=0x7fb5eaf5f8, fmt=0x7fb7e8a640 "usb%s", ap=...) at bsd-asprintf.c:61
#6 0x0000000001c88c50 in asprintf (str=0x7fb5eaf5f8, fmt=0x7fb7e8a640 "usb%s") at bsd-asprintf.c:120
#7 0x0000007fb7e853cc in linux_enumerate_device () from /usr/lib/libusb-1.0.so.0
#8 0x0000007fb7e854c4 in sysfs_scan_device () from /usr/lib/libusb-1.0.so.0
#9 0x0000007fb7e85b80 in op_init () from /usr/lib/libusb-1.0.so.0
#10 0x0000007fb7e7dd1c in libusb_init () from /usr/lib/libusb-1.0.so.0
#11 0x0000007fb7ea65fc in cyusb_open(unsigned short, unsigned short) () from /usr/lib/libMRegAccess.so
#12 0x0000007fb7ea33f4 in InitDefaultUSBConn () from /usr/lib/libMRegAccess.so
#13 0x0000007fb7ea58e0 in openDefaultUSBDriver () from /usr/lib/libMRegAccess.so
#14 0x00000000010ddd94 in InitDrv () at <compiled_source_code.c>
#15 ... at <compiled_source_code.c>
#16 ... at <compiled_source_code.c>
#17 ... at <compiled_source_code.c>
#18 ... at <compiled_source_code.c>
#19 0x0000007fb7f80fd0 in start_thread () from /lib//libpthread.so.0
#20 0x0000007fb7d8cf60 in ?? () from /lib//libc.so.6
I'm unable to understand why the custom function CustMalloc() is getting called instead of the actual malloc()
This appears to be happening because you compiled and linked bsd-asprintf.c (with your macro redefinition) into your main executable.
You can tell that asprintf and CustMalloc are part of your binary, because their addresses are very different from other library routines (such as linux_enumerate_device or sem_wait).
If you want to know where asprintf is defined (which archive library or object file it comes from), relink your executable with -Wl,-y,asprintf flag, and the linker will tell you.
Related
I'm trying to trace my segmentation fault using gdb and I'm unable to find the exact line where the fault is happening.
(gdb) backtrace
#0 0x00110402 in __kernel_vsyscall ()
#1 0x007a5690 in raise () from /lib/libc.so.6
#2 0x007a6f91 in abort () from /lib/libc.so.6
#3 0x007dd9eb in __libc_message () from /lib/libc.so.6
#4 0x007e59aa in _int_free () from /lib/libc.so.6
#5 0x007e90f0 in free () from /lib/libc.so.6
#6 0x080dc4e7 in CRYPTO_free ()
#7 0x08c36668 in ?? ()
#8 0x08c44bac in ?? ()
#9 0x08100168 in BN_free ()
#10 0x00000009 in ?? ()
#11 0x08c44ba8 in ?? ()
#12 0x08108c07 in BN_MONT_CTX_free ()
#13 0xffffffff in ?? ()
#14 0x08c36630 in ?? ()
#15 0x08112697 in RSA_eay_finish ()
#16 0x08c4c110 in ?? ()
#17 0x08c36630 in ?? ()
#18 0x081150af in RSA_free ()
#19 0xffffffff in ?? ()
#20 0x00000009 in ?? ()
#21 0x0821870d in ?? ()
#22 0x000000dd in ?? ()
#23 0x08c4c110 in ?? ()
#24 0x08c35e98 in ?? ()
#25 0x08136893 in EVP_PKEY_free ()
#26 0xffffffff in ?? ()
#27 0x0000000a in ?? ()
#28 0x08226017 in ?? ()
#29 0x00000189 in ?? ()
#30 0x007e90f0 in free () from /lib/libc.so.6
#31 0x00000000 in ?? ()
(gdb)
How do I get rid of the ?? () and get a more precise solution? Thank you.
First, getting the complete stack trace here will likely not help you: any crash inside free implementation is due to heap corruption. Here we have heap corruption that GLIBC has already detected and told you about on the console.
Knowing where the corrupted block is being freed usually doesn't help to find where the block was corrupted; use specialized tools like Valgrind or AddressSanitizer for that.
Second, you are not getting file/line info because the crash is happening inside libc.so.6, and you have not installed debuginfo symbols for it. How to install debuginfo depends on your Linux distribution, which you have not told us about.
Last, the reason you have an "apparently corrupt" stack with addresses that don't correspond to any symbols is likely that the calls are coming from hand-coded assembly code (from libopenssl.a), which doesn't use frame pointers and doesn't have correct unwind descriptors. GDB needs one or the other to produce correct stack trace.
Compile your project with -g -O0 flag. Without -g flag the gcc compiler will strip all the symbol out and that's why you cannot see any symbol. If you want debug 3rd party library then you should configure it with --with-debug or other debug option.
Yeah it looks like your stack is corrupted. The way I would approach this is to run the program under a memory profiler like valgrind. Watch out for double free, writing arrays out-of-bounds, and conditional jumps.
In a single EXE process, are TCL regular expressions shared by each interpreter instance returned by Tcl_CreateInterp? How could threads with 4 different interpreter instances (0x94fbcd8,0x94dff20,0x94c4170,0x94a8760) all be making a call like TclReFree (re=0x86b0444) at ./../generic/regfree.c:52?
This comment in the TCL manual hints that objects may be shared...
Tcl objects are allocated on the heap and are shared as much as possible to
reduce storage requirements. Reference counting is used to determine when an
object is no longer needed and can safely be freed.
Source: https://www.tcl.tk/man/tcl8.4/TclLib/Object.htm
We're encountering crashes in our 32-bit server application. We've isolated the root cause to a TCL regular expression shared between threads concurrently running in separate TCL interpreter instances.
The interpreters are failing on this line of TCL
regsub "\\*" $s "\\*" s
The application concurrently runs TCL 8.4.11 interpreter instances. Each interpreter is executing "user TCL scripts" in separate threads. The app creates threads that "own" 1 interpreter instance created using Tcl_CreateInterp. Each thread then tells the interpreter instance to run a "user TCL script" with Tcl_EvalObjv. The crash happens when each interpreter is configured to run the same "user TCL script" on the line containing the regsub shown above.
This app has been running in dozens of different production environments for over 15 years. In the current environment, the app is running on Red Hat Linux 6.5 64-bit.
The core dump looks like...
Program terminated with signal 11, Segmentation fault.
#0 0x0811c020 in miss ()
(gdb) bt
#0 0x0811c020 in miss ()
#1 0x0811b7ed in shortest ()
#2 0x0811a4fa in find ()
#3 0x0811a429 in TclReExec ()
#4 0x080fc83f in RegExpExecUniChar ()
#5 0x080fc970 in Tcl_RegExpExecObj ()
#6 0x080bb9f1 in Tcl_RegsubObjCmd ()
#7 0x080b027a in TclEvalObjvInternal ()
#8 0x080d2726 in TclExecuteByteCode ()
#9 0x080d1bd1 in TclCompEvalObj ()
#10 0x080fbd6c in TclObjInterpProc ()
#11 0x080b027a in TclEvalObjvInternal ()
#12 0x080d2726 in TclExecuteByteCode ()
#13 0x080d1bd1 in TclCompEvalObj ()
#14 0x080fbd6c in TclObjInterpProc ()
#15 0x080b027a in TclEvalObjvInternal ()
#16 0x080b0527 in Tcl_EvalObjv ()
After recompiling the app with a version of TCL with the compile flag --enable-symbols=mem and linked with D.U.M.A. - Detect Unintended Memory Access http://duma.sourceforge.net/ (a fork of Electric Fence to help catch buffer overruns), I'm getting a core dump like
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xea8eeb70 (LWP 31004)]
0x08151496 in TclReFree (re=0x86b0444) at ./../generic/regfree.c:52
52 (*((struct fns *)re->re_fns)->free)(re);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6_5.2.i686
(gdb) list
47 regfree(re)
48 regex_t *re;
49 {
50 if (re == NULL)
51 return;
52 (*((struct fns *)re->re_fns)->free)(re);
53 }
(gdb) bt
#0 0x08151496 in TclReFree (re=0x86b0444) at ./../generic/regfree.c:52
#1 0x08124360 in FreeRegexp (regexpPtr=0x86b0440) at ./../generic/tclRegexp.c:989
#2 0x08123ec2 in FreeRegexpInternalRep (objPtr=0xf64041b8) at ./../generic/tclRegexp.c:746
#3 0x08128cab in SetStringFromAny (interp=0x0, objPtr=0xf64041b8) at ./../generic/tclStringObj.c:1762
#4 0x08127894 in Tcl_GetUnicodeFromObj (objPtr=0xf64041b8, lengthPtr=0xea8ecee8) at ./../generic/tclStringObj.c:567
#5 0x080c3e9a in Tcl_RegsubObjCmd (dummy=0x0, interp=0x94fbcd8, objc=4, objv=0x94fbf28) at ./../generic/tclCmdMZ.c:718
#6 0x080b1386 in TclEvalObjvInternal (interp=0x94fbcd8, objc=5, objv=0x94fbf24, command=0x0, length=0, flags=0) at ./../generic/tclBasic.c:3088
#7 0x080e5a88 in TclExecuteByteCode (interp=0x94fbcd8, codePtr=0x95174e0) at ./../generic/tclExecute.c:1417
#8 0x080e4959 in TclCompEvalObj (interp=0x94fbcd8, objPtr=0x95097f0) at ./../generic/tclExecute.c:981
#9 0x08122a35 in TclObjInterpProc (clientData=0x9514520, interp=0x94fbcd8, objc=2, objv=0x94fbf1c) at ./../generic/tclProc.c:1100
#10 0x080b1386 in TclEvalObjvInternal (interp=0x94fbcd8, objc=2, objv=0x94fbf1c, command=0x0, length=0, flags=0) at ./../generic/tclBasic.c:3088
#11 0x080e5a88 in TclExecuteByteCode (interp=0x94fbcd8, codePtr=0xf64011f8) at ./../generic/tclExecute.c:1417
#12 0x080e4959 in TclCompEvalObj (interp=0x94fbcd8, objPtr=0x9513f68) at ./../generic/tclExecute.c:981
#13 0x08122a35 in TclObjInterpProc (clientData=0x9514d10, interp=0x94fbcd8, objc=2, objv=0xea8ee34c) at ./../generic/tclProc.c:1100
#14 0x080b1386 in TclEvalObjvInternal (interp=0x94fbcd8, objc=2, objv=0xea8ee34c, command=0x81a4ffe "", length=0, flags=0) at ./../generic/tclBasic.c:3088
#15 0x080b15e4 in Tcl_EvalObjv (interp=0x94fbcd8, objc=2, objv=0xea8ee34c, flags=0) at ./../generic/tclBasic.c:3204
#16 0x0808a812 in run_tcl_proc (pDevice=0x82405e0, pInterp=0x830d340, iNumArgs=2, objv=0xea8ee34c, bIsCommand=0 '\000', pCommand=0x0)
#17 0x08093492 in Tcl_begin_next_state (pDevice=0x82405e0, iNextState=RunPoll, pCommand=0x0)
#18 0x08093579 in Tcl_port_thread (dummy=0x8232c00)
#19 0x0014fb39 in start_thread () from /lib/libpthread.so.0
#20 0x00967d7e in clone () from /lib/libc.so.6
(gdb)
This gdb sessions also clearly shows concurrent threads executing regfree on the same regular expression, even though each thread's TCL interpreter instance is completely thread bound. There should be zero sharing between threads. The only thing they have in common is they are executing a "user TCL script" file with the same filename. The files were all loaded with Tcl_EvalFile into per-thread interpreter instances.
(gdb) info threads
45 Thread 0xe30e2b70 (LWP 31017) 0x00110430 in __kernel_vsyscall ()
--snip--
34 Thread 0xe9eedb70 (LWP 31005) 0x00110430 in __kernel_vsyscall ()
* 33 Thread 0xea8eeb70 (LWP 31004) 0x08151496 in TclReFree (re=0x86b0444) at ./../generic/regfree.c:52
32 Thread 0xeb2efb70 (LWP 31003) 0x08151496 in TclReFree (re=0x86b0444) at ./../generic/regfree.c:52
31 Thread 0xebcf0b70 (LWP 31002) 0x08151496 in TclReFree (re=0x86b0444) at ./../generic/regfree.c:52
30 Thread 0xec6f1b70 (LWP 31001) 0x08151496 in TclReFree (re=0x86b0444) at ./../generic/regfree.c:52
29 Thread 0xed0f2b70 (LWP 31000) 0x00110430 in __kernel_vsyscall ()
--snip--
1 Thread 0xf7fec8d0 (LWP 30970) 0x00110430 in __kernel_vsyscall ()
(gdb)
Note that this question is a completely separate crash from my previous question alloc: invalid block - Are Tcl_IncrRefCount and Tcl_DecrRefCount thread safe for threaded Tcl / 1 interp per thread?.
After digging through the app's code, I found a case where in thread A an interpreter is created and asked to run a proc but then in thread B used to run many other procs. I'm guessing this may be the root cause of this crash. Strangely, the app doesn't crash on Windows but crashes immediately (most of the time) on Linux. The app creates threads:
On Windows, using the Win32 API.
On Linux, using POSIX Threads / pthreads.
To answer your immediate question, REs are shared by two mechanisms. Firstly, they're bound to the internal representation of the Tcl_Obj values generated from the values in your script (e.g., the literals and the results of operations). Secondly, they're also stored in a size-bounded per-thread LRU cache.
Both of these mechanisms are strictly thread-bound. REs are not shared between threads; Tcl shares extremely little between threads.
However, there are a number of larger issues in your question.
If you're sending messages (err, scripts) between threads for execution, you're strongly recommended to use the Thread extension for this, as this takes care to copy things that need to be copied. The Thread extension ships with a full distribution of Tcl 8.6 (it's now a contributed package, along with [incr Tcl], SQLite and TDBC) but it should be available separately for older versions of Tcl.
Also, you're using a doubly-unsupported version of Tcl. The most recent version of 8.4 is 8.4.20 (which should be a drop-in replacement) and even that has been out of security/build support for several years now. You really are recommended to upgrade. 8.5.17 is the current long-term support release, and 8.6.3 is the current production release. (They're also quite a bit faster on a lot of code.)
Our 32-bit server application statically embeds tcl 8.4.11. On Red Hat Linux 6.5 64-bit we're encountering crashes / core dumps. The failure looks like
alloc: invalid block: 0xf6f00f58: 88 f6 0
At the bottom of the question, I've documented two different core dumps we've seen.
We've isolated a potential root cause to a TCL object shared between two threads concurrently running separate TCL interpreter instances. We think it's because TCL object is passed to Tcl_IncrRefCount / Tcl_DecrRefCount from these concurrently executing TCL interpreters.
Are Tcl_IncrRefCount / Tcl_DecrRefCount thread safe when TCL is compiled threaded?
Are TCL objects shared by TCL interpreter instances? Is there any way to disable TCL object sharing across interpreter instances?
Is the situation any better in TCL version 8.6.3?
(gdb) bt
#0 __kernel_vsyscall () at arch/x86/vdso/vdso32/sysenter.S:49
#1 0x001b7871 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x001b914a in abort () at abort.c:92
#3 0x080f611c in Tcl_PanicVA ()
#4 0x080f613b in Tcl_Panic ()
#5 0x0810133c in Ptr2Block ()
#6 0x08100e04 in TclpFree ()
#7 0x080b46a7 in Tcl_Free ()
#8 0x08100686 in FreeStringInternalRep ()
#9 0x080fdac1 in ResetObjResult ()
#10 0x080fd316 in Tcl_GetStringResult ()
#11 0x0808aaad in run_tcl_proc (pDevice=0x8e0ba08, pInterp=0x8d798c0, iNumArgs=2, objv=0x115434c, bIsCommand=0 '\000', pCommand=0x0)
#12 0x08093672 in Tcl_begin_next_state (pDevice=0x8e0ba08, iNextState=RunPoll, pCommand=0x0)
#13 0x08093759 in Tcl_port_thread (dummy=0x8d1cab8)
#14 0x008bcb39 in start_thread (arg=0x1154b70) at pthread_create.c:301
#15 0x0026fc2e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:133
(gdb)
(gdb) bt
#0 __kernel_vsyscall () at arch/x86/vdso/vdso32/sysenter.S:49
#1 0x00395871 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x0039714a in abort () at abort.c:92
#3 0x080f611c in Tcl_PanicVA ()
#4 0x080f613b in Tcl_Panic ()
#5 0x0810133c in Ptr2Block ()
#6 0x08100e04 in TclpFree ()
#7 0x080b46a7 in Tcl_Free ()
#8 0x080d21b6 in TclExecuteByteCode ()
#9 0x080d1bc1 in TclCompEvalObj ()
#10 0x080fbd5c in TclObjInterpProc ()
#11 0x080b026a in TclEvalObjvInternal ()
#12 0x080d2716 in TclExecuteByteCode ()
#13 0x080d1bc1 in TclCompEvalObj ()
#14 0x080fbd5c in TclObjInterpProc ()
#15 0x080b026a in TclEvalObjvInternal ()
#16 0x080b0517 in Tcl_EvalObjv ()
#17 0x0808aa02 in run_tcl_proc (pDevice=0x94a2500, pInterp=0xac2bba0, iNumArgs=2, objv=0x11b034c, bIsCommand=0 '\000', pCommand=0x0)
#18 0x08093672 in Tcl_begin_next_state (pDevice=0x94a2500, iNextState=RunPoll, pCommand=0x0)
#19 0x08093759 in Tcl_port_thread (dummy=0x9365e98)
#20 0x00356b39 in start_thread (arg=0x11b0b70) at pthread_create.c:301
#21 0x0044dc2e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:133
(gdb)
The calls Tcl_IncrRefCount (actually a simple macro) and Tcl_DecrRefCount (a complicated macro) are sort of thread safe, but only because each Tcl_Obj should only ever be accessed from the thread that created it; parallel calls to T_IRC and T_DRC are fine, so long as they're on different values. The plus side of this is that accesses don't need locking (and the memory manager for Tcl_Obj structures takes advantage of this).
Note that multi-threaded access is not a good plan at all unless you're very careful, since even reader operations like Tcl_GetIntFromObj can write to the underlying structure if a type transformation needs to be applied. These operations are not locked. Doing it at all needs very intimate knowledge of the current type of the value — not something that you're usually encouraged to think about in Tcl in the first place, though tcl::unsupported::representation can be helpful with probing this in 8.6 — and some very careful interlocking between the threads so that one isn't writing while the other is peeking. Don't do this at all, while not 100% accurate, is the approach least likely to lead to headaches.
You probably ought to read more about how you're supposed to do it. The ActiveState blog has a reasonable introduction.
I have a small C program that loads a custom DLL and uses a couple of functions. I can run the program from the console and it works as intended. (I'm compiling with MinGW on Windows XP)
But if I run it from gdb, when it gets to loading the DLL, I get:
56 ldll = LoadLibrary("gsp810.dll");
(gdb) n
Program received signal SIGSEGV, Segmentation fault.
0x7c929af2 in ntdll!RtlpWaitForCriticalSection () from C:\WINDOWS\system32\ntdll.dll
The weird thing is, if I make a backtrace at this point, I get a strange stack of Windows functions, which doesn't even contain my own program's stack (see below). However, if I keep running, it'll eventually return to my main() function and everything seems to be back to normal. The program works as expected and the functions from the DLL can be called.
(gdb) backtrace
#0 0x7c929af2 in ntdll!RtlpWaitForCriticalSection () from C:\WINDOWS\system32\ntdll.dll
#1 0x7c911046 in ntdll!RtlEnterCriticalSection () from C:\WINDOWS\system32\ntdll.dll
#2 0x00e161a0 in ?? ()
#3 0x77da6cf8 in RegCloseKey () from C:\WINDOWS\system32\advapi32.dll
#4 0x77da78e4 in RegOpenKeyExA () from C:\WINDOWS\system32\advapi32.dll
#5 0x77f44fcd in SHLWAPI!PathMakeSystemFolderW () from C:\WINDOWS\system32\shlwapi.dll
#6 0x77f452e8 in SHLWAPI!PathMakeSystemFolderW () from C:\WINDOWS\system32\shlwapi.dll
#7 0x77f45252 in SHLWAPI!PathMakeSystemFolderW () from C:\WINDOWS\system32\shlwapi.dll
#8 0x7c91118a in ntdll!LdrInitializeThunk () from C:\WINDOWS\system32\ntdll.dll
#9 0x77f40000 in ?? ()
#10 0x7c92b5d2 in ntdll!LdrFindResourceDirectory_U () from C:\WINDOWS\system32\ntdll.dll
#11 0x7c9262db in ntdll!RtlValidateUnicodeString () from C:\WINDOWS\system32\ntdll.dll
#12 0x7c92643d in ntdll!LdrLoadDll () from C:\WINDOWS\system32\ntdll.dll
#13 0x00000000 in ?? ()
Is this SEGFAULT normal, or it is indicating an underlying problem with the DLL?
EDIT: Ok, looks like the problem is in the DLL itself. What I don't understand is the backtrace gdb is showing, as it does not contain the functions in my application. Then, at a certain point, it somehow "switches" to my stack, and the program keeps running as if nothing had happened.
Is it possible that Windows is somehow "handling" the segmentation fault, and the it returns control to the application?
I am analyzing this core dump
Program received signal SIGABRT, Aborted.
0xb7fff424 in __kernel_vsyscall ()
(gdb) where
#0 0xb7fff424 in __kernel_vsyscall ()
#1 0x0050cd71 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x0050e64a in abort () at abort.c:92
#3 0x08083b3b in ?? ()
#4 0x08095461 in ?? ()
#5 0x0808bdea in ?? ()
#6 0x0808c4e2 in ?? ()
#7 0x080b683b in ?? ()
#8 0x0805d845 in ?? ()
#9 0x08083eb6 in ?? ()
#10 0x08061402 in ?? ()
#11 0x004f8cc6 in __libc_start_main (main=0x805f390, argc=15, ubp_av=0xbfffef64, init=0x825e220, fini=0x825e210,
rtld_fini=0x4cb220 <_dl_fini>, stack_end=0xbfffef5c) at libc-start.c:226
#12 0x0804e5d1 in ?? ()
I'm not able to know which function ?? maps to OR for instance #10 0x08061402 in ?? ()
falls in which address range ...
Please help me debug this.
Your program has no debugging symbols. Recompile it with -g. Make sure you haven't stripped your executable, e.g. by passing -s to the linker.
Even though #user794080 didn't say so, it appears exceedingly likely that his program is a 32-bit linux executable.
There are two possible reasons (I can think of) for symbols from main executable (and all symbols in the stack trace in the range [0x08040000,0x08100000) are from the main executable) not to show up.
The main executable has in fact been stripped (this is the same as
ninjalj's answer), and often happens when '-s' is passed into the linker, perhaps inadvertently.
The executable has been compiled with a new(er) GCC, but is being debugged by an old(er) GDB, which chokes on some newer dwarf construct (there should be a warning from GDB about that).
To know what libraries are mapped into the application, record a pid of you program, stopped in gdb and run in other console
cat /proc/$pid/maps
wher $pid is the pid of stopped process. Format of the maps file is described at http://linux.die.net/man/5/proc - starting from "/proc/[number]/maps
A file containing the currently mapped memory regions and their access permissions."
Also, if your OS don't use a ASLR (address space layout randomization) or it is disabled for your program, you can use
ldd ./program
to list linked libraries and their memory ranges. But if ASLR is turned on, you will be not able to get real memory mapping ranges info, as it will change for each run of program. But even then you will know, what libraries are linked in dynamically and install a debuginfo for them.
The stack might be corrupted. The "??" can happen if the return address on the stack has been overwritten by, for example, a buffer overflow.