So this is a follow up to my previous Question
I am trying to build a legacy codebase with Address Sanitizer. As soon as I run the binary, I get a Segmentation Fault.
==2940167==ERROR: AddressSanitizer: global-buffer-overflow on address 0x000002aa79a0 at pc 0x000000d6aaf4 bp 0x7fffe7244140 sp 0x7fffe7244130
READ of size 4 at 0x000002aa79a0 thread T3
Thread 1 "block_hk.xcg." received signal SIGSEGV, Segmentation fault.
Here is the trace with gdb:
#0 0x00007ffff6f0dc26 in __sanitizer::internal_strlen(char const*) () from /lib64/libasan.so.5
#1 0x00007ffff6e57208 in printf_common(void*, char const*, __va_list_tag*) () from /lib64/libasan.so.5
#2 0x00007ffff6e59566 in vsnprintf () from /lib64/libasan.so.5
#3 0x00007ffff540ed5c in ?? () from /lib64/libdus.so
#4 0x00007ffff540f017 in ?? () from /lib64/libdus.so
#5 0x00007ffff540f860 in ?? () from /lib64/libdus.so
#6 0x00007ffff541a443 in logPrint () from /lib64/libdus.so
#7 0x00007ffff4761734 in mcsCreateSem () from /lib64/libcwl.so
#8 0x00007ffff4764b14 in mcsInitialize () from /lib64/libcwl.so
#9 0x00007ffff47678fc in cwlAppRunCPP () from /lib64/libcwl.so
#10 0x000000000112d719 in main (argc=1, argv=0x7ffffffe7478) at ./src/main.c:755
Related
get the encode_audio.c example from ffmpeg, which converts in-memory raw PCM data to MP2
compile
gcc -o encode_audio encode_audio.c -I/usr/include/x86_64-linux-gnu/libavcodec -lavformat -lavcodec -lavutil -lm
check that is works with the MP2 encoder -> OK
change the encoder to codec = avcodec_find_encoder(AV_CODEC_ID_ADPCM_G722);
adapt the output parameters to g722
c->sample_rate = 16000;//select_sample_rate(codec);
c->channel_layout = AV_CH_LAYOUT_MONO;//select_channel_layout(codec);
c->channels = 1; //av_get_channel_layout_nb_channels(c->channel_layout);
recompile, test... crash!
#0 0x00007ffff5cd5e97 in __GI_raise (sig=sig#entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007ffff5cd7801 in __GI_abort () at abort.c:79
#2 0x00007ffff5d20897 in __libc_message (action=action#entry=do_abort, fmt=fmt#entry=0x7ffff5e4db9a "%s\n")
at ../sysdeps/posix/libc_fatal.c:181
#3 0x00007ffff5d2790a in malloc_printerr (str=str#entry=0x7ffff5e4f800 "free(): invalid next size (fast)")
at malloc.c:5350
#4 0x00007ffff5d2a078 in _int_free (have_lock=<optimized out>, p=<optimized out>, av=<optimized out>)
at malloc.c:4213
#5 0x00007ffff5d2a078 in __GI___libc_free (mem=<optimized out>) at malloc.c:3124
#6 0x00007ffff5d2a078 in tcache_thread_shutdown () at malloc.c:2969
#7 0x00007ffff5d2a078 in arena_thread_freeres () at arena.c:950
#8 0x00007ffff5d2c55a in _int_memalign (av=0x7ffff6082c40 <main_arena>, alignment=32, bytes=<optimized out>)
at malloc.c:4750
#9 0x00007ffff5d31e43 in _mid_memalign (address=<optimized out>, bytes=40, alignment=<optimized out>)
at malloc.c:3305
#10 0x00007ffff5d31e43 in __posix_memalign (memptr=0x7fffffffd720, alignment=<optimized out>, size=40)
at malloc.c:5369
#11 0x00007ffff6457663 in av_malloc () at /usr/lib/x86_64-linux-gnu/libavutil.so.55
#12 0x00007ffff645798d in av_mallocz () at /usr/lib/x86_64-linux-gnu/libavutil.so.55
#13 0x00007ffff643b89d in av_buffer_create () at /usr/lib/x86_64-linux-gnu/libavutil.so.55
#14 0x00007ffff643be40 in av_buffer_realloc () at /usr/lib/x86_64-linux-gnu/libavutil.so.55
#15 0x00007ffff6848efa in av_packet_ref () at /usr/lib/x86_64-linux-gnu/libavcodec.so.57
#16 0x00007ffff68f8474 in avcodec_encode_audio2 () at /usr/lib/x86_64-linux-gnu/libavcodec.so.57
#17 0x00007ffff68f8af9 in () at /usr/lib/x86_64-linux-gnu/libavcodec.so.57
#18 0x00007ffff68f8c5a in avcodec_send_frame () at /usr/lib/x86_64-linux-gnu/libavcodec.so.57
#19 0x0000555555555085 in encode ()
#20 0x00005555555555a2 in main ()
Hmm... what am I missing?
Solution:
The code encoding a single ton sound is explicitly written for stereo channel layout. Reducing to one channel avoid the buffer overflow.
samples = (uint16_t*)frame->data[0];
for (j = 0; j < c->frame_size; j++)
{
samples[j] = (int)(sin(t) * 10000);
for (k = 1; k < c->channels; k++)
samples[j + k] = samples[j];
t += tincr;
}
I'm trying to run efence on my code, and it always cores here:
Electric Fence 2.1 Copyright (C) 1987-1998 Bruce Perens.
Program received signal SIGSEGV, Segmentation fault.
memalign (alignment=4, userSize=28) at ../utils/libefence/efence.c:492
492 ../utils/libefence/efence.c: No such file or directory.
in ../utils/libefence/efence.c
(gdb) bt
#0 memalign (alignment=4, userSize=28) at ../utils/libefence/efence.c:492
#1 0xf7ff928c in malloc (size=27) at ../utils/libefence/efence.c:816
#2 0x41c92c67 in operator new(unsigned int) () from /usr/lib/libstdc++.so.6
#3 0x41c78204 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) () from /usr/lib/libstdc++.so.6
#4 0x41c7a468 in char* std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) () from /usr/lib/libstdc++.so.6
#5 0x41c7a5d6 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) ()
from /usr/lib/libstdc++.so.6
#6 0xefb12078 in __static_initialization_and_destruction_0 ()
at ../include/isan/objstoredefs/core/Parameters.h:125
#7 _GLOBAL__sub_I_RecurrWindowPBI.cc(void) ()
at ../dme/svc/common/src/gen/ifc/beh/./imp/trig/RecurrWindowPBI.cc:77
#8 0xefbc9dfd in __do_global_ctors_aux ()
from /isan/lib/libsvc_ifc_behcommon.so
#9 0xefaf59b5 in _init () from /isan/lib/libsvc_ifc_behcommon.so
#10 0x419fd486 in __ctype_init () from /lib/libc.so.6
#11 0x4100ed39 in ?? () from /lib/ld-linux.so.2
#12 0x4100ee8f in ?? () from /lib/ld-linux.so.2
#13 0x410011ef in ?? () from /lib/ld-linux.so.2
(gdb) frame 2
#2 0x41c92c67 in operator new(unsigned int) () from /usr/lib/libstdc++.so.6
I try to run the program using GDB with environment set to LD_PRELOAD for the efence lib.
The fault seems to be here:
491 for ( slot = allocationList, count = slotCount ; count > 0; count-- ) {
492 if ( slot->mode == FREE
493 && slot->internalSize >= internalSize ) {
494 if ( !fullSlot
495 ||slot->internalSize < fullSlot->internalSize){
496 fullSlot = slot;
497 if ( slot->internalSize == internalSize
498 && emptySlots[0] )
499 break; /* All done, */
500 }
501 }
502 else if ( slot->mode == NOT_IN_USE ) {
503 if ( !emptySlots[0] )
504 emptySlots[0] = slot;
505 else if ( !emptySlots[1] )
506 emptySlots[1] = slot;
507 else if ( fullSlot
508 && fullSlot->internalSize == internalSize )
509 break; /* All done. */
510 }
511 slot++;
512 }
But, in GDB, I'm able to dup the slot structure without any issues:
(gdb) p slot
$1 = (Slot *) 0xef846000
(gdb) p slot->mode
$2 = NOT_IN_USE
(gdb)
(gdb) x/10i $eip
=> 0xf7ff9590 <memalign+448>: mov 0x10(%edi),%edx
0xf7ff9593 <memalign+451>: cmp $0x1,%edx
0xf7ff9596 <memalign+454>: jne 0xf7ff95c0 <memalign+496>
0xf7ff9598 <memalign+456>: mov 0xc(%edi),%edx
0xf7ff959b <memalign+459>: cmp %edx,%esi
0xf7ff959d <memalign+461>: ja 0xf7ff95f8 <memalign+552>
0xf7ff959f <memalign+463>: test %ecx,%ecx
0xf7ff95a1 <memalign+465>: je 0xf7ff95a8 <memalign+472>
0xf7ff95a3 <memalign+467>: cmp 0xc(%ecx),%edx
0xf7ff95a6 <memalign+470>: jae 0xf7ff95f8 <memalign+552>
(gdb) p/x $edi
$5 = 0xef846000
(gdb) ptype (*slot)
type = struct _Slot {
void *userAddress;
void *internalAddress;
size_t userSize;
size_t internalSize;
Mode mode;
}
(gdb) p allocationList
$6 = (Slot *) 0xef846000
(gdb) p allocationList[0]
$7 = {userAddress = 0x0, internalAddress = 0x0, userSize = 0,
internalSize = 0, mode = NOT_IN_USE}
Any help?
The efence code is as in:
http://linux.softpedia.com/get/Programming/Debuggers/Electric-Fence-3305.shtml
Thanks!!
So the suspect slot at 0xef846000 SEGV's but is not NULL. The explanation is that with efence the neighboring segments are read/write protected, causing a SIGBUS. But I guess on linux this BUS turns out to be a SEGV instead.
Within the GDB repl those segment read protection bits are disabled/overruled.
So the remaining question is in what memory segment allocationList[0] is located? I guess it's in a fence without read bits. How could that happen? No idea.
My application is receiving the segmentation fault. The back trace log -
Program received signal SIGSEV, Segmentation fault.
0x00000000004a5c03 in engine_unlocked_finish ()
(gdb) bt
#0 0x00000000004a5c03 in engine_unlocked_finish ()
#1 0x00000000004a5d71 in ENGINE_finish ()
#2 0x000000000046a537 in EVP_PKEY_free_it ()
#3 0x000000000046a91b in EVP_PKEY_free ()
#4 0x00000000004b231a in pubkey_cb ()
#5 0x0000000000470c97 in asn1_item_combine_free ()
#6 0x0000000000750f70 in X509_CINF_seq_tt ()
#7 0x00000000010f7d90 in ?? ()
#8 0x00000000010f7cf0 in ?? ()
#9 0x0000000000000000 in ?? ()
The stackframe at #9 is interesting. It's address is 0x0000000000000000. Does this mean stack got corrupted even before getting to engine_unlocked_finish () ?
The stackframe at #9 is interesting.
Not really. What's most likely happening is that X509_CINF_seq_tt is hand-coded assembly, and lacks correct unwind descriptors, so everything after it in the stack trace is just bogus.
In fact, looking at this source, X509_CINF_seq_tt is not even a function, so it's probably asn1_item_combine_free that starts the "bad unwind".
I am using RSA_sign() to create dkim signatures. Sometimes , extremely unpredictably the code just crashes
I get a gdb dump like this
Thread 39 (Thread 0x41401940 (LWP 31921)):
#0 0x0000003b9dacc3bb in BN_div () from /lib64/libcrypto.so.6
#1 0x0000003b9daceb40 in BN_mod_inverse () from /lib64/libcrypto.so.6
#2 0x0000003b9dacb609 in BN_BLINDING_create_param () from /lib64/libcrypto.so.6
#3 0x0000003b9dadc9f7 in RSA_setup_blinding () from /lib64/libcrypto.so.6
#4 0x0000003b9daee954 in ?? () from /lib64/libcrypto.so.6
#5 0x0000003b9daef56b in ?? () from /lib64/libcrypto.so.6
#6 0x0000003b9da6e965 in RSA_sign () from /lib64/libcrypto.so.6
#7 0x0000000000403e7f in dkim_create (headers=0x2aaaac001840, headerc=7,
......., v=0) at firm-dkim.c:145
The firm-dkim.c is available here
http://code.google.com/p/firm-dkim/source/browse/trunk/firm-dkim.c
How can I debug this further ?
Thanks
Ram
Ok I think I got the error
The code over here
firm-dkim.c.
Does not allocate any memory for RSA *rsa_private , line 48
And this unallocated memory area is used in RSA_sign() and RSA_free()
I think that must be causing the segfault. I have alloc'ed memory now and I am running the daemon in production. Hopefully no more segfaults now.
Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x18d1c070
0x96350250 in strlen ()
(gdb) bt
#0 0x96350250 in strlen ()
#1 0x963574d1 in strdup ()
#2 0x9636e420 in asl_set_query ()
#3 0x9636e2d9 in asl_set ()
#4 0x9636d5d2 in vsyslog ()
#5 0x9636d3e1 in syslog ()
#6 0x23405e78 in gp_log (level=4, fmt=0x23429b68 "%-s:%4d: size: %d\n") at ../../rpc/mac/gp_lib.c:49
#7 0x23404c61 in rpc_encap (out=0xbfffb23c, args=0xbfffd2d0) at ../../rpc/rpc.c:178
#8 0x23405269 in rpc_encap_args (pkt_out=0xbfffb20c, pd=0x2342e460, args=0xbfffd2d0) at ../../rpc/rpc.c:120
#9 0x23405549 in rpc_call_common (c=0x23053048, pd=0x2342e460, args=0xbfffd2d0, timeout=0, pkt_in=0xbfffd30c, size_in=0xbfffd308) at ../../rpc/rpc.c:227
#10 0x234057f9 in rpc_call_actor (pd=0x2342e460, args=0xbfffd2d0, timeout=0, pkt_in=0xbfffd30c, size_in=0xbfffd308) at ../../rpc/rpc.c:204
#11 0x23402839 in MobileIPAPIStopRoaming_w () at ../../MIPSDKv4/MobileIPSDK/MobileIP4_w.c:229
#12 0x23401420 in MobileIPAPIStopRoaming (mipHandle=1) at ../../MIPSDKv3/MobileIPSDKv3.c:300
#13 0x21c0e1a6 in GP_ICM_CONNMGR::CMobileIpSdk::MobileIPAPIStopRoaming ()
#14 0x21c0e6cf in GP_ICM_CONNMGR::CMobileIpSdk::Uninit ()
#15 0x1e749c19 in GP_ICM_RULEMGR::CRulesManager::Uninit ()
#16 0x0004df18 in CConnectionManager::UninitConnectionManager ()
#17 0x0013e6b4 in ICMApplication::UninitICMApplication ()
#18 0x001495b2 in main ()
(gdb) info registers
eax 0xffffffff -1
ecx 0x18d1c07c 416399484
edx 0x18d1c070 416399472
ebx 0x9636e2f0 -1774787856
esp 0xbfffaa4c 0xbfffaa4c
ebp 0xbfffaa78 0xbfffaa78
esi 0x964c6cc0 -1773376320
edi 0x18d1c07c 416399484
eip 0x96350250 0x96350250 <strlen+16>
eflags 0x10286 66182
cs 0x17 23
ss 0x1f 31
ds 0x1f 31
es 0x1f 31
fs 0x0 0
gs 0x37 55
(gdb) frame 1
#1 0x963574d1 in strdup ()
(gdb) frame 2
#2 0x9636e420 in asl_set_query ()
(gdb) frame 3
#3 0x9636e2d9 in asl_set ()
(gdb) frame 4
#4 0x9636d5d2 in vsyslog ()
(gdb) frame 5
#5 0x9636d3e1 in syslog ()
(gdb) frame 6
#6 0x23405e78 in gp_log (level=4, fmt=0x23429b68 "%-s:%4d: size: %d\n") at ../../rpc/mac/gp_lib.c:49
49 syslog(5, "%s", buf);
(gdb) p buf
$5 = "rpc_encap: 178: size: 48\n", '\0' <repeats 998 times>
(gdb) p level
$6 = 4
(gdb) p fmt
$7 = 0x23429b68 "%-s:%4d: size: %d\n"
The gp_log function is this:
#define LOG_DEBUG(format, ...) gp_log(DL_DEBUG, "%-s:%4d: " format, __FUNCTION__, __LINE__, ##__VA_ARGS__)
#define LOG_MAX_LENGTH 1024
void gp_log(int level, const char *fmt, ...)
{
va_list ap;
char buf[LOG_MAX_LENGTH] = {0};
if (level > log_level)
return;
va_start(ap, fmt);
vsnprintf(buf, LOG_MAX_LENGTH, fmt, ap);
va_end(ap);
syslog(level, "%s", buf);
//printf("%s", buf);
}
This statement in rpc_encap print out the contents of an integer, which lead to this crash.
LOG_DEBUG("size: %d\n", size);
The crash is 100% reproduceable. I examine the parameters to syslog() every time before
crash:
buf is always "rpc_encap: 178: size: 48\n".
level is always 4.
If I use printf instead of syslog, the crash disappeared.
In order to build a universal library for Mac 10.5/6/7(Intel arch), I add these flags when compiling on 10.6 or 10.7,
CFLAGS = -force_cpusubtype_ALL -mmacosx-version-min=10.5 -arch i386
and crash happened.
Now I remove these flags and build on 10.5, no more crash(also the library works on 10.5/6/7).