I'm a newbie in Linux driver Field.Now, I has just done with writing Linux driver and I'm testing the accuracy of the function of this driver. The goal of my driver is using FPGA card with CPU ARMv7 through PCIe communication and doing both encryption and decryption in this card. When I test my kernel module, I see this kernel panic. In my test case, I send multiple packets continuously to FPGA card to implement encryption/decryption. However, after doing several packets well, Linux kernel was crash. At first, I think that this kernel crash bug is related to allocating kernel memory and freeing this memory (kzalloc function and kfree function). May be memory allocated for some pointer does not be freed immediately. Can anyone suggests the cause and the solution for this kernel panic?
[ 532.593938] Unable to handle kernel NULL pointer dereference at virtual address 00000001
[ 532.602069] pgd = ecb8c000
[ 532.604780] [00000001] *pgd=2ca83831, *pte=00000000, *ppte=00000000
[532.611066] Internal error: Oops: 17 [#1] SMP ARM
[ 532.615777] Modules linked in: testcrypto(+) huy_crypto xdma ath9k ath9k_common pppoe ppp_async ath9k_hw ath10k_pci ath10k_core ath pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_policy xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache iptable_raw iptable_mangle iptable_filter ipt_ah ipt_ECN ip_tables crc_ccitt compat sch_cake
[ 532.686456] act_connmark nf_conntrack sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred cryptodev nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb ip6_vti ip_vti xfrm6_mode_tunnel xfrm6_mode_transport xfrm6_mode_beet ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel xfrm4_mode_tunnel xfrm4_mode_transport xfrm4_mode_beet ipcomp esp4 ah4 ip6_tunnel tunnel6 tunnel4 ip_tunnel mpls_iptunnel mpls_router mpls_gso xfrm_user xfrm_ipcomp af_key xfrm_algo algif_skcipher algif_hash af_alg sha512_generic md5 echainiv cbc authenc gpio_button_hotplug [last unloaded: testcrypto]
[ 532.748684] CPU: 0 PID: 5563 Comm: insmod Not tainted 4.14.176 #0
[ 532.754789] Hardware name: Marvell Armada 380/385 (Device Tree)
[ 532.760721] task: ef3bde00 task.stack: ef182000
[ 532.765263] PC is at __kmalloc_track_caller+0x100/0x144
[ 532.770499] LR is at 0x89a5
[ 532.773297] pc : [<c01ee8b4>] lr : [<000089a5>] psr: 20000013
[ 532.779576] sp : ef183d98 ip : a0000013 fp : ffffee4b
[ 532.784811] r10: 00000009 r9 : 00008124 r8 : 00000002
[ 532.790046] r7 : 00000001 r6 : ecfe2a40 r5 : 014000c0 r4 : ef001e40
[ 532.796586] r3 : 00000000 r2 : ef7d6a34 r1 : 2ee8d000 r0 : 000089a6
[ 532.803128] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
[ 532.810278] Control: 10c5387d Table: 2cb8c04a DAC: 00000051
[ 532.816035] Process insmod (pid: 5563, stack limit = 0xef182210)
[ 532.822054] Stack: (0xef183d98 to 0xef184000)
[ 532.826419] 3d80: 00000008 c024eb40
[ 532.834616] 3da0: ecfe24c0 00000124 00000002 c01c73b8 ec94b540 ef0181c0 ec94b540 c024eb40
[ 532.842812] 3dc0: 00000000 00000124 bf6780c8 ec94b540 c07088c8 ec94b540 00000124 bf6780c8
[ 532.851008] 3de0: 00000000 c024fcc8 00000000 c07088c8 ec94b540 c025173c 00000000 edfed95c
[ 532.859204] 3e00: ec94b540 c025207c 00001000 00000000 c07088c8 edfed95c 00000000 00000000
[ 532.867401] 3e20: edfed800 edfeda58 ec94b540 c0252a5c 00000000 00000000 c07e21ec c07e21d8
[ 532.875597] 3e40: 00000008 ef183f40 edfed800 edfeda34 edfeda70 0000002c 014000c0 bf678080
[ 532.883793] 3e60: c0a03c48 c0193a30 bf67808c 00007fff bf678080 c0191330 c0a5dce8 bf6780c8
[ 532.891990] 3e80: c0190a50 bf67808c c0702ce4 bf678170 c082dfd8 c07da244 c07da3a0 c0a03c48
[ 532.900186] 3ea0: c07da250 f1523fff ffe00000 ef0e8600 fffff000 c0a5dcc0 014002c0 00000011
[ 532.908382] 3ec0: 00000000 00000000 00000000 00000000 00000000 00000000 6e72656b 00006c65
[ 532.916578] 3ee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 532.924774] 3f00: 00000000 00000000 00000000 00000000 00000000 c9929c40 00000080 00000fe4
[ 532.932970] 3f20: 00000000 00642ff4 f1523fe4 ffffe000 b6f74230 00000051 00000000 c0194110
[ 532.941166] 3f40: f1515bcc f1513000 00010fe4 f1523904 f1523748 f151f760 00004000 00004080
[ 532.949362] 3f60: 00000000 00000000 00000000 000036d0 00000029 0000002a 00000017 00000000
[ 532.957558] 3f80: 00000012 00000000 00000000 00000000 00000003 00000080 c01077a4 ef182000
[ 532.965755] 3fa0: 00000080 c01075a0 00000000 00000000 00632010 00010fe4 b6f74230 00000700
[ 532.973951] 3fc0: 00000000 00000000 00000003 00000080 00010fe4 00000000 00000020 00000000
[ 532.982148] 3fe0: beca6d1c beca6d00 00011d50 b6f2abac 60000010 00632010 00000000 00000000
[ 532.990350] [<c01ee8b4>] (__kmalloc_track_caller) from [<c01c73b8>] (kstrdup+0x30/0x54)
[ 532.998378] [<c01c73b8>] (kstrdup) from [<c024eb40>] (__kernfs_new_node+0x28/0x130)
[ 533.006055] [<c024eb40>] (__kernfs_new_node) from [<c024fcc8>] (kernfs_new_node+0x1c/0x38)
[ 533.014340] [<c024fcc8>] (kernfs_new_node) from [<c025173c>] (__kernfs_create_file+0x18/0xa4)
[ 533.022885] [<c025173c>] (__kernfs_create_file) from [<c025207c>] (sysfs_add_file_mode_ns+0x13c/0x194)
[ 533.032213] [<c025207c>] (sysfs_add_file_mode_ns) from [<c0252a5c>] (internal_create_group+0x194/0x2e8)
[ 533.041630] [<c0252a5c>] (internal_create_group) from [<c0193a30>] (load_module+0x1b74/0x2118)
[ 533.050262] [<c0193a30>] (load_module) from [<c0194110>] (SyS_init_module+0x13c/0x174)
[ 533.058201] [<c0194110>] (SyS_init_module) from [<c01075a0>] (ret_fast_syscall+0x0/0x54)
[ 533.066312] Code: ea00000f e121f00c eaffffd2 e5943014 (e7973003)
[ 533.072445] ---[ end trace bb93ca4b64a48f93 ]---
[ 533.079218] Kernel panic - not syncing: Fatal exception
[ 533.084457] CPU1: stopping
[ 533.087173] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D 4.14.176 #0
[ 533.094497] Hardware name: Marvell Armada 380/385 (Device Tree)
[ 533.100436] [<c010ecf8>] (unwind_backtrace) from [<c010a9b0>] (show_stack+0x10/0x14)
[ 533.108202] [<c010a9b0>] (show_stack) from [<c0636974>] (dump_stack+0x94/0xa8)
[ 533.115442] [<c0636974>] (dump_stack) from [<c010db38>] (handle_IPI+0xe4/0x190)
[ 533.122769] [<c010db38>] (handle_IPI) from [<c0101494>] (gic_handle_irq+0x8c/0x90)
[ 533.130357] [<c0101494>] (gic_handle_irq) from [<c010b64c>] (__irq_svc+0x6c/0x90)
[ 533.137855] Exception stack(0xef067f80 to 0xef067fc8)
[ 533.142919] 7f80: 00000001 00000000 00000000 c01145a0 ffffe000 c0a03cb8 c0a03c6c 00000000
[ 533.151115] 7fa0: 00000000 414fc091 00000000 00000000 ef067fc8 ef067fd0 c0107f68 c0107f6c
[ 533.159309] 7fc0: 60000013 ffffffff
[ 533.162807] [<c010b64c>] (__irq_svc) from [<c0107f6c>] (arch_cpu_idle+0x34/0x38)
[ 533.170224] [<c0107f6c>] (arch_cpu_idle) from [<c015f6d4>] (do_idle+0xdc/0x19c)
[ 533.177551] [<c015f6d4>] (do_idle) from [<c015f9f0>] (cpu_startup_entry+0x18/0x1c)
[ 533.185139] [<c015f9f0>] (cpu_startup_entry) from [<0010182c>] (0x10182c)
[ 533.193813] Rebooting in 3 seconds..
(Updated)
Moreover, the kernel has been crashed after several packets processed. In the log, I see some bug information related to Insmod function. Here is my code in the Init function module: ( In my Init function, I implement encryption/decryption with one packet request many times by using a loop for and module_param value. )
static int __init test_init(void)
{
for (i = 0; i < req_num; i ++)
{
if (cipher_choice == 3)
{
test_esp_rfc4106(test_choice,endec);
mdelay(1000);
pr_err("--------------------------%d-------------------:
%s - PID:%d\n",__LINE__ , __func__ , current->pid);
pr_err("------------------------Number of req-----------
--------: %d\n",i);
}
}
return 0;
}
You can tell from the log that this happened right at soon as the module was loaded (well, during loading) because insmod is still running. It was creating an entry in '/sys'. This gives you some idea what may have been happening before the crash.
Since the crash is inside kstrdup/kmalloc and not directly related to your code, the most likely cause is either a double free or a buffer overflow in your module's code. Since it was soon after loading the module, probably the problem is in your module's init. No one is going to be able to tell you exactly what went wrong because you didn't post any code.
Related
I'm trying to inject a new submit bio function to the blkdev, from in-tree kernel module.
I'm using
Linux raspberrypi 5.10.76-v7l+
with LSM support
I was able to do it in lower version with no problem, but with the change of the make_request_fn I'm having a problem with the inject.
this is the code :
#if LINUX_VERSION_CODE < KERNEL_VERSION(5,10)
queue = bdev_get_queue(bdev);
if (queue) {
vm_hold.blkdev_orig.make_request_fn = queue->make_request_fn;
queue->make_request_fn = test_injected_blkdev_make_request_fn;
}
#else
fops = bdev_get_fops(bdev);
if (fops) {
vm_hold.blkdev_orig.submit_bio = fops->submit_bio;
fops->submit_bio = test_injected_blkdev_make_request_fn;
//blk_mq_submit_bio(fops->submit_bio);
}
#endif // LINUX_VERSION_CODE < KERNEL_VERSION(5,10,44)
the test function is just a sample:
blk_qc_t test_injected_blkdev_make_request_fn(struct bio *bio)
{
pr_emerg("my test submit bio function\n");
return vm_hold.blkdev_orig.submit_bio(bio);
}
when i'm calling my module to perform the injection i'm getting a crash that lock like that:
--- cut here ---
[ 130.295126] Unable to handle kernel paging request at virtual address c0c6551c
[ 130.295141] pgd = 226518d8
[ 130.295152] [c0c6551c] *pgd=80000000007003, *pmd=440000000c0079d(bad)
[ 130.295194] Internal error: Oops: a0e [#1] SMP ARM
[ 130.295200] Modules linked in: rfcomm cmac fuse bnep hci_uart btbcm bluetooth ecdh_generic ecc 8021q garp stp llc brcmfmac brcmutil cfg80211 rfkill raspberrypi_hwmon bcm2835_codec(C) bcm2835_isp(C) bcm2835_v4l2(C) v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_bcm2835(C) videobuf2_common snd_pcm snd_timer snd videodev vc_sm_cma(C) mc uio_pdrv_genirq uio i2c_dev ip_tables x_tables ipv6
[ 130.356973] CPU: 3 PID: 1161 Comm: host Tainted: G C 5.10.76-v7l+ #58
[ 130.356977] Hardware name: BCM2835
[ 130.356993] PC is at test_inject_blkdev+0xf4/0x114
[ 130.357008] LR is at __irq_work_queue_local+0x3c/0x70
[ 130.378604] pc : [<c06e2614>] lr : [<c0351bcc>] psr: 60000013
[ 130.378608] sp : ca589e68 ip : ca589d78 fp : ca589e7c
[ 130.378613] r10: 00000003 r9 : 00000000 r8 : beba2644
[ 130.378618] r7 : ca471a80 r6 : c1205048 r5 : c0c6551c r4 : c2574240
[ 130.378627] r3 : c06e0294 r2 : f0fbb462 r1 : 00000d62 r0 : 00004670
[ 130.408779] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 130.408785] Control: 30c5383d Table: 02eeb680 DAC: bb5b2507
[ 130.408792] Process host (pid: 1161, stack limit = 0xa9b229bd)
[ 130.408798] Stack: (0xca589e68 to 0xca58a000)
[ 130.408807] 9e60: c13fd79c c13fd684 ca589e94 ca589e80 c06e26dc c06e252c
[ 130.440451] 9e80: 00004000 c13fd79c ca589ecc ca589e98 c06e3d5c c06e26bc ca589eb4 ca589ea8
[ 130.440458] 9ea0: c0ba4f7c f0fbb462 c1205048 c1205048 beba2644 ca471a80 00005391 00000000
[ 130.440464] 9ec0: ca589f0c ca589ed0 c06e5a40 c06e3cec 00000032 c020bf60 c833c354 00000000
[ 130.465317] 9ee0: 00f3b150 f0fbb462 00000005 c1205048 ca471a80 00005391 ca471a80 beba2644
[ 130.465324] 9f00: ca589fa4 ca589f10 c04574d8 c06e56a4 00000000 00000000 00000000 00000000
[ 130.465330] 9f20: 00000000 00000000 c020bf88 beba2644 c020d244 ca5133c0 c1205048 ca5133c0
[ 130.465336] 9f40: 00000032 00f3b150 ca588000 00000004 ca589f94 ca589f60 c04419b0 c04413c8
[ 130.465342] 9f60: 00000000 00000000 00000000 f0fbb462 ca588000 f0fbb462 00f3b150 00000000
[ 130.465350] 9f80: 00000000 00f3b590 00000036 c0200204 ca588000 00000036 00000000 ca589fa8
[ 130.515053] 9fa0: c0200040 c045737c 00000000 00000000 00000003 00005391 beba2644 00000000
[ 130.515059] 9fc0: 00000000 00000000 00f3b590 00000036 00000000 00000000 b6fdf000 00000000
[ 130.515065] 9fe0: 0002fcac beba2634 0001c214 b6f0651c 80000010 00000003 00000000 00000000
[ 130.539914] Backtrace:
[ 130.539926] [<c06e2520>] (test_inject_blkdev) from [<c06e26dc>] (nanolock_inject+0x2c/0x4c)
[ 130.539932] r5:c13fd684 r4:c13fd79c
[ 130.554836] [<c06e26b0>] (perform_test_inject) from [<c06e3d5c>] (nanolock_end_load_otpm+0x7c/0x2a8)
[ 130.554839] r4:c13fd79c r3:00004000
[ 130.554851] [<c06e3ce0>] (test_inject_blkdev_call) from [<c06e5a40>] (nanolock_ioctl+0x3a8/0x700)
[ 130.576265] r9:00000000 r8:00005391 r7:ca471a80 r6:beba2644 r5:c1205048 r4:c1205048
[ 130.576274] [<c06e5698>] (my_module_ioctl) from [<c04574d8>] (sys_ioctl+0x168/0x8fc)
[ 130.576281] r8:beba2644 r7:ca471a80 r6:00005391 r5:ca471a80 r4:c1205048
[ 130.576288] [<c0457370>] (sys_ioctl) from [<c0200040>] (ret_fast_syscall+0x0/0x28)
[ 130.576293] Exception stack(0xca589fa8 to 0xca589ff0)
[ 130.611458] 9fa0: 00000000 00000000 00000003 00005391 beba2644 00000000
[ 130.611464] 9fc0: 00000000 00000000 00f3b590 00000036 00000000 00000000 b6fdf000 00000000
[ 130.611470] 9fe0: 0002fcac beba2634 0001c214 b6f0651c
[ 130.633154] r10:00000036 r9:ca588000 r8:c0200204 r7:00000036 r6:00f3b590 r5:00000000
[ 130.633156] r4:00000000
[ 130.633163] Code: e3003294 e3040670 e34c306e e3001d62 (e5853000)
[ 130.633212] ---[ end trace 2588a376c135822c ]---
At the older kernel version the code works perfectly. but with the new bio I'm straggling
any reference to the new bio system will help (i used google but i didn't find how to inject the new code properly).
I run a server in ESP8266 which accept argument and value when you click the submit button in the webpage. after some iteration of clicking the submit button, the esp8266 crashes with the Exception of:
--------------- CUT HERE FOR EXCEPTION DECODER ---------------
Exception (29):
epc1=0x40205224 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000
>>>stack>>>
ctx: cont
sp: 3ffffca0 end: 3fffffc0 offset: 0190
3ffffe30: 3fff1598 3fff1598 3ffffe80 4020521f
3ffffe40: 3fff6354 3ffffe98 3ffffe80 40217ef0
3ffffe50: 00000003 3ffffec0 3ffffe80 40217f20
3ffffe60: 3ffffe98 3ffffea4 3fff1588 3fff0da4
3ffffe70: 00000000 3ffffec0 00000005 402053a9
3ffffe80: 3fff631c 0029002f 00003a30 3fff6300
3ffffe90: 0029002f 800001da 3fff6354 0029002f
3ffffea0: 00000001 3fffbf5c 0005002f 00217d42
3ffffeb0: 00000001 3fff0dac 3ffffefc 4020cb24
3ffffec0: 3fff68b4 01da01df 00000000 fffffffe
3ffffed0: 00000000 3fffc6fc 00000000 3fff1c88
3ffffee0: 00000000 3fffdad0 40223278 00000000
3ffffef0: 3ffffefc 3fffff64 3fff0a00 40223428
3fffff00: 00000000 000003e8 3fffff60 03e87701
3fffff10: 402583d1 00000004 3fff0d58 4020171a
3fffff20: 40258677 00000004 3fff0d58 3fff0d5c
3fffff30: 00000000 4bc6a7f0 0189374b 00000000
3fffff40: 00000000 00000000 4bc6a7f0 00000000
3fffff50: 000e001a 00050008 4010051c 0001fc51
3fffff60: 00000000 6c80e048 4024c800 3ffe8514
3fffff70: 024bbda9 80000000 00000000 4010059d
3fffff80: 00000000 00000000 00000001 3fff1cc8
3fffff90: 3fffdad0 00000000 0001fc51 40211f35
3fffffa0: 3fffdad0 00000000 3fff1c88 40218fd4
3fffffb0: feefeffe feefeffe 3ffe899c 40100f45
<<<stack<<<
last failed alloc call: 4020521F(100)
--------------- CUT HERE FOR EXCEPTION DECODER ---------------
ets Jan 8 2013,rst cause:2, boot mode:(3,6)
For debugging: Free Heap is printed(runs every 2 seconds using millis) in the serial monitor by
Serial.println(ESP.getFreeHeap());
The submit button is sending the following argument and value(Partial only) :
http://192.168.4.1?user=1234&password=1234&data1=qwertyuiopasdfghjklzxcvbnm&data1=qwertyuiopasdfghjklzxcvbnm&data2=qwertyuiopasdfghjklzxcvbnm&data3=qwertyuiopasdfghjklzxcvbnm&data4=qwertyuiopasdfghjklzxcvbnm
as you can see the FreeHeap is going down until it reaches the minimum where ESP8266 can't handle.
Is there a way to refresh or reuse the heap without rebooting the Chip?
Is safeString can help to this problem ?
Im getting a "Unable to handle kernel paging request" kernel oops (see error message below) and I can't see why. As far as I know, this error message occurs, when I'm accessing memory that I shouldn't access.But in this case the error message seems to trigger in/directly after the allocation.
So my question is: is there another reason for this oops to occur?
Here's the error message:
[ 68.229185] Unable to handle kernel paging request at virtual address 5761308c
[ 68.236398] pgd = c0004000
[ 68.239094] [5761308c] *pgd=00000000
[ 68.242670] Internal error: Oops: 5 [#1] SMP ARM
[ 68.247266] Modules linked in: riddler(O)
[ 68.251285] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O 4.1.0 #14
[ 68.258388] Hardware name: Altera SOCFPGA
[ 68.262380] task: c07cffa8 ti: c07ca000 task.ti: c07ca000
[ 68.267763] PC is at kmem_cache_alloc_trace+0x60/0x184
[ 68.272883] LR is at riddler_nf_queue_handler+0x164/0x6d4 [riddler]
[ 68.279122] pc : [<c0118ea0>] lr : [<bf000630>] psr: 200f0113
[ 68.279122] sp : c07cbb80 ip : c07cbbc0 fp : c07cbbbc
[ 68.290546] r10: 00000000 r9 : ed66d800 r8 : 0000000c
[ 68.295746] r7 : edc01f00 r6 : bf000630 r5 : 00008020 r4 : 5761308c
[ 68.302243] r3 : 00000000 r2 : c07cbb80 r1 : c07cbce0 r0 : 2da0f000
[ 68.308741] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
[ 68.316015] Control: 10c5387d Table: 2cd3004a DAC: 00000015
[ 68.321734] Process swapper/0 (pid: 0, stack limit = 0xc07ca218)
[ 68.327713] Stack: (0xc07cbb80 to 0xc07cc000)
[ 68.332054] bb80: 000005dc c0448cdc c07cc404 00004058 2d66e000 ed5c99d0 bf001cfc ed5a8800
[ 68.340196] bba0: bf001cfc 000005c0 ed66d800 00000000 c07cbc14 c07cbbc0 bf000630 c0118e4c
[ 68.348338] bbc0: c07cbc9c 00000044 c07cbc14 c07cbbd8 00000000 00000000 00000001 00484c68
[ 68.356479] bbe0: 000005c0 0000001c 00000044 ed5a8800 c05b83f0 00000044 bf001984 edfebe40
[ 68.364621] bc00: bf001940 00000000 c07cbc64 c07cbc18 c0484e3c bf0004d8 c07cbc3c 00000000
[ 68.372762] bc20: 80000000 c04bbc02 ede3f000 00000000 00000000 c0489f30 c0483d08 c07cbc9c
[ 68.380903] bc40: edfebe40 c07cbc6c c07cdae8 00000003 ede3f000 c07cd524 c07cbc94 c07cbc68
[ 68.389045] bc60: c0483da8 c0484d00 edc817c0 bf001940 c07c9340 00000050 ecc91340 edfebe40
[ 68.397186] bc80: ede3f000 00000008 c07cbcd4 c07cbc98 c048a880 c0483d54 c07cbccc 00000000
[ 68.405327] bca0: 80000000 c04bbc02 ede3f000 00000000 00000000 c0489f30 edfebe40 c07ce898
[ 68.413468] bcc0: c048a4c4 ede3f000 c07cbd24 c07cbcd8 c0457b00 c048a4d0 00000000 ede3f000
[ 68.421610] bce0: edfebe40 ede3f580 00000020 edfebe40 00000000 ede3f05c c07cbd14 c07cd514
[ 68.429752] bd00: edfebe40 ede3f580 00000020 00000100 00000000 00000001 c07cbd3c c07cbd28
[ 68.437893] bd20: c045a34c c0457440 c07cd514 edfebe40 c07cbd6c c07cbd40 c045a400 c045a320
[ 68.446034] bd40: 2690390a 14562ba2 edfebe40 ede3f580 00000020 00000100 00000003 edfebe40
[ 68.454175] bd60: c07cbd8c c07cbd70 c045ada8 c045a3b4 ede3f580 edfebe40 000005ea 00000020
[ 68.462316] bd80: c07cbdec c07cbd90 c03a50c8 c045acec 00000000 e2d32d80 0000000f ee1d5400
[ 68.470456] bda0: 00000001 ede3f4c0 c07cbde4 c07cbdb8 c07d153c 00000100 00000040 00000002
[ 68.478597] bdc0: ffffffff ede3f580 00000001 00000040 0000012c ee1d8b80 c07cc100 c07cbe18
[ 68.486738] bde0: c07cbe54 c07cbdf0 c045be80 c03a4e04 ffffa577 c06b2a60 c07cd514 c07cd514
[ 68.494880] be00: c082ce31 ee1d8b80 c082f608 2da0f000 c07c9b80 ffffa579 c07cbe18 c07cbe18
[ 68.503021] be20: c07cbe20 c07cbe20 00000001 c07cc08c 00000003 00000000 00000100 c07ca000
[ 68.511163] be40: 00000008 c082cf0c c07cbebc c07cbe58 c002a684 c045bc68 00000011 edd41180
[ 68.519304] be60: edd411e0 00200000 c07cc100 ffffa578 c05769b4 0000000a c08302c0 c07c52c8
[ 68.527445] be80: c07cc080 c07cbe58 0000000c 00000004 00000000 c07c7c2c 00000024 00000000
[ 68.535587] bea0: 00000000 edc10800 c07cc450 c05769a8 c07cbed4 c07cbec0 c002ab68 c002a5a0
[ 68.543728] bec0: c07c7c2c 00000024 c07cbefc c07cbed8 c006738c c002aaec fee0010c c07cce2c
[ 68.551870] bee0: c07cbf20 fee00100 c082d4c0 c07cc450 c07cbf1c c07cbf00 c0009470 c0067328
[ 68.560011] bf00: c0010648 600f0013 ffffffff c07cbf54 c07cbf74 c07cbf20 c0014680 c0009448
[ 68.568153] bf20: 00000000 00000000 000198a0 c0021be0 00000000 c07ca000 c07cc49c ffffffff
[ 68.576295] bf40: c082d4c0 c07cc450 c05769a8 c07cbf74 c07cbf78 c07cbf68 c0010644 c0010648
[ 68.584437] bf60: 600f0013 ffffffff c07cbf9c c07cbf78 c005db84 c001060c 00000000 c082cc99
[ 68.592578] bf80: c07c5304 c07cbf78 c0570680 ffffffff c07cbfac c07cbfa0 c056be78 c005d9f4
[ 68.600719] bfa0: c07cbff4 c07cbfb0 c0763cc0 c056bdfc ffffffff ffffffff c07636d8 00000000
[ 68.608861] bfc0: 00000000 c07a98c8 00000000 c082fc14 c07cc440 c07a98c4 c07d1218 0000406a
[ 68.617002] bfe0: 413fc090 00000000 00000000 c07cbff8 0000807c c0763978 00000000 00000000
[ 68.625154] [<c0118ea0>] (kmem_cache_alloc_trace) from [<bf000630>] (riddler_nf_queue_handler+0x164/0x6d4 [riddler])
[ 68.635638] [<bf000630>] (riddler_nf_queue_handler [riddler]) from [<c0484e3c>] (nf_queue+0x148/0x17c)
[ 68.644907] [<c0484e3c>] (nf_queue) from [<c0483da8>] (nf_hook_slow+0x60/0xfc)
[ 68.652104] [<c0483da8>] (nf_hook_slow) from [<c048a880>] (ip_rcv+0x3bc/0x4c4)
[ 68.659308] [<c048a880>] (ip_rcv) from [<c0457b00>] (__netif_receive_skb_core+0x6cc/0x874)
[ 68.667542] [<c0457b00>] (__netif_receive_skb_core) from [<c045a34c>] (__netif_receive_skb+0x38/0x94)
[ 68.676724] [<c045a34c>] (__netif_receive_skb) from [<c045a400>] (netif_receive_skb_internal+0x58/0xcc)
[ 68.686077] [<c045a400>] (netif_receive_skb_internal) from [<c045ada8>] (napi_gro_receive+0xc8/0xf4)
[ 68.695176] [<c045ada8>] (napi_gro_receive) from [<c03a50c8>] (stmmac_poll+0x2d0/0x644)
[ 68.703148] [<c03a50c8>] (stmmac_poll) from [<c045be80>] (net_rx_action+0x224/0x354)
[ 68.710864] [<c045be80>] (net_rx_action) from [<c002a684>] (__do_softirq+0xf0/0x304)
[ 68.718577] [<c002a684>] (__do_softirq) from [<c002ab68>] (irq_exit+0x88/0xc0)
[ 68.725774] [<c002ab68>] (irq_exit) from [<c006738c>] (__handle_domain_irq+0x70/0xc4)
[ 68.733575] [<c006738c>] (__handle_domain_irq) from [<c0009470>] (gic_handle_irq+0x34/0x6c)
[ 68.741893] [<c0009470>] (gic_handle_irq) from [<c0014680>] (__irq_svc+0x40/0x54)
[ 68.749340] Exception stack(0xc07cbf20 to 0xc07cbf68)
[ 68.754371] bf20: 00000000 00000000 000198a0 c0021be0 00000000 c07ca000 c07cc49c ffffffff
[ 68.762512] bf40: c082d4c0 c07cc450 c05769a8 c07cbf74 c07cbf78 c07cbf68 c0010644 c0010648
[ 68.770650] bf60: 600f0013 ffffffff
[ 68.774133] [<c0014680>] (__irq_svc) from [<c0010648>] (arch_cpu_idle+0x48/0x4c)
[ 68.781508] [<c0010648>] (arch_cpu_idle) from [<c005db84>] (cpu_startup_entry+0x19c/0x264)
[ 68.789743] [<c005db84>] (cpu_startup_entry) from [<c056be78>] (rest_init+0x88/0x8c)
[ 68.797465] [<c056be78>] (rest_init) from [<c0763cc0>] (start_kernel+0x354/0x360)
[ 68.804917] Code: e7914000 e3540000 0a000038 e5973014 (e7949003)
[ 68.811024] ---[ end trace 11b433a852b633dc ]---
And here's the code segment that's causing it (I think, I used gdb "list *(riddler_nf_queue_handler)+0x164" to find it):
new = (struct fpga_kfifo_entry *) kzalloc(sizeof(struct fpga_kfifo_entry), GFP_ATOMIC);
new->entry = entry;
new->dma_buffer = dma_buffer; //+0x164 is here
new->dma_buffer_size = data_length_pad;
if(!(kfifo_put(&irq_fifo, new))) {
// FIFO was full
#ifdef DEBUG
pr_info("IRQ Fifo is full, couldn't add new entry!\n");
#endif
kfree(dma_buffer);
kfree(new);
nf_reinject(entry, NF_DROP);
return 0;
}
dma_buffer is allocated in the same function, but earlier:
data_length_pad = round_to_word_size(data_length + 1);
dma_buffer = (unsigned char *) kzalloc(data_length_pad, GFP_ATOMIC);
if (!dma_buffer) {
kfree(dma_buffer);
nf_reinject(entry, NF_DROP);
return 0;
} else {
// Only continue when allocation was successful
dma_buffer[data_length_pad - 1] = (unsigned char) data_length_pad - data_length;
}
Thanks in advance!
The Deadlock occurs in my application when initialization of local static variable happens in the function called from DLLMain Entry point with param DLL_THREAD_DETACH.
Below is Windbg analysis
This is usually caused by another thread holding the loader lock.
Following are the Locks Held.
CritSec ntdll!LdrpLoaderLock+0 at 7c97e178
LockCount 3
RecursionCount 1
OwningThread 17e8
EntryCount d
ContentionCount d
*** Locked
CritSec MSVCR80!__app_type+94 at 781c3bc8
LockCount 1
RecursionCount 1
OwningThread 1100
EntryCount 1
ContentionCount 1
*** Locked
#
Call stack Thread 17e8
781c3bc8 78132bd9 0777fde4 ntdll!RtlEnterCriticalSection+0x46
00000008 b87d2630 00000000 MSVCR80!_lock+0x2e
0864ae10 08631d7f 0864ae10 EPComUtilities32!_onexit+0x36
0864ae10 b87d2588 00000001 EPComUtilities32!atexit+0x9
0777fea8 0864719f 08630000 EPComUtilities32!XCriticalSectionEx::ThreadTerminated+0x5f
08630000 00000003 00000000 EPComUtilities32!DllMain+0x20
08630000 7c90118a 08630000 EPComUtilities32!__DllMainCRTStartup+0x7a
08630000 00000003 00000000 EPComUtilities32!_DllMainCRTStartup+0x1d
#
Call Stack thread 1100
000000b0 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
000000b0 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xa8
000000b0 ffffffff 06ce64e0 kernel32!WaitForSingleObject+0x12
000480ba 000f4240 00000000 CATSysMultiThreading!CATThreads::Join+0xf5
0012fcc8 00000004 00000000 JS0GROUP!CATLM::StopHB+0xf4
d138509f 00416694 00000001 JS0GROUP!CATLM::Unmake+0x6b
00000000 00000000 00000000 MSVCR80!_cinit+0xd6
00000000 0012fd6c 081e68d9 MSVCR80!exit+0xd
00000000 06d404f0 0998fb90 JS0GROUP!CATExit+0x1d
00000000 004ef366 0000000d DNBPLMProvider!DNBEPLMTransactionMgt::OnApplicationExit+0x229
00000000 0012fd9c 004eabfc JS0GROUP!CATCallExits+0x2bc
00000000 0012ff7c 0040cefd JS0GROUP!CATErrorNormalEnd+0x31
00000000 06ce71d0 06ce71d0 JS0GROUP!CATExit+0xc
00000007 06cdb120 059b61d8 DLMMfgContextSolver!main+0x146d
ffffffff ffffffff bffde000 DLMMfgContextSolver!__tmainCRTStartup+0x10f
#
Please give you comments to understand what might have caused the deadlock.
Note: the moment i make the static variable as non static the problem disappears this in context of example posted in forum Deadlock occurs in Function Scoped Static variables (Thread Unsafe in VC++)
In short, what caused the deadlock is that you did something non-trivial in DllMain.
I am trying to debug an IE 8 crash caused by a Silverlight application. I managed to find some articles on how to do a memory dump when a process crashes. I loaded the dump in windbg and ran !analyze -v. Below is the result. I am stuck at what further steps I can take to figure out what module or library that is running in Silverlight is causing the crash. So all I have right now is the crash in IE is caused by an Access violation (attempt to execute non-executable address) and from what is in the stack trace that some animation is running in Silverlight.
Any tips or articles that would help me debug this will be appreciated.
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1864.1560): Access violation - code c0000005 (first/second chance not available)
eax=00000000 ebx=00000000 ecx=1b11fc58 edx=5c6f007d esi=00000000 edi=193b8e08
eip=00000000 esp=0f61f750 ebp=0f61f76c iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
00000000 ?? ???
FAULTING_IP:
+56b3952f04ebde68
748bc9f1 654c dec esp
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 748bc9f1
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000008
Parameter[1]: 00000000
Attempt to execute non-executable address 00000000
PROCESS_NAME: iexplore.exe
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_PARAMETER1: 00000008
EXCEPTION_PARAMETER2: 00000000
WRITE_ADDRESS: 00000000
FOLLOWUP_IP:
agcore!CFrameworkElement::SetValue+1d7
5c704fa8 84c0 test al,al
FAILED_INSTRUCTION_ADDRESS:
+56b3952f04ebde68
748bc9f1 654c dec esp
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
FAULTING_THREAD: 00001560
BUGCHECK_STR: APPLICATION_FAULT_SOFTWARE_NX_FAULT_NULL
PRIMARY_PROBLEM_CLASS: SOFTWARE_NX_FAULT_NULL
DEFAULT_BUCKET_ID: SOFTWARE_NX_FAULT_NULL
LAST_CONTROL_TRANSFER: from 5c704fa8 to 00000000
STACK_TEXT:
WARNING: Frame IP not in any known module. Following frames may be wrong.
0f61f74c 5c704fa8 1b17a134 193b8e08 0e690e14 0x0
0f61f76c 5c712360 0e690e14 1b17a134 0e690e14 agcore!CFrameworkElement::SetValue+0x1d7
0f61f788 5c7123a8 0e690e14 1b17a134 0e690e14 agcore!CShape::SetValue+0x72
0f61f7a0 5c70a6ff 0e690e14 1b17a134 00000000 agcore!CEllipse::SetValue+0x3b
0f61f7d0 5c752c2b 1b17a090 193b8e08 00000000 agcore!CAnimation::DoSetValue+0x50
0f61f810 5c7a7fb1 0f61f884 0f61f868 1b17a090 agcore!CAnimation::UpdateAnimationUsingKeyFrames+0x3b5
0f61f82c 5c707146 00000000 00000000 00000000 agcore!CAnimation::UpdateAnimation+0x184
0f61f87c 5c7071e5 3e4c8000 0f61f8cc 00000000 agcore!CTimeline::ComputeState+0x13a
0f61f89c 5c706d49 193f82b0 0f61f8cc 0f61f8d4 agcore!CTimelineGroup::ComputeState+0x8c
0f61f8ac 5c7069c7 3e4c8000 0f61f8cc 0b111f60 agcore!CStoryboard::ComputeState+0x48
0f61f8d4 5c706a29 0e6a0ca0 00000000 0e490070 agcore!CTimeManager::Tick+0x79
0f61f8e8 5c78f960 0b0e6d68 0f61f990 00000000 agcore!CCoreServices::Tick+0x21
0f61f940 5c706ac2 0b111f60 0e42ca08 ffffffff agcore!CCoreServices::Draw+0x140
0f61f964 67ac141c 0af99b90 00000000 0f61f990 agcore!CCoreServices::Draw+0x2d
0f61f9b4 67a933c2 0f61f9c8 00000000 00000000 npctrl!CXcpBrowserHost::OnTick+0x1b1
0f61f9e0 67a927c6 0064069c 00000402 00000000 npctrl!CXcpDispatcher::Tick+0xf3
0f61fa08 67a92709 0064069c 00000402 00000000 npctrl!CXcpDispatcher::OnReentrancyProtectedWindowMessage+0xcd
0f61fa28 764b6238 0064069c 00000402 00000000 npctrl!CXcpDispatcher::WindowProc+0xb8
0f61fa54 764b68ea 67a9269d 0064069c 00000402 user32!InternalCallWinProc+0x23
0f61facc 764b7d31 00000000 67a9269d 0064069c user32!UserCallWinProcCheckWow+0x109
0f61fb2c 764b7dfa 67a9269d 00000000 0f61fbb4 user32!DispatchMessageWorker+0x3bc
0f61fb3c 6fe504a6 0f61fb54 00000000 0ab11908 user32!DispatchMessageW+0xf
0f61fbb4 6fe60446 0af956a0 00000000 0b18a338 ieframe!CTabWindow::_TabWindowThreadProc+0x452
0f61fc6c 769d49bd 0ab11908 00000000 0f61fc88 ieframe!LCIETab_ThreadProc+0x2c1
0f61fc7c 76e53677 0b18a338 0f61fcc8 77829d72 iertutil!CIsoScope::RegisterThread+0xab
0f61fc88 77829d72 0b18a338 7dbc895d 00000000 kernel32!BaseThreadInitThunk+0xe
0f61fcc8 77829d45 769d49af 0b18a338 00000000 ntdll!__RtlUserThreadStart+0x70
0f61fce0 00000000 769d49af 0b18a338 00000000 ntdll!_RtlUserThreadStart+0x1b
SYMBOL_STACK_INDEX: 1
SYMBOL_NAME: agcore!CFrameworkElement::SetValue+1d7
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: agcore
IMAGE_NAME: agcore.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4a67e422
STACK_COMMAND: ~44s; .ecxr ; kb
FAILURE_BUCKET_ID: SOFTWARE_NX_FAULT_NULL_c0000005_agcore.dll!CFrameworkElement::SetValue
BUCKET_ID: APPLICATION_FAULT_SOFTWARE_NX_FAULT_NULL_BAD_IP_agcore!CFrameworkElement::SetValue+1d7
This morning one of the third-party libraries that we were using in Silverlight was updated and that fixed the problem.
One of the ways to identify the cause you would have to get the memory dump and look for call-stack. Here are couple of links to identify the cause here and here
Here is existing answer Is anyone else experiencing weird debug + crash behavior with Silverlight?