How to debug a custom Linux Kernel on a remote machine?
I have a remote machine with a custom-built Linux Kernel originally from 5.6.8. The system under the kernel is running into problems which I want to fix. The kgdb-related options are
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_KGDB_TESTS is not set
CONFIG_KGDB_LOW_LEVEL_TRAP=y
CONFIG_KGDB_KDB=y
The kgdboc module is compiled builtin. The documentation of kgdb describes how to use terminal with a serial port. Here is how it is configured on the remote machine:
root#remote-pc:# cat /sys/module/kgdboc/parameters/kgdboc
ttyS0,115200
Using ssh I issued
root#remote-pc:~# echo g > /proc/sysrq-trigger
And after running gdb vminux what port should I use to connect to the remote Kernel?
(gdb) target remote remote-pc:__what_port?__
You cannot connect to a literally remote machine through a serial port. What you want to use is KGDB Over Ethernet (kdgboe). See How to use kgdb over ethernet (kgdboe)?
Related
I'm studying DPDK and trying to create a simple application, however it can't see a NIC bound to DPDK.
Here is a list of network devices I have on my machine
$ dpdk-devbind.py --status-dev net
Network devices using kernel driver
===================================
0000:01:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller 8168' if=enp1s0 drv=r8169 unused=vfio-pci *Active*
0000:02:00.0 'RTL8822BE 802.11a/b/g/n/ac WiFi adapter b822' if=wlp2s0 drv=rtw_8822be unused=rtw88_8822be,vfio-pci *Active*
I disable my ethernet NIC (it can't be bound to DPDK while it is active) and bind it to vfio-pci driver successfully
$ ip link set enp1s0 down
$ dpdk-devbind.py -b vfio-pci enp1s0
Now dpdk-devbind.py shows that the NIC is using DPDK-compatible driver
$ dpdk-devbind.py --status-dev net
Network devices using DPDK-compatible driver
============================================
0000:01:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller 8168' drv=vfio-pci unused=r8169
Network devices using kernel driver
===================================
0000:02:00.0 'RTL8822BE 802.11a/b/g/n/ac WiFi adapter b822' if=wlp2s0 drv=rtw_8822be unused=rtw88_8822be,vfio-pci *Active*
However when I run any example DPDK application it says that there are no available NIC ports. For instance I wrote a simple application
int main(int argc, char *argv[])
{
int ret;
int total_ports, avail_ports;
ret = rte_eal_init(argc, argv);
if( ret < 0 )
rte_exit(EXIT_FAILURE, "EAL initialization failed\n");
total_ports = rte_eth_dev_count_total();
avail_ports = rte_eth_dev_count_avail();
printf("ETH PORTS %d %d\n", total_ports, avail_ports);
rte_eal_cleanup();
return 0;
}
and both rte_eth_dev_count_total() and rte_eth_dev_count_avail() return 0.
What am I doing wrong?
The main reason why DPDK ports are not identified in your environment is because the NIC in use is not having a supported vendor Poll Mode Driver. Please refer to list of supported NIC from various vendor Realtek is not among them.
Work around: you can use PCAP PMD to solve the problem. Please follow the steps for your environment as
ensure your ethernet port is linked to kernel driver r8169
isntall libpacp-dev for your enviorment (linux/bsd)
rebuild dpdk with PCAP support.
start your application with following args --no-pci --vdev=net_pcap0,iface=enp1s0
This will use libpcap PMD as wrapper for RX and TX over non supported NIC
I am trying to make a Raspberry Pi processor communicate with my MacBook Pro using a C program. I have an Ethernet cable connected to both devices and a USB wireless adaptor for Wifi connection. Both the Mac and Pi are connected via the same Wifi network.
The C code establishes the Client-Server connection and this code can be found here:
Server: http://www.cs.rpi.edu/~moorthy/Courses/os98/Pgms/server.c
Client: http://www.cs.rpi.edu/~moorthy/Courses/os98/Pgms/client.c
The guide I am using is here: http://www.cs.rpi.edu/~moorthy/Courses/os98/Pgms/socket.html
I placed the server.c file in one of my Mac's folders and the client.c file in a folder within the Raspberry Pi. After compiling both using 'gcc -o client client.c' and likewise server.c, I run the following on the MacBook Pro's Terminal:
./server 51717
Where 51717 is the port number I am using; the server code requires me to specify the port number. The client requires me to pass in my machine's hostname and port number. Therefore, I run the following from the Raspberry Pi's terminal:
./client localhost 51717
When running both ./server and ./client from my MacBook Pro, the program executes just fine. However, the error occurs when executing ./client from the Pi. This yields a: "Connection refused" error. I have tried looking up 'My Hostname' and inputted the value instead of putting 'localhost'. I also placed my IP address over 'localhost' and merely got a 'Connection timed out' error. I am not sure what else to input as my 'hostname' in order to make the connection work.
The issue was in fact that I needed to use the IP Address of the ethernet cable connected to the MacBook Pro. I found that by going to the WiFi button at the top of the screen, clicking 'Open Network Preferences', and then selecting the ethernet tab.
I am trying to port eCos on an i386 PC.
I have downloaded prebuilt redboot.bin from
http://ecos.sourceware.org/ecos/boards/redbootbins/x86pc/
I boot it onto usb disk, using
dd conv=sync if/redboot.bin of=/dev/sdb1
After booting target from usb, I get "IA2!" string on the target monitor always, and on serial port on 38400 8n1 configurations, I receive nothing.
I tried using i386-elf-gdb, but it is not able to connect to the target and starts printing "Ignoring error packet, Continuing..."
I also tried to build redboot using configtool for i386, but it is only able to build library, when I try Tests, It gives ERROR: multiple definition of cyg_start()
I am very new to eCos, and I don't know what I am doing wrong!!.
Ok, I figured out how to boot Redboot on a target i386 pc with RealteK RTL8139 ehternet card.
install grub on usb stick,
mkdir /mnt/USB && mount /dev/sdx1 /mnt/USB
grub-install --force --no-floppy --boot-directory=/mnt/USB/boot /dev/sdx
Build Redboot using ecosconfig, make sure the number of pci bus are less than 8 or more, if more, then need to increase the pci bus range from from 8 inside pci.h, I had my realtek ethernet card on bus 10 dev 10, I had to increase the bus to 11, so that redboot finds realtek card on bootup.
ecosconfig new pc redboot
configtool ecos.ecc
add common ethernet support
Build Library
copy redboot.elf on usb.
on grub startup menu,
insmod multiboot
multiboot /redboot.elf
boot
Thats it, redboot will use BOOTP and provide IP Address, then I can test redboot commands like ip_address, reset, ping, version etc.
I am trying to setup linux kernel module debugging, using two machines - target and host. On target machine, I have compiled and installed a 3.5.0 kernel with CONFIG_MAGIC_SYSRQ=y flag and other flags for over the serial console debugging.
When I want to break the kernel to attach remote gdb, I use
$ echo g > /proc/sysrq-trigger
But above command is not breaking the kernel.
$ cat /proc/sys/kernel/sysrq"
Above command is returning 1, hence magic sysrq keys are enabled. Even "echo b > /proc/sysrq-trigger" is working and rebooting the machine. Can anybody please point out what I may be missing?
Thanks
You have first configure your target kernel as follows
CONFIG_FRAME_POINTER=y
CONFIG_DEBUG_KERNEL=y
CONFIG_KGDB=y
CONFIG_DEBUG_INFO=y
CONFIG_KGDB_SERIAL_CONSOLE=y (here I am using serial port for kgdb)
CONFIG_MAGIC_SYSRQ= y (for sysrq functions).
Now compile kernel with imx6 configuration file.
Boot the target with this compiled kernel.You have to tell tell target which serial port you are going to use for kgdb pupose.In my case I am using the same console port for kgdb also.This settings you can do either through kernel parameters or via sysfs entry.For imx6 sabrelite board,I am using ttymxc1 for console.This will change depending on your target
1) As a kernel parameter
Add the following parameter to bootargs
kgdboc=/dev/ttymxc1,115200 to your arguments.
2) If you are using sysfs entry, do like this
echo /dev/ttymxc1,115200 > /sys/module/kgdboc/parameters/kgdboc
Since same serial port is used for both the console and debugging, we use agent proxy. Through agent proxy we get the target console as well as we do the debugging.
Source for compiling agentproxy is available at the following link
"https://kernel.googlesource.com/pub/scm/utils/kernel/kgdb/agent-proxy/+/agent-proxy-1.96"
After compiling for host pc ,run it as follows
sudo ./agent-proxy 5550^5551 0 /dev/ttyS0,15200
Now you can see target terminal through telnet with this agentproxy support
sudo telnet localhost 5550
(It is better to use this telnet instead of minicom where only this agent proxy support comes.)
When you want to start debugging, the target system has to enter debug mode from normal mode. We can do that in this way in target
echo g > /proc/sysrq-trigger
Now it will enter debugger mode.
Now from host side run gdb on vmlinux of the arm compiled kernel.
Go to the corresponding kernel source directory and do like this
arm-fsl-linux-gnueabi-gdb ./vmlinux
Now it will show gdb terminal .From there you have to connect to target for kgdb,
$target remote /dev/ttyS0
In my case my host serial port is /dev/ttyS0.
Now it will get connected to target. Here after you can use gdb commands to debug the kernel.
You try this way.
i want to debug an MIPS linux driver from my 64bit suse machine over serial ttyS0. The used gdb works greate over LAN with the debugging of applications but not with kgdb over serial. I used this page and a few more to start the debugging but without final results.
My kernel is compiled with the following setting:
CONFIG_MAGIC_SYSRQ=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_KGDB_TESTS is not set
CONFIG_CMDLINE="kgdboc=ttyS0,115200"
if i run the gdb:
gdb vmlinux
(gdb) set remotebaud 115200
(gdb) set debug remote 1
(gdb) target remote /dev/ttyS0
i can observe the following output:
OUTPUT (GDB_TERMINAL):
(gdb) target remote /dev/ttyS0
Remote debugging using /dev/ttyS0
Sending packet: $qSupported:qRelocInsn+#9a...Ack
Timeout in mid-packet, retrying
Timed out.
Timed out.
Ignoring packet error, continuing...
Packet qSupported (supported-packets) is supported
warning: unrecognized item "qSupported:qRelocIns" in "qSupported" response
Sending packet: $Hg0#df...Nak
Sending packet: $Hg0#df...Ack
Packet received: Hg0
Sending packet: $?#3f...Packet instead of Ack, ignoring it
Ack
Timed out.
Timed out.
Timed out.
Ignoring packet error, continuing...
Sending packet: $Hc-1#09...Nak
Sending packet: $Hc-1#09...Ack
Reply contains invalid hex digit 36
OUTPUT (REMOTE_TARGET):
+$?#3f09n+#9a$Hg0#df+09
Nothing more happens!!!
I also test the sysrq but the mentioned sysrq-option 'g' seems to do not to fit!
echo b > /proc/sysrq-trigger
#successfully reboot
echo g > /proc/sysrq-trigger
#prints only the help message (SysRq : HELP : loglevel(0-9) reBoot Crash termin .....)
Is the sysrq running correctly?
Is there somethin that i have missed?
Exists there a way to test the running kgdb on my remote device?
I had many issues to run gdb with kgdb over serial link. My host is an Intel x86 Linux machine and the target is ARM 32-bit Raspberry Pi 2. The target is connected with a USB to TTL Serial Cable. Here are the key problems and their solutions.
1) Do not use gdb concurrently with screen or minicom.
With minicom connected on the tty, GDB hangs and then crashes:
(gdb) target remote /dev/ttyUSB0
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
/build/gdb-cXfXJ3/gdb-7.11.1/gdb/thread.c:89: internal-error: inferior_thread: Assertion `tp' failed.
A problem internal to GDB has been detected, further debugging may prove unreliable.```
And when screen connected, GDB fails to connect:
(gdb) tar rem /dev/ttyUSB0
/dev/ttyUSB0: Device or resource busy.
2) Corrupted packets and timeouts
Invalid packet, Bad checksum, Saw new packet start in middle of old one, Timed out
Use GDB supporting the target architecture. The default GDB on x86 (at least on Ubuntu) does not support an arm target. Use instead gdb-multiarch or the GDB corresponding to the cross-compiling toolset i.e. arm-linux-gnueabihf-gdb. List available architecutres with (gdb) set architecture command. The auto architecture was detecting the arm target correctly in my case.
3) KGDB must be triggered on the target prior to connect
Remote replied unexpectedly to 'vMustReplyEmpty': vMustReplyEmpty
Malformed response to offset query, qOffsets
GDB tries to connect, but the target is not in debug mode. The debug sysrq must be triggered before connecting with GDB. The keyboard shortcut was not working for me. Running the command echo g > /proc/sysrq-trigger through SSH as root works.
4) Configure the baudrate
warning: Invalid baud rate 115200. Maximum value is 38400.
/dev/ttyUSB0: Invalid argument.
The baudrate configured must match between the kgdboc kernel parameter and GDB. In my case, the baudrate 115200 was not supported. Setting the baudrate to 38400 as suggested was necessary. Reboot the target with kernel cmdline: kgdboc=ttyAMA0,38400 and then:
(gdb) set remotebaud 38400
(gdb) target remote /dev/ttyUSB0
Remote debugging using /dev/ttyUSB0
0x800b4730 in kgdb_breakpoint ()
Once these issues were solved, kernel debugging worked as expected.
I know that this is for 2 years ago but I just wanted to say that I'm having the same problem with my board which unfortunately doesn't have a LAN interface to use gdbserver over a TCP connection. I don't know if you could figure this out but if your board does have a LAN interface (or you can build one using USB to Ethernet adapters) it seems that debugging is possible through that!