1263 *(int *)-1 = 0;
(gdb) bt
#0 0xc0215d6c in pcibios_init () at arch/ppc/kernel/pci.c:1263
#1 0xc020e728 in do_initcalls () at init/main.c:563
#2 0xc020e7c4 in do_basic_setup () at init/main.c:605
#3 0xc0001374 in init (unused=0x20) at init/main.c:677
#4 0xc00049d0 in kernel_thread ()
Previous frame inner to this frame (corrupt stack?)
(gdb)
The crash in this example was contrived by a simple write to an invalid memory location (all ones). We first establish a connection from gdb to KGDB and allow the kernel to continue to boot. Notice that we didn't even bother to set breakpoints. When the crash occurs, we see the line of offending code and get a nice backtrace to help us determine its cause.
14.6. Chapter Summary
• Linux kernel debugging presents many complexities, especially in a cross-development environment. Understanding how to navigate these complexities is the key to successful kernel debugging.
• KGDB is a very useful kernel-level gdb stub that enables direct symbolic source-level debugging inside the Linux kernel and device drivers. It uses the gdb remote protocol to communicate to your host-based cross-gdb.
• Understanding (and minimizing) compiler optimizations helps make sense of seemingly strange debugger behavior when stepping through compiler-optimized code.
• gdb supports user-defined commands, which can be very useful for automating tedious debugging tasks such as iterating kernel linked lists and accessing complex variables.
• Kernel-loadable modules present their own challenges to source-level debugging. The module's initialization routine can be debugged by placing a breakpoint in module.c at the call to module->init().
• printk and the Magic SysReq key provide additional tools to help isolate problems during kernel development and debugging.
• Hardware-assisted debugging via a JTAG probe enables debugging Flash or ROM resident code where other debugging methods can be cumbersome or otherwise impossible.
• Enabling CONFIG_SERIAL_TEXT_DEBUG on architectures where this feature is supported is a powerful tool for debugging a new kernel port.
• Examining the printk log_buf often leads to the cause of a silent kernel crash on boot.
• KGDB passes control to gdb on a kernel panic, enabling you to examine a backtrace and isolate the cause of the kernel panic.
14.6.1. Suggestions for Additional Reading
Robert Love
Novell Press, 2005
Claudia Salzberg Rodriguez et al.
Prentice Hall, 2005
'Using the GNU Compiler Collection'
Richard M. Stallman and the GCC Developer Community GNU Press, a division of Free Software Foundation
http://gcc.gnu.org/onlinedocs/
KGDB Sourceforge home page
http://sourceforge.net/projects/KGDB
Debugging with GDB
Richard Stallman, Roland Pesch, Stan Shebs, et al.
Free Software Foundation
www.gnu.org/software/gdb/documentation/
Tool Interface Standards
DWARF Debugging Information Format Specification
Version 2.0
TIS Committee, May 1995
Chapter 15. Debugging Embedded Linux Applications
In the previous chapter, we explored the use of GDB for debugging kernel code and code resident in Flash, such as bootloader code. In this chapter, we continue our coverage of GDB for debugging application code in user space. We extend our coverage of remote debugging and the tools and techniques used for this peculiar debugging environment.
15.1. Target Debugging
We already explored several important debugging tools in Chapter 13, 'Development Tools.' strace and ltrace can be used to observe and characterize a process's behavior and often isolate problems. dmalloc can help isolate memory leaks and profile memory usage. ps and top are both useful for examining the state of processes. These relatively small tools are designed to run directly on the target hardware.
Debugging Linux application code on an embedded system has its own unique challenges. Resources on your embedded target are often limited. RAM and nonvolatile storage limitations might prevent you from running target-based development tools. You might not have an Ethernet port or other high-speed connection. Your target embedded system might not have a graphical display, keyboard, or mouse.
This is where your cross-development tools and an NFS root mount environment can yield large dividends.
