Embedded Linux Primer: A Practical, Real-World Approach

13.7.1. Suggestions for Additional Reading

GDB: The GNU Project Debugger:

www.gnu.org/software/gdb/gdb.html

GDB Pocket Reference

Arnold Robbins

O'Reilly Media, 2005

Data Display Debugger:

www.gnu.org/software/ddd/

cbrowser home page:

http://cbrowser.sourceforge.net/

cscope home page:

http://cscope.sourceforge.net/index.html

dmallocDebug Malloc Library:

http://dmalloc.com/

Tool Interface Standard (TIS) Executable and Linking Format (ELF) Specification

Version 1.2

TIS Committee, May 1995

Tool interface standards:

DWARF Debugging Information Format Specification

Version 2.0

TIS Committee, May 1995

Chapter 14. Kernel Debugging Techniques

Often the pivotal factor in achieving development timetables comes down to one's efficiency in finding and fixing bugs. Debugging inside the Linux kernel can be quite challenging. No matter how you approach it, kernel debugging will always be complex. This chapter examines some of the complexities and presents ideas and methods to improve your debugging skills inside the kernel and device drivers.

14.1. Challenges to Kernel Debugging

Debugging a modern operating system involves many challenges. Virtual memory operating systems present their own unique challenges. Gone are the days when we could replace a processor with an in-circuit emulator. Processors have become far too fast and complex. Moreover, pipeline architectures hide important code-execution details, partly because memory accesses on the bus can be ordered differently from code execution, and particularly because of internal caching of instruction streams. It is not always possible to correlate external bus activity to internal processor instruction execution, except at a rather coarse level.

Some of the challenges you will encounter while debugging Linux kernel code are:

• Linux kernel code is highly optimized for speed of execution in many areas.

• Compilers use optimization techniques that complicate the correlation of C source to actual machine instruction flow. Inline functions are a good example of this.

• Single-stepping through compiler optimized code often produces unusual and unexpected results.

• Virtual memory isolates user space memory from kernel memory and can make various debugging scenarios especially difficult.

• Some code cannot be stepped through with traditional debuggers.

• Startup code can be especially difficult because of its proximity to the hardware and the limited resources available (for example, no console, limited memory mapping, and so on).

The Linux kernel has matured into a very high-performance operating system capable of competing with the best commercial operating systems. Many areas within the kernel do not lend themselves to easy analysis by simply reading the source code. Knowledge of the architecture and detailed design are often necessary to understand the code flow in a particular area. Several good books are available that describe the kernel design in detail. Refer to Section 14.6.1, 'Suggestions for Additional Reading,' for recommendations.

GCC is an optimizing compiler. By default, the Linux kernel is compiled with the -O2 compiler flag. This enables many optimization algorithms that can change the fundamental structure and order of your code.[89] For example, the Linux kernel makes heavy use of inline functions. Inline functions are small functions declared with the inline keyword, which results in the function being included directly in the execution thread instead of generating a function call and the associated overhead.[90] Inline functions require a minimum of -O1 optimization level. Therefore, you cannot turn off optimization, which would be desirable for easier debugging.

In many areas within the Linux kernel, single-stepping through code is difficult or impossible. The most obvious examples are code paths that modify the virtual memory settings. When your application makes a system call that results in entry into the kernel, this results in a change in address space as seen by the process. In fact, any transition that involves a processor exception changes the operational context and can be difficult or impossible to single-step through.

14.2. Using KGDB for Kernel Debugging

Two popular methods enable symbolic source-level debugging within the Linux kernel:

• Using KGDB as a remote gdb agent

• Using a hardware JTAG probe to control the processor

We cover JTAG debugging in Section 14.4, 'Hardware-Assisted Debugging.'

KGDB (Kernel GDB) is a set of Linux kernel patches that provide an interface to gdb via its remote serial protocol. KGDB implements a gdb stub that communicates to a cross-gdb running on your host development workstation. Until very recently, KGDB on the target required a serial connection to the development host. Some targets support KGDB connection via Ethernet, although this is relatively new. Complete support for KGDB is still not in the mainline kernel.org kernel. You need to port KGDB to your chosen target or obtain an embedded Linux distribution for your chosen architecture and platform that contains KGDB support. Most embedded Linux