14.5. When It Doesn't Boot
One of the most frequently asked questions on the various mailing lists that serve embedded Linux goes something like this:
Thus starts the long and sometimes frustrating learning curve of embedded Linux! Many things that can go wrong could lead to this common failure. With some knowledge and a JTAG debugger, there are ways to determine what went awry.
14.5.1. Early Serial Debug Output
The first tool you might have available is CONFIG_SERIAL_TEXT_DEBUG. This Linux kernel-configuration option adds support for debug messages very early in the boot process. At the present time, this feature is limited to the PowerPC architecture, but nothing prevents you from duplicating the functionality in other architectures. Listing 14-22 provides an example of this feature in use on a PowerPC target using the U-Boot bootloader.
Listing 14-22. Early Serial Text Debug
## Booting image at 00200000 ...
Image Name: Linux-2.6.14
Created: 2005-12-19 22:24:03 UTC
Image Type: PowerPC Linux Kernel Image (gzip compressed)
Data Size: 607149 Bytes = 592.9 kB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
Uncompressing Kernel Image ... OK
id mach(): done <== Start of messages enabled by
MMU:enter <== CONFIG_SERIAL_TEXT_DEBUG
MMU:hw init
MMU:mapin
MMU:setio
MMU:exit
setup_arch: enter
setup_arch: bootmem
arch: exit
arch: real exit
Using this feature, you can often tell where your board is getting stuck during the boot process. Of course, you can add your own early debug messages in other places in the kernel. Here is an example of its usage found in .../arch/ppc/mm/init.c :
/* Map in all of RAM starting at KERNELBASE */
if (ppc_md.progress)
ppc_md.progress('MMU:mapin', 0x301);
mapin_ram();
The AMCC Yosemite platform is an excellent example of this infrastructure. Consult the following files in the Linux source tree[99] for details of how this debugging system is implemented:
| File | Function | Purpose |
|---|---|---|
| gen550_dbg.c | gen550_init | Serial port setup, called by yosemite.c platform-initialization file |
| gen550_dbg.c | gen550_progress | Low-level serial output routine |
| ibm44x_common.c | ibm44x_platform_init | Binds platform-specific progress routine to generic ppc machine-dependent infrastructure |
14.5.2. Dumping the printk Log Buffer
When we discussed printk debugging in Section 14.3.6, we pointed out some of the limitations of this method. printk itself is a very robust implementation. One of its shortcomings is that you can't see any printk messages until later in the boot sequence when the console device has been initialized. Very often, when your board hangs on boot, quite a few messages are stuck in the printk buffer. If you know where to find them, you can often pinpoint the exact problem that is causing the boot to hang. Indeed, many times you will discover that the kernel has encountered an error that led to a call to panic(). The output from panic() has likely been dumped into the printk buffer, and you can often pinpoint the exact line of offending code.
This is best accomplished with a JTAG debugger, but it is still possible to use a bootloader and its memory dump capability to display the contents of the printk buffer after a reset. Some corruption of memory contents might occur as a result of the reset, but log buffer text is usually very readable.
The actual buffer where printk stores its message text is declared in the printk source file .../kernel/printk.c.
static char __log_buf[__LOG_BUF_LEN];
We can easily determine the linked location of this buffer from the Linux kernel map file System.map.
$ grep __log_buf System.map
c022e5a4 b __log_buf
