Embedded Linux Primer: A Practical, Real-World Approach

7     set $mp=(struct module *)((char *)$m->next - (char *)4)
8     printf '0x%08X	%s
', $mp, $mp->name
9     if ($mp->list->next == &modules)
10       set $done=1
11     end
12     set $m=$m->next
13   end
14 end
15
16 document lsmod
17 List the loaded kernel modules and their start addresses
18 end

This simple loop starts with the kernel's global variable module. This variable is a struct list_head that marks the start of the linked list of loadable modules. The only complexity is the same as that described in Listing 14-15. We must subtract an offset from the struct list_head pointer to point to the top of the struct module. This is performed in line 7. This macro produces a simple listing of modules containing the address of the struct module and the module's name. Here is an example of its use:

(gdb) lsmod
Address         Module
0xD1012A80      ip_conntrack_tftp
0xD10105A0      ip_conntrack
0xD102F9A0      loop
(gdb) help lsmod
List the loaded kernel modules and their start addresses
(gdb)

Macros such as the ones presented here are very powerful debugging aids. You can create macros in a similar fashion to display anything in the kernel that lends itself to easy access, especially the major data structures maintained as linked lists. Examples include process memory map information, module information, file system information, and timer lists and so on. The information presented here should get you started.

14.3.5. Debugging Loadable Modules

The most common reason for using KGDB is to debug loadable kernel modules, that is, device drivers. One of the more convenient features of loadable modules is that, under most circumstances, it is not necessary to reboot the kernel for each new debugging session. You can start a debugging session, make some changes, recompile, and reload the module without the hassle and delay of a complete kernel reboot.

The complication associated with debugging loadable modules is in gaining access to the symbolic debug information contained in the module's object file. Because loadable modules are dynamically linked when they are loaded into the kernel, the symbolic information contained in the object file is useless until the symbol table is adjusted.

Recall from our earlier examples how we invoke gdb for a kernel debugging session:

$ ppc_4xx-gdb vmlinux

This launches a gdb debugging session on your host, and reads the symbol information from the Linux kernel ELF file vmlinux. Of course, you will not find symbols for any loadable modules in this file. Loadable modules are separate compilation units and are linked as individual standalone ELF objects. Therefore, if we intend to perform any source-level debugging on a loadable module, we need to load its debug symbols from the ELF file. gdb provides this capability in its add-symbol-file command.

The add-symbol-file command loads symbols from the specified object file, assuming that the module itself has already been loaded. However, we are faced with the chicken-and-egg syndrome. We don't have any symbol information until the loadable module has been loaded into the kernel and the add-symbol-file command is issued to read in the module's symbol information. However, after the module has been loaded, it is too late to set breakpoints and debug the module's *_init and related functions because they have already executed.

The solution to this dilemma is to place a breakpoint in the kernel code that is responsible for loading the module, after it has been linked but before its initialization function has been called. This work is done by .../kernel/module.c. Listing 14-17 reproduces the relevant portions of module.c.

Listing 14-17. module.c : Module Initialization

...
1901 down(&notify_mutex);
1902 notifier_call_chain(&module_notify_list, MODULE_STATE_COMING, mod);
1903 up(&notify_mutex);
1904
1905 /* Start the module */
1906 if (mod->init != NULL)
1907  ret = mod->init();
1908 if (ret < 0) {
1909  /* Init routine failed: abort. Try to protect us from
1910     buggy refcounters. */
1911   mod->state = MODULE_STATE_GOING;
...

We load the module using the modprobe utility, which was demonstrated in Listing 8-5 in Chapter 8, 'Device Driver Basics,' and looks like this:

$ modprobe loop

This command issues a special system call that directs the kernel to load the module. The module loading begins at sys_init_module() in module.c. After the module has been loaded into kernel memory and dynamically linked, control is passed to the module's _init function. This is shown in lines 1906 and 1907 of Listing 14-17. We place our breakpoint here. This enables us to add the symbol file to gdb and subsequently set breakpoints in the module. We demonstrate this process using the Linux kernel's loopback driver called loop.ko. This module has no dependencies on other modules and is reasonably easy to demonstrate.

Listing 14-18 shows the gdb commands to initiate this debugging session on loop.ko.

Listing 14-18. Initiate Module Debug Session: loop.ko

1 $ ppc-linux-gdb --silent vmlinux
2 (gdb) connect
3 breakinst () at arch/ppc/kernel/ppc-stub.c:825
4 825     }
5 Breakpoint 1 at 0xc0016b18: file kernel/panic.c, line 74.
6 Breakpoint 2 at 0xc005a8c8: file fs/buffer.c, line 296.
7 (gdb) b module.c:1907
8 Breakpoint 3 at 0xc003430c: file kernel/module.c, line 1907.
9 (gdb) c
10 Continuing.
11 >>>> Here we let the kernel finish booting
12      and then load the loop.ko module on the target
13