First we start gdb, with the location of the mod_perl executable and the core file as the arguments:

panic% gdb /home/httpd/httpd_perl/bin/httpd_perl /home/httpd/perl/core

To see the backtrace, execute the where or bt commands:

(gdb) where
#0  0x4039f781 in crash_now_for_real (
    suicide_message=0x403a0120 "Cannot stand this life anymore")
    at DumpCore.xs:10
#1  0x4039f7a3 in crash_now (
    suicide_message=0x403a0120 "Cannot stand this life anymore",
    attempt_num=42) at DumpCore.xs:17
#2  0x4039f824 in XS_Debug_ _DumpCore_segv (cv=0x84ecda0)
    at DumpCore.xs:26
#3  0x401261ec in Perl_pp_entersub ( )
   from /usr/lib/perl5/5.6.1/i386-linux/CORE/
#4  0x00000001 in ?? ( )

Notice that only the symbols from the DumpCore.xs file are available (plus Perl_pp_entersub from, since by default Debug::DumpCore always compiles itself with the -g flag. However, we cannot see the rest of the trace, because our Perl and mod_perl libraries and Apache server were built without the debug symbols. We need to recompile them all with the debug symbols, as explained earlier in this chapter.

Then we repeat the process of starting the server, issuing a request, and getting the core file, after which we run gdb again against the executable and the dumped core file:

panic% gdb /home/httpd/httpd_perl/bin/httpd_perl /home/httpd/perl/core

Now we can see the whole backtrace:

(gdb) bt
#0  0x40448aa2 in crash_now_for_real (
    suicide_message=0x404499e0 "Cannot stand this life anymore")
    at DumpCore.xs:10
#1  0x40448ac9 in crash_now (
    suicide_message=0x404499e0 "Cannot stand this life anymore",
    attempt_num=42) at DumpCore.xs:17
#2  0x40448bd1 in XS_Debug_ _DumpCore_segv (my_perl=0x8133b60, cv=0x861d1fc)
    at DumpCore.xs:26
#3  0x4011d5d4 in Perl_pp_entersub (my_perl=0x8133b60) at pp_hot.c:2773
#4  0x400fb439 in Perl_runops_debug (my_perl=0x8133b60) at dump.c:1398
#5  0x400a6288 in S_call_body (my_perl=0x8133b60, myop=0xbffff160, is_eval=0)
    at perl.c:2045
#6  0x400a5e34 in Perl_call_sv (my_perl=0x8133b60, sv=0x85d696c, flags=4)
    at perl.c:1963
#7  0x0808a6e3 in perl_call_handler (sv=0x85d696c, r=0x860bf54, args=0x0)
    at mod_perl.c:1658
#8  0x080895f2 in perl_run_stacked_handlers (hook=0x8109c47 "PerlHandler",
    r=0x860bf54, handlers=0x82e5c4c) at mod_perl.c:1371
#9  0x080864d8 in perl_handler (r=0x860bf54) at mod_perl.c:897
#10 0x080d2560 in ap_invoke_handler (r=0x860bf54) at http_config.c:517
#11 0x080e6796 in process_request_internal (r=0x860bf54) at http_request.c:1308
#12 0x080e67f6 in ap_process_request (r=0x860bf54) at http_request.c:1324
#13 0x080ddba2 in child_main (child_num_arg=0) at http_main.c:4595
#14 0x080ddd4a in make_child (s=0x8127ec4, slot=0, now=1028133659)
#15 0x080ddeb1 in startup_children (number_to_start=4) at http_main.c:4792
#16 0x080de4e6 in standalone_main (argc=2, argv=0xbffff514) at http_main.c:5100
#17 0x080ded04 in main (argc=2, argv=0xbffff514) at http_main.c:5448
#18 0x40215082 in _ _libc_start_main ( ) from /lib/i686/

Reading the trace from bottom to top, we can see that it starts with Apache functions, moves on to the mod_perl and then Perl functions, and finally calls functions from the Debug::DumpCore package. At the top we can see the crash_now_for_real( ) function, which was the one that caused the segmentation fault; we can also see that the faulty code was at line 10 of the DumpCore.xs file. And indeed, if we look at that line number we can see the reason for the segfault—the dereferencing of the NULL pointer:

9: int *p = NULL;
  10: printf("%d", *p); /* cause a segfault */

In our example, we knew what Perl script had caused the segmentation fault. In the real world, it is likely that you'll have only the core file, without any clue as to which handler or script has triggered it. The special curinfo gdb macro can help:

panic% gdb /home/httpd/httpd_perl/bin/httpd_perl /home/httpd/perl/core
(gdb) source mod_perl-1.xx/.gdbinit
(gdb) curinfo

Start the gdb debugger as before. .gdbinit, the file with various useful gdb macros, is located in the source tree of mod_perl. We use the gdb source function to load these macros, and when we run the curinfo macro we learn that the core was dumped when /home/httpd/perl/ was executing the code at line 9.

These are the bits of information that are important in order to reproduce and resolve a problem: the filename and line number where the fault occurred (the faulty function is Debug::DumpCore::segv( ) in our case) and the actual line where the segmentation fault occurred (the printf("%d", *p) call in XS code). The former is important for problem reproducing, since it's possible that if the same function was called from a different script the problem wouldn't show up (not the case in our example, where using a dereferenced NULL pointer will always cause a segmentation fault).