gdb-commands
Introduction
Core file is generated when program crashes, if it’s not turned off, see limitation for the current user by $ ulimit -a or set it with unlimited by $ ulimit -c unlimited to allow core generation.
ulimit reads /etc/security/limits.conf that controls the limitation for the system, like core file, max opened file etc
Frequently used command
1 | >run a b |
Inside Gdb
1 | $ gdb app_binary core_file |
how gdb load debug info
if you compile you application with '-g' option, compiler will write DWARF(debug info) to the objects, actually, it includes two sectionsdebug_info and debug_line sections
- debug_info save function ,variable
- debug_line map source line to machine code
How does these section save information, it uses DIE (debug info entry), each has lots attributes(key-value) to show the info
Each DIE has one or more attribute/value pairs
Each attribute has a name
– Describes meaning of attribute
– Value specified for each attribute
– Data format specified in attribute encoding
● Examples
– DW_AT_name – Name of object DIE describes
– DW_AT_location – Source location of object
– DW_AT_low_pc – Start address of object
– DW_AT_high_pc – End address of object
– DW_AT_type – Pointer to DIE describing type
DIE for variable
1c28: DW_TAG_variable
DW_AT_name : decode
DW_AT_decl_file : 1
DW_AT_decl_line : 1782
DW_AT_type : <0x657>
DW_AT_location : 0x24ed (location list)
show all dwarf info
$ readelf -w
core file location
By default, core file is generated at the current directory or /var/log/core, change it if needed following the below rules:
core file生成的地方是在/proc/sys/kernel/core_pattern文件定义的。
改动到生成到自己定义的目录的方法是：
echo "pattern" > /proc/sys/kernel/core_pattern
并且只有超级用户可以修改这两个文件。
"pattern"类似我们C语言打印字符串的格式，相关标识如下：
%%: 相当于%
%p: 相当于<pid>
%u: 相当于<uid>
%g: 相当于<gid>
%s: 相当于导致dump的信号的数字
%t: 相当于dump的时间
%h: 相当于hostname
%e: 相当于执行文件的名称
这时用如下命令设置生成的core file到系统/tmp目录下，并记录pid以及执行文件名
$ echo "/tmp/core-%e-%p" > /proc/sys/kernel/core_pattern
after such operations. core file should be generated when segment fault happens.
change this pattern live
# echo '/tmp/core_%e.%p.%t' | sudo tee /proc/sys/kernel/core_pattern
core file with name core.program.processid.timestamp
Or change it forever
edit /etc/sysctl.conf
kernel.core_pattern = /tmp/core_%e.%p.%t
$ systemctl restart systemd-sysctl
core file not generated
$ ulimit(output must be: unlimited)$ cat /proc/sys/kernel/core_pattern(the path must be writable)
Commands
Getting help
>help
>help class
>help command
Executing your program
# run with args
>run a b
# another way to run with args
>set args a b
>run
# run without args
>run
>kill
Symbol table
>info address s
(info address var_name; info address fun_name) where it is.
>info func [regex]
show names,types of defined functions regular pattern search all matched functions
>info var [regex]
show names types of global, static variables, not value of it!!
>info var ngx_show_version
All variables matching regular expression "ngx_show_version":
>info locals
>ptype struct task_struct
show definition of task_struct
>ptype /o struct task_struct
show definition of task_struct with offset and total size
Breakpoints and watchpoints
>break [file:] line
>break [file:] func
>break *addr (set breakpoint at address addr)
>break [above three] if expr
# program stops when watched var is written
>watch expr | watch var
# program stops when watched var is read
>rwatch expr (read)
# program stops when watched var is read or write
>awatch expr (read/or write)
>info break
>info watch
>delete [n]
>clear [file:]fun
>clear [file:]line
>disable [n]
>enable [n]
>ignore n count (ignore breakpoint n, count times) [n] can be number for breakpoint or watchpoint
Program stack
>backtrace (print trace of all frames in stack)
>frame [n] (select frame number n)
>info frame (describe selected frame)
>info args (arguments of selected frame)
>info locals (local variables of selected frame)
>info reg [rn] (register values [for regs rn] in selected:info reg rax or info reg r10)
Working files
>file [file] (use file for both symbols and executable)
>core-file [file] ( read file as coredump;or discard)
>exce-file [file] ( use file as executable only; or discard)
>symbol-file [file]( use symbol table from file, or discard)
Loads symbol table information from a specified file, replacing any previously loaded symbols.
Used to set the main symbol file for the debugging session.
>add-symbol-file addr ( read additional symbols from file, dynamically loaded at addr.)
Reads additional symbol table information from a specified file.
Adds new symbol data to the existing symbol table without discarding previously loaded symbols. if existing symbol table already has a symbol, it's not added.
Typically used for loading symbols from dynamically loaded modules or shared libraries
Execution control
>continue (continue running)
>step
>stepi (step by machine instructions rather than source lines)
>next
>nexti
>finish
>return [expr] (pop selected stack frame without executing [setting return value])
>signal num (resume execution with signal )
>jump line (resume execution at specified line number)
>jump *address
>set var=expr (evaluate expr without displaying it; use for altering program variables)
Display
>print [/f] [expr] (print a;print /x a;print /t a)
f can be
x hexadecimal
t binary
c character
>x [/Nuf] expr (examine memory at address expr; x/1bx 0x123;x/1hx 0x123;x/1wx 0x123)
N is the length you want to see!!
u can be
b individual bytes
h halfwords (two bytes)
w words (four bytes)
g giant words (eight bytes)
>disassem [addr] (display memory as machine instructions from addr)
>disassem /s function_name (display machine code as well as source code for this function)
Automatic display
>display [/f] expr (show value of expr each time program stops)
>undisplay n
>disable disp n
>enable disp n
>info display
Source files
>dir names (add directory names to front of source path as prefix)
>dir (clear source path)
>show dir
>list (show next ten lines of source)
>list - (show previous ten lines)
>list line
>list func
debug multiple processes
when parent folks the child, you can use this method to debug the child, switch to child process, by default, gdb does not follow!
NOTE:
- If you want gdb follow, make sure the followed process can debug and has symbol.
- DO NOT follow if run execxx() after fork() as you have nothing to know the command to run, does it have debug symbol or not
(make sure set breakpoint or watch before running)gdb>show follow-fork-mode gdb>set follow-fork-mode child gdb>set follow-fork-mode parent
but if there are more child process want to debug use sleep in that process and attach it to debug first change code add sleep
#ps -ef | grep xx (to see the child)
gdb>attach child_process_id
gdb>b function_name
gdb>c
multiple threads
gdb>info threads
gdb>thread 2
show mapped shared memory
gdb>info proc all
show offset of a given field
gdb> p &((struct _IO_proc_file *)0)->next
call a function in gdb
gdb>call printf("hello")
(output is the return value)
test.c
#include <stdio.h>
#include <stdlib.h>
struct rule {
int start;
int end;
};
void f(int a) {
printf("%d\n", a);
}
void f2(struct *rule) {
printf("%d %d\n", rule->start, rule->end);
}
int main() {
return 0;
}
$ gcc -d -o test test.c
$ gdb test
> b main
> r
>set $var=12
>call f($var)
>call f(12)
>set $ptr=malloc(sizeof(struct rule))
>p $ptr
>set ((struct rule*)$ptr)->start=12
>set ((struct rule*)$ptr)->end=10
> p *((struct rule*)$ptr)
>call f2($ptr)
>set $ptr=malloc(4)
>set *((int*)$ptr)=12
>p *((int*)$ptr)
gdb debug c++
set breakpoint on a member function
gdb>b class::function
show fields/members of the given class/struct instance
gdb>ptype class_instance
show attributes of a given instance
gdb>p *class_pointer
show private attribute of a given class instance
(you can show all attributes[public, protected, private] by gdb])
gdb>p class_pointer->private_attribute
show global variable with namespace
gdb> p 'name_space:log_level'
disassemble c++ function
gdb>disassem 'c++ function signature without return type'
show function signature
gdb>info functions xxx
ignore SIGPIPE in gdb
SIGPIPE happens when read/write a closed socket.
gdb>handle SIGPIPE nostop
gdb>handle SIGPIPE nostop noprint pass
how to fix no such file or directory
$cdir compiled path when building binary, get its value
gdb> info source
$cwd working path when debugging the binary, get its value by
gdb> pwd
gdb> list
No such file or directory
First you need to let gdb store the source path info by '-g', later on put the source code at proper path.
gdb saves the file path with absolute path, search it when debugging
$gcc -g -o test /tmp/a.c
gdb saves the file path with relative path, search it by adding compiled path or current working path
$gcc -g -o test a.c
$gdb test
gdb>list
if test build with absolute path, if not found, search compiled path, then working path
if test build with relative path, first check $cdir/file_name, then $cwd/file_name
if not found 'No such file or directory'
Sometime you only get the binary file, first check
$ objdump -WL test
# aa: file format elf64-x86-64
#
# Decoded dump of debug contents of section .debug_line:
#
# CU: /tmp/a.c:
# File name Line number Starting address
# a.c 5 0x400526
# a.c 6 0x40052e
# a.c 8 0x400535
# a.c 9 0x400549
# a.c 10 0x40054e
#
# CU(compiled unit) shows gdb saves absolute path in debuginfo
$ objdump -WL test
# aa: file format elf64-x86-64
#
# Decoded dump of debug contents of section .debug_line:
#
# CU: a.c:
# File name Line number Starting address
# a.c 5 0x400526
# a.c 6 0x40052e
# a.c 8 0x400535
# a.c 9 0x400549
# a.c 10 0x40054e
#
# CU(compiled unit) shows gdb saves relative path in debuginfo
# In this case you also need to know the compiled path
$ objdump -W aa | grep DW_AT_comp_dir
# <15> DW_AT_comp_dir : (indirect string, offset: 0x38): /tmp
# DW_AT_comp_dir DW_FORM_strp
#
# As you can see, the compiled path is /tmp
# If you debug the binary on your compliing machine, and you don't move you source code
# every thing is fine!!!
#
# BUT if you get the binary from other guys or you move the source code, it will shows
# 'No such file or directory!'
# 1. compiled with absolute path
$ gdb
gdb>set substitute-path $from $to
# let's say you moved a.c from /tmp to /home/
$ gdb test
gdb> set substitute-path /tmp /home
# 2. binary file saves with relative path
$ gdb
gdb> dir $new_path
debug with separate debug files.
GDB supports two ways of specifying the separate debug info file:
- The executable contains a debug link that specifies the name of the separate debug info file(.gnu_debuglink)
- The executable contains a build ID, a unique bit string that is also present in the corresponding debug info file(the build is most stored at .note.gnu.build-id)
# check the separate debug info file with command
$ readelf -n /lib/x86_64-linux-gnu/libc-2.23.so | grep BUILD_ID -C 2
Displaying notes found at file offset 0x00000270 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 40572882c66d064f9e4134cc94e4127798aad742
# check the debug link
$ readelf -x.gnu_debuglink /lib/x86_64-linux-gnu/libc-2.23.so
Depending on the way the debug info file is specified, GDB uses two different methods of looking for the debug file
- For the “debug link” method, GDB looks up the named file in the directory of the executable file, then in a subdirectory of that directory named `.debug’, and finally under the global debug directory, in a subdirectory whose name is identical to the leading directories of the executable’s absolute file name.
- For the “build ID” method, GDB looks in the .build-id’ subdirectory of the global debug directory for a file named `nn/nnnnnnnn.debug‘, where nn are the first 2 hex characters of the build ID bit string, and nnnnnnnn are the rest of the bit string
Example
suppose you ask GDB to debug /usr/bin/ls, which has a debug link that specifies the file ls.debug, and a build ID whose value in hex is abcdef1234. If the global debug directory is /usr/lib/debug, then GDB will look for the following debug information files, in the indicated order:
# global debug dir: /usr/lib/debug by default
/usr/lib/debug/.build-id/ab/cdef1234.debug
/usr/bin/ls.debug
/usr/bin/.debug/ls.debug
/usr/lib/debug/usr/bin/ls.debug
Example to use build-id not from default path
>gdb
>set debug-file-directory /tmp/lib/debug
>file /usr/bin/nginx
>core-file core.nginx
>bt
detach symbol from binary(compiled with -g)
Symbol tables store key-value pairs that map names (like variables and functions) to their memory addresses, they serve basic purposes like verifying variable declarations,implementing type checking, and determining scope resolution, it’s always stored in binary file when you compile application, no need any option to enable it.
Debug information is more comprehensive than just symbol tables, it includes additional context like:
- Source code filenames
- Line numbers mapping code to source
- Variable types and their attributes
- Scope information
- Class structures (methods and members)
Debug information provides richer context specifically for debugging purposes, Debug symbols extend symbol tables with additional information
# generate binary with debuginfo and symbol tables, even without -g option, symbol tables are still stored in binary file
$ gcc -o hello hello.c -g
# --------------- check debuginfo with several tools ---------------
$ objdump -h hello | grep debug
27 .debug_aranges 00000030 0000000000000000 0000000000000000 0000303f 2**0
CONTENTS, READONLY, DEBUGGING, OCTETS
28 .debug_info 000000f3 0000000000000000 0000000000000000 0000306f 2**0
CONTENTS, READONLY, DEBUGGING, OCTETS
29 .debug_abbrev 000000b3 0000000000000000 0000000000000000 00003162 2**0
CONTENTS, READONLY, DEBUGGING, OCTETS
30 .debug_line 00000069 0000000000000000 0000000000000000 00003215 2**0
CONTENTS, READONLY, DEBUGGING, OCTETS
31 .debug_str 00000104 0000000000000000 0000000000000000 0000327e 2**0
CONTENTS, READONLY, DEBUGGING, OCTETS
32 .debug_line_str 0000001e 0000000000000000 0000000000000000 00003382 2**0
CONTENTS, READONLY, DEBUGGING, OCTETS
$ file hello
hello: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=9211bbda2c800fb9cbad99aea8a3936cf9689491, for GNU/Linux 3.2.0, with debug_info, not stripped
$ readelf -S hello | grep .debug
[28] .debug_aranges PROGBITS 0000000000000000 0000303f
[29] .debug_info PROGBITS 0000000000000000 0000306f
[30] .debug_abbrev PROGBITS 0000000000000000 00003162
[31] .debug_line PROGBITS 0000000000000000 00003215
[32] .debug_str PROGBITS 0000000000000000 0000327e
[33] .debug_line_str PROGBITS 0000000000000000 00003382
# --------------- check debuginfo with several tools ---------------
# --------------- check symbol tables ---------------
$ readelf -s hello
Symbol table '.dynsym' contains 8 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 FUNC GLOBAL DEFAULT UND _[...]@GLIBC_2.34 (2)
2: 0000000000000000 0 NOTYPE WEAK DEFAULT UND _ITM_deregisterT[...]
3: 0000000000000000 0 FUNC GLOBAL DEFAULT UND puts@GLIBC_2.2.5 (3)
4: 0000000000000000 0 FUNC GLOBAL DEFAULT UND [...]@GLIBC_2.2.5 (3)
5: 0000000000000000 0 NOTYPE WEAK DEFAULT UND __gmon_start__
6: 0000000000000000 0 NOTYPE WEAK DEFAULT UND _ITM_registerTMC[...]
7: 0000000000000000 0 FUNC WEAK DEFAULT UND [...]@GLIBC_2.2.5 (3)
Symbol table '.symtab' contains 39 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 FILE LOCAL DEFAULT ABS Scrt1.o
2: 000000000000038c 32 OBJECT LOCAL DEFAULT 4 __abi_tag
3: 0000000000000000 0 FILE LOCAL DEFAULT ABS crtstuff.c
4: 00000000000010b0 0 FUNC LOCAL DEFAULT 16 deregister_tm_clones
5: 00000000000010e0 0 FUNC LOCAL DEFAULT 16 register_tm_clones
6: 0000000000001120 0 FUNC LOCAL DEFAULT 16 __do_global_dtors_aux
7: 0000000000004014 1 OBJECT LOCAL DEFAULT 26 completed.0
8: 0000000000003db8 0 OBJECT LOCAL DEFAULT 22 __do_global_dtor[...]
9: 0000000000001160 0 FUNC LOCAL DEFAULT 16 frame_dummy
10: 0000000000003db0 0 OBJECT LOCAL DEFAULT 21 __frame_dummy_in[...]
11: 0000000000000000 0 FILE LOCAL DEFAULT ABS h.c
12: 0000000000000000 0 FILE LOCAL DEFAULT ABS crtstuff.c
13: 0000000000002130 0 OBJECT LOCAL DEFAULT 20 __FRAME_END__
14: 0000000000000000 0 FILE LOCAL DEFAULT ABS
15: 0000000000003dc0 0 OBJECT LOCAL DEFAULT 23 _DYNAMIC
16: 000000000000202c 0 NOTYPE LOCAL DEFAULT 19 __GNU_EH_FRAME_HDR
17: 0000000000003fb0 0 OBJECT LOCAL DEFAULT 24 _GLOBAL_OFFSET_TABLE_
18: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __libc_start_mai[...]
19: 0000000000000000 0 NOTYPE WEAK DEFAULT UND _ITM_deregisterT[...]
20: 0000000000004000 0 NOTYPE WEAK DEFAULT 25 data_start
21: 0000000000000000 0 FUNC GLOBAL DEFAULT UND puts@GLIBC_2.2.5
22: 0000000000004014 0 NOTYPE GLOBAL DEFAULT 25 _edata
23: 0000000000001169 26 FUNC GLOBAL DEFAULT 16 hello_function
24: 00000000000011c8 0 FUNC GLOBAL HIDDEN 17 _fini
25: 0000000000000000 0 FUNC GLOBAL DEFAULT UND printf@GLIBC_2.2.5
26: 0000000000004010 4 OBJECT GLOBAL DEFAULT 25 global_var
27: 0000000000004000 0 NOTYPE GLOBAL DEFAULT 25 __data_start
28: 0000000000000000 0 NOTYPE WEAK DEFAULT UND __gmon_start__
29: 0000000000004008 0 OBJECT GLOBAL HIDDEN 25 __dso_handle
30: 0000000000002000 4 OBJECT GLOBAL DEFAULT 18 _IO_stdin_used
31: 0000000000004018 0 NOTYPE GLOBAL DEFAULT 26 _end
32: 0000000000001080 38 FUNC GLOBAL DEFAULT 16 _start
33: 0000000000004014 0 NOTYPE GLOBAL DEFAULT 26 __bss_start
34: 0000000000001183 67 FUNC GLOBAL DEFAULT 16 main
35: 0000000000004018 0 OBJECT GLOBAL HIDDEN 25 __TMC_END__
36: 0000000000000000 0 NOTYPE WEAK DEFAULT UND _ITM_registerTMC[...]
37: 0000000000000000 0 FUNC WEAK DEFAULT UND __cxa_finalize@G[...]
38: 0000000000001000 0 FUNC GLOBAL HIDDEN 12 _init
$ objdump --syms hello
hello: file format elf64-x86-64
SYMBOL TABLE:
0000000000000000 l df *ABS* 0000000000000000 Scrt1.o
000000000000038c l O .note.ABI-tag 0000000000000020 __abi_tag
0000000000000000 l df *ABS* 0000000000000000 crtstuff.c
00000000000010b0 l F .text 0000000000000000 deregister_tm_clones
00000000000010e0 l F .text 0000000000000000 register_tm_clones
0000000000001120 l F .text 0000000000000000 __do_global_dtors_aux
0000000000004014 l O .bss 0000000000000001 completed.0
0000000000003db8 l O .fini_array 0000000000000000 __do_global_dtors_aux_fini_array_entry
0000000000001160 l F .text 0000000000000000 frame_dummy
0000000000003db0 l O .init_array 0000000000000000 __frame_dummy_init_array_entry
0000000000000000 l df *ABS* 0000000000000000 h.c
0000000000000000 l df *ABS* 0000000000000000 crtstuff.c
0000000000002130 l O .eh_frame 0000000000000000 __FRAME_END__
0000000000000000 l df *ABS* 0000000000000000
0000000000003dc0 l O .dynamic 0000000000000000 _DYNAMIC
000000000000202c l .eh_frame_hdr 0000000000000000 __GNU_EH_FRAME_HDR
0000000000003fb0 l O .got 0000000000000000 _GLOBAL_OFFSET_TABLE_
0000000000000000 F *UND* 0000000000000000 __libc_start_main@GLIBC_2.34
0000000000000000 w *UND* 0000000000000000 _ITM_deregisterTMCloneTable
0000000000004000 w .data 0000000000000000 data_start
0000000000000000 F *UND* 0000000000000000 puts@GLIBC_2.2.5
0000000000004014 g .data 0000000000000000 _edata
0000000000001169 g F .text 000000000000001a hello_function
00000000000011c8 g F .fini 0000000000000000 .hidden _fini
0000000000000000 F *UND* 0000000000000000 printf@GLIBC_2.2.5
0000000000004010 g O .data 0000000000000004 global_var
0000000000004000 g .data 0000000000000000 __data_start
0000000000000000 w *UND* 0000000000000000 __gmon_start__
0000000000004008 g O .data 0000000000000000 .hidden __dso_handle
0000000000002000 g O .rodata 0000000000000004 _IO_stdin_used
0000000000004018 g .bss 0000000000000000 _end
0000000000001080 g F .text 0000000000000026 _start
0000000000004014 g .bss 0000000000000000 __bss_start
0000000000001183 g F .text 0000000000000043 main
0000000000004018 g O .data 0000000000000000 .hidden __TMC_END__
0000000000000000 w *UND* 0000000000000000 _ITM_registerTMCloneTable
0000000000000000 w F *UND* 0000000000000000 __cxa_finalize@GLIBC_2.2.5
0000000000001000 g F .init 0000000000000000 .hidden _init
# --------------- check symbol tables ---------------
# detach debuginfo and symbol tables to a separate file, foo still has them even after saved to a separate file
$ objcopy --only-keep-debug foo foo.debug
# strip debuginfo from foo, foo has no debuginfo now
# Remove debugging info only
$ strip -g foo
# strip all(debuginfo and symbol tables)
$ strip foo
watchpoints is deleted by gdb automatically
gdb will delete watchpoints if out of scope of variable used by watchpoints, this usually happens
when you add a watchpoint on a local variable, to prevent gdb delete such watchpoints, add -l/-location
to the watchpoint, but with this type of watchpoint, gdb will pick up changes that other functions make to that same address on the stack, so you can add the qualification if $_caller_is(“your_caller_func”, 0)
gdb> watch -location $local_var
# better way to use
gdb> watch -location $local_var if $_caller_is("caller_function", 0)
run gdb command automatically when launch gdb
# create a file ~/.gdbinit
# put gdb command in the file each line
# then start gdb
$ gdb
pax on application
1 | paxctl is a tool that allows PaX flags to be modified on a per-binary |
use glibc source code in gdb
1 | most of time, you application link with glibc dynamic! |
gdb with lots of question marks even with symbol table loaded
1 |
|