gdb-commands
Introduction
Core file is generated when program crashes, if it’s not turned off, see limitation for the current user by $ ulimit -a or set it with unlimited by $ ulimit -c unlimited to allow core generation.
ulimit reads /etc/security/limits.conf that controls the limitation for the system, like core file, max opened file etc
Frequently used command
1 | >run a b |
Inside Gdb
1 | $ gdb app_binary core_file |
how gdb load debug info
if you compile you application with '-g' option, compiler will write DWARF(debug info) to the objects, actually, it includes two sectionsdebug_info and debug_line sections
- debug_info save function ,variable
- debug_line map source line to machine code
How does these section save information, it uses DIE (debug info entry), each has lots attributes(key-value) to show the info
Each DIE has one or more attribute/value pairs
Each attribute has a name
– Describes meaning of attribute
– Value specified for each attribute
– Data format specified in attribute encoding
● Examples
– DW_AT_name – Name of object DIE describes
– DW_AT_location – Source location of object
– DW_AT_low_pc – Start address of object
– DW_AT_high_pc – End address of object
– DW_AT_type – Pointer to DIE describing type
DIE for variable
1c28: DW_TAG_variable
DW_AT_name : decode
DW_AT_decl_file : 1
DW_AT_decl_line : 1782
DW_AT_type : <0x657>
DW_AT_location : 0x24ed (location list)
show all dwarf info
$ readelf -w
core file location
By default, core file is generated at the current directory or /var/log/core, change it if needed following the below rules:
core file生成的地方是在/proc/sys/kernel/core_pattern文件定义的。
改动到生成到自己定义的目录的方法是：
echo "pattern" > /proc/sys/kernel/core_pattern
并且只有超级用户可以修改这两个文件。
"pattern"类似我们C语言打印字符串的格式，相关标识如下：
%%: 相当于%
%p: 相当于<pid>
%u: 相当于<uid>
%g: 相当于<gid>
%s: 相当于导致dump的信号的数字
%t: 相当于dump的时间
%h: 相当于hostname
%e: 相当于执行文件的名称
这时用如下命令设置生成的core file到系统/tmp目录下，并记录pid以及执行文件名
$ echo "/tmp/core-%e-%p" > /proc/sys/kernel/core_pattern
after such operations. core file should be generated when segment fault happens.
change this pattern live
# echo '/tmp/core_%e.%p.%t' | sudo tee /proc/sys/kernel/core_pattern
core file with name core.program.processid.timestamp
Or change it forever
edit /etc/sysctl.conf
kernel.core_pattern = /tmp/core_%e.%p.%t
$ systemctl restart systemd-sysctl
core file not generated
$ ulimit(output must be: unlimited)$ cat /proc/sys/kernel/core_pattern(the path must be writable)
Commands
Getting help
>help
>help class
>help command
Executing your program
# run with args
>run a b
# another way to run with args
>set args a b
>run
# run without args
>run
>kill
Symbol table
>info address s
(info address var_name; info address fun_name) where it is.
>info func [regex]
show names,types of defined functions regular pattern search all matched functions
>info var [regex]
show names types of global, static variables, not value of it!!
>info var ngx_show_version
All variables matching regular expression "ngx_show_version":
>info locals
Breakpoints and watchpoints
>break [file:] line
>break [file:] func
>break *addr (set breakpoint at address addr)
>break [above three] if expr
# program stops when watched var is written
>watch expr | watch var
# program stops when watched var is read
>rwatch expr (read)
# program stops when watched var is read or write
>awatch expr (read/or write)
>info break
>info watch
>delete [n]
>clear [file:]fun
>clear [file:]line
>disable [n]
>enable [n]
>ignore n count (ignore breakpoint n, count times) [n] can be number for breakpoint or watchpoint
Program stack
>backtrace (print trace of all frames in stack)
>frame [n] (select frame number n)
>info frame (describe selected frame)
>info args (arguments of selected frame)
>info locals (local variables of selected frame)
>info reg [rn] (register values [for regs rn] in selected:info reg rax or info reg r10)
Working files
>file [file] ( use file for both symbols and executable)
>core-file [file] ( read file as coredump;or discard)
>exce-file [file] ( use file as executable only; or discard)
>symbol-file [file]( use symbol table from file, or discard)
>add-sym file addr ( read additional symbols from file, dynamically loaded at addr.)
Execution control
>continue (continue running)
>step
>stepi (step by machine instructions rather than source lines)
>next
>nexti
>finish
>return [expr] (pop selected stack frame without executing [setting return value])
>signal num (resume execution with signal )
>jump line (resume execution at specified line number)
>jump *address
>set var=expr (evaluate expr without displaying it; use for altering program variables)
Display
>print [/f] [expr] (print a;print /x a;print /t a)
f can be
x hexadecimal
t binary
c character
>x [/Nuf] expr (examine memory at address expr; x/1bx 0x123;x/1hx 0x123;x/1wx 0x123)
N is the length you want to see!!
u can be
b individual bytes
h halfwords (two bytes)
w words (four bytes)
g giant words (eight bytes)
>disassem [addr] (display memory as machine instructions from addr)
>disassem /s function_name (display machine code as well as source code for this function)
Automatic display
>display [/f] expr (show value of expr each time program stops)
>undisplay n
>disable disp n
>enable disp n
>info display
Source files
>dir names (add directory names to front of source path as prefix)
>dir (clear source path)
>show dir
>list (show next ten lines of source)
>list - (show previous ten lines)
>list line
>list func
debug multiple processes
when parent folks the child, you can use this method to debug the child, switch to child process, by default, gdb does not follow!
NOTE:
- If you want gdb follow, make sure the followed process can debug and has symbol
- DO NOT follow if run execxx() after fork() as you have nothing to know the command to run, does it have debug symbol or not
(make sure set breakpoint or watch before running)gdb>show follow-fork-mode gdb>set follow-fork-mode child gdb>set follow-fork-mode parent
but if there are more child process want to debug use sleep in that process and attach it to debug first change code add sleep
#ps -ef | grep xx (to see the child)
gdb>attach child_process_id
gdb>b function_name
gdb>c
multiple threads
gdb>info threads
gdb>thread 2
show mapped shared memory
gdb>info proc all
show offset of a given field
gdb> p &((struct _IO_proc_file *)0)->next
call a function in gdb
gdb>call printf("hello")
(output is the return value)
test.c
#include <stdio.h>
#include <stdlib.h>
struct rule {
int start;
int end;
};
void f(int a) {
printf("%d\n", a);
}
void f2(struct *rule) {
printf("%d %d\n", rule->start, rule->end);
}
int main() {
return 0;
}
$ gcc -d -o test test.c
$ gdb test
> b main
> r
>set $var=12
>call f($var)
>call f(12)
>set $ptr=malloc(sizeof(struct rule))
>p $ptr
>set ((struct rule*)$ptr)->start=12
>set ((struct rule*)$ptr)->end=10
> p *((struct rule*)$ptr)
>call f2($ptr)
>set $ptr=malloc(4)
>set *((int*)$ptr)=12
>p *((int*)$ptr)
gdb debug c++
set breakpoint on a member function
gdb>b class::function
show fields/members of the given class/struct instance
gdb>ptype class_instance
show attributes of a given instance
gdb>p *class_pointer
show private attribute of a given class instance
(you can show all attributes[public, protected, private] by gdb])
gdb>p class_pointer->private_attribute
show global variable with namespace
gdb> p 'name_space:log_level'
disassemble c++ function
gdb>disassem 'c++ function signature without return type'
show function signature
gdb>info functions xxx
ignore SIGPIPE in gdb
SIGPIPE happens when read/write a closed socket.
gdb>handle SIGPIPE nostop
gdb>handle SIGPIPE nostop noprint pass
how to fix no such file or directory
$cdir compiled path when building binary, get its value
gdb> info source
$cwd working path when debugging the binary, get its value by
gdb> pwd
gdb> list
No such file or directory
First you need to let gdb store the source path info by '-g', later on put the source code at proper path.
gdb saves the file path with absolute path, search it when debugging
$gcc -g -o test /tmp/a.c
gdb saves the file path with relative path, search it by adding compiled path or current working path
$gcc -g -o test a.c
$gdb test
gdb>list
if test build with absolute path, if not found, search compiled path, then working path
if test build with relative path, first check $cdir/file_name, then $cwd/file_name
if not found 'No such file or directory'
Sometime you only get the binary file, first check
$ objdump -WL test
# aa: file format elf64-x86-64
#
# Decoded dump of debug contents of section .debug_line:
#
# CU: /tmp/a.c:
# File name Line number Starting address
# a.c 5 0x400526
# a.c 6 0x40052e
# a.c 8 0x400535
# a.c 9 0x400549
# a.c 10 0x40054e
#
# CU(compiled unit) shows gdb saves absolute path in debuginfo
$ objdump -WL test
# aa: file format elf64-x86-64
#
# Decoded dump of debug contents of section .debug_line:
#
# CU: a.c:
# File name Line number Starting address
# a.c 5 0x400526
# a.c 6 0x40052e
# a.c 8 0x400535
# a.c 9 0x400549
# a.c 10 0x40054e
#
# CU(compiled unit) shows gdb saves relative path in debuginfo
# In this case you also need to know the compiled path
$ objdump -W aa | grep DW_AT_comp_dir
# <15> DW_AT_comp_dir : (indirect string, offset: 0x38): /tmp
# DW_AT_comp_dir DW_FORM_strp
#
# As you can see, the compiled path is /tmp
# If you debug the binary on your compliing machine, and you don't move you source code
# every thing is fine!!!
#
# BUT if you get the binary from other guys or you move the source code, it will shows
# 'No such file or directory!'
# 1. compiled with absolute path
$ gdb
gdb>set substitute-path $from $to
# let's say you moved a.c from /tmp to /home/
$ gdb test
gdb> set substitute-path /tmp /home
# 2. binary file saves with relative path
$ gdb
gdb> dir $new_path
debug with separate debug files.
GDB supports two ways of specifying the separate debug info file:
- The executable contains a debug link that specifies the name of the separate debug info file(.gnu_debuglink)
- The executable contains a build ID, a unique bit string that is also present in the corresponding debug info file(the build is most stored at .note.gnu.build-id)
# check the separate debug info file with command
$ readelf -n /lib/x86_64-linux-gnu/libc-2.23.so | grep BUILD_ID -C 2
Displaying notes found at file offset 0x00000270 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 40572882c66d064f9e4134cc94e4127798aad742
# check the debug link
$ readelf -x.gnu_debuglink /lib/x86_64-linux-gnu/libc-2.23.so
Depending on the way the debug info file is specified, GDB uses two different methods of looking for the debug file
- For the “debug link” method, GDB looks up the named file in the directory of the executable file, then in a subdirectory of that directory named `.debug’, and finally under the global debug directory, in a subdirectory whose name is identical to the leading directories of the executable’s absolute file name.
- For the “build ID” method, GDB looks in the .build-id’ subdirectory of the global debug directory for a file named `nn/nnnnnnnn.debug‘, where nn are the first 2 hex characters of the build ID bit string, and nnnnnnnn are the rest of the bit string
Example
suppose you ask GDB to debug /usr/bin/ls, which has a debug link that specifies the file ls.debug, and a build ID whose value in hex is abcdef1234. If the global debug directory is /usr/lib/debug, then GDB will look for the following debug information files, in the indicated order:
# global debug dir: /usr/lib/debug by default
/usr/lib/debug/.build-id/ab/cdef1234.debug
/usr/bin/ls.debug
/usr/bin/.debug/ls.debug
/usr/lib/debug/usr/bin/ls.debug
Example to use build-id not from default path
>gdb
>set debug-file-directory /tmp/lib/debug
>file /usr/bin/nginx
>core-file core.nginx
>bt
detach symbol from binary(compiled with -g)
# detach symbol to a separate file, foo still has symbol
$ objcopy --only-keep-debug foo foo.debug
# strip symbol from foo, foo has no symbol now
# Remove debugging symbols only
$ strip -g foo
# strip all symbols!!!
$ strip foo
# attach symbol back to binary
$ objcopy --add-gnu-debuglink=foo.debug foo
watchpoints is deleted by gdb automatically
gdb will delete watchpoints if out of scope of variable used by watchpoints, this usually happens
when you add a watchpoint on a local variable, to prevent gdb delete such watchpoints, add -l/-location
to the watchpoint, but with this type of watchpoint, gdb will pick up changes that other functions make to that same address on the stack, so you can add the qualification if $_caller_is(“your_caller_func”, 0)
gdb> watch -location $local_var
# better way to use
gdb> watch -location $local_var if $_caller_is("caller_function", 0)
run gdb command automatically when launch gdb
# create a file ~/.gdbinit
# put gdb command in the file each line
# then start gdb
$ gdb
pax on application
1 | paxctl is a tool that allows PaX flags to be modified on a per-binary |
use glibc source code in gdb
1 | most of time, you application link with glibc dynamic! |
gdb with lots of question marks even with symbol table loaded
1 |
|