linux-memory-checker

Posted on Edited on In ,

Introduction

There are two powerful tools for memory check, one is valgrind, the other is address sanitizer, the difference is

  • valgrind, no need to compile your app, have big impact on performance(down 10-50 times), can’t work well with multiple process, but can detect leak in shared library.
  • address sanitizer, need to compile your app, has less impact for app run(down 2 times)

Always use address sanitizer if you can recomplie your application!

valgrind

valgrind error when runs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
--4636:0: aspacem <<< SHOW_SEGMENTS: out_of_memory (19 segments)
--4636:0: aspacem 1 segment names in 1 slots
--4636:0: aspacem freelist is empty
--4636:0: aspacem (0,4,2) /usr/lib/valgrind/memcheck-amd64-linux
--4636:0: aspacem   0: RSVN 0000000000-0003ffffff     64m ----- SmFixed
--4636:0: aspacem   1:      0004000000-0037ffffff    832m
--4636:0: aspacem   2: FILE 0038000000-003821efff 2224128 r-x-- d=0x802 i=404505  o=0       (0,4)
--4636:0: aspacem   3:      003821f000-003841efff 2097152
--4636:0: aspacem   4: FILE 003841f000-0038421fff   12288 rw--- d=0x802 i=404505  o=2224128 (0,4)
...
==4636==     Valgrind's memory management: out of memory:
==4636==        newSuperblock's request for 4194304 bytes failed.
==4636==           42,684,416 bytes have already been mmap-ed ANONYMOUS.
==4636==     Valgrind cannot continue.  Sorry.
==4636==
==4636==     There are several possible reasons for this.
==4636==     - You have some kind of memory limit in place.  Look at the
==4636==       output of 'ulimit -a'.  Is there a limit on the size of
==4636==       virtual memory or address space?
==4636==     - You have run out of swap space.
==4636==     - Valgrind has a bug.  If you think this is the case or you are
==4636==     not sure, please let us know and we'll try to fix it.
==4636==     Please note that programs can take substantially more memory than
==4636==     normal when running under Valgrind tools, eg. up to twice or
==4636==     more, depending on the tool.  On a 64-bit machine, Valgrind
==4636==     should be able to make use of up 32GB memory.  On a 32-bit
==4636==     machine, Valgrind should be able to use all the memory available
==4636==     to a single process, up to 4GB if that's how you have your
==4636==     kernel configured.  Most 32-bit Linux setups allow a maximum of
==4636==     3GB per process.
==4636==
==4636==     Whatever the reason, Valgrind cannot continue.  Sorry.

$ sysctl -w kernel.pax.softmode=1

$ valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose --log-file=valgrind-out.txt ./test

address sanitizer

it comes from google, and gcc supports it from 4.8, so make sure you gcc support it before compiling your program.

how to use it

1
2
3
4
-fsanitize=address    #开启地址越界检查功能
-fno-omit-frame-pointer  #开启后，可以出界更详细的错误信息
-fsanitize=leak   #开启内存泄露检查功能
可选择-O1或者更高的优化级别编译

this will build nginx with static asan, can NOT detect link in shared library.

build it in nginx

1
2
--with-cc-opt="-O0 -fsanitize=address -fno-omit-frame-pointer -fPIC -g $CC_OPT"
--with-ld-opt="-fsanitize=address -fno-omit-frame-pointer -Wl,-Bsymbolic-functions -fpic -Wl,-z,relro"

check result
sanitizer only prints result when program ends normally. (Hint: Do a kill or killall with normal SIGTERM of the daemons at the end).

  • $ killall nginx (if nginx runs as a daemon)
  • check file set by error_log in nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
==7852==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 320 byte(s) in 5 object(s) allocated from:
    #0 0x7f92a193b602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0x9f7eef in ngx_http_init_connection src/http/ngx_http_request.c:228
    #2 0x97fdb3 in ngx_event_accept src/event/ngx_event_accept.c:350
    #3 0x9aea6f in ngx_epoll_process_events src/event/modules/ngx_epoll_module.c:937
    #4 0x9773a9 in ngx_process_events_and_timers src/event/ngx_event.c:242
    #5 0x9a6cca in ngx_worker_process_cycle src/os/unix/ngx_process_cycle.c:989
    #6 0x99b93c in ngx_spawn_process src/os/unix/ngx_process.c:213
    #7 0x9a25b9 in ngx_start_worker_processes src/os/unix/ngx_process_cycle.c:425
    #8 0x9a0fb6 in ngx_master_process_cycle src/os/unix/ngx_process_cycle.c:146
    #9 0x8eff1a in main src/core/nginx.c:390
    #10 0x7f929da2a82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    ...
    SUMMARY: AddressSanitizer: 576 byte(s) leaked in 11 allocation(s).

option for AddressSanitizer

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# check all available options

$ ASAN_OPTIONS=help=1 ./test
Available flags for AddressSanitizer:
	symbolize
		- If set, use the online symbolizer from common sanitizer runtime to turn virtual addresses to file/line locations.
	external_symbolizer_path
		- Path to external symbolizer. If empty, the tool will search $PATH for the symbolizer.
	allow_addr2line
		- If set, allows online symbolizer to run addr2line binary to symbolize stack traces (addr2line will only be used if llvm-symbolizer binary is unavailable.
	strip_path_prefix
		- Strips this prefix from file paths in error reports.
	fast_unwind_on_check
		- If available, use the fast frame-pointer-based unwinder on internal CHECK failures.
	fast_unwind_on_fatal
		- If available, use the fast frame-pointer-based unwinder on fatal errors.
	fast_unwind_on_malloc
		- If available, use the fast frame-pointer-based unwinder on malloc/free.
	handle_ioctl
		- Intercept and handle ioctl requests.
	malloc_context_size
		- Max number of stack frames kept for each allocation/deallocation.
	log_path
		- Write logs to "log_path.pid". The special values are "stdout" and "stderr". The default is "stderr".
....

set log file for AddressSanitizer

1
2
3
$ ASAN_OPTIONS=log_path=/tmp/test ./test

# it will create a file named test.$pid and print result to that file

REF