FTR: This gdb script will print full stack traces for every read syscall made:
catch syscall read
commmands
backtrace
continue
end
Put that into a file `trace-read.gdb` and attach to a running process like so:
gdb -x trace-read.gdb -p $(pgrep -n tar)
This assumes you are running an executable with built-in debugging symbols (gcc -g). It should be possible to side-load debugging symbols provided in an external package, though I don't have a command at hand. (Anyone?)
This works well enough on Linux, to quickly debug situations like the one above. However, it can make the attached process painfully slow, and occasionally bring it down all-together.
The right tool for this kind of in situations where slowing down or crashing the process is not acceptable, is DTrace:
Ironically, DTrace is one of the main selling points for Solaris/OmniOS (or FreeBSD) over Linux.
The situation has gotten better recently, with bpftrace becoming available:
You can do this on Linux using perf, if you want less overhead and less impact on the application-under-test:
$ perf record -e syscalls:sys_enter_read -g -- application arg1 arg2 ...
[ application runs while perf writes out a log, recording every read() syscall, and keeping track of the backtrace each time]
$ perf report -g --stdio
[ perf reads the log, writing out the backtraces ]
This is the basic usage. Lots more is available, obviously. This has been available for a LONG time.
This works well enough on Linux, to quickly debug situations like the one above. However, it can make the attached process painfully slow, and occasionally bring it down all-together.
The right tool for this kind of in situations where slowing down or crashing the process is not acceptable, is DTrace:
- Tutorial: https://wiki.freebsd.org/DTrace/Tutorial- ustack: http://dtrace.org/guide/chp-user.html#chp-user-4
Ironically, DTrace is one of the main selling points for Solaris/OmniOS (or FreeBSD) over Linux. The situation has gotten better recently, with bpftrace becoming available:
- http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux...
- https://github.com/iovisor/bpftrace
Until you have a 4.x Kernel with the right configuration options running, I am afraid the above gdb scripts is your best option on Linux.