debugging snippets

Inspecting a runaway Ruby process with GDB

Tagged debugging, gdb, ruby  Languages ruby

Here's how to inspect a live Ruby process that is eating 100% CPU:

First:

sudo apt-get install gdb

Then ~/.gdbinit:

define eval
  call(rb_p(rb_eval_string_protect($arg0,(int*)0)))
end

define redirect_stdout
  call rb_eval_string("$_old_stdout, $stdout = $stdout, File.open('/tmp/ruby-debug.' + Process.pid.to_s, 'a'); $stdout.sync = true")
end

Now:

$ sudo gdb attach <PID>
(gdb) redirect_stdout
(gdb) eval("Kernel.caller")
(gdb) eval "ObjectSpace.each_object {|o| puts \"#{o.class.name}, #{o.inspect} -- #{o.object_id}\" unless o.is_a?(String) }; puts '----'"

Thanks: http://rrn.dk/running-ruby-process-callstack http://www.coffeepowered.net/2010/08/23/mongomapper-development-mode-and-memory-leaks/

Debugging tools

Tagged debugging, dstat, ngrep, perf, strace, wireshark, netcat, netstat, dtruss, dtrace, tcpdump  Languages bash

IO and system calls

  • dstat

Monitor network and disk IO:

dstat -t
  • dtrace / dtruss (OSX)

To get the list of available system calls use:

sudo dtrace -ln 'syscall:::entry'

Find which files a program is opening (same as strace -f -p $PID -e open):

sudo dtruss -t open_nocancel -p $PID

Also see ls /usr/bin/.d*

  • strace (Linux)

Monitor system calls made by an app:

strace ruby app.rb

Writes all system calls made by SSH, and subprocesses (-f), to a file named ssh.txt:

strace -f -o ssh.txt ssh jebus.com

Spy on all ‘open’ system calls made by a process:

strace -f -p $PID -e open

Use these commands to see a list of all available system calls (Linux only):

man syscalls
  • opensnoop

Monitor what files are being opened:

opensnoop -p $PID
strace -e open -p $PID

Networking

  • netcat

Pipe/copy data over a network:

cat request.txt | nc metafilter.com 80
  • netstat

Find which programs are listening to which port:

sudo netstat -tunapl
lsof -i -P # OSX
  • ngrep

Listen to traffic containing the string “localhost” on any network interface:

sudo ngrep -d any localhost
  • tcpdump

Listen to traffic containing the string “localhost” on any network interface:

sudo tcpdump port 80 -w http.pcap

Writes a pcap file that can be analyzed with Wireshark.

  • Wireshark

Analyze pcap files from ngrep, tcpdump, etc:

wireshark http.pcap

CPU (Linux)

  • perf

Run perf, a sampling profiler, to see where your application is spending its time:

sudo perf record ruby app.rb

Find out what the program using the most CPU time is doing:

sudo perf top

Find out if an app is using the L1 cache which is ~200 times faster than RAM:

sudo perf stat -e L1-dcache-load-misses my_golang_app

References