Tracing applications with OProfile and SystemTap

Data on Tap

Article from ADMIN 01/2010

By Thorsten Scherf

Does your application data take ages to creep off your disk or your network card, even if no noticeable activity is taking place? Tools such as OProfile and SystemTap help you find out why.

Experienced administrators tend to use tools such as ps, vmstat, or the like when they need statistics for individual subsystems such as the network, memory, or block I/O. These tools can help identify hardware or software bottlenecks, and they are indisputably useful for a general appraisal, but if you want to delve deeper, you need something with more punch.

Again, the standard toolbox offers a couple of utilities. For example, the popular strace traces applications. In the simplest cases, the tool lists all the system calls (syscalls) with their arguments and return codes for a specific application. Setting options allows for highly selective Strace output. For example, if you need to investigate whether an application is parsing the configuration file that you painstakingly put together, you can call Strace:

strace -e trace=open -o mutt.trace mutt

This command line sends all open syscalls for the Mutt application to the /tmp/mutt.trace output file. Then, you can easily grep the configuration file from the results.

Profiling applications, including the popular OProfile tool http://1, take this a step further by giving you details of the performance of individual applications, the kernel, or the complete system (see Figure 1). For this to happen, OProfile accesses the CPU performance counters on state-of-the-art hardware. The counters have information on how often a specific event has occurred. In this context, an event can be RAM access or the number of interrupts. This information is very useful for identifying bottlenecks or debugging the system.

...

Use Express-Checkout link below to read the full article (PDF).