On the Speedemy.com site there's a post showing you how to get down to a seriously low level of processing and identify performance issues with strace, the debugging tool that helps monitor interactions between processes, the kernel, system calls, etc.

What do you do when your Linux web app starts acting out? You troubleshoot it, of course! Me too. And today I would like to talk to you about one tool that works really well with all sorts of applications and application servers – ranging from apache http server running modules (mod_php, mod_perl, etc.), to applications running in fast cgi setting or even simple single-threaded programs.

The tool I’m talking about is strace, and if you’re not familiar with it, it will be my pleasure to introduce you two.

They start off by answering the question of when you shouldn't use strace for testing (like when an application can actually be profiled properly) before shows where it can actually help. The post then briefly introduces strace and what it can do, pointing out what kind of information it can provide. From there they start in on using it to do the debugging and show examples of the output it can provide. They help you use some other command line options to refine this output into something a bit more useful and even include an awk command to narrow it down even more .