I am a big fan of the linux perf (perf events, perf tools) tool. It has already helped me several times debugging problems or solving performance bottlenecks.
Unfortunately, documentation is rather sparse about it. As such, I created a perf-examples repository that contains small example programs that show how certain perf features can help to find problems.
It’s currently work in progress and supposed to grow over time. Contributions are very welcome. All examples are structured into sub folders that correspond to the different perf subcommands (record, trace, stat …).
For example, the trace folder currently contains an example program that shows how mutex locking times can be traced.
which will execute the perf command:
sudo perf trace ./mutex_lock
and give the following (truncated) output:
The first column shows the time that has passed since the start of the program (milliseconds). The second column depicts the time that we spent in a certain syscall. We see that we spent four seconds waiting on the mutex call (third line, futex syscall). Running the example yourself you would see that the output is even nicely colored, highlighting long wait times. There are more examples in the repository that can be used and played around with to try different options.
Every perf related article has to mention Brendan Gregg’s perf page that contains a huge collection of perf examples. Those are mostly based on existing tools, though.
If you have similar stories of how perf helped you solve a particular problem which can be packed into a simple example, please feel free to contribute and let me know.