strace

What strace is

A quick excerpt from the man page to learn what strace is:

``strace`` is a useful diagnostic, instructional, and debugging tool [...] Each line in the trace contains the system call name, followed by its arguments in parentheses and its return value.

A system call is simply what programs use to ask the system to do operations like opening a file, writing in it, creating a now process... etc. It is not necessary to understand every single system call to use strace.

Once the package strace is installed you can either trace a program when launching it, or hook yourself to an already running thread.

Fair warning, strace is quite resource intensive, and will slow down your program further if you simply print its output to your terminal instead of sending it into files, so it's better to send it to a file and "tail -f" it. Also, as strace creates one file per process, so for multi-process app I advise creating a dedicated directory like /tmp/strace_my_program.

Basic commands

If you want to trace a program/script that as you launch it (arguments are explained below):

1  strace -ffqttv -s1000 -o /tmp/strace_0/strace.log -p <your_command_here>

For instance:

1  strace -ffqttv -s1000 -o /tmp/strace_0/strace.log -p main.py -s "hello"

And if you want to attach yourself to an already running process:

1  strace -ffqttv -s1000 -o /tmp/strace_0/strace.log -p <pid of the process>

The additional arguments I put are:

• -ff: follow 'forks' (children processes) and send their output into separate files;
• -q: remove strace messages about "attaching" and "detaching", which aren't really useful;
• -tt: add timestamps with milliseconds precision to the logs, especially useful to debug multi-process apps;
• -v: do not abbreviate calls' details, which can be useful;
• -s1000: specify maximum string size (default is 32, which is often too small);
• -o /tmp/strace_0/strace.log: send the output to this file: if there are multiple processes, the files will be created as strace.log.<pid>.

I really advised to use these parameters for more useful logs but you can obviously adapt them.

Filtering

If you know what you're looking for, you can filter the output. For instance, you can filter system calls based on their category:

• -e trace=file;
• -e trace=process;
• -e trace=network;
• -e trace=signal;
• -e trace=ipc;
• -e trace=desc;
• -e trace=memory.

And even filter directly the system calls: for instance, see only when the program writes to standard (stdout) and error (stderr) output:

-e write=1,2

You can also narrow down the output based on a file path:

-P /var

Reading the file(s)

strace is quite verbose, but as its output is standardized, you can use plugins or even settings in your text editor to highlight it. For instance in vim, simply type:

:set filetype=strace