offwaketime(8) System Manager's Manual offwaketime(8)
NAME
offwaketime - Summarize blocked time by off-CPU stack + waker stack.
Uses Linux eBPF/bcc.
SYNOPSIS
offwaketime [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-d] [-f]
[--stack-storage-size STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M
MAX_BLOCK_TIME] [--state STATE] [duration]
DESCRIPTION
This program shows kernel stack traces and task names that were blocked
and "off-CPU", along with the stack traces and task names for the
threads that woke them, and the total elapsed time from when they
blocked to when they were woken up. This combines the summaries from
both the offcputime and wakeuptime tools. The time measurement will be
very similar to off-CPU time, however, off-CPU time may include a lit-
tle extra time spent waiting on a run queue to be scheduled. The com-
bined stacks, task names, and total time is summarized in kernel con-
text for efficiency, using an eBPF map.
The output summary will further help you identify reasons why threads
were blocking, and quantify the time from when they were blocked to
woken up. This spans all types of blocking activity: disk I/O, network
I/O, locks, page faults, swapping, sleeping, involuntary context
switches, etc.
This is complementary to CPU profiling (e.g., CPU flame graphs) which
shows the time spent on-CPU. This shows the time spent blocked off-CPU,
and the output, especially the -f format, can be used to generate an
"off-wake time flame graph".
See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html
REQUIREMENTS
CONFIG_BPF and bcc.
OPTIONS
-h Print usage message.
-f Print output in folded stack format.
-p PID Trace this process ID only (filtered in-kernel). Can be a comma
separated list of PIDS.
-t TID Trace this thread ID only (filtered in-kernel). Can be a comma
separated list of TIDS.
-u Only trace user threads (no kernel threads).
-k Only trace kernel threads (no user threads).
-U Show stacks from user space only (no kernel space stacks).
-K Show stacks from kernel space only (no user space stacks).
-d, --delimited
insert delimiter between kernel/user stacks
--stack-storage-size STACK_STORAGE_SIZE
Change the number of unique stack traces that can be stored and
displayed.
duration
Duration to trace, in seconds.
-m MIN_BLOCK_TIME
The amount of time in microseconds over which we store traces
(default 1)
-M MAX_BLOCK_TIME
The amount of time in microseconds under which we store traces
(default U64_MAX)
--state
Filter on this thread state bitmask (eg, 2 == TASK_UNINTERRUPT-
IBLE). See include/linux/sched.h for states.
EXAMPLES
Trace all thread blocking events, and summarize (in-kernel) by user and
kernel off-CPU stack trace, waker stack traces, task names, and total
blocked time:
# offwaketime
Trace for 5 seconds only:
# offwaketime 5
Trace for 5 seconds, and emit output in folded stack format (suitable
for flame graphs), user-mode threads only:
# offwaketime -fu 5
Trace PID 185 only:
# offwaketime -p 185
OVERHEAD
This summarizes unique stack trace pairs in-kernel for efficiency, al-
lowing it to trace a higher rate of events than methods that post-
process in user space. The stack trace and time data is only copied to
user space once, when the output is printed. While these techniques
greatly lower overhead, scheduler events are still a high frequency
event, as they can exceed 1 million events per second, and so caution
should still be used. Test before production use.
If the overhead is still a problem, take a look at the min block op-
tion. If your aim is to chase down longer blocking events, then this
could be increased to filter shorter blocking events, further lowering
overhead.
SOURCE
This is from bcc.
https://github.com/iovisor/bcc
Also look in the bcc distribution for a companion _examples.txt file
containing example usage, output, and commentary for this tool.
OS
Linux
STABILITY
Unstable - in development.
AUTHOR
Brendan Gregg
SEE ALSO
offcputime(8), wakeuptime(8)
USER COMMANDS 2016-01-30 offwaketime(8)
Generated by dwww version 1.16 on Tue Dec 16 05:46:54 CET 2025.