BPFTRACE(8) BPFTRACE(8)
NAME
bpftrace - a high-level tracing language
SYNOPSIS
bpftrace [OPTIONS] FILENAME
bpftrace [OPTIONS] -e 'program code'
DESCRIPTION
bpftrace is a high-level tracing language and runtime for Linux based
on BPF. It supports static and dynamic tracing for both the kernel and
user-space.
When FILENAME is "-", read from stdin.
EXAMPLES
List all probes with "sleep" in their name
# bpftrace -l '*sleep*'
Trace processes calling sleep
# bpftrace -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }'
Trace processes calling sleep while spawning sleep 5 as a child process
# bpftrace -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }' -c 'sleep 5'
SUPPORTED ARCHITECTURES
x86_64, arm64 and s390x
OPTIONS
Output format
-B MODE, Set the buffer mode for stdout. Valid values are
none No buffering. Each I/O is written as soon as possible
line Data is written on the first newline or when the buffer is
full. This is the default mode.
full Data is written once the buffer is full.
-f FORMAT, Set the output format. Valid values are
json
text
-o FILENAME
Write bpftrace tracing output to FILENAME instead of stdout. This
doesn’t include child process (-c option) output. Errors are still
written to stderr.
--no-warnings
Suppress all warning messages created by bpftrace.
Tracing
-e PROGRAM
Execute PROGRAM instead of reading the program from a file
-I DIR
Add the directory DIR to the search path for C headers. This option
can be used multiple times.
--include FILENAME
Add FILENAME as an include for the pre-processor. This is equal to
adding '#include FILENAME' to the start bpftrace program. This
option can be used multiple times.
-l [SEARCH]
List all probes that match the SEARCH pattern. If the pattern is
omitted all probes will be listed. This pattern supports wildcards
in the same way that probes do. E.g. '-l kprobe:*file*' to list all
'kprobes' with 'file' in the name. For more details see the LISTING
PROBES section.
--unsafe
Some calls, like 'system', are marked as unsafe as they can have
dangerous side effects ('system("rm -rf")') and are disabled by
default. This flag allows their use.
-k
Errors from bpf-helpers(7) are silently ignored by default which
can lead to strange results. This flag enables the detection of
errors (except for errors from 'probe_read_*'). When errors occurs
bpftrace will log an error containing the source location and the
error code:
stdin:48-57: WARNING: Failed to probe_read_user_str: Bad address (-14)
u:lib.so:"fn(char const*)" { printf("arg0:%s\n", str(arg0));}
~~~~~~~~~
-kk
Same as '-k' but also includes the errors from 'probe_read_*'
helpers.
Process management
-p PID
Attach to the process with PID. If the process terminates, bpftrace
will also terminate. When using USDT probes, uprobes, and
uretprobes they will be attached to only this process. For listing
uprobes/uretprobes if you set the target to '*' the process’s
address space will be searched for the symbols.
-c COMMAND
Run COMMAND as a child process. When the child terminates bpftrace
stops as well, as if 'exit()' has been called. If bpftrace
terminates before the child process does the child process will be
terminated with a SIGTERM. If used, 'USDT' probes these will only
be attached to the child process. To avoid a race condition when
using 'USDTs' the child is stopped after 'execve' using 'ptrace(2)'
and continued when all 'USDT' probes are attached.
The child PID is available to programs as the 'cpid' builtin.
The child process runs with the same privileges as bpftrace itself
(usually root).
--usdt-file-activation
activate usdt semaphores based on file path
Miscellaneous
--info
Print detailed information about features supported by the kernel
and the bpftrace build.
--no-feature feature,feature,..., Disable detected features, valid
values are
uprobe_multi to disable uprobe_multi link
kprobe_multi to disable kprobe_multi link
-h, --help
Print the help summary
-V, --version
Print bpftrace version information
-v
verbose messages
-d
debug mode
-dd
verbose debug mode
ENVIRONMENT VARIABLES
Some behavior can only be controlled through environment variables.
Most of these can also be set via the config block directly in a script
(before any probes). This section lists all those variables.
BPFTRACE_BTF
Default: None
The path to a BTF file. By default, bpftrace searches several locations
to find a BTF file. See src/btf.cpp for the details.
BPFTRACE_CACHE_USER_SYMBOLS
Default: PER_PROGRAM if ASLR disabled or -c option given, PER_PID
otherwise.
-* PER_PROGRAM - each program has its own cache. If there are more
processes with enabled ASLR for a single program, this might produce
incorrect results. -* PER_PID - each process has its own cache. This is
accurate for processes with ASLR enabled, and enables bpftrace to
preload caches for processes running at probe attachment ti me. If
there are many processes running, it will consume a lot of a memory. -*
NONE - caching disabled. This saves the most memory, but at the cost of
speed.
BPFTRACE_CPP_DEMANGLE
Default: 1
C++ symbol demangling in userspace stack traces is enabled by default.
This feature can be turned off by setting the value of this environment
variable to 0.
BPFTRACE_DEBUG_OUTPUT
Default: 0
Outputs bpftrace’s runtime debug messages to the trace_pipe. This
feature can be turned on by setting the value of this environment
variable to 1.
BPFTRACE_KERNEL_BUILD
Default: /lib/modules/$(uname -r)
Only used with BPFTRACE_KERNEL_SOURCE if it is out-of-tree Linux kernel
build.
BPFTRACE_KERNEL_SOURCE
Default: /lib/modules/$(uname -r)
bpftrace requires kernel headers for certain features, which are
searched for in this directory.
BPFTRACE_LOG_SIZE
Default: 1000000
Log size in bytes.
BPFTRACE_MAX_BPF_PROGS
Default: 512
This is the maximum number of BPF programs (functions) that bpftrace
can generate. The main purpose of this limit is to prevent bpftrace
from hanging since generating a lot of probes takes a lot of resources
(and it should not happen often).
BPFTRACE_MAX_CAT_BYTES
Default: 10k
Maximum bytes read by cat builtin.
BPFTRACE_MAX_MAP_KEYS
Default: 4096
This is the maximum number of keys that can be stored in a map.
Increasing the value will consume more memory and increase startup
times. There are some cases where you will want to: for example,
sampling stack traces, recording timestamps for each page, etc.
BPFTRACE_MAX_PROBES
Default: 512
This is the maximum number of probes that bpftrace can attach to.
Increasing the value will consume more memory, increase startup times
and can incur high performance overhead or even freeze or crash the
system.
BPFTRACE_MAX_STRLEN
Default: 64
Number of bytes allocated on the BPF stack for the string returned by
str().
Make this larger if you wish to read bigger strings with str().
Beware that the BPF stack is small (512 bytes), and that you pay the
toll again inside printf() (whilst it composes a perf event output
buffer). So in practice you can only grow this to about 200 bytes.
Support for even larger strings is [being
discussed](https://github.com/iovisor/bpftrace/issues/305).
BPFTRACE_MAX_TYPE_RES_ITERATIONS
Default: 0
Maximum should be the number of levels of nested field accesses for
tracepoint args. 0 is unlimited.
BPFTRACE_PERF_RB_PAGES
Default: 64
Number of pages to allocate per CPU for perf ring buffer. The value
must be a power of 2.
If you’re getting a lot of dropped events bpftrace may not be
processing events in the ring buffer fast enough. It may be useful to
bump the value higher so more events can be queued up. The tradeoff is
that bpftrace will use more memory.
BPFTRACE_STACK_MODE
Default: bpftrace
Output format for ustack and kstack builtins. Available modes/formats:
bpftrace, perf, and raw. This can be overwritten at the call site.
BPFTRACE_STR_TRUNC_TRAILER
Default: ..
Trailer to add to strings that were truncated. Set to empty string to
disable truncation trailers.
BPFTRACE_VMLINUX
Default: None
This specifies the vmlinux path used for kernel symbol resolution when
attaching kprobe to offset. If this value is not given, bpftrace
searches vmlinux from pre defined locations. See
src/attached_probe.cpp:find_vmlinux() for details.
BPFTRACE LANGUAGE
Overview
The bpftrace (bt) language is inspired by the D language used by dtrace
and uses the same program structure. Each script consists of an
preamble and one or more action blocks.
preamble
actionblock1
actionblock2
Preprocessor and type definitions take place in the preamble:
#include <linux/socket.h>
#define RED "\033[31m"
struct S {
int x;
}
Each action block consists of three parts:
probe[,probe]
/predicate/ {
action
}
Probes
A probe specifies the event and event type to attach too.
Predicate
The predicate is optional condition that must be met for the action
to be executed.
Action
Actions are the programs that run when an event fires (and the
predicate is met). An action is a semicolon (;) separated list of
statements and always enclosed by brackets {}
A basic script that traces the open(2) and openat(2) system calls can
be written as follows:
BEGIN
{
printf("Tracing open syscalls... Hit Ctrl-C to end.\n");
}
tracepoint:syscalls:sys_enter_open,
tracepoint:syscalls:sys_enter_openat
{
printf("%-6d %-16s %s\n", pid, comm, str(args.filename));
}
This script has two action blocks and a total of 3 probes. The first
action block uses the special BEGIN probe, which fires once during
bpftrace startup. This probe is used to print a header, indicating that
the tracing has started.
The second action block uses two probes, one for open and one for
openat, and defines an action that prints the file being open ed as
well as the pid and comm of the process that execute the syscall. See
the PROBES section for details on the available probe types.
Identifiers
Identifiers must match the following regular expression:
[_a-zA-Z][_a-zA-Z0-9]*
Comments
Both single line and multi line comments are supported.
// A single line comment
i:s:1 { // can also be used to comment inline
/*
a multi line comment
*/
print(/* inline comment block */ 1);
}
Config Block
To improve script portability, you can set bpftrace configs via the
config block, which can only be placed at the top of the script before
any probes (even BEGIN).
config = {
stack_mode=perf;
max_map_keys=2
}
BEGIN { ... }
uprobe:./testprogs/uprobe_test:uprobeFunction1 { ... }
The names of the config variables can be in the format of environment
variables or their lowercase equivalent without the BPFTRACE_ prefix.
For example, BPFTRACE_STACK_MODE, STACK_MODE, and stack_mode are
equivalent.
Note: Environment variables for the same config take precedence over
those set inside a script config block.
Data Types
The following fundamental integer types are provided by the language.
┌────────┬─────────────────────────┐
│ │ │
│ Type │ Description │
├────────┼─────────────────────────┤
│ │ │
│ uint8 │ Unsigned 8 bit integer │
├────────┼─────────────────────────┤
│ │ │
│ int8 │ Signed 8 bit integer │
├────────┼─────────────────────────┤
│ │ │
│ uint16 │ Unsigned 16 bit integer │
├────────┼─────────────────────────┤
│ │ │
│ int16 │ Signed 16 bit integer │
├────────┼─────────────────────────┤
│ │ │
│ uint32 │ Unsigned 32 bit integer │
├────────┼─────────────────────────┤
│ │ │
│ int32 │ Signed 32 bit integer │
├────────┼─────────────────────────┤
│ │ │
│ uint64 │ Unsigned 64 bit integer │
├────────┼─────────────────────────┤
│ │ │
│ int64 │ Signed 64 bit integer │
└────────┴─────────────────────────┘
Floating-point
Floating-point numbers are not supported by BPF and therefore not by
bpftrace.
Constants
Integers constants can be defined in the following formats:
• decimal (base 10)
• octal (base 8)
• hexadecimal (base 16)
• scientific (base 10)
Octal constants have to be prefixed with a 0, e.g. 0123. Hexadecimal
constants start with either 0x or 0X, e.g. 0x10. Scientific are written
in the <m>e<n> format which is a shorthand for m*10^n, e.g. $i = 2e3;.
Note that scientific literals are integer only due to the lack of
floating point support, 1e-3 is not valid.
To improve the readability of big literals a underscore _ can be used
as field separator, e.g. 1_000_123_000.
Integer suffixes as found in the C language are parsed by bpftrace to
ensure compatibility with C headers/definitions but they’re not used as
size specifiers. 123UL, 123U and 123LL all result in the same integer
type with a value of 123.
Character constants can be defined by enclosing the character in single
quotes, e.g. $c = 'c';.
String constants can be defined by enclosing the character string in
double quotes, e.g. $str = "Hello world";.
Characters and strings support the following escape sequences:
┌──────┬──────────────────────┐
│ │ │
│ \n │ Newline │
├──────┼──────────────────────┤
│ │ │
│ \t │ Tab │
├──────┼──────────────────────┤
│ │ │
│ \0nn │ Octal value nn │
├──────┼──────────────────────┤
│ │ │
│ \xnn │ Hexadecimal value nn │
└──────┴──────────────────────┘
Type conversion
Integer and pointer types can be converted using explicit type
conversion with an expression like:
$y = (uint32) $z;
$py = (int16 *) $pz;
Integer casts to a higher rank are sign extended. Conversion to a lower
rank is done by zeroing leading bits.
It is also possible to cast between integers and integer arrays using
the same syntax:
$a = (uint8[8]) 12345;
$x = (uint64) $a;
Both the cast and the destination type must have the same size. When
casting to an array, it is possible to omit the size which will be
determined automatically from the size of the cast value.
Operators and Expressions
Arithmetic Operators
The following operators are available for integer arithmetic:
┌───┬────────────────────────┐
│ │ │
│ + │ integer addition │
├───┼────────────────────────┤
│ │ │
│ - │ integer subtraction │
├───┼────────────────────────┤
│ │ │
│ * │ integer multiplication │
├───┼────────────────────────┤
│ │ │
│ / │ integer division │
├───┼────────────────────────┤
│ │ │
│ % │ integer modulo │
└───┴────────────────────────┘
Logical Operators
┌────┬─────────────┐
│ │ │
│ && │ Logical AND │
├────┼─────────────┤
│ │ │
│ || │ Logical OR │
├────┼─────────────┤
│ │ │
│ ! │ Logical NOT │
└────┴─────────────┘
Bitwise Operators
┌────┬───────────────────────────┐
│ │ │
│ & │ AND │
├────┼───────────────────────────┤
│ │ │
│ | │ OR │
├────┼───────────────────────────┤
│ │ │
│ ^ │ XOR │
├────┼───────────────────────────┤
│ │ │
│ << │ Left shift the left-hand │
│ │ operand by the number of │
│ │ bits specified by the │
│ │ right-hand expression │
│ │ value │
├────┼───────────────────────────┤
│ │ │
│ >> │ Right shift the left-hand │
│ │ operand by the number of │
│ │ bits specified by the │
│ │ right-hand expression │
│ │ value │
└────┴───────────────────────────┘
Relational Operators
The following relational operators are defined for integers and
pointers.
┌────┬────────────────────────────┐
│ │ │
│ < │ left-hand expression is │
│ │ less than right-hand │
├────┼────────────────────────────┤
│ │ │
│ <= │ left-hand expression is │
│ │ less than or equal to │
│ │ right-hand │
├────┼────────────────────────────┤
│ │ │
│ > │ left-hand expression is │
│ │ bigger than right-hand │
├────┼────────────────────────────┤
│ │ │
│ >= │ left-hand expression is │
│ │ bigger or equal to than │
│ │ right-hand │
├────┼────────────────────────────┤
│ │ │
│ == │ left-hand expression equal │
│ │ to right-hand │
├────┼────────────────────────────┤
│ │ │
│ != │ left-hand expression not │
│ │ equal to right-hand │
└────┴────────────────────────────┘
The following relation operators are available for comparing strings
and integer arrays.
┌────┬────────────────────────────┐
│ │ │
│ == │ left-hand string equal to │
│ │ right-hand │
├────┼────────────────────────────┤
│ │ │
│ != │ left-hand string not equal │
│ │ to right-hand │
└────┴────────────────────────────┘
Assignment Operators
The following assignment operators can be used on both map and scratch
variables:
┌─────┬────────────────────────────┐
│ │ │
│ = │ Assignment, assign the │
│ │ right-hand expression to │
│ │ the left-hand variable │
├─────┼────────────────────────────┤
│ │ │
│ <<= │ Update the variable with │
│ │ its value left shifted by │
│ │ the number of bits │
│ │ specified by the │
│ │ right-hand expression │
│ │ value │
├─────┼────────────────────────────┤
│ │ │
│ >>= │ Update the variable with │
│ │ its value right shifted by │
│ │ the number of bits │
│ │ specified by the │
│ │ right-hand expression │
│ │ value │
├─────┼────────────────────────────┤
│ │ │
│ += │ Increment the variable by │
│ │ the right-hand expression │
│ │ value │
├─────┼────────────────────────────┤
│ │ │
│ -= │ Decrement the variable by │
│ │ the right-hand expression │
│ │ value │
├─────┼────────────────────────────┤
│ │ │
│ *= │ Multiple the variable by │
│ │ the right-hand expression │
│ │ value │
├─────┼────────────────────────────┤
│ │ │
│ /= │ Divide the variable by the │
│ │ right-hand expression │
│ │ value │
├─────┼────────────────────────────┤
│ │ │
│ %= │ Modulo the variable by the │
│ │ right-hand expression │
│ │ value │
├─────┼────────────────────────────┤
│ │ │
│ &= │ Bitwise AND the variable │
│ │ by the right-hand │
│ │ expression value │
├─────┼────────────────────────────┤
│ │ │
│ |= │ Bitwise OR the variable by │
│ │ the right-hand expression │
│ │ value │
├─────┼────────────────────────────┤
│ │ │
│ ^= │ Bitwise XOR the variable │
│ │ by the right-hand │
│ │ expression value │
└─────┴────────────────────────────┘
All these operators are syntactic sugar for combining assignment with
the specified operator. @ -= 5 is equal to @ = @ - 5.
Increment and Decrement Operators
The increment (+`) and decrement (`--`) operators can be used on
integer and pointer variables to increment their value by one. They can
only be used on variables and can either be applied as prefix or
suffix. The difference is that the expression `x+ returns the original
value of x, before it got incremented while ++x returns the value of x
post increment. E.g.
$x = 10;
$y = $x--; // y = 10; x = 9
$a = 10;
$b = --$a; // a = 9; b = 9
Note that maps will be implicitly declared and initialized to 0 if not
already declared or defined. Scratch variables must be initialized
before using these operators.
Variables and Maps
bpftrace knows two types of variables, scratch and map.
'scratch' variables are kept on the BPF stack and only exists during
the execution of the action block and cannot be accessed outside of the
program. Scratch variable names always start with a $, e.g. $myvar.
'map' variables use BPF 'maps'. These exist for the lifetime of
bpftrace itself and can be accessed from all action blocks and
user-space. Map names always start with a @, e.g. @mymap.
All valid identifiers can be used as name.
The data type of a variable is automatically determined during first
assignment and cannot be changed afterwards.
Associative Arrays
Associative arrays are a collection of elements indexed by a key,
similar to the hash tables found in languages like C++ (std::map) and
Python (dict). They’re a variant of 'map' variables.
@name[key] = expression
@name[key1,key2] = expression
Just like with any variable the type is determined on first use and
cannot be modified afterwards. This applies to both the key(s) and the
value type.
The following snippet creates a map with key signature [int64,
string[16]] and a value type of int64:
@[pid, comm]++
Variable scoping
Pointers
Pointers in bpftrace are similar to those found in C.
Tuples
bpftrace has support for immutable N-tuples (n > 1). A tuple is a
sequence type (like an array) where, unlike an array, every element can
have a different type.
Tuples are a comma separated list of expressions, enclosed in brackets,
(1,2) Individual fields can be accessed with the . operator. Tuples are
zero indexed like arrays are.
i:s:1 {
$a = (1,2);
$b = (3,4, $a);
print($a);
print($b);
print($b.0);
}
Prints:
(1, 2)
(3, 4, (1, 2))
3
Arrays
bpftrace supports accessing one-dimensional arrays like those found in
C.
Constructing arrays from scratch, like int a[] = {1,2,3} in C, is not
supported. They can only be read into a variable from a pointer.
The [] operator is used to access elements.
struct MyStruct {
int y[4];
}
kprobe:dummy {
$s = (struct MyStruct *) arg0;
print($s->y[0]);
}
Structs
C like structs are supported by bpftrace. Fields are accessed with the
. operator. Fields of a pointer to a struct can be accessed with the ->
operator.
Custom struct can be defined in the preamble
Constructing structs from scratch, like struct X var = {.f1 = 1} in C,
is not supported. They can only be read into a variable from a pointer.
struct MyStruct {
int a;
}
kprobe:dummy {
$ptr = (struct MyStruct *) arg0;
$st = *$ptr;
print($st.a);
print($ptr->a);
}
Conditionals
Conditional expressions are supported in the form of if/else statements
and the ternary operator.
The ternary operator consists of three operands: a condition followed
by a ?, the expression to execute when the condition is true followed
by a : and the expression to execute if the condition is false.
condition ? ifTrue : ifFalse
Both the ifTrue and ifFalse expressions must be of the same type,
mixing types is not allowed.
The ternary operator can be used as part of an assignment.
$a == 1 ? print("true") : print("false");
$b = $a > 0 ? $a : -1;
If/else statements, like the one in C, are supported.
if (condition) {
ifblock
} else if (condition) {
if2block
} else {
elseblock
}
Loops
Since kernel 5.3 BPF supports loops as long as the verifier can prove
they’re bounded and fit within the instruction limit.
In bpftrace loops are available through the while statement.
while (condition) {
block;
}
Within a while-loop the following control flow statements can be used:
┌──────────┬────────────────────────────┐
│ │ │
│ continue │ skip processing of the │
│ │ rest of the block and jump │
│ │ back to the evaluation of │
│ │ the conditional │
├──────────┼────────────────────────────┤
│ │ │
│ break │ Terminate the loop │
└──────────┴────────────────────────────┘
i:s:1 {
$i = 0;
while ($i <= 100) {
printf("%d ", $i);
if ($i > 5) {
break;
}
$i++
}
printf("\n");
}
Loop unrolling is also supported with the unroll statement.
unroll(n) {
block;
}
The compiler will evaluate the block n times and generate the BPF code
for the block n times. As this happens at compile time n must be a
constant greater than 0 (n > 0).
The following two probes compile into the same code:
i:s:1 {
unroll(3) {
print("Unrolled")
}
}
i:s:1 {
print("Unrolled")
print("Unrolled")
print("Unrolled")
}
INVOCATION MODE
There are three invocation modes for bpftrace built-in functions.
┌──────────────┬─────────────────────┬────────────────────┐
│ │ │ │
│ Mode │ Description │ Example functions │
├──────────────┼─────────────────────┼────────────────────┤
│ │ │ │
│ Synchronous │ The value/effect of │ reg(), str(), │
│ │ the built-in │ ntop() │
│ │ function is │ │
│ │ determined/handled │ │
│ │ right away by the │ │
│ │ bpf program in the │ │
│ │ kernel space. │ │
├──────────────┼─────────────────────┼────────────────────┤
│ │ │ │
│ Asynchronous │ The value/effect of │ printf(), clear(), │
│ │ the built-in │ exit() │
│ │ function is │ │
│ │ determined/handled │ │
│ │ later by the │ │
│ │ bpftrace process in │ │
│ │ the user space. │ │
├──────────────┼─────────────────────┼────────────────────┤
│ │ │ │
│ Compile-time │ The value of the │ kaddr(), │
│ │ built-in function │ cgroupid(), │
│ │ is determined │ offsetof() │
│ │ before bpf programs │ │
│ │ are running. │ │
└──────────────┴─────────────────────┴────────────────────┘
While BPF in the kernel can do a lot there are still things that can
only be done from user space, like the outputting (printing) of data.
The way bpftrace handles this is by sending events from the BPF program
which user-space will pick up some time in the future (usually in
milliseconds). Operations that happen in the kernel are 'synchronous'
('sync') and those that are handled in user space are 'asynchronous'
('async')
The asynchronous behaviour can lead to some unexpected behavior as
updates can happen before user space had time to process the event. The
following situations may occur:
• event loss: when using printf(), the amount of data printed may be
less than the actual number of events generated by the kernel
during BPF program’s execution.
• delayed exit: when using the exit() to terminate the program,
bpftrace needs to handle the exit signal asynchronously causing the
BPF program may continue to run for some additional time.
One example is updating a map value in a tight loop:
BEGIN {
@=0;
unroll(10) {
print(@);
@++;
}
exit()
}
Maps are printed by reference not by value and as the value gets
updated right after the print user-space will likely only see the final
value once it processes the event:
@: 10
@: 10
@: 10
@: 10
@: 10
@: 10
@: 10
@: 10
@: 10
@: 10
Therefore, when you need precise event statistics, it is recommended to
use synchronous functions (e.g. count() and hist()) to ensure more
reliable and accurate results.
ADDRESS-SPACES
Kernel and user pointers live in different address spaces which,
depending on the CPU architecture, might overlap. Trying to read a
pointer that is in the wrong address space results in a runtime error.
This error is hidden by default but can be enabled with the -kk flag:
stdin:1:9-12: WARNING: Failed to probe_read_user: Bad address (-14)
BEGIN { @=*uptr(kaddr("do_poweroff")) }
~~~
bpftrace tries to automatically set the correct address space for a
pointer based on the probe type, but might fail in cases where it is
unclear. The address space can be changed with the kptr() and uptr()
functions.
BUILTINS
Builtins are special variables built into the language. Unlike the
scratch and map variable they don’t need a $ or @ as prefix (except for
the positional parameters).
┌───────────────┬─────────────┬────────────┬───────────────────────┬───────────────────┐
│ │ │ │ │ │
│ Variable │ Type │ Kernel │ BPF Helper │ Description │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ $1, $2, ...$n │ int64 │ n/a │ n/a │ The nth │
│ │ │ │ │ positional │
│ │ │ │ │ parameter │
│ │ │ │ │ passed to the │
│ │ │ │ │ bpftrace │
│ │ │ │ │ program. If │
│ │ │ │ │ less than n │
│ │ │ │ │ parameters │
│ │ │ │ │ are passed │
│ │ │ │ │ this │
│ │ │ │ │ evaluates to │
│ │ │ │ │ 0. For string │
│ │ │ │ │ arguments use │
│ │ │ │ │ the str() │
│ │ │ │ │ call to │
│ │ │ │ │ retrieve the │
│ │ │ │ │ value. │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ $# │ int64 │ n/a │ n/a │ Total amount │
│ │ │ │ │ of positional │
│ │ │ │ │ parameters │
│ │ │ │ │ passed. │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ arg0, arg1, │ int64 │ n/a │ n/a │ nth argument │
│ ...argn │ │ │ │ passed to the │
│ │ │ │ │ function │
│ │ │ │ │ being traced. │
│ │ │ │ │ These are │
│ │ │ │ │ extracted │
│ │ │ │ │ from the CPU │
│ │ │ │ │ registers. │
│ │ │ │ │ The amount of │
│ │ │ │ │ args passed │
│ │ │ │ │ in registers │
│ │ │ │ │ depends on │
│ │ │ │ │ the CPU │
│ │ │ │ │ architecture. │
│ │ │ │ │ (kprobes, │
│ │ │ │ │ uprobes, │
│ │ │ │ │ usdt). │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ args │ struct args │ n/a │ n/a │ The struct of │
│ │ │ │ │ all arguments │
│ │ │ │ │ of the traced │
│ │ │ │ │ function. │
│ │ │ │ │ Available in │
│ │ │ │ │ tracepoint, │
│ │ │ │ │ kfunc, and │
│ │ │ │ │ uprobe (with │
│ │ │ │ │ DWARF) │
│ │ │ │ │ probes. Use │
│ │ │ │ │ args.x to │
│ │ │ │ │ access │
│ │ │ │ │ argument x or │
│ │ │ │ │ args to get a │
│ │ │ │ │ record with │
│ │ │ │ │ all │
│ │ │ │ │ arguments. │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ cgroup │ uint64 │ 4.18 │ get_current_cgroup_id │ ID of the │
│ │ │ │ │ cgroup the │
│ │ │ │ │ current task │
│ │ │ │ │ is in. Only │
│ │ │ │ │ works with │
│ │ │ │ │ cgroupv2. │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ comm │ string[16] │ 4.2 │ get_current_com │ comm of the │
│ │ │ │ │ current task. │
│ │ │ │ │ Equal to the │
│ │ │ │ │ value in │
│ │ │ │ │ /proc/<pid>/comm │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ cpid │ uint32 │ n/a │ n/a │ PID of the child │
│ │ │ │ │ process │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ numaid │ uint32 │ 5.8 │ numa_node_id │ ID of the NUMA │
│ │ │ │ │ node executing │
│ │ │ │ │ the BPF program │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ cpu │ uint32 │ 4.1 │ raw_smp_processor_id │ ID of the │
│ │ │ │ │ processor │
│ │ │ │ │ executing the │
│ │ │ │ │ BPF program │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ curtask │ uint64 │ 4.8 │ get_current_task │ Pointer to │
│ │ │ │ │ struct │
│ │ │ │ │ task_struct of │
│ │ │ │ │ the current task │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ elapsed │ uint64 │ (see nsec) │ ktime_get_ns / │ Nanoseconds │
│ │ │ │ ktime_get_boot_ns │ elapsed since │
│ │ │ │ │ bpftrace │
│ │ │ │ │ initialization, │
│ │ │ │ │ based on nsecs │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ func │ string │ n/a │ n/a │ Name of the │
│ │ │ │ │ current function │
│ │ │ │ │ being traced │
│ │ │ │ │ (kprobes,uprobes) │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ gid │ uint64 │ 4.2 │ get_current_uid_gid │ GID of current │
│ │ │ │ │ task │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ kstack │ kstack │ │ get_stackid │ Kernel stack │
│ │ │ │ │ trace │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ nsecs │ uint64 │ 4.1 / 5.7 │ ktime_get_ns / │ nanoseconds since │
│ │ │ │ ktime_get_boot_ns │ kernel boot. On │
│ │ │ │ │ kernels that │
│ │ │ │ │ support │
│ │ │ │ │ ktime_get_boot_ns │
│ │ │ │ │ this includes the │
│ │ │ │ │ time spent │
│ │ │ │ │ suspended, on │
│ │ │ │ │ older kernels it │
│ │ │ │ │ does not. │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ jiffies │ uint64 │ 5.9 │ get_jiffies_64 │ Jiffies of the │
│ │ │ │ │ kernel. In 32-bit │
│ │ │ │ │ system, using │
│ │ │ │ │ this builtin │
│ │ │ │ │ might be slower. │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ pid │ uint64 │ 4.2 │ get_current_pid_tgid │ Process ID (or │
│ │ │ │ │ thread group ID) │
│ │ │ │ │ of the current │
│ │ │ │ │ task. │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ probe │ string │ n/na │ n/a │ Name of the │
│ │ │ │ │ current probe │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ rand │ uint32 │ 4.1 │ get_prandom_u32 │ Random number │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ retval │ int64 │ n/a │ n/a │ Value returned by │
│ │ │ │ │ the function │
│ │ │ │ │ being traced │
│ │ │ │ │ (kretprobe, │
│ │ │ │ │ uretprobe, │
│ │ │ │ │ kretfunc) │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ sarg0, sarg1, │ int64 │ n/a │ n/a │ nth stack value │
│ ...sargn │ │ │ │ of the function │
│ │ │ │ │ being traced. │
│ │ │ │ │ (kprobes, │
│ │ │ │ │ uprobes). │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ tid │ uint64 │ 4.2 │ get_current_pid_tgid │ Thread ID of the │
│ │ │ │ │ current task. │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ uid │ uint64 │ 4.2 │ get_current_uid_gid │ UID of current │
│ │ │ │ │ task │
├───────────────┼─────────────┼────────────┼───────────────────────┼───────────────────┤
│ │ │ │ │ │
│ ustack │ ustack │ 4.6 │ get_stackid │ Userspace stack │
│ │ │ │ │ trace │
└───────────────┴─────────────┴────────────┴───────────────────────┴───────────────────┘
MAP FUNCTIONS
Map functions are built-in functions who’s return value can only be
assigned to maps. The data type associated with these functions are
only for internal use and are not compatible with the (integer)
operators.
Functions that are marked async are asynchronous which can lead to
unexpected behavior, see the [Sync and Async] section for more
information.
avg
variants
• avg(int64 n)
Calculate the running average of n between consecutive calls.
i:s:1 {
@x++;
@y = avg(@x);
print(@x);
print(@y);
}
Internally this keeps two values in the map: value count and running
total. The average is computed in user-space when printing by dividing
the total by the count.
clear
variants
• clear(map m)
async
Clear all keys/values from map m.
i:ms:100 {
@[rand % 10] = count();
}
i:s:10 {
print(@);
clear(@);
}
count
variants
• count()
Count how often this function is called.
Using @=count() is conceptually similar to @++. The difference is that
the count() function uses a map type optimized for this (PER_CPU),
increasing performance. Due to this the map cannot be accessed as a
regular integer.
i:ms:100 {
@ = count();
}
i:s:10 {
print(@);
clear(@);
}
delete
variants
• delete(mapkey k)
Delete a single key from a map. For a single value map this deletes the
only element. For an associative-array the key to delete has to be
specified.
k:dummy {
@scalar = 1;
@associative[1,2] = 1;
delete(@scalar);
delete(@associative[1,2]);
delete(@associative); // error
}
hist
variants
• hist(int64 n[, int k])
Create a log2 histogram of n using $2^k$ buckets per power of 2, 0 ⇐ k
⇐ 5, defaults to 0.
kretprobe:vfs_read {
@bytes = hist(retval);
}
Results in:
@:
[1M, 2M) 3 | |
[2M, 4M) 2 | |
[4M, 8M) 2 | |
[8M, 16M) 6 | |
[16M, 32M) 16 | |
[32M, 64M) 27 | |
[64M, 128M) 48 |@ |
[128M, 256M) 98 |@@@ |
[256M, 512M) 191 |@@@@@@ |
[512M, 1G) 394 |@@@@@@@@@@@@@ |
[1G, 2G) 820 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
lhist
variants
• lhist(int64 n, int64 min, int64 max, int64 step)
Create a linear histogram of n. lhist creates M ((max - min) / step)
buckets in the range [min,max) where each bucket is step in size.
Values in the range (-inf, min) and (max, inf) get their get their own
bucket too, bringing the total amount of buckets created to M+2.
i:ms:1 {
@ = lhist(rand %10, 0, 10, 1);
}
i:s:5 {
exit();
}
Prints:
@:
[0, 1) 306 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[1, 2) 284 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[2, 3) 294 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[3, 4) 318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[4, 5) 311 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[5, 6) 362 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[6, 7) 336 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[7, 8) 326 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 9) 328 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[9, 10) 318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
max
variants
• max(int64 n)
Update the map with n if n is bigger than the current value held.
min
variants
• min(int64 n)
Update the map with n if n is smaller than the current value held.
stats
variants
• stats(int64 n)
stats combines the count, avg and sum calls into one.
kprobe:vfs_read {
@bytes[comm] = stats(arg2);
}
@bytes[bash]: count 7, average 1, total 7
@bytes[sleep]: count 5, average 832, total 4160
@bytes[ls]: count 7, average 886, total 6208
@
sum
variants
• sum(int64 n)
Calculate the sum of all n passed.
zero
variants
• zero(map m)
async
Set all values for all keys to zero.
len
variants
• len(map m)
Return the number of elements in the map.
FUNCTIONS
Functions that are marked async are asynchronous which can lead to
unexpected behaviour, see the [sync and async] section for more
information.
compile time functions are evaluated at compile time, a static value
will be compiled into the program.
unsafe functions can have dangerous side effects and should be used
with care, the --unsafe flag is required for use.
bswap
variants
• uint8 bswap(uint8 n)
• uint16 bswap(uint16 n)
• uint32 bswap(uint32 n)
• uint64 bswap(uint64 n)
bswap reverses the order of the bytes in integer n. In case of 8 bit
integers, n is returned without being modified. The return type is an
unsigned integer of the same width as n.
buf
variants
• buf_t buf(void * data, [int64 length])
buf reads length amount of bytes from address data. The maximum value
of length is limited to the BPFTRACE_MAX_STRLEN variable. For arrays
the length is optional, it is automatically inferred from the
signature.
buf is address space aware and will call the correct helper based on
the address space associated with data.
The buf_t object returned by buf can safely be printed as a hex encoded
string with the %r format specifier.
Bytes with values >=32 and <=126 are printed using their ASCII
character, other bytes are printed in hex form (e.g. \x00). The %rx
format specifier can be used to print everything in hex form, including
ASCII characters. The similar %rh format specifier prints everything in
hex form without \x and with spaces between bytes (e.g. 0a fe).
i:s:1 {
printf("%r\n", buf(kaddr("avenrun"), 8));
}
\x00\x03\x00\x00\x00\x00\x00\x00
\xc2\x02\x00\x00\x00\x00\x00\x00
cat
variants
• void cat(string namefmt, [...args])
async
Dump the contents of the named file to stdout. cat supports the same
format string and arguments that printf does. If the file cannot be
opened or read an error is printed to stderr.
t:syscalls:sys_enter_execve {
cat("/proc/%d/maps", pid);
}
55f683ebd000-55f683ec1000 r--p 00000000 08:01 1843399 /usr/bin/ls
55f683ec1000-55f683ed6000 r-xp 00004000 08:01 1843399 /usr/bin/ls
55f683ed6000-55f683edf000 r--p 00019000 08:01 1843399 /usr/bin/ls
55f683edf000-55f683ee2000 rw-p 00021000 08:01 1843399 /usr/bin/ls
55f683ee2000-55f683ee3000 rw-p 00000000 00:00 0
cgroup_path
variants
• cgroup_path cgroup_path(int cgroupid, string filter)
Convert cgroup id to cgroup path. This is done asynchronously in
userspace when the cgroup_path value is printed, therefore it can
resolve to a different value if the cgroup id gets reassigned. This
also means that the returned value can only be used for printing.
A string literal may be passed as an optional second argument to filter
cgroup hierarchies in which the cgroup id is looked up by a wildcard
expression (cgroup2 is always represented by "unified", regardless of
where it is mounted).
The currently mounted hierarchy at /sys/fs/cgroup is used to do the
lookup. If the cgroup with the given id isn’t present here (e.g. when
running in a Docker container), the cgroup path won’t be found (unlike
when looking up the cgroup path of a process via /proc/.../cgroup).
BEGIN {
$cgroup_path = cgroup_path(3436);
print($cgroup_path);
print($cgroup_path); /* This may print a different path */
printf("%s %s", $cgroup_path, $cgroup_path); /* This may print two different paths */
}
cgroupid
variants
• uint64 cgroupid(const string path)
compile time
cgroupid retrieves the cgroupv2 ID of the cgroup available at path.
BEGIN {
print(cgroupid("/sys/fs/cgroup/system.slice"));
}
exit
variants
• void exit()
async
Terminate bpftrace, as if a SIGTERM was received. The END probe will
still trigger (if specified) and maps will be printed.
join
variants
• void join(char *arr[], [char * sep = ' '])
async
join joins all the string array arr with sep as separator into one
string. This string will be printed to stdout directly, it cannot be
used as string value.
The concatenation of the array members is done in BPF and the printing
happens in userspace.
tracepoint:syscalls:sys_enter_execve {
join(args.argv);
}
kaddr
variants
• uint64 kaddr(const string name)
compile time
Get the address of the kernel symbol name.
The following script:
kptr
variants
• T * kptr(T * ptr)
Marks ptr as a kernel address space pointer. See the address-spaces
section for more information on address-spaces. The pointer type is
left unchanged.
ksym
variants
• ksym_t ksym(uint64 addr)
async
Retrieve the name of the function that contains address addr. The
address to name mapping happens in user-space.
The ksym_t type can be printed with the %s format specifier.
kprobe:do_nanosleep
{
printf("%s\n", ksym(reg("ip")));
}
Prints:
do_nanosleep
macaddr
variants
• macaddr_t macaddr(char [6] mac)
Create a buffer that holds a macaddress as read from mac This buffer
can be printed in the canonical string format using the %s format
specifier.
kprobe:arp_create {
printf("SRC %s, DST %s\n", macaddr(sarg0), macaddr(sarg1));
}
Prints:
SRC 18:C0:4D:08:2E:BB, DST 74:83:C2:7F:8C:FF
ntop
variants
• inet_t ntop([int64 af, ] int addr)
• inet_t ntop([int64 af, ] char addr[4])
• inet_t ntop([int64 af, ] char addr[16])
ntop returns the string representation of an IPv4 or IPv6 address. ntop
will infer the address type (IPv4 or IPv6) based on the addr type and
size. If an integer or char[4] is given, ntop assumes IPv4, if a
char[16] is given, ntop assumes IPv6. You can also pass the address
type (e.g. AF_INET) explicitly as the first parameter.
pton
variants
• char addr[4] pton(const string *addr_v4)
• char addr[16] pton(const string *addr_v6)
compile time
pton converts a text representation of an IPv4 or IPv6 address to byte
array. pton infers the address family based on . or : in the given
argument. pton comes in handy when we need to select packets with
certain IP addresses.
override
variants
• override(uint64 rc)
unsafe
Kernel 4.16
Helper bpf_override
Supported probes
• kprobe
When using override the probed function will not be executed and
instead rc will be returned.
k:__x64_sys_getuid
/comm == "id"/ {
override(2<<21);
}
uid=4194304 gid=0(root) euid=0(root) groups=0(root)
This feature only works on kernels compiled with
CONFIG_BPF_KPROBE_OVERRIDE and only works on functions tagged
ALLOW_ERROR_INJECTION.
bpftrace does not test whether error injection is allowed for the
probed function, instead if will fail to load the program into the
kernel:
ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
Error attaching probe: 'kprobe:vfs_read'
reg
variants
• reg(const string name)
Supported probes
• kprobe
• uprobe
Get the contents of the register identified by name. Valid names depend
on the CPU architecture.
signal
variants
• signal(const string sig)
• signal(uint32 signum)
unsafe
Kernel 5.3
Helper bpf_send_signal
Probe types: k(ret)probe, u(ret)probe, USDT, profile
Send a signal to the process being traced. The signal can either be
identified by name, e.g. SIGSTOP or by ID, e.g. 19 as found in kill -l.
kprobe:__x64_sys_execve
/comm == "bash"/ {
signal(5);
}
$ ls
Trace/breakpoint trap (core dumped)
sizeof
variants
• sizeof(TYPE)
• sizeof(EXPRESSION)
compile time
Returns size of the argument in bytes. Similar to C/C++ sizeof
operator. Note that the expression does not get evaluated.
offsetof
variants
• offsetof(STRUCT, FIELD)
• offsetof(EXPRESSION, FIELD)
compile time
Returns offset of the field offset bytes in struct. Similar to kernel
offsetof operator. Note that subfields are not yet supported.
str
variants
• str(char * data [, uint32 length)
Helper probe_read_str, probe_read_{kernel,user}_str
str reads a NULL terminated (\0) string from data. The maximum string
length is limited by the BPFTRACE_MAX_STRLEN env variable, unless
length is specified and shorter than the maximum. In case the string is
longer than the specified length only length - 1 bytes are copied and a
NULL byte is appended at the end.
When available (starting from kernel 5.5, see the --info flag) bpftrace
will automatically use the kernel or user variant of
probe_read_{kernel,user}_str based on the address space of data, see
ADDRESS-SPACES for more information.
strerror
variants
• strerror strerror(int error)
Convert errno code to string. This is done asynchronously in userspace
when the strerror value is printed, hence the returned value can only
be used for printing.
#include <errno.h>
BEGIN {
print(strerror(EPERM));
}
strftime
variants
• strtime_t strftime(const string fmt, int64 timestamp_ns)
async
Format the nanoseconds since boot timestamp timestamp_ns according to
the format specified by fmt. The time conversion and formatting happens
in user space, therefore the timestr_t value returned can only be used
for printing using the %s format specifier.
bpftrace uses the strftime(3) function for formatting time and supports
the same format specifiers.
i:s:1 {
printf("%s\n", strftime("%H:%M:%S", nsecs));
}
bpftrace also supports the following format string extensions:
┌───────────┬────────────────────────────┐
│ │ │
│ Specifier │ Description │
├───────────┼────────────────────────────┤
│ │ │
│ %f │ Microsecond as a decimal │
│ │ number, zero-padded on the │
│ │ left │
└───────────┴────────────────────────────┘
strncmp
variants
• int64 strncmp(char * s1, char * s2, int64 n)
strncmp compares up to n characters string s1 and string s2. If they’re
equal 0 is returned, else a non-zero value is returned.
bpftrace doesn’t read past the length of the shortest string.
The use of the == and != operators is recommended over calling strncmp
directly.
strcontains
variants
• int64 strcontains(const char *haystack, const char *needle)
strcontains compares whether the string haystack contains the string
needle. If needle is contained 1 is returned, else zero is returned.
bpftrace doesn’t read past the length of the shortest string.
system
variants
• void system(string namefmt [, ...args])
unsafe async
system lets bpftrace run the specified command (fork and exec) until it
completes and print its stdout. The command is run with the same
privileges as bpftrace and it blocks execution of the processing
threads which can lead to missed events and delays processing of async
events.
i:s:1 {
time("%H:%M:%S: ");
printf("%d\n", @++);
}
i:s:10 {
system("/bin/sleep 10");
}
i:s:30 {
exit();
}
Note how the async time and printf first print every second until the
i:s:10 probe hits, then they print every 10 seconds due to bpftrace
blocking on sleep.
Attaching 3 probes...
08:50:37: 0
08:50:38: 1
08:50:39: 2
08:50:40: 3
08:50:41: 4
08:50:42: 5
08:50:43: 6
08:50:44: 7
08:50:45: 8
08:50:46: 9
08:50:56: 10
08:50:56: 11
08:50:56: 12
08:50:56: 13
08:50:56: 14
08:50:56: 15
08:50:56: 16
08:50:56: 17
08:50:56: 18
08:50:56: 19
system supports the same format string and arguments that printf does.
t:syscalls:sys_enter_execve {
system("/bin/grep %s /proc/%d/status", "vmswap", pid);
}
time
variants
• void time(const string fmt)
async
Format the current wall time according to the format specifier fmt and
print it to stdout. Unlike strftime() time() doesn’t send a timestamp
from the probe, instead it is the time at which user-space processes
the event.
bpftrace uses the strftime(3) function for formatting time and supports
the same format specifiers.
uaddr
variants
• T * uaddr(const string sym)
Supported probes
• uprobes
• uretprobes
• USDT
Does not work with ASLR, see issue #75
<https://github.com/iovisor/bpftrace/issues/75>
The uaddr function returns the address of the specified symbol. This
lookup happens during program compilation and cannot be used
dynamically.
The default return type is uint64*. If the ELF object size matches a
known integer size (1, 2, 4 or 8 bytes) the return type is modified to
match the width (uint8*, uint16*, uint32* or uint64* resp.). As ELF
does not contain type info the type is always assumed to be unsigned.
uprobe:/bin/bash:readline {
printf("PS1: %s\n", str(*uaddr("ps1_prompt")));
}
uptr
variants
• T * uptr(T * ptr)
Marks ptr as a user address space pointer. See the address-spaces
section for more information on address-spaces. The pointer type is
left unchanged.
usym
variants
• usym_t usym(uint64 * addr)
async
Supported probes
• uprobes
• uretprobes
Equal to ksym but resolves user space symbols.
If ASLR is enabled, user space symbolication only works when the
process is running at either the time of the symbol resolution or the
time of the probe attachment. The latter requires
BPFTRACE_CACHE_USER_SYMBOLS to be set to PER_PID, and might not work
with older versions of BCC. A similar limitation also applies to
dynamically loaded symbols.
uprobe:/bin/bash:readline
{
printf("%s\n", usym(reg("ip")));
}
Prints:
readline
path
variants
• char * path(struct path * path)
Kernel 5.10
Helper bpf_d_path
Return full path referenced by struct path pointer in argument.
This function can only be used by functions that are allowed to, these
functions are contained in the btf_allowlist_d_path set in the kernel.
unwatch
variants
• void unwatch(void * addr)
async
Removes a watchpoint
skboutput
variants
• uint32 skboutput(const string path, struct sk_buff *skb, uint64
length, const uint64 offset)
Kernel 5.5
Helper bpf_skb_output
Write sk_buff skb 's data section to a PCAP file in the path, starting
from offset to offset + length.
The PCAP file is encapsulated in RAW IP, so no ethernet header is
included. The data section in the struct skb may contain ethernet
header in some kernel contexts, you may set offset to 14 bytes to
exclude ethernet header.
Each packet’s timestamp is determined by adding nsecs and boot time,
the accuracy varies on different kernels, see nsecs.
This function returns 0 on success, or a negative error in case of
failure.
Environment variable BPFTRACE_PERF_RB_PAGES should be increased in
order to capture large packets, or else these packets will be dropped.
Usage
# cat dump.bt
kfunc:napi_gro_receive {
$ret = skboutput("receive.pcap", args.skb, args.skb->len, 0);
}
kfunc:dev_queue_xmit {
// setting offset to 14, to exclude ethernet header
$ret = skboutput("output.pcap", args.skb, args.skb->len, 14);
printf("skboutput returns %d\n", $ret);
}
# export BPFTRACE_PERF_RB_PAGES=1024
# bpftrace dump.bt
...
# tcpdump -n -r ./receive.pcap | head -3
reading from file ./receive.pcap, link-type RAW (Raw IP)
dropped privs to tcpdump
10:23:44.674087 IP 22.128.74.231.63175 > 192.168.0.23.22: Flags [.], ack 3513221061, win 14009, options [nop,nop,TS val 721277750 ecr 3115333619], length 0
10:23:45.823194 IP 100.101.2.146.53 > 192.168.0.23.46619: 17273 0/1/0 (130)
10:23:45.823229 IP 100.101.2.146.53 > 192.168.0.23.46158: 45799 1/0/0 A 100.100.45.106 (60)
OUTPUT FORMATTING
print
variants
• void print(T val)
async
variants
• void print(T val)
• void print(@map)
• void print(@map, uint64 top)
• void print(@map, uint64 top, uint64 div)
print prints a the value, which can be a map or a scalar value, with
the default formatting for the type.
i:ms:10 { @=hist(rand); }
i:s:1 {
print(@);
print(123);
print("abc");
exit();
}
Prints:
@:
[16M, 32M) 3 |@@@ |
[32M, 64M) 2 |@@ |
[64M, 128M) 1 |@ |
[128M, 256M) 4 |@@@@ |
[256M, 512M) 3 |@@@ |
[512M, 1G) 14 |@@@@@@@@@@@@@@ |
[1G, 2G) 22 |@@@@@@@@@@@@@@@@@@@@@@ |
[2G, 4G) 51 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
123
abc
Note that maps are printed by reference while scalar values are copied.
This means that updating and printing maps in a fast loop will likely
result in bogus map values as the map will be updated before userspace
gets the time to dump and print it.
The printing of maps supports the optional top and div arguments. top
limits the printing to the top N entries with the highest integer
values
BEGIN {
$i = 11;
while($i) {
@[$i] = --$i;
}
print(@, 2);
clear(@);
exit()
}
@[9]: 9
@[10]: 10
The div argument scales the values prior to printing them. Scaling
values before storing them can result in rounding errors. Consider the
following program:
k:f {
@[func] += arg0/10;
}
With the following sequence as numbers for arg0: 134, 377, 111, 99. The
total is 721 which rounds to 72 when scaled by 10 but the program would
print 70 due to the rounding of individual values.
Changing the print call to print(@, 5, 2) will take the top 5 values
and scale them by 2:
@[6]: 3
@[7]: 3
@[8]: 4
@[9]: 4
@[10]: 5
printf
variants
• void printf(const string fmt, args...)
async
printf() formats and prints data. It behaves similar to printf() found
in C and many other languages.
The format string has to be a constant, it cannot be modified at
runtime. The formatting of the string happens in user space. Values are
copied and passed by value.
bpftrace supports all the typical format specifiers like %llx and %hhu.
The non-standard ones can be found in the table below:
┌───────────┬────────┬─────────────────────┐
│ │ │ │
│ Specifier │ Type │ Description │
├───────────┼────────┼─────────────────────┤
│ │ │ │
│ r │ buffer │ Hex-formatted │
│ │ │ string to print │
│ │ │ arbitrary binary │
│ │ │ content returned by │
│ │ │ the buf (buf) │
│ │ │ function. │
└───────────┴────────┴─────────────────────┘
Supported escape sequences
Colors are supported too, using standard terminal escape sequences:
print("\033[31mRed\t\033[33mYellow\033[0m\n")
PROBES
bpftrace supports various probe types which allow the user to attach
BPF programs to different types of events. Each probe starts with a
provider (e.g. kprobe) followed by a colon (:) separated list of
options. The amount of options and their meaning depend on the provider
and are detailed below. The valid values for options can depend on the
system or binary being traced, e.g. for uprobes it depends on the
binary. Also see LISTING PROBES
It is possible to associate multiple probes with a single action as
long as the action is valid for all specified probes. Multiple probes
can be specified as a comma (,) separated list:
kprobe:tcp_reset,kprobe:tcp_v4_rcv {
printf("Entered: %s\n", probe);
}
Wildcards are supported too:
kprobe:tcp_* {
printf("Entered: %s\n", probe);
}
Both can be combined:
kprobe:tcp_reset,kprobe:*socket* {
printf("Entered: %s\n", probe);
}
Most providers also support a short name which can be used instead of
the full name, e.g. kprobe:f and k:f are identical.
BEGIN and END
These are special built-in events provided by the bpftrace runtime.
BEGIN is triggered before all other probes are attached. END is
triggered after all other probes are detached.
Note that specifying an END probe doesn’t override the printing of
'non-empty' maps at exit. To prevent the printing all used maps need be
cleared, which can be done in the END probe:
END {
clear(@map1);
clear(@map2);
}
hardware
variants
• hardware:event_name:
• hardware:event_name:count
shortname
• h
The hardware probe attaches to pre-defined hardware events provided by
the kernel.
They are implemented using performance monitoring counters (PMCs):
hardware resources on the processor. There are about ten of these, and
they are documented in the perf_event_open(2) man page. The event names
are:
• cpu-cycles or cycles
• instructions
• cache-references
• cache-misses
• branch-instructions or branches
• branch-misses
• bus-cycles
• frontend-stalls
• backend-stalls
• ref-cycles
The count option specifies how many events must happen before the probe
fires. If count is left unspecified a default value is used.
hardware:cache-misses:1e6 { @[pid] = count(); }
interval
variants
• interval:us:count
• interval:ms:count
• interval:s:count
• interval:hz:rate
shortnames
• i
The interval probe fires at a fixed interval as specified by its time
spec. Interval fire on one CPU at the time, unlike [profile] probes.
iterator
variants
• iter:task
• iter:task:pin
• iter:task_file
• iter:task_file:pin
• iter:task_vma
• iter:task_vma:pin
shortnames
• it
These are eBPF iterator probes, that allow iteration over kernel
objects.
Iterator probe can’t be mixed with any other probe, not even other
iterator.
Each iterator probe provides set of fields that could be accessed with
ctx pointer. User can display set of available fields for iterator via
-lv options as described below.
Examples:
# bpftrace -e 'iter:task { printf("%s:%d\n", ctx->task->comm, ctx->task->pid); }'
Attaching 1 probe...
systemd:1
kthreadd:2
rcu_gp:3
rcu_par_gp:4
kworker/0:0H:6
mm_percpu_wq:8
...
# bpftrace -e 'iter:task_file { printf("%s:%d %d:%s\n", ctx->task->comm, ctx->task->pid, ctx->fd, path(ctx->file->f_path)); }'
Attaching 1 probe...
systemd:1 1:/dev/null
systemd:1 2:/dev/null
systemd:1 3:/dev/kmsg
...
su:1622 1:/dev/pts/1
su:1622 2:/dev/pts/1
su:1622 3:/var/lib/sss/mc/passwd
...
bpftrace:1892 1:pipe:[35124]
bpftrace:1892 2:/dev/pts/1
bpftrace:1892 3:anon_inode:bpf-map
bpftrace:1892 4:anon_inode:bpf-map
bpftrace:1892 5:anon_inode:bpf_link
bpftrace:1892 6:anon_inode:bpf-prog
bpftrace:1892 7:anon_inode:bpf_iter
# bpftrace -e 'iter:task_vma {printf("%s %d %lx-%lx\n", comm, pid, ctx->vma->vm_start, ctx->vma->vm_end);}'
Attaching 1 probe...
bpftrace 119480 55b92c380000-55b92c386000
bpftrace 119480 55b92c386000-55b92c391000
bpftrace 119480 55b92c391000-55b92c397000
bpftrace 119480 55b92c398000-55b92c399000
bpftrace 119480 55b92c399000-55b92c39a000
bpftrace 119480 55b92cce3000-55b92d010000
...
bpftrace 119480 7ffd55dde000-7ffd55de2000
bpftrace 119480 7ffd55de2000-7ffd55de4000
It’s possible to pin iterator with specifying optional probe ':pin'
part, that defines the pin file. It can be specified as absolute path
or relative to /sys/fs/bpf.
relative pin
# bpftrace -e 'iter:task:list { printf("%s:%d\n", ctx->task->comm, ctx->task->pid); }'
Program pinned to /sys/fs/bpf/list
# cat /sys/fs/bpf/list
systemd:1
kthreadd:2
rcu_gp:3
rcu_par_gp:4
kworker/0:0H:6
mm_percpu_wq:8
rcu_tasks_kthre:9
...
Examples with absolute pin file:
absolute pin
# bpftrace -e '
iter:task_file:/sys/fs/bpf/files {
printf("%s:%d %s\n", ctx->task->comm, ctx->task->pid, path(ctx->file->f_path));
}'
Program pinned to /sys/fs/bpf/files
# cat /sys/fs/bpf/files
systemd:1 anon_inode:inotify
systemd:1 anon_inode:[timerfd]
...
systemd-journal:849 /dev/kmsg
systemd-journal:849 anon_inode:[eventpoll]
...
sssd:1146 /var/log/sssd/sssd.log
sssd:1146 anon_inode:[eventpoll]
...
NetworkManager:1155 anon_inode:[eventfd]
NetworkManager:1155 /var/lib/sss/mc/passwd (deleted)
kfunc and kretfunc
variants
• kfunc[:mod]:fn
• fentry[:mod]:fn
• kretfunc[:mod]:fn
• fexit[:mod]:fn
shortnames
• f (kfunc)
• fr (kretfunc)
requires (--info)
• Kernel features:BTF
• Probe types:kfunc
kfuncs attach to kernel function similar to kprobe and kretprobe. They
make use of eBPF trampolines which allows kernel code to call into BPF
programs with near zero overhead. kfunc and kretfunc are aliased as
fentry and fexit to match how these are referenced in the kernel.
kfunc s make use of BTF type information to derive the type of function
arguments at compile time. This removes the need for manual type
casting and makes the code more resilient against small signature
changes in the kernel. The function arguments are available in the args
struct which can be inspected by doing verbose listing (see LISTING
PROBES). These arguments are also available in the return probe
(kretfunc).
# bpftrace -lv 'kfunc:tcp_reset'
kfunc:tcp_reset
struct sock * sk
struct sk_buff * skb
kfunc:x86_pmu_stop {
printf("pmu %s stop\n", str(args.event->pmu->name));
}
kretfunc:fget {
printf("fd %d name %s\n", args.fd, str(retval->f_path.dentry->d_name.name));
}
fd 3 name ld.so.cache
fd 3 name libselinux.so.1
fd 3 name libselinux.so.1
...
kfunc:kvm:x86_emulate_insn { @ = count(); }
@ = 347603
kprobe and kretprobe
variants
• kprobe:fn
• kprobe:fn+offset
• kretprobe:fn
shortnames
• k
• kr
kprobe s allow for dynamic instrumentation of kernel functions. Each
time the specified kernel function is executed the attached BPF
programs are ran.
kprobe:tcp_reset {
@tcp_resets = count()
}
Function arguments are available through the argX and sargX builtins,
for register args and stack args respectively. Whether arguments passed
on stack or in a register depends on the architecture and the number or
arguments in used, e.g. on x86_64 the first non-floating point 6
arguments are passed in registers, all following arguments are passed
on the stack. Note that floating point arguments are typically passed
in special registers which don’t count as argX arguments which can
cause confusion. Consider a function with the following signature:
void func(int a, double d, int x)
Due to d being a floating point x is accessed through arg1 where one
might expect arg2.
bpftrace does not detect the function signature so it is not aware of
the argument count or their type. It is up to the user to perform Type
conversion when needed, e.g.
kprobe:tcp_connect
{
$sk = ((struct sock *) arg0);
...
}
kprobe s are not limited to function entry, they can be attached to any
instruction in a function by specifying an offset from the start of the
function.
kretprobe s trigger on the return from a kernel function. Return probes
do not have access to the function (input) arguments, only to the
return value (through retval). A common pattern to work around this is
by storing the arguments in a map on function entry and retrieving in
the return probe:
kprobe:d_lookup
{
$name = (struct qstr *)arg1;
@fname[tid] = $name->name;
}
kretprobe:d_lookup
/@fname[tid]/
{
printf("%-8d %-6d %-16s M %s\n", elapsed / 1e6, pid, comm,
str(@fname[tid]));
}
profile
variants
• profile:us:count
• profile:ms:count
• profile:s:count
• profile:hz:rate
shortnames
• p
Profile probes fire on each CPU on the specified interval.
software
variants
• software:event:
• software:event:count
shortnames
• s
The software probe attaches to pre-defined software events provided by
the kernel. Event details can be found in the perf_event_open(2) man
page.
The event names are:
• cpu-clock or cpu
• task-clock
• page-faults or faults
• context-switches or cs
• cpu-migrations
• minor-faults
• major-faults
• alignment-faults
• emulation-faults
• dummy
• bpf-output
tracepoint
variants
• tracepoint:subsys:event
shortnames
• t
Tracepoints are hooks into events in the kernel. Tracepoints are
defined in the kernel source and compiled into the kernel binary which
makes them a form of static tracing. Which means that unlike kprobe s
new tracepoints cannot be added without modifying the kernel.
The advantage of tracepoints is that they generally provide a more
stable interface than kprobe s do, they do not depend on the existence
of a kernel function.
Tracepoint arguments are available in the args struct which can be
inspected with verbose listing, see the LISTING PROBES section for more
details.
tracepoint:syscalls:sys_enter_openat {
printf("%s %s\n", comm, str(args.filename));
}
irqbalance /proc/interrupts
irqbalance /proc/stat
snmpd /proc/diskstats
snmpd /proc/stat
snmpd /proc/vmstat
snmpd /proc/net/dev
[...]
Additional information
• https://www.kernel.org/doc/html/latest/trace/tracepoints.html
rawtracepoint
variants
• rawtracepoint:event
shortnames
• rt
The hook point triggered by tracepoint and rawtracepoint is the same.
tracepoint and rawtracepoint are nearly identical in terms of
functionality. The only difference is in the program context.
rawtracepoint offers raw arguments to the tracepoint while tracepoint
applies further processing to the raw arguments. The additional
processing is defined inside the kernel.
Tracepoint arguments are available via the argN builtins. The available
arguments can be found in the relative path of the kernel source code
include/trace/events/. Each arg is a 64-bit integer.
rawtracepoint:block_rq_insert {
printf("%llx %llx\n", arg0, arg1);
}
ffff88810977d6f8 ffff8881097e8e80
[...]
uprobe, uretprobe
variants
• uprobe:binary:func
• uprobe:binary:func+offset
• uprobe:binary:offset
• uretprobe:binary:func
shortnames
• u
• ur
uprobe s or user-space probes are the user-space equivalent of kprobe
s. The same limitations that apply kprobe and kretprobe also apply to
uprobe s and uretprobe s.
When tracing libraries, it is sufficient to specify the library name
instead of a full path. The path will be then automatically resolved
using /etc/ld.so.cache:
# bpftrace -e 'uprobe:libc:malloc { printf("Allocated %d bytes\n", arg0); }'
Allocated 4 bytes
...
If the traced binary has DWARF included, function arguments are
available in the args struct which can be inspected with verbose
listing, see the LISTING PROBES section for more details.
When tracing C++ programs, it is possible to turn on automatic symbol
demangling by using the :cpp prefix:
# bpftrace -e 'u:src/bpftrace:cpp:"bpftrace::BPFtrace::add_probe" { print("adding probe\n"); }'
Attaching 1 probe...
adding probe
It is important to note that for uretprobe s to work the kernel runs a
special helper on user-space function entry which overrides the return
address on the stack. This can cause issues with languages that have
their own runtime like Golang:
example.go
func myprint(s string) {
fmt.Printf("Input: %s\n", s)
}
func main() {
ss := []string{"a", "b", "c"}
for _, s := range ss {
go myprint(s)
}
time.Sleep(1*time.Second)
}
bpftrace
# bpftrace -e 'uretprobe:./test:main.myprint { @=count(); }' -c ./test
runtime: unexpected return pc for main.myprint called from 0x7fffffffe000
stack: frame={sp:0xc00008cf60, fp:0xc00008cfd0} stack=[0xc00008c000,0xc00008d000)
fatal error: unknown caller pc
usdt
variants
• usdt:binary_path:probe_name
• usdt:binary_path:[probe_namespace]:probe_name
• usdt:library_path:probe_name
• usdt:library_path:[probe_namespace]:probe_name
shortnames
• U
You can target the entire host (or an entire process’s address space by
using the -p arg) by using a single wildcard in place of the
binary_path/library_path e.g. bpftrace -e 'usdt:*:loop {
printf("hi\n"); }. Please note that if you use wildcards for the
probe_name or probe_namespace and end up targeting multiple USDTs for
the same probe you might get errors if you also utilize the USDT
argument builtins (e.g. arg0) as they could be of different types.
watchpoint and asyncwatchpoint
variants
• watchpoint:absolute_address:length:mode
• watchpoint:function+argN:length:mode
shortnames
• w
• aw
These are memory watchpoints provided by the kernel. Whenever a memory
address is written to (w), read from (r), or executed (x), the kernel
can generate an event.
In the first form, an absolute address is monitored. If a pid (-p) or a
command (-c) is provided, bpftrace takes the address as a userspace
address and monitors the appropriate process. If not, bpftrace takes
the address as a kernel space address.
In the second form, the address present in argN when function is
entered is monitored. A pid or command must be provided for this form.
If synchronous (watchpoint), a SIGSTOP is sent to the tracee upon
function entry. The tracee will be SIGCONTed after the watchpoint is
attached. This is to ensure events are not missed. If you want to avoid
the SIGSTOP + SIGCONT use asyncwatchpoint.
Note that on most architectures you may not monitor for execution while
monitoring read or write.
Examples
Print hit when a read from or write to 0x10000000 happens:
# bpftrace -e 'watchpoint:0x10000000:8:rw { printf("hit!\n"); exit(); }' -c ./testprogs/watchpoint
Print the call stack every time the jiffies variable is updated:
# bpftrace -e "watchpoint:0x$(awk '$3 == "jiffies" {print $1}' /proc/kallsyms):8:w {
@[kstack] = count();
}
i:s:1 { exit(); }"
......
@[
do_timer+12
tick_do_update_jiffies64.part.22+89
tick_sched_do_timer+103
tick_sched_timer+39
__hrtimer_run_queues+256
hrtimer_interrupt+256
smp_apic_timer_interrupt+106
apic_timer_interrupt+15
cpuidle_enter_state+188
cpuidle_enter+41
do_idle+536
cpu_startup_entry+25
start_secondary+355
secondary_startup_64+164
]: 319
"hit" and exit when the memory pointed to by arg1 of increment is
written to.
# cat wpfunc.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
__attribute__((noinline))
void increment(__attribute__((unused)) int _, int *i)
{
(*i)++;
}
int main()
{
int *i = malloc(sizeof(int));
while (1)
{
increment(0, i);
(*i)++;
usleep(1000);
}
}
# bpftrace -e 'watchpoint:increment+arg1:4:w { printf("hit!\n"); exit() }' -c ./wpfunc
LISTING PROBES
Probe listing is the method to discover which probes are supported by
the current system. Listing supports the same syntax as normal
attachment does:
# bpftrace -l 'kprobe:*'
# bpftrace -l 't:syscalls:*openat*
# bpftrace -l 'kprobe:tcp*,trace
# bpftrace -l 'k:*socket*,tracepoint:syscalls:*tcp*'
The verbose flag (-v) can be specified to inspect arguments (args) for
providers that support it:
# bpftrace -l 'fr:tcp_reset,t:syscalls:sys_enter_openat' -v
kretfunc:tcp_reset
struct sock * sk
struct sk_buff * skb
tracepoint:syscalls:sys_enter_openat
int __syscall_nr
int dfd
const char * filename
int flags
umode_t mode
# bpftrace -l 'uprobe:/bin/bash:rl_set_prompt' -v # works only if /bin/bash has DWARF
uprobe:/bin/bash:rl_set_prompt
const char *prompt
2025-02-05 BPFTRACE(8)
Generated by dwww version 1.16 on Tue Dec 16 16:50:13 CET 2025.