|
|
@@ -0,0 +1,139 @@
|
|
|
+ Kprobe-based Event Tracer
|
|
|
+ =========================
|
|
|
+
|
|
|
+ Documentation is written by Masami Hiramatsu
|
|
|
+
|
|
|
+
|
|
|
+Overview
|
|
|
+--------
|
|
|
+This tracer is similar to the events tracer which is based on Tracepoint
|
|
|
+infrastructure. Instead of Tracepoint, this tracer is based on kprobes(kprobe
|
|
|
+and kretprobe). It probes anywhere where kprobes can probe(this means, all
|
|
|
+functions body except for __kprobes functions).
|
|
|
+
|
|
|
+Unlike the function tracer, this tracer can probe instructions inside of
|
|
|
+kernel functions. It allows you to check which instruction has been executed.
|
|
|
+
|
|
|
+Unlike the Tracepoint based events tracer, this tracer can add and remove
|
|
|
+probe points on the fly.
|
|
|
+
|
|
|
+Similar to the events tracer, this tracer doesn't need to be activated via
|
|
|
+current_tracer, instead of that, just set probe points via
|
|
|
+/sys/kernel/debug/tracing/kprobe_events. And you can set filters on each
|
|
|
+probe events via /sys/kernel/debug/tracing/events/kprobes/<EVENT>/filter.
|
|
|
+
|
|
|
+
|
|
|
+Synopsis of kprobe_events
|
|
|
+-------------------------
|
|
|
+ p[:EVENT] SYMBOL[+offs|-offs]|MEMADDR [FETCHARGS] : Set a probe
|
|
|
+ r[:EVENT] SYMBOL[+0] [FETCHARGS] : Set a return probe
|
|
|
+
|
|
|
+ EVENT : Event name.
|
|
|
+ SYMBOL[+offs|-offs] : Symbol+offset where the probe is inserted.
|
|
|
+ MEMADDR : Address where the probe is inserted.
|
|
|
+
|
|
|
+ FETCHARGS : Arguments.
|
|
|
+ %REG : Fetch register REG
|
|
|
+ sN : Fetch Nth entry of stack (N >= 0)
|
|
|
+ sa : Fetch stack address.
|
|
|
+ @ADDR : Fetch memory at ADDR (ADDR should be in kernel)
|
|
|
+ @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol)
|
|
|
+ aN : Fetch function argument. (N >= 0)(*)
|
|
|
+ rv : Fetch return value.(**)
|
|
|
+ ra : Fetch return address.(**)
|
|
|
+ +|-offs(FETCHARG) : fetch memory at FETCHARG +|- offs address.(***)
|
|
|
+
|
|
|
+ (*) aN may not correct on asmlinkaged functions and at the middle of
|
|
|
+ function body.
|
|
|
+ (**) only for return probe.
|
|
|
+ (***) this is useful for fetching a field of data structures.
|
|
|
+
|
|
|
+
|
|
|
+Per-Probe Event Filtering
|
|
|
+-------------------------
|
|
|
+ Per-probe event filtering feature allows you to set different filter on each
|
|
|
+probe and gives you what arguments will be shown in trace buffer. If an event
|
|
|
+name is specified right after 'p:' or 'r:' in kprobe_events, the tracer adds
|
|
|
+an event under tracing/events/kprobes/<EVENT>, at the directory you can see
|
|
|
+'id', 'enabled', 'format' and 'filter'.
|
|
|
+
|
|
|
+enabled:
|
|
|
+ You can enable/disable the probe by writing 1 or 0 on it.
|
|
|
+
|
|
|
+format:
|
|
|
+ It shows the format of this probe event. It also shows aliases of arguments
|
|
|
+ which you specified to kprobe_events.
|
|
|
+
|
|
|
+filter:
|
|
|
+ You can write filtering rules of this event. And you can use both of aliase
|
|
|
+ names and field names for describing filters.
|
|
|
+
|
|
|
+
|
|
|
+Usage examples
|
|
|
+--------------
|
|
|
+To add a probe as a new event, write a new definition to kprobe_events
|
|
|
+as below.
|
|
|
+
|
|
|
+ echo p:myprobe do_sys_open a0 a1 a2 a3 > /sys/kernel/debug/tracing/kprobe_events
|
|
|
+
|
|
|
+ This sets a kprobe on the top of do_sys_open() function with recording
|
|
|
+1st to 4th arguments as "myprobe" event.
|
|
|
+
|
|
|
+ echo r:myretprobe do_sys_open rv ra >> /sys/kernel/debug/tracing/kprobe_events
|
|
|
+
|
|
|
+ This sets a kretprobe on the return point of do_sys_open() function with
|
|
|
+recording return value and return address as "myretprobe" event.
|
|
|
+ You can see the format of these events via
|
|
|
+/sys/kernel/debug/tracing/events/kprobes/<EVENT>/format.
|
|
|
+
|
|
|
+ cat /sys/kernel/debug/tracing/events/kprobes/myprobe/format
|
|
|
+name: myprobe
|
|
|
+ID: 23
|
|
|
+format:
|
|
|
+ field:unsigned short common_type; offset:0; size:2;
|
|
|
+ field:unsigned char common_flags; offset:2; size:1;
|
|
|
+ field:unsigned char common_preempt_count; offset:3; size:1;
|
|
|
+ field:int common_pid; offset:4; size:4;
|
|
|
+ field:int common_tgid; offset:8; size:4;
|
|
|
+
|
|
|
+ field: unsigned long ip; offset:16;tsize:8;
|
|
|
+ field: int nargs; offset:24;tsize:4;
|
|
|
+ field: unsigned long arg0; offset:32;tsize:8;
|
|
|
+ field: unsigned long arg1; offset:40;tsize:8;
|
|
|
+ field: unsigned long arg2; offset:48;tsize:8;
|
|
|
+ field: unsigned long arg3; offset:56;tsize:8;
|
|
|
+
|
|
|
+ alias: a0; original: arg0;
|
|
|
+ alias: a1; original: arg1;
|
|
|
+ alias: a2; original: arg2;
|
|
|
+ alias: a3; original: arg3;
|
|
|
+
|
|
|
+print fmt: "%lx: 0x%lx 0x%lx 0x%lx 0x%lx", ip, arg0, arg1, arg2, arg3
|
|
|
+
|
|
|
+
|
|
|
+ You can see that the event has 4 arguments and alias expressions
|
|
|
+corresponding to it.
|
|
|
+
|
|
|
+ echo > /sys/kernel/debug/tracing/kprobe_events
|
|
|
+
|
|
|
+ This clears all probe points. and you can see the traced information via
|
|
|
+/sys/kernel/debug/tracing/trace.
|
|
|
+
|
|
|
+ cat /sys/kernel/debug/tracing/trace
|
|
|
+# tracer: nop
|
|
|
+#
|
|
|
+# TASK-PID CPU# TIMESTAMP FUNCTION
|
|
|
+# | | | | |
|
|
|
+ <...>-1447 [001] 1038282.286875: do_sys_open+0x0/0xd6: 0x3 0x7fffd1ec4440 0x8000 0x0
|
|
|
+ <...>-1447 [001] 1038282.286878: sys_openat+0xc/0xe <- do_sys_open: 0xfffffffffffffffe 0xffffffff81367a3a
|
|
|
+ <...>-1447 [001] 1038282.286885: do_sys_open+0x0/0xd6: 0xffffff9c 0x40413c 0x8000 0x1b6
|
|
|
+ <...>-1447 [001] 1038282.286915: sys_open+0x1b/0x1d <- do_sys_open: 0x3 0xffffffff81367a3a
|
|
|
+ <...>-1447 [001] 1038282.286969: do_sys_open+0x0/0xd6: 0xffffff9c 0x4041c6 0x98800 0x10
|
|
|
+ <...>-1447 [001] 1038282.286976: sys_open+0x1b/0x1d <- do_sys_open: 0x3 0xffffffff81367a3a
|
|
|
+
|
|
|
+
|
|
|
+ Each line shows when the kernel hits a probe, and <- SYMBOL means kernel
|
|
|
+returns from SYMBOL(e.g. "sys_open+0x1b/0x1d <- do_sys_open" means kernel
|
|
|
+returns from do_sys_open to sys_open+0x1b).
|
|
|
+
|
|
|
+
|
Masami Hiramatsu