diff options
Diffstat (limited to 'Documentation/trace')
-rw-r--r-- | Documentation/trace/events-kmem.rst (renamed from Documentation/trace/events-kmem.txt) | 50 | ||||
-rw-r--r-- | Documentation/trace/events-msr.rst | 40 | ||||
-rw-r--r-- | Documentation/trace/events-msr.txt | 37 | ||||
-rw-r--r-- | Documentation/trace/events-nmi.rst | 45 | ||||
-rw-r--r-- | Documentation/trace/events-nmi.txt | 43 | ||||
-rw-r--r-- | Documentation/trace/events-power.rst (renamed from Documentation/trace/events-power.txt) | 52 | ||||
-rw-r--r-- | Documentation/trace/events.rst (renamed from Documentation/trace/events.txt) | 677 | ||||
-rw-r--r-- | Documentation/trace/ftrace-design.rst (renamed from Documentation/trace/ftrace-design.txt) | 252 | ||||
-rw-r--r-- | Documentation/trace/ftrace-uses.rst | 23 | ||||
-rw-r--r-- | Documentation/trace/ftrace.rst | 3332 | ||||
-rw-r--r-- | Documentation/trace/ftrace.txt | 3220 | ||||
-rw-r--r-- | Documentation/trace/hwlat_detector.rst (renamed from Documentation/trace/hwlat_detector.txt) | 26 | ||||
-rw-r--r-- | Documentation/trace/index.rst | 23 | ||||
-rw-r--r-- | Documentation/trace/intel_th.rst (renamed from Documentation/trace/intel_th.txt) | 43 | ||||
-rw-r--r-- | Documentation/trace/kprobetrace.rst (renamed from Documentation/trace/kprobetrace.txt) | 100 | ||||
-rw-r--r-- | Documentation/trace/mmiotrace.rst (renamed from Documentation/trace/mmiotrace.txt) | 86 | ||||
-rw-r--r-- | Documentation/trace/stm.rst (renamed from Documentation/trace/stm.txt) | 23 | ||||
-rw-r--r-- | Documentation/trace/tracepoint-analysis.rst (renamed from Documentation/trace/tracepoint-analysis.txt) | 41 | ||||
-rw-r--r-- | Documentation/trace/tracepoints.rst (renamed from Documentation/trace/tracepoints.txt) | 77 | ||||
-rw-r--r-- | Documentation/trace/uprobetracer.rst (renamed from Documentation/trace/uprobetracer.txt) | 44 |
20 files changed, 4240 insertions, 3994 deletions
diff --git a/Documentation/trace/events-kmem.txt b/Documentation/trace/events-kmem.rst index 194800410061..555484110e36 100644 --- a/Documentation/trace/events-kmem.txt +++ b/Documentation/trace/events-kmem.rst @@ -1,22 +1,26 @@ - Subsystem Trace Points: kmem +============================ +Subsystem Trace Points: kmem +============================ The kmem tracing system captures events related to object and page allocation within the kernel. Broadly speaking there are five major subheadings. - o Slab allocation of small objects of unknown type (kmalloc) - o Slab allocation of small objects of known type - o Page allocation - o Per-CPU Allocator Activity - o External Fragmentation + - Slab allocation of small objects of unknown type (kmalloc) + - Slab allocation of small objects of known type + - Page allocation + - Per-CPU Allocator Activity + - External Fragmentation This document describes what each of the tracepoints is and why they might be useful. 1. Slab allocation of small objects of unknown type =================================================== -kmalloc call_site=%lx ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s -kmalloc_node call_site=%lx ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s node=%d -kfree call_site=%lx ptr=%p +:: + + kmalloc call_site=%lx ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s + kmalloc_node call_site=%lx ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s node=%d + kfree call_site=%lx ptr=%p Heavy activity for these events may indicate that a specific cache is justified, particularly if kmalloc slab pages are getting significantly @@ -27,9 +31,11 @@ the allocation sites were. 2. Slab allocation of small objects of known type ================================================= -kmem_cache_alloc call_site=%lx ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s -kmem_cache_alloc_node call_site=%lx ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s node=%d -kmem_cache_free call_site=%lx ptr=%p +:: + + kmem_cache_alloc call_site=%lx ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s + kmem_cache_alloc_node call_site=%lx ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s node=%d + kmem_cache_free call_site=%lx ptr=%p These events are similar in usage to the kmalloc-related events except that it is likely easier to pin the event down to a specific cache. At the time @@ -38,10 +44,12 @@ but the call_site can usually be used to extrapolate that information. 3. Page allocation ================== -mm_page_alloc page=%p pfn=%lu order=%d migratetype=%d gfp_flags=%s -mm_page_alloc_zone_locked page=%p pfn=%lu order=%u migratetype=%d cpu=%d percpu_refill=%d -mm_page_free page=%p pfn=%lu order=%d -mm_page_free_batched page=%p pfn=%lu order=%d cold=%d +:: + + mm_page_alloc page=%p pfn=%lu order=%d migratetype=%d gfp_flags=%s + mm_page_alloc_zone_locked page=%p pfn=%lu order=%u migratetype=%d cpu=%d percpu_refill=%d + mm_page_free page=%p pfn=%lu order=%d + mm_page_free_batched page=%p pfn=%lu order=%d cold=%d These four events deal with page allocation and freeing. mm_page_alloc is a simple indicator of page allocator activity. Pages may be allocated from @@ -65,8 +73,10 @@ contention on the zone->lru_lock. 4. Per-CPU Allocator Activity ============================= -mm_page_alloc_zone_locked page=%p pfn=%lu order=%u migratetype=%d cpu=%d percpu_refill=%d -mm_page_pcpu_drain page=%p pfn=%lu order=%d cpu=%d migratetype=%d +:: + + mm_page_alloc_zone_locked page=%p pfn=%lu order=%u migratetype=%d cpu=%d percpu_refill=%d + mm_page_pcpu_drain page=%p pfn=%lu order=%d cpu=%d migratetype=%d In front of the page allocator is a per-cpu page allocator. It exists only for order-0 pages, reduces contention on the zone->lock and reduces the @@ -92,7 +102,9 @@ can be allocated and freed on the same CPU through some algorithm change. 5. External Fragmentation ========================= -mm_page_alloc_extfrag page=%p pfn=%lu alloc_order=%d fallback_order=%d pageblock_order=%d alloc_migratetype=%d fallback_migratetype=%d fragmenting=%d change_ownership=%d +:: + + mm_page_alloc_extfrag page=%p pfn=%lu alloc_order=%d fallback_order=%d pageblock_order=%d alloc_migratetype=%d fallback_migratetype=%d fragmenting=%d change_ownership=%d External fragmentation affects whether a high-order allocation will be successful or not. For some types of hardware, this is important although diff --git a/Documentation/trace/events-msr.rst b/Documentation/trace/events-msr.rst new file mode 100644 index 000000000000..e938aa0b6f4f --- /dev/null +++ b/Documentation/trace/events-msr.rst @@ -0,0 +1,40 @@ +================ +MSR Trace Events +================ + +The x86 kernel supports tracing most MSR (Model Specific Register) accesses. +To see the definition of the MSRs on Intel systems please see the SDM +at http://www.intel.com/sdm (Volume 3) + +Available trace points: + +/sys/kernel/debug/tracing/events/msr/ + +Trace MSR reads: + +read_msr + + - msr: MSR number + - val: Value written + - failed: 1 if the access failed, otherwise 0 + + +Trace MSR writes: + +write_msr + + - msr: MSR number + - val: Value written + - failed: 1 if the access failed, otherwise 0 + + +Trace RDPMC in kernel: + +rdpmc + +The trace data can be post processed with the postprocess/decode_msr.py script:: + + cat /sys/kernel/debug/tracing/trace | decode_msr.py /usr/src/linux/include/asm/msr-index.h + +to add symbolic MSR names. + diff --git a/Documentation/trace/events-msr.txt b/Documentation/trace/events-msr.txt deleted file mode 100644 index 78c383bf06aa..000000000000 --- a/Documentation/trace/events-msr.txt +++ /dev/null @@ -1,37 +0,0 @@ - -The x86 kernel supports tracing most MSR (Model Specific Register) accesses. -To see the definition of the MSRs on Intel systems please see the SDM -at http://www.intel.com/sdm (Volume 3) - -Available trace points: - -/sys/kernel/debug/tracing/events/msr/ - -Trace MSR reads - -read_msr - -msr: MSR number -val: Value written -failed: 1 if the access failed, otherwise 0 - - -Trace MSR writes - -write_msr - -msr: MSR number -val: Value written -failed: 1 if the access failed, otherwise 0 - - -Trace RDPMC in kernel - -rdpmc - -The trace data can be post processed with the postprocess/decode_msr.py script - -cat /sys/kernel/debug/tracing/trace | decode_msr.py /usr/src/linux/include/asm/msr-index.h - -to add symbolic MSR names. - diff --git a/Documentation/trace/events-nmi.rst b/Documentation/trace/events-nmi.rst new file mode 100644 index 000000000000..9e0a7289d80a --- /dev/null +++ b/Documentation/trace/events-nmi.rst @@ -0,0 +1,45 @@ +================ +NMI Trace Events +================ + +These events normally show up here: + + /sys/kernel/debug/tracing/events/nmi + + +nmi_handler +----------- + +You might want to use this tracepoint if you suspect that your +NMI handlers are hogging large amounts of CPU time. The kernel +will warn if it sees long-running handlers:: + + INFO: NMI handler took too long to run: 9.207 msecs + +and this tracepoint will allow you to drill down and get some +more details. + +Let's say you suspect that perf_event_nmi_handler() is causing +you some problems and you only want to trace that handler +specifically. You need to find its address:: + + $ grep perf_event_nmi_handler /proc/kallsyms + ffffffff81625600 t perf_event_nmi_handler + +Let's also say you are only interested in when that function is +really hogging a lot of CPU time, like a millisecond at a time. +Note that the kernel's output is in milliseconds, but the input +to the filter is in nanoseconds! You can filter on 'delta_ns':: + + cd /sys/kernel/debug/tracing/events/nmi/nmi_handler + echo 'handler==0xffffffff81625600 && delta_ns>1000000' > filter + echo 1 > enable + +Your output would then look like:: + + $ cat /sys/kernel/debug/tracing/trace_pipe + <idle>-0 [000] d.h3 505.397558: nmi_handler: perf_event_nmi_handler() delta_ns: 3236765 handled: 1 + <idle>-0 [000] d.h3 505.805893: nmi_handler: perf_event_nmi_handler() delta_ns: 3174234 handled: 1 + <idle>-0 [000] d.h3 506.158206: nmi_handler: perf_event_nmi_handler() delta_ns: 3084642 handled: 1 + <idle>-0 [000] d.h3 506.334346: nmi_handler: perf_event_nmi_handler() delta_ns: 3080351 handled: 1 + diff --git a/Documentation/trace/events-nmi.txt b/Documentation/trace/events-nmi.txt deleted file mode 100644 index c03c8c89f08d..000000000000 --- a/Documentation/trace/events-nmi.txt +++ /dev/null @@ -1,43 +0,0 @@ -NMI Trace Events - -These events normally show up here: - - /sys/kernel/debug/tracing/events/nmi - --- - -nmi_handler: - -You might want to use this tracepoint if you suspect that your -NMI handlers are hogging large amounts of CPU time. The kernel -will warn if it sees long-running handlers: - - INFO: NMI handler took too long to run: 9.207 msecs - -and this tracepoint will allow you to drill down and get some -more details. - -Let's say you suspect that perf_event_nmi_handler() is causing -you some problems and you only want to trace that handler -specifically. You need to find its address: - - $ grep perf_event_nmi_handler /proc/kallsyms - ffffffff81625600 t perf_event_nmi_handler - -Let's also say you are only interested in when that function is -really hogging a lot of CPU time, like a millisecond at a time. -Note that the kernel's output is in milliseconds, but the input -to the filter is in nanoseconds! You can filter on 'delta_ns': - -cd /sys/kernel/debug/tracing/events/nmi/nmi_handler -echo 'handler==0xffffffff81625600 && delta_ns>1000000' > filter -echo 1 > enable - -Your output would then look like: - -$ cat /sys/kernel/debug/tracing/trace_pipe -<idle>-0 [000] d.h3 505.397558: nmi_handler: perf_event_nmi_handler() delta_ns: 3236765 handled: 1 -<idle>-0 [000] d.h3 505.805893: nmi_handler: perf_event_nmi_handler() delta_ns: 3174234 handled: 1 -<idle>-0 [000] d.h3 506.158206: nmi_handler: perf_event_nmi_handler() delta_ns: 3084642 handled: 1 -<idle>-0 [000] d.h3 506.334346: nmi_handler: perf_event_nmi_handler() delta_ns: 3080351 handled: 1 - diff --git a/Documentation/trace/events-power.txt b/Documentation/trace/events-power.rst index 21d514ced212..a77daca75e30 100644 --- a/Documentation/trace/events-power.txt +++ b/Documentation/trace/events-power.rst @@ -1,13 +1,14 @@ - - Subsystem Trace Points: power +============================= +Subsystem Trace Points: power +============================= The power tracing system captures events related to power transitions within the kernel. Broadly speaking there are three major subheadings: - o Power state switch which reports events related to suspend (S-states), - cpuidle (C-states) and cpufreq (P-states) - o System clock related changes - o Power domains related changes and transitions + - Power state switch which reports events related to suspend (S-states), + cpuidle (C-states) and cpufreq (P-states) + - System clock related changes + - Power domains related changes and transitions This document describes what each of the tracepoints is and why they might be useful. @@ -22,14 +23,16 @@ Cf. include/trace/events/power.h for the events definitions. A 'cpu' event class gathers the CPU-related events: cpuidle and cpufreq. +:: -cpu_idle "state=%lu cpu_id=%lu" -cpu_frequency "state=%lu cpu_id=%lu" + cpu_idle "state=%lu cpu_id=%lu" + cpu_frequency "state=%lu cpu_id=%lu" A suspend event is used to indicate the system going in and out of the suspend mode: +:: -machine_suspend "state=%lu" + machine_suspend "state=%lu" Note: the value of '-1' or '4294967295' for state means an exit from the current state, @@ -45,10 +48,11 @@ correctly draw the states diagrams and to calculate accurate statistics etc. ================ The clock events are used for clock enable/disable and for clock rate change. +:: -clock_enable "%s state=%lu cpu_id=%lu" -clock_disable "%s state=%lu cpu_id=%lu" -clock_set_rate "%s state=%lu cpu_id=%lu" + clock_enable "%s state=%lu cpu_id=%lu" + clock_disable "%s state=%lu cpu_id=%lu" + clock_set_rate "%s state=%lu cpu_id=%lu" The first parameter gives the clock name (e.g. "gpio1_iclk"). The second parameter is '1' for enable, '0' for disable, the target @@ -57,8 +61,9 @@ clock rate for set_rate. 3. Power domains events ======================= The power domain events are used for power domains transitions +:: -power_domain_target "%s state=%lu cpu_id=%lu" + power_domain_target "%s state=%lu cpu_id=%lu" The first parameter gives the power domain name (e.g. "mpu_pwrdm"). The second parameter is the power domain target state. @@ -67,28 +72,31 @@ The second parameter is the power domain target state. ================ The PM QoS events are used for QoS add/update/remove request and for target/flags update. +:: -pm_qos_add_request "pm_qos_class=%s value=%d" -pm_qos_update_request "pm_qos_class=%s value=%d" -pm_qos_remove_request "pm_qos_class=%s value=%d" -pm_qos_update_request_timeout "pm_qos_class=%s value=%d, timeout_us=%ld" + pm_qos_add_request "pm_qos_class=%s value=%d" + pm_qos_update_request "pm_qos_class=%s value=%d" + pm_qos_remove_request "pm_qos_class=%s value=%d" + pm_qos_update_request_timeout "pm_qos_class=%s value=%d, timeout_us=%ld" The first parameter gives the QoS class name (e.g. "CPU_DMA_LATENCY"). The second parameter is value to be added/updated/removed. The third parameter is timeout value in usec. +:: -pm_qos_update_target "action=%s prev_value=%d curr_value=%d" -pm_qos_update_flags "action=%s prev_value=0x%x curr_value=0x%x" + pm_qos_update_target "action=%s prev_value=%d curr_value=%d" + pm_qos_update_flags "action=%s prev_value=0x%x curr_value=0x%x" The first parameter gives the QoS action name (e.g. "ADD_REQ"). The second parameter is the previous QoS value. The third parameter is the current QoS value to update. And, there are also events used for device PM QoS add/update/remove request. +:: -dev_pm_qos_add_request "device=%s type=%s new_value=%d" -dev_pm_qos_update_request "device=%s type=%s new_value=%d" -dev_pm_qos_remove_request "device=%s type=%s new_value=%d" + dev_pm_qos_add_request "device=%s type=%s new_value=%d" + dev_pm_qos_update_request "device=%s type=%s new_value=%d" + dev_pm_qos_remove_request "device=%s type=%s new_value=%d" The first parameter gives the device name which tries to add/update/remove QoS requests. diff --git a/Documentation/trace/events.txt b/Documentation/trace/events.rst index 2cc08d4a326e..bdf1963ba6ba 100644 --- a/Documentation/trace/events.txt +++ b/Documentation/trace/events.rst @@ -1,7 +1,9 @@ - Event Tracing +============= +Event Tracing +============= - Documentation written by Theodore Ts'o - Updated by Li Zefan and Tom Zanussi +:Author: Theodore Ts'o +:Updated: Li Zefan and Tom Zanussi 1. Introduction =============== @@ -25,23 +27,22 @@ The events which are available for tracing can be found in the file /sys/kernel/debug/tracing/available_events. To enable a particular event, such as 'sched_wakeup', simply echo it -to /sys/kernel/debug/tracing/set_event. For example: +to /sys/kernel/debug/tracing/set_event. For example:: # echo sched_wakeup >> /sys/kernel/debug/tracing/set_event -[ Note: '>>' is necessary, otherwise it will firstly disable - all the events. ] +.. Note:: '>>' is necessary, otherwise it will firstly disable all the events. To disable an event, echo the event name to the set_event file prefixed -with an exclamation point: +with an exclamation point:: # echo '!sched_wakeup' >> /sys/kernel/debug/tracing/set_event -To disable all events, echo an empty line to the set_event file: +To disable all events, echo an empty line to the set_event file:: # echo > /sys/kernel/debug/tracing/set_event -To enable all events, echo '*:*' or '*:' to the set_event file: +To enable all events, echo ``*:*`` or ``*:`` to the set_event file:: # echo *:* > /sys/kernel/debug/tracing/set_event @@ -49,8 +50,8 @@ The events are organized into subsystems, such as ext4, irq, sched, etc., and a full event name looks like this: <subsystem>:<event>. The subsystem name is optional, but it is displayed in the available_events file. All of the events in a subsystem can be specified via the syntax -"<subsystem>:*"; for example, to enable all irq events, you can use the -command: +``<subsystem>:*``; for example, to enable all irq events, you can use the +command:: # echo 'irq:*' > /sys/kernel/debug/tracing/set_event @@ -60,33 +61,33 @@ command: The events available are also listed in /sys/kernel/debug/tracing/events/ hierarchy of directories. -To enable event 'sched_wakeup': +To enable event 'sched_wakeup':: # echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable -To disable it: +To disable it:: # echo 0 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable -To enable all events in sched subsystem: +To enable all events in sched subsystem:: # echo 1 > /sys/kernel/debug/tracing/events/sched/enable -To enable all events: +To enable all events:: # echo 1 > /sys/kernel/debug/tracing/events/enable When reading one of these enable files, there are four results: - 0 - all events this file affects are disabled - 1 - all events this file affects are enabled - X - there is a mixture of events enabled and disabled - ? - this file does not affect any event + - 0 - all events this file affects are disabled + - 1 - all events this file affects are enabled + - X - there is a mixture of events enabled and disabled + - ? - this file does not affect any event 2.3 Boot option --------------- -In order to facilitate early boot debugging, use boot option: +In order to facilitate early boot debugging, use boot option:: trace_event=[event-list] @@ -110,12 +111,12 @@ It also displays the format string that will be used to print the event in text mode, along with the event name and ID used for profiling. -Every event has a set of 'common' fields associated with it; these are -the fields prefixed with 'common_'. The other fields vary between +Every event has a set of ``common`` fields associated with it; these are +the fields prefixed with ``common_``. The other fields vary between events and correspond to the fields defined in the TRACE_EVENT definition for that event. -Each field in the format has the form: +Each field in the format has the form:: field:field-type field-name; offset:N; size:N; @@ -123,27 +124,27 @@ where offset is the offset of the field in the trace record and size is the size of the data item, in bytes. For example, here's the information displayed for the 'sched_wakeup' -event: +event:: -# cat /sys/kernel/debug/tracing/events/sched/sched_wakeup/format + # cat /sys/kernel/debug/tracing/events/sched/sched_wakeup/format -name: sched_wakeup -ID: 60 -format: - field:unsigned short common_type; offset:0; size:2; - field:unsigned char common_flags; offset:2; size:1; - field:unsigned char common_preempt_count; offset:3; size:1; - field:int common_pid; offset:4; size:4; - field:int common_tgid; offset:8; size:4; + name: sched_wakeup + ID: 60 + format: + field:unsigned short common_type; offset:0; size:2; + field:unsigned char common_flags; offset:2; size:1; + field:unsigned char common_preempt_count; offset:3; size:1; + field:int common_pid; offset:4; size:4; + field:int common_tgid; offset:8; size:4; - field:char comm[TASK_COMM_LEN]; offset:12; size:16; - field:pid_t pid; offset:28; size:4; - field:int prio; offset:32; size:4; - field:int success; offset:36; size:4; - field:int cpu; offset:40; size:4; + field:char comm[TASK_COMM_LEN]; offset:12; size:16; + field:pid_t pid; offset:28; size:4; + field:int prio; offset:32; size:4; + field:int success; offset:36; size:4; + field:int cpu; offset:40; size:4; -print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid, - REC->prio, REC->success, REC->cpu + print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid, + REC->prio, REC->success, REC->cpu This event contains 10 fields, the first 5 common and the remaining 5 event-specific. All the fields for this event are numeric, except for @@ -168,7 +169,7 @@ A filter expression consists of one or more 'predicates' that can be combined using the logical operators '&&' and '||'. A predicate is simply a clause that compares the value of a field contained within a logged event with a constant value and returns either 0 or 1 depending -on whether the field value matched (1) or didn't match (0): +on whether the field value matched (1) or didn't match (0):: field-name relational-operator value @@ -189,8 +190,8 @@ And for string fields they are: ==, !=, ~ -The glob (~) accepts a wild card character (*,?) and character classes -([). For example: +The glob (~) accepts a wild card character (\*,?) and character classes +([). For example:: prev_comm ~ "*sh" prev_comm ~ "sh*" @@ -203,27 +204,27 @@ The glob (~) accepts a wild card character (*,?) and character classes A filter for an individual event is set by writing a filter expression to the 'filter' file for the given event. -For example: +For example:: -# cd /sys/kernel/debug/tracing/events/sched/sched_wakeup -# echo "common_preempt_count > 4" > filter + # cd /sys/kernel/debug/tracing/events/sched/sched_wakeup + # echo "common_preempt_count > 4" > filter -A slightly more involved example: +A slightly more involved example:: -# cd /sys/kernel/debug/tracing/events/signal/signal_generate -# echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter + # cd /sys/kernel/debug/tracing/events/signal/signal_generate + # echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter If there is an error in the expression, you'll get an 'Invalid argument' error when setting it, and the erroneous string along with -an error message can be seen by looking at the filter e.g.: +an error message can be seen by looking at the filter e.g.:: -# cd /sys/kernel/debug/tracing/events/signal/signal_generate -# echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter --bash: echo: write error: Invalid argument -# cat filter -((sig >= 10 && sig < 15) || dsig == 17) && comm != bash -^ -parse_error: Field not found + # cd /sys/kernel/debug/tracing/events/signal/signal_generate + # echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter + -bash: echo: write error: Invalid argument + # cat filter + ((sig >= 10 && sig < 15) || dsig == 17) && comm != bash + ^ + parse_error: Field not found Currently the caret ('^') for an error always appears at the beginning of the filter string; the error message should still be useful though @@ -255,35 +256,35 @@ fields can be guaranteed to propagate successfully to all events. Here are a few subsystem filter examples that also illustrate the above points: -Clear the filters on all events in the sched subsystem: +Clear the filters on all events in the sched subsystem:: -# cd /sys/kernel/debug/tracing/events/sched -# echo 0 > filter -# cat sched_switch/filter -none -# cat sched_wakeup/filter -none + # cd /sys/kernel/debug/tracing/events/sched + # echo 0 > filter + # cat sched_switch/filter + none + # cat sched_wakeup/filter + none Set a filter using only common fields for all events in the sched -subsystem (all events end up with the same filter): +subsystem (all events end up with the same filter):: -# cd /sys/kernel/debug/tracing/events/sched -# echo common_pid == 0 > filter -# cat sched_switch/filter -common_pid == 0 -# cat sched_wakeup/filter -common_pid == 0 + # cd /sys/kernel/debug/tracing/events/sched + # echo common_pid == 0 > filter + # cat sched_switch/filter + common_pid == 0 + # cat sched_wakeup/filter + common_pid == 0 Attempt to set a filter using a non-common field for all events in the sched subsystem (all events but those that have a prev_pid field retain -their old filters): +their old filters):: -# cd /sys/kernel/debug/tracing/events/sched -# echo prev_pid == 0 > filter -# cat sched_switch/filter -prev_pid == 0 -# cat sched_wakeup/filter -common_pid == 0 + # cd /sys/kernel/debug/tracing/events/sched + # echo prev_pid == 0 > filter + # cat sched_switch/filter + prev_pid == 0 + # cat sched_wakeup/filter + common_pid == 0 5.4 PID filtering ----------------- @@ -291,16 +292,18 @@ common_pid == 0 The set_event_pid file in the same directory as the top events directory exists, will filter all events from tracing any task that does not have the PID listed in the set_event_pid file. +:: -# cd /sys/kernel/debug/tracing -# echo $$ > set_event_pid -# echo 1 > events/enabled + # cd /sys/kernel/debug/tracing + # echo $$ > set_event_pid + # echo 1 > events/enable Will only trace events for the current task. To add more PIDs without losing the PIDs already included, use '>>'. +:: -# echo 123 244 1 >> set_event_pid + # echo 123 244 1 >> set_event_pid 6. Event triggers @@ -342,12 +345,12 @@ way, so beware about making generalizations between the two. 6.1 Expression syntax --------------------- -Triggers are added by echoing the command to the 'trigger' file: +Triggers are added by echoing the command to the 'trigger' file:: # echo 'command[:count] [if filter]' > trigger Triggers are removed by echoing the same command but starting with '!' -to the 'trigger' file: +to the 'trigger' file:: # echo '!command[:count] [if filter]' > trigger @@ -379,30 +382,30 @@ The following commands are supported: For example, the following trigger causes kmalloc events to be traced when a read system call is entered, and the :1 at the end - specifies that this enablement happens only once: + specifies that this enablement happens only once:: - # echo 'enable_event:kmem:kmalloc:1' > \ - /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger + # echo 'enable_event:kmem:kmalloc:1' > \ + /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger The following trigger causes kmalloc events to stop being traced when a read system call exits. This disablement happens on every - read system call exit: + read system call exit:: - # echo 'disable_event:kmem:kmalloc' > \ - /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger + # echo 'disable_event:kmem:kmalloc' > \ + /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger - The format is: + The format is:: enable_event:<system>:<event>[:count] disable_event:<system>:<event>[:count] - To remove the above commands: + To remove the above commands:: - # echo '!enable_event:kmem:kmalloc:1' > \ - /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger + # echo '!enable_event:kmem:kmalloc:1' > \ + /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger - # echo '!disable_event:kmem:kmalloc' > \ - /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger + # echo '!disable_event:kmem:kmalloc' > \ + /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger Note that there can be any number of enable/disable_event triggers per triggering event, but there can only be one trigger per @@ -418,34 +421,34 @@ The following commands are supported: triggering event occurs. For example, the following trigger dumps a stacktrace every time the - kmalloc tracepoint is hit: + kmalloc tracepoint is hit:: - # echo 'stacktrace' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + # echo 'stacktrace' > \ + /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger The following trigger dumps a stacktrace the first 5 times a kmalloc - request happens with a size >= 64K + request happens with a size >= 64K:: - # echo 'stacktrace:5 if bytes_req >= 65536' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + # echo 'stacktrace:5 if bytes_req >= 65536' > \ + /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger - The format is: + The format is:: stacktrace[:count] - To remove the above commands: + To remove the above commands:: - # echo '!stacktrace' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + # echo '!stacktrace' > \ + /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger - # echo '!stacktrace:5 if bytes_req >= 65536' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + # echo '!stacktrace:5 if bytes_req >= 65536' > \ + /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger The latter can also be removed more simply by the following (without - the filter): + the filter):: - # echo '!stacktrace:5' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + # echo '!stacktrace:5' > \ + /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger Note that there can be only one stacktrace trigger per triggering event. @@ -458,23 +461,23 @@ The following commands are supported: The following command creates a snapshot every time a block request queue is unplugged with a depth > 1. If you were tracing a set of events or functions at the time, the snapshot trace buffer would - capture those events when the trigger event occurred: + capture those events when the trigger event occurred:: - # echo 'snapshot if nr_rq > 1' > \ - /sys/kernel/debug/tracing/events/block/block_unplug/trigger + # echo 'snapshot if nr_rq > 1' > \ + /sys/kernel/debug/tracing/events/block/block_unplug/trigger - To only snapshot once: + To only snapshot once:: - # echo 'snapshot:1 if nr_rq > 1' > \ - /sys/kernel/debug/tracing/events/block/block_unplug/trigger + # echo 'snapshot:1 if nr_rq > 1' > \ + /sys/kernel/debug/tracing/events/block/block_unplug/trigger - To remove the above commands: + To remove the above commands:: - # echo '!snapshot if nr_rq > 1' > \ - /sys/kernel/debug/tracing/events/block/block_unplug/trigger + # echo '!snapshot if nr_rq > 1' > \ + /sys/kernel/debug/tracing/events/block/block_unplug/trigger - # echo '!snapshot:1 if nr_rq > 1' > \ - /sys/kernel/debug/tracing/events/block/block_unplug/trigger + # echo '!snapshot:1 if |