From: eLinux.org

Kernel Trace Systems

Here are some links to information about different kernel tracing systems:

Contents

• 1 General Purpose tracing systems
• 2 Special Purpose tracing systems
• 3 Trace Infrastructure
• 4 Sampling Systems
• 5 Related facilities
• 6 Other Systems
• 7 Collaboration Efforts

General Purpose tracing systems

Some major Linux general-purpose tracing systems are:

• ptrace - ability to trace syscall entry and exit, and signal delivery, to a process (also used for debugging a process)
  • see "man ptrace" and "man strace"
• Ftrace
  • Ftrace Function Graph ARM - presentations and patches by Tim Bird to add function graph and duration tracing to ARM systems
    • The presentation has good introductory material on ftrace, as well as links to additional resources
  • tracer for kernel functions
  • can also be used for debugging or analyzing latencies and performance issues
  • in mainline since 2.6-
  • See Measuring Function Duration with FTrace
    • outline of presentation by Tim Bird for Linux Symposium 2009
• System Tap - System Tap is a system for building and executing tracing and sampling systems that can be applied to a running Linux system
• LTTng - Linux Trace Toolkit, next generation
• LKST - Linux Kernel State Tracer

Special Purpose tracing systems

There are some other notable special-purpose kernel tracing systems:

• KFT - Kernel Function Trace - traces functions to show function durations and call graphs
• latency trace - RT-preempt tool for measuring interrupt and mutex latency
  • The latency tracer is embedded in the RT-preempt patch - see Realtime Preemption and RT-preempt Article
• block tracer (blktrace) - allows you to see exactly what is going on in the block layer for a given queue
  • Introduction by Jens Axboe: Introduction
  • Execellent presentation: blktrace.pdf
  • Guide to using is at: blktrace guide
  • This appears to have been mainlined as of 2.6.17
  • Timeline utility (blktrace post-processing tool): blktrace timeline utility
• delay accounting patches - collect statistics about the delays that are experienced by each task on the system
  • see delay accounting patches

Trace Infrastructure

• KProbes - grew out of dprobes, with information at: dprobes

  • see an excellent tutorial at: kprobes
  • The mainline version of the KProbes supports x86,Alpha and PPC64 architectures. A MIPS implementation has been completed on the 2.6.16 kernel and tested on the Toshiba TX49 platform. Patch is available in the Patch Archive.

• [would be nice to have some djprobe stuff here]

Sampling Systems

Note that profile systems (or "sampling systems") are slightly different, in that they involve sampling instead of event tracing. Some major ones for Linux are:

• top - provides a dynamic real-time view of a running system, including processes
  • see "man top"
  • see also ksysguard, Gnome system Monitor
• OProfile - system-wide profiler for Linux systems
  • see oprofile
  • also: oprofile at IBM
• BootChart - samples bootup and provides visualization of process startup and system utilization
  • see Bootchart

Related facilities

• in-kernel statistics infrastructure - proposal for a generic implementation of statistics facilities inside the kernel
  • see inkernel stats
• perfmon2 - interfaces to hardware performance monitoring features of the CPU
  • see perfmon
• inotify - inotify

Other Systems

Here are some systems I haven't classified yet:

• Datastreams - a system for creating and monitoring tracepoints - see datastreams

Collaboration Efforts

Some trace system project leaders are trying to collaborate: see Tracing Collaboration Project

Category:

• Linux tracing