vDSO call tracing via hw breakpoints + in-memory ELF symbol resolution
- C 84.1%
- Python 10.7%
- Shell 2.8%
- Makefile 2.4%
| .github/workflows | ||
| include | ||
| src | ||
| tests | ||
| .gitignore | ||
| Makefile | ||
| README.txt | ||
. .
\ . /
\ : /
.-----\----------+-------------/-----.
/ \ | / \
/ .--------+-----------. \
/ / / \
/ / 7f 45 4c 46 / \
/ / / \
/ '----------.----------' \
'---------------------|----------------------------'
\ | /
\ | /
'---------+-----------'
|
.--------------------+--------------------.
| |
.---v---. .---v---.
| entry | | return|
'---+---' '---+---'
| |
'--------------------.--------------------'
|
.---------v---------.
| vdsotrace |
'-------------------'
=========================================================
v d s o t r a c e / notes from the return address
=========================================================
a small linux instrumentation zine
issue 02 / native x86-64 / 2026
<-------------------------------------------------------------------------->
this follows a function call into the small elf image that linux
places inside a process, then follows its return back to the caller.
along the way, an auxiliary-vector entry becomes an executable address,
a stack word becomes a breakpoint, and a register snapshot becomes an
observation of a service that can complete entirely in userspace.
vdsotrace is the implementation of that walk, written in c around the
linux tracing interface and the processor's hardware debug registers.
its subject is the vdso, where kernel-supplied code participates in an
ordinary userspace call chain without requiring every invocation to
cross the system-call boundary.
the register names and interface labels in this text are set in
lowercase as part of its typography, including labels whose spelling
in system headers uses uppercase letters.
.---------------------------------------------------------.
| 00 / the call that stays upstairs |
| 01 / an address left by the loader |
| 02 / reading the image from the inside |
| 03 / borrowing the caller's return address |
| 04 / keeping track of a moving process |
| 05 / turning observations into a session |
| 06 / closing the circuit |
| 07 / further reading |
'---------------------------------------------------------'
<== 00 / the call that stays upstairs ======================================>
a program asks for the time through clock_gettime(), and its library can
answer by calling into a kernel-supplied shared object already mapped into
the process. when the selected clock can be served through that path, the
calculation completes in userspace and the result travels back through the
same function-call machinery that brought execution there [1].
the application has received a service associated with the operating system,
but the successful invocation has not necessarily produced a system call.
this distinction matters when reconstructing what a program actually did,
because an observation point at kernel entry sees only the calls that reach
that boundary.
process address space
.---------------------------------------------------------.
| |
| application -----> libc -----> mapped vdso |
| ^ ^ | |
| | '--------------' |
| '---------------' | |
| | |
'--------------------------------------|------------------'
|
syscall fallback, if needed
|
.--------------------------------------v------------------.
| kernel |
'---------------------------------------------------------'
vdsotrace places its observation point at the exported function itself,
where the call has already selected the vdso but its internal execution
path has not yet determined how the service will be completed. a direct
call through a resolved function pointer reaches the same observation
point as a call made through a library wrapper.
the image belongs to the process's virtual address space even though its
code comes from the kernel, which makes the ordinary userspace calling
convention the most useful description of its boundary. the tracer uses
that convention to connect a function's identity, its incoming arguments,
and the state it leaves behind when execution returns to its caller.
<== 01 / an address left by the loader =====================================>
the starting address is supplied rather than guessed, because linux records
the vdso elf header in the at_sysinfo_ehdr entry of the auxiliary vector.
reading /proc/pid/auxv provides that value, while the corresponding [vdso]
entry in /proc/pid/maps identifies the readable extent of the image [2].
auxiliary vector virtual memory map
.--------------------. .-----------------------.
| at_sysinfo_ehdr | | base ... end |
| | | | r-xp [vdso] |
'---------|----------' '-----------+-----------'
| |
'----------------.------------------'
|
v
.-----------------.
| bounded image |
| in local memory |
'-----------------'
with the target stopped, the acquisition layer first requests a bulk copy
through process_vm_readv(). when that transfer is unavailable or supplies
only part of the requested range, ptrace word reads obtain the remaining
bytes. the same reader later captures stack words and output structures,
so address acquisition and result decoding share one memory-access path.
copying the image gives the parser a fixed byte range to interpret, with
target pointers remaining numeric addresses until their contents have been
obtained explicitly. this separates the question of how memory is acquired
from the question of what the elf structures inside that memory describe.
<== 02 / reading the image from the inside =================================>
the vdso carries the metadata needed to describe its own exports, beginning
with an elf header and a program-header table. load segments establish the
mapped ranges, while the dynamic segment connects the symbol table, string
table, hash data, and version definitions used during resolution [3].
.------------------.
| elf header |
'--------+---------'
|
.--------v---------.
| program headers |
'---+----------+---'
| |
pt_load| |pt_dynamic
v v
.-----------. .---------------------------.
| executable| | dt_symtab -> symbols |
| ranges | | dt_strtab -> names |
'-----------' | dt_hash -> symbol count |
| dt_gnu_hash -> chains |
| dt_versym -> indices |
| dt_verdef -> versions |
'-------------+-------------'
|
.---------------v-------------.
| name + version + st_value |
'-----------------------------'
the sysv hash table supplies a symbol count directly, while the gnu hash
path derives the table's extent from its highest populated bucket and the
chain that follows it. symbol names are checked against the string table,
and eligible function definitions are checked against executable segments
before their addresses enter the tracing machinery.
the parser copies structures into local objects before examining their
fields, keeping interpretation independent of their alignment in the input
buffer. for the zero-based mapped image, adding a function's st_value to
the discovered base yields the address at which its entry can be observed.
versioning gives the export a second component of identity, because the
symbol's version index connects it to a named definition in the image.
an unqualified lookup selects the default definition, while a qualified
lookup joins a function name to a particular version with an at sign.
the lookup itself preserves the case-sensitive spelling published by
the image even though this text uses lowercase interface labels.
the resulting inventory owns the function names and version names together
with their addresses, sizes, and default-version status. whenever exec
replaces the process image, resolution begins again from its auxiliary
vector so the instrumentation follows the newly mapped definition.
<== 03 / borrowing the caller's return address =============================>
an execution breakpoint in dr0 watches the selected entry address, causing
the processor to report a debug exception when the thread reaches it.
linux presents that event as a tracing stop, and the tracer examines the
signal metadata and dr6 to identify its own breakpoint before inspecting
the stopped register state [4].
the amd64 calling convention places the first six integer or pointer
arguments in rdi, rsi, rdx, rcx, r8, and r9. at entry, rsp points to the
return address placed on the stack by the caller, providing a completion
point without requiring the tracer to decode the function body [5].
just before entry executes
registers stack
.--------------------. .-----------------------.
| rdi rsi rdx | rsp -->| caller return address |
| rcx r8 r9 | +-----------------------+
'---------+----------' | caller's stack state |
| '-----------------------'
| |
'----------------.------------------'
|
v
.------------------------.
| save arguments and rsp |
| watch the return site |
'------------------------'
dr1 through dr3 carry return-site breakpoints whose associated call frames
retain the entry stack pointer, arguments, and monotonic start time. when
execution reaches a watched return site, both the instruction address and
the restored stack position participate in identifying the completed call.
for an ordinary near return, the relationship is:
rip == saved return address
rsp == saved entry rsp + 8
matching the stack position gives the return address a particular call
context, which distinguishes it from another visit to the same instruction
with a different frame. the tracer then reads rax and any recognized output
structures before allowing the caller's next instruction to execute.
the completed record retains the return address and entry stack pointer,
allowing callers to be distinguished even when they request the same
service with identical arguments. these addresses preserve the observed
call context alongside the exported function's resolved identity.
caller vdso tracer
| | |
+-------- call -------->+---------- entry ------->|
| | |
| |<--------- resume -------+
| | |
|<------- return -------+ |
+--------------------- return stop -------------->|
| |
|<------------------ capture and resume -----------+
v
the resume flag in eflags lets the stopped instruction execute without
immediately reporting the same execution-breakpoint fault again. because
the observation addresses live in debug state, the function's executable
bytes remain intact throughout the call and the return-site instruction
requires no patching or reconstruction.
each task also keeps a cached description of its installed debug state,
so changing a return slot updates the affected addresses rather than
rewriting the entire register set. an unchanged plan clears only the
debug status, while exec and failed updates invalidate the cache before
another configuration is installed.
<== 04 / keeping track of a moving process =================================>
a tracing session must keep its register-level model attached to tasks that
can create threads, replace their address spaces, deliver signals, and exit
while other work continues. vdsotrace uses ptrace_seize to establish control
and ptrace_interrupt to stop individual tasks during attachment and cleanup.
attachment enumerates the target's task directory and freezes the discovered
threads before repeating the enumeration, allowing the known set to settle
before breakpoints are installed. launching follows a different route,
with a pipe holding the child until exec tracing is installed and the
replacement image can be resolved at its first exec event.
.-------------------------.
launch -->| |<-- attach
| traced tasks |
'------------+------------'
|
.---------------+---------------.
| | |
v v v
clone exec return
| | |
v v v
resolve child resolve image capture call
| | |
'---------------+---------------'
|
v
continue execution
newly traced threads resolve their mapped exports before their breakpoint
state is armed. optional fork and vfork following extends the same model
to descendant processes, whose later exec events establish the identities
of the images they actually run. task identity keeps each contribution
associated with the thread that produced it across the session.
process identity is captured separately from thread identity, connecting
each task to its thread group before its breakpoints are armed. both
identities accompany call records, which makes the relationship between
worker threads and forked descendants explicit in the observed stream.
application signal-delivery stops retain their signals when execution
resumes, while the tracer consumes its own hardware traps internally.
group-stop handling uses ptrace_listen to participate in job control, and
ordinary shutdown interrupts the remaining tasks, restores their saved
debug state, and detaches them through a shared cleanup path.
the cleanup path also handles failed record writes and explicit requests
to stop tracing. accounting survives the removal of the corresponding
task-control objects, so work completed by a thread remains part of the
session after that thread has exited or detached.
a monotonic deadline can end a session even when its target makes no vdso
calls. the event loop combines nonblocking status collection with an
atomic signal-mask transition into ppoll, allowing task events, shutdown
requests, and deadline expiration to wake the same wait path without
periodic polling.
<== 05 / turning observations into a session ===============================>
completion capture produces one shared record before presentation begins,
preserving raw rax alongside the signed result appropriate to a recognized
prototype. dedicated decoders recover clock, timeval, timezone, time, and
cpu-location outputs from the stopped task, while generic exports retain
the six captured integer argument registers as their calling context.
the getrandom export adds a five-argument interface whose result is a
signed machine-width byte count. its arguments identify an output buffer,
requested length, flags, an opaque state area, and that area's size. the
decoder preserves this width for both successful returns and negative
errors, while recognizing the special query that describes how the caller
should allocate the opaque state [6].
getrandom invocation
.---------------------------------------------------------.
| buffer | length | flags | opaque state | opaque size |
'----------------------------+----------------------------'
|
.-----------+-----------.
| |
v v
allocation query byte request
| |
v v
state size and signed byte count
mapping parameters or negative error
the allocation query exposes state size and mapping parameters through
the same output-field machinery used for timekeeping structures. ordinary
successful requests report the number of bytes produced, connecting this
new prototype to the existing error classification and session accounting.
the observation also carries the selected symbol version and two distinct
time measurements. a monotonic interval describes the traced duration
between entry and return, including tracer processing and scheduling, while
a realtime sample places completion on the application's wall-clock timeline.
completed call
|
v
.------------------. .-----------------------.
| accumulate first +------>| per-thread statistics |
'--------+---------' '-----------+-----------'
| |
v v
.------------------. .-----------------------.
| select records | | session statistics |
| by tid, error, | | counts and durations |
| or duration | '-----------------------'
'--------+---------'
|
v
.------------------.
| text / json lines|
'------------------'
accumulation happens before record selection, so filtering by thread,
recognized error, or minimum duration changes the emitted stream without
changing the observed population behind the session's statistics. a call
budget uses that same population, and summary-only operation retains the
aggregate view through the same accounting path.
per-thread and session summaries retain completion counts, classified
results, recognized errors, emitted records, and duration statistics.
text and json lines consume the same records and counters, with hexadecimal
strings preserving addresses and raw register values in structured output.
the architecture gives an interactive trace and a machine-consumed stream
the same underlying interpretation of each captured invocation.
logarithmic latency buckets retain the distribution behind those totals,
and cumulative bucket counts provide upper bounds for the fiftieth,
ninety-fifth, and ninety-ninth percentiles. the bounds are capped by the
largest observed duration, while the bucket populations can be reconciled
across thread summaries to recover the complete session distribution.
structured objects carry a schema revision and a session identifier, with
call sequence numbers assigned from the observed population before record
selection. filtered streams therefore preserve their original completion
order, and append-mode output can retain successive sessions in one file
without confusing their sequence spaces.
<== 06 / closing the circuit ==============================================>
the implementation can be checked at both ends of the observation, because
the resolver accepts a byte image and the tracing engine observes a workload
whose calls are known independently. constructed elf images exercise hash
tables, version definitions, and range validation, while a direct-call
workload makes successful and invalid clock requests from four threads.
that workload produces 320 completed invocations with 160 raw error returns,
giving the integration checks a concrete population against which to compare
thread identities, decoded structures, filters, and summary totals. further
checks cover exec from a worker thread, fork and vfork descendants, signal
delivery, and detachment after an output failure. parser and record checks
also run under address and undefined-behavior sanitizers.
additional checks exercise interrupted and partial memory transfers,
debug-register cache recovery, idle-target deadlines, and reconciliation
of latency buckets. on kernels exporting getrandom, the live workload
queries its allocation parameters, maps the requested state, and verifies
successful generation together with a deliberately invalid flag request.
the same checks are configured for gcc and clang with compiler warnings
treated as errors.
what connects those layers is the decision to observe a service at its
function boundary, where elf metadata explains what was called and the
calling convention explains what entered and what returned. the auxiliary
vector supplies an address, the mapped image supplies an identity, and the
caller's stack supplies the point where an invocation becomes a result.
.-------------------.
| address -> name |
| entry -> frame |
| return -> value |
'---------+---------'
|
.---------v---------.
| back to the caller|
'-------------------'
<== 07 / further reading ==================================================>
[1] linux man-pages, vdso(7).
https://man7.org/linux/man-pages/man7/vdso.7.html
[2] linux man-pages, getauxval(3) and proc_pid_auxv(5).
https://man7.org/linux/man-pages/man3/getauxval.3.html
https://man7.org/linux/man-pages/man5/proc_pid_auxv.5.html
[3] system v abi, elf program loading and dynamic linking.
https://www.sco.com/developers/gabi/latest/ch5.dynamic.html
[4] linux man-pages, ptrace(2) and process_vm_readv(2).
https://man7.org/linux/man-pages/man2/ptrace.2.html
https://man7.org/linux/man-pages/man2/process_vm_readv.2.html
[5] amd64 system v abi, function calling sequence.
https://www.ucw.cz/~hubicka/papers/abi/
[6] linux vdso getrandom implementation and allocation parameters.
https://github.com/torvalds/linux/blob/master/lib/vdso/getrandom.c
https://github.com/torvalds/linux/blob/master/include/uapi/linux/random.h
<-------------------------------------------------------------------------->
eof
<-------------------------------------------------------------------------->