My first eBPF program — counting execve() from inside the kernel

I've been meaning to properly look at eBPF for a while. I knew the headline version — "run your own code inside the Linux kernel, safely" — but not what that actually means in practice. So I wrote the smallest thing I could think of: a program that runs every time any process on the machine calls execve(), bumps a counter, and lets a normal userspace program read that counter.

The program itself is about 30 lines. Understanding why each of those lines is there took a lot longer, and that's what this post is mostly about.

The project

Three files:

ebpf/
├── Makefile
├── hello.bpf.c   # runs inside the kernel
└── hello.c       # normal userspace program that loads it

The kernel side:

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 1);
    __type(key, __u32);
    __type(value, __u64);
} exec_count SEC(".maps");

SEC("tracepoint/syscalls/sys_enter_execve")
int on_execve(void *ctx)
{
    __u32 key = 0;
    __u64 *count = bpf_map_lookup_elem(&exec_count, &key);
    if (count)
        __sync_fetch_and_add(count, 1);

    char comm[16];
    bpf_get_current_comm(&comm, sizeof(comm));
    __u32 pid = bpf_get_current_pid_tgid() >> 32;

    bpf_printk("execve: pid=%d comm=%s", pid, comm);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

The userspace side uses libbpf to open the compiled object, load it into the kernel, attach it, and then read the map once per second:

struct bpf_object *obj = bpf_object__open_file("hello.bpf.o", NULL);
bpf_object__load(obj);                       // the verifier runs here

struct bpf_program *prog = bpf_object__find_program_by_name(obj, "on_execve");
struct bpf_link *link = bpf_program__attach(prog);

int map_fd = bpf_object__find_map_fd_by_name(obj, "exec_count");
while (!stop) {
    __u32 key = 0;
    __u64 val = 0;
    bpf_map_lookup_elem(map_fd, &key, &val);
    printf("\rexecve calls so far: %llu", (unsigned long long)val);
    fflush(stdout);
    sleep(1);
}

And the Makefile:

ARCH_INC := /usr/include/$(shell uname -m)-linux-gnu

all: hello.bpf.o hello

hello.bpf.o: hello.bpf.c
	clang -O2 -g -target bpf -I$(ARCH_INC) -c $< -o $@

hello: hello.c
	gcc -O2 -Wall $< -lbpf -o $@

Two compilers, two worlds

The first thing that confused me was that the Makefile uses two different compilers. They're building two very different things.

gcc ... -lbpf builds a completely ordinary Linux program, linked against libbpf. Nothing special. ($< is the first prerequisite, $@ is the target, which I always forget.)

clang -target bpf is the interesting one. Clang is the C frontend for LLVM, and LLVM has had a BPF backend for years. So the same C you'd normally compile to x86 can be compiled for the BPF instruction set. GCC has a BPF backend too these days, but clang is still what basically everyone uses for this.

The flags on that line matter more than I expected:

• -O2 is not optional in practice. Unoptimised code tends to do things (spilling and reloading pointers through the stack, and so on) that the verifier can't follow, and it gets rejected.
• -g emits debug info, which is where BTF (BPF Type Format) comes from. libbpf uses it to understand the map definitions, and it's the basis for CO-RE (compile once, run everywhere across kernel versions).
• -c gives you an object file. There's no linking step; the .o is the thing the kernel ends up loading.

The weird include path

-I/usr/include/x86_64-linux-gnu looked like noise to me, so I deleted it and tried again:

In file included from hello.bpf.c:2:
In file included from /usr/include/linux/bpf.h:11:
/usr/include/linux/types.h:5:10: fatal error: 'asm/types.h' file not found
    5 | #include <asm/types.h>
      |          ^~~~~~~~~~~~~

The chain is linux/bpf.h → linux/types.h → asm/types.h. The asm/ headers depend on the architecture, so on Debian/Ubuntu they live in a multiarch directory (/usr/include/x86_64-linux-gnu/asm/, next to bits/, sys/, gnu/ and friends). When you compile for x86, the compiler adds that directory to the search path automatically. When you compile with -target bpf, it doesn't, because as far as clang is concerned you're not building for x86 any more. So you have to point it there yourself. The $(shell uname -m) makes it work on ARM too.

It was a nice small reminder that "the include path" is just a list of directories the preprocessor walks, and the defaults depend on the target.

So what is the "BPF virtual machine"?

Every intro says eBPF programs run in a "virtual machine" inside the kernel, and I pictured something like a VM with its own OS. It isn't. It's a VM in the JVM/WebAssembly sense: an instruction set defined by the kernel.

• 11 64-bit registers, r0–r10 (r10 is a read-only frame pointer)
• a 512-byte stack
• plain arithmetic, loads and stores, jumps, and call for a fixed set of helper functions

It's deliberately close to real hardware, so that the kernel's JIT can turn it into native instructions almost one-to-one. On my machine /proc/sys/net/core/bpf_jit_enable is 1, so nothing is interpreted. By the time the program runs, it's x86 code.

The full pipeline turned out to be:

C ──clang──► LLVM IR ──LLVM BPF backend──► BPF bytecode (ELF .o)
                                                │
                              bpf(BPF_PROG_LOAD) syscall
                                                ▼
                                verifier ──► JIT ──► native x86

Why not just ship x86 code? Because the kernel has to be able to prove the program is safe before running it, and you can't do that with arbitrary machine code. A small, restricted instruction set can be analysed. It's also portable: the same .o works on any architecture with a BPF JIT.

Looking at the intermediate steps

I wanted to see each stage, so I stopped the compiler at different points.

LLVM IR, with -S -emit-llvm:

clang -O2 -target bpf -I/usr/include/x86_64-linux-gnu -S -emit-llvm hello.bpf.c -o hello.bpf.ll

define dso_local noundef i32 @on_execve(ptr nocapture readnone %0) #0 section "tracepoint/syscalls/sys_enter_execve" {
  ...
  %4 = call ptr inttoptr (i64 1 to ptr)(ptr noundef nonnull @exec_count, ptr noundef nonnull %2)
  %5 = icmp eq ptr %4, null
  br i1 %5, label %8, label %6

6:
  %7 = atomicrmw add ptr %4, i64 1 seq_cst, align 8
  ...
  %9 = call i64 inttoptr (i64 16 to ptr)(ptr noundef nonnull %3, i32 noundef 16)
  %10 = call i64 inttoptr (i64 14 to ptr)()

inttoptr (i64 1 to ptr) was the first surprise. The helpers aren't real functions with addresses. bpf_helpers.h declares each one as a function pointer with a fixed number: bpf_map_lookup_elem is helper 1, bpf_get_current_pid_tgid is 14, bpf_get_current_comm is 16, and bpf_printk ends up as bpf_trace_printk, number 6.

That's even clearer in the actual bytecode (llvm-objdump -d hello.bpf.o):

0000000000000000 <on_execve>:
       0:	r1 = 0x0
       1:	*(u32 *)(r10 - 0x4) = r1        ; key = 0, on the stack
       2:	r2 = r10
       3:	r2 += -0x4                      ; r2 = &key
       4:	r1 = 0x0 ll                     ; r1 = &exec_count (patched at load time)
       6:	call 0x1                        ; bpf_map_lookup_elem
       7:	if r0 == 0x0 goto +0x2          ; the NULL check
       8:	r1 = 0x1
       9:	lock *(u64 *)(r0 + 0x0) += r1   ; atomic increment

0000000000000050 <LBB0_2>:
      10:	r6 = r10
      11:	r6 += -0x14                     ; comm[16] on the stack
      12:	r1 = r6
      13:	r2 = 0x10
      14:	call 0x10                       ; bpf_get_current_comm
      15:	call 0xe                        ; bpf_get_current_pid_tgid
      16:	r0 >>= 0x20                     ; >> 32 to get the PID
      ...
      22:	call 0x6                        ; bpf_trace_printk
      23:	r0 = 0x0
      24:	exit

25 instructions. You can read the whole thing and match every line back to the C. The map address is 0x0 here because the map doesn't exist yet. libbpf creates it at load time and patches the real reference into that instruction. The __sync_fetch_and_add turned into a single lock ... +=, which is reassuring given that this can run on every CPU at once.

The verifier, or: why the if (count) is mandatory

When the loader calls bpf_object__load(), the bytecode goes to the kernel through the bpf() syscall, and the verifier walks every possible path through the program. It tracks what each register holds (a scalar, a pointer to the stack, a pointer to a map value, a pointer that might be NULL) and the bounds of every value. If any path could do something unsafe, the whole program is rejected.

bpf_map_lookup_elem returns "pointer to map value, or NULL". If you dereference it without checking, the verifier refuses to load the program. That's the if r0 == 0x0 at instruction 7, and it's not defensive programming, the program won't load without it.

The rest of the restrictions follow from the same idea, which is that the kernel can't crash or hang because of your code:

It's not just syscalls

Mine hooks the sys_enter_execve tracepoint, which made me assume eBPF is a "run something next to a syscall" mechanism. It's much broader than that. The program type and the SEC() name decide where it attaches:

• tracepoints: stable hooks the kernel developers placed on purpose (syscalls, scheduler, block I/O, networking)
• kprobes / fentry / fexit: the entry or exit of almost any kernel function
• uprobes / USDT: functions in userspace binaries, e.g. reading the plaintext passed to OpenSSL's SSL_write
• XDP: runs in the network driver before the kernel has even allocated an sk_buff, so it can drop or redirect packets extremely early
• tc, sockets, cgroups: packet filtering, per-cgroup policy, socket-level redirection
• LSM: security hooks that can actually deny an operation
• struct_ops / sched_ext: whole TCP congestion control algorithms, and since 6.12 whole CPU schedulers, written in BPF

A lot of tools I already use or have heard of are built on this: bpftrace, bcc, Cilium, Falco, Tetragon, and the various continuous profilers.

Running it

make
sudo ./hello

It has to run from the project directory, because the loader opens hello.bpf.o with a relative path. Once it's attached it prints a single line that updates every second:

Attached. Run some commands in another terminal. Ctrl-C to quit.
execve calls so far: 37

The counter goes up even if you don't touch anything. It counts every execve() on the whole machine, so it picks up cron jobs, shell prompts that shell out to git, editors spawning language servers, and so on. That alone is a small window into how much process creation a "quiet" Linux box is doing.

The bpf_printk output goes to the kernel's trace buffer:

sudo cat /sys/kernel/tracing/trace_pipe

bash-123456  [004] ...21  9876.543210: bpf_trace_printk: execve: pid=123456 comm=bash

I expected comm=ls when running ls, but it's comm=bash. The hook fires on sys_enter, before execve has replaced the process image, so at that moment the process is still the forked shell. It makes sense once you think about it, but I found it a good way to make "the hook runs at this exact point" feel concrete.

While it's running, bpftool lets you look at it from the kernel's side:

sudo bpftool prog show name on_execve          # the loaded program
sudo bpftool map dump name exec_count          # the counter, read straight from the map
sudo bpftool prog dump xlated name on_execve   # bytecode as the verifier saw it
sudo bpftool prog dump jited name on_execve    # the native x86 the JIT produced

The last one closes the loop on the whole C → bytecode → native story.

Ctrl-C destroys the link and closes the object, and once nothing references the program any more, the kernel frees it.

What's next

The obvious experiment is to remove the if (count) and read the verifier's rejection log. After that I want to replace bpf_printk with a ring buffer, so that events flow to userspace properly instead of through a debug pipe, and then try a kprobe and some CO-RE with vmlinux.h, so that I'm reading real kernel structs instead of just calling helpers.

What I like most so far is how inspectable all of it is. The bytecode fits on one screen, the helpers are numbered calls, the verifier tells you exactly why it said no, and bpftool lets you see what the kernel is actually running.