publications

2025

1. Pairwise BPF Programs Should Be Optimized Together

   Milo Craun, and Dan Williams

   In Proceedings of the 3rd Workshop on EBPF and Kernel Extensions , Coimbra, Portugal, 2025

   Abs PDF

   BPF programs are extensively used for tracing and observability in production systems where performance overheads matter. Many individual BPF programs do not incur serious performance degrading overhead on their own, but increasingly more than a single BPF program is used to understand production system performance. BPF deployments have begun to look more like distributed applications; however, this is a mismatch with the underlying Linux kernel, potentially leading to high overhead cost. In particular, we identify that many BPF programs follow a pattern based on pairwise program deployment where entry and exit probes will be attached to measure a single quantity. We find that the pairwise BPF program pattern results in unnecessary overheads. We identify three optimizations—BPF program inlining, context aware optimization, and intermediate state internalization—that apply to pairwise BPF programs. We show that applying these optimizations to an example pairwise BPF program can reduce overhead on random read throughput from 28.13% to 8.98% and on random write throughput from 26.97% to 8.60%. We then examine some key design questions that arise when seeking to integrate optimizations with the existing BPF system.

2. Rex: closing the language-verifier gap with safe and usable kernel extensions

   Jinghao Jia, Ruowen Qin, Milo Craun , and 8 more authors

   In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference , Boston, MA, USA, 2025

   Abs PDF

   Safe kernel extensions have gained significant traction, evolving from simple packet filters to large, complex programs that customize storage, networking, and scheduling. Existing kernel extension mechanisms like eBPF rely on in-kernel verifiers to ensure safety of kernel extensions by static verification using symbolic execution. We identify significant usability issues–safe extensions being rejected by the verifier–due to the language-verifier gap, a mismatch between developers’ expectation of program safety provided by a contract with the programming language, and the verifier’s expectation.We present Rex, a new kernel extension framework that closes the language-verifier gap and improves the usability of kernel extensions in terms of programming experience and maintainability. Rex builds upon language-based safety to provide safety properties desired by kernel extensions, along with a lightweight extralingual runtime for properties that are unsuitable for static analysis, including safe exception handling, stack safety, and termination. With Rex, kernel extensions are written in safe Rust and interact with the kernel via a safe interface provided by Rex’s kernel crate. No separate static verification is needed. Rex addresses usability issues of eBPF kernel extensions without compromising performance.

3. Towards Safe Agentic AI Performance Engineering

   Dan Williams, Milo Craun, Michael V. Le , and 3 more authors

   In Proceedings of the 4th Workshop on Practical Adoption Challenges of ML for Systems , Seoul, Republic of Korea, 2025

   Abs PDF

   The emergence of agentic AI—reasoning AI agents that can connect to tools and take actions—offers an enormous potential in performing tasks that currently require highly skilled humans to perform. In this position paper, we discuss AI agents in one such role: performance engineer. A performance engineer is typically highly trained and highly trusted to run performance diagnostic tools—which more often than not require root or administrator privileges—on production machines to diagnose performance issues. Critically, performance engineers are trusted not to cause harm to the production systems they are investigating, including crashing or hanging the systems, extracting sensitive information from them, or negatively affecting their performance. In this paper, we argue that current AI agents have the training, but lack the trust to be performance engineers. We outline four components: prevention, detection/auditing, aborting/rollback, and retry/refocus and highlight gaps where the approaches taken for human-based performance engineers fall short.

2024

1. Eliminating eBPF Tracing Overhead on Untraced Processes

   Milo Craun, Khizar Hussain, Uddhav Gautam , and 3 more authors

   In Proceedings of the ACM SIGCOMM 2024 Workshop on EBPF and Kernel Extensions , Sydney, NSW, Australia, 2024

   Abs PDF

   Current eBPF-based kernel extensions affect entire systems, and are coarse-grained. For some use cases, like tracing, operators are more interested in tracing a subset of processes (e.g., belonging to a container) rather than all processes. While overhead from tracing is expected for targeted processes, we find untraced processes—those that are not the target of tracing—also incur performance overhead. To better understand this overhead, we identify and explore three techniques for per-process filtering for eBPF: post-eBPF, in-eBPF, and pre-eBPF filtering, finding that all three approaches result in excessive overhead on untraced processes. Finally, we propose a system that allows for zero-untraced-overhead per-process eBPF tracing by modifying kernel virtual memory mappings to present per-process kernel views, effectively enabling untraced processes to execute on the kernel as if no eBPF programs are attached.

2023

1. Enabling eBPF on Embedded Systems Through Decoupled Verification

   Milo Craun, Adam Oswald, and Dan Williams

   In Proceedings of the 1st Workshop on EBPF and Kernel Extensions , New York, NY, USA, 2023

   Abs PDF

   eBPF (Extended Berkeley Packet Filter) is a Linux kernel subsystem that aims to allow developers to write safe and efficient kernel extensions by employing an in-kernel verifier and just-in-time compiler (JIT). We find that verification is prohibitively expensive for resource-constrained embedded systems. To solve this we describe a system that allows for verification to occur outside of the embedded kernel and before BPF program load time. The in-kernel verifier and JIT are coupled so they must be decoupled together. A designated verifier kernel accepts a BPF program, then verifies, compiles, and signs a native precompiled executable. The executable can then be loaded onto an embedded device without needing the verifier and JIT on the embedded device. Decoupling verification and JIT from load-time opens the door to much more than running BPF programs on embedded devices. It allows larger and more expressive BPF programs to be verified, provides a way for new approaches to verification to be used without extensive kernel modification and creates the possibility for BPF program verification as a service.