wBPF: Efficient Edge-Case Observability for CXL Pooling systems via eBPF

Although the eBPF instruction set is standardized, the way it is loaded and executed varies widely between kernels (Linux, Windows[20], FreeBSD[1]) and runtimes for the user space (uBPF[16], rBPF[23]). Each environment imposes unique constraints: different system calls, loader APIs, and supported features making it difficult to build a single eBPF program that runs smoothly everywhere. The Linux ecosystem, for example, commonly relies on specialized bpf syscalls and libbpf-based tooling, whereas eBPF on Windows offers a Linux-compatible API wrapper but diverges on kernel functionalities. User-space implementations further fragment the landscape by omitting kernel-specific interfaces like ring buffers or perf events.

While CO-RE helps mitigate kernel-version mismatches by allowing eBPF programs to adapt dynamically through BTF data, its benefits do not fully extend to non-Linux runtimes or to architectural disparities (e.g., differences in pt_regs structures, endianness, and pointer width). Consequently, compatibility becomes a pressing concern in distributed heterogeneous environments—where eBPF must trace edge cases across a mix of OS kernels, user-space runtimes, and processor architectures. Missing or partially implemented features can break observability in isolated nodes, undermine global visibility, or even yield inconsistent data collection. These challenges highlight the difficulty in achieving a uniform eBPF deployment across varied systems, emphasizing the need for solutions prioritizing cross-platform loader interfaces, feature-discovery mechanisms, and architectural abstractions.

In the realm of distributed systems, identifying and diagnosing edge cases—rare and atypical scenarios that deviate from normal operation—is pivotal for maintaining system reliability and performance. Edge cases often emerge under specific, uncommon conditions that may not be anticipated during the initial design and testing phases. These cases can lead to critical failures, degraded performance, or unexpected behaviors that are challenging to reproduce and resolve.

One of the primary challenges in identifying edge cases lies in their inherent rarity. Traditional monitoring and logging mechanisms are typically optimized for capturing aggregate metrics and common failure modes, inadvertently neglecting the subtle anomalies that characterize edge cases. As a result, edge cases may only surface intermittently, making them difficult to detect and analyze without specialized tooling.

Furthermore, the complexity of modern distributed systems, characterized by numerous interacting microservices, intricate network dependencies, and dynamic scaling, exacerbates the difficulty of pinpointing the root causes of edge cases. The non-deterministic nature of these systems means that similar conditions can yield different outcomes, complicating the tracing and replication of edge-case scenarios.

Recent advancements in distributed tracing have shown promise in enhancing edge case identification. Craun and Thompson [7] propose methodologies to eliminate noise within trace data, thereby amplifying the signals that signify edge-case behaviors. By refining trace sampling techniques and employing intelligent filtering mechanisms, their approach ensures that the most pertinent traces—those that may reveal rare and critical issues—are retained for analysis.

Similarly, Zhang and Martinez [31] explore the benefits of adaptive sampling strategies in distributed tracing frameworks. Their research demonstrates that adaptive sampling can dynamically adjust the trace collection rate based on real-time system performance metrics and anomaly detection signals. This flexibility allows for increased trace capture during periods of irregular activity, thereby improving the likelihood of identifying edge cases without incurring significant overhead during normal operations.

Edge case identification is intrinsically linked with temporal provenance and tail-latency troubleshooting. Temporal provenance involves tracking the sequence and timing of events across distributed components, providing a comprehensive view of request flows and interactions. When combined with enhanced tracing techniques, temporal provenance can illuminate the temporal patterns and dependencies that underpin edge-case scenarios.

Tail-latency troubleshooting focuses on addressing the high-percentile latency spikes that can adversely affect user experience. Edge cases often manifest as tail latencies, where a small subset of requests experiences unusually long processing times. By leveraging refined distributed tracing and adaptive sampling, developers can better correlate tail latencies with specific code paths, resource constraints, or network issues that constitute edge cases.

The architecture of wBPF addresses the challenges associated with deploying eBPF programs across diverse environments, managing their lifecycle in containers, and handling versioning and pluggability. wBPF includes a WebAssembly (Wasm) library, toolchain, and runtime support for loading and executing eBPF programs, ensuring a consistent execution environment regardless of the underlying platform.

In the runtime, a Wasm module can correspond to multiple eBPF programs. An eBPF program instance can be dynamically loaded into the kernel from the Wasm sandbox. This allows the selection of the desired attach point and control over the lifecycle of multiple eBPF bytecode objects. wBPF supports various types of maps and enables bidirectional communication with the kernel, including support for most map types. It efficiently transfers information between kernel and user space through ring buffers polling (or vice versa) and map accesses. wBPF is adaptable to nearly all use cases involving eBPF programs and can evolve and extend as kernel functionality evolves.

While eBPF instructions themselves are inherently cross-platform, the userspace control applications that interact with the runtime are typically native binaries. This can lead to compatibility issues across different platforms. wBPF addresses this challenge by leveraging WebAssembly (Wasm) and the WebAssembly System Interface (WASI) for userspace control plane applications, creating a robust and portable compatibility layer.

CO-RE. wBPF also enhances the CO-RE capabilities by incorporating mechanisms to ensure compatibility across different runtimes. This includes automatic BTF downloading in the runtime to ensure that necessary BTF data is available on the host, and provide BTF for userspace eBPF runtimes and host applications. As for a rich tapestry of kernel feature-interoperability issues, we require minor updates to the Wasm runtime code to make them compatible.

Architecture-Independent Relocation. It refers to the process of adjusting the addresses within a program so that it can be executed correctly regardless of the underlying hardware architecture. Unlike architecture-specific relocation, which tailors the relocation process to the nuances of a particular hardware platform, architecture-independent relocation emphasizes portability and flexibility. This approach typically involves using abstract representations of memory addresses and leveraging standardized formats for executable files. By decoupling the relocation process from specific architectures, developers can create more versatile and maintainable software that can be easily adapted to diverse environments. This is especially important in heterogeneous computing systems where software must operate seamlessly across different hardware configurations.

Semantic probing involves analyzing and interpreting the underlying meanings and relationships within data or systems. In the context of software engineering and system design, semantic probing can be utilized to understand the intricacies of code behavior, system interactions, and the effectiveness of various components. By systematically examining the semantics, developers and researchers can identify patterns, optimize performance, and ensure that systems behave as intended under different scenarios. This approach is particularly valuable for debugging, enhancing system reliability, and facilitating the evolution of software architectures. We can find identical kprobe functions or helper functions using LLM manner[33], which resolves the kernel space heterogeneous problem.

eBPF programs are function-based and event-driven, and a specific eBPF program is run when a kernel or user space application passes a hook point. To use an eBPF program, we first need to compile the corresponding source code into BPF bytecode using the clang/LLVM toolchain, which contains the corresponding data structure definitions, maps, and progs definitions. Programs are program segments, and maps can be used to store data or communicate bi-directionally with the user space. After that, we can implement a complete eBPF application with the help of the user state development framework and the loading framework. There's only two part of state WASI that need to be maintained with BPF, kernel state, and userspace state.

1. The user state program needs to interact with the kernel through a series of system calls (mainly BPF system calls), create a corresponding map to store data in the kernel state or to communicate with the user state, dynamically select different segments to load according to the configuration, dynamically modify the bytecode or configure the parameters of the eBPF program, load the corresponding bytecode information into the kernel, ensure security through validators, and communicate with the kernel through maps and the kernel, passing data from the kernel state to the user state (or vice versa) through mechanisms such as ring buffer/perf buffer.
   
2. The kernel state is mainly responsible for the specific computational logic and data collection.
   

wBPF essentially wants to treat the Wasm sandbox as an alternative user-state runtime space on top of the OS, allowing Wasm applications to implement the same programming model and execution logic in the sandbox as eBPF applications that normally run in the user state. wBPF would require a runtime module built on top of the host (outside the sandbox), and some runtime libraries compiled to Wasm bytecode inside the sandbox to provide complete support.

Figure 2 depicts a distributed tracing setup inspired by Hindsight, in which each colored rectangle (blue, orange, teal, green) represents a host participating in the processing of a single request. The horizontal timeline running through these hosts reflects the lifecycle of a trace—uniquely identified by a shared trace ID—as it propagates through the system. At various sampling points (indicated by black circles), telemetry is generated locally on each host.

Common Trace Identifier and Telemetry Stitching. Each time a request arrives at a host, the system either generates a new trace ID or propagates an existing one (as in Hindsight), ensuring the trace remains cohesive across service boundaries. Host 1 (the blue rectangle) collects several data points—spans, metrics, or logs—through Hindsight's lightweight ‘tracepoint‘ calls, buffering them locally rather than immediately ingesting them into a central collector. As the request continues onto Host 2 (orange), more telemetry is accumulated, keyed by the same trace ID.

Meanwhile, Host 3 (teal) might detect an outlier symptom—perhaps an abnormal latency spike or an erroneous status code—which invokes Hindsight's trigger mechanism. This trigger notifies a local agent, which in turn contacts a centralized coordinator to gather the scattered portions of this trace from all hosts that participated in servicing the request. This retrieval of “breadcrumbs” (i.e., references to all machines involved in the trace) ensures that the entire lifetime of the trace can be retroactively sampled and reported—even though the system had not eagerly ingested the data upfront.

In-context Lifetime-Active Heterogeneous Sampling and Log Reduction. When a sufficiently serious anomaly (“edge case”) is detected, Host 2 may decide to “dump the lifetime trace”—effectively firing a trigger to instruct every host holding that trace's data to flush it to a backend trace collector. This action provides operators or automated analysis tools with a complete end-to-end view of how the request flowed through Hosts 1–4.

In parallel, the system may send a “reduce log” signal to Host 3, advising it to scale back logging verbosity. Hindsight easily accommodates such dynamic adjustments to logging levels because local agents can be informed out-of-band; for non-problematic traces, the system simply never reports the collected data and eventually discards it to free up memory. This approach helps avoid both the storage overhead and performance penalties of always capturing high-fidelity trace data.

Clearing Traces and Resource Efficiency. Finally, once the trace reaches Host 4 (green), the system can “clear the trace” upon confirming that no further triggers are required, thereby releasing resources. Here again, Hindsight's lazy ingestion model means that any untriggered traces are eventually evicted from local buffers in bulk, preventing fragmented partial-data scenarios that lead to incoherent traces. Crucially, if no triggers occur, the trace never leaves local memory—no cost is paid in shipping unused data to external collectors.

By weaving these principles into an In-context Lifetime Active Sampling approach, developers gain the dual benefits of selective, on-demand retrieval of traces during edge cases, alongside 100% coverage (i.e., every request is potentially traceable). This design balances deep observability with minimal resource footprint because high-volume, “normal” traces do not burden the backend, while atypical traces undergo rapid ingestion and detailed analysis when needed.

To evaluate the compatibility of wBPF, we tested various eBPF programs across different platforms, including multiple Linux versions, Windows, and userspace eBPF environments. The platforms considered include Linux 5.5, Linux 6.10, Linux 6.10 on ARM64 architecture, Windows, and userspace eBPF. The matrix in Table 1 provides a view of the support for each eBPF program in both native and wBPF environments. The symbols used in the table are "X" for native eBPF and "O" for wBPF, while "-" indicates that the platform is not applicable.

The compatibility results demonstrate that wBPF can effectively run various eBPF programs across different platforms, ensuring binary-level compatibility.

Compared to other embedded Wasm runtimes [3, 14, 19, 27], which typically rely on ad-hoc, ioctl-style interfaces for memory management and hardware access [6, 25, 29, 30], WALI offers a more structured and unified application binary interface (ABI). This approach transcends heterogeneous platforms [10, 11], providing a consistent and well-defined model that accommodates diverse hardware and resource constraints. By standardizing the interaction between Wasm modules and the host system, WALI reduces fragmentation and enhances interoperability, which is crucial for effective edge tracing and observability across varied environments.