Imagine for a moment, being able to fundamentally alter how the very core of your operating system functions, not just patching it, but dynamically changing its behavior, you know, without rebooting or even recompiling a single line of code.
Yeah, that sounds pretty wild.
That almost sounds like a superpower, right, the ability to peer into and even reshape your Linux kernel on the fly. How is that even possible?
It's certainly a profound change in how we interact with the kernel. And that transformative technology is called eBPF EBPFKA. This isn't just some minor upgrade. It's really a revolutionary platform. It's fundamentally changing how we build high performance, security, observability and networking tools.
And that revolutionary aspect is what we're here to unpack today. Good plan for this deep dive, our mission is to explore the power behind eBPF. We'll uncover what it is, why it's become so incredibly important, especially you know, in recent years with cloud native stuff, and how it actually delivers on this promise of un paralleled insight and control. All our insights today, by the way, are drawn from the excellent book Learning EVPF by Liz.
Rice A great resource. And you know, even if you listening aren't planning to write eBPF code yourself, just understanding the fundamentals that gives you a significant advantage. How so, well, you're likely to come across eBPF based tooling more and more. It's rapidly becoming standard in software infrastructure roles, right, and knowing how it works underneath will help you use those tools much more effectively. It gives you a real edge.
Okay, so let's unpack this. To really get eBPF, we probably need to start with the Linux kernel itself. How would you describe its role for someone maybe not deep in OS internals.
Sure, think of the kernel as the essential software layer, sitting directly between your applications and the physical hardware.
The mediator exactly.
It's the traffic cop for all interactions. Immediates every request from userspace apps through things called system calls or ciscalls, like reading files, network stuff files, sending network traffic, even just accessing memory. The kernel handles it all. Everything ultimately goes through the kernel.
And because applications rely so heavily on the kernel for well everything, being able to observe those interactions must give you incredible visibility into what an app is doing.
That's the key.
But where did this idea of hooking into the kernel start. It had pretty humble origins, didn't they. Oh?
Absolutely. The BPF and EDPF originally stood for Berkeley Packet Filter, started way back.
In nineteen ninety two, ninety two, Okay.
Yeah, simply as a way to efficiently filter network packets. We're talking in general purpose thirty two bit instructions looked a lot like assembly.
Language, so like for TCP dump or something.
Precisely, a tiny piece of BPF code could say filter out non ippaydance from an Ethernet stream, letting GZP dump capture only what was relevant. That efficiency was groundbreaking for network analysis back then.
So the original BPF came into Lenux in ninety seven kernel two point one one seventy five for that purpose. But I guess the packet filter part of the name started to make less sense over time. It did.
It became much more than just packets.
What was the turning point? When did it become more versatile?
Well, the first step beyond just network packets was probably SUCCUMBPF and twenty twelve Kernel three point.
Five sacked and TTF right for ciscall filter exactly.
It let BPS programs decide whether to allow or deny user space apps from making specific system calls, so you could restrict what an application could do at the kernel level.
A security thing.
Yeah, foundational for containers security later on. But the real extended revolution, the birth of eBPF as we know it, began in kernel three point one eight. That was twenty fourteen.
Okay, twenty fourteen. And what changed then? It sounds like it was more than just an incremental update.
Oh, it was a complete overhaul. The instruction set became sixty four bit, the interpreter was rewritten, Crucial new features were added. That's just This is where we got eBPF maps for sharing data between programs and user space.
Ah okay, that sounds.
Important, hugely important. Also the VPF system call for userspace interaction, and maybe most critically for safety, the eBPF FAIRI fire was introduced. We'll definitely need to talk more about that.
Absolutely. So if adding new kernel features is traditionally you know, a massive undertaking months, maybe years to get changes upstream, how does eBPF change that? What makes it such a game changer today?
That's the core of its appeal. Really Yeah, eBPF fundamentally changes how we can extend the kernel. It allows for dynamic kernel functionality, dynamic meaning meaning eBPF programs can be loaded and unloaded on demand. This dramatically speeds up adding new kernel features. We're talking maybe months, not the years it usually takes for upstream kernel changes.
Wow.
So you can create and deploy custom kernel functionality incredibly quickly without needing you know, the entire Linux world to agree and merge it into the main kernel.
And it's not just about speed of deployment, is it. Performance is key.
Too, absolutely crucial. Once an eBPF program is loaded and JIT compiled, I'm compiled right, it runs as native machine instructions directly on the CPU. This avoids those costly back and forth transitions between kernel space and user space for every single event.
Which makes it incredibly efficient.
Remarkably efficient. It sounds almost counterintuitive for something running inside the kernel, but yeah, it allows EVPF tools to operate with minimal overhead.
That efficiency sounds like a massive win, especially in cloud native environments. We often see that sidecar model in kubernettes for injecting things like logging tracing.
Security yep, the sidecar pattern.
How does EVPF compare. Is it a better alternative?
In many ways? Yes, The sidecar approach, while useful, often means modifying application yamo files, restarting.
Pods, right operational friction.
Exactly, and if something goes wrong, the sidecar might not even get injected, leaving you blind. Plus, sidecars can potentially be bypassed by bad actors. How so well, imagine an attacker deploys an uninstrumented app like a crypto minor sidecar based security tool which relies on injection, might not even see it connecting out to its mining pool.
Ah I see, and eBPF avoids that Precisely.
eBPF based tools run on the host and can police all traffic, all processes on that machine, including everything inside containers. They're much harder to.
Side steps, so instant complete visibility.
Pretty much without needing app modifications or restarts. That's a game changer for infrastructure teams wanting comprehensive coverage.
Okay, but these programs are running directly inside the kernel. That immediately raises safety concerns. What stops a buggy eBPF program from crashing the whole system or reading memory. It shouldn't.
That's the million dollar question, and the answer is the eBPF verifier.
The kernel's unseen guardian you called it.
That's a good way to put it. Before an eBPF program is ever loaded, the verifier performs an exhaustive static analysis. It checks every single possible execution path, every path, every path to ensure its safe and won't harm the kernel. It's like a super strict bouncer at the kernel's door. Honestly, it's a marvel of engineering that makes this whole thing possible safely.
So what kinds of checks does this bouncer perform? It must be incredibly rigorous, it really is.
It tracks the state of each register their types, value ranges to prevent invalid operations. It enforces trick memory access, making sure programs only touch memory they're allowed to.
No arbitrary reads are writes none.
It mandates checks for null pointers before they're reference, preventing common crashes. Very importantly, it guarantees programs run to completion.
How does it do that?
It rejects programs with loops that it can't prove will terminate. There's also a maximum instruction limit, currently a million instructions for privileged users a million.
Okay, that's quite a lot.
Actually, it gives you scope for complex logic. The verifier also checks that helper functions called by the eBPF program are allowed for that specific program type, and it even checks for GPL compatible licenses if the program uses certain restricted helpers.
That's incredibly thorough. It really builds confidence.
It has to be. The kernel stability is.
Paramount, Okay, beyond safety. You mentioned eBPF maps earlier as a key differentiator from the original BPF. How did maps change the game?
Maps was a fundamental breakthrough. They are essentially key value stores, data structures that act as a bridge, a bridge between between the kernel and user space, or even between different eBPF programs running in the kernel. They allow communication and state.
Sharing, which wasn't possible before.
Right. Classic BPF was stateless, just processing individual packets. Maps allow eBPF programs to store state across multiple events or invocations, so you can use.
Them for things like counters sending metrics from the kernel up to a userspace monitoring tool exactly.
That's a common use case, or storing configuration data pushed down from user space, or maintaining state across multiple program executions like global variables.
And you mentioned tail calls too. What are those?
Tail calls allow one eBPF program to effectively jump to another eBPF program, replacing itself. It's a way to chain programs together.
Ah okay, so you can build more complex logic flows.
Precisely, you can chain up to thirty three tail calls currently, combined with that million instruction limit per program, it gives you a lot of flexibility for sophisticated in kernel logic.
So maps and tail calls really elevated eBPF from just a filter to a proper programmable platform.
Absolutely, they are key to building that sophisticated eBPF based applications we see today.
Now, if you're thinking about building with eBPF, what are the language choices for the kernel side code itself? I assume it's pretty low level.
Yes, For the code running in the kernel. Most eBPF programs are written in C or Rust. Why those Because they are compilers like Clang LLVM for C can directly target the EDPF bytecode instruction set. They give you that necessary low level control.
Makes sense? And what about the user space side? The applications that load and manage these kernel programs.
There you have much more flexibility. User space applications can be written in various languages using specific libraries to interact with the kernel's eBPF capabilities.
When eBPF was first taking off, I remember BCC being very popular.
Yes, BCC, the BPF compiler collection. It supports Python, Lua, C plus plus C. It was fantastic for learning and experimentation.
Why was it good for learning?
It abstracted away a lot of the low level complexity, making EDPF accessible to people who weren't necessarily deep kernel hackers. You could write Python scripts to load and interact with CEBPF code.
But it had drawbacks for production it did.
Its main issue was a compile the EDPFC code at runtime on the target machine.
Oh right, so you needed the compiler toolchain LVM CLIANG kernel headers on every single server exactly.
That meant bigger dependencies, potential installation headaches, and noticeable startup delays while the code compiled plus higher resource consumption not ideal for large scale production deployments.
So there wasn't need something more robust, more portable for enterprise use. That leads us to KEOII, right, compile once, run everywhere.
Yes. Keyoor is the modern approach and a huge leap forward. It's really what made eBPF practical for widespread production use.
How does it achieve that run everywhere magic? It sounds tricky, with different kernel versions having slightly different data structures, It is clever.
The key is BTF BPF type format. During compilation, detailed type information about the kernel structures the eBPF program uses is embedded into the compiled object file itself. Okay, Then when you load that program on a target machine, a userspace library uses that BTF data to understand the specific kernel version running there. It can then automatically adjust memory offsets and structure accesses on the fly, so it.
Adapts the pre compiled code at load time.
Essentially, yes, it relocates the memory accesses based on the actual kernel's layout. This means you can compile your EVPF program once on your rig machine and then deploy that same binary across a whole fleet running different kernel versions within limits.
Of course, that's a massive operational win. No more compiling on every node huge.
It makes deployment much much simpler.
And the core c library that enables score is LIBPF.
That's right. Lip APF is maintained alongside the kernel source tree, and it's the reference implementation. It uses auto generated BPF skeletons c headers that make loading and managing the eBPF programs and maps much cleaner from user space.
And the benefits over BCC are.
Lower memory footprint, no runtime compilation dependency, no startup delay, and that crucial keore portability. If you're building a serious eBPF application today, especially in C, LIBPF is the way to go.
But it's not just C anymore, is it. The ecosystem seems to be growing definitely.
The community has built excellent libraries for other popular languages. For Go developers, there's silly MEPF and also libbed PFG, which provides Go bindings for lipef.
And for Rust. Rust seems like a natural for systems programming.
It does, and there are several options there too, like libpfrs red, BPF, and AYA. Each has slightly different approaches and trade offs, but they all bring eBPF development to Rust programmers, leveraging Rust safety features.
So lots of options depending on your team's preferred language exactly.
It's much more accessible now than it was just a few years ago.
Okay, so we have this powerful, safe portable technology, how is it actually being used? Let's talk real world applications. Observability seems like a prime candidate.
Absolutely because eBPF programs can attach to so many points ciscolls, kernel trace points, kernel functions, eight K probes, even user space functions uprobes. They provide incredibly powerful tracing capabilities.
Deeper than traditional monitoring, much.
Deeper and often with less overhead. You get unparalleled insight into system behavior, network disc io function calls within applications, all without modifying or restarting the application itself. It's like having x ray vision into your running systems.
And in networking. You mentioned BPF's origins there, but eBPF takes it much further, especially with XDP.
Yes XDP the express data path. This is really powerful. XDP allows eBPF programs to run at the earliest possible point in the network stack, right in the driver, even before the kernel allocates memory for the packet BSKB.
So super fast packet process extremely fast.
This lets you do things like high performance firewalling, dropping malicious packets for dedas mitigation, or implementing custom load balancing all at line rate in many cases.
Before the packet even really hits the main kernel networking exactly.
You can make decisions and potentially drop packets with minimal overhead. Some network cards even support offloading XDP programs directly onto the NIC hardware itself.
Wow, hardware offload. That's serious performance it is.
It allows you to handle enormous amounts of traffic. Think about mitigating those packet of death vulnerabilities. XDP can drop the malicious packet before it can even cause harm in the network stack.
And this has big implications for Kubernetes networking too, doesn't it replacing things like.
Iptables fundamentally changing it. Yes. Traditional Kubernetes networking often relies heavily on iptables for implementing network policies and services and contract for connection.
Tracking, and those have scaling issues, right they do.
Iptable's rules are processed linearly, so performance degrades is the number of rules or services grows. That's o n complexity. Contract can also become a bottleneck.
How does eBPF help.
eBPF based networking solutions like Cilium replace Iptable's rules and contract with EVPF programs and maps they use efficient hash maps for lookups.
Hash maps the lookups are faster.
Much faster oh one or constant time lookups for network policies, service load balancing, and connection tracking regardless of the number of rules or connections. This is critical for performance and scalability in large Kubernetes clusters.
That's a significant architectural shift. Now here's a capability that sounds almost like magic transparent ryption insight. How can eBPF see inside encrypted traffic without keys?
It's a clever trick. It doesn't actually break the encryption itself. Instead, eBPF uses uprobes to hook into userspace SSLTLS libraries like open ssl ortls.
So it attaches to the applications encryption functions precisely.
It attaches to functions like SSL read and SSL write. It can then intercept the data after the library has decrypted it for SSL rate or before the library encrypts it for SSL right.
Ah. So it catches the plaintext data inside the application's memory just before encryption or just after decryption exactly.
You get visibility into the clear text data flow without needing access to the private keys or certificates. It's incredibly powerful for security monitoring, troubleshooting, and even compliance without disrupting the encryption itself.
That's fascinating. Okay, let's shift fully to security. eBPF seems poised to move security from just detection to actual prevention.
That's a key trend for years. Second PPF provided a basic layer by filtering ciscoles, which is used heavily in Docker and Kubernetes.
Right limiting what a container can do.
Yes, and some eBPF tools like inspector gadget can even help generate those secom profiles by observing what ciscoles an application legitimately uses.
But generating perfect profiles is hard right capturing all error pads and edge cases.
It's very challenging, and traditional detection often suffers from that Tatau problem time of check to time of use the race condition. Exactly by the time a separate security agent detects a malicious ciscle or file access and decides to act, the damage might already be done. The malicious operation might have completed.
So how does eBPF enable prevention closing that time gap?
This is where a specific eBPF program type comes in bpfpot typos. These programs attached to hooks within the Linux Security Module Framework.
LSM, like a Parmer or Cylenux.
Use exactly the same framework. These eBPF LSM programs can perform he checks directly inside the kernel before a sensitive operation happens, and crucially, they can return.
An air code and that error code.
It prevents the operation from proceeding. The kernel respects the error code returned by the LSM hook, stopping the action before it even starts. It happens synchronously within the context of the original operation.
So it's in kernel enforcement, not just after the fact detection.
Precisely, it's a fundamental shift from reactive detection to proactive synchronous prevention.
And tools like Cilium Tetragon build on this.
They do. Tetragen uses EVPF attaching to k probes and trace points, not just LSM hooks, to get rich context about security events. It can filter these events efficiently in the kernel, reducing overhead, and.
Can it actively stop threats Yes.
Based on his policies, it can use helpers like bpfen signal to for example, send a sig kill signal to terminate a malicious process immediately synchronously when a violation is detected.
So detect and terminate right there in the kernel context exactly.
It moves security beyond just observing and alerting to actively enforcing policy and preventing harm in real time.
So to kind of wrap this deep dive up, eBPF has really come a long way from just filtering packets.
An incredible journey.
It's evolved into this powerful, versatile platform that's genuinely transforming observability, networking, and security, giving us unprecedented control and insight right at the kernel level.
It really feels like we're still just scratching the surface of what's possible with it. The innovation is happening incredibly fast, and.
The future looks bright. What's next?
Well, The eBPF Foundation, which includes major players like Google, Meta, Microsoft, Netflix, isovalent. They're working on standardizing.
eBPF standardization is always good, yeah.
And even exploring its adoption for Windows, which could potentially allow eBPF programs written for Linux to run on Windows too.
Wow. Cross platform eBPF.
That's the long term vision for some. It's a rapidly evolving field. New kernel features are landing all the time, though it's worth remembering there's always a lag before the very latest features make it into stable Linux distributions.
You might be running sure the usual adoption cycle.
Right, but the pace of development is really impressive.
It certainly seems like it's going to shape how many of our tools work now and in the future. So maybe a final thought for our listeners, EDPF seems to be shifting from this cutting edge niche thing to a fundamental platform for infrastructure tooling.
Seems that way so well.
Understanding how this magic works become as essential for developers and operators today as say, understanding networking protocols or distributed systems.
That's a really interesting question to ponder. The tools built on EEDPF are definitely becoming ubiquitous, things like Cilium, Falco, Pixie, Tetric.
Carey'll encounter them more and more.
You will, and having even that foundational understanding of what's happening under the hood how eBPF enables them, it definitely gives you a significant advantage in using them effectively, troubleshooting them, and just understanding your systems better.
Well. Hopefully this deep dive gave you a good starting point. We encourage you to think about how these eBPF based tools might shape your work and maybe explore some of the projects or Liz Rice's book if you want to go deeper.
It's a fascinating space to watch.
Absolutely thanks for joining us for this deep dive.