Deploying Example eBPF Programs On Local Host
This section describes running bpfd and the example eBPF programs on a local host.
When running bpfd, it can be run as a process or run as a systemd service.
Examples run the same, independent of how bpfd is deployed, other than requiring sudo or not.
NOTE: When running bpfd as a systemd service, bpfd can only access files owned by the
bpfduser, which is created by the install script (./script/setup.sh install). In this case, the install script also copies the examples eBPF bytecode from the examples directory to a directory owned by bpfd (/run/bpfd/examples/). When bpfd is run as a privileged process, the bytecode can be loaded from the examples directory. The example code looks in both locations (/run/bpfd/examples/first), so if the bytecode is recompiled and bpfd is running as a systemd service, run./script/setup.sh examplesto update the copied version.
Building
To build directly on a system, make sure all the prerequisites are met, then build.
Prerequisites
This assumes bpfd is already installed and running on the system. If not, see Setup and Building bpfd.
- All requirements defined by the
cilium/ebpfpackage -
libbpf development package to get the required eBPF c headers
Fedora:
sudo dnf install libbpf-develUbuntu:
sudo apt-get install libbpf-dev -
Cilium's
bpf2gobinarygo install github.com/cilium/ebpf/cmd/bpf2go@master
Building Locally
To build all the C based eBPF counter bytecode, run:
To build all the Userspace GO Client examples, run:
To build only a single example:
Running On Host
The most basic way to deploy this example is running directly on a host system.
First, start or ensure bpfd is up and running.
Tutorial will guide you through deploying bpfd.
In all the examples of running on a host system, a bpfd-client certificate is used
that is generated by bpfd to encrypt the application's connection to bpfd.
The diagram below shows go-xdp-counter example, but the go-tc-counter and
go-tracepoint-counter examples operate exactly the same way.
Following the diagram (Purple numbers):
- When
go-xdp-counteruserspace is started, it will send a gRPC request over mTLS tobpfdrequestingbpfdto load thego-xdp-countereBPF bytecode located on disk atbpfd/examples/go-xdp-counter/bpf_bpfel.oat a priority of 50 and on interfaceens3. These values are configurable as we will see later, but for now we will use the defaults (except interface, which is required to be entered). bpfdwill load it'sdispatchereBPF program, which links to thego-xdp-countereBPF program and return a UUID referencing the running program.bpfctl listcan be used to show that the eBPF program was loaded.- Once the
go-xdp-countereBPF bytecode is loaded, the eBPF program will write packet counts and byte counts to a shared map. go-xdp-counteruserspace program periodically reads counters from the shared map and logs the value.
Running Privileged
The most basic example, just use sudo to start the go-xdp-counter program.
Determine the host interface to attach the eBPF program to and then start the go program with:
or (NOTE: TC programs also require a direction, ingress or egress)
cd bpfd/examples/go-tc-counter/
sudo ./go-tc-counter -direction ingress -iface <INTERNET INTERFACE NAME>
or
The output should show the count and total bytes of packets as they pass through the interface as shown below:
sudo ./go-xdp-counter --iface vethff657c7
2023/07/17 17:43:58 Using Input: Interface=vethff657c7 Priority=50 Source=/run/bpfd/examples/go-xdp-counter/bpf_bpfel.o
2023/07/17 17:43:58 Unable to read /etc/bpfd/bpfd.toml, using default configuration values.
2023/07/17 17:43:58 Program registered with id 6211
2023/07/17 17:44:01 4 packets received
2023/07/17 17:44:01 580 bytes received
2023/07/17 17:44:04 4 packets received
2023/07/17 17:44:04 580 bytes received
2023/07/17 17:44:07 8 packets received
2023/07/17 17:44:07 1160 bytes received
:
Use bpfctl to show the go-xdp-counter eBPF bytecode was loaded.
Finally, press <CTRL>+c when finished with go-xdp-counter.
:
2023/07/17 17:44:34 28 packets received
2023/07/17 17:44:34 4060 bytes received
^C2023/07/17 17:44:35 Exiting...
2023/07/17 17:44:35 Unloading Program: 6211
Running Unprivileged
To run the examples unprivileged (without sudo), the following three steps must be performed.
Step 1: Create bpfd User Group
The Tutorial guide describes the different modes bpfd can be run in.
Specifically, the Systemd Service section explains how to
start bpfd and create the bpfd Users and bpfd User Group.
bpfd must be started as a Systemd Service and the examples must be run from a User that is a
member of the bpfd User Group.
If the user running the userspace program is not a member of the bpfd user group, then the userspace
program cannot access the map files shared between the BFP program and the userspace program.
If the above step is skipped, then the userspace program must be run with sudo or the program must
be granted CAP_DAC_OVERRIDE capabilities (sudo /sbin/setcap cap_dac_override=ep ./go-xdp-counter).
Step 2: Grant CAP_BPF Linux Capability
NOTE: Only for kernel versions prior to kernel 5.19
The examples use a map to share data between the userspace side of the program and the eBPF portion.
Accessing this map requires access to the CAP_BPF capability for kernel versions prior to kernel 5.19.
Run the following command to grant go-xdp-counter access to the CAP_BPF capability:
and
and
Reminder: The capability must be re-granted each time the examples are rebuilt.
Step 3: Start go-xdp-counter without sudo
Start go-xdp-counter without sudo:
./go-xdp-counter -iface ens3
2022/12/02 15:59:34 Unable to read /etc/bpfd/bpfd.toml, using default configuration values.
2022/12/02 15:59:34 Using Input: Interface=ens3 Priority=50 Source=/run/bpfd/examples/go-xdp-counter/bpf_bpfel.o
2022/12/02 15:59:35 Program registered with id 6221
2022/12/02 15:59:38 0 packets received
2022/12/02 15:59:38 0 bytes received
2022/12/02 15:59:41 4 packets received
2022/12/02 15:59:41 580 bytes received
:
2022/12/02 16:00:59 64 packets received
2022/12/02 16:00:59 9280 bytes received
^C2022/12/02 16:01:00 Exiting...
2022/12/02 16:01:00 Unloading Program: 6221
Passing eBPF Bytecode In A Container Image
bpfd can load eBPF bytecode from a container image built following the spec described in eBPF Bytecode Image Specifications. Pre-built eBPF container images for the examples can be loaded from:
quay.io/bpfd-bytecode/go-xdp-counter:latestquay.io/bpfd-bytecode/go-tc-counter:latestquay.io/bpfd-bytecode/go-tracepoint-counter:latest
To use the container image, pass the URL to the userspace program:
./go-xdp-counter -iface ens3 -image quay.io/bpfd-bytecode/go-xdp-counter:latest
2022/12/02 16:28:32 Unable to read /etc/bpfd/bpfd.toml, using default configuration values.
2022/12/02 16:28:32 Using Input: Interface=ens3 Priority=50 Source=quay.io/bpfd-bytecode/go-xdp-counter:latest
2022/12/02 16:28:34 Program registered with id 6223
2022/12/02 16:28:37 4 packets received
2022/12/02 16:28:37 580 bytes received
2022/12/02 16:28:40 4 packets received
2022/12/02 16:28:40 580 bytes received
^C2022/12/02 16:28:42 Exiting...
2022/12/02 16:28:42 Unloading Program: 6223
Building eBPF Bytecode Container Image
eBPF Bytecode Image Specifications provides detailed
instructions on building and shipping bytecode in a container image.
To build go-xdp-counter and go-tc-counter eBPF bytecode container image, first make sure the
bytecode has been built (i.e. bpf_bpfel.o has been built - see Building), then
run the build commands below:
cd bpfd/examples/go-xdp-counter/
go generate
docker build \
--build-arg PROGRAM_NAME=go-xdp-counter \
--build-arg BPF_FUNCTION_NAME=xdp_stats \
--build-arg PROGRAM_TYPE=xdp \
--build-arg BYTECODE_FILENAME=bpf_bpfel.o \
--build-arg KERNEL_COMPILE_VER=$(uname -r) \
-f ../../packaging/container-deployment/Containerfile.bytecode . -t quay.io/$USER/go-xdp-counter-bytecode:latest
and
cd bpfd/examples/go-tc-counter/
go generate
docker build \
--build-arg PROGRAM_NAME=go-tc-counter \
--build-arg BPF_FUNCTION_NAME=stats \
--build-arg PROGRAM_TYPE=tc \
--build-arg BYTECODE_FILENAME=bpf_bpfel.o \
--build-arg KERNEL_COMPILE_VER=$(uname -r) \
-f ../../packaging/container-deployment/Containerfile.bytecode . -t quay.io/$USER/go-tc-counter-bytecode:latest
and
cd bpfd/examples/go-tracepoint-counter/
go generate
docker build \
--build-arg PROGRAM_NAME=go-tracepoint-counter \
--build-arg BPF_FUNCTION_NAME=tracepoint_kill_recorder \
--build-arg PROGRAM_TYPE=tracepoint \
--build-arg BYTECODE_FILENAME=bpf_bpfel.o \
--build-arg KERNEL_COMPILE_VER=$(uname -r) \
-f ../../packaging/container-deployment/Containerfile.bytecode . -t quay.io/$USER/go-tracepoint-counter-bytecode:latest
bpfd currently does not provide a method for pre-loading bytecode images
(see issue #603), so push the bytecode image to a remote
repository.
For example:
docker login quay.io
docker push quay.io/$USER/go-xdp-counter-bytecode:latest
docker push quay.io/$USER/go-tc-counter-bytecode:latest
Then run with the privately built bytecode container image:
./go-tc-counter -iface ens3 -direction ingress -location image://quay.io/$USER/go-tc-counter-bytecode:latest
2022/12/02 16:38:44 Unable to read /etc/bpfd/bpfd.toml, using default configuration values.
2022/12/02 16:38:44 Using Input: Interface=ens3 Priority=50 Source=quay.io/$USER/go-tc-counter-bytecode:latest
2022/12/02 16:38:45 Program registered with id 6225
2022/12/02 16:38:48 4 packets received
2022/12/02 16:38:48 580 bytes received
2022/12/02 16:38:51 4 packets received
2022/12/02 16:38:51 580 bytes received
^C2022/12/02 16:38:51 Exiting...
2022/12/02 16:38:51 Unloading Program: 6225
Preloading eBPF Bytecode
Another way to load the eBPF bytecode is to pre-load the eBPF bytecode and
pass the associated bpfd program id to the userspace program.
This is similar to how eBPF programs will be loaded in Kubernetes, except kubectl commands will be
used to create Kubernetes CRD objects instead of using bpfctl, but that is covered in the next section.
The userspace programs will skip the loading portion and use the program id to find the shared
map and continue from there.
Referring back to the diagram above, the load and unload are being done by bpfctl and not
go-xdp-counter userspace program.
First, use bpfctl to load the go-xdp-counter eBPF bytecode:
bpfctl load-from-image --image-url quay.io/bpfd-bytecode/go-xdp-counter:latest xdp --iface ens3 --priority 50
Bpfd State
---------------
Name: xdp_stats
Image URL: quay.io/bpfd-bytecode/go-xdp-counter:latest
Pull Policy: IfNotPresent
Global: None
Metadata: None
Map Pin Path: /run/bpfd/fs/maps/6229
Map Owner ID: None
Map Used By: 6229
Priority: 50
Iface: ens3
Position: 0
Proceed On: pass, dispatcher_return
Kernel State
----------------------------------
ID: 6229
Name: xdp_stats
Type: xdp
Loaded At: 2023-07-17T17:48:10-0400
Tag: 4b9d1b2c140e87ce
GPL Compatible: true
Map IDs: [2724]
BTF ID: 2834
Size Translated (bytes): 168
JITed: true
Size JITed (bytes): 104
Kernel Allocated Memory (bytes): 4096
Verified Instruction Count: 21
Then run the go-xdp-counter userspace program, passing in the UUID:
./go-xdp-counter -iface ens3 -id 6229
2022/12/02 17:01:38 Using Input: Interface=ens3 Source=6229
2022/12/02 17:01:41 180 packets received
2022/12/02 17:01:41 26100 bytes received
2022/12/02 17:01:44 184 packets received
2022/12/02 17:01:44 26680 bytes received
^C2022/12/02 17:01:46 Exiting...
2022/12/02 17:01:46 Closing Connection for Program: 6229
Then use bpfctl to unload the eBPF bytecode: