Developing the bpfman-operator
This section is intended to give developer level details regarding the layout and design of the bpfman-operator. At its core the operator was implemented using the operator-sdk framework which make those docs another good resource if anything is missed here.
High level design overview
This repository houses two main processes, the
bpfman-agent and the
bpfman-operator along with CRD api definitions
The following diagram depicts how all these components work together to create a functioning operator.
Building and deploying
For building and deploying the bpfman-operator simply see the attached
help Display this help.
kustomize Download kustomize locally if necessary.
controller-gen Download controller-gen locally if necessary.
envtest Download envtest-setup locally if necessary.
opm Download opm locally if necessary.
manifests Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
generate Generate ALL auto-generated code.
generate-register Generate register code see all `zz_generated.register.go` files.
generate-deepcopy Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations see all `zz_generated.register.go` files.
generate-typed-clients Generate typed client code
generate-typed-listers Generate typed listers code
generate-typed-informers Generate typed informers code
fmt Run go fmt against code.
verify Verify all the autogenerated code
test Run Unit tests.
test-integration Run Integration tests.
bundle Generate bundle manifests and metadata, then validate generated files.
build-release-yamls Generate the crd install bundle for a specific release version.
build Build bpfman-operator and bpfman-agent binaries.
build-images Build bpfman, bpfman-agent, and bpfman-operator images.
push-images Push bpfman, bpfman-agent, bpfman-operator images.
load-images-kind Load bpfman, bpfman-agent, and bpfman-operator images into the running local kind devel cluster.
bundle-build Build the bundle image.
bundle-push Push the bundle image.
catalog-build Build a catalog image.
catalog-push Push a catalog image.
install Install CRDs into the K8s cluster specified in ~/.kube/config.
uninstall Uninstall CRDs from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
Vanilla K8s Deployment
setup-kind Setup Kind cluster
deploy Deploy bpfman-operator to the K8s cluster specified in ~/.kube/config with the csi driver initialized.
undeploy Undeploy bpfman-operator from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
kind-reload-images Reload locally build images into a kind cluster and restart the ds and deployment so they're picked up.
run-on-kind Kind Deploy runs the bpfman-operator on a local kind cluster using local builds of bpfman, bpfman-agent, and bpfman-operator
deploy-openshift Deploy bpfman-operator to the Openshift cluster specified in ~/.kube/config.
undeploy-openshift Undeploy bpfman-operator from the Openshift cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
Running Locally in KIND
To run locally in a kind cluster with an up to date build simply run:
The container images used for
bpfman-operator can also be manually configured,
by default local image builds will be used for the kind deployment.
Then rebuild and load a fresh build run:
Which will rebuild the bpfman-operator, bpfman-agent, and bpfman images and load them into the kind cluster.
To run all of the Unit Tests defined in the bpfman-operator controller code simply run
To run Integration Tests locally:
- Build the images locally with the
- Run the integration test suite.
Additionally the integration test can be configured with the following environment variables:
- KEEP_TEST_CLUSTER: If set to
truethe test cluster will not be torn down after the integration test suite completes.
- USE_EXISTING_KIND_CLUSTER: If this is set to the name of the existing kind cluster the integration test suite will use that cluster instead of creating a new one.
The bpfman-operator project layout is guided by the recommendations from both the operator-sdk framework and the standard golang project-layout. The following is a brief description of the main directories and their contents.
NOTE: Bolded directories contain auto-generated code
/apis: Contains the K8s CRD api definitions(
*_types.go) for each version along with the auto-generated register and deepcopy methods(
/bundle: Contains the OLM bundle manifests and metadata for the operator. More details can be found in the operator-sdk documentation.
/cmd: Contains the main entry-points for the bpfman-operator and bpfman-agent processes.
/config: Contains the configuration files for launching the bpfman-operator on a cluster.
/bpfman-deployment: Contains static deployment yamls for the bpfman-daemon, this includes two containers, one for
bpfmanand the other for the
bpfman-agent. This DaemonSet yaml is NOT deployed statically by kustomize, instead it's statically copied into the operator image which is then responsible for deploying and configuring the bpfman-daemon DaemonSet. Lastly, this directory also contains the default config used to configure the bpfman-daemon, along with the cert-manager certificates used to encrypt communication between the bpfman-agent and bpfman.
/bpfman-operator-deployment:Contains the static deployment yaml for the bpfman-operator. This is deployed statically by kustomize.
/crd: Contains the CRD manifests for all of the bpfman-operator APIs.
/bases: Is where the actual CRD definitions are stored. These definitions are auto-generated by controller-gen.
/default: Contains the default deployment configuration for the bpfman-operator.
/manifests: Contains the bases for generating OLM manifests.
/openshift: Contains the Openshift specific deployment configuration for the bpfman-operator.
/prometheus: Contains the prometheus manifests used to deploy Prometheus to a cluster. At the time of writing this the bpfman-operator is NOT exposing any metrics to prometheus, but this is a future goal.
/rbac: Contains rbac yamls for getting bpfman and the bpfman-operator up and running on Kubernetes.
/bpfman-agent: Contains the rbac yamls for the bpfman-agent. They are automatically generated by kubebuilder via build tags in the bpfman-agent controller code.
/bpfman-operator: Contains the rbac yamls for the bpfman-operator. They are automatically generated by kubebuilder via build tags in the bpfman-operator controller code.
/samples: Contains sample CR definitions that can be deployed by users for each of our supported APIs.
/scorecard: Contains the scorecard manifests used to deploy scorecard to a cluster. At the time of writing this the bpfman-operator is NOT running any scorecard tests.
/test: Contains the test manifests used to deploy the bpfman-operator to a kind cluster for integration testing.
/controllers: Contains the controller implementations for all of the bpfman-operator APIs. Each controller is responsible for reconciling the state of the cluster with the desired state defined by the user. This is where the source of truth for the auto-generated RBAC can be found, keep an eye out for
/bpfmanagent: Contains the controller implementations which reconcile user created
*Programtypes to multiple
/bpfmanoperator: Contains the controller implementations which reconcile global
BpfProgramobject state back to the user by ensuring the user created
*Programobjects are reporting the correct status.
/hack: Contains any scripts+static files used by the bpfman-operator to facilitate development.
/internal: Contains all private library code and is used by the bpfman-operator and bpfman-agent controllers.
/pkg: Contains all public library code this is consumed externally and internally.
/client: Contains the autogenerated clientset, informers and listers for all of the bpfman-operator APIs. These are autogenerated by the k8s.io/code-generator project, and can be consumed by users wishing to programmatically interact with bpfman specific APIs.
/helpers: Contains helper functions which can be consumed by users wishing to programmatically interact with bpfman specific APIs.
/test/integration: Contains integration tests for the bpfman-operator. These tests are run against a kind cluster and are responsible for testing the bpfman-operator in a real cluster environment. It uses the kubernetes-testing-framework project to programmatically spin-up all of the required infrastructure for our unit tests.
Makefile: Contains all of the make targets used to build, test, and generate code used by the bpfman-operator.
Metrics/Health port issues
In some scenarios, the health and metric ports may are already in use by other services on the system. When this happens the bpfman-agent container fails to deploy. The ports currently default to 8175 and 8174.
The ports are passed in through the daemonset.yaml for the
bpfman-daemon and deployment.yaml and
manager_auth_proxy_patch.yaml for the
The easiest way to change which ports are used is to update these yaml files and rebuild the container images.
The container images need to be rebuilt because the
bpfman-daemon is deployed from the
and the associated yaml files are copied into the
If rebuild the container images is not desirable, then the ports can be changed on the fly.
bpfman-operator, the ports can be updated by editing the
kubectl edit deployment -n bpfman bpfman-operator
- --upstream=http://127.0.0.1:8174/ <-- UPDATE
- --health-probe-bind-address=:8175 <-- UPDATE
- --metrics-bind-address=127.0.0.1:8174 <-- UPDATE
port: 8175 <-- UPDATE
port: 8175 <-- UPDATE
bpfman-daemon, the ports could be updated by editing the
bpfman-daemon is restarted for any reason by the
bpfman-operator, the changes
will be lost. So it is recommended to update the ports for the
bpfman-daemon via the bpfman
kubectl edit configmap -n bpfman bpfman-config
bpfman.agent.healthprobe.addr: :8175 <-- UPDATE
bpfman.agent.metric.addr: 127.0.0.1:8174 <-- UPDATE