Skip to content

Release Process

This document describes the process for making a release for the bpfman project.

Overview

The bpfman project includes both the bpfman and bpfman-operator repositories. When a release is made for the project, a release is created for each repository with the same version number.

Each bpfman project release is comprised of the following major components:

  • bpfman (Core library) and bpfman-api (Core GRPC API protobuf definitions) library crates
  • bpfman (CLI), and bpfman-rpc (gRPC server) binary crates
  • bpf-metrics-exporter and bpf-log-exporter binary crates
  • bpfman RPMs stored in the bpfman COPR repository.
  • Kubernetes User Facing Custom Resource Definitions (CRDs)
    • BpfApplication
    • FentryProgram
    • FexitProgram
    • KprobeProgram
    • TcProgram
    • TcxProgram
    • TracepointProgram
    • UprobeProgram
    • XdpProgram
  • Corresponding go pkgs in the form of github.com/bpfman/bpfman which includes the following:
    • github.com/bpfman/bpfman/clients/gobpfman/v1: The go client for the bpfman GRPC API API helpers.
  • Corresponding go pkgs in the form of github.com/bpfman/bpfman-operator which includes the following:
    • github.com/bpfman/bpfman-operator/apis: The go bindings for the bpfman CRD API
    • github.com/bpfman/bpfman-operator/pkg/client: The autogenerated clientset for the bpfman CRD API
    • github.com/bpfman/bpfman-operator/pkg/helpers: The provided bpfman CRD API helpers.
  • The following core component container images with tag <RELEASE_VERSION>:
    • quay.io/bpfman/bpfman-agent
    • quay.io/bpfman/bpfman-operator-bundle
    • quay.io/bpfman/bpfman-operator
    • quay.io/bpfman/bpfman
    • quay.io/bpfman/tc-dispatcher
    • quay.io/bpfman/xdp-dispatcher
  • The relevant example bytecode container images with tag <RELEASE_VERSION> from source code located in the bpfman project:
    • quay.io/bpfman-bytecode/fentry
    • quay.io/bpfman-bytecode/fexit
    • quay.io/bpfman-bytecode/go-app-counter
    • quay.io/bpfman-bytecode/go-kprobe-counter
    • quay.io/bpfman-bytecode/go-tc-counter
    • quay.io/bpfman-bytecode/go-tracepoint-counter
    • quay.io/bpfman-bytecode/go-uprobe-counter
    • quay.io/bpfman-bytecode/go-xdp-counter
    • quay.io/bpfman-bytecode/kprobe
    • quay.io/bpfman-bytecode/kretprobe
    • quay.io/bpfman-bytecode/tc-pass
    • quay.io/bpfman-bytecode/tcx-test
    • quay.io/bpfman-bytecode/tracepoint
    • quay.io/bpfman-bytecode/uprobe
    • quay.io/bpfman-bytecode/uretprobe
    • quay.io/bpfman-bytecode/xdp-pass-private
    • quay.io/bpfman-bytecode/xdp-pass
  • The relevant example userspace container images with tag <RELEASE_VERSION> from source code located in the bpfman project:
    • quay.io/bpfman-userspace/go-app-counter
    • quay.io/bpfman-userspace/go-kprobe-counter
    • quay.io/bpfman-userspace/go-target
    • quay.io/bpfman-userspace/go-tc-counter
    • quay.io/bpfman-userspace/go-tcx-counter
    • quay.io/bpfman-userspace/go-tracepoint-counter
    • quay.io/bpfman-userspace/go-uprobe-counter
    • quay.io/bpfman-userspace/go-xdp-counter
  • The OLM (Operator Lifecycle Manager) for the Kubernetes Operator.
    • This includes a bundle directory on disk as well as the quay.io/bpfman/bpfman-operator-bundle image with the tag <RELEASE_VERSION>.

Versioning strategy

Release Version Number

bpfman uses the MAJOR.MINOR.PATCH scheme defined by SemVer for version numbers in which the components are defined as follows:

  • MAJOR: Incremented for incompatible API changes.
  • MINOR: Incremented for adding functionality in a backward-compatible manner.
  • PATCH: Incremented for backward-compatible bug fixes.

Major version zero (0.y.z) is for initial development. If the MAJOR version is 0, anything MAY change at any time, and the public API SHOULD NOT be considered stable.

Releases are tagged in git with the version number prefixed by "v". For example, release version 0.5.2 is tagged as v0.5.2.

Kubernetes API Versions (e.g. v1alpha2, v1beta1)

Within the bpfman-operator, API versions are primarily used to indicate the stability of a resource. For example, if a resource has not yet graduated to beta, it is still possible that it could either be removed from the API or changed in backward incompatible ways. For more information on API versions, refer to the Kubernetes API versioning documentation.

Releasing a new version

Release Process Overview

Since bpfman and bpfman-operator are maintained in separate repositories, each requires an independent release. However, to ensure version consistency, we plan to synchronize the release versions of both projects. Therefore, whenever a release is needed for either bpfman or bpfman-operator, both repositories will be released with the same version number.

As bpfman-operator depends on bpfman, it is essential to release bpfman first, followed by bpfman-operator.

Whenever possible, releases are made on the main branch of each repository and should follow the Standard Release from Main Branch process. However, it is sometimes necessary to "patch" a previous release with some but not all of the changes that exist on the main branch. In those cases, a patch branch is created from the tag of the release being patched and the release is done on that branch as described in the Patch Branch Release section. Finally, if it is necessary to test the release automation, the simplified process described in the Release Candidate Release section can be used.

Generating Release Notes

The release notes are contained in CHANGELOG files stored in the changelogs directory of each repository. The change log name must contain the release version (e.g., CHANGELOG-v0.5.2.md).

To simplify the generation of the release notes details, we are using the GitHub release page as described below. Note that we only use the release page to generate a starting point for the release notes, and don't actually create a tag or do a release from it.

  1. Go to the bpfman releases page.
  2. Push the "Draft a new release" button.
  3. Enter the new release number in the "Choose a tag" pull-down.
  4. Choose the most recent release in the "Previous tag" pull-down.
  5. Push the "Generate release notes" button.

The automatically generated output will likely need to be reorganized and cleaned up a bit, but it provides a good starting point.

The format for the CHANGELOG file is as follows:

  1. Summary of the major changes and highlights. For example: "The v0.5.2 release is a patch release that introduced..."
  2. What's Changed (minor changes may be removed from the list generated by GitHub)
  3. Full Changelog
  4. New Contributors
  5. Known Issues

Notes on generating the changelog

  • Empty sections should be omitted.
  • Sections 2-3 may be copied and pasted from the text generated with the GitHub releases page process described above.
  • The CHANGELOG for a given release is used by GitHub to generate the initial content for that release on the bpfman releases page. However, after the release has been generated, updates to the CHANGELOG file are not automatically reflected on the GitHub releases page, so the GitHub releases page must be manually edited using the GitHub GUI.
  • Unlike most markdown, the generated output on the GitHub releases page renders each newline in the CHANGELOG file. So each paragraph should be on a single line, or it will not flow as intended.

Standard Release from Main Branch

This section describes the standard release process used when making a release from the main branch and may be used for major, minor, or patch releases. As mentioned above, we first complete the release for bpfman and then follow that up with a release for bpfman-operator.

bpfman Release

  • Create a new branch in your bpfman fork, for example <githubuser>/release-x.y.z, and use the new branch in the upcoming steps.
  • Make the following changes
    • Add a new changelog for the release using the process described in Generating Release Notes.
    • Update the Cargo.toml version for the workspace:
      • version = "x.y.z"
      • bpfman = { version = "x.y.z", path = "./bpfman" }"
      • bpfman-api = { version = "x.y.z", path = "./bpfman-api" }
      • Note: bpfman-csi does not need to be updated.
    • Run cargo generate-lockfile
    • Update the bpfman version in the bpfman/examples/Makefile:
      • VERSION ?= x.y.z
    • Add a new bpfman/examples/config/v0.x.y/ and bpfman/examples/config/v0.x.y-selinux/ config directory for the release version by copying the latest release directory and running a search for the current release and replace with the new release.
    • Add new example config directories for any new examples added since the last release.
    • Update dispatcher tags.
    • Search the code and docs for the current version number without the "v" (e.g., 0.5.1) and replace it with the new version number where it makes sense. (Be careful, though, because not all should be replaced.)
  • Commit the changes, push them to your repo, and open a PR against the bpfman repo.
  • After the PR is reviewed, merged, and all GitHub actions have completed successfully, tag the release with the version number (e.g., v0.5.2).
    • Tag the release using the commit on main where the changelog update merged.
    • A maintainer or someone with write permission on the repo must create the tag.
    • This can be done using the git CLI or Github's release page.
  • The Release will be automatically created by GitHub actions when the tag is applied.

After these steps are completed, the following should occur: - All GitHub actions should complete successfully. - The release appears on the GitHub Releases Page. - Images are built and updated with the new version tag at: - quay.io/bpfman - quay.io/bpfman-bytecode - quay.io/bpfman-userspace - The new version appears at crates.io - New RPMs are built and pushed to the bpfman COPR repository.

After the release is complete do the following:

  • Run make build-release-yamls from the bpfman/examples directory, and then add the yaml files generated to the release as assets from the GitHub release page.
    • The yaml files generated include:
      • bpfman-crds-install.yaml
      • bpfman-operator-install.yaml
      • go-app-counter-install-selinux.yaml
      • go-app-counter-install.yaml
      • go-kprobe-counter-install-selinux.yaml
      • go-kprobe-counter-install.yaml
      • go-tc-counter-install-selinux.yaml
      • go-tc-counter-install.yaml
      • go-tcx-counter-install-selinux.yaml
      • go-tcx-counter-install.yaml
      • go-tracepoint-counter-install-selinux.yaml
      • go-tracepoint-counter-install.yaml
      • go-uprobe-counter-install-selinux.yaml
      • go-uprobe-counter-install.yaml
      • go-uretprobe-counter-install-selinux.yaml
      • go-uretprobe-counter-install.yaml
      • go-xdp-counter-install-selinux.yaml
      • go-xdp-counter-install.yaml
  • Do another PR that changes the tag for XDP_DISPATCHER_IMAGE and TC_DISPATCHER_IMAGE in bpfman/src/lib.rs back to latest.

bpfman-operator Release

  • Create a new branch in your bpfman-operator fork, for example <githubuser>/release-x.y.z, and use the new branch in the upcoming steps.
  • Make the following changes
    • Add a new changelog for the release using the process described in Generating Release Notes.
    • Update the bpfman version in go.mod
    • Run the following commands from the bpfman-operator directory: 
      go mod vendor
go mod tidy
    • Update the bpfman-operator version in the Makefile:
      • VERSION ?= x.y.z
    • Run make bundle from the bpfman-operator directory to update the bundle version.
    • Update the version in the links in README.md
    • Update the version in the OpenShift Containerfiles.
  • Commit the changes, push them to your repo, and open a PR against the bpfman-operator repo.
  • After the PR is reviewed, merged, and all GitHub actions have completed successfully, tag the release with the version number (e.g., v0.5.4).
    • Tag the release using the commit on main where the changelog update merged.
    • A maintainer or someone with write permission on the repo must create the tag.
    • This can be done using the git CLI or Github's release page.
  • The Release will be automatically created by GitHub actions when the tag is applied.

After these steps are completed, the following should occur:

  • All GitHub actions should complete successfully.
  • The release appears on the GitHub Releases Page.
  • Images are built and updated with the new version tag at:
    • quay.io/bpfman/bpfman-operator
    • quay.io/bpfman/bpfman-agent

After the release completes:

  • Update the community-operator and community-operators-prod repositories with the latest bundle manifests.
    • Run IMAGE_TAG=vx.y.z make bundle from bpfman-operator.
    • Manually update the following tags in bundle/manifests/bpfman-operator.clusterserviceversion.yaml (TODO: automate this step).
      • Change :latest to :vx.y.z on for the example image URLs.
      • Change "containerImage: quay.io/bpfman/bpfman-operator:latest" to "containerImage: quay.io/bpfman/bpfman-operator:vx.y.z".
    • Open a PR in each of the community operator repos with the following:
      • Copy bpfman-operator/{manifests, metadata, tests} to the new release directory. Copy bpfman-operator/Containerfile.bundle.openshift to the new release directory.
      • Create a new release directory under operator/bpfman-operator/ in each repo named x.y.z
    • Lessons learned about updating the community operators:
      • These PRs usually auto-merge as soon as all checks pass, and once a bundle for a release is merged, it cannot be modified. If any errors are found in the bundle files after merging, the only solution is to create a new release and open a new PR in each community operator repository.
      • If you start a PR in the community-operator repository as a draft and later mark it as ready for review, it will still auto-merge. However, this auto-merge behavior doesn’t apply in the community-operators-prod repository, where a maintainer must manually merge the PR if you start it as draft.
      • To streamline the process, it’s recommended that you begin with a draft PR in the community-operator repository to allow for review. Once the PR is reviewed and all checks pass, mark it as ready for review. After it merges, submit a PR with the same bundle to the community-operators-prod repository.

Patch Branch Release

The patch branch release process is essentially the same as that for the standard release with the following exceptions.

Do the following for each repo:

  • If this is the first patch release for a given release, someone with write permissions on the repo (e.g., one of the maintainers) must create a branch from the git tag of the release you want to patch.
    • If patching vx.y.z, the patch branch should be named release-vx.y.z-patch.
  • Create a branch for your changes from the upstream branch.
  • Cherry pick the relevant commits.
  • Do other fixups if necessary.

Then, follow the steps from Standard Release from Main Branch section, except open your PRs against the release branch.

Release Candidate Release

Often times cutting a release candidate is a great way to test any changes to our release infrastructure before cutting an official release. Make sure release candidate versions contain an rc suffix (e.g., 0.4.0-rc1). This is a lighter-weight process meaning many of the versioned manifests do not necessarily need to be created.

As in the other releases, first complete the release for bpfman and then follow that up with a release for bpfman-operator.

bpfman Release

  • Create a new branch in your bpfman fork based on the upstream patch branch named, for example <githubuser>/release-x.y.z-rc1, and use the new branch in the upcoming steps.
  • Make the following changes
    • Add a new changelog for the release. A full set of release notes is not required. A single line that says something like "Pre-release 1 for v0.5.2" is sufficient.
    • Update the Cargo.toml version for the workspace:
      • version = "x.y.z-rc1"
      • bpfman = { version = "x.y.z-rc1", path = "./bpfman" }"
      • bpfman-api = { version = "x.y.z-rc1", path = "./bpfman-api" }
      • Note: bpfman-csi does not need to be updated.
    • Run cargo generate-lockfile
  • Commit the changes, push them to your repo, and open a PR against the bpfman repo.
  • After the PR is reviewed, merged, and all GitHub actions have completed successfully, tag the release with the version number (e.g., vx.y.z-rc1).
    • Tag the release using the commit on main where the changelog update merged.
    • A maintainer or someone with write permission on the repo must create the tag.
    • This can be done using the git CLI or Github's release page.
  • The Release will be automatically created by GitHub actions when the tag is applied.

After these steps are completed, the following should occur:

  • All GitHub actions should complete successfully.
  • The release appears on the GitHub Releases Page.
  • Images are built and updated with the new version tag at:
    • quay.io/bpfman
    • quay.io/bpfman-bytecode
    • quay.io/bpfman-userspace
  • The new version appears at crates.io
  • A new RPM is built and pushed to the bpfman COPR repository.

bpfman-operator Release

  • Create a new branch in your bpfman fork based on the upstream patch branch named, for example <githubuser>/release-x.y.z-rc1, and use the new branch in the upcoming steps.
  • Make the following changes
    • Add a new changelog for the release. A full set of release notes is not required. A single line that says something like "Pre-release 1 for v0.5.2" is sufficient.
    • Update the bpfman-operator version in the Makefile:
      • VERSION ?= x.y.z-rc1
  • Commit the changes, push them to your repo, and open a PR against the bpfman-operator repo.
  • After the PR is reviewed, merged, and all GitHub actions have completed successfully, tag the release with the version number (e.g., vx.y.z-rc1).
    • Tag the release using the commit on main where the changelog update merged.
    • A maintainer or someone with write permission on the repo must create the tag.
    • This can be done using the git CLI or Github's release page.
  • The Release will be automatically created by GitHub actions when the tag is applied.

After these steps are completed, the following should occur:

  • All GitHub actions should complete successfully.
  • The release appears on the GitHub Releases Page.
  • Images are built and updated with the new version tag at:
    • quay.io/bpfman/bpfman-operator
    • quay.io/bpfman/bpfman-agent