How to set up BCC for eBPF on MacOS

January 2021

I recently began exploring eBPF, starting with Brendan Gregg’s BPF Performance Tools: Linux System and Application Observability. One of the first requirements is to install BCC, and my first instinct was to use Docker [1].

Doing this with Docker proved to be difficult, and I could barely find any articles that talked about installing BCC on an OS X host. Most of the content assumes Ubuntu or some other Linux host.

I did find one article by Peter Malmgren that explains how to do this using Docker, which was really helpful [2]. However, it fell short on running some BCC tools and I had to change some stuff to get it working.

If you’re only interested in doing the install, I suggest you skip to the “Using Vagrant” section.

Using Docker

eBPF tools such as BCC and bpftrace rely on Kernel headers which don’t ship with the Docker Desktop for macOS VM. This means the regular install command (apt-get install -y bpfcc-tools "linux-headers-$(uname -r)") won’t work, because there are no linux-headers [3].

Hence I needed to find a way to compile and install them. The steps below are mostly the same as Peter’s, except I added a start script and had to create volumes to /lib/modules/$(uname -r)/source and /lib/modules/$(uname -r)/build. In his case, symlinks were created automatically from /usr/src/$(uname -r) to those directories but that was not my reality.

You can get the files I mention below in the ebf-playground repo.

Docker Problems

With this setup, most of the tools used for the 60-second analysis worked. However, it fell short on ext4 commands. If you run ext4slower, it will fail with this error below;

ERROR: no ext4_file_operations in /proc/kallsyms. Exiting.
HINT: the kernel should be built with CONFIG_KALLSYMS_ALL.

I added the config CONFIG_KALLSYMS_ALL to .config in the start script, but this error kept happening. After a while and some research, I gave up and went with the Vagrant option.

Using Vagrant also makes sense because of all the hacking done to get Docker working. The best solutions are simple and sweet.

Using Vagrant

Vagrant is a tool for building and managing virtual machine environments in a single workflow. The difference is that the kernels in the Vagrant environments don’t use LinuxKit, meaning we can install bcc normally (apt-get install -y bpfcc-tools "linux-headers-$(uname -r)) without compiling headers and changing configs.

This is my recommended approach, as it is easier to get up and running.

Next Steps

When testing these tools, there needs to be some activity on the Vagrant environment such as open(2) syscalls and new processes being executed, which I can trace with opensnoop and execsnoop respectively.

I’m thinking about running an Nginx server and simulating load so there is activity that can be traced. Before doing that, I will need to understand how Nginx interacts with the system to know what sort of calls to expect.

Notes

[1] Whenever I test out new tools I like to use Docker containers because they are reusable and won’t require installing potentially bulky software on my system.

[2] His article explains the traits of Docker for Mac that affect the way BCC can be installed, so I suggest you take a look if you’re curious. Thanks, Peter!

[3] If you try, you get an error similar to the one below;

bash: linux-headers-4.9.125-linuxkit: command not found

[4] Since it’s heavy (1GB or so), it’s wise not to add it to the build. We are ensuring we only have to download it once and make it available in the container using volumes.

Hi! My name is Opeyemi. I like distributed systems, NodeJS, Golang and Puff Puff. You can learn more about me or message me on Twitter.

Share on