Fuzzing the FreeBSD Kernel with Syzkaller and Nested Virtualization on a Linux Host

During my internal research time I decided to dive into kernel fuzzing. In particular, my goal was to set up Syzkaller in a first step and extend the system call descriptions in a second step to increase coverage and find new bugs. As Syzkaller is fuzzing the Linux kernel quite extensively, I hoped that fuzzing the FreeBSD kernel could maybe find some low hanging fruits.

However, as my main operating system is Ubuntu, I encountered some annoying issues. Therefore, I had the idea to run a FreeBSD host in a virtual machine and use this VM as a host to spawn nested FreeBSD machines which are fuzzed by Syzkaller. Unfortunately, this was not as easy as I thought at first. I tried different combinations of VirtualBox, VMWare, qemu and bhyve for the nested virtualization, many of which failed and took me a lot of time. Hence, I want to describe in this blog post how to set this up to give the community something back and hopefully make the life of someone else easier.

Setup

To distinguish commands, which are executed on the Linux host and the FreeBSD VM, commands on the Linux host and the FreeBSD VM start with $ and %, respectively.

Setting Up the FreeBSD Host VM

Create a directory where you put the FreeBSD host image into, download and decompress it:

$ mkdir ~/fuzzing_freebsd
$ cd ~/fuzzing_freebsd
$ wget https://download.freebsd.org/releases/VM-IMAGES/14.0-RELEASE/amd64/Latest/FreeBSD-14.0-RELEASE-amd64-zfs.qcow2.xz
$ xz --decompress FreeBSD-14.0-RELEASE-amd64-zfs.qcow2.xz

Please make sure to download an image that is using zfs as a file system, as this is required at a later point to setup the bhyve machine.

Before you run the VM, increase its size:

$ qemu-img resize FreeBSD-14.0-RELEASE-amd64-zfs.qcow2 +25G

Now, start this VM with qemu and set everything up that is needed. First, run the VM as follows and adjust the number of cores and amount of RAM to your preferences:

$ qemu-system-x86_64 -m 16G -smp 12 -drive file=FreeBSD-14.0-RELEASE-amd64-zfs.qcow2,format=qcow2 -enable-kvm -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic,model=e1000 -cpu host

Log in as root (no password is required) and make sure that the zpool has grown by +25G. Otherwise run the following commmands to make sure that the zpool grows:

% pkg install gpart
% gpart show
[in my case the drive I want to increase is: ada0]
% gpart recover ada0
% gpart resize -i 4 /dev/ada0 [take the index of freebsd-zfs]
% zpool set autoexpand=on zroot
% zpool online -e zroot ada0

If the zpool has not grown, please reboot.

SSH Configuration on The Host VM

To configure ssh, first edit /etc/ssh/sshd_config and set

PermitRootLogin yes

Additionally, run

% sysrc sshd_enable=YES
% sysrc ifconfig_DEFAULT=DHCP
% /etc/rc.d/sshd start

Give the current user root a password (I just chose root as password):

% passwd

Now, check if you can connect to the VM via

$ ssh -p 10022 root@localhost

Checking Out and Building Syzkaller

Before you download and configure Syzkaller, install the following required dependencies on the host VM:

$ ssh -p 10022 root@localhost

Checking out and Building Syzkaller

Before you download and configure Syzkaller, install the following required dependencies on the host VM:

% pkg install bash gcc git gmake go golangci-lint llvm

Due to the nested VM setup, bhyve will be used as the VM backend. Hence, a DHCP server has to be installed, too:

% pkg install dnsmasq

Next, checkout the Syzkaller sources:

% mkdir /root/fuzzing
% git clone https://github.com/google/syzkaller /root/fuzzing/syzkaller

Check if you can build the binaries:

% cd /root/fuzzing/syzkaller
% gmake manager fuzzer execprog executor TARGETOS=freebsd

Setting up the Nested FreeBSD VM

To set up a nested FreeBSD VM inside our FreeBSD host VM, download an image of the latest FreeBSD version. This time download a raw image instead of a qcow image as the raw image will managed by bhyve:

% pkg install wget
% cd /root
% wget https://download.freebsd.org/snapshots/VM-IMAGES/15.0-CURRENT/amd64/Latest/FreeBSD-15.0-CURRENT-amd64.raw.xz
% xz --decompress FreeBSD-15.0-CURRENT-amd64.raw.xz

To use bhyve on the FreeBSD host VM, some additional steps are required. First execute:

% zfs create -o mountpoint=/syzkaller zroot/syzkaller
% mv FreeBSD-15.0-CURRENT-amd64.raw /syzkaller

Furthermore, configure networking and DHCP for the nested VM:

% ifconfig bridge create bridge0
% ifconfig bridge0 inet 169.254.0.1
% echo 'dhcp-range=169.254.0.2,169.254.0.254,255.255.255.0' > /usr/local/etc/dnsmasq.conf
% echo 'interface=bridge0' >> /usr/local/etc/dnsmasq.conf
% sysrc dnsmasq_enable=YES
% service dnsmasq start
% echo 'net.link.tap.up_on_open=1' >> /etc/sysctl.conf
% sysctl net.link.tap.up_on_open=1

Add the following lines to /etc/rc.conf such that the configuration of bridged networking is enabled every time the system boots:

cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="inet 169.254.0.1 addm tap0 up"
ifconfig_tap0="up"

Before the VM is started by bhyve, check that the bhyve kernel module is really loaded:

% kldload vmm

Next, start the nested VM from the host VM with bhyve. First, create a new tap interface:

% ifconfig tap create

In my case, tap0 was created. Now, add this interface to the bridge:

% ifconfig bridge0 addm tap0

Finally, run

% bhyveload -c stdio -m 512M -d /syzkaller/FreeBSD-15.0-CURRENT-amd64.raw -e autoboot_delay=0 testvm0
% bhyve -H -A -P -c 1 -m 512M -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,/syzkaller/FreeBSD-15.0-CURRENT-amd64.raw -l com1,stdio testvm0

and log in as root (no password required).

SSH Configuration on the Nested VM

As Syzkaller has to be able to connect to the nested VM via SSH and a SSH key, configure SSH and first edit /etc/ssh/sshd_config to set

PermitRootLogin yes

Additionally, run

% sysrc sshd_enable=YES
% sysrc ifconfig_DEFAULT=DHCP
% /etc/rc.d/sshd start

Give the current user root a password (I just chose root as password):

% passwd

Find out the ip address of the nested VM by running ifconfig on the nested VM for example. In my case the ip address of the nested VM is 169.254.0.46. Now, try to connect to the nested VM via SSH by running:

% ssh root@169.254.0.46

Next, create a ssh key pair on the host VM and add the public key to the running VM:

% mkdir -p /root/fuzzing/files
% cd /root/fuzzing/files
% ssh-keygen -f root_key_nested.id_rsa -t rsa -N ''
% ssh-copy-id -i ./root_key_nested.id_rsa.pub root@169.254.0.46

Now, you should be able to connect to the nested VM by executing the following command:

% ssh -i /root/fuzzing/files/root_key_nested.id_rsa root@169.254.0.46

Downloading, Building And Installing Kernel Sources

The next step is to check out the FreeBSD kernel sources:

% git clone --depth=1 --branch=main https://github.com/freebsd/freebsd-src /usr/src

Create a custom kernel config for Syzkaller and build the kernel:

% cd /usr/src/sys/amd64/conf
% cat <<__EOF__ > SYZKALLER
include "./GENERIC"

ident	SYZKALLER

options 	COVERAGE
options 	KCOV
__EOF__
% cd /usr/src
% make -j $(sysctl -n hw.ncpu) KERNCONF=SYZKALLER buildkernel

To install the built kernel on the nested VM, run the following commands on the host VM (shut the nested VM down before installing the new kernel):

% mdconfig -a -f /syzkaller/FreeBSD-15.0-CURRENT-amd64.raw
% mount /dev/md0p4 /mnt
% cd /usr/src
% make KERNCONF=SYZKALLER installkernel DESTDIR=/mnt
% umount /mnt
% mdconfig -d -u 0

Start the nested VM and make sure that it is running correctly:

% bhyveload -c stdio -m 512M -d /syzkaller/FreeBSD-15.0-CURRENT-amd64.raw -e autoboot_delay=0 testvm0
% bhyve -H -A -P -c 1 -m 512M -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,/syzkaller/FreeBSD-15.0-CURRENT-amd64.raw -l com1,stdio testvm0

Running Syzkaller

Now that we have a working setup, it’s time for actually running Syzkaller.

Configuring Syzkaller

Create a config file for Syzkaller that determines how many VMs should be spawned, and that contains the information where to find the kernel sources and object files. Moreover, you can tell Syzkaller by this config file how to spawn the nested VMs with bhyve and how to connect via SSH using the private SSH key. Create the file /root/fuzzing/files/freebsd.cfg with the following contents:

{
  "name": "freebsd",
  "target": "freebsd/amd64",
  "http": "127.0.0.1:56741",
  "workdir": "/root/fuzzing/syzkaller/workdir",
  "syzkaller": "/root/fuzzing/syzkaller/",
  "kernel_obj": "/usr/obj/usr/src/amd64.amd64/sys/SYZKALLER",
  "kernel_src": "/",
  "sshkey": "/root/fuzzing/files/root_key_nested.id_rsa",
  "sandbox": "none",
  "procs": 12,
  "image": "/syzkaller/FreeBSD-15.0-CURRENT-amd64.raw",
  "type": "bhyve",
  "vm": {
    "count": 4,
    "cpu": 4,
    "mem": "4048M",
    "bridge": "bridge0",
    "hostip": "169.254.0.1",
    "dataset": "zroot/syzkaller"
  }
}

Furthermore, this config file contains the path to the workdir of Syzkaller which has to be created:

% mkdir /root/fuzzing/syzkaller/workdir

Running Syzkaller

Syzkaller presents the current fuzzing results really nicely by spawning a webserver. However, this webserver is running on our host VM and not on our host OS (Ubuntu). Hence, we forward the port 56741 on which the spawned webserver is running (see the entry http in /root/fuzzing/files/freebsd.cfg) by the following command on our main OS:

$ ssh -N -L56741:127.0.0.1:56741 -p 10022 root@localhost

Finally, everything is ready to go! On the nested VM, execute the following commands:

% cd /root/fuzzing/syzkaller
% bin/syz-manager -config ../files/freebsd.cfg

Yeahhh, enjoy viewing the results by visiting http://127.0.0.1:56741 with a browser on your Linux host.

Fuzzing the Bluetooth Stack

Now that the setup is ready to run Syzkaller to fuzz nested FreeBSD VMs on a FreeBSD host VM hosted by a Linux main OS, I could finally try to add some system call descriptions to find uncovered bugs. While analyzing which system calls have already been described I discovered that the description of the socket system calls for the bluetooth stack were missing. Hence, I decided to give this a try.

Adding System Call Descriptions for the Bluetooth Stack

I do not want to go into detail here how the syzlang works as this has already been done several time, for example here.

To include new system call descriptions for the bluetooth stack, I added the file syzkaller/freebsd/socket_bluetooth.txt with the following contents:

include <sys/types.h>
include <sys/socket.h>
include <sys/sockio.h>
include <sys/bitstring.h>
include <netinet/in.h>
include <netgraph/bluetooth/include/ng_hci.h>
include <netgraph/bluetooth/include/ng_l2cap.h>
include <netgraph/bluetooth/include/ng_btsocket.h>

resource sock_bt[sock]
resource sock_bt_hci[sock_bt]
resource sock_bt_sco[sock_bt]
resource sock_bt_rfcomm[sock_bt]
resource sock_bt_l2cap[sock_bt]

socket$bt_hci(domain const[PF_BLUETOOTH], type const[SOCK_RAW], protocol const[BLUETOOTH_PROTO_HCI]) sock_bt_hci
bind$bt_hci(s sock_bt_hci, addr ptr[in, sockaddr_hci], addrlen len[addr])
ioctl$sock_bt_hci(fd sock_bt_hci, cmd flags[bt_hci_ioctl], arg buffer[inout])
setsockopt$bt_hci_HCI_FILTER(s sock_bt_hci, level const[SOL_HCI_RAW], optname const[SO_HCI_RAW_FILTER], optval ptr[in, ng_btsocket_hci_raw_filter], optlen len[optval])
setsockopt$bt_hci_HCI_DIRECTION(s sock_bt_hci, level const[SOL_HCI_RAW], optname const[SO_HCI_RAW_DIRECTION], optval ptr[in, int32], optlen len[optval])
getsockopt$bt_hci(s sock_bt_hci, level const[SOL_HCI_RAW], optname flags[bt_hci_sockopt], optval buffer[out], optlen ptr[inout, len[optval, int32]])
write$bt_hci(fd sock_bt_hci, buf buffer[in], nbytes bytesize[buf])

socket$bt_sco(domain const[PF_BLUETOOTH], type const[SOCK_SEQPACKET], protocol const[BLUETOOTH_PROTO_SCO]) sock_bt_sco
bind$bt_sco_sockaddr_sco(s sock_bt_sco, addr ptr[in, sockaddr_sco], addrlen len[addr])
connect$bt_sco_sockaddr_sco(s sock_bt_sco, name ptr[in, sockaddr_sco], namelen len[name])
getsockopt$bt_sco_SOL_SCO(s sock_bt_sco, level const[SOL_SCO], optname flags[bt_sol_sockopt], optval buffer[out], optlen ptr[inout, len[optval, int32]])

socket$bt_l2cap(domain const[PF_BLUETOOTH], type flags[bt_l2cap_type], protocol const[BLUETOOTH_PROTO_L2CAP]) sock_bt_l2cap
bind$bt_l2cap(fd sock_bt_l2cap, addr ptr[in, sockaddr_l2], addrlen len[addr])
connect$bt_l2cap(fd sock_bt_l2cap, addr ptr[in, sockaddr_l2], addrlen len[addr])
setsockopt$bt_l2cap(fd sock_bt_l2cap, level const[SOL_L2CAP], optname flags[bt_l2cap_sockopt], optval ptr[in, int32], optlen len[optval])
getsockopt$bt_l2cap(s sock_bt_l2cap, level const[SOL_L2CAP], optname flags[bt_l2cap_sockopt], optval buffer[out], optlen ptr[inout, len[optval, int32]])

socket$bt_rfcomm(domain const[PF_BLUETOOTH], type const[SOCK_STREAM], protocol const[BLUETOOTH_PROTO_RFCOMM]) sock_bt_rfcomm
bind$bt_rfcomm(fd sock_bt_rfcomm, addr ptr[in, sockaddr_rfcomm], addrlen len[addr])
connect$bt_rfcomm(fd sock_bt_rfcomm, addr ptr[in, sockaddr_rfcomm], addrlen len[addr])
setsockopt$bt_rfcomm(fd sock_bt_rfcomm, level const[SOL_RFCOMM], optname flags[bt_rfcomm_sockopt], optval ptr[in, int32], optlen len[optval])
getsockopt$bt_rfcomm(s sock_bt_rfcomm, level const[SOL_RFCOMM], optname flags[bt_rfcomm_sockopt], optval buffer[out], optlen ptr[inout, len[optval, int32]])

ng_btsocket_hci_raw_filter {
	packet_mask	int64[32]
	event_mask	int64[64]
}

sockaddr_hci {
	hci_len		int8
	hci_family	int8
	hci_node	int8[32]
}

sockaddr_sco {
	sco_len		int8
	sco_family	int8
	sco_bdaddr	int8[6]
}

sockaddr_l2 {
	l2cap_len		int8
	l2cap_familty		int8
	l2cap_psm		int16
	l2cap_bdaddr		int8[6]
	l2cap_cid		int16
	l2cap_bdaddr_type	int8
}

sockaddr_rfcomm {
	rfcomm_len	int8
	rfcomm_family	int8
	rfcomm_bdaddr	int8[6]
	rfcomm_channel	int8
}

bt_hci_ioctl = SIOC_HCI_RAW_NODE_GET_STATE, SIOC_HCI_RAW_NODE_INIT, SIOC_HCI_RAW_NODE_GET_DEBUG, SIOC_HCI_RAW_NODE_SET_DEBUG, SIOC_HCI_RAW_NODE_GET_BUFFER, SIOC_HCI_RAW_NODE_GET_BDADDR, SIOC_HCI_RAW_NODE_GET_FEATURES, SIOC_HCI_RAW_NODE_GET_STAT, SIOC_HCI_RAW_NODE_RESET_STAT, SIOC_HCI_RAW_NODE_FLUSH_NEIGHBOR_CACHE, SIOC_HCI_RAW_NODE_GET_NEIGHBOR_CACHE, SIOC_HCI_RAW_NODE_GET_CON_LIST, SIOC_HCI_RAW_NODE_GET_LINK_POLICY_MASK, SIOC_HCI_RAW_NODE_SET_LINK_POLICY_MASK, SIOC_HCI_RAW_NODE_GET_PACKET_MASK, SIOC_HCI_RAW_NODE_SET_PACKET_MASK, SIOC_HCI_RAW_NODE_GET_ROLE_SWITCH, SIOC_HCI_RAW_NODE_SET_ROLE_SWITCH
bt_hci_sockopt = SO_HCI_RAW_FILTER, SO_HCI_RAW_DIRECTION
bt_sol_sockopt = SO_SCO_MTU, SO_SCO_CONNINFO
bt_l2cap_sockopt = SO_L2CAP_IMTU, SO_L2CAP_OMTU, SO_L2CAP_IFLOW, SO_L2CAP_OFLOW, SO_L2CAP_FLUSH, SO_L2CAP_ENCRYPTED
bt_l2cap_type = SOCK_SEQPACKET, SOCK_RAW
bt_rfcomm_sockopt = SO_RFCOMM_MTU, SO_RFCOMM_FC_INFO

Extracting, Generating and Rebuilding Binaries

After adding the new system call descriptions, you have to first run a make extract command on the host VM to extract information based on the new file socket_bluetooth.txt:

% cd /root/fuzzing/syzkaller
% gmake extract ARCH=amd64 TARGETOS=freebsd SOURCEDIR=/usr/src/

The new file syzkaller/freebsd/sys/socket_bluetooth.txt.const should have been created now. Next, execute a generate command to update the generated code:

% cd /root/fuzzing/syzkaller
% gmake generate

To rebuild the binaries, run a gmake command:

% cd /root/fuzzing/syzkaller
% gmake manager fuzzer execprog executor TARGETOS=freebsd

Rebuilding the Kernel

Unfortunately, our current kernel build does not support bluetooth. Therefore, edit the file /usr/src/sys/amd64/conf/SYZKALLER and rebuild/install the kernel afterward:

include "./GENERIC"

ident SYZKALLER

options COVERAGE
options KCOV
options KASAN

options NETGRAPH
options NETGRAPH_DEBUG
options NETGRAPH_ASYNC
options NETGRAPH_BLUETOOTH
options NETGRAPH_BLUETOOTH_BT3C
options NETGRAPH_BLUETOOTH_H4
options NETGRAPH_BLUETOOTH_HCI
options NETGRAPH_BLUETOOTH_L2CAP
options NETGRAPH_BLUETOOTH_SOCKET
options NETGRAPH_BLUETOOTH_UBT
options NETGRAPH_BLUETOOTH_UBTBCMFW
options NETGRAPH_BPF
options NETGRAPH_BRIDGE
options NETGRAPH_CAR
options NETGRAPH_CHECKSUM
options NETGRAPH_CISCO
options NETGRAPH_DEFLATE
options NETGRAPH_DEVICE
options NETGRAPH_ECHO
options NETGRAPH_EIFACE
options NETGRAPH_ETHER
options NETGRAPH_ETHER_ECHO
options NETGRAPH_FEC
options NETGRAPH_FRAME_RELAY
options NETGRAPH_GIF
options NETGRAPH_GIF_DEMUX
options NETGRAPH_HOLE
options NETGRAPH_IFACE
options NETGRAPH_IP_INPUT
options NETGRAPH_IPFW
options NETGRAPH_KSOCKET
options NETGRAPH_L2TP
options NETGRAPH_LMI
options NETGRAPH_MPPC_COMPRESSION
options NETGRAPH_MPPC_ENCRYPTION
options NETGRAPH_NAT
options NETGRAPH_NETFLOW
options NETGRAPH_ONE2MANY
options NETGRAPH_PATCH
options NETGRAPH_PIPE
options NETGRAPH_PPP
options NETGRAPH_PPPOE
options NETGRAPH_PPTPGRE
options NETGRAPH_PRED1
options NETGRAPH_RFC1490
options NETGRAPH_SOCKET
options NETGRAPH_SPLIT
options NETGRAPH_SPPP
options NETGRAPH_TAG
options NETGRAPH_TCPMSS
options NETGRAPH_TEE
options NETGRAPH_TTY
options NETGRAPH_UI
options NETGRAPH_VJC
options NETGRAPH_VLAN

Now, rebuild the kernel on the host VM and install the kernel on the nested VMs (shut the nested VM down before the following steps):

% cd /usr/src
% make -j $(sysctl -n hw.ncpu) KERNCONF=SYZKALLER buildkernel
% mdconfig -a -f /syzkaller/FreeBSD-15.0-CURRENT-amd64.raw
% mount /dev/md0p4 /mnt
% cd /usr/src
% make KERNCONF=SYZKALLER installkernel DESTDIR=/mnt
% umount /mnt
% mdconfig -d -u 0

Check again if the nested VM is booting properly.

Finding New Bugs

Before you run Syzkaller, edit the config file /root/fuzzing/files/freebsd.cfg and specify which system calls should be fuzzed by the field enable_syscalls. For example, we can focus on socket$bt_hci and sendto (this is only a small subset of the new system call descriptions I added):

{
  "name": "freebsd",
  "target": "freebsd/amd64",
  "http": "127.0.0.1:56741",
  "workdir": "/root/fuzzing/syzkaller/workdir",
  "syzkaller": "/root/fuzzing/syzkaller/",
  "kernel_obj": "/usr/obj/usr/src/amd64.amd64/sys/SYZKALLER",
  "kernel_src": "/",
  "sshkey": "/root/fuzzing/files/root_key_nested.id_rsa",
  "sandbox": "none",
  "procs": 12,
  "image": "/zroot/syzkaller/FreeBSD-15.0-CURRENT-amd64.raw",
  "type": "bhyve",
  "vm": {
    "count": 4,
    "cpu": 4,
    "mem": "4048M",
    "bridge": "bridge0",
    "hostip": "169.254.0.1",
    "dataset": "zroot/syzkaller"
  },
	"enable_syscalls": [ "socket$bt_hci", "sendto" ]
}

Finally, start Syzkaller and find new bugs!

% cd /root/syzkaller
% bin/syz-manager -config ../files/freebsd.cfg

After a short time Syzkaller finds a RedZonePartial crash of the following form:

login: panic: ASan: Invalid access, 34-byte read at 0xfffffe0002044eb0, RedZonePartial(2)

After some more time Syzkaller was able to trigger a UMAUseAfterFree crash in ng_btsocket_hci_raw_send() which was caused by the same bug as the RedZonePartial crash before:

login: panic: ASan: Invalid access, 34-byte read at 0xfffffe00835ad0f0, UMAUseAfterFree(fd)
cpuid = 2
time = 1711976404
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0xc6/frame 0xfffffe00abe76210
kdb_backtrace() at kdb_backtrace+0xd0/frame 0xfffffe00abe76370
vpanic() at vpanic+0x26a/frame 0xfffffe00abe76530
panic() at panic+0xb5/frame 0xfffffe00abe76600
kasan_code_name() at kasan_code_name/frame 0xfffffe00abe766d0
kasan_memmove() at kasan_memmove+0x1c9/frame 0xfffffe00abe76710
ng_btsocket_hci_raw_send() at ng_btsocket_hci_raw_send+0x498/frame 0xfffffe00abe767d0
sosend_generic() at sosend_generic+0xd4a/frame 0xfffffe00abe769a0
sousrsend() at sousrsend+0x116/frame 0xfffffe00abe76a30
kern_sendit() at kern_sendit+0x5a2/frame 0xfffffe00abe76ba0
sendit() at sendit+0x157/frame 0xfffffe00abe76bf0
sys_sendto() at sys_sendto+0x181/frame 0xfffffe00abe76d10
amd64_syscall() at amd64_syscall+0x47a/frame 0xfffffe00abe76f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00abe76f30
--- syscall (198, FreeBSD ELF64, __syscall), rip = 0x2ac88a, rsp = 0x826567ef8, rbp = 0x826567f80 ---
KDB: enter: panic
[ thread pid 1086 tid 100349 ]
Stopped at      kdb_enter+0x6e: movq    $0,0x23c69a7(%rip)

As we are fuzzing with a FreeBSD host, Syzkaller was even able to find a C-reproducer! This would have not been possible in an automatic way if we were fuzzing the FreeBSD Kernel with a Linux host.

Moreover, I have reported the UMAUserAfterFree bug to the security team of FreeBSD. The method ng_btsocket_hci_raw_send() was not verifying that the sockaddr, which has been allocated in sendit(), is large enough. Therefore, copying the sockaddr into an mbuf in this line could trigger an out-of-bounds read.

Conclusion

It required a lot of time and effort to get this setup running which consists of a Linux host that runs a FreeBSD host VM which fuzzes nested FreeBSD VMs with Syzkaller. Therefore I hope that this blog post helps some people to get this setup working quickly. To make a contribution to the open source project Syzkaller I sent a pull request for my additions which have been merged, see here.