Menu







Debugging in Extreme Context through QEMU+Linux-KVM

2023-09-20 - Virtualization Technology

During recent research in developing NoirVisor, I find it exceedingly difficult to develop without using sufficient debugging facilities (i.e.: It is very unstable to print debug messages using DbgPrint routines provided by Windows Kernel). As such, certain measures must be taken in order to help debugging. QEMU provides a very neat peripheral device, called the ISA-DebugCon, dedicated for debugging purpose. This blog goes over how to setup QEMU+Linux-KVM for hypervisor development.

You will need to do the following preparations:

Create the Virtual Machine

This blog uses Debian as example. Installation of QEMU might differ among distributions but the way of running it should be the same.
First of all, install QEMU.

sudo apt install qemu-system ovmf

Then create a disk image for your virtual machine.

qemu-img create -f raw disk.img 40G

You may customize the size of your image. The command I provided will create a 40G image with raw format, meaning that you may analyze the disk by directly using binary viewer tools. You may also choose other formats if you want your disk image to be compatible for other virtual machine products (e.g.: specify vhdx for Hyper-V or vmdk for VMware) but you will have to decode their formats or even decrypt them in order to read the contents of the disk image.

Next, create a bash script file so that running QEMU will be easier. The base code of the script should be:

qemu-system-x86_64 -accel kvm -cpu host,hypervisor=off,svm=on -smp 2 -m 4096 -bios OVMF.fd -drive format=raw,file=disk.img -vga qxl -audio pa,model=hda -nic model=e1000e -device usb-ehci -device usb-mouse -device usb-tablet

At this point, the virtual machine is created successfully. Assuming the script file is saved to runvm.sh, you may execute the script to start it:

bash runvm.sh

Install Windows

This blog will not talk about the fool-proof method of installing Windows. It is reader’s responsibility to acquire the installation image and do the installation.

You will need to add your installation image to your guest by adding the following parameter:

-cdrom Windows-Installer.iso

After you installed Windows, install the SPICE-Guest Tools. This is similar to the VMware Tools so that the experience of using QEMU is significantly improved.

You may also follow Microsoft’s guide to setup KDNET.

Remote Access to QEMU

You can use the virt-viewer to remotely access VM in QEMU if your QEMU is running on a remote Linux host. You will have to append the following arguments to QEMU:

-spice addr=0.0.0.0,port=3001,disable-ticketing=on,disable-agent-file-xfer=off -device virtio-serial -chardev spicevmc,id=vdagent,debug=0,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -chardev spiceport,id=spicechannel1,name=org.spice-space.webdav.0 -device virtserialport,nr=2,chardev=spicechannel1,name=org.spice-space.webdav.0

Change the port=3001 part if you need to do so (e.g.: port is occupied by other software).

Also, change the audio part:

-audio spice,model=hda

Or otherwise the sound from your guest will come out from your Linux host’s machine, rather than your workstation.

Then run the virt-viewer you just installed on your workstation. Input spice://kvmhost:3001 to connect to your QEMU VM. If you are using a different port, adjust accordingly. After you installed the SPICE guest tools in QEMU VM, you will be able to drag’n’drop files into the VM. Unfortunately, all files go to the desktop, and you can’t drag files out of your VM. The clipboard also works.

For some strange reasons that I don’t know, Windows Update can sometimes kill the SPICE guest tools so that your mouse will disappear and keyboard won’t react. Therefore, you must disable Automatic Updates and never update your guest system.

Debug Your UEFI Program

After you compiled your UEFI program, you may pack it into a disk image. You may use GNU mtools to create a disk image. You may download my precompiled mtools for Windows and add them into PATH. But in Linux, you may install mtools directly. For example, you may use apt in Debian or Ubuntu:

sudo apt install mtools

Then create the disk images:

dd if=/dev/zero of=disk.img bs=1k count=1440
mformat -i disk.img -f 1440 ::
mmd -i disk.img ::/EFI
mmd -i disk.img ::/EFI/BOOT
mcopy -i disk.img bootx64.efi ::/EFI/BOOT

You may use mcopy command to copy more files into the disk image. Note that this image is a 1.44M floppy disk image. You may want to extend this size as situation requires.

If you are using Windows (not WSL), please replace dd command with fsutil:

fsutil file createNew disk.img 1474560
mformat -i disk.img -f 1440 ::
mmd -i disk.img ::/EFI
mmd -i disk.img ::/EFI/BOOT
mcopy -i disk.img bootx64.efi ::/EFI/BOOT

You may use any method to send your disk image to your remote Linux host (e.g.: the scp method). Then run your UEFI image:

qemu-system-x86_64 -accel kvm -bios OVMF.fd -drive format=raw,file=disk.img

You can add extra arguments as needed. For example, you may add extra arguments specified in the following chapters to help debugging.

The ISA-DebugCon Peripheral Device

As we mentioned, the reason we use QEMU is the ISA-DebugCon device it provides. The principle of this device is very simple: as you read from or write to this device, the request will be transferred to a specific stream device (e.g.: files, standard I/O, TCP connection, named pipe, etc.) It is even easier to use than the serial port (you don’t have to configure baud rate, parity, etc.) In addition, this device won’t occupied by the operating systems (in contrast, you can’t directly use I/O instructions to operate serial port after Windows is booted.)

To use ISA-DebugCon in QEMU, I recommend to append the following parameters:

-chardev socket,id=debugger,port=23456,host=0.0.0.0,server=on,telnet=on -device isa-debugcon,iobase=0x402,chardev=debugger

This will make the ISA-DebugCon using TCP connection through Telnet protocol and listen on 23456 port. You may use PuTTY on your development machine to see the output. As you start QEMU, it will hang in order to wait for the TCP connection to be established. After you connect PuTTY to QEMU, the VM will start booting. Considering that we are using 23456 port and Telnet protocol, you will have to specify them on PuTTY Session configurations. In addition, go to Terminal configurations and check on “Implicit CR in every LF” so that line-feeds will make the cursor jump to the start of the next line. It is also recommended to configure your Linux host with a static IP address on your router so that you can enter the session without repeating the configuration efforts. By the way, if you use the default firmware (SeaBIOS) of QEMU and the I/O address of it is on 0x402, you can view SeaBIOS’ debug output.

If you are using WSL2 but that’s not your development machine, you will have to configure network bridging since servers in WSL2 can only respond inside the Hyper-V network. Therefore, you don’t have to configure network bridging if you are using WSL2 in your development machine.

Using ISA-DebugCon

There is no standard method to detect this device. Therefore, you will have to save the I/O address somewhere else (it can be Registry or even hard-code into your program). To operate this device, we should use the I/O instructions. See the __outbyte documentations from MSDN.

To output some data to the device, it’s pretty simple:

void DebugWriteUnsafe(IN PBYTE Buffer,IN SIZE_T Length)
{
        for(SIZE_T i=0;i<Length;i++)
                __outbyte(0x402,Buffer[i]);    // Use your actual port number!
}

The reason it’s unsafe is because there are race conditions among different cores. Without a lock, simultaneous output will render your output being mixed together, causing the outputted content chaotic and hard to read. As such, a lock is required. Considering the extreme context we will be using this device, a spin-lock is required. However, you can also be lazy: just add as many ISA-DebugCon devices as the CPU cores of the VM so that race condition is mitigated.

However, this device can only be used as a debug-printer. Albeit you can write to it in order to output your debug messages, any reads from it will only return byte 0xE9. This is this device’s signature. In other words, this device can’t be used as an interactive debugger. This characteristic can assist detecting ISA-DebugCon device, but it’s only necessary – not sufficient – to prove a port is used for ISA-DebugCon. In other words, if a byte read from the port is 0xE9, you can’t confirm it must be ISA-DebugCon device (not a sufficient condition), but if that byte is not 0xE9, you can confirm it’s absolutely not ISA-DebugCon device (a necessary condition).

You may use __outbytestring macro in order to remove the for-loop. However, please note that the future x86-S architecture will deprecate the string I/O instructions, so it’s not recommended for the sake of compatibility.

Using GDB

It is recommended to enable QEMU monitor regardless of whether you will be using GDB or not. If you will be using SPICE so that you can operate QEMU VM screen from a remote Windows workstation, you can append the following parameters to enable QEMU Monitor:

-monitor tcp:0.0.0.0:3002,server

You will need to use a telnet client (e.g.: PuTTY) to connect to QEMU Monitor console. In QEMU Monitor, you can operate the VM state. To enable GDB server, enter the following command:

gdbserver tcp:0.0.0.0:3003

The best way to use GDB in Windows is probably via IDA. However, that’s a feature from IDA Pro version. IDA Freeware cannot attach to a remote GDB session. VisualGDB is not free either. So, let’s just use GDB itself. To use GDB on Windows, we may run it on WSL, or msys2 if your Windows is too old to run WSL. Simply type gdb to run GDB. In GDB’s console, run the command:

tar rem kvmhost:3003

Replace “kvmhost” with the address of your KVM host machine. Adjust the port number if you are using a different one. Listed here are some simple commands you should know when you debug VM in QEMU.

You may write your own debugger by using GDB protocol so that PDB symbols can be supported. The specification of remote GDB protocol is available online.

Using GDB could be particularly useful when you are going to debug or even reverse engineer proprietary software components such as PatchGuard.

Summary

This blog introduced the method of using QEMU+Linux-KVM and installing debugging environment for Windows Kernel and most importantly – the usage of ISA-DebugCon peripheral device. This blog also briefly described about using GDB to debug QEMU guest.

Leave a Reply

Your email address will not be published. Required fields are marked *