Emulating MIPS guests in Proxmox 5

This guide is obsolete, please see the new guide for Proxmox 8 here

I wanted to emulate MIPS guests on my Proxmox hypervisor so that I could do some security research on router firmware. Unfortunately, Proxmox has customised some of the QEMU packages and their dependencies, which makes it difficult to install the standard Debian qemu-system-mips package. In particular, Proxmox provides its own pve-libspice-server1 package which conflicts with the libspice-server1 package that vanilla QEMU depends on, so attempting to install it will complain:

Some packages could not be installed.

The following packages have unmet dependencies:
 qemu-system-mips : Depends: libspice-server1 (>= 0.12.5)

To solve this, we need to build a modified version of the package from source.

Building qemu-system-mips for Proxmox 5

We’ll build this package inside a Debian 9 container, rather than building it on the Proxmox host directly. This will allow us to avoid conflicts with Proxmox packages during apt-get build-dep.

In Proxmox’s web interface, select the storage you use to hold container templates, then on the Content page, hit the Templates button. Download the Debian 9 template.

Create a new container that uses this Debian 9 template. You’ll need to give it at least 4GB of total memory (RAM + swap) for the build to complete successfully.

Start up the new container and SSH into it. If you are unable to connect to the container by SSH, you may need to run pct enter YOUR-CONTAINER-ID from Proxmox and run service ssh restart to kick it into life.

Edit /etc/apt/sources.list to add deb-src entries for each of the Debian repositories:

deb-src http://ftp.debian.org/debian stretch main contrib
deb-src http://ftp.debian.org/debian stretch-updates main contrib
deb-src http://security.debian.org stretch/updates main contrib

Now fetch those new repositories and get the container up to date:

apt-get update && apt-get dist-upgrade -y

Grab the build dependencies for qemu and some other utilities:

apt-get build-dep qemu -y
apt-get install git -y

Clone my patched copy of the upstream Debian package for QEMU, which disables Proxmox-incompatible features (xen, spice, virtfs):

git clone -b proxmox-compat https://github.com/thenickdude/qemu-debian.git

Now enter that directory and build QEMU:

cd qemu-debian
debian/rules debian/control
dpkg-buildpackage -uc -us -b

If successful, this will result in a series of .deb packages in the parent directory. Copy these to a temporary directory on Proxmox.

Installing the new packages (Proxmox 5)

To install qemu-system-mips on Proxmox, you need to install the qemu-system-mips and qemu-system-common packages that you built, and their dependencies:

# Install dependencies from the standard repos:
apt-get install ipxe-qemu libbluetooth3 libbrlapi0.6 libcacard0 libfdt1 libvdeplug2

# Install the .deb files you built:
dpkg -i qemu-system-common_2.8+dfsg-6+deb9u2_amd64.deb qemu-system-mips_2.8+dfsg-6+deb9u2_amd64.deb

Building qemu-system-mips for Proxmox 6

For Proxmox 6, use a Debian 10 container instead of Debian 9.

Edit /etc/apt/sources.list to add deb-src entries for each of the Debian repositories, and add the Proxmox repo too:

deb-src http://ftp.nz.debian.org/debian buster main contrib
deb-src http://ftp.nz.debian.org/debian buster-updates main contrib
deb-src http://security.debian.org buster/updates main contrib

deb http://download.proxmox.com/debian/pve buster pve-no-subscription

Now fetch those new repositories and get the container up to date:

apt-get update && apt-get dist-upgrade -y

Now use these commands to fetch and build my fork of pve-qemu:

apt build-dep qemu -y
apt install libjemalloc-dev libpci-dev libproxmox-backup-qemu0-dev libsdl1.2-dev libsystemd-dev meson python3-sphinx quilt texi2html check lintian libjson-perl -y

git clone -b alien-architectures https://github.com/thenickdude/pve-qemu.git
cd pve-qemu
make

This will build pve-qemu-kvm_5.2.0-3_amd64.deb and pve-qemu-kvm-dbg_5.2.0-3_amd64.deb files in the current directory for you, copy those to Proxmox and install them using:

dpkg -i pve-qemu-kvm_5.2.0-3_amd64.deb pve-qemu-kvm-dbg_5.2.0-3_amd64.deb

Note that as well as adding qemu-system-* binaries for MIPS (mips-softmmu, mips64-softmmu, mips64el-softmmu, mipsel-softmmu), this also adds support for ppc-softmmu, ppc64-softmmu, riscv32-softmmu, and riscv64-softmmu.

Creating a Debian MIPS virtual machine

We’ll create a Debian Wheezy MIPS virtual machine that we can run with qemu-system-mips. First, let’s set up networking. We’ll use qemu-bridge-helper to bridge the VM’s network to Proxmox’s. To enable support for that, run these commands in Proxmox:

# Allow all users to start up the bridge as root:
chmod u+s /usr/lib/qemu/qemu-bridge-helper

# Allow us to connect to Proxmox's default bridge:
mkdir /etc/qemu
echo "allow vmbr0" > /etc/qemu/bridge.conf

Grab debian_wheezy_mips_standard.qcow2 and vmlinux-3.2.0-4-4kc-malta (any of the *-4kc-malta kernels can be used) from this website and put them into the same directory as each other:

https://people.debian.org/~aurel32/qemu/mips/

Create a script called “startvm” in that directory with this content:

#!/usr/bin/env bash

qemu-system-mips -m 256M -M malta -kernel vmlinux-3.2.0-4-4kc-malta -hda debian_wheezy_mips_standard.qcow2 -append "root=/dev/sda1 console=ttyS0" -nographic -vga none -serial mon:stdio -net nic -net bridge,br=vmbr0

# To emulate a CPU compatible with Ubiquiti's Nanobeam AC (and lots of their other products), add "-cpu 34Kf"

Make that script executable:

chmod u+x startvm

Now you can start up your new Debian Wheezy VM by just running that script on Proxmox:

./startvm

The default password for the root account is just “root”. There is also an account called “user” with password “user”.

Upgrading to Debian Jessie

You can upgrade this VM to the newer Debian Jessie release like so:

# Grab newer signing keys:
apt-get install debian-keyring debian-archive-keyring -y

# Update installed packages to the newest Wheezy versions:
apt-get update
apt-get dist-upgrade -y

# Switch to Jessie repositories:
sed -i 's/wheezy/jessie/g' /etc/apt/sources.list

# Upgrade to Jessie:
apt-get update
apt-get dist-upgrade -y

shutdown -r now

Note that since this MIPS CPU is emulated in software, this will take hours.

You can then upgrade from Jessie to Stretch using the same procedure (just replace the sed command with sed -i 's/jessie/stretch/g' /etc/apt/sources.list.

Fixing ipv6

During boot, you’ll see a bunch of warnings relating to ipv6 symbol mismatches, and ipv6 won’t work:

[ 14.312000] ipv6: Unknown symbol __ip4_datagram_connect (err 0)
[ 14.336000] ipv6: Unknown symbol ip_idents_reserve (err 0)
[ 14.336000] ipv6: disagrees about version of symbol tcp_alloc_md5sig_pool
[ 14.336000] ipv6: Unknown symbol tcp_alloc_md5sig_pool (err -22)

I think this is because the ipv6 support is provided by a loadable module, which was not compiled at the same time as the kernel supplied by aurel32.

You can fix this by copying Debian’s official kernel and initrd (vmlinux-4.9.0-3-4kc-malta and initrd.img-4.9.0-3-4kc-malta for Stretch) out of the VM’s /boot directory to the host, and editing the startvm script to use those instead. Also add a “nokaslr” option to the “append” string to fix the fatal error “Initramfs unpacking failed: uncompression error” when loading the initrd.

#!/usr/bin/env bash 

qemu-system-mips -m 256M -M malta -kernel vmlinux-4.9.0-3-4kc-malta -initrd initrd.img-4.9.0-3-4kc-malta -hda debian_wheezy_mips_standard.qcow2 -append "root=/dev/sda1 console=ttyS0 nokaslr" -nographic -vga none -serial mon:stdio -net nic -net bridge,br=vmbr0

This also fixes a problem where the VM reboots instead of halting upon shutdown.