Installing macOS 12 “Monterey” Developer Beta on Proxmox 6

With the public release of Monterey, this guide is now obsolete, please use my new installation guide instead

This tutorial for installing macOS Monterey Developer Beta has been adapted for Proxmox from Kholia’s OSX-KVM project and Leoyzen’s OpenCore configuration for KVM. You can get the full sourcecode of my OpenCore release on my GitHub here.

Requirements

Since Monterey is still in closed Developer Beta, you need to be an Apple Developer and have access to a Mac (or Mac VM) to download it.

Your Proxmox host computer’s CPU must support SSE 4.2, so for Intel your CPU must be at least as new as Nehalem, which was the first CPU generation to bear the “Core” i5/i7 branding. Older CPUs will cause Illegal Instruction crashes when apps/extensions attempt to use these missing instructions.

Modern AMD CPUs also support SSE 4.2 and will work with this guide.

First step: Create an installation ISO

Since Monterey is still in closed Developer Beta, you need to opt-in to the Apple beta program and grab Monterey from System Update. Just exit out of the install wizard when it says “to set up the installation of macOS 12 Beta, click Continue” and you should be left with “Install macOS 12 Beta” in your Applications folder.

Download my copy of the OSX-KVM repository using the download button, and unzip it:

https://github.com/thenickdude/OSX-KVM

Open up the Terminal and run this command to install the commandline tools:

xcode-select --install

Now from the root of OSX-KVM, run:

cd scripts/monterey
make Monterey-full.img

This will build a Monterey-full.img file in the scripts/monterey directory for you. Upload this file to your Proxmox’s ISO store directory (typically /var/lib/vz/template/iso). Although we’re putting it in the ISO directory so that we can use it with Proxmox’s ISO picker later, this a raw disk image rather than a true ISO.

Prepare an OpenCore image

Download the OpenCore.iso.gz file from the newest release in my repository (you need v13 or newer), double click it to unpack it, and upload it to Proxmox’s ISO store at /var/lib/vz/template/iso. Although it has a .iso file extension, this is actually a hard disk image.

Fetch the OSK authentication key

macOS checks that it is running on real Mac hardware, and refuses to boot on third-party hardware. You can get around this by reading an authentication key out of your real Mac hardware (the OSK key). Save the first block of C code from this page as smc_read.c. In a command prompt, change into the same directory as that file and run:

xcode-select --install # If you don't already have gcc
gcc -o smc_read smc_read.c -framework IOKit
./smc_read

It’ll print out the 64 character OSK for you. Make a note of it.

Every Mac uses the same OSK, so don’t be surprised that it doesn’t look like a random string!

Create the VM

From the Proxmox web UI, create a new virtual machine as shown below.

• 

• 

• 

• 

• 

• 

• 

In the Options page for the VM, ensure that “use tablet for pointer” is set to “Yes”.

In the Hardware page for the VM, add a second DVD drive at IDE0, set it to use your Monterey-full.img.

Don’t try to start the VM just yet. First, SSH into your Proxmox server so we can make some edits to the configuration files.

Edit /etc/pve/qemu-server/YOUR-VM-ID-HERE.conf (e.g. with nano or vim). Add this line, being sure to substitute the OSK you extracted earlier into the right place:

args: -device isa-applesmc,osk="THE-OSK-YOU-EXTRACTED-GOES-HERE" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2

A USB keyboard is added here because macOS doesn’t support QEMU’s default PS/2 keyboard. Ensure the args are all on a single line!

We also need to add a -cpu argument. If your host CPU is Intel, add this to the end of the “args” line:

-cpu host,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc

This will pass through all of the features that your CPU supports. OpenCore’s config will pretend to macOS that the CPU’s model name is Penryn for compatibility.

If your host CPU is AMD, or the above argument doesn’t work for you, use this more-compatible alternative:

-cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,check

This pretends that your CPU is Penryn, which will keep macOS happy even if your host CPU is AMD, and adds a bunch of newer required and optional CPU features on top. Features that your host CPU doesn’t support will be ignored (a warning will be printed to the console during launch with qm start 1xx), but note that macOS won’t work without SSE4.2 support.

You can remove the “+invtsc” feature from the list if your CPU doesn’t support it, or if you want to be able to migrate a running VM between Proxmox nodes.

Now find the lines that define the two “ISOs” (ide0 and ide2), and remove the “,media=cdrom” part from them. Add “,cache=unsafe” in its place. This will treat these as hard disks rather than DVD drives.

Save your changes, return to the Options tab, and change the boot order to put IDE2 (the OpenCore image) first. Your final VM configuration file should resemble this:

args: -device isa-applesmc,osk="..." -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu host,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc
agent: 1
balloon: 0
bios: ovmf
boot: order=ide2
cores: 4
cpu: Penryn
efidisk0: vms:vm-100-disk-1,size=1M
ide0: isos:iso/Monterey-full.img,cache=unsafe,size=14G
ide2: isos:iso/OpenCore-v13.img,cache=unsafe,size=150M
machine: q35
memory: 4096
name: monterey
net0: vmxnet3=...,bridge=vmbr0,firewall=1
numa: 1
ostype: other
virtio0: vms:vm-100-disk-0,cache=unsafe,discard=on,size=64G
scsihw: virtio-scsi-pci
smbios1: uuid=...
sockets: 1
vga: vmware

Configure Proxmox

On Proxmox, run “echo 1 > /sys/module/kvm/parameters/ignore_msrs” to avoid a bootloop during macOS boot. To make this change persist across Proxmox reboots, run:

echo "options kvm ignore_msrs=Y" >> /etc/modprobe.d/kvm.conf && update-initramfs -k all -u

Install Monterey

Now start up your VM, it should boot to the OpenCore boot picker:

Press enter to boot the “Install macOS 12 Beta” entry and the installer should appear. (If your keyboard isn’t working, leave the Proxmox Console page and re-enter it)

Our virtual hard drive needs to be erased/formatted before we can install to it, so select the Disk Utility option. Follow the steps below to format the disk:

• 

• 

Now you can quit Disk Utility from the top menu (Disk Utility > Quit Disk Utility), and we’re ready to begin installation!

• 

• 

• 

After the first stage of installation, the VM will reboot 2 or 3 times in quick succession, and each time you must manually pick the “macOS Installer” entry (the second one here, with the hard disk icon) to continue installation. It will not be selected for you automatically:

If your keyboard isn’t responding on this screen, exit the Console tab in Proxmox and re-enter it. If you get a “prohibited” sign like this appearing, hit the Reset button on the VM to try again:

Now the installation is nearly complete and the macOS Installer entry disappears, so pick the name of your main disk to boot (mine’s called Main).

There’s just one more reboot to come. Afterwards pick the “Main” entry again and this time you’ll finally boot into Monterey!

Answer the initial install questions, and you’ll be logged on! Note that you will want to hold off on logging into your Apple ID until you’ve configured your Mac’s serial number in OpenCore (because otherwise a Mac with the default shared serial number in my OpenCore image will be added to your Apple ID).

Note that it will be really sluggish for a few minutes after the first boot while the system performs housekeeping tasks.

There’s currently a problem where the loginwindow crashes when it is displayed, and this will prevent macOS from booting. So before you reboot macOS you must either enable FileVault disk encryption, or enable automatic login for your account, in order to avoid the loginwindow from being displayed.

To enable FileVault, hit the Apple menu, then System Preferences, Security & Privacy, and on the FileVault tab, turn on FileVault:

Or to enable automatic login, hit the Apple menu, then Users & Groups, and select the Automatic Login option:

Make the OpenCore install permanent

We’re currently booting using OpenCore from the attached OpenCore ISO. Let’s install that to the hard drive instead. Pop open Terminal and run “diskutil list” to see what drives we have available.

Use “sudo dd if=<source> of=<dest>” to copy the “EFI” partition from the OpenCore CD and overwrite the EFI partition on the hard disk. The OpenCore CD is the small disk (~150MB) that only has an EFI partition on it, and the main hard disk is the one with the large (>30GB) Apple_APFS “Container” partition on it.

In my case these EFI partitions ended up being called disk2s1 and disk0s1 respectively, so I ran “sudo dd if=/dev/disk2s1 of=/dev/disk0s1” (note that if you get these names wrong, you will overwrite the wrong disk and you’ll have to start the installation over again!).

Now shut down the VM, and remove both the OpenCore and the Monterey installer drives from the Hardware tab. On the Options tab, edit the boot order to place your virtio0 disk as the first disk. Boot up. If everything went well, you should see the OpenCore boot menu, and you can select your “Main” disk to boot Monterey:

Sleep management

I found that I was unable to wake Monterey from sleep using my mouse or keyboard. If you encounter the same problem, you can either disable system sleep in Monterey’s Energy Saver settings to avoid the issue, or you can manually wake the VM up from sleep from Proxmox by running:

qm monitor YOUR-VM-ID-HERE 
system_wakeup 
quit

Editing your OpenCore/EFI settings

The Configuration.pdf that explains the OpenCore config.plist file can be found along with the OpenCore release on my GitHub.

To mount your EFI partition in macOS so you can edit your config.plist, first check the “identifier” of your EFI partition in the terminal:

~$ diskutil list
/dev/disk0 (external, physical):
   #:                   TYPE NAME              SIZE       IDENTIFIER
   0:  GUID_partition_scheme                  *512.1 GB   disk0
   1:                    EFI EFI               209.7 MB   disk0s1
   2:             Apple_APFS Container disk1   511.9 GB   disk0s2

Then you can mount it like so:

sudo mkdir /Volumes/EFI
sudo mount -t msdos /dev/disk0s1 /Volumes/EFI

Now you can edit /Volumes/EFI/OC/config.plist with your favourite text editor to make your changes. (TextEdit is not a great choice because it likes to insert curly quotes into the file and otherwise break things, there are some dedicated plist editors available such as XCode).

If you’re unable to boot macOS, you can edit the config.plist using the “UEFI Shell” option in the OpenCore boot menu instead.

Enter “FS0:” and press enter to open up the first available filesystem, then run “edit EFI\OC\config.plist” (if the file isn’t found, try switching to another filesystem like fs1:). When you’re done editing, press control+Q to exit, “Y” to save, then run “exit” to return to the OpenCore menu. You need to reboot for your changes to take effect.

If you prefer, you can edit config.plist from the comfort of your Proxmox host instead. If you’re booting from an attached OpenCore.img file, you can mount that file as a disk on the host. If you’re booting from the VM’s disk instead, it must be in raw format in order to be mounted (e.g. typical LVM or ZFS usage) rather than qcow2.

# Mount an OpenCore image:
losetup --partscan /dev/loop0 /var/lib/vz/template/iso/OpenCore-v11.img
# or a VM boot disk:
losetup --partscan /dev/loop0 /dev/zvol/tank/vms/vm-100-disk-1

mount /dev/loop0p1 /mnt

Now the contents of that first partition are available in /mnt, so you can edit /mnt/EFI/OC/config.plist in your favourite editor. When you’re done, do this to unmount the disk:

umount /mnt
losetup --detach /dev/loop0

Automatic boot

In config.plist, you can set Misc/Boot/Timeout to a non-zero value to allow the default boot option be chosen automatically after that delay in seconds. I’ve disabled this by default because it causes the installer ISO to re-enter its main menu instead of continuing the second stage of installation.

You can set the default boot option by pressing control+enter on it.

Verbose boot

To boot macOS in Verbose mode to diagnose boot problems, at the OpenCore boot menu press Cmd+V before pressing enter to boot macOS (you don’t need to hold it down).

If there is a kernel panic during boot and it reboots too quickly to be read, edit config.plist to add “debug=0x100” to the kernel arguments.

Changing screen resolution

To change macOS’ screen resolution, you need to edit the UEFI/Output/Resolution setting in config.plist, the default is 1920×1080@32.

You should be able to change this to any of the modes that the system OVMF menu offers (hit F2 at the start of guest boot and choose “Device Manager/OVMF Platform Configuration” to see which resolutions are available).

Video performance

Because there is no guest video acceleration available for macOS, video performance is poor.

In Google Chrome in the guest you will need to toggle off the setting to “use hardware acceleration when available” to improve issues with elements not being drawn or flickering (especially video). Safari may be a better choice.

macOS’s built in “Screen Sharing” feature offers dramatically better framerates and latency than Proxmox’s browser-based VNC console, so if you have a real Mac to act as a viewing console, you can enable that in the VM’s “Sharing” settings and connect to the VM using the Screen Sharing app from your Mac instead:

Apparently Screen Sharing is also compatible with VNC clients like RealVNC, so you should be able to connect to it from Linux or Windows consoles using RealVNC.

The real magic bullet for video performance is to pass through a compatible video card using PCIe passthrough (though note that macOS now only supports a handful of very old NVidia cards). This offers near-native performance. You can read more about how I’m using PCIe passthrough on my own installation here.

USB passthrough

Since I want to use this as my primary computer, I want to use a USB keyboard and mouse plugged directly into Proxmox, rather than sending my input through the web VNC console.

Proxmox has good documentation for USB passthrough. Basically, run “qm monitor YOUR-VM-ID-HERE”, then “info usbhost” to get a list of the USB devices connected to Proxmox:

qm> info usbhost
 Bus 3, Addr 12, Port 6, Speed 480 Mb/s
 Class 00: USB device 8564:1000, Mass Storage Device
 Bus 3, Addr 11, Port 5.4, Speed 12 Mb/s
 Class 00: USB device 04d9:0141, USB Keyboard
 Bus 3, Addr 10, Port 5.1.2, Speed 12 Mb/s
 Class 00: USB device 046d:c52b, USB Receiver
 Bus 3, Addr 9, Port 14.4, Speed 12 Mb/s
 Class 00: USB device 046d:c227, G15 GamePanel LCD
 Bus 3, Addr 8, Port 14.1, Speed 1.5 Mb/s
 Class 00: USB device 046d:c226, G15 Gaming Keyboard

In this case I can add my keyboard and mouse to USB passthrough by quitting qm, then running:

qm set YOUR-VM-ID-HERE -usb1 host=04d9:0141
qm set YOUR-VM-ID-HERE -usb2 host=046d:c52b

This saves the devices to the VM configuration for you. You need to reboot to have the new settings apply. Note that the emulated USB3 device doesn’t work with macOS, so don’t set “usb3=1”.

You can also pass through USB devices by passing through an entire USB controller using Proxmox’s PCIe passthrough feature, which gives much better compatibility.

Fixing “guest boots to UEFI shell”

If your guest ends up booting to the UEFI shell instead of showing the OpenCore boot menu, especially if you’ve just updated OpenCore to a new version, you’ll need to edit the guest’s UEFI boot entries to fix this.

At the very start of guest boot, hit F2 to enter guest UEFI settings.

First we’ll remove the old entries. Choose the Boot Maintenance option, then Boot Options -> Delete Boot Option. Use the spacebar to tick any old Clover or OpenCore entries (avoid ticking the EFI Internal Shell option, you want to keep that!). Select “Commit Changes and Exit”.

Now we’ll add the correct entry for OpenCore back in. Select Add Boot Option. Navigate through the device tree to EFI/OC/OpenCore.efi (use EFI/BOOT/BOOTx64.efi instead on my “OpenCore v10” and older) and select it, name this new option “OpenCore” or similar. Again Commit Changes and Exit.

Go to the Change Boot Order and move OpenCore to the top. Commit Changes and Exit.

Now back out to the main menu and choose Reset, and you should successfully boot into OpenCore this time.

Fixing iMessage

iCloud and the App Store should already be working for you, but for iMessage support you must follow these steps to mark your network adapter as built-in:

https://dortania.github.io/OpenCore-Post-Install/universal/iservices.html#fixing-en0

Disabling SIP (System Integrity Protection)

You can disable SIP by selecting the Recovery option from the OpenCore boot menu, then use the top menu to open the Terminal and run “csrutil disable –no-internal”. Then reboot.

This may be needed to run unsigned kexts or perform other hacks.

Upgrading OpenCore

Sometimes you need to update OpenCore to a new release in order to support a new macOS update. I’ll assume you don’t have any customisations to config.plist you want to save.

First take a snapshot! It’s great to be able to roll back if something goes wrong.

If you’re still able to boot macOS, you can update it from within the guest. Follow the instructions in the “Editing your OpenCore/EFI settings” section to mount your EFI partition. Then you can delete the EFI folder in there and replace it with the one from the OpenCoreEFIFolder.zip file from my OpenCore release (you’ll probably need to empty the trash to make room for the new folder). You’re done!

If you aren’t able to boot macOS, unpack and upload the new OpenCore ISO to Proxmox’s ISO store instead. Add a new CD drive to the VM that uses that ISO. Then in Proxmox’s terminal edit the VM’s config (in /etc/pve/qemu-server) to replace “media=cdrom” with “cache=unsafe” for the OpenCore drive. Now on the “Options” tab, change the boot order to put the new OpenCore drive first.

Start the VM and boot into macOS using the new OpenCore drive. From within macOS you can now follow the instructions from the “Make the OpenCore install permanent” section to install the new OpenCore image to your main macOS disk, after which the OpenCore drive can be detached from the VM.

Upgrading from macOS Big Sur

Monterey is still in Beta, so upgrading a system you care about is definitely not advised yet. But if you have a throwaway VM to try it on…

First make a backup or snapshot of your system! Being able to roll back when the upgrade goes wrong is a real lifesaver.

You’ll need to update OpenCore to my v13 release before the upgrade. You can follow the instructions in the “Editing your OpenCore settings” section above to mount your EFI disk. Then you can replace the OpenCore files in the mounted “EFI” disk with the ones from the OpenCoreEFIFolder.zip file in my newest OpenCore release.

Reboot to make sure that you can still boot Big Sir.

If you’re using any PCIe passthrough devices (particularly video cards) you’ll want to disable those and set “vga: vmware” instead, so you can install using Proxmox’s web console from a different machine during the upgrade. This avoids installer problems triggered by flaky video card passthrough, especially host lockups caused by the AMD Reset Bug.

Now you can upgrade to Monterey using Software Update like you would on a real Mac.