Getting PCI Passthrough working on G6 HP Proliant Servers with Proxmox

Standard

A better way to do this – Update: 2021-06-28

Someone has put together a patched kernel for Proxmox to let people run passthrough on their HP servers, it’s available on GitHub as a .deb download for direct install or you can build it from their sources yourself. Visit kiler129’s relax-intel-rmrr repository for a better way to do this 🙂

The rest of this post is included as an archive.

This is a quick stub post to show how I finally got PCI pass-through working on my HP DL360 G6 server, since it would show an error such as

DMAR-IR: This system BIOS has enabled interrupt remapping interrupt remapping is being disabled.

If you actually tried to launch a machine with a PCI device attached, it would say:

Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.

From what I can tell this is because the system uses RMRR to control fan speeds, and having these bits set causes the driver to disable IOMMU. There is a simple change you can make to the Proxmox kernel source code to get passed this error and use PCI devices with VMs without issue.

sed -i.new 's/device_is_rmrr_locked(dev)/false/' ./$UBUNTU_NAME/drivers/iommu/intel-iommu.c

After making that change and recompiling the kernel, it works perfectly fine, but the system will still say DMAR-IR interrupts have been disabled, however it will work fine.

I am indebted to this Reddit post and the script that they put together that was created from their research, which let me get my DL360 G6 working. They have a DL380 but I am pleased it works on my system. I don’t have a Reddit account, but show the original author some love if you do! https://old.reddit.com/r/homelab/comments/iw5cew/proxmox_i_created_a_script_to_fix_gpu_pass/

I’ve quoted their script below just in-case the original pastebin link in their Reddit post goes down.

#!/bin/bash
 
#Proxmox 6.2 over Ubuntu Focal Fossa patch for IOMMU.
#this can be done with Proxmox vanilla, just run the script!
 
#based heavily on
#https://forum.proxmox.com/threads/help-with-pci-passthrough.23980/
 
#move to working dir
cd "${0%/*}"
 
#Proxmox 4.4 kernel modding guide dependencies (see the above link)
#apt-get install git screen fakeroot build-essential devscripts libncurses5 libncurses5-dev libssl-dev bc flex bison libelf-dev libaudit-dev libgtk2.0-dev libperl-dev libperl-dev asciidoc xmlto gnupg gnupg2
#Debian kernel build dependencies
#apt-get install build-essential linux-source bc kmod cpio flex cpio libncurses5-dev
#zfsonlinux has hidden dependencies
#apt-get install dh-python python3-cffi python3-setuptools python3-sphinx python3-all-dev
#even more dependencies
#apt-get install libdw-dev libiberty-dev libnuma-dev libslang2-dev lz4
 
#full list of dependencies, though it may contain duplicates
apt-get install git screen fakeroot build-essential devscripts libncurses5 libncurses5-dev libssl-dev bc flex bison libelf-dev libaudit-dev libgtk2.0-dev libperl-dev libperl-dev asciidoc xmlto gnupg gnupg2 build-essential linux-source bc kmod cpio flex cpio libncurses5-dev dh-python python3-cffi python3-setuptools python3-sphinx python3-all-dev libdw-dev libiberty-dev libnuma-dev libslang2-dev lz4
 
#rm -r ./tmp/proliant-iommu-patch/
#TODO Only if the user says so on finishing compilation - Ubuntu-focal file is weird so we have to remove
#rm -r ./pve-kernel
 
#mkdir ./tmp/proliant-iommu-patch
#cd ./tmp/proliant-iommu-patch
 
#/tmp/proliant-iommu-patch/pve-kernel/patches/kernel/
git clone git://git.proxmox.com/git/pve-kernel.git
#mkdir ./pve-kernel
cd ./pve-kernel
 
#use a proxmox makefile bug to only get the submodules. THIS iS A BAD IDEA
make -j 2
 
cd ./submodules
 
#saves the hassle of Proxmox using git on its own
#git clone git://kernel.ubuntu.com/ubuntu/ubuntu-focal.git/
 
#TODO UBUNTU_NAME= "$(ls | grep -m 1 ubuntu)" > used for compatibility
 
UBUNTU_NAME="$(ls | grep -m 1 ubuntu)"
 
echo $UBUNTU_NAME
 
#if [ ! -d ./$UBUNTU_NAME/drivers ]
#then
#    #git clone git://kernel.ubuntu.com/ubuntu/ubuntu-focal.git
#    #git clone git://git.proxmox.com/git/mirror_$UBUNTU_NAME-kernel
#    git clone https://github.com/torvalds/linux.git
#    echo ""
#    #git clone $REPOSRC $LOCALREPO
#else
#    echo $UBUNTU_NAME + " is already cloned, skipping!"
#    #cd $LOCALREPO
#    #git pull $REPOSRC
#fi
#
#mv -T ./mirror_$UBUNTU_NAME-kernel ./$UBUNTU_NAME
 
sed -i.new 's/device_is_rmrr_locked(dev)/false/' ./$UBUNTU_NAME/drivers/iommu/intel-iommu.c
 
#cat ./iommu/intel-iommu.c | grep 'device_is_rmrr_locked'
 
#named according to proxmox tutorial :)
#diff -u ./iommu/intel-iommu.c ./ubuntu-focal/drivers/iommu/intel-iommu.c > ./remove_mbrr_check.patch
 
#move back to working dir - TODO this
#cd "${0%/*}"
cd ../
 
echo "building"
make
 
find ./ -name '*.deb' -exec cp -prv '{}' '../' ';'
 
#optional TODO
##dpkg -i *.deb
#update-grub
 
cd ../
 
echo "
 
#find ./ -name '*.deb' -exec dpkg -i '{}' ';'
dpkg -i *.deb
 
update-grub
echo \"=======================================================================\"
echo \"Please reboot your system. Tell u/oezingle if your machine doesn't boot\"
echo \"=======================================================================\"
" > ./install-new-kernel.sh
 
chmod 0777 ./install-new-kernel.sh
 
rm -R ./pve-kernel
 
echo ""
echo ""
echo "===================================================================="
echo "Install all .deb files and update-grub, or run install-new-kernel.sh"
echo "===================================================================="