Configure GPU Passthrough (Archived 2024-04-14) #

DO NOT USE THIS - ONLY FOR REFERENCE - ALREADY AUTOMATED IN HOST SETUP SCRIPTS

This guide is aimed mainly for GPU without vGPU-support, as a alternative approach to enable GPU computing inside a virtual machine. It is tested on Dell Precision 7920 Rack with an Intel CPU, and based on this guide. This means, if the machine does not have an Intel CPU, it might be better to follow the original guide directly.

Enable virtualization

1. Go to BIOS
2. Enable virtualzation
3. If applicable, enable PCIe virtualization passthrough

Install required packages

Use the following command:

sudo apt install qemu-kvm qemu-utils libvirt-daemon-system libvirt-clients bridge-utils virt-manager ovmf

Setting up the PCI passthrough

Identify bus IDs and GPU IDs #

1. Run lspci -nnv and look for the GPU.
2. Note ALL of the bus channels and ID’s (including Serial, USB, and audio). See image below.

Edit GRUB #

1. sudo nano /etc/default/grub
2. Edit or addGRUB_CMDLINE_LINUX_DEFAULT so it includes the option and all of the GPU IDs from the previous step: GRUB_CMDLINE_LINUX_DEFAULT=“intel_iommu=on vfio-pci.ids=,,<id…>” For example: GRUB_CMDLINE_LINUX_DEFAULT=“intel_iommu=on vfio-pci.ids=10de:1eb0,10de:10f8,10de:1ad8,10de:1ad9”
3. sudo update-grub
4. Reboot the machine
5. Verify settings after reboot with dmesg | grep IOMMU

Apply VFIO-pci driver by PCI bus id #

1. sudo nano /etc/initramfs-tools/scripts/init-top/vfio.sh
2. Paste the following lines and change the bus id placeholders, for example: 65:00.0 65:00.1 65:00.2 65:00.3.

    #!/bin/sh

    PREREQ=""

    prereqs()
    {
        echo "$PREREQ"
    }

    case $1 in
    prereqs)
        prereqs
        exit 0
        ;;
    esac

    for dev in <bus id>
    do
        echo "vfio-pci" > /sys/bus/pci/devices/$dev/driver_override
        echo "$dev" > /sys/bus/pci/drivers/vfio-pci/bind
    done

    exit 0

3. sudo chmod +x /etc/initramfs-tools/scripts/init-top/vfio.sh

4. sudo nano /etc/initramfs-tools/modules

5. Add the following line:

options kvm ignore_msrs=1

6. sudo update-initramfs -u -k all

Verify installation #

Run lspci -nnv and find the GPU. Ensure that Kernel driver in use: vfio-pci 

Start GPU VM in CloudStack #

1. Make sure the VM is turned off
2. Edit the VM’s settings and add a new entry extraconfig-1 
3. Paste the following lines below and change the bus placeholder, for example bus=’0x65'

    <devices>
        <hostdev mode="subsystem" type="pci" managed="yes">
            <driver name="vfio"/>
            <source>
                <address domain="0x0000" bus="0x65" slot="0x00" function="0x0"/>
            </source>
            <alias name="nvidia0"/>
            <address type="pci" domain="0x0000" bus="0x00" slot="0x00" function="0x0"/>
        </hostdev>
    </devices>

If more cards are to be passed through to a single machine, see this guide. Once finished, the settings panel should look like this:

4. Start the VM

Troubleshooting #

vfio-pci driver not in use #

Some devices do not automatically use the new vfio-pci, so it can be force bound using a bash script.

1. Make note of which driver it is using instead of vfio-pci, such as nvidia-gpu.

2. Create a bash script somewhere and paste the lines below. Make sure to swap the placeholder with the bus id of the device with the wrong driver.

#!/bin/sh

PCI_HID="0000:YOUR-BUS-ID-HERE"

echo -n "$PCI_HID" > /sys/bus/pci/drivers//unbind

echo -n "$PCI_HID" > /sys/bus/pci/drivers/vfio-pci/bind

If it is more than one device, lines 2-4 can be copied and placed underneath.

3. sudo nano/etc/systemd/system/bind-gpu-drivers.service

4. Paste the following lines and change the script location placeholder:

[Unit]
Description=Bind GPU drivers so CloudStack can use them
After=network-online.target

[Service]
ExecStart=your script location

[Install]
WantedBy=multi-user.target

5. sudo systemctl enable bind-gpu-drivers.service && sudo systemctl start bind-gpu-drivers.service