前言
本文使用硬件为Pascal架构的P106显卡，如您使用的显卡非本架构请自行变通
本文也在NVIDIA CMP 40HX进行验证也可用
例如ampere架构的显卡需要自行开启VF，且每个VF仅能选择一个配置开启
文中提及的libvgpu_unlock_rs.so、NVIDIA-Linux-x86_64-525.85.07-vgpu-kvm-patched.run 、528.24_grid_win10_win11_server2019_server2022_dch_64bit_international.exe 均在上传中
传完了评论区见
本文内核为pve-kernel-5.15.102-1-pve，如果是6.2.9内核请使用NVIDIA-Linux-x86_64-525.85.07-vgpu-kvm-kernel-6.2.9-patched.run
以下本文使用的主要硬件及配置，如果您平台与本篇差距过大也请自行变通
主板：ASUS B150 PRO GAMING D3 （其他主板怎么动bios你自己看着办)
CPU：Intel QQLT （不是Intel的U开启IOMMU方法不一样）
内存：3条8G的DDR3绿条
存储：傲腾16G
所谓显卡：NVIDIA P106-090(用NVIDIA CMP 40HX也不影响，其他卡我也没有不知道)
网络存储：嗨群晖开启SMB及NFS
后续补充及勘误见评论区，如有重大修正会直接编辑文章
BIOS设置
开启VT
如图所示，开启VT功能
开启VT-d
如图所示，开启VT-d
开启主要输出为核显
不开启核显可能导致P106插上去后，就黑屏了
后续有兴趣你也可以顺便玩玩Intel的GVT-G
安全引导设置
关闭安全引导，或者是选择非Windows系统
安装PVE
略，这个有啥好说的，非本文重点好吧
查看系统信息
查看内核cmdline
此步骤可以迅速排查引导参数导致翻车的问题
如图所示，参数未有所需的
查看IOMMU状态
这一步不用命令行，直接在这个添加PCI设备看一眼就行
如图所示这种状态就是没开起来
查看显卡情况
使用 即可，此步骤防止你以为卡插好了实际上并没有的情况
如图所示，这是一张P106-090
修改相关配置及参数
/etc/modules
echo -e "vfio\nvfio_iommu_type1\nvfio_pci\nvfio_virqfd" >> /etc/modules
/etc/modprobe.d/pve-blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/pve-blacklist.conf
内核cmdline
此处需要注意不同启动方式需要修改的文件不同
本例为/etc/default/grub
添加intel_iommu=on iommu=pt spectre_v2=off到如图所示位置，spectre_v2参数非必须相关参数
安装依赖及软件包
更换软件源
如下4行，执行便可
root@pve:~# sed -i 's|^deb http://ftp.debian.org|deb https://mirrors.ustc.edu.cn|g' /etc/apt/sources.list
root@pve:~# sed -i 's|^deb http://security.debian.org|deb https://mirrors.ustc.edu.cn/debian-security|g' /etc/apt/sources.list
root@pve:~# source /etc/os-release
root@pve:~# echo "deb https://mirrors.ustc.edu.cn/proxmox/debian/pve $VERSION_CODENAME pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list订阅源要如图所示选中并点击禁用
安装包
一句 即可
apt install -y build-essential dkms pve-headers-$(uname -r) mdevctl
root@pve:~# apt install -y build-essential dkms pve-headers-$(uname -r) mdevctl
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
cpp cpp-10 dctrl-tools dpkg-dev fakeroot g++ g++-10 gcc gcc-10 jq libalgorithm-diff-perl libalgorithm-diff-xs-perl
libalgorithm-merge-perl libasan6 libatomic1 libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libdeflate0
libdpkg-perl libfakeroot libfile-fcntllock-perl libgcc-10-dev libgd3 libgomp1 libisl23 libitm1 libjbig0 libjq1 liblsan0
libmpc3 libmpfr6 libnsl-dev libonig5 libquadmath0 libstdc++-10-dev libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp6 libxpm4
linux-compiler-gcc-10-x86 linux-headers-5.10.0-21-amd64 linux-headers-5.10.0-21-common linux-headers-amd64 linux-kbuild-5.10
linux-libc-dev lsb-release make manpages-dev patch pve-headers-5.15 pve-headers-5.15.102-1-pve sudo
Suggested packages:
cpp-doc gcc-10-locales debtags menu debian-keyring g++-multilib g++-10-multilib gcc-10-doc gcc-multilib autoconf automake
libtool flex bison gdb gcc-doc gcc-10-multilib glibc-doc git bzr libgd-tools libstdc++-10-doc make-doc ed diffutils-doc
The following NEW packages will be installed:
build-essential cpp cpp-10 dctrl-tools dkms dpkg-dev fakeroot g++ g++-10 gcc gcc-10 jq libalgorithm-diff-perl
libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libatomic1 libc-dev-bin libc-devtools libc6-dev libcc1-0
libcrypt-dev libdeflate0 libdpkg-perl libfakeroot libfile-fcntllock-perl libgcc-10-dev libgd3 libgomp1 libisl23 libitm1
libjbig0 libjq1 liblsan0 libmpc3 libmpfr6 libnsl-dev libonig5 libquadmath0 libstdc++-10-dev libtiff5 libtirpc-dev libtsan0
libubsan1 libwebp6 libxpm4 linux-compiler-gcc-10-x86 linux-headers-5.10.0-21-amd64 linux-headers-5.10.0-21-common
linux-headers-amd64 linux-kbuild-5.10 linux-libc-dev lsb-release make manpages-dev mdevctl patch pve-headers
pve-headers-5.15 pve-headers-5.15.102-1-pve sudo
0 upgraded, 61 newly installed, 0 to remove and 18 not upgraded.刷新grub及引导
分别运行update-grub及 即可，随后使用reboot重启
root@pve:~# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.102-1-pve
Found initrd image: /boot/initrd.img-5.15.102-1-pve
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done
root@pve:~# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.15.102-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
root@pve:~# reboot重启后检查
如下所示，cmdline应有刚才添加的内容，添加PCI设备也消去了警告
root@pve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.102-1-pve root=UUID=61e72f0f-a4f7-4455-9959-aebf71e6af5e ro quiet intel_iommu=on iommu=pt spectre_v2=off
host安装VGPU驱动
创建相关配置
root@pve:# mkdir /etc/vgpu_unlock
root@pve:# touch /etc/vgpu_unlock/profile_override.toml
root@pve:# mkdir /etc/systemd/system/{nvidia-vgpud.service.d,nvidia-vgpu-mgr.service.d}
root@pve:# echo -e "[Service]\nEnvironment=LD_PRELOAD=/opt/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so" > /etc/systemd/system/nvidia-vgpud.service.d/vgpu_unlock.conf
root@pve:# echo -e "[Service]\nEnvironment=LD_PRELOAD=/opt/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so" > /etc/systemd/system/nvidia-vgpu-mgr.service.d/vgpu_unlock.conf
root@pve:# cat /etc/systemd/system/{nvidia-vgpud.service.d,nvidia-vgpu-mgr.service.d}/*
[Service]
Environment=LD_PRELOAD=/opt/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so
[Service]
Environment=LD_PRELOAD=/opt/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so上传必须文件
上传前需要通过mkdir -p /opt/vgpu_unlock-rs/target/release创建文件夹
随后通过ssh或者是其他方式上传及
上传完了记得用chmod给run文件加可执行权限
root@pve:~# mkdir -p /opt/vgpu_unlock-rs/target/release
root@pve:~# cd /opt/vgpu_unlock-rs/target/release
root@pve:/opt/vgpu_unlock-rs/target/release# ls -l
total 96240
-rw-r--r-- 1 root root 4756296 Apr 16 01:29 libvgpu_unlock_rs.so
-rw-r--r-- 1 root root 93787315 Apr 16 01:29 NVIDIA-Linux-x86_64-525.85.07-vgpu-kvm-patched.run
root@pve:/opt/vgpu_unlock-rs/target/release# chmod +x NVIDIA-Linux-x86_64-525.85.07-vgpu-kvm-patched.run
root@pve:/opt/vgpu_unlock-rs/target/release# ls -l
total 96240
-rw-r--r-- 1 root root 4756296 Apr 16 01:29 libvgpu_unlock_rs.so
-rwxr-xr-x 1 root root 93787315 Apr 16 01:29 NVIDIA-Linux-x86_64-525.85.07-vgpu-kvm-patched.run安装host驱动
一句./NVIDIA-Linux-x86_64-525.85.07-vgpu-kvm-patched.run 就跑起来了
检查安装情况
保险起见可以先使用 启动相关服务
随后使用nvidia-smi以及mdevctl types查看
guest安装VGPU驱动
向虚拟机添加设备
首先给你要跑gpu的虚拟机添加设备
记得虚拟机改成q35，CPU不会选的话直接host，nvidia启动对CPU是有要求的
虚拟机安装guest驱动
想个办法，无论是走smb也好，nfs也好，还是浏览器也好，把对应版本的GRID驱动丢进去安装
注意，如果版本高于宿主机驱动，或者是低于宿主机驱动版本，均有可能出现包括但不限于装不上、感叹号等异常
如下，Windows虚拟机安装驱动528.24_grid_win10_win11_server2019_server2022_dch_64bit_international.exe完成
可以去PVE看看
启用授权
此过程略过，如需了解请看另外一篇文章，那篇有介绍
授权文件丢进文件夹后，重启如图所示的服务
后记
本文成篇于深夜，行文仍需调整，欢迎各位指正优化
好困，顶不住了，后面再改改优化优化
其实40HX等矿卡也差不多这个玩法
我看NVIDIA被开源源码的话50HX那些是有搞头的，什么时候价格好起来了捡一张回来试试
