2025 年 3 月 – 丁丁的 BLOG

AIO(All in Boom[爆炸就是艺术])现在很流行ESXI或者PVE, 我不太喜欢索性直接用Rocky Linux做宿主, 使用SSH + Cockpit Web控制台管理, 安装Rocky Linux很简单这里就不再赘述.

网卡直通这个一点不奇怪，不过大多数是用vfio-pci的ids来设置PCI设备的唯一标识符, 但是有个问题, 如果是多张卡或者多口卡那么就会全部被替代, 另外插PCIe插槽要注意, 只有PCH通道才能全部拆分IOMMU组, 如果是CPU通道, 那么大概率会跟多个设备一起被编在同一个IOMMU组.

使用命令行脚本查看IOMMU的组别情况, 如果我们要直通的设备并不是独立组, 那就无法直通了.

# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done;
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 07)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP108 [GeForce GT 1030] [10de:1d01] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GP108 High Definition Audio Controller [10de:0fb8] (rev a1)
IOMMU Group 1 02:00.0 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 1 02:00.1 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 1 02:00.2 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 1 02:00.3 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)

这个是插在CPU通道的情况,可以看到四口Intel 82580千兆网卡是在同一个IOMMU组, 这样就无法进行直通的, 我们在看一下插在PCH通道上的X710-DA4万兆网卡.

# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done;
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers [8086:1918] (rev 07)
...
IOMMU Group 20 0a:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 02)
IOMMU Group 21 0a:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 02)
IOMMU Group 22 0a:00.2 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 02)
IOMMU Group 23 0a:00.3 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 02)

这里可以看到0a:00.0、0a:00.1、0a:00.2、0a:00.3分别是X710-DA4的D、C、B、A, 是的PCI插槽序号正好跟光口的顺序相反, 他们的IOMMU组已经是独立的编号, 这样就可以分别直通, 记下需要直通的ID,我们这里是C、B、A三口, 即0000:0a:00.1 0000:0a:00.2 0000:0a:00.3。

建立启动脚本的dracut配置

# mkdir -p /usr/lib/dracut/modules.d/99vfio-pci
# touch /usr/lib/dracut/modules.d/99vfio-pci/module-setup.sh
# chmod +x /usr/lib/dracut/modules.d/99vfio-pci/module-setup.sh

# vi /usr/lib/dracut/modules.d/99vfio-pci/module-setup.sh

将dracut的配置代码输入进去.

#!/bin/bash
check() {
    if [ -d "/sys/module/vfio_pci" ]; then
        return 0
    else
        return 1
    fi
}
 
depends() {
    return 0
}
 
install() {
    inst_hook initqueue/start 05 
    declare moddir = \${moddir}
    inst_hook pre-udev 00 "\${moddir}/vfio-pci-init-script.sh"
}

编辑自动启动脚本

# vi /usr/lib/dracut/modules.d/99vfio-pci/vfio-pci-init-script.sh

#!/bin/sh
DEVS="0000:0a:00.1 0000:0a:00.2 0000:0a:00.3"
for DEV in $DEVS; do
    echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
    echo $DEV > /sys/bus/pci/drivers/vfio-pci/bind
done
modprobe -i vfio-pci

这里的DEVS填写我们需要直通的PCI插槽序号, 即之前查看IOMMU组看到的序号.

; 设置可执行
# chmod +x /usr/lib/dracut/modules.d/99vfio-pci/vfio-pci-init-script.sh
; 重新生成Linux初始内存文件系统（initramfs）
# dracut /boot/initramfs-$(uname -r).img $(uname -r) --force

重启后查看vfio_pci设备信息, 可以看到驱动已经加载成功.

# ls /sys/bus/pci/drivers/vfio-pci -l
总用量 0
lrwxrwxrwx. 1 root root    0  3月  7 22:09 0000:0a:00.1 -> ../../../../devices/pci0000:00/0000:00:1d.0/0000:0a:00.1
lrwxrwxrwx. 1 root root    0  3月  7 22:09 0000:0a:00.2 -> ../../../../devices/pci0000:00/0000:00:1d.0/0000:0a:00.2
lrwxrwxrwx. 1 root root    0  3月  7 22:09 0000:0a:00.3 -> ../../../../devices/pci0000:00/0000:00:1d.0/0000:0a:00.3
--w-------. 1 root root 4096  3月  4 23:01 bind
lrwxrwxrwx. 1 root root    0  3月  7 22:09 module -> ../../../../module/vfio_pci
--w-------. 1 root root 4096  3月  7 22:09 new_id
--w-------. 1 root root 4096  3月  7 22:09 remove_id
--w-------. 1 root root 4096  3月  7 22:09 uevent
--w-------. 1 root root 4096  3月  7 22:09 unbind

获取需要直通PCIe设备的详细信息.

# virsh nodedev-list --tree |grep pci
...
  +- pci_0000_00_1d_0
  |   +- pci_0000_0a_00_0
  |   +- pci_0000_0a_00_1
  |   +- pci_0000_0a_00_2
  |   +- pci_0000_0a_00_3
...

Dump出需要直通的设备信息, pci_0000_0a_00_3 和 pci_0000_0a_00_2, 即X710的A口和B口.

# virsh nodedev-dumpxml pci_0000_0a_00_3
<device>
  <name>pci_0000_0a_00_3</name>
  <path>/sys/devices/pci0000:00/0000:00:1d.0/0000:0a:00.3</path>
  <parent>pci_0000_00_1d_0</parent>
  <driver>
    <name>vfio-pci</name>
  </driver>
  <capability type='pci'>
    <class>0x020000</class>
    <domain>0</domain>
    <bus>10</bus>
    <slot>0</slot>
    <function>3</function>
    <product id='0x1572'>Ethernet Controller X710 for 10GbE SFP+</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <capability type='virt_functions' maxCount='32'/>
    <iommuGroup number='23'>
      <address domain='0x0000' bus='0x0a' slot='0x00' function='0x3'/>
    </iommuGroup>
    <pci-express>
      <link validity='cap' port='0' speed='8' width='4'/>
      <link validity='sta' speed='8' width='4'/>
    </pci-express>
  </capability>
</device>

# virsh nodedev-dumpxml pci_0000_0a_00_2
<device>
  <name>pci_0000_0a_00_2</name>
  <path>/sys/devices/pci0000:00/0000:00:1d.0/0000:0a:00.2</path>
  <parent>pci_0000_00_1d_0</parent>
  <driver>
    <name>vfio-pci</name>
  </driver>
  <capability type='pci'>
    <class>0x020000</class>
    <domain>0</domain>
    <bus>10</bus>
    <slot>0</slot>
    <function>2</function>
    <product id='0x1572'>Ethernet Controller X710 for 10GbE SFP+</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <capability type='virt_functions' maxCount='32'/>
    <iommuGroup number='22'>
      <address domain='0x0000' bus='0x0a' slot='0x00' function='0x2'/>
    </iommuGroup>
    <pci-express>
      <link validity='cap' port='0' speed='8' width='4'/>
      <link validity='sta' speed='8' width='4'/>
    </pci-express>
  </capability>
</device>

; 下载OpenWrt x86 ext4固件
# wget https://downloads.openwrt.org/releases/24.10.0/targets/x86/64/openwrt-24.10.0-x86-64-generic-ext4-combined.img.gz
; 解压缩
# gzip -d openwrt-24.10.0-x86-64-generic-ext4-combined.img.gz

gzip: openwrt-24.10.0-x86-64-generic-ext4-combined.img.gz: decompression OK, trailing garbage ignored

; 将img镜像转换成qcow2硬盘镜像
# qemu-img convert -p -f raw -O qcow2 openwrt-24.10.0-x86-64-generic-ext4-combined.img openwrt-24.10.0-x86-64-generic-ext4-combined.qcow2

; 修改容量到20G
# qemu-img resize openwrt-24.10.0-x86-64-generic-ext4-combined.qcow2 20G
; 填充扩展容量到真实大小
# qemu-img resize -f raw openwrt-24.10.0-x86-64-generic-ext4-combined.qcow2 20G

编写XML openwrt.xml 来定义一台新的虚拟机.

# vi openwrt.xml
<domain type='kvm'>
    <name>openwrt_default_gateway</name>
    <memory unit='MiB'>2048</memory>
    <currentMemory unit='MiB'>2048</currentMemory>
    <vcpu>2</vcpu>
    <os>
        <type arch='x86_64' machine='pc-i440fx-8.2'>hvm</type>
        <boot dev='hd'/>
    </os>
    <features>
        <acpi/>
        <apic/>
    </features>
    <clock offset='utc'/>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>destroy</on_crash>
    <devices>
        <emulator>/usr/bin/qemu-system-x86_64</emulator>
        <disk type='file' device='disk'>
            <driver name='qemu' type='qcow2'/>
            <source file='/data/libvirt/disk/openwrt-24.10.0-x86-64-generic-ext4-combined.qcow2'/>
            <target dev='vda' bus='virtio'/>
        </disk>
        <hostdev mode='subsystem' type='pci' managed='yes'>
            <driver name='vfio'/>
            <source>
                <address domain='0x0000' bus='0x0a' slot='0x00' function='0x3'/>
            </source>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
            <driver name='vfio'/>
            <source>
                <address domain='0x0000' bus='0x0a' slot='0x00' function='0x2'/>
            </source>
        </hostdev>
        <interface type='network'>
            <source network='v10gswitch'/>
            <model type='virtio'/>
        </interface>
        <controller type='usb' index='0'>
            <address type='pci' domain='0x0000' bus='0x00' solt='0x02' function='0x0'/>
        </controller>
        <input type='mouse' bus='ps2'/>
        <controller type='ide' index='0'>
            <address type='pci' domain='0x0000' bus='0x00' solt='0x01' function='0x0'/>
        </controller>
        <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
            <listen type='address' address='0.0.0.0'/>
        </graphics>
    </devices>
</domain>

创建虚拟机

# virsh define openwrt.xml

由于X710的驱动i40e在OpenWrt默认是没有的, 需要手动下载并拷贝进虚拟机, 否则因为没网络而无法继续.

# wget https://downloads.openwrt.org/releases/24.10.0/targets/x86/64/kmods/6.6.73-1-a21259e4f338051d27a6443a3a7f7f1f/kmod-i40e_6.6.73-r1_x86_64.ipk

拷贝 i40e 驱动到虚拟机中

# virt-copy-in -d openwrt_default_gateway kmod-i40e_6.6.73-r1_x86_64.ipk /root
; 如果提示 bash: virt-copy-in：未找到命令 则安装软件包.
# dnf install -y libguestfs-tools
; 如果提示 libguestfs: error: stat: /usr/libexec/qemu-kvm: 没有那个文件或目录 则做一下软连接.
# ln -s $(which qemu-kvm) /usr/libexec/qemu-kvm

在虚拟机中运行

# opkg kmod-i40e_6.6.73-r1_x86_64.ipk

在虚拟机中运行自动扩容脚本, OpenWrt官网有详细介绍, 见 https://openwrt.org/docs/guide-user/advanced/expand_root

# 直接配置自动扩容脚本
cat << "EOF" > /etc/uci-defaults/70-rootpt-resize
if [ ! -e /etc/rootpt-resize ] \
&& type parted > /dev/null \
&& lock -n /var/lock/root-resize
then
ROOT_BLK="$(readlink -f /sys/dev/block/"$(awk -e \
'$9=="/dev/root"{print $3}' /proc/self/mountinfo)")"
ROOT_DISK="/dev/$(basename "${ROOT_BLK%/*}")"
ROOT_PART="${ROOT_BLK##*[^0-9]}"
parted -f -s "${ROOT_DISK}" \
resizepart "${ROOT_PART}" 100%
mount_root done
touch /etc/rootpt-resize
reboot
fi
exit 1
EOF
cat << "EOF" > /etc/uci-defaults/80-rootfs-resize
if [ ! -e /etc/rootfs-resize ] \
&& [ -e /etc/rootpt-resize ] \
&& type losetup > /dev/null \
&& type resize2fs > /dev/null \
&& lock -n /var/lock/root-resize
then
ROOT_BLK="$(readlink -f /sys/dev/block/"$(awk -e \
'$9=="/dev/root"{print $3}' /proc/self/mountinfo)")"
ROOT_DEV="/dev/${ROOT_BLK##*/}"
LOOP_DEV="$(awk -e '$5=="/overlay"{print $9}' \
/proc/self/mountinfo)"
if [ -z "${LOOP_DEV}" ]
then
LOOP_DEV="$(losetup -f)"
losetup "${LOOP_DEV}" "${ROOT_DEV}"
fi
resize2fs -f "${LOOP_DEV}"
mount_root done
touch /etc/rootfs-resize
reboot
fi
exit 1
EOF
cat << "EOF" >> /etc/sysupgrade.conf
/etc/uci-defaults/70-rootpt-resize
/etc/uci-defaults/80-rootfs-resize
EOF

; 或者用wget获取自动扩容脚本
# wget -U "" -O expand-root.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/expand_root?codeblock=0"
; 执行扩容脚本
# . ./expand-root.sh
; 安装需要的软件包
# opkg update
# opkg install parted losetup resize2fs
; 开始进行扩容, 这时候会自动重启, 等待扩容完毕即可.
# sh /etc/uci-defaults/70-rootpt-resize

至此OpenWrt已经创建完毕, 剩下就是更新源、安装简体中文语言支持以及安装需要的功能了.

# opkg update
# opkg install luci-i18n-base-zh-cn install luci-i18n-package-manager-zh-cn

; WOL网卡唤醒APP
# opkg install luci-i18n-wol-zh-cn

; 络流量监视器,它使用内核提供的网络接口统计信息
# opkg install luci-i18n-vnstat2-zh-cn

; 通用即插即用UPnP（端口自动转发）
# opkg install luci-i18n-upnp-zh-cn

; BT下载工具
# opkg install luci-i18n-transmission-zh-cn

; 流量监控工具
# opkg install luci-i18n-statistics-zh-cn

; 网络共享（Samba4）
# opkg install luci-i18n-samba4-zh-cn

; MWAN3负载均衡
# opkg install luci-i18n-mwan3-zh-cn

; 网页文件管理器
# opkg install luci-i18n-filebrowser-zh-cn

; 动态域名 DNS
# opkg install luci-i18n-ddns-zh-cn

; Aria2下载工具
# opkg install luci-i18n-aria2-zh-cn

; BanIP
# opkg install luci-i18n-banip-zh-cn

最后秀一把深水宝的双模光模块。

丁丁的 BLOG

心态决定感觉，感觉决定行为

月度归档： 2025 年 3 月

AIO之万兆网卡直通OpenWrt安装

超微X9DR3-F、X11SSH-LN4F主板开机执行自定义风扇PWM脚本