bbqdt

So it seems like this is a problem with unraid. I booted from an ubuntu flash drive and it works fine - I will file a bug. I am not really sure where the problem is but it seems like the kernel.

I have enabled sr-iov and added vfs in the driver properly per the instructions The problem is that after I add a virtual functions via `sriov_numvfs` I cannot seem to configure them nor use them in KVM. Here is my current state - # lspci | grep X-4 83:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] 83:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] # ip link show eth1 15: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 98:03:9b:94:8d:30 brd ff:ff:ff:ff:ff:ff Note that the VF does not show under eth1, it shows as eth2: # ip link show eth2 79: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 56:17:3a:18:05:d6 brd ff:ff:ff:ff:ff:ff If I unbind the vf via `/sys/bus/pci/drivers/mlx5_core/unbind` it does not show at all under `ip link`. With the vf bound _or_ unbound I get the following when trying to do _any_ ip link set operations on it - # ip link set eth4 vf 0 mac 52:54:00:96:19:c1 RTNETLINK answers: Operation not supported Another interesting point is that if I try to use it in kvm with an `<interface>` element I get this error when trying to start the vm - `error: internal error: missing IFLA_VF_INFO in netlink response`. If I use a `<hostdev>` element I can start the vm and it seems to recognize it but network traffic does not work. Am I missing some config somewhere? Do I need to set some param somewhere that I am missing? System info - # uname -a Linux staton-nas 5.15.43-Unraid #1 SMP Wed May 25 11:54:05 PDT 2022 x86_64 Intel(R) Xeon(R) CPU E5-2648L v2 @ 1.90GHz GenuineIntel GNU/Linux # ip -V ip utility, iproute2-5.16.0 # virsh -V Virsh command line tool of libvirt 8.2.0 See web site at https://libvirt.org/ Compiled with support for: Hypervisors: QEMU/KVM Cloud-Hypervisor Networking: Remote Network Bridging Interface udev Nwfilter Storage: Dir Miscellaneous: Daemon Nodedev Secrets Debug # mstconfig -d 83:00.0 q Device #1: ---------- Device type: ConnectX4LX Name: MCX4431A-GCA_Bx Description: ConnectX-4 Lx EN network interface card for OCP; 50GbE single-port QSFP28; PCIe3.0 x8; ROHS R6 Device: 83:00.0 Configurations: Next Boot MEMIC_BAR_SIZE 0 MEMIC_SIZE_LIMIT _256KB(1) FLEX_PARSER_PROFILE_ENABLE 0 FLEX_IPV4_OVER_VXLAN_PORT 0 ROCE_NEXT_PROTOCOL 254 PF_NUM_OF_VF_VALID False(0) NON_PREFETCHABLE_PF_BAR False(0) VF_VPD_ENABLE False(0) STRICT_VF_MSIX_NUM False(0) VF_NODNIC_ENABLE False(0) NUM_PF_MSIX_VALID True(1) NUM_OF_VFS 20 NUM_OF_PF 1 SRIOV_EN True(1) PF_LOG_BAR_SIZE 5 VF_LOG_BAR_SIZE 0 NUM_PF_MSIX 63 NUM_VF_MSIX 11 INT_LOG_MAX_PAYLOAD_SIZE AUTOMATIC(0) PCIE_CREDIT_TOKEN_TIMEOUT 0 ACCURATE_TX_SCHEDULER False(0) PARTIAL_RESET_EN False(0) SW_RECOVERY_ON_ERRORS False(0) RESET_WITH_HOST_ON_ERRORS False(0) PCI_BUS0_RESTRICT_SPEED PCI_GEN_1(0) PCI_BUS0_RESTRICT_ASPM False(0) PCI_BUS0_RESTRICT_WIDTH PCI_X1(0) PCI_BUS0_RESTRICT False(0) PCI_DOWNSTREAM_PORT_OWNER Array[0..15] CQE_COMPRESSION BALANCED(0) IP_OVER_VXLAN_EN False(0) MKEY_BY_NAME False(0) UCTX_EN True(1) PCI_ATOMIC_MODE PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0) TUNNEL_ECN_COPY_DISABLE False(0) LRO_LOG_TIMEOUT0 6 LRO_LOG_TIMEOUT1 7 LRO_LOG_TIMEOUT2 8 LRO_LOG_TIMEOUT3 13 ICM_CACHE_MODE DEVICE_DEFAULT(0) TX_SCHEDULER_BURST 0 LOG_MAX_QUEUE 17 LOG_DCR_HASH_TABLE_SIZE 14 DCR_LIFO_SIZE 16384 ROCE_CC_PRIO_MASK_P1 255 CLAMP_TGT_RATE_AFTER_TIME_INC_P1 True(1) CLAMP_TGT_RATE_P1 False(0) RPG_TIME_RESET_P1 300 RPG_BYTE_RESET_P1 32767 RPG_THRESHOLD_P1 1 RPG_MAX_RATE_P1 0 RPG_AI_RATE_P1 5 RPG_HAI_RATE_P1 50 RPG_GD_P1 11 RPG_MIN_DEC_FAC_P1 50 RPG_MIN_RATE_P1 1 RATE_TO_SET_ON_FIRST_CNP_P1 0 DCE_TCP_G_P1 1019 DCE_TCP_RTT_P1 1 RATE_REDUCE_MONITOR_PERIOD_P1 4 INITIAL_ALPHA_VALUE_P1 1023 MIN_TIME_BETWEEN_CNPS_P1 4 CNP_802P_PRIO_P1 6 CNP_DSCP_P1 48 LLDP_NB_DCBX_P1 False(0) LLDP_NB_RX_MODE_P1 OFF(0) LLDP_NB_TX_MODE_P1 OFF(0) DCBX_IEEE_P1 True(1) DCBX_CEE_P1 True(1) DCBX_WILLING_P1 True(1) KEEP_ETH_LINK_UP_P1 True(1) KEEP_IB_LINK_UP_P1 False(0) KEEP_LINK_UP_ON_BOOT_P1 False(0) KEEP_LINK_UP_ON_STANDBY_P1 False(0) DO_NOT_CLEAR_PORT_STATS_P1 False(0) AUTO_POWER_SAVE_LINK_DOWN_P1 False(0) NUM_OF_VL_P1 _4_VLs(3) NUM_OF_TC_P1 _8_TCs(0) NUM_OF_PFC_P1 8 VL15_BUFFER_SIZE_P1 0 DUP_MAC_ACTION_P1 LAST_CFG(0) SRIOV_IB_ROUTING_MODE_P1 LID(1) IB_ROUTING_MODE_P1 LID(1) PHY_FEC_OVERRIDE_P1 DEVICE_DEFAULT(0) WOL_MAGIC_EN True(1) ROCE_CONTROL ROCE_ENABLE(2) PCI_WR_ORDERING per_mkey(0) MULTI_PORT_VHCA_EN False(0) PORT_OWNER True(1) ALLOW_RD_COUNTERS True(1) RENEG_ON_CHANGE True(1) TRACER_ENABLE True(1) IP_VER IPv4(0) BOOT_UNDI_NETWORK_WAIT 0 UEFI_HII_EN True(1) BOOT_DBG_LOG False(0) UEFI_LOGS DISABLED(0) BOOT_VLAN 1 LEGACY_BOOT_PROTOCOL PXE(1) BOOT_RETRY_CNT NONE(0) BOOT_INTERRUPT_DIS False(0) BOOT_LACP_DIS True(1) BOOT_VLAN_EN False(0) BOOT_PKEY 0 DYNAMIC_VF_MSIX_TABLE False(0) EXP_ROM_UEFI_ARM_ENABLE False(0) EXP_ROM_UEFI_x86_ENABLE False(0) EXP_ROM_PXE_ENABLE True(1) ADVANCED_PCI_SETTINGS False(0) SAFE_MODE_THRESHOLD 10 SAFE_MODE_ENABLE True(1) # mstflint -d 83:00.0 q Image type: FS3 FW Version: 14.32.1010 FW Release Date: 1.12.2021 Product Version: 14.32.1010 Rom Info: type=UEFI version=14.25.17 cpu=AMD64,AARCH64 type=PXE version=3.6.502 cpu=AMD64 Description: UID GuidsNumber Base GUID: 98039b0300948d30 4 Base MAC: 98039b948d30 4 Image VSD: N/A Device VSD: N/A PSID: MT_2490111032 Security Attributes: N/A Also note that my setup works fine with my connectx-3 card.