bbqdt

Sorry, I was going off of release notes. Just updated and checked, and you are correct, they are included. Also, this works now with stock kernel.

@limetech You seemed to have missed CONFIG_MLX5_ACCEL=y CONFIG_MLX5_ESWITCH=y CONFIG_MLX5_BRIDGE=y CONFIG_MLX5_CLS_ACT=y CONFIG_MLX5_TC_SAMPLE= CONFIG_MLX5_EN_TLS=y CONFIG_MLXSW_SPECTRUM=m Was that intentional?

Bump

Anyone? @limetech ? This would be pretty easy to turn on.

Here are the main additions to the .config for the kernel build needed for this - CONFIG_TLS=y CONFIG_TLS_DEVICE=y CONFIG_NET_SWITCHDEV=y CONFIG_MLX5_ACCEL=y CONFIG_MLX5_ESWITCH=y CONFIG_MLX5_BRIDGE=y CONFIG_MLX5_CLS_ACT=y CONFIG_MLX5_TC_SAMPLE=y CONFIG_MLX5_TLS=y CONFIG_MLX5_EN_TLS=y CONFIG_MLXSW_SPECTRUM=m

@limetech Please enable the following kernel build configs to fix this - required - CONFIG_MLX5_ESWITCH=y please also add for nice-to-have CONFIG_MLX5_EN_TLS=y

So it seems like this is a problem with unraid. I booted from an ubuntu flash drive and it works fine - I will file a bug. I am not really sure where the problem is but it seems like the kernel.

I have enabled sr-iov and added vfs in the driver properly per the instructions The problem is that after I add a virtual functions via `sriov_numvfs` I cannot seem to configure them nor use them in KVM. Here is my current state - # lspci | grep X-4 83:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] 83:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] # ip link show eth1 15: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 98:03:9b:94:8d:30 brd ff:ff:ff:ff:ff:ff Note that the VF does not show under eth1, it shows as eth2: # ip link show eth2 79: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 56:17:3a:18:05:d6 brd ff:ff:ff:ff:ff:ff If I unbind the vf via `/sys/bus/pci/drivers/mlx5_core/unbind` it does not show at all under `ip link`. With the vf bound _or_ unbound I get the following when trying to do _any_ ip link set operations on it - # ip link set eth4 vf 0 mac 52:54:00:96:19:c1 RTNETLINK answers: Operation not supported Another interesting point is that if I try to use it in kvm with an `<interface>` element I get this error when trying to start the vm - `error: internal error: missing IFLA_VF_INFO in netlink response`. If I use a `<hostdev>` element I can start the vm and it seems to recognize it but network traffic does not work. Am I missing some config somewhere? Do I need to set some param somewhere that I am missing? System info - # uname -a Linux staton-nas 5.15.43-Unraid #1 SMP Wed May 25 11:54:05 PDT 2022 x86_64 Intel(R) Xeon(R) CPU E5-2648L v2 @ 1.90GHz GenuineIntel GNU/Linux # ip -V ip utility, iproute2-5.16.0 # virsh -V Virsh command line tool of libvirt 8.2.0 See web site at https://libvirt.org/ Compiled with support for: Hypervisors: QEMU/KVM Cloud-Hypervisor Networking: Remote Network Bridging Interface udev Nwfilter Storage: Dir Miscellaneous: Daemon Nodedev Secrets Debug # mstconfig -d 83:00.0 q Device #1: ---------- Device type: ConnectX4LX Name: MCX4431A-GCA_Bx Description: ConnectX-4 Lx EN network interface card for OCP; 50GbE single-port QSFP28; PCIe3.0 x8; ROHS R6 Device: 83:00.0 Configurations: Next Boot MEMIC_BAR_SIZE 0 MEMIC_SIZE_LIMIT _256KB(1) FLEX_PARSER_PROFILE_ENABLE 0 FLEX_IPV4_OVER_VXLAN_PORT 0 ROCE_NEXT_PROTOCOL 254 PF_NUM_OF_VF_VALID False(0) NON_PREFETCHABLE_PF_BAR False(0) VF_VPD_ENABLE False(0) STRICT_VF_MSIX_NUM False(0) VF_NODNIC_ENABLE False(0) NUM_PF_MSIX_VALID True(1) NUM_OF_VFS 20 NUM_OF_PF 1 SRIOV_EN True(1) PF_LOG_BAR_SIZE 5 VF_LOG_BAR_SIZE 0 NUM_PF_MSIX 63 NUM_VF_MSIX 11 INT_LOG_MAX_PAYLOAD_SIZE AUTOMATIC(0) PCIE_CREDIT_TOKEN_TIMEOUT 0 ACCURATE_TX_SCHEDULER False(0) PARTIAL_RESET_EN False(0) SW_RECOVERY_ON_ERRORS False(0) RESET_WITH_HOST_ON_ERRORS False(0) PCI_BUS0_RESTRICT_SPEED PCI_GEN_1(0) PCI_BUS0_RESTRICT_ASPM False(0) PCI_BUS0_RESTRICT_WIDTH PCI_X1(0) PCI_BUS0_RESTRICT False(0) PCI_DOWNSTREAM_PORT_OWNER Array[0..15] CQE_COMPRESSION BALANCED(0) IP_OVER_VXLAN_EN False(0) MKEY_BY_NAME False(0) UCTX_EN True(1) PCI_ATOMIC_MODE PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0) TUNNEL_ECN_COPY_DISABLE False(0) LRO_LOG_TIMEOUT0 6 LRO_LOG_TIMEOUT1 7 LRO_LOG_TIMEOUT2 8 LRO_LOG_TIMEOUT3 13 ICM_CACHE_MODE DEVICE_DEFAULT(0) TX_SCHEDULER_BURST 0 LOG_MAX_QUEUE 17 LOG_DCR_HASH_TABLE_SIZE 14 DCR_LIFO_SIZE 16384 ROCE_CC_PRIO_MASK_P1 255 CLAMP_TGT_RATE_AFTER_TIME_INC_P1 True(1) CLAMP_TGT_RATE_P1 False(0) RPG_TIME_RESET_P1 300 RPG_BYTE_RESET_P1 32767 RPG_THRESHOLD_P1 1 RPG_MAX_RATE_P1 0 RPG_AI_RATE_P1 5 RPG_HAI_RATE_P1 50 RPG_GD_P1 11 RPG_MIN_DEC_FAC_P1 50 RPG_MIN_RATE_P1 1 RATE_TO_SET_ON_FIRST_CNP_P1 0 DCE_TCP_G_P1 1019 DCE_TCP_RTT_P1 1 RATE_REDUCE_MONITOR_PERIOD_P1 4 INITIAL_ALPHA_VALUE_P1 1023 MIN_TIME_BETWEEN_CNPS_P1 4 CNP_802P_PRIO_P1 6 CNP_DSCP_P1 48 LLDP_NB_DCBX_P1 False(0) LLDP_NB_RX_MODE_P1 OFF(0) LLDP_NB_TX_MODE_P1 OFF(0) DCBX_IEEE_P1 True(1) DCBX_CEE_P1 True(1) DCBX_WILLING_P1 True(1) KEEP_ETH_LINK_UP_P1 True(1) KEEP_IB_LINK_UP_P1 False(0) KEEP_LINK_UP_ON_BOOT_P1 False(0) KEEP_LINK_UP_ON_STANDBY_P1 False(0) DO_NOT_CLEAR_PORT_STATS_P1 False(0) AUTO_POWER_SAVE_LINK_DOWN_P1 False(0) NUM_OF_VL_P1 _4_VLs(3) NUM_OF_TC_P1 _8_TCs(0) NUM_OF_PFC_P1 8 VL15_BUFFER_SIZE_P1 0 DUP_MAC_ACTION_P1 LAST_CFG(0) SRIOV_IB_ROUTING_MODE_P1 LID(1) IB_ROUTING_MODE_P1 LID(1) PHY_FEC_OVERRIDE_P1 DEVICE_DEFAULT(0) WOL_MAGIC_EN True(1) ROCE_CONTROL ROCE_ENABLE(2) PCI_WR_ORDERING per_mkey(0) MULTI_PORT_VHCA_EN False(0) PORT_OWNER True(1) ALLOW_RD_COUNTERS True(1) RENEG_ON_CHANGE True(1) TRACER_ENABLE True(1) IP_VER IPv4(0) BOOT_UNDI_NETWORK_WAIT 0 UEFI_HII_EN True(1) BOOT_DBG_LOG False(0) UEFI_LOGS DISABLED(0) BOOT_VLAN 1 LEGACY_BOOT_PROTOCOL PXE(1) BOOT_RETRY_CNT NONE(0) BOOT_INTERRUPT_DIS False(0) BOOT_LACP_DIS True(1) BOOT_VLAN_EN False(0) BOOT_PKEY 0 DYNAMIC_VF_MSIX_TABLE False(0) EXP_ROM_UEFI_ARM_ENABLE False(0) EXP_ROM_UEFI_x86_ENABLE False(0) EXP_ROM_PXE_ENABLE True(1) ADVANCED_PCI_SETTINGS False(0) SAFE_MODE_THRESHOLD 10 SAFE_MODE_ENABLE True(1) # mstflint -d 83:00.0 q Image type: FS3 FW Version: 14.32.1010 FW Release Date: 1.12.2021 Product Version: 14.32.1010 Rom Info: type=UEFI version=14.25.17 cpu=AMD64,AARCH64 type=PXE version=3.6.502 cpu=AMD64 Description: UID GuidsNumber Base GUID: 98039b0300948d30 4 Base MAC: 98039b948d30 4 Image VSD: N/A Device VSD: N/A PSID: MT_2490111032 Security Attributes: N/A Also note that my setup works fine with my connectx-3 card.

Seems there is also an issue with deletions. If I’m running mover on a large dir with lots of nested dirs and files, and rm -rf it, stuff breaks. rm command fails and mover seems to freak out a little.

Was this solved? I ask because I’m considering adding something to my cache that could easily get opened after mover started to move it. I was looking at the original mover script with rsync and it’s obvious it would not handle this. The latest mover script seems to use something else that is UnRAID internal, so I hope it is fixed.

memmap works just fine, bad advice above https://imgur.com/a/XTjg6qU