turnipisum

Great news! hopefully you'll be all good then. 👍

VM setup it's been a while lol. As far as i can remember i passed 2070 super, audio and usb through when i created the vm's. I had to mess about a bit to find the correct usb's for each vm as i have 2 vm's with 2070 supers. Make sure you have enabled all the options in BIOS for virtual machine, iommu etc... My setup is as below if it helps Unraid boot: Legacy VM Manager Libvirt version: 6.5.0 QEMU version: 5.1.0 Libvirt storage location: /mnt/user/system/libvirt/libvirt.img Default VM storage path: /mnt/user/domains/ Default ISO storage path: /mnt/user/isos/ Default Windows VirtIO driver ISO (optional): virtio-win-0.1.189-1.iso Default network bridge: br0 Upon host shutdown: Shutdown PCIe ACS override: Multi-function VFIO allow unsafe interrupts: No VM Windows 10 CPU Mode: Host Passthrough (AMD Ryzen Threadripper 3970X 32-Core) Logical CPUs: Initial Memory: 32256MB Machine: i440fx-5.1 BIOS: OVMF Hyper-V: Yes USB Controller: 3.0 (qemu XHCI) OS Install ISO: /mnt/user/isos/Windows.iso OS Install CDRom Bus: IDE VirtIO Drivers ISO: /mnt/user/isos/virtio-win-0.1.189-1.iso VirtIO Drivers CDRom Bus: IDE Primary vDisk Location: Manual /mnt/disks/Force_MP600_XXXX/Windows 10/vdisk1.img Primary vDisk Bus: VirtIO Graphics Card: NVIDIA GeForce RTX 2070 SUPER (4d:00.0) Graphics ROM BIOS: Sound Card: NVIDIA TU104 HD Audio Controller (4d:00.1) Network MAC: XXXXXXX Network Bridge: br0 Network Model: virtio-netvirtio Other PCI Devices: AMD Starship USB 3.0 Host Controller | USB controller (03:00.3) NVIDIA TU104 USB 3.1 Host Controller | USB controller (4d:00.2) NVIDIA TU104 USB Type-C UCSI Controller | Serial bus controller (4d:00.3) Hope it helps

Last thing i can think of to try is set "VFIO allow unsafe interrupts: " to yes in vm manager if your still getting nowhere.

Ok what about bios then is set for iommu? what motherboard is it?

The creation error in your first post is related to the iommu group 14 which is the disk your trying to pass through is it not? So usb might be working but it just doesn't get past the disk issue.

https://forum.level1techs.com/t/is-it-even-possible-to-pass-through-my-primary-gpu-what-am-i-doing-wrong/157588/3

Also maybe try machine as i440fx

Have you got all parts of the 3090 card selected in iommu/vfio sound any usb etc...

You could pass it as vdisk or try and pass it via unassigned devices via dev/mnt... Also are you editing existing vm setup? If so have you tried new vm setup with what you want.

Nothing new other than than the c-states, typical current idle or memory speed tricks to try. Yes i'm on beta 35 now still having issues! I'm leaning to kernel issue or Nvidia drivers on the two 2070 supers on my vm's maybe. tried a lot of tweaks losing track now lol. I have seen posts on other forum's about bare metal ryzen and linux rig's having lock up's as well. I'm hoping that 6.9.0 release will solve it but who knows.

It works just doesn't auto update. What can i change there is no settings for the plugin. What UR version are you on?

This is not refreshing the data on mine unless i reload dashboard page. RM1000i

Update! so looking in my bios again i had power supply set to "low current idle" so i've now set it to "typical current idle" and got almost 4 days uptime so far! 🤞 Also i did some logging of psu usage just in case. most i've seen while gaming on 2 vm's with 2070 supers on each one was 760 watts so don't think i'm hitting limit on the hx1000i but i am going get a 1600i as soon as i can but at £460 it's gonna have to wait to new year as i will need bigger ups as well another £700-1k 🤪

You trying to use a vdisk image or just pass nvme through? as i didn't see path to drive in config. I use unassigned devices with vdisk for my nvme if it helps.

I've got 8x 16gb strips running at 2133mhz so well within spec. I was running it at 2666mhz which is still in with suggest max. Dropping ram speed was one of the first things i did when i started getting issues as well as memory test.

Yep given it a go! I've disabled c-states and set power to typical idle but still no cigar 🚬🤪

@Squid or @limetech any ideas on any of this? It would seem more than a few people are getting lock up's lately that i have seen in the community. Could this be a AMD chipset issue with kernel on beta builds?

Oh crap i don't get it that many times a day revert back to a beta that worked for you. mines random can be 3-5 days a day or like 12 days longest up on beta 35 is 17 days i think. What is your server hardware?

It's possible @ich777 that it's a bit close on the psu but i have ups monitor on in UR gui and i've not seen it above 560 watts as yet. rounding the spec draw figures the 3970x max 300w plus 300w each for the two 2070's =900w maybe do with more scope after adding other bits into the mix. i might put the 2070's on separate 750 psu that i have spare then see what happens as a 1200-1600 watt psu are fair few quid more! 🤪 Bit of a pain not to have the other vm's running as they are all needed sort of living with the short down time when it happens while i try and track it down. Thanks for the suggestion.

Hi @ich777 as below. I'm starting to think maybe it could be i440x(currently in use) vs q35 issue but just guessing really. I have 3 x win 10 vm's 1x vnc only the other two have 2070 supers passed through and usb 3. Case: Corsair Obsidian 750d | MB: Asrock Trx40 Creator | CPU: AMD Threadripper 3970X | Cooler: Noctua NH-U14S | RAM: Corsair LPX 128GB DDR4 C16 | GPU: 2 x MSI RTX 2070 Super's | Cache: Intel 660p Series 1TB M.2 X2 in 2TB Pool | Parity: Ironwolf 6TB | Array Storage: Ironwolf 6TB + Ironwolf 4TB | Unassigned Devices: Corsair 660p M.2 1TB + Kingston 480GB SSD + Skyhawk 2TB | NIC: Intel 82576 Chip, Dual RJ45 Ports, 1Gbit PCI | PSU: Corsair RM1000i

Nope that didn't work just had one again. ------------[ cut here ]------------ Nov 27 00:24:47 10.0.0.31 kernel: WARNING: CPU: 6 PID: 7646 at drivers/iommu/dma-iommu.c:471 __iommu_dma_unmap+0x7a/0xe8 Nov 27 00:24:47 10.0.0.31 kernel: Modules linked in: nfsd lockd grace sunrpc md_mod nct6683 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha bonding r8169 realtek atlantic igb i2c_algo_bit wmi_bmof mxm_wmi edac_mce_amd kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd r8125(O) glue_helper ahci rapl nvme libahci nvme_core ccp k10temp i2c_piix4 i2c_core wmi button acpi_cpufreq [last unloaded: realtek] Nov 27 00:24:47 10.0.0.31 kernel: CPU: 6 PID: 7646 Comm: ethtool Tainted: G O 5.8.18-Unraid #1 Nov 27 00:24:47 10.0.0.31 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40 Creator, BIOS P1.70 05/29/2020 Nov 27 00:24:47 10.0.0.31 kernel: RIP: 0010:__iommu_dma_unmap+0x7a/0xe8 Nov 27 00:24:47 10.0.0.31 kernel: Code: 46 28 4c 8d 60 ff 48 8d 54 18 ff 49 21 ec 48 f7 d8 4c 29 e5 49 01 d4 49 21 c4 48 89 ee 4c 89 e2 e8 c3 de ff ff 4c 39 e0 74 02 <0f> 0b 49 83 be 68 07 00 00 00 75 32 49 8b 45 08 48 8b 40 48 48 85 Nov 27 00:24:47 10.0.0.31 kernel: RSP: 0018:ffffc90001afba40 EFLAGS: 00010206 Nov 27 00:24:47 10.0.0.31 kernel: RAX: 0000000000002000 RBX: 0000000000001000 RCX: 0000000000000001 Nov 27 00:24:47 10.0.0.31 kernel: RDX: ffff889fd571ae20 RSI: ffffffffffffe000 RDI: 0000000000000009 Nov 27 00:24:47 10.0.0.31 kernel: RBP: 00000000fed8e000 R08: ffff889fd571ae20 R09: ffff889f86573c70 Nov 27 00:24:47 10.0.0.31 kernel: R10: 0000000000000009 R11: ffff888000000000 R12: 0000000000001000 Nov 27 00:24:47 10.0.0.31 kernel: R13: ffff889fd571ae10 R14: ffff889f9c979800 R15: ffffffffa0170600 Nov 27 00:24:47 10.0.0.31 kernel: FS: 000014a5efd0b740(0000) GS:ffff889fdd180000(0000) knlGS:0000000000000000 Nov 27 00:24:47 10.0.0.31 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 27 00:24:47 10.0.0.31 kernel: CR2: 000014a5efd9ff30 CR3: 0000001f80272000 CR4: 0000000000340ee0 Nov 27 00:24:47 10.0.0.31 kernel: Call Trace: Nov 27 00:24:47 10.0.0.31 kernel: iommu_dma_free+0x1a/0x2b Nov 27 00:24:47 10.0.0.31 kernel: aq_ptp_ring_free+0x31/0x60 [atlantic] Nov 27 00:24:47 10.0.0.31 kernel: aq_nic_deinit+0x4e/0xa4 [atlantic] Nov 27 00:24:47 10.0.0.31 kernel: aq_ndev_close+0x26/0x2d [atlantic] Nov 27 00:24:47 10.0.0.31 kernel: __dev_close_many+0xa1/0xb5 Nov 27 00:24:47 10.0.0.31 kernel: dev_close_many+0x48/0xa6 Nov 27 00:24:47 10.0.0.31 kernel: dev_close+0x42/0x64 Nov 27 00:24:47 10.0.0.31 kernel: aq_set_ringparam+0x4c/0xc8 [atlantic] Nov 27 00:24:47 10.0.0.31 kernel: ethnl_set_rings+0x1fc/0x252 Nov 27 00:24:47 10.0.0.31 kernel: genl_rcv_msg+0x1d9/0x251 Nov 27 00:24:47 10.0.0.31 kernel: ? genlmsg_multicast_allns+0xea/0xea Nov 27 00:24:47 10.0.0.31 kernel: netlink_rcv_skb+0x7d/0xd1 Nov 27 00:24:47 10.0.0.31 kernel: genl_rcv+0x1f/0x2c Nov 27 00:24:47 10.0.0.31 kernel: netlink_unicast+0x10c/0x1a5 Nov 27 00:24:47 10.0.0.31 kernel: netlink_sendmsg+0x29d/0x2d3 Nov 27 00:24:47 10.0.0.31 kernel: sock_sendmsg_nosec+0x32/0x3c Nov 27 00:24:47 10.0.0.31 kernel: __sys_sendto+0xce/0x109 Nov 27 00:24:47 10.0.0.31 kernel: ? exc_page_fault+0x3e2/0x40c Nov 27 00:24:47 10.0.0.31 kernel: __x64_sys_sendto+0x20/0x23 Nov 27 00:24:47 10.0.0.31 kernel: do_syscall_64+0x7a/0x94 Nov 27 00:24:47 10.0.0.31 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 27 00:24:47 10.0.0.31 kernel: RIP: 0033:0x14a5efe25bc6 Nov 27 00:24:47 10.0.0.31 kernel: Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 11 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 72 c3 90 55 48 83 ec 30 44 89 4c 24 2c 4c 89 Nov 27 00:24:47 10.0.0.31 kernel: RSP: 002b:00007ffc7437e458 EFLAGS: 00000246 ORIG_RAX: 000000000000002c Nov 27 00:24:47 10.0.0.31 kernel: RAX: ffffffffffffffda RBX: 00007ffc7437e4d0 RCX: 000014a5efe25bc6 Nov 27 00:24:47 10.0.0.31 kernel: RDX: 000000000000002c RSI: 000000000046f3a0 RDI: 0000000000000004 Nov 27 00:24:47 10.0.0.31 kernel: RBP: 000000000046f2a0 R08: 000014a5efef61a0 R09: 000000000000000c Nov 27 00:24:47 10.0.0.31 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000000000046f340 Nov 27 00:24:47 10.0.0.31 kernel: R13: 000000000046f330 R14: 0000000000000000 R15: 000000000043504b Nov 27 00:24:47 10.0.0.31 kernel: ---[ end trace 87428ae110e59bcb ]---

3 days up time! so i'm hoping the c-state or power supply idle setting has done the trick. i will come back with update if i can get to 20 days up time or if it locks up again.

Just found the posts about power supply idle and c-states so trying that see what happens.

Anyone with any ideas on these crashes, I'm about ready sledge hammer it! 🔨

And again another crash about 24hrs apart this time.