JKunraid

Seems to have worked. Thanks so much.

I created a new USB key using the Unraid software utility. The system boots but of course the drive mappings and everything else isn't configured or a valid license.. Are you saying this can all be fixed by simply overwriting the /config folder on the new USB by copying over the /config folder on the old USB key?

Unraid server stopped booting after a cold restart. When looking at the boot sequence locally it does see the Unraid USB key but gets stuck during boot process (tried with GUI, without, and safe mode). I pulled out the USB key and put it in an Ubuntu machine to run a checkdisk using the built-in Disk app. It did flag a dirty bit. I ran the repair and it seemed to work but when I try to boot with it I once again get stuck during the boot process. I have a flash key config backup but it's out of date and I've updated the array with additional drives and done other changes since the old backup. Fortunately, I was able to copy all the files of the USB key to another computer. What would be the steps to recover to another USB key using the files from the old key?

I have a bunch of HDDs in a Netapp disk shelf. I'd like to passthrough one of them to a PopOS VM. I can see them in unassigned disk devices (able to mount and unmount it). I was able to passthrough a GPU and a couple of M.2 SSD drives by binding them in Unraid's system devices but I don't see this particular HDD in the list. How would I do the passthrough?

UPDATE: I did of search of devices with bus 43 (and 44) as you suggested using the broken XML. As the XML was nested I assumed the parent lines of XML were also required. I added these lines to working XML config just above the <memballon model='none'/> line. It booted and showed the two drives. I need to spend a bit of time making sure I haven't broken anything but as it stands it seems to be working.. I'll update as soon as I know for sure. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x43' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x44' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </hostdev>

Yes. A virtio vdisk is set to "1" on boot field. It has a working OS. The m.2 drives are for secondary drives not operating system... I have nearly the same setup on the simpler VM and i have no issue using the form view to add them in. It's the manually modified xml VM that has the issue. I could start from scratch again but I took hours to get the VM working for my particular configuration. It would be a lot simpler if I could add the lines into the xml manually.

Thanks again for reply Simon. Not following you though. I've already done the vfio bind in system devices and confirmed I can add the m.2 drives to another basic VM using Unraids VM template GUI. That VM boots fine and adds the drives. The issue is if I use the VM template tool in form view to add the m.2 drives to another VM that has a manually modified XML config template. My second VM boots fine. When I use to VM template form view to modify the config (rather than the config directly using the "xml text" toggle) it messes up my manually modified XML config. The system won't boot then. In other words, what I'm trying to figure out is what text lines do I need to add to my VM's XML configuration to get the two m.2 drives to do the passthrough? (i.e. rather than using the form view) (Edit:" The devices below are available as an option in the VM's form view but if I check the boxes and click update the VM doesn't boot. I have to repaste a backup of the xml config to get it to boot again) Other PCI Devices: AMD RAID Bottom Device | Non-Volatile memory controller (43:00.0) AMD RAID Bottom Device | Non-Volatile memory controller (44:00.0)

I want to passthrough a couple of M.2 SSDs. I am able to do the passthrough with a simple Linux VM configuration using the UI (confirmed with testing). When I try to do the passthrough to another Linux VM though, one that has some manual modifications done to its XML configuration file, it messes up the XML configuration and VM won't boot. I've seen this Unraid bug before where if you switch back to UI mode and save after manually modifying VM's XML config it can mess it up. So I backed it up and was able to recover but I'm stuck as to how to add the passthrough for the m.2s manually to my xml file.

Got it. Thanks again.

I opened up the fsab file with nano. I've never done this before so I'm not sure what the exact line I should add looks like.

I just realized I have one more related question. How would I keep the connection to share persistent across reboots?

Perfect. Thanks.

Setup following virtiofs settings when I created a PopOS VM (settings selected from dropdown). Unraid Share Mode: Virtiofs Mode Unraid Share: user: vmShare Unraid Source Path: /mnt/user/vmShare Unraid Mount Tag: vmShare The VM boots fine but I can't see the share named "vmShare". either in locations, VM's mnt folder, or using Disks utility. Is there some step I'm missing or is there a problem?

Unraid Share Mode: Virtiofs Mode Unraid Share: user: vmShare Unraid Source Path: /mnt/user/vmShare Unraid Mount Tag: vmShare

Success! My existing Ubuntu VM still doesn't see it. And I tried with multiple new templates of Ubuntu but I tried with your suggested PopOS VM and it recognizes the GPU. Thanks so much for the help.

Threadripper 3970x on Asus Zenith II Extreme Mobo. Bios is from 2022. There is a newer version so i'm going to give it a shot tomorrow. I created the diagnostic file. i have some privacy related concerns about it since it contains a lot of data. Is there any way I can send you specific files or folder? (I will review before sending) I'll give Pop OS a try tomorrow. Another Linux distro with an existing template in the vm setup after that. Big Brother AI spyware after that... umm I mean Windows.

Results of running (dmesg | grep -e DMAR -e IOMMU) on host system. [ 0.507012] pci 0000:60:00.2: AMD-Vi: IOMMU performance counters supported [ 0.508237] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported [ 0.513697] pci 0000:20:00.2: AMD-Vi: IOMMU performance counters supported [ 0.515574] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported [ 0.518364] pci 0000:60:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 0.518532] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 0.518699] pci 0000:20:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 0.518867] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 0.521408] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank). [ 0.521522] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank). [ 0.521637] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank). [ 0.521747] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank). [ 1.748930] AMD-Vi: AMD IOMMUv2 loaded and initialized Results of (dmesg | grep 'remapping') on host, 0.519033] AMD-Vi: Interrupt remapping enabled Installed "disable security mitigations" with no luck. Is this enabled or disabled by default? You mention add line (options vfio_iommu_type1 allow_unsafe_interrupts=1) to grub. Am I supposed to append it to it's own line somewhere specific? (e.g append it to the line I previously added?) label GPU passthrough mode menu default kernel /bzimage append initrd=/bzroot video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init pci=noaer pcie_aspm=off pcie_acs_override=downstream,multifunction options vfio_iommu_type1 allow_unsafe_interrupts=1 btw - thanks for the help. Hopefully we can figure this out but even if not I'm learning a lot just trying different things.

The good news. The System boots fine. The bad news. VM still doesn't recognize the GPU. Any thoughts on what I should try next?

Thanks. I made the edit (also changing it to default menu option I think) Can you quickly review before I try rebooting in case I messed something up. default menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label Unraid OS kernel /bzimage append initrd=/bzroot label Unraid OS GUI Mode kernel /bzimage append initrd=/bzroot,/bzroot-gui label Unraid OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Unraid OS GUI Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest label GPU passthrough mode menu default kernel /bzimage append initrd=/bzroot video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init pci=noaer pcie_aspm=off pcie_acs_override=downstream,multifunction (btw- I'm on Unraid 6.12.8)

I tried to add line... "kernel /bzimage append initrd=/bzroot video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init" ...but couldn't find the grub file on flash drive. Where exactly is located in Unraid and its exact name?

I ran "lspci -v | grep vfio" (on host) and it returned following. Kernel driver in use: vfio-pci Kernel driver in use: vfio-pci I also ran "lspci -v | grep nvidia" Kernel modules: nvidia_drm, nvidia

I changed bus setting to "2" as you suggested but it wouldn't save XML ("attempted double use of PCI Address error). I was able to change it to "5" but it didn't make a difference. Out of curiousity are you on a system with two GPUs? My system only has one GPU the 3090. It has no IGPU.. The first reply (other poster) suggested that when my system boots the host system might be locking the GPU which is what is preventing the passthrough to VM. I didn't understand his instructions as to how to change that though.

I downloaded driver directly from Nvidia. During installation it doesn't recognize the existence of the 3090 not do I see it when I run various hardware utilities (I can see the GPU in Unraid host but not the VM). Also connected to Ubuntu VM with SSH as opposed to RDP but didn't make a difference.

Thanks for the reply. There is a lot to unpack so I'm trying to make sure I'm doing things right step by step. I made multifunction edit to my VM's config but doens't work yet. Does this at least look correct? (before I move on to something else) <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev>

My Ubuntu VM installed with no issues and boots fine. I am also able to RDP into it no problem (have XRDP installed). I'm trying to do a GPU passthrough with a rtx3090. I've checked below IOMMU group in tools > system devices (and rebooted) IOMMU group 84 : [10de:2204] 01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090] (rev a1) [10de:1aef] 01:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1) Then I selected "NVIDIA GeForce RTX 3090 (01:00.0)" in dropdown of "Graphics Card" field of my VM's configuration. When I try to install the Nvidia Linux driver it can't find the GPU. I installed inxi to check then ran "sudo inxi --full". Although Unraid OS recognizes it, my Ubuntu VM does not see my GPU. Any ideas on what I'm missing?