RX 5700 XT cannot pass through at all

[Yros]

Hello,

I just finished my new build (Ryzen 3700x, 32gb RAM ECC, GTX 1080ti and 8 HDD (2x 6tb and 6x4tb)) and am now facing three different issues that are preventing me from going further.

 

1] GPU not working properly: this is the first and most critical issue I'm facing. When I start my server, I cannot even get into the unRAID GUI as all it displays is a blinking "_" on the top left corner of the screen. I have no problem starting unRAID normally (without GUI) and I can remotely access it through another device (smartphone, tablet or laptop). However, when I try to setup a VM and passthrough the GPU, then it's black screen all over again. I tried various methods (thank you Spaceinvader One for your excellent YouTube videos) to make it work, I dumped the GPU bios, linked it properly on the VM, and so on. I noticed that I managed to start the VM and install it 'properly' via VNC graphic display (though with a huge response delay) but in direct passthrough, nothing, just the black screen. When checking the VM log, I can see the same error line:

2020-01-05T17:59:08.926831Z qemu-system-x86_64: vfio_region_write(0000:09:00.0:region1+0x1e81c0, 0x0,8) failed: Device or resource busy
2020-01-05T17:59:08.926839Z qemu-system-x86_64: vfio_region_write(0000:09:00.0:region1+0x1e81b8, 0xffffff,8) failed: Device or resource busy

I also tried with a Manjaro and a LinuxMint VM but the result is the same.

 

My 2nd issue may be related to this: my Logs are reaching 100% very quickly soon after I boot. It may be related to this thread: 

In the Tools > System Logs, I also have this error:

Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 4096 bytes) in /usr/local/emhttp/plugins/dynamix/include/Syslog.php on line 20

 

My 3rd and final 'issue' is related to the disk setup. I have currently 4 disks (2 of 6tb and 2 of 4tb) (+2 500g SSDs for cache pool but that's irrelevant I think). My issue is that when I try to add the 2nd 6TB HDD into the disks, the array refuses to start and it tells me something along the lines of 'you cannot add any more drives'. I don't really understant why it would refuse to mount the 6tb HDD into the array and only allows it into the parity drive despite the fact that there is already another 6tb HDD as parity in place and since both are the same size it shouldn't be a problem, right ? My default filesystem is 'xfs' (even though in the settings it says that it should be btrfs but w.e). I don't know if this is why ?

EDIT: Found the reason for that one: you cannot add a new disk to an array that is larger than the current disks until the parity is successfully rebuilt. So I just waited for the parity to be done and it now works just fine.

 

 

Please find the logs and diagnostics below and thank you in advance for any help provided.

yros-diagnostics-20200105-1950.zip yros-syslog-20200105-1851.zip

Edited January 9, 2020 by Yros
Modifying the title to meet current state

[FreeMan]

Often, logs filling quickly is related to misconfigured dockers. I haven't made any recent changes that do logging (that I'm aware of) or that I had to set any docker/Unraid path mappings on, so that's not likely it. I'm not implying that you've made mistakes in that area, but it's definitely something to look at. 

 

Go to Tools | Diagnostics and post the entire downloaded ZIP file to your next post - someone will stop by sooner or later to parse through it to give you some pointers - that's way above my pay grade!

 

Finally, you said 

Quote

three different issues

but only posted two. Did you miss detailing one or did you mistype your count?  

Edited January 5, 2020 by FreeMan

[Yros]

Sorry, the thread was posted before I finished writing it so I was editing it to provide the related informations ^^ As far as dockers are concerned, this is a brand new installation so the only docker I have installed at the moment is Krusader.

[FreeMan]

Unraid's Plus licence:

Allows 6 disks

 

The Pro license:

Allows many, many more. Which license are you running?

 

Beyond that and the tid-bits I've given so far, I'm beyond my pay grade. I'm sure someone else will chime in to help you soon.

 

(I see you did add your diagnostics.zip to the first post - usually, it's best to add it to a new post at the end, otherwise your thread might not get bumped back up top for people to notice, however, the additional discussion after the edit will have taken care of that. Also, the syslog is included in the diagnostics, so no need to post it separately. No need to remove it at this point, just a time saving note for the future.)

[Yros]

Currently with trial key. Also it shouldn't be the reason since I can add the HDD as a parity drive but it refuses to add it as a normal drive, which is strange. As for the syslog, I had a doubt considering how the zip file was twice bigger than the full diagnostic '-' Thanks anyway for the help ^^

[Yros]

Ok so change of plans : my previous graphic card ended up having some hardware issues so I ditched it in favor of a brand new AMD RX 5700 XT GPU. Now I can finally reach unRaid in GUI mode, but GPU passthrough is still a problem.

 

In system devices, I get the following IOMMU grouping :

IOMMU group 27:	[1002:1478] 09:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1)
IOMMU group 28:	[1002:1479] 0a:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch
IOMMU group 29:	[1002:731f] 0b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev c1)
IOMMU group 30:	[1002:ab38] 0b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio

In my VM, the GPU devices being passed are group 29 for the graphics and 30 for the audio (so the last two lines of the code above)...
 

When I start a VM, I can reach the 'boot' part (like Linux boot to choose whether or not I start from the ISO, or install the system like Manjaro) but the moment I try to enter and reach the next step, black screen. The logs spam the following warning:
 

Jan  9 07:45:06 Yros kernel: vfio-pci 0000:0b:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref]

and sometimes it's the usual 'device is already in use'.
 

I'm also aware of the AMD reset bug (which has a partial fix implemented in the custom 5.3 linux kernel available for unraid 6.8.0-rc5, which I move onto for that reason) but it seems to no avail. I've also added this to the syslinux configuration :
append vfio-pci.ids=1002:731f,1002:ab38,1002:1478,1002:1479 intel_iommu=on amd_iommu=on 

But I think I made two mistakes there: first, I'm on AMD, and iommu is already active from the BIOS so I don't think I need it either way. Second, I added the 4 AMD Navi devices (including upstream/downstream) but I'm not sure if that was necessary. Especially considering they're already in their independant iommu groups so I 'probably' don't need to use the vfio-pci.ids ? Rather, I should maybe try a pci.stub to prevent unraid from using it in order to pass it through? But that's a bit problematic considering how I have only one GPU and it's in the primary slot and I'd like to be able to boot my computer/server in GUI mode to access unraid and then start a VM at any point directly with GPU passed-through.