Pc specs:
MB: Asus TUF gaming X570 pro
Ram: G.Skill Trident Z Neo 2x 16GB 3600
                                        2x 32GB 3600
CPU: Ryzen 9 5900X
GPU: OLD- 9800 GTX+
        NEW- RTX 4070 SUPER OC
        TEST GPU- GTX 1080
PSU: OLD- OCZ 700 watt
        NEW- Corsair RM850X


This was supposed to be a quick and easy GPU swap, so i could install a VM for processing drone photogrammetry(cuda core needed).....Now ive been troubleshooting for 3 days without any good results. 

This PC has been running Unraid the last 2-3 years without and problems. Then after the swap from the Nvidia 9800 GTX+ (a card I've had for a really long time) to the RTX 4070 Super, now Unraid hangs on the initial boot from USB at random places in the boot process, depending if i choose standard boot, gui boot, safemode-non gui, or safe mode with gui. First i tried putting the old gpu back in place, but due to the dvi connection on that old gpu and not having a working monitor with dvi, i scrounged a gpu from the children's gaming pc, a gtx1080. Put that in place, booted up and was stable for a couple days. Then i decided the 20 year old USB drive UNRAID is on may be the issue. I was using the previous GPU to passthrough to a VM, before i swapped the GPU i unassigned the GPU from the VM and then updated the nvidia-driver plugin drivers to 565.57.01.

So, I have rebuilt the OS USB from a backup onto a different newer USB, thinking maybe that was the problem, swapped the new RTX 4070 in place and still having the same issue, randomly hang in the initial boot, though it was able to boot all the way, giving me webgui access a couple times, but that only lasted 5 or so minutes before crashing and rebooting. I tried installing the GTX1080 again to get the system back online in the meantime, but now with it installed it fails in the same way. I borrowed 2080ti from a friend to test with and same experience. It seemingly hangs on random lines in the boot process. Also, when upgrading to the RTX 4070 i also installed the corsair RM850X from a OCZ 700watt psu. I have swapped back to the OCZ for testing and get the same results.

I did notice a couple times during the 10's of reboot cycles a kernel RCU test running after the boot process failed and rebooted, but i dont know if this was just caused by the boot process hanging, crashing and then rebooting. screenshots at the end of the post.

I collected a syslog just before it crashed while i had CLI access. using 
"cp /var/log/syslog /boot/logs/syslog_backup<incrementing#>"
attached to post

Things that I have done in Troubleshooting:
-double checked all power from PSU is connected to the MB connections securely and to the GPU securely. 
-Rebuilt Unraid USB from backup on a newer USB
-Built a new Unraid USB using the USB creator, then transfered the config directory from the backup.
-Built a new Unraid USB without my config
-Swapped in different GPU's: 9800GTX+, GTX1080, RTX2080ti, RTX4070Super
-defaulted BIOS setting, then disabled bluetooth and wifi on the board
-flipped between UEFI on and off and subsequently removed the "-" from the EFI directory when booting Legacy 
-swapped the old PSU back in to see if it affected it
-stripped all of the hardware from the server besides CPU, RAM and GPU and m.2 drives
-ran a memtest86 and the 96GB ran for 2.5hrs stably and passed the test 
-Flashed the Asus motherboard to the latest firmware

Planning on today finding a USB bootable stress testing utility to verify the CPU and GPU both function under load without crashing

I bought the recommended USB drive and it is arriving today, Samsung Bar 3.1 drive, so i can rule that out. 


Point where it hangs during standard boot

 

Point where it hangs boot with GUI


Passed Memtest86


RCU tests




 

syslog_backup5

Edited November 13, 20241 yr by J_Dub

If a new flash rive with a stock install also has issues, it suggests a hardware problem.

I would agree with that, if this system wasn't solid running unraid for the last couple years since it was built. it may be that the issue occurred during the PSU and GPU swap process, but ive never had that happen with all of the builds and upgrades ive done for years. ill do some further testing to try and confirm this. Im not sure how reliable it is, but this motherboard has the QLED post test, CPU,RAM,VGA and BOOT. all of the indicator lights go through post properly. With memtest86 running for 2.5hrs and it passed, the GPU being new and the same issue occuring on multiple GPU's, the odd man out is the CPU. Ill reseat it and run some stress test to see if it crashes during heavy load.

 

Edited November 13, 20241 yr by J_Dub

Did some testing tonight and this doesn't seem to be hardware failure related, as i created a Hirens USB boot environment and for the last 2 hours have had it running fine with Prime95 torture test running on all 12 cores with a end report of Zero errors from the same USB port that i have been booting UNRAID from. With that in mind, is there a recommended set of BIOS settings that could be causing any problems if set incorrectly? I have one last hardware related fix attempt once Amazon gets here, that is the USB drive, Im currently using a random SanDisk USB i have had laying around to write the backup copy of my original USB throughout the troubleshooting i have done. Ill see if that has a positive response. I just dont understand this situation, This server has been running flawlessly for 2-3 years now and now its not.

Edited November 14, 20241 yr by J_Dub

New USB drive didn't fix it, im at a loss in troubleshooting. this is a weird one. I have never had issues like this before, the system in all tests shows the hardware is fine, but it will not boot and run Unraid. This is all after its been solid the last 2 or so years.
I built the new USB, base image from the Unraid USB Creator for version 6.12.13 and its acting teh same way. just hanging at the following screen for the last 2-3 hours.



Any help is appreciated.

Edited November 14, 20241 yr by J_Dub

A couple more logs that i was able to gather via CLI before it hung up completely, on an attempt where it actually got booted completely.
 

syslog24 syslog28

Could this be an issue with the RTX4070 not liking the driver that is being provided by the nvidia-driver plugin? I updated the driver on the plugin to the newest driver 565.57.01 before swapping the card in this server. If that is a problem, the server rarely gets booted far enough that i would have the time to go into the nvidia-driver plugin and change it, is there a way to get the driver packages and manually put them in the directory in the Unraid USB?

*update* nevermind, i wasnt considering the fact i used a base image this last try.

Edited November 14, 20241 yr by J_Dub

Could this be an issue with bios settings? 

was up late last night, set BIOS to default, and used a guide for BIOS configuration for Unraid.  I then set everything as indicated in this page, https://docs.unraid.net/unraid-os/getting-started/advanced-bios-config/. Still no go, the first time it loaded and looked promising, with just a base Unraid OS load 6.12.13, without my configuration, was stable for 10-15 minutes so I shutdown and put my /config on the same USB, the first boot after that it immediately froze part way through the USB boot cycle. then i tried to recreate a base OS USB again and see if that loaded, it also hung part way through the boot process.

 

*****RESOLUTION****
Bought a Ryzen 5950X and replaced the 5900X.
Now the system runs stable. So weird that it ran fine with Hirens Diagnostic USB running a Prime95 torture test for 2 hrs without errors. but it is the issue. mind blown

Edited November 15, 20241 yr by J_Dub