JorgeB

00:17.0 RAID bus controller [0104]: Intel Corporation Device [8086:43de] (rev 11) DeviceName: Onboard - Other Subsystem: ASUSTeK Computer Inc. Device [1043:8694]  
No SATA controller being detected, only this, see if there's an option in the BIOS to change it to AHCI/SATA

Can't answer that, never used those, just delete one of them.

Try this:
https://forums.unraid.net/topic/124055-unraid-6101-fails-to-boot-bzroot-checksum-error/?do=findComment&comment=1131675
 

If it's a Dell server see here:
 
https://forums.unraid.net/topic/119502-bzimage-checksum-error/?do=findComment&comment=1108354
 

Data Metadata System Id Path single single single Unallocated -- --------- --------- --------- -------- ----------- 1 /dev/sdg1 397.00GiB 2.00GiB 32.00MiB 48.10GiB -- --------- --------- --------- -------- ----------- Total 397.00GiB 2.00GiB 32.00MiB 48.10GiB Used 364.44GiB 314.58MiB 80.00KiB  
They confirm only cache1 is being used, you can usually fix it doing this:
 
-stop the array
-unassign cache2
-start array
-stop array
-re-assign cache2
-start array
 
You should now see activity for cache2, in case of doubt post new diags.

If it works in safe mode it suggests one of the plugins is the problem, uninstall or rename the plg files to see if you can find the culprit.

Prefail is just the type os some SMART attributes, also disk4 looks healthy, most likely not a disk problem.

Change /mnt/user0/ to /mnt/user/, user0 doesn't include contents from any pool.

See my reply to your other post, you need to remove device 81 from the VM XML:
 
-device vfio-pci,host=0000:81:00.0,id=hostdev0,bus=pci.4,addr=0x0 \  

Server was rebooted in the middle so can't see everything that happened, current cache is mounting, but it's empty, possibly some error during the replacement.
 
Also note that you have device 81:00 bound to vfio-pci, 81 is now one of the NVMe devices and although it's a different device and it wasn't bound it still brought the device offline, so delete that bind and run a correcting scrub on the pool.
 
As for the old data, I assume this device was part of the original pool?
 
C300-CTFDDAC128MAG_00000000102202FBC295 (sdl)  
If yes this also suggests the replacement wasn't done correctly since there's still a btrfs filesystem there, and it should have been wiped during the replacement, if this was part of the old pool and the pool was redundant you should be able to mount it if you assign alone to a new pool (not with UD), if it doesn't mount in a new pool post new diags, maybe it can be mounted manually.

Looks like the docker image ran out of space, the system share exists on both disk1 and cache, if the docker image is on disk1 the disk is completely full, if that's the case I would recommend deleting that one and recreating the image on cache.

See here:
 
 

Disk does appear to be failing, you can run an extended SMART test to confirm.

Check that it's well seated, if errors persist you likely have a bad CPU.

That's good news!
 
UD historical devices don't really matter for this, but you can remove it now or later, I assume you plan do re-add the other device o the pool?
 
If yes first make sure backups are up to date, then you'll need to wipe the other device before adding it back to the pool, you can do it like this:
 
-check that array auto start is disable, shutdown server
-reconnect the other NVMe device
-power on the server, don't start the array
-wipe the unassigned device with
blkdiscard /dev/nvme#n1 Replace # with correct number, not sure if 6.9.2 needs -f for blkdiscard if a data is detected, if yes use it.
-assign it back to the pool
-start array to begin balance
 

Now start the array with both cache devices unassigned, then stop the array, assign both devices to the same pool, start the array and post new diags, if it mounts there's a chance the fs will go read only again, if that happens we can try to mount manually with skip_balance.

Once a disk gets disabled it needs to be rebuilt/re-synced:
 
https://wiki.unraid.net/Manual/Storage_Management#Rebuilding_a_drive_onto_itself

Board is spamming the log with ATA lonk info, the disk looks OK though, there a lot of UDMA CRC errors, and the LSI firmware you're using has known issues, you should update to latest, and since the emulated disk is mounting and assuming contents look correct you can then swap/replace cables to rule that out and re-enable the disk.

Check filesystem on disk5

You can download the stable zip then just replace all the bz* files on the flash drive and reboot.

Settings -> Display Settings -> Used / Free columns

Parity disk appears to be really failing so you'll also need to replace that one, but the rebuilt disk should be 100% OK as long as no more errors on a different disk, when Unraid fails to read a sector form parity1 it will use parity2, it just took longer with all the errors, but parity2 saved your butt in this case.

May 18 10:18:12 SW-URSVR root: '/mnt/user/system/libvirt/libvirt.img' is in-use, cannot mount  
Rebooting should fix it.

Could be this issue:
Also a good idea to run memtest.
 

Crashing looks more hardware related, you can try running memtest if you haven't yet, can also try upgrading to v6.10-rc8 mostly to rule out any kernel compatibility issue.