Hello all,

I have the following problem.

Few days ago first parity drive of my unraid pool (2xparity, 2xdata) failed, was not mountable again.

I replaced the drive, everything was fine with the pool. I checked the failed drive it showed an UDMA CRC error count failure, but after formating it works fine again.

Restoring was not possible due to no superblock found. Didn't understood what's going on ...

I don't dive in every detail now.

After parity check and two days later first parity drive, other physical drive of course, failed again, not mountable, not restorable.

Exchanged the drive by a new hdd again, everything seems to be fine.

After next reboots two drives of my unraid pool and 2 drives of my zfs SSD pool failed and this two are fully gone now.

Some data was lost due to my testing what is going on, but that is not the problem. I have some backups and can restore the data.

So I thought HW is the problem.

Replaced power supply, same issues.

Replaced motherboard, same issues, different M.2 SSD failed as well, but drives working fine after testing externally.

Replaced CPU, same issues.

During all this reboots SW failed sporadically at different positions during boot.

Of course I can start to reinstall the USB Stick, but I guess a reinstalled backup on any stick probably doesn't help.

Has anybody an idea what could be the problem? Latest diagnostic file is attached.

Many thanks in advance

diagnostics-20251028-1609.zip

There aren't any device errors logged, note that the syslog starts over after every boot, post new diags before rebooting if there are more errors.

Drive nvme2n1 already fully failed and is not mountable, but is was recogniced by bios before booting this time.

Guess it doesn't help to boot again, but I have the diagnostic file before I rebooted. Does this help?

diagnostics-20251028-1545.zip

When an NVMe device disappears, most often it dropped offline, not failed, but if this happens, usually you need to power cycle the server for the device to be detected again; just rebooting won't be enogh.

The older diags also don't show any device errors that I can see; there's a btrfs balance with two NVMe devices, but it's completed successfully.

Power cycle doesn't help either, I tried a dozen times. It seems drives which failed need to be formated externally to bring them back in unraid. But then they work fine again for the first period of time until anything fails again. And the nvme that failed here is brand new I bought it yesterday.

In the first picture you can see the the failed drive "Gaming" and it was originally named or mounted nvme2n1.

In the second picture you see same drive under unassigned devices and it is named as nvme0n1 same as the second drive in first nvme pool "Application 2", but not mountable.

There seems to be something mixed up in the SW!? Propably my problems started with update to 7.1.4? No accusation, just a observation.

The fail this time started with a missing green bulb in front of the Name "Gaming" during a file transfer >1TB between this two pools by the move button.

I rebooted after cold start, without switching of power supply.

After reboot it showed the red X and I did first diagnostic file.

I rebooted again checked in the bios, drive is recognized and with the same numbering as shown in the two nvme pools.

Unraid reboots but doesn't find it as nvme2n1 but as unassigned under nvme0n1 that is already used!? And I did the second diagnostic file.

In the terminal window all drives can be found with the correct naming.

This was now round about the 15th failed hdd or ssd drive in the last 10 days. And I nearly changed the full HW except the case and half the drives.

Do you think it could help to fully new install unraid?

19 minutes ago, Ulrich_W said:

It seems drives which failed need to be formated externally to bring them back in unraid.

This seems not possible, since Unraid doesn't care if the devices are formatted or not to detect them, so something else is going on.

19 minutes ago, Ulrich_W said:

But then they work fine again for the first period of time until anything fails again.

Post new diags next time that happens, before rebooting.

Ok, thanks for your help so far.

Ok, here we are again ☹️

After everything was fine I got a notification.

Then after cold reboot.

First drive starts to fail, no temperature information anymore.

Second drive the same.

one drive is shown as unassigned.

With folloowing drive log.

Attached the latest log files.diagnostics-20251029-0436.zipdiagnostics-20251029-0442.zipdiagnostics-20251029-0506.zip

This looks like a device being passed through to a VM, since there are a lot of them, please confirm which VMs were running when this happened.

Not any VM or Docker was running anymore when this happend the last 10 times.

Actually I only tryed to start the VM "001b Windows 10 Gaming", but it didn't work as the iso file was missing last time.

Before I was regularly using VM "001b Windows 10 Gaming" and VM "101 Windows 10 Programming NVDIA" in parallel with two passed though graphics cards.

Only one graphics card is passed through now, but this configuration was running months without any changes or trouble till the frist drive failed.

I didnt work on the configuration of VMs or Dockers till 10 days before when I updated some dockers, apps and unraid as usually and several times before.

As I remember only this first VM was up when it happend the first time. But there was no connection between the first failed drive and the VM.

The VM was running without any issues the first five times when any drives failed. I am just summarizing what I remember ....

Edited October 29, 2025Oct 29 by Ulrich_W
typo

9 minutes ago, Ulrich_W said:

Actually I only tryed to start the VM "001b Windows 10 Gaming", but it didn't work as the iso file was missing last time.

That VM is passing through this NVMe device:

05:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC E18 PCIe SSD [1bb1:5018] (rev 01)

Subsystem: Seagate Technology PLC E18 PCIe SSD [1bb1:5018]

Kernel driver in use: nvme

Kernel modules: nvme

When you tried to start the VM, the device dropped and got reconnected with a different ID.

Oct 29 04:32:37 3UNetNAS usb_manager: Info: rc.usb_manager vm_action 001b Windows 10 Gaming prepare begin -

Oct 29 04:32:37 3UNetNAS usb_manager: Info: rc.usb_manager vm_action 001b Windows 10 Gaming stopped end -

Oct 29 04:32:37 3UNetNAS kernel: nvme nvme4: pci function 0000:04:00.0

Oct 29 04:32:37 3UNetNAS kernel: nvme nvme4: missing or invalid SUBNQN field.

Oct 29 04:32:37 3UNetNAS kernel: nvme nvme4: D3 entry latency set to 10 seconds

Oct 29 04:32:37 3UNetNAS kernel: nvme nvme4: 16/0/0 default/read/poll queues

Oct 29 04:32:37 3UNetNAS kernel: nvme4n2: p1

Oct 29 04:32:37 3UNetNAS kernel: BTRFS warning: duplicate device /dev/nvme4n2p1 devid 2 generation 9427 scanned by udevd (43625)

Oct 29 04:36:14 3UNetNAS root: Fix Common Problems Version 2025.08.07

Oct 29 04:36:29 3UNetNAS kernel: BTRFS error (device nvme2n1p1): bdev /dev/nvme1n1p1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0

Oct 29 04:36:29 3UNetNAS kernel: BTRFS error (device nvme2n1p1): bdev /dev/nvme1n1p1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0

Oct 29 04:36:29 3UNetNAS kernel: BTRFS error (device nvme2n1p1): bdev /dev/nvme1n1p1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0

Edit the VM and remove this part:

<hostdev mode='subsystem' type='pci' managed='yes'>

<driver name='vfio'/>

<source>

<address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>

</source>

<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>

</hostdev>

<hostdev mode='subsystem' type='pci' managed='yes'>

<driver name='vfio'/>

<source>

<address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>

</source>

<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>

</hostdev>

Then apply, reboot the server, and post new diags (just one copy please), if it happens again.

Actually the script shows:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x45' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x45' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>

guess I shall delete all of it, correct?

I had some trouble during reboot but different than the other times all drives came back in good condition.

Attached the diag.

diagnostics-20251029-1926.zip

I was looking at the wrong XML, there are multiple ones named Windows Gaming, but yes, you can remove them all for now. The important ones are device 04 and 05, those two are the Seagate nvme devices

Hello Jorge, I couldn't wait 🙂

Diag file is already posted.

Scrub the application pool and post the results from the GUI.

Just the scrub results or any further 

UUID:             42ffca51-c710-487d-baaf-1b778e48eb43
Scrub started:    Wed Oct 29 20:05:00 2025
Status:           finished
Duration:         0:02:10
Total to scrub:   708.86GiB
Rate:             5.45GiB/s
Error summary:    verify=612
  Corrected:      612
  Uncorrectable:  0
  Unverified:     0

That's enough; all errors were corrected, so that's good. Next, I would deal with the degraded ZFS pool.

  pool: zfs_ssd_pool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
	invalid.  Sufficient replicas exist for the pool to continue
	functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
  scan: resilvered 3.19T in 1 days 16:32:20 with 0 errors on Tue Oct 28 10:19:33 2025
config:
	NAME                    STATE     READ WRITE CKSUM
	zfs_ssd_pool            DEGRADED     0     0     0
	  raidz2-0              DEGRADED     0     0     0
	    sde1                ONLINE       0     0     0
	    sdg1                ONLINE       0     0     0
	    sdd1                ONLINE       0     0     0
	    sdf1                ONLINE       0     0     0
	    sdb1                ONLINE       0     0     0
	    831355008407993453  FAULTED      0     0     0  was /dev/sdg1
errors: No known data errors

ok, I started replacing the failed drive, that will take a while.

I change drive 5 that showed the fiist error.

Resilvering was finished, pool was still degraded.

then changed drive 6 that seem not to be readable.

New, or actually same problems with nvme Gaming 2.

Diag file is attached.

diagnostics-20251031-0643.zip

Edited October 31, 2025Oct 31 by Ulrich_W
typo and screenshot added

2 hours ago, Ulrich_W said:

I change drive 5 that showed the fiist error.

The problem was with drive 6, that's the one that was faulted, didn't see any issues with drive 5.

2 hours ago, Ulrich_W said:

New, or actually same problems with nvme Gaming 2.

One of the devices is failing to initialzes:

Oct 31 06:29:31 3UNetNAS kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Oct 31 06:29:31 3UNetNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 2312, 248 blocks, I/O Error (sct 0x3 / sc 0x71) 
Oct 31 06:29:31 3UNetNAS kernel: I/O error, dev nvme0n1, sector 2312 op 0x0:(READ) flags 0x80700 phys_seg 31 prio class 0
Oct 31 06:29:31 3UNetNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 2184, 120 blocks, I/O Error (sct 0x3 / sc 0x71) 
Oct 31 06:29:31 3UNetNAS kernel: I/O error, dev nvme0n1, sector 2184 op 0x0:(READ) flags 0x80700 phys_seg 15 prio class 0
Oct 31 06:29:51 3UNetNAS kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Oct 31 06:29:51 3UNetNAS kernel: nvme nvme0: Disabling device after reset failure: -19

Swap m.2 slots between those two pool devices and see where the problem follows.

I swapped now the two devices.

diagnostics-20251031-1458.zip

Problem followed the device, suggesting that is the problem:

Oct 31 14:47:24 3UNetNAS kernel: nvme nvme3: I/O tag 960 (b3c0) opcode 0x2 (I/O Cmd) QID 3 timeout, aborting req_op:READ(0) size:4096
Oct 31 14:47:24 3UNetNAS kernel: nvme3n1: I/O Cmd(0x2) @ LBA 7814036432, 8 blocks, I/O Error (sct 0x0 / sc 0x7) 
Oct 31 14:47:24 3UNetNAS kernel: I/O error, dev nvme3n1, sector 7814036432 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Oct 31 14:47:24 3UNetNAS kernel: nvme nvme3: Abort status: 0x0
Oct 31 14:47:24 3UNetNAS kernel: nvme nvme3: I/O tag 512 (a200) opcode 0x2 (I/O Cmd) QID 13 timeout, aborting req_op:READ(0) size:4096
Oct 31 14:47:24 3UNetNAS kernel: nvme3n1: I/O Cmd(0x2) @ LBA 2112, 8 blocks, I/O Error (sct 0x0 / sc 0x7) 
Oct 31 14:47:24 3UNetNAS kernel: nvme nvme3: Abort status: 0x0
Oct 31 14:47:24 3UNetNAS kernel: I/O error, dev nvme3n1, sector 2112 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Oct 31 14:47:24 3UNetNAS kernel: nvme nvme3: I/O tag 513 (a201) opcode 0x2 (I/O Cmd) QID 13 timeout, aborting req_op:READ(0) size:4096
Oct 31 14:47:24 3UNetNAS kernel: nvme3n1: I/O Cmd(0x2) @ LBA 2104, 8 blocks, I/O Error (sct 0x0 / sc 0x7) 
Oct 31 14:47:24 3UNetNAS kernel: nvme nvme3: Abort status: 0x0

I believe there's a forum thread about updating firmware for Samsung NVMe devices that can help with this; if it doesn't, it would need to be replaced.

Well, it sounds plausible, but it is nearly impossible that the Samsung NVME drive causes the problem.

This is why I first thought it is a hardware problem and I changed nearly every single component now.

Dont't get me wrong Jorge, I fully trust you, you are one of the experts and I am grateful for any help, thank you very much.

But, (just thinking, no accusation)

the original problem started with the SATA drives in the unraid pool first parity 1, then parity 1 and disk 1, then parity 2 and disk 2, and so on ...

Then ZFS pool started to fail, mainly disk 2 and 5, and so on ... now disk 6

In the meanwhile I exchanged PCIE SATA controller, power supply, motherboard two times, CPU and exactly this Samsung NVME SSD990 4TB that fails now and it is brand new.

Before I had in 6x Seagate FireCuda 530 2TB that also failed with the latest motherboard, in the application and gaming NVME pool.

I also know the Fractal Define 7 case I use has a problem with the front 5V USB supply in the boot process, but front USBs aren't plugged in anymore.

I also exchanged temporarily the graphics card.

And before, ... everything worked without any issue for months.

Now again, after reboot, Gaming 2 NVME doesn't exist anymore in unraid, in the bios every single drive exists.

But now I can choose the exchanged SATA ZFS drive 6 for to add to the pool.

This is exactly when it starts again, I destroy the Gaming raid to single. Then I reboot, I bet NVME Gaming 2 comes back.

I remove the pool, add the two Gaming NVMEs in a new pool again with raid 1. Then everything is fine, latest when I reboot and it starts over again ...

The only time I thought it becomes better was, when I deleted the <hostdev> part from the VM script.

Suddenly after this all drives came back without issues?! Didn't really get why?

Latest diag file attached.

diagnostics-20251031-1707.zip

I can only speak for the current issues, the ones I see logged, and at the moment, the problem appears to be that NVMe device, since the issue follows the device when swapping m.2 slots.