Last year I added one of these drives to use for the things I needed most speed and reliability for: appdata, VMs, all that good stuff. Formatted as ZFS so I can use Spaceinvader One's snapshot & replication script to keep safe versions of everything on the array.

 

After a month or two, I started having server crashes. After much head scratching, the issue turned out to be the 990 Pro. It would disappear from the system, seemingly without reason. Changing slots on the motherboard didn't help. Rebooting wouldn't help, but it would always reappear after a power cycle. The firmware was out of date, but updating it didn't help. The vendor replaced it under warranty (the replacement had a later manufacturing date and came with the latest firmware) and that drive's been fine since.

 

Even with the replication and multiple backups, I had a fair bit of trouble getting everything restored and swapped over to the new drive and wanted to minimise the chances of that happening again, so I purchased a second 4TB 990 Pro at a discount from a guy on ebay who was selling a lot of SSDs. It was an older manufacturing date, so the first thing I did was update the firmware, then I set that as a mirror of the first drive and all was well.

 

After a few weeks, I checked the status and found that the drive was offline. Exactly the same was happening- it would disappear, a reboot wouldn't help, but a power cycle would bring it back.

 

So I returned that for a full refund and bought another one from a reputable supplier in mid January this year with a recent manufacturing date and the latest firmware. I set that as the mirror and I hoped that was the end of the issues.

 

Then yesterday I updated Unraid to 7.0.1 - after reboot, the mirror drive was missing! I power cycled and it came back, however I now have this message:

zfs pool status:

      pool: cache_4tb
     state: ONLINE
    status: One or more devices has experienced an unrecoverable error.  An
    	attempt was made to correct the error.  Applications are unaffected.
    action: Determine if the device needs to be replaced, and clear the errors
    	using 'zpool clear' or replace the device with 'zpool replace'.
       see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
      scan: resilvered 34.1G in 00:00:24 with 0 errors on Tue Mar  4 00:14:54 2025
    config:

    	NAME                STATE     READ WRITE CKSUM
    	cache_4tb           ONLINE       0     0     0
    	  mirror-0          ONLINE       0     0     0
    	    /dev/nvme2n1p1  ONLINE       0     0     0
    	    /dev/nvme1n1p1  ONLINE       0     0     5

    errors: No known data errors

 

I checked the logs and can't find when the drive failed. It's a while since I last rebooted Unraid (probably not long after v7 was released. When the mirror drive has failed I haven't been getting any warning or notification, either in v6 or 7. The only way I've found is to go into the main drive's settings and check there. Is there a way to enable this for a failed drive in a ZFS array?

 

As this is the 3rd time this has happened, with what should be one of the most reliable drives from one of the most reliable manufacturers, I'm starting to wonder if something else is at play here.

 

Any help or advice would be much appreciated.

 

 

tower-diagnostics-20250305-0007.zip

Diags are after rebooting so we can't see what happened, see here for how to monitor the pool, and post new diags after it happens again, before rebooting.

Thanks- is there a way to be notified if the drive disappears again?

 

To get the mirror back up and running so that I'm protected in the meantime, is it wise to run zpool clear as recommended in the error message?

21 minutes ago, -C- said:

is there a way to be notified if the drive disappears again?

See the link above.

 

21 minutes ago, -C- said:

is it wise to run zpool clear as recommended in the error message?

It's already running in a mirror again, but clear the errors and then run a scrub

Apologies for missing the link earlier. Have now implemented that script.

I've also cleared & scrubbed and no errors were found. Thanks for confirming that the mirror is running.

  pool: cache_4tb
 state: DEGRADED
status: One or more devices has been removed by the administrator.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using zpool online' or replace the device with
	'zpool replace'.
  scan: scrub repaired 0B in 00:21:25 with 0 errors on Wed Mar  5 23:06:24 2025
config:

	NAME           STATE     READ WRITE CKSUM
	cache_4tb      DEGRADED     0     0     0
	  mirror-0     DEGRADED     0     0     0
	    nvme2n1p1  ONLINE       0     0     0
	    nvme1n1p1  REMOVED      0     0     0

errors: No known data errors

 

It's gone again...

 

Thanks

 

tower-diagnostics-20250306-1101.zip

Device is dropping offline:

 

Mar  6 01:36:07 Tower kernel: nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
Mar  6 01:36:07 Tower kernel: nvme nvme1: Does your device have a faulty power saving mode enabled?
Mar  6 01:36:07 Tower kernel: nvme nvme1: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
Mar  6 01:36:07 Tower kernel: nvme 0000:05:00.0: Unable to change power state from D3cold to D0, device inaccessible
Mar  6 01:36:07 Tower kernel: nvme nvme1: Disabling device after reset failure: -19

 

try this, on the main GUI page click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (top right) and add this to your default boot option, after "append initrd=/bzroot"

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

e.g.:

append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

Reboot and see if it makes a difference.

Thanks, I now have this:

 

label Unraid OS
  menu default
  kernel /bzimage
  append initrd=/bzroot  acpi_enforce_resources=lax nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

 

Am just waiting for some unbalancing to complete and shall reboot after that and see how things go.

 

 

 

Forgot to mention that you may need to power cycle the server to get the NVMe device back, sometimes, just a reboot is not enough.

Thanks for this JorgeB - it's been 2.5 months and no further issues, so I'm marking this is resolved.

 

Do you have a link to what that line of code is doing?

I believe it's mostly to do with power saving modes.

I have just received an alert that there's a problem with the same pool and found that the primary drive is showing as removed:

Previously, I was having issues with the mirrored drive showing as removed. Does the code that I added back in March target a particular drive?

1 hour ago, -C- said:

Does the code that I added back in March target a particular drive?

Nope, it applies to all NVMe devices.

That's not good news.

This'll be the 4th time I've had a different 990 Pro fail in this system. Although it's likely there wasn't actually a problem with the ones I returned and the issue with drives dropping out would have been cured with the edit to the boot section. This one that's now giving trouble was replaced under warranty in October last year.

Any advice on how best to troubleshoot from here?

1 hour ago, -C- said:

his'll be the 4th time I've had a different 990 Pro fail in this system.

Try Googling Samsung 990 Pro NVMe issue

Not sure if the issue has been completely resolved....

I would recommend trying a different NVMe device brand/model, it appears that some kernel/board/device combinations can be a problem, e.g., the same device can work fine for another user with a different board

ive been experiencing similar issues. i have 2 cache pools no redundancy just 1 for appdata and 1 for burst stuff like sab/transcodes ect.

i had 1 980 pro 1tb that had ~5pb R/W, once i got a 2tb 990 pro my problems started, at first i thought it was just the 980 was on its last legs so i bought another 990 evo plus 1tb to replace.

at first the 980 was dropping out 99% of the time. once i switched the 980 for the 990 evo plus then the 990 pro began dropping out.

i then updated the firmware on the 990 pro and it was stable for a few weeks. But this week its back again... its weird itll be fine for like a solid period then just begin dropping out once a day until i do something and it will be fine for a another period of time.

i think my next steps will be pull out the 990 pro and try the 980 with the 990 evo and see if that changes anything. was also considering just changing the 990 pro back to xfs so that it just keeps ticking along.

i have noticed i can cause it to drop out when there is alot of IO occuring i.e i process a bunch of unzips at once ,while theres transcodes running. so its 100% IO related

Does anyone know if its a zfs specific issue and using something like xfs will just let it keep ticking along?

the worst part is when this occurs my cpus get pinned at 100% due to it trying to perform ls commands on the cache share. maybe its mover?

There's a kernel bugfix to try and deal with NVMe devices dropping out:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.12.y&id=bc255fec6fd7300b56c7adad5cb8482cd7cac81a

This will be included in the kernel used for the next 7.2 public release, whether that will be -beta.3 or -rc.1, so it may be worth retesting once that's out.

Update the firmware on the 990 pro to 6B2QJXD7 too. That's also supposed to help.

7 minutes ago, warpspeed said:

Update the firmware on the 990 pro to 6B2QJXD7 too. That's also supposed to help.

That firmware release has been withdrawn and no longer available publically

pub

Oh interesting. I updated one of my drives in a two drive ZFS pool and it seemed to help. Haven't had any additional issues since doing so. Wonder why they pulled it.

Here to say I'm having the identical issue. same drive 990, same issues..

On 8/31/2025 at 5:11 PM, tjb_altf4 said:

That firmware release has been withdrawn and no longer available publically

looks like there is a new firmware release with a potential fix, I'll try it next time I'm doing server maintenance.

On 9/22/2025 at 8:54 AM, tjb_altf4 said:

looks like there is a new firmware release with a potential fix

Thanks for the heads up. I've just updated both the 990s that make up my mirrored cache drive.

After my primary drive disappeared again on 27th of June, I restarted the server and not had either 990 drop out since then. But it's not been good knowing that one or both drives could drop out at any time, especially if high throughput is the cause as with them mirrored they were being hit simultaneously.

Anyway, I made it through a busy summer without issue, now hoping this latest firmware version finally, permanently resolves the problem. Will keep updating here if anything happens.

Have you had any issues since the latest firmware?

Started having similar issues a few months ago.
I have two 990 pro bought both in 2023, updated to firmware 4B2QJXD7 before installing them in unraid.
One is for appdata + docker.img while the other one acting as a temp cache
temp cache was fine i think but the one for appdata and docker would just drop and be inaccessible until a restart was made, both on unraid 6 and 7


Just updated the firmware to 7B2QJXD7 hoping that solves it