1 of 2 NVME drives in "mirrored cache Z pool" has "spun down" and will not "spin up" / turn on

I know NVME or SSD dont really spin down

but that's what is being reported by unraid

As well, i cant seem to wake it up

it will not respond to "spin up"

it stays gray and has no temp or wattage reporting

and the disk location is blinking red as smart is reporting temp error on temp threshold which is no value / null

could the drive have fully shut down due to temp

i had seen a previous video where someone changed the nvme thresholds as they were too low

even if thats the case though, how do i get it back online

Please post the diagnostics.

tower-diagnostics-20250605-1401.zip

sorry for the delay

here is the file

for the nvme drive in question

ending in t0696

the smart log says drive failed

i have since rebooted and its working fine

i have now disabled spin down on the nvme cache pool which was on by default

i did not enable this

even though they are reporte ddifferently

they are both the same drive but 1 is over a year old and the other is new from yesterday

the 1 year old and never used is working fine

its the brand new one that gave the issue

must be different firmwares

i also noticed that the new one (which seems to have an issue) runs hotter by a few degrees

and olso idles at 0.01watts while the other sits at 0.00watts

actually

the older drive

the one without issue

is running hotter by 6-8 degrees

yet is running fine and idles at 0.00 watts

while the newer one is cooler but idles at 0.01 watts

new drive just stopped responding again

currently still shows green

but no temp or wattage reported when copying to cache

also no data transfer speed for that disk

and report smart temp issue for that disk again

the temp never went above 41c

??????????

Jun 4 23:43:15 Tower kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1

Jun 4 23:43:15 Tower kernel: nvme nvme1: Disabling device after reset failure: -19

Device is dropping offline, but since the log is being spammed with PCIe errors start by doing this:

https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/?do=findComment&comment=1165009

Reboot and then post new diags if it drops again.

I have applied this and rebooted

on start the disk was not there and my array would not start

so i did a full shutdown and then powered on

its ok for now, it seems

it will take at least 4-5 hours to know if it fixed it for me as thats about how long it takes for the issue to start every time

im a little concerned that there may be an actual issue with this new drive

especially after it was missing after reboot

also does this affect only power management of NVME or other things as well?

as i really dont want to disable power management

7 minutes ago, charettepa said:

on start the disk was not there and my array would not start

so i did a full shutdown and then powered on

This is normal, it typically requires a full power cycle for the device to come back.

It may increase power consumption, but should be by a very little.

so far i see no changes in NVME disk power consumption

i'm not sure about motherboard and total system use as i had not checked that yet

is the issue the specific NVME's or the OS/Kernel not handling things properly

looks like in the thread you sent me to, there was someone trying to investigate further as there seems to be a bugzilla related issue for power management

ok, so it looks like my drive just dropped off again

Jun 6 09:58:06 Tower webgui: Successful login user root from 192.168.0.148

Jun 6 10:01:23 Tower autofan: Highest disk temp is 38C, adjusting fan speed from: 180 (70% @ 1167rpm) to: 110 (43% @ 746rpm)

Jun 6 10:06:28 Tower autofan: Highest disk temp is 37C, adjusting fan speed from: 110 (43% @ 735rpm) to: 90 (35% @ 602rpm)

Jun 6 10:08:01 Tower emhttpd: spinning down /dev/sdb

Jun 6 10:08:01 Tower emhttpd: spinning down /dev/sdc

Jun 6 10:13:09 Tower emhttpd: spinning down /dev/sdd

Jun 6 10:13:09 Tower emhttpd: spinning down /dev/sde

Jun 6 10:15:20 Tower kernel: nvme nvme0: I/O tag 8 (3008) opcode 0x6 (Admin Cmd) QID 0 timeout, reset controller

Jun 6 10:16:33 Tower autofan: Highest disk temp is 0C, adjusting fan speed from: 90 (35% @ 603rpm) to: OFF (0% @ 141rpm)

Jun 6 10:18:32 Tower kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1

Jun 6 10:20:40 Tower kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1

Jun 6 10:20:40 Tower kernel: nvme nvme0: Disabling device after reset failure: -19

Edited June 6, 20251 yr by charettepa

58 minutes ago, charettepa said:

is the issue the specific NVME's or the OS/Kernel not handling things properly

Seems to be a combination of the kernel, board and device, as the same device can work fine for one use an not another oner, but using a different brand/model can also help.

from the section of log i posted it seems to still be failing

am i correct?

so should i be trying to get this drive replaced?

50 minutes ago, charettepa said:

from the section of log i posted it seems to still be failing

No, it's dropping offline, typically not a media error, though it can still be a device issue, but not usually.

so what could the issue be?

im not sure i understand

how do i know if the drive needs replacing

its not the nvme slot as i swapped the 2 and the same still happens to that drive only

this is my cache mirror array and the system wont start the array when this happens, which is always within 1-2 hours after start

what should i do next?

Edited June 6, 20251 yr by charettepa

19 minutes ago, charettepa said:

so what could the issue be?

1 hour ago, JorgeB said:

eems to be a combination of the kernel, board and device,

If you can, try with a different device, ideally from a different brand or model.

I dont currently have any others

not at that size

i would need to initiate a return

are any specific brand/model recommend for success with unraid

Also though, what's odd, as i mentioned

they are the same manufacturer, same device, same model

but 1 is a year old and the other is brand new

its only the new one that has the issue

52 minutes ago, charettepa said:

are any specific brand/model recommend for success with unraid

Difficult to say, a model that works great with a board can have issues with different one, personally, I have no issues with multiple WD, Samsung, Intel, Kingston and Kioxia (Toshiba) SSDs

I have been looking into it and the issue seems to be mostly present if you are in a zfs raid and/or dont have a dram nvme

is it worth while breaking my mirror than seeng if it stays up as a solo drive

I really specifcally wanted a raid z mirror for the cache pool as i wanted to also take advantage of the ram caching that zfs does in addition

and put 32gb ram in the system for this purpose

ill spend more if needed but i want to make sure it will work

i see samsung 990 pro has dram

but i could have sworn i saw in the other thread people that had the same issue

if i know 990 pro will work

ill downgrade my 2tb cache to 1tb of 990 pros

Edited June 6, 20251 yr by charettepa

broke the mirror and created a new "temp" pool for the affected drive

at first it seems like it would be ok

but after about 2.5 hours it dropped again

i have found that in other older threads you recommended a longer string for the bootup

"append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off"

i am now trying this but the nvme drives nver drop below 6.5 watts now

thats 13 extra watts as the drives idled to 0.01watts before

again

i dont mind buying more disks if i know they will work

are 990 pro a safe bet

the only threads i have found so far with issues were able to resolve by using what you first recommended to me

"append initrd=/bzroot pcie_aspm=off"

9 hours ago, charettepa said:

"append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off"

Yep, this is also worth trying, I thought I had already mentioned this here.

no, like i mentioned above

it did not help

Plus, power usage stayed at max power of 6.5 watts per NVME at all times

Hello, subscribing toi this thread as I have the exact same issue. What I did is disable sping down on the faulty nvme.

Did adding the kernel parameters fix the problem?

I am considering swapping nvme but not sure if this going to help.

@googleg

I had already tried disabling spin down, that did not help

it just started reporting drive down instead

the kernel tweak "append initrd=/bzroot pcie_aspm=off" did not help with my NVME

it does seem to help with NVME that "should" be working

it would seem that you should never use NVME without DRAM in ZFS , even single drive

i have since purchased 990 pro's, which do have DRAM

its been a few days and is holding up

i do however still have the kernel tweak "append initrd=/bzroot pcie_aspm=off"

as i noticed some people with 990's still needed this, so i just left it on

one thing that might be of value or help is making sure your drives are from the same batch and same production center

ie. samsungs can be made in korea, malaysia, and a few others

when i bought mine i went to a local store and specifically requested same country and same production date

apparently this helps somehow and , well , if it costs nothing extra, why not

sorry for the long reply

i'd rather give details than just 1 line answers and leave you hanging

Edited June 11, 20251 yr by charettepa

Thanks for the feedback. Is there a list of NVMEs with DRAM?