6.12.0-rc3 - Spin Down not working on ZFS Formatted Array Disks (And some ZFS Pools)

There have been a few reports of some spin down related issues, but I believe nothing conclusive or reproducible, possibly we'll need to wait to see if there's a trend or specif hardware/configuration involved to try and drill down the issue, I for example have seven disk based zfs pools between both my always on main servers and they all spin down.

 

 

2 hours ago, JorgeB said:

There have been a few reports of some spin down related issues, but I believe nothing conclusive or reproducible, possibly we'll need to wait to see if there's a trend or specif hardware/configuration involved to try and drill down the issue, I for example have seven disk based zfs pools between both my always on main servers and they all spin down.

 

 

I was monitoring I/O and don't see anything actually hitting the disks, tough to track down because disks are empty.  But it makes ZFS unusable for me with this many drives, power usage goes up by hundreds of watts.

It might not be directly related to zfs, since other users report issues with different filesystems, one thing you can try that has worked for some is to go back to v6.11.5, do a new config, start array (obviously any zfs disks/pools won't mount but won't be damaged), then upgrade back to v6.12

I'd rather do some more troubleshooting and try to figure out what is actually causing this.  I'll mess with trying to trace it down some more when I have the time.

So I spent more time on this and I found the culprit. it's the "ZFS Master" Plugin from IkerSaint
 

This in my log, after installing ZFS Master.
Apr 22 21:34:03 bertha emhttpd: spinning down /dev/sdb
Apr 22 21:34:06 bertha emhttpd: read SMART /dev/sdb

FYI, this only affects ZFS formatted disks/pools.  btrfs, xfs all work as expected.

Hope this is the common denominator for the others experiencing this as well!

Changed Status to Solved

Changed Priority to Other

On 4/22/2023 at 10:58 PM, samsausages said:

So I spent more time on this and I found the culprit. it's the "ZFS Master" Plugin from IkerSaint
 

This in my log, after installing ZFS Master.
Apr 22 21:34:03 bertha emhttpd: spinning down /dev/sdb
Apr 22 21:34:06 bertha emhttpd: read SMART /dev/sdb

FYI, this only affects ZFS formatted disks/pools.  btrfs, xfs all work as expected.

Hope this is the common denominator for the others experiencing this as well!

Thanks for posting solution, I too had the same trouble and uninstalled “ZFS Master" Plugin and It  fixed spindowns too.

1 hour ago, bbrodka said:

Thanks for posting solution, I too had the same trouble and uninstalled “ZFS Master" Plugin and It  fixed spindowns too.

Glad it helped you find what plugin is causing it!  
I have been trying to find out what is actually causing this, because it is unexpected behavior.  I'm wondering if it's hardware or a bios setting.
Are you per chance using an LSI HBA Card?  Mind sharing some of your hardware specs?  I'm trying to find common denominators.

Thanks!

On 5/9/2023 at 9:51 AM, samsausages said:

Glad it helped you find what plugin is causing it!  
I have been trying to find out what is actually causing this, because it is unexpected behavior.  I'm wondering if it's hardware or a bios setting.
Are you per chance using an LSI HBA Card?  Mind sharing some of your hardware specs?  I'm trying to find common denominators.

Thanks!

LSI 9300-16i 16-port SAS9300-16I 12GB/s

And

LSI SAS 9200-8e PCI Express to 6Gb/s 

Are in my server

2 hours ago, bbrodka said:

LSI 9300-16i 16-port SAS9300-16I 12GB/s

And

LSI SAS 9200-8e PCI Express to 6Gb/s 

Are in my server

Interesting, I have the 9300-16i as well, and I was starting to see that as the culprit, since I changed out most of my other bits during testing.

Thanks for the follow up, I'll mention that to the developer.

Hi, ZFS Master developer here; I have already answered some of the doubts on the plugin thread; however, I thought it would be beneficial also to respond and even dive a little bit deeper here.

 

Just one question regarding this issue. Are you experiencing the same behavior even with the main tab closed? I agree that loading the main tab will wake up your disks, but there are no other background processes in the plugin besides the ones running when the main tab is open. So if your spin-up issue persists even with the unRaid GUI closed, something else is happening.

 

Please confirm if that is the case.

17 hours ago, Iker said:

Hi, ZFS Master developer here; I have already answered some of the doubts on the plugin thread; however, I thought it would be beneficial also to respond and even dive a little bit deeper here.

 

Just one question regarding this issue. Are you experiencing the same behavior even with the main tab closed? I agree that loading the main tab will wake up your disks, but there are no other background processes in the plugin besides the ones running when the main tab is open. So if your spin-up issue persists even with the unRaid GUI closed, something else is happening.

 

Please confirm if that is the case.

Agreed! I noticed it too. Thanks for confirming it. Is there a way to check the status via command line?

hdparm -C /dev/sdx

smartctl -i -n standby /dev/sdx

3 hours ago, skanthan said:

Agreed! I noticed it too. Thanks for confirming it. Is there a way to check the status via command line?

Any sas drives?

Did we figure out the cause?

What firmware is the HBA's on?

 

Investigating issue of array drives never spinning down or not staying spun down.

A small amount of data i.e a few Kb is written everyone & then causing the to stay alive.

Nothing is show in file activity or open files. 

 

Array is 

Parity

Disk 1 - XFS

Disk 2 - XFS

Disk 4 - ZFS

 

Cache - ZFS

Cache mirror - ZFS

 

Sata-cache - ZFS

Sata-cache mirror - ZFS

 

Jorge mentioned this plugin can cause drives to stay spun up. 

Having a look through this thread, it looked like those with HBA LSI SAS cards were having the issue? Did anyone get anywhere with this?

 

I have a 9207-8i - I did notice the firmware today was on v13 so updated to the latest v20.07

 

My unraid is on 6.12.2 & didn't notice issue on 6.12 or 6.12.1

 

May be a general bug with spin down in 6.12.2 as I notice the default spin down is not saved to other drives. And manual setting does not persist.

moulin-rouge-diagnostics-20230703-2212.zip

Edited July 5, 2023 by dopeytree

On 7/5/2023 at 12:03 PM, dopeytree said:

Did we figure out the cause?

I stopped using ZFS Master, as I rarely use it.  Developer seems to think it's related to ZFS Master Plugin Refreshing info when you go to the tower/main page.
Have not been able to figure out if it's exclusive to the 9300 chipset, but do find a lot of people that are using that and reporting this issue with ZFS Master.

On 5/9/2023 at 8:51 AM, samsausages said:

Are you per chance using an LSI HBA Card?  
 

- Dell Inc. 0NK70N , Version A03
- (x2) Xeon® CPU E5-2697 v2

- 80GB DDR3

- NVME (via PCI-e card) for main cache

- Latest update for ZFS Master

 

I have an LSI HBA Card plugged into a dell mobo.  I have a mix of some of the disks on the mobo onboard controller, and the LSI HBA.  I formatted a new drive and added it to the array (in ZFS, the rest are xfs), and i noticed the whole array won't spin down now. 

 

The single single drive ZFS test-pool i made doesn't stay spun down neither (same HBA). 

My 6 disk zfs pool i didn't expect to spin down, as it is constantly being read.

 

Uninstalled ZFS Master = everything (expected) spins down, and stays down, as expected!  

---

 

I have another server, Supermicro mobo, that i haven't converted over to ZFS yet...i can make a new zfs pool of one or two disk to test ideas if you want tho.  The old Dell is my test server to figure out things before i set it up on the production NAS.

Edited October 4, 2023 by miicar

@miicar zfsmaster just added a setting so it doesn't refresh unless you click the refresh button.  That keeps it from spinning up disks now.

The new zfsmaster update for no refresh fixed my issue on this as well. It would be great to have a note in the refresh field that it will spin the drive up if its spun down when the refresh happens. Might help solve threads like these.

Confirmed, by turning the ZFS Master "Refresh interval" to "no Refresh", my ZFS-formatted disks stop spinning up all the time. Thank you"

Confirmed for me too. Uninstalled ZFS MAster, and disks pin down again... good as here in the UK, cost is prohibitive

On 5/17/2023 at 9:58 PM, Iker said:

hdparm -C /dev/sdx

smartctl -i -n standby /dev/sdx

Hey, I can confirm without unraid webui open, disk can normally turn into standby mode. I set disk spin down in 15 minutes, I checked it after the 20th minutes.

 

 

 

@samsausages Could this setting be per pool somehow? I have a pool, which I don't mind refreshing every 30 sec, but I want my secondary pool to sleep.