[Plugin] Spin Down SAS Drives


doron

Recommended Posts

15 minutes ago, CS01-HS said:

I changed DEBUG to true in /usr/local/emhttp/plugins/sas-spindown/functions

and ran: touch /tmp/spindownsas-debug

Good (actually one of those would have sufficed, but that's fine).

You may want to try (with the plugin installed, and when there's no i/o against that drive):

sdspin /dev/sdb down
echo $?
sdspin /dev/sdb
echo $?

and post output.

Link to comment

Done. Same results whether the drive is active or already spun down.

 

root@NAS:~# sdspin /dev/sdb down
SG_IO: bad/missing sense data, sb[]:  f0 00 01 00 50 40 00 0a 80 00 00 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
root@NAS:~# echo $?
0
root@NAS:~# sdspin /dev/sdb
root@NAS:~# echo $?
0

 

Link to comment
7 hours ago, CS01-HS said:

Done. Same results whether the drive is active or already spun down.

Was this done with the plugin installed and the debug set as you described above?

(I expected another debug message - it might be that one of the above conditions is not met)

 

At any rate, this seems to show that your drive is not really being spun down.

Link to comment
9 hours ago, doron said:

Was this done with the plugin installed and the debug set as you described above?

 

Plugin was installed but debug was disabled. I re-enabled it (with touch /tmp/spindownsas-debug) and tested again - same results, no additional printouts:

root@NAS:~# touch /tmp/spindownsas-debug
root@NAS:~# sdspin /dev/sdb down
/usr/sbin/hdparm -y /dev/sdb
SG_IO: bad/missing sense data, sb[]:  f0 00 01 00 50 40 00 0a 80 00 00 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
root@NAS:~# echo $?
0
root@NAS:~# sdspin /dev/sdb
root@NAS:~# echo $?
0
root@NAS:~#

 

 

9 hours ago, doron said:

At any rate, this seems to show that your drive is not really being spun down.

 

It's a USB drive so I can tell by feel when it's spun down and it is. Spin down worked even without this plugin, problem was unRAID wouldn't detect it. My guess is this wrapper suppresses the error code from "bad/missing sense data" which is fortunate for me but it probably shouldn't.

 

I still don't know the exact command unRAID calls to spin down Unassigned Devices. Do you? I assume it's smartctl but with what parameters exactly? (This is getting outside the scope of your plugin.)

Link to comment
1 minute ago, CS01-HS said:

 

Plugin was installed but debug was disabled. I re-enabled it (with touch /tmp/spindownsas-debug) and tested again - same results, no additional printouts:


root@NAS:~# touch /tmp/spindownsas-debug
root@NAS:~# sdspin /dev/sdb down
/usr/sbin/hdparm -y /dev/sdb
SG_IO: bad/missing sense data, sb[]:  f0 00 01 00 50 40 00 0a 80 00 00 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
root@NAS:~# echo $?
0
root@NAS:~# sdspin /dev/sdb
root@NAS:~# echo $?
0
root@NAS:~#

Okay, now it's as expected (you'd notice one extra line with "hdparm -y" in it).

 

1 minute ago, CS01-HS said:

It's a USB drive so I can tell by feel when it's spun down and it is. Spin down worked even without this plugin, problem was unRAID wouldn't detect it.

What you may be witnessing is the USB HDD controller spinning the drive down on its own accord, after a certain timeout, without Unraid's intervention - I think many/most USB external drives do that to save power, heat and vibration.

 

1 minute ago, CS01-HS said:

I still don't know the exact command unRAID calls to spin down Unassigned Devices. Do you?

 

Yes. For non-SAS drives, the command used is the command echoed above - in our case:

/usr/sbin/hdparm -y /dev/sdb

 

However in the case of your external drive, that command does not succeed. As I said, my guess is the spin down that does happen is performed by the controller on the external HDD itself.

  • Like 1
Link to comment

Good eye re: hdparm printout, I missed that.

 

Okay but I'm pretty sure unRAID doesn't call hdparm directly - it issues some other command (smartctl?) that then calls hdparm. That call (and its parameters for UDs) is what I'm after - if you happen to know.

 

Re: USB spindown, I can bore you with details if you want? 

Spoiler

With 6.8.3 where UD spindown was controlled by the UD plugin I noticed the drive never spun down. So I wrote user script to monitor activity and spin it down with an hdparm call when inactive. The only problem was as far as unRAID knew it was still spinning so the status LED was wrong. With 6.9 unRAID took over UD spindown which made spindown work (and my script unnecessary) but LED was still wrong, probably because hdparm returns that error. With 6.9 and your plugin spindown and LED status both work.

 

Link to comment
20 minutes ago, doron said:

It does.

Check out /usr/local/sbin/sdspin - this is what Unraid calls (as of 6.9). 

That code in turn calls hdparm.

 

sdspin! That's what I was looking for, thank you. I mistakenly assumed that script was specific to your plugin. And I see the plugin helpfully saves the default version so I'll dig in and figure out the difference.

Link to comment

Okay, mystery solved. Compare the return code from your version to unRAID's:

 

root@NAS:~# sdspin /dev/sdb down; echo $?
SG_IO: bad/missing sense data, sb[]:  f0 00 01 00 50 40 00 0a 80 00 00 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0
root@NAS:~# cp /usr/local/sbin/sdspin.unraid /usr/local/sbin/sdspin
root@NAS:~# sdspin /dev/sdb down; echo $?
1
root@NAS:~#

 

Now the two scripts:

 

Unraid's returns 1 (RET=1)

hdparm () {
  OUTPUT=$(/usr/sbin/hdparm $1 $RDEVNAME 2>&1)
  RET=$?
  [[ $RET == 0 && ${OUTPUT,,} =~ "bad/missing sense" ]] && RET=1
}

...

exit $RET

 

Yours executes this block

    else  # Not SAS

      $DEBUG && echo $HDPARM -y $RDEVNAME
      $HDPARM -y $RDEVNAME > /dev/null

    fi

 

without catching a return code then hits the last line, returning 0

exit 0 # Just in case :-)

 

 

Edited by CS01-HS
Link to comment
21 hours ago, doron said:

Yup, that is, in fact, intentional 🙂

 

I don't want to derail further but for anyone following, I ended up customizing sdspin to ignore "bad/missing sense" returns from USB drives, so I can uninstall the plugin (I have no SAS drives.) Thanks for the plugin and your help - otherwise I never would have found the fix.

 

 

Link to comment
  • 2 weeks later...

Hi,

 

just wanted to give a quick update on the sas spindown issue i had with the SAS Controller i reported earlier. So it doesnt matter what sas drive i spin down, the reason its not waking up again is the controller believes that a bus error occurred. 

 

Here the dmesg output after a sdspin down of a seagate drive:

 

[303126.462121] sas: Enter sas_scsi_recover_host busy: 1 failed: 1
[303126.462123] sas: trying to find task 0x00000000a2a13202
[303126.462124] sas: sas_scsi_find_task: aborting task 0x00000000a2a13202
[303126.462274] pm80xx0:: mpi_ssp_completion  1932:sas IO status 0x1
[303126.462276] pm80xx0:: mpi_ssp_completion  1944:SAS Address of IO Failure Drive:5000c500a61b9e91
[303126.462277] pm80xx0:: mpi_ssp_completion  2158:task 0x00000000a2a13202 done with io_status 0x1 resp 0x0 stat 0x8d but aborted by upper layer!
[303126.462282] sas: sas_scsi_find_task: task 0x00000000a2a13202 is done
[303126.462283] sas: sas_eh_handle_sas_errors: task 0x00000000a2a13202 is done
[303126.462289] sas: ata7: end_device-7:4: dev error handler
[303126.462291] sas: ata8: end_device-7:5: dev error handler
[303126.462293] sas: ata9: end_device-7:6: dev error handler
[303126.462295] sas: ata10: end_device-7:7: dev error handler
[303126.462296] sas: ata11: end_device-7:8: dev error handler
[303126.462297] sas: ata12: end_device-7:9: dev error handler
[303126.462303] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1

 

I can fix it by basically unplugging and replugging the drive. That way the controller recovers from the sas_scsi_recovery_host error. 

 

So the disk tested was this time a ST4000NM0025 

 

Connected to a : supermicro aoc-s8076l-l16e

That is basically an adaptec sas controller. 

 

Driver is pm80xx0 , 

 

The funny thing is the 6 sata drives connected to the same controller spin down fine and come back up. So i assume a deeper issue with libsas and the pm80xx driver.

 

Just wanted to let you know if you want to put that combo into your blacklist. 

 

Regards,

 

Link to comment
  • 2 weeks later...

Hey Doron, 

 

Thank you for taking the time and making this plugin.  I'm sorry if you've already answered this a million times.  I've tried to look through the thread, but it's difficult to follow along.

I have HP H240 HBA and Constellation ES.3 ST3000NM0023 drives.  I have been following this plugin since you've released it last year and I know that ES.3s were a problem.  I cried a little and then figured I'll be patient and see if there are any fixes in the future 😭😉

 

In 6.8.3 unRAID would never even pretend that the drives were spun down and in the log I would get that red/green zebra response that I've seen in this thread.  In 6.9.1, with your plugin installed (v0.85), unRAID "spin downs" the drives, the drives make a weird noise, the green dot becomes gray, but the drives appear to still be spinning and making spinning noise.  The weird thing is that if I try to access the drive that is "spun down" (and yet still spinning) it takes a couple of seconds for it to "wake up" and allow me access to the files within.

 

Maybe I am misunderstanding what "spinning down" is, but I assumed that they would literally stop spinning.  Also, I did check if the drives were the source of the noise and yes they are, everything else in the system is too quiet to make any audible noise when the side panel is on.  Is there anything you can do to either help me troubleshoot or is it a lost cause?  Thanks and sorry again if you've answered this before.

Edited by numerica
Link to comment
7 hours ago, numerica said:

Constellation ES.3 ST3000NM0023

I have a  ST4000NM0023 and it spins down fine but is connected via an LSI controller.

 

What output to you get from sdparm -C sense /dev/sdX when unraid thinks they are spun down.

 

Sounds like your drives are going into an idle state, as takes 10-15 seconds for mine to spin up.

Edited by SimonF
Link to comment
9 hours ago, numerica said:

I have HP H240 HBA and Constellation ES.3 ST3000NM0023 drives.

(...)

  In 6.9.1, with your plugin installed (v0.85), unRAID "spin downs" the drives, the drives make a weird noise, the green dot becomes gray, but the drives appear to still be spinning and making spinning noise.  The weird thing is that if I try to access the drive that is "spun down" (and yet still spinning) it takes a couple of seconds for it to "wake up" and allow me access to the files within.

(...)

Maybe I am misunderstanding what "spinning down" is, but I assumed that they would literally stop spinning. 

 

Yes, you've hit one of the main sources of grief with this entire "SAS Spin Down" journey.

In brief, spindown is not well defined in the SCSI / SAS standards, and therefore it's implemented differently on different drives.

For a slightly broader explanation: SAS drives have several power states. Two of those are of interest here: Idle (2) and Standby (3). In the standard, those states are defined rather loosely and abstractly; each of them is supposed to be a deeper "power saving" mode, in which the power draw is yet lower and the distance from the Active state is yet larger than the previous state. However, the standard does not dictate specific actions like turning off the spindle rotation, or anything specific actually. The net effect is that each drive+controller+firmware  version combo demonstrates a slightly different behavior.

 

From your description, it seems that your drive's firmware takes the drive to an idling mode which still includes spinning. Perhaps a firmware upgrade will change that; perhaps not.

The frustrating thing is, that we don't know. So basically I'm trying to collect success/failure reports and code them into the plugin 😞

 

If you can send me (pm is fine) the output of the following command, that'll be helpful for documenting the situation.

/usr/local/emhttp/plugins/sas-spindown/sas-util

 

Link to comment
9 hours ago, SimonF said:

I have a  ST4000NM0023 and it spins down fine but is connected via an LSI controller.

 

What output to you get from sdparm -C sense /dev/sdX when unraid thinks they are spun down.

 

Sounds like your drives are going into an idle state, as takes 10-15 seconds for mine to spin up.

 

    /dev/sdf: SEAGATE   ST3000NM0023      0003
Additional sense: Standby condition activated by command

 

That's the output.  Thanks for helping!

Link to comment
1 hour ago, SimonF said:

Thats the result that should be seen, have you been able to check if the drive has spun down by touching it? or heat after it has been spun down a while.

 

The way I have been determining it is by knowing what the box sounds like without mechanical drives and with.  After putting the drives on stand-by, I see no reduction in noise.

Link to comment

Looks like something in unRAID 6.9.2 is resulting in drives not spinning down. Even trying to manually spin down doesn't help. Here's the Tools > System Log

 

Apr  8 08:12:12 cobblednas emhttpd: spinning down /dev/sde

Apr  8 08:12:12 cobblednas emhttpd: spinning down /dev/sdi

Apr  8 08:12:12 cobblednas SAS Assist v0.85: Spinning down device /dev/sde

Apr  8 08:12:12 cobblednas SAS Assist v0.85: Spinning down device /dev/sdi

Apr  8 08:12:33 cobblednas emhttpd: read SMART /dev/sde

Apr  8 08:12:33 cobblednas emhttpd: read SMART /dev/sdi

Apr  8 08:15:15 cobblednas emhttpd: spinning down /dev/sdd

Apr  8 08:15:15 cobblednas SAS Assist v0.85: Spinning down device /dev/sdd

Apr  8 08:15:35 cobblednas emhttpd: read SMART /dev/sdd

Apr  8 08:15:50 cobblednas emhttpd: spinning down /dev/sdg

Apr  8 08:15:50 cobblednas SAS Assist v0.85: Spinning down device /dev/sdg

Apr  8 08:15:56 cobblednas emhttpd: read SMART /dev/sdg

Apr  8 08:18:26 cobblednas emhttpd: spinning down /dev/sdh

Apr  8 08:18:26 cobblednas SAS Assist v0.85: Spinning down device /dev/sdh

Apr  8 08:18:33 cobblednas emhttpd: read SMART /dev/sdh

Apr  8 08:19:51 cobblednas emhttpd: spinning down /dev/sdf

Apr  8 08:19:51 cobblednas emhttpd: spinning down /dev/sdc

Apr  8 08:19:51 cobblednas SAS Assist v0.85: Spinning down device /dev/sdc

Apr  8 08:19:51 cobblednas SAS Assist v0.85: Spinning down device /dev/sdf

Apr  8 08:20:13 cobblednas emhttpd: read SMART /dev/sdf

Apr  8 08:20:13 cobblednas emhttpd: read SMART /dev/sdc

Apr  8 08:31:55 cobblednas emhttpd: spinning down /dev/sdi

Apr  8 08:31:55 cobblednas SAS Assist v0.85: Spinning down device /dev/sdi

Apr  8 08:32:18 cobblednas emhttpd: read SMART /dev/sdi

Apr  8 08:42:14 cobblednas emhttpd: spinning down /dev/sde

Apr  8 08:42:14 cobblednas SAS Assist v0.85: Spinning down device /dev/sde

Apr  8 08:42:35 cobblednas emhttpd: read SMART /dev/sde

Apr  8 08:45:17 cobblednas emhttpd: spinning down /dev/sdd

Apr  8 08:45:17 cobblednas SAS Assist v0.85: Spinning down device /dev/sdd

Apr  8 08:45:21 cobblednas emhttpd: read SMART /dev/sdd

Apr  8 08:45:52 cobblednas emhttpd: spinning down /dev/sdg

Apr  8 08:45:52 cobblednas SAS Assist v0.85: Spinning down device /dev/sdg

Apr  8 08:46:13 cobblednas emhttpd: read SMART /dev/sdg

Apr  8 08:48:28 cobblednas emhttpd: spinning down /dev/sdh

Apr  8 08:48:28 cobblednas SAS Assist v0.85: Spinning down device /dev/sdh

Apr  8 08:48:50 cobblednas emhttpd: read SMART /dev/sdh

Apr  8 08:49:53 cobblednas emhttpd: spinning down /dev/sdf

Apr  8 08:49:53 cobblednas emhttpd: spinning down /dev/sdc

Apr  8 08:49:53 cobblednas SAS Assist v0.85: Spinning down device /dev/sdf

Apr  8 08:49:53 cobblednas SAS Assist v0.85: Spinning down device /dev/sdc

Apr  8 08:50:27 cobblednas emhttpd: read SMART /dev/sdf

Apr  8 08:50:27 cobblednas emhttpd: read SMART /dev/sdc

 

Edit: I did privately send Doron all diagnostic information to troubleshoot this.

Edited by xaositek
Send diagnostics
Link to comment
1 hour ago, xaositek said:

Looks like something in unRAID 6.9.2 is resulting in drives not spinning down. Even trying to manually spin down doesn't help.

 

I have just migrated to 6.9.2 and SAS Spindown works perfectly for me (same as it did in 6.9.0 and 6.9.1).

Do you have a distinctly different experience than you had in 6.9.1?

Link to comment
16 minutes ago, doron said:

 

I have just migrated to 6.9.2 and SAS Spindown works perfectly for me (same as it did in 6.9.0 and 6.9.1).

Do you have a distinctly different experience than you had in 6.9.1?

Yes, I have raised a bug report. Only one of my drives is affected. But its also affecting non SAS drives.

Link to comment
On 4/8/2021 at 6:37 PM, SimonF said:

Yes, I have raised a bug report. Only one of my drives is affected. But its also affecting non SAS drives.

@SimonF, are you seeing the same behavior (with your SATA drives) if you remove the plugin?

 

If NOT (i.e. removing the plugin makes the issue go away), can you please test something for me --

1. Remove the plugin

2. Manually spin down one of the SATA drives on which you've seen the problem. 

3. Issue this command against this SATA drive:

sdparm -ip di_target /dev/sdX

4. Check whether this command caused (a) the drive to spin up (b) message "read SMART /dev/sdX" to be logged.

 

Thanks! (Un)fortunately I can't reproduce this issue locally.

 

(There's also a related report that I'd like to get to the bottom of.)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.