March 15, 20215 yr Author 15 minutes ago, CS01-HS said: I changed DEBUG to true in /usr/local/emhttp/plugins/sas-spindown/functions and ran: touch /tmp/spindownsas-debug Good (actually one of those would have sufficed, but that's fine). You may want to try (with the plugin installed, and when there's no i/o against that drive): sdspin /dev/sdb down echo $? sdspin /dev/sdb echo $? and post output.
March 15, 20215 yr Done. Same results whether the drive is active or already spun down. root@NAS:~# sdspin /dev/sdb down SG_IO: bad/missing sense data, sb[]: f0 00 01 00 50 40 00 0a 80 00 00 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 root@NAS:~# echo $? 0 root@NAS:~# sdspin /dev/sdb root@NAS:~# echo $? 0
March 16, 20215 yr Author 7 hours ago, CS01-HS said: Done. Same results whether the drive is active or already spun down. Was this done with the plugin installed and the debug set as you described above? (I expected another debug message - it might be that one of the above conditions is not met) At any rate, this seems to show that your drive is not really being spun down.
March 16, 20215 yr 9 hours ago, doron said: Was this done with the plugin installed and the debug set as you described above? Plugin was installed but debug was disabled. I re-enabled it (with touch /tmp/spindownsas-debug) and tested again - same results, no additional printouts: root@NAS:~# touch /tmp/spindownsas-debug root@NAS:~# sdspin /dev/sdb down /usr/sbin/hdparm -y /dev/sdb SG_IO: bad/missing sense data, sb[]: f0 00 01 00 50 40 00 0a 80 00 00 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 root@NAS:~# echo $? 0 root@NAS:~# sdspin /dev/sdb root@NAS:~# echo $? 0 root@NAS:~# 9 hours ago, doron said: At any rate, this seems to show that your drive is not really being spun down. It's a USB drive so I can tell by feel when it's spun down and it is. Spin down worked even without this plugin, problem was unRAID wouldn't detect it. My guess is this wrapper suppresses the error code from "bad/missing sense data" which is fortunate for me but it probably shouldn't. I still don't know the exact command unRAID calls to spin down Unassigned Devices. Do you? I assume it's smartctl but with what parameters exactly? (This is getting outside the scope of your plugin.)
March 16, 20215 yr Author 1 minute ago, CS01-HS said: Plugin was installed but debug was disabled. I re-enabled it (with touch /tmp/spindownsas-debug) and tested again - same results, no additional printouts: root@NAS:~# touch /tmp/spindownsas-debug root@NAS:~# sdspin /dev/sdb down /usr/sbin/hdparm -y /dev/sdb SG_IO: bad/missing sense data, sb[]: f0 00 01 00 50 40 00 0a 80 00 00 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 root@NAS:~# echo $? 0 root@NAS:~# sdspin /dev/sdb root@NAS:~# echo $? 0 root@NAS:~# Okay, now it's as expected (you'd notice one extra line with "hdparm -y" in it). 1 minute ago, CS01-HS said: It's a USB drive so I can tell by feel when it's spun down and it is. Spin down worked even without this plugin, problem was unRAID wouldn't detect it. What you may be witnessing is the USB HDD controller spinning the drive down on its own accord, after a certain timeout, without Unraid's intervention - I think many/most USB external drives do that to save power, heat and vibration. 1 minute ago, CS01-HS said: I still don't know the exact command unRAID calls to spin down Unassigned Devices. Do you? Yes. For non-SAS drives, the command used is the command echoed above - in our case: /usr/sbin/hdparm -y /dev/sdb However in the case of your external drive, that command does not succeed. As I said, my guess is the spin down that does happen is performed by the controller on the external HDD itself.
March 16, 20215 yr Good eye re: hdparm printout, I missed that. Okay but I'm pretty sure unRAID doesn't call hdparm directly - it issues some other command (smartctl?) that then calls hdparm. That call (and its parameters for UDs) is what I'm after - if you happen to know. Re: USB spindown, I can bore you with details if you want? Spoiler With 6.8.3 where UD spindown was controlled by the UD plugin I noticed the drive never spun down. So I wrote user script to monitor activity and spin it down with an hdparm call when inactive. The only problem was as far as unRAID knew it was still spinning so the status LED was wrong. With 6.9 unRAID took over UD spindown which made spindown work (and my script unnecessary) but LED was still wrong, probably because hdparm returns that error. With 6.9 and your plugin spindown and LED status both work.
March 16, 20215 yr Author 25 minutes ago, CS01-HS said: Okay but I'm pretty sure unRAID doesn't call hdparm directly It does. Check out /usr/local/sbin/sdspin - this is what Unraid calls (as of 6.9). That code in turn calls hdparm.
March 16, 20215 yr 20 minutes ago, doron said: It does. Check out /usr/local/sbin/sdspin - this is what Unraid calls (as of 6.9). That code in turn calls hdparm. sdspin! That's what I was looking for, thank you. I mistakenly assumed that script was specific to your plugin. And I see the plugin helpfully saves the default version so I'll dig in and figure out the difference.
March 16, 20215 yr Okay, mystery solved. Compare the return code from your version to unRAID's: root@NAS:~# sdspin /dev/sdb down; echo $? SG_IO: bad/missing sense data, sb[]: f0 00 01 00 50 40 00 0a 80 00 00 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0 root@NAS:~# cp /usr/local/sbin/sdspin.unraid /usr/local/sbin/sdspin root@NAS:~# sdspin /dev/sdb down; echo $? 1 root@NAS:~# Now the two scripts: Unraid's returns 1 (RET=1) hdparm () { OUTPUT=$(/usr/sbin/hdparm $1 $RDEVNAME 2>&1) RET=$? [[ $RET == 0 && ${OUTPUT,,} =~ "bad/missing sense" ]] && RET=1 } ... exit $RET Yours executes this block else # Not SAS $DEBUG && echo $HDPARM -y $RDEVNAME $HDPARM -y $RDEVNAME > /dev/null fi without catching a return code then hits the last line, returning 0 exit 0 # Just in case :-) Edited March 16, 20215 yr by CS01-HS
March 16, 20215 yr Author 3 hours ago, CS01-HS said: without catching a return code then hits the last line, returning 0 Yup, that is, in fact, intentional 🙂
March 17, 20215 yr 21 hours ago, doron said: Yup, that is, in fact, intentional 🙂 I don't want to derail further but for anyone following, I ended up customizing sdspin to ignore "bad/missing sense" returns from USB drives, so I can uninstall the plugin (I have no SAS drives.) Thanks for the plugin and your help - otherwise I never would have found the fix.
March 25, 20215 yr Hi, just wanted to give a quick update on the sas spindown issue i had with the SAS Controller i reported earlier. So it doesnt matter what sas drive i spin down, the reason its not waking up again is the controller believes that a bus error occurred. Here the dmesg output after a sdspin down of a seagate drive: [303126.462121] sas: Enter sas_scsi_recover_host busy: 1 failed: 1 [303126.462123] sas: trying to find task 0x00000000a2a13202 [303126.462124] sas: sas_scsi_find_task: aborting task 0x00000000a2a13202 [303126.462274] pm80xx0:: mpi_ssp_completion 1932:sas IO status 0x1 [303126.462276] pm80xx0:: mpi_ssp_completion 1944:SAS Address of IO Failure Drive:5000c500a61b9e91 [303126.462277] pm80xx0:: mpi_ssp_completion 2158:task 0x00000000a2a13202 done with io_status 0x1 resp 0x0 stat 0x8d but aborted by upper layer! [303126.462282] sas: sas_scsi_find_task: task 0x00000000a2a13202 is done [303126.462283] sas: sas_eh_handle_sas_errors: task 0x00000000a2a13202 is done [303126.462289] sas: ata7: end_device-7:4: dev error handler [303126.462291] sas: ata8: end_device-7:5: dev error handler [303126.462293] sas: ata9: end_device-7:6: dev error handler [303126.462295] sas: ata10: end_device-7:7: dev error handler [303126.462296] sas: ata11: end_device-7:8: dev error handler [303126.462297] sas: ata12: end_device-7:9: dev error handler [303126.462303] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1 I can fix it by basically unplugging and replugging the drive. That way the controller recovers from the sas_scsi_recovery_host error. So the disk tested was this time a : ST4000NM0025 Connected to a : supermicro aoc-s8076l-l16e That is basically an adaptec sas controller. Driver is pm80xx0 , The funny thing is the 6 sata drives connected to the same controller spin down fine and come back up. So i assume a deeper issue with libsas and the pm80xx driver. Just wanted to let you know if you want to put that combo into your blacklist. Regards,
March 25, 20215 yr Author Thanks for reporting,@half . Would you post or pm the output of/just/local/emhttp/plugins/sas-spindown/sas-util?
April 3, 20215 yr Hey Doron, Thank you for taking the time and making this plugin. I'm sorry if you've already answered this a million times. I've tried to look through the thread, but it's difficult to follow along. I have HP H240 HBA and Constellation ES.3 ST3000NM0023 drives. I have been following this plugin since you've released it last year and I know that ES.3s were a problem. I cried a little and then figured I'll be patient and see if there are any fixes in the future 😭😉 In 6.8.3 unRAID would never even pretend that the drives were spun down and in the log I would get that red/green zebra response that I've seen in this thread. In 6.9.1, with your plugin installed (v0.85), unRAID "spin downs" the drives, the drives make a weird noise, the green dot becomes gray, but the drives appear to still be spinning and making spinning noise. The weird thing is that if I try to access the drive that is "spun down" (and yet still spinning) it takes a couple of seconds for it to "wake up" and allow me access to the files within. Maybe I am misunderstanding what "spinning down" is, but I assumed that they would literally stop spinning. Also, I did check if the drives were the source of the noise and yes they are, everything else in the system is too quiet to make any audible noise when the side panel is on. Is there anything you can do to either help me troubleshoot or is it a lost cause? Thanks and sorry again if you've answered this before. Edited April 3, 20215 yr by numerica
April 3, 20215 yr 7 hours ago, numerica said: Constellation ES.3 ST3000NM0023 I have a ST4000NM0023 and it spins down fine but is connected via an LSI controller. What output to you get from sdparm -C sense /dev/sdX when unraid thinks they are spun down. Sounds like your drives are going into an idle state, as takes 10-15 seconds for mine to spin up. Edited April 3, 20215 yr by SimonF
April 3, 20215 yr Author 9 hours ago, numerica said: I have HP H240 HBA and Constellation ES.3 ST3000NM0023 drives. (...) In 6.9.1, with your plugin installed (v0.85), unRAID "spin downs" the drives, the drives make a weird noise, the green dot becomes gray, but the drives appear to still be spinning and making spinning noise. The weird thing is that if I try to access the drive that is "spun down" (and yet still spinning) it takes a couple of seconds for it to "wake up" and allow me access to the files within. (...) Maybe I am misunderstanding what "spinning down" is, but I assumed that they would literally stop spinning. Yes, you've hit one of the main sources of grief with this entire "SAS Spin Down" journey. In brief, spindown is not well defined in the SCSI / SAS standards, and therefore it's implemented differently on different drives. For a slightly broader explanation: SAS drives have several power states. Two of those are of interest here: Idle (2) and Standby (3). In the standard, those states are defined rather loosely and abstractly; each of them is supposed to be a deeper "power saving" mode, in which the power draw is yet lower and the distance from the Active state is yet larger than the previous state. However, the standard does not dictate specific actions like turning off the spindle rotation, or anything specific actually. The net effect is that each drive+controller+firmware version combo demonstrates a slightly different behavior. From your description, it seems that your drive's firmware takes the drive to an idling mode which still includes spinning. Perhaps a firmware upgrade will change that; perhaps not. The frustrating thing is, that we don't know. So basically I'm trying to collect success/failure reports and code them into the plugin 😞 If you can send me (pm is fine) the output of the following command, that'll be helpful for documenting the situation. /usr/local/emhttp/plugins/sas-spindown/sas-util
April 3, 20215 yr 9 hours ago, SimonF said: I have a ST4000NM0023 and it spins down fine but is connected via an LSI controller. What output to you get from sdparm -C sense /dev/sdX when unraid thinks they are spun down. Sounds like your drives are going into an idle state, as takes 10-15 seconds for mine to spin up. /dev/sdf: SEAGATE ST3000NM0023 0003 Additional sense: Standby condition activated by command That's the output. Thanks for helping!
April 3, 20215 yr 13 minutes ago, numerica said: Standby condition activated by command Thats the result that should be seen, have you been able to check if the drive has spun down by touching it? or heat after it has been spun down a while.
April 3, 20215 yr 1 hour ago, SimonF said: Thats the result that should be seen, have you been able to check if the drive has spun down by touching it? or heat after it has been spun down a while. The way I have been determining it is by knowing what the box sounds like without mechanical drives and with. After putting the drives on stand-by, I see no reduction in noise.
April 8, 20215 yr Looks like something in unRAID 6.9.2 is resulting in drives not spinning down. Even trying to manually spin down doesn't help. Here's the Tools > System Log Apr 8 08:12:12 cobblednas emhttpd: spinning down /dev/sde Apr 8 08:12:12 cobblednas emhttpd: spinning down /dev/sdi Apr 8 08:12:12 cobblednas SAS Assist v0.85: Spinning down device /dev/sde Apr 8 08:12:12 cobblednas SAS Assist v0.85: Spinning down device /dev/sdi Apr 8 08:12:33 cobblednas emhttpd: read SMART /dev/sde Apr 8 08:12:33 cobblednas emhttpd: read SMART /dev/sdi Apr 8 08:15:15 cobblednas emhttpd: spinning down /dev/sdd Apr 8 08:15:15 cobblednas SAS Assist v0.85: Spinning down device /dev/sdd Apr 8 08:15:35 cobblednas emhttpd: read SMART /dev/sdd Apr 8 08:15:50 cobblednas emhttpd: spinning down /dev/sdg Apr 8 08:15:50 cobblednas SAS Assist v0.85: Spinning down device /dev/sdg Apr 8 08:15:56 cobblednas emhttpd: read SMART /dev/sdg Apr 8 08:18:26 cobblednas emhttpd: spinning down /dev/sdh Apr 8 08:18:26 cobblednas SAS Assist v0.85: Spinning down device /dev/sdh Apr 8 08:18:33 cobblednas emhttpd: read SMART /dev/sdh Apr 8 08:19:51 cobblednas emhttpd: spinning down /dev/sdf Apr 8 08:19:51 cobblednas emhttpd: spinning down /dev/sdc Apr 8 08:19:51 cobblednas SAS Assist v0.85: Spinning down device /dev/sdc Apr 8 08:19:51 cobblednas SAS Assist v0.85: Spinning down device /dev/sdf Apr 8 08:20:13 cobblednas emhttpd: read SMART /dev/sdf Apr 8 08:20:13 cobblednas emhttpd: read SMART /dev/sdc Apr 8 08:31:55 cobblednas emhttpd: spinning down /dev/sdi Apr 8 08:31:55 cobblednas SAS Assist v0.85: Spinning down device /dev/sdi Apr 8 08:32:18 cobblednas emhttpd: read SMART /dev/sdi Apr 8 08:42:14 cobblednas emhttpd: spinning down /dev/sde Apr 8 08:42:14 cobblednas SAS Assist v0.85: Spinning down device /dev/sde Apr 8 08:42:35 cobblednas emhttpd: read SMART /dev/sde Apr 8 08:45:17 cobblednas emhttpd: spinning down /dev/sdd Apr 8 08:45:17 cobblednas SAS Assist v0.85: Spinning down device /dev/sdd Apr 8 08:45:21 cobblednas emhttpd: read SMART /dev/sdd Apr 8 08:45:52 cobblednas emhttpd: spinning down /dev/sdg Apr 8 08:45:52 cobblednas SAS Assist v0.85: Spinning down device /dev/sdg Apr 8 08:46:13 cobblednas emhttpd: read SMART /dev/sdg Apr 8 08:48:28 cobblednas emhttpd: spinning down /dev/sdh Apr 8 08:48:28 cobblednas SAS Assist v0.85: Spinning down device /dev/sdh Apr 8 08:48:50 cobblednas emhttpd: read SMART /dev/sdh Apr 8 08:49:53 cobblednas emhttpd: spinning down /dev/sdf Apr 8 08:49:53 cobblednas emhttpd: spinning down /dev/sdc Apr 8 08:49:53 cobblednas SAS Assist v0.85: Spinning down device /dev/sdf Apr 8 08:49:53 cobblednas SAS Assist v0.85: Spinning down device /dev/sdc Apr 8 08:50:27 cobblednas emhttpd: read SMART /dev/sdf Apr 8 08:50:27 cobblednas emhttpd: read SMART /dev/sdc Edit: I did privately send Doron all diagnostic information to troubleshoot this. Edited April 8, 20215 yr by xaositek Send diagnostics
April 8, 20215 yr 48 minutes ago, xaositek said: Looks like something in unRAID 6.9.2 is resulting in drives not spinning down. Even trying to manually spin down doesn't help. Here's the Tools > System Log I don't think this is a SAS helper issue.
April 8, 20215 yr Author 1 hour ago, xaositek said: Looks like something in unRAID 6.9.2 is resulting in drives not spinning down. Even trying to manually spin down doesn't help. I have just migrated to 6.9.2 and SAS Spindown works perfectly for me (same as it did in 6.9.0 and 6.9.1). Do you have a distinctly different experience than you had in 6.9.1?
April 8, 20215 yr 16 minutes ago, doron said: I have just migrated to 6.9.2 and SAS Spindown works perfectly for me (same as it did in 6.9.0 and 6.9.1). Do you have a distinctly different experience than you had in 6.9.1? Yes, I have raised a bug report. Only one of my drives is affected. But its also affecting non SAS drives.
April 8, 20215 yr Updated to 6.9.2 as well, my HP branded Seagate drives aren't spinning down either but the Hitachi drives are. Here's logs in case you all need them. server-diagnostics-20210408-1550.zip
April 9, 20215 yr Author On 4/8/2021 at 6:37 PM, SimonF said: Yes, I have raised a bug report. Only one of my drives is affected. But its also affecting non SAS drives. @SimonF, are you seeing the same behavior (with your SATA drives) if you remove the plugin? If NOT (i.e. removing the plugin makes the issue go away), can you please test something for me -- 1. Remove the plugin 2. Manually spin down one of the SATA drives on which you've seen the problem. 3. Issue this command against this SATA drive: sdparm -ip di_target /dev/sdX 4. Check whether this command caused (a) the drive to spin up (b) message "read SMART /dev/sdX" to be logged. Thanks! (Un)fortunately I can't reproduce this issue locally. (There's also a related report that I'd like to get to the bottom of.)
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.