[Plugin] Spin Down SAS Drives

[SimonF]

12 minutes ago, deanpelton said:

 

So I ran the command, the status on the Unraid page did not change.

Running the status command immedately after gave the following:

/dev/sdf: SEAGATE   ST10000NM002G     E003

 

Unsure what this means.

You should see output like this if it spins down. Normally you wont see sense data if spinning.

root@Tower:~# sg_start  --readonly --pc=3 /dev/sde
root@Tower:~# sdparm --command=sense /dev/sde
    /dev/sde: SEAGATE   ST4000NM0023      XMGJ
Additional sense: Standby condition activated by command

 

[Gislum]

I have some consistency problems with the plugin, sometimes it works, sometimes not.

Currently, drives of type "ST32000444SS" are not able to spin down, I have tried restarting the server, reinstalling the plugin, checking for disk access and running some of the suggested commands, but nothing seems to work.

 

 

Here are some logs from what I have tried:

 

sas-util output.txt sas-util.txt syslog.txt

[deanpelton]

So I upgarded to a SAS 2308 controller (from the Dell H310), and also got a couple more SAS drives.

It appears to work intermittetly with the HUH721010AL5200 (WD Ultrastar DC HC510), but not at all with the Seagate x16's I have.

 

Has anyone have Seagate x16's they have managed to spin down?

 

On 2/21/2023 at 1:39 PM, deanpelton said:

Hi,

 

I'm running ST10000NM002G with a Dell H310 and this plugin appears not to work.

I get the following in my log:

 

Feb 21 12:50:00 Adam-HTPC  emhttpd: spinning down /dev/sdf
Feb 21 12:50:00 Adam-HTPC SAS Assist v2022.08.02: Spinning down device /dev/sdf
Feb 21 12:50:19 Adam-HTPC  emhttpd: read SMART /dev/sdf

 

However the drive does not appear to spin down.

I read through the entire thread and I can't see anyone comming on my specific combo, wondering if anyone else is running Seagate X16 SAS drive OR a Dell H310 and has it working?

 

 

[deanpelton]

Okay so I've been playing with this for a few days now and moved the Seagate SAS drive to UD.

The drive is not mounted and has no data or filesystem on it.

 

I can confirm also that the below command DOES spin the drive down, I hear the platter stop and the head park, and can feel the drive is no longer spinning.

sdparm --command=sense /dev/sde

sg_start -rp3 /dev/sdx

 

I also run this to confirm it's down

sdparm --command=sense /dev/sg4
   /dev/sg4: SEAGATE   ST10000NM002G     E004
Additional sense: Standby condition activated by command

 

The issue is... it spins back up! And continues to click and move the head like it's doing something perpetually.

I've turned off BMS (self scan), and confirmed it's not actually scanning using smartctl -x.

 

Really stuck as to what the hell the drive is doing, File Activiy shows nothing, I'm not running system fan. Open to idea's as to where to look, but I very much would like to fill my server with these but not if they are going to waste power.

Edited March 8, 2023 by deanpelton

[deanpelton]

Okay I managed to figure this out for my drive as it was a couple of issues by moving the drives into Unassigned Drives.

 

First issue: drive won't stay spun down.

Even with everything perfect, issuing --readonly --pc=3 to the drive would not consistently keep it spun down, it would spin down then back up again almost immediately.

I swapped from using that to using the line below:

sg_start -S /dev/sgX

This resulted in MUCH more consistent spin down for my Seagate x16 drives.

Is there a reason pc=3 is preferable to -S?

 

 

Second issue: spinning back up again. When using sg_start --readonly --pc=3 when I spun down my drives, the log would look like this:

Mar  8 22:00:07 Adam-HTPC  emhttpd: spinning down /dev/sde
Mar  8 22:00:07 Adam-HTPC SAS Assist v2022.08.02: Spinning down device /dev/sde
Mar  8 22:00:26 Adam-HTPC  emhttpd: read SMART /dev/sde

 

The timing was never consistent either, it would be anywhere from 5 - 30 seconds, and then it would spin up again.

Thus I assumed something was accessing it that doesn't spin up SATA drives, at this moment I suspect it's Telegraf, which I run for logging. Pausing / Stopping this appears to fix my issue entirely!

I suspect the smart_ctl that Telegraf is using appears to spin up the sas drives like it used to with sata drives, and because they take so long to spin down it appears to the user like they never spun down at all.

 

I also suspect that because the Seagate drives take SO long to spin down, that the standby setting is not yet set in SmartCtl, so when Telegraf pings them every 3 seconds, mid-spindown, it spins the drive back up again.

[doron]

2 hours ago, deanpelton said:

Is there a reason pc=3 is preferable to -S?

Yes. When using -S, several drive families then require an explicit spin-up command sent to them to spin up (in sg_start jargon, that'd be "-s" - small s). That is not what Unraid expects - it expects an implicit wake up whenever subsequent i/o is issued against the drive. So while "-S" would cause more drives to spin down, it will also cause some of them to stay spun down, which would translate into Unraid marking the drive as faulty and various other small hells breaking loose.

 

2 hours ago, deanpelton said:

Pausing / Stopping this appears to fix my issue entirely!

Yes, this is typically the solution to spin-ups closely following spin-downs.

Glad you found the issue.

[djtodd]

OK, I have a number of "Factory Refurbed" Seagate Exos WL 10TB SAS 12G 3.5 drives in my backup system.

 

They won't spin down for the life of me. Is there any troubleshooting I can do?

 

They show as the ever so "I trust this name" as "OOS1000G"

[deanpelton]

8 hours ago, djtodd said:

OK, I have a number of "Factory Refurbed" Seagate Exos WL 10TB SAS 12G 3.5 drives in my backup system.

 

They won't spin down for the life of me. Is there any troubleshooting I can do?

 

They show as the ever so "I trust this name" as "OOS1000G"

 

Are you running any apps that would poll these drives for status? E.g. Autofan or Telegraf?

 

When you spin the drives down can you hear them spin down and back up again?

 

You might just be unlucky, but I managed to get mine working by putting exclusions in telegraf and Autofan.

[djtodd]

23 hours ago, deanpelton said:

 

Are you running any apps that would poll these drives for status? E.g. Autofan or Telegraf?

 

When you spin the drives down can you hear them spin down and back up again?

 

You might just be unlucky, but I managed to get mine working by putting exclusions in telegraf and Autofan.

 

Hah, no its definitely the drives. I can click spin down forever, and they never do, not even for a second and then spin up.

 

This is on a very bare bones server that I use for backups. It sits in S3 sleep until woken by my main Unraid server. Then 5 minutes later backups run. Then after 30 minutes of activity the drives should spin down, then 15 minutes later go back to sleep.

 

The Toshiba SATA drives spin down nicely, and the SAS drives in my main server spin down, its just these particular odd drives.

 

[deanpelton]

9 minutes ago, djtodd said:

 

Hah, no its definitely the drives. I can click spin down forever, and they never do, not even for a second and then spin up.

 

This is on a very bare bones server that I use for backups. It sits in S3 sleep until woken by my main Unraid server. Then 5 minutes later backups run. Then after 30 minutes of activity the drives should spin down, then 15 minutes later go back to sleep.

 

The Toshiba SATA drives spin down nicely, and the SAS drives in my main server spin down, its just these particular odd drives.

 

 

Have you tried this command?

Note this can put the drives into a standby state that could require a power cycle to bring them back up.
You would need to find the number of the drive (sgX instead of shX).

 

sg_start -S /dev/sgX

 

I have no success with the command the script uses which is --pc=3 flag instead of -S, however scrolling back through this thread you will see that there are people whose drives this disabled due to it powering down. So best to test this with the array off and do a reboot after.

[SimonF]

5 hours ago, djtodd said:

 

Hah, no its definitely the drives. I can click spin down forever, and they never do, not even for a second and then spin up.

 

This is on a very bare bones server that I use for backups. It sits in S3 sleep until woken by my main Unraid server. Then 5 minutes later backups run. Then after 30 minutes of activity the drives should spin down, then 15 minutes later go back to sleep.

 

The Toshiba SATA drives spin down nicely, and the SAS drives in my main server spin down, its just these particular odd drives.

 

Do see any entries in the log for SAS Spindown Helper?

[djtodd]

On 3/22/2023 at 3:16 AM, SimonF said:

Do see any entries in the log for SAS Spindown Helper?

 

Yeah, and I think I know whats going on, but I have no idea how to stop it:

 

Mar 23 15:10:35 Backup SAS Assist v2022.08.02: Spinning down device /dev/sdd

Mar 23 15:10:54 Backup emhttpd: spinning down /dev/sdf

Mar 23 15:10:54 Backup SAS Assist v2022.08.02: Spinning down device /dev/sdf

Mar 23 15:11:21 Backup emhttpd: read SMART /dev/sdd

Mar 23 15:11:21 Backup emhttpd: read SMART /dev/sdf

 

So as soon as they're spun down for some reason SMART is read and spins them back up again.

 

[SimonF]

25 minutes ago, djtodd said:

 

Yeah, and I think I know whats going on, but I have no idea how to stop it:

 

Mar 23 15:10:35 Backup SAS Assist v2022.08.02: Spinning down device /dev/sdd

Mar 23 15:10:54 Backup emhttpd: spinning down /dev/sdf

Mar 23 15:10:54 Backup SAS Assist v2022.08.02: Spinning down device /dev/sdf

Mar 23 15:11:21 Backup emhttpd: read SMART /dev/sdd

Mar 23 15:11:21 Backup emhttpd: read SMART /dev/sdf

 

So as soon as they're spun down for some reason SMART is read and spins them back up again.

 

Are you running telegraph or something similar, by default it does not use the -n option on smartctl so will spin up the disks.

 

looks like something is running approx 25-30secs. Any dockers or plugins for monitoring?

[djtodd]

2 minutes ago, SimonF said:

Are you running telegraph or something similar, by default it does not use the -n option on smartctl so will spin up the disks.

 

looks like something is running approx 25-30secs. Any dockers or plugins for monitoring?

 

None of the above. However, I think it may be the s3 sleep plugin. I used the file activity plug in and saw nothing, as expected.

 

Status monitors the hardware status of the device

Counters monitors the read/write counters of the device

 

I've changed the device activity monitoring from "Status" to "Counters" as this is my backup server. It basically gets woken at 2am, backups run with it as the destination, then when the drives go inactive for 15 minutes, it put the system to sleep.

 

I've hit the big spin down button, and now I'm waiting.

[SimonF]

6 minutes ago, djtodd said:

 

None of the above. However, I think it may be the s3 sleep plugin. I used the file activity plug in and saw nothing, as expected.

 

Status monitors the hardware status of the device

Counters monitors the read/write counters of the device

 

I've changed the device activity monitoring from "Status" to "Counters" as this is my backup server. It basically gets woken at 2am, backups run with it as the destination, then when the drives go inactive for 15 minutes, it put the system to sleep.

 

I've hit the big spin down button, and now I'm waiting.

looking at tbe plugin it uses hdparm that is not 

compatible with sas drives.

[doron]

26 minutes ago, SimonF said:

looking at tbe plugin it uses hdparm that is not 

compatible with sas drives.

That is correct. I proposed a patch to the author, hopefully he will consider merging it in.

[djtodd]

28 minutes ago, SimonF said:

looking at tbe plugin it uses hdparm that is not 

compatible with sas drives.

 

3 minutes ago, doron said:

That is correct. I proposed a patch to the author, hopefully he will consider merging it in.

 

 

Well, just using counters seems to work. I spun down the drives, waited a few minutes, started a backup which spun up the Parity and Data disk(s). 15 minutes after it finished the system went to sleep.

 

I'll monitor over the next few days and make sure my backups run and the system is asleep in the morning.

[SimonF]

21 minutes ago, doron said:

That is correct. I proposed a patch to the author, hopefully he will consider merging it in.

Thanks will discuss with the author to get it merged.

[bonienl]

Currently S3-sleep accepts Unraid version 6.9.0 and higher.

I don't remember since which Unraid version sdspin is included. Anyone knows?

 

[SimonF]

8 minutes ago, bonienl said:

Currently S3-sleep accepts Unraid version 6.9.0 and higher.

I don't remember since which Unraid version sdspin is included. Anyone knows?

 

 

From 6.9 I believe sdspin was introduced.

 

was added for this https://wiki.unraid.net/Manual/Release_Notes/Unraid_OS_6.9.0#SAS_Spinning_Control

Edited March 23, 2023 by SimonF

[bonienl]

Thanks @doron

I merged your PR and made a new version available, should be there 'soon' ...

 

[Octalbush]

Unfortunately this plugin causes my drives to get read errors, and today it even made one go disabled and I am currently rebuilding it now. It seems to work ok when I let Unraid manage the spindowns, but if I spin it down using the web gui button, it can cause errors, and in todays case, the drive to be disabled entirely. I notice I get some errors over time even if I let Unraid spin them down based on time, but they don't disable the drive. I am using Toshiba MG06 8TB SAS drives.

[JimJamUrUnraid]

Is anyone having issues with western digital drives and this plugin? It seems as though all of my western digital drives don’t spin down.

[doron]

3 hours ago, Octalbush said:

Unfortunately this plugin causes my drives to get read errors, and today it even made one go disabled and I am currently rebuilding it now. It seems to work ok when I let Unraid manage the spindowns, but if I spin it down using the web gui button, it can cause errors, and in todays case, the drive to be disabled entirely. I notice I get some errors over time even if I let Unraid spin them down based on time, but they don't disable the drive. I am using Toshiba MG06 8TB SAS drives.

It turns out that the standby (aka "spin down") commands are interpreted differently by different SAS HDDs, unlike SATA drives. These commands are not well standardized. Some drives, after receiving this command, do spin down, but expect an explicit "spin up" command to start revolving again. This behavior is not compatible with Unraid, which expects a spun-down drive to spin back up automatically when the next I/O is directed at it. This translates to read or write errors (depending on the I/O that was underway); if it was a write, you'll get the drive red-x'ed, like you experienced.

This has been reported a lot with Seagate drives, and a bit less with Toshiba drives.

 

There's very little the plugin can do about it. I can add the MG06 to the exclusion list - which will mean that the plugin will simply avoid touching these drives. 

Towards that end can you share the output of:

 

/usr/local/emhttp/plugins/sas-spindown/sas-util

 

[Octalbush]

On 3/30/2023 at 2:10 AM, doron said:

It turns out that the standby (aka "spin down") commands are interpreted differently by different SAS HDDs, unlike SATA drives. These commands are not well standardized. Some drives, after receiving this command, do spin down, but expect an explicit "spin up" command to start revolving again. This behavior is not compatible with Unraid, which expects a spun-down drive to spin back up automatically when the next I/O is directed at it. This translates to read or write errors (depending on the I/O that was underway); if it was a write, you'll get the drive red-x'ed, like you experienced.

This has been reported a lot with Seagate drives, and a bit less with Toshiba drives.

 

There's very little the plugin can do about it. I can add the MG06 to the exclusion list - which will mean that the plugin will simply avoid touching these drives. 

Towards that end can you share the output of:

 

/usr/local/emhttp/plugins/sas-spindown/sas-util

 

Here's the output:
 

sdb     | MG06SCA800A   | 1000:0097:1028:1f45   |  n/a  | 
sdc     | MG06SCA800A   | 1000:0097:1028:1f45   |  n/a  | 
sdd     | MG06SCA800A   | 1000:0097:1028:1f45   |  n/a  | 
sdg     | MG06SCA800A   | 1000:0097:1028:1f45   |  n/a  | 
sdi     | MG06SCA800A   | 1000:0097:1028:1f45   |  n/a  |