Disk wont stay spun down - i am 99.9 % sure its SMART - not a plugin, docker or vm

As it is a raid card you are likely to have to setup additional smart settings for each disk i.e. contro!ler of cciss for each disk click on drive name on main view and look for smart settings

Edited September 11, 2022 by SimonF

Thanks for your response, i had to do that in the past (as of 6.9.x), but 6.10 detects them correctly on auto settings.

nevertheless, it does not change the behaviour if set manually (or now automatic) - the disks spin up and do a smart read without increasing read or write bytes and no plugin/docker/vm/share access.

 

so while the raid card is/was not the most convenient thing to use, its not the culprit.

Some RAID controllers don't fully/correctly support spin down, was going to suggest testing on an onboard SATA port but it looks like there are none.

small addendum to further prove my point:

 

when i spin down a disk and then execute a smart read with correct parameters:

smartctl -A /dev/sdb  -d cciss,0 -i

 

the generic disk info is displayed, then it hangs for a second while the disk spins up and then displays the rest of the smart data.

 

 

 

Quote

Some RAID controllers don't fully/correctly support spin down, was going to suggest testing on an onboard SATA port but it looks like there are none.

 

yes there are none. but it does spin down. it correctly reports standby when i check it (dont have the command by hand now), the power consumption of the server goes down (so it is really not spinning) and the parity stays spun down (=correctly spun down).

 

so as i said, its not my raid controller (although i completely agree they can be bothersome)

 

further interesting thing, i have set Tunable (poll_attributes) to 0 in disk config, yet i have the familiar

 

Sep 12 10:31:46 hp emhttpd: spinning down /dev/sdd
Sep 12 10:42:25 hp emhttpd: read SMART /dev/sdd

 

so if anyone knows how to completely stop that smart monitoring job please just let me know. once it is disabled and i dont see those log entries and disks still spin up only then i am convinced its not a smart issue.

 

 

Edited September 12, 2022 by Michael H

22 hours ago, Michael H said:

smartctl -A /dev/sdb  -d cciss,0 -i

Does smartctl -n standby -A /dev/sdb  -d cciss,0 -i spin up a disk that is spun down?

9 hours ago, SimonF said:

Does smartctl -n standby -A /dev/sdb  -d cciss,0 -i spin up a disk that is spun down?

 

just tested, yes it does.

 

root@hp:~# smartctl -n standby -A /dev/sdb  -d cciss,0 -i
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.15.46-Unraid] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

CHECK POWER MODE: incomplete response, ATA output registers missing
CHECK POWER MODE not implemented, ignoring -n option
=== START OF INFORMATION SECTION ===
Device Model:     WDC WD101EFBX-68B0AN0
Serial Number:    VCHPY2BP
LU WWN Device Id: 5 000cca 0b0d81072
Firmware Version: 85.00A85
User Capacity:    10,000,831,348,736 bytes [10.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database 7.3/5387
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Sep 13 17:32:46 2022 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0004   128   128   054    Old_age   Offline      -       108
  3 Spin_Up_Time            0x0007   139   139   024    Pre-fail  Always       -       594 (Average 572)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       3068
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
  9 Power_On_Hours          0x0012   099   099   000    Old_age   Always       -       10844
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       40
192 Power-Off_Retract_Count 0x0032   056   056   000    Old_age   Always       -       53546
193 Load_Cycle_Count        0x0012   056   056   000    Old_age   Always       -       53546
194 Temperature_Celsius     0x0002   196   196   000    Old_age   Always       -       33 (Min/Max 24/43)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

 

 

meanwhile the log (i tested for all 3 disks)

 

Sep 13 17:34:24 hp emhttpd: spinning down /dev/sdb
Sep 13 17:34:27 hp emhttpd: spinning down /dev/sdc
Sep 13 17:34:43 hp emhttpd: read SMART /dev/sdb
Sep 13 17:34:43 hp emhttpd: read SMART /dev/sdc
Sep 13 17:34:53 hp kernel: sdc: sdc1
Sep 13 17:34:56 hp kernel: sdb: sdb1
Sep 13 17:35:17 hp emhttpd: spinning down /dev/sdb
Sep 13 17:35:20 hp emhttpd: spinning down /dev/sdc
Sep 13 17:35:39 hp kernel: sdd: sdd1
Sep 13 17:35:39 hp kernel: sdd: sdd1
Sep 13 17:35:39 hp kernel: sdd: sdd1
Sep 13 17:35:50 hp emhttpd: read SMART /dev/sdd
Sep 13 17:35:50 hp emhttpd: read SMART /dev/sdc
Sep 13 17:35:50 hp kernel: sdd: sdd1
Sep 13 17:35:50 hp kernel: sdd: sdd1
Sep 13 17:35:50 hp kernel: sdc: sdc1

 

You may need to use a different device name for the controller rather than the read sdX name

 

To look at disks behind HP Smart Array controllers, use syntax such as:
smartctl -a -d cciss,0 /dev/cciss/c0d0 (cciss driver under Linux)
smartctl -a -d cciss,0 /dev/sg2 (hpsa or hpahcisr drivers under Linux)
smartctl -a -d cciss,0 /dev/ciss0 (under FreeBSD)

 

what does lsscsi -g output

 

Some controllers you need to use the card device in smartctl not the disk device.

Output of lsscsi -g

 

root@hp:~#  lsscsi -g
[0:0:0:0]    disk    Kingston DataTraveler 3.0       /dev/sda   /dev/sg0 
[1:0:0:0]    storage HP       P840             7.00  -          /dev/sg1 
[1:0:1:0]    disk    ATA      WDC WD101EFBX-68 0A85  /dev/sdb   /dev/sg2 
[1:0:2:0]    disk    ATA      ST10000VN0008-2P SC61  /dev/sdc   /dev/sg3 
[1:0:3:0]    disk    ATA      WDC WD101EFBX-68 0A85  /dev/sdd   /dev/sg4 
[1:0:4:0]    disk    ATA      SanDisk SSD PLUS 00RL  /dev/sde   /dev/sg5 
[1:0:5:0]    disk    IBM-ESXS ST31000424SS     BC2D  /dev/sdf   /dev/sg6 
[1:0:6:0]    enclosu HP       P840             7.00  -          /dev/sg7 

 

so if its the sg thing its the hpahcisr driver? which one would that be under the smart controller types?

 

 

so the command for smartctl with sg2 smartctl -a -n standby -d cciss,0 /dev/sg2 also wakes up the disk, but produces more output

 

smartctl -a -n standby -d cciss,0 /dev/sg2
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.15.46-Unraid] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD101EFBX-68B0AN0
Serial Number:    VCHPY2BP
LU WWN Device Id: 5 000cca 0b0d81072
Firmware Version: 85.00A85
User Capacity:    10,000,831,348,736 bytes [10.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database 7.3/5387
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Sep 13 18:56:25 2022 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers missing
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (   87) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 998) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0004   128   128   054    Old_age   Offline      -       108
  3 Spin_Up_Time            0x0007   138   138   024    Pre-fail  Always       -       582 (Average 594)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       3072
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
  9 Power_On_Hours          0x0012   099   099   000    Old_age   Always       -       10846
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       40
192 Power-Off_Retract_Count 0x0032   056   056   000    Old_age   Always       -       53556
193 Load_Cycle_Count        0x0012   056   056   000    Old_age   Always       -       53556
194 Temperature_Celsius     0x0002   196   196   000    Old_age   Always       -       33 (Min/Max 24/43)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

 

 

 

i played around trying to disable smart completely with no success. however, i found following information in smartd:

 

Quote

/dev/sdd [SAT], not found in smartd database 7.3/5387.

 

for the 2 disks that dont stay spun down. so could this be the underlying issue? that smart monitoring just does not know whether the disks are spun down because it does not know how to check?

15 minutes ago, Michael H said:

i played around trying to disable smart completely with no success. however, i found following information in smartd:

 

 

for the 2 disks that dont stay spun down. so could this be the underlying issue? that smart monitoring just does not know whether the disks are spun down because it does not know how to check?

I think you may need to use sg1 or sg7 for the enclosure the number after the comma is the drive so starting at 0 upto 15. but may depend on port conmected on controller

if you hover over the hp card in tools sysdevs you shou!d see more info and the driver being used.

10 hours ago, SimonF said:

I think you may need to use sg1 or sg7 for the enclosure the number after the comma is the drive so starting at 0 upto 15. but may depend on port conmected on controller

 

does not change anything.

 

hspa driver is used.

can you tell me how to disable smart completely?

or which command is used by unraid to monitor smart so i can check the disk spin up with that?

2 hours ago, Michael H said:

can you tell me how to disable smart completely?

or which command is used by unraid to monitor smart so i can check the disk spin up with that?

Only option I know of is setting poll to zero but this will also mean temps are not updated.

 

scripts unraid uses are in /usr/local/sbin

 

 

sdspin has options up down and status

 

smartctl_type is used to call smartctl with extra settings from gui.

 

I will have a look to see if I can find the correct way to setup for you card and look at the smartctl src over the weekend.

 

Uses 

Specifically it looks at

/sys/block/sdb/sdb1/stat

Then adds the 3rd and 7th values (sectors read and sectors written).

 

Detail in the tread.

 

 

thanks for trying to help!!

 

a few more odd things to consider:

i already have the poll to 0, but unraid does not care and i still see the spinup and smart calls (and temperature)

i also tried to brute force it to not read smart by renaming smartctl_type, smartctl and smartd but still spin up and reading. for now i did not dare to delete them, but it gets more and more tempting

 

i even set the scheduler to none so that it would not be executed, and  the temps disappeared. i dared to hope but was dissapointed again:

 

Sep 15 23:07:40 hp emhttpd: shcmd (24249): echo none > /sys/block/sdc/queue/scheduler
Sep 15 23:07:40 hp emhttpd: shcmd (24250): echo none > /sys/block/sdd/queue/scheduler
Sep 15 23:07:40 hp emhttpd: shcmd (24251): echo none > /sys/block/sdb/queue/scheduler
Sep 15 23:07:40 hp kernel: mdcmd (36): set md_num_stripes 1280
Sep 15 23:07:40 hp kernel: mdcmd (37): set md_queue_limit 80
Sep 15 23:07:40 hp kernel: mdcmd (38): set md_sync_limit 5
Sep 15 23:07:40 hp kernel: mdcmd (39): set md_write_method
Sep 15 23:07:46 hp emhttpd: spinning down /dev/sdd
Sep 15 23:07:48 hp emhttpd: spinning down /dev/sdb
Sep 15 23:18:18 hp emhttpd: read SMART /dev/sdb
Sep 15 23:18:31 hp emhttpd: read SMART /dev/sdd

 

as far as i can tell from the thread you linked there is also no increase in the read/write counter and it spins up... also no solution there as far as i can see, just like the other similar threads. as you are a respected community member, maybe can you try to take it to the unraid people to be looked at as the bug as it seems to be?

 

also if you want to remote into my server to try things and poke around, we can surely arrange that. i know how difficult it is to remotely diagnose a problem.

 

 

 

Edited September 15, 2022 by Michael H

The read smart in the log is only after Unraid detecting drive is spun up.

 

 

Sep 15 23:07:46 hp emhttpd: spinning down /dev/sdd
Sep 15 23:07:48 hp emhttpd: spinning down /dev/sdb
Sep 15 23:18:18 hp emhttpd: read SMART /dev/sdb
Sep 15 23:18:31 hp emhttpd: read SMART /dev/sdd

and is 11 mins after spinning down but will continue to look are you able to post a more recent diagnostics?

 

Edited September 16, 2022 by SimonF

1 hour ago, SimonF said:

The read smart in the log is only after Unraid detecting drive is spun up.

 

 

Sep 15 23:07:46 hp emhttpd: spinning down /dev/sdd
Sep 15 23:07:48 hp emhttpd: spinning down /dev/sdb
Sep 15 23:18:18 hp emhttpd: read SMART /dev/sdb
Sep 15 23:18:31 hp emhttpd: read SMART /dev/sdd

and is 11 mins after spinning down but will continue to look are you able to post a more recent diagnostics?

 

 

i think I'm misunderstood. i know that smart is only read after the disk is spun up (theoretically).

my point is that unraid does not correctly detect that the disk is spun down, hence issues the smart command which REALLY wakes up the disk.

 

to fully prove this theory (altough in my mind it is certain) i want to completely block smart. however, setting poll interval to 0, renaming the smart commands in sbin and using "none" as the scheduler have all not helped in really disabling smart. so regardless wheter my theory is correct or not, i would say its at least a bug that i am not able to disable smart.

 

here are more recent diagnostics.

hp-diagnostics-20220916-1034.zip

On 9/16/2022 at 9:39 AM, Michael H said:

 

i think I'm misunderstood. i know that smart is only read after the disk is spun up (theoretically).

my point is that unraid does not correctly detect that the disk is spun down, hence issues the smart command which REALLY wakes up the disk.

 

to fully prove this theory (altough in my mind it is certain) i want to completely block smart. however, setting poll interval to 0, renaming the smart commands in sbin and using "none" as the scheduler have all not helped in really disabling smart. so regardless wheter my theory is correct or not, i would say its at least a bug that i am not able to disable smart.

 

here are more recent diagnostics.

hp-diagnostics-20220916-1034.zip

This is not related to the spinup of the ATA drives, but notice you have a SAS drive. You need to install the SAS spindown helper if you want that drive to spin down.

 

@doron has with the sas spindown plugins wrapper that was installed that may be able to help detect what is calling smartctl but not sure if it will be installed on 6.10.x

 

#!/bin/bash

# Spin down SAS drives - stopgap scripts until Unraid does organically.
#
# This script is a wrapper for "smartctl" - which in 7.1 does not support the "-n standby"
# flag for SAS drive. This wrapper works around that, by checking whether the drive is SAS
# and if so, avoid calling smartctl (return silently).
#
# v&version;
# &copyright;

. &plugindir;/functions

DEVICE="${@: -1}"

$DEBUG && Log "debug: smartctl wrapper caller is $(cat /proc/$PPID/comm), grandpa is $(cat /proc/$(cat /proc/$PPID/stat | cut -d" " -f4)/comm), device $DEVICE, args \"$@\""


if [[ "$@" =~ \-n\ +standby\  || "$@" =~ "--nocheck=standby" ]] ; then

	if IsSBY $DEVICE ; then

		$DEBUG && Log  "debug: device $DEVICE is spun down, smartctl evaded"
		echo "SAS Assist Plugin: $DEVICE is in standby mode (spun down), smartctl exits(2)"

		exit 2		# Match smartctl's exit code, thanks @segator

	fi

fi

$REALSMART "$@"

 

 

main sdspin process uses hdparm -C to get device standby status, What is the output for each of the ATA drives?

 

@limetech @dlandon any other suggestions?

Update to 6.11-rc5, boot in safe mode, and post diagnostics so we can look at a pure Unraid system with latest version.

 @dlandon updated to rc5, booted in safe mode, disabled virtlib and docker, spun down disks and same issue:

 

Quote

Sep 18 19:58:52 hp  emhttpd: shcmd (118): umount /etc/libvirt
Sep 18 19:58:56 hp kernel: sdd: sdd1
Sep 18 19:58:56 hp kernel: sdb: sdb1
Sep 18 19:58:56 hp kernel: sdc: sdc1
Sep 18 19:58:56 hp kernel: sdf: sdf1 sdf2
Sep 18 19:59:17 hp  emhttpd: spinning down /dev/sdc
Sep 18 19:59:19 hp  emhttpd: spinning down /dev/sdb
Sep 18 19:59:22 hp  emhttpd: spinning down /dev/sdd
Sep 18 19:59:30 hp kernel: sdf: sdf1 sdf2
Sep 18 19:59:35 hp  nmbd[11860]: [2022/09/18 19:59:35.852049,  0] ../../source3/libsmb/nmblib.c:923(send_udp)
Sep 18 19:59:35 hp  nmbd[11860]:   Packet send failed to 192.168.122.255(138) ERRNO=Network is unreachable
Sep 18 20:00:03 hp kernel: sdf: sdf1 sdf2
Sep 18 20:00:36 hp kernel: sdf: sdf1 sdf2
Sep 18 20:01:08 hp kernel: sdf: sdf1 sdf2
Sep 18 20:01:41 hp kernel: sdf: sdf1 sdf2
Sep 18 20:02:14 hp kernel: sdf: sdf1 sdf2
Sep 18 20:02:46 hp kernel: sdf: sdf1 sdf2

 

also your openfiles plugin never reported anything (which is to be expected as read and write counters also dont go up)

 

 

@SimonF nevermind the sas drive, it is passed through directly to a vm, it will never have to be spun down.

 

hdparm -C correctly reports the drive as standby when spun down. so i wrote a script to see if the smart read is done while it still reports as standby and what is curious is that it is indeed reporting active a few seconds before the smart status is read. but then the drive does not respond for a good 10 seconds (my script reads every second).

 

so i guess you are right that smart only reads when the drive is active. which leaves the question why it is active, when there is no plugin, docker or vm running or any access on the shares... i hate this problem

 

 

Quote

Sep 18 20:38:43 hp  emhttpd: read SMART /dev/sdb

 

Quote

Sun Sep 18 20:38:29 CEST 2022

/dev/sdb:
 drive state is:  standby

Sun Sep 18 20:38:30 CEST 2022

/dev/sdb:
 drive state is:  active/idle

 

Sun Sep 18 20:38:44 CEST 2022

/dev/sdb:
 drive state is:  active/idle

 

 

here are the new diagnostics

 

 

hp-diagnostics-20220918-2012.zip

Edited September 18, 2022 by Michael H

Nothing in the diagnostics stands out to me, but because we have diagnostics of a safe mode with no plugins or other things running, I can present it for evaluation.  I'll present it to LT and see if we can come up with some answers.

Install the Doron SAS Spindown plugin and see if that helps.

thanks for taking it to LT.

 

the spindown plugin does not help. the spindown itself works fine and the drives in question are sata, not sas drives.

Thanks for the info.  I'm passing it along.