• [6.9.2] Parity Drive will not spin down via GUI or Schedule


    SimonF
    • Minor

    Since upgrading to 6.9.2 I cannot spin down parity via GUI, as soon as I spin down it comes back up.

     

    Apr 8 10:18:45 Tower emhttpd: spinning down /dev/sdf
    Apr 8 10:18:58 Tower SAS Assist v0.85: Spinning down device /dev/sdf
    Apr 8 10:18:58 Tower emhttpd: read SMART /dev/sdf

     

    Revert to 6.9.1 issue no longer happens.

     

    I can manually spin down the drive. All other array drives which are also SAS spin down fine.

    root@Tower:~# sg_start -rp 3 /dev/sdf
    root@Tower:~# sdparm -C sense /dev/sdf
        /dev/sdf: SEAGATE   ST4000NM0023      XMGJ
    Additional sense: Standby condition activated by command

     

     

    Also is it possible to get an updated version of smartctl added.

     

    Will continue to do more testing.

     



    User Feedback

    Recommended Comments



     I see a similar thing happening in 6.9.2.

     

    The disks connected to my Adaptec HBA are spun up as soon as a SMART read is done by emhttpd for those devices.

    Disks directly connected to SATA ports on the motherboard are not SMART read and stay spun down.

     

    Apr 8 12:01:22 vesta emhttpd: spinning down /dev/sde
    Apr 8 12:01:34 vesta emhttpd: spinning down /dev/sdg
    Apr 8 12:01:42 vesta emhttpd: spinning down /dev/sdm
    Apr 8 12:01:42 vesta emhttpd: spinning down /dev/sdk
    Apr 8 12:01:42 vesta emhttpd: spinning down /dev/sdd
    Apr 8 12:01:42 vesta emhttpd: spinning down /dev/sdb
    Apr 8 12:01:42 vesta emhttpd: spinning down /dev/sdc
    Apr 8 12:01:42 vesta emhttpd: spinning down /dev/sdl
    Apr 8 12:07:12 vesta emhttpd: read SMART /dev/sdb
    Apr 8 12:07:12 vesta emhttpd: read SMART /dev/sdc
    Apr 8 12:07:31 vesta emhttpd: read SMART /dev/sdd
    Apr 8 12:07:31 vesta emhttpd: read SMART /dev/sde
    Apr 8 12:07:41 vesta emhttpd: read SMART /dev/sdg
    Apr 8 12:08:01 vesta emhttpd: read SMART /dev/sdk
    Apr 8 12:08:01 vesta emhttpd: read SMART /dev/sdl
    Apr 8 12:08:11 vesta emhttpd: read SMART /dev/sdm

    image.png.8235f0854dcfcec833936864e3561000.png

    Link to comment
    Share on other sites

    All of mine are connected via LSI controllers. Either motherboard or external HBA, Only the SEAGATE one doesnt spin down which is the parity. Mine are all SAS on my test system.

     

    image.thumb.png.8687d3d9e3d89cc9e6d0035a1da647a3.png

     

    image.thumb.png.b6a3eaffd531290d944cc542bcfcf6e8.png

    Link to comment
    Share on other sites

    I found this on my system as well across all my SAS drives. I've sent diagnostic info over to Doron who maintains the "Spin Down SAS Drives" plug-in.

     

     

    Link to comment
    Share on other sites
    3 minutes ago, xaositek said:

    I found this on my system as well across all my SAS drives. I've sent diagnostic info over to Doron who maintains the "Spin Down SAS Drives" plug-in.

     

     

    I dont believe this is an issue with Doron's plugin as also happening with non SAS drives for bonienl,

    Link to comment
    Share on other sites

    Fair point completely @SimonF. The research I had available to me across my two unRAID servers had limited this issue only to the server with SAS drives. My other unRAID box which has only SATA drives is working as expected.

    Link to comment
    Share on other sites
    6 minutes ago, xaositek said:

    Fair point completely @SimonF. The research I had available to me across my two unRAID servers had limited this issue only to the server with SAS drives. My other unRAID box which has only SATA drives is working as expected.

    May be related to this or the new Kernel and SAS HBAs.

     

    emhttpd: detect out-of-band device spin-up

    Link to comment
    Share on other sites

    Data point: Both my parity drives (SAS HGST) spin down upon request, no issue.

    HDD is HGST, ctrl is onboard SM (LSI).

    Link to comment
    Share on other sites

    No spin down issue here for 6.9.2, working as expected, six SATA HDDs with LSI HBA

    Link to comment
    Share on other sites
    6 hours ago, bonienl said:

    Apr 8 12:07:12 vesta emhttpd: read SMART /dev/sdb

     

    That message is output as a result of seeing I/O on partition 1 of the device.  That is, code has detected I/O counters have incremented since the last time they were checked (polled every second).  If there has been I/O to partition 1 then we assume the device must be spun-up.  The 'read SMART' command is then issued to get the device temperature.

     

    Specifically it looks at

    /sys/block/sdb/sdb1/stat

     

    Then adds the 3rd and 7th values (sectors read and sectors written).

     

    I don't see this happening on my test servers, but I don't have any SAS devices.

    Link to comment
    Share on other sites
    26 minutes ago, limetech said:

     

     

    /sys/block/sdb/sdb1/stat

     

     

    root@Tower:/usr/sbin# date
    Thu Apr  8 18:02:18 BST 2021
    root@Tower:/usr/sbin# cat /sys/block/sdf/sdf1/stat 
         132     2492    21596     1175      136     2488    20992     2102        0     2021     3277        0        0        0        0        0        0
    root@Tower:/usr/sbin# date
    Thu Apr  8 18:02:39 BST 2021
    root@Tower:/usr/sbin# date
    Thu Apr  8 18:02:57 BST 2021
    root@Tower:/usr/sbin# cat /sys/block/sdf/sdf1/stat 
         132     2492    21596     1175      136     2488    20992     2102        0     2021     3277        0        0        0        0        0        0
    root@Tower:/usr/sbin# date 
    Thu Apr  8 18:03:04 BST 2021
    root@Tower:/usr/sbin# 

     

    Apr  8 18:02:44 Tower emhttpd: spinning down /dev/sdf
    Apr  8 18:02:44 Tower SAS Assist v0.85: Spinning down device /dev/sdf
    Apr  8 18:02:49 Tower emhttpd: read SMART /dev/sdf
    Apr  8 18:02:49 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdf, args "-A /dev/sdf"

    This is the output I get for my parity drive when I click green ball to spin down.

    Link to comment
    Share on other sites
    1 hour ago, limetech said:

    I don't see this happening on my test servers, but I don't have any SAS devices.

     

    All my disks are SATA. Do you have a HBA controller to test against?

     

    1 hour ago, limetech said:

    That message is output as a result of seeing I/O on partition 1 of the device

     

    Of the twelve disks in the array, there are 8 disks connected to my Adaptec HBA controller, while the remaining 4 disks are connected to SATA ports on the motherboard.

     

    Perhaps the HBA is doing something (differently) which appears as I/O activity. Reading the disk status using hdparm shows them all as 'standby'.

     

    Link to comment
    Share on other sites

    Performed the same as SimonF for reference.

     

    root@cobblednas:~# date && cat /sys/block/sdf/sdf1/stat

    Thu Apr  8 13:05:17 CDT 2021

         352     6073    57504     1456      549     6102    53208     7431        0     6344     8887        0        0        0        0        0        0

    root@cobblednas:~# date && cat /sys/block/sdf/sdf1/stat

    Thu Apr  8 13:05:34 CDT 2021

         352     6073    57504     1456      549     6102    53208     7431        0     6344     8887        0        0        0        0        0        0

    root@cobblednas:~# date && cat /sys/block/sdf/sdf1/stat

    Thu Apr  8 13:06:38 CDT 2021

         352     6073    57504     1456      549     6102    53208     7431        0     6344     8887        0        0        0        0        0        0

     

     

    Apr  8 13:05:20 cobblednas emhttpd: spinning down /dev/sdf

    Apr  8 13:05:20 cobblednas SAS Assist v0.85: Spinning down device /dev/sdf

    Apr  8 13:05:26 cobblednas emhttpd: read SMART /dev/sdf

    Link to comment
    Share on other sites
    1 hour ago, limetech said:

    Specifically it looks at

    /sys/block/sdb/sdb1/stat

     

    For the s3_sleep plugin I use these stats too, but history has proven that these are not always reliable (perhaps hardware/controller dependent), hence s3_sleep supports the hdparm method too, to check the disk state instead (method to use is selectable)

    Link to comment
    Share on other sites
    2 minutes ago, bonienl said:

    hdparm

    Would need to be sdparm for SAS

    Link to comment
    Share on other sites
    1 minute ago, SimonF said:

    Would need to be sdparm for SAS

    I know :) 

    • Like 1
    Link to comment
    Share on other sites

    I am seeing this also. Using 3 SATA 8tb with an H310 controller. I turned all VM's and containers off to see if that was causing it. I spin them down and in a few seconds they all come right back up. 

    Link to comment
    Share on other sites

    When the drives spin back up, are you guys seeing a correlating "read SMART" message in syslog?

    Link to comment
    Share on other sites
    SimonF

    Posted (edited)

    Apr  8 20:09:27 Tower emhttpd: Spinning up all drives...
    Apr  8 20:09:27 Tower emhttpd: spinning up /dev/sdj
    Apr  8 20:09:27 Tower emhttpd: spinning up /dev/sdk
    Apr  8 20:09:27 Tower emhttpd: spinning up /dev/sdh
    Apr  8 20:09:27 Tower emhttpd: spinning up /dev/sdg
    Apr  8 20:09:27 Tower emhttpd: spinning up /dev/sdc
    Apr  8 20:09:27 Tower emhttpd: spinning up /dev/sdi
    Apr  8 20:09:43 Tower emhttpd: read SMART /dev/sdj
    Apr  8 20:09:43 Tower emhttpd: read SMART /dev/sdk
    Apr  8 20:09:43 Tower emhttpd: read SMART /dev/sdh
    Apr  8 20:09:43 Tower emhttpd: read SMART /dev/sdg
    Apr  8 20:09:43 Tower emhttpd: read SMART /dev/sdd
    Apr  8 20:09:43 Tower emhttpd: read SMART /dev/sde
    Apr  8 20:09:43 Tower emhttpd: read SMART /dev/sdf
    Apr  8 20:09:43 Tower emhttpd: read SMART /dev/sdc
    Apr  8 20:09:43 Tower emhttpd: read SMART /dev/sdi
    Apr  8 20:09:43 Tower emhttpd: read SMART /dev/sda
    Apr  8 20:09:43 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdj, args "-A /dev/sdj"
    Apr  8 20:09:43 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdg, args "-A /dev/sdg"
    Apr  8 20:09:43 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdd, args "-A /dev/sdd"
    Apr  8 20:09:43 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdc, args "-A /dev/sdc"
    Apr  8 20:09:43 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sde, args "-A /dev/sde"
    Apr  8 20:09:43 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdh, args "-A /dev/sdh"
    Apr  8 20:09:43 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdk, args "-A /dev/sdk"
    Apr  8 20:09:43 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdi, args "-A /dev/sdi"
    Apr  8 20:09:43 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sda, args "-A /dev/sda"
    Apr  8 20:09:43 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdf, args "-A /dev/sdf"

    @doronThis is if I do a manual spin up.

    Edited by SimonF
    Link to comment
    Share on other sites
    4 minutes ago, SimonF said:

    @doronThis is if I do a manual spin up.

    Thanks. That's expected. Are you getting a similar "read SMART" messages when the drives spin back up spontaneously (the issue at hand)?

     

    Link to comment
    Share on other sites
    SimonF

    Posted (edited)

    12 minutes ago, doron said:

    Thanks. That's expected. Are you getting a similar "read SMART" messages when the drives spin back up spontaneously (the issue at hand)?

     

    Yes if i access a specific disk.

     

    Apr 8 20:22:33 Tower emhttpd: read SMART /dev/sdi
    Apr 8 20:22:33 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdi, args "-A /dev/sdi"

     

    sdi is a pool drive, but is also a SAS disk.

     

    Also I am seeing calls to all array drives every 30mins as part of the polling, I thought this should only be for devices not spun down.

     

     

    Apr  8 19:49:16 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdb, args "-A /dev/sdb"
    Apr  8 19:54:24 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdd, args "-A /dev/sdd"
    Apr  8 19:54:24 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdf, args "-A /dev/sdf"
    Apr  8 19:54:24 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sde, args "-A /dev/sde"
    Apr  8 19:54:24 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sda, args "-A /dev/sda"
    Apr  8 19:54:24 Tower SAS Assist v0.85: debug: smartctl wrapper caller is smartctl_type, grandpa is emhttpd, device /dev/sdb, args "-A /dev/sdb"

     

    Only sdf is active.

     

     

    Edited by SimonF
    Link to comment
    Share on other sites
    3 hours ago, limetech said:

    Specifically it looks at

    I made monitoring script to read the stats values of all my array disks. I am using /proc/diskstats which shows all disks in a single view with the following columns (using columns: 3, 4, 6, 8, 10).

     1  major number
     2  minor mumber
     3  device name
     4  reads completed successfully
     5  reads merged
     6  sectors read
     7  time spent reading (ms)
     8  writes completed
     9  writes merged
    10  sectors written
    11  time spent writing (ms)
    12  I/Os currently in progress
    13  time spent doing I/Os (ms)
    14  weighted time spent doing I/Os (ms)

     

    The output of the monitoring script looks like this (12 disks in my array)

    device  sectors read    sectors written reads completed writes completed
    sdb     0               0               4828            31
    sdc     0               0               4683            31
    sdd     0               0               4408            31
    sde     0               0               4584            231
    sdg     0               0               4440            231
    sdk     0               0               5314            4
    sdl     0               0               4264            31
    sdh     0               0               79174           5
    sdj     0               0               79242           32
    sdm     0               0               4223            31
    sdo     0               0               79004           4

     

    The counters for sectors read and sectors written stay zero all the time.

    While the disks are spun down the reads completed counter for all disks goes up

     

    After the "poll attributes" timer expires (5 minutes in my case), the SMART read operation kicks in, at this point the sectors read and sector written counters are still zero (no activity), while the reads completed counter stops incrementing for the disks which are now spun up due to the SMART read operation.

     

    This monitoring shows that the counters for sectors read and sectors written correctly tell "no activity" for disks spun down. No sure what is different with the emhttpd logic.

     

    And last, for the die-hards, this is the script I use

    #!/bin/bash
    declare -a reads writes
    clear
    while :; do
      # select disks in array
      stats=($(awk '/(sd[bcdeghjklmoq]) /{print $3,$6,$10,$4,$8}' /proc/diskstats))
      c=0; s=${#stats[@]}
      # clear screen and show header
      echo -e "\x1b[Hdevice\tsectors read\tsectors written\treads completed\twrites completed"
      # show counter difference for each disk on each iteration (every second)
      for ((i=0;i<s;i+=5)); do
        reads[c]=$((stats[i+1]-reads[c]))
        writes[c]=$((stats[i+2]-writes[c]))
        echo -e "\x1b[0K${stats[i]}\t$((reads[c]))\t\t$((writes[c]))\t\t${stats[i+3]}\t\t${stats[i+4]}"
        reads[c]=${stats[i+1]}
        writes[c]=${stats[i+2]}
        ((c++))
      done
      sleep 1
    done

     

    Link to comment
    Share on other sites
    3 hours ago, bonienl said:

    /proc/diskstats

     

    Try using

    /sys/block/sdX/stat  # for the entire device

    /sys/block/sdX/sdX1/stat  # for partition 1 on the device

     

    The smartctl command definitely increments both read command count and sectors read count for the entire device.

     

    I cannot reproduce this issue, which is to say, works for me.

    Link to comment
    Share on other sites
    6 hours ago, bclinton said:

    I am seeing this also. Using 3 SATA 8tb with an H310 controller. I turned all VM's and containers off to see if that was causing it. I spin them down and in a few seconds they all come right back up. 

     

    I reverted back to 6.91 and all drives spin down correctly.

    Link to comment
    Share on other sites

    Downgraded to unRAID 6.9.1 and all drives immediately spun down after they wouldn't spin down all day under 6.9.2.

     

    Apr  8 20:43:00 cobblednas emhttpd: spinning down /dev/sdi

    Apr  8 20:43:00 cobblednas SAS Assist v0.85: Spinning down device /dev/sdi

    Apr  8 20:43:03 cobblednas emhttpd: spinning down /dev/sdf

    Apr  8 20:43:03 cobblednas SAS Assist v0.85: Spinning down device /dev/sdf

    Apr  8 20:43:03 cobblednas emhttpd: spinning down /dev/sdd

    Apr  8 20:43:03 cobblednas SAS Assist v0.85: Spinning down device /dev/sdd

    Apr  8 20:43:03 cobblednas emhttpd: spinning down /dev/sdg

    Apr  8 20:43:03 cobblednas SAS Assist v0.85: Spinning down device /dev/sdg

    Apr  8 20:43:03 cobblednas emhttpd: spinning down /dev/sdc

    Apr  8 20:43:04 cobblednas SAS Assist v0.85: Spinning down device /dev/sdc

    Apr  8 20:43:04 cobblednas emhttpd: spinning down /dev/sdh

    Apr  8 20:43:04 cobblednas SAS Assist v0.85: Spinning down device /dev/sdh

    Apr  8 20:43:04 cobblednas emhttpd: spinning down /dev/sde

    Apr  8 20:43:04 cobblednas SAS Assist v0.85: Spinning down device /dev/sde

    Link to comment
    Share on other sites
    8 hours ago, limetech said:

     

    Try using

    /sys/block/sdX/stat  # for the entire device

    /sys/block/sdX/sdX1/stat  # for partition 1 on the device

     

    The smartctl command definitely increments both read command count and sectors read count for the entire device.

     

    I cannot reproduce this issue, which is to say, works for me.

     

    I have rewritten the script to read /sys/block/sdX/sdX1/stat columns 3, 7, 1, 5

    These are sectors read, sectors written, read I/Os, write I/Os.

    Output looks like

    image.png.7fa91f89d7210adde1b4fdd3a7211ea8.png

     

    I do not see activity in sectors read/written, yet "read SMART" kicks in causing the disks to spin up

     

    Apr 9 10:20:34 vesta emhttpd: spinning down /dev/sdj
    Apr 9 10:20:34 vesta emhttpd: spinning down /dev/sdh
    Apr 9 10:20:34 vesta emhttpd: spinning down /dev/sde
    Apr 9 10:20:34 vesta emhttpd: spinning down /dev/sdb
    Apr 9 10:20:34 vesta emhttpd: spinning down /dev/sdc
    Apr 9 10:20:34 vesta emhttpd: spinning down /dev/sdq
    Apr 9 10:20:34 vesta emhttpd: spinning down /dev/sdo
    Apr 9 10:20:34 vesta emhttpd: spinning down /dev/sdl
    Apr 9 10:26:57 vesta emhttpd: read SMART /dev/sdb
    Apr 9 10:26:57 vesta emhttpd: read SMART /dev/sdc
    Apr 9 10:27:06 vesta emhttpd: read SMART /dev/sde
    Apr 9 10:27:16 vesta emhttpd: read SMART /dev/sdh
    Apr 9 10:27:36 vesta emhttpd: read SMART /dev/sdj
    Apr 9 10:27:36 vesta emhttpd: read SMART /dev/sdl
    Apr 9 10:27:46 vesta emhttpd: read SMART /dev/sdo
    Apr 9 10:27:56 vesta emhttpd: read SMART /dev/sdq

     

    This is the script

    #!/bin/bash
    declare -a reads writes
    clear
    dev=(sdh sdj sde sdc sdb sdm sdp sdf sdi sdq sdo sdl)
    while :; do
      echo -e "\x1b[Hdevice\tsectors read\tsectors written\tread I/Os\twrite I/Os"
      for ((i=0;i<12;i++)); do
        stats=($(awk '{print $3,$7,$1,$5}' /sys/block/${dev[i]}/${dev[i]}1/stat))
        reads[i]=$((stats[0]-reads[i]))
        writes[i]=$((stats[1]-writes[i]))
        echo -e "\x1b[0K${dev[i]}\t${reads[i]}\t\t${writes[i]}\t\t${stats[2]}\t\t${stats[3]}"
        reads[i]=${stats[0]}
        writes[i]=${stats[1]}
      done
      sleep 1
    done

     

    Link to comment
    Share on other sites



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.