• [unRAID 6.10.0-rc1] - emhttpd: read SMART events keep spinning up array disks


    danioj
    • Urgent

    I am raising this bug report to formally capture discussions on the 6.10.0-rc1 release thread relating to a possible bug.

     

    In summary, I have found that emhttpd keeps issuing read SMART events consistently to my array disks, which is spinning them up. The effect is that my usually spun down Array is pretty much spun up 24x7.

     

    Here are the relevant discussions in that thread.

     

    There was some suggestion that the CATurboWrite Plugin or my Marvel Controller (which only has 3 disks of 16 connected to it) could be at fault but I have since removed the plugin and have confirmed that this issue appears to also be experienced by another user @Mathervius who doesn't have a Marvel controller.

     

    Diagnostics are attached.

     

    Chosen minor but really feels a bit more than that. Having the array spun up all the time isn't desirable at all.

     

     

    unraid-diagnostics-20210902-1852.zip




    User Feedback

    Recommended Comments



    Try without it to be sure, there still might be an issue without it activated.

     

    In any case, it's always good to go to the bottom of the issue. It might help someone else some day. :) 

    Link to comment
    8 hours ago, danioj said:

    *Penny Drops*

     

    Woot! Pretty sure you found the cause of the spin ups :) 

     

    Nice @ChatNoir, I like how you even provided a link to the wiki for all the details :) 

    Link to comment

    I thought I would report back that I was too hasty in reporting that the SMART issue had gone away.

     

    Even with all Disk related Plugins removed and as per my actions above I noticed that I was still getting the array SMART data read at random times that was having the effect of spinning up my disks.

     

    I downgraded to the latest Stable 6.9.2 and have been running for over a week and not a single unexpected SMART data read or array spin up to speak of.

     

    I don't have the time to debug so I am sticking with the Stable.

     

    I noticed a set of posts close to this yesterday - I will like this thread in that thread.

    Link to comment
    15 hours ago, danioj said:

    I thought I would report back that I was too hasty in reporting that the SMART issue had gone away.

     

    Even with all Disk related Plugins removed and as per my actions above I noticed that I was still getting the array SMART data read at random times that was having the effect of spinning up my disks.

     

    I downgraded to the latest Stable 6.9.2 and have been running for over a week and not a single unexpected SMART data read or array spin up to speak of.

     

    I don't have the time to debug so I am sticking with the Stable.

     

    I noticed a set of posts close to this yesterday - I will like this thread in that thread.

    6.9.2 does not help, i also checked, the icon shows the disk is spin down, but it actually is still spining. it is a bug does not show the disk status correctly.

    Link to comment
    3 hours ago, moguiyu said:

    6.9.2 does not help, i also checked, the icon shows the disk is spin down, but it actually is still spining. it is a bug does not show the disk status correctly.

    What do you have set as the Poll interval?    The GUI only updates the disk status at that frequency.

    Link to comment

    I’ve had this issue since beta 35 of the last release.  I’m on the latest stable release now and I still see my drives spinning up for SMART and then down again after the 15 min set time.  Over and over again. 

    Link to comment

    The same issue here on 6.9.2, did not have this issue before, removed the turbo write plugin, did a manual spin down, 30 seconds later it spins back up; 

     

    Oct 6 10:31:57 BlackBoX root: Stopping CA Turbo Mode
    Oct 6 10:31:57 BlackBoX root: Terminating 31523
    Oct 6 10:32:05 BlackBoX flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup.php update
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdm
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdj
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdk
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdh
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdg
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdd
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdt
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sde
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdb
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdr
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdf
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdc
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sds
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdn
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdq
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdo
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdl
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdi
    Oct 6 10:32:11 BlackBoX emhttpd: spinning down /dev/sdp
    Oct 6 10:32:14 BlackBoX emhttpd: read SMART /dev/sdj
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdm
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdh
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdg
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdd
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdt
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sde
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdb
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdr
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdf
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdc
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sds
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdn
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdq
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdo
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdl
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdi
    Oct 6 10:32:46 BlackBoX emhttpd: read SMART /dev/sdp
    Oct 6 10:32:47 BlackBoX emhttpd: read SMART /dev/sdk

     

    Link to comment

    Same for me. Drives spinning down and after a while...Spins up again without any file access. Only with read Smart entry in disk log. No Turbo Write or any plugin activated

    Link to comment

    Ok, might've found something, in settings -> disk settings, I set 'Tunable (md_write_method)' to AUTO instead of 'reconstruct write', disks hasn't spun back up as of yet, will monitor it for a few days and report back. 

    Link to comment
    11 hours ago, Abnorm said:

    Ok, might've found something, in settings -> disk settings, I set 'Tunable (md_write_method)' to AUTO instead of 'reconstruct write', disks hasn't spun back up as of yet, will monitor it for a few days and report back. 

     

    with reconstruct write set then a write to any drive will spin up all drives.

    Link to comment
    On 10/7/2021 at 2:09 PM, wiesel2482 said:

    Same for me. Drives spinning down and after a while...Spins up again without any file access. Only with read Smart entry in disk log. No Turbo Write or any plugin activated

    I have set to Auto since beginning, it does not help. the disk still read SMART...

    Link to comment

    Strange, I cannot remember if I've changed it at any point, it might've been waay back before squids plugin showed up to utilize the performance benefits. But in my usecase it has actually made a difference as disks are spinning down and not spinning up again as expected now. 

    Link to comment

    I'm facing this issue on 6.9.2 too - since few days parity + one hdd is spinning up immedietly aferer spun down to read SMART - no access, no data written to hdd. Manual spin down and 30 sec later those two hdd are back again.

     

    Tried to figure out if it's not a docker container and no luck, it's so random and so annoying - investigated telegraf, scrutiny, plex, tdarr, frigate and even with all those disabled I still see drives spinning up for no reason. 

     

    I can confirm this bug exists on 6.10rc2 and it even got worse:

    Nov 14 16:24:21 SIGMA emhttpd: spinning down /dev/sdd
    Nov 14 16:24:27 SIGMA emhttpd: spinning down /dev/sdh
    Nov 14 16:24:27 SIGMA emhttpd: spinning down /dev/sdb
    Nov 14 16:24:48 SIGMA emhttpd: read SMART /dev/sdh
    Nov 14 16:24:48 SIGMA emhttpd: read SMART /dev/sdd
    Nov 14 16:24:48 SIGMA emhttpd: read SMART /dev/sdb
    Nov 14 16:25:38 SIGMA emhttpd: spinning down /dev/sdd
    Nov 14 16:25:49 SIGMA emhttpd: read SMART /dev/sdd
    Nov 14 16:32:57 SIGMA emhttpd: spinning down /dev/sdd
    Nov 14 16:33:06 SIGMA emhttpd: read SMART /dev/sdd

     

    Previously I had only two disks waking up all time, now another one has same behaviour but this drive is not part of array and it's unassigned, used as a backup once a day.

    Edited by cpu
    Link to comment
    11 hours ago, another_hoarder said:

    Any updates on this, OP or Staff?  Just noticed my log went to 100% because of the very same issue.  For me it happens every hour.

    By setting 'Tunable (md_write_method)' to AUTO still solved it for me, disks are spun down as expected. Might not work for your usecase of course. Double check dockers etc if any paths they work in are on the array rather than the cache.  

    Link to comment
    On 2/4/2022 at 10:28 AM, Abnorm said:

    By setting 'Tunable (md_write_method)' to AUTO still solved it for me, disks are spun down as expected. Might not work for your usecase of course. Double check dockers etc if any paths they work in are on the array rather than the cache.  

    unfortunately this does not work for me...

    Link to comment

    Sharing my experience as a user that was also facing this SMART read issue but found a solution, for what it's worth;

     

    The culprit was my "system" share was set to Cache-Prefer but my docker.img file was still sitting on my array, meaning any reads/writes were causing a spin up. I guess Unraid then took the opportunity to read SMART data as the drive was active which is why I saw it in the logs.

     

    Note, I do have mover scheduled but evidently it couldn't move the docker.img file because Docker was always enabled and the file in-use. I assume a file that is in-use is prevented from moving, I don't know that for sure. After disabling docker, running mover, seeing the 40GB move from my array to my cache drive, and re-enabling docker, the spin-ups and SMART reads have gone away.

     

    Also, the heavy hard drive activity I experienced when enabling or disabling docker no longer occurs, presumably because it's no longer hitting the docker.img file on my array. If anyone else sees that symptom, perhaps my solution with work for you.

    Edited by sww
    Link to comment
    8 minutes ago, sww said:

    Sharing my experience as a user that was also facing this SMART read issue but found a solution, for what it's worth;

     

    The culprit was my "system" share was set to Cache-Prefer but my docker.img file was still sitting on my array, meaning any reads/writes were causing a spin up. I guess Unraid then took the opportunity to read SMART data as the drive was active which is why I saw it in the logs.

     

    Note, I do have mover scheduled but evidently it couldn't move the docker.img file because Docker was always enabled and the file in-use. I assume a file that is in-use is prevented from moving, I don't know that for sure. After disabling docker, running mover, seeing the 40GB move from my array to my cache drive, and re-enabling docker, the spin-ups and SMART reads have gone away.

     

    Also, the heavy hard drive activity I experienced when enabling or disabling docker no longer occurs, presumably because it's no longer hitting the docker.img file on my array. If anyone else sees that symptom, perhaps my solution with work for you.

    Yes mover cannot move a file tbat is inuse. Likely you changed to cache prefer after image was created. Only stopping docker as you did would allow it to move.

    Edited by SimonF
    Link to comment

    Ah that makes sense.

    To think of all the electricity and hard drive lifespan I've wasted in the past 2 years of running Unraid :(

    Link to comment
    On 2/11/2022 at 3:25 PM, sww said:

    Sharing my experience as a user that was also facing this SMART read issue but found a solution, for what it's worth;

     

    The culprit was my "system" share was set to Cache-Prefer but my docker.img file was still sitting on my array, meaning any reads/writes were causing a spin up. I guess Unraid then took the opportunity to read SMART data as the drive was active which is why I saw it in the logs.

     

    Note, I do have mover scheduled but evidently it couldn't move the docker.img file because Docker was always enabled and the file in-use. I assume a file that is in-use is prevented from moving, I don't know that for sure. After disabling docker, running mover, seeing the 40GB move from my array to my cache drive, and re-enabling docker, the spin-ups and SMART reads have gone away.

     

    Also, the heavy hard drive activity I experienced when enabling or disabling docker no longer occurs, presumably because it's no longer hitting the docker.img file on my array. If anyone else sees that symptom, perhaps my solution with work for you.

    thanks for sharing this. after stop docker and start mover, twice, the OS still reads SMART to wake up the disks. very unfortunately. i wonder if it is because it is dell server only can get info from authorized disks, cause the embeded disks was fine, only the new disks keeps reading smart.

     

    • Haha 1
    Link to comment
    On 2/11/2022 at 3:25 PM, sww said:

    Sharing my experience as a user that was also facing this SMART read issue but found a solution, for what it's worth;

     

    The culprit was my "system" share was set to Cache-Prefer but my docker.img file was still sitting on my array, meaning any reads/writes were causing a spin up. I guess Unraid then took the opportunity to read SMART data as the drive was active which is why I saw it in the logs.

     

    Note, I do have mover scheduled but evidently it couldn't move the docker.img file because Docker was always enabled and the file in-use. I assume a file that is in-use is prevented from moving, I don't know that for sure. After disabling docker, running mover, seeing the 40GB move from my array to my cache drive, and re-enabling docker, the spin-ups and SMART reads have gone away.

     

    Also, the heavy hard drive activity I experienced when enabling or disabling docker no longer occurs, presumably because it's no longer hitting the docker.img file on my array. If anyone else sees that symptom, perhaps my solution with work for you.

    This was the solution for me! Thanks.

     

    Edit.. I was too soon with my answer. This wasn't it. For me the problem seems to be solved after accessing the disk in Unraid and clicking through the tabs:
    image.png.c16c8434a820ddf0141ca9ae1093f82f.png

    It seems that after doing this, the drives don't start up anymore. If this is really the solution, then this is definitely a bug in Unraid and should be patched.

     

    It is now 17:00 and the disk hasn't spun up yet, so this seems to be it...?

    image.png.5d84812786a3f268af8853d75f7419c5.png

     

    I also tried the following to solve it, but to no avail:

    • move docker .img to SSD
    • clean up shares
    • play around with Tunable (poll_attributes) values
    • remove Unassigned Devices plugin (and reinstall after concluding it didn't have any effect)
    • sign out of unraid server plugin
    • remove NERD Pack plugin
    • install scrutiny docker app
    • stop docker entirely
    • stop part of some docker apps (i.e. Plex)
    • stopping all cron jobs
    • disabling SMB on the shares that use the drives that are constantly spinning up

     

    Hope this helps some of you. Again: if this really is the case, then please devs, fix it!

     

    Edit 20-aug, the saga continues.... 

     

    Look like I finally find the culprit. My Unraid is running as a VM in ESXI. ESXI uses smart daemon every 30 minutes. 

    image.png.55ec52dd3ae2d1085fc7078edd7590e2.png

     

    After disabling the smart daemon using command:

    /etc/init.d/smartd stop

    the polling stopped! Problem solved!

    Edited by Luciano200x
    Link to comment
    On 7/20/2022 at 8:47 PM, Luciano200x said:

    This was the solution for me! Thanks.

    This solution unfortunately did not help us which i tried twice before: turn off both the docker and VM then triggered the mover, but the disks are still read smart.

    i also used all these 3 apps to track, there is no sign of disk use or file read.

    410076152_ScreenShot2022-07-31at11_23_27AM.png.4e42cf1a6e07238b7b7a1bb01a18f3cb.png

     

    but i did checked some spindown-read smart intervals and find out it usually take 5min'38 sec and average is around 350 second. which looks like some system level thing triggered this instead of application. any guys have ideas how to dig?

    76698632_ScreenShot2022-07-31at11_06_22AM.thumb.png.7b29b19084ba26253594c7f26cefae84.png

    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.