July 9, 201313 yr Apparently, smartctl is unnecessary waking up some Hitachi disks. Test this: spin it down manually, and see if it will wake up if you do a `hdparm -C` on it. Huh, I have all Hitachi drives (less then 1 seagate 250GB drive) `hdparm -C` never wakes any of them. Only Hitachi base variable with hdparm is "-H Read temperature from drive (Hitachi only)" which is sweet. Right. You didn't read what you quoted. Ouch SORRY!!!
July 9, 201313 yr Jul 9 11:47:17 Tower spind: Version 3.4 <c> 2013 by Pourko Balkanski (For personal use only, not for redistribution!) Jul 9 11:47:17 Tower spind: Idle timeouts set for 5 minutes. Jul 9 11:47:17 Tower spind: Monitoring: sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp sdq sdr sds Jul 9 11:53:18 Tower spind: spinning down /dev/sdc Jul 9 11:53:18 Tower spind: spinning down /dev/sdd Jul 9 11:53:19 Tower spind: spinning down /dev/sdf Jul 9 11:53:20 Tower spind: spinning down /dev/sdo Jul 9 11:53:20 Tower spind: spinning down /dev/sdr Jul 9 11:53:21 Tower spind: spinning down /dev/sds No flash, nor VMDK mount being monitored, very good (I can tell from the count) How do you imagine spinning down flash? If a disk can be spun down, we watch it. If a disk can't be spun down, we don't bother with it. Don't imagine, just checking that it didn't is all.
July 9, 201313 yr Author I have just found this however, which seems to suggest another means of determining the spinstate, without using hdparm -C. Quite how/if this could be leveraged in a script or unRAID I'm not sure. I'll leave that to the far more knowledgable people in the room ... http://forums.freenas.org/threads/how-to-find-out-if-a-drive-is-spinning-down-properly.2068/ That uses the tool camcontrol, which appears to be strictly a FreeBSD tool, for the FreeBSD CAM subsystem. I'm no expert here though, but it does not look in any way adaptable for us. Aaah! That would explain why I couldn't find a compiled version anywhere to download Looking at this from another angle... would it be possible to inspect the response of the hdparm -C command? As if it does return with: SG_IO: bad/missing sense data, sb[]: 70 00 00 00 00 00 00 18 00 00 00 00 00 00 00 00 00 00 09 0c 00 00 00 ff 00 00 00 00 00 00 40 50 You could possibly take the 24th (?) hexadecimal value, and determine the spin status from that instead? I only suggest this, as from what I understand - the SG_IO response is only returned for drives that also don't accurately report their spin status. So presumably this wouldn't influence anyone except people with this particular issue?
July 9, 201313 yr http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2009226 Virtual compatibility mode The real hardware characteristics are hidden. http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2003813 Lastly: http://communities.vmware.com/docs/DOC-10799
July 9, 201313 yr Author @jack0w: Can you please post the output of the newer hdparm: hdparm -I /dev/sdb There it is, hope this helps! root@Atlas:/boot/custom# hdparm -I /dev/sdb /dev/sdb: ATA device, with non-removable media Model Number: WDC WD30EFRX-68AX9N0 Serial Number: WD-WMC1T****109 Firmware Revision: 80.00A80 Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0 Standards: Supported: 9 8 7 6 5 Likely used: 9 Configuration: Logical max current cylinders 16383 16383 heads 16 16 sectors/track 63 63 -- CHS current addressable sectors: 16514064 LBA user addressable sectors: 268435455 LBA48 user addressable sectors: 5860533168 Logical Sector size: 512 bytes Physical Sector size: 4096 bytes Logical Sector-0 offset: 0 bytes device size with M = 1024*1024: 2861588 MBytes device size with M = 1000*1000: 3000592 MBytes (3000 GB) cache/buffer size = unknown Capabilities: LBA, IORDY(can be disabled) Queue depth: 32 Standby timer values: spec'd by Standard, with device specific minimum R/W multiple sector transfer: Max = 16 Current = 0 DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 Cycle time: min=120ns recommended=120ns PIO: pio0 pio1 pio2 pio3 pio4 Cycle time: no flow control=120ns IORDY flow control=120ns Commands/features: Enabled Supported: * SMART feature set Security Mode feature set * Power Management feature set * Write cache * Look-ahead * Host Protected Area feature set * WRITE_BUFFER command * READ_BUFFER command * NOP cmd * DOWNLOAD_MICROCODE Power-Up In Standby feature set * SET_FEATURES required to spinup after power up SET_MAX security extension * 48-bit Address feature set * Device Configuration Overlay feature set * Mandatory FLUSH_CACHE * FLUSH_CACHE_EXT * SMART error logging * SMART self-test Media Card Pass-Through * General Purpose Logging feature set * 64-bit World wide name * URG for READ_STREAM[_DMA]_EXT * URG for WRITE_STREAM[_DMA]_EXT * WRITE_UNCORRECTABLE_EXT command * {READ,WRITE}_DMA_EXT_GPL commands * Segmented DOWNLOAD_MICROCODE * Gen1 signaling speed (1.5Gb/s) * Gen2 signaling speed (3.0Gb/s) * Gen3 signaling speed (6.0Gb/s) * Native Command Queueing (NCQ) * Host-initiated interface power management * Phy event counters * NCQ priority information * unknown 76[15] DMA Setup Auto-Activate optimization Device-initiated interface power management * Software settings preservation * SMART Command Transport (SCT) feature set * SCT Write Same (AC2) * SCT Error Recovery Control (AC3) * SCT Features Control (AC4) * SCT Data Tables (AC5) unknown 206[7] unknown 206[12] (vendor specific) unknown 206[13] (vendor specific) unknown 206[14] (vendor specific) Security: Master password revision code = 65534 supported not enabled not locked not frozen not expired: security count supported: enhanced erase 428min for SECURITY ERASE UNIT. 428min for ENHANCED SECURITY ERASE UNIT. Logical Unit WWN Device Identifier: 50014ee602ccfc25 NAA : 5 IEEE OUI : 0014ee Unique ID : 602ccfc25 Checksum: correct
July 9, 201313 yr Author http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2009226 Virtual compatibility mode The real hardware characteristics are hidden. http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2003813 Lastly: http://communities.vmware.com/docs/DOC-10799 Thanks for the links @madburg, I'll have a read and see if anything stands out as being relevant to this issue - although I'll admit I'm hardly an expert
July 9, 201313 yr Author There it is, hope this helps! Just to confirm, that's the newer hdparm, right? 0 root@Tower:~# hdparm -V hdparm v9.43 It certainly is! Is that good or bad news? root@Atlas:/boot/custom# hdparm -V hdparm v9.43
July 9, 201313 yr I have just found this however, which seems to suggest another means of determining the spinstate, without using hdparm -C. Quite how/if this could be leveraged in a script or unRAID I'm not sure. I'll leave that to the far more knowledgable people in the room ... http://forums.freenas.org/threads/how-to-find-out-if-a-drive-is-spinning-down-properly.2068/ That uses the tool camcontrol, which appears to be strictly a FreeBSD tool, for the FreeBSD CAM subsystem. I'm no expert here though, but it does not look in any way adaptable for us. Aaah! That would explain why I couldn't find a compiled version anywhere to download Looking at this from another angle... would it be possible to inspect the response of the hdparm -C command? As if it does return with: SG_IO: bad/missing sense data, sb[]: 70 00 00 00 00 00 00 18 00 00 00 00 00 00 00 00 00 00 09 0c 00 00 00 ff 00 00 00 00 00 00 40 50 You could possibly take the 24th (?) hexadecimal value, and determine the spin status from that instead? I only suggest this, as from what I understand - the SG_IO response is only returned for drives that also don't accurately report their spin status. So presumably this wouldn't influence anyone except people with this particular issue? The VMWare RDM virtual layer gets back all the data that is required back from the SAS controller, but does not from your SATA controller, its spitting back (raw) what it got back which is incomplete. This may work on another MB sata controller, not an option for you. As WeeboTech mentioned adding another sas card would, at least it is an option... You could have another look in your MB bios settings and see if there are any options for SATA that could be tried as well.
July 9, 201313 yr I'm really sorry to post this in here and I know this will come accross as rude, but can someone sumarize the problem here for me as I do not have time to read this whole thread. I will tomorrow. Is it going to delay 5 release?
July 9, 201313 yr @madburg: When I click on your links, I get: An error occurred while processing your request. Reference #97.245efea5.1373393147.456ad2c0 You could have said a few words what those links are. Really? all the links?
July 9, 201313 yr http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2009226 Virtual compatibility mode The real hardware characteristics are hidden. http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2003813 Lastly: http://communities.vmware.com/docs/DOC-10799 Thanks for the links @madburg, I'll have a read and see if anything stands out as being relevant to this issue - although I'll admit I'm hardly an expert Links give explanation to options of RDM/DirectPath and what limitations they each have, and ruffly how they work. Its not an easy read if you don't work with this product.
July 9, 201312 yr Emhttp could fix this: Instead of a straight... smartctl -n sleep,standby -a $dev ...emhttp should do something like... hdparm -C $dev | grep -q active && smartctl -a $dev Or, better yet, leave all that crap to external scripts. To this point, I have been testing for a while now with the unRAID spindown disabled and xnars script running (set to 5 mins). The behavior I had where every 30mins (how I had unRAID set "Default spin down delay: 30" ) I had a hard pause lasting 2-5 seconds of any video playing (via AFP and Plex clients) is now gone. Being xnars script is parsing every five minutes I should have noticed it even more frequently, but not a single hiccup. I will test more this evening.
July 9, 201312 yr It loops once a minute actually. That's different from your 5-minute idle-timers. But never mind. I see ur point and understand better now. My understanding is smartctrl is causing the freeze, where your comment to change emhttp condition code would use hdparm first. Or i misunderstood the comment. Fact is the freezes are gone after disabling unRAIDs Default spin down delay and utilizing ur script. I new it was the use of unRAIDs Default spin down delay causing my issue, because disabling it made the behaviour stop, I never had any other means to put something else in place that would check and spin down the drives and see if it still occurred. Until now.
July 9, 201312 yr Author SG_IO: bad/missing sense data, sb[]: 70 00 00 00 00 00 00 18 00 00 00 00 00 00 00 00 00 00 09 0c 00 00 00 ff 00 00 00 00 00 00 40 50 You could possibly take the 24th (?) hexadecimal value, and determine the spin status from that instead? I only suggest this, as from what I understand - the SG_IO response is only returned for drives that also don't accurately report their spin status. So presumably this wouldn't influence anyone except people with this particular issue? I went to the hdparm man page to look if there's any hdparm option that would return the raw "sense data" of any disk, but there wasn't any. That would've been something worth investigating further. LOL! Got it! hdparm --verbose -I /dev/sdb 2>&1 | grep 'SG_IO: sb' Not sure if it's reliable though: 0 root@Tower:~# 0 root@Tower:~# hdparm -C /dev/sdb /dev/sdb: drive state is: active/idle 0 root@Tower:~# hdparm --verbose -I /dev/sdb 2>&1 | grep 'SG_IO: sb' SG_IO: sb[]: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0 root@Tower:~# This is what I get when I run that command: root@Atlas:/boot/custom# hdparm --verbose -I /dev/sdb 2>&1 | grep 'SG_IO: sb' SG_IO: sb[]: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Doesn't seem to have the same string I got from running 'hdparm -C'. Is it possible to analyse the output of the 'hdparm -C' command, only IF the sense data is present? Am thinking, if that could be incorporated into your script - it could issue the spindown command if required based on the analysis of the sense data. Would that be possible? As that would certainly be a work around for my issue, and for anyone else wanting to leverage unRAID under ESXi with RDM'd drives
July 9, 201312 yr Author I've not found rock solid documentation on the "sense data", the only thing I've managed to do on my end is repeat the tests on all drives connected to the onboard controller. I have had consistent results there, but then that is to be expected. I can only think that it could be proved if someone else with different hardware attempted to recreate the problem, and then from looking at their output of the 'hdparm -C' command, we could determine if the 24th byte does always indicates the spin status or not. My feeling though is that there must be something to it if the freenas guys have been able to leverage the sense data to determine the spin status in their own product without the need for relying on hdparm. To be honest, I'd have a go at editing your script for myself and anyone else hoping to use ESXi with RDM drives, since I have a rough idea of the end result I'm after, I just lack the experience with Linux to know which commands to use and be able to modify the script to analyse that string and act on it accordingly. Something that would probably take weeks of trial and error for me, I'm sure someone more knowledgeable could do very quickly!
July 9, 201312 yr Author I've not found rock solid documentation on the "sense data" If someone would know what that "sense data" means, that would be the hdparm people. And when they dump that data it means: Here, we've tried everything, and we can't make any sense of it, so here's the raw output. If we can prove a pattern in that raw output by analysing outputs provided by anyone else who is prepared to test this, ie the 24th byte indicates spin status - would that be good enough for us to act on? The only other thing I've come across is this https://code.google.com/p/spindown/ this doesn't seem to be reliant on hdparm or smartctl - possibly a more reliable way of determining disk activity? From what I've read it seems to work by checking how many blocks are read/written to/from drives - though I'm not sure if there's a downside to this method that I'm not seeing? Any thoughts pros/cons to this method?
July 9, 201312 yr I don't have time to read through 5 pages of wtf. Please provide a summary of the issue, bearing in mind: a) in all my testing spindown works just fine. b) I don't support any 3rd party tinkering with spindown and/or setting the drive spindown timers. c) the unraid-driver issues spin-ups in order to implement spinup-groups, emhttp uses smartctl to spin drives down d) if you are running in a VM try it on the native h/w before posting an issue about spindown
July 9, 201312 yr Author I don't have time to read through 5 pages of wtf. Please provide a summary of the issue Many apologies Tom, I think this thread has ended up in the wrong part of the forum. The problem is that hdparm doesn't report the correct spin status for drives in certain conditions, resulting in drives not being spundown, because the unRAID believes they already are. That being said, this has always been ESXi related problem, I've not had these issues on native hardware. Weebo mentioned there was a possible change to behaviour running unRAID under ESX, I was merely testing this out and reporting my findings, the thread was split and I've only now spotted it was moved to the 'unRAID OS 5.x Issue List' forum. Perhaps it would be more appropriate if it was moved to the customizations forum? I've been following this thread sparsely. I figured if an updated smartmontools helped madburg, maybe it will help with unRAID under ESX. Previously, unRAID did not detect the spin status or temperature under ESX with RDM/rdmp drives. I updated smartmontools with pourko's version and I was able to see temps and the drives were not flashing any more. I can't seem to determine if the drives spin up and down. Thought i would add it here in case others want to test with it.
July 9, 201312 yr Author If we can prove a pattern in that raw output by analysing outputs provided by anyone else who is prepared to test this, ie the 24th byte indicates spin status - would that be good enough for us to act on? What ypu're trying to do here is convince me to write a customized program for one particular person. If I myself were to use ESXi, then you'd probably convince me very easily. The only other thing I've come across is this https://code.google.com/p/spindown/ this doesn't seem to be reliant on hdparm or smartctl - possibly a more reliable way of determining disk activity? More reliable than hdparm? In Linux, hdparm is the Golden standard. We can't pretend to be any smarter. See, your problem actually is with ESXi. From what I've read it seems to work by checking how many blocks are read/written to/from drives - though I'm not sure if there's a downside to this method that I'm not seeing? Any thoughts pros/cons to this method? How is that a "method"? You read/writte blocks to SSDs too, right? So what? Will that method tell you if your SSD is spinning? Damn! You spotted my cunning plan! I was looking at it in terms of one method being to use the output of hdparm to determine spin status, the other to use this 'spindown' script I linked to. Personally I'm not using any SDD's at the moment, but I take your point! Really you're absolutely right - it is an issue with ESXi. I've just been trying to find a way to work around the problem, if I can. It would certainly avoid upgrading hardware to get passthrough working under ESXi, or buying another SAS card!!
July 9, 201312 yr If you can confirm that you do not have these issues outside of ESX, then I would like to move this elsewhere for continued discussion. I moved it here to keep this thread on topic and out of the other announcement forum in case there was a problem showing up. Seriously, it can be allot of work wading through pages and pages of intermixed conversations and consolidating them.
July 9, 201312 yr Author If you can confirm that you do not have these issues outside of ESX, then I would like to move this elsewhere for continued discussion. I moved it here to keep this thread on topic and out of the other announcement forum in case there was a problem showing up. Seriously, it can be allot of work wading through pages and pages of intermixed conversations and consolidating them. @Weebo, I can confirm I definitely don't have these issues outside of ESX and I am very happy for you to move this topic to where ever you feel is most appropriate. I hope my previous post didn't come across as if I were pointing the finger - it was definitely not meant like that as I have a great deal of respect for your work you do as an admin on these forums, and in helping other users. I just simply hadn't noticed where the topic ended up (which I should have done!), and I personally felt bad for having wasted Tom's time. Please accept my apologies for the inconvenience caused by this misunderstanding.
July 9, 201312 yr No problem. At least Tom got a look and posted what he wanted to see from us. I thought it may have been a real hardware interface issue somehow. My goal was to keep this discussion moving and not let it get intermingled with the other discussions in the announcement thread.
July 9, 201312 yr I don't have time to read through 5 pages of wtf. Please provide a summary of the issue Many apologies Tom, I think this thread has ended up in the wrong part of the forum. The problem is that hdparm doesn't report the correct spin status for drives in certain conditions, resulting in drives not being spundown, because the unRAID believes they already are. That being said, this has always been ESXi related problem, I've not had these issues on native hardware. Weebo mentioned there was a possible change to behaviour running unRAID under ESX, I was merely testing this out and reporting my findings, the thread was split and I've only now spotted it was moved to the 'unRAID OS 5.x Issue List' forum. Perhaps it would be more appropriate if it was moved to the customizations forum? I've been following this thread sparsely. I figured if an updated smartmontools helped madburg, maybe it will help with unRAID under ESX. Previously, unRAID did not detect the spin status or temperature under ESX with RDM/rdmp drives. I updated smartmontools with pourko's version and I was able to see temps and the drives were not flashing any more. I can't seem to determine if the drives spin up and down. Thought i would add it here in case others want to test with it. Ok, I'll probably move it, but here is how unraid manages spinup/spindown/reading temps. There are two cases to consider: 1. What happens when you load a webGui page, and 2. What happens in the background during normal operation. Case 1 When you load the webGui page we want to go get the disk temperatures so that we can display them. For this, emhttp, uses smartctl like this: /usr/sbin/smartctl -n standby -s on -A /dev/sdX where "sdX" is the linux device identifier. The "-n standby" tells smartctl to exit and not access any smart attributes if it thinks the device is in standby mode, ie, spun down. This is because with most drives, reading the smart attributes causes the drive to spin up. If the temperate can't be read then webGui displays an "*" for the temperature. In terms of which attribute to use, emhttp looks first for "Airflow_Temperature_Cel" and if found, uses that setting. If not found then it looks for "Temperature_Celsius". If neither found then it displays "*" for temperature. This is the only use of "smartctl" in stock unraid. Case 2 When the array is Started there is a background thread that wakes up every 5 seconds to see if it's time to spin down a drive. This test is accomplished purely by watching I/O counters. If there has been no I/O on a drive for "spindown delay" amount of time then the drive is spun-down, and a flag is set saying "this device is spun down". If any I/O subsequently happens on that drive, then the flag is cleared and we record the timestamp of the I/O. The code to spindown a drive varies: - for cache drive, the 'hdparm -y /dev/sdX" command is used - for array drives, the unraid driver sends "ATA_OP_STANDBYNOW1" to the drive (which is supported by all linux disk drivers that we use). The code to spinup a drive also varies: - for cache drive, either I/O causes it to spin up, or an explicit 'hdparm -S0 /dev/sdX' command is used - for array drives, either I/O causes it to spin up, or the unraid driver sends "ATA_OP_SETIDLE1" to the drive (also supported by all linux disk drivers that we use). It does this to spun-down disks of the same spin-up group as the disk with I/O occurring. Anyway there are few other special checks: - if formatting, we don't check temperatures at all because this is so i/o intensive it hangs the webGui and people complain. - if array not started we don't check spin-down. This is by design. - checking temps, spinning up/down never done with disabled disks, again by design (because could cause flood of messages to syslog and severe slow down due to error recovery because, well, the drive is bad). At one time, the background polling thread also read the temperatures of spun-up disks, with the idea of implementing an "over temp" watchdog feature. Well SMART commands are very disruptive of the normal I/O flow. Turns out, the disk drivers completely "synchronize" by stopping all new commands, wait for current commands to complete, send the SMART command down to the drive, wait for it to complete, then resume normal I/O. The SMART command itself often drives the read/write heads either to the innermost track or outermost track, introducing possibly long seek times. All this combines to sometimes cause "studdering" of media streams, so I took this check out. The idea is that when I get around to implementing this feature, we would poll temps far less often, say every 10 min instead of every 5 sec.
July 9, 201312 yr Author When the array is Started there is a background thread that wakes up every 5 seconds to see if it's time to spin down a drive. This test is accomplished purely by watching I/O counters. If there has been no I/O on a drive for "spindown delay" amount of time then the drive is spun-down, and a flag is set saying "this device is spun down". If any I/O subsequently happens on that drive, then the flag is cleared and we record the timestamp of the I/O. Since hdparm isn't being used to determine if a disk needs to be spundown, it seems this issue is purely a cosmetic one - the graphic indicating spin status isn't accurate for some RDM drives under ESXi. Thanks for explaining all that though Tom, at least your spindown magic it is still working away in the background! On the bright side, myself and others now understand the current situation in regarding spin status and temperatures of RDM drives under ESXi!
Archived
This topic is now archived and is closed to further replies.