July 7, 200817 yr Before going to bed last night I pressed the Spin Down button. All drives dutifully spun down. This morning I was going to experiment with the smart script, and all the drives were spun down, so I hit the Spin Up button. Interesting behavior - first press brought up about 1/2 the drives, second press brought up 1 more, third-fifth presses did nothing. Continued pressed ultimately brough them all up - took 12+ presses before all the drives were spinning. This is NOT a big deal, just thought I would report and ask other users to test this out also so that Tom can get some feedback on this issue. If everyone is seeing this, the code for spin up may have gotten overoptimized.
July 7, 200817 yr Any pattern regarding order of spinup - based on assignment, physical position, drive brand/model, etc.? Bill
July 7, 200817 yr There was some recent discussion of this here: http://lime-technology.com/forum/index.php?topic=2164. And Tom recently commented on how he spins up the drives here: http://lime-technology.com/forum/index.php?topic=2182.msg16238#msg16238.
July 7, 200817 yr There was some recent discussion of this here: http://lime-technology.com/forum/index.php?topic=2164. And Tom recently commented on how he spins up the drives here: http://lime-technology.com/forum/index.php?topic=2182.msg16238#msg16238. Since Tom is basically just reading blocks of data from the disk drives, and then checking if they have spun up, then repeating as needed. It might be that the data is being supplied from cache, and his method of determining if the drive is subsequently spinning is flawed. Or perhaps he gives up after a dozen tries or so on a given disk. My own method of spinning up drives involves reading a random block off the disk. That random block is calculated to be somewhere within the physical limits of the specific disk. That way, the odds of the disk having to spin up is pretty good. Tom's description seemed to indicate he started at the first block on the disk and then advanced until he saw the disk spinning. Perhaps with a huge cache, and Tom possibly giving up after a few iterations, and always starting at the same point on the disks, it would act as Robj is indicating. My shell script logic is similar to this: # This is the disk to be spun up. disk=/dev/hdc # determine the number of blocks on the drive blocks=`df -k ${disk} | grep -v Filesystem | sed "s/\([^ ]*\) *\([^ ]*\) .*/\2/"` # calculate a (random) block number to be read somewhere # between block 1 and the max blocks on the drive skip_b=$(( 1+(`dd if=/dev/urandom count=1 2>/dev/null | cksum | cut -f1 -d" "`)%($blocks) )) # read the random block from the disk. We use a random # block to try to ensure it is not in the cache memory # if the block was in the cache memory, the disk would # not spin up. dd if=${disk} of=/dev/null count=1 bs=1k skip=$skip_b >/dev/null 2>&1 & Joe L.
July 7, 200817 yr I'd be interested to know on what unraid version you started seeing this on. I have a different issue with the spinup in 4.3.1 in that it crashes the server. Previously i was running 4.3b3 which also tried to spin up all drives at the same time and there was no problem with that version. Tom has said that the spinup code has not changed but there has been at linux kernel change that started in 4.3b4. Mark
July 7, 200817 yr Author I have had no problem with crashes. I press the "spin up" very infrequently, so cannot say when it started happening. I just know the last time it worked (maybe 4.2.3 or 4.3b3) and this time (4.3.2) it didn't. Caching is likely NOT the culprit, although can't say for sure. I rebooted the server yesterday and only accessed a couple of disks afterwards (except running smartctl). The last thing I did before spinning down was watch a movie - which likely uncached most everything. And if caching was the issue, following the same algorithm, wouldn't it likely have never spun up the disks? I was not careful about recording what disks came up and when. Not sure if there is a pattern. (e.g. disks on one controller). Will try again later. Anyone else seeing this (or not seeing this)?
July 7, 200817 yr In my case, it started when the Spin Up button was first added, and I have seen no change in behavior, that is, it hasn't gotten better or worse. Also have not had a crash. I have one drive in particular, as mentioned in the link above, that nearly always requires 2 Spin Up's, often more, rarely less. As a data point for comparison, it is a Seagate ST3500630AS, firmware 3.AAE, on a Syba SiI3132-based addon card. Several other drives may also at random not spin up either, but don't remember which.
July 8, 200817 yr Author Okay, I did a series of spinups ... here are the results ... Press # - Drives that spun up --------- -------------------------- 1 - Parity, 2, 3, 5, 8, 11, 12, 13 2 - 7 3 - 1 4 - 6 5 - none 6 - none 7 - none 8 - 9 9 - 4 10 - 10 (last one) I then did a spin down followed immediately by a spin up - all drives spun up. I then did another spin down followed immediate by a spin up - all drives spun up. Weird that the problem happens after a long delay, but not after an immediate action. So the drives that did not spin up the first time are ... 1, 4, 6, 7, 9, 10 ALL of these disks are connected to the MOTHERBOARD (ICH8)! This certainly seems significant. I recently changed the mode in BIOS for the MB SATA controller from IDE (I think) to ACHI. I'll bet if I set it back to IDE this problem would go away. This turns on native command queueing (NCQ). I wonder if that is related to the problem. I will try again tomorrow to confirm that the problem only occurs after a long delay between spin down and spin up. BTW, this is all on a P5B VM DO motherboard
July 8, 200817 yr I would be interesting to use a self made shell script calling smartctl -a on each of the 10 drives to see if it handled spin up better. In my spin up script I use hdparm -S242 /dev/sdX to spin up tje drive. This command waits until the drive actually responds with an acknowledgement. What I find interesting is statements where the code was optimized to read from the drive "and test" if it had spun up, yet some drives do not spin up anyway.
July 8, 200817 yr Author Observations on WeeboTech's Test I did as requested and ran the smartctl script. It did NOT spin up all the disks. The disks that did not spin up are: 2, 11, 12, 13 These are ALL the Western Digital hard drives in my system. 1x500G, 1x750G, 2x1T I manually ran the hdparm -S242 /dev/sdX on these four disks and they each spun up the first time. New Observation About Prior Spin Up Test: These drives did not spin up on first press of the Spin Up button ... 1, 4, 6, 7, 9, 10 These are MOST of the Seagate drives in my array (4x750G, 2x500G). (The only others are two Seagate 1T drives (parity and disk3) that did spin up.) Next Test For my next test tonight I am going to try to change the spindown timeout while the disks are spun down. I wonder if that will wake up all the drives (equivalent to running hdparm command?). Other suggestions?
July 8, 200817 yr Next test if you are willing... Instead of using smartctl try using hdparm in a script as in hdparm -S244 /dev/sdX where X = drive letter of device. Smartctl -a /dev/sdX should have spun up the drive. If the drive does not spin up do a hdparm -C /dev/sdX to see the idle status.. or better yet do echo "Processing /dev/sdX" hdparm -C /dev/sdX hdparm -S243 /dev/sdX hdparm -C /dev/sdX echo " " examples of output would be root@Atlas:~# hdparm -C /dev/sdc /dev/sdc: drive state is: standby root@Atlas:~# hdparm -C /dev/sdf /dev/sdf: drive state is: active/idle
July 9, 200817 yr Author That technique spun up all disks ... it printed out the following for each device: /dev/sda1: drive state is: standby /dev/sda1: setting standby to 243 (1 hours + 30 minutes) ~4 second pause /dev/sda1: drive state is: active/idle I wonder what would happen if I ran them all in parallel.
July 9, 200817 yr > I wonder what would happen if I ran them all in parallel. They should all spin up. Try and write a script to do it in a loop or simply do the hdparm /dev/sdX -S243 & for each drive. FWIW, What I do in my spin control script is the hdparm /dev/sdX -S243 in a loop. For each drive in the loop. I spawn a copy of the command but prefix it with a sleep argument incrementing 1 second per drive. Drive 1 is 0 seconds Drive 2 is 1 seconds Drive 3 is 2 Seconds Drive 4 is 3 Seconds. etc, etc... This way I don't kill the power supply ;-) Either create a shell script or try and locate my spin control script.
July 9, 200817 yr Author Since WeeboTech's "hdparm" technique worked so well, I decided to see what would happen if I went into the Web GUI and changed the "Disk Spin Down Delay". I assumed that behind the covers it was doing the same "hdparm" command that WeeboTech suggested I try. SUCCESS! It resulted in spinning up all the drives the first time! So if you need a reliable way to spin up all your disks, THAT is an easy way to do it without running scripts. It took more than a couple seconds, however, consistent with running the commands in series and not in parallel. For my array (15 disks) it was close to 45 seconds. I am wondering if RobJ and others that experience spin up issues have luck with this technique. If so, Tom can make a very easy change to how Spin Up operates that works for everyone.
July 9, 200817 yr I use individual spin down control, by putting a series of hdparm -S commands in a script, and calling it from the go script. So I ran one of them with all drives spun down, and they all came up first time. It was a bit slow, about 5 seconds per command, well over half a minute total. I then created a modified version to use WeeboTech's suggested simultaneous spin up, and that also worked perfectly, and much much faster, 6 or 7 seconds. #!/bin/bash # Set disk spindown to 15 minutes, simultaneously /usr/sbin/hdparm -S180 /dev/sda & /usr/sbin/hdparm -S180 /dev/sdg & /usr/sbin/hdparm -S180 /dev/sdd & /usr/sbin/hdparm -S180 /dev/sdb & ## /usr/sbin/hdparm -S180 /dev/sde & /usr/sbin/hdparm -S180 /dev/sdf & /usr/sbin/hdparm -S180 /dev/sdc & /usr/sbin/hdparm -S180 /dev/sdh & /usr/sbin/hdparm -S180 /dev/sdi & /usr/sbin/hdparm -C /dev/sda /usr/sbin/hdparm -C /dev/sdg /usr/sbin/hdparm -C /dev/sdd /usr/sbin/hdparm -C /dev/sdb /usr/sbin/hdparm -C /dev/sdf /usr/sbin/hdparm -C /dev/sdc /usr/sbin/hdparm -C /dev/sdh /usr/sbin/hdparm -C /dev/sdi Using the configured spin down time in a loop based on the above appears to be a more reliable method to attach to the Spin Up button, except for those of us who control the spin down of our drives individually. We will not use it, but that is a minor issue as we have our own scripts.
July 15, 200817 yr A version of the following was previously written up on the 10th, but a careless 'Dotable Dork' did not migrate it over. After the hdparm experimentation above, I later discovered in my syslog 2 series of 'exception Emask' errors for 6 of my 8 drives, of the frozen & (timeout) variety, with both soft and hard resets. Although it appears to have been harmless, with no adverse effects that I could discover, neither soft or hard resets are desirable. I have to assume they clear the channels, buffers, caches, etc, and therefore could potentially cause issues if there were any file operations in progress. Thankfully, if you are spinning the drives up or down, you probably have no file operations in progress yet. A sample of the messages for a single drive: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen ata1.00: cmd e3/00:f2:00:00:00/00:00:00:00:00/40 tag 0 res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) ata1.00: status: { DRDY } followed later by this: ata1: soft resetting link or this: ata7: hard resetting link followed later by this: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata1.00: configured for UDMA/133 ata1: EH complete sd 1:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB) sd 1:0:0:0: [sda] Write Protect is off sd 1:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA All of the above messages are repeated for multiple drives. With the new v4.3.3 apparently using this newer method of spinning up the drives, I had the same series of 'exception Emask' error sets in the syslog. I was only testing the new Spin Up button, and am not planning on using it, but am curious to see if others find these frozen exceptions in their syslogs. One theory I have is that the drive becomes briefly unresponsive during the spin up, and if there was a wait of a few seconds before querying it for status, that no errors would be reported (or resets performed). Perhaps like memory, some drives need a 'wait' after a spin up is started. Interestingly, although all of my drives immediately spun up, 2 of the drives ignored the new value of spindown, and retained my previously set value. This is a common problem when trying to set a spindown value for a drive, and has been noted before. I have tried a couple of methods, both disabling spindown before setting the value, and setting the value twice in a row, but still have inconsistent results. Setting the spindown value with the hdparm -S command does not always 'take', but it does seem to spin the drive up reliably.
July 15, 200817 yr I believe the best way to spin up all the disks is to simply start a parity sync. For many releases this is exactly what I used to spin up all the hard drives. It's truly amazing how problematic a 'spin up' button has become.
July 19, 200817 yr Author Starting parity check does an excellent job of spinning up the disks! The only negative (and this is a real nit), is that it updates the "last checked on" date, even if you only run it for a few seconds. Maybe the date should update on completion not on start?
Archived
This topic is now archived and is closed to further replies.