January 18, 201313 yr A short summary of what happened: About a week ago my server just stopped responding. Could not even telnet to it. I had to hard-reset the machine and then after startup I had a disabled disk (disk 2 out of my 5 disks on 5.0-rc10 pro array). There was nothing being done on the array at the time when the machine stopped responding (or at most some files may have been copied FROM the array not not written to it). I checked the drive's SMART report and nothing was wrong. I also checked the other disks and there were no SMART issues as checked against the troubleshooting section in the wiki. To be safe I went and bought a new motherboard, RAM and CPU and replaced it. So the only hardware that is not new is the power supply and the actual disks. I believe that there is nothing wrong with any of the disks as shown by the SMART reports, and that something else went wrong that caused the system to hang and the possible error/corruption of the now disabled disk2. But I also understand that the disk was disabled because of a write error and I will not trust the disk and the aim is to rebuild the failed disk from parity. But first I want to copy the contents of that failed disk to a backup just to be extra sure. That's why I replaced the server hardware to be sure before I started trying to fix the situation. I started the "new" machine and started array and started copying files from the disabled disk2 to external drive on my PC. I understand that my array is unprotected and the the failed disk's content is actually constructed from the other disks and parity, but I just wanted the data backed up from that disk before I attempted any rebuild of the disk just in case (yes I know I should have faith in unRAID to do that, but call me over-careful if you want). Now when I copy from the array it would go for a few GB and then just stop responding. Sometimes I can still telnet into unRAID and then I would stop the array from the command line and reboot, other times the whole machine doesn't respond, not even on the console keyboard and I can't ping the server and thus have to reset the machine. I keep on restarting and the same thing keeps on happening. Sometimes when it hangs and I can still access the webUI, it would show some disks as missing. Then I would restart, and I'd be able to access the array for a while copying files off, and then it would hang again. My zipped syslog is attached. It is from after the last restart and trying to copy files from the array and it stopped responding again. This time at least the webUI was still up and I saw again more than one disk showing as missing. I then stopped the array from the command line and copied the syslog. Here is also a link to an uploaded copy of the syslog if you don't feel like download/unzipping: https://www.dropbox.com/s/pm61p1ly7hgjr7v/syslog1.txt Any ideas please? This is driving me mad and I am worried big time. I have now been struggling with this for more about a week and have searched and read the forums to the extent that I feel that I have memorized the entire forum/wiki... Any help will be much appreciated thanks. PS: I have never attempted writing anything to the array since the initial problem as I understand the risks of this. So I'm trying to fix the array and do extra backups before I will start using it again for writing. Thanks Johan syslog1.zip
January 18, 201313 yr You have 6 onboard SATA ports (working fine), and you have a 2 port addon SATA controller, and I believe IT is the problem. You started the array (you have autostart disabled by the way, but you probably know that), and 2 minutes later both of the drives attached to the 2 port SATA controller stopped responding, at exactly the same time. That means something happened to the SATA controller, NOT the attached drives. The system tried to recover them, but no response at all, so quickly disabled them. Once drives are disabled, you can completely ignore all of the subsequent drive errors. This usually means that there are NO problems at all with these drives, so no need to rebuild them once the system is up again. Is it possible that the previous disabled disk was connected to this controller? If so, I would trust that drive, and save yourself a lot of rebuilding hassle. A lost write is not the only reason a drive may be disabled, as in your present case. This may also mean your previous motherboard and RAM are also fine. I wish there were errors present that directly implicate the controller, but there aren't. But I would replace it as soon as possible any way. It looks like it just quit! You can't use a critical component like this if you can't trust it.
January 19, 201313 yr Author Thanks RobJ. My previous motherboard had 5 on-board SATA ports and I added a plug-in SATA card for the sixth drive. It was the drive attached to that plug-in card's SATA port that had the initial failure and that was one of my reasons for suspecting the hardware and replacing it. The new motherboard where the syslog comes from actually has 7 on-board SATA ports (+ 1 external) and no plug-in SATA card anymore. 6 ports are together, one is a bit off on its own and one is at the back. So maybe the mobo makes the 2 separate ports look like an add-on controller, but it is in fact on the mobo and this is the brand new mobo. But your comments and insight into the syslog makes me think suspect even more that there is a hardware problem (not the disks) and since the PSU is the only old component still and I have read that they can do funny things, I'm going to replace that as well now. Thanks for your comments so far. Regards Johan
January 19, 201313 yr Some motherboards come with 2 different SATA controllers; usually 1 intel one and one from another manufacturer. What PSU do you have? The 2 drives that drop out at the same time, do they happen to be connected with the same molex connector in daisy chain or from a common cable back to the power supply? I suspect that with you having a marked drive as failed and parity correction comes into play, your system can't handle the power load from all drives being active at the same time.
January 19, 201313 yr Author I replaced the PSU with a brand new one just now (500W). So I have a new MB, CPU, RAM, PSU and SATA cables. I started the array again with the failed drive disabled and started the copy off the failed disk (by using the parity feature of course). It copied for about 20 minutes around 30-50GB, and then the same thing: the file system stopped responding. For a while the webUI was also unresponsive, but that has come back up now. It shows the array as started, but I cannot access the share. Then I tried stopping the array from the webui, but get the following: Stop SMB...Spinning up all drives...Sync filesystems...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)...Retry unmounting user share(s)... And now the webUI has also stopped responding again. Attached is the syslog just before I tried to stop the array from the webUI. And yes I know the array autostart is disabled thanks RobJ. That's why I start emhttp manually after each system startup. Autostart disabled is the reason I have to start emhttp manually correct? I'm no syslog expert, but it seems like ata7 and ata8 is failing during the copy and that is why the array stops working right? How do I figure out which disks are on ata7 and ata8? How is this possible with a new PSU now also? Even if those 2 disks were on the same power cable previously and that caused the problem, then this is now a new PSU. Regards Johan syslog2.txt
January 19, 201313 yr Author I suspect that with you having a marked drive as failed and parity correction comes into play, your system can't handle the power load from all drives being active at the same time. But for the past 2 years those same drives have been working in my old hardware without any power issues? And now I have a new PSU. Surely 6 SATA drives and the motherboard is not too much power demand for a 500W PSU? And I don't have anything else in the machine. Just mobo and 6 SATA drives. Thanks for the comment.
January 19, 201313 yr Which PSU are you using? How many 12V rails are there? How much amperage can each 12V rail supply? Does each drive have its own lead from the PSU? Are your 6 drives green (low power) or fast?
January 19, 201313 yr Author The PSU is a HuntKey Green Power: http://www.huntkeydiy.com/en/product/p-82-433.html It's specs state: 3.3V 24A, 5V 17A, 12V1 16A, 12V2 18A The drives are Seagate Barracudas and they are 5V 0.72A, 12V 0.52A spec. The PSU has 2 power rails. One supplies 4 drives and the other supplies 2 drives and 2 small fans each less than 0.2A. I don't think that's a problem? I restarted again after the last error and managed to copy around 120GB before the array stopped responding again. This time the error in the syslog looked different to me. Latest syslog: https://www.dropbox.com/s/2nek9hxbw5ua2xe/syslog3.txt I then stopped the array from the webUI and then started it again. But then it all froze up again and I had to stop from telnet and restart. Could this be a software issue since all my hardware apart from the drives have been replaced now? And even if the one drive's failure was a real failure (which it is not as the SMART report shows), then surely the changes for another drive to fail now is very remote. The next time I start the machine the drives and not disabled either, except of course for the one drive in the initial failure. So that means the other drives are fine. This one is a real puzzle...
January 19, 201313 yr The PSU is a HuntKey Green Power: http://www.huntkeydiy.com/en/product/p-82-433.html It's specs state: 3.3V 24A, 5V 17A, 12V1 16A, 12V2 18A The drives are Seagate Barracudas and they are 5V 0.72A, 12V 0.52A spec. The PSU has 2 power rails. One supplies 4 drives and the other supplies 2 drives and 2 small fans each less than 0.2A. I don't think that's a problem? I restarted again after the last error and managed to copy around 120GB before the array stopped responding again. This time the error in the syslog looked different to me. Latest syslog: https://www.dropbox.com/s/2nek9hxbw5ua2xe/syslog3.txt I then stopped the array from the webUI and then started it again. But then it all froze up again and I had to stop from telnet and restart. Could this be a software issue since all my hardware apart from the drives have been replaced now? And even if the one drive's failure was a real failure (which it is not as the SMART report shows), then surely the changes for another drive to fail now is very remote. The next time I start the machine the drives and not disabled either, except of course for the one drive in the initial failure. So that means the other drives are fine. This one is a real puzzle... Read this post: http://lime-technology.com/forum/index.php?topic=12219.msg130458#msg130458 Note how the two rails are really split! In the eyes of the PS designer, he is more concerned about the current required for that super fast video card than about supplying enough current for hard drives. Dual rail PS are intended for the gaming community and general use. Single rail supplies are what the server community should be looking at. The HD spec's are usually running conditions and not the startup current draw. Read the first post in the thread above for startup current draw. Remember that there are times when unRAID starts all of the drives up at the same time!
January 19, 201313 yr That PSU does not seem to be designed for a server with multiple drives. Read here.
January 19, 201313 yr Author Thanks a lot guys for the responses @jonathanm and @frank1940. I've read the posts and wiki you referred to and it makes sense. I'm going to look into getting a local PSU with the necessary properties. I live in South Africa and it should be available from the bigger suppliers here. The theory makes sense and I want to be safe within the limits of the PSU as described to be sure. It still worries me though that this machine has worked for more than a year on a similar specced PSU and now suddenly all the funnies. Maybe a perfect storm? Please keep any other ideas coming while I try and source an adequate PSU. Regards Johan
January 19, 201313 yr You didn't say if you had physically checked the power feeds to your hard drives. If you haven't, now might be a good time to do it. If you are using power feed splitters, did you actually check each wire by pulling on it to see if it pulls out of the connector. If you aren't using splitters, pull on the wires going to the connectors. If you are using Molex-to-SATA converters, check those for bad crimps. As I think you realize, don't pull so hard as to attempt to break the connection, but hard enough that you will pull the wire out if the crimp is defective.
January 19, 201313 yr Author I had done some basic cable checks and inspection, but will do more thorough ones as suggested thanks.
January 19, 201313 yr Motherboard devices usually have PCI addresses prefixed by 0000:00, with onboard disk controllers typically at 0000:00:1f. Your onboard SATA controller is at 0000:00:1f.2, very common. Addon devices such as other disk controllers are generally at 0000:01, 0000:02, and up. Your other SATA controller is at 0000:05:00.0. Because it's probably a separate soldered chip on the motherboard, it's an addon that can't be 'un-added'. It is almost certainly supporting those 2 extra SATA ports. The 2 attached drives are ST2000DL003-9VT166_5YD3RZWW (sdf, Disk 3) and ST2000DL003-9VT166_5YD3JZ2N (sdg, Parity). In syslog2, the extra disk controller quit again, exactly the same way. The only difference is that it waited 20 minutes after array start. An additional problem I forgot to mention in the first report, there is evidence of file corruption in the file system on Disk 2, you need to run reiserfsck on it. See Check Disk File systems. (I'm slightly suspicious of this evidence, since it's reported during all the errors dealing with the 2 disabled drives. May be a false-positive. Still need to check it out though.) Syslog3 only covers about an hour, and so far, the extra disk controller has not quit. However, syslog3 does confirm definite corruption on Disk 2, so make sure you run reiserfsck on it. Others are guiding you as to power, which is often a prime suspect in problems like this. I would add another suspect, heat - a possibly overheating critical component. So that makes the suspects: the power supply, the power cables to the 2 drives (especially if they share a cable), an overheating component, and that extra disk controller (with its 2 ports).
January 20, 201313 yr Author Thanks again @RobJ. I think I have everything back up and running again correctly. I'll provide a short summary below in case in can help anyone in future. I have 3 last questions though: 1) After the last restart (I'll summarize below what was done to fix hardware issue) I managed to get the rebuild running and even after 2+ hours and 600GB out of 2TB it was still going. Something that told me the progress was a lot farther than before and things were now working. However after about 2 or so hours emhttp stopped working again. No webUI response. I could hear the disks churning so I left it overnight and this morning stopped and restarted from command line and the disk rebuild was done and array is all green again. But why did emhttp stop responding? Attached is that syslog after the rebuild and just before I restarted from command line. I can't see anything in the syslog on that? 2) @RobJ: I figured out which disks were attached to the failed SATA controller by seeing in the webUI which disks were missing after the failure. But I could not figure out from the syslog how to match the ataX number back to a device sdX number. Please how did you get that? 3) Should I still run reiserfs check as per wiki on all disks and let it fix whatever it wants to (suggested parameters that reiserfsck provides)? Is this safe and will it not damage my data even if reiserfsck suggests --rebuild-sb? I have read the wiki on this, but just want to makes sure. Here is a summary of what happened and how it was resolved in case it can help anyone else in future: 1) unRAID array stopped responding on old hardware. Reset server. 2) UR came up, but one disk was disabled. All disks SMART report was OK. I checked and the disabled disk was attached to a plug-in SATA card. I assumed the card to be faulty and didn't wan't to take any chances, so I bought a new mobo with enough on-board SATA ports. 3) UR started on new hardware and I decided to copy contents from failed disk first before starting with disk rebuild - using unprotected array with disabled disk. This copy process kept on failing with failed SATA controller as per my posts above. 4) Many replies suggested power issues and I eventually identified the failed controller through the disks connected to it. I took one of the disks connected to this controller and connected it to an open port on another controller on the mobo as the mobo has 7 internal SATA ports and I have only 6 disks. This seemed to have solved the problem and I'm not getting any controller failures anymore (touchwood). 5) I believe all the posts suggesting that my PSU is maybe borderline as far as it's capabilities goes, are correct and I am going to look at getting a proper single rail PSU as per the wiki this coming week. The moving of a disk to another SATA port may have just "balanced" some of the power distribution to the extent that the PSU can now cope with the load. What think you? Lastly, thanks a lot to the forum for the great help. This is one of the main factors that makes unRAID the product of choice in this space. Great support from community, simple and stable product. Regards Johan syslog5.txt
January 20, 201313 yr 3) Should I still run reiserfs check as per wiki on all disks and let it fix whatever it wants to (suggested parameters that reiserfsck provides)? Is this safe and will it not damage my data even if reiserfsck suggests --rebuild-sb? I have read the wiki on this, but just want to makes sure. I'm sorry, but I have no time just now, won't be able to respond until possibly (only slight) late tonight, more likely tomorrow night. You do want to run reiserfsck on at least Disk 2, with the --check option, then more if it tells you to. See the Check Disk File systems wiki page for full instructions. Make sure you start the array in Maintenance mode.
January 20, 201313 yr 5) I believe all the posts suggesting that my PSU is maybe borderline as far as it's capabilities goes, are correct and I am going to look at getting a proper single rail PSU as per the wiki this coming week. The moving of a disk to another SATA port may have just "balanced" some of the power distribution to the extent that the PSU can now cope with the load. What think you? Changing physical PSU connection will not use the additional rail. One rail powers all SATA and Molex connectors regardless of physical cable. The other rail is reserved for the PCIe card connectors. Replace the PSU ASAP.
January 22, 201313 yr 4) ... I eventually identified the failed controller through the disks connected to it. I took one of the disks connected to this controller and connected it to an open port on another controller on the mobo as the mobo has 7 internal SATA ports and I have only 6 disks. This seemed to have solved the problem and I'm not getting any controller failures anymore. To clarify, your syslog indicates you reconnected both drives (serial#'s ending in ZWW and Z2N) to the onboard 6 ports, but reconnected one of the drives (3C9) previously connected to an onboard port, is now connected to a port on the other controller, possibly to that one SATA port off by itself? So there is still one drive connected to the problematic controller. The syslog indicates that the last of the 6 good ports is still open/available. The syslog also indicates that there was a single exception, exactly like the previous exceptions that proved fatal to both drives, on that one drive connected to the other controller. It occurred about 4 hours into the rebuild, but this time after a hard reset, the connection was restored, and there were no further issues, at least to the end of this syslog. With 2 drives attached, that hard reset did absolutely nothing. I'm not sure what this means. It could mean that if needed, you can use one port (and one port only) on this controller, but I really don't trust it. You might check for a BIOS update for this motherboard, perhaps it will contain fixes for this extra disk controller. Or you might consider returning the board, exchanging it for another that is similar, if possible. I have 3 last questions though: 1) After the last restart (I'll summarize below what was done to fix hardware issue) I managed to get the rebuild running and even after 2+ hours and 600GB out of 2TB it was still going. Something that told me the progress was a lot farther than before and things were now working. However after about 2 or so hours emhttp stopped working again. No webUI response. I could hear the disks churning so I left it overnight and this morning stopped and restarted from command line and the disk rebuild was done and array is all green again. But why did emhttp stop responding? Attached is that syslog after the rebuild and just before I restarted from command line. I can't see anything in the syslog on that? A chronology (from syslog5.txt): (Tom, please check this out!) Jan 19 18:25:13 Tower kernel: md: recovery thread rebuilding disk2 ... (disk2 rebuild begins, shortly after boot of server) ... Jan 19 20:39:03 Tower in.telnetd[1968]: connect from 10.0.0.10 (10.0.0.10) (user Telnet's in) Jan 19 20:39:09 Tower login[1969]: ROOT LOGIN on '/dev/pts/0' from '10.0.0.10' ... Jan 19 20:40:54 Tower emhttp: unRAID System Management Utility version 5.0-rc10 (#1 emhttp appears to restart) Jan 19 20:40:54 Tower emhttp: Copyright © 2005-2012, Lime Technology, LLC (these 6 lines are identical for each restart below) Jan 19 20:40:54 Tower emhttp: Plus key detected, GUID: 054C-02A5-0001-A080825C4476 Jan 19 20:40:54 Tower emhttp: get_config_idx: fopen /boot/config/flash.cfg: No such file or directory - assigning defaults Jan 19 20:40:54 Tower emhttp: diskFsStatus.1 not found Jan 19 20:40:54 Tower kernel: emhttp[1997]: segfault at 0 ip b74f9760 sp bfee7bc0 error 4 in libc-2.11.1.so[b7480000+15c000] Jan 19 20:41:48 Tower emhttp: unRAID System Management Utility version 5.0-rc10 (#2 emhttp appears to restart) ... Jan 19 20:41:48 Tower kernel: emhttp[2006]: segfault at 0 ip b7476760 sp bfdf9e40 error 4 in libc-2.11.1.so[b73fd000+15c000] Jan 19 20:42:46 Tower emhttp: unRAID System Management Utility version 5.0-rc10 (#3 emhttp appears to restart) ... Jan 19 20:42:47 Tower kernel: emhttp[2015]: segfault at 0 ip b7549760 sp bf949b00 error 4 in libc-2.11.1.so[b74d0000+15c000] Jan 19 20:44:58 Tower emhttp: unRAID System Management Utility version 5.0-rc10 (#4 emhttp appears to restart) ... Jan 19 20:44:58 Tower kernel: emhttp[2023]: segfault at 0 ip b748b760 sp bfcf1240 error 4 in libc-2.11.1.so[b7412000+15c000] Jan 19 20:49:53 Tower emhttp: unRAID System Management Utility version 5.0-rc10 (#5 emhttp appears to restart) ... Jan 19 20:49:54 Tower kernel: emhttp[2045]: segfault at 0 ip b747e760 sp bf8d55c0 error 4 in libc-2.11.1.so[b7405000+15c000] Jan 19 22:07:07 Tower kernel: ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen (this is related to previous hardware issue, Jan 19 22:07:07 Tower kernel: ata7.00: failed command: READ DMA EXT not to emhttp segfault issue) Jan 19 22:07:07 Tower kernel: ata7.00: cmd 25/00:00:bf:0a:ce/00:04:8a:00:00/e0 tag 0 dma 524288 in Jan 19 22:07:07 Tower kernel: res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Jan 19 22:07:07 Tower kernel: ata7.00: status: { DRDY } Jan 19 22:07:07 Tower kernel: ata7: hard resetting link Jan 19 22:07:08 Tower kernel: ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jan 19 22:07:08 Tower kernel: ata7.00: configured for UDMA/133 Jan 19 22:07:08 Tower kernel: ata7: EH complete Jan 20 01:07:53 Tower kernel: md: sync done. time=24161sec Jan 20 01:07:53 Tower kernel: md: recovery thread sync completion status: 0 User Johan (jdelport) reports that WebUI is non-responsive about 2 hours or so after rebuild begins. The emhttp restarts appear shortly after Telnet in at about 2 hours and 15 minutes after rebuild begins, so correspond very well with WebUI loss. Johan, it looks like the emhttp restarts correspond with commands you tried within the Telnet session. If you could list every command you did, in particular the command(s) that correspond with the 5 restart times above, that might help Tom. I'm changing the Message icon to Bug report, and requesting you to remove the word RESOLVED from the thread title. The original hardware issue is essentially figured out, resolved, but now there is a new issue, a possible RC10 bug with the emhttp restarts and associated segfaults in libc. 2) @RobJ: I figured out which disks were attached to the failed SATA controller by seeing in the webUI which disks were missing after the failure. But I could not figure out from the syslog how to match the ataX number back to a device sdX number. Please how did you get that? I wish this were easier, but it's not! Long ago, I helped Brian with a feature in the MyMain page of UnMenu, that would identify all of the symbols (eg. scsi0, sd 0:0:0:0, scsi 0:0:0:0, ata1, ata1.0, sda) associated with a drive, and locate every related syslog line that contained them. But even then, it was not perfect, maybe 99.8% correct. And the exceptions to the rules have gotten much worse since then, with simultaneous assignment of symbols by differing driver modules, that completely mix up the order of them. It's partly trial and error now, partly simple logic, and sometimes working backwards, from the very last symbol assignments. Somewhere I wrote something up about this, and I think it would be partially helpful, but don't know where it is. When I have more time, I'll look for it. 3) Should I still run reiserfs check as per wiki on all disks and let it fix whatever it wants to (suggested parameters that reiserfsck provides)? Is this safe and will it not damage my data even if reiserfsck suggests --rebuild-sb? I have read the wiki on this, but just want to makes sure. This last syslog showed no file system corruption, so either you have already run this, or it's not as much a problem any more (still there, but lurking quietly). You probably should still run one more reiserfsck check, if you haven't yet.
Archived
This topic is now archived and is closed to further replies.