June 26, 200719 yr Howdy, my first time to request help (had 2 problems before, but stubborn or shy about asking for help). I was copying 195 gigs to Disk6, an empty but working and tested disk. Lightning started crackling closer and closer, so decided to shut down. Stopped copy, refreshed unRAID web page, all good, no errors. Stopped unRAID, all fine, clicked PowerDown, successful, shut down other computers, no apparent errors anywhere. After calm returned to skies, restarted computers. JacoBack, the unRAID server, looked perfect, on the web page and the console. Started the copy of last 35 gigs to Disk6, but it froze almost immediately, and errors appeared on console. Uh-oh... Refreshed web page, and it showed 2 or 3 errors on Disk6 AND over 29000 errors on Disk2. Uh-oh uh-oh uh-oh ... Stopped the array and ... (hope embedded image of unRAID web page displays here, I'm new at this stuff) After analyzing, Disk2 (md2, sdf) and Disk6 (md6, sde) appear to be on the same controller, my Syba 2-port SATA II PCIe X1 card. The first errors reported are, I believe, on the controller bus, during the write to Disk6. That seems to have taken down the controller and Disk6, AND Disk2 as well. I think Tom at Lime Technology will have to check out some of this, but was hoping for some quick advice as to my next steps, to minimize further loss. What I *think* might be the best plan is to shut it all down, bring it back up and make sure the Syba controller is working correctly. Then run reiserfsck on Disk2 and Disk6, fix whatever I can, and try to bring the array up again, with a parity sync/rebuild. I'd like to hear others comments. Losing 2 drives at once is a scary thing! What was being copied to Disk6 is safe elsewhere, but Disk2 is another matter, 460gigs of data, not mission-critical, but quite a loss to me. Tom: Note the copy of the serial number of Disk1 into the model number and serial number for Disk2, and the copy of the serial number of Disk6 into its model number. This appears in the image and at the end of the attached syslog. Also note what appears to be a data overrun in the output syslog stream in line 486, probably many missing messages. Minor note: In line 751 of this excerpted syslog, printk reports over 14700 suppressed messages. I removed over 32750 lines of redundant errors to cut the file size down, but still 80K.
June 26, 200719 yr Author I've decided to leave it up, in its current failed state, for a day or 2, in case Tom or anyone else has an idea as to troubleshooting it.
June 27, 200719 yr Author Well, don't know what to say, been checking every few hours for a couple of days hoping for some help. I probably phrased it wrong or posted in the wrong place. I'm rather new to forums, rather new to unRAID, and very new to Linux. I had hoped Tom might see my little tale of woe, and possibly have some debugging steps because of the drive model and serial number corruption showing above. Just my bad timing. I've run reiserfsck on the drives that are accessible, all good. I've captured the output info from the reiserfsck, hdparm -I, and top commands, in case they might be helpful, and I have a screen capture of the Devices tab of the web page. The syslog is attached above. Are there other Linux or unRAID commands that might reveal something useful? I may pull the flash drive out before powering down, to avoid any further corruption of config/super.dat, although it may be too late, since I had already stopped the array. Comparing it with an older copy, it appears to have reasonably valid, uncorrupted info, to my un-expert eyes. I'll wait a few more hours before powering down, and beginning recovery of my once-great unRAID system.
June 28, 200719 yr Hum, so You can't access the two disks ? What I would do in that case, since you parity is worthless anyway, is to unplug everything, switch the two "failed drive", boot a live cd with raiser fs (or a clean copy of unraid) and run some fsck, you might even be able to mount them read only to see if your data is there.
June 29, 200719 yr I would try to "restore" first, no data will be lost- I've= done this twice now in 3.0 and 4.0, and rebuild parity. Do not format any drives under any circumstances. If you want a really powerful analysis/low level fixer try Spinrite. If you think you have some data corruption you need to email Tom about it, I've also seen threads that talki about this kind of incident, see some of the threads I've posted I may even have had something comparable.
June 30, 200719 yr Author Thank you both for your helpful comments. I can say at last that my unRAID server is fully operational again, with absolutely no data loss! It was not without additional hiccups though. I appreciated your (orb) advice, which was basically to treat the unRAID system as suspect, and work with it using only third party tools at first. However, I didn't exactly follow it. I first compared the contents of the flash drive with a backup, and found it almost certainly uncorrupted, which meant unRAID (and its tools) should start cleanly, even if my hardware doesn't. The more I thought about the crash, the more I felt the addon SATA controller was the initial cause, then unRAID failed to correctly deal with the loss of the controller and both drives. It seemed to me that the drives should be intact, except for an incomplete file from the interrupted copy, because once the controller was down, the drives could not be accessed for good or bad. So, since I didn't think I could hurt it worse, I decided to see what would happen by starting unRAID. I was rather shocked, happily, when the console showed all 6 drives mounted. I checked the web page, and found Disks 1 through 5 and Parity were green. All drive info, model and serial numbers were correct. Disk6 was red, but it had a temperature, disk size and free space that looked correct. The data from all 6 disks was accessible. I stopped the array, and ran reiserfsck on Disk2, perfect, then Disk6 and as expected had to let 'reiserfsck --fix-fixable' do its thing. I started the array, and clicked Restore to start a parity sync, but almost immediately saw errors on the console (tail -f), and a minute later the parity sync quit. This time, it was the first 2 drives that failed, Disk1 and Parity, both connected to the motherboard, not to the controller card that started this whole mess. Another blow to my confidence in the system. Since the array was still up, I clicked the Stop button, then looked over to my unRAID machine, only to see BIOS messages appearing. It had rebooted itself. A third blow to my confidence... But when the system came up again, it was all green but the parity drive. I started it, parity sync finished in about 2 hours, without any errors, and everything seems to be fine. Except my confidence in it... Thankfully, I didn't have to use SpinRite. That's a great tool, and I've used it several times, but it sure takes a long time, and would have shut me down for another day or two. I did search your (melechmet) threads, and I don't think we can say if your missing and failed drives were caused by the same problem, but it would be interesting to see the very first error messages in your syslog, at the moment of drive failure, and compare them with mine.
Archived
This topic is now archived and is closed to further replies.