November 23, 201015 yr Hi. Having a medium level of Panic attack...! I was doing the swap-disable as documented here: http://lime-technology.com/forum/index.php?topic=8569.0 and when I left the system this morning, everything looked fine. When I got back, the server looked like the attachment, and I got a 2GB Syslog (NOT attached) that essentially contains information like this (the "sd..."-lines are much less frequent than the handle/md lines): Nov 23 08:47:42 Tower kernel: handle_stripe read error: 175740016/1, count: 1 Nov 23 08:47:42 Tower kernel: md: disk1 read error Nov 23 08:47:42 Tower kernel: handle_stripe read error: 175740024/1, count: 1 Nov 23 08:47:42 Tower kernel: sd 3:0:0:0: [sdc] Unhandled error code Nov 23 08:47:42 Tower kernel: sd 3:0:0:0: [sdc] Result: hostbyte=0x04 driverbyte=0x00 Nov 23 08:47:42 Tower kernel: sd 3:0:0:0: [sdc] CDB: cdb[0]=0x28: 28 00 0a 79 99 bf 00 04 00 00 Nov 23 08:47:42 Tower kernel: end_request: I/O error, dev sdc, sector 175741375 Nov 23 08:47:42 Tower kernel: md: disk1 read error Nov 23 08:47:42 Tower kernel: handle_stripe read error: 175740032/1, count: 1 Nov 23 08:47:42 Tower kernel: md: disk1 read error Nov 23 08:47:42 Tower kernel: handle_stripe read error: 175740040/1, count: 1 Nov 23 08:47:42 Tower kernel: md: disk1 read error It seems that the regenerated disk6 (accessing it from it's disk-share) has only a very small fraction of the original data on it. Also the disk1 appears empty, even if the folders are there. Initial spot checks indicate that data on the other disks appear intact. My original disk6 is obviously also still intact. I really need some advice on how to proceed. Please. :'( I could imagine, that one option would be to reinstall the disk6, and regenerate the parity, but then I'm still in a situation where I have lost the contents on disk1. I would like to get all my data back, including ways to verify if there are intact data on the disks - in particular disk1 and disk6. ANY suggestions are welcome...
November 24, 201015 yr Author Can I take a disk from the server, attach it to an unRAID test machine with only that disk attached (no parity), assign it as disk1 and start the test server to see what is in on the disk? Can I do that with all the disks from my live server to retrieve important data ?
November 24, 201015 yr Wait for someone with more experience with unRAID disaster recovery to give you a definitive response, but I would NOT hook the drive up and assign it as a data drive in another box. You could, however, mount it in another Linux box with the mount command. mkdir /mnt/mydrive mount -r -t reiserfs /dev/sdX /mnt/mydrive /dev/sdX is the device mapping (and would be /dev/hdX for an IDE device). To find the mapping you can use lls -l /dev/disk/by-id/ to get a list of drives and their mappings.
November 24, 201015 yr Hi. Having a medium level of Panic attack...! I was doing the swap-disable as documented here: http://lime-technology.com/forum/index.php?topic=8569.0 and when I left the system this morning, everything looked fine. When I got back, the server looked like the attachment, and I got a 2GB Syslog (NOT attached) that essentially contains information like this (the "sd..."-lines are much less frequent than the handle/md lines): Nov 23 08:47:42 Tower kernel: handle_stripe read error: 175740016/1, count: 1 Nov 23 08:47:42 Tower kernel: md: disk1 read error Nov 23 08:47:42 Tower kernel: handle_stripe read error: 175740024/1, count: 1 Nov 23 08:47:42 Tower kernel: sd 3:0:0:0: [sdc] Unhandled error code Nov 23 08:47:42 Tower kernel: sd 3:0:0:0: [sdc] Result: hostbyte=0x04 driverbyte=0x00 Nov 23 08:47:42 Tower kernel: sd 3:0:0:0: [sdc] CDB: cdb[0]=0x28: 28 00 0a 79 99 bf 00 04 00 00 Nov 23 08:47:42 Tower kernel: end_request: I/O error, dev sdc, sector 175741375 Nov 23 08:47:42 Tower kernel: md: disk1 read error Nov 23 08:47:42 Tower kernel: handle_stripe read error: 175740032/1, count: 1 Nov 23 08:47:42 Tower kernel: md: disk1 read error Nov 23 08:47:42 Tower kernel: handle_stripe read error: 175740040/1, count: 1 Nov 23 08:47:42 Tower kernel: md: disk1 read error It seems that the regenerated disk6 (accessing it from it's disk-share) has only a very small fraction of the original data on it. Also the disk1 appears empty, even if the folders are there. Initial spot checks indicate that data on the other disks appear intact. My original disk6 is obviously also still intact. I really need some advice on how to proceed. Please. :'( I could imagine, that one option would be to reinstall the disk6, and regenerate the parity, but then I'm still in a situation where I have lost the contents on disk1. I would like to get all my data back, including ways to verify if there are intact data on the disks - in particular disk1 and disk6. ANY suggestions are welcome... Is it possible you accidentally dislodged the cable to disk1? That would explain all the errors. Joe L.
November 24, 201015 yr Author Is it possible you accidentally dislodged the cable to disk1? That would explain all the errors. Joe L. Yes and no. I may have touched the disk1 cable when initally swapping the cables to the new disk. I did however verify all connections before, and also now after the incident, everything looked/felt fine and normal. The system was running and regenerating fine without any indication that drive1 was inaccessible. I was not even home (nor was anyone else) at the time the issue happened. It was at 89% of copying the parity when I left. I came home to this mess Given things are as they are, I'm still looking for how to proceed in order to have the highest likelyhood of saving as much as possible of my data.
November 24, 201015 yr I would normally say to capture a syslog. But in this case, it will be too huge if you attempted to capture it all.. You could try something like this to capture the initial 10,000 line portion of it (before all the repeating errors): sed 10000q /var/log/syslog >/boot/syslog_capture.txt You could then zip it, and it will compress very well. Then stop the array, power down, re-seat the connectors. Power up and see if the disk1 responds. Also, send a PM/email to limetech. Joe L.
November 24, 201015 yr Author I will capture syslog as soon as I get home from work and post here. I actually believe I have a local copy of the 2TB version on another Windows machine. So I should be able to capture some without having to start the server. Implicitly you are saying, that it will be OK to start the currently powered off server again - no (further) harm is expected to be done ?
November 24, 201015 yr Implicitly you are saying, that it will be OK to start the currently powered off server again - no (further) harm is expected to be done ? Yes... if a "parity check" starts, cancel it as soon as possible. I've got a feeling the array will not start itself anyway. Joe L.
November 24, 201015 yr Author Holding out for about an hour still before I turn on the server, but here's part of the syslog. Most of the lines in the log are the aforementioned Md/handle lines. Hope the syslog helps. SysLog.zip
November 24, 201015 yr I see this starting the copy from the old parity disk to the new: Nov 22 20:57:59 Tower emhttp: copying disk6 to disk0 ... Nov 22 21:05:12 Tower emhttp: ... copying 1% complete Nov 22 21:12:28 Tower emhttp: ... copying 2% complete Nov 22 21:19:43 Tower emhttp: ... copying 3% complete Nov 22 21:26:58 Tower emhttp: ... copying 4% complete Nov 22 21:34:14 Tower emhttp: ... copying 5% complete And then this as it completes the copy: Nov 23 08:02:00 Tower emhttp: ... copying 100% complete Nov 23 08:02:00 Tower emhttp: ... syncing Nov 23 08:02:02 Tower kernel: mdcmd (38): start SWAP_DSBL Nov 23 08:02:02 Tower kernel: unraid: allocating 54200K for 1280 stripes (10 disks) Nov 23 08:02:02 Tower kernel: md1: running, size: 976762552 blocks Nov 23 08:02:02 Tower kernel: md2: running, size: 976762552 blocks Nov 23 08:02:02 Tower kernel: md3: running, size: 976762552 blocks Nov 23 08:02:02 Tower kernel: md5: running, size: 976762552 blocks Nov 23 08:02:02 Tower kernel: md6: running, size: 976762552 blocks Nov 23 08:02:02 Tower kernel: md7: running, size: 390711352 blocks Nov 23 08:02:02 Tower kernel: md8: running, size: 390711352 blocks Nov 23 08:02:02 Tower kernel: md9: running, size: 390711352 blocks The first error was here: Nov 23 08:02:26 Tower kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x380000 action 0x6 Nov 23 08:02:26 Tower kernel: ata3.00: BMDMA stat 0x25 Nov 23 08:02:26 Tower kernel: ata3: SError: { 10B8B Dispar BadCRC } Nov 23 08:02:26 Tower kernel: ata3.00: failed command: READ DMA EXT Nov 23 08:02:26 Tower kernel: ata3.00: cmd 25/00:00:ff:0e:01/00:04:00:00:00/e0 tag 0 dma 524288 in Nov 23 08:02:26 Tower kernel: res 51/84:2f:ff:0e:01/84:03:00:00:00/e0 Emask 0x10 (ATA bus error) Nov 23 08:02:26 Tower kernel: ata3.00: status: { DRDY ERR } Nov 23 08:02:26 Tower kernel: ata3.00: error: { ICRC ABRT } CRC errors are frequently bad cabling. ( A loose cable where one of the shield grounds are not making good contact can do this too ) I think if you replace the cable to disk1... or , at the least re-seat it, it might just have the data you stored there. The parity disk is copied. The only part of the puzzle is to re-build disk6. Joe L.
November 24, 201015 yr Author Well - the server actually does start up - so I stopped it again and here is the syslog from that attempt. I removed many thousands of identical lines like the ones here: Nov 24 20:33:00 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 21536 does not match to the expected one 1 Nov 24 20:33:00 Tower kernel: REISERFS error (device md6): vs-5150 search_by_key: invalid format found in block 29294593. Fsck? It had over 363.000 lines ....! It appears that the disk1 is not read properly. I actually did try to put it in a machine with only that disk in and my backup unRAID USB, and briefly assigned it without ever starting the unrais server on that machine. Don't know if it had an impact on disk1 not being readable here. Feels like I'm in quicksand... syslog-2010-11-24-reduced.zip
November 24, 201015 yr Well - the server actually does start up - so I stopped it again and here is the syslog from that attempt. I removed many thousands of identical lines like the ones here: Nov 24 20:33:00 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 21536 does not match to the expected one 1 Nov 24 20:33:00 Tower kernel: REISERFS error (device md6): vs-5150 search_by_key: invalid format found in block 29294593. Fsck? It had over 363.000 lines ....! It appears that the disk1 is not read properly. I actually did try to put it in a machine with only that disk in and my backup unRAID USB, and briefly assigned it without ever starting the unrais server on that machine. Don't know if it had an impact on disk1 not being readable here. Feels like I'm in quicksand... That simple means the file system on it needs to be repaired. Don't give up on the disk just yet. Get a SMART report on it. If it looks good, then a reiserfsck of it would be in order.
November 24, 201015 yr Author Oh you posted while I was reducing the logfile. I actually also hope that the disk1 has the data on it. But it did not look promising when I accessed it from its disk share. There were a few files, some folders but they were empty (as seen at the time). I still have the original disk6 that was disconnected originally. So it MUST have all the data from there. Still disk1 needs to be verified somehow... Wow - you posted again :-) Smart report looks fine I think: smartctl version 5.38 [i486-slackware-linux-gnu] Copyright © 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: WDC WD10EADS-00P8B0 Serial Number: WD-WCAVU0248604 Firmware Version: 01.00A01 User Capacity: 1,000,204,886,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Wed Nov 24 21:22:28 2010 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (22980) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x303f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 180 177 021 Pre-fail Always - 5991 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 615 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 090 090 000 Old_age Always - 7681 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 127 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 74 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 2828 194 Temperature_Celsius 0x0022 118 094 000 Old_age Always - 32 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 9935 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
November 24, 201015 yr Author Get a SMART report on it. If it looks good, then a reiserfsck of it would be in order. What I read from that is that I would activate the "File System Check" button on the "Disk Management" screen?
November 24, 201015 yr Author I guess it is the commandline version. But which parameters are to be applied?
November 24, 201015 yr Author The reiserfsck --check /dev/sdc stated thath the superblock is corrupted, and that states that I should run the --rebuild-sb paramter. Is that actually what I should do?
November 24, 201015 yr I guess it is the commandline version. But which parameters are to be applied? You always start with just the basic command. If an additional option is needed, it will tell you. So, since this is disk1, if the array is able to be started, you would type reiserfsck /dev/md1 to perform the check. If the array cannot be started, then you'll need to use reiserfsck on the actual physical disk partition but that will invalidate parity so let's wait on that first.
November 25, 201015 yr Author I did this on the console, but it did not end up in the syslog (which I'm attaching anyway, since I have started the array in the mean time) So I made a photo of the console in stead. It essentially states, that i should run --rebuild-tree Should I? syslog-2010-11-25-reduced.zip
November 25, 201015 yr Author Thanks - I've started it. I'll hit the sack (it's 2:45AM) and report back i a few hours how it went.
November 25, 201015 yr Thanks - I've started it. I'll hit the sack (it's 2:45AM) and report back i a few hours how it went. You must be somewhere in Europe... It is evening here on the east coast of the US. Getting ready for a Thanksgiving holiday dinner tomorrow. I have a Pumpkin souffle in the oven baking... and it smells wonderful. Have a good nights sleep. Joe L.
November 25, 201015 yr Author Oh yes - it's Your Thanksgiving weekend. We don't have that here in Copenhagen, Denmark. But we have the first snow of the year And you just made med scavenge the kitchen for a snack before bedtime with all that food talk ...
November 25, 201015 yr Souffle done the night before? Won't it have collapsed by tomorrow? Or are you doing one for a test?
November 25, 201015 yr Author Happy Thanksgiving all you Americans As far as my regeneration of data goes, it's not happy at all... As can be seen from the attached log, it appears that it has discarded all my data. Unless the parity overwrote the disk, the data must still be there. Or am I reading the reporting from the disk check wrongly? It shall be noted, that the disk stayed "unformatted" in unRAID. But I would expect that as long as the server is not restarted. I have now stopped unRAID server, but I'm really concerned if I can start the unRAID server again, in case the disk starts being overwritten or some of the other disks are overwritten... Can I safely start the array to see that disk1 is indeed empty, or do I have other options of exploring it? In any case, there must be a way to rebuild the disk contents! Is there a reiserfsck command or another tool that can (attempt) to rebuild the contents? putty-tower-2010-11-25-reduced.zip
November 25, 201015 yr In any case, there must be a way to rebuild the disk contents! Is there a reiserfsck command or another tool that can (attempt) to rebuild the contents? The only way I know is to perform a full scan of the drive while rebuilding the file tree. http://lime-technology.com/forum/index.php?topic=5087.msg47070#msg47070 The would be on /dev/md1, exactly as in the example post. reiserfsck --rebuild-tree -S /dev/md1
Archived
This topic is now archived and is closed to further replies.