June 28, 201016 yr I would think 2GB to be enough. Does syslog still say out of memory? Segmentation fault is a program trying to access memory that it does not have. I dunno if it's a program bug or something else (failure to acquire new memory). The script won't solve the problem, but if it shows a memory high water mark being reached, it may bandaid the issue by forcefully dropping cache buffers.
June 29, 201016 yr Author Another day, another few runs of resierfsck. The following is the failure overnight, interestingly a "Cannot red the block" , followed by Segmentation fault this time. /Movies/Meet??The??Parents/VIDEO_TS/VTS_03_2.VOBvpf-10680: The file [2571 2587] has the wrong block count in the StatData (2095848) - corrected to (2095352) /VTS_03_3.VOBvpf-10680: The file [2571 2588] has the wrong block count in the St atData (2095984) - corrected to (2095656) /VTS_03_5.VOBbread: Cannot read the block (3435973836): (Invalid argument). Segmentation fault root@Tower:~# The only syslog error is Jun 29 00:12:35 Tower kernel: reiserfsck[1759]: segfault at 4b906a49 ip 0807563d sp bfec8084 error 4 in reiserfsck[8048000+4a000] Going to have another shot at the swap file with swappiness 100 now. I will then embark on the script. Matt
June 29, 201016 yr Author With an on disk swap file and swappiness of 100 I failed on pass 3 with an abort. I am thinking that maybe this is a bug with reiserfsck, or my data is tipping reiserfsck over, as the line it fails on is as follows: /VTS_01_5.VOBufile.c 391 are_file_items_correct are_file_items_correct: Position (offset == 409821185) in the middle ofthe file [1719 1730] was not found. Aborted root@Tower:~# WeeboTech, I am struggling to get your scrip to run, although i can issue the commands manually. I get an error (likely my own ignorance) root@Tower:~# sh cache.sh.3 ': not a valid identifierre: `FIELDS ': not a valid identifiert: `FIELDS cache.sh.3: line 10: syntax error near unexpected token `newline' 'ache.sh.3: line 10: ` case "${FIELDS[0]}" in root@Tower:~# bash cache.sh.3 ': not a valid identifierre: `FIELDS ': not a valid identifiert: `FIELDS cache.sh.3: line 10: syntax error near unexpected token `newline' 'ache.sh.3: line 10: ` case "${FIELDS[0]}" in root@Tower:~# grep Cached /proc/meminfo Cached: 146072 kB SwapCached: 0 kB root@Tower:~# echo 3 > /proc/sys/vm/drop_caches root@Tower:~# grep Cached /proc/meminfo Cached: 145988 kB SwapCached: 0 kB root@Tower:~# Another radical idea, would be to maybe move the disk over to another machine. I have a main media center machine running windows, with dual core and 4GB RAM (not compatible with my old server). If I unplugged all the windows HDD and plugged this one, could I boot off a linux flash drive and run reiserfsck from there. If so would unraid basic be a good starting point to use as a linux base, or maybe another distro ? Thanks Matt
June 29, 201016 yr With an on disk swap file and swappiness of 100 I failed on pass 3 with an abort. I am thinking that maybe this is a bug with reiserfsck, or my data is tipping reiserfsck over, as the line it fails on is as follows: This is my thought too. Re: script run. Make sure the first line has #!/bin/bash Make sure it is executable with chmod u+x cache.sh.3 Another radical idea, would be to maybe move the disk over to another machine. I have a main media center machine running windows, with dual core and 4GB RAM (not compatible with my old server). If I unplugged all the windows HDD and plugged this one, could I boot off a linux flash drive and run reiserfsck from there. If so would unraid basic be a good starting point to use as a linux base, or maybe another distro ? I would borrow the 4GB from that machine and install it temporarily into the unRAID machine. What version of resierfsck are you using? do reiserfsck -V
June 29, 201016 yr Please schedule a smart LONG test. after looking at the code, there are some messages returned that it cannot read a block off the hard drive. Anything in the syslog point to this? smartctl -d ata -tlong /dev/sd? where ? = the character of the drive. I believe this is sde. Then post the smartlog. This will take some time. After looking at the source code, I don't really know if it's a bad block on the hard drive, or a bad pointer somewhere in the tables. This bad value could be used somewhere internally which makes it point outside of an expected area. Hence the aborted and segmentation faults.
June 29, 201016 yr Author I would borrow the 4GB from that machine and install it temporarily into the unRAID machine. What version of resierfsck are you using? do reiserfsck -V Unfortunately the 4GB is new DDR2 memory and will not fit the old motherboard. Latest version of reiserfsck root@Tower:~# reiserfsck -V reiserfsck 3.6.21 (2009 www.namesys.com) I will kick the smart LONG test off, as soon as the latest rebuild-tree is finished which I have run directly on the server (which now has a head) instead of via telnet as before. (Dont ask me why I am doing this, just thought worth a try!) I will post the smart log. Thanks for this Matt
June 29, 201016 yr Make sure you disable the spin-down timer in unRAID. It will cause the long test to abort if it spins it down in the middle of the test. Joe L.
June 29, 201016 yr Author Chaps, The following is my long smart report root@Tower:~# smartctl -a -d ata /dev/sde smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Alle n Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: SAMSUNG HD103UJ Serial Number: S13PJ1LS704586 Firmware Version: 1AA01118 User Capacity: 1,000,204,886,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 3b Local Time is: Tue Jun 29 18:50:11 2010 GMT ==> WARNING: May need -F samsung or -F samsung2 enabled; see manual for details. SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (13533) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off supp ort. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 226) minutes. Conveyance self-test routine recommended polling time: ( 24) minutes. SCT capabilities: (0x003f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_ FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0007 067 067 011 Pre-fail Always - 10690 4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 860 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 10583 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 6992 10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 176 13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 184 Unknown_Attribute 0x0033 100 100 000 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 074 045 000 Old_age Always - 26 (Lifetime Min/Max 18/26) 194 Temperature_Celsius 0x0022 075 044 000 Old_age Always - 25 (0 21 27 18) 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 12563621 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 281 200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0 201 Soft_Read_Error_Rate 0x000a 100 100 000 Old_age Always - 0 SMART Error Log Version: 1 ATA Error Count: 14 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 14 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 2f 31 00 00 e0 Error: ICRC, ABRT 47 sectors at LBA = 0x00000031 = 49 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 40 20 00 00 e0 00 16:34:48.900 READ DMA c8 00 20 00 00 00 e0 00 16:34:48.870 READ DMA c8 00 08 00 00 00 e0 00 16:34:46.250 READ DMA e0 00 00 00 00 00 a0 00 16:34:45.220 STANDBY IMMEDIATE ec 00 00 00 00 00 a0 02 16:34:45.200 IDENTIFY DEVICE Error 13 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 bf 18 83 de ee Error: ICRC, ABRT 191 sectors at LBA = 0x0ede8318 = 2494 63576 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 d7 82 de ee 00 02:28:43.880 READ DMA c8 00 00 d7 75 de ee 00 02:28:43.780 READ DMA c8 00 00 d7 63 de ee 00 02:28:43.700 READ DMA c8 00 00 d7 5a de ee 00 02:28:43.660 READ DMA c8 00 f0 d7 52 de ee 00 02:28:43.600 READ DMA Error 12 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 8f 08 bd eb ed Error: ICRC, ABRT 143 sectors at LBA = 0x0debbd08 = 2335 53160 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 a0 f7 bc eb ed 00 02:19:20.650 READ DMA c8 00 60 97 b8 eb ed 00 02:19:20.610 READ DMA c8 00 a0 f7 b3 eb ed 00 02:19:20.600 READ DMA c8 00 60 97 af eb ed 00 02:19:20.560 READ DMA c8 00 a0 f7 aa eb ed 00 02:19:20.550 READ DMA Error 11 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 9f 88 99 86 ec Error: ICRC, ABRT 159 sectors at LBA = 0x0c869988 = 2101 47720 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 27 99 86 ec 00 02:08:20.870 READ DMA c8 00 00 27 90 86 ec 00 02:08:20.800 READ DMA c8 00 00 27 87 86 ec 00 02:08:20.770 READ DMA c8 00 00 27 7e 86 ec 00 02:08:20.720 READ DMA c8 00 00 27 75 86 ec 00 02:08:20.680 READ DMA Error 10 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 2f a0 69 f8 e9 Error: ICRC, ABRT 47 sectors at LBA = 0x09f869a0 = 16727 4912 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 50 7f 69 f8 e9 00 01:42:39.950 READ DMA c8 00 b0 cf 64 f8 e9 00 01:42:39.920 READ DMA c8 00 50 7f 60 f8 e9 00 01:42:39.910 READ DMA c8 00 00 7f 49 f8 e9 00 01:42:39.780 READ DMA c8 00 00 7f 40 f8 e9 00 01:42:39.740 READ DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA _of_first_error # 1 Extended offline Completed without error 00% 6989 - # 2 Short offline Completed without error 00% 6956 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. root@Tower:~#
June 29, 201016 yr It performed the test just fine: Extended offline Completed without error 00% 6989
June 29, 201016 yr it seems to point to a failure reading the disk. Yet the failure may not be a real failure of read, it may be the block number supplied is way out of range for the disk. (that is beyond my knowledge). The segmentation fault is because the read failed and returned NULL so the upper calling routine did not handle it correctly. Further review shows that it would print a lengthy message if it considers the block to be bad on the disk. ret = f_read(bh); if (ret > 0) { die ("%s: End of file, cannot read the block (%lu).\n", __FUNCTION__, block); } else if (ret < 0) { /* BAD BLOCK LIST SUPPORT * die ("%s: Cannot read a block # %lu. Specify list of badblocks\n",*/ if (errno == EIO) { check_hd_msg(); die ("%s: Cannot read the block (%lu): (%s).\n", __FUNCTION__, block, strerror(errno)); } else { fprintf (stderr, "%s: Cannot read the block (%lu): (%s).\n", __FUNCTION__, block, strerror(errno)); return NULL; } } mark_buffer_uptodate (bh, 0); return bh; } ....snip.... void check_hd_msg (void) { fprintf(stderr, "\nThe problem has occurred looks like a hardware problem. I'll think about it a lil more, but I'm running out of ideas.
June 29, 201016 yr Author Thanks guys, appreciate you taking a look. I may look to pull apart my main media center PC tomorrow and put in this disk with unRaid Basic - it is a far better PC and should rule out the hardware for one. In the mean time any ideas - let me know, ready and willing to give them a go! (UK based so the day is nearly over here) Matt
June 29, 201016 yr Thanks guys, appreciate you taking a look. I may look to pull apart my main media center PC tomorrow and put in this disk with unRaid Basic - it is a far better PC and should rule out the hardware for one. In the mean time any ideas - let me know, ready and willing to give them a go! (UK based so the day is nearly over here) Matt Do you have another disk in the server with enough free space on it ( 1TB free ) to make a copy? You could make an image of the failed disk to use to recover from any failed attempt at restoring it. To do that, just type: dd bs=1M if=/dev/sde1 of=/mnt/disk1/sde_image.img replacing "disk1" with whatever disk might have the space. Then, we can try some more drastic steps and recover back the place you are now. Joe L.
June 30, 201016 yr Author Do you have another disk in the server with enough free space on it ( 1TB free ) to make a copy? Unfortunately not, the last disk (1TB) has a small amount of data written to it, meaning that although nearly empty, it cannot be used. I will have to go and buy another 1TB - do you have something in mind to try? I am heading back to thinking I may restore from parity, this will at least get me back to where I was with a readonly drive that I can use to retrieve my data - remembering that disk 1 was still read only before I ran reiserfsck --rebuild-tree ? Trying the rebuild-tree on the much better machine today to rule out hardware/memory, will keep you posted Matt
June 30, 201016 yr Author Joe L, WeeboTech, Firstly thank you so much for all your suggestions and support over the past few days on this. I am glad to say this is now fixed and I have obtained a successful run through reiserfsck by moving the drive to my other 4GB and considerably more modern PC and running with a basic version of unRaid. The steps I followed were : Option 3 from the following on the Wiki http://lime-technology.com/wiki/index.php?title=FAQ#How_do_I_recover_data_from_an_unRAID_disk.3F Upgraded the the latest version of resisefsck as per Joe L's suggestion at the start of the thread Ran a --check to make sure that I definitely got the right disk out of my server - sure enough it reported that a --rebuild-tree had previously failed Ran --rebuild-tree which turned up far more errors than previously both through Pass 1 and Pass 3 Pass 3 (semantic) made a tonne of corrections Reiserfsck completed without issue I have a tonne of lost and founds to while away the evenings sorting through. The original server has now been upgraded to 2GB of DDR 2700 RAM (from original 1GB). This server is an old Asus A7N8X-X board with an AMD k62 3000 CPU. However evidently this still wass not enough for a successful reiserfsck in this case. So I guess that it is a hardware issue, I assume it needed a) More RAM (4GB in this case) b) Better higher speed RAM The "donor" machine for the successful run uses a Gigabit motherboard with an Intel dual core processor I assume that the initial corruption came from a failed SATA PCI Controller that I replaced - that Disk 1 used. Again thanks for all your help, learnt a lot about memory and cache! You chaps keep these forums and the user community going for unRAID, cant thank you enough. Cheers Matt
June 30, 201016 yr Since you performed the repair on a different PC, your parity data on the unRAID server is no longer in sync. You MUST do a full parity check on your server once you re-install the disk in it. Expect it to find LOTS of "errors" (they are not really errors. They just represent all the files in lost+found and all the corrections the reiserfsck made.) Only after it completes do you actually have parity protection again. So... press the "Check" button as soon as possible, and let it run while you celebrate your success at rebuilding your file-tree, and sort out what is in lost+found. The "file" command will probably be very helpful. (You'll need to install it, but it can help identify files by their contents) Joe L.
June 30, 201016 yr This has me a bit concerned. Surely the 4GB could have helped, but there is a deeper underlying problem here. I think I have a better understanding of the corruption issue here. I took a look at the reiserfsck code. if it ran out of memory, you would have gotten a very specific out of memory message. The constant checks and continuing corruption has me a bit concerned. I would suggest you do a lengthy memtest. I would also suggest using teracopy on your windows station and do some very large copies with "verification ON" That's one of the reasons I like teracopy. It copies to the machine, then reads it back and checks the CRC. You may want to do some md5 tests too. I would suggest you check out a program called md5deep. It's cases like this where my disktracker/locate database will come in handy. How old is that board? Are you aware of prior issues reported with nforce chipsets and corruption? http://lime-technology.com/forum/index.php?topic=1061.msg7227 Am I being too paranoid?
June 30, 201016 yr Am I being too paranoid? No. I don't think so. I think, at the least, multiple parity checks should be run. Once the first has gotten parity into sync, the subsequent runs should not find any errors. If they do, there is another hardware issue involved. Joe L.
June 30, 201016 yr Author Am I being too paranoid? No. I don't think so. Chaps, you most definitely are not, whilst I have my data back, I continue to have a more workable problem, whereby unRAID is crashing on parity check, meaning I have no resilience. Further more additional reiserfsck --check reveals further --fix-fixable problems, only --fix-fixable causes a crash. I think I am nearing the end with this board to be honest, I was sold by the performance of my new board. This board is from 2002/2003 so very old (although I have an identical as backup). Considering going for a a new bundle, however I will miss my 5 X PCI slots for a 4 SATA connection cards on each, few boards seem to pack as many expansion slots these days, with moderate on board SATA connectors. I am going to continue investigations, as it has been more or less stable up until now. Matt
June 30, 201016 yr Don't do any write operations onto that machine. In fact the mere mounting of a drive, writes data to the superblock. This in effect will make the situation worse. I would really suggest a pause until you have stable hardware. How many drives do you have now?
July 2, 201016 yr Author I have turned the machine off now, with the disks still readable, parity will not run, so I figured I need to take time out and think about hardware. How many drives do you have now? I have 4 x 1 tb drives, 1 tb parity and 500 gb cache in it. All these run through PCI SATA Cards with 4 connections on each. Matt
July 23, 201015 yr Author I thought I would post back in the spirit of things to update on my story. I ditched all my old hardware and bought a AsRock ALiveDual-eSATA2 mobo with a dual core AMD cpu. Put 2GB of new DDR2 RAM in it. This represents a reasonably good entry level system in my opinion. After rebuild, I could not believe how much quicker boot time and time to access my shares has been. Everything is so much more responsive. I have rebuilt parity a number of times successfully. I have had no more failures. Reiserfsck has run fine on the new system. So to conclude this thread, I guess I was pushing it a bit with my old nforce2 AMD K6 setup. Matt
July 23, 201015 yr I just received this board and it really Rocks!: Fast Atom Dual Core which draws minimal with power. 2 GB DDR2 RAM. It also got two PCI-e slots for expanding the SATA-connectors, besides the existing 4 on the mainboard. http://www.supermicro.com/products/motherboard/ATOM/945/X7SLA.cfm?typ=H
July 23, 201015 yr So to conclude this thread, I guess I was pushing it a bit with my old nforce2 AMD K6 setup. There were documented problems with the older nforce boards. Who knows, your board may have aged to a point where it was no longer reliable. I'm glad you are stable now! Good Luck!
Archived
This topic is now archived and is closed to further replies.