May 19, 201115 yr I noticed a problem with SABnzbd being unable to save files a few days ago. I took a closer look today and got "read only file system" errors in the terminal when attempting to create/remove some files and the same thing when accessing the shares in Finder (SMB share) even though it said I had read/write permission. I found a suggestion to use reiserfsck on the forum (http://lime-technology.com/forum/index.php?topic=11021.0 - http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems), and when I did I got the following: root@unraid:~# umount /dev/md1 root@unraid:~# reiserfsck --check /dev/md1 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md1 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes The problem has occurred looks like a hardware problem. If you have bad blocks, we advise you to get a new hard drive, because once you get one bad block that the disk drive internals cannot hide from your sight,the chances of getting more are generally said to become much higher (precise statistics are unknown to us), and this disk drive is probably not expensive enough for you to you to risk your time and data on it. If you don't want to follow that follow that advice then if you have just a few bad blocks, try writing to the bad blocks and see if the drive remaps the bad blocks (that means it takes a block it has in reserve and allocates it for use for of that block number). If it cannot remap the block, use badblock option (-B) with reiserfs utils to handle this block correctly. bread: Cannot read the block (2): (Input/output error). Aborted root@unraid:~# When I viewed the web interface it told me that disk 1 and 2 were missing (parity and cache still present). I rebooted the machine and when it came back up it says that disk 1 is missing. Now in terminal I can't see /dev/md2 (or 1) to try reiserfsck again. The disk is only a couple months old. I did preclear. I have been doing full parity check every month (last check 18 days ago, no errors). Attached current syslog. There should be 1 2TB parity, 2 2TB data and 1 SSD cache disks. What can I try next? EDIT: I'm using 5.0b6a. I tried rebooting again, and both disk 1 and 2 were missing. Another reboot and only disk 1 is missing again. syslog.txt
May 19, 201115 yr That error has typically required the disk to be rebuilt to get it back. You have to get disk2 to be stable for that to happen. You should be able to run the array with disk1 simulated, and if so you then assign a drive as disk1 and unRAID will rebuild it. Peter
May 19, 201115 yr Author I can't assign a device to disk1 though, there are no devices in the drop-down. It tells me the serial number of the device that it is expecting, but says that it is missing. I will try to re-seat the cables when I get home tonight. But it seems weird that the device was not "missing" to begin with, but at least one file system was "read only". Only when I stopped SMB and unmounted /dev/md1 and ran reiserfsck did it show up as "missing". Since then disk2 has intermittently been reported as "missing" as well. How can I be sure it is "stable" and trust starting the array with a disk missing in order to rebuild it?
May 19, 201115 yr Author I re-seated all the data and power cables for all disks on the motherboard and disk. No change. dmesg (attached) reports the following: ata2.00: link is slow to respond, please be patient (ready=0) ata2.00: SRST failed (errno=-16) ata2.00: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata2.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata2.00: failed to IDENTIFY (I/O error, err_mask=0x100) ata2.00: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata2.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata2.01: ATA-8: WDC WD20EARS-00MVWB0, 51.0AB51, max UDMA/133 ata2.01: 3907029168 sectors, multi 0: LBA48 NCQ (depth 0/32) ata2.01: configured for UDMA/133 Is the disk dead? Should I try plugging it into my Mac Pro instead to see what happens? I do have another 2TB drive (but not a green one). Should I chuck that in and wait the 25 hours for preclear before starting the array and attempting a rebuild? Even if this disk is dead, why does disk 2 appear missing also half the time after a reboot? dmesg.txt
May 19, 201115 yr Author I swapped the data on the parity disk and disk 1 and started the server. It took a long time to start up and get access to the web server. The array started this time, but disk 1 has a red icon. The syslog mentions a bunch of read errors on disk 1 (attached). What should I do now? UPDATE: I tried to run a short smartctl test on the disk through unMENU, and got the following. smartctl -t short -d ata /dev/sdb 2>&1 smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net Smartctl: Device Read Identity Failed (not an ATA/ATAPI device) A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. I started a smartctl test on another disk and it started OK (same make/model disk). Not sure where/when I'll see the results, though. unMENU says it will appear in the Smart-Status-Report. But where is this? syslog.txt
May 19, 201115 yr Author Tried a brand new data cable on disk 1 and it came up as "not installed" (not even "missing" anymore), and not even listed in the drop-down to assign a device to a disk.
May 19, 201115 yr Author I put the disk into my Mac Pro and while the Mac can't read the disk (no reiserfs support) it says SMART status is "verified" and doesn't report any errors. I put another 2TB disk in the unRAID server, assigned it to disk 1, the ball turned blue, and I started the array / rebuild. I didn't pre-clear this new disk. Do I need to do some additional tests or take any other steps once the rebuild is complete? I'm not going to format the old disk and attempt to read it on the Mac yet, in case the rebuild fails and the old disk is actually salvageable.
May 19, 201115 yr I was going to suggest disconnecting the disk and seeing if the system becomes better behaved, but it does sound like that drive has got problems. You'll know it's was likely the drive if the rebuild succeeds. Peter
May 19, 201115 yr Author I was going to suggest disconnecting the disk and seeing if the system becomes better behaved, but it does sound like that drive has got problems. You'll know it's was likely the drive if the rebuild succeeds. Peter The rebuild appears to have completed successfully. At least, it shows all disks with green balls now. But it also shows 357804540 errors on the parity disk, and "Last checked on Fri May 20 07:20:51 2011 EST, finding 357799241 errors" under the array status section (different to the parity disk by 1). But when I try to read from the disk1 share, I get "The operation can't be completed because an unexpected error occurred (error code -51)." (OS X 10.6). I managed to open one video file with quick look, but my tbn (thumbnail) files won't open and nfo files I've tried open and the contents looks like gibberish. What should I do now? UPDATE: My syslog is currently 2.0GB (the max?) and filled with lines like: May 20 02:43:26 unraid kernel: handle_stripe read error: 1171388384/0, count: 1 (Errors) May 20 02:43:26 unraid kernel: md: disk0 read error (Errors) May 20 02:43:26 unraid kernel: handle_stripe read error: 1171388392/0, count: 1 (Errors) May 20 02:43:26 unraid kernel: md: disk0 read error (Errors) May 20 02:43:26 unraid kernel: handle_stripe read error: 1171388400/0, count: 1 (Errors) May 20 02:43:26 unraid kernel: md: disk0 read error (Errors) May 20 02:43:26 unraid kernel: handle_stripe read error: 1171388408/0, count: 1 (Errors) May 20 02:43:26 unraid kernel: md: disk0 read error (Errors) May 20 02:43:26 unraid kernel: handle_stripe read error: 1171388416/0, count: 1 (Errors) May 20 02:43:26 unraid kernel: md: disk0 read error (Errors) May 20 02:43:26 unraid kernel: handle_stripe read error: 1171388424/0, count: 1 (Errors) May 20 02:43:26 unraid kernel: md: disk0 read error (Errors) UPDATE: Now I'm also getting input/output error on my cache drive (SSD). root@unraid:/mnt/cache# ls -la /bin/ls: reading directory .: Input/output error total 0 root@unraid:/mnt/cache# There should be 3 folders on the cache drive. .couchpotato, .sabnzbd and .sickbeard. Unraid web interface reports cache drive as having a green ball.
May 20, 201115 yr It look like you parity drive is bad or misbehaving. Post a SMART report of the parity disk.
May 20, 201115 yr Author It look like you parity drive is bad or misbehaving. Post a SMART report of the parity disk. I can't do a smart test now because I'm still trying to copy files off the last remaining "good" drive now that this whole system looks about to die. But when I click Smart Status Report for the parity drive I get the following: smartctl -a -d ata /dev/sdb smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net Smartctl: Device Read Identity Failed (not an ATA/ATAPI device) A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. Not sure if/when the last SMART report was generated, though, but this doesn't look good. So I've potentially got the parity disk AND disk 1 failing at the same time? I think I viewed the smart report for the parity disk and disk 2 last night and didn't have this problem (I might be remembering wrong), but this is the same error I got when I tried to run a short SMART test on the old disk 1.
May 20, 201115 yr Author So after my attempt to do a smart status report on the parity drive, it was showing the same symptoms as the old disk1. I stopped the array which was difficult enough because (I think) unraid was trying to write a 2+ GB syslog to my USB stick. I only discovered this by examining the process list in a shell (thankfully I could still connect) and saw that the unraid stop command was still running. I tried to run it again in the shell and its output froze on removing the old log and trying to write a new log. I hit CTRL C and after a few more minutes inactivity, the machine powered off. Having the web interface say nothing but "shutting down" for 5 plus minutes (seemingly frozen) is not helpful. When I restarted, I decided to put the old disk1 back in to see if it was actually damaged and if I could salvage data off it, since the rebuilt disk1 appears to be mostly corrupted. Now it's telling me that the old disk1 is a "new parity drive" (blue ball) and the old parity disk is not showing anywhere. I unassigned the old disk1 from the parity drive slot, and in unmenu disk management I can now do a smart report on the old disk1 which reports no problems. However, it doesn't say that there is a reiserfs file system on the disk (gives me the option to create one, instead of mounting one, like it does for the cache drive). Is this disk screwed, or is my parity disk screwed, or are both disks screwed? How can I find out for sure if there is still a reiserfs file system on the disk (there should be, if the disk is actually OK) and mount it and get data off it? UPDATE: Here is the smart status report for the old disk1, which is now visible again to unraid: smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Green (Adv. Format) family Device Model: WDC WD20EARS-00MVWB0 Serial Number: WD-WMAZA3492132 Firmware Version: 51.0AB51 User Capacity: 2,000,398,934,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Fri May 20 19:29:21 2011 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (36660) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3035) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 253 172 021 Pre-fail Always - 941 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 106 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 097 097 000 Old_age Always - 2192 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 31 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 15 193 Load_Cycle_Count 0x0032 196 196 000 Old_age Always - 13012 194 Temperature_Celsius 0x0022 114 101 000 Old_age Always - 36 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 2192 - # 2 Short offline Completed without error 00% 2172 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Here is the partition table for the disk: root@unraid:~# fdisk /dev/sdc WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help): p Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes 1 heads, 63 sectors/track, 62016336 cylinders Units = cylinders of 63 * 512 = 32256 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdc1 2 62016336 1953514552 83 Linux Partition 1 does not end on cylinder boundary. Command (m for help): This looks the same as the partition table for my cache drive, but unmenu disk management reports that the cache drive has a reiserfs file system on it and gives me a button to mount it. For the old disk1, it says there is none and gives me the option to create one.
May 20, 201115 yr Author I tried to mount /dev/sdc1 and was told there is no reiserfs on the partition. I ran resierfsck --check /dev/sdc1 and I get the following: reiserfs_open: the reiserfs superblock cannot be found on /dev/sdc1. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb. The Check Disk Filesystems page on the wiki says "Important Note #2!!! If the option --rebuild-sb is suggested, then PLEASE ask for assistance on the unRAID forums. The --rebuild-sb option requires answers that must be PERFECT." Should I attempt to rebuild the superblock? Or is there something else I can attempt to get this back? All I have done with this disk is swap the SATA cables and when that didn't work, swapped in a new disk, put this disk into my Mac Pro which recognised the disk but not the file system. I opted to "ignore" the disk while on the Mac, but the Mac did at least report SMART status verified. Now I have put it back into the unraid server and it is visible again, SMART is working again, the partition table is still there, but I can't mount it. I'm really not sure what the probable cause or solution to this mess is. Dead parity disk, dead disk1, dead cable, dead controller, etc. It seems that with trial and error each component is confirmed as OK but at the same time a different component starts to error.
May 20, 201115 yr You can try it and see what happens. The answers are; 3.6 block - 4096 journal - Yes resizer - no If you wanted, you could do a direct disk copy of this disk to that new one before you start this, just in case. Google the dd command and you'll get a good link with examples of how to use it to copy the whole disk. Peter
May 22, 201115 yr Author I made a mistake. What I thought was the old disk1 appearing as a new parity disk was actually the old parity disk appearing as a new parity disk. I mistakingly ran reiserfsck --rebuild-sb, --check and --rebuild-tree on it. Of course all that did was destroy my parity disk (which was already appearing as a "new" disk, anyway). However, I had already rebuild disk1 onto a new disk earlier, but many files appeared to be corrupted. So I ran reiserfsck --check on the new disk1 and found about 115 errors that needed to be corrected with --rebuild-tree. So I did that, and my lost+found folder had a tonne of files in it. Most without filename/extension, but I guess .avi or .mkv based on the file size. Some worked (didn't play the entire file though, would take forever), but most did not work. So I've lost around 500GB of data so far. The array seems to be running normally again now. I ran initconfig, re-assigned my old disk2 to disk2, newly rebuilt disk1 as disk1, and old parity as parity and started the array. Parity has been rebuilt. However, this experience has left me with little confidence in unraid's ability to recover from a single disk failure. I had been running a parity check every month and no errors were reported. When I found I couldn't copy some files off my array and subsequently disk1 was reported as "missing", I rebuilt a new disk1 from the parity data but that rebuilt disk was corrupted. How is this possible? It seems that only disk1 suffered an actual hardware fault, so why did unraid rebuild a corrupted disk from parity data? I guess that there were some errors on disk1 that were not fatal (did not result in unraid marking the disk as inactive) but did result in a corrupted filesystem, which was mirrored on the parity disk. When the disk eventually failed, I could re-create my data but it was already corrupted. Is it typical for disks to corrupt their file systems (and make the parity data useless) just before a fatal error that marks the disk as inactive? Perhaps I should be running a monthly (or even weekly, or daily) reiserfsck in addition to the regular parity check? UPDATE: Looks like many of the files that were NOT in "lost+found" are also still corrupted somehow. I could have lost much more than 500GB and it will be a big pain in the ass to try and detect which files are corrupted
May 22, 201115 yr I swapped the data on the parity disk and disk 1 and started the server. What does this mean?
May 22, 201115 yr Parity gets created when a new array config is initialized and then it is updated on a write to a data disk and it can be updated during a parity check. If you always run a nocorrect parity check then the parity can't be corrupted by a data drive that is failing. Seeing parity errors being corrected is a bad thing during a parity check. On a data write, unRAID will first read the parity disk and the data disk to determine what data is changing on the data disk so that it can make the appropriate changes to the parity disk. During this time, the parity could get damaged if the data disk is reading back erratic data. However, the only damage would be the files being written, assuming the disk is quickly failing and dies shortly after this happens. If unRAID is having a hard time reading the disk then it's likely a write will also fail which immediately removes the disk from the array and drops the array into disk simulation mode. There are also times that a disk/controller/memory etc will just cause erratic data read-back. In these cases, the parity will always have errors. Peter
May 22, 201115 yr UnRAID is great if a disk fails. In the unlikely case of a disk returning false results there is nothing it can do. UnRAID is not a backup for itself. No system is.
May 22, 201115 yr Author Parity gets created when a new array config is initialized and then it is updated on a write to a data disk and it can be updated during a parity check. If you always run a nocorrect parity check then the parity can't be corrupted by a data drive that is failing. Seeing parity errors being corrected is a bad thing during a parity check. On a data write, unRAID will first read the parity disk and the data disk to determine what data is changing on the data disk so that it can make the appropriate changes to the parity disk. During this time, the parity could get damaged if the data disk is reading back erratic data. However, the only damage would be the files being written, assuming the disk is quickly failing and dies shortly after this happens. If unRAID is having a hard time reading the disk then it's likely a write will also fail which immediately removes the disk from the array and drops the array into disk simulation mode. There are also times that a disk/controller/memory etc will just cause erratic data read-back. In these cases, the parity will always have errors. Peter These files were each written to only once (when copied to the array) over time, and most of them had subsequently been successfully and completely read (videos streamed during playback). So if the initial write was successful (verified by subsequent playback) and there were no more writes to a file, how would such a large number of files become corrupted AND how would the parity disk come to mirror the corrupted files instead of the original data that was successfully written?
Archived
This topic is now archived and is closed to further replies.