September 13, 20169 yr Hello, I am running Unraid v6.1.9 (Upgraded from 5 on August 21 2016) I recently noticed disk 2 with errors on the main screen and then finally last week I got the Red X and it's disabled. I have since ordered a new drive to replace it but I thought I would post the diagnosis files here to confirm it is in fact dead. Also, I want to confirm the following; On the dashboard, parity shows "Data is invalid". This is because disk 2 is out of the array? Last successful parity check was on September 1st (I have it scheduled to run monthly) I've attached the diagnosis, the drive in question is; WDC_WD20EARS-00MVWB0_WD-WMAZ20043234-20160912-1954.txt kenny-diagnostics-20160912-1954.zip
September 13, 20169 yr Community Expert Serial Number: WD-WMAZ20043234 5 Reallocated_Sector_Ct 0x0033 193 193 140 Pre-fail Always - 56 196 Reallocated_Event_Count 0x0032 156 156 000 Old_age Always - 44 197 Current_Pending_Sector 0x0032 197 197 000 Old_age Always - 1141 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 287 Definitely replace. Data is invalid is normal.
September 13, 20169 yr Author Serial Number: WD-WMAZ20043234 5 Reallocated_Sector_Ct 0x0033 193 193 140 Pre-fail Always - 56 196 Reallocated_Event_Count 0x0032 156 156 000 Old_age Always - 44 197 Current_Pending_Sector 0x0032 197 197 000 Old_age Always - 1141 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 287 Definitely replace. Data is invalid is normal. Thanks, i'm afraid to bring this up but I was checking the wiki and forum out and I noticed something that popped out. Pending sectors... My disk 3 & Disk 4 have a ton of those and it sounds like my parity might not be valid for the rebuild of disk 2... say it ain't so. Disk 3: WD-WCAZA8997232 Disk 4: WD-WCAVY1930651
September 13, 20169 yr Community Expert I would shut the server down until you get that new disk. Then try to rebuilt the new disk with current parity. It will either work or it won't. (Those pending sectors may still be readable.) You will not have lost anything by trying. And don't do anything with the disk you take out. In many cases, you can recover a lot of the files on those disks...
September 13, 20169 yr Community Expert You must be the 5th or 6th person this week that is running v6 and never bothered to setup Notifications You would have gotten notified by email about all those disk problems. In fact, you should have seen some indications on the Dashboard that things weren't looking good. Notifications is one of the most critical features of v6. Way too many people this week who just set it and forget it until they accidentally discover they are past the point of no return.
September 13, 20169 yr Author I would shut the server down until you get that new disk. Then try to rebuilt the new disk with current parity. It will either work or it won't. (Those pending sectors may still be readable.) You will not have lost anything by trying. And don't do anything with the disk you take out. In many cases, you can recover a lot of the files on those disks... Thank you, i'm going to get the hard drive today. I also bought another 3TB drive because i'm running out of space. So the way i'm going to try this; 1. Preclear 2 TB replacement hard drive 2. assign it to disk 2, let it rebuild 3. sidemount dead drive, compare data 4. Preclear 3TB new drive (This will be my nrw parity) 5. once rebuild is finished, assign 3TB as parity 6. Let parity rebuild 7. use spare 2TB (old parity drive) to clear out pending sectors Does that make sense? I have 2 weeks worth of work ahead of me lol
September 13, 20169 yr You must be the 5th or 6th person this week that is running v6 and never bothered to setup Notifications You would have gotten notified by email about all those disk problems. In fact, you should have seen some indications on the Dashboard that things weren't looking good. Notifications is one of the most critical features of v6. Way too many people this week who just set it and forget it until they accidentally discover they are past the point of no return. I think this should come set up as the default. I have mine set up now, but it's one of those "learned the hard way" lessons.
September 13, 20169 yr Flixx, you will have some problems recovering the data if your parity is invalid. Make sure you pull the bad drives and only power them up if you need to pull data. The normal copy command will not work with drives that have bad sectors. I found that the "dd" command with the conv=noerror will give you the best chances of recovering your data. Apparently FAT32 is the only file system that can remap bad sectors on a disk. fsck will only label them as bad, so if you have large files written across these bad blocks/sectors, they will be unable to be read or restored. Just some things I learned the past week when this happened to me.
September 13, 20169 yr Author You must be the 5th or 6th person this week that is running v6 and never bothered to setup Notifications You would have gotten notified by email about all those disk problems. In fact, you should have seen some indications on the Dashboard that things weren't looking good. Notifications is one of the most critical features of v6. Way too many people this week who just set it and forget it until they accidentally discover they are past the point of no return. My notifications are set to on for the alerts only. I am nearly at 100% disk space and when i enable any other type of notifications (notices or warnings) I get flooded with emails about my 11 hard drives being out of space... Furthermore, On the dashboard I do have the exclamation mark next to smart status. When I click it, I get in green letters next to "Last SMART test results" = "Completed without errors" which obviously is misleading. There are some items in orange below but when they say things like "pending" or "reallocation", they are not alarming words... besides the summary above says everything is A ok. Finally, I've been running V5 for years, I've had drives fail on me and replaced with no issue and a pending sector was never something that i've seen brought up or ever mentioned. This leads me to believe that either V6 introduced this new issue somehow or that the issue was there all along and unraid is not as reliable of a software raid as it's believed to be - Because who knows what V7 will realise was missed all along with the previous versions Flixx, you will have some problems recovering the data if your parity is invalid. Make sure you pull the bad drives and only power them up if you need to pull data. The normal copy command will not work with drives that have bad sectors. I found that the "dd" command with the conv=noerror will give you the best chances of recovering your data. Apparently FAT32 is the only file system that can remap bad sectors on a disk. fsck will only label them as bad, so if you have large files written across these bad blocks/sectors, they will be unable to be read or restored. Just some things I learned the past week when this happened to me. Thank you, I really hope I don't have to go down this route but i'll look into the DD command nonetheless.
September 13, 20169 yr Community Expert I am nearly at 100% disk space and when i enable any other type of notifications (notices or warnings) I get flooded with emails about my 11 hard drives being out of space... You will only get that email once per disk.
September 13, 20169 yr Community Expert I am nearly at 100% disk space and when i enable any other type of notifications (notices or warnings) I get flooded with emails about my 11 hard drives being out of space... You will only get that email once per disk. And you can set the threshold on a per disk basis.
September 17, 20169 yr Author The data rebuild on the failed drive was succesful I started working on the pending sectors on my disk 3 and disk 4 following the wiki by removing the disk, starting array, stopping array and re-adding the disk. Disk 3 had 1 pending sector. It now has 0 but Reallocated sector count still shows 1 and there is an alert for that disk because of it. Disk 4 is in the process of rebuilding, although it just started an hour ago, the pending sectors already went frolm 188 down to 24. How can I remove the alert for the reallocated sector number?
September 17, 20169 yr Just click on it on the dashboard and acknowledge it. You won't be notified again unless it changes Sent from my LG-D852 using Tapatalk
September 18, 20169 yr Author Just click on it on the dashboard and acknowledge it. You won't be notified again unless it changes Sent from my LG-D852 using Tapatalk Thanks, that's been done... So now a new issue appeared.. Since my last report, disk 4 data reconstruction was a success and i now have 0 pending sectors. So, to recap, i replaced the failed disk 2 and fixed the pending sectors on the other disk 3 and disk4. Now I can finally focus on my original intent, which was to upgrade my parity drive to a larger disk. Last night, I installed my 3tb drive, removed the 2tb parity from the configuration and added the 3 tb one. I started the parity rebuild... This morning Disk 1 had 69 read errors and now has 1 pending sector on it. I am now redoing a parity check. Attached is the latest diagnosis, below a snippet of the syslog; Sep 17 21:38:02 kenny kernel: mdcmd (36): spindown 5 Sep 17 21:55:31 kenny kernel: ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Sep 17 21:55:31 kenny kernel: ata1.01: failed command: READ DMA EXT Sep 17 21:55:31 kenny kernel: ata1.01: cmd 25/00:00:50:97:45/00:04:a4:00:00/f0 tag 0 dma 524288 in Sep 17 21:55:31 kenny kernel: res 51/40:1f:28:99:45/40:02:a4:00:00/f0 Emask 0x9 (media error) Sep 17 21:55:31 kenny kernel: ata1.01: status: { DRDY ERR } Sep 17 21:55:31 kenny kernel: ata1.01: error: { UNC } Sep 17 21:55:31 kenny kernel: ata1.00: configured for UDMA/133 Sep 17 21:55:31 kenny kernel: ata1.01: configured for UDMA/133 Sep 17 21:55:31 kenny kernel: sd 3:0:1:0: [sdd] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 Sep 17 21:55:31 kenny kernel: sd 3:0:1:0: [sdd] tag#0 Sense Key : 0x3 [current] [descriptor] Sep 17 21:55:31 kenny kernel: sd 3:0:1:0: [sdd] tag#0 ASC=0x11 ASCQ=0x4 Sep 17 21:55:31 kenny kernel: sd 3:0:1:0: [sdd] tag#0 CDB: opcode=0x28 28 00 a4 45 97 50 00 04 00 00 Sep 17 21:55:31 kenny kernel: blk_update_request: I/O error, dev sdd, sector 2756024616 Sep 17 21:55:31 kenny kernel: ata1: EH complete Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024552 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024552 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024560 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024560 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024568 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024568 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024576 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024576 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024584 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024584 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024592 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024592 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024600 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024600 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024608 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024608 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024616 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024616 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024624 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024624 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024632 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024632 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024640 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024640 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024648 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024648 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024656 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024656 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024664 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024664 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024672 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024672 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024680 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024680 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024688 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024688 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024696 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024696 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024704 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024704 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024712 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024712 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024720 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024720 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024728 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024728 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024736 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024736 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024744 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024744 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024752 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024752 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024760 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024760 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024768 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024768 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024776 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024776 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024784 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024784 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024792 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024792 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024800 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024800 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024808 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024808 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024816 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024816 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024824 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024824 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024832 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024832 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024840 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024840 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024848 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024848 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024856 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024856 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024864 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024864 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024872 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024872 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024880 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024880 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024888 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024888 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024896 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024896 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024904 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024904 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024912 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024912 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024920 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024920 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024928 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024928 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024936 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024936 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024944 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024944 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024952 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024952 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024960 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024960 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024968 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024968 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024976 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024976 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024984 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024984 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756024992 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756024992 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025000 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025000 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025008 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025008 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025016 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025016 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025024 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025024 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025032 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025032 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025040 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025040 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025048 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025048 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025056 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025056 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025064 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025064 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025072 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025072 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025080 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025080 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025088 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025088 Sep 17 21:55:31 kenny kernel: md: disk1 read error, sector=2756025096 Sep 17 21:55:31 kenny kernel: md: multiple disk errors, sector=2756025096 kenny-diagnostics-20160918-0727.zip
Archived
This topic is now archived and is closed to further replies.