January 13, 20179 yr Hello, Here's my situation. I have 2 new 8TB drives that have been precleared and I'm using them to upgrade my system. I have done the following: 1. Did a parity check before changing any drives. 2. Replaced existing 4TB parity drive with new 8TB drive and did a rebuild. 3. Did another parity check which resulted in no errors. 4. Replaced disk2 (2TB) with the other 8TB drive and started the rebuild. After about 6 hours of rebuilding the rebuild rate goes down to 100KB/s and I started getting the following errors in the syslog: Jan 12 02:01:31 Tower kernel: sd 4:0:4:0: [sdl] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 Jan 12 02:01:31 Tower kernel: sd 4:0:4:0: [sdl] tag#0 Sense Key : 0xb [current] Jan 12 02:01:31 Tower kernel: sd 4:0:4:0: [sdl] tag#0 ASC=0x47 ASCQ=0x3 Jan 12 02:01:31 Tower kernel: sd 4:0:4:0: [sdl] tag#0 CDB: opcode=0x28 28 00 5e 64 4e a0 00 00 20 00 Jan 12 02:01:31 Tower kernel: blk_update_request: I/O error, dev sdl, sector 1583632032 Jan 12 02:01:31 Tower kernel: md: disk9 read error, sector=1583631968 Jan 12 02:01:31 Tower kernel: Buffer I/O error on dev md9, logical block 197953996, lost async page write Jan 12 02:01:31 Tower kernel: md: disk9 read error, sector=1583631976 Jan 12 02:01:31 Tower kernel: Buffer I/O error on dev md9, logical block 197953997, lost async page write Jan 12 02:01:31 Tower kernel: md: disk9 read error, sector=1583631984 Jan 12 02:01:31 Tower kernel: Buffer I/O error on dev md9, logical block 197953998, lost async page write Jan 12 02:01:31 Tower kernel: md: disk9 read error, sector=1583631992 Jan 12 02:01:31 Tower kernel: Buffer I/O error on dev md9, logical block 197953999, lost async page write I have the existing 4TB and 2TB drives intact and there have been writes to the array since the parity drive upgrade. The drive that is being rebuilt was full (less than 50MB free) and the drive with the failure is 2TB and has ~36GB free. Also, the error count showing up in the WebUI for the failing drive is less than 50 and the errors shown in the syslog seem to be the same blocks over and over again. The rebuild has made it past the size of the failing drive and is continuing at a normal rate. I'm only using single parity with Unraid 6.2.1. Attached is the syslog and smart report for the failing drive. I understand that this failing drive is likely toast but what are the various options I have to recover my data? TIA unraid_failure_2017-01-12.zip
January 13, 20179 yr Community Expert If there were writes do the array after the parity upgrade the old parity is useless, this is what I would do: -let the rebuild finish and if you have checksums check disk2 and replace corrupt file(s) by copying them from old disk2 (if like most you don't have checksums you'll need to copy everything from old disk2 and overwrite the rebuilt disk since there's no way of knowing which file(s) were affected). -after that copy/move everything you can from disk9 to another array disk There will be one ore more files you can't copy/move due to the bad sectors, you'll know what files they are and if they are easily replaceable, if they're not you can rebuild that disk using for example old disk2, there will be some corruption but files like video files should still be mostly playable. Another option but possible only if you are sure there were no writes to new disk2 is to do a new config with old disk2, trusting parity and rebuilding disk9.
January 13, 20179 yr Community Expert Set up notifications! The SMART you posted for the drive definitely indicates it needs to be replaced. If you had notifications setup it would have been telling you so. Since you seemed unaware of this drive's issues, I have to wonder about your other drives. Go to Tools - Diagnostics and post the complete diagnostics zip so we can check on them.
January 13, 20179 yr Author If there were writes do the array after the parity upgrade the old parity is useless, this is what I would do: -let the rebuild finish and if you have checksums check disk2 and replace corrupt file(s) by copying them from old disk2 (if like most you don't have checksums you'll need to copy everything from old disk2 and overwrite the rebuilt disk since there's no way of knowing which file(s) were affected). -after that copy/move everything you can from disk9 to another array disk There will be one ore more files you can't copy/move due to the bad sectors, you'll know what files they are and if they are easily replaceable, if they're not you can rebuild that disk using for example old disk2, there will be some corruption but files like video files should still be mostly playable. The checksum idea sounds like a good one, I'll use md5deep for that. Is it possible to replace any bad files with the good ones from old disk2 and then rebuild disk9 as-is? When parity was built there were no smart errors from disk9 so is it possible that the existing parity will work? Another option but possible only if you are sure there were no writes to new disk2 is to do a new config with old disk2, trusting parity and rebuilding disk9. The mover emptied the cache drive to the new disk2 during rebuild so this sounds out of the question. Set up notifications! The SMART you posted for the drive definitely indicates it needs to be replaced. If you had notifications setup it would have been telling you so. Since you seemed unaware of this drive's issues, I have to wonder about your other drives. Go to Tools - Diagnostics and post the complete diagnostics zip so we can check on them. I thought I did have notifications set up, I started receiving emails of the SMART failures about 6 hours into the rebuild. Are there other notifications that I'm unaware of that I should be enabling? I have emails enabled for all notification levels: status, notices, warnings, and alerts. I have attached the complete diagnostics zip (minus some mover entries from the syslog), can you take a look to see if I have other drive issues? Thanks. tower-diagnostics-20170112-1918.zip
January 14, 20179 yr Community Expert The checksum idea sounds like a good one, I'll use md5deep for that. Is it possible to replace any bad files with the good ones from old disk2 and then rebuild disk9 as-is? Checksum route would be best if they already existed, you can still create checksums for old disk2 and use them to verify the rebuilt disk, but it will take probably longer to create and then check them than to just copy all files from disk2 overwriting the rebuilt disk. Another option is to use a binary file compare utility to compare files from old disk2 and the rebuilt one, any different file(s) assume they are corrupted. When parity was built there were no smart errors from disk9 so is it possible that the existing parity will work? Not without some possible corrupt files, a 100% rebuild of disk disk9 would need not only correct parity but also all other disks, and we know new disk2 has some corruptions, even if you replace all corrupt files there's very little chance that there won't be some sectors different from what they should be.
January 14, 20179 yr Community Expert Set up notifications! The SMART you posted for the drive definitely indicates it needs to be replaced. If you had notifications setup it would have been telling you so. Since you seemed unaware of this drive's issues, I have to wonder about your other drives. Go to Tools - Diagnostics and post the complete diagnostics zip so we can check on them. I thought I did have notifications set up, I started receiving emails of the SMART failures about 6 hours into the rebuild. Are there other notifications that I'm unaware of that I should be enabling? I have emails enabled for all notification levels: status, notices, warnings, and alerts. I have attached the complete diagnostics zip (minus some mover entries from the syslog), can you take a look to see if I have other drive issues? Thanks. I apologize for jumping to conclusions about your notifications. SMART for all other drives looks OK, except the original disk2 has a few pending. Is that your latest diagnostics? The syslog ends yesterday and the last drive inventory in the syslog seems to show you replacing disk1 instead of disk2. Jan 11 19:57:03 Tower kernel: mdcmd (1): import 0 sde 7814026532 0 ST8000AS0002-1NA17Z_Z840Q2Z4 Jan 11 19:57:03 Tower kernel: md: import disk0: (sde) ST8000AS0002-1NA17Z_Z840Q2Z4 size: 7814026532 Jan 11 19:57:03 Tower kernel: mdcmd (2): import 1 sdf 7814026532 1 ST8000AS0002-1NA17Z_Z840Q39C Jan 11 19:57:03 Tower kernel: md: import disk1: (sdf) ST8000AS0002-1NA17Z_Z840Q39C size: 7814026532 erased Jan 11 19:57:03 Tower kernel: md: import_slot: 1 wrong Jan 11 19:57:03 Tower kernel: mdcmd (3): import 2 sdg 1953514552 0 WDC_WD20EARS-00MVWB0_WD-WMAZ20439916 Jan 11 19:57:03 Tower kernel: md: import disk2: (sdg) WDC_WD20EARS-00MVWB0_WD-WMAZ20439916 size: 1953514552 Jan 11 19:57:03 Tower kernel: mdcmd (4): import 3 sdc 1953514552 0 WDC_WD20EARS-00MVWB0_WD-WCAZA2048728 Jan 11 19:57:03 Tower kernel: md: import disk3: (sdc) WDC_WD20EARS-00MVWB0_WD-WCAZA2048728 size: 1953514552 Jan 11 19:57:03 Tower kernel: mdcmd (5): import 4 sdb 3907018532 0 ST4000VN000-1H4168_Z304X4PY Jan 11 19:57:03 Tower kernel: md: import disk4: (sdb) ST4000VN000-1H4168_Z304X4PY size: 3907018532 Jan 11 19:57:03 Tower kernel: mdcmd (6): import 5 sdi 3907018532 0 ST4000VN000-1H4168_Z302K0D1 Jan 11 19:57:03 Tower kernel: md: import disk5: (sdi) ST4000VN000-1H4168_Z302K0D1 size: 3907018532 Jan 11 19:57:03 Tower kernel: mdcmd (7): import 6 sdh 3907018532 0 Hitachi_HDS5C4040ALE630_PL1321LAGA5D0H Jan 11 19:57:03 Tower kernel: md: import disk6: (sdh) Hitachi_HDS5C4040ALE630_PL1321LAGA5D0H size: 3907018532 Jan 11 19:57:03 Tower kernel: mdcmd (: import 7 sdj 3907018532 0 ST4000VN000-1H4168_Z3028YEK Jan 11 19:57:03 Tower kernel: md: import disk7: (sdj) ST4000VN000-1H4168_Z3028YEK size: 3907018532 Jan 11 19:57:03 Tower kernel: mdcmd (9): import 8 sdk 3907018532 0 ST4000VN000-1H4168_Z302F9Z0 Jan 11 19:57:03 Tower kernel: md: import disk8: (sdk) ST4000VN000-1H4168_Z302F9Z0 size: 3907018532 Jan 11 19:57:03 Tower kernel: mdcmd (10): import 9 sdl 1953514552 0 ST2000DM001-9YN164_S2F06HKA Jan 11 19:57:03 Tower kernel: md: import disk9: (sdl) ST2000DM001-9YN164_S2F06HKA size: 1953514552 Jan 11 19:57:03 Tower kernel: mdcmd (11): import 10 sdm 2930266532 0 ST3000DM001-1CH166_Z1F33NVK Jan 11 19:57:03 Tower kernel: md: import disk10: (sdm) ST3000DM001-1CH166_Z1F33NVK size: 2930266532 Jan 11 19:57:03 Tower kernel: mdcmd (12): import 11 sdn 3907018532 0 ST4000DM000-1F2168_W300575G Jan 11 19:57:03 Tower kernel: md: import disk11: (sdn) ST4000DM000-1F2168_W300575G size: 3907018532 then Jan 11 19:57:34 Tower kernel: mdcmd (47): start RECON_DISK Jan 11 19:57:34 Tower kernel: unraid: allocating 66940K for 1280 stripes (13 disks) Jan 11 19:57:34 Tower kernel: md1: running, size: 7814026532 blocks Jan 11 19:57:34 Tower kernel: md2: running, size: 1953514552 blocks Jan 11 19:57:34 Tower kernel: md3: running, size: 1953514552 blocks Jan 11 19:57:34 Tower kernel: md4: running, size: 3907018532 blocks Jan 11 19:57:34 Tower kernel: md5: running, size: 3907018532 blocks Jan 11 19:57:34 Tower kernel: md6: running, size: 3907018532 blocks Jan 11 19:57:34 Tower kernel: md7: running, size: 3907018532 blocks Jan 11 19:57:34 Tower kernel: md8: running, size: 3907018532 blocks Jan 11 19:57:34 Tower kernel: md9: running, size: 1953514552 blocks Jan 11 19:57:34 Tower kernel: md10: running, size: 2930266532 blocks Jan 11 19:57:34 Tower kernel: md11: running, size: 3907018532 blocks then later we start with the disk9 problems Jan 12 02:01:31 Tower kernel: md: disk9 read error, sector=1583631968 Jan 12 02:01:31 Tower kernel: Buffer I/O error on dev md9, logical block 197953996, lost async page write Jan 12 02:01:31 Tower kernel: md: disk9 read error, sector=1583631976 Jan 12 02:01:31 Tower kernel: Buffer I/O error on dev md9, logical block 197953997, lost async page write Jan 12 02:01:31 Tower kernel: md: disk9 read error, sector=1583631984 Jan 12 02:01:31 Tower kernel: Buffer I/O error on dev md9, logical block 197953998, lost async page write Jan 12 02:01:31 Tower kernel: md: disk9 read error, sector=1583631992 Jan 12 02:01:31 Tower kernel: Buffer I/O error on dev md9, logical block 197953999, lost async page write
January 14, 20179 yr Author Is that your latest diagnostics? The syslog ends yesterday and the last drive inventory in the syslog seems to show you replacing disk1 instead of disk2. Sorry about that, you're right it's actually disk1 that I replaced and not disk2. I got confused and was thinking that disk1 was the parity disk. So, in my previous posts, when I say disk2 I mean disk1. Attached is the latest diagnostics. tower-diagnostics-20170113-2148.zip
January 14, 20179 yr Community Expert Is that your latest diagnostics? The syslog ends yesterday and the last drive inventory in the syslog seems to show you replacing disk1 instead of disk2. Sorry about that, you're right it's actually disk1 that I replaced and not disk2. I got confused and was thinking that disk1 was the parity disk. So, in my previous posts, when I say disk2 I mean disk1. Attached is the latest diagnostics. Parity is disk0. In fact that is the way it is referred to in syslog as you can see from that snippet I posted. Do you have backups? It looks like disk9 is going to prevent rebuilding disk1. What we really need to do is rebuild disk9 instead. Then maybe we can see if we can clear the few pending on disk2. Do you still have the original disk1? There are only 14 files (flash, cache, parity, 11 data disks) in the smart folder of your diagnostics so I assume the original disk1 isn't currently attached. I would like to get a SMART from it. Has anything been written to the array since parity was rebuilt?
January 14, 20179 yr Community Expert Reviewing the thread I see you have already answered 2 of my questions. And you said the rebuild had gotten past the size of disk9. Does it look like it is going to complete then? Not that it is likely to be a good rebuild in any case. Since there have been writes since parity rebuild the most important question is, do you have backups? Since you still have the original disk1 copies of any critical files on disk9 is most important.
January 14, 20179 yr Author Reviewing the thread I see you have already answered 2 of my questions. And you said the rebuild had gotten past the size of disk9. Does it look like it is going to complete then? Not that it is likely to be a good rebuild in any case. The rebuild did complete and I suspect that ~1TB of the rebuild completed before the drive failed. Since there have been writes since parity rebuild the most important question is, do you have backups? Since you still have the original disk1 copies of any critical files on disk9 is most important. Unfortunately, I don't have a backup. At this point I'm trying to find the optimal way to maximize the data that can be recovered. If I were to copy the files from old disk1 onto rebuilt disk1 and then rebuild disk9, is there any chance that would work? I could then try and copy files off of the bad disk9 but at this point I don't really trust it. Here is the smart report from the old disk1 gathered with CrystalDiskInfo on my Win7 machine connected with a sata-to-usb adapter: ---------------------------------------------------------------------------- (4) WDC WD20EARX-00PASB0 ---------------------------------------------------------------------------- Enclosure : Seagate USB USB Device (V=0BC2, P=A0A4, sa1) - wd Model : WDC WD20EARX-00PASB0 Firmware : 51.0AB51 Serial Number : WD-WMAZA8438694 Disk Size : 2000.3 GB (8.4/137.4/2000.3/2000.3) Buffer Size : Unknown Queue Depth : 32 # of Sectors : 3907029168 Rotation Rate : Unknown Interface : USB (Serial ATA) Major Version : ATA8-ACS Minor Version : ---- Transfer Mode : SATA/600 Power On Hours : 40369 hours Power On Count : 218 count Temparature : 32 C (89 F) Health Status : Good Features : S.M.A.R.T., 48bit LBA, NCQ APM Level : ---- AAM Level : ---- -- S.M.A.R.T. -------------------------------------------------------------- ID Cur Wor Thr RawValues(6) Attribute Name 01 200 200 _51 000000000001 Read Error Rate 03 170 166 _21 000000001942 Spin-Up Time 04 _92 _92 __0 00000000203A Start/Stop Count 05 200 200 140 000000000000 Reallocated Sectors Count 07 100 253 __0 000000000000 Seek Error Rate 09 _45 _45 __0 000000009DB1 Power-On Hours 0A 100 100 __0 000000000000 Spin Retry Count 0B 100 100 __0 000000000000 Recalibration Retries 0C 100 100 __0 0000000000DA Power Cycle Count C0 200 200 __0 000000000045 Power-off Retract Count C1 _10 _10 __0 00000008B9F6 Load/Unload Cycle Count C2 118 103 __0 000000000020 Temperature C4 200 200 __0 000000000000 Reallocation Event Count C5 200 200 __0 000000000000 Current Pending Sector Count C6 200 200 __0 000000000000 Uncorrectable Sector Count C7 200 200 __0 000000000002 UltraDMA CRC Error Count C8 200 200 __0 000000000000 Write Error Rate
January 14, 20179 yr Community Expert The options I listed earlier are still valid, just substitute disk2 for disk1.
January 14, 20179 yr Author The options I listed earlier are still valid, just substitute disk2 for disk1. Does the order matter? Can I copy the data from disk9 to disk1 (now 8TB) and then copy the data from old disk1 to new disk1? I'm assuming the the parity drive won't help me at all at this point?
January 14, 20179 yr Community Expert Order doesn't matter, you may want to use a different folder/share for disk9 data.
January 14, 20179 yr Community Expert Have you inspected the files already on the new rebuilt disk1? Just want to make sure it actually mounted since there was some possibility the rebuild might have been corrupt. As for your parity, if it wasn't already complaining about disk9 when you rebuilt parity it might be good. The only problem we had with it was that there had been writes to the array after it was built so it might not be in sync with the original disk1 and so you couldn't just put that disk back in. Another possibility would be to not worry about the files on the rebuild, format the new disk1, copy all files from original disk1, use original disk1 to rebuild disk9. Let johnnie.black respond to that idea before deciding anything though. Possible I am overlooking something. And start thinking about a backup plan! You don't have to backup everything but you absolutely must have backups for anything important and irreplaceable.
January 14, 20179 yr Community Expert And in case you don't know, copy from disk to disk, never from disk to share.
January 14, 20179 yr Community Expert Have you inspected the files already on the new rebuilt disk1? Just want to make sure it actually mounted since there was some possibility the rebuild might have been corrupt. This is true, check that it mounts and running a file system check is also a good idea before you start to copy. Another possibility would be to not worry about the files on the rebuild, format the new disk1, copy all files from original disk1, use original disk1 to rebuild disk9. Don't think this would work, as even after replacing all files there's no guarantee they will be using the exact same sectors, I think it's likely they wont.
January 14, 20179 yr Community Expert Have you inspected the files already on the new rebuilt disk1? Just want to make sure it actually mounted since there was some possibility the rebuild might have been corrupt. This is true, check that it mounts and running a file system check is also a good idea before you start to copy. Another possibility would be to not worry about the files on the rebuild, format the new disk1, copy all files from original disk1, use original disk1 to rebuild disk9. Don't think this would work, as even after replacing all files there's no guarantee they will be using the exact same sectors, I think it's likely they wont. I don't really follow you here. If you format the rebuilt disk1 while it is in the array, then parity will be updated to stay in sync with the format. Then if you mount the original disk1 outside the array and copy all of its files to the newly formatted new disk1, parity would be updated to stay in sync with the copy. I am not understanding why the "exact same sectors" is relevant. And after all the files have been copied from the original disk1 and they aren't needed anymore, it could be used to rebuild the bad disk9 since it is the same size.
January 14, 20179 yr Community Expert I misunderstood what you said, he can do that and I also mentioned that earlier, but there will always be some corruption on the rebuilt disk9, because of the prior rebuild errors. When parity was built there were no smart errors from disk9 so is it possible that the existing parity will work? Not without some possible corrupt files, a 100% rebuild of disk disk9 would need not only correct parity but also all other disks, and we know new disk2 has some corruptions, even if you replace all corrupt files there's very little chance that there won't be some sectors different from what they should be.
January 14, 20179 yr Community Expert Disk2 is still a problem and I had sort of forgotten about that. But it's not as big a problem as disk9. It is just 3 pending sectors so maybe any corruption from that would be minimal. So it would probably be a good idea to copy what can be copied from disk9 before trying to rebuild it. Even if you get disk1 and disk9 square something needs to be done about disk2. Not really any ideal path here.
January 14, 20179 yr Community Expert Assuming the pending sectors on disk2 are false positives, since there where no errors from it on the last rebuild, I would do the following: 1-copy all files from old disk1 to new disk1 overwriting all files (you can also format disk1 before the copy but I would guess the rebuild is mostly ok, there where very few errors, maybe do a file system check) 2-rebuild disk9 using a spare or old disk1 (after the copy above) 3-use all files that can be copied from old disk9 without error to replace the ones on the rebuilt disk9, so you know for sure those are good, the files that you can't copy due to the bad sectors use the ones on the rebuild disk, they should be mostly ok, and if they are for example video files should still be playable. -
January 14, 20179 yr Author 1-copy all files from old disk1 to new disk1 overwriting all files (you can also format disk1 before the copy but I would guess the rebuild is mostly ok, there where very few errors, maybe do a file system check) Does the new disk1 need to be exactly the same as the old disk1? Will keeping any newly added files on the disk affect the rebuild of disk9? 2-rebuild disk9 using a spare or old disk1 (after the copy above) I can probably use the old 4TB parity drive for this.
January 14, 20179 yr Community Expert 1-copy all files from old disk1 to new disk1 overwriting all files (you can also format disk1 before the copy but I would guess the rebuild is mostly ok, there where very few errors, maybe do a file system check) Does the new disk1 need to be exactly the same as the old disk1? Will keeping any newly added files on the disk affect the rebuild of disk9? That's not a problem because parity was updated when those files were added, what may cause some corruptions are the errors during disk1 rebuild, but nothing you can do about that now, with some luck after step 3 most/all files on it will be OK.
January 15, 20179 yr Author 3-use all files that can be copied from old disk9 without error to replace the ones on the rebuilt disk9, so you know for sure those are good, the files that you can't copy due to the bad sectors use the ones on the rebuild disk, they should be mostly ok, and if they are for example video files should still be playable. What's the best way to find out which files are unreadable before doing the copy, so that I can skip them?
January 15, 20179 yr Community Expert Copy them to another disk or another folder on the rebuilt disk9. You can also change order of things and do it like this: 1-copy all files from old disk1 to new disk1 overwriting all files (you can also format disk1 before the copy but I would guess the rebuild is mostly ok, there where very few errors, maybe do a file system check) 2-since you have space copy all the files that you can from disk9 to a new folder/share on disk1 (you can make a list of those you couldn't copy due to the bad sectors) 2-rebuild disk9 using a spare or old disk1 (after the copy above) 3-move all files from the the disk9 folder on disk1 to the rebuilt disk9, on the rebuilt disk9 check if the files you could not copy are ok (or try to get them again if you can, with some luck it can be just a couple of files)
Archived
This topic is now archived and is closed to further replies.