muntedewok Posted May 20, 2019 Share Posted May 20, 2019 Hi all, So I'm running Ver 6.6.1 with a 3TB WD Red NAS Parity, 3x 3TB WD NAS/Seagate drives and a 2TB WD drive and a 500GB WD drive for cache (will be swapping out for an SSD soon). Yesterday noticed that Disk 2 had spat the dummy, was no longer "mountable" so I eventually managed to perform a check on it, disk was OK so only way to get it back was to rebuild it. No worries. However at the same time I noticed that Drive 4 was having SMART issues as well. The rebuild on Drive 2 appeared to work OK and then drive 4 drive appeared to fail. Further investigation on this found that the SATA connector on the SAS breakout cable from my RAID controller appears to be faulty and wasn't reading correctly and also became unmountable, during the Drive 2 rebuild the drive 4 drive racked up 700,000,000 errors. I am not kidding. I ended up unplugging Drive 4 from the SAS breakout and plugged the SATA connector directly into the motherboard, no worries. However since then I've rebuild the drive and although it says it completed successfully, it has 33.6mb data on it out of 3TB. Checked drive folder view nothing on it of course. Also noticed that my shares, even though they are mountable, are empty with exception to 1 which is complaining about Some or All Files Unprotected. If I browse the disks individually, all the files are there, no worries and if I check the Included Disk field for the shares it's set to All as required. I've disconnected the apparently faulty disk 4 with the SMART errors and plugged in a brand new 3TB WD Red NAS drive and it's supposedly rebuilding, however it's also just sitting at 33.6mb. Is it possible to recover the data on the 3x drives which are showing data when I manually browse them and recreate the shares or something? I imagine it'd be a case of recreating the RAID with the existing working disks then adding the new 3TB drive into the array as well, cutting my losses on the data that's gone missing? I've obviously resigned myself to the fact I've lost a disk worth of data so I'm just in damage control/salvage mode at the moment. Good thing is it's just TV shows and movies so if worse comes to worst, I can rebuild it over time, just annoying when it's 11TB of data Thanks fellas, -Chris Quote Link to comment
JorgeB Posted May 20, 2019 Share Posted May 20, 2019 17 minutes ago, muntedewok said: during the Drive 2 rebuild the drive 4 drive racked up 700,000,000 errors This means the rebuitl disk will be severely (or completely) corrupt, so this is expected: 18 minutes ago, muntedewok said: although it says it completed successfully, it has 33.6mb data on it out of 3TB. Though this also makes think you formatted the disk. As for the rest, please post the diagnostics: Tools -> Diagnostics. Quote Link to comment
muntedewok Posted May 20, 2019 Author Share Posted May 20, 2019 Hey Johnnie, The new 3TB WD I installed was fresh out of the box, hadn't been spun up in any other machine. I have uploaded the diagnostics as requested. Cheers, -Chris tower-diagnostics-20190520-2134.zip Quote Link to comment
JorgeB Posted May 20, 2019 Share Posted May 20, 2019 Check filesystem on disk2: https://wiki.unraid.net/Check_Disk_Filesystems#Checking_and_fixing_drives_in_the_webGui Quote Link to comment
muntedewok Posted May 20, 2019 Author Share Posted May 20, 2019 Did you want me to post the outcome of the check or run a specific command against the drive as well? -Chris Quote Link to comment
JorgeB Posted May 20, 2019 Share Posted May 20, 2019 No need, unless you need more help. You'll also need to finish rebuilding disk4. Quote Link to comment
muntedewok Posted May 20, 2019 Author Share Posted May 20, 2019 (edited) 5 found corruptions can be fixed only when running with --rebuild-tree Think it might be told to hold onto my balls! Edited May 20, 2019 by muntedewok Quote Link to comment
JorgeB Posted May 20, 2019 Share Posted May 20, 2019 It's usually fine, though there can always be some data loss. Quote Link to comment
JorgeB Posted May 20, 2019 Share Posted May 20, 2019 7 minutes ago, johnnie.black said: there can always be some data loss. Though in this case there will be for sure, because of the corrupt rebuild. Quote Link to comment
muntedewok Posted May 20, 2019 Author Share Posted May 20, 2019 Probably explains what all this is then: init_source_bitmap: Bitmap 14882 (of 32768 bits) is wrong - mark all blocks [487653376 - 487686144] as used init_source_bitmap: Bitmap 14883 (of 32768 bits) is wrong - mark all blocks [487686144 - 487718912] as used Looks like it's going to take a while... Quote Link to comment
JorgeB Posted May 20, 2019 Share Posted May 20, 2019 --rebuild-tree always takes a few hours, around 2 or 3 hours per TB. Quote Link to comment
muntedewok Posted May 20, 2019 Author Share Posted May 20, 2019 No worries thanks for the heads up. I'll check on it in the morning and kick off the rebuild of Disk 4 as well, will take a good 6-7 hours or so judging by the last effort. Thanks for the assistance and hope it goes well! -Chris Quote Link to comment
muntedewok Posted May 21, 2019 Author Share Posted May 21, 2019 Hey Johhnie, Disk 2 has rebuilt OK and as predicted lost a heap of data, not too fussed about that easily recoverable from elsewhere, the shares have come back up correctly and data visible in the network mapped volumes now, all good! However, I've replaced the stuffed Disk 4 and it's complaining that the new disk is unmountable. I removed it and plugged it into my PC, used diskpart to clean the drive of all partitions etc and re-inserted it into the server. However when I start the volume again it's complaining the drive is unmountable even when I force a format on it. I've uploaded the latest log files if required. Kind regards, -Chris tower-diagnostics-20190521-0840.zip Quote Link to comment
JorgeB Posted May 21, 2019 Share Posted May 21, 2019 That's strange, I see the format failing but not why it's failing, try formatting with xfs, since reiser it's no longer recommended anyway, and to see if there is more info. Quote Link to comment
muntedewok Posted May 21, 2019 Author Share Posted May 21, 2019 Is there a way to format it while it's in the server still? I've only got remote access to my server at the moment from my office -Chris Quote Link to comment
JorgeB Posted May 21, 2019 Share Posted May 21, 2019 With the array stopped click on disk4 and change the filesystem to xfs, then start array and try to format again, if it still fails post new diags. Quote Link to comment
muntedewok Posted May 21, 2019 Author Share Posted May 21, 2019 New logs uploaded, unfortunately no dice on the format! -Chris tower-diagnostics-20190521-1506.zip Quote Link to comment
muntedewok Posted May 21, 2019 Author Share Posted May 21, 2019 Just downloaded the unassigned drives plugin, going to run a pre-clear on the drive and see what it does afterwards. -Chris Quote Link to comment
JorgeB Posted May 21, 2019 Share Posted May 21, 2019 Preclear won't help, it might be a bug partitioning a disabled disk, though don't remember seeing it before, wait for the rebuild to finish them format. Quote Link to comment
muntedewok Posted May 21, 2019 Author Share Posted May 21, 2019 Managed to get it to mount this way. Started RAID with drive 4 unassigned, and performed format on it. Then dismounted RAID, then selected drive and mounted again. Now it says it's rebuilding RAID however it's back at the point where it's only showing 33.6mb on the drive. New logs uploaded. Thanks, -Chris tower-diagnostics-20190521-1539.zip Quote Link to comment
JorgeB Posted May 21, 2019 Share Posted May 21, 2019 18 minutes ago, muntedewok said: it's back at the point where it's only showing 33.6mb on the drive. That's expected, it was already like that before, and it woulds always be after a format. Quote Link to comment
muntedewok Posted May 21, 2019 Author Share Posted May 21, 2019 The interesting thing being that even though the write count is going up with the rebuild (now at 12% and 857,688 writes) the data count isn't increasing. I'm wondering if the parity that was available for the drive previously is hosed... -Chris Quote Link to comment
JorgeB Posted May 21, 2019 Share Posted May 21, 2019 The first diags you posted already had an empty disk4: Filesystem Size Used Avail Use% Mounted on /dev/md1 1.9T 1.7T 197G 90% /mnt/disk1 /dev/md2 2.8T 1.7T 1.1T 60% /mnt/disk2 /dev/md3 2.8T 2.5T 256G 91% /mnt/disk3 /dev/md4 2.8T 33M 2.8T 1% /mnt/disk4 No amount of rebuilding will change that, whatever the problem was it happened before that, likely you formatted an unmountable disk, that deletes all data on that disk and updates parity accordingly. Quote Link to comment
muntedewok Posted May 22, 2019 Author Share Posted May 22, 2019 Fair call as well, hadn't factored in the fact Disk 2 had a corrupted rebuild. Disk 4 has rebuilt into the RAID array OK, lost about 4TB worth of data however it was only movies and I have a log of all that was in there anyway so no big deal. Shares are showing correctly as being protected with exception to the appdata and Downloads folder, which if I view the folder structure, are showing different disks being used (single or multiple disks) for each folder. I'll probably just rip these out because they're small in size, rebuild the share then move the files/folders back in. Thank you very much for assistance on this Johnnie, greatly appreciated. -Chris Quote Link to comment
itimpi Posted May 22, 2019 Share Posted May 22, 2019 3 hours ago, muntedewok said: Fair call as well, hadn't factored in the fact Disk 2 had a corrupted rebuild. Disk 4 has rebuilt into the RAID array OK, lost about 4TB worth of data however it was only movies and I have a log of all that was in there anyway so no big deal. Shares are showing correctly as being protected with exception to the appdata and Downloads folder, which if I view the folder structure, are showing different disks being used (single or multiple disks) for each folder. I'll probably just rip these out because they're small in size, rebuild the share then move the files/folders back in. Thank you very much for assistance on this Johnnie, greatly appreciated. -Chris The appdata User Share is normally expected to all be on the cache drive (which is why it would show as unprotected as you only have a single drive in cache). What setting do you have for “Use Cache” on the appdata Share? Also what is that setting set to for the ‘Downloads’ folder? Not sure what you mean when you say ‘rebuild’ in the context of these shares❔ Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.