WorriedAboutDataLoss Posted March 24, 2018 Share Posted March 24, 2018 So: We had a power outage and my server went down. Usually when this happens and the server comes back up and I start the array it performs a parity check (as expected due to unclean shutdown). Right now when I brought the server back up it is also showing Disk 4 as "No Device" and I can hear a strange clicking so I'm guessing the drive's gone bad. How do I bring my array back up now? Do I just replace the drive and reboot and start the array - I'm guessing that won't work since the server thinks parity is not valid and will start a new parity check! I hope I'm not going to lose all that data on the 4th disk! Thanks in advance for any advice! Quote Link to comment
trurl Posted March 24, 2018 Share Posted March 24, 2018 Tools - Diagnostics, post complete zip. Quote Link to comment
WorriedAboutDataLoss Posted March 24, 2018 Author Share Posted March 24, 2018 Done! mediatower-diagnostics-20180324-1511 (1).zip Quote Link to comment
trurl Posted March 24, 2018 Share Posted March 24, 2018 Unclean shutdown normally does a noncorrecting parity check, but I'm not sure about what it does on that old version of unRAID you are running. The missing drive doesn't appear in the SMART so I can't tell if it is usable or not, but if you have a spare it is better to replace anyway. You could try checking the connections to see if it shows up and post another diagnostic. Your data is likely going to be OK after rebuilding the disk, but there is a small chance of corruption, which can probably be repaired. Quote Link to comment
WorriedAboutDataLoss Posted March 25, 2018 Author Share Posted March 25, 2018 Hmm. Even if I can start the array and it can do a non-correcting parity check - wouldn't it mark parity as invalid since it can't access disk4? Then will it be able to use the invalid parity to rebuild the disk? Quote Link to comment
trurl Posted March 25, 2018 Share Posted March 25, 2018 It won't do a parity check with a missing disk. Quote Link to comment
trurl Posted March 25, 2018 Share Posted March 25, 2018 If you start the array it should emulate the missing disk from the parity calculation and you can access the emulated disk and read its files. The emulated disk contents would be rebuilt to a new disk. Quote Link to comment
WorriedAboutDataLoss Posted March 26, 2018 Author Share Posted March 26, 2018 Thanks! Just waiting for my new drive to arrive. Will post here once I've got my server back up! Quote Link to comment
WorriedAboutDataLoss Posted April 4, 2018 Author Share Posted April 4, 2018 HELP!! I replaced the drive with the new drive. Assigned it to Disk 4. But the array now started with the following (see attached image and log files). I can't access any of my emulated data as well! Attaching logs here as well. mediatower-diagnostics-20180404-1034.zip Quote Link to comment
trurl Posted April 4, 2018 Share Posted April 4, 2018 Most likely you upset the connection on parity while replacing disk4. I don't understand why it would try to write and then disable parity while rebuilding disk4, but that does seem to be what syslog is saying. DON'T FORMAT anything. Do you still have the original disk4? Let's see if @johnnie.black has any ideas. Quote Link to comment
JorgeB Posted April 4, 2018 Share Posted April 4, 2018 Parity disk dropped offline since the beginning of the rebuild, possibly because of a bad cable, ideally you should have stopped the rebuild when you got the errors, since the rebuilt disk is completely corrupt, but if parity disk is OK we can force a new rebuild with the invalid slot command, first thing to do is to disable array autostart (if enable) power down, check/replace parity cables, power back on, and post new diags. Quote Link to comment
JorgeB Posted April 4, 2018 Share Posted April 4, 2018 12 minutes ago, trurl said: I don't understand why it would try to write and then disable parity while rebuilding disk4, but that does seem to be what syslog is saying. That's normal behavior, if there are read errors on another disk during a rebuild unRAID logs "multiple disk errors", i.e., there are errors on more disks than parity can recover, it will continue the rebuild but there will be corruption, in this case the complete disk is corrupt since parity dropped immediately offline, but on cases where there are only a few read errors the rebuilt disk will be mostly OK, which can be better then nothing. Quote Link to comment
WorriedAboutDataLoss Posted April 4, 2018 Author Share Posted April 4, 2018 Stopped the array. Rebooted. Attaching logs here. Console is showing following on Parity disk "PARITY DEVICE IS DISABLED", ALL DATA ON THIS DISK WILL BE ERASED WHEN ARRAY IS STARTED". I'm not starting the array for now! mediatower-diagnostics-20180404-1117.zip Quote Link to comment
JorgeB Posted April 4, 2018 Share Posted April 4, 2018 21 minutes ago, WorriedAboutDataLoss said: I'm not starting the array for now! Don't, it may take a few minutes for me to check the diags since I'm at work. Quote Link to comment
JorgeB Posted April 4, 2018 Share Posted April 4, 2018 Disk has SMART disable by default, and since it's not on the array unRAID doesn't enable it, you need to do this on the terminal: smartctl -s on /dev/sdg then post the output of: smartctl -a /dev/sdg Quote Link to comment
WorriedAboutDataLoss Posted April 4, 2018 Author Share Posted April 4, 2018 Output attached here smart.log Quote Link to comment
JorgeB Posted April 4, 2018 Share Posted April 4, 2018 Disk looks fine, follow these instructions carefully: -Tools -> New Config -> Retain current configuration: All -> Apply -Assign any missing disk(s) -Important - After checking the assignments leave the browser on that page, the "Main" page. -Open an SSH session/use the console and type: mdcmd set invalidslot 4 29 -Back on the GUI and without refreshing the page, just start the array, do not check the "parity is already valid" box, disk4 will start rebuilding, disk4 should mount immediately but if it's unmountable don't format, wait for the rebuild to finish and then run a filesystem check. Quote Link to comment
WorriedAboutDataLoss Posted April 4, 2018 Author Share Posted April 4, 2018 Completed above steps. It started by stying Array started. Parity-sync / Data rebuild 0.5% in the status bar. It also showed Disk4 as unmountable and showing Parity device as disabled. But after a few mins looks like whatever it was doing has stopped. Should I get another diagnostic log to post here? Quote Link to comment
WorriedAboutDataLoss Posted April 4, 2018 Author Share Posted April 4, 2018 Adding diagnostics now. mediatower-diagnostics-20180404-1557.zip Quote Link to comment
JorgeB Posted April 4, 2018 Share Posted April 4, 2018 Parity disk failed again, possibly it's a bad disk, you can try one more time after swapping both cables (or backplane slot) with a different disk to rule out everything else, then repeat the procedure above. Quote Link to comment
WorriedAboutDataLoss Posted April 6, 2018 Author Share Posted April 6, 2018 Sorry for the delayed response. I was unable to work on the system till now. So I switched cables around. Parity seems to be working now but disk 1 shows as disabled now. I performed the above steps again and attaching the logs here. Should I be running filesystem check now (if so - how?) OR should I repeat above steps again after trying to reseat disk1? mediatower-diagnostics-20180406-0930.zip Quote Link to comment
JorgeB Posted April 6, 2018 Share Posted April 6, 2018 Problem stayed with ATA8, i.e., looks like the problem is with that port on the Asmedia controller or the cable there connected, since you still have two onboard Intel PCH free ports connect disk1 to one of them using a new SATA cable, and run the invalid slot procedure again. Quote Link to comment
WorriedAboutDataLoss Posted April 6, 2018 Author Share Posted April 6, 2018 That seemed to help! moved the cable for disk1 to another slot. Went thru the same procedure as above. All disks are showing green (none of the disks show as unmountable!!). Parity sync/data-rebuild is in process and estimated time is 5.5 hrs. Hopefully this is it!! I'll post updates once this process finishes. Quote Link to comment
WorriedAboutDataLoss Posted April 7, 2018 Author Share Posted April 7, 2018 THANK YOU SO MUCH!! Really appreciate all the help from members johnnie.black and trurl! Array is back online. Data on disk 4 (that was lost originally) has been rebuilt and is accessible now. I'm a happy camper now Best! Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.