February 7, 20197 yr Hey everyone, thank you in advanced for any help, (unraid version 6.6.6, 24TB array, 4 data disks 1 parity) So, long story short, a disk was bumped while the array was running and was dropped from the array. We checked the forums, and followed the steps for replacing a data disk. This Drive is a known good drive, so a zeroing of the drive was not completed prior to parity rebuild. The rebuild initially started at around 110-120MB/s following our historical avg speed. The array was left over night and when checked on in the morning, noted the speed had dropped down to 205KB/s. Typically our array takes 21hrs to compete, current estimated finish 309 days and steadily increasing. It is noted our flash docker is maxed at 100%, not sure if this symptom is related but has not been previously present. Any information you may require for troubleshooting let me know and i will add it ASAP, this post is being closely monitored (please send help lol). Diags attached. Treat is post as a Noob first time post, apologizes for any mistakes, I will update post with any suggestions ASAP. xserve-diagnostics-20190206-1259.zip
February 7, 20197 yr Community Expert 15 minutes ago, PWRFNK said: followed the steps for replacing a data disk Might not have been anything wrong with the original disk. Connection problems are very much more common than bad disks. 8 minutes ago, PWRFNK said: flash docker is maxed at 100% That Dashboard entry actually has 3 numbers referring to 3 different things. The 1st number is telling you how full your flash drive is. Not very. The 2nd number is telling you how full your log is. Completely full. The 3rd number is telling you how full your docker image is. Not very. Your log has filled up with errors because all of your disks except cache have disconnected. You should always double check all connections any time you are mucking about in the case.
February 7, 20197 yr Author 19 minutes ago, trurl said: Might not have been anything wrong with the original disk. Connection problems are very much more common than bad disks. That Dashboard entry actually has 3 numbers referring to 3 different things. The 1st number is telling you how full your flash drive is. Not very. The 2nd number is telling you how full your log is. Completely full. The 3rd number is telling you how full your docker image is. Not very. Your log has filled up with errors because all of your disks except cache have disconnected. You should always double check all connections any time you are mucking about in the case. When we had initially shutdown the array to start the disk replacement process, all disks were checked, the disk in question removed, verified, reseated. Upon Boot all disks were verified present in BIOS then booted into unraid. Upon booting to unriad OS, disks were again checked to be present, disk then re-added and array started. The rebuilding process initially started without issue. The System is a Dell T420: https://www.dell.com/downloads/global/products/pedge/t420_spec_sheet.pdf https://topics-cdn.dell.com/pdf/poweredge-t420_owners-manual_en-us.pdf The Drives are sledded. Edited February 7, 20197 yr by PWRFNK
February 7, 20197 yr Community Expert 5 minutes ago, PWRFNK said: Upon Boot all disks were verified present in BIOS then booted into unraid. Upon booting to unriad OS, disks were again checked to be present, disk then re-added and array started. The rebuilding process initially started without issue. Something changed since you started because 20 minutes ago, trurl said: Your log has filled up with errors because all of your disks except cache have disconnected. Something's not working. Power, controller, ...
February 7, 20197 yr Author So to eliminate any possible variables would it be recommended to stop the rebuild, stop the array, shutdown the system and then reseat the drives to be certain it isn't power/controller/connection related, and restart the rebuild process or will that likely result in data loss (which is what I am trying to avoid)?
February 7, 20197 yr Community Expert No point in continuing the rebuild with those hardware issues. Restarting the rebuild won't make anything worse. If it offers to FORMAT anything, DON'T DO IT!
February 7, 20197 yr Author Update/next steps taken: We have shutdown the system, removed all disks, removed the back plane, visually verified all Sata/SAS ports for oxidization/damage/corrosion, none found. Visually inspected board, cleaned all sports and caddy assembly while all drives were out. Reinstalled backplane, reseated power, reseated SAS connectors A/B on the backplane and SAS card. Reseated all drives, and moved disk in question to a different bay to see if issue is related to poor seating in bay 4. Array has now been started and all disks found, rebuild started and running at historical avg rebuild rate. Will update post later if issues persist, change or are resolved.
Archived
This topic is now archived and is closed to further replies.