We have an array of one parity drive and two data drives that were getting old so we decided to get new and larger drives. We first replaced the parity drive and did a parity sync successfully, and then replaced one data drive with a successful data-rebuild. However after replacing the second data drive we are now unable to get through the data rebuild.
Diagnostics from when it first happened:
dmtower-diagnostics-20240331-1522.zip
Tried again a few times but it failed within an hour with the same errors in the syslog. Looked like a connection issue so we replaced the cables and tried again. It then ran much further so we thought we were good but it failed again after ~9 hours this time when it was almost done.
Syslog from the latest attempt:
dmtower-syslog-20240401-1620.zip
Any ideas what's going on here? It's also starting to become a bit time sensitive to get back up, so is there a way to swap the old data drive back in without any rebuilds for now?