March 17, 20251 yr Hey all. Looking for an assist. I have a disk disabled in my array, but the SMART report seems fine? I went ahead and removed it from the array and re-adding, intentionally starting a rebuild. But it looks like something went awry and it's disabled again before the rebuild could finish. Would love any pointers here. The server's been running fine for about a year now, so not sure what the issue could be. Attaching the SMART report for context in case I missed something. Thanks in advance! TOSHIBA_HDWG11A_72E0A00GFBDG-20250317-1059.txt
March 18, 20251 yr Author Sure thing. Sorry for the delay. Attached to this post. stash-diagnostics-20250317-2056.zip
March 18, 20251 yr Community Expert 8 hours ago, gavination said: disabled again before the rebuild could finish Mar 13 17:39:27 Stash kernel: md: recovery thread: recon D4 ... Looks like rebuild finished Mar 14 09:34:28 Stash kernel: md: sync done. time=57301sec Mar 14 09:34:28 Stash kernel: md: recovery thread: exit status: 0 Then a non-correcting parity check started at midnight (scheduled I presume) on the 16th Mar 16 00:00:01 Stash kernel: mdcmd (37): check NOCORRECT Mar 16 00:00:01 Stash kernel: Mar 16 00:00:01 Stash kernel: md: recovery thread: check P ... Then it began to have problems with disk4 at almost exactly 3am which resulted in disabling the disk. Not clear on cause. Wonder if something is scheduled then. What do you get from command line with this? cat /etc/cron.d/root SMART looks OK and it even passed extended self-test recently. Emulated disk is mounted so should be OK to rebuild on top. I presume this is unrelated, but your appdata has files all over the array, though not on disk4.
March 18, 20251 yr Author 17 minutes ago, trurl said: Mar 13 17:39:27 Stash kernel: md: recovery thread: recon D4 ... Looks like rebuild finished Mar 14 09:34:28 Stash kernel: md: sync done. time=57301sec Mar 14 09:34:28 Stash kernel: md: recovery thread: exit status: 0 Then a non-correcting parity check started at midnight (scheduled I presume) on the 16th Mar 16 00:00:01 Stash kernel: mdcmd (37): check NOCORRECT Mar 16 00:00:01 Stash kernel: Mar 16 00:00:01 Stash kernel: md: recovery thread: check P ... Then it began to have problems with disk4 at almost exactly 3am which resulted in disabling the disk. Not clear on cause. Wonder if something is scheduled then. What do you get from command line with this? cat /etc/cron.d/root SMART looks OK and it even passed extended self-test recently. Emulated disk is mounted so should be OK to rebuild on top. I presume this is unrelated, but your appdata has files all over the array, though not on disk4. Appreciate you taking a look. Apologies for the misleading info. Checked back in the next day and just assumed it hadn't finished, apparently. As for the cron jobs running, I've dropped that below as well: # Generated system monitoring schedule: */1 * * * * /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null # Generated mover schedule: 0 */1 * * * /usr/local/sbin/mover &> /dev/null # Generated parity check schedule: 0 0 * * 0 /usr/local/sbin/mdcmd check NOCORRECT &> /dev/null # CRON for CA background scanning of applications 8 * * * * php /usr/local/emhttp/plugins/community.applications/scripts/notices.php > /dev/null 2>&1 Some basic automations that I've had running for a while now. And yeah the `appdata` directory being all over is a product of me experimenting with some stuff a while back, and of course I haven't fixed it in all my laziness.
March 18, 20251 yr Community Expert 2 minutes ago, gavination said: # Generated parity check schedule: 0 0 * * 0 /usr/local/sbin/mdcmd check NOCORRECT &> /dev/null Why are you running weekly parity checks? WIth 18TB parity that must take a significant part of each week.
March 18, 20251 yr Author 10 hours ago, trurl said: Why are you running weekly parity checks? WIth 18TB parity that must take a significant part of each week. It takes about a day, iirc. Appreciate your asking. I'd relatively recently switched to using the chonkier drives once I needed to do some expansion. Would cadence would you recommend?
March 18, 20251 yr Author 5 hours ago, JorgeB said: Looks more like a power/connection issue. Appreciate the second set of eyes. I'll go ahead and reseat the HBA and cables. I figured I'd ask some more experienced folks if there was a configuration issue before I took her offline.
March 18, 20251 yr Community Expert Monthly parity checks are pretty common, some do it less frequently. I assume you are aware that parity checks are not needed to maintain parity. Parity is maintained each time an array disk is written.
March 18, 20251 yr Author 1 hour ago, trurl said: Monthly parity checks are pretty common, some do it less frequently. I assume you are aware that parity checks are not needed to maintain parity. Parity is maintained each time an array disk is written. I am indeed. Learned that in the docs, but it never hurts to be sure. Appreciate it! I'll move the cron to monthly. Will definitely spare a lot of drive use. In the meantime, I've gone and reseated everything, power cables, data cables, and the HBA. Fingers crossed it all goes well. At least I know it's not the drive.
March 19, 20251 yr Author It's been over 24 hours at this point, and the disk is still live. Think it might have been a connection/wiring issue after all. Thanks a ton for helping rule out potential problems!
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.