[SOLVED] new drive stuck in formatting, "array must be started" error

[nuggolips]

Hello, something funky is happening on my server. Yesterday I updated to 6.9.2 from 6.8.3 and everything seemed to be as it should, once I got up and running. I checked the existing array, docker seemed to run fine, etc.

 

I added a disk to the array and left it to clear. Clearing completed overnight and this morning I formatted the disk. Normally this only takes a few seconds but it's been stuck on "formatting" for an hour or more. I haven't stopped the array and all existing data is accessible, dockers running, etc. However, I cannot cancel the formatting operation, can't access the system logs from the UI, and I now have an error on the docker tab (see attachment). 

 

I'm curious what others think I should do as a next step. I can let it go and see if formatting eventually completes, but since it normally only takes a few seconds I'm not sure how long to wait.

[nuggolips]

I forgot to mention, as part of the work I did yesterday I also added a HBA card (Dell H310) to the system which is serving (2) new disks as well as my cache drive.

 

The current config is as follows:

 

Motherboard SATA:

Parity, Disk 1-4

HBA Card:

Cache, Disk 5-6

 

Disk 5 is the one stuck in formatting. Disk 6 is currently running a pre-clear and that process is continuing without issue (according to the UI). I checked and have no issues reading/writing to the cache drive, so I don't think there's any cable issues at play.

 

I guess I will leave it alone until Disk 6 preclear finishes, and if the other disk is still stuck I will restart the server.

[JorgeB]

See if you can get the diags by typing "diagnostics" on the console.

[nuggolips]

I can issue the command, and it returns "Starting diagnostics collection..." but appears to be stuck there (it's been about 3 minutes so far). I'll leave it for a while and post anything that comes up.

[Squid]

worst case scenario if it never completes,

cp /var/log/syslog /boot/syslog

and upload the syslog file that'll now be on the flash drive

[nuggolips]

Here's the syslog from the above process

syslog

[itimpi]

I think it is quite possible that a running pre-clear can cause any other array operation that normally finishes with a ‘sync’ operation (that I expect format does) to hang until the pre-clear either finishes or is stopped.   Could be wrong about that as it is just a theory  

[JorgeB]

32 minutes ago, itimpi said:

I think it is quite possible that a running pre-clear can cause any other array operation that normally finishes with a ‘sync’ operation (that I expect format does) to hang until the pre-clear either finishes or is stopped.

 

Very possible, wait for the pre-clear to finish and the disk should finish formatting.

[nuggolips]

Will do, thanks for the insight! I'll report back to close the loop on this.

[nuggolips]

Update: Solved.

 

As soon as the active step in the preclear process (which was the zeroing out step) finished, the other drive finished format and other errors (Docker tab message, unable to access logs) disappeared.

 

So it would appear that was indeed the issue, and it was able to resolve itself even before the entire pre-clear operation completed; just the current process.