Array stuck at mounting disks

[napalmd]

So I was performing a replacement and upgrade of drives.  No drive had errors.

Before: 4tb parity and 6x2tb data drives

replaced the parity with 8tb wd drive no problems.

Then replaced drive 1 with another 8tb wd, this one took about 30 minutes to mount after I assigned it, then the parity sync finished without problems.

Today I replaced data drive 2 with a 8tb seagate drive, assigned it, and it stayed at mounting drives, for more than an hour, and monitoring the server with top comand didn't seem to be any activity.

I tried the comand reboot and shutdown but it didnt work. the putty session closed but it stayed on, and could login again. the web interface at this point did not respond any more.

then I tried this instructions to reboot https://wiki.unraid.net/Console#To_cleanly_Stop_the_array_from_the_command_line

and this time it took ssh out because I couldn't login anymore, but still did not shutdown, so I forced shutdown on the power button.

so now after reboot the webinterface does not load, and it seems to be stuck at mounting disk2:

 

Quote

root@Unraid:/var/log# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           128M  1.8M  127M   2% /var/log
/dev/sda1        15G   74M   15G   1% /boot
/dev/md1        7.3T  6.8T  584G  93% /mnt/disk1
/dev/md2        1.9T  1.3T  622G  67% /mnt/disk2

I can still acess files on disk 1 and 2, in fact I needed to copy some of them with rsync and it was really slow like 2MB/s

 

dmesg.txt syslog.txt

[JorgeB]

Disk2 looks to be correctly mounted, but it didn't continue to mount the other disks, since you're on a very old release upgrade to latest Unraid, try booting again in normal or safe mode, and if still issues pleas post the complete diagnostics: Tools -> Diagnostics.

[napalmd]

Thanks for the answer. before this I decided to do a quick smart test, but after 30 minutes is still at 90% remaining. does this mean the drive is busy doing something? how can I know if it is doing something?

[JorgeB]

17 minutes ago, napalmd said:

but after 30 minutes is still at 90% remaining. does this mean the drive is busy doing something?

Possibly, try to get the full diags by typing "diagnostics" on the console, it shows if there's any array disk activity.

[napalmd]

the command showed no output but it created this file. hope this helps. thank you.

unraid-diagnostics-20200728-1837.zip

[JorgeB]

Unfortunately it's incomplete and missing both files that show disk loads, does the server have a disk activity led? IIRC reiserfs can take a long time to mount after an unclean shutdown, but 2 hours seems too long.

[napalmd]

sorry it does not have activity leds, and I am working on it remotely because it is 30Km away.  I will only go to the location on Thursday...

but what does unraid do after a drive is added/replaced? does it have to do a full format or something? this is a seagate archive, they are a little slow but I read that many people use them without problem on unraid...

I can ask for someone to force shutdown the server and plug the pendrive on a windows machine so I could update it, if it is the best option now.

thanks

[JorgeB]

6 minutes ago, napalmd said:

but what does unraid do after a drive is added/replaced? does it have to do a full format or something?

It does a rebuild, but only after mounting all the disks, either the array is really stuck or it's still mounting the array disks, wait a few more hours then post a new syslog so we can see if it advanced.

[JorgeB]

33 minutes ago, johnnie.black said:

IIRC reiserfs can take a long time to mount after an unclean shutdown

Apparently I recall correctly, found this:

https://forums.unraid.net/topic/90127-dell-h310lsi-9240-8i-with-8tb-hdd/?do=findComment&comment=836709

 

 

[napalmd]

2 minutes ago, johnnie.black said:

Apparently I recall correctly, found this:

https://forums.unraid.net/topic/90127-dell-h310lsi-9240-8i-with-8tb-hdd/?do=findComment&comment=836709

 

 

ok. so I will wait a few more hours, but it is now I think 6 hours in.

Since this is a replacement drive can't I just format it xfs and do a rebuild? it automatically chose reiserfs, maybe because old drive was using it, but the default type in the conig is xfs.

[JorgeB]

2 minutes ago, napalmd said:

Since this is a replacement drive can't I just format it xfs and do a rebuild?

No, rebuild is always in the original filesystem, though you should convert asap all those reiserfs drives to xfs, reiser is not recommended since v6 was released.

[napalmd]

well, after about 8 hours it finally came alive! yay!

the smart short test is still 90% remaining... it might be waiting for the drive to became idle?

But the values 195 Hardware_ECC_Recovered and 240 Head_Flying_Hours are increasing. could be a bad cable?

I'll replace the cable anyway next time, I will have to replace another drive after this parity sync.

Thanks a lot for the help.

 

[JorgeB]

9 hours ago, napalmd said:

it might be waiting for the drive to became idle?

Yes.

 

9 hours ago, napalmd said:

But the values 195 Hardware_ECC_Recovered and 240 Head_Flying_Hours are increasing. could be a bad cable?

Those attributes are not cable related, and likely no indication of any issue, but if you want post the diags once the SMART test and rebuild finish.