Unraid is having to mount ALL the disk, on every startup...

[GeoffH]

Hi, I had a bit of an issue the other day, when my server didn't turn on, and it my efforts to get it back, I have tried several different motherboards, but due to some annoying compatibility issues, the Adaptec raid controllers would stall during the Unraid "posting" process.

So I think with all the false starts & forced shutdowns the system is not happy.

So when I turn this Unraid server on, it boots up reasonably fast, then once I've logged into Unraid, it takes 15 - 20 minutes to mount all 28 disks, but then went I try and access it form a remote PC, it takes many minutes to open the folders, and it basically won't copy anything, or is it I just haven't got the patience to wait for hours....

Then, when I shut it down, it takes a further 10 - 15 minutes, to do that.

So I'd like to know if I put the hard drives in another Unraid server, and try and access them from there.

Regards

[JorgeB]

Please post the diagnostics after array start.

[GeoffH]

Thanks for the reply, and I will post the diagnostics for you tomorrow (my time).

I got cut short with my original post, I had to go out suddenly.

So about all I have been able to do it a disk repair on EVERY disk, using the default command -n.

Hopefully I will be able to fix this, as it's an 80Tb array, and it's 90% full, so I'd hate to lose it, but I should be able to transfer the disks to another server.

Regards.

[GeoffH]

OK, I have gathered some diagnostic information (doesn't mean anything to me)..

Hopefully you will be able to see what might be causing this, it takes forever to mount all the disks, EVERY startup, then I can't copy off them, but I can "see" them from remote pc's, then it takes well over 1000 seconds (or whatever the countdown is at the shutdown screen).

 

So .zip -0334 is the one I "forced", -0427 seemed to be generated some other way?

Cheers

tower-diagnostics-20230316-0334.zip tower-diagnostics-20230316-0427.zip

Edited March 16, 2023 by GeoffH

[itimpi]

16 hours ago, GeoffH said:

So about all I have been able to do it a disk repair on EVERY disk, using the default command -n

That is NOT a repair - it is only a check that will leave the file system on the drive unchanged.  You have to remove the -n (no modify) option for it to be a repair.   However while running a repair is a good idea I do not think it is the answer to the slow mounting.

 

is it still happening after the system has had a clean shutdown on subsequent boots?  

[GeoffH]

28 minutes ago, itimpi said:

That is NOT a repair - it is only a check that will leave the file system on the drive unchanged.  You have to remove the -n (no modify) option for it to be a repair.   However while running a repair is a good idea I do not think it is the answer to the slow mounting.

 

is it still happening after the system has had a clean shutdown on subsequent boots?  

Hi itimpi, 

 

OK, thanks for the info, I might have to do the "repair" properly, this time

 

TBH, I don't think I'm getting to a stage to have a clean shutdown, like I said it takes SO, SO long to shut down, then it just shuts down...I would have thought it might be OK on the next start, but the several times this has now happened, it still takes ages to mount every disk. 

[JorgeB]

Nothings obvious on the log, the Adaptec controllers are the biggest unknown to me, I guess you don't have different controllers you could try with?

[itimpi]

4 hours ago, GeoffH said:

Hi itimpi, 

 

OK, thanks for the info, I might have to do the "repair" properly, this time

 

TBH, I don't think I'm getting to a stage to have a clean shutdown, like I said it takes SO, SO long to shut down, then it just shuts down...I would have thought it might be OK on the next start, but the several times this has now happened, it still takes ages to mount every disk. 

Do you have the syslog server enabled?    If not I suggest you do so with the mirror to flash option enabled so that we can get a syslog that covers both the closedown and subsequent startup sequence to see if that shows anything useful.

[GeoffH]

48 minutes ago, JorgeB said:

Nothings obvious on the log, the Adaptec controllers are the biggest unknown to me, I guess you don't have different controllers you could try with?

Hi JorgeB,

Well, that's interesting, but annoying...

The Adaptec Raid controllers are set up in JBOD, so each hdd is a separate JBOD disk.

I don't think I have similar cards to swap with, the other older Adaptec card, each hdd is set as a single RAID volume.

If I swap from one to another I will probably have to re initialise disk, which will wipe the data.

I have a dilemma

[GeoffH]

25 minutes ago, itimpi said:

Do you have the syslog server enabled?    If not I suggest you do so with the mirror to flash option enabled so that we can get a syslog that covers both the closedown and subsequent startup sequence to see if that shows anything useful.

Hi, no I don't think so, I will check it out tomorrow (my time).

So what's the next step ??

I guess I should at least do another disk check, without the -n.

Thanks.

Edited March 16, 2023 by GeoffH

[GeoffH]

Well, another day, another major problem

I start this server up to do some more checking, and when I get into the Main page, ALL the disks are missing !!!!!

Some change to the configuration, and I haven't done anything.

So can I get this back by going into Configuration, retaining all the settings, and apply ??

I still think I'm going to lose ALL this...that will be a VERY sad day

 

Restarted after a hour or so, and this time it recognised ALL the drives, and now I am doing a xfs_repair without the "-n".

 

Do I need to post my syslog ??

 

Thanks

Edited March 17, 2023 by GeoffH
more info

[JorgeB]

I would recommend using different controllers.

[GeoffH]

1 hour ago, JorgeB said:

I would recommend using different controllers.

Yeah, that might all well and good, but I have been using these controllers for years, with no issue, I doubt that it's the controllers, tho.

I can't just swap the hdd's to another controller, due to the way I have had to set them up for Unraid to recognise them.

As I said earlier, some controllers I have to set the hdd's as JBOD disks, and on others, each disk has to be a single raid 0 drive.

I'm going to do some tests on another empty Unraid server to see how it handles swapping hdd's from another server.

Another issue I have is the faulty server has 28 drives, but all my other server boxes only hold 16, so I'm hoping that if I swap the controller from the faulty server to another server, then also swap only 16 drives, that the 16 drives will be recognised, with minimal data loss. If that works I can then swap the remaining drives, and copy the data off them.

Just not sure what to do with the "new configuration" options.

Regards.

[JorgeB]

24 minutes ago, GeoffH said:

As I said earlier, some controllers I have to set the hdd's as JBOD disks, and on others, each disk has to be a single raid 0 drive.

That's one of the reasons those controllers are not recommended.

 

25 minutes ago, GeoffH said:

so I'm hoping that if I swap the controller from the faulty server to another server, then also swap only 16 drives, that the 16 drives will be recognised,

You can do that, using the same controller for those raid0 drives, parity won't be valid of course, but it will import any working data drive assigned.

 

 

 

[GeoffH]

Hi, well, yet another day, and SUCCESS, this time.

I have swapped the Adaptec controller ASR-51646, over to another Unraid server, used the USB stick from that server (not the faulty one), installed 16 of the 28 hdd's, booted up, logged in, assigned all 16 drives in alphabetical order, started the array, and it mounted the hdd's, almost instantly (like it should), and I am now copying off the stuff I wanted last week.

It will be interesting (when I get to it), what happens between disks 16 & 17, meaning that disks 17 thru 28 are still in the faulty server.

 

So what could be the problem, the Unraid USB stick is faulty or corrupted, or the replacement motherboard doesn't like something

 

So I would like to thankyou, both JorgeB & itimpi for your assistance with this strange behaviour

 

Regards

[JorgeB]

7 hours ago, GeoffH said:

the Unraid USB stick is faulty or corrupted,

I would guess this is unlikely based on the problem, but let us know what you find.

[GeoffH]

12 minutes ago, JorgeB said:

I would guess this is unlikely based on the problem, but let us know what you find.

So can I safely create a new fresh USB stick, and use it on that faulty server, and just re-assign all the drives, and then just set it up the way I had it ??

 

BTW, I don't use the Parity option.

 

I'm thinking of putting the balance of the hdd's from the faulty server into another Unraid box I have, and just do the same thing, and copy them somewhere else.

[JorgeB]

You, just assign the disks again.