Only can boot into safe mode and crashes instantly when array starts up

[SKAntoniou]

Replaced motherboard, took ages to get delivered. can start without gpu in it but still same issue as before. still returning old motherboard because the whole can't start without gpu in wasn't good.

[SKAntoniou]

any suggestions would be great, really annoying at this point, would rather not lose all the stuff on it as it is the opposite reason of why I built this. 

[JorgeB]

If there's still nothing logged it's not easy to diagnose remotely, you basically need to try and simply things as much as possible to try and find the culprit, e.g., do a new array with just one data disk for example and see if it starts (note that doing this will invalid pariyy, so it will need to be re-synced later).

[SKAntoniou]

thanks, I just started it in maintenance mode, and was looking at drive logs, spun down a disc and then spun it up and crashed soon after. I am not getting instant crashes though when I start normally, takes like 20 seconds now until crash, some progress. 

Also all my plugins are gone, would that be a sign of anything? any chance my usb drive is failing? doesn't show that on the webui but who knows what's happening. 

I just installed community apps and common problems again to see if they show anything but sadly not. will keep trying things, thanks. 

[SKAntoniou]

if I use a different drive to start an array to test if it works, can I later bring all the drives back and restore my original array?

[JorgeB]

Yes, but you'll need to re-sync parity.

[SKAntoniou]

thanks

[SKAntoniou]

So that actually worked pretty well, Started with one drive formatted it, (one that was never in the array). worked

Then unplugged that drive, moved to putting old drives back in worked out well but dumb me didn't take a picture of what drive was where before starting this process. 

So out of my 7 drives ( 2 parity) 4 are in perfectly but 3 drives show as unmountable. 2 I am guessing are because they were parity drives but the one is the weird one. The drive log I posted before about disk 4 shows what drive was meant to be in the array but sadly is showing unmountable, does this mean I lost everything on that drive? any option to save that?

 

[SKAntoniou]

the log showed this 

Apr 7 16:09:16 unBaud kernel: XFS (sdh1): Corruption warning: Metadata has LSN (2:769339) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve.
Apr 7 16:09:16 unBaud kernel: XFS (sdh1): log mount/recovery failed: error -22
Apr 7 16:09:16 unBaud kernel: XFS (sdh1): log mount failed
Apr 7 16:09:16 unBaud unassigned.devices: Mount of '/dev/sdh1' failed. Error message: mount: /mnt/disks/ST4000DM004-2CV104_ZFN2Z9C3: wrong fs type, bad option, bad superblock on /dev/sdh1, missing codepage or helper program, or other error.

[SKAntoniou]

Oh my god, everything works now, xfs_repair -v /dev/sdh1 fixed it.

thank you for continued support JorgeB, literally couldn't do it without you. now just going to add back parity disk and everything should be okay. weird had to make a new array then xfs repair the drive after it to fix it but all is sorted, thank you. 

[SKAntoniou]

ok was close, added parity drives and now it crashes when I try parity sync it, any clues?

 

[trurl]

50 minutes ago, SKAntoniou said:

dumb me didn't take a picture of what drive was where

For future reference, you could get your disk assignments by examining the diagnostics you posted earlier.

[JorgeB]

26 minutes ago, SKAntoniou said:

ok was close, added parity drives and now it crashes when I try parity sync it, any clues?

This suggests a hardware issue, like bad PSU, overheating CPU, etc.

[SKAntoniou]

3 minutes ago, trurl said:

For future reference, you could get your disk assignments by examining the diagnostics you posted earlier.

thanks il try remember that.

[SKAntoniou]

1 minute ago, JorgeB said:

This suggests a hardware issue, like bad PSU, overheating CPU, etc.

oh alright, that sucks. The power supply with this little stuff in the build is a bit overkill, don't know if that is broken. Cpu might be a pain to check but il try run some stress tests on it, if not under clock it a bit.

[SKAntoniou]

Ok so got new power supply and put it in, stopped crashes when I add parity drives back to array. 

Only problem now is since swapping power supply 2 drives are showing up a unmountable. One drive was showing up as unmountable but since adding parity drives another one decided it was unmountable. One drive is showing up with a Red Cross (the new one that won't mount) other is green but isn't mountable. Any suggestions?

[JorgeB]

Please post current diags.

[SKAntoniou]

6 minutes ago, JorgeB said:

Please post current diags.

thanks

unbaud-diagnostics-20210412-1618.zip

[JorgeB]

You have 3 invalid disks, dual parity can only have 2, assuming the actual disable disk is fine since it looks healthy do a new config and re-sync parity, this should take care of the unmountable disk1, still seeing a lot of NMI events in the log though, you'll then need to check filesystem on disk2.

[SKAntoniou]

thanks for the tip, I tried that but now I am back to crashing. I don't think this is a hardware problem, I have changed my PSU and motherboard, Tested my Ram on memtest for days, under clocked my cpu and swapped the cables for my sas card with motherboard ones and nothing is working.

From the brief time it wasn't crashing, I worked out it is mainly crashing trying to mount hard drives which end up being unmountable (or at least thats my best guess). It is really annoying. I have probably lost all my data right? 

[SKAntoniou]

ordered a new sas card because fuck it, already wasted a bunch of money, will update when this comes and doesn't work.

[JorgeB]

37 minutes ago, JorgeB said:

still seeing a lot of NMI events in the log though

This usually the indication of a hardware issue, or hardware that is not working with Unraid, do you have a different PC (or board + CPU) you could try with? Data should still be all there.

[SKAntoniou]

19 minutes ago, JorgeB said:

This usually the indication of a hardware issue, or hardware that is not working with Unraid, do you have a different PC (or board + CPU) you could try with? Data should still be all there.

nice, yeh sadly don't have another cpu + motherboard, going to be moving to a thread ripper build in the future so might buy them soon so I can just try get it to work.

[SKAntoniou]

Sorry been a long time, Just want to give an update.

 

So it was a hardware issue as you guessed. Replaced Motherboard, CPU, RAM and PSU and used that stuff for a normal computer. 

 

Disk 2 was still acting weird so did this (as suggested by you guys): 

Booted in Maintenance mode,

used xfs_repair -v /dev/md2,

log was inconsistent 

used xfs_repair -L /dev/md2,

fixed

used xfs_repair -v /dev/md2 to check

all good

stopped array and started normally and everything is back to normal

 

Thanks for the help. Was very useful.

 

[trurl]

On 6/12/2023 at 12:12 PM, SKAntoniou said:

everything is back to normal

Check lost+found share for files repair couldn't figure out.