Server crash

[chiz]

I recently decided to upgrade mobo and cpu in my unraid server with the ones that i previously used in my HTPC. I've had a couple of crashes but haven't had time to look into it properly.

 

Yesterday a 2tb drive failed, i was going to replace it anyways but was waiting until black friday. So I went and bought a new drive today, 8tb WD red. My initial plan was to first change parity in a couple of weeks, then replace the old drive, but since it failed two weeks prior to my plan I now need to do something different. I read in the wiki about the parity swap procedure and have followed all the steps. Everything is fine until it isn't, i cannot get a log but took a picture of the monitor that is attached to the server atm.

 

I've tried three times now with the same result. 

 

I am now in the process of pre-clearing the new drive just to see if the server will stay alive long enough. Any advice would be very helpful at this point.

 

Attached latest syslog if it helps.

server-syslog-20181112-2040.zip

Edited November 12, 2018 by chiz

[John_M]

Nov 12 20:25:37 Server kernel: Your BIOS is broken; DMAR reported at address 0!
Nov 12 20:25:37 Server kernel: BIOS vendor: American Megatrends Inc.; Ver: V1.2; Product Version: 1.0

I'd look to see if there's an update available.

[jonp]

In addition, while the picture is helpful to see, logs are even more vital.  You can actually get the logs to show up on the screen by typing the following command after logging in as root:

 

tail /var/log/syslog -f > /boot/syslog.txt

 

This will do two things:

 - Put the log as it's growing directly on the screen

 - Save the contents as it's written to your USB flash

 

If the server crashes again and the message appears the same on screen, shut down, remove the flash, plug it into your PC, and post the syslog.txt file you find there here on the forums so we can review.

[chiz]

3 hours ago, John_M said:

Nov 12 20:25:37 Server kernel: Your BIOS is broken; DMAR reported at address 0!
Nov 12 20:25:37 Server kernel: BIOS vendor: American Megatrends Inc.; Ver: V1.2; Product Version: 1.0

I'd look to see if there's an update available.

Thanks, been trying to update bios without success. The M-flash tool in bios finds the correct file on the usb but freezes once i press it. Tried with three different USB-keys with the same result.

 

Also tried with a couple of older versions, i.e. not older than the one I have installed, but older than the newest one on the MSI website. 

 

Decided i will move all critical files from another WD20EARS in the server before I attempt anything else, just in case i get a second crash.

[John_M]

7 minutes ago, chiz said:

Thanks, been trying to update bios without success.

I'm not very familiar with MSI motherboards. I tend to use Asus and sometimes Gigabyte, occasionally ASRock. There's usually more than one way to flash the BIOS. For example, most modern Gigabyte boards support three different ways - one from within Windows (so discard that one); one from within the BIOS (like you've already tried); and one where you make a bootable DOS USB stick (use Rufus to install FreeDOS) to flash the BIOS the old fashioned way. Asus gives you the option of going online and downloading the BIOS update from within the update tool, launched from within the BIOS. It might be worth checking to see if there's an alternative method.

 

A couple of thoughts: does the M-Flash tool require that you use a specific USB port? Do you need to load BIOS defaults before using it?

[chiz]

Some updates. Managed to prepare a bootable usb stick since the standard bios upgrade didn’t work. Before instarted the update i decided to do a memtest86, which showed that i had a faulty ram-disk. After identifying the faulty one i updated the bios which worked fine.

 

copying the parity now and it has now completed 4%, all the previous tries crashed at 1%. Looking good so far.

[John_M]

With faulty RAM all bets are off, so well done for finding it. Hopefully you're now in a good position to complete the parity swap. Post again if you need any more help.

[chiz]

Copying went fine, and also parity rebuild. But now i have a problem, the old parity, which i used to replace the broken disc according to the parity swap procedure, doesn’t seem to be included.

 

So how do i proceed from here in order to not loose any data from the broken disk?

these are my options, feels like something is missing

 

[JorgeB]

Please post the diagnostics: Tools -> Diagnostics

[chiz]

diagnostics attached

server-diagnostics-20181115-1504.zip

[JorgeB]

Check filesystem on disk6:

https://wiki.unraid.net/Check_Disk_Filesystems#Checking_and_fixing_drives_in_the_webGui

or

https://wiki.unraid.net/Check_Disk_Filesystems#Drives_formatted_with_XFS

[chiz]

Just to clarify in which order everything happened and why i want to be sure it’s correct before i format the drive.

 

1. disc 6 died a few days ago.

i wanted a bigger parity so followed this guide:

https://wiki.unraid.net/index.php/The_parity_swap_procedure

 

2. Bought a new 8tb drive and put that in.

3. After disabling disc 6, starting the array i proceeded to do the parity swap.

4. The old parity is kow disc 6, but first it gets copied to the new parity.

5. New parity copy complete. Data rebuild started i thiught, but maybe it was a parity check instead?

the old parity should have no filesystem, so that is correct, but i thought the process of rebuilding the array was complete.

 

 

[JorgeB]

Log only shows disk6 rebuild, it should also show the parity swap, so something is missing from your description, the rebuild should have been done immediately after the parity copy and without rebooting, perhaps there was some issue with the server during the rebuild and you rebooted and it would start from the beginning, in that case it might be OK.

[chiz]

Well i do have an 8tb parity now, it was 4tb before.

[JorgeB]

1 minute ago, chiz said:

Well i do have an 8tb parity now, it was 4tb before.

Yes you do, but the log doesn't show the parity swap, i.e., the server was rebooted after doing it, but it does show disk6 rebuild, and that should have been done right after parity swap, not after rebooting.

Edited November 15, 2018 by johnnie.black

[chiz]

Afaik i have not rebooted.

uptime on server is 1 day 18 hours 39mins

the parity check or rebuild whichever it did, was completed 03.46 so approx 12 hours ago, it lasted for 12 hours 37 mins. Before that the server was copying the parity.

 

just checked timestamps in forum and i wrote 4% copying on tuesday 09.30, which corresponds with uptime.

 

Edited November 15, 2018 by chiz

[JorgeB]

Sorry, you're right, I don't know how but I missed the parity copy part, just check filesystem on disk6.

[JorgeB]

P.S. there was a CRC error on disk8, just one so far so not a big deal, but if there are more in the future replace the SATA cable.

[chiz]

[JorgeB]

You need to remove the -n (no modify) flag or nothing will be done.

[chiz]

Repaired and restarted array in normal mode, everything seems to work fine now. Thank you so much for all the help!