Dear *,

 

I really need a hint into the right directions.
I am running out of ideas.
I have a HP MicroServer Gen10 with a AMD Opteron™ X3421 and 32GBs of RAM.
5x 2.5 HDDs and one SSD with a riser card is installed.

Regardless what I do - the server stops for no reason at some point. I cannot see why. I activated the syslog and checked if there might be something interesting, but as far as I can see - nothing really susiciously. It happens with v6.12.4 up to v6.12.6.

- Disabled mover - no diference
- Removed second m2 SSD - no difference
- Stopped cache features (all goes directly to destination disk) - no difference
- VM support on/off - no difference
- Docker reduced to a tiny minimum of containers - no difference
- One or both NICs connected no difference
- SYSlog does not show a kernel panic or anything when the machine stops.
I have the file where you can see the point it stops and the point where I hard stop the machine and start it up again

Found this Forums post from someone with mostly same hardware:

Sadly no really important things.

This is what I found in the syslog when I do a search.... I will check IOMMU support in the BIOS next time I reboot the device (or if it stops next time)
 

[Firmware Bug]: HEST: Table contents overflow for hardware error source: 2.
ERST: Error Record Serialization Table (ERST) support is initialized.
kfd kfd: amdgpu: error getting iommu info. is the iommu enabled?
kfd kfd: amdgpu: Error initializing iommuv2
kfd kfd: amdgpu: device 1002:9874 NOT added due to errors

I have attached a VGA Monitor (HDMI goes black when unraid boots up) maybe I see something there.
Maybe i go for a serial usb converter and try to get some more info (if possible)

BUT apart from this, can someone have a quick look at the diagnostic logs - or point me to something which might shed some more light onto this.
I really love Unraid, but it stops out of the blue, sometimes after a few hours, sometimes after 2-4 days...

Help me Unraid Kenobis - you're my only hope.

myunraid-diagnostics-20231202-1910.zip

Enable the syslog server and post that after a crash.

 

Hi JorgeB

first THANKS for looking into this!
Here some updates better known as "what I have now done to further track down the issue":

- Changed macvlan to ipvlan
Just in case, because some ppl report problems and unresponsive systems
- Enabled IOMMU in BIOS
Since I had that IOMMU error on AMD GPU
- Disabled Network Stacking
In BIOS

- Updated the BIOS to absolute latest state
https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_8374214d9a3a4392878c9bba59&tab=releaseNotes

 

Syslog is enabled to log to my array.
Attached the latest diagnostics plus the syslog file

Means the syslog now contains a few stops and restarts and I tried to change things, update to latest states.
I begin to think that the MicroServer Gen10 was not the best choice... maybe Gen8 would the better decision


Additional INFO
I mentioned that I have attached a monitor to maybe see some crash info - but unfortunately no - because monitor goes into power safe and the system crashes hard so a USB keyboard was not able to wake the system up to wake the monitor connection again

 

syslog-myunraid.log myunraid-diagnostics-20231203-1544.zip

Edited December 3, 20232 yr by clyde

38 minutes ago, clyde said:

- Changed macvlan to ipvlan

When was this done and did you reboot after changing?

 

Dec  3 08:37:06 Anubis kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Dec  3 08:37:06 Anubis kernel: ? _raw_spin_unlock+0x14/0x29
Dec  3 08:37:06 Anubis kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

These are recent

WHEN:
iirc after that because of some threads I found.

DID YOU REBOOT
Yes because machine stopped again after that change.

You should not be getting macvlan call traces with ipvlan, post the output of:

docker network ls

 

docker network ls
NETWORK ID     NAME         DRIVER    SCOPE
3d24d0daa2b1   br0          ipvlan    local
e736a7064171   bridge       bridge    local
d8b2bb19546c   mynet        bridge    local
fba54f09fdbb   host         host      local
1c7e099ecfb9   none         null      local

Current update is 17h - and parity check started (currently 7x%.
I am not entirely sure, but I think somewhere close to the end of the parity check the system went into unresponsive state, so fingers crossed I will update this post asap.

Still thank you for your help JorgeB.

 

Edited December 4, 20232 yr by clyde

Array Parity Check passed - system still online.
 

Dec  4 14:48:22 MyUnraid kernel: md: sync done. time=47901sec
Dec  4 14:48:22 MyUnraid kernel: md: recovery thread: exit status: 0

 

Still, fingers-crossed, since I just guessed that the parity check was always in the end the death of the os.
It might be something else and this means I need to just wait for the next few days. Sometimes the system stopped after 2-3 or 4 days.

I will update this thread accordingly.

 

Not a bad start, if the machine runs for 7 days in a row - I would consider this as a success.
Will update in 3 days.

 

 

Can be closed. So it was maybe macvlan change to ipvlan. Or the IOMMU change.
But I would bet on macvlan to ipvlan change right now - NIC is iirc a broadcom NetXtreme BCM5720NIC.

Just as reference if anybody maybe expierence same issues.
Thx for help JorgeB!