Skip to content
View in the app

A better way to browse. Learn more.
Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.

New View the Bug / Feature Tracker

[7.1.2] server crashes

By Struck

· 569 views
  • Closed
  • Minor

I have a server that has given me some problems past few weeks with hard lockups of various degrees.

This morning, it happened again, but not a full lockup.

I could still access parts of the webui, terminal, and I saw some of the dockers still running in the active processes, but samba shares did not work.

If I opened a terminal and wanted to list the /mnt directory, the terminal froze.


After this, I tried to create the diagnostics, but it stalled halfway. And after this, the webui also did not work, but the web terminal was still awake.

Here is what the screen showed when creating the diagnostics:

Failure of the diagnostics creation

mkdir -p /boot/logs

mkdir -p '/hotbox-diagnostics-20250613-0814/system' '/hotbox-diagnostics-20250613-0814/config' '/hotbox-diagnostics-20250613-0814/logs' '/hotbox-diagnostics-20250613-0814/shares' '/hotbox-diagnostics-20250613-0814/smart' '/hotbox-diagnostics-20250613-0814/qemu' '/hotbox-diagnostics-20250613-0814/xml'

top -bn1 -o%CPU 2>/dev/null|todos >'/hotbox-diagnostics-20250613-0814/system/top.txt'

tail /boot/bz*.sha256 >> '/hotbox-diagnostics-20250613-0814/unraid-7.1.2.txt'

uptime

nproc

lscpu 2>/dev/null|todos >'/hotbox-diagnostics-20250613-0814/system/lscpu.txt'

lsscsi -vgl 2>/dev/null|todos >'/hotbox-diagnostics-20250613-0814/system/lsscsi.txt'

lspci -knn 2>/dev/null|todos >'/hotbox-diagnostics-20250613-0814/system/lspci.txt'

lspci -vv 2>/dev/null| awk -b '/ASPM/{print $0}' RS=|grep -P '(^[a-z0-9:.]+|ASPM |Disabled;|Enabled;)'|todos >'/hotbox-diagnostics-20250613-0814/system/aspm-status.txt'

lsusb 2>/dev/null|todos >'/hotbox-diagnostics-20250613-0814/system/lsusb.txt'

free -mth 2>/dev/null|todos >'/hotbox-diagnostics-20250613-0814/system/memory.txt'

lsof -Pni 2>/dev/null|todos >'/hotbox-diagnostics-20250613-0814/system/lsof.txt'

lsmod|sort 2>/dev/null|todos >'/hotbox-diagnostics-20250613-0814/system/lsmod.txt'

df -h 2>/dev/null|todos >'/hotbox-diagnostics-20250613-0814/system/df.txt'

ip -br a|awk '/^(eth|bond)[0-9]+ /{print $1}'|sort

dmidecode -qt2|awk -F: '/^ Manufacturer:/{m=$2};/^ Product Name:/{p=$2} END{print m" -"p}' 2>/dev/null|todos >'/hotbox-diagnostics-20250613-0814/system/motherboard.txt'

dmidecode -qt0 2>/dev/null|todos >>'/hotbox-diagnostics-20250613-0814/system/motherboard.txt'

cat /proc/meminfo 2>/dev/null|todos >'/hotbox-diagnostics-20250613-0814/system/meminfo.txt'

dmidecode --type 17 2>/dev/null|todos >>'/hotbox-diagnostics-20250613-0814/system/meminfo.txt'

sed -ri 's/(["\[ ])(127|10|172\.1[6-9]|172\.2[0-9]|172\.3[0-1]|192\.168)((\.[0-9]{1,3}){2,3}([/" .]|$))/\1@@@\2\3/g; s/(["\[ ][0-9]{1,3}\.)([0-9]{1,3}\.){2}([0-9]{1,3})([/" .]|$)/\1XXX.XXX.\3\4/g; s/@@@//g' '/hotbox-diagnostics-20250613-0814/system/lsof.txt'

sed -ri 's/(([0-9a-f]{1,4}:){4})(([0-9a-f]{1,4}:){3}|:)([0-9a-f]{1,4})([ .:/]|$)/\1XXXX:XXXX:XXXX:\5\6/g' '/hotbox-diagnostics-20250613-0814/system/lsof.txt'

sed -ri 's/(\[([0-9a-f]{1,4}:){4})(([0-9a-f]{1,4}:){3}|:)([0-9a-f]{1,4})(\])([ .:/]|$)/\1XXXX:XXXX:XXXX:\5\6/g' '/hotbox-diagnostics-20250613-0814/system/lsof.txt'

sed -ri 's/(([0-9a-f]{1,4}:){4})(([0-9a-f]{1,4}:){3}|:)([0-9a-f]{1,4})(\/[0-9]{1,3})([ .:/]|$)/\1XXXX:XXXX:XXXX:\5\7/g' '/hotbox-diagnostics-20250613-0814/system/lsof.txt'

ethtool 'eth0' 2>/dev/null|todos >>'/hotbox-diagnostics-20250613-0814/system/ethtool.txt'

ethtool -i 'eth0' 2>/dev/null|todos >>'/hotbox-diagnostics-20250613-0814/system/ethtool.txt'

ip -br a|todos >'/hotbox-diagnostics-20250613-0814/system/ifconfig.txt'

sed -ri 's/(["\[ ])(127|10|172\.1[6-9]|172\.2[0-9]|172\.3[0-1]|192\.168)((\.[0-9]{1,3}){2,3}([/" .]|$))/\1@@@\2\3/g; s/(["\[ ][0-9]{1,3}\.)([0-9]{1,3}\.){2}([0-9]{1,3})([/" .]|$)/\1XXX.XXX.\3\4/g; s/@@@//g' '/hotbox-diagnostics-20250613-0814/system/ifconfig.txt'

sed -ri 's/(([0-9a-f]{1,4}:){4})(([0-9a-f]{1,4}:){3}|:)([0-9a-f]{1,4})([ .:/]|$)/\1XXXX:XXXX:XXXX:\5\6/g' '/hotbox-diagnostics-20250613-0814/system/ifconfig.txt'

sed -ri 's/(\[([0-9a-f]{1,4}:){4})(([0-9a-f]{1,4}:){3}|:)([0-9a-f]{1,4})(\])([ .:/]|$)/\1XXXX:XXXX:XXXX:\5\6/g' '/hotbox-diagnostics-20250613-0814/system/ifconfig.txt'

sed -ri 's/(([0-9a-f]{1,4}:){4})(([0-9a-f]{1,4}:){3}|:)([0-9a-f]{1,4})(\/[0-9]{1,3})([ .:/]|$)/\1XXXX:XXXX:XXXX:\5\7/g' '/hotbox-diagnostics-20250613-0814/system/ifconfig.txt'

find /sys/kernel/iommu_groups/ -type l 2>/dev/null|sort -V|todos >'/hotbox-diagnostics-20250613-0814/system/iommu_groups.txt'

todos '/hotbox-diagnostics-20250613-0814/system/cmdline.txt'

echo -ne ' /boot ' >>'/hotbox-diagnostics-20250613-0814/system/folders.txt';ls -l '/boot'|todos >>'/hotbox-diagnostics-20250613-0814/system/folders.txt'

echo -ne ' /boot/config ' >>'/hotbox-diagnostics-20250613-0814/system/folders.txt';ls -l '/boot/config'|todos >>'/hotbox-diagnostics-20250613-0814/system/folders.txt'

echo -ne ' /boot/config/plugins ' >>'/hotbox-diagnostics-20250613-0814/system/folders.txt';ls -l '/boot/config/plugins'|todos >>'/hotbox-diagnostics-20250613-0814/system/folders.txt'

echo -ne ' /boot/config/firmware folder does not exist ' >>'/hotbox-diagnostics-20250613-0814/system/folders.txt'

echo -ne ' /boot/syslinux ' >>'/hotbox-diagnostics-20250613-0814/system/folders.txt';ls -l '/boot/syslinux'|todos >>'/hotbox-diagnostics-20250613-0814/system/folders.txt'

echo -ne ' /var/log ' >>'/hotbox-diagnostics-20250613-0814/system/folders.txt';ls -l '/var/log'|todos >>'/hotbox-diagnostics-20250613-0814/system/folders.txt'

echo -ne ' /var/log/plugins ' >>'/hotbox-diagnostics-20250613-0814/system/folders.txt';ls -l '/var/log/plugins'|todos >>'/hotbox-diagnostics-20250613-0814/system/folders.txt'

echo -ne ' /boot/extra ' >>'/hotbox-diagnostics-20250613-0814/system/folders.txt';ls -l '/boot/extra'|todos >>'/hotbox-diagnostics-20250613-0814/system/folders.txt'

echo -ne ' /var/log/packages ' >>'/hotbox-diagnostics-20250613-0814/system/folders.txt';ls -l '/var/log/packages'|todos >>'/hotbox-diagnostics-20250613-0814/system/folders.txt'

echo -ne ' /var/lib/pkgtools/packages ' >>'/hotbox-diagnostics-20250613-0814/system/folders.txt';ls -l '/var/lib/pkgtools/packages'|todos >>'/hotbox-diagnostics-20250613-0814/system/folders.txt'

echo -ne ' /tmp ' >>'/hotbox-diagnostics-20250613-0814/system/folders.txt';ls -l '/tmp'|todos >>'/hotbox-diagnostics-20250613-0814/system/folders.txt'

cp -- /boot/config/*.cfg '/hotbox-diagnostics-20250613-0814/config'

cp -- /boot/config/*.conf '/hotbox-diagnostics-20250613-0814/config'

cp -- /boot/config/*.dat '/hotbox-diagnostics-20250613-0814/config'

cp /boot/config/go '/hotbox-diagnostics-20250613-0814/config/go.txt'

sed -i -e '/password/c ***line removed***' -e '/user/c ***line removed***' -e '/pass/c ***line removed***' '/hotbox-diagnostics-20250613-0814/config/go.txt'

find '/hotbox-diagnostics-20250613-0814/config' -name '*.cfg' -exec sed -ri 's/^((disk|flash)(Read|Write)List.*=")[^"]+/\1.../' {} \;

sed -ri 's/(["\[ ])(127|10|172\.1[6-9]|172\.2[0-9]|172\.3[0-1]|192\.168)((\.[0-9]{1,3}){2,3}([/" .]|$))/\1@@@\2\3/g; s/(["\[ ][0-9]{1,3}\.)([0-9]{1,3}\.){2}([0-9]{1,3})([/" .]|$)/\1XXX.XXX.\3\4/g; s/@@@//g' '/hotbox-diagnostics-20250613-0814/config/network.cfg'

sed -ri 's/(([0-9a-f]{1,4}:){4})(([0-9a-f]{1,4}:){3}|:)([0-9a-f]{1,4})([ .:/]|$)/\1XXXX:XXXX:XXXX:\5\6/g' '/hotbox-diagnostics-20250613-0814/config/network.cfg'

sed -ri 's/(\[([0-9a-f]{1,4}:){4})(([0-9a-f]{1,4}:){3}|:)([0-9a-f]{1,4})(\])([ .:/]|$)/\1XXXX:XXXX:XXXX:\5\6/g' '/hotbox-diagnostics-20250613-0814/config/network.cfg'

sed -ri 's/(([0-9a-f]{1,4}:){4})(([0-9a-f]{1,4}:){3}|:)([0-9a-f]{1,4})(\/[0-9]{1,3})([ .:/]|$)/\1XXXX:XXXX:XXXX:\5\7/g' '/hotbox-diagnostics-20250613-0814/config/network.cfg'

/usr/local/emhttp/webGui/scripts/show_interfaces ip|tr ',' ' ' >'/hotbox-diagnostics-20250613-0814/config/listen.txt'

/usr/local/emhttp/webGui/scripts/error_interfaces|sed 's///' >>'/hotbox-diagnostics-20250613-0814/config/listen.txt'

sed -ri 's/(["\[ ])(127|10|172\.1[6-9]|172\.2[0-9]|172\.3[0-1]|192\.168)((\.[0-9]{1,3}){2,3}([/" .]|$))/\1@@@\2\3/g; s/(["\[ ][0-9]{1,3}\.)([0-9]{1,3}\.){2}([0-9]{1,3})([/" .]|$)/\1XXX.XXX.\3\4/g; s/@@@//g' '/hotbox-diagnostics-20250613-0814/config/listen.txt'

sed -ri 's/(([0-9a-f]{1,4}:){4})(([0-9a-f]{1,4}:){3}|:)([0-9a-f]{1,4})([ .:/]|$)/\1XXXX:XXXX:XXXX:\5\6/g' '/hotbox-diagnostics-20250613-0814/config/listen.txt'

sed -ri 's/(\[([0-9a-f]{1,4}:){4})(([0-9a-f]{1,4}:){3}|:)([0-9a-f]{1,4})(\])([ .:/]|$)/\1XXXX:XXXX:XXXX:\5\6/g' '/hotbox-diagnostics-20250613-0814/config/listen.txt'

sed -ri 's/(([0-9a-f]{1,4}:){4})(([0-9a-f]{1,4}:){3}|:)([0-9a-f]{1,4})(\/[0-9]{1,3})([ .:/]|$)/\1XXXX:XXXX:XXXX:\5\7/g' '/hotbox-diagnostics-20250613-0814/config/listen.txt'

sed -ri 's/^(share(Comment|ReadList|WriteList)=")[^"]+/\1.../' '/hotbox-diagnostics-20250613-0814/shares/A---t.cfg'

Before rebooting, I used the web terminal to copy the incomplete diagnostics dump to a persistent location, together with the /var/syslog, as it is not included in the diagnostics dump because the dump stalled before reaching that step.

Diagnostics.zip

I rebooted and extracted the diagnostics shown above.

Looking at the syslog, it complains about a kernel bug in btrfs

Jun 13 06:01:24 Hotbox kernel: kernel BUG at fs/btrfs/ctree.c:4107!
Jun 13 06:01:24 Hotbox kernel: Oops: invalid opcode: 0000 [#1] PREEMPT SMP PTI

It seems to have happened right when the docker auto update is running at 6am.

Once rebooted I setup the external syslog server to hopefully catch more information when it happens again.

Only 2 hours later, it stopped working again, but this time, neither webui nor SMB was working. The tower still responded to pings, and had an IP.

syslog.zip

I had to hard reboot the tower.

This second syslog points towards some memory allocation issues, but I am not too familiar with it.

Jun 13 10:11:07 Hotbox kernel: Trying to vfree() bad address (00000000846bd043)
Jun 13 10:11:07 Hotbox kernel: WARNING: CPU: 6 PID: 147932 at mm/vmalloc.c:3226 remove_vm_area+0x3b/0x70

Any idea what could be wrong, or how to fix it.
Maybe I am facing hardware issues?

Bonus information:

I recently added an RTX 5070 to the server, with the 570.144 open-source driver.
The card is used in the folding@home docker. I have power limited the card to 175W.

The addition has put more heat and stress on the ageing hardware, but I am not sure if it's the root cause.

    •

    Account

    Navigation

    Search

    Search

    Configure browser push notifications
    Chrome (Android)
    1. Tap the lock icon next to the address bar.
    2. Tap Permissions → Notifications.
    3. Adjust your preference.
    Chrome (Desktop)
    1. Click the padlock icon in the address bar.
    2. Select Site settings.
    3. Find Notifications and adjust your preference.