Jump to content
We're Hiring! Full Stack Developer ×

Server becomes unresponsive every 1-2 days


Go to solution Solved by JorgeB,

Recommended Posts

Hi, I'm having an issue where my server becomes unresponsive every few days. When it occurs, all of my Dockers, SMB shares, web UI and SSH are unavailable, so usually my only option is to cut power and do a hard reset. I haven't been able to narrow down what's causing the issue. The things I've tried so far are:

  • Recreated my docker image, since initially I had errors like "BTRFS error (device loop2): block=15655632896 write time tree block corruption detected". This seems to have resolved those errors, but the server would still shut off every few days
  • Changed my cache from a btrfs pool to single-drive xfs, as suggested here
  • Changed one of my docker networks (not the default network) from macvlan to ipvlan, as suggested here

 

According to my Grafana dashboard, the server crashed around 2024-01-19 16:18:00 and I don't see any syslog entries near that time. Can anyone help me figure out what the issue is?

unraidnas-diagnostics-20240119-1844.zip syslog-4.log

Link to comment

Thanks for the suggestion. I just ran memtest and it passed with 0 errors.

 

There is another possible hint though - I plugged in a monitor hoping that there would be some extra info that's not in the syslog. When I was rebooting to access memtest the server became unresponsive again, and it was stuck on unmounting disk1 and cache with "target is busy". I still couldn't SSH at that point to see what was using the disks though.

PXL_20240120_041847853.jpg

Link to comment
  • Solution
Jan 17 21:05:45 UnraidNAS kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Jan 17 21:05:45 UnraidNAS kernel: ? _raw_spin_unlock+0x14/0x29
Jan 17 21:05:45 UnraidNAS kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

Link to comment

Hmm my docker settings are already configured to use ipvlan. I'm not sure why there are still macvlan calls in the logs

 

Do you know what would be causing these warnings? They seem to appear whenever the parity check runs, and the parity check immediately goes to "aborted" status.

 

```

Jan 20 20:57:12 UnraidNAS kernel: WARNING: CPU: 5 PID: 7842 at kernel/exit.c:814 do_exit+0x87/0x923

Jan 20 20:57:12 UnraidNAS dnsmasq[19468]: read /etc/hosts - 3 names

Jan 20 20:57:12 UnraidNAS dnsmasq[19468]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 names

Jan 20 20:57:12 UnraidNAS dnsmasq-dhcp[19468]: read /var/lib/libvirt/dnsmasq/default.hostsfile

Jan 20 20:57:12 UnraidNAS kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap tun wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha macvlan veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm kvm drm_display_helper btusb crct10dif_pclmul drm_kms_helper btrtl crc32_pclmul crc32c_intel btbcm ghash_clmulni_intel cp210x btintel sha512_ssse3 sha256_ssse3 bluetooth

Jan 20 20:57:12 UnraidNAS kernel: mei_hdcp mei_pxp drm gigabyte_wmi wmi_bmof usbserial input_leds joydev sha1_ssse3 aesni_intel ecdh_generic led_class ecc crypto_simd cryptd rapl intel_cstate intel_gtt mei_me i2c_i801 agpgart i2c_smbus ahci r8169 intel_uncore syscopyarea i2c_core mei libahci realtek sysfillrect sysimgblt thermal fb_sys_fops video fan tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_tad acpi_pad button unix

Jan 20 20:57:12 UnraidNAS kernel: CPU: 5 PID: 7842 Comm: unraidd0 Tainted: P      D W  O       6.1.64-Unraid #1

```

Link to comment

Yes, I've rebooted. Although I think the default network was always set to ipvlan. After running `docker network ls`, I found that the network I thought I had switched to ipvlan was still using macvlan. I had to delete and re-create the network for the changes to apply. Here are my updated networks, in case anything else looks off:

NETWORK ID     NAME                           DRIVER    SCOPE
af8ab9c8b30a   bridge                         bridge    local
8d67e34f2e80   calibre-web_default            bridge    local
435083f671a2   calibre_default                bridge    local
b3756318c8a7   homeassistant_static-network   bridge    local
a4b7252f41e6   homepage_default               bridge    local
c8db19f9bd6e   host                           host      local
cbf192c48919   jackett_default                bridge    local
53ff69cd9a89   lidarr_default                 bridge    local
fa498a7bb2c1   mealie_default                 bridge    local
47138cf95491   networking_adguard             ipvlan    local
93749dc48500   nextcloud_default              bridge    local
4a02a01d014b   nginx_default                  bridge    local
3d7c5e46eb9b   none                           null      local
7cce0ad2e17a   ombi_default                   bridge    local
aeade728a34f   photostructure_default         bridge    local
37506981f054   qbittorrentvpn_default         bridge    local
853aea5a9554   radarr_default                 bridge    local
ca5018f9c583   readarr_default                bridge    local
3865b501173f   sonarr_default                 bridge    local
2311a872a784   syncthing_default              bridge    local
6148d2132067   tautulli_default               bridge    local
68ea18f2fc32   vaultwarden_default            bridge    local
72281cf161e4   watchtower_default             bridge    local

Hopefully that fixes the issue. Parity check is running now so I will let that complete and I'll see if the server crashes again over the next few days. Thanks for the help.

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...