Jump to content

Crashing daily

joshsalvi

By joshsalvi
in General Support

Recommended Posts

joshsalvi Rookie

joshsalvi

Posted
Posted

Hi,

I keep having an issue where my unRAID machine keeps crashing, about once per day. It remains turned on, but subsequently disconnects. I'm not really sure what is wrong with the device, though I do have a number of standard apps and plugins installed. I was wondering if anyone might be able to help decode the diagnostics (attached here) to get down to the problem with the device. This is for a home server, primarily for media, backup, and sharing. I've also included a longer syslog for these purposes. The machine appears to have crashed sometime overnight. It does look like there are numerous calls to the My Servers plugin, along with the cache being full, but I'm not sure how that would cause the system to crash. 

 

Perhaps related to this portion of the log? 

Apr 30 00:00:02 Tower Docker Auto Update: Community Applications Docker Autoupdate running
Apr 30 00:00:02 Tower Docker Auto Update: Checking for available updates
Apr 30 00:00:51 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:01:10 Tower Docker Auto Update: Stopping netdata
Apr 30 00:01:31 Tower Docker Auto Update: Stopping nzbget
Apr 30 00:01:41 Tower kernel: veth49d3dc1: renamed from eth0
Apr 30 00:01:41 Tower kernel: docker0: port 9(veth5093395) entered disabled state
Apr 30 00:01:43 Tower kernel: docker0: port 9(veth5093395) entered disabled state
Apr 30 00:01:43 Tower kernel: device veth5093395 left promiscuous mode
Apr 30 00:01:43 Tower kernel: docker0: port 9(veth5093395) entered disabled state
Apr 30 00:01:49 Tower Docker Auto Update: Stopping sonarr
Apr 30 00:01:51 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:01:57 Tower kernel: veth21be5a4: renamed from eth0
Apr 30 00:01:57 Tower kernel: docker0: port 12(veth327a7f5) entered disabled state
Apr 30 00:01:59 Tower kernel: docker0: port 12(veth327a7f5) entered disabled state
Apr 30 00:01:59 Tower kernel: device veth327a7f5 left promiscuous mode
Apr 30 00:01:59 Tower kernel: docker0: port 12(veth327a7f5) entered disabled state
Apr 30 00:02:04 Tower Docker Auto Update: Stopping tautulli-1
Apr 30 00:02:14 Tower kernel: veth2de64d5: renamed from eth0
Apr 30 00:02:14 Tower kernel: docker0: port 14(vethecd6efb) entered disabled state
Apr 30 00:02:17 Tower kernel: docker0: port 14(vethecd6efb) entered disabled state
Apr 30 00:02:17 Tower kernel: device vethecd6efb left promiscuous mode
Apr 30 00:02:17 Tower kernel: docker0: port 14(vethecd6efb) entered disabled state
Apr 30 00:02:23 Tower Docker Auto Update: Stopping thelounge
Apr 30 00:02:30 Tower kernel: veth426e084: renamed from eth0
Apr 30 00:02:30 Tower kernel: docker0: port 15(veth5b5be94) entered disabled state
Apr 30 00:02:33 Tower kernel: docker0: port 15(veth5b5be94) entered disabled state
Apr 30 00:02:33 Tower kernel: device veth5b5be94 left promiscuous mode
Apr 30 00:02:33 Tower kernel: docker0: port 15(veth5b5be94) entered disabled state
Apr 30 00:02:44 Tower Docker Auto Update: Installing Updates for netdata nzbget sonarr tautulli-1 thelounge
Apr 30 00:02:51 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:04:51 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:05:51 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:06:51 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:10:51 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:11:51 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:12:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:13:29 Tower root: /etc/libvirt: 920.1 MiB (964812800 bytes) trimmed on /dev/loop3
Apr 30 00:13:29 Tower root: /var/lib/docker: 5.3 GiB (5717483520 bytes) trimmed on /dev/loop2
Apr 30 00:13:29 Tower root: /mnt/cache_bak: 48.4 GiB (51929731072 bytes) trimmed on /dev/sdg1
Apr 30 00:13:29 Tower root: /mnt/cache: 370.1 GiB (397339983872 bytes) trimmed on /dev/sdf1
Apr 30 00:13:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:14:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:15:38 Tower Docker Auto Update: Restarting netdata
Apr 30 00:15:51 Tower Docker Auto Update: Restarting nzbget
Apr 30 00:15:51 Tower kernel: docker0: port 9(veth3e21fc5) entered blocking state
Apr 30 00:15:51 Tower kernel: docker0: port 9(veth3e21fc5) entered disabled state
Apr 30 00:15:51 Tower kernel: device veth3e21fc5 entered promiscuous mode
Apr 30 00:15:51 Tower kernel: docker0: port 9(veth3e21fc5) entered blocking state
Apr 30 00:15:51 Tower kernel: docker0: port 9(veth3e21fc5) entered forwarding state
Apr 30 00:15:51 Tower kernel: docker0: port 9(veth3e21fc5) entered disabled state
Apr 30 00:15:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:16:26 Tower kernel: eth0: renamed from veth0139892
Apr 30 00:16:26 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth3e21fc5: link becomes ready
Apr 30 00:16:26 Tower kernel: docker0: port 9(veth3e21fc5) entered blocking state
Apr 30 00:16:26 Tower kernel: docker0: port 9(veth3e21fc5) entered forwarding state
Apr 30 00:16:30 Tower Docker Auto Update: Restarting sonarr
Apr 30 00:16:30 Tower kernel: docker0: port 12(vethcfd084f) entered blocking state
Apr 30 00:16:30 Tower kernel: docker0: port 12(vethcfd084f) entered disabled state
Apr 30 00:16:30 Tower kernel: device vethcfd084f entered promiscuous mode
Apr 30 00:16:30 Tower kernel: docker0: port 12(vethcfd084f) entered blocking state
Apr 30 00:16:30 Tower kernel: docker0: port 12(vethcfd084f) entered forwarding state
Apr 30 00:16:30 Tower kernel: docker0: port 12(vethcfd084f) entered disabled state
Apr 30 00:16:32 Tower nginx: 2022/04/30 00:16:32 [error] 13521#13521: *176353 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /admin/api.php?version HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "127.0.0.1"
Apr 30 00:16:32 Tower nginx: 2022/04/30 00:16:32 [error] 13521#13521: *176355 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /admin/api.php?version HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "localhost"
Apr 30 00:16:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:16:56 Tower kernel: eth0: renamed from veth97e4f51
Apr 30 00:16:56 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethcfd084f: link becomes ready
Apr 30 00:16:56 Tower kernel: docker0: port 12(vethcfd084f) entered blocking state
Apr 30 00:16:56 Tower kernel: docker0: port 12(vethcfd084f) entered forwarding state
Apr 30 00:17:01 Tower Docker Auto Update: Restarting tautulli-1
Apr 30 00:17:01 Tower kernel: docker0: port 14(vethf9fe2e8) entered blocking state
Apr 30 00:17:01 Tower kernel: docker0: port 14(vethf9fe2e8) entered disabled state
Apr 30 00:17:01 Tower kernel: device vethf9fe2e8 entered promiscuous mode
Apr 30 00:17:27 Tower kernel: eth0: renamed from veth762c43a
Apr 30 00:17:27 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethf9fe2e8: link becomes ready
Apr 30 00:17:27 Tower kernel: docker0: port 14(vethf9fe2e8) entered blocking state
Apr 30 00:17:27 Tower kernel: docker0: port 14(vethf9fe2e8) entered forwarding state
Apr 30 00:17:32 Tower Docker Auto Update: Restarting thelounge
Apr 30 00:17:32 Tower kernel: docker0: port 15(vethebe6890) entered blocking state
Apr 30 00:17:32 Tower kernel: docker0: port 15(vethebe6890) entered disabled state
Apr 30 00:17:32 Tower kernel: device vethebe6890 entered promiscuous mode
Apr 30 00:17:32 Tower kernel: docker0: port 15(vethebe6890) entered blocking state
Apr 30 00:17:32 Tower kernel: docker0: port 15(vethebe6890) entered forwarding state
Apr 30 00:17:32 Tower kernel: docker0: port 15(vethebe6890) entered disabled state
Apr 30 00:18:10 Tower kernel: eth0: renamed from vethde6f2f1
Apr 30 00:18:10 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethebe6890: link becomes ready
Apr 30 00:18:10 Tower kernel: docker0: port 15(vethebe6890) entered blocking state
Apr 30 00:18:10 Tower kernel: docker0: port 15(vethebe6890) entered forwarding state
Apr 30 00:18:16 Tower Docker Auto Update: Community Applications Docker Autoupdate finished
Apr 30 00:18:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:19:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:20:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:21:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:22:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:23:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:24:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:26:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:27:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:28:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:29:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:30:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:31:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:32:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:33:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:34:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:35:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:36:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:37:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Apr 30 00:38:52 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update

 

Thanks!

 

I'm also going to temporarily disable the My Servers plugin to see if that helps.

tower-diagnostics-20220430-1705.zip syslog.txt syslog-2 syslog-192.168.1.209.log

Link to comment
JorgeB Grand Master

JorgeB

Posted
Posted 
Apr 26 07:42:45 Tower kernel: macvlan_broadcast+0x10e/0x13c [macvlan]
Apr 26 07:42:45 Tower kernel: macvlan_process_broadcast+0xf8/0x143 [macvlan]

 

Macvlan call traces are usually the result of having dockers with a custom IP address, upgrading to v6.10 and switching to ipvlan might fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enable, top right)), or see below for more info.

 

https://forums.unraid.net/topic/70529-650-call-traces-when-assigning-ip-address-to-docker-containers/

 

See also here:

https://forums.unraid.net/bug-reports/stable-releases/690691-kernel-panic-due-to-netfilter-nf_nat_setup_info-docker-static-ip-macvlan-r1356/

 

There's also this:

  

Apr 26 06:31:12 Tower kernel: XFS (sdg1): Metadata corruption detected at xfs_dinode_verify+0xa3/0x581 [xfs], inode 0x3f0d11 dinode
Apr 26 06:31:12 Tower kernel: XFS (sdg1): Unmount and run xfs_repair

 

Check filesystyem on cache_bak

Link to comment
joshsalvi Rookie

joshsalvi

Posted
  • Author
Posted

Thank you!

 

  • I checked the filesystem on sdg1, and all looks okay.  
root@Tower:~# xfs_repair -v /dev/sdg1
Phase 1 - find and verify superblock...
        - block cache size set to 1514888 entries
Phase 2 - using internal log
        - zero log...
zero_log: head block 0 tail block 0
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
Phase 5 - rebuild AG headers and trees...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...

        XFS_REPAIR Summary    Mon May  2 08:40:12 2022

Phase		Start		End		Duration
Phase 1:	05/02 08:40:11	05/02 08:40:11
Phase 2:	05/02 08:40:11	05/02 08:40:11
Phase 3:	05/02 08:40:11	05/02 08:40:12	1 second
Phase 4:	05/02 08:40:12	05/02 08:40:12
Phase 5:	05/02 08:40:12	05/02 08:40:12
Phase 6:	05/02 08:40:12	05/02 08:40:12
Phase 7:	05/02 08:40:12	05/02 08:40:12

Total run time: 1 second
done
  • I'll change the docker custom network type now and see if this fixes the problem. I'll report back!
Link to comment
  • 2 years later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing
×
×
  • Create New...