November 30, 20241 yr My Server Zelda, hardware listed in signature, keeps locking up after ~1 day of running. Syslog is showing that one of my share's cache is full. It isn't and actually has 8.07TB available. It floods the syslog with it then it crashes. There are other errors in there too. I'm not sure if it is related but I had no issues until I added in an RTX 3070. After a couple days it started locking up like this. I removed the card to try to diagnose and even restored a backup but the problem persists now. This is going on a month I've been trying to fix this and it is driving me insane. zelda-diagnostics-20241130-0847.zip syslog-100.93.205.62.log
November 30, 20241 yr Community Expert this sounds like a mover / phatom mount are you using cahce shares or disk shares? do you have a share set for 2 locations? you may need a plugin ICH777 for the nvidia driver. *double check bios make sure uefi and resizeable bar is enabled. run mem test.
November 30, 20241 yr Author 1 hour ago, bmartino1 said: this sounds like a mover / phatom mount are you using cahce shares or disk shares? do you have a share set for 2 locations? you may need a plugin ICH777 for the nvidia driver. *double check bios make sure uefi and resizeable bar is enabled. run mem test. I use cache shares. I have one share for zfs_backup that only uses my array disk. I have three cache pools. One called cache that is a zfs mirror, one that is a single cache disk for temp files like downloads, and one that is a zfs pool with 3 6tb drives and one ssd setup as cache for that pool. The array is a none parity protected signle zfs disk. One of the shares that was saying full was using the single disk cache for primary then the hdd pool for long term storage. I have since changed that to be the array. I could see there being a mover issue just not sure how to correct at this point. I removed the graphics card so I removed the nvidia plugin as well but would really like to use it for llama. I had resiable bar on when the card was in but disabled after I removed the card. UEFI is on. Issue has been going with or without it enabled. Computer is stable booting into a different OS. I've tried so many things I'm ready to nuke it and start over. Memtest passing with zero errors. Been running it for quite awhile.
December 1, 20241 yr Community Expert i think you have a mover issues and a misconfigured cahce share. as something seems angry ... lets review the sys logs: Observations Memory and Disk Usage The syslogs show recurring messages about a cache pool being full. However, the user reports the cache has adequate space (8.07TB free). This discrepancy might indicate issues with disk usage reporting, file system corruption, or mover misconfiguration. *Since memtest is fine System Stability The server locks up after about a day, which aligns with issues like memory exhaustion, resource contention, or hardware/software conflicts. *are you hard rebooitng when this happens? Graphics Card History The user recently added and removed an RTX 3070. Changes to hardware configurations (especially GPUs) can leave residual conflicts in the software stack (e.g., NVIDIA drivers or plugins). *Is the card pluded in or not? could be a bad g-card.... ZFS Involvement The system uses ZFS pools for caching and storage. ZFS is resource-intensive, and improper configuration can exacerbate instability or crashes. *1 gb of ram to 1 TB in ZFS is a golden rule for alocation.... Mover Configuration Mover-related operations are critical to Unraid's cache system. If misconfigured, this could lead to the "cache full" messages and impact stability. ?rsync from cache share? if not mover? Lets Double check and Verify Cache Pool and Filesystem Health: Check the health of ZFS pools and the single-disk cache pool *Can be done in the web ui... zpool status zfs list zpool status zfs list Look for errors or warnings. Not seing anything out right.. Lets Analyze Mover Logs Review the mover logs to ensure files are being moved correctly and not causing contention: /var/log/syslog | grep mover *no mention of mover? ATM due to cache shares, this may be a phtom/misconfigured path in a docker... advance togle show more setting confrim pathing... Review Docker and Plugins Temporarily stop all Docker containers and disable plugins to isolate their impact *boot into unraid safemode. does it stay on for a day? Monitor Resource Usage Use htop or iotop to track CPU, memory, and disk usage. Pay attention to spikes before the crash. Per previous-syslog: Known acpi linux bug Nov 24 16:19:49 zelda kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.XHCI.RHUB.SS01._UPC], AE_ALREADY_EXISTS Nov 24 16:19:49 zelda kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog Nov 24 16:19:49 zelda kernel: ACPI: Skipping parse of AML opcode: OpcodeName unavailable (0x0014) *look for bios updates this can fix these sometimes... You apear to have a network issue: Nov 24 16:19:49 zelda rsyslogd: omfwd/udp: socket X: sendto() error: Network is unreachable Nov 24 16:19:49 zelda rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd') These errors suggest that the rsyslog daemon is attempting to forward logs over the network but cannot reach the destination. Verify that the target server for log forwarding is reachable. *Review Rsyslog Configuration... confirm its not a mover issues: as there are no mover-related entries in the syslog Enable Mover Logging: Go to Settings > Scheduler in the Unraid web interface. Set Mover logging to Enabled to capture mover activities in the syslog I'm not seeing any major issue on why unraid would be unstable. a 3070 is power hungry make sure your not under power for the g card and machine. Recommend to run unraid in safe mode (no plugins or dockers) to see if it is a docker or plugin
December 1, 20241 yr Author 5 minutes ago, bmartino1 said: i think you have a mover issues and a misconfigured cahce share. as something seems angry ... lets review the sys logs: Observations Memory and Disk Usage The syslogs show recurring messages about a cache pool being full. However, the user reports the cache has adequate space (8.07TB free). This discrepancy might indicate issues with disk usage reporting, file system corruption, or mover misconfiguration. *Since memtest is fine System Stability The server locks up after about a day, which aligns with issues like memory exhaustion, resource contention, or hardware/software conflicts. *are you hard rebooitng when this happens? Graphics Card History The user recently added and removed an RTX 3070. Changes to hardware configurations (especially GPUs) can leave residual conflicts in the software stack (e.g., NVIDIA drivers or plugins). *Is the card pluded in or not? could be a bad g-card.... ZFS Involvement The system uses ZFS pools for caching and storage. ZFS is resource-intensive, and improper configuration can exacerbate instability or crashes. *1 gb of ram to 1 TB in ZFS is a golden rule for alocation.... Mover Configuration Mover-related operations are critical to Unraid's cache system. If misconfigured, this could lead to the "cache full" messages and impact stability. ?rsync from cache share? if not mover? Lets Double check and Verify Cache Pool and Filesystem Health: Check the health of ZFS pools and the single-disk cache pool *Can be done in the web ui... zpool status zfs list zpool status zfs list Look for errors or warnings. Not seing anything out right.. Lets Analyze Mover Logs Review the mover logs to ensure files are being moved correctly and not causing contention: /var/log/syslog | grep mover *no mention of mover? ATM due to cache shares, this may be a phtom/misconfigured path in a docker... advance togle show more setting confrim pathing... Review Docker and Plugins Temporarily stop all Docker containers and disable plugins to isolate their impact *boot into unraid safemode. does it stay on for a day? Monitor Resource Usage Use htop or iotop to track CPU, memory, and disk usage. Pay attention to spikes before the crash. Per previous-syslog: Known acpi linux bug Nov 24 16:19:49 zelda kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.XHCI.RHUB.SS01._UPC], AE_ALREADY_EXISTS Nov 24 16:19:49 zelda kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog Nov 24 16:19:49 zelda kernel: ACPI: Skipping parse of AML opcode: OpcodeName unavailable (0x0014) *look for bios updates this can fix these sometimes... You apear to have a network issue: Nov 24 16:19:49 zelda rsyslogd: omfwd/udp: socket X: sendto() error: Network is unreachable Nov 24 16:19:49 zelda rsyslogd: action 'action-2-builtin:omfwd' suspended (module 'builtin:omfwd') These errors suggest that the rsyslog daemon is attempting to forward logs over the network but cannot reach the destination. Verify that the target server for log forwarding is reachable. *Review Rsyslog Configuration... confirm its not a mover issues: as there are no mover-related entries in the syslog Enable Mover Logging: Go to Settings > Scheduler in the Unraid web interface. Set Mover logging to Enabled to capture mover activities in the syslog I'm not seeing any major issue on why unraid would be unstable. a 3070 is power hungry make sure your not under power for the g card and machine. Recommend to run unraid in safe mode (no plugins or dockers) to see if it is a docker or plugin Hey thanks for your detailed insight. I'll go through these things and get back if I find a resolution or not.
May 4, 20251 yr Author After going through all of this and found quite a bit that needed cleaning up it was still doing it. Eventually I had a drive throw some errors and I pulled it out. Not lock up since. Have had it up for 2 weeks and no issues since. Was doing this for awhile before any errors showed up. Very frustrating but glad it seems to be fixed now.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.