Recently my server (Unraid 6.12.11) has been behaving strangely.

- Syslog starts getting these error from around 3AM everyday (JST, so UTC+9). Natually I haven't been awake to catch it live, so all I can get are from logs later in the morning: https://pastebin.com/E60PBcUa

- I could SSH into the server, the Docker containers seem to be running, but du -h / shows that the flash drive is full. The culprits seems to be some randomly named directories under /usr/local/emhttp/exit/.

- I tried to run diagnostics through SSH, but as the flash drive was full, the generated diag zip was empty.

- Hardware parts seem to be fine, the server run throughout the day without any error.

- After a reboot everything seems fine again. I have attached a diag after reboot.

Please let me know if there's anything else I can do provide to help solve this weird problem. Thank you.

tower-diagnostics-20241129-0937.zip

root fs full this tells me you have a phantom mount writing to a non-existent location in /mnt user and have filled ram (unraid root drive)

Thank you for posting diag. Will reveiw and get back sometime tomorrow.

My recommendation atm is to shutdown all vms, Dockers and if have lxc turn them off...
*turn off autostart of any and all vms and dockers
setting > vm off

setting > docker off
reboot unriad

at webui after a reboot the root fs should be normal. at this time you neeed to verifi disk , pool names settings and any errors to make sure they are talking to a exisintg path and not filling the fuse /mnt/user.

?swap pluging? docekr, vm path poing to a temp location and not the actual drive?
run mover to move files... verify that non of your array disk are full and high water hit.

as you will need to fix pathing and file data locaiton if it is full...


then enable dockers and edit docker setting and each docker and confirm file pathing and disk is not full or misconfigured...
same for VMs...

will need to review diag for further torubleshooting.

are you runnign and user scripts?

 

The syslog indicates issues related to network connectivity, as multiple errors like Network is unreachable and related socket errors from rsyslogd are present. However, these are unlikely to be directly tied to the rootfs filling up or the /usr/local/emhttp/exit/ directory issues. Let's analyze the problem further:

 

Diagnosing the rootfs Full Issue

 

Probable Causes:

Excessive Logging or Temporary Files:

Randomly named directories under /usr/local/emhttp/exit/ might be logs, temporary files, or cache growing excessively, filling up the limited rootfs space.

 

Memory Leaks or Misconfigured Applications:

A process or script might be writing excessively to /usr/local/emhttp without cleanup.

 

Scheduled Tasks:

Since the issue happens around 3 AM (JST), it's likely linked to a scheduled cron job or background process.

 

Next Steps:

Implement these diagnostics and collect data to pinpoint the exact cause...

If a plugin or script is causing the issue, disabling or reconfiguring it may resolve the problem.

Review logs in the /usr/local/emhttp/exit/ directory for clues on which application or process creates these files.

 

Suggested Steps:

*Untested this may or may not work emhttp is unraid webui web server...
Inspect /usr/local/emhttp/exit/ Growth:

Before the rootfs fills up again, monitor the directory to identify what gets written there

du -sh /usr/local/emhttp/exit/*
 

du -sh /usr/local/emhttp/exit/*

Check the timestamps of the files:

ls -lt /usr/local/emhttp/exit/

 

*Warning this can break/corrupt files... as temp data is stored in ram... this will potential increas the ram disk os in ram if you have more ram..

*Untested use at your own risk!...
Increase Rootfs Space Temporarily:

If you are unable to gather diagnostics due to the rootfs filling up, consider remounting with a larger tmpfs size:

lsblk
df -h
mount -o remount,size=4G /var

Note: This is a temporary solution to collect more logs
*a reboot should fixes this and be better...

 

Check for Cron Jobs:

Review scheduled cron jobs, especially those that run around 3 AM:

*what user script is running at 3 am ? zfs/btrfs scrub ? snapshot ? other rsync file copy?
 

cat /etc/cron.d/* | grep -i 3:00

 

*enable syslog server to try and captures....
 

Examine Diagnostics After 3 AM:

Leave a session running to log the directory's contents at intervals after 3 AM

while true; do date >> /boot/emhttp_debug.log; ls -l /usr/local/emhttp/exit/ >> /boot/emhttp_debug.log; sleep 300; done

*Run at console of unraid terminal...

?is syslog filling up log rotate setting?

Syslog Configuration:

Redirect logs to a persistent storage location to avoid rootfs fill

mkdir -p /mnt/cache/syslogs
ln -sf /mnt/cache/syslogs /var/log

 

Investigate Plugin or Software Behavior:

The issue could stem from a third-party plugin or Docker container interacting with /usr/local/emhttp

 

Disk Failure ?
Monitor Disk I/O:

Use iotop or similar tools to identify processes writing heavily

iotop -o

 

Thank you bmartino1 for your detailed replies. I will review settings and setup further logging before 3AM. The remounting tmpfs tip seems promising too, hopefully next time I will be able to get a proper diag before reboot.

2 hours ago, bmartino1 said:

are you runnign and user scripts?

 

I do have user scripts, and this one rclone script runs at 3AM.

I recently poked around in the rclone config a bit, and might have bungled something up.

Checking the log file created by that script revealed this:

 

2024/11/29 16:13:33 DEBUG : rclone: Version "v1.68.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "sync" "-vv" "--log-file" "/mnt/user/Logs/rclone.txt" "/mnt/user/Chonky/Backups" "encrypted_b2:"]
2024/11/29 16:13:33 DEBUG : Creating backend with remote "/mnt/user/Chonky/Backups"
2024/11/29 16:13:33 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2024/11/29 16:13:33 DEBUG : Creating backend with remote "encrypted_b2:"
2024/11/29 16:13:33 DEBUG : Creating backend with remote "exit"
2024/11/29 16:13:33 DEBUG : fs cache: renaming cache item "exit" to be canonical "/root/exit"
2024/11/29 16:13:33 DEBUG : <filename>: Need to transfer - File not found at Destination
2024/11/29 16:13:33 DEBUG : <filename>: Need to transfer - File not found at Destination
2024/11/29 16:13:33 DEBUG : <filename>: Need to transfer - File not found at Destination

 

I don't think the remote "exit" is supposed to be there.

Will continue to investigate.

It was that rclone script indeed.

I fixed the remote config and the script is actually backing up my file now, instead of trying to write to a nonexistent exit directory on the root file system.

As for why the rclone config ended up the way it did, I probably tried to exit out of the rclone config by typing in "exit" at the remote name step. 😅

I'm just glad rclone didn't overwrite anything important apparently.