This is the second time today I recieved this message, I'm not sure what is causing it and resolves itself after rebooting. 

 

Attached is the diagnostics (after reboot), error when trying to login and df -h output (before reboot).

 

When this happens I can still use SSH and Plex but I can't login / access SMB shares 

 

unraid-server-diagnostics-20210920-1552.zip

Edited September 20, 20214 yr by sheldz8

You have filled rootfs, which is where the OS is and so it makes it impossible for the OS to work with its own files.

 

This probably means you have something writing to a path that isn't on any actual storage device. One possibility would be a docker mapping with a host path that doesn't specify a user share or disk.

Not related, but your appdata, domains, system shares have files on the array and are set to move files to the array.

 

Best if these shares are all on pool (cache) and set to stay on pool. Having them on the array will keep disks spunup since these files are always open, and will impact docker/VM performance due to slower array.

11 hours ago, trurl said:

You have filled rootfs, which is where the OS is and so it makes it impossible for the OS to work with its own files.

 

This probably means you have something writing to a path that isn't on any actual storage device. One possibility would be a docker mapping with a host path that doesn't specify a user share or disk.

Is it possible to see what caused it because my server was fine since I built it. I haven't made any changes to it in 2 months 

11 hours ago, trurl said:

Not related, but your appdata, domains, system shares have files on the array and are set to move files to the array.

 

Best if these shares are all on pool (cache) and set to stay on pool. Having them on the array will keep disks spunup since these files are always open, and will impact docker/VM performance due to slower array.

The last time I set these shares to only use Cache my 1TB cache pool filled up fast and the server was unusable

7 hours ago, sheldz8 said:

Is it possible to see what caused it because my server was fine since I built it. I haven't made any changes to it in 2 months 

 

Examine each of your dockers and make sure any host path is a user share or disk.

 

7 hours ago, sheldz8 said:

The last time I set these shares to only use Cache my 1TB cache pool filled up fast and the server was unusable

 

This also indicates some misconfiguration. These shares shouldn't take nearly that much space.

 

Get rid of that gdrive temporarily and see if you can get things working correctly including moving those shares to cache.

15 hours ago, trurl said:

 

Examine each of your dockers and make sure any host path is a user share or disk.

 

 

This also indicates some misconfiguration. These shares shouldn't take nearly that much space.

 

Get rid of that gdrive temporarily and see if you can get things working correctly including moving those shares to cache.

Thanks

 

I did check the dockers and all the volumes were correct but I left my server alone for 1 day and the disk didn't fill up again so I don't know what happened.

 

I'll sort out the shares when I get time.

Edited September 22, 20214 yr by sheldz8

17 hours ago, trurl said:

 

Examine each of your dockers and make sure any host path is a user share or disk.

 

 

This also indicates some misconfiguration. These shares shouldn't take nearly that much space.

 

Get rid of that gdrive temporarily and see if you can get things working correctly including moving those shares to cache.

I found the cause it was my rclone mount i'm not sure what happened but this topic is solved

 

I was affected by this issue as well...

 

I just got alerted to several of my Dockers stopped after the CA Update plugin. My unRAID is at 6.12.14 and had 33 days of uptime when it occurred. 

 

Digging into more, I see this error in my logs:

 

unraid pam_elogind(sshd:session): Failed to create session: No space left on device

 

The issue started shortly after the unRAID plugin 'CA Auto Update Applications' was scheduled to run and update my Dockers.

 

My troubleshooting steps:

1. I verified there was no storage space issues on the array, cache pools, or flash drive.
2. I stopped all dockers and then array. When I started the array, the issue returned and then all dockers could not start.
3. I then stopped the array and rebooted unRAID. Upon starting up the array again, all dockers started normally without error.
4. As a precaution, I disabled auto updating of my dockers using the 'CA Auto Update Applications' plugin.

 

I'm not sure what caused this....

 

Any ideas team?

Edited January 30, 20251 yr by gurulee

When it happens, post Diagnostics before rebooting

On 1/29/2025 at 4:58 PM, trurl said:

When it happens, post Diagnostics before rebooting

@trurl I suspect the issue was caused by the 'CA Auto Update Applications' plugin, which auto-updates my dockers as well. See my edited post above.

Edited January 31, 20251 yr by gurulee

The plugin would not be to blame, must be one of the containers. 

 

20 hours ago, trurl said:

When it happens, post Diagnostics before rebooting

 

22 hours ago, trurl said:

The plugin would not be to blame, must be one of the containers. 

 

 

I suspected the 'CA Auto Update Applications' plugin since it auto-updates my dockers and the scheduled time of the docker updates coincided with the errors in the unRAID logs, and when the issue started.
I will grab/post diagnostics if it happens again.

Edited January 31, 20251 yr by gurulee

I had a similar issue, for me, it is related to a home-assistant unraid integration. Here is the bug issue for the it:

https://github.com/domalab/ha-unraid/issues/66

 

Suggested workaround was using a user script to filter-out associate syslog entry not resolving:

https://github.com/domalab/ha-unraid/wiki/Troubleshooting#ssh-session-log-spam

Edited March 27, 20251 yr by gurulee