(SOLVED) No space left on device (28)

[sheldz8]

This is the second time today I recieved this message, I'm not sure what is causing it and resolves itself after rebooting. 

 

Attached is the diagnostics (after reboot), error when trying to login and df -h output (before reboot).

 

When this happens I can still use SSH and Plex but I can't login / access SMB shares 

 

unraid-server-diagnostics-20210920-1552.zip

Edited September 20, 2021 by sheldz8

[trurl]

You have filled rootfs, which is where the OS is and so it makes it impossible for the OS to work with its own files.

 

This probably means you have something writing to a path that isn't on any actual storage device. One possibility would be a docker mapping with a host path that doesn't specify a user share or disk.

[trurl]

Not related, but your appdata, domains, system shares have files on the array and are set to move files to the array.

 

Best if these shares are all on pool (cache) and set to stay on pool. Having them on the array will keep disks spunup since these files are always open, and will impact docker/VM performance due to slower array.

[sheldz8]

11 hours ago, trurl said:

You have filled rootfs, which is where the OS is and so it makes it impossible for the OS to work with its own files.

 

This probably means you have something writing to a path that isn't on any actual storage device. One possibility would be a docker mapping with a host path that doesn't specify a user share or disk.

Is it possible to see what caused it because my server was fine since I built it. I haven't made any changes to it in 2 months 

[sheldz8]

11 hours ago, trurl said:

Not related, but your appdata, domains, system shares have files on the array and are set to move files to the array.

 

Best if these shares are all on pool (cache) and set to stay on pool. Having them on the array will keep disks spunup since these files are always open, and will impact docker/VM performance due to slower array.

The last time I set these shares to only use Cache my 1TB cache pool filled up fast and the server was unusable

[trurl]

7 hours ago, sheldz8 said:

Is it possible to see what caused it because my server was fine since I built it. I haven't made any changes to it in 2 months 

 

Examine each of your dockers and make sure any host path is a user share or disk.

 

7 hours ago, sheldz8 said:

The last time I set these shares to only use Cache my 1TB cache pool filled up fast and the server was unusable

 

This also indicates some misconfiguration. These shares shouldn't take nearly that much space.

 

Get rid of that gdrive temporarily and see if you can get things working correctly including moving those shares to cache.

[sheldz8]

15 hours ago, trurl said:

 

Examine each of your dockers and make sure any host path is a user share or disk.

 

 

This also indicates some misconfiguration. These shares shouldn't take nearly that much space.

 

Get rid of that gdrive temporarily and see if you can get things working correctly including moving those shares to cache.

Thanks

 

I did check the dockers and all the volumes were correct but I left my server alone for 1 day and the disk didn't fill up again so I don't know what happened.

 

I'll sort out the shares when I get time.

Edited September 22, 2021 by sheldz8

[sheldz8]

17 hours ago, trurl said:

 

Examine each of your dockers and make sure any host path is a user share or disk.

 

 

This also indicates some misconfiguration. These shares shouldn't take nearly that much space.

 

Get rid of that gdrive temporarily and see if you can get things working correctly including moving those shares to cache.

I found the cause it was my rclone mount i'm not sure what happened but this topic is solved