logs reporting disk is full when it is not

[huladaddy]

I have begun getting errors related to my disk being full.

 

binhex-sabnzbdvpn:

tee: /config/supervisord.log: No space left on device

Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/supervisor/loggers.py", line 102, in emit
self.stream.write(msg)
OSError: [Errno 28] No space left on device

 

 

JellyFin:

[16:49:57] [FTL] [1] Main: Error while starting server.
Full: SQLitePCL.pretty.SQLiteException: database or disk is full

Because of this error, JellyFin no longer runs.

 

 

but this is what my filesystems look like:

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop2       20G  9.6G  9.8G  50% /
tmpfs            64M     0   64M   0% /dev
tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
shfs             53T   53T  299G 100% /tv
/dev/loop2       20G  9.6G  9.8G  50% /etc/hosts
tmpfs           3.8G     0  3.8G   0% /proc/acpi
tmpfs           3.8G     0  3.8G   0% /sys/firmware

 

 

and this is what my docker volume looks like:

Label: none  uuid: 717af81f-c683-4ca7-98de
    Total devices 1 FS bytes used 9.12GiB
    devid    1 size 20.00GiB used 13.52GiB path /dev/loop2

 

 

What is going on? What is causing these errors?

 

[JorgeB]

Please post diags.

[huladaddy]

I have some new information. I can see that 3 of my physical drives are actually filling up. All of my shares are set to high-water and automatically split directories.

 

 

diagnostics-20210819-1324.zip

[trurl]

I would say that most of your disks are dangerously full. I say dangerously because if you corrupt a disk filesystem you may not be able to repair since there is no space for repair to work on the disk.

 

And you have no Minimum Free set for any of your shares or cache. You should set Minimum Free to larger than the largest file you expect to write to the share or cache.

 

And your cache is probably too small. Is that why you have appdata on the array?

 

You should quit downloading anything or allowing anything to write to your server until you deal with your space issues. Ideally you want to keep several GB free on all disks, maybe more.

[huladaddy]

Thanks for the advice.

 

Is there any way to balance the free space across all the drives?

[trurl]

There isn't enough free space on any drive to bother balancing. You need more or larger disks.

 

I usually prefer larger instead of more to keep the drive count down, but you end up not gaining as much space when you replace a disk with a larger one as you would if you just added a larger one.

 

But I always say, each additional disk is an additional point of failure.

[trurl]

1 minute ago, trurl said:

There isn't enough free space on any drive to bother balancing.

 

1 hour ago, trurl said:

you have no Minimum Free set for any of your shares or cache. You should set Minimum Free to larger than the largest file you expect to write to the share or cache.

What is the largest file you expect to write to any share or cache? Many people will have movies that are more than 10G, maybe much more.

 

Unraid has no way to know how large a file will become when it chooses a disk for it. If a disk has more than Minimum Free, it can be chosen for the file. If the disk is chosen, and the file won't fit, the write will fail.

[huladaddy]

If I made space by deleting files, how could I go about balancing?

[trurl]

unBALANCE plugin.

[huladaddy]

Thanks very much for the advice. Appreciate it.

 

Regarding cache, I guess I can replace it with a larger drive. Is it recommended for appdata to be put on cache? Isn't it better to have it on the array for data protection? I have the appdata share set to not use cache.

[trurl]

Your dockers/VMs have some shares they use by default. These are appdata, domains, system.

 

Dockers/VMs will perform better if these shares are on cache (or other pool) and set to stay on pool since they won't be affected by slower parity updates, and even better performance if pool is SSD. Also, files are always open on these shares, so if they are on the array disks can't spindown.

 

CA Backup plugin will backup your appdata.