Cache filling up every night then resolving itself

[The Transplant]

Every morning I wake up to errors about the cache running out then it resolves itself.  Running Version 6.11.5 2022-11-20

 

Looking at the log I see this:

 

Dec 30 04:54:16 Odin shfs: copy_file: /mnt/disk2/domains/Windows 10/vdisk1.img /mnt/cache/domains/Windows 10/vdisk1.img.partial (28) No space left on device.

 

Here are my cache settings:

 

 

This is similar to this post but the resolution involves changing cache settings (based on advice in 2018) and I can't find this in my current version settings to see if this is the issue.

https://github.com/blakeblackshear/frigate/discussions/4822

 

Any help appreciated.  Happy to post more information as needed.  Thanks.

[Kilrah]

Do you have some nignly backup routine (might be VMs given that file giving the error) that's backing up directly to the cache drive?

[The Transplant]

@Kilrah Thanks, this has to be it.  I do have a backup routine running:

 

 

I think I accepted defaults.  Should I be pointing this destination to something that is not running through cache?  I don't fully understand (newbie) the relationship between shared and actual folders.  Any advice appreciated.  Thanks.

[Kilrah]

This looks right.

 

But looking again at your first screenshot it seems your domains share is set to cache:prefer. How much stuff is in there? Prefer will make mover try to move everything in that share onto the cache everytime it runs, likely your cache is too small to fit your VMs and it fails each time. 

 

Click the "Compute" on the domains line (or better yet, "Compute All" button at the bottom) on the shares page and post a screenshot.

Edited December 30, 2022 by Kilrah

[The Transplant]

@Kilrah Thanks, I clicked the compute all and the results are attached.  Also, don't know if it is useful but I have a spare SSD (249GB) connected but currently unassigned.

 

[Kilrah]

So yes, the domains share has 537GB on disk2, and your choice of Cache:Prefer causes the mover to try to move that to the SSD that only has 146GB free.

 

Set it to cache:no so it leaves things as is if that's how you want them. 

 

Your system share likely also has duplicate files on both the cache and disk2, if the files on disk2 are older you'll want to delete them. 

Edited December 30, 2022 by Kilrah

[The Transplant]

Thanks. @Kilrah Will make those changes.

[The Transplant]

@Kilrah How do I confirm that there are duplicate files on cache and disk2?  I don't feel comfortable deleting files until I understand this more.

 

Also, I just realized that my backup runs weekly on a Monday.  So this should not have been causing the bump in disk usage every night.  Since switching to cache:no on domains yesterday,  I only got warnings about utilization and no alerts.  So something is still happening.  I do have the second SSD drive attached that is unused.  I could add this as a second cache drive since I am not using it for anything else?

[itimpi]

32 minutes ago, The Transplant said:

Since switching to cache:no on domains yesterday,

Note that mover ignores any shares with Use Cache=No setting so any files for that share that are already on the cache are left there.

[Kilrah]

2 hours ago, The Transplant said:

How do I confirm that there are duplicate files on cache and disk2?

Browse the folders in the UI and compare file date/times.

[The Transplant]

@kilrah

 

I did a quick run through of disk2 and cache and didn't see anything older on disk2.  Just out of curiosity is there a reason you said to check between disk2 and cache vs. any other disk?  Should I check the other disks?

 

Also, if I set a disk to prefer:cache and then ran mover manually, then turned cache off - would it correct out of date files?

 

Thanks.

 

 

 

 

[trurl]

37 minutes ago, The Transplant said:

Just out of curiosity is there a reason you said to check between disk2 and cache vs. any other disk?

In your screenshot above that shows disk usage for system share.

[Kilrah]

12 hours ago, The Transplant said:

I did a quick run through of disk2 and cache and didn't see anything older on disk2.

You checked disk2/system and cache/system? The point is that only cache/system should contain files. If disk2/system also does then they're in conflict, and you want to delete the older one.

 

13 hours ago, The Transplant said:

Also, if I set a disk to prefer:cache and then ran mover manually, then turned cache off - would it correct out of date files?

Mover will not move a file if there's already a duplicate at the destination, which is likely your case.

[The Transplant]

@Kilrah In disk2\system\docker I have one file docker.img with a timestamp of 22-10-17.  In cache\system\docker I have one file docker.img with a timestamp of 23-1-08.  For system I have Prefer : Cache.  So should I delete the docker.img file in disk2\system\docker, the older file, and then run mover?  Similarly in disk2\system\libvirt I have a file libvirt.img with a timestamp of 22-10-17.  In cache\system\libvirt I have a file libvirt.img with a timestamp of 23-1-1.  Should I delete the older file in disk2\system\libvirt?  Thanks.

[trurl]

yes delete the older files