ZFS shows full!

[NLS]

So with 6.12 (now on 6.12.1), I changed my cache (single) disk to zfs.

It is a 1TB M.2, on a 32GB RAM server.

Also my docker is a folder, not an image.

 

Anyway, had some huge file transfers today and my cache got full.

I've seen before this being critical (and UNRAID SHOULD protect itself more actually), but this time seems it is MORE critical.

 

When cache became full, my VM became paused and even dockers were not fully ok.

I managed to get local access to my server, I moved manually a few GB of data and rebooted the server.

Machine started working more or less ok (VM was ok), but some dockers seemed to not work ok. SWAG is stopping and doublecommander (as I planned to move more files off the cache) started with the VNC desktop giving me errors. Removed, reinstalled... that docker never succeeded starting again.

 

Then I noticed, although cache had like 20GB free, there was an entry in the log the kept adding!
"shfs: share cache full"!!!

...every single second!

 

I used UNRAID own GUI, to start moving a few more GB off the server. I am now around 70GB free (and around 91% cache utilization), which should be plenty and the log entry keeps going!

Cache settings are now:

5% minimum free space (I changed that from a much smaller number).

warning 70%

critical 90%

THEN I NOTICED in dashboard... ZFS shows 100% (it even showed 101% briefly!)...

(also in SWAG log I saw that there was no space to create some files which is why it stops)

 

HTF does this go down???

I am now thinking of reverting to btrfs, as zfs seems to have overcomplicated things for no reason.

 

Question is, what I can do now, how this goes down?

(plan right now, is that when I finish the move that needs about 2 more hours to finish, to reboot the server again)

Another thing I noticed, is that CPU load is around 35% while the move (and log entries) is keep going. Which for a Ryzen 5600G (6 core/12 thread) is a bit heavy for doing nothing serious... but still better than when I booted the server (after the initial small move to free a bit of cache) which was a sinus graph going up and down. At least now it is stable.

Edited June 28, 2023 by NLS

[itimpi]

Have you set a Minimum Free Space value on the cache pool to stop it getting too full?

[NLS]

5 minutes ago, itimpi said:

Have you set a Minimum Free Space value on the cache pool to stop it getting too full?

 

I did after the issue.

The problem is that now I DO have space in the cache (more than 100GB) and still the log is getting full with those entries, ZFS reports 100%, containers run funny (depending on if they need to write files) and probably cache is read only for some reason.
(although I am under the minimum free space threshold)
Again: Cache is 89% (and getting lower) but ZFS (which is used only in the cache, no other drive) reports 100% (the bar under "memory" in dashboard).

 

Edited June 28, 2023 by NLS

[itimpi]

3 minutes ago, NLS said:

but ZFS (which is used only in the cache, no other drive) reports 100% (the bar under "memory" in dashboard).

That is expected to be 100% most of the time.    It is not an indication of how full the drive is, but an indication of how much of the RAM allocated for ARC use is being used.

[NLS]

4 minutes ago, itimpi said:

That is expected to be 100% most of the time.    It is not an indication of how full the drive is, but an indication of how much of the RAM allocated for ARC use is being used.

 

OK.

I will wait for the transfer to finish, reboot and hope the system will work more normally afterwards.

 

(note that this system worked fine for years)

[crazycajun]

On 6/28/2023 at 4:29 PM, NLS said:

 

OK.

I will wait for the transfer to finish, reboot and hope the system will work more normally afterwards.

 

(note that this system worked fine for years)

Any update to the outcone?

[NLS]

On 1/16/2024 at 10:55 PM, crazycajun said:

Any update to the outcone?

 

fixed itself after reboot if I remember