Hey all,

 

Had an issue start happening yesterday morning, but I'm having issues getting things moving so I can resolve.

Mar 15 13:04:06 Slash webgui: File_put_contents_atomic failed to write / rename /mnt/cache/1160867511.tmp
Mar 15 13:04:06 Slash root: Fix Common Problems: Error: Unable to write to cache

 

I've been fiddling with Docker recently and I've got some issues I know I need to resolve with Docker sizing (or something misconfigured), cache drive sizing, and whatnot, but I can't get mover to work, and the Docker stack won't start, and the cache drive seems to be in RO.

 

I did see some threads about the cache drive I have (or a newer version of it, really) that looks like it might have a firmware issue too, though since they were talking about some newer Crucial drives, I'm not sure if it's a red herring.

 

Thank you to anyone who can help take a look and give some guidance.

slash-diagnostics-20250315-1322.zip

 

Edit 1:

Trying to look through things, since I suspected something was up with the SSD cache drive, I've brought the array into maint mode and done the filesystem scan and extended scan and it looks like it's clean.  Can I erase the cache drive pool to get things restarted?  I'm okay losing cached data and rebuilding my dockers/VMs as needed, as the primary purpose is still the rest of the data.

 

Edit 2:  I was able to manually delete a few files from the cache drive which freed up enough for the docker stack to start.  Mover is no longer throwing errors, but it's not doing a heck of a lot either.  I've enabled logging and it's starting and finishing instantly, but no errors.  I'm guessing that it's not healthy to sit with only 8.57G free after a reboot.  I've also used the time to go through some of Spaceinvader One's videos and check the image size and whatnot, cleanup excess docker images, etc.  The docker image is setup to a max of 300G (apparently I did that way back when, it's set at 100G now), but it looks like the docker.img file is 322G.  Looking at the cache share, it's calculating some 750G of space usage on the cache drive though.

 

Edited March 16, 20251 yr by justinmit
Adding updates

So after looking through more threads on the mover, I saw a lot of discussion about the settings for mover with the shares setup to use cache, but move to the array.  Finally figured out that if I have the share setup to "Array" as it has been, it looks like it uses the cache, but gives no settings for mover.  But when I set it for New files to be created in Cache, and secondary is array, the mover action "Cache -> Array" is available, and setting that moves the files off to the array.  I did that and triggered the Mover process and it's in the process of moving a lot of files to the array.

 

However, is this the proper setup?  Shouldn't it handle moving things back and forth to the cache as needed instead?  I presume with this setting, if the file's been moved, it's not going to be cached if I go back out to read the files again, so this is more of a Fast Write to Slow Store setup than a real caching setup.  

 

Is there some documentation that explains in better detail?  This post was super helpful to me to explain that this seems to lay out the table of options and helped with why the mover was doing (and not doing) what it was doing:

But it also feels like something off on the setup.  I had 750G listed as the total size of files on the cache drive that's 500G, and mover wasn't engaging (seems like it's due to that configuration), but I'm not sure about how the files got to the cache drive in the first place in that case then.  It seems like it was the expected caching mechanism that put it there, but from the post above, it sounds like those were orphaned files.

 

I do think part of the problem is the fact that the docker image is over 300G, so I'll need to work on trying to shrink it as well.

46 minutes ago, justinmit said:

some documentation that explains in better detail

https://docs.unraid.net/unraid-os/manual/shares/user-shares/#primary-and-secondary-storage-unraid-612

 

In general, the current documentation is at the Documentation link at the bottom of every forum page, and also by clicking "manual" in the bottom right corner of your Unraid webUI.

 

48 minutes ago, justinmit said:

is this the proper setup

Probably some details could be better. For specific recommendations post new diagnostics.

7 minutes ago, trurl said:

https://docs.unraid.net/unraid-os/manual/shares/user-shares/#primary-and-secondary-storage-unraid-612

 

In general, the current documentation is at the Documentation link at the bottom of every forum page, and also by clicking "manual" in the bottom right corner of your Unraid webUI.

 

Probably some details could be better. For specific recommendations post new diagnostics.

Thank you for that.  I had been looking through the docs, and once I realized the mover mechanism, some of it was making more sense.  Like the cache page there talks specifically about how it's necessary to set the limits otherwise it sounds like what I was seeing is expected.  But I couldn't find anything on how the cache was working prior to the new settings I'm using.  It looked like it was using the cache drive pretty effectively when the share was setup as the primary storage being the Array, not Cache as it is now.  Given the new setup, I'm guessing it won't push anything to the cache from the array at this point though, right?

 

I'm certain details could be better, my setup was pretty ancient and I've neglected keeping up to date with new practices.

 

Appreciate the help.

slash-diagnostics-20250315-2249.zip

You have appdata and system all on cache. I usually recommend to also have domains all on cache.

 

appdata, domains, system are the default shares used by Dockers/VMs. If these are on the array, Docker/VM performance will be impacted by slower array writes and reads, and array disks won't be able to spin down since these files are always open.

 

If you do want to move domains to cache, VM Manager in Settings has to be disabled (already is) since nothing can move open files. Set domains share the same way you have appdata and system shares for Primary, Secondary, Mover action.

 

Mar 15 20:51:08 Slash root: Fix Common Problems: Other Warning: Mover logging is enabled

I'm sure you are aware of this since you were trying to figure out mover. All the files mover logs are in your syslog, and so in the Diagnostics you posted, so you don't want to leave it on all the time.

 

Diagnostics says all of your plugins are unknown. Must be something with Community Applications not working with your server when the diagnostics were taken. Have you had problems with CA?

 

d-------s                         shareUseCache="no"      # Share exists on cache, disk1, disk2, disk3

This share has files on cache, but mover ignores shares without any Mover action specified. Should probably be Primary:cache; Secondary:array; Mover action:cache->array

 

I see you already fixed your docker.img size.

 

You should set Minimum Free for cache

https://docs.unraid.net/unraid-os/manual/storage-management/#minimum-free-space-for-a-pool

 

And Minimum Free for each of your User Shares

https://docs.unraid.net/unraid-os/manual/shares/user-shares/#minimum-free-space

Thank you, solid advice, and a good list of things to fix after I got things up and running.  I saw some of these while looking into the problem, but definitely not all of it yet.

9 minutes ago, trurl said:

If you do want to move domains to cache, VM Manager in Settings has to be disabled (already is) since nothing can move open files. Set domains share the same way you have appdata and system shares for Primary, Secondary, Mover action.

 

Yeah, I shut down VMs while getting things going again, I'll probably re-enable it after, and I'll pay attention to the domain share here.

 

10 minutes ago, trurl said:

Diagnostics says all of your plugins are unknown. Must be something with Community Applications not working with your server when the diagnostics were taken. Have you had problems with CA?

No actually.  It's been pretty solid, though today while mover is going it's been slower to pull up.  But all the plugins are identified.  Actually cleaned up some old ones inbetween my original post and the most recent one since the Fix Common Problems plugin actually mentioned some were no longer needed/supported.

 

 

16 minutes ago, trurl said:

This share has files on cache, but mover ignores shares without any Mover action specified. Should probably be Primary:cache; Secondary:array; Mover action:cache->array

 

 

I think this is the very old setup I had, at some point in time, I had it setup to include the cache drive.  I'll probably set that one as well, though it's old enough where I don't think there's anything besides some directory structures on the cache.

 

22 minutes ago, trurl said:

You should set Minimum Free for cache

https://docs.unraid.net/unraid-os/manual/storage-management/#minimum-free-space-for-a-pool

 

And there it is.  That's the failure point.  The docs mention:

Quote

Unraid does not take into account file size when selecting a pool and once Unraid has selected a pool for a file it will not change its mind, and if the file does not fit you get an out-of-space error.

So that's where I hit the failure, I had multiple things being written and they filled it up.

 

I'm still a bit foggy on why it all worked previously, but I'm starting to think it was luck, long term.  I hadn't made much in the way of config change from literally like version 5 or something, so I'm guessing either the migration or that I just missed it in the old configs. 

 

I think that helps wrap it up.  Very much appreciated, and I appreciate the extra look at my settings.  I'm certain there's even more I can do, and I'll get back into the docs tomorrow, but for now, we're powering down due to a major storm coming through.  Thank you again!

6 hours ago, justinmit said:

it's old enough where I don't think there's anything besides some directory structures on the cache

User shares are simply the combined top level folders on array and pools. You can see how much space a share is using on a particular disk/pool by browsing into the disk/pool with Dynamix File Manager. Under Action, click + for the folder (share) and select Calculate.

9 hours ago, justinmit said:

It looked like it was using the cache drive pretty effectively when the share was setup as the primary storage being the Array, not Cache as it is now.

With that setting mover will take no action on the share so if any files end up on the cache for some reason then they remain there.   This is intentional as there are some Use Cases that exploit this behaviour.

2 hours ago, trurl said:

User shares are simply the combined top level folders on array and pools. You can see how much space a share is using on a particular disk/pool by browsing into the disk/pool with Dynamix File Manager. Under Action, click + for the folder (share) and select Calculate.

I did that while looking and it (and still does) show the Cache drive as well as the other drives of the array, but "0 B" on that row.  At the time, with my incorrect understanding, that made logical sense to me since that share hadn't had any movement for an extended period of time, so in a "normal caching" situation, I'd expect it to be migrated at some point.  I'm not sure that's what had happened though anymore, which would just show my understanding was incorrect.  

41 minutes ago, itimpi said:

With that setting mover will take no action on the share so if any files end up on the cache for some reason then they remain there.   This is intentional as there are some Use Cases that exploit this behaviour.

That makes a lot more sense now, the thing I'm still trying to figure out is how I was using it previously which led to the "any files end up on the cache for some reason".  It makes sense there would be use-cases for this, I can think of a way to set it up based on directory splitting and whatnot, but I'm curious about what other legitimate use cases there are, and I might have unintentionally edged into one without knowing it, causing the drive to fill up.

 

32 minutes ago, justinmit said:

which led to the "any files end up on the cache for some reason". 

It could be the behaviour mentioned in the Caution in this part of the online documentation.

39 minutes ago, itimpi said:

It could be the behaviour mentioned in the Caution in this part of the online documentation.

It very well could be.  I used to do a lot more work to try to use rebalancer to partition drive usage, and I do a lot of file operations with Krusader.  I suspect it'll be virtually impossible to determine which route caused it.

1 hour ago, itimpi said:

It could be the behaviour mentioned in the Caution in this part of the online documentation.

Dynamix File Manager avoids this by using the workaround mentioned at that link