-
Cache drive full, mover fails, docker stack won't start
It very well could be. I used to do a lot more work to try to use rebalancer to partition drive usage, and I do a lot of file operations with Krusader. I suspect it'll be virtually impossible to determine which route caused it.
-
Cache drive full, mover fails, docker stack won't start
I did that while looking and it (and still does) show the Cache drive as well as the other drives of the array, but "0 B" on that row. At the time, with my incorrect understanding, that made logical sense to me since that share hadn't had any movement for an extended period of time, so in a "normal caching" situation, I'd expect it to be migrated at some point. I'm not sure that's what had happened though anymore, which would just show my understanding was incorrect. That makes a lot more sense now, the thing I'm still trying to figure out is how I was using it previously which led to the "any files end up on the cache for some reason". It makes sense there would be use-cases for this, I can think of a way to set it up based on directory splitting and whatnot, but I'm curious about what other legitimate use cases there are, and I might have unintentionally edged into one without knowing it, causing the drive to fill up.
-
Cache drive full, mover fails, docker stack won't start
Thank you, solid advice, and a good list of things to fix after I got things up and running. I saw some of these while looking into the problem, but definitely not all of it yet. Yeah, I shut down VMs while getting things going again, I'll probably re-enable it after, and I'll pay attention to the domain share here. No actually. It's been pretty solid, though today while mover is going it's been slower to pull up. But all the plugins are identified. Actually cleaned up some old ones inbetween my original post and the most recent one since the Fix Common Problems plugin actually mentioned some were no longer needed/supported. I think this is the very old setup I had, at some point in time, I had it setup to include the cache drive. I'll probably set that one as well, though it's old enough where I don't think there's anything besides some directory structures on the cache. And there it is. That's the failure point. The docs mention: So that's where I hit the failure, I had multiple things being written and they filled it up. I'm still a bit foggy on why it all worked previously, but I'm starting to think it was luck, long term. I hadn't made much in the way of config change from literally like version 5 or something, so I'm guessing either the migration or that I just missed it in the old configs. I think that helps wrap it up. Very much appreciated, and I appreciate the extra look at my settings. I'm certain there's even more I can do, and I'll get back into the docs tomorrow, but for now, we're powering down due to a major storm coming through. Thank you again!
-
Cache drive full, mover fails, docker stack won't start
Thank you for that. I had been looking through the docs, and once I realized the mover mechanism, some of it was making more sense. Like the cache page there talks specifically about how it's necessary to set the limits otherwise it sounds like what I was seeing is expected. But I couldn't find anything on how the cache was working prior to the new settings I'm using. It looked like it was using the cache drive pretty effectively when the share was setup as the primary storage being the Array, not Cache as it is now. Given the new setup, I'm guessing it won't push anything to the cache from the array at this point though, right? I'm certain details could be better, my setup was pretty ancient and I've neglected keeping up to date with new practices. Appreciate the help. slash-diagnostics-20250315-2249.zip
-
Cache drive full, mover fails, docker stack won't start
So after looking through more threads on the mover, I saw a lot of discussion about the settings for mover with the shares setup to use cache, but move to the array. Finally figured out that if I have the share setup to "Array" as it has been, it looks like it uses the cache, but gives no settings for mover. But when I set it for New files to be created in Cache, and secondary is array, the mover action "Cache -> Array" is available, and setting that moves the files off to the array. I did that and triggered the Mover process and it's in the process of moving a lot of files to the array. However, is this the proper setup? Shouldn't it handle moving things back and forth to the cache as needed instead? I presume with this setting, if the file's been moved, it's not going to be cached if I go back out to read the files again, so this is more of a Fast Write to Slow Store setup than a real caching setup. Is there some documentation that explains in better detail? This post was super helpful to me to explain that this seems to lay out the table of options and helped with why the mover was doing (and not doing) what it was doing: But it also feels like something off on the setup. I had 750G listed as the total size of files on the cache drive that's 500G, and mover wasn't engaging (seems like it's due to that configuration), but I'm not sure about how the files got to the cache drive in the first place in that case then. It seems like it was the expected caching mechanism that put it there, but from the post above, it sounds like those were orphaned files. I do think part of the problem is the fact that the docker image is over 300G, so I'll need to work on trying to shrink it as well.
-
justinmit started following Cache drive full, mover fails, docker stack won't start
-
Cache drive full, mover fails, docker stack won't start
Hey all, Had an issue start happening yesterday morning, but I'm having issues getting things moving so I can resolve. Mar 15 13:04:06 Slash webgui: File_put_contents_atomic failed to write / rename /mnt/cache/1160867511.tmp Mar 15 13:04:06 Slash root: Fix Common Problems: Error: Unable to write to cache I've been fiddling with Docker recently and I've got some issues I know I need to resolve with Docker sizing (or something misconfigured), cache drive sizing, and whatnot, but I can't get mover to work, and the Docker stack won't start, and the cache drive seems to be in RO. I did see some threads about the cache drive I have (or a newer version of it, really) that looks like it might have a firmware issue too, though since they were talking about some newer Crucial drives, I'm not sure if it's a red herring. Thank you to anyone who can help take a look and give some guidance. slash-diagnostics-20250315-1322.zip Edit 1: Trying to look through things, since I suspected something was up with the SSD cache drive, I've brought the array into maint mode and done the filesystem scan and extended scan and it looks like it's clean. Can I erase the cache drive pool to get things restarted? I'm okay losing cached data and rebuilding my dockers/VMs as needed, as the primary purpose is still the rest of the data. Edit 2: I was able to manually delete a few files from the cache drive which freed up enough for the docker stack to start. Mover is no longer throwing errors, but it's not doing a heck of a lot either. I've enabled logging and it's starting and finishing instantly, but no errors. I'm guessing that it's not healthy to sit with only 8.57G free after a reboot. I've also used the time to go through some of Spaceinvader One's videos and check the image size and whatnot, cleanup excess docker images, etc. The docker image is setup to a max of 300G (apparently I did that way back when, it's set at 100G now), but it looks like the docker.img file is 322G. Looking at the cache share, it's calculating some 750G of space usage on the cache drive though.
justinmit
Members
-
Joined
-
Last visited