TheFullTimer
Members
-
Joined
-
Last visited
Solutions
-
TheFullTimer's post in Ollama & Invoke Docker intermittent shutdown (7.0.0 Beta 4) was marked as the answerFollowing up with this issue, the trouble has been resolved.
If you didn't read the thread, this catches you up:
Working through my thought process, I didn't freeze updates or other work on the system. I don't believe that a reasonable end-user would ignore container updates that might fix their issue, and being in the Beta i did update from 7.0.0 Beta-4 to 7.0.0-rc.1. I had deleted the images, switched to overlay2 storage driver, and rebuilt the containers. When the issue persisted I did look around to other concerns.
Resolution and checking work(?):
The most impactful difference to stability occurred after I checked through my specifications in the Tips and Tweaks plugin. Here I had kept the defaults of "vm.dirty_background_ratio to 3% and vm.dirty_ratio to 5%". As a reference, my testing system has 128 GB of RAM, so I adjusted these down to 1% and 2%, respectively. Again, a full Shutdown and Power On was performed after the change and the problematic Containers ran idle for 24 hours.
As they now seemed to be stable, I began testing them with intermittent workloads to see if they might eventually shutdown, as they had previously. They did continue to operate normally. An update for both Invoke and Ollama had been released, so the containers were updated and continued to function without unexpectedly quitting. I also took the opportunity to update the Checkpoints and Loras of the Invoke container, and took the easiest path of deleting, removing appdata, and cleanly installing with updated Paths.
It had then been ~60 hours and I went back to test the Cache ratios. They were returned to 3% & 5%, respectively, and the system was cleanly power cycled & left to idle with problematic containers running but otherwise inactive. Nothing changed, and the containers remained available for 24hr+. I then returned the setting back to 1% & 2%, where it now remains.
TL:DR
System has 128 GB of RAM.
Adjusting "vm.dirty_background_ratio to 3% and vm.dirty_ratio to 5%" to "vm.dirty_background_ratio to 1% and vm.dirty_ratio to 2%" was most impactful.
Additional Thoughts:
This does raise some questions about those applications potentially crashing if the virtual memory is dirty. To resolve that concern, and 100% confirm the issue, a clean test environment would've been necessary. I didn't really expect the Cache ratio to have a large impact in the first place, but it had effectively corrected the problem on an idle system. It is also possible that work I performed when the cache ratio was lowered had fixed some underlying issue with one or more containers.