(SOLVED) Docker Configs Gone After Reboot

[SquishyDave]

Running Unraid 6.9.2. My docker apps were acting a little funny, so rebooted the server, and then discovered all my docker apps had lost config. No idea how, I hadn't updated, or changed anything recently.

 

I was looking at the appdata share and noticed settings for cache. (cache pool has nothing to select.)

 

Don't have a cache drive, never have, not even briefly. I'm very confused as to where all my dockers settings went. All my data in binhex-sonarr, gone, same with binhex-sabnzbdvpn, as far as I can tell back to factory. binhex-sabnzbdvpn won't even boot because no ovpn file.

 

I don't know linux, I'm a windows sys admin, so I'm comfortable fiddling with computers but don't know much about linux. Is that cache a linux thing and normal for unraid without a cache drive? Why when I browse the share does a lot of it sit on a cache?

 

Is that a red herring and some other thing caused my settings to go away?

 

Apologise for my linux ignorance and for posting at all, I searched google and the forums and couldn't find anyone else with this exact problem or any clear explanation on those cache settings being normal or not.

 

Any insight would be greatly appreciated, it's going to be a lot of work for me to redo all my settings.

Edited August 17, 2021 by SquishyDave

[trurl]

Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post in this thread.

[SquishyDave]

Sorry, yes, here it is.

squishystore-diagnostics-20210817-1251.zip

[itimpi]

Having a user share setting of Use Cache=Prefer is perfectly OK without a cache drive as the system then just reverts to using the array.   If you later add a cache pool it can then switch to using it for that share.

 

I suspect that your problem is caused by one or more of your docker containers having a mapping that refers to /mnt/cache (when it should refer to /mnt/user) and this is creating the location /mnt/cache (in RAM) which the system then subsequently treats as though the cache drive WAS present so starts writing to it.   This means that the contents of /mnt/cache do not survive a reboot, and also start filling up RAM until the system runs out and stops working correctly.

 

Makes me think that the Docker Manager should have a check that will not let you set up a mapping directly under /mnt that does not refer to a physical drive to stop users being able make configurion mistakes of this sort.

 

[SquishyDave]

7 minutes ago, itimpi said:

Having a user share setting of Use Cache=Prefer is perfectly OK without a cache drive as the system then just reverts to using the array.   If you later add a cache pool it can then switch to using it for that share.

 

I suspect that your problem is caused by one or more of your docker containers having a mapping that refers to /mnt/cache (when it should refer to /mnt/user) and this is creating the location /mnt/cache (in RAM) which the system then subsequently treats as though the cache drive WAS present so starts writing to it.   This means that the contents of /mnt/cache do not survive a reboot, and also start filling up RAM until the system runs out and stops working correctly.

 

Makes me think that the Docker Manager should have a check that will not let you set up a mapping directly under /mnt that does not refer to a physical drive to stop users being able make configurion mistakes of this sort.

 

Thank you very much for the reply. Stepping through it myself, I thought something like that might be the case. I'm sure I mapped it wrong way back when I set it up. I understand now, but didn't then.

 

However binhex-sabnzbdvpn only has user mappings, and it lost config. Also why is all of binhex-deluge, jacket and sonarr being listed as located on cache after a reboot? (see first post) All three of deluge, jacket and sonarr have the incorrect cache mapping for data, but the correct user mapping for config, so the config at least should be on the array. And sabnzbvpn has the correct user mapping for both, but some of it's tree is still on the cache somehow.

[itimpi]

47 minutes ago, SquishyDave said:

Thank you very much for the reply. Stepping through it myself, I thought something like that might be the case. I'm sure I mapped it wrong way back when I set it up. I understand now, but didn't then.

 

However binhex-sabnzbdvpn only has user mappings, and it lost config. Also why is all of binhex-deluge, jacket and sonarr being listed as located on cache after a reboot? (see first post) All three of deluge, jacket and sonarr have the incorrect cache mapping for data, but the correct user mapping for config, so the config at least should be on the array. And sabnzbvpn has the correct user mapping for both, but some of it's tree is still on the cache somehow.

If ANY mapping refers to /mnt/cache then this will get created when the container is started thus causing this to get created in RAM which will have undesirable side-effect of fooling the system into thinking it has a cache drive and san start putting files there  (and these do not survive a reboot)

 

[SquishyDave]

Ah I see, so it moved files to the cache afterwards even though they were created under user. Thank you for your knowledge.

 

You've been a massive help, I've learned a lot, and now I can fix up my mappings and spend the time to recreate everything knowing it won't disappear again.

 

I really appreciate you taking the time to explain to me what happened here. Thank you.

[itimpi]

1 hour ago, SquishyDave said:

Ah I see, so it moved files to the cache afterwards even though they were created under user. T

Yes /mnt/user is an amalgamated view of all top level folders on all physical drives under /mnt including any pools   In this case you have 'fooled' the system into thinking that there is a cache device.

 

You need to be aware of this that as any time you decide to operate in Unraid at the physical drive level y/u will find that files/folders show up at both the drive level and the User Share level - they are simply two views of the same folders/files.

 

[SquishyDave]

Thanks again. I'll just say that I think at least the binhex dockers by default point to cache for data, maybe other dockers do too, not sure. See the default value below. And it worked fine for literal years before this issue showed up. If anyone is searching for this issue and comes across this thread they'll know why at least. I'm not sure I actively chose to browse to cache, it seems like that's how they came out of the box.

 

I'll go back and double check these dockers and see if I missed a note telling me to make sure to change this path if I don't have a cache drive. I almost certainly did miss such a note, but if not I'll see if I can let the binhex guys know, but I'm guessing I didn't RTFM closely enough. Thanks again for all your help.

[itimpi]

Since having a cache drive dramatically improves the performance of VMs and docker containers there is probably an implicit assumption that anyone making use of them if likely to have a cache drive.   There is also the fact that accessing a folder on the cache/pool by referencing the drive seems to be slightly more performant than doing it indirectly via a User Share.

 

[trurl]

13 hours ago, SquishyDave said:

Don't have a cache drive, never have, not even briefly.

Some reason you don't want a cache (or other pool)? You don't have to use it to cache user share writes, but it can make some significant differences with dockers/VMs.

 

If you have cache (or other pool) for your dockers/VMs, not only will they perform better (they aren't impacted by slower parity writes, and with SSD even better performance), but if your dockers/VMs are on the array, they will keep array disks spunup since they always have open files.

[SquishyDave]

 

8 hours ago, trurl said:

Some reason you don't want a cache (or other pool)? You don't have to use it to cache user share writes, but it can make some significant differences with dockers/VMs.

 

If you have cache (or other pool) for your dockers/VMs, not only will they perform better (they aren't impacted by slower parity writes, and with SSD even better performance), but if your dockers/VMs are on the array, they will keep array disks spunup since they always have open files.

I'd been using unraid for years before they introduced a cache drive, and when they did I never bothered to put one in because I never noticed a performance issue that required it as a fix. It's plenty fast enough for all my needs. I didn't realise it didn't have to be used for shares, but I don't need any of its advantages so I probably won't bother.

 

Sure it bit me on the arse this time, but I know what happened now and can avoid it in future.

 

So that's the reason, works fine, can't be bothered.

[trurl]

12 hours ago, SquishyDave said:

didn't realise it didn't have to be used for shares

It is almost always used for shares, but not necessarily for caching user share writes. Dockers/VMs have shares (appdata, domains, system) they use by default, and these are usually configured to move to and stay on cache. It is the fact that these are configured to move to cache that was the root of your problem, when you accidentally created cache in RAM and these shares got moved there so they didn't persist.