Container data paths and backup questions

[MrCravon]

I have all my docker appdata and most paths defined in the docker templates located on my cache drive. I have been relying on the CA Backup / Restore Appdata plugin to ensure the data is safely stored on the array in case of a failure of the cache drive. I recently changed my cache drive to one with a larger size and had some hiccups with file permissions resulting in some files left on the old drive not moved to the new drive etc. I fixed it by restoring a backup from some days earlier. The only problem is my MongoDB container which won't start now due to corrupted files. I read up on it and it seems that the WiredTiger.wt gest corrupted when CA Backup / Restore Appdata does it's thing. Now this post is not about fixing my MongoDB install, but rather about container path mapping. After these issues I started thinking about how my container paths are mapped to paths on my server. Two examples are my GitLab and my MongoDb instances.

 

Generally should the data paths of containers be located on the array instead of the cache drive?

 

Also I see the GitLab instance points to /mnt/cache/appdata/ instead of /mnt/user/appdata/ is that a problem?

 

[trurl]

/mnt/user is the user shares, which includes cache. If your appdata is cache-prefer it will prefer to be on cache anyway, but if you don't expect it to fill up cache then using /mnt/cache is OK and better in some ways.

[Squid]

1 hour ago, MrCravon said:

that the WiredTiger.wt gest corrupted when CA Backup / Restore Appdata does it's thing

Where did you read this?

[MrCravon]

1 hour ago, Squid said:

Where did you read this?

 

Well I didn't read that exactly. But after restoring the backups the MongoDB container wouldn't start. The log produces an error like this:

 

2021-06-25T12:07:32.538+0200 E STORAGE [initandlisten] WiredTiger error (0) [1624615652:538965][1:0x14e9929f5a80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1624615652:538965][1:0x14e9929f5a80], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

Reading up on that it seems this happen if you try to use files that was backed up without stopping all writes to the mongod before copying the files. I read that here under Back Up with cp or rsync: (https://docs.mongodb.com/manual/core/backups/) I realize the plugin uses tar? But I suspect the same applies?

 

So my deduction (which of course may be wrong) is that the MongoDB instance was not shut down or not shut down properly before making the backups resulting in the WiredTiger error. I have only briefly looked in to this though.

 

I found this: https://medium.com/@imunscarred/repairing-mongodb-when-wiredtiger-wt-file-is-corrupted-9405978751b5

which I plan to read through and try this weekend. 

Edited June 25, 2021 by MrCravon