appdata versioning

[NAS]

Currently every docker container posted here requires that some data be maintained outwith the container itself. Typically this is config files or databases stored in the appdata folder.

 

However whilst users are free to use docker inherent versioning control to roll about releases with each change comes the risk that the appdata will be modified in such a way as to become incompatible withe other container versions. i.e. if you update a container and it updates say a database which is incompatible with the app in the previous container version.

 

This is a fundamental problem with docker which presumes (in my opinion incorrectly) that almost all relevant data is kept in the container or that the config files outwith the container are mostly cross version compatible. Essentially they leave this potential mess up to the user to handle.

 

Suggest we leverage rsync, btrfs snapshots or some other method to version and maintain an appdata structure that matches the container structure. In this way if you update a container a point in time snapshot of the appdata is kept allowing you to recover if required.

 

 

[jumperalex]

Hmm an excellent point. Now I know you are referencing docker but I think it still applies to how most of us run VM's and visa versa.

 

So right now I accomplish this with my VM, in a fasion, by making weekly rotating copies (5 total) of my VM.img and data.img files.  If I'm ever worried about the freshness of data.img before a big VM.img change I fire off the backup script manually. All of this is done on my small SSD which is essentially full when seen outside the array, but my data.img file is internally only about 25% full. It is also ext4 right now.

 

However, if my drive were larger I should be able to reformat as btrfs, and then use COW to run snapshots of data.img and even VM.img. Right?  I'd still want backups on a different device (in my case the array) but if I ever have trouble with the new version I can rollback the snap shot.

 

What was the reason unraid moved away from btrfs as the disk format for dock containers and move to btrfs inside the loopback image? Will that impact what we're talking about here?

[jonp]

We didn't "move away" from btrfs.  We moved away from requiring btrfs on a physical device because we wanted to give folks with existing setups an easy way to use docker without having to reformat an entire drive with btrfs. The virtual disk we create for docker uses btrfs itself.

[jumperalex]

Oh yes I know that. Sorry if the way I wrote it was unclear.

... and move to btrfs inside the loopback image?

 

That said, I thought there were other reasons, beyond not having to format the entire drive, you moved to using btrfs inside the virtual disk. I'm just asking if any of those reasons would impact our ability to use it at the disk level to give us access to snap-shots of the virtual loopback image.  Meanwhile still using the loopback functionality with btrfs inside the virtual drive. so basically a BTRFS image stored on a BTRFS partition.

 

Am I making any sense at all?

[itimpi]

Oh yes I know that. Sorry if the way I wrote it was unclear.

... and move to btrfs inside the loopback image?

 

That said, I thought there were other reasons, beyond not having to format the entire drive, you moved to using btrfs inside the virtual disk. I'm just asking if any of those reasons would impact our ability to use it at the disk level to give us access to snap-shots of the virtual loopback image.  Meanwhile still using the loopback functionality with btrfs inside the virtual drive. so basically a BTRFS image stored on a BTRFS partition.

 

Am I making any sense at all?

There were definitely other issues such as ability to back up docker; compatibility with mover; ability to store docker images easily on the data drives.  There are some threads talking about this issue in the docker sub-forum area.

 

As far as btrfs level functionality is concerned you should have exactly the same functionality with the loop approach as you would have with a btrfs formatted physical drive.

[NAS]

I think we went quite a bit off topic there.

 

To be clear this is not a shortcoming of unRAID i am highlighting but of Docker itself and something we may be able to improve.

[itimpi]

I think we went quite a bit off topic there.

 

To be clear this is not a shortcoming of unRAID i am highlighting but of Docker itself and something we may be able to improve.

One could argue that this is an issue that is nothing to do with docker, but is a fundamental flaw in the apps themselves?

[jumperalex]

Well I was trying to be on-topic.  What I was asking, or trying to understand in relation to your question is:

 

Given how dockers are using now, with loopback devices to solve all the problems itimpi mentioned, is there any reason we can't still format our physical drive which stores the docker.img and appdata.img files as BTRFS so that we can take advantage of snap-shoting of those *.img files?

 

I'm going to guess we'd still have the problem of not being able to backup to the array (iirc it causes an explosion of store file size due to COW), but would there be any other limitations? or have we defeated the entire purpose of moving docker inside a *.img file?

[jumperalex]

As far as btrfs level functionality is concerned you should have exactly the same functionality with the loop approach as you would have with a btrfs formatted physical drive.

 

OK I probably glossed over this too fast before writing my post above. So internal to the appdata.img file, which is btrfs, nothing should stop someone from creating a snapshot of say /mnt/appdata/[MySpecialFlowerApplicationsData]. The application will just hum along doing its thing, and if we need to rollback, stop the app, restore the older snapshot using btrfs tools, restart app, profit?

[NAS]

Well I was trying to be on-topic.  What I was asking, or trying to understand in relation to your question is:

 

Given how dockers are using now, with loopback devices to solve all the problems itimpi mentioned, is there any reason we can't still format our physical drive which stores the docker.img and appdata.img files as BTRFS so that we can take advantage of snap-shoting of those *.img files?

 

I'm going to guess we'd still have the problem of not being able to backup to the array (iirc it causes an explosion of store file size due to COW), but would there be any other limitations? or have we defeated the entire purpose of moving docker inside a *.img file?

Yup that would be another, actually quite easy, way to do it. The downside is you would need to take the whole img offline to backup which means every docker has stop for any docker upgrade.

 

Note that this would not really be the same as a BTRFS snapshot. What you describe is more like a backup as opposed to subvolumes/snapshots.

 

However TBH there is value in both approaches.

 

 

I think we went quite a bit off topic there.

 

To be clear this is not a shortcoming of unRAID i am highlighting but of Docker itself and something we may be able to improve.

One could argue that this is an issue that is nothing to do with docker, but is a fundamental flaw in the apps themselves?

Yeah you are right you could although to be fair since docker is the tool that enables the roll-about and the apps are not its more a docker responsibilty.

 

Net effect is the same though.

[jumperalex]

Well I was trying to be on-topic.  What I was asking, or trying to understand in relation to your question is:

 

Given how dockers are using now, with loopback devices to solve all the problems itimpi mentioned, is there any reason we can't still format our physical drive which stores the docker.img and appdata.img files as BTRFS so that we can take advantage of snap-shoting of those *.img files?

 

I'm going to guess we'd still have the problem of not being able to backup to the array (iirc it causes an explosion of store file size due to COW), but would there be any other limitations? or have we defeated the entire purpose of moving docker inside a *.img file?

Yup that would be another, actually quite easy, way to do it. The downside is you would need to take the whole img offline to backup which means every docker has stop for any docker upgrade.

 

Note that this would not really be the same as a BTRFS snapshot. What you describe is more like a backup as opposed to subvolumes/snapshots.

 

However TBH there is value in both approaches.

 

Right but if we are trying to solve the problem of breaking data structure compatibility due to an updated app, well aren't you stopping docker anyway? So then how different / problematic is it to take a "backup" snapshot?  Or am I missunderstanding snapshoting?  I mean won't any active application, like say, Plex have heartache with its database just suddenly changing without itself being paused/shutdown?

 

Also how often does that really happen though? I almost have to side with itimpi on that. If an app can't update without data structure problems isn't it the app's responsibility to figure it out (first-run data scrub for example)??

 

[thinking] ... or are you considering the possibility of having two containers, with different versions, accessing the same data? Would that be wise at anytime unless it were specifically designed for multi-user heterogeneous access?

[NAS]

To backup an img file you need to dismount the file. If you have 10 dockers running in that img you dont want to have to stop all 10 just to backup one docker appdata folder.

 

This is why btrfs snapshots would be better. Essentially the way docker itself works extended to appdata.

 

The issue is not rolling forward apps its rolling back or sideways. This is a docker feature so it is not the responsibility of the app to be able to roll back 1,2 to even 1000 versions which sounds crazy but isnt if you are running from a git repo.

 

A user should KNOW that if they run a docker and it munges their appdata they can revert both the docker container and the appdata seamlessly.