Marshalleq Posted December 30, 2020 Share Posted December 30, 2020 Hi, for years I've begrudgingly run a btrfs docker image, which was hosted on an SSD with underlying XFS or ZFS no problem. There has been for a while the option of instead storing this into a folder, which I'd prefer for various reasons. However, it seems that docker is deciding that it won't store files on the file system, I assume because it's currently formatted ZFS and is looking for XFS or BTRFS underneath. I don't understand why it would care, given that it's no longer an image what file system is underneath? The below from the docker.log file failed to start daemon: error initializing graphdriver: prerequisites for driver not satisfied (wrong filesystem?) Anyone know why? I'd like to log a ticket, but thought I'd ask here first. Thanks. Quote Link to comment
trurl Posted December 30, 2020 Share Posted December 30, 2020 You probably just aren't specifying the path correctly to get it to use a folder. Had to dig back a little to find when this was introduced in 6.9b24: Quote Link to comment
Marshalleq Posted December 30, 2020 Author Share Posted December 30, 2020 Thanks for the reply and link, I'm just setting it up under a folder on a ZFS drive via it's location, so /mnt/mountpoint/docker/docker/ When I point it at non zfs i.e. /mnt/user/docker/docker/ it works, but if I change it to the zfs one I get the error. So based on the error in the log, I assume it's detecting ZFS somehow, however, I didn't check to see what the log says on XFS, perhaps it's the same and the cause is something else. Either way, I strongly suspect it's some unraid config / bug here. Quote Link to comment
Squid Posted December 30, 2020 Share Posted December 30, 2020 There's different drivers for docker to be on different filesystems. AFAIK, there's only the 2 included in the OS -> btrfs and xfs. You could probably do a feature req to have the driver for zfs included for docker Your other option is either a btrfs or xfs image stored on the zfs pool. 1 Quote Link to comment
Marshalleq Posted December 30, 2020 Author Share Posted December 30, 2020 Thanks Squid. I half thought I read that, but just got lost wondering why it even needed to know it's underlying filesystem and therefore a driver, therefore thought I must have misunderstood. Do you know why it needs a driver for an underlying filesystem? I'll do a feature request, if Unraid actually does end up including ZFS it won't harm to have it done already. Quote Link to comment
OneMeanRabbit Posted March 17, 2021 Share Posted March 17, 2021 (edited) @Marshalleq I'm very new to UnRaid & ZFS, but the material here is AMAZING! Squid mentioned above that we can create a .xfs/.btrfs/ext4 volume on top of a ZFS zvol. I've done it by creating both a btrfs & xfs volume that doesn't appear to have any issues, currently using the btrfs one. There was some fun in getting it mounted, but this link set me straight finally! Other sites said to mkfs.btrfs /dev/zd0p1 (which didn't work) vs /dev/zvol/POOL/ZVOL. I did run into issues on how to mount the damned ZVOL after...kinda forgot which worked now, but it was a bit of a hassle. Hope this helps, as I can now has any FS I need on the frontend, but with all of the power of ZFS on the backend. Edited March 17, 2021 by OneMeanRabbit Quote Link to comment
glennv Posted March 17, 2021 Share Posted March 17, 2021 Not sure if this is still relevant as 2 days ago (not knowing about this post) i moved from docker image to folder on zfs (dataset)and everything works just fine. I run 6.9.1 build with zfs included via kernel helper docker. edit : directly on zfs so no zvol stuff as mentioned in other posts Quote Link to comment
Marshalleq Posted March 17, 2021 Author Share Posted March 17, 2021 14 hours ago, OneMeanRabbit said: @Marshalleq I'm very new to UnRaid & ZFS, but the material here is AMAZING! Squid mentioned above that we can create a .xfs/.btrfs/ext4 volume on top of a ZFS zvol. I've done it by creating both a btrfs & xfs volume that doesn't appear to have any issues, currently using the btrfs one. There was some fun in getting it mounted, but this link set me straight finally! Other sites said to mkfs.btrfs /dev/zd0p1 (which didn't work) vs /dev/zvol/POOL/ZVOL. I did run into issues on how to mount the damned ZVOL after...kinda forgot which worked now, but it was a bit of a hassle. Hope this helps, as I can now has any FS I need on the frontend, but with all of the power of ZFS on the backend. You have a functioning docker 'folder' on ZFS filesystem, rather than a docker folder on BTRFS / XFS and then on ZFS? Quote Link to comment
Forusim Posted May 1, 2021 Share Posted May 1, 2021 Are there any benefits in migrating from btrfs docker.img to directory? Will it use less space, because a fixed 20 GB allocation is not required anymore? Is it possible to transfer all the data in existing docker.img to docker directory? Quote Link to comment
dnLL Posted May 2, 2021 Share Posted May 2, 2021 2 hours ago, Forusim said: Are there any benefits in migrating from btrfs docker.img to directory? Will it use less space, because a fixed 20 GB allocation is not required anymore? Is it possible to transfer all the data in existing docker.img to docker directory? I have the exact same question as your first one, but I can reply to the other 2 since I made the switch following a recommendation in one of SpaceInvader's video (he doesn't say why however). It doesn't really use less space, at least for me. I'm not sure what's using 17GB of space honestly, I'm running 4 dockers: qbittorentvpn, plex, youtube-dl and duplicati. root@server:~# du -h /mnt/cache/system/docker/docker/ | sort -hr | head 17G /mnt/cache/system/docker/docker/docker/btrfs/subvolumes 17G /mnt/cache/system/docker/docker/docker/btrfs 17G /mnt/cache/system/docker/docker/docker 17G /mnt/cache/system/docker/docker/ 986M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init 986M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3 986M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/681dbb488afc61adc745b592f266b76ac9cd52b6689fe7a7bb9acd3eb117ad4f 964M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3/usr 964M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init/usr 964M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/681dbb488afc61adc745b592f266b76ac9cd52b6689fe7a7bb9acd3eb117ad4f/usr As for whether you can transfer all the data... well, there is no data in your docker.img file technically, all you need are your customized user templates located in /boot/config/plugins/dockerMan/templates-user/. Then you just add your containers again, it should remember their configuration and boot normally. 1 Quote Link to comment
Marshalleq Posted May 2, 2021 Author Share Posted May 2, 2021 One of your dockers almost certainly has a cache / temporary / cancelled files or similar contained within in. My guess would be qbittorrent vpn. Since you have it in a folder, you should be able to figure that out with du -sh /folderpath. Also, I do believe that you would save space - because the difference between the 20GB image file and the actual docker space (in your case 3GB difference) is still a difference. The main exception to this is when you have compression on the filesystem hosting the image such as ZFS - it will recognise the image files blank space and compress it effectively to zero. Quote Link to comment
dnLL Posted May 2, 2021 Share Posted May 2, 2021 4 minutes ago, Marshalleq said: One of your dockers almost certainly has a cache / temporary / cancelled files or similar contained within in. My guess would be qbittorrent vpn. Since you have it in a folder, you should be able to figure that out with du -sh /folderpath. Well if I go deeper I can see from one of the directory to another that similar lib files are here. root@server:~# ls -l /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/ | wc -l 52 root@server:~# du -sh /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init/usr/lib/* | sort -hr | head 95M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init/usr/lib/python3.9 90M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init/usr/lib/libLLVM-11.so 53M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init/usr/lib/libgo.so.16.0.0 36M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init/usr/lib/dri 32M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init/usr/lib/libgphobos.so.1.0.0 28M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init/usr/lib/perl5 28M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init/usr/lib/libicudata.so.69.1 20M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init/usr/lib/libstdc++.so.6.0.28 19M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init/usr/lib/udev 16M /mnt/cache/system/docker/docker/docker/btrfs/subvolumes/f241e33ad5c10ab40fdc2f4c33e5a3cc6b8a0b7dffa7cad22188a348a9de96c3-init/usr/lib/d3d I'm just very confused by what are the subvolumes exactly? I have 4 dockers but there are 52 subvolumes directories? Also, how do I know which subvolume belongs to which docker? At any rate, I don't really mind docker consuming 17G of space, it's curiosity more than anything else considering that all my dockers are configured to have their data into the appdata and they do consume 17GB right off after booting which seems like an insane amount of space for something that should theoretically be lightweight. Quote Link to comment
Marshalleq Posted May 2, 2021 Author Share Posted May 2, 2021 I’m pretty sure I was able to run about 30 dockers in less than 5G of space - but I’d have to check that. What you can do is just stop the docker service, rename the image file / directory and restart the service. If configured as you say the image file / directory should be much smaller. I have seen some docker images not playing nice with the appdata by default so with this you can see which folder is growing amd investigate from there. if one folder is growing you will likely find you’ll have to increase your image file size or allow more space for the folder Indefinitely. Quote Link to comment
dnLL Posted May 2, 2021 Share Posted May 2, 2021 2 hours ago, Marshalleq said: I’m pretty sure I was able to run about 30 dockers in less than 5G of space - but I’d have to check that. What you can do is just stop the docker service, rename the image file / directory and restart the service. If configured as you say the image file / directory should be much smaller. I have seen some docker images not playing nice with the appdata by default so with this you can see which folder is growing amd investigate from there. if one folder is growing you will likely find you’ll have to increase your image file size or allow more space for the folder Indefinitely. I deleted the docker directory and downloaded every docker 1 by 1. youtube-dl: 1.1 G duplicati: 3.4 G qbittorrentvpn: 7.5 G linuxserver/plex: 4 G Total of 17GB exactly. I'm a little bit confused because with the same dockers my docker.img was 4.1GB before updating to 6.9.X. I am... unsure what's going on exactly to say the least, especially if you have 30 dockers in less than 5G, that's insane in comparison. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.