• [6.8.3] docker image huge amount of unnecessary writes on cache


    S1dney
    • Solved Urgent

    EDIT (March 9th 2021):

    Solved in 6.9 and up. Reformatting the cache to new partition alignment and hosting docker directly on a cache-only directory brought writes down to a bare minimum.

     

    ###

     

    Hey Guys,

     

    First of all, I know that you're all very busy on getting version 6.8 out there, something I'm very much waiting on as well. I'm seeing great progress, so thanks so much for that! Furthermore I won't be expecting this to be on top of the priority list, but I'm hoping someone of the developers team is willing to invest (perhaps after the release).

     

    Hardware and software involved:

    2 x 1TB Samsung EVO 860, setup with LUKS encryption in BTRFS RAID1 pool.

     

    ###

    TLDR (but I'd suggest to read on anyway 😀)

    The image file mounted as a loop device is causing massive writes on the cache, potentially wearing out SSD's quite rapidly.

    This appears to be only happening on encrypted caches formatted with BTRFS (maybe only in RAID1 setup, but not sure).

    Hosting the Docker files directory on /mnt/cache instead of using the loopdevice seems to fix this problem.

    Possible idea for implementation proposed on the bottom.

     

    Grateful for any help provided!

    ###

     

    I have written a topic in the general support section (see link below), but I have done a lot of research lately and think I have gathered enough evidence pointing to a bug, I also was able to build (kind of) a workaround for my situation. More details below.

     

    So to see what was actually hammering on the cache I started doing all the obvious, like using a lot of find commands to trace files that were written to every few minutes and also used the fileactivity plugin. Neither was able trace down any writes that would explain 400 GBs worth of writes a day for just a few containers that aren't even that active.

     

    Digging further I moved the docker.img to /mnt/cach/system/docker/docker.img, so directly on the BTRFS RAID1 mountpoint. I wanted to check whether the unRAID FS layer was causing the loop2 device to write this heavy. No luck either.

    This gave me a situation I was able to reproduce on a virtual machine though, so I started with a recent Debian install (I know, it's not Slackware, but I had to start somewhere ☺️). I create some vDisks, encrypted them with LUKS, bundled them in a BTRFS RAID1 setup, created the loopdevice on the BTRFS mountpoint (same of /dev/cache) en mounted it on /var/lib/docker. I made sure I had to NoCow flags set on the IMG file like unRAID does. Strangely this did not show any excessive writes, iotop shows really healthy values for the same workload (I migrated the docker content over to the VM).

     

    After my Debian troubleshooting I went back over to the unRAID server, wondering whether the loopdevice is created weirdly, so I took the exact same steps to create a new image and pointed the settings from the GUI there. Still same write issues. 

     

    Finally I decided to put the whole image out of the equation and took the following steps:

    - Stopped docker from the WebGUI so unRAID would properly unmount the loop device.

    - Modified /etc/rc.d/rc.docker to not check whether /var/lib/docker was a mountpoint

    - Created a share on the cache for the docker files

    - Created a softlink from /mnt/cache/docker to /var/lib/docker

    - Started docker using "/etc/rd.d/rc.docker start"

    - Started my BItwarden containers.

     

    Looking into the stats with "iotstat -ao" I did not see any excessive writing taking place anymore.

    I had the containers running for like 3 hours and maybe got 1GB of writes total (note that on the loopdevice this gave me 2.5GB every 10 minutes!)

     

    Now don't get me wrong, I understand why the loopdevice was implemented. Dockerd is started with options to make it run with the BTRFS driver, and since the image file is formatted with the BTRFS filesystem this works at every setup, it doesn't even matter whether it runs on XFS, EXT4 or BTRFS and it will just work. I my case I had to point the softlink to /mnt/cache because pointing it /mnt/user would not allow me to start using the BTRFS driver (obviously the unRAID filesystem isn't BTRFS). Also the WebGUI has commands to scrub to filesystem inside the container, all is based on the assumption everyone is using docker on BTRFS (which of course they are because of the container 😁)

    I must say that my approach also broke when I changed something in the shares, certain services get a restart causing docker to be turned off for some reason. No big issue since it wasn't meant to be a long term solution, just to see whether the loopdevice was causing the issue, which I think my tests did point out.

     

    Now I'm at the point where I would definitely need some developer help, I'm currently keeping nearly all docker container off all day because 300/400GB worth of writes a day is just a BIG waste of expensive flash storage. Especially since I've pointed out that it's not needed at all. It does defeat the purpose of my NAS and SSD cache though since it's main purpose was hosting docker containers while allowing the HD's to spin down.

     

    Again, I'm hoping someone in the dev team acknowledges this problem and is willing to invest. I did got quite a few hits on the forums and reddit without someone actually pointed out the root cause of issue.

     

    I missing the technical know-how to troubleshoot the loopdevice issues on a lower level, but have been thinking on possible ways to implement a workaround. Like adjusting the Docker Settings page to switch off the use of a vDisk and if all requirements are met (pointing to /mnt/cache and BTRFS formatted) start docker on a share on the /mnt/cache partition instead of using the vDisk.

    In this way you would still keep all advantages of the docker.img file (cross filesystem type) and users who don't care about writes could still use it, but you'd be massively helping out others that are concerned over these writes.

     

    I'm not attaching diagnostic files since they would probably not point out the needed.

    Also if this should have been in feature requests, I'm sorry. But I feel that, since the solution is misbehaving in terms of writes, this could also be placed in the bugreport section.

     

    Thanks though for this great product, have been using it so far with a lot of joy! 

    I'm just hoping we can solve this one so I can keep all my dockers running without the cache wearing out quick,

     

    Cheers!

     

    • Like 3
    • Thanks 17



    User Feedback

    Recommended Comments



    11 minutes ago, TexasUnraid said:

    Good to know, interesting use case as well. How would the script know that an attack is taking place?

     

    So no gotchas with symlinks on unraid? works just like any other linux system (aka, I can look up generic symlink tutorials online)?

    Very simple really. I took inspiration from the protect against cryptovirus plugin I saw on the app store. I put a few traps on various SMB locations and have a script run periodically to check on those traps if they change. It is just part of my overall strategy.

     

    No gotcha. Have been using symlinks for years.

     

    I think you don't need to wait for 6.9 RC. This is beta 25, not beta 1. It's pretty rock solid for me.

    • Like 1
    Link to comment
    5 minutes ago, testdasi said:

    I think you don't need to wait for 6.9 RC. This is beta 25, not beta 1. It's pretty rock solid for me.

    Yeah, while a few bugs are still normal there should be no show stoppers, and nothing that would put the data at risk, I updated one of my main servers today, and based on these past few hours the NVMe device that was writing about 3TB per day with v6.8 (without any optimizations) is now on course to write less than 200GB per day.

     

    11 minutes ago, boomam said:

    Do we know if there will be a definitive guide created for this issue once 6.9 drops, to help people convert over/transfer data/etc?

    The release notes mention how to re-partition the device(s), backup/restore would depend on your current backups, but you can always move the cache data to the array and then restore, like mentioned in the FAQ for the cache replacement procedure.

    Link to comment
    2 minutes ago, johnnie.black said:

    The release notes mention how to re-partition the device(s), backup/restore would depend on your current backups, but you can always move the cache data to the array and then restore, like mentioned in the FAQ for the cache replacement procedure.

    It needs to be clearer than that - lets not encourage the usual 'linux/FOSS' views of "its easy, here's 10 articles and 3 different blogs that when mushed together easily fix the issue, oh and most of it is out-of-date as a recent kernel change makes 50% of the commands invalid now" ;-).

     

    Joking aside, whilst for us following this thread will be fine, for anyone coming into this new, there needs to be a clear blog/guide somewhere saying what the issue is, listing exact resources and steps to fix it, etc. 

     

    Speaking for myself, i know if i hadn't found a spare bit of time to dive into it a few months back, posted on here/Reddit and got linked to this thread, that i wouldn't have a clue about this.

    So by being more proactive about it, and having it as one article/guide, pinned in the forum, perhaps even included in the monthly news, will ensure that no member of the current or future community can miss this issue or its fix.

    • Like 1
    Link to comment

    So there are no outstanding changes from beta 25 to be implemented in the next release? I have not been following the beta thread very closely.

    Link to comment
    17 hours ago, boomam said:

    It needs to be clearer than that - lets not encourage the usual 'linux/FOSS' views of "its easy, here's 10 articles and 3 different blogs that when mushed together easily fix the issue, oh and most of it is out-of-date as a recent kernel change makes 50% of the commands invalid now" ;-).

     

    Joking aside, whilst for us following this thread will be fine, for anyone coming into this new, there needs to be a clear blog/guide somewhere saying what the issue is, listing exact resources and steps to fix it, etc. 

     

    Speaking for myself, i know if i hadn't found a spare bit of time to dive into it a few months back, posted on here/Reddit and got linked to this thread, that i wouldn't have a clue about this.

    So by being more proactive about it, and having it as one article/guide, pinned in the forum, perhaps even included in the monthly news, will ensure that no member of the current or future community can miss this issue or its fix.

    We can even hope for a nice Spaceinvaderone video on this ?

    Along side the guide it would be great.

    Link to comment

    Yes, Agreed on a proper guild being really handy for this even though I can figure it out the hard way.

     

    Well, I tried to use the symlink for a UD device but ran into an issue. I put the symlinks on the cache drive pointing towards the UD drive. This works fine for everything except that unraid does not see these as valid shares and thus does not let me edit the settings for them.

     

    So I have no idea how it will treat these shares and if it will break docker / appdata since they will not be set to cache only?

    Link to comment
    9 minutes ago, TexasUnraid said:

    Yes, Agreed on a proper guild being really handy for this even though I can figure it out the hard way.

     

    Well, I tried to use the symlink for a UD device but ran into an issue. I put the symlinks on the cache drive pointing towards the UD drive. This works fine for everything except that unraid does not see these as valid shares and thus does not let me edit the settings for them.

     

    So I have no idea how it will treat these shares and if it will break docker / appdata since they will not be set to cache only?

    With due respect, please don't talk about hacks in this topic.

    Link to comment

    Sure thing, although with all due respect, until a few weeks ago when this was officially acknowledged, this entire topic was nothing but "hacks" to work around the issue. 😉

     

    So if hacks are not allowed to be discussed, what is an official option to fix the issue? I really would love one, I really hate going outside officially supported channels.

     

    Thus far the only official option I have heard is wait for 6.9 which is ? months away for an RC and 6+ months away from official release?

     

    6.9 does sound like the fix, I am just uncomfortable using betas on an active server, I will always question if an issue if due to the beta or something else. An RC is not ideal but I would consider it.

    Link to comment
    38 minutes ago, TexasUnraid said:

    Sure thing, although with all due respect, until a few weeks ago when this was officially acknowledged, this entire topic was nothing but "hacks" to work around the issue. 😉

     

    So if hacks are not allowed to be discussed, what is an official option to fix the issue? I really would love one, I really hate going outside officially supported channels.

     

    Thus far the only official option I have heard is wait for 6.9 which is ? months away for an RC and 6+ months away from official release?

     

    6.9 does sound like the fix, I am just uncomfortable using betas on an active server, I will always question if an issue if due to the beta or something else. An RC is not ideal but I would consider it.

    applying space_cache=v2 to your btrfs mounts makes a significant difference in writes, I got a reduction of about 65%, and you can do it live on whatever version you're currently on.

     

    on another note - I did end up installing the beta, bailed on my raid10 btrfs cache and now have three pool devices:

    cache pool(xfs):

    • regular share caching
    • downloads
    • docker-xfs.img
    • plex transcoding

     

    xfs pool:

    • caching for only /usr/mnt/appdata

     

    btrfs pool(single disk):

    • nothing currently

     

    I'm going to give it another 5 days with appdata on the XFS pool, then move it to the BTRFS pool for a week, then add a second disk to BTRFS pool and run that for a week.  With transcoding removed from the equation it should only be IO from normal container operations, so it should be a pretty fair comparison.

     

    1096488086_Screenshotfrom2020-07-2116-21-35.thumb.png.9e85c3c07dc9012f3db7c67a10375ea6.png

     

    Edited by Dephcon
    Link to comment

    Yeah, I am thinking I will just use the space_cache=v2 and move things over to the cache for now and see what kind of writes I get.

     

    If they are tolerable then I will wait for 6.9 RC. If they are still too high I will consider the beta. The multiple cache pools would be really handy for me as well.

     

    Keep us posted on how things go and if you notice any bugs with the beta 🙂

    Edited by TexasUnraid
    Link to comment
    18 hours ago, TexasUnraid said:

    Yeah, I am thinking I will just use the space_cache=v2 and move things over to the cache for now and see what kind of writes I get.

     

    If they are tolerable then I will wait for 6.9 RC. If they are still too high I will consider the beta. The multiple cache pools would be really handy for me as well.

     

    Keep us posted on how things go and if you notice any bugs with the beta 🙂

    Just to give you some additional info based on my friend's use case who had pretty much identical cache load to me on 6.8.3:

     

    2MB/s brfs cache, btrfs docker.img

    650kB/s brfs cache (w/space_cache=v2), btrfs docker.img

    250kB/s xfs cache, btrfs docker.img

     

    So if you're not using or don't need BTRFS RAID, re-formatting your cache disk to XFS makes a huge difference.  That's a change 60TB/yr to 7.5TB/yr.

    Edited by Dephcon
    Link to comment

    Upgrading to 6.9.0-beta25, and wiping and rebuilding cache seems to have fixed the excessive drive writes.  I updated at 1PM yesterday.

     

    Thanks @limetech

     

    Sun Jul 19 00:00:01 CDT 2020	52,349,318 [26.8 TB]
    Sun Jul 19 01:00:01 CDT 2020	52,423,388 [26.8 TB]
    Sun Jul 19 02:00:01 CDT 2020	52,489,648 [26.8 TB]
    Sun Jul 19 03:00:01 CDT 2020	52,555,542 [26.9 TB]
    Sun Jul 19 04:00:01 CDT 2020	52,620,891 [26.9 TB]
    Sun Jul 19 05:00:02 CDT 2020	52,704,944 [26.9 TB]
    Sun Jul 19 06:00:02 CDT 2020	52,781,371 [27.0 TB]
    Sun Jul 19 07:00:01 CDT 2020	52,857,676 [27.0 TB]
    Sun Jul 19 08:00:01 CDT 2020	52,969,998 [27.1 TB]
    Sun Jul 19 09:00:01 CDT 2020	53,060,428 [27.1 TB]
    Sun Jul 19 10:00:02 CDT 2020	53,143,267 [27.2 TB]
    Sun Jul 19 11:00:01 CDT 2020	53,226,597 [27.2 TB]
    Sun Jul 19 12:00:01 CDT 2020	53,302,735 [27.2 TB]
    Sun Jul 19 13:00:02 CDT 2020	53,370,136 [27.3 TB]
    Sun Jul 19 14:00:01 CDT 2020	53,497,045 [27.3 TB]
    Sun Jul 19 15:00:01 CDT 2020	53,570,280 [27.4 TB]
    Sun Jul 19 16:00:02 CDT 2020	53,660,287 [27.4 TB]
    Sun Jul 19 17:00:01 CDT 2020	53,757,767 [27.5 TB]
    Sun Jul 19 18:00:01 CDT 2020	53,843,113 [27.5 TB]
    Sun Jul 19 19:00:01 CDT 2020	54,494,403 [27.9 TB]
    Sun Jul 19 20:00:01 CDT 2020	54,591,716 [27.9 TB]
    Sun Jul 19 21:00:01 CDT 2020	54,684,939 [27.9 TB]
    Sun Jul 19 22:00:01 CDT 2020	54,769,497 [28.0 TB]
    Sun Jul 19 23:00:01 CDT 2020	54,881,700 [28.0 TB]
    Mon Jul 20 00:00:01 CDT 2020	54,962,156 [28.1 TB]
    Mon Jul 20 01:00:01 CDT 2020	55,012,101 [28.1 TB]
    Mon Jul 20 02:00:01 CDT 2020	55,114,507 [28.2 TB]
    Mon Jul 20 03:00:01 CDT 2020	55,199,643 [28.2 TB]
    Mon Jul 20 04:00:01 CDT 2020	55,285,523 [28.3 TB]
    Mon Jul 20 05:00:01 CDT 2020	55,390,072 [28.3 TB]
    Mon Jul 20 06:00:01 CDT 2020	55,492,177 [28.4 TB]
    Mon Jul 20 07:00:01 CDT 2020	55,562,868 [28.4 TB]
    Mon Jul 20 08:00:01 CDT 2020	55,641,502 [28.4 TB]
    Mon Jul 20 09:00:01 CDT 2020	55,709,571 [28.5 TB]
    Mon Jul 20 10:00:01 CDT 2020	55,778,340 [28.5 TB]
    Mon Jul 20 11:00:01 CDT 2020	55,855,175 [28.5 TB]
    Mon Jul 20 12:00:01 CDT 2020	55,937,448 [28.6 TB]
    Mon Jul 20 13:00:01 CDT 2020	56,014,597 [28.6 TB]
    Mon Jul 20 14:00:01 CDT 2020	56,092,328 [28.7 TB]
    Mon Jul 20 15:00:01 CDT 2020	56,156,565 [28.7 TB]
    Mon Jul 20 17:00:01 CDT 2020	56,273,142 [28.8 TB]
    Mon Jul 20 18:00:01 CDT 2020	56,344,795 [28.8 TB]
    Mon Jul 20 19:00:01 CDT 2020	56,364,160 [28.8 TB]
    Mon Jul 20 20:00:01 CDT 2020	56,407,275 [28.8 TB]
    Mon Jul 20 21:00:01 CDT 2020	56,447,405 [28.9 TB]
    Mon Jul 20 22:00:01 CDT 2020	56,471,394 [28.9 TB]
    Mon Jul 20 23:00:02 CDT 2020	56,544,547 [28.9 TB]
    Tue Jul 21 00:00:01 CDT 2020	56,558,841 [28.9 TB]
    Tue Jul 21 01:00:01 CDT 2020	56,572,818 [28.9 TB]
    Tue Jul 21 02:00:01 CDT 2020	56,588,893 [28.9 TB]
    Tue Jul 21 03:00:01 CDT 2020	56,619,137 [28.9 TB]
    Tue Jul 21 04:00:01 CDT 2020	56,649,114 [29.0 TB]
    Tue Jul 21 05:00:01 CDT 2020	56,694,088 [29.0 TB]
    Tue Jul 21 06:00:01 CDT 2020	56,734,883 [29.0 TB]
    Tue Jul 21 07:00:01 CDT 2020	56,740,772 [29.0 TB]
    Tue Jul 21 08:00:01 CDT 2020	56,764,329 [29.0 TB]
    Tue Jul 21 09:00:01 CDT 2020	56,791,261 [29.0 TB]
    Tue Jul 21 10:00:01 CDT 2020	57,390,492 [29.3 TB]
    Tue Jul 21 11:00:02 CDT 2020	57,481,471 [29.4 TB]
    Tue Jul 21 12:00:01 CDT 2020	57,522,137 [29.4 TB]
    	
    Tue Jul 21 14:00:01 CDT 2020	58,216,955 [29.8 TB]
    Tue Jul 21 15:00:01 CDT 2020	58,222,173 [29.8 TB]
    Tue Jul 21 16:00:01 CDT 2020	58,235,354 [29.8 TB]
    Tue Jul 21 17:00:01 CDT 2020	58,270,523 [29.8 TB]
    Tue Jul 21 18:00:01 CDT 2020	58,300,798 [29.8 TB]
    Tue Jul 21 19:00:01 CDT 2020	58,346,858 [29.8 TB]
    Tue Jul 21 20:00:01 CDT 2020	58,382,861 [29.8 TB]
    Tue Jul 21 21:00:01 CDT 2020	58,403,922 [29.9 TB]
    Tue Jul 21 22:00:01 CDT 2020	58,420,439 [29.9 TB]
    Tue Jul 21 23:00:01 CDT 2020	58,493,227 [29.9 TB]
    Wed Jul 22 00:00:02 CDT 2020	58,494,926 [29.9 TB]
    Wed Jul 22 01:00:01 CDT 2020	58,529,097 [29.9 TB]
    Wed Jul 22 02:00:01 CDT 2020	58,556,746 [29.9 TB]
    Wed Jul 22 03:00:01 CDT 2020	58,574,415 [29.9 TB]
    Wed Jul 22 04:00:01 CDT 2020	58,605,297 [30.0 TB]
    Wed Jul 22 05:00:01 CDT 2020	58,632,079 [30.0 TB]
    Wed Jul 22 06:00:01 CDT 2020	58,655,069 [30.0 TB]
    Wed Jul 22 07:00:01 CDT 2020	58,672,137 [30.0 TB]
    Wed Jul 22 08:00:01 CDT 2020	58,689,196 [30.0 TB]
    Wed Jul 22 09:00:01 CDT 2020	58,712,601 [30.0 TB]
    Wed Jul 22 10:00:01 CDT 2020	58,731,743 [30.0 TB]

     

    Link to comment
    1 hour ago, Dephcon said:

    Just to give you some additional info based on my friend's use case who had pretty much identical cache load to me on 6.8.3:

     

    2MB/s brfs cache, btrfs docker.img

    650kB/s brfs cache (w/space_cache=v2), btrfs docker.img

    250kB/s xfs cache, btrfs docker.img

     

    So if you're not using or don't need BTRFS RAID, re-formatting your cache disk to XFS makes a huge difference.  That's a change 60TB/yr to 7.5TB/yr.

    I have 5x 128gb laptop SSD's I was given in a raid5, so kinda need btrfs lol. This is why I had the extra SSD in the array formatted as XFS.

     

    I put docker on the cache this morning, waiting a few hours to see what kind of writes I end up with. Really considering just updating to the beta and creating a 2nd cache to be done with this.

    Link to comment
    27 minutes ago, Wavey said:

    @StevenDWhat is that output you are using to monitor the amount of writes?

    There is probably a better way, but I just have a script run at the top of every hour

    date >> /mnt/cache/cache1.txt
    smartctl -a -d nvme /dev/nvme0n1 | grep "Units Written" >> /mnt/cache/cache1.txt
    date >> /mnt/cache/cache2.txt
    smartctl -a -d nvme /dev/nvme1n1 | grep "Units Written" >> /mnt/cache/cache2.txt
    

     

    • Like 1
    Link to comment

    Upgraded one of my main servers to b25 and re-formatted the cache device with the new alignment and consider this issue fixed for me, I have a single btrfs cache device with 3 Windows VMs always running, as well as the docker image and appdata:

     

    v6.8 was writing about 3TB a day

    v6.8 with space cache v2 brought it down to a more reasonable 700GB a day

    v6.9-beta25 with the new alignment brought it down even further to 191.87GB in the last 24 hours

     

    While 192GB a day is still considerable and I know that if I went with xfs it would less, as the previously linked study found I believe we must accept btrfs will always have higher write amplification due to being a COW filesystem, and while I don't need a pool for this I rely on btrfs snapshots and send/receive for my backups, so I can live with these daily writes, just couldn't with 3TB a day, that was just crazy.

    Link to comment
    5 hours ago, johnnie.black said:

    v6.8 was writing about 3TB a day

    v6.8 with space cache v2 brought it down to a more reasonable 700GB a day

    v6.9-beta25 with the new alignment brought it down even further to 191.87GB in the last 24 hours

    Is the alignment issue something regarding NVMe? I recall something about that, but I only have SATA so I've skimmed over most of it.

     

    *Edit*  Before you answer, I noticed my OG Cache pool is 4K aligned and my new pool devices are 1M aligned so i guess it's for all SSDs?

     

    *edit2* that's a 93% decrease in writes!  I'm still testing with XFS but I'd much rather go back to BTRFS RAID10 or a pair of BTRFS RAID1 for protection, assuming it's not a massive difference from XFS.

    Edited by Dephcon
    Link to comment
    23 minutes ago, Dephcon said:

    Is the alignment issue something regarding NVMe?

    All SSDs, my cache is NVMe but earlier I tested on a regular SSD and the difference was similar, though it can vary with brand/model.

    Link to comment
    25 minutes ago, Dephcon said:

    I'm still testing with XFS

    Based on some quick earlier tests xfs would still write much less, I would estimate at least 5 times less in my case, still I can live with 190GB instead of 30/40GB a day so I can have checksums and snapshots.

    • Thanks 1
    Link to comment

    If I install 6.9 beta 25 and change my cache to the 1 MiB Partition Alignment is it still possible to roll back to 6.8.3 without changing the new alignment?

    Link to comment
    12 minutes ago, Gragorg said:

    I install 6.9 beta 25 and change my cache to the 1 MiB Partition Alignment is it still possible to roll back to 6.8.3 without changing the new alignment?

    Unfortunately no, you'd need to re-format.

    Link to comment
    17 minutes ago, johnnie.black said:

    Based on some quick earlier tests xfs would still write much less, I would estimate at least 5 times less in my case, still I can live with 190GB instead of 30/40GB a day so I can have checksums and snapshots.

    damn that's still pretty significant. 

     

    I'm really torn on this whole issue. I'm super butt-hurt over how much wear I've been putting on cache SSDs over the years and want to limit it as much as possible, but I also would prefer to not to ever restore my pool devices from backup, reconfigure containers, etc.

    Edited by Dephcon
    Link to comment
    1 minute ago, Dephcon said:

    damn that's still pretty significant. 

    It is, but like the previously linked study found, some write amplification is unavoidable:

     

    imagem.png.d367586d792e53b74f7b191ab7c61ab0.png

     

     

     

    As long as it's not ridiculous like before I'm fine with it, but anyone that doesn't need a pool or the other btrfs features might as well stick with xfs.

    Link to comment

    Just switched appdata from xfs to a single disk btrfs, it's about 2x the writes:

    662051737_Screenshotfrom2020-07-2413-03-45.thumb.png.a85ec49ebbf8be1753c325eb5d30a74a.png

     

    ignore the "avg" it's btrfs heavy as it starts with all my containers starting back up.  if i exclude the container boot-up, until now it's AVG is ~121kB/s and my 2hr average on XFS before the cut-over was 49kB/s.  So that's a 2.5x difference.

    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.