Jump to content
  • [6.8.3] docker image huge amount of unnecessary writes on cache


    S1dney
    • Urgent

    Hey Guys,

     

    First of all, I know that you're all very busy on getting version 6.8 out there, something I'm very much waiting on as well. I'm seeing great progress, so thanks so much for that! Furthermore I won't be expecting this to be on top of the priority list, but I'm hoping someone of the developers team is willing to invest (perhaps after the release).

     

    Hardware and software involved:

    2 x 1TB Samsung EVO 860, setup with LUKS encryption in BTRFS RAID1 pool.

     

    ###

    TLDR (but I'd suggest to read on anyway 😀)

    The image file mounted as a loop device is causing massive writes on the cache, potentially wearing out SSD's quite rapidly.

    This appears to be only happening on encrypted caches formatted with BTRFS (maybe only in RAID1 setup, but not sure).

    Hosting the Docker files directory on /mnt/cache instead of using the loopdevice seems to fix this problem.

    Possible idea for implementation proposed on the bottom.

     

    Grateful for any help provided!

    ###

     

    I have written a topic in the general support section (see link below), but I have done a lot of research lately and think I have gathered enough evidence pointing to a bug, I also was able to build (kind of) a workaround for my situation. More details below.

     

    So to see what was actually hammering on the cache I started doing all the obvious, like using a lot of find commands to trace files that were written to every few minutes and also used the fileactivity plugin. Neither was able trace down any writes that would explain 400 GBs worth of writes a day for just a few containers that aren't even that active.

     

    Digging further I moved the docker.img to /mnt/cach/system/docker/docker.img, so directly on the BTRFS RAID1 mountpoint. I wanted to check whether the unRAID FS layer was causing the loop2 device to write this heavy. No luck either.

    This gave me a situation I was able to reproduce on a virtual machine though, so I started with a recent Debian install (I know, it's not Slackware, but I had to start somewhere ☺️). I create some vDisks, encrypted them with LUKS, bundled them in a BTRFS RAID1 setup, created the loopdevice on the BTRFS mountpoint (same of /dev/cache) en mounted it on /var/lib/docker. I made sure I had to NoCow flags set on the IMG file like unRAID does. Strangely this did not show any excessive writes, iotop shows really healthy values for the same workload (I migrated the docker content over to the VM).

     

    After my Debian troubleshooting I went back over to the unRAID server, wondering whether the loopdevice is created weirdly, so I took the exact same steps to create a new image and pointed the settings from the GUI there. Still same write issues. 

     

    Finally I decided to put the whole image out of the equation and took the following steps:

    - Stopped docker from the WebGUI so unRAID would properly unmount the loop device.

    - Modified /etc/rc.d/rc.docker to not check whether /var/lib/docker was a mountpoint

    - Created a share on the cache for the docker files

    - Created a softlink from /mnt/cache/docker to /var/lib/docker

    - Started docker using "/etc/rd.d/rc.docker start"

    - Started my BItwarden containers.

     

    Looking into the stats with "iotstat -ao" I did not see any excessive writing taking place anymore.

    I had the containers running for like 3 hours and maybe got 1GB of writes total (note that on the loopdevice this gave me 2.5GB every 10 minutes!)

     

    Now don't get me wrong, I understand why the loopdevice was implemented. Dockerd is started with options to make it run with the BTRFS driver, and since the image file is formatted with the BTRFS filesystem this works at every setup, it doesn't even matter whether it runs on XFS, EXT4 or BTRFS and it will just work. I my case I had to point the softlink to /mnt/cache because pointing it /mnt/user would not allow me to start using the BTRFS driver (obviously the unRAID filesystem isn't BTRFS). Also the WebGUI has commands to scrub to filesystem inside the container, all is based on the assumption everyone is using docker on BTRFS (which of course they are because of the container 😁)

    I must say that my approach also broke when I changed something in the shares, certain services get a restart causing docker to be turned off for some reason. No big issue since it wasn't meant to be a long term solution, just to see whether the loopdevice was causing the issue, which I think my tests did point out.

     

    Now I'm at the point where I would definitely need some developer help, I'm currently keeping nearly all docker container off all day because 300/400GB worth of writes a day is just a BIG waste of expensive flash storage. Especially since I've pointed out that it's not needed at all. It does defeat the purpose of my NAS and SSD cache though since it's main purpose was hosting docker containers while allowing the HD's to spin down.

     

    Again, I'm hoping someone in the dev team acknowledges this problem and is willing to invest. I did got quite a few hits on the forums and reddit without someone actually pointed out the root cause of issue.

     

    I missing the technical know-how to troubleshoot the loopdevice issues on a lower level, but have been thinking on possible ways to implement a workaround. Like adjusting the Docker Settings page to switch off the use of a vDisk and if all requirements are met (pointing to /mnt/cache and BTRFS formatted) start docker on a share on the /mnt/cache partition instead of using the vDisk.

    In this way you would still keep all advantages of the docker.img file (cross filesystem type) and users who don't care about writes could still use it, but you'd be massively helping out others that are concerned over these writes.

     

    I'm not attaching diagnostic files since they would probably not point out the needed.

    Also if this should have been in feature requests, I'm sorry. But I feel that, since the solution is misbehaving in terms of writes, this could also be placed in the bugreport section.

     

    Thanks though for this great product, have been using it so far with a lot of joy! 

    I'm just hoping we can solve this one so I can keep all my dockers running without the cache wearing out quick,

     

    Cheers!

     

    • Like 3
    • Thanks 15


    User Feedback

    Recommended Comments



    I am not saying it is not a problem ...

    • Thanks 1

    Share this comment


    Link to comment
    Share on other sites

    Today found this post... 

     

    I run iotop for few hours:

    Start -> Mon May 11 11:45:17 CEST 2020
    End -> Mon May 11 18:49:54 CEST 2020
    
    root@VDS:~# iotop -oa
    Total DISK READ :       0.00 B/s | Total DISK WRITE :      24.44 M/s
    Actual DISK READ:       0.00 B/s | Actual DISK WRITE:      24.18 M/s
      TID  PRIO  USER     DISK READ DISK WRITE>  SWAPIN      IO    COMMAND
     6169 be/0 root         65.20 M    100.45 G  0.00 %  1.06 % [loop2]
    18510 be/4 root         58.20 M   1852.94 M  0.00 %  0.03 % shfs /mnt/user -disks 511 2048000000 -o noatime,allow_other -o remember=0
    18491 be/4 root         59.39 M   1839.84 M  0.00 %  0.03 % shfs /mnt/user -disks 511 2048000000 -o noatime,allow_other -o remember=0
    10221 be/4 root         59.02 M   1809.12 M  0.00 %  0.03 % shfs /mnt/user -disks 511 2048000000 -o noatime,allow_other -o remember=0
    18515 be/4 root         57.02 M   1775.67 M  0.00 %  0.03 % shfs /mnt/user -disks 511 2048000000 -o noatime,allow_other -o remember=0
    18601 be/4 root         58.44 M   1751.80 M  0.00 %  0.03 % shfs /mnt/user -disks 511 2048000000 -o noatime,allow_other -o remember=0
    18600 be/4 root         60.01 M   1746.48 M  0.00 %  0.03 % shfs /mnt/user -disks 511 2048000000 -o noatime,allow_other -o remember=0
    18492 be/4 root         59.45 M   1741.75 M  0.00 %  0.03 % shfs /mnt/user -disks 511 2048000000 -o noatime,allow_other -o remember=0
    18502 be/4 root         58.53 M   1730.81 M  0.00 %  0.03 % shfs /mnt/user -disks 511 2048000000 -o noatime,allow_other -o remember=0
    18451 be/4 root         58.43 M   1666.26 M  0.00 %  0.03 % shfs /mnt/user -disks 511 2048000000 -o noatime,allow_other -o remember=0
    18505 be/4 root         60.94 M   1611.01 M  0.00 %  0.03 % shfs /mnt/user -disks 511 2048000000 -o noatime,allow_other -o remember=0
     6192 be/4 root        140.00 K   1516.05 M  0.00 %  0.11 % [btrfs-transacti]
     [...]

     

    Everything after last line is less then 200M writes.

    I've new SSD disk with 400 TBW and 5y warranty :/

    Share this comment


    Link to comment
    Share on other sites
    PTRFRLL

    Posted (edited)

    Another affected user here. My SSD has 499.90 TB in writes just over a year in use!

     

    Echoing what others have said, I have 2 x SSDs in a btrfs unencrypted RAID-0 setup for cache. I see anywhere from 25Mb/s to 80+Mb/s writes on the cache pool.

     

    I have another Unraid server, set up the same way (6.8.3, 2xSSD btrfs) that does not seem to be affected. The main difference between the two systems is the dockers that are running.

     

    In my case, the official Plex, MariaDB, and Zoneminder dockers seem to be the main offenders. If I disable those, I can drop the writes to ~ 200Kb/s. Plex I switched to linuxserver but unfortunately I can't simply disable my CCTV software...

    Edited by PTRFRLL
    Add more details

    Share this comment


    Link to comment
    Share on other sites

    I see the same regarding MariaDB. Lots of small writes to my databases but it generates huge amount of writes to the ssds. For now, I have my databases on one of my unassigned devices but that's not optimal. Just to save some ssd cache wear. I still see lots of writes to my cache from other dockers. 

    Share this comment


    Link to comment
    Share on other sites
    hovee

    Posted (edited)

    EDIT: Nevermind... still screwed.

     

    I posted earlier that I'm affected by this issue as well. I am using a btrfs unencrypted cache pool with 2x 1TB Samsung 860 EVO SSDs. One thought occurred to me that I wanted to run by everyone. For your containers that you have running do you have them set to /mnt/cache/appdata or /mnt/user/appdata? I had all of mine set to /mnt/cache/appdata. Every time I would install a new container I would change the variable path to use cache. I just went through all of my containers that I have running and switched them to /mnt/user/appdata. Right now I'm at 2gb writes for loop2 writes after 10 minutes.

     

    I need to do more testing and see if this is any better. I'm grasping at straws as my SSDs are trashed now.

     

    I currently have 25 containers running:

     

    1activegeek/airconnect

    grafana/grafana

    homeassistant/home-assistant

    influxdb:latest

    jlesage/filebot

    jlesage/filezilla

    jlesage/firefox

    jlesage/handbrake

    linuxserver/ddclient

    linuxserver/duckdns

    linuxserver/letsencrypt

    linuxserver/mariadb

    linuxserver/nextcloud

    linuxserver/plex

    linuxserver/unifi-controller

    modenaf360/youtube-dl-nas

    oznu/homebridge

    portainer/portainer

    saspus/duplicacy-web

    spants/mqtt

    telegraf:alpine

    Edited by hovee

    Share this comment


    Link to comment
    Share on other sites
    Niklas

    Posted (edited)

    I have tried that. Pointing to /mnt/cache/appdata or /mnt/user/appdata. Made no big difference. Just other names in iotop for the processes that eats away data. 

     

    Edit 

    shfs vs the real process name

    Edited by Niklas

    Share this comment


    Link to comment
    Share on other sites
    1 minute ago, Niklas said:

    I have tried that. Pointing to /mnt/cache/appdata or /mnt/user/appdata. Made no big difference. Just other names in iotop for the processes that eats away data. 

    yep.. I was just doing some math from my post above and came to the conclusion it made no difference and I'm still screwed...

    Share this comment


    Link to comment
    Share on other sites

    Bad news on my end, my writes have crept back up again, and arguably just as bad as it was before I switched my cache to XFS.

     

    This suggests it is a docker issue again.

     

    I need to do more testing but at this stage I am suspecting the  Nextcloud/MariaDB dockers are causing the excessive writes.

     

    I would be interested if other people are using these dockers and can test by stopping them and tracking writes?

     

    I need to back-track from blaming BTRFS now too, at least in my case, the file system does not appear to be a factor in excessive writes.

     

    Share this comment


    Link to comment
    Share on other sites
    24 minutes ago, nas_nerd said:

    I would be interested if other people are using these dockers and can test by stopping them and tracking writes?

    I am not using either nextcloud or mariadb.

     

    I wonder if there's a faster way to test this. I have two SSDs in the server, since the original plan was to have them both in a (btrfs) cache pool. I had to remove the pool, and reformat one of the drives as xfs, which is now running as a single cache drive.

     

    The second SSD is now just sitting idle, unformatted, not mounted, doing nothing. Is there an easy way to format the second SSD as btrfs, mirror the contents of the cache disk to the btrfs disk, and tell unraid to use SSD_2 as the cache? It might make it easier to switch back and forth a little faster between an xfs cache disk and a btrfs cache disk and watch what happens with I/O.

    Share this comment


    Link to comment
    Share on other sites

      

    20 hours ago, bonienl said:

    The priority "Urgent" means something is seriously wrong and prevents the system from working normally.

     

    This is not really the case here...

     

    The priority "Minor" may sound as insignificant, but it does mean Limetech is looking into the issue and address it as appropriate.

    You're right, which is why @itimpi's suggestion of having another category in between sounds like a good one. 

     

    8 hours ago, hovee said:

    yep.. I was just doing some math from my post above and came to the conclusion it made no difference and I'm still screwed...

    The unRAID file system is not the issue here. To get the writes down you have to take the loopdevice (loop2) out of the equation by mounting the docker directory in the filesystem (e.g. creating a symlink from /var/lib/docker to a location on disk/cache). The error seems to be docker in combination with the loopdevice, reading though some comments it's not entirely clear where (and if) btrfs (with or without encryption and/or pooling) has a relation with this problem also, my guess is yes. 

     

    There has been a lot of reports of certain docker containers (like the official Plex) writing a lot also, so it's easy to mistake that and this bug, also since it's possible you're affected by both of them ;) 

     

    To have this solved you'd just need some devs on board. Someone that spins up a test machine and who is able to understand how this relates to page flushes etc (sadly this goes beyond my knowledge).

    Then again I'm sure the devs have other issues also which might have more priority at the moment, although priority is usually driven by community calls right? So this might shift :) 

    Share this comment


    Link to comment
    Share on other sites
    3 hours ago, S1dney said:

    reading though some comments it's not entirely clear where (and if) btrfs (with or without encryption and/or pooling) has a relation with this problem also, my guess is yes. 

    I use a single btrfs drive unencrypted and have the same issue.

    3 hours ago, S1dney said:

    There has been a lot of reports of certain docker containers (like the official Plex) writing a lot also, so it's easy to mistake that and this bug, also since it's possible you're affected by both of them ;) 

    Earlier I already reported my findings. For me it doesn't matter which docker is up and running or if all dockers are stopped. As soon as I enable docker in Unraid I see that increased writes. Dissabling docker itself, Boom, problem disapears. Enabling it with no container running, tada, writes from loop2 are back with 2-5mb/s. Most docker containers people are reported, for myself I don't even use. No Plex, no download managers. Sure, you can reduce the amount of writes by disabling a docker, but it doesn't change the behaviour. Containers like unifi, netdata or nextcloud for example will always produce writes if some monitoring is enabled or mobile devices randomly connecting and checking for new files. Let's hope someone will figure this out. Maybe the next Unraid with a newer docker engine will already have a fix for this. Who knows.

    Share this comment


    Link to comment
    Share on other sites
    S1dney

    Posted (edited)

    14 minutes ago, bastl said:

    I use a single btrfs drive unencrypted and have the same issue.

     

    Earlier I already reported my findings. For me it doesn't matter which docker is up and running or if all dockers are stopped. As soon as I enable docker in Unraid I see that increased writes. Dissabling docker itself, Boom, problem disapears. Enabling it with no container running, tada, writes from loop2 are back with 2-5mb/s. Most docker containers people are reported, for myself I don't even use. No Plex, no download managers. Sure, you can reduce the amount of writes by disabling a docker, but it doesn't change the behaviour. Containers like unifi, netdata or nextcloud for example will always produce writes if some monitoring is enabled or mobile devices randomly connecting and checking for new files. Let's hope someone will figure this out. Maybe the next Unraid with a newer docker engine will already have a fix for this. Who knows.

    That seems to be exactly the issue I was facing indeed.

    A container that wrote a lot, would just bump up writes a lot faster but in general every write docker makes seemed to just get multiplied by who knows what factor :P 

     

    So that could potentially rule out encryption and pooling of disks and would leave the combination between BTRFS and a loop device, I don't believe XFS was affected by this (I heard/read some users reporting XFS did not had these issues).

    Edited by S1dney

    Share this comment


    Link to comment
    Share on other sites

    I'm 3 days now since the switch to a single un-encrypted XFS cache and consistently getting better results. loop2 is producing only ~9MB/min during idle for writes with all my dockers started (included binhex Plex, sonarr, radarr, sabnzbd, deluge, mariaDB, nextcloud, letsencrypt, cloudflare-ddns, pihole, ombi, grafana, teleconf, influxDB) compared to the ~8MB/s I was seeing before after stopping all my dockers but only having docker enabled on my un-encrypted BTRFS cache.

     

    Not sure what the trigger for @nas_nerd's XFS issue but I can't repro it with mariaDB and nextcloud enabled (no user connects in the last 3 days though, maybe I should try and upload something). 

     

    image.thumb.png.84b8130959fcab67cc267eb87cc1912d.png

    over 10 minutes using iotop -oa -d 600

    image.thumb.png.9bc1d6be031f31b9c1b94698b55b8525.png

    over four hours using iotop -oa -d 14400 with several small uploads to nextcloud and a couple of downloads.

    image.thumb.png.3fc7e53d2b693f58440757d574c2e473.png

    Edited by chanrc
    Adding four hour screenshot

    Share this comment


    Link to comment
    Share on other sites

    I am not sure if it would be helpful to anyone, but I moved the system share from the cache to the array as a temporary measure and it seems to have helped a lot.  I went from an average of 160 GB / day of write to around 60 GB / day.  Not perfect, but there is a good chance that is what my dockers are actually using as I have a lot running. Just wanted to share what I found to prevent beating on my SSDs a little.

    • Like 1

    Share this comment


    Link to comment
    Share on other sites
    S1dney

    Posted (edited)

    38 minutes ago, pottlepaul said:

    I am not sure if it would be helpful to anyone, but I moved the system share from the cache to the array as a temporary measure and it seems to have helped a lot.  I went from an average of 160 GB / day of write to around 60 GB / day.  Not perfect, but there is a good chance that is what my dockers are actually using as I have a lot running. Just wanted to share what I found to prevent beating on my SSDs a little.

    By this I think you’re essentially moving the docker image (and thus the mount on /var/lib/docker) onto the array. So these writes should not go to the cache anymore, I guess. 

     

    Docker will keep you array up non-stop though, which kind op defeats unRAID selling point in being able to spin down disks. 
     

    When you combine this with the unassigned disks plugin, you might be able to put it on a single disk for now (I think, haven’t used the plugin before) and have the array still fall asleep.

     

    Good suggestion for some people that are not into making unsupported CLI/script tweaks, thanks for sharing!

     

    Also as @chanrc is reporting, this really looks to be btrfs related, which is sad, cause it’s your only option if you want to have a redundant cache.

    Edited by S1dney

    Share this comment


    Link to comment
    Share on other sites
    On 3/21/2020 at 5:36 AM, S1dney said:

    I've read though my posts real quick and noticed I have not yet provided my final solution, so let me share that.

    Basically what I did was:

    I tried following your tutorial, but after reboot it tells me I don't have any docker containers installed.

    1065017496_ScreenShot2020-05-12at1_22_45PM.thumb.png.a9d6bea9fabc2f755b413d1a8194d734.png

     

    It shows me the docker service is running.

    1044152176_ScreenShot2020-05-12at1_23_04PM.thumb.png.e8e5a1ae8b047b3c68eb9e1a136bebfd.png

     

    Checking the logs I can see it created the softlinks.

    490799377_ScreenShot2020-05-12at1_24_39PM.thumb.png.be86bce91c4fb763629d4be5611bcfda.png

     

    If I remove that entry to copy the rc.docker file in the /boot/config/go directory everything works again after a reboot. However, then it puts it back to original and uses loop2.

    Share this comment


    Link to comment
    Share on other sites
    19 minutes ago, hovee said:

    I tried following your tutorial, but after reboot it tells me I don't have any docker containers installed.

    1065017496_ScreenShot2020-05-12at1_22_45PM.thumb.png.a9d6bea9fabc2f755b413d1a8194d734.png

     

    It shows me the docker service is running.

    1044152176_ScreenShot2020-05-12at1_23_04PM.thumb.png.e8e5a1ae8b047b3c68eb9e1a136bebfd.png

     

    Checking the logs I can see it created the softlinks.

    490799377_ScreenShot2020-05-12at1_24_39PM.thumb.png.be86bce91c4fb763629d4be5611bcfda.png

     

    If I remove that entry to copy the rc.docker file in the /boot/config/go directory everything works again after a reboot. However, then it puts it back to original and uses loop2.

    It behaves as expected.

    All your docker containers are downloaded into the docker image (located in the system folder somewhere on the cache, docker.img is the file I think).

    After the changes you’ve made, you’re not mounting that anymore, but have docker targeted to a different directory.

    Docker will create the needed working directories upon service start, meaning that all container are still inside the docker.img file.

     

    I initially re-created them, using the templates from the dockerMan GUI this isn’t too much work and all persistent data should not reside in the docker.img anyways or you might lose it if the docker.img gets corrupted. I guess you could also copy all data over before implementing the script that mounts the cache directory but I would recreate the containers if I were you.

     

    You should also recreate the docker.img image if you’re done with everything, so that when something changes in future unRAID versions which potentially breaks this, you’ll notice that you have no containers after a reboot and know the docker.img file is mounted or something else is wrong :-)

    Share this comment


    Link to comment
    Share on other sites
    2 hours ago, S1dney said:

    It behaves as expected.

    All your docker containers are downloaded into the docker image (located in the system folder somewhere on the cache, docker.img is the file I think).

    After the changes you’ve made, you’re not mounting that anymore, but have docker targeted to a different directory.

    Docker will create the needed working directories upon service start, meaning that all container are still inside the docker.img file.

     

    I initially re-created them, using the templates from the dockerMan GUI this isn’t too much work and all persistent data should not reside in the docker.img anyways or you might lose it if the docker.img gets corrupted. I guess you could also copy all data over before implementing the script that mounts the cache directory but I would recreate the containers if I were you.

     

    You should also recreate the docker.img image if you’re done with everything, so that when something changes in future unRAID versions which potentially breaks this, you’ll notice that you have no containers after a reboot and know the docker.img file is mounted or something else is wrong 🙂

    Thank you for the explanation. That makes sense. I will give that a try!

    Share this comment


    Link to comment
    Share on other sites
    nuhll

    Posted (edited)

    how to check if this is a problem for me? 

    one shows 934731420222 LBAS (2y6m)
    the other 899487348326 LBAS (2y4m)

     

    if im right its about 500gb a day, which seems much (?!)

     

    sadly iotop is not working for me (was already installed)

    root@Unraid-Server:~# iotop
    libffi.so.7: cannot open shared object file: No such file or directory
    To run an uninstalled copy of iotop,
    launch iotop.py in the top directory

    root@Unraid-Server:~# iotop -ao
    libffi.so.7: cannot open shared object file: No such file or directory
    To run an uninstalled copy of iotop,
    launch iotop.py in the top directory
    root@Unraid-Server:~# iotop.py
    -bash: iotop.py: command not found
    root@Unraid-Server:~# py iotop.py
    -bash: py: command not found
    root@Unraid-Server:~# python iotop.py
    python: can't open file 'iotop.py': [Errno 2] No such file or directory

     

    /dev/sdd    Power_On_Hours            22510 hours / 937 days / 2.57 years
    /dev/sdd    Wear_Leveling_Count       44 (% health)
    /dev/sdd    Total_LBAs_Written        445842.14 gb / 435.39 tb
    /dev/sdd    mean writes per hour:     19.806 gb / 0.019 tb

     

    /dev/sdc    Power_On_Hours            20447 hours / 851 days / 2.33 years
    /dev/sdc    Wear_Leveling_Count       46 (% health)
    /dev/sdc    Total_LBAs_Written        428919.80 gb / 418.87 tb
    /dev/sdc    mean writes per hour:     20.977 gb / 0.020 tb

     

    is that normal?

     

    Idle windows 10 VM + the usual linux iso download dockers and some light plex... 

     

    I also wonder why limetech isnt posting anymore here, i mean its 6 months after he read it, was it addressed in some RC?

    Edited by nuhll

    Share this comment


    Link to comment
    Share on other sites
    1 hour ago, nuhll said:

    how to check if this is a problem for me? 

    one shows 934731420222 LBAS (2y6m)
    the other 899487348326 LBAS (2y4m)

     

    if im right its about 500gb a day, which seems much (?!)

     

    sadly iotop is not working for me (was already installed)

    root@Unraid-Server:~# iotop
    libffi.so.7: cannot open shared object file: No such file or directory
    To run an uninstalled copy of iotop,
    launch iotop.py in the top directory

    root@Unraid-Server:~# iotop -ao
    libffi.so.7: cannot open shared object file: No such file or directory
    To run an uninstalled copy of iotop,
    launch iotop.py in the top directory
    root@Unraid-Server:~# iotop.py
    -bash: iotop.py: command not found
    root@Unraid-Server:~# py iotop.py
    -bash: py: command not found
    root@Unraid-Server:~# python iotop.py
    python: can't open file 'iotop.py': [Errno 2] No such file or directory

     

    /dev/sdd    Power_On_Hours            22510 hours / 937 days / 2.57 years
    /dev/sdd    Wear_Leveling_Count       44 (% health)
    /dev/sdd    Total_LBAs_Written        445842.14 gb / 435.39 tb
    /dev/sdd    mean writes per hour:     19.806 gb / 0.019 tb

     

    /dev/sdc    Power_On_Hours            20447 hours / 851 days / 2.33 years
    /dev/sdc    Wear_Leveling_Count       46 (% health)
    /dev/sdc    Total_LBAs_Written        428919.80 gb / 418.87 tb
    /dev/sdc    mean writes per hour:     20.977 gb / 0.020 tb

     

    is that normal?

     

    Idle windows 10 VM + the usual linux iso download dockers and some light plex... 

     

    I also wonder why limetech isnt posting anymore here, i mean its 6 months after he read it, was it addressed in some RC?

    Make sure you have libffi installed and updated to get iotop to work.

     

    I'm sure limetech are aware of this issue, they are no doubt busy on 6.9 at the moment, hopefully after it is released they can turn their focus to this issue.

    Share this comment


    Link to comment
    Share on other sites
    7 hours ago, nas_nerd said:

    Make sure you have libffi installed and updated to get iotop to work.

     

    I'm sure limetech are aware of this issue, they are no doubt busy on 6.9 at the moment, hopefully after it is released they can turn their focus to this issue.

    LT have known about this since atleast December of last year, I really hope what you're saying isn't true. :/

    Share this comment


    Link to comment
    Share on other sites
    woble

    Posted (edited)

    Switched to XFS, rebuilt the docker.img with the same dockers as I had before. Now `loop2` is around 9-30MB/min.

     

    For those who are still on BTRFS perhaps it's worth trying removing the docker.img and re-adding all dockers. It might help. I should have tried that first.

     

    My original post: https://forums.unraid.net/bug-reports/stable-releases/683-docker-image-huge-amount-of-unnecessary-writes-on-cache-r733/?do=findComment&comment=9019

     

    Edit: after having it run for almost a day, the `loop2` write seems to be stable at around 1.7GB/h.

     

    Edited by woble

    Share this comment


    Link to comment
    Share on other sites
    17 hours ago, nas_nerd said:

    I'm sure limetech are aware of this issue, they are no doubt busy on 6.9 at the moment, hopefully after it is released they can turn their focus to this issue.

     

    Maybe, but you said the same thing about them working on 6.8 when this bug was reported back in version 6.7 and we still haven't heard anything from them about it. This is not a minor issue -- I suspect that it's actually happening for a LOT of installs, but most people don't know it's happening because it requires actually looking for it.


    This is a serious bug -- potentially costing users a lot of money in trashed SSDs. And this is commercial software -- we're paying for a license to use it, so it's not just a FOSS project where expectations should be low. In my opinion, LimeTech should be investigating this issue and prioritizing it before releasing the next version.

    • Like 2

    Share this comment


    Link to comment
    Share on other sites

    This is just a disaster.

    I've been using Unraid for almost a month now. I put btrfs raid1 in the cache without encryption. Two brand new samsung ssd 860 2tb. Just yesterday I paid for the Unraid pro license for $ 89.
    Today I accidentally learned from a person that a forum is discussing this problem. I decided to check my SSD. And now I ask you, are you kidding me guys? My two new SSDs already have 90tbw, and 28 percent of my life. THAT is 1 percent life ssd for every day?

    And this problem has been known to you since the month of November, six months have passed and you still haven’t fixed it?
     I'm just in shock. All impression spoiled about Unraid and Limetech.

     

    samsung ssd cash.jpg

    Edited by muwahhid

    Share this comment


    Link to comment
    Share on other sites
    41 minutes ago, muwahhid said:

    This is just a disaster.

    I've been using Unraid for almost a month now. I put btrfs raid1 in the cache without encryption. Two brand new samsung ssd 860 2tb. Just yesterday I paid for the Unraid pro license for $ 89.
    Today I accidentally learned from a person that a forum is discussing this problem. I decided to check my SSD. And now I ask you, are you kidding me guys? My two new SSDs already have 90tbw, and 28 percent of my life. THAT is 1 percent life ssd for every day?

    And this problem has been known to you since the month of November, six months have passed and you still haven’t fixed it?
     I'm just in shock. All impression spoiled about Unraid and Limetech.

    samsung ssd cash.jpg

    How are you calculating 90TBW?

    Share this comment


    Link to comment
    Share on other sites



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.