• [6.8.3] Unnecessary overwriting of JSON-files in docker.img every 5 seconds


    mgutt
    • Minor

    @Valerio found this out first, but never received an answer. Today I found it out, too. But this is present since 2019 (or even longer).

     

    I would say it's a bug, as:

    • it prevents HDD/SSD spindown/sleep (depending on the location of docker.img)
    • it wears out the SSD in the long run (if docker.img is located here) - see this bug, too.
    • it prevents reaching CPU's deep sleep states

     

    What happens:

    • /var/lib/docker/containers/*/hostconfig.json is updated every 5 seconds with the same content
    • /var/lib/docker/containers/*/config.v2.json is updated every 5 seconds with the same content except of some timestamps (which shouldn't be part of a config file I think)

     

    Which docker containers:

    • verified are Plex (Original) and PiHole, but maybe this is a general behaviour

     

    As an example the source of hostconfig.json which was updated yesterday 17280x times with the same content:

    find /var/lib/docker/containers/40b4197fdea122178139e9571ae5f4040a2ef69449acf14e616010c7e293bb44 -ls -name hostconfig.json -exec cat {} \;
    2678289      4 -rw-r--r--   1 root     root         1725 Oct  8 13:46 /var/lib/docker/containers/40b4197fdea122178139e9571ae5f4040a2ef69449acf14e616010c7e293bb44/hostconfig.json
    {
       "Binds":[
          "/mnt/user/tv:/tv:ro",
          "/mnt/cache/appdata/Plex-Media-Server:/config:rw",
          "/mnt/cache/appdata/Plex-Transcode:/transcode:rw",
          "/mnt/user/movie:/movie:ro"
       ],
       "ContainerIDFile":"",
       "LogConfig":{
          "Type":"json-file",
          "Config":{
             "max-file":"1",
             "max-size":"50m"
          }
       },
       "NetworkMode":"host",
       "PortBindings":{
          
       },
       "RestartPolicy":{
          "Name":"no",
          "MaximumRetryCount":0
       },
       "AutoRemove":false,
       "VolumeDriver":"",
       "VolumesFrom":null,
       "CapAdd":null,
       "CapDrop":null,
       "Capabilities":null,
       "Dns":[
          
       ],
       "DnsOptions":[
          
       ],
       "DnsSearch":[
          
       ],
       "ExtraHosts":null,
       "GroupAdd":null,
       "IpcMode":"private",
       "Cgroup":"",
       "Links":null,
       "OomScoreAdj":0,
       "PidMode":"",
       "Privileged":false,
       "PublishAllPorts":false,
       "ReadonlyRootfs":false,
       "SecurityOpt":null,
       "UTSMode":"",
       "UsernsMode":"",
       "ShmSize":67108864,
       "Runtime":"runc",
       "ConsoleSize":[
          0,
          0
       ],
       "Isolation":"",
       "CpuShares":0,
       "Memory":0,
       "NanoCpus":0,
       "CgroupParent":"",
       "BlkioWeight":0,
       "BlkioWeightDevice":[
          
       ],
       "BlkioDeviceReadBps":null,
       "BlkioDeviceWriteBps":null,
       "BlkioDeviceReadIOps":null,
       "BlkioDeviceWriteIOps":null,
       "CpuPeriod":0,
       "CpuQuota":0,
       "CpuRealtimePeriod":0,
       "CpuRealtimeRuntime":0,
       "CpusetCpus":"",
       "CpusetMems":"",
       "Devices":[
          {
             "PathOnHost":"/dev/dri",
             "PathInContainer":"/dev/dri",
             "CgroupPermissions":"rwm"
          }
       ],
       "DeviceCgroupRules":null,
       "DeviceRequests":null,
       "KernelMemory":0,
       "KernelMemoryTCP":0,
       "MemoryReservation":0,
       "MemorySwap":0,
       "MemorySwappiness":null,
       "OomKillDisable":false,
       "PidsLimit":null,
       "Ulimits":null,
       "CpuCount":0,
       "CpuPercent":0,
       "IOMaximumIOps":0,
       "IOMaximumBandwidth":0,
       "MaskedPaths":[
          "/proc/asound",
          "/proc/acpi",
          "/proc/kcore",
          "/proc/keys",
          "/proc/latency_stats",
          "/proc/timer_list",
          "/proc/timer_stats",
          "/proc/sched_debug",
          "/proc/scsi",
          "/sys/firmware"
       ],
       "ReadonlyPaths":[
          "/proc/bus",
          "/proc/fs",
          "/proc/irq",
          "/proc/sys",
          "/proc/sysrq-trigger"
       ]
    }

     

    • Like 2
    • Thanks 3



    User Feedback

    Recommended Comments



    2 hours ago, Oreonipples said:

    thinking maybe its that my drive is btfrs as this is higher than it was without this on my xfs cache drive

    It will be less with XFS, but 12MB/s (sure?!) is too much for writes to log files. Your problem lies somewhere else.

    Link to comment

    After finishing and switching drive to xfs. Bam 0 - 50 kb/s down from 12mb/s. Perhaps the chia docker was adding to it but I think a large part was the btfrs file system. My unraid install does not seem to enjoy cache drives formatted like that. Probably something I screwed up when I originally set it up. Im tempted to change my docker to a directory in the future but I'm just not in the mood to setup all my dockers again right now. 

    Link to comment
    3 hours ago, Oreonipples said:

    I'm just not in the mood to setup all my dockers again right now. 

    Don't need to. Change to Dir and then Apps > Previous Apps to install them as before. The only thing what happens is re-downloading the packages. Any if you created such, you need to recreate custom networks. But you don't need to change templates or similar things. The docker.img does not contain any important data.

     

    Link to comment
    1 hour ago, mgutt said:

    The docker.img does should not contain any important data.

    FTFY. Occasionally we see people manually updating or adding things to their containers, or saving things to non- mapped paths in the container.

     

    Either will create other issues and should be fixed, but it's always possible.

    Link to comment
    29 minutes ago, JonathanM said:

    Occasionally we see people manually updating or adding things to their containers

    No pity with these ^^

    • Haha 2
    Link to comment

    I dont believe ive done any manual updates or wierd add's outside the template. Good to know I should be able to switch to a docker directory relatively painlessly.

     

    Moving back to xfs on the new cache drive appears to have fixed my issue almost 100% in conjunction with your log ramdisk! I have started the machinaris docker again, its syncing network which I believe is causing the writes. However now im only seeing 2mb/s - 7mb/s on that drive down from the 12 - 14. 

    Link to comment
    On 8/25/2021 at 10:45 PM, pervin_1 said:

    The scripts you shared are working perfectly. TBH, my writes ( I have been monitoring for a good time now ) are not excessive. The first script is barely showing any activiy for my containers. I see some repepetive logs for json files from /docker/containers. But I am assuming this is a RAMDISK for status and logs. Correct me if I am wrong, please. 

    Another question, the appdata is located on my NVMe SSD cache pool. My understanding there will be some kind of writes from dockers regardless, correct? Besides that I left the script to dump the log/status files for safety reason every 30 minutes. 

    BTW, I can see that my writes are basically coming from the OnlyOffice Document server docker ( and some from Nextcloud ). It's connected to my Nextcloud server as a document editor. I am not sure if you are familiar or using it. 

     

    Edit:

    Nextcloud was in the tmp folder. Added extra parameter to mount the tmp folder in RAM disk ( not sure if you script handles the tmp in docker containers ). The OnlyOffice, mainly writes go to /run/postgresql every one minute or so. I am assuming is some kind of database system. Do you think it's a good idea to mount it to the RAM disk under the extra parameters? 

    Could you clarify what you mounted to the Nextcloud container? /tmp:/tmp or /var/lib/docker/containers:/tmp

    Link to comment

    Has anyone tested the ramdisk workaround from @mgutt with the final release of 6.10 or has the bug perhaps been solved completely with this release?

    Link to comment

     

    On 5/19/2022 at 12:04 AM, kennymc.c said:

    Has anyone tested the ramdisk workaround from @mgutt with the final release of 6.10 or has the bug perhaps been solved completely with this release?

     

    On 5/30/2022 at 12:20 AM, JustOverride said:

    I'll also like to know if this has been resolve or if it will be my next task. Thanks.

     

    I did test my setup on 6.10.2 and it seem's to work just as fine as on 6.9

    Link to comment
    On 7/19/2022 at 11:27 PM, ricemonster said:

    Is this still an issue on the latest stable version?

     

    Would like to know as well if this was merged into the main branch.

    Link to comment
    5 hours ago, Kyle Boddy said:

     

    Would like to know as well if this was merged into the main branch.


    Same here, I hope it has been fixed though.

    • Upvote 2
    Link to comment

    I dont think it is. I had read/writes on my datadrives every x seconds. Found out that my docker.img file was still on datadrives in stead of SSD cache. After moving it to SSD, writes are still happening, but at least its silent..

    Reading this thread, the writes makes more sence.

    Edit: Putting this thread in favourites and experimenting in the coming weekend.

    Edited by Caddy
    Link to comment

    Bump!  Very interested in this as should all other users.  I know I'm new here, but all the more reason as I immediately discovered this is a pain point for my SSD cache once I set it up.

    Link to comment

    I've created a quick script to log my SSD write accumulation so I can watch it without having to grab it manually.  This works great for my Samsung SSDs, which I prefer, may not work for other brands.  Since people watching this issue might find it useful, I'm thought I'd posting it here.  Just change the /dev/sd? to your devices (I have two SSDs in a pool, sde and sdg).  This outputs to a csv files in my /mnt/user/down folder (change down to your favorite folder for this) called SSD.csv. I run it daily using the user scripts plugin.  This makes it easy to just open with Excel and check it from time to time and add formulas to compute daily accumulation, etc. This is pretty basic, so I didnt include any header row in the script itself, but its this (LBAW=LBA written, GBW=Gigabyte Written):

    Date,Time,sde LBAW,sde GBW,sdg LBAW,sdg GBW

     

    #!/bin/bash
    dt=`date +"%D,%T"`
    sdeLBA=`smartctl -Ai /dev/sde | grep LBA | awk '{print $NF}'`
    sdeGB=$(($sdeLBA*512/1024**3))
    sdgLBA=`smartctl -Ai /dev/sdg | grep LBA | awk '{print $NF}'`
    sdgGB=$(($sdgLBA*512/1024**3))
    echo "$dt,$sdeLBA,$sdeGB,$sdgLBA,$sdgGB" >> /mnt/user/down/SSD.csv

     

    Update:

    After several days of mixed but typical usage (normal runtime writes and a few container updates), I found that writes are under 10GB per day to cache for my setup.  This is after implementing several of the mods people have mentioned in this thread and with about 6 active containers only.

    Edited by rkotara
    • Thanks 1
    Link to comment

    To collect more feedback for the Go file code, I started a new thread here:

     

    Note: This is a new version. The very first release caused in rare cases multiple mounts on the same path, which caused a stalling server.

    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.