• [6.8.3] Unnecessary overwriting of JSON-files in docker.img every 5 seconds


    mgutt
    • Minor

    @Valerio found this out first, but never received an answer. Today I found it out, too. But this is present since 2019 (or even longer).

     

    I would say it's a bug, as:

    • it prevents HDD/SSD spindown/sleep (depending on the location of docker.img)
    • it wears out the SSD in the long run (if docker.img is located here) - see this bug, too.
    • it prevents reaching CPU's deep sleep states

     

    What happens:

    • /var/lib/docker/containers/*/hostconfig.json is updated every 5 seconds with the same content
    • /var/lib/docker/containers/*/config.v2.json is updated every 5 seconds with the same content except of some timestamps (which shouldn't be part of a config file I think)

     

    Which docker containers:

    • verified are Plex (Original) and PiHole, but maybe this is a general behaviour

     

    As an example the source of hostconfig.json which was updated yesterday 17280x times with the same content:

    find /var/lib/docker/containers/40b4197fdea122178139e9571ae5f4040a2ef69449acf14e616010c7e293bb44 -ls -name hostconfig.json -exec cat {} \;
    2678289      4 -rw-r--r--   1 root     root         1725 Oct  8 13:46 /var/lib/docker/containers/40b4197fdea122178139e9571ae5f4040a2ef69449acf14e616010c7e293bb44/hostconfig.json
    {
       "Binds":[
          "/mnt/user/tv:/tv:ro",
          "/mnt/cache/appdata/Plex-Media-Server:/config:rw",
          "/mnt/cache/appdata/Plex-Transcode:/transcode:rw",
          "/mnt/user/movie:/movie:ro"
       ],
       "ContainerIDFile":"",
       "LogConfig":{
          "Type":"json-file",
          "Config":{
             "max-file":"1",
             "max-size":"50m"
          }
       },
       "NetworkMode":"host",
       "PortBindings":{
          
       },
       "RestartPolicy":{
          "Name":"no",
          "MaximumRetryCount":0
       },
       "AutoRemove":false,
       "VolumeDriver":"",
       "VolumesFrom":null,
       "CapAdd":null,
       "CapDrop":null,
       "Capabilities":null,
       "Dns":[
          
       ],
       "DnsOptions":[
          
       ],
       "DnsSearch":[
          
       ],
       "ExtraHosts":null,
       "GroupAdd":null,
       "IpcMode":"private",
       "Cgroup":"",
       "Links":null,
       "OomScoreAdj":0,
       "PidMode":"",
       "Privileged":false,
       "PublishAllPorts":false,
       "ReadonlyRootfs":false,
       "SecurityOpt":null,
       "UTSMode":"",
       "UsernsMode":"",
       "ShmSize":67108864,
       "Runtime":"runc",
       "ConsoleSize":[
          0,
          0
       ],
       "Isolation":"",
       "CpuShares":0,
       "Memory":0,
       "NanoCpus":0,
       "CgroupParent":"",
       "BlkioWeight":0,
       "BlkioWeightDevice":[
          
       ],
       "BlkioDeviceReadBps":null,
       "BlkioDeviceWriteBps":null,
       "BlkioDeviceReadIOps":null,
       "BlkioDeviceWriteIOps":null,
       "CpuPeriod":0,
       "CpuQuota":0,
       "CpuRealtimePeriod":0,
       "CpuRealtimeRuntime":0,
       "CpusetCpus":"",
       "CpusetMems":"",
       "Devices":[
          {
             "PathOnHost":"/dev/dri",
             "PathInContainer":"/dev/dri",
             "CgroupPermissions":"rwm"
          }
       ],
       "DeviceCgroupRules":null,
       "DeviceRequests":null,
       "KernelMemory":0,
       "KernelMemoryTCP":0,
       "MemoryReservation":0,
       "MemorySwap":0,
       "MemorySwappiness":null,
       "OomKillDisable":false,
       "PidsLimit":null,
       "Ulimits":null,
       "CpuCount":0,
       "CpuPercent":0,
       "IOMaximumIOps":0,
       "IOMaximumBandwidth":0,
       "MaskedPaths":[
          "/proc/asound",
          "/proc/acpi",
          "/proc/kcore",
          "/proc/keys",
          "/proc/latency_stats",
          "/proc/timer_list",
          "/proc/timer_stats",
          "/proc/sched_debug",
          "/proc/scsi",
          "/sys/firmware"
       ],
       "ReadonlyPaths":[
          "/proc/bus",
          "/proc/fs",
          "/proc/irq",
          "/proc/sys",
          "/proc/sysrq-trigger"
       ]
    }

     

    • Like 1
    • Thanks 3



    User Feedback

    Recommended Comments



    I am using 6.9.0-beta29 (will go to 30 ASAP)  and do see those writes too.
    Noticed that only 2 of my containers end up writing to these 2 files which are the only containers that seem to have a docker health check available.

     

    Here is the reason why you see a write every 5 seconds for your plex container

    https://github.com/plexinc/pms-docker/blob/master/Dockerfile#L62

    HEALTHCHECK --interval=5s --timeout=2s --retries=20 CMD /healthcheck.sh || exit 1

     

    Edited by DasMarx
    • Thanks 3
    Link to comment

    You can disable those healthchecks by adding

    --no-healthcheck

    to the Extra Parameters of each docker container in the advanced view.

    • Thanks 3
    Link to comment
    4 hours ago, DasMarx said:

    You can disable those healthchecks by adding

    This solved it!!! 🥳

    116320200_2020-10-0910_02_27.thumb.png.1efe01ae9fc11241756681413a3bf7c6.png

     

    I had this suspect, too. But renaming healthcheck.sh to healthcheck.delete didn't help, so I gave up :|

     

    Added to the Plex container as follows:

    788449800_2020-10-0909_47_55.png.7161b69678a8bb55e5addc6b8a7ad844.png

     

    After that I verified it as follows:

     

    Docker -> Plex Icon -> Console:

    find /config/*/ -mmin -1 -ls > /config/recent_modified_files$(date +"%Y%m%d_%H%M%S").txt

    Results: Empty, which means no modified files in /mnt/user/appdata/Plex-Media-Server)

     

    WebGUI -> Webterminal:

    inotifywait -e create,modify,attrib,moved_from,moved_to --timefmt %c --format '%T %_e %w %f' -mr /var/lib/docker > /mnt/user/system/docker/recent_modified_files_$(date +"%Y%m%d_%H%M%S").txt

    Result: Empty, which means no writes in docker.img at all!!!

     

    Whats next?

    How should @limetech use this information to solve the issue? Add --no-healthcheck to all containers by default? I think something similar is needed as most people do not know how to monitor their disk traffic. They will only conclude "hey Unraid does not send this disk to sleep!" or "Unraid produces really much traffic on my SSD". ^^

    Edited by mgutt
    • Thanks 2
    Link to comment

    Not obvious on the best way to handle this as technically this is an issue with specific docker containers rather than an UnRAID issue.   There may be good reasons the maintainer of a particular container has set this up so that over-riding it becomes a bad idea.    I wonder if the Fix Common Problems plugin could be enhanced to detect this and suggest the fix?  Alternatively make it a configuration option in the docker templates?

    Link to comment
    46 minutes ago, itimpi said:

    here may be good reasons the maintainer of a particular container has set this up

    Yes you are right. Some people use this state inside of the container to check the run status or internet access. Its not how its meant to be, as for example the internet access check is absolutely crazy as 1 million users of the container, would result in 1 million google.com requests every x seconds ^^

     

    I'm also not sure why Plex is doing this every 5 seconds:

    https://github.com/plexinc/pms-docker/blob/master/root/healthcheck.sh

     

    They connect to http://tower:32400/identity, which returns an XML file and the content of this file is thrown away. This means they only check, if the internal webserver is running. And this is needed every 5 seconds? Are they crazy? ^^

     

    Besides of that. Are these JSON files really part of an usual docker installation or is this a special Unraid thing? I wonder why only Unraid users are complaining those permanent writes. Ok... maybe it has the most transparent traffic monitoring ;)

     

    If these writes are related to Docker only, we should open an issue here. Because only updating a timestamp (or nothing) inside a config file, does not really sound like it's working as it ment to be.

    Edited by mgutt
    Link to comment
    6 minutes ago, mgutt said:

    Besides of that. Are these JSON files really part of an usual docker installation or is this a special Unraid thing? I wonder why only Unraid users are complaining those permanent writes. Ok... maybe it has the most transparent traffic monitoring ;)

     

    This is nothing to do with Unraid - it is internal to the docker container.   It may have been more obvious to Unraid users because of the fact that a docker image is used and there was an issue in 6.8.3 and earlier that could cause significant write amplification when writing to that image stored on a SSD.   If the docker container files were stored directly in the file system (as the latest Unraid betas allow) then this is probably far less noticeable particularly if using a HDD with a file system like XFS that has far less inherent write amplification than BTRFS does.

     

    11 minutes ago, mgutt said:

    If these writes are related to Docker only, we should open an issue here. Because only updating a timestamp (or nothing) inside a config file, does not really sound like it's working as it meant to be.

    This could not do any harm although they may simply reply that the feature is being mis-used and the fix should come from the container maintainer.

     

    Link to comment
    53 minutes ago, itimpi said:

    mis-used and the fix should come from the container maintainer

    As the JSON files are overwritten by docker and not the container, it shouldn't be an issue of the maintainer. I'll try my luck ^^

    Link to comment

    hey, i found this on reddit and i tryed the following command:

     

    Quote

    inotifywait -e create,modify,attrib,moved_from,moved_to --timefmt %c --format '%T %_e %w %f' -mr /var/lib/docker > /mnt/user/system/docker/recent_modified_files_$(date +"%Y%m%d_%H%M%S").txt

     the output is attached.

     

    are the writings normal?

     

    recent_modified_files_20201010_100445.txt

    Link to comment

    @sonic6

    You have many requests because of a postgresql database. If you need it, there would be nothing you could optimize. But I would say these writes happen to often :

    MODIFY /var/lib/docker/containers/ced93fa7c199b90a5414e103be9871a25cae5b8e3fdaab059b1517e34ad0150f/ ced93fa7c199b90a5414e103be9871a25cae5b8e3fdaab059b1517e34ad0150f-json.log

    Check in your Container Dashboard for a container ID (advanced view) starting with "ced93fa7". Which docker is it? Does it have a logging setting which you could disable?

    Link to comment
    20 hours ago, mgutt said:

     

    
    MODIFY /var/lib/docker/containers/ced93fa7c199b90a5414e103be9871a25cae5b8e3fdaab059b1517e34ad0150f/ ced93fa7c199b90a5414e103be9871a25cae5b8e3fdaab059b1517e34ad0150f-json.log

    Check in your Container Dashboard for a container ID (advanced view) starting with "ced93fa7". Which docker is it? Does it have a logging setting which you could disable?

    this is the binhex-krusader container. Everything is like in the stock template with the "--no-healthcheck" command.

     

    maybe someone other than me runs the binhex-krusader contain and have the same?

    Link to comment

    Someone in Reddit found out that a container, which is using HEALTHCHECK, causes displaying the word "healthy" in the Unraid > Docker Dashboard (Advanced View):

    1114486258_2020-10-1401_28_54.png.5369600ae544fd95840adc9297fa0e43.png

     

    This makes it much easier to find those containers.

    Edited by mgutt
    • Thanks 1
    Link to comment

    I am getting periodic writes to  a share on one of my disk drives.  They occur every few seconds. The writes are to the file "docker.img" in the system share which only exists on one hard drive. The system share contains two directories (docker, libvirt) each directory contains one image (docker.img in the docker directory; libvirt.img in the libvirt directory.

    I have entered the "--no-healthcheck" for my one docker container to no avail.

     

    Any help greatly appreciated

    Link to comment

    I wonder if I need to enter the NO HEALTHCHECK parameter for all these containers? Will they work stably after that?

     

    image.thumb.png.b6fc90d575a3a00c394ec4254952f2eb.png

    • Like 7
    • Thanks 2
    Link to comment

    @limetech

    I solved this issue as follows and successfully tested it in:

    • Unraid 6.9.2
    • Unraid 6.10.0-rc1

     

    1. Add this to /boot/config/go (by Config Editor Plugin):
      # -------------------------------------------------
      # RAM-Disk for Docker json/log files
      # -------------------------------------------------
      # create RAM-Disk on starting the docker service
      sed -i '/^  echo "starting \$BASE ..."$/i \
        # move json/logs to ram disk\
        rsync -aH --delete /var/lib/docker/containers/ ${DOCKER_APP_CONFIG_PATH%/}/containers_backup\
        mount -t tmpfs tmpfs /var/lib/docker/containers\
        rsync -aH --delete ${DOCKER_APP_CONFIG_PATH%/}/containers_backup/ /var/lib/docker/containers\
        logger -t docker RAM-Disk created' /etc/rc.d/rc.docker
      # remove RAM-Disk on stopping the docker service
      sed -i '/^  # tear down the bridge$/i \
        # backup json/logs and remove RAM-Disk\
        rsync -aH --delete /var/lib/docker/containers/ ${DOCKER_APP_CONFIG_PATH%/}/containers_backup\
        umount /var/lib/docker/containers\
        rsync -aH --delete ${DOCKER_APP_CONFIG_PATH%/}/containers_backup/ /var/lib/docker/containers\
        logger -t docker RAM-Disk removed' /etc/rc.d/rc.docker
      # Automatically backup Docker RAM-Disk
      sed -i '/^<?PHP$/a \
      $sync_interval_minutes=30;\
      if ( ! ((date('i') * date('H') * 60 + date('i')) % $sync_interval_minutes) && file_exists("/var/lib/docker/containers")) {\
        exec("mkdir /var/lib/docker_bind");\
        exec("mount --bind /var/lib/docker /var/lib/docker_bind");\
        exec("rsync -aH --delete /var/lib/docker/containers/ /var/lib/docker_bind/containers");\
        exec("umount /var/lib/docker_bind");\
        exec("rmdir /var/lib/docker_bind");\
        exec("logger -t docker RAM-Disk synced");\
      }' /usr/local/emhttp/plugins/dynamix/scripts/monitor
    2. Optional: Limit the Docker LOG size to avoid using too much RAM:
      1082044065_2021-08-1910_34_09.png.30f268aeff1d17f254eacd8d1e7bf9a8.png
    3. Reboot server

     

    Notes:

    • By this change /var/lib/docker/containers, which contains only status and log files, becomes a RAM-Disk and therefore avoids wearing out your SSD and allows a permanent sleeping SSD (energy efficient)
    • It automatically syncs the RAM-Disk every 30 minutes to your default appdata location  (for server crash / power-loss scenarios). If container logs are important to you, feel free to change the value of "$sync_interval_minutes" in the above code to a smaller value to sync the RAM-Disk every x minutes.
    • If you like to update Unraid OS, you should remove the change from the Go File until it's clear that this enhancement is still working/needed!

     

    Your Reward:

    415470549_2021-08-1808_28_47.png.b552e3017be7cbf5a825f6e9d8d13885.png

     

    After you enabled the docker service you can check if the RAM-Disk has been created (and its usage):

    image.png.c52c3e62319c9f0fdbb253cb8462d785.png

     

    Screenshot of changes in /etc/rc.d/rc.docker

    763244413_2021-08-2200_03_37.thumb.png.c16142adb6acaf74f90cd37382d2c490.png

     

    and /usr/local/emhttp/plugins/dynamix/scripts/monitor

    783908454_2021-08-2200_01_57.png.7942055f5871d164050cda1f415930bc.png

    • Like 16
    Link to comment
    Quote

    (if they are important to you, create a cron job)

     

     

    What specifically needs to go in the script for the job? The entire contents of the paste or just a certain part?

    Link to comment

    Great work!

     

    Will this work if the docker.img is on a different drive?

    For myself, '/mnt/cache-b/docker/docker.img" instead of "/mnt/cache/docker/docker.img"

    Link to comment
    On 3/2/2021 at 10:17 PM, muwahhid said:

    I wonder if I need to enter the NO HEALTHCHECK parameter for all these containers? Will they work stably after that?

     

    image.thumb.png.b6fc90d575a3a00c394ec4254952f2eb.png

    Beautiful work, this should be implemented as a feature for future releases if possible!

    Link to comment

    This is amazing work mgutt. This issue has plagued me for a long time and has destroyed two of my nice SSDs already.

     

    Hoping to see this officially fixed in an unraid update...

     

    Anyone have any idea if they'll officially reply?

    Link to comment
    On 8/17/2021 at 9:09 AM, mgutt said:

    @limetech

    I solved this issue as follows. Maybe it can be implemented similar in one of the next Unraid releases (should be enableable through the docker settings)?!

     

    Tested:

    - Unraid 6.9.2

    - Unraid 6.10.0-rc1

     

    Note: This enhancement works only if the Share "system" and the subfolder "docker" exists. I will remove this note if I found a way to support situations where the user deleted the "system" share or the subfolder.

     

    1. Add this to /boot/config/go (by Config Editor Plugin):
      # -------------------------------------------------
      # RAM-Disk for Docker json/log files
      # -------------------------------------------------
      # create RAM-Disk on starting the docker service
      sed -i '/^  echo "starting \$BASE ..."$/i \
        # move json/logs to ram disk\
        rsync -aH --delete /var/lib/docker/containers/ /mnt/user/system/docker/containers_backup\
        mount -t tmpfs tmpfs /var/lib/docker/containers\
        rsync -aH --delete /mnt/user/system/docker/containers_backup/ /var/lib/docker/containers\
        logger -t docker RAM-Disk created' /etc/rc.d/rc.docker
      # remove RAM-Disk on stopping the docker service
      sed -i '/^  # tear down the bridge$/i \
        # backup json/logs and remove RAM-Disk\
        rsync -aH --delete /var/lib/docker/containers/ /mnt/user/system/docker/containers_backup\
        umount /var/lib/docker/containers\
        rsync -aH --delete /mnt/user/system/docker/containers_backup/ /var/lib/docker/containers\
        logger -t docker RAM-Disk removed' /etc/rc.d/rc.docker
      # automatically backup RAM-Disk if Docker tab has been loaded (https://unix.stackexchange.com/a/198543/101920)
      sed -i '/^<?PHP$/a \
      if (file_exists("/var/lib/docker/containers")) {\
        exec("mkdir /var/lib/docker_bind");\
        exec("mount --bind /var/lib/docker /var/lib/docker_bind");\
        exec("rsync -aH --delete /var/lib/docker/containers/ /var/lib/docker_bind/containers");\
        exec("umount /var/lib/docker_bind");\
        exec("rmdir /var/lib/docker_bind");\
        exec("logger -t docker Created RAM-Disk backup");\
      }' /usr/local/emhttp/plugins/dynamix.docker.manager/DockerContainers.page

       

    2. Optional: Limit the Docker LOG size to avoid using too much RAM:
      1082044065_2021-08-1910_34_09.png.30f268aeff1d17f254eacd8d1e7bf9a8.png
    3. Reboot server

     

    Notes:

    • By this change /var/lib/docker/containers becomes a RAM-Disk which avoids wearing out your SSD and allows a permanent sleeping SSD (energy efficient)
    • Everytime Unraid WebGUI > Docker is opened, it creates automatically a backup (which is done for server crash / power-loss scenarios)
    • If you like to update Unraid OS you should remove the change from the Go File until it's clear that this enhancement is still working/needed!
    • If the server crashes / suffers from power-loss, recently added container log entries will be lost (if they are important to you, create a cron job)

     

    Your Reward:

    415470549_2021-08-1808_28_47.png.b552e3017be7cbf5a825f6e9d8d13885.png

     

    After you enabled the docker service you can check if the RAM-Disk has been created (and its usage):

    image.png.c52c3e62319c9f0fdbb253cb8462d785.png

     

    Screenshot of changes in /etc/rc.d/rc.docker

    1325731527_2021-08-1910_24_56.thumb.png.50bd3bf060cbd9d1ee8be835b19d8afc.png

     

    and /usr/local/emhttp/plugins/dynamix.docker.manager/DockerContainers.page

    1263749897_2021-08-1910_26_36.png.e652746508c4e16850618f701500889c.png

    That’s great! Thank you for the good work!

    Link to comment
    5 hours ago, Sunsparc said:

    What specifically needs to go in the script for the job? The entire contents of the paste or just a certain part?

    I updated the above code. Now it automatically syncs the RAM-Disk every 30 minutes. You can change it in the Go file to a lower value if you like to.

     

    5 hours ago, boomam said:

    Will this work if the docker.img is on a different drive?

    For myself, '/mnt/cache-b/docker/docker.img" instead of "/mnt/cache/docker/docker.img"

    The location of your docker.img is irrelevant. You can even use a docker directory, which is a method I recommend as it reduces write amplification.

    Link to comment
    1 hour ago, mgutt said:

    I updated the above code. Now it automatically syncs the RAM-Disk every 30 minutes. You can change it in the Go file to a lower value if you like to.

     

     

    Thank you!

    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.