• [6.8.3] Unecessary overwriting of JSON-files in docker.img every 5 seconds


    mgutt
    • Minor

    @Valerio found this out first, but never received an answer. Today I found it out, too. But this is present since 2019 (or even longer).

     

    I would say it's a bug, as:

    • it prevents HDD/SSD spindown/sleep (depending on the location of docker.img)
    • it wears out the SSD in the long run (if docker.img is located here) - see this bug, too.
    • it prevents reaching CPU's deep sleep states

     

    What happens:

    • /var/lib/docker/containers/*/hostconfig.json is updated every 5 seconds with the same content
    • /var/lib/docker/containers/*/config.v2.json is updated every 5 seconds with the same content except of some timestamps (which shouldn't be part of a config file I think)

     

    Which docker containers:

    • verified are Plex (Original) and PiHole, but maybe this is a general behaviour

     

    As an example the source of hostconfig.json which was updated yesterday 17280x times with the same content:

    find /var/lib/docker/containers/40b4197fdea122178139e9571ae5f4040a2ef69449acf14e616010c7e293bb44 -ls -name hostconfig.json -exec cat {} \;
    2678289      4 -rw-r--r--   1 root     root         1725 Oct  8 13:46 /var/lib/docker/containers/40b4197fdea122178139e9571ae5f4040a2ef69449acf14e616010c7e293bb44/hostconfig.json
    {
       "Binds":[
          "/mnt/user/tv:/tv:ro",
          "/mnt/cache/appdata/Plex-Media-Server:/config:rw",
          "/mnt/cache/appdata/Plex-Transcode:/transcode:rw",
          "/mnt/user/movie:/movie:ro"
       ],
       "ContainerIDFile":"",
       "LogConfig":{
          "Type":"json-file",
          "Config":{
             "max-file":"1",
             "max-size":"50m"
          }
       },
       "NetworkMode":"host",
       "PortBindings":{
          
       },
       "RestartPolicy":{
          "Name":"no",
          "MaximumRetryCount":0
       },
       "AutoRemove":false,
       "VolumeDriver":"",
       "VolumesFrom":null,
       "CapAdd":null,
       "CapDrop":null,
       "Capabilities":null,
       "Dns":[
          
       ],
       "DnsOptions":[
          
       ],
       "DnsSearch":[
          
       ],
       "ExtraHosts":null,
       "GroupAdd":null,
       "IpcMode":"private",
       "Cgroup":"",
       "Links":null,
       "OomScoreAdj":0,
       "PidMode":"",
       "Privileged":false,
       "PublishAllPorts":false,
       "ReadonlyRootfs":false,
       "SecurityOpt":null,
       "UTSMode":"",
       "UsernsMode":"",
       "ShmSize":67108864,
       "Runtime":"runc",
       "ConsoleSize":[
          0,
          0
       ],
       "Isolation":"",
       "CpuShares":0,
       "Memory":0,
       "NanoCpus":0,
       "CgroupParent":"",
       "BlkioWeight":0,
       "BlkioWeightDevice":[
          
       ],
       "BlkioDeviceReadBps":null,
       "BlkioDeviceWriteBps":null,
       "BlkioDeviceReadIOps":null,
       "BlkioDeviceWriteIOps":null,
       "CpuPeriod":0,
       "CpuQuota":0,
       "CpuRealtimePeriod":0,
       "CpuRealtimeRuntime":0,
       "CpusetCpus":"",
       "CpusetMems":"",
       "Devices":[
          {
             "PathOnHost":"/dev/dri",
             "PathInContainer":"/dev/dri",
             "CgroupPermissions":"rwm"
          }
       ],
       "DeviceCgroupRules":null,
       "DeviceRequests":null,
       "KernelMemory":0,
       "KernelMemoryTCP":0,
       "MemoryReservation":0,
       "MemorySwap":0,
       "MemorySwappiness":null,
       "OomKillDisable":false,
       "PidsLimit":null,
       "Ulimits":null,
       "CpuCount":0,
       "CpuPercent":0,
       "IOMaximumIOps":0,
       "IOMaximumBandwidth":0,
       "MaskedPaths":[
          "/proc/asound",
          "/proc/acpi",
          "/proc/kcore",
          "/proc/keys",
          "/proc/latency_stats",
          "/proc/timer_list",
          "/proc/timer_stats",
          "/proc/sched_debug",
          "/proc/scsi",
          "/sys/firmware"
       ],
       "ReadonlyPaths":[
          "/proc/bus",
          "/proc/fs",
          "/proc/irq",
          "/proc/sys",
          "/proc/sysrq-trigger"
       ]
    }

     

    • Thanks 3


    User Feedback

    Recommended Comments

    I am using 6.9.0-beta29 (will go to 30 ASAP)  and do see those writes too.
    Noticed that only 2 of my containers end up writing to these 2 files which are the only containers that seem to have a docker health check available.

     

    Here is the reason why you see a write every 5 seconds for your plex container

    https://github.com/plexinc/pms-docker/blob/master/Dockerfile#L62

    HEALTHCHECK --interval=5s --timeout=2s --retries=20 CMD /healthcheck.sh || exit 1

     

    Edited by DasMarx
    • Thanks 2
    Link to comment
    Share on other sites

    You can disable those healthchecks by adding

    --no-healthcheck

    to the Extra Parameters of each docker container in the advanced view.

    • Thanks 1
    Link to comment
    Share on other sites
    4 hours ago, DasMarx said:

    You can disable those healthchecks by adding

    This solved it!!! 🥳

    116320200_2020-10-0910_02_27.thumb.png.1efe01ae9fc11241756681413a3bf7c6.png

     

    I had this suspect, too. But renaming healthcheck.sh to healthcheck.delete didn't help, so I gave up :|

     

    Added to the Plex container as follows:

    788449800_2020-10-0909_47_55.png.7161b69678a8bb55e5addc6b8a7ad844.png

     

    After that I verified it as follows:

     

    Docker -> Plex Icon -> Console:

    find /config/*/ -mmin -1 -ls > /config/recent_modified_files$(date +"%Y%m%d_%H%M%S").txt

    Results: Empty, which means no modified files in /mnt/user/appdata/Plex-Media-Server)

     

    WebGUI -> Webterminal:

    inotifywait -e create,modify,attrib,moved_from,moved_to --timefmt %c --format '%T %_e %w %f' -mr /var/lib/docker > /mnt/user/system/docker/recent_modified_files_$(date +"%Y%m%d_%H%M%S").txt

    Result: Empty, which means no writes in docker.img at all!!!

     

    Whats next?

    How should @limetech use this information to solve the issue? Add --no-healthcheck to all containers by default? I think something similar is needed as most people do not know how to monitor their disk traffic. They will only conclude "hey Unraid does not send this disk to sleep!" or "Unraid produces really much traffic on my SSD". ^^

    Edited by mgutt
    • Thanks 1
    Link to comment
    Share on other sites

    Not obvious on the best way to handle this as technically this is an issue with specific docker containers rather than an UnRAID issue.   There may be good reasons the maintainer of a particular container has set this up so that over-riding it becomes a bad idea.    I wonder if the Fix Common Problems plugin could be enhanced to detect this and suggest the fix?  Alternatively make it a configuration option in the docker templates?

    Link to comment
    Share on other sites
    46 minutes ago, itimpi said:

    here may be good reasons the maintainer of a particular container has set this up

    Yes you are right. Some people use this state inside of the container to check the run status or internet access. Its not how its meant to be, as for example the internet access check is absolutely crazy as 1 million users of the container, would result in 1 million google.com requests every x seconds ^^

     

    I'm also not sure why Plex is doing this every 5 seconds:

    https://github.com/plexinc/pms-docker/blob/master/root/healthcheck.sh

     

    They connect to http://tower:32400/identity, which returns an XML file and the content of this file is thrown away. This means they only check, if the internal webserver is running. And this is needed every 5 seconds? Are they crazy? ^^

     

    Besides of that. Are these JSON files really part of an usual docker installation or is this a special Unraid thing? I wonder why only Unraid users are complaining those permanent writes. Ok... maybe it has the most transparent traffic monitoring ;)

     

    If these writes are related to Docker only, we should open an issue here. Because only updating a timestamp (or nothing) inside a config file, does not really sound like it's working as it ment to be.

    Edited by mgutt
    Link to comment
    Share on other sites
    6 minutes ago, mgutt said:

    Besides of that. Are these JSON files really part of an usual docker installation or is this a special Unraid thing? I wonder why only Unraid users are complaining those permanent writes. Ok... maybe it has the most transparent traffic monitoring ;)

     

    This is nothing to do with Unraid - it is internal to the docker container.   It may have been more obvious to Unraid users because of the fact that a docker image is used and there was an issue in 6.8.3 and earlier that could cause significant write amplification when writing to that image stored on a SSD.   If the docker container files were stored directly in the file system (as the latest Unraid betas allow) then this is probably far less noticeable particularly if using a HDD with a file system like XFS that has far less inherent write amplification than BTRFS does.

     

    11 minutes ago, mgutt said:

    If these writes are related to Docker only, we should open an issue here. Because only updating a timestamp (or nothing) inside a config file, does not really sound like it's working as it meant to be.

    This could not do any harm although they may simply reply that the feature is being mis-used and the fix should come from the container maintainer.

     

    Link to comment
    Share on other sites
    53 minutes ago, itimpi said:

    mis-used and the fix should come from the container maintainer

    As the JSON files are overwritten by docker and not the container, it shouldn't be an issue of the maintainer. I'll try my luck ^^

    Link to comment
    Share on other sites

    hey, i found this on reddit and i tryed the following command:

     

    Quote

    inotifywait -e create,modify,attrib,moved_from,moved_to --timefmt %c --format '%T %_e %w %f' -mr /var/lib/docker > /mnt/user/system/docker/recent_modified_files_$(date +"%Y%m%d_%H%M%S").txt

     the output is attached.

     

    are the writings normal?

     

    recent_modified_files_20201010_100445.txt

    Link to comment
    Share on other sites

    @sonic6

    You have many requests because of a postgresql database. If you need it, there would be nothing you could optimize. But I would say these writes happen to often :

    MODIFY /var/lib/docker/containers/ced93fa7c199b90a5414e103be9871a25cae5b8e3fdaab059b1517e34ad0150f/ ced93fa7c199b90a5414e103be9871a25cae5b8e3fdaab059b1517e34ad0150f-json.log

    Check in your Container Dashboard for a container ID (advanced view) starting with "ced93fa7". Which docker is it? Does it have a logging setting which you could disable?

    Link to comment
    Share on other sites
    20 hours ago, mgutt said:

     

    
    MODIFY /var/lib/docker/containers/ced93fa7c199b90a5414e103be9871a25cae5b8e3fdaab059b1517e34ad0150f/ ced93fa7c199b90a5414e103be9871a25cae5b8e3fdaab059b1517e34ad0150f-json.log

    Check in your Container Dashboard for a container ID (advanced view) starting with "ced93fa7". Which docker is it? Does it have a logging setting which you could disable?

    this is the binhex-krusader container. Everything is like in the stock template with the "--no-healthcheck" command.

     

    maybe someone other than me runs the binhex-krusader contain and have the same?

    Link to comment
    Share on other sites

    Someone in Reddit found out that a container, which is using HEALTHCHECK, causes displaying the word "healthy" in the Unraid > Docker Dashboard (Advanced View):

    1114486258_2020-10-1401_28_54.png.5369600ae544fd95840adc9297fa0e43.png

     

    This makes it much easier to find those containers.

    Edited by mgutt
    Link to comment
    Share on other sites

    I am getting periodic writes to  a share on one of my disk drives.  They occur every few seconds. The writes are to the file "docker.img" in the system share which only exists on one hard drive. The system share contains two directories (docker, libvirt) each directory contains one image (docker.img in the docker directory; libvirt.img in the libvirt directory.

    I have entered the "--no-healthcheck" for my one docker container to no avail.

     

    Any help greatly appreciated

    Link to comment
    Share on other sites


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.