RAM-Disk for Docker status/log files

[mgutt]

To reduce wear on your SSD this script moves your docker status and log files into your RAM. This script syncs every 30 minutes the RAM content to your SSD to reduce loss of log files.

 

Note: Tested by now only with Unraid 6.9 and 6.10. Please return feedback for further versions.

 

Paste this in your Go File:

 

# -------------------------------------------------
# RAM-Disk for Docker json/log files v1.4
# -------------------------------------------------

# create RAM-Disk on starting the docker service
sed -i '/^  echo "starting \$BASE ..."$/i \
  # move json/logs to ram disk\
  rsync -aH --delete /var/lib/docker/containers/ ${DOCKER_APP_CONFIG_PATH%/}/containers_backup\
  mountpoint -q /var/lib/docker/containers || mount -t tmpfs tmpfs /var/lib/docker/containers || logger -t docker Error: RAM-Disk could not be mounted!\
  rsync -aH --delete ${DOCKER_APP_CONFIG_PATH%/}/containers_backup/ /var/lib/docker/containers\
  logger -t docker RAM-Disk created' /etc/rc.d/rc.docker

# remove RAM-Disk on stopping the docker service
sed -i '/^  # tear down the bridge$/i \
  # backup json/logs and remove RAM-Disk\
  rsync -aH --delete /var/lib/docker/containers/ ${DOCKER_APP_CONFIG_PATH%/}/containers_backup\
  umount /var/lib/docker/containers || logger -t docker Error: RAM-Disk could not be unmounted!\
  rsync -aH --delete ${DOCKER_APP_CONFIG_PATH%/}/containers_backup/ /var/lib/docker/containers\
  if [[ -d /var/lib/docker_bind ]]; then umount /var/lib/docker_bind || logger -t docker Error: RAM-Disk bind unmount failed while docker stops!; fi\
  logger -t docker RAM-Disk removed' /etc/rc.d/rc.docker

# Automatically backup Docker RAM-Disk
sed -i '/^<?PHP$/a \
$sync_interval_minutes=30;\
if ( ! ((date("i") * date("H") * 60 + date("i")) % $sync_interval_minutes) && file_exists("/var/lib/docker/containers")) {\
  exec("\
    [[ ! -d /var/lib/docker_bind ]] && mkdir /var/lib/docker_bind\
    if ! mountpoint -q /var/lib/docker_bind; then\
      if ! mount --bind /var/lib/docker /var/lib/docker_bind; then\
        logger -t docker Error: RAM-Disk bind mount failed!\
      fi\
    fi\
    if mountpoint -q /var/lib/docker_bind; then\
      rsync -aH --delete /var/lib/docker/containers/ /var/lib/docker_bind/containers && logger -t docker Success: Backup of RAM-Disk created.\
      umount -l /var/lib/docker_bind\
    else\
      logger -t docker Error: RAM-Disk bind mount failed!\
    fi\
  ");\
}' /usr/local/emhttp/plugins/dynamix/scripts/monitor

 

If you don't want to restart your server, paste it in your terminal, too (has to be executed only once!)

 

How it works

 

Docker service is started

- It creates the path /mnt/<path_to_your_docker_data>/containers_backup

- It copies all files from /var/lib/docker/containers to /containers_backup

- It creates a RAM disk /var/lib/docker/containers (on top of the already existing path, so the already existing path is not deleted)

- It copies all files from /containers_backup back to the now empty RAM disk /var/lib/docker/containers

 

Docker service is stopped

- Reverses steps from before

 

Every 30 minutes

- It creates the path /var/lib/docker_bind

- It mounts the path /var/lib/docker to /var/lib/docker_bind (this trick allows as to access the local path /var/lib/docker/containers)

- It copies all files from the RAM Disk /var/lib/docker/containers to /var/lib/docker_bind/containers (which is the local /var/lib/docker/containers path)

 

Reduce RAM Usage

If you fear to much RAM usage, you can change your docker settings as follows:

 

By that each container can only create a single log file which is limited to 50 MB.

 

How much RAM is used

Usually less then 512 MB:

# df -h /var/lib/docker/containers
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            32G  369M   31G   2% /var/lib/docker/containers

 

Does it really help?

For me it made a huge change (most of the time absolutely no writes on my SSD). But read the reactions of other users:

https://forums.unraid.net/bug-reports/stable-releases/683-unnecessary-overwriting-of-json-files-in-dockerimg-every-5-seconds-r1079/?do=findComment&comment=15472

 

[ICDeadPpl]

It gave me an error until I added a "\" on this line:
 

umount -l /var/lib/docker_bind\

 

[kaiguy]

Thanks for sharing this. Using your script (with the minor edit that @ICDeadPpl highlighted--it was erroring out for me as well without adding the backslash).

 

What is your recommendation, @mgutt, for identifying what is making /var/docker/overlay2 writes? I'm getting pretty consistent writes still even after incorporating the script.

Edited March 10 by kaiguy
more words

[mgutt]

On 3/9/2023 at 10:35 AM, ICDeadPpl said:

It gave me an error until I added a "\" on this line:
 

Thanks!

 

On 3/10/2023 at 9:38 PM, kaiguy said:

What is your recommendation, @mgutt, for identifying what is making /var/docker/overlay2 writes?

Repeat this command every minute or so:

 

find /var/lib/docker -type f -not -path "*/diff*" -print0 | xargs -0 stat --format '%Y:%.19y %n' | sort -nr | cut -d: -f2- 2> /dev/null | head -n30 | sed -e 's|/merged|/...|; s|^[0-9-]* ||'

 

Now you should know which files are permanently updated.

 

The next step is to find which path belongs to which container:

 

csv="CONTAINER;PATHS\n"; for f in /var/lib/docker/image/*/layerdb/mounts/*/mount-id; do subid=$(cat $f); idlong=$(dirname $f | xargs basename); id="$(echo $idlong | cut -c 1-12)"; name=$(docker ps --format "{{.Names}}" -f "id=$id"); [[ -z $name ]] && continue; csv+="\n"$(printf '=%.0s' {1..20})";"$(printf '=%.0s' {1..100})"\n"; [[ -n $name ]] && csv+="$name;" csv+="/var/lib/docker/(btrfs|overlay2).../$subid\n"; csv+="$id;"; csv+="/var/lib/docker/containers/$idlong\n"; for vol in $(docker inspect -f '{{ range .Mounts }}{{ if eq .Type "volume" }}{{ .Destination }}{{ printf ";" }}{{ .Source }}{{ end }}{{ end }}' $id); do csv+="$vol\n"; done; done; echo ""; echo -e $csv | column -t -s';'; echo "";

 

Now you could think about how to solve your problem. Maybe the container is writing logs inside the container volume? Or permanently writes to a /tmp dir? Then you could add this path to your container settings and link it to unraids /tmp, which is inside the RAM (and will be lost on reboot of course):

 

 

Thats how I reduced my writes by 99%. But note: After reboot the paths seem to get the wrong permissions. That's why I added this to my go file:

# -------------------------------------------------
# Correct wrong container path permissions on server reboot
# -------------------------------------------------
mkdir -m 777 -p /tmp/nginx-proxy-manager/var/log
mkdir -m 777 -p /tmp/pihole/var/log/lighttpd
mkdir -m 777 -p /tmp/minecraft/paper/tmp
mkdir -m 777 -p /tmp/minecraft/vanilla/tmp

 

More information in this German thread:

https://forums.unraid.net/topic/112617-ssd-abnutzung-maßgeblich-reduzieren/

[PeterHS]

I've used the ideas here and I have definitely reduced my disk writes on v.6.11.5 by at least 60%.

I'm still trying to catch more of them and minimize/move to RAM. But I'm wondering how many writes people see during idle?

 

I've tracked mine during the last 4 days and I'm seeing between 300k-600k writes per 24h on 6-7 dockers + HomeAssistant VM. I honestly don't know if that's a lot or not? Should I be satisfied or do I need to go deeper?

[Patty92]

Hey,

I installed a fresh 6.12.0-rc2 on a test system to test a few things.
Here I noticed when I use the script "RAM disk for Docker json/log files v1.3" it comes to the error:

 

Mar 22 05:48:01 TOWER crond[1018]: exit status 255 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null
Mar 22 05:49:01 TOWER crond[1018]: exit status 255 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null
Mar 22 05:50:01 TOWER crond[1018]: exit status 255 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null
Mar 22 05:51:01 TOWER crond[1018]: exit status 255 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null

 

This error occurs every full minute.
After I removed the script and restarted the system, the error no longer occurs.
Greetings Patty
 

[mgutt]

On 3/22/2023 at 2:04 PM, Patty92 said:

After I removed the script and restarted the system, the error no longer occurs.

Thank you for your report. This bug has been fixed. I simply forgot to use quotes around "i" and "H" in this line:

 

It seems the recent PHP version is a little bit more picky ^^

[PeterHS]

How do I change the sync interval ?

 

I tried: $sync_interval_minutes=60;\  and this works.

But: $sync_interval_minutes=90;\   or  $sync_interval_minutes=120;\  doesn't. It still runs every full hour.

[mgutt]

2 hours ago, PeterHS said:

But: $sync_interval_minutes=90;\   or  $sync_interval_minutes=120;\  doesn't

Yupp. 60 minutes is the maximum by now.

[Phoenix Down]

Just wanted to say thank you @mgutt! I had an earlier version of your RAM Disk installed. Wasn't even sure of the version, as it didn't have the version number in the comments. Must have been from a couple of years ago. In any case, after upgrading to 6.11 recently, I noticed that my Disk 1 would have writes to it frequently, which wakes it up along with my two parity disks. This is despite me emptying all contents from Disk 1, and all of my Dockers running from the SSD cache pool. Also, new writes should have gone to the cache pool and not the array. I spent hours watching iotop and dstat, and I was about to pull my hair out when I noticed that Disk 1 would only wake up when certain Docker container is running (specifically DiskSpeed and HomeAssistant). On a whim, I looked to see if there is a newer version of the RAM Disk available, and found this thread. I updated the RAM Disk code, and viola! No more disks waking up! Still not sure why certain Dockers are writing directly to the array or why it's always Disk 1, but I'm glad the new code fixed the issue

Edited April 24 by Phoenix Down

[Niklas]

v1.4 of this script will not play nice with 6.12.0-rc6 (and later?).

 

My docker share (on zfs pool) with docker-xfs.img would not allow it to unmount the docker mount (busy) when rebooting the server or stopping the docker service. Ends up with unclean shutdown when unraid forces the reboot (starts parity check when rebooted).

 

Edited May 21 by Niklas

[kennymc.c]

Does the above mentioned problem still occur with 6.12 stable or do we have to wait for an update of the script? Unfortunately, the root issue itself still seems to be successfully ignored by @limetech

[kennymc.c]

@mgutt Were you able to test this with one of the 6.12 releases? 

[mgutt]

2 hours ago, kennymc.c said:

Were you able to test this with one of the 6.12 releases? 

Only a few posts above you can see, it was tested with one of the release candidate versions. 

[kennymc.c]

I already saw that post but my question was referring to the stable versions of 6.12 and if this could potentially be fixed.

Edited June 30 by kennymc.c

[caplam]

Just a post to thank @mgutt

 

I installed it under 6.11.5 with a sync interval of 60min.

Works like a charm. My ssd says thanks.

I use a single ssd (xfs) with docker directory.

Upgraded to 6.12.1 with no problem.

Edited July 7 by caplam

[kennymc.c]

I finally tried it today with 6.12.2 but unlike the post above me I still get an error message when stopping Docker:

 

Jul  8 19:31:15 smartserver root: stopping dockerd ...
Jul  8 19:31:17 smartserver root: umount: /var/lib/docker_bind: not mounted.
Jul  8 19:31:17 smartserver docker: Error: RAM-Disk bind unmount failed while docker stops!
Jul  8 19:31:17 smartserver docker: RAM-Disk removed
Jul  8 19:31:17 smartserver emhttpd: shcmd (28898): umount /var/lib/docker

 

The only difference is the default sync time of 30 minutes and instead of a ssd formatted in XFS I use BTRFS. No errors seem to occur during docker startup and the sync process.

[sonic6]

On 7/8/2023 at 8:01 PM, kennymc.c said:

I finally tried it today with 6.12.2 but unlike the post above me I still get an error message when stopping Docker:

 

@mgutt any chance to fix this?

[rkotara]

Seems like a logic issue when a mount point (directory) is present and not a "mount".  The code in this line is looking for a directory:

if [[ -d /var/lib/docker_bind ]]; then umount /var/lib/docker_bind || logger -t docker Error: RAM-Disk bind unmount failed while docker stops!; fi\

 

 

But maybe something like this would work better since it actually checks for a mount:

mount | awk '{if ($3 == "/var/lib/docker_bind") { exit 0}} ENDFILE{exit -1}' && umount /var/lib/docker_bind || logger -t docker Error: RAM-Disk bind unmount failed while docker stops!\

Of course, I cant really test this since I'm not using the script yet, I'm just following along until I upgrade and add this afterward or hope its part of future "improvements".

 

Edited July 17 by rkotara

[Mainfrezzer]

Either im going completely coo-coo or in 12.3 something changed. I tested it with 12.2 a while back and it just worked but now ive updated to 12.3 and the /etc/rc.d/rc.docker file wont get edited at all. /usr/local/emhttp/plugins/dynamix/scripts/monitor is still recieving the change.


Update:
I can gladly announce, i am not going crazy. The comments in the file are gone. 

 

sed -i '/^  echo "starting \$BASE ..."$/i \

can be worked around with 
 

sed -i '/^  echo "starting \$DOCKERD ..."$/i \

but stopping docker has changed the 
 

sed -i '/^  # tear down the bridge$/i \

into a 15 count loop that has the comment. I would at least expect that one to be inserted there but for some reason it didnt. So there might be more funky magic going on.


Update #2

Got it to work

 

# -------------------------------------------------
# RAM-Disk for Docker json/log files v1.4 for 6.12.3
# -------------------------------------------------

# create RAM-Disk on starting the docker service
sed -i '/^  echo "starting \$DOCKERD ..."$/i \
  # move json/logs to ram disk\
  rsync -aH --delete /var/lib/docker/containers/ ${DOCKER_APP_CONFIG_PATH%/}/containers_backup\
  mountpoint -q /var/lib/docker/containers || mount -t tmpfs tmpfs /var/lib/docker/containers || logger -t docker Error: RAM-Disk could not be mounted!\
  rsync -aH --delete ${DOCKER_APP_CONFIG_PATH%/}/containers_backup/ /var/lib/docker/containers\
  logger -t docker RAM-Disk created' /etc/rc.d/rc.docker

# remove RAM-Disk on stopping the docker service
sed -i '/^        # tear down the bridge/i \
  # backup json/logs and remove RAM-Disk\
  rsync -aH --delete /var/lib/docker/containers/ ${DOCKER_APP_CONFIG_PATH%/}/containers_backup\
  umount /var/lib/docker/containers || logger -t docker Error: RAM-Disk could not be unmounted!\
  rsync -aH --delete ${DOCKER_APP_CONFIG_PATH%/}/containers_backup/ /var/lib/docker/containers\
  if [[ -d /var/lib/docker_bind ]]; then umount /var/lib/docker_bind || logger -t docker Error: RAM-Disk bind unmount failed while docker stops!; fi\
  logger -t docker RAM-Disk removed' /etc/rc.d/rc.docker

# Automatically backup Docker RAM-Disk
sed -i '/^<?PHP$/a \
$sync_interval_minutes=30;\
if ( ! ((date("i") * date("H") * 60 + date("i")) % $sync_interval_minutes) && file_exists("/var/lib/docker/containers")) {\
  exec("\
    [[ ! -d /var/lib/docker_bind ]] && mkdir /var/lib/docker_bind\
    if ! mountpoint -q /var/lib/docker_bind; then\
      if ! mount --bind /var/lib/docker /var/lib/docker_bind; then\
        logger -t docker Error: RAM-Disk bind mount failed!\
      fi\
    fi\
    if mountpoint -q /var/lib/docker_bind; then\
      rsync -aH --delete /var/lib/docker/containers/ /var/lib/docker_bind/containers && logger -t docker Success: Backup of RAM-Disk created.\
      umount -l /var/lib/docker_bind\
    else\
      logger -t docker Error: RAM-Disk bind mount failed!\
    fi\
  ");\
}' /usr/local/emhttp/plugins/dynamix/scripts/monitor

 

Edited July 26 by Mainfrezzer

[kennymc.c]

On 7/17/2023 at 11:30 PM, rkotara said:

Seems like a logic issue when a mount point (directory) is present and not a "mount".  The code in this line is looking for a directory:

if [[ -d /var/lib/docker_bind ]]; then umount /var/lib/docker_bind || logger -t docker Error: RAM-Disk bind unmount failed while docker stops!; fi\

 

 

But maybe something like this would work better since it actually checks for a mount:

mount | awk '{if ($3 == "/var/lib/docker_bind") { exit 0}} ENDFILE{exit -1}' && umount /var/lib/docker_bind || logger -t docker Error: RAM-Disk bind unmount failed while docker stops!\

Of course, I cant really test this since I'm not using the script yet, I'm just following along until I upgrade and add this afterward or hope its part of future "improvements".

 

 

This doesn't work for me with 6.12.3. I can't see any logs related to the RAM disk script when replacing this line and stopping Docker.

[Mainfrezzer]

32 minutes ago, kennymc.c said:

 

This doesn't work for me with 6.12.3. I can't see any logs related to the RAM disk script when replacing this line and stopping Docker.

mhmm thats odd. i did test it with a brand new usb/install.

Is the /etc/rc.d/rc.docker file changed at all?

You gotta excuse me for my awful picture quality.

[kennymc.c]

I wasn’t quoting your post. Your fix is working fine which doesn’t include the updated line

[Mainfrezzer]

1 minute ago, kennymc.c said:

I wasn’t quoting your post. Your fix is working fine which doesn’t include the updated line

oh right, my bad, looked like it. just flew over the page an looked like mine.

[Unpack5920]

Thank you Manfreezer,

I can confirm, it is working for me now with 6.12.3.

 

root@TOWER:/# mount | grep tmpfs | grep docker
tmpfs on /var/lib/docker/containers type tmpfs (rw,relatime,inode64)
tmpfs on /mnt/cache/system/docker/containers type tmpfs (rw,relatime,inode64)