[[6.8.3] docker image huge amount of unnecessary writes on cache]

12 hours ago, TexasUnraid said:

Ok, I added the SSD with LBA logging to the cache pool and have the docker image on an XFS drive and app data on the cache again.

 

The writes to the pool work out to around ~400mb/hour yet when the appdata was on an XFS drive it was a mere ~5mb/hour. Almost a 100X increase in writes seems pretty extreme and I can't explain it.

 

I think going with the direct mount docker is the best option at this point. Guess that is my next project.

 

For what it is worth, the direct mount did not actually fix it for me, just obfuscate it by making it so loop2/3 didn't show up in iotop. When I checked the SMART status I wound up with just as many writes as before, so I would be interested to hear your results.

It seems like I might be an outlier here, so possibly I have a different issue affecting my setup.

[[6.8.3] docker image huge amount of unnecessary writes on cache]

12 hours ago, S1dney said:

EDIT:

After some PMs with @caplam we can safely advise NOT to go down the copy over data rabbit hole.

It seemed like copying over the data will “inflate” it, most likely due to btrfs features like deduplication and/or layering/CoW. Not really into btrfs that much to fully explains what’s going on here, but I imagine you would be required to use of more specific tools to handle copying that btrfs specific data over.
Recreating the containers (and thus redowloading) creates all the needed btrfs subvolumes. If your not a specialist I would not recommend missing with the btrfs data 🙂

 

A note on this from my experience. I actually did this and it worked, but when I went to make changes to the dockers it failed and they would no longer start. I think I could have fixed it by clearing out the mountpoints, but I opted to just wipe the share and then re-add all the dockers from "my templates" which worked just fine.

 

So, can confirm  --would not recommend.

 

I did copy the libvirt folder though and have not noticed any ill effects (...yet. haha)

[[6.8.3] docker image huge amount of unnecessary writes on cache]

14 hours ago, S1dney said:

That's interesting, thanks for the work!

Do you have any metrics to share?

It's funny you mention that. I haven't rolled the workaround back yet, but I'm starting to wonder if it isn't fixing the issue so much as obfuscating it.

I'm planning on rolling the fix back later today/tonight to see what happens.

I got some metrics by basically doing the same thing as your script (except manually). According to SMART reporting, my cache drives are writing 16.93GB/hr.

This is even though when I implemented the workaround I also moved several VMs and a few docker paths to my nvme drives just to reduce the writes further.

 

I'd be curious to know what others are seeing.

[[6.8.3] docker image huge amount of unnecessary writes on cache]

For anyone else that needs it, I was having more issues with libvirt/loop3 than docker/loop2, so I adapted @S1dney's solution from here for libvirt.

 

A little CYA: To reiterate what has already been said, this workaround is not ideal and comes with some big caveats, so be sure to read through the thread and ask questions before implementing.

 

I'm not going to get into it here, but I used S1dney's same basic directions for the docker by making backups and copying files to folders in /boot/config/.

 

 

Create a share called libvirt on the cache drive just like for the docker instructions.

 

edit rc.libvirt 's start_libvirtd method as follows:

start_libvirtd() {
  if [ -f $LIBVIRTD_PIDFILE ];then
    echo "libvirt is already running..."
    exit 1
  fi
  if mountpoint /etc/libvirt &> /dev/null ; then
     echo "Image is mounted, will attempt to unmount it next."
     umount /etc/libvirt 1>/dev/null 2>&1
     if [[ $? -ne 0 ]]; then
       echo "Image still mounted at /etc/libvirt, cancelling cause this needs to be a symlink!"
       exit 1
     else
       echo "Image unmounted succesfully."
     fi
  fi
  # In order to have a soft link created, we need to remove the /etc/libvirt directory or creating a soft link will fail
  if [[ -d /etc/libvirt ]]; then
    echo "libvirt directory still exists, removing it so we can use it for the soft link."
    rm -rf /etc/libvirt
    if [[ -d /etc/libvirt ]]; then
      echo "/etc/libvirt still exists! Creating a soft link will fail thus refusing to start libvirt."
      exit 1
    else
      echo "Removed /etc/libvirt. Moving on."
    fi
  fi
  # Now that we know that the libvirt image isn't mounted, we want to make sure the symlink is active
  if [[ -L /etc/libvirt && -d /etc/libvirt ]]; then
    echo "/etc/libvirt is a soft link, libvirt is allowed to start"
  else
    echo "/etc/libvirt is not a soft link, will try to create it."
    ln -s /mnt/cache/libvirt /etc/ 1>/dev/null 2>&1
    if [[ $? -ne 0 ]]; then
      echo "Soft link could not be created, refusing to start libvirt!"
      exit 1
    else
      echo "Soft link created."
    fi
  fi
  # convert libvirt 1.3.1 w/ eric's hyperv vendor id patch to how libvirt does it in libvirt 1.3.3+
  sed -i -e "s/<vendor id='none'\/>/<vendor_id state='on' value='none'\/>/g" /etc/libvirt/qemu/*.xml &> /dev/null
  # remove <locked/> from xml because libvirt + virlogd + virlockd has an issue with locked
  sed -i -e "s/<locked\/>//g" /etc/libvirt/qemu/*.xml &> /dev/null
  # copy any new conf files we dont currently have
  cp -n /etc/libvirt-/*.conf /etc/libvirt &> /dev/null
  # add missing tss user account if coming from an older version of unRAID
  if ! grep -q "^tss:" /etc/passwd ; then
    useradd -r -c "Account used by the trousers package to sandbox the tcsd daemon" -d / -u 59 -g tss -s /bin/false tss
  fi
  echo "Starting libvirtd..."
  mkdir -p $(dirname $LIBVIRTD_PIDFILE)
  check_processor
  /sbin/modprobe -a $MODULE $MODULES
  /usr/sbin/libvirtd -d -l $LIBVIRTD_OPTS
}

 

Add this code the the go file in addition to the code for the docker workaround:

# Put the modified libvirt service file over the original one to make it not use the libvirt.img
cp /boot/config/service-mods/libvirt-service-mod/rc.libvirt /etc/rc.d/rc.libvirt
chmod +x /etc/rc.d/rc.libvirt