S1dney

Agreed! "Urgent" might be to generic in that it sums up "Server crash", "data loss" and "showstopper" under one caller. For now it seems to be a showstopper for a bunch of people so it's still accurate. If a new category is created, let me know and I'll adjust 👍

Changed Priority to Urgent >> Since I noticed this thread getting more and more attention lately, and more and more people urging it to be urgent instead of minor, I'll raise priority on this one. Just an FYI, I made/kept it minor initially cause I had a workable workaround that I felt satisfied with. If the Command Line Interface isn't really your thing or you have any other reason to not tweak the OS in an unsupported way I can fully understand this frustration. In the end... The community decides priority. Also updated the title to version 6.8.3 as requested. Cheers

Seems you're just hitting this bug and for it, this is expected behavior I think It seems like every write docker does on the cache multiplies by 10 (at least). I recall seeing similar behavior, whenever docker starts to write, it hammers big time.

The devs were involved in this topic already, but this has not yielded any results yet. No proper solution yet. Only way to get around those excessive writes (that I found) is by putting the loop device out of the equation. I've described a (non-supported) way to do so, which works well for me. You're running in a BTRFS pool if I understand correctly? Unencrypted? Thought this was solely based on encrypted pools. What you could to is install the "Nerd Pack" from the Community Applications, then head over to Setting -> "Nerd Pack" and install "iotop". Then from the commandline run "iotop -ao" (shows an accumulated view of all processes that actually read and write). See if one partical item (besides loop2) stand out in a big way, this could indicate one container is running wild, I have found topics were certain databases would write tremendously on the cache.

Well, I'm actually expecting it to break at one point, which is why I've made a copy of the original rc.docker script so I can diff that against a possible new rc.docker script after a future upgrade. In short I just modified the start_docker function inside the rc.docker script to unmount the docker image, remove the /var/lib/docker directory and create a symlink to a directory directly on my cache. Simple hack to put the loopdevice out of the equation. Also I very much agree that this hack isn't a real fix, but I've bought expensive Samsung EVO SSD's for the 5 year warranty on them and for me it was not acceptable to void the warranty in two years (or less) due to writes that were going into a black hole. I noticed this problem had been around for a long while and I wasn't expecting a fix soon, hence I created my own temporary patch. I can also imagine @Thor isn't really keen on waiting for a fix since his drives are overheating non stop. Now I did actually made a suggestion in my original post, but this would rather be a workaround, since the root cause of the massive writes would be better solved: Like said, I understand your point and understand that as a developer you're wary for modifications to system scripts that might cause failures later on. For me it was the only way to work around docker hammering on the cache like crazy and felt like I needed to share it for someone that has similar issues, of course with a warning that modifying these scripts comes with a risk I'm sure this bug report will get bumped once in a while and will eventually be solved, but I have a workable solution for now. Appreciate the work! 🙌

There a bug report for this now: See report.

Hey man, If you reboot your server everything will be reset to default and unRaid will mount the normal docker image upon boot. Assuming you've created the share which I've mentioned, then docker won't be able to see those, since they work created on the path targeted by the symlink and not inside the docker.img file. So it will see/start whatever container you have created into the docker image, or nothing if you created a new one before starting to work with the symlink approach. I've read though my posts real quick and noticed I have not yet provided my final solution, so let me share that. Basically what I did was: Create a share named docker (which has to be cache only or this will break after the mover kicks in!) Created a directory "docker-service-mod" at /boot/config/, which will obviously survive a reboot since it's on flash. --> command: mkdir /boot/config/docker-service-mod Copied the original rc.docker file to flash also (this allows me to easily check if the unRaid devs have made a change in a later version if my docker service fails all of a sudden). --> command: cp /etc/rc.d/rc.docker /boot/config/docker-service-mod/rc.docker.original Remove the rc.docker file --> command: rm /etc/rc.d/rc.docker Create a new /etc/rc.d/rc.docker file with everything that was in the other one, but replace the docker_start() function with a custom one (as defined code block below). To get this on unRaid you have multiple ways. You could use vi to edit the function directly the file at /etc/rc.d/rc.docker (vi has funky syntax though if you're not a regular vi user, I believe unRAID also has nano installed which is more user friendly for non-vi users) . You can also create a file named rc.docker locally (on Windows for example), copy the content of the original rc.docker file in it, make the changes to the start_docker() function and use WinSCP to copy it to /etc/rc.d/. If you copy it from Windows, make it executable with "chmod +x /etc/rc.d/rc.docker" (not sure if it's needed, but setting the execute bit on there won't hurt for sure here, since it's a script) # Start docker start_docker(){ if is_docker_running; then echo "$DOCKER is already running" return 1 fi if mountpoint $DOCKER_ROOT &>/dev/null; then echo "Image is mounted, will attempt to unmount it next." umount $DOCKER_ROOT 1>/dev/null 2>&1 if [[ $? -ne 0 ]]; then echo "Image still mounted at $DOCKER_ROOT, cancelling cause this needs to be a symlink!" exit 1 else echo "Image unmounted succesfully." fi fi # In order to have a soft link created, we need to remove the /var/lib/docker directory or creating a soft link will fail if [[ -d $DOCKER_ROOT ]]; then echo "Docker directory still exists, removing it so we can use it for the soft link." rm -rf $DOCKER_ROOT if [[ -d $DOCKER_ROOT ]]; then echo "$DOCKER_ROOT still exists! Creating a soft link will fail thus refusing to start docker." exit 1 else echo "Removed $DOCKER_ROOT. Moving on." fi fi # Now that we know that the docker image isn't mounted, we want to make sure the symlink is active if [[ -L $DOCKER_ROOT && -d $DOCKER_ROOT ]]; then echo "$DOCKER_ROOT is a soft link, docker is allowed to start" else echo "$DOCKER_ROOT is not a soft link, will try to create it." ln -s /mnt/cache/docker /var/lib 1>/dev/null 2>&1 if [[ $? -ne 0 ]]; then echo "Soft link could not be created, refusing to start docker!" exit 1 else echo "Soft link created." fi fi echo "starting $BASE ..." if [[ -x $DOCKER ]]; then # If there is an old PID file (no docker running), clean it up: if [[ -r $DOCKER_PIDFILE ]]; then if ! ps axc|grep docker 1>/dev/null 2>&1; then echo "Cleaning up old $DOCKER_PIDFILE." rm -f $DOCKER_PIDFILE fi fi nohup $UNSHARE --propagation slave -- $DOCKER -p $DOCKER_PIDFILE $DOCKER_OPTS >>$DOCKER_LOG 2>&1 & fi } Copy the new rc.docker file to flash. --> command: cp /etc/rc.d/rc.docker /boot/config/docker-service-mod/rc.docker Modify the /boot/config/go file, so that unRaid injects the modified version of the rc.docker file into the /etc/rc.d/ directory before starting emhttp (which will also start docker). I used vi for that. My go file has several other things in it, but the relevant part is below. The chmod +x command might not be necessary cause it worked also before I used it, however I feel more comfortable knowing it explicitly sets the execution bit: #!/bin/bash # Put the modified docker service file over the original one to make it not use the docker.img cp /boot/config/docker-service-mod/rc.docker /etc/rc.d/rc.docker chmod +x /etc/rc.d/rc.docker # Start the Management Utility /usr/local/sbin/emhttp & I've been using this for a while and have been very happy with it so far. Be aware though that this might break one day if Limetech decides that the rc.docker script should be modified (which is why I keep a copy of the original one like mentioned). It could also break if Limetech steps away from the Go file. Simple thing you can do is recreate the docker.img file from the GUI before you take the steps above (this will destroy each container and also any data that you have not made persistent!), then if a future update breaks this, you'll have no docker container running (since docker is started on a blank image). You should be aware of this pretty quick. Only one side effect of this approach I was able to determine is that opening the docker settings page takes a while to load, cause the GUI tries to run some commands on the BTRFS filesystem of the image (which is not there anymore). This will freeze up the settings page for a bit and push some resource usage in one CPU core until that command times out. Not a deal breaker for me, if you want to turn off docker, then simple wait until the command times out and reveals the rest of the page (without BTRFS status info). Good luck!

Yeah it should indeed. You can check with: df -hm /dev/loop2 This will probably show you mounted on /var/lib/docker. Now the fact that you're moving the docker image between the user share appdata and the mountpoint directly doesn't really help, since docker is still running inside an image file, mounted via a loop device on your cache. There seems to be a bug with using docker in this kind of setup, although I wasn't able to reproduce this on another linux distro. Might be a Slackware thing. The only way (I could come up with) to get the writes down is to create a symlink between /mnt/cache/docker (or whatever cache only dir you create) and /var/lib/docker and then start docker. The start_docker() function I've modified inside the rc.docker script does that and some other things, like checking whether the docker image is already mounted, and if so, unmount it.

You're saying you're moving the mariadb location between the array, the cache location (/mnt/user/appdata) and directly onto the mountpoint (/mnt/cache). The screenshots seem to point out that the loop device your docker image is using is still the source of a lot of writes though (loop2)? One way to avoid this behavior would be to have docker mount on the cache directly, bypassing the loop device approach. I had similar issues and I'm running directly on the BTRFS cache now for quite some time. Still really happy with it. I wrote about my approach in the bug report I did here: Note that in order to have it working on boot automatically I modified the start_docker() function and copied the entire /etc/rc.d/rc.docker file to /boot/config/docker-service-mod/rc.docker. My go file copies that back over the original rc.docker file so that when the docker deamon is started, the script sets up docker to use the cache directly. Haven't got any issues so far 🙂 My /boot/config/go file looks like this now (just the cp command to rc.docker is relevant here, the other lines before are for hardware acceleration on Plex) #!/bin/bash # Load the i915 driver and set the right permissions on the Quick Sync device so Plex can use it modprobe i915 chmod -R 777 /dev/dri # Place the modified docker rc.d script over the original one to make it not use the docker.img cp /boot/config/docker-service-mod/rc.docker /etc/rc.d/rc.docker # Start the Management Utility /usr/local/sbin/emhttp & Cheers.

Well, if I read your initial post on 01-11-2010, there is a security vulnerability discovered in the form-based authentication. If all users are strongly encouraged to update/patch this flaw, I'd really also like to deploy it of course. I wasn't able to find any of the releases that included the kernel 5.4 with it, they all seem to revert me back to a kernel version below 5. Thanks again! REF:

Hey Guys, Is this patch also available for those holding on 6.8rc7? I want to stay on kernel 5.4, but leaving my host vulnerable is also not that appealing. Appreciate the help! Cheers

No problem! Cheers.

I cannot recall docker being extremely slow, but judging the screenshot of iotop this is indeed exactly the problem I was facing. Is that cache encrypted?

You're welcome! I think a lot of us are.. Without knowing. If you're not troubleshooting the cache you'll likely not notice this, unless you're questioned about why the metrics are way off between disks

I had the same. Docker links are deprecated though.. https://docs.docker.com/network/links/ I solved it by creating a network for the container. I believe you "can" still use it by omitting the --net switch on your command, which will link it to the default bridge network: https://github.com/docker/cli/issues/2030 However it's deprecated for a reason 🙂 Note that this does require running a command to create the docker container cause the --net switch will be fed to docker deamon through DockerMan if using the GUI.

Also you might want to run "btrfs balance start -mconvert=raid1 /mnt/cache" against your pool cause your setup isn't that redundant at the moment 🙂 Data Metadata System Id Path RAID1 single(!) single(!) Unallocated -- --------- --------- -------- -------- ----------- 1 /dev/sdt1 250.00GiB 2.01GiB 4.00MiB 213.75GiB 2 /dev/sdu1 250.00GiB 2.00GiB - 213.76GiB -- --------- --------- -------- -------- ----------- Total 250.00GiB 4.01GiB 4.00MiB 427.51GiB Used 219.61GiB 1.96GiB 64.00KiB If one of your drives fails now, your in bad luck. See:

Good to hear you have it resolved. Seemed like a different problem indeed

That's correct. I've initially tried to set the docker vDisk path to both the unRAID filesystem (/mnt/user/cache) as directly onto the btrfs mountpoint (/mnt/cache) from within the docker settings, neither made a noticeable change. So therefore was able to rule out the unRAID filesystem.

I spent a lot of time trying to find similar situations on Google as well, but was able to find any. The problem isn't really btrfs atop of dm-crypt, as having docker writing to the encrypted cache only produces 27 GB of (daily) writes in comparison to having docker mounted on the loop device, which resulted in 400 GB of daily writes with a lot of less activity compared to now (I've increased the amount of docker in the meantime). @hotio's problem matches my case exactly though. Unfortunately I'm lacking the knowledge at the moment to dive in at the level of looking into/fixing the amount of page flushes etc (like @limetech mentioned). I was able to pinpoint it to the combination of docker, the loop device and (most likely) dm-crypt. Hence this bugreport. Appreciate the efforts! 🍻

Great! Thanks for taking the time, I'm curious to see where the investigation leads to! I must say that I also tried to decrypt my pool once and had the feeling that excessive writes still took place (I switched back after a short while though, cause running unencrypted was not an option for me). My writes never went to the level @szymon described though. You can tweak the docker service file once and move the docker files out of the docker image like I described, to see if it's really the loop device causing the issues, in my case it solved my issue (still running directly onto the cache mountpoint).

Sure, you're welcome. Let me know how that went, kind of curious

Have you tried to delete (and thus automatically recreate) the docker.img already? Mine was just doing a lot of writes on disk, but none of the errors you're describing (nor the CPU and RAM usage) was happening to me. Seems like to reddit thread you mentioned below (not sure if you created that) seems to have csum errors, which I assume are checksum errors. This might point to corruption of your docker image, the loop2 device is also formatted with btrfs.

Hahah thanks, trial and error though this time You're right. It was a long night I will adjust it right away, I meant the mountpoint of the cache indeed instead of a user share. Yeah I was aware of this, I wasn't affected by the bug though so I haven't dug into this too deep. I might try a few versions later to switch back to the /mnt/user shares, but until that time for me this is a perfect workaround.

Well isn't that great, of course the Bitwarden version is crashing. I just have to wait until they decide to upgrade the SQL container to a newer version. I did notice the MCR version of SQL doesn't require the ulimits override anymore so that's nice. I've collected my Bitwarden MSSQL logging and rolled back to 6.7.2, everything's fine now (so said cause I liked the new features). Thanks for confirming this. EDIT: Now that were an additional four hours well spend. So apparently the combination of the newer docker build with the MSSQL container trips over mounting volumes relatively to /mnt/user/appdata (so the unRAID filesystem). The docker version shipped with unRAID version 6.7.2 (that was 18.x?) doesn't seem to care but now on 19.x this becomes a problem all of a sudden. The reason why your container worked (I assume) was that you mounted directly onto the btrfs (orwhichever filesystem you're using) mountpoint on /mnt/cache. Docker seems to now like this better. Although this would break if the mover kicks in I don't really mind cause my appdata is cache only anyways. I've updated the docker-compose.override.yml file to map them differently and the container is running fine now. # # Override file for the auto-generated docker-compose.yml file provided # by Bitwarden # This file mounts the volumes inside the MSSQL container directly onto # the btrfs mountpoint instead of a relative path in the unRAID # filesystem as the MSSQL container fails otherwise. # The file also sets ulimits on the same MSSQL container, because with the # ulimit stack size set to ulimited systemwide (as is the case on unRaid), # the container refuses to start. # ######################################################################### version: '3' services: mssql: volumes: - /mnt/cache/appdata/bitwarden/bwdata/mssql/data:/var/opt/mssql/data - /mnt/cache/appdata/bitwarden/bwdata/logs/mssql:/var/opt/mssql/log - /mnt/cache/appdata/bitwarden/bwdata/mssql/backups:/etc/bitwarden/mssql/backups ulimits: stack: soft: "8192000" hard: "8192000" Cheers! 🍻

Oh darn, so much irony. I updated to 6.8 rc7 yesterday and this seemed to have killed my bitwarden mssql container. Stuck in a restart loop, and the errors are kind of not getting me anywhere. The ones of you actually using the mssql container already running a 6.8 build?