Sarge

I've been running 6.10 RC2 for a couple months now, learning how Unraid works and getting a bunch of Docker containers up like Pihole, Nextcloud, Plex, etc. Today, while copying some mkv files to the server I noticed that transfer speeds suddenly tanked. After some troubleshooting I think I've narrowed it down to the drives themselves. Here's what I've done: VM service is off, has been for a while as I have no need for VM's yet Turned off the Docker service to rule out any of the containers putting strain on the system Disabled RAM write caching using Tips and Tweaks and setting vm.dirty_background_ratio and vm.dirty_ratio to 0 (did this early on to try and figure out if the RAM to disk flush was causing it) Uninstalled a couple plugins that I had most recently installed, one of which was Dynamix File Integrity, kind of hoped that would fix it, then realized the cache drive is BTRFS and should not have been affected by it at all. I did do a reboot after with no improvement. Tested by SSHing into the Unraid box and running the following two commands. dd if=/dev/zero of=/mnt/cache/temp/test1.img bs=1G count=1 oflag=dsync dd if=/dev/zero of=/mnt/disk1/dtest/test1.img bs=1G count=1 oflag=dsync Results are around 3 to 5 MB a second, both on the cache drive and spinning disks where normal results are several gigs a second for the cache and ~100 MB/s for the disks System Specs: Dell R720xd Dell RAID card flashed to IT mode, been working fine for a couple months 128 Gigs of RAM, fully tested with Memtest 2 X Intel Xeon E5-2697 v2 @ 2.70GHz (24 cores, 48 threads) Dual 10 gig Dell Ethernet bonded Nvidia Quadro P400 7 X 10TB Seagate Enterprise drives, two of which are parity 2 X 2TB Samsung SSD 970 EVO Plus NVME drives for cache in RAID 1 using BTRFS Please see diagnostics attached. Any pointers anyone can give would be greatly appreciated. Let me know if this should go in the 6.10 RC2 thread. thor-diagnostics-20220224-0018.zip

I do have Auto Update Applications installed and running but I'm nearly 100% positive that's not it as one of the containers is one I compiled and host on Dockerhub and i haven't updated it, so there aren't any changes to the image to download and no reason to restart the container. I'm not running a VPN (yet). I do have a reverse proxy installed but am currently just using it for SSL certificate renewal. Other than that, they are all independant. Note: I have the array down at the moment as I'm in the middle of testing the drives to see if there are any errors. I should have done that before copying over all the data but didn't think about it. Unrelated question on the subject of drive testing. I know I can pull one empty drive from the array, change it from BTRFS to XFS and restart the array and reformat it without any issues, but I don't know if I can do that to more than one drive at a time. i.e. if I use unbalance to move all data to one drive that has been tested, then shut down the array and run `badblocks -wvs -b 4096 /dev/sdX` on all the empty drives at once will the array be broken when I spin it back up? Any other thoughts on the Docker corruption?

I'm not sure if this should go in the 6.10 RC2 post, in general or in here. Someone let me know if I should move it. I'm new to Unraid but have been watching a bunch of videos and reading tutorials and these forums. I decided to go with 6.10 RC2 instead of the stable branch and am having some issues with Docker containers corrupting themselves. Diagnostics attached What's happening: I'll get all the docker apps installed, everything is fine and starts as one would expect and all of them work. Sometime later, typically the next day, I will discover that one or more have stopped or are in some kind of failed state. This is almost always the Docker OCI complaining that a file inside the docker image is missing and can't be ran. From Pihole: OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: read init-p: connection reset by peer: unknown Deleting the docker container and image and re-installing does not fix it. Only fix I've found is to stop the Docker service and delete the /mnt/user/system/docker/ folder then restart the Docker service. I can then reinstall all of the Docker containers and they all work, (Call #1 above, rinse and repeat) Changes I've made: I changed Docker from running in a vdisk to running in a directory, this seems to be when things started breaking. I changed Docker Custom Network Type from MacVlan to IpVlan at the same time. I have not reverted either of these changes in an attempt to fix yet. I do have the the appdata backup plugin set to back up my appdata folder weekly, but it did not do so last night and things still failed System Specs: Dell R720xd Perc Card flashed to IT mode 128 Gigs of RAM Cache drive is two new 2TB Samsung 970 Evo Plus drives in Raid 1 with BTRFS. Main Array is five 10TB Seagate Enterprise drives with about 3 years of spin on them. There are two of the same 10TB drives being used for parity, but they are both new. Things I've Tried: I've ran a scrub a couple times on the parity NVME drives, no issues found. I ran Memtest 86 overnight, it made it all the way through one pass plus about half way through the next with no errors. I ran Prime 95 from the Ultimate Boot CD (USB) for several hours with no issues. I checked the firmware on the Samsung NVME drives I'm using for cache and it is on its latest. Note: you may see some errors in the logs for some Samsung 860 Evo drives, haven't had a chance to look into those, they are generating CRC errors. Nothing is on them, they are set up as a cache pool called Vmdks that I'm not using yet. Any help or tests anyone can think of would be very appreciated. thor-diagnostics-20220130-2050.zip