I have fixed the problem, I think. So far, so good. I wanted to try everything before I started updating the hardware, so I found and updated my bios for my motherboard to the latest version available. I still have some iowaits that go up to the 50’s in certain situations, but so far, nothing I have thrown at it has locked up the server while trying to complete an awaited task.
Most situations that I have tested still allowed me to transfer my test file at full speed with no degradation of speed even if there was an io wait reported. The 2 situations in which I was able to make the speed slow down slightly were if tdar was transcoding multiple files (3 at a time plus any 1 copy could be running) or if plex was detecting intros. Both of those situations made my transfer speed drop about 20% at most, but it no longer drops to 0 and locks up the server until the io catches up. I would like to have little to no iowaits but I will take having the server always respond without issue as a win for now.
While trying to figure this out the past 2 days, I found multiple posts about people having the same issues, but none of the posts have solutions, so I want to post what helped me here.
First, update your bios if there is an update. Something about the bios my server was using (which was old) and btfs did not work well together. This fixed my issue so save yourself 2 days and try this first.
If that does not fix your issue, the couple of things you can do to try and diagnose what is causing it are
First, install glances from apps. It’s an easy way to view the iowait and show tasks that are using your cpu and you can watch/log iowaits while you try different scenarios. You can get the same information with the top command, but glances made it easier to view, and it logs the last 9 warning/critical issues.
Next, run the iostat with the following command: iostat -hymx 1 4
This will give you a response like below. Look to see if any disks have 100% usage, as I had below. If you do, that is the disk that is having issues.
Next, install disk speed from apps and run the benchmark on that drive. In my case, the drive looked fine when I ran the benchmark without other things turned on, but that helped me rule out the disk (somewhat since I still thought we could have a disk issue).
Lastly, install nerd tools, and then inside nerd tools, install iotop. After you install iotop run it using the command: iotop -o
This will list out all the tasks that are using io in real-time. Keep it open and do things on your server that cause it to have a high iowait percentage to see what task is causing the problem. In my case, it was the btfs worker (shown in the youtube video linked in this thread), so if my bios update did not work, I was going to swap out the cache drive and change the drive from btfs to xfs as a last resort.
If iotop is not showing the percentage and shows unavailable, run the following command and then run iotop again. It might still show unavailable for a minute, but then it started showing for me.
echo 1 > /proc/sys/kernel/task_delayacct
Hopefully, this helps anyone else having this issue. If it starts happening again i will open this thread back up.
