Thanks for all the work you put into this Chris. I recently converted my old AMD FX4300 Black based unraid server to an i7-6700K based platform with 32GB RAM. I added a Samsung SATA SSD, a Crucial NVME drive, and an LSI HBA to attach my stack of spinning rust.
I had been struggling with IOWAIT issues ever since first booting the new platform up and had been struggling to diagnose it. After reading your post I moved all my appdata and VMs off my NVME drive, converted it from BTRFS to XFS, then moved everything back. Now it seems my IOWAIT issues are solved. Occasionally see 5-6% IOWAIT in top, but only if I'm doing something file intensive like a Plex library scan.
Thanks for pointing me in the right direction.