So after some more testing I found out that my nvme drive was acting up. I would sometime benchmark fine at other times it has atrocious speeds, down to 40mb/s. I had backed up the cache drive and reformatted it from btrfs-encryted to xfs-encrypted. Still had the problems after I moved everything over. I decided to replace that drive with a sata SSD and since the performance of both dockers and vm's is back to normal. Smart data doesn't show any problems for the nvme drive. The weird part is that I tested the drive once again in the unassigned devices and it performed beautifully.
Question is the drive bad (as in hardware?) or was it something to do with its place as cache drive and unraid?
How do you verify that the hardware nvme is bad besides smart data and simple benchmark?
Drive Benchmark is attached.
Smart Data:
SMART Data as of Jan 3, 2021 2:08 PM
Temperature31 Celsius
Warning Comp. Temperature Time0
Critical Comp. Temperature Time0
Critical Warning0x00
Available Spare100%
Available Spare Threshold10%
Percentage Used0%
Data Units Read41,354,551 [21.1 TB]
Data Units Written397,036,618 [203 TB]
Host Read Commands150,229,416
Host Write Commands2,006,570,124
Controller Busy Time221,673
Power Cycles52
Power On Hours11,151
Unsafe Shutdowns9
Media and Data Integrity Errors0
Error Information Log Entries0