CowboyRedBeard

Guys any help would be much appreciated. I'm at a loss with this. Today I did a test.... I stopped and disabled VMs / Dockers Moved everything off the cache Reformatted that drive from XFS to BTRFS Moved everything back... Whamo... When doing a big download with Sabnzbd I got decent performance. Then later today, doing the same sort of workload I'm stuck at roughly 80MiB/s on writes again... How is this? It can't be a hardware thing??

So at this point, I've suffered through this by scheduling tasks in the wee hours for nearly 2 years. But I've begun using the server for more workloads than just Plex and infrastructure dockers again. (I had originally built this box to help with VMs for Crypto projects + Plex) But at this point, the high IO wait is actually bad enough to raise the fans and CPU / PCM temps for sustained periods. It's time to add more storage and change a few other things on top of it. So here's my question: I started unRAID on this box back in 2019. A few hardware changes and however many versions later I still have this problem. Would rebuilding it from scratch potentially help alleviate this problem? And if it's worth a shot, what is a good general strategy? What other things should I consider? @Squid @trurl

Tried that (rebooted the server even)... doesn't seem to make a difference

What does that do? I'm assuming this is something to do with VMs, which I don't understand how that'd have an effect on general OS disk I/O Mine are currently 10 & 20 in a server with 128G of RAM

Guys, is there any hope that there will be a resolution to this issue?

I've been waiting over a year to see resolution to this problem... I've actually changed how I use unraid to mitigate it's affects on applications and users. I've not had the time, but thought about doing a fresh install in hopes of fixing it since this install of unraid is pretty old and even has gone through 2 physical servers... But new users seeing it makes me think that won't help.

Yeah, this is more than a year on at this point. And there's a number of folks with the issue. It'd be great to get a resolution, since I'd had unraid for YEARS without this problem.

I've been able to limp it along by scheduling large write I/O jobs, but that might not work for everyone I'm actually thinking about building another install just to see if it goes away Additionally... I've upgraded from beta 25 to 6.9.0 and the issue persists

I've not seen a fix yet, I've been experiencing it for nearly a year.

Well, mine is in production with data... but I'd be happy to run some tests and provide metrics / data / output... I reproduce it daily. 😪

My Christmas wish is for this to get fixed.... 😃

Still a problem, this shows server temps and fan speeds, etc.... This is when I have it set to download and unpack files at 02:00 Any chance this gets fixed soon? Alternatively, @jonp / @itimpi do you think maybe a quicker fix for me would be to rebuild unraid from scratch? I mean one of the other side affects is that SSD write speeds are way way down, to 80MBps range... which also sucks.

I did the beta and still have issue.

I saw that beta 29 is out, does it address this in any way? Also, happy to provide any data that might be helpful... If you can tell me how to collect it.

I mean, at this point I'm happy to actually rebuild the server.... fresh install... if that has a high likelihood of making this go away

I've posted most of it to this thread, but when took it from 2 - SSDs in a pool (BTRFS) and had single SSD on XFS it was less of a problem, but still a problem. I could run some tests or tools and show you the results if you can tell me which ones would be meaningful. Basically it's any time there's a large write (20G) to SSD.

Can I assist with testing data?

Have you guys had any progress on this issue? Thanks

Awesome, thanks for the feedback. Please let me know if I can assist with any data. I'm happy to perform tests and send it back to you. My setup: PCI Devices and IOMMU Groups IOMMU group 0: [8086:0e00] 00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04) IOMMU group 1: [8086:0e02] 00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04) IOMMU group 2: [8086:0e03] 00:01.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b (rev 04) IOMMU group 3: [8086:0e04] 00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04) IOMMU group 4: [8086:0e08] 00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04) IOMMU group 5: [8086:0e20] 00:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04) IOMMU group 6: [8086:0e21] 00:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04) IOMMU group 7: [8086:0e22] 00:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04) IOMMU group 8: [8086:0e23] 00:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04) IOMMU group 9: [8086:0e24] 00:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04) IOMMU group 10: [8086:0e25] 00:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04) IOMMU group 11: [8086:0e26] 00:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04) IOMMU group 12: [8086:0e27] 00:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04) IOMMU group 13: [8086:0e28] 00:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04) IOMMU group 14: [8086:0e2a] 00:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04) IOMMU group 15: [8086:0e2c] 00:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04) IOMMU group 16: [8086:1d3e] 00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06) IOMMU group 17: [8086:1d3a] 00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05) [8086:1d3b] 00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05) IOMMU group 18: [8086:1d2d] 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06) Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 003: ID 0781:5571 SanDisk Corp. Cruzer Fit Bus 001 Device 004: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon IOMMU group 19: [8086:1d10] 00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6) IOMMU group 20: [8086:1d26] 00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06) Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 003: ID 0764:0501 Cyber Power System, Inc. CP1500 AVR UPS IOMMU group 21: [8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6) [102b:0532] 08:01.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a) IOMMU group 22: [8086:1d41] 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06) [8086:1d02] 00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06) [3:0:0:0] disk ATA ST8000DM004-2CX1 0001 /dev/sdb 8.00TB [4:0:0:0] disk ATA TOSHIBA HDWE150 FP2A /dev/sdc 5.00TB [8086:1d22] 00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06) [8086:1d24] 00:1f.6 Signal processing controller: Intel Corporation C600/X79 series chipset Thermal Management Controller (rev 06) IOMMU group 23: [1b4b:9215] 02:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11) [8:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sdd 1.00TB [10:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sde 1.00TB IOMMU group 24: [8086:2700] 04:00.0 Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series [N:0:0:1] disk INTEL SSDPED1D280GA__1 /dev/nvme0n1 280GB IOMMU group 25: [8086:1d6b] 05:00.0 Serial Attached SCSI controller: Intel Corporation C602 chipset 4-Port SATA Storage Control Unit (rev 06) [11:0:0:0] disk ATA WDC WD80EMAZ-00W 0A83 /dev/sdf 8.00TB [11:0:1:0] disk ATA WDC WD80EMAZ-00W 0A83 /dev/sdg 8.00TB [11:0:2:0] disk ATA WDC WD60EFRX-68L 0A82 /dev/sdh 6.00TB [11:0:3:0] disk ATA WL6000GSA6457 1C01 /dev/sdi 6.30TB IOMMU group 26: [8086:1521] 06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) IOMMU group 27: [8086:1521] 06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) IOMMU group 28: [8086:1521] 06:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) IOMMU group 29: [8086:1521] 06:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) IOMMU group 30: [8086:0e80] 7f:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04) IOMMU group 31: [8086:0e90] 7f:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04) IOMMU group 32: [8086:0ec0] 7f:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04) [8086:0ec1] 7f:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04) [8086:0ec2] 7f:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04) [8086:0ec3] 7f:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04) IOMMU group 33: [8086:0e1e] 7f:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04) [8086:0e1f] 7f:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04) IOMMU group 34: [8086:0ee0] 7f:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee2] 7f:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee4] 7f:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee6] 7f:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee8] 7f:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) IOMMU group 35: [8086:0ee1] 7f:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee3] 7f:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee5] 7f:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee7] 7f:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee9] 7f:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) IOMMU group 36: [8086:0ea0] 7f:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04) [8086:0e30] 7f:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04) IOMMU group 37: [8086:0ea8] 7f:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04) IOMMU group 38: [8086:0e71] 7f:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04) IOMMU group 39: [8086:0eaa] 7f:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 40: [8086:0eab] 7f:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 41: [8086:0eac] 7f:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 42: [8086:0ead] 7f:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 43: [8086:0eb0] 7f:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04) IOMMU group 44: [8086:0eb1] 7f:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04) IOMMU group 45: [8086:0eb2] 7f:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04) IOMMU group 46: [8086:0eb3] 7f:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04) IOMMU group 47: [8086:0eb4] 7f:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04) IOMMU group 48: [8086:0eb5] 7f:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04) IOMMU group 49: [8086:0eb6] 7f:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04) IOMMU group 50: [8086:0eb7] 7f:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04) IOMMU group 51: [8086:0e1d] 7f:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04) [8086:0e34] 7f:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04) [8086:0e81] 7f:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04) [8086:0e36] 7f:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04) IOMMU group 52: [8086:0ec8] 7f:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04) [8086:0ec9] 7f:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04) [8086:0eca] 7f:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04) IOMMU group 53: [8086:0e02] 80:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04) IOMMU group 54: [8086:0e04] 80:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04) IOMMU group 55: [8086:0e08] 80:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04) IOMMU group 56: [8086:0e20] 80:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04) IOMMU group 57: [8086:0e21] 80:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04) IOMMU group 58: [8086:0e22] 80:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04) IOMMU group 59: [8086:0e23] 80:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04) IOMMU group 60: [8086:0e24] 80:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04) IOMMU group 61: [8086:0e25] 80:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04) IOMMU group 62: [8086:0e26] 80:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04) IOMMU group 63: [8086:0e27] 80:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04) IOMMU group 64: [8086:0e28] 80:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04) IOMMU group 65: [8086:0e2a] 80:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04) IOMMU group 66: [8086:0e2c] 80:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04) IOMMU group 67: [10de:1b82] 82:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070 Ti] (rev a1) [10de:10f0] 82:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1) IOMMU group 68: [8086:0e80] ff:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04) IOMMU group 69: [8086:0e90] ff:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04) IOMMU group 70: [8086:0ec0] ff:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04) [8086:0ec1] ff:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04) [8086:0ec2] ff:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04) [8086:0ec3] ff:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04) IOMMU group 71: [8086:0e1e] ff:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04) [8086:0e1f] ff:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04) IOMMU group 72: [8086:0ee0] ff:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee2] ff:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee4] ff:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee6] ff:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee8] ff:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) IOMMU group 73: [8086:0ee1] ff:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee3] ff:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee5] ff:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee7] ff:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee9] ff:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) IOMMU group 74: [8086:0ea0] ff:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04) [8086:0e30] ff:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04) IOMMU group 75: [8086:0ea8] ff:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04) IOMMU group 76: [8086:0e71] ff:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04) IOMMU group 77: [8086:0eaa] ff:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 78: [8086:0eab] ff:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 79: [8086:0eac] ff:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 80: [8086:0ead] ff:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 81: [8086:0eb0] ff:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04) IOMMU group 82: [8086:0eb1] ff:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04) IOMMU group 83: [8086:0eb2] ff:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04) IOMMU group 84: [8086:0eb3] ff:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04) IOMMU group 85: [8086:0eb4] ff:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04) IOMMU group 86: [8086:0eb5] ff:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04) IOMMU group 87: [8086:0eb6] ff:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04) IOMMU group 88: [8086:0eb7] ff:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04) IOMMU group 89: [8086:0e1d] ff:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04) [8086:0e34] ff:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04) [8086:0e81] ff:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04) [8086:0e36] ff:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04) IOMMU group 90: [8086:0ec8] ff:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04) [8086:0ec9] ff:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04) [8086:0eca] ff:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04) USB Devices Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 003: ID 0781:5571 SanDisk Corp. Cruzer Fit Bus 001 Device 004: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 003: ID 0764:0501 Cyber Power System, Inc. CP1500 AVR UPS SCSI Devices [0:0:0:0] disk SanDisk Cruzer Fit 1.00 /dev/sda 16.0GB [3:0:0:0] disk ATA ST8000DM004-2CX1 0001 /dev/sdb 8.00TB [4:0:0:0] disk ATA TOSHIBA HDWE150 FP2A /dev/sdc 5.00TB [8:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sdd 1.00TB [10:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sde 1.00TB [11:0:0:0] disk ATA WDC WD80EMAZ-00W 0A83 /dev/sdf 8.00TB [11:0:1:0] disk ATA WDC WD80EMAZ-00W 0A83 /dev/sdg 8.00TB [11:0:2:0] disk ATA WDC WD60EFRX-68L 0A82 /dev/sdh 6.00TB [11:0:3:0] disk ATA WL6000GSA6457 1C01 /dev/sdi 6.30TB [N:0:0:1] disk INTEL SSDPED1D280GA__1 /dev/nvme0n1 280GB

I'd say this isn't any better..... Gets into the 20% range, downloaded a 40G file / unpack / etc... All on cache What else can I provide to help isolate the issue? This is killing my server. Crushing it just to copy one file... this is rediculous.

So far, docker disabled while using the mover to get files back onto the cache.... Looking like it's the same sorta CPU usage only it's not a sticking to the same thread pairing hung at 100% .... will test when everything is back the way it should be

Ah-ha! Didn't see that in the "release notes" window within the GUI upgrade tool, works now. Will do some testing and report back!

Done, but when I click on the cache drive it still says: Partition format: MBR: 4KiB-aligned

So 6.9.0b25 is out it seems.... Does that look like a good one to try for fixing this issue? And will it realign after upgrade, or will I need to reformat the cache?