6.8.3 Disk writes causing high CPU


Recommended Posts

  • Replies 210
  • Created
  • Last Reply

Top Posters In This Topic

  • 2 weeks later...
  • 2 weeks later...
On 5/21/2020 at 5:38 AM, ephigenie said:

Did you try looking with IOtop which process is causing the amount of IO ? 
Can you trace it as well with docker stats across your containers ?  Just to try to identify the verdict.... 

I'll do a test and post the results tonight, I was waiting for the parity check to finish. It did come up with errors, which is a first time for that for me

 

image.png.353d85471a3e2870487db4be77417ee7.png

Link to comment
  • 3 weeks later...
  • 2 weeks later...

This seems to be an ongoing problem, even with XFS and no pool.

 

What sort of data can I provide to help diagnose what's going on here? Moving to no pool / XFS has made the system "survive" during these operations, which is better than totally offline during them... but still not an awesome thing.

 

Help?

 

image.thumb.png.b291cf1ac66cdaf746a6a593e49fa326.png

Link to comment
  • 3 weeks later...

I don't want to keep necro-posting here... let me know if I need to start a different thread or what.... But this is still an issue. Here for example is 2 different files coming down through SAB onto a single XFS cache drive....


image.thumb.png.9db24b3319a293bc88f2831290f9fac4.png


This never used to happen, and now even after moving away from a pool it's still beating up the system. It's livable, as in other services don't fall off completely, but it's still way more impact than prior.

This isn't working for me. I mean at this point, my server basically does only Plex and download media files because it can't handle anything else. And this is a decently stout machine (dual Xeon E5 2690v2 with 128G RAM).

 

Help please.

Link to comment

I see it mainly when writing to the cache pool, I think because that's the only thing that is fast enough to exhibit the issue. I could run the mover and see what happens though, since I'm usually asleep when it runs.

 

I'm happy to run any tests deemed informative and post results here.

 

Someone above had suggested the output from iotop but I didn't see how to get that to work.

Link to comment
  • 1 month later...
On 7/9/2020 at 1:56 AM, johnnie.black said:

Yes, basically you need to do the same thing, backup, re-format and restore cache data, note that beta24 was release yesterday but there's a bug and still no aligning to 1MiB, should be fixed on next beta.

So 6.9.0b25 is out it seems.... Does that look like a good one to try for fixing this issue? And will it realign after upgrade, or will I need to reformat the cache?

Link to comment

Hi there,

 

I am actually in the midst of doing some pretty extensive storage performance testing in the lab.  We're testing with a combination of SSDs, NVMe drives, and a 10gbps network.  This will take some time to test and analyze, but we are actively researching this.

Link to comment

Awesome, thanks for the feedback. Please let me know if I can assist with any data. I'm happy to perform tests and send it back to you. My setup:

 

PCI Devices and IOMMU Groups

IOMMU group 0:				[8086:0e00] 00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04)
IOMMU group 1:				[8086:0e02] 00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
IOMMU group 2:				[8086:0e03] 00:01.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b (rev 04)
IOMMU group 3:				[8086:0e04] 00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
IOMMU group 4:				[8086:0e08] 00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
IOMMU group 5:			 	[8086:0e20] 00:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
IOMMU group 6:			 	[8086:0e21] 00:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
IOMMU group 7:			 	[8086:0e22] 00:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
IOMMU group 8:			 	[8086:0e23] 00:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
IOMMU group 9:			 	[8086:0e24] 00:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
IOMMU group 10:			 	[8086:0e25] 00:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
IOMMU group 11:			 	[8086:0e26] 00:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
IOMMU group 12:			 	[8086:0e27] 00:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
IOMMU group 13:			 	[8086:0e28] 00:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
IOMMU group 14:			 	[8086:0e2a] 00:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04)
IOMMU group 15:			 	[8086:0e2c] 00:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04)
IOMMU group 16:				[8086:1d3e] 00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06)
IOMMU group 17:			 	[8086:1d3a] 00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05)
 	[8086:1d3b] 00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05)
IOMMU group 18:			 	[8086:1d2d] 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06)
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 003: ID 0781:5571 SanDisk Corp. Cruzer Fit
Bus 001 Device 004: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon
IOMMU group 19:				[8086:1d10] 00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6)
IOMMU group 20:			 	[8086:1d26] 00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06)
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 003: ID 0764:0501 Cyber Power System, Inc. CP1500 AVR UPS
IOMMU group 21:				[8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
 	[102b:0532] 08:01.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
IOMMU group 22:			 	[8086:1d41] 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06)
 	[8086:1d02] 00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06)
[3:0:0:0]    disk    ATA      ST8000DM004-2CX1 0001  /dev/sdb   8.00TB
[4:0:0:0]    disk    ATA      TOSHIBA HDWE150  FP2A  /dev/sdc   5.00TB
 	[8086:1d22] 00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06)
 	[8086:1d24] 00:1f.6 Signal processing controller: Intel Corporation C600/X79 series chipset Thermal Management Controller (rev 06)
IOMMU group 23:			 	[1b4b:9215] 02:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
[8:0:0:0]    disk    ATA      Samsung SSD 860  2B6Q  /dev/sdd   1.00TB
[10:0:0:0]   disk    ATA      Samsung SSD 860  2B6Q  /dev/sde   1.00TB
IOMMU group 24:			 	[8086:2700] 04:00.0 Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series
[N:0:0:1]    disk    INTEL SSDPED1D280GA__1                     /dev/nvme0n1   280GB
IOMMU group 25:			 	[8086:1d6b] 05:00.0 Serial Attached SCSI controller: Intel Corporation C602 chipset 4-Port SATA Storage Control Unit (rev 06)
[11:0:0:0]   disk    ATA      WDC WD80EMAZ-00W 0A83  /dev/sdf   8.00TB
[11:0:1:0]   disk    ATA      WDC WD80EMAZ-00W 0A83  /dev/sdg   8.00TB
[11:0:2:0]   disk    ATA      WDC WD60EFRX-68L 0A82  /dev/sdh   6.00TB
[11:0:3:0]   disk    ATA      WL6000GSA6457    1C01  /dev/sdi   6.30TB
IOMMU group 26:			 	[8086:1521] 06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 27:			 	[8086:1521] 06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 28:			 	[8086:1521] 06:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 29:			 	[8086:1521] 06:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 30:			 	[8086:0e80] 7f:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
IOMMU group 31:			 	[8086:0e90] 7f:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
IOMMU group 32:			 	[8086:0ec0] 7f:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04)
 	[8086:0ec1] 7f:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04)
 	[8086:0ec2] 7f:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04)
 	[8086:0ec3] 7f:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04)
IOMMU group 33:			 	[8086:0e1e] 7f:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
 	[8086:0e1f] 7f:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
IOMMU group 34:			 	[8086:0ee0] 7f:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee2] 7f:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee4] 7f:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee6] 7f:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee8] 7f:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
IOMMU group 35:			 	[8086:0ee1] 7f:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee3] 7f:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee5] 7f:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee7] 7f:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee9] 7f:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
IOMMU group 36:			 	[8086:0ea0] 7f:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
 	[8086:0e30] 7f:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
IOMMU group 37:			 	[8086:0ea8] 7f:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04)
IOMMU group 38:			 	[8086:0e71] 7f:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04)
IOMMU group 39:			 	[8086:0eaa] 7f:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
IOMMU group 40:			 	[8086:0eab] 7f:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
IOMMU group 41:			 	[8086:0eac] 7f:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
IOMMU group 42:			 	[8086:0ead] 7f:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
IOMMU group 43:			 	[8086:0eb0] 7f:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04)
IOMMU group 44:			 	[8086:0eb1] 7f:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04)
IOMMU group 45:			 	[8086:0eb2] 7f:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04)
IOMMU group 46:			 	[8086:0eb3] 7f:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04)
IOMMU group 47:			 	[8086:0eb4] 7f:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04)
IOMMU group 48:			 	[8086:0eb5] 7f:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04)
IOMMU group 49:			 	[8086:0eb6] 7f:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04)
IOMMU group 50:			 	[8086:0eb7] 7f:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04)
IOMMU group 51:			 	[8086:0e1d] 7f:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
 	[8086:0e34] 7f:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
 	[8086:0e81] 7f:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04)
 	[8086:0e36] 7f:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)
IOMMU group 52:			 	[8086:0ec8] 7f:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04)
 	[8086:0ec9] 7f:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
 	[8086:0eca] 7f:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
IOMMU group 53:				[8086:0e02] 80:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
IOMMU group 54:				[8086:0e04] 80:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
IOMMU group 55:				[8086:0e08] 80:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
IOMMU group 56:			 	[8086:0e20] 80:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
IOMMU group 57:			 	[8086:0e21] 80:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
IOMMU group 58:			 	[8086:0e22] 80:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
IOMMU group 59:			 	[8086:0e23] 80:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
IOMMU group 60:			 	[8086:0e24] 80:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
IOMMU group 61:			 	[8086:0e25] 80:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
IOMMU group 62:			 	[8086:0e26] 80:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
IOMMU group 63:			 	[8086:0e27] 80:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
IOMMU group 64:			 	[8086:0e28] 80:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
IOMMU group 65:			 	[8086:0e2a] 80:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04)
IOMMU group 66:			 	[8086:0e2c] 80:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04)
IOMMU group 67:			 	[10de:1b82] 82:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070 Ti] (rev a1)
 	[10de:10f0] 82:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
IOMMU group 68:			 	[8086:0e80] ff:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
IOMMU group 69:			 	[8086:0e90] ff:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
IOMMU group 70:			 	[8086:0ec0] ff:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04)
 	[8086:0ec1] ff:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04)
 	[8086:0ec2] ff:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04)
 	[8086:0ec3] ff:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04)
IOMMU group 71:			 	[8086:0e1e] ff:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
 	[8086:0e1f] ff:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
IOMMU group 72:			 	[8086:0ee0] ff:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee2] ff:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee4] ff:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee6] ff:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee8] ff:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
IOMMU group 73:			 	[8086:0ee1] ff:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee3] ff:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee5] ff:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee7] ff:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
 	[8086:0ee9] ff:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
IOMMU group 74:			 	[8086:0ea0] ff:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
 	[8086:0e30] ff:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
IOMMU group 75:			 	[8086:0ea8] ff:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04)
IOMMU group 76:			 	[8086:0e71] ff:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04)
IOMMU group 77:			 	[8086:0eaa] ff:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
IOMMU group 78:			 	[8086:0eab] ff:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
IOMMU group 79:			 	[8086:0eac] ff:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
IOMMU group 80:			 	[8086:0ead] ff:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
IOMMU group 81:			 	[8086:0eb0] ff:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04)
IOMMU group 82:			 	[8086:0eb1] ff:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04)
IOMMU group 83:			 	[8086:0eb2] ff:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04)
IOMMU group 84:			 	[8086:0eb3] ff:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04)
IOMMU group 85:			 	[8086:0eb4] ff:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04)
IOMMU group 86:			 	[8086:0eb5] ff:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04)
IOMMU group 87:			 	[8086:0eb6] ff:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04)
IOMMU group 88:			 	[8086:0eb7] ff:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04)
IOMMU group 89:			 	[8086:0e1d] ff:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
 	[8086:0e34] ff:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
 	[8086:0e81] ff:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04)
 	[8086:0e36] ff:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)
IOMMU group 90:			 	[8086:0ec8] ff:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04)
 	[8086:0ec9] ff:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
 	[8086:0eca] ff:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
USB Devices

Bus 001 Device 001:	ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002:	ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 003:	ID 0781:5571 SanDisk Corp. Cruzer Fit
Bus 001 Device 004:	ID 0557:2221 ATEN International Co., Ltd Winbond Hermon
Bus 002 Device 001:	ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002:	ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 003:	ID 0764:0501 Cyber Power System, Inc. CP1500 AVR UPS


SCSI Devices

[0:0:0:0]	disk    SanDisk  Cruzer Fit       1.00  /dev/sda   16.0GB
[3:0:0:0]	disk    ATA      ST8000DM004-2CX1 0001  /dev/sdb   8.00TB
[4:0:0:0]	disk    ATA      TOSHIBA HDWE150  FP2A  /dev/sdc   5.00TB
[8:0:0:0]	disk    ATA      Samsung SSD 860  2B6Q  /dev/sdd   1.00TB
[10:0:0:0]	disk    ATA      Samsung SSD 860  2B6Q  /dev/sde   1.00TB
[11:0:0:0]	disk    ATA      WDC WD80EMAZ-00W 0A83  /dev/sdf   8.00TB
[11:0:1:0]	disk    ATA      WDC WD80EMAZ-00W 0A83  /dev/sdg   8.00TB
[11:0:2:0]	disk    ATA      WDC WD60EFRX-68L 0A82  /dev/sdh   6.00TB
[11:0:3:0]	disk    ATA      WL6000GSA6457    1C01  /dev/sdi   6.30TB
[N:0:0:1]	disk    INTEL SSDPED1D280GA__1                     /dev/nvme0n1   280GB

 

 

Edited by CowboyRedBeard
Link to comment
  • 1 month later...
On 8/18/2020 at 2:17 PM, jonp said:

Hi there,

 

I am actually in the midst of doing some pretty extensive storage performance testing in the lab.  We're testing with a combination of SSDs, NVMe drives, and a 10gbps network.  This will take some time to test and analyze, but we are actively researching this.

Have you guys had any progress on this issue? Thanks

Link to comment
Have you guys had any progress on this issue? Thanks
Yes and no. We have determined some of the excess writes on SSDs that some users were seeing may have been due to the boundary size we set which we have resolved. That will be in an upcoming update, but will require a process to convert which we are still sorting out.

Separately we are testing smb performance in a 10gbps environment to see how we hold up in that setting. That testing is still underway and should be completed soon.

However the CPU usage issue reported here hasn't been able to be reproduced just yet, so we are still trying.

Sent from my Pixel 3 XL using Tapatalk

Link to comment
11 minutes ago, jonp said:

Yes and no. We have determined some of the excess writes on SSDs that some users were seeing may have been due to the boundary size we set which we have resolved. That will be in an upcoming update, but will require a process to convert which we are still sorting out.

Separately we are testing smb performance in a 10gbps environment to see how we hold up in that setting. That testing is still underway and should be completed soon.

However the CPU usage issue reported here hasn't been able to be reproduced just yet, so we are still trying.

Sent from my Pixel 3 XL using Tapatalk
 

Can I assist with testing data?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.