May 1, 20206 yr Author Further experimentation .... Feeding it 1Gbps shows stress still (again though I'm not sure what's normal for a rig like this)
May 1, 20206 yr Author And... so after sustained load we see the issue crop up again, not as bad as before, but still pretty terrible performance. Help?
May 11, 20206 yr Author I mean, it's not killing other services like it once was... but the IOWAIT is still very very high Anyone from LT able to help with this?
May 21, 20206 yr Did you try looking with IOtop which process is causing the amount of IO ? Can you trace it as well with docker stats across your containers ? Just to try to identify the verdict....
May 22, 20206 yr Author On 5/21/2020 at 5:38 AM, ephigenie said: Did you try looking with IOtop which process is causing the amount of IO ? Can you trace it as well with docker stats across your containers ? Just to try to identify the verdict.... I'll do a test and post the results tonight, I was waiting for the parity check to finish. It did come up with errors, which is a first time for that for me
June 9, 20206 yr Author OK, finally have time to do this. From the shell on unraid I don't get iotop... I installed it via nerd tools But get this :~# iotop libffi.so.7: cannot open shared object file: No such file or directory To run an uninstalled copy of iotop, launch iotop.py in the top directory
June 19, 20206 yr Author This seems to be an ongoing problem, even with XFS and no pool. What sort of data can I provide to help diagnose what's going on here? Moving to no pool / XFS has made the system "survive" during these operations, which is better than totally offline during them... but still not an awesome thing. Help?
July 7, 20205 yr Author I don't want to keep necro-posting here... let me know if I need to start a different thread or what.... But this is still an issue. Here for example is 2 different files coming down through SAB onto a single XFS cache drive.... This never used to happen, and now even after moving away from a pool it's still beating up the system. It's livable, as in other services don't fall off completely, but it's still way more impact than prior. This isn't working for me. I mean at this point, my server basically does only Plex and download media files because it can't handle anything else. And this is a decently stout machine (dual Xeon E5 2690v2 with 128G RAM). Help please.
July 7, 20205 yr Community Expert Please remind me, is this issue mainly when writing to cache SSD/pool or also when writing to array disks?
July 7, 20205 yr Author I see it mainly when writing to the cache pool, I think because that's the only thing that is fast enough to exhibit the issue. I could run the mover and see what happens though, since I'm usually asleep when it runs. I'm happy to run any tests deemed informative and post results here. Someone above had suggested the output from iotop but I didn't see how to get that to work.
July 8, 20205 yr Community Expert Next beta is going to align the partition on SSDs on the 1MiB boundary, like Windows does, this should improve performance, try that that once it's available, you'll need to reformat the cache though.
July 8, 20205 yr Author Should I follow the above directions where I went from BTRFS to XFS (no pool) ?
July 9, 20205 yr Community Expert Yes, basically you need to do the same thing, backup, re-format and restore cache data, note that beta24 was release yesterday but there's a bug and still no aligning to 1MiB, should be fixed on next beta.
August 13, 20205 yr Author On 7/9/2020 at 1:56 AM, johnnie.black said: Yes, basically you need to do the same thing, backup, re-format and restore cache data, note that beta24 was release yesterday but there's a bug and still no aligning to 1MiB, should be fixed on next beta. So 6.9.0b25 is out it seems.... Does that look like a good one to try for fixing this issue? And will it realign after upgrade, or will I need to reformat the cache?
August 13, 20205 yr Community Expert Just now, CowboyRedBeard said: Does that look like a good one to try for fixing this issue? There are reports that it helps, in some cases by a lot. 2 minutes ago, CowboyRedBeard said: will I need to reformat the cache? This.
August 17, 20205 yr Author Done, but when I click on the cache drive it still says: Partition format: MBR: 4KiB-aligned
August 17, 20205 yr Community Expert 8 minutes ago, CowboyRedBeard said: Done, but when I click on the cache drive it still says: Partition format: MBR: 4KiB-aligned Did you follow the instructions in the Release Notes on how to wipe the current contents and then repartion/format to the new alignment?
August 17, 20205 yr Author Ah-ha! Didn't see that in the "release notes" window within the GUI upgrade tool, works now. Will do some testing and report back!
August 17, 20205 yr Author So far, docker disabled while using the mover to get files back onto the cache.... Looking like it's the same sorta CPU usage only it's not a sticking to the same thread pairing hung at 100% .... will test when everything is back the way it should be
August 17, 20205 yr Author I'd say this isn't any better..... Gets into the 20% range, downloaded a 40G file / unpack / etc... All on cache What else can I provide to help isolate the issue? This is killing my server. Crushing it just to copy one file... this is rediculous. Edited August 17, 20205 yr by CowboyRedBeard
August 18, 20205 yr Hi there, I am actually in the midst of doing some pretty extensive storage performance testing in the lab. We're testing with a combination of SSDs, NVMe drives, and a 10gbps network. This will take some time to test and analyze, but we are actively researching this.
August 18, 20205 yr Author Awesome, thanks for the feedback. Please let me know if I can assist with any data. I'm happy to perform tests and send it back to you. My setup: PCI Devices and IOMMU Groups IOMMU group 0: [8086:0e00] 00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04) IOMMU group 1: [8086:0e02] 00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04) IOMMU group 2: [8086:0e03] 00:01.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b (rev 04) IOMMU group 3: [8086:0e04] 00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04) IOMMU group 4: [8086:0e08] 00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04) IOMMU group 5: [8086:0e20] 00:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04) IOMMU group 6: [8086:0e21] 00:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04) IOMMU group 7: [8086:0e22] 00:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04) IOMMU group 8: [8086:0e23] 00:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04) IOMMU group 9: [8086:0e24] 00:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04) IOMMU group 10: [8086:0e25] 00:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04) IOMMU group 11: [8086:0e26] 00:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04) IOMMU group 12: [8086:0e27] 00:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04) IOMMU group 13: [8086:0e28] 00:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04) IOMMU group 14: [8086:0e2a] 00:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04) IOMMU group 15: [8086:0e2c] 00:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04) IOMMU group 16: [8086:1d3e] 00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06) IOMMU group 17: [8086:1d3a] 00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05) [8086:1d3b] 00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05) IOMMU group 18: [8086:1d2d] 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06) Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 003: ID 0781:5571 SanDisk Corp. Cruzer Fit Bus 001 Device 004: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon IOMMU group 19: [8086:1d10] 00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6) IOMMU group 20: [8086:1d26] 00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06) Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 003: ID 0764:0501 Cyber Power System, Inc. CP1500 AVR UPS IOMMU group 21: [8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6) [102b:0532] 08:01.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a) IOMMU group 22: [8086:1d41] 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06) [8086:1d02] 00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06) [3:0:0:0] disk ATA ST8000DM004-2CX1 0001 /dev/sdb 8.00TB [4:0:0:0] disk ATA TOSHIBA HDWE150 FP2A /dev/sdc 5.00TB [8086:1d22] 00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06) [8086:1d24] 00:1f.6 Signal processing controller: Intel Corporation C600/X79 series chipset Thermal Management Controller (rev 06) IOMMU group 23: [1b4b:9215] 02:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11) [8:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sdd 1.00TB [10:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sde 1.00TB IOMMU group 24: [8086:2700] 04:00.0 Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series [N:0:0:1] disk INTEL SSDPED1D280GA__1 /dev/nvme0n1 280GB IOMMU group 25: [8086:1d6b] 05:00.0 Serial Attached SCSI controller: Intel Corporation C602 chipset 4-Port SATA Storage Control Unit (rev 06) [11:0:0:0] disk ATA WDC WD80EMAZ-00W 0A83 /dev/sdf 8.00TB [11:0:1:0] disk ATA WDC WD80EMAZ-00W 0A83 /dev/sdg 8.00TB [11:0:2:0] disk ATA WDC WD60EFRX-68L 0A82 /dev/sdh 6.00TB [11:0:3:0] disk ATA WL6000GSA6457 1C01 /dev/sdi 6.30TB IOMMU group 26: [8086:1521] 06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) IOMMU group 27: [8086:1521] 06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) IOMMU group 28: [8086:1521] 06:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) IOMMU group 29: [8086:1521] 06:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) IOMMU group 30: [8086:0e80] 7f:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04) IOMMU group 31: [8086:0e90] 7f:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04) IOMMU group 32: [8086:0ec0] 7f:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04) [8086:0ec1] 7f:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04) [8086:0ec2] 7f:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04) [8086:0ec3] 7f:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04) IOMMU group 33: [8086:0e1e] 7f:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04) [8086:0e1f] 7f:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04) IOMMU group 34: [8086:0ee0] 7f:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee2] 7f:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee4] 7f:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee6] 7f:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee8] 7f:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) IOMMU group 35: [8086:0ee1] 7f:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee3] 7f:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee5] 7f:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee7] 7f:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee9] 7f:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) IOMMU group 36: [8086:0ea0] 7f:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04) [8086:0e30] 7f:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04) IOMMU group 37: [8086:0ea8] 7f:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04) IOMMU group 38: [8086:0e71] 7f:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04) IOMMU group 39: [8086:0eaa] 7f:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 40: [8086:0eab] 7f:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 41: [8086:0eac] 7f:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 42: [8086:0ead] 7f:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 43: [8086:0eb0] 7f:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04) IOMMU group 44: [8086:0eb1] 7f:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04) IOMMU group 45: [8086:0eb2] 7f:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04) IOMMU group 46: [8086:0eb3] 7f:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04) IOMMU group 47: [8086:0eb4] 7f:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04) IOMMU group 48: [8086:0eb5] 7f:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04) IOMMU group 49: [8086:0eb6] 7f:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04) IOMMU group 50: [8086:0eb7] 7f:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04) IOMMU group 51: [8086:0e1d] 7f:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04) [8086:0e34] 7f:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04) [8086:0e81] 7f:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04) [8086:0e36] 7f:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04) IOMMU group 52: [8086:0ec8] 7f:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04) [8086:0ec9] 7f:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04) [8086:0eca] 7f:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04) IOMMU group 53: [8086:0e02] 80:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04) IOMMU group 54: [8086:0e04] 80:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04) IOMMU group 55: [8086:0e08] 80:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04) IOMMU group 56: [8086:0e20] 80:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04) IOMMU group 57: [8086:0e21] 80:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04) IOMMU group 58: [8086:0e22] 80:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04) IOMMU group 59: [8086:0e23] 80:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04) IOMMU group 60: [8086:0e24] 80:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04) IOMMU group 61: [8086:0e25] 80:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04) IOMMU group 62: [8086:0e26] 80:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04) IOMMU group 63: [8086:0e27] 80:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04) IOMMU group 64: [8086:0e28] 80:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04) IOMMU group 65: [8086:0e2a] 80:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04) IOMMU group 66: [8086:0e2c] 80:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04) IOMMU group 67: [10de:1b82] 82:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070 Ti] (rev a1) [10de:10f0] 82:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1) IOMMU group 68: [8086:0e80] ff:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04) IOMMU group 69: [8086:0e90] ff:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04) IOMMU group 70: [8086:0ec0] ff:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04) [8086:0ec1] ff:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04) [8086:0ec2] ff:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04) [8086:0ec3] ff:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04) IOMMU group 71: [8086:0e1e] ff:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04) [8086:0e1f] ff:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04) IOMMU group 72: [8086:0ee0] ff:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee2] ff:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee4] ff:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee6] ff:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee8] ff:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) IOMMU group 73: [8086:0ee1] ff:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee3] ff:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee5] ff:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee7] ff:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) [8086:0ee9] ff:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) IOMMU group 74: [8086:0ea0] ff:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04) [8086:0e30] ff:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04) IOMMU group 75: [8086:0ea8] ff:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04) IOMMU group 76: [8086:0e71] ff:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04) IOMMU group 77: [8086:0eaa] ff:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 78: [8086:0eab] ff:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 79: [8086:0eac] ff:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 80: [8086:0ead] ff:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) IOMMU group 81: [8086:0eb0] ff:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04) IOMMU group 82: [8086:0eb1] ff:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04) IOMMU group 83: [8086:0eb2] ff:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04) IOMMU group 84: [8086:0eb3] ff:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04) IOMMU group 85: [8086:0eb4] ff:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04) IOMMU group 86: [8086:0eb5] ff:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04) IOMMU group 87: [8086:0eb6] ff:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04) IOMMU group 88: [8086:0eb7] ff:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04) IOMMU group 89: [8086:0e1d] ff:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04) [8086:0e34] ff:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04) [8086:0e81] ff:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04) [8086:0e36] ff:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04) IOMMU group 90: [8086:0ec8] ff:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04) [8086:0ec9] ff:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04) [8086:0eca] ff:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04) USB Devices Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 003: ID 0781:5571 SanDisk Corp. Cruzer Fit Bus 001 Device 004: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 003: ID 0764:0501 Cyber Power System, Inc. CP1500 AVR UPS SCSI Devices [0:0:0:0] disk SanDisk Cruzer Fit 1.00 /dev/sda 16.0GB [3:0:0:0] disk ATA ST8000DM004-2CX1 0001 /dev/sdb 8.00TB [4:0:0:0] disk ATA TOSHIBA HDWE150 FP2A /dev/sdc 5.00TB [8:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sdd 1.00TB [10:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sde 1.00TB [11:0:0:0] disk ATA WDC WD80EMAZ-00W 0A83 /dev/sdf 8.00TB [11:0:1:0] disk ATA WDC WD80EMAZ-00W 0A83 /dev/sdg 8.00TB [11:0:2:0] disk ATA WDC WD60EFRX-68L 0A82 /dev/sdh 6.00TB [11:0:3:0] disk ATA WL6000GSA6457 1C01 /dev/sdi 6.30TB [N:0:0:1] disk INTEL SSDPED1D280GA__1 /dev/nvme0n1 280GB Edited August 19, 20205 yr by CowboyRedBeard
September 22, 20205 yr Author On 8/18/2020 at 2:17 PM, jonp said: Hi there, I am actually in the midst of doing some pretty extensive storage performance testing in the lab. We're testing with a combination of SSDs, NVMe drives, and a 10gbps network. This will take some time to test and analyze, but we are actively researching this. Have you guys had any progress on this issue? Thanks
September 22, 20205 yr Have you guys had any progress on this issue? ThanksYes and no. We have determined some of the excess writes on SSDs that some users were seeing may have been due to the boundary size we set which we have resolved. That will be in an upcoming update, but will require a process to convert which we are still sorting out. Separately we are testing smb performance in a 10gbps environment to see how we hold up in that setting. That testing is still underway and should be completed soon. However the CPU usage issue reported here hasn't been able to be reproduced just yet, so we are still trying.Sent from my Pixel 3 XL using Tapatalk
September 22, 20205 yr Author 11 minutes ago, jonp said: Yes and no. We have determined some of the excess writes on SSDs that some users were seeing may have been due to the boundary size we set which we have resolved. That will be in an upcoming update, but will require a process to convert which we are still sorting out. Separately we are testing smb performance in a 10gbps environment to see how we hold up in that setting. That testing is still underway and should be completed soon. However the CPU usage issue reported here hasn't been able to be reproduced just yet, so we are still trying. Sent from my Pixel 3 XL using Tapatalk Can I assist with testing data?
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.