BlueSialia Posted March 27, 2020 Share Posted March 27, 2020 Hello, I've just recently built my first Unraid system to use as the "family computer" hosting a VM for my girlfriend and I. The system specs are: Motherboard : ASUS ROG Strix X570-F Gaming CPU : AMD Ryzen 9 3900X RAM : CMW32GX4M2C3200C16 x4 = 64GB Storage : SB-ROCKET-NVMe4-1TB (Cache drive) ST8000DMZ04 x2 (Array drives) GPUs : ZT-P10700G-10M EVGA 670 FTW SIG2 I've copied files from HDD to HDD and from Nvme to Nvme with the following results: root@UnBlue:/mnt/user/backups/VM backups# rsync -P BlueSialia-Win.template.img Test.img BlueSialia-Win.template.img 4,378,624,000 3% 184.11MB/s 0:11:45 ^C root@UnBlue:/mnt/user/vdisks/BlueSialia - Windows 10# rsync -P vdisk1.img Test.img vdisk1.img 9,482,371,072 6% 373.33MB/s 0:05:34 ^C The HDD to HDD are fine. Actually, I expected less since it has to deal with parity as well. But the Nvme to Nvme is 10 times slower than it should. The Nvme is formatted as Btrfs and Dynamix SSD Trim is installed and scheduled to run every night. `lspci -s 01:00.0 -vv` looks fine as well (to my inexperienced eye). 01:00.0 Non-Volatile memory controller: Phison Electronics Corporation E16 PCIe4 NVMe Controller (rev 01) (prog-if 02 [NVM Express]) Subsystem: Phison Electronics Corporation E16 PCIe4 NVMe Controller Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 56 NUMA node: 0 Region 0: Memory at f7c00000 (64-bit, non-prefetchable) [size=16K] Capabilities: [80] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #1, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 unlimited ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 16GT/s (ok), Width x4 (ok) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled AtomicOpsCtl: ReqEn- LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [d0] MSI-X: Enable+ Count=9 Masked- Vector table: BAR=0 offset=00002000 PBA: BAR=0 offset=00003000 Capabilities: [e0] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [f8] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [100 v1] Latency Tolerance Reporting Max snoop latency: 1048576ns Max no snoop latency: 1048576ns Capabilities: [110 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=10us PortTPowerOnTime=300us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=32768ns L1SubCtl2: T_PwrOn=300us Capabilities: [128 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [1e0 v1] Data Link Feature <?> Capabilities: [200 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00001010 000022ff 000c0000 0ef9590e Capabilities: [300 v1] Secondary PCI Express <?> Capabilities: [340 v1] Physical Layer 16.0 GT/s <?> Capabilities: [378 v1] Lane Margining at the Receiver <?> Kernel driver in use: nvme Kernel modules: nvme What can I try? Quote Link to comment
JorgeB Posted March 27, 2020 Share Posted March 27, 2020 rsync isn't built for speed, try the same copy with pv Quote Link to comment
BlueSialia Posted March 27, 2020 Author Share Posted March 27, 2020 Didn't know that rsync was slow. Anyway, I tried: root@UnBlue:/mnt/user/vdisks/BlueSialia - Windows 10# pv vdisk1.img > Test.img ^C.9GiB 0:00:16 [ 740MiB/s] [===> ] 8% ETA 0:02:51 It fluctuated between 400MiB/s and 900MiB/s. Faster, but still 4 to 5 times slower than the advertised speed. Quote Link to comment
JorgeB Posted March 27, 2020 Share Posted March 27, 2020 Copying with the same device as source and dest will never give you optimal speeds, and never max speed. Quote Link to comment
BlueSialia Posted March 27, 2020 Author Share Posted March 27, 2020 Oh, okay. Are those numbers the expected result then? Thank you for indulging me anyway. Quote Link to comment
JorgeB Posted March 27, 2020 Share Posted March 27, 2020 6 minutes ago, BlueSialia said: Are those numbers the expected result then? They look about right to me. Quote Link to comment
testdasi Posted March 27, 2020 Share Posted March 27, 2020 1 hour ago, BlueSialia said: Didn't know that rsync was slow. Anyway, I tried: root@UnBlue:/mnt/user/vdisks/BlueSialia - Windows 10# pv vdisk1.img > Test.img ^C.9GiB 0:00:16 [ 740MiB/s] [===> ] 8% ETA 0:02:51 It fluctuated between 400MiB/s and 900MiB/s. Faster, but still 4 to 5 times slower than the advertised speed. Remember that headline advertised numbers are always highly optimistic under very specific ideal conditions so any "real life" test will never come anywhere close. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.