- Minor
As title.
HP DL180 G6, 64gb ram, dual X5570 xeon
6 disk 4TB WD RED array
1 disk 4TB WD RED parity
1 PCIe 960gb NVMe for cache and VM
1 SSD 120GB unassigned disk
Write speeds (to nvme cache drive) completely saturate my GbE network at 100+ MB/s
Write speeds to the 120GB ssd (unassigned disk) also around 100+ MB/s
Read speeds from array, NVMe cache and 120GB ssd (unassigned disk) all max out at around ~65 MB/s
hdparm test in terminal - These are the 6 disk (+1 parity) that make up the array. Also the 120GB ssd and NVMe Cache drive
root@hpdl180g6:~# hdparm -tT /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/nvme0n1
/dev/sdb:
Timing cached reads: 19330 MB in 1.99 seconds = 9701.13 MB/sec
Timing buffered disk reads: 530 MB in 3.00 seconds = 176.38 MB/sec
/dev/sdc:
Timing cached reads: 19264 MB in 1.99 seconds = 9667.14 MB/sec
Timing buffered disk reads: 516 MB in 3.00 seconds = 171.74 MB/sec
/dev/sdd:
Timing cached reads: 18796 MB in 1.99 seconds = 9432.50 MB/sec
Timing buffered disk reads: 470 MB in 3.01 seconds = 156.05 MB/sec
/dev/sde:
Timing cached reads: 18840 MB in 1.99 seconds = 9453.48 MB/sec
Timing buffered disk reads: 478 MB in 3.01 seconds = 158.81 MB/sec
/dev/sdf:
Timing cached reads: 18816 MB in 1.99 seconds = 9441.74 MB/sec
Timing buffered disk reads: 434 MB in 3.01 seconds = 144.16 MB/sec
/dev/sdg:
Timing cached reads: 19058 MB in 1.99 seconds = 9563.10 MB/sec
Timing buffered disk reads: 448 MB in 3.01 seconds = 148.74 MB/sec
/dev/sdh:
Timing cached reads: 18868 MB in 1.99 seconds = 9467.43 MB/sec
Timing buffered disk reads: 518 MB in 3.00 seconds = 172.40 MB/sec
/dev/sdi: (shitty 120GB ssd)
Timing cached reads: 18418 MB in 1.99 seconds = 9241.06 MB/sec
Timing buffered disk reads: 578 MB in 3.00 seconds = 192.36 MB/sec
/dev/nvme0n1:
Timing cached reads: 18958 MB in 1.99 seconds = 9513.47 MB/sec
Timing buffered disk reads: 4350 MB in 3.00 seconds = 1449.83 MB/sec
nvme is a gen3 but the server only has gen2 pcie slot - hence the slower (boo hoo 1500MB/s) reads
Crystal Disk Mark 6 on the 120GB unassigned disk - Mapped as a network drive in Windows 7.
Sequential Read (Q= 32,T= 1) : 118.354 MB/s
Sequential Write (Q= 32,T= 1) : 117.362 MB/s
Random Read 4KiB (Q= 8,T= 8) : 88.570 MB/s [ 21623.5 IOPS]
Random Write 4KiB (Q= 8,T= 8) : 110.104 MB/s [ 26880.9 IOPS]
Random Read 4KiB (Q= 32,T= 1) : 83.825 MB/s [ 20465.1 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 109.400 MB/s [ 26709.0 IOPS]
Random Read 4KiB (Q= 1,T= 1) : 9.412 MB/s [ 2297.9 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 6.245 MB/s [ 1524.7 IOPS]
Test : 4096 MiB [S: 22.3% (24.9/111.7 GiB)] (x1) <0Fill> [Interval=5 sec]
and
Sequential Read (Q= 32,T= 1) : 118.345 MB/s
Sequential Write (Q= 32,T= 1) : 117.305 MB/s
Random Read 4KiB (Q= 8,T= 8) : 87.685 MB/s [ 21407.5 IOPS]
Random Write 4KiB (Q= 8,T= 8) : 108.942 MB/s [ 26597.2 IOPS]
Random Read 4KiB (Q= 32,T= 1) : 83.757 MB/s [ 20448.5 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 109.225 MB/s [ 26666.3 IOPS]
Random Read 4KiB (Q= 1,T= 1) : 8.299 MB/s [ 2026.1 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 6.240 MB/s [ 1523.4 IOPS]
Test : 4096 MiB [S: 22.3% (24.9/111.7 GiB)] (x1) [Interval=5 sec]
iperf3 reports 112 MB/s in both send and receive.
As I said, write speeds over the network to the cache drive are flawless at 100+ MB/s.
Write speeds to the array, not using cache drive and not using parity are also in excess of 100+ MB/s - IE: writing to 1 disk at a time
I have noticed after removing the parity drive to do read testing, the rebuild of the parity is operating at ~83 MB/s.
Parity is valid
Last checked on Tuesday, 2019-01-22, 12:06 (four days ago), finding 0 errors.
Duration: 13 hours, 15 minutes, 6 seconds. Average speed: 83.9 MB/sec
ifconfig
oot@hpdl180g6:~# ifconfig bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 1500 inet myIPaddress netmask 255.255.255.0 broadcast BROADCAST ADDRESS inet6 IPV6-MAC ADDRESS prefixlen 64 scopeid 0x20<link> ether MACADDRESS txqueuelen 1000 (Ethernet) RX packets 8983667 bytes 12223721990 (11.3 GiB) RX errors 0 dropped 0 overruns 154 frame 0 TX packets 9531485 bytes 13138810014 (12.2 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
What am I doing wrong????
This is a pretty much vanilla install of unRAID with all the tools guys like Space Invader One suggests - monitoring tools and unassigned disk plugin.
Any ideas guys ?