Hi, I am new to unraid so I feel like I'm missing stuff or have made a mistake or two when making configurations. But essentially I feel like I am not getting the best speeds possible with ZFS. My cache pool is 4 sata SSDs in raidz but the speeds are nowhere near what I would expect to get. Im not sure if its a hardware limitation of a mistake I've made. Any help would be amazing. Thanks

Server: r730xd with H730p mini
CPU: Dual Intel(R) Xeon(R) CPU E5-2630 
RAM: 64Gb


SSD Cache - raidz
 

 

HDD 6 mirrored vdevs

 

seraphim-diagnostics-20240914-2233.zip

I assume you are testing over SMB? If yes not far from line speed, you should also run a single stream iperf test in both directions first to see LAN bandwidth

2 minutes ago, JorgeB said:

I assume you are testing over SMB? If yes not far from line speed, you should also run a single stream iperf test in both directions first to see LAN bandwidth

Yep it’s over SMB. I’m running on 10gig so bandwidth shouldn’t be an issue. I also did a check with iperf and I was getting close to 10gb both ways. 
 

I also did a test with fio and dd and got similar results. I’ll have to run those tests again later to and post the results.

 

I feel like the SSD speeds in particular are very underwhelming. Or is that what I really should be expecting?

11 minutes ago, ikko said:

I’m running on 10gig so bandwidth shouldn’t be an issue.

Yes, and you are already getting close to 1GB/s, at least for writes, that's line speed.

19 hours ago, JorgeB said:

Yes, and you are already getting close to 1GB/s, at least for writes, that's line speed.

You were right. The writes were being capped at line speed. I ended up remaking the pool into a raid0 just to test out the speeds but I didn't see an improvement at all in the reads. 



Just for my sanity is this the best I can expect from the reads? Anyways I appreciate you clearing up my confusion with the line speeds. 

In my experience dd is not a really good test, I prefer to test real life transfers, use one or more large files and transfer over SMB or locally on the server, locally you can test with pv, for example

 

pv /mnt/path/to/large/file > /dev/null

 

If you have a second fast pool you can use that as the destination.

 

I would expect around 1 to 2GB/s max speed for real world transfers, based on the rest of the hardware, CPU single threaded rating also has a large impact, since transfers are single threaded.

18 hours ago, JorgeB said:

In my experience dd is not a really good test, I prefer to test real life transfers, use one or more large files and transfer over SMB or locally on the server, locally you can test with pv, for example

 

pv /mnt/path/to/large/file > /dev/null

 

If you have a second fast pool you can use that as the destination.

 

I would expect around 1 to 2GB/s max speed for real world transfers, based on the rest of the hardware, CPU single threaded rating also has a large impact, since transfers are single threaded.

Yep once again you were right. Sending a file to /dev/null was 2GB/s, and sending it to another pool was about 900MB/s with a max of about 1GB/s. 

On 9/15/2024 at 7:18 PM, JorgeB said:

I would expect around 1 to 2GB/s max speed for real world transfers, based on the rest of the hardware, CPU single threaded rating also has a large impact, since transfers are single threaded.

It seems I have run into a new problem/concern. I have am getting good speeds but I am concerned with the iowait that I believe is related to the read speeds but I'm not 100%. I was benchmarking my lancache and saw really high iowait and the only thing I can think of is the slower than expected read performance. The benchmark show great results but the iowait bothers me. 

I recently also got two Toshiba px04srb096 and in raid 0 the crystaldiskmark and pv test yeiled similar results compared to my other SSD pool. I searched everywhere trying to figure out what is going on because it seems a bit suspicious to me to be getting extremely similar results. I saw a lot of things on the h730p mini being a possible issue but I'm completely lost as of now

Benchmark results:


iowait:


Updated diagnostics: 

seraphim-diagnostics-20240926-1806.zip

 

Edited September 26, 20241 yr by ikko

2 hours ago, ikko said:

I have am getting good speeds but I am concerned with the iowait

If the speeds are good I wouldn't worry about it, and the i/o wait you are seeing looks normal to me.

2 hours ago, JorgeB said:

If the speeds are good I wouldn't worry about it, and the i/o wait you are seeing looks normal to me.

I just find it very weird that both SSD pools have the same performance. It feels like there is a bottleneck of something is capping out the read performance in some way. Another thing I did find odd was that I found no difference in performance when I had 3x Samsung SSDs in a pool vs 4x.  If this is normal then I guess I really shouldn't be too fussed about it. Just need that confirmation that I'm not going crazy 

2x Toshiba 

 

4x Samsung 

Those look OK to me for random writes.

8 hours ago, JorgeB said:

Those look OK to me for random writes.

Thank you so much man. I've been legit losing my mind thinking something was wrong. I appreciate it