I'm curious about what speeds to expect from zfs pools, so I asked chatGPT.

 

Question:

What is expected in read and write speed relative to drive speed on a 6 drive zaidz2 zfs pool given enough cpu power?

 

ChaiGPT:

In a standard RAID-Z2 configuration with 6 drives, assuming sufficient CPU power, large media files, and proper configuration, you can generally expect the following in terms of read and write speeds relative to the drive speeds:

Read Speeds: Read speeds can be expected to approach or even exceed the aggregate read speed of all the drives combined. This is because RAID-Z2 can leverage parallelism to read data from multiple drives simultaneously. Therefore, for large sequential read operations, you could see read speeds approximately 5 to 6 times the speed of a single drive.

Write Speeds: Write speeds in a RAID-Z2 configuration are influenced by parity calculations. While RAID-Z2 has a good balance of protection and performance, the write speeds might be somewhat lower compared to read speeds, especially for large sequential writes. Due to the dual parity calculations of RAID-Z2, write speeds could be around 2 to 3 times the speed of a single drive.

Keep in mind that these are rough estimates and actual performance can vary based on factors such as hardware efficiency, CPU power, cache configuration, fragmentation, and other environmental conditions.

 

 

Is this a realistic speed assumption? I look kind of optimistic.

 

For large files write speed seems about right, read speed in my experience will be more like 3 to 4 times single drive.

TrueNAS blog article on performance with 12 drives. As I understand it the base speed on each drive is 100MB/s with read/write of 900MB/s on the Z2 example. Meaning read/write speed of 75% of combined drives speed. If I understand correctly. 

 

I typical drive should be capable double the speed of 100MB/s, or do "capable of 100MB/s streaming reads and writes" something else?

You can get varying results, depending on how the tests are done, here is for example some tests I did a few months back, these were done by doing an actual large file copy read/write, max speed of the disks used was around 180MB/s:

 

Some test on both servers:

 

 

Drive share to drive share, SMB multi channel. Server 1 Win11 VM is on wirtio-net (wirtio not working) and server. 2 VM is on wirtio. I had a session with SpaceinvaderOne, but we could not solve it with the time we had left. 

 

How can server 1 perform that poorly?

Server 2 does not perform that great either.

 

In the TrueNAS blog post it uses 100 MB/s in hdd streaming speed. Is streaming speed the same as a hdd´s write speed, today typically 200-250MB/s?

 

 

helmaxx-diagnostics-20230823-1442.zip kjell-diagnostics-20230822-2004.zip

Are you using VMs for testing? Testing would be better done with bare metal server and clients.

1 hour ago, JorgeB said:

Are you using VMs for testing? Testing would be better done with bare metal server and clients.

 

First two lines are on the servers, the third line between server and VM running on the same server. 

OK, I think I get it, so server1 HDD based 6xraidz1 pool is writing at 470MB/s, that's seems reasonable to me, also note that IIRC having a separate logs device can actually slow down the pool in some cases.

 

NVMe to NVMe would expect more around 1.5GB/s, depending on the devices used, also DFM uses rsync, probably not the best tool to test, try copying a large file with pv:

 

pv /path/to/source > /path/to/dest

 

Also if you are using mostly large files try 1M rescordsize:

zfs set recordsize=1M pool_name

 

Thanks for taking time coming back to me. Much appreciated. 

 

Update the table with the pv command.

 

 

1. line: How can it be that server 1 from NVMe to NVMe is only 1.25 GB/s, while server 2 is 3.01 GB/s (at the seq write speed spec at 3.0 GB/s)? Seq. write speed on on NVMe on server 1 is 4.2 GB/s. 

2. line: Server 1 seq. write speed on hdd´s are 250 MB/s. Let's say 200 x 6 RaidZ2 should be 1.2 GB/s, not 473 MB/s. Or am I missing something here. For server 2 with 8 x sata ssd (500 MB/s) 

3. line: A good reading here should be about 70% of seq write at about 3 GB/s, not 1.2 GB and certainly not 320 MB/s. The server is basically copying to itself.

 

All NVMe drives are pcie 4, for server 1 is min. seq read 4700 MB/s write 4200 MB/s.

 

Cheers,

 

For testing the performance on a single hdd in a ZFS Pool is it same to run this command: 

 

dd if=/dev/zero of=/<mount_point>/testfile bs=1M count=<number_of_blocks> status=progress
 

14 hours ago, frodr said:

1. line: How can it be that server 1 from NVMe to NVMe is only 1.25 GB/s, while server 2 is 3.01 GB/s

Likely because of the faster CPU.

 

14 hours ago, frodr said:

2. line: Server 1 seq. write speed on hdd´s are 250 MB/s. Let's say 200 x 6 RaidZ2 should be 1.2 GB/s, not 473 MB/s. Or am I missing something here. For server 2 with 8 x sata ssd (500 MB/s)

There's always some overhead, don't expect speeds to increase linearly, and like mentioned logs and recordsize can make a difference, as well as other factor/tunes.

 

 

 

 

34 minutes ago, JorgeB said:

Likely because of the faster CPU.

 

There's always some overhead, don't expect speeds to increase linearly, and like mentioned logs and recordsize can make a difference, as well as other factor/tunes.

 

 

 

 

 

But the server 1 has the faster CPU, AMD Ryzen ThreadRipper PRO 3955WX, server 2 has I7-13700. 

 

 

Yes, some overhead is expected. but here it is more than 60%. And how can the huge difference in line 3 be explained.

 

 

 

pv, and most copy operations, are single threaded, so single threaded performance counts more, 13700 is almost twice as fast for a single thread.

 

Did you try with 1M recordize and without a logs vdev?

5 hours ago, JorgeB said:

pv, and most copy operations, are single threaded, so single threaded performance counts more, 13700 is almost twice as fast for a single thread.

 

Did you try with 1M recordize and without a logs vdev?

 

• I see. But how come a basic Windows PC can utilise NVMe speed almost fully and not Unraid?
• Are you also saying that a Treadripper Pro based server can only perform 1/4 of a I7-13700 which again is less than 1/3 of the NVMe drive speed, see line 3. This is very strange to me.
• No, I haven't tried 1M recordsize and without a logs vdev. The first only works with data added after the change in recordsize, right? And to try without log vdev means that I have to take down the hole Pool. I rather set up another pool for testing some day. 
• Can you help me with a command for testing hdd drive speed on a drive that´s in a zfs pool?

 

Cheers

 

6 minutes ago, frodr said:

I see. But how come a basic Windows PC can utilise NVMe speed almost fully and not Unraid?

In my experience Windows is the same, as is for others, see for example:

https://superuser.com/questions/1331298/why-do-windows-file-transfer-speeds-top-out-at-around-1-5-gigabytes-second

 

8 minutes ago, frodr said:

Are you also saying that a Treadripper Pro based server can only perform 1/4 of a I7-13700 which again is less than 1/3 of the NVMe drive speed, see line 3. This is very strange to me

I don't consider VM speeds since those add a lot of variables, for bare metal it's about half the speed, and that's what I would expect based on the single threaded performance.

 

10 minutes ago, frodr said:

The first only works with data added after the change in recordsize, right?

Correct, you can just change and repeat the transfer, especially for raidz1 I found that it can have a nice boot, it should not make much difference for single or mirrored pools.

 

11 minutes ago, frodr said:

Can you help me with a command for testing hdd drive speed on a drive that´s in a zfs pool?

If you mean test a single drive read speed you can use dd, though I don't put much value on static benchmarks, more on real world operations.