Current Unraid Server:

Epyc 7302p

256GB Ram

Mellanox ConnectX-5 NIC (at 10g speeds)

ZFS pool = 5x IronWolf Pro's 22TB, and 2x Exos X22 22TB drives. All new drives.
 

I am moving the data off my Synology with 10g NIC (8x 12tb Exos X20 drives) directly to the ZFS pool I have setup on Unraid. Its around ~80TB, so I know this is going to take a few days/week. I have been doing is at night, Will start the move, and then in the morning either Pause, or just stop it and then pickup from there next time. So far about ~35TB used, so less than 50% full. 
 

For some reason, yesterday the transfer speeds which were sustained and consistently 500-600MBs have now dropped to 90-150MB/s. I have tried transfering files to the Pool from my smaller flash based NAS, and no luck. anything I try to write to the ZFS pool now, won't go over ~150MB/s

Any idea or help on what I am missing on why write speeds suddenly tanked? I transferred multiple TBs at that speed, so it wasn't ram cache running out of space. I cant figure out why, maybe I am being a dumbo and totally missing something..
 

I am the same file types, .mkv. Size of each file ranges from 20gb to 80gb depending on the movie or Series, but same file type across all use cases.

Also I went back and copied over some the same files that earlier did do 500MB/s and they now do 100-150MB/s.

you asked for ZFS, you got it.

Its bad, at least UNRAID is not the real good host for it (but its bad on others too).

ZFS is fast as long as there is enough ram free for the ARC cache, but then it crumbles down to drivespeed  / number of disks because it needs to wait for all drives to store data and parity.

You can simply proove this by rebooting and continuing the transfer later on. I bet it will go up to 500 again for some period, then drop down again for the rest of the time.

 

You could enlarge that period by editing a kernel variable that limits ZFS's ram usage (search this forum for this, I can't remember, dropped zfs after frustrating results long ago already), stock UNRAID only uses 1/8 of maximum RAM for ZFS.

 

But beware! runing ZFS without a proper UPS can result in total loss of data! The larger the ARC in RAM is, the more likely is that it wont be synched to disks fast enough if an

outage happens!!!

 

Use at your own risk!

 

 

9 hours ago, Alderaan said:

For some reason, yesterday the transfer speeds which were sustained and consistently 500-600MBs have now dropped to 90-150MB/s.

This is not normal, I get consistent write speeds even with pools getting closer to full, you can post the diagnostics during a transfer to see if there's something visible there.

15 hours ago, JorgeB said:

This is not normal, I get consistent write speeds even with pools getting closer to full, you can post the diagnostics during a transfer to see if there's something visible there.

Sorry for delay. I setup my Unraid ZFS pool as an CIFS mount on my synology, so Synology is transfering directly to the pool now. Attached is Diagnostics, took this during a transfer. 

 

 

Also Running cmd below during transfer

zpool iostat -v 1

Shows:
zfs-core 49.8T 90.3T 0 2.63K 0 645M

raidz2-0 49.8T 90.3T 0 2.63K 0 645M

sdc1 - - 0 563 0 91.6M

sde1 - - 0 232 0 92.6M

sdg1 - - 0 234 0 92.1M

sdh1 - - 0 553 0 91.6M

sdi1 - - 0 246 0 93.7M

sdd1 - - 0 547 0 91.7M

sdf1 - - 0 310 0 91.8M

 

But also its not consistent, sometimes the ping will show 0000 on writes. So the up is (500 to 600m) ... down (write showing 0 on pool)... up....down...up...down.. no ping is consistent from one to next if that makes sense.
 

Looking at the network stats on dashboard of my unraid server, the 10g nic, only peaks though at 1.6Gbps (200 MBps). Both Synlogy and Unraid set to 10g, and also enabled Jumbo frames so MTU running at 9000.
 

Synlogy transfer stats for the move shows the speed anywhere from 150MBs to 180MBs.....

Homelab Diagnostics May 7.zip

7 hours ago, Alderaan said:

But also its not consistent, sometimes the ping will show 0000 on writes. So the up is (500 to 600m) ... down (write showing 0 on pool)... up....down...up...down.. no ping is consistent from one to next if that makes sense.

That suggests to me the client may be the problem, it's not sending the data fast enough, can you try a large transfer from a PC for example with Windows and compare?

Does the Unraid write method impact zfs write speeds? (Disk settings: tunable: auto, read/modify/write or reconstruct) 

 

I also get with my raidz1 zfs pool 150MB/s or something with 4x 20TB Toshiba enterprise drives and 2x500GB NVME L2ARC drives in mirror. 

Edited November 21, 20241 yr by TallMan206

10 hours ago, TallMan206 said:

Does the Unraid write method impact zfs write speeds?

It does for any array disks, independent of the filesystem used, it will not affect pools.