Hello there,

 

I'm trying to move a share from my TrueNAS server to my UnRaid server, and for some reason, it starts off as normal, around 200MB/s. But very quickly, it drops down to about 15MB/s and slowly but surely goes slower even still. It's now hovering at 3-5MB/s. I'm talking about 3.5TB of data, and I don't want to wait a few days for it to finish.

 

My UnRaid server is an R720 and the array is a two-drive non-raid ZFS array. When I started the first attempt, I quickly stopped it, because I realized I still had a cache drive enabled, which is smaller than the data I wanted to copy, so I quickly disabled that, and started over. Shouldn't have anything to do with it, but I just figured I'd mention it.

 

I know that my UnRaid server is the problem, not the TrueNAS machine, because all the UnRaid shares I've set up are very slow to respond. It literally takes 15 seconds for Windows File Explorer to load the blue bars that show capacity utilization.

 

The setup is as follows:

TrueNAS pool: 4x HDD, data striped (I know, not responsible, but I only had to park it for a few days). Not sure if relevant, but 2x Xeon 2637 V3, 32GB RAM.

Connected to Windows machine (that's initiation the copy) over 25Gb connection

Windows PC is connected to a switch with a 2.5Gbit connection.

UnRaid machine also 2.5Gbit connection, two HDD drive ZFS array. Not sure if relevant, but 2x Xeon 2620 V1, 192GB of RAM)

 

Does anyone know what's going on here, or know where to look?

 

Cheers

 

EDIT:

Things I've looked into since

• Disabled cache drive
• Checked the flowrate
• Ruled out TrueNAS server as the issue; same issue with local files
• Ruled out ZFS memory taking away memory from UnRaid OS.

unraidserver-diagnostics-20241017-1813.zip

Edited October 17, 20241 yr by LeroyLaFontaine

To clarify it:

You have a TrueNAS directly connected to Windows Box with a 25G link?

The same Windows box is connected to a switch (and to unraid) over a (seperate?) 2,5G link?

 

In this case I would guess the windows box is overloaded by incoming data that needs speed conversion 10:1. It runs out of buffers an stalls.

Make sure you have turned on Flow Control everywhere (and everybody uses it too), so Windows can stop TrueNAS and spool on to Unraid to free up some Buffers / RAM.

This will decrease the "burst" rate but make transfer overall "smoother" and a bit faster. Depending on the Disk Speed of UNRAID, maybe 60Mb/s (with parity) could be possible. Dont expect more, even down to 20Mb/s can happen!

Also tell UNRAID not to buffer much in RAM, this could also stall the transfer! Default is 10%, better lower it to 1 or 2%.

 

Edited October 17, 20241 yr by MAM59

I’m not sure it has anything to with the speed situation you described. My TrueNAS shares are still very responsive, whilst the UnRaid share are very sluggish. Also, the pool in question has hard drives, with around 300 MB/s usually. However, I will look into the flowers you mentioned. 
 

The RAM thing does look interesting. But if that’s the issue, what’s the point if it then? At the time, I bought it to improve performance. I had it set if with a lot of RAM for ZFS, and a 1TB NVMe to improve writes (which I disabled temporarily for this transfer).

You should post the diagnostics during a transfer to see if there's something visible there.

11 minutes ago, LeroyLaFontaine said:

I had it set if with a lot of RAM for ZFS, and a 1TB NVMe to improve writes (which I disabled temporarily for this transfer).

Then it is not available for UNRAID. So what you see is the ZFS buffer running full. The disks cannot keep up with the 2,5G link (thats normal, no normal drive can keep up for a long time) and mirroring it effectivly halfs the write speed too.

But, if you spend the RAM on ZFS there is also no way to tweak with UNraid parameters.

11 minutes ago, LeroyLaFontaine said:

whilst the UnRaid share are very sluggish

Of course they are. If the disks are blocking, NOTHING will work anymore and has to wait.

11 minutes ago, LeroyLaFontaine said:

But if that’s the issue, what’s the point if it then?

The point is that it helps FOR A SHORT period. Its not normal to transfer a few TB at once everyday. Once in sync you will only do smaller quantities of writes, these might be taken without any stalling (hopefully). But if you copy over complete disks, no system can help you to prevent blocking.

 

  

7 hours ago, MAM59 said:

Then it is not available for UNRAID. So what you see is the ZFS buffer running full. The disks cannot keep up with the 2,5G link (thats normal, no normal drive can keep up for a long time) and mirroring it effectivly halfs the write speed too.

But, if you spend the RAM on ZFS there is also no way to tweak with UNraid parameters.

The ZFS ram is set to 163GB, of the 192GB available, that leaves for 29GB for UnRaid, and the very few dockers I run.

Anyways, I did a reboot, and started the transfer again, and it would drop down very quickly to 15MB/s. I could see the ZFS cache filling up again, since it was empty before because of the reboot. But the speed started dropping when ZFS memory was at like 9%. 

 

8 hours ago, JorgeB said:

You should post the diagnostics during a transfer to see if there's something visible there.

I've attached the diagnostics to this post.

 

Also I've rules out any issues with the TrueNAS server. I'm getting the same results when transferring a large, local file.

unraidserver-diagnostics-20241017-1813.zip

I missed before that you were using the array, thought it was a mirrored zfs pool, there's a known issues with write speeds to the array when they are formatted zfs, pools are not affected, though 5-15MB/s seems lower than usual, but it can vary a lot with the disk model, I see that the disks are mostly empty, try formatting one of them xfs or btrfs and retest.

38 minutes ago, JorgeB said:

I missed before that you were using the array, thought it was a mirrored zfs pool, there's a known issues with write speeds to the array when they are formatted zfs, pools are not affected, though 5-15MB/s seems lower than usual, but it can vary a lot with the disk model, I see that the disks are mostly empty, try formatting one of them xfs or btrfs and retest.

If I'm gonna empty the drive, and format it, could I not take it out of the array, and set it up as a new one-drive ZFS pool? And if yes, could I remove the other drive that's pretty much filled, and place that in another pool by itself, leaving the data and the ZFS filesystem?

Edited October 17, 20241 yr by LeroyLaFontaine

6 minutes ago, LeroyLaFontaine said:

If I'm gonna empty the drive, and format it, could I not take it out of the array, and set it up as a new one-drive ZFS pool?

For that you don't even need to re-format, just create a pool and assign it there, just need to do a new config first (Tools - New config)

49 minutes ago, JorgeB said:

For that you don't even need to re-format, just create a pool and assign it there, just need to do a new config first (Tools - New config)

Aight, I'll try that out. And what about the remaining Disk? Can I just remove that from the array as well, put it in another pool, by itself, whilst keeping the data in the progress?

 

I can then add some random USB stick to the array so I'll at least have one drive to be able to start the array.

9 minutes ago, LeroyLaFontaine said:

Can I just remove that from the array as well, put it in another pool, by itself, whilst keeping the data in the progress?

Only if you upgrade to v7-beta, the array is optional there.

So I emptied the drive, put it in a pool, zfs, no improvements, formatted it to xfs, put it back to the array, no improvements yet again.

This is getting very frustrating

unraidserver-diagnostics-20241018-0037.rar

Controller is in RAID mode, which is not recommended, but should not cause that, is the speed normal if you write to the Nvme device?

I’m sorry to have wasted your time, but apparently the drive itself is the issue. I made this wonky setup to attach this 3.5” drive to the SFF backplane of the TrueNAS server, and setup a new pool in there. And I get low write speeds as well.

is that something that can happen to a drive, messed up write speeds but being able to read normally?

Edited October 18, 20241 yr by LeroyLaFontaine

8 minutes ago, LeroyLaFontaine said:

s that something that can happen to a drive, messed up write speeds but being able to read normally?

Not very common but I've seen it before.