ZFS ARC vs Unraid Cache priority

[YujiTFD]

Good day, everyone!

 

I'm on my way from TrueNAS SCALE to Unraid (tired of TNS's complexity) and since I want best of both worlds, I chose to use ZFS-formatted drives in hybrid array for snapshots and flexible drives configuration. I honestly browsed this forum and googled alot, but some questions remains unanswered (or I missed them), so I would really appreciate some insights towards Unraid.

 

1. While using cache drive, how exactly ZFS ARC works: does ARC gets filled up first and moves data to cache until mover moves everything to the array? And If I don't have cache drive, does ARC works the same way it does in regular ZFS pool, gradually transferring data to the array during writing? I'd like to limit SSD wearing, so I'm reviewing my options.

 

2. I have 10 GbE network setup and I'm somewhat worried about transfer speed: I don't need high bandwidth throughput constantly, but times I do - will cache drive (1 TB NVMe) level out writing speed between TNS pure ZFS pool (3 x mirror 2 wide) and Unraid's hybrid ZFS? In theory it should to the extent of it's capacity, but I'd like to hear real-life results.

 

It's all for now, thank you in advance.

[itimpi]

Not sure if you are talking about using ZFS in the main array or in a pool?    If in a pool then the cache drive is irrelevant as you cannot current have a cache drive in front of a pool and even if you could it is unlikely any performance advantage would be seen..  If you are talking about ZFS in the main array (where every drive is an independent file system) then ARC would not come into play until a file is moved to the array as the two mechanisms are independent of each other. 

 

Also ZFS in the main array is known to currently perform badly so not recommended at the moment unless it is specifically being used to support ZFS snapshots from a pool as then performance is not really the prime concern.

[YujiTFD]

36 minutes ago, itimpi said:

Not sure if you are talking about using ZFS in the main array or in a pool?

In main array, Unraid calls it "Hybrid Array"

36 minutes ago, itimpi said:

If you are talking about ZFS in the main array <...> then ARC would not come into play until a file is moved to the array

Not mean to deny your words, but my test setup is clearly shows actual ARC usage in Unraid's main array of ZFS-formatted HDDs:

- fresh Unraid with no VMs or Dockers,

- 1 parity and 2 data HDDs in main array (all three are old 500 GB SATA II Hitachi Desktars, Unraid settings for filesystem is set to ZFS),

- 12 GB RAM reserved for ARC with "options zfs zfs_arc_max=12582912" in zfs.conf,

- 1 GbE Unraid network card

- 10 GbE network card for source file transfer (Windows 11 PC)

resulted in stable transfer over 100 MB/s throughput (102-105 MB/s) until ARC was over, then transfer dropped to ~25-30 MB/s. After that I delete copied file from Unraid's share - ARC drops to 0% full, start copying again - it starts growing immidiately.

 

How is it possible?

[JorgeB]

1 hour ago, YujiTFD said:

does ARC gets filled up first and moves data to cache until mover moves everything to the array?

ARC is mostly read cache in RAM, nothing to do with the mover, it works the same as any other zfs implementation, you can change how much RAM the ARC can use.

 

ARC will also cache writes, but only for 5 secs or so, before they are flushed to the devices, then you are limited by the device write speed.

[YujiTFD]

5 minutes ago, JorgeB said:

ARC will also cache writes, but only for 5 secs or so, before they are flushed to the devices

As I said before in https://forums.unraid.net/topic/166724-zfs-arc-vs-unraid-cache-priority/#comment-1424125, while 12 GB ARC had space in it, my transfer speed over 1GbE link was ~103 MB/s sharp, it sustained much longer than 5 seconds.

 

It's really strange, let me fire it and test it again.

 

P.S. Still, if ARC IS a write cache, how does it work with the Mover?

Edited May 28 by YujiTFD

[itimpi]

5 minutes ago, YujiTFD said:

As I said before in https://forums.unraid.net/topic/166724-zfs-arc-vs-unraid-cache-priority/#comment-1424125, while 12 GB ARC had space in it, my transfer speed over 1GbE link was ~103 MB/s sharp, it sustained much longer than 5 seconds.

 

It's really strange, let me fire it and test it again.

 

Are you sure that you are writing to the array?  If you have set the share to be cached then the speed may be what you get writing to the cache.   If that is what is happening then as I said ARC will not come into play for the array drives until the file is transferred by mover to the main array.

[JorgeB]

6 minutes ago, YujiTFD said:

how does it work with the Mover?

 

10 minutes ago, JorgeB said:

nothing to do with the mover

 

[JorgeB]

7 minutes ago, YujiTFD said:

my transfer speed over 1GbE link was ~103 MB/s sharp, it sustained much longer than 5 seconds.

That's not a problem while the device can keep up with the writes, 103MB/s should be not a problem for most devices, even for disks.

[YujiTFD]

20 minutes ago, JorgeB said:

ARC will also cache writes, but only for 5 secs or so

I take back my statement, but not completely: my 12 GB ARC was not filled till 100% before speed dropped, only about 40%, which took ~20 seconds, but not 5

 

Yeah, I know ARC has nothing to do with the mover, question was: if both present, which one goes first? Apparently, it's ARC, but not till 100%.

 

18 minutes ago, itimpi said:

Are you sure that you are writing to the array?

Yes, I have no cache in test setup, only 3 old HDDs. And looks like that ARC was the first in the queue.

[JorgeB]

15 minutes ago, YujiTFD said:

eah, I know ARC has nothing to do with the mover, question was: if both present, which one goes first? Apparently, it's ARC, but not till 100%.

I don't understand the question, mover can only be used if you are using a pool as primary storage and the array as secondary, and in that case you can run it manually or on a schedule.

[YujiTFD]

2 minutes ago, JorgeB said:

I don't understand the question

An NVMe SSD as cache, a main array of ZFS HDDs, a share with primary storage set to cache, and secondary storage - to the array with mover on schedule.

 

In this particular case will ARC be used? Can it be used?

 

[JorgeB]

The ARC is always used when you read or write from/to a zfs filesystem.