low performance write related to memory cache

[KokomiCat]

My configuration:

hp microserver gen8

cpu: e3-1265l v2

memory: 16g ddr3l

storage: WDC_WD10EZEX(1t hdd) x4, 2 x data, 2 x parity

 

WDC_WDS120G2G0A(120g ssd) x1, as cache

 

way to produce this problem:

I use Dynamix System Statistics to monitor the memory utilization and disk stats.

 

I use command below to test write speed.

dd if=/dev/zero of=/mnt/user/test/test.dbf bs=8k count=800000 conv=fdatasync

I use command below to test read speed.

dd if=/mnt/user/test/test.dbf of=/dev/null bs=8k count=800000

 

I have two shared folders, /mnt/user/test1 has cache set to No, and /mnt/user/test2 has cache set to Prefer.(write to SSD first)

Both of these partitions are btrfs.

 

I found that:

a) When the memory is enough

Both of them have 600MBps read and 600MBps write.

 

b) When the memory is fully occupied for cache

test1 has 170MBps read, test2 has 300MBps read

the write speed is werid.

test1 has only 40MBps write, test2 has 70MBps write

 

Dynamix System Statistics says that the size of memory cache is the same, but disk has both read and write at 100MBps.

 

I thought it might be a problem of the memory cache.

When the memory is full and new files are writing, it writes to both the memory for caching and the HDD.

As the memory is full, out-dated memory cache writes to HDD, which takes up half of the HDD write speed.

However, as the file is too large, the memory cache is useless(not going to be used), instead it write back to the HDD again, which causing the low writing performance.

 

In my opinion, memory cache may be the source of the problem, thus memory cache should be stop, or manually stop on this occassion.

 

This didn't happened on other OS on my server.

Edited August 16, 2020 by S4kura0ne

[Squid]

38 minutes ago, S4kura0ne said:

test1 has only 40MBps write, test2 has 70MBps write

Those speeds are more or less inline if you have a parity enabled share.

 

Once the memory cache gets filled, the speed will always drop down to those speeds-ish.  Enabling turbo write (reconstruct write) in disk settings will speed things up, at the expense of all drives must be spinning.

[KokomiCat]

Got it. Does it means that as soon as I use disk for parity, this speed(write at 40MB/s, read at 170MB/s) is the maximum speed I can got actually?

But what is the bottleneck? Most of my CPU cores has quite low usage.

 

btw, can enabling turbo write give me continued speed up(eg. 150GB single file write)? And do I need to have a UPS connected in case the reconstruct process was interrupted unexpectedly?

 

And can I get higher performance by changing the filesystem from btrfs to ext4 or other fs?

Edited August 17, 2020 by S4kura0ne

[itimpi]

You might find this part of the online documentation useful in explaining why you do not get raw disk speeds when writing to the parity protected array and how the ‘turbo’ write mode can speed things up.

[JonathanM]

9 hours ago, S4kura0ne said:

And do I need to have a UPS connected

Yes, Unraid needs to be properly shut down to avoid issues, regardless of write mode.

[trurl]

8 hours ago, itimpi said:

You might find this part of the online documentation useful in explaining why you do not get raw disk speeds when writing to the parity protected array and how the ‘turbo’ write mode can speed things up.

@itimpi didn't realize you had been so busy with the wiki. Great job. I need to link it more often.