Understanding cache reads/writes

[dnLL]

I have a RAID1 of cache NVME SSD drives. I originally set it up in BTRFS but recently changed for ZFS.

 

I'm trying to understand why is my writes usage so much higher than reads. Currently both SSDs are sitting down at around 250 TiB write and 8 TiB read per the smart report with 395 days of uptime.

 

On my cache are the VMs, the dockers and temporary cache storage for other shares. When I first installed my SSDs, I was plagued by an issue with dockers where I would get super high write usage (multiple TiB per day). This was eventually resolved. Now, it's more or less following the same curve with about 20 TiB of writes per month. That's about the size of my whole array (24 TB), which is obviously ridiculous.

 

My questions are the following.

 

1- When the mover is invoked, should there be a 1:1 ratio between the previous writes and the files being moved turning into reads?

 

2- When loading VM disk files from the cache, I assume it reads the whole file? Does every read/write within the VM get transferred to the VM disk file (in this case my cache)?

 

3- Anything particular pertaining to dockers that I should be aware of concerning high writes usage? I'm using the folder option for docker storage.

 

Sent from my Pixel 7 Pro using Tapatalk

 

 

[dnLL]

I did some digging thanks to the File Activity plugin combined with iostat.

 

root@server01:~# iostat -mxd nvme0n1 -d nvme1n1
Linux 6.1.74-Unraid (server01)  03/17/2024      _x86_64_        (16 CPU)

Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
nvme0n1          1.30      0.10     0.00   0.00    0.64    76.38   94.35      2.97     0.00   0.00    0.72    32.26   25.29      2.76     0.00   0.00    2.61   111.85    4.48    2.10    0.14   8.74
nvme1n1          1.32      0.10     0.00   0.00    0.63    76.47   98.80      2.97     0.00   0.00    0.51    30.81   25.29      2.76     0.00   0.00    2.44   111.85    4.48    2.09    0.12   8.56

 

So, writing around 3 MB/s consistently while looking with iostat. With the File Activity plugin, I noticed the following:

 

** /mnt/user/domains **
Mar 17 22:49:25 MODIFY => /mnt/cache/domains/vm-linux/vdisk1.img
Mar 17 22:49:26 MODIFY => /mnt/cache/domains/vm-automation/haos_ova-8.2.qcow2
...
Mar 17 22:49:26 MODIFY => /mnt/cache/domains/vm-automation/haos_ova-8.2.qcow2
Mar 17 22:49:27 MODIFY => /mnt/cache/domains/vm-windows/vdisk1.img
...
Mar 17 22:49:27 MODIFY => /mnt/cache/domains/vm-windows/vdisk1.img
Mar 17 22:49:28 MODIFY => /mnt/cache/domains/vm-automation/haos_ova-8.2.qcow2
...
Mar 17 22:49:28 MODIFY => /mnt/cache/domains/vm-automation/haos_ova-8.2.qcow2
Mar 17 22:49:28 MODIFY => /mnt/cache/domains/vm-dev/vdisk1.img
...
Mar 17 22:49:28 MODIFY => /mnt/cache/domains/vm-dev/vdisk1.img

** Cache and Pools **
Mar 17 22:44:13 MODIFY => /mnt/cache/domains/vm-automation/haos_ova-8.2.qcow2
...
Mar 17 22:44:29 MODIFY => /mnt/cache/domains/vm-automation/haos_ova-8.2.qcow2

 

For instance, lots of writes coming from my Home Assistant VM running HAOS. That would be because the data from HA is hosted locally on the VM rather than on a NFS share. That also explains while I'm getting a lot of writes without necessary reading it. I might want to work on that and on similar issues with my other VMs.

 

Thought I would post my findings just to help others with similar issues using the right tools to potentially find the problem. Obviously with the File Activity plugin, I don't get the actual quantity of data that is being written but it gives me a good idea.