Rune Posted February 26, 2020 Share Posted February 26, 2020 Ran with a single XFS-encrypted cache disk for a while, then moved to a 3-disk BTRFS-encrypted cache pool. Since that change, I've noticed a constant write IO on the cache disks of about 2-3MB/s (per the Main page). IOStat on the unraid host confirms the IO is happening. No other tool can give me any idea on what is doing the writing (eg: iotop points at shfs). I have 3 VMs with disks on the cache pool (that were there prior to the change). Monitoring on the VMs, iostat says there is maybe 10kB/s average write IO on each. I NFS-mount in a bunch of stuff, but monitoring nfsiostat tells me that there's not much going on there, and mostly read traffic. Since Btrfs is copy-on-write, I checked, and made sure that the VM disks, libvirt, docker, and other random-write stuff is COW-disabled (chattr +C, and verify with lsattr). With no IO coming from SMB clients, an only minor IO coming from VMs and NFS mounts, and COW disabled... how can I tell where all the extra IO is coming from? (I can account for, say, 100kB/s, but I'm seeing 3MB/s (1.5MB/s when you account for btrfs writing redundant blocks). Quote Link to comment
JorgeB Posted February 26, 2020 Share Posted February 26, 2020 Maybe same as this? Quote Link to comment
TechMed Posted February 26, 2020 Share Posted February 26, 2020 I am having a similar problem. Everything turned off, Dockers, VM, UD but still having high writes to my Crucial MX500. Spikes into the 20+MB/s. Second machine showing over all lower write number, but still higher than previously. Diags Attached tower01-diagnostics-20200226-0428.zip Quote Link to comment
Rune Posted February 26, 2020 Author Share Posted February 26, 2020 14 hours ago, johnnie.black said: Maybe same as this? Sounds similar, but I don't have any torrent clients, nor did I have any Windows machines on at the time. Quote Link to comment
Rune Posted February 29, 2020 Author Share Posted February 29, 2020 Did some more testing on this, and there is definitely some write amplification going on here. I transferred everything off of the cache, and marked it as 'No' for all shares, and that dropped the write IO to zero on cache. At that point, there was reasonably constant 50-200kB/s write IO on the array (including parity). That IO only appeared on the disk that I moved the VM disks to. I moved everything except the VMs back, one at a time, and the write IO stayed on the array. Shut down the VMs, the write IO stopped, moved the disks to cache pool, set the 'No COW' flag (chattr +C) on the disks, and when I started up one, the IO was already above the 50-200kB/s that I saw as an average on the array. On cache, it's about 900kB/s-3.8MB/s ...which is about a 18-19x write amplification for being on the cache pool instead of the array. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.