Wavey Posted July 22, 2020 Share Posted July 22, 2020 (edited) As I was trouble shooting my Terabytes of writes to my Cache drive the consensus has been to not use BTRFS as the filesystem on the cache. I also changed a few dockers to Linuxserver versions (plex, radarr, sonarr, etc... anything that can be linuxserver is linuxserver). No encrypted devices either. I switched everything to XFS but I still see btrfs "kworkers" doing things. Is this normal? I'm also still seeing a ton of writes from loop2. I mostly troubleshot this on Reddit where everyone who switched to XFS saw a crazy number of writes go way down. I'm just trying to see did I do this properly. Below is a snapshot of my iotop showing all those workers. Total DISK READ : 22.29 M/s | Total DISK WRITE : 214.80 M/s Actual DISK READ: 25.52 M/s | Actual DISK WRITE: 227.95 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 20004 be/4 root 0.00 B/s 13.07 K/s 0.00 % 19.09 % [kworker/u65:14+btrfs-endio-write] 17469 be/4 root 0.00 B/s 39.20 K/s 0.00 % 16.76 % [kworker/u65:6+btrfs-endio-write] 1957 be/4 root 0.00 B/s 13.07 K/s 0.00 % 14.39 % [kworker/u65:12+btrfs-endio-write] 10390 be/0 root 9.40 M/s 114.31 M/s 0.00 % 10.95 % [loop2] 28072 be/4 root 13.07 K/s 509.64 K/s 0.00 % 8.69 % [kworker/u64:1-btrfs-submit] 30178 be/4 nobody 4.90 M/s 721.98 K/s 0.00 % 2.78 % Plex Transcoder -codec:0 hevc -~-4ed1-b8d9-7f2e8280d292/progress 21217 be/4 root 382.23 K/s 1450.50 K/s 0.00 % 2.54 % [kworker/u64:0-btrfs-submit] 19173 be/4 nobody 13.07 K/s 75.62 M/s 0.00 % 2.27 % mono --debug Radarr.exe -nobrowser -data=/config 19174 be/4 nobody 3.27 K/s 19.16 M/s 0.00 % 1.94 % mono --debug Radarr.exe -nobrowser -data=/config 18640 be/4 root 189.48 K/s 875.53 K/s 0.00 % 1.75 % [kworker/u64:2-btrfs-submit] 15556 be/4 root 13.07 K/s 104.54 K/s 0.00 % 1.64 % [kworker/u66:10-btrfs-worker] 16391 be/4 nobody 1950.34 K/s 0.00 B/s 0.00 % 0.71 % mono --debug Radarr.exe -nobrowser -data=/config [Finalizer] 1386 be/4 root 225.42 K/s 16.33 K/s 0.00 % 0.62 % shfs /mnt/user -disks 16777215 ~time,allow_other -o remember=330 16390 be/4 nobody 793.86 K/s 0.00 B/s 0.00 % 0.34 % mono --debug Radarr.exe -nobrowser -data=/config [SGen worker] 1726 be/4 root 13.07 K/s 65.34 K/s 0.00 % 0.20 % [kworker/u66:0-btrfs-worker] 19172 be/4 nobody 26.14 K/s 39.20 K/s 0.00 % 0.13 % mono --debug Radarr.exe -nobrowser -data=/config 13439 be/4 root 13.07 K/s 52.27 K/s 0.00 % 0.11 % [kworker/u66:9-btrfs-worker] 17662 be/4 nobody 836.33 K/s 0.00 B/s 0.00 % 0.11 % Plex Media Server 1395 be/4 root 437.76 K/s 0.00 B/s 0.00 % 0.10 % shfs /mnt/user -disks 16777215 ~time,allow_other -o remember=330 1392 be/4 root 339.76 K/s 13.07 K/s 0.00 % 0.08 % shfs /mnt/user -disks 16777215 ~time,allow_other -o remember=330 17663 be/4 nobody 166.61 K/s 0.00 B/s 0.00 % 0.07 % Plex Media Server 24303 be/4 nobody 568.44 K/s 13.07 K/s 0.00 % 0.07 % python3 /app/tautulli/Tautulli.py --datadir /config 5316 be/4 root 52.27 K/s 182.95 K/s 0.00 % 0.07 % [kworker/u66:4-flush-btrfs-1] 18465 be/4 nobody 55.54 K/s 0.00 B/s 0.00 % 0.06 % mono --debug Sonarr.exe -nobrow~ -data=/config [Thread Pool Wor] 11290 be/4 root 0.00 B/s 13.07 K/s 0.00 % 0.05 % [kworker/u65:8+btrfs-endio-write] 7998 be/4 nobody 55.54 K/s 0.00 B/s 0.00 % 0.05 % mono --debug Sonarr.exe -nobrow~ -data=/config [Thread Pool Wor] 13188 be/4 root 26.14 K/s 91.47 K/s 0.00 % 0.05 % [kworker/u66:7-flush-btrfs-1] 8008 be/4 nobody 45.74 K/s 0.00 B/s 0.00 % 0.04 % mono --debug Sonarr.exe -nobrow~ -data=/config [Thread Pool Wor] 13076 be/4 nobody 104.54 K/s 0.00 B/s 0.00 % 0.04 % mono --debug Sonarr.exe -nobrow~ -data=/config [Thread Pool Wor] 1387 be/4 root 238.48 K/s 22.87 K/s 0.00 % 0.04 % shfs /mnt/user -disks 16777215 ~time,allow_other -o remember=330 32016 be/4 root 13.07 K/s 104.54 K/s 0.00 % 0.04 % [kworker/u66:24-btrfs-endio-write] 8010 be/4 nobody 179.68 K/s 0.00 B/s 0.00 % 0.04 % mono --debug Sonarr.exe -nobrow~ -data=/config [Thread Pool Wor] 1303 be/4 root 117.61 K/s 6.53 K/s 0.00 % 0.04 % shfs /mnt/user -disks 16777215 ~time,allow_other -o remember=330 18187 be/4 root 0.00 B/s 26.14 K/s 0.00 % 0.04 % [kworker/u65:7+btrfs-endio-write] 8013 be/4 nobody 19.60 K/s 0.00 B/s 0.00 % 0.04 % mono --debug Sonarr.exe -nobrow~ -data=/config [Thread Pool Wor] Edited July 22, 2020 by Wavey Quote Link to comment
JorgeB Posted July 22, 2020 Share Posted July 22, 2020 1 minute ago, Wavey said: Is this normal? Yes, docker image is still btrfs, but it shouldn't case high writes on a xfs filesystem, though if you want the latest beta allows xfs for the docker image. Quote Link to comment
Wavey Posted July 22, 2020 Author Share Posted July 22, 2020 (edited) 18 minutes ago, johnnie.black said: Yes, docker image is still btrfs, but it shouldn't case high writes on a xfs filesystem, though if you want the latest beta allows xfs for the docker image. Ah ok at least I know they should be there. Yeah my cache is getting about 80-100TB a month written to it. It's 1 years old and is at 755TB written to it, soooooo yeah. I have a second unraid system that the cache is 4.5 years old that has about 65TB written to it. Both have Plex installed, commented this because from other posts I've noticed this is a culprit so it's probably not in my case. Edited July 22, 2020 by Wavey Quote Link to comment
JorgeB Posted July 22, 2020 Share Posted July 22, 2020 btrfs docker image or vdisks on a btrfs filesystem can and usually does have a very high write amplification, but a btrfs docker image on a xfs filesystem should not be a problem, recently released v6.9-beta25 also mostly fixes this problem for btrfs filesystems, though btrfs will always have more writes than xfs due to being a COW filesystem. 1 Quote Link to comment
Wavey Posted July 22, 2020 Author Share Posted July 22, 2020 Hehe, cow and butter. Yeah I'll give it a try, redo the cache using spaceinvaders method and go from there. Thanks! Quote Link to comment
Matthew_K Posted July 22, 2020 Share Posted July 22, 2020 Sorry to hijack this thread, but I wan ran into an interesting issue. So I am in the process of settign up a new system I setup my cache drive, made sure that everything was working. Mounted and formatted the drive with btrfs. Then I swapped the Hardware as soon as it became available, now when I look at the system, It tells me that the Cache is healthy and active but the mount say No filesystem. If i go to /mnt/cache it also tells me that location does not exist. Looking at the error logs Jul 22 10:16:01 Tower kernel: ata2.00: exception Emask 0x10 SAct 0x7000 SErr 0x4090000 action 0xe frozen ... Jul 22 10:27:22 Tower kernel: BTRFS error (device sdc1): parent transid verify failed on 26034176 wanted 56 found 19 Jul 22 10:27:22 Tower kernel: BTRFS warning (device sdc1): failed to read fs tree: -5 ... Jul 22 10:27:22 Tower emhttpd: /mnt/cache mount error: No file system So am I looking at a Bad Drive, Cable, something else? I was mapping an NTFS drive to copy data to the Disk 1 partition and started and stopped the array in rapid succession. also the drive does have one Uncorrectable error count Quote Link to comment
JorgeB Posted July 22, 2020 Share Posted July 22, 2020 7 minutes ago, Matthew_K said: parent transid verify failed This means there were lost writes sometime in the past and it's usually a fatal error, you need to re-format the cache, if there's important data here are some recovery options, btrfs restore is likely the best for this. P.S. next time please create your own thread since it's unrelated to the OP. Quote Link to comment
Wavey Posted August 1, 2020 Author Share Posted August 1, 2020 On 7/22/2020 at 1:03 PM, johnnie.black said: btrfs docker image or vdisks on a btrfs filesystem can and usually does have a very high write amplification, but a btrfs docker image on a xfs filesystem should not be a problem, recently released v6.9-beta25 also mostly fixes this problem for btrfs filesystems, though btrfs will always have more writes than xfs due to being a COW filesystem. my cache writes dropped by 75%, thanks for the information! 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.