Mover moves data from ZFS pool to XFS array extremely slowly

[Can0n]

So here is the story.

My current server (6.12.2)  has two 10TB parity disks and 22x6,8 and 10TB XFS disks for the array

I used two 1TB SSD in BTRFS RAID1 and two 4TB Spinners as a secondary pool called download-cache using BTRFS

I also have 2 USB3 4TB drives attached with unassigned devices

I wanted to convert both pools to ZFS so here is what i did.

set mover for the download-cache to move all the data to the array stopped all applications using the download-cache so no files were locked. ran mover when it was done and set the share (Media and a couple other shares) to array and primary no no secondary storage.

Once the download-cache was empty i let the apps use it again as the data would only go to the array.

I stopped the array and removed the 2x4TB disks and started the array without them then stopped it again and assigned them setting the file system to ZFS then formatted them adding the 2x USB3 4TB drives as well for a raidz 1 group of 4 disks.

once done my next step was appdata, domains and system shares (current on BTRFS)

i shutdown vm manager and docker and set the shares to move to the array. once done I did same process to remove cache, re-add and format as ZFS Mirror 1 group of 2 devices.

i decided to rsync it back to the cache from array as Mover took over 4 hours rsync took about 1.5 hours. most of my dockers failed to work correctly and i saw perms completely borked.

i ran mover to move them to the download-cache to test them and they all worked albeit slower being on spinner drives.

now i used a different rsync option to keep time, owner/groups and permissions rsync -avz /mnt/download-cache/apps /mnt/cache/appdata

 

that seemed to copy ok and all my dockers except Plex worked. Plex is acting like its a new server despite database and all files seeming to be there. I updated the config to point back to download-cache where the folder apps still had all the data

Plex is working albeit slower

so i decided to run mover on it last night the only thing moving is the Plex folder and its been 12 hours and still moving

Attached diagnostics but would like to know why BTRFS over 400GB took 4 hours to move and ZFS to XFS 214GB is taking more than 12 hours

some takeaways:

Would be nice to have a progress bar on Mover to see its status and potentially pause it if needing some bandwidth on the drives.

if a progress bar is not easily able to be added perhaps an estimated time to completion based on its current file list

thor-diagnostics-20230706-1155.zip

[Vr2Io]

md_write_method="auto"

 

Try enable reconstruction write first.

 

1 hour ago, Can0n said:

BTRFS over 400GB took 4 hours

Even that, it seems a bit slow, less then 30MB/s, but this also depends on large / small file size.

[Can0n]

39 minutes ago, Vr2Io said:

md_write_method="auto"

 

Try enable reconstruction write first.

 

Even that, it seems a bit slow, less then 30MB/s, but this also depends on large / small file size.

yeah 400GB 4 hours isnt bad Plex has millions of little files but moving from ZFS to XFS is dog slow since its just plex now and mover

i moved all the other docker appdata using rsync -avh /mnt/download-cache/apps /mnt/cache/appdata and it was from ZFS pool to ZFS pool at about 45min

plex went as well but when i started the container up it was trying to set me up as a new server. when i mapped the contain back to /mnt/download-cache/apps/PlexMediaServer

it fired right up so rsync is breaking something in plex

i enabled reconstruction and monitoring thanks for the tip I thought i had that on a long time ago

 

[Vr2Io]

19 minutes ago, Can0n said:

rsync is breaking something in plex

I always use rsync -ah ( same as -avh ) to sync appdata between disk and /tmp ( because I run docker in ram disk), it never have problem, but I will stop all docker & docker service before backup.

Edited July 6, 2023 by Vr2Io

[Can0n]

1 minute ago, Vr2Io said:

I always use rsync -ah ( same as -avh ) to sync appdata between disk and /tmp ( ram ) because I run docker in ram disk, it never have problem, but I will stop all docker & docker service before backup.

yeah the -v is just verbose so you can see what it is spitting out.

wish I had enough ram for that I have 64GB max for my i7 10700k

i am running a plex transcoder scratch disk

write reconstruction seems to be helping

[Vr2Io]

Just now, Can0n said:

wish I had enough ram for that I have 64GB max for my i7 10700k

I have 64G, but mainly because just several docker and not use more then 6GB. Remain memory use for CCTV recording.

[Vr2Io]

I have two ZFS pool RAID_0 and RAIDz1+RAIDz1, if compare to BTRFS RAID, performance have drop half, but it still far fast then single spinner disk, so copy / rsync to array won't got problem.

[Can0n]

25 minutes ago, Vr2Io said:

I have 64G, but mainly because just several docker and not use more then 6GB. Remain memory use for CCTV recording.

my Plex Folder alone is 214GB

i have 31 containers installed and run about 20 of the full time

[Vr2Io]

1 minute ago, Can0n said:

my Plex Folder alone is 214GB

i have 31 containers installed and run about 20 of the full time

Opps 🤪

[Can0n]

2 hours ago, Vr2Io said:

Opps 🤪

 

mover seems to have gotten stuck it says its still moving but no dat is being moved in meantime only things not moved for Plex is the Cache/Phototranscoder and gigs and gigs in it I might try moving the contents outside of plex and that way mover running plex back to cache pool might be quite a bit faster

[Veah]

Try using Unbalance plugin.  It will kind of let you see what is moving.

[Can0n]

25 minutes ago, Veah said:

Try using Unbalance plugin.  It will kind of let you see what is moving.

I do have that but it doesnt see the download-cache only the cache and my disks...odd

[Can0n]

4 hours ago, Veah said:

Try using Unbalance plugin.  It will kind of let you see what is moving.

So got Plex in appdata working now used mover and rsync -avh and then a quick move from /mnt/cache/apps2/PlexMediaServer to  /mnt/cache/appdata/PlexMediaServer

after 15 hours of mover started using rsync for all the tiny meta data files and found a vast majority of the 214GB was old PhotoCache i removed all that and it made Plex about 64GB

[Can0n]

On 7/6/2023 at 6:41 PM, Veah said:

Try using Unbalance plugin.  It will kind of let you see what is moving.

unfortunately unbalance doesnt support datasets just shares so that could explain why i could not see the temp location i had moved my appdata to 

[Can0n]

Mover still incredibly slow moving any thing even large linux isos to ZFS from XFS or XFS to ZFS ZFS is using datasets so i think its hanging on the delete portion of the mover script as it not checking for datasets and doing a recursive zfs destroy command 

[Vr2Io]

For ZFS single / array disk, try below solution

 

https://forums.unraid.net/topic/141781-utterly-low-file-copy-speed-what-is-wrong/?do=findComment&comment=1282301