Questions around migrating server to new hardware and changing from xfs array to zfs pool

[Underscoreus]

Hey everyone,
I'm planning to migrate my entire Unraid server to a new box (new motherboard, cpu, RAM etc.) as well as migrating the data on to a new set of SSD drives, in the process going from a set of 4 array devices (1 parity, 3 "active") using xfs to a pool of 4 drives using zfs (RAIDz, 1 parity, 3 "active"). I am looking to transfer all of my data (only around 5TB in my case) as well as all of my VM's and preferably all of my dockers as well. I was hoping you guys could help me out by answering some questions I had about the process, the more I look into it and think about it, the more I'm not sure if I'm out of my depth with this project:
 

1. Are there any plugins or even dockers to help with/automate this process? I know you can get rsync dockers and stuff like that but I was more wondering if there were tools or processes specifically designed to move entire Unraid servers onto new drives, that would update all the paths for all my shares, VM's, dockers etc. Or is this going to be a pretty manual process of doing a copy of all the files onto the new drives and manually going through and changing the paths of all my stuff to point to the same files on the new drives? 
2. If there does exist any software to automate this transition, will it be hindered by the fact that I am going from an array of drives using xfs to a pool of drives using zfs?
3. If the solution is to manually copy the files over, what would be the best way to do so? Clone the content of each drive in the old array to a corresponding SSD in the new array? Just copy the entire content of the "user" folder to the new pool? Is the files even really stored in the user folder or is that just symlinks to what is actually stored on each disk?
4. Is a specific tool better to use in the scenario I have to manually move all my files? Would there be much of a difference setting up and using something like rsync vs just loading up a krusader docker and doing the copy via the gui there?
5. When transferring to the new hardware I'll be unassigning all the cpu pinning and removing all references to hardware that won't be carried over from my VM configs, but is there anything else I need to be sure to do before pulling the plug? Presumably stuff like static IP's won't matter since they are assigned to a specific ethernet port on the motherboard that won't be carried over, so I'll just need to configure the port on the new motherboard with a static IP instead?
6. Hardware wise, neither of my motherboards have 8 SATA ports so to have all 8 drives plugged in for the initial file transfer I'm just thinking of getting a pcie SATA expansion card, are there any known issues with Unraid not working with these? I'm thinking of getting this card from startech.

If you could help me answer any of these questions I'd be really grateful! I'm not 100% sure of all the proper lingo so if anything is unclear please leave a question and I'll try to amend the post with an update.

Thanks for the read

[JonathanM]

4 hours ago, Underscoreus said:

going from a set of 4 array devices (1 parity, 3 "active") using xfs to a pool of 4 drives using zfs (RAIDz, 1 parity, 3 "active").

Why?

[Underscoreus]

1 hour ago, JonathanM said:

Why?

Why am I going over to ZFS from XFS?
As far as I understand, Unraid, using XFS, does not support SSD trim, which would cause more wear on the drives over time. Whilst if I use ZFS I can enable trimming.

Edited October 14, 2023 by Underscoreus

[JonathanM]

What drive(s) are you planning to use in the main parity array?

[Underscoreus]

56 minutes ago, JonathanM said:

What drive(s) are you planning to use in the main parity array?

I was planing on going with the Samsung 870 EVO 2TB or 4TB since it uses MLC flash so hopefully it'll last a bit longer than QLC SSDs.

Edited October 14, 2023 by Underscoreus

[JonathanM]

8 hours ago, Underscoreus said:

Clone the content of each drive in the old array to a corresponding SSD in the new array?

I got the impression you weren't using the array for your SSD's, except for the one you just mentioned, 870 evo.

[Underscoreus]

4 hours ago, JonathanM said:

I got the impression you weren't using the array for your SSD's, except for the one you just mentioned, 870 evo.

I used poor wording, let me try to clarify:
I'm moving from a server using 4 array devices (HDDs) to a server using 4 SSDs in a pool (these 4 SSDs will be the 870 evos). So poor wording of me to say "corresponding SSD in the new array", I should have said "corresponding SSD in the new pool" or "corresponding SSD in the new setup". I was trying to ask if I should just copy the content from a harddrive, to a corresponding SSD. I'm not sure if each drive is even exposed in the file system when you configure a ZFS pool or if it just shows up as a single folder, maybe that is where the confusion comes from.

Edited October 14, 2023 by Underscoreus

[JonathanM]

10 hours ago, Underscoreus said:

I'm not sure if each drive is even exposed in the file system when you configure a ZFS pool or if it just shows up as a single folder,

Each pool is a single destination, no matter the number of individual drives in the pool.

 

You will still need a drive assigned to a disk slot in the parity array for now, this can be a spare USB stick if you want, but it's required to have a drive assigned to the main array. This requirement may (will?) change in the next major version of Unraid.

 

Do you currently have any pools defined?

[Underscoreus]

4 hours ago, JonathanM said:

Each pool is a single destination, no matter the number of individual drives in the pool.

Alright, good to know!

 

4 hours ago, JonathanM said:

You will still need a drive assigned to a disk slot in the parity array for now, this can be a spare USB stick if you want, but it's required to have a drive assigned to the main array. This requirement may (will?) change in the next major version of Unraid.

Ah, alright, I'll be sure to get either a spare drive or like you suggested just a spare USB stick to assign to the main array.

 

4 hours ago, JonathanM said:

Do you currently have any pools defined?

I currently have a 250GB cache pool set up that I'm not really using, but I haven't pulled the trigger on all the components and set up the new ZFS pool yet since I wanted to ask these questions first in case some of my hardware choices were unwise or if there were major issues with actually going through with this operation on the software side.

Edited October 15, 2023 by Underscoreus

[JonathanM]

3 hours ago, Underscoreus said:

I currently have a 250GB cache pool set up that I'm not really using

Normally the docker system and container appdata as well as VM files all reside there. Where are yours?

[Underscoreus]

2 hours ago, JonathanM said:

Normally the docker system and container appdata as well as VM files all reside there. Where are yours?

My docker system and containers are stored on in my cache pool but my VM files are not. Since the cache drive was kind of small I chose not to store them there since it would more or less fill it. In the new setup I might be looking at just having everything be stored in the ZFS pool instead of having just a few files in a separate pool, just for simplicity.

[JonathanM]

You could set up your new array with your current single drive cache pool reassigned as a single array disk1, that would take care of the array disk requirement.

 

Now that you have more fully described your setup, I have a 10,000ft overview of what I would recommend.

 

1. Move your current Unraid setup intact (Boot USB license stick, array drives and cache SSD) to the new board. Make sure it's functional, like you said in #5 in the OP you would need to redo any CPU and board hardware specific stuff.

2. Add your new SSD's into a newly defined pool, name it as the final primary storage, the name doesn't particularly matter, but you will be referencing it pretty much from now on, so something like mainzfs or otherwise descriptive would work.

3. Set the new ZFS pool as primary for all desired shares that are currently on the array, set the array as secondary storage, and mover action from array to ZFS pool.

4. For shares that currently live on the SSD cache pool, set the array as primary storage, and the cache pool as secondary, mover action from cache pool to array.

5. Disable vm and docker services in settings. NOT JUST STOP THE CONTAINERS AND VMS, you must disable the services. There should be no VM or Docker tabs on the GUI when done.

6. Run mover. Wait. For many hours. Check if mover is still running. Wait more hours.

7. After mover is done, for any shares that were on the cache pool and should now be on the array, set the ZFS pool to primary, the array as secondary, mover action from array to ZFS pool.

8. Run mover. Wait. Hopefully not nearly as long this time.

9. Set all shares to ZFS pool as primary, secondary to none. Verify all files now are on ZFS pool.

10. Execute new config, preserve all, apply. Go to main page, remove all the current array assignments, remove cache pool, assign cache pool ssd as disk1 in the array. Leave ZFS pool alone.

11. Start array, format array disk1 if needed.

12. Verify Docker and VM services are pointing to the ZFS pool correctly, start the services.

 

There may be nuances and dragons in each of these steps. Don't assume, ask questions. All my instructions ASSumed you are using user shares as intended, if you were actually pointing to specific disk paths those will need to be amended to the corresponding user shares at some point in the process. Exclusive mode should also be enabled after everything is on the new ZFS pool.

[Underscoreus]

Thank you so much for the helpful overview!

22 hours ago, JonathanM said:

There may be nuances and dragons in each of these steps. Don't assume, ask questions.

I'll probably be back either in this thread or in a separate thread for any specific issues or questions that arrise. Hopefully there wont be too many of them. I'm unsure if I should mark this thread as solved now and just add an addendum post with any additional info I required when doing the transfer once I finish the process, or if I should wait to mark the thread as solved till the process is finished.

Edited October 16, 2023 by Underscoreus

[IronBeardKnight]

NVME CACHE
I transferred from btrfs to zfs and have had a very noticeable decrease in write speed before and after memory cache has been filled not only that but the ram being used as a cache makes me nerves even though I have a ups and ecc memory, I have noticed that my duel nvme raid1 in zfs cache pool gets full 7000mbps read but only 1500mbps max write which is a far cry from what it should be when using zfs. I will be swtich my appdata pools back to btrfs as it has nearly all the same features as zfs but is much faster from my tests.
The only thing that is missing to take advantage of the btrfs is the nice gui plugin and scripts that have been done to deal with snapshots which i'm sure someone could manage to bang up pretty quick using existing zfs scripts and plugins.

Its important to note here that my main nvme cache pool was both raid1 in btrfs and zfs local to that file system type of raid1 obviously.

ARRAY
I also started doing some of my array drives to single disk zfs as per spaceinvadors videos as the array parity and expansion abilities would be handled by unraid which is where I noticed the biggest downside to me personally which was that zfs single disk  unlike any zpools is obviously missing a lot of features but more so is very heavily impacted write performance and you still only get single disk read speed obviously once the ram cache was exhausted. I noticed 65% degrading in write speed to the zfs single drive.

I did a lot of research into BTRFS vs ZFS and have decided to migrate all my drives and cache to BTRFS  and let unraid handle parity much the same as the way spaceinvader is doing zfs but this way I don't see the performance impact I was seeing in zfs and should still be able to do all the same snapshot shifting and replication that zfs does. Doing it this way I avoid the dreaded unstable btrfs local file system raid 5/6 and I get nearly all the same features as ZFS but without the speed bug issues in Unraid.

DISCLAIMER
I'm sure ZFS is very fast when it comes to an actual zpool and not on a single disk drives situation but it also very much feels like zfs is a deep storage only file system and not really molded for an active so to speak array.

Given my testing all my cache pools and the drives within them will be btrfs raid 1 or 0 ( Raid 1 giving you active bitrot protection) and my Array will be Unraid handled parity with individual BTRFS files system single disks

Hope this helps others in some way to avoid days of data transfer only to realis the pitfalls.

[JorgeB]

5 hours ago, IronBeardKnight said:

I noticed 65% degrading in write speed to the zfs single drive.

There's currently a bug affecting write speeds for zfs on the array:

 

 

[IronBeardKnight]

6 hours ago, JorgeB said:

There's currently a bug affecting write speeds for zfs on the array:

 

 

yeah one of the reasons I switched back. when the bug is fixed I may look again in future but have wasted enough time transferring files around already on this.

Appreciate the reply mate.

[Underscoreus]

7 hours ago, JorgeB said:

There's currently a bug affecting write speeds for zfs on the array:

 

 

So this is only an issue when ZFS is setup using array disks and not if I setup ZFS using pools, is that correct? Also, is this bug still active in 6.12.4 or or has it been fixed? I didn't see it mentioned in the known issues section for any of the 6.12 versions released so far.

[JorgeB]

2 minutes ago, Underscoreus said:

So this is only an issue when ZFS is setup using array disks and not if I setup ZFS using pools, is that correct?

Correct.

 

2 minutes ago, Underscoreus said:

Also, is this bug still active in 6.12.4 or or has it been fixed?

Still an issue.

[Underscoreus]

11 minutes ago, JorgeB said:

Correct.

 

Still an issue.

Alright, thanks for the confirming. Hopefully shouldn't affect me too much since I'm going with the pools approach.

[IronBeardKnight]

21 hours ago, Underscoreus said:

Alright, thanks for the confirming. Hopefully shouldn't affect me too much since I'm going with the pools approach.

Hopefully now with significant investment ZFS will get its  pool expansion issues finally polished off soon 

Edited October 18, 2023 by IronBeardKnight

[JorgeB]

8 minutes ago, IronBeardKnight said:

ZFS will get its  pool expansion issues finally polished off soon

It's expected for openzfs 2.3.0 in about a year.

[Underscoreus]

8 hours ago, IronBeardKnight said:

Hopefully now with significant investment ZFS will get its  pool expansion issues finally polished off soon 

It's probably not going to affect me since I'm not planning on immidiately expanding my pool after switching drives, but out of curiosity, what kind of issues are there with ZFS pool expansion? I couldn't find anything specific online other than not being able to expand a ZFS pool like you do with a btrfs or xfs pool/array but instead having to add another VDEV with the same amount of drives as the existing one, which is just how ZFS works.

[JorgeB]

10 hours ago, Underscoreus said:

what kind of issues are there with ZFS pool expansion?

In about a year you will be able to expand a raidz pool by just adding one disk.

[Underscoreus]

Alright!
It took me a while to get all the components, assemble them and then pull the trigger but I have now emerged on the other end, transfer completed and would like to fullfill my promise about a post-mortem report about how the process went and the issues, as well as the solutions, I encountered.

First off, a huge thank you to JonathanM for his guide further up in the thread, it was invaluable.
As for the "nuances and dragons" I encountered along my journey:

• After I updated my Unraid version from 6.9.2 to 6.12.4 the GPU passthrough to my VM stopped working. After looking on the forum, this seems to be a wide spread problem with no real conclusive way to solve it. I tried to get it working again for a little while but since I wasn't going to use the GPU in my new setup I let it go.
• The motherboard in my new setup did not allow me to boot using legacy mode no matter what I tried (double checking the USB was the first and only thing in the boot order, disabling fast boot, disabling XHCI Hand-of, Enabling Legacy USB Support and making sure the USB is plugged in to a USB 2.0 port on the motherboard). What ended up solving the issue was changing the name of the folder on the Unraid Boot USB from EFI- to EFI to enable UEFI boot, and just like that it started working exactly as expected.
  • Note that the USB was always visible in the BIOS, but whenever I tried booting to it I was just brought back to the BIOS screen again.
  • Also, the boot times of my new machine was very slow, even with fast boot enabled, but disabling XHCI Hand-of seems to have massively improved the boot times. Not sure what it does or why, but if you find your new machine slow to boot, maybe disabling this could be an idea to try.
• After booting in to Unraid my new drives were initially not showing up in the Main overview. This was due to the fact that they were all plugged into a SATA expansion card and the firmware on the sata controller was apparently old. I downloaded the new firmware and installation software from a third-party site and went ahead and flashed the new firmware to the controller. After doing so the expansion card worked a treat and the drives immidiately showed up in my overview.
  • To update the firmware on this card you will need another PC running Windows, as the firmware upgrading tool is only windows compatible. Another requirement is that the chipset on the motherboard can't be a 600-series chipset (Atleast at the time of writing). This is the chipset used by the latest 13000 and 14000 series processors from Intel.
  • If you have a lot of other PCIE cards plugged in to your Windows machine that you want to use to update the firmware on the card you might need to remove some of the other PCIE expansion cards in your system. I had both a USB 3.0 expansion card, an ethernet adapter and a GPU plugged in to my machine and it caused the Sata controller card to not be visible in the firmware updating software, possibly due to there not being enough PCIE lanes on my CPU. To solve this problem I simply turned off the machine, unplugged the USB and Ethernet expansion cards, rebooted and tried again, and just like that the card showed up and I was able to update the firmware. Thank god for this reply from da_stingo on the forum, without it I would have never thought to try to update the firmware and assumed my Sata expansion card was a lost cause.
  • One last thing to note about this SATA expansion subject. Apparently some expansion cards, depending on the chipset they use, don't play nice with Unraid/Linux. This is something I was completely unaware of and it was only by luck that I ended up buying an expansion card with a copatible/reccomended chipset. If you are planing on doing this, please have a look at this forum thread for reccomendations on which chipsets to look out for before buying.
• Slight ammendment that was needed to the great guide provided by JonathanM mentioned above, I was unable to assign the cache pool as the secondary storage and the array as the primary storage for the shares that were hosted on my cache drive so instead I simply kept the cache pool as the primary storage location, set the array as the secondary storage location and just reversed the transfer direction to go from the cache to the array, which yielded the same end result of transfering the files from my cache pool to my array.
• Another issue I encountered when doing the moving was that some data did not want to move, namely the docker and virtio folders in my system share, as well as all my empty shares. This turned out to be because I had the CA Mover plugin installed and running. Once I removed that plugin and retried the mover it moved all the remaining files in my system share as well as recreated the empty folders/shares in my new ZFS pool without issue. Thanks to this post on the forum for giving me the solution to this specific problem.
• While running mover I encountered some files that had too long filenames to be transfered by mover from my disks to my zfs pool causing some "filename too long" errors to appear in the log terminal. It was fairly easy to fix by opening a terminal and renaming the offending files using the mv command.
  • Hot tip (that is probably very common knowledge), if your file name or file path contains spaces, just wrap it in "", like this: "/mnt/disk1/this path has spaces/file.txt"
• Even after all that I needed to do some manual moving of a few files that were persistently sticking to my cache drive using either mv or the rsync commands. Most of these were temp files in a few different locations that I probably could have done without but I figured since there were only a few folders I would try to get them transfered as well.

 

And that was that! Over all a fairly smooth transition, all things considered!
Hopefully this final answer can be of help to someone in the future that are planning on doing a similar transition or if someone has any of these individual problems.
I'll mark this final post as the solution so it becomes more visible (Not sure if I can mark multiple posts as solutions) but the true heroes in all of this are JonathanM for the amazing guide he made, JorgeB for general advice and the great list of compatible Sata controllers as well as the other people in the other forum threads that actually had all the answeres. A huge thank you and shout out to them!!