ProfessionalIdiot Posted November 29, 2023 Share Posted November 29, 2023 I am new to ZFS so apologies if I misuse some terms. I created a ZFS pool via unRAID GUI of 4 devices. I then expanded this with 4 more devices via GUI. After that, I added cache using the CLI (from instructions here.) At this point, everything was working great. Here was my zpool at that point: root@nas01:~# zpool status pool: zfs state: ONLINE scan: scrub repaired 0B in 00:48:59 with 0 errors on Mon Nov 27 01:49:00 2023 config: NAME STATE READ WRITE CKSUM zfs ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdg1 ONLINE 0 0 0 sdh1 ONLINE 0 0 0 sde1 ONLINE 0 0 0 sdf1 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 sdc1 ONLINE 0 0 0 sdl1 ONLINE 0 0 0 sdm1 ONLINE 0 0 0 sdn1 ONLINE 0 0 0 cache sdj1 ONLINE 0 0 0 sdi1 ONLINE 0 0 0 errors: No known data errors I then tried adding 3 new devices to my pool via CLI: root@nas01:/var/log# zpool add zfs -f mirror sdk1 sdd1 sdb1 After that I tried the import instructions from JorgeB's post and the pool showed unmountable when I started the array. However, when I bring up the pool via CLI I see it is all happy and healthy: root@nas01:~# zpool status pool: zfs state: ONLINE scan: scrub repaired 0B in 00:48:59 with 0 errors on Mon Nov 27 01:49:00 2023 config: NAME STATE READ WRITE CKSUM zfs ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdg1 ONLINE 0 0 0 sdh1 ONLINE 0 0 0 sde1 ONLINE 0 0 0 sdf1 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 sdc1 ONLINE 0 0 0 sdl1 ONLINE 0 0 0 sdm1 ONLINE 0 0 0 sdn1 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 sdk1 ONLINE 0 0 0 sdd1 ONLINE 0 0 0 sdb1 ONLINE 0 0 0 cache sdj1 ONLINE 0 0 0 sdi1 ONLINE 0 0 0 (Note the new mirror-2 vdev) And I see the data/shares: root@nas01:~# df -h Filesystem Size Used Avail Use% Mounted on rootfs 32G 244M 32G 1% / tmpfs 32M 416K 32M 2% /run /dev/sda1 239G 448M 239G 1% /boot overlay 32G 244M 32G 1% /lib overlay 32G 244M 32G 1% /usr devtmpfs 8.0M 0 8.0M 0% /dev tmpfs 32G 0 32G 0% /dev/shm tmpfs 128M 2.6M 126M 2% /var/log tmpfs 1.0M 0 1.0M 0% /mnt/disks tmpfs 1.0M 0 1.0M 0% /mnt/remotes tmpfs 1.0M 0 1.0M 0% /mnt/addons tmpfs 1.0M 0 1.0M 0% /mnt/rootshare tmpfs 6.3G 0 6.3G 0% /run/user/0 zfs 17T 256K 17T 1% /mnt/zfs zfs/system 17T 499M 17T 1% /mnt/zfs/system zfs/isos 17T 256K 17T 1% /mnt/zfs/isos zfs/domains 17T 256K 17T 1% /mnt/zfs/domains zfs/x 25T 8.0T 17T 33% /mnt/zfs/x zfs/appdata 17T 10M 17T 1% /mnt/zfs/appdata I have tried to import in the unRAID GUI in two different orders: 1) The original order (with cache) with the 3 new mirror-2 disks below those, in the order shown in zpool status 2) What the new zpool order shows (vdev raidz1-0, 1-1, mirror-2, cache) Trying either of these, the GUI shows the disks as unmountable Due to how ZFS works I am not able to remove the mirror-2 vdev and revert to the old config It seems as if my data is safe, and I do not see any unhappiness in syslog, so two questions: 1) How do I get the array started again? What order do I need to list disks in the GUI? 2) If I wanted to add/replace disks in the future, what is the procedure to determine the correct disk order in the unRAID GUI? Thanks in advance Quote Link to comment
ProfessionalIdiot Posted November 29, 2023 Author Share Posted November 29, 2023 Wow - to make this more confusing, when I start the array in order 2 mentioned above (vdev order: raidz1-0, 1-1, mirror-2, cache), the GUI shows all the Zfs disks as unmountable, but everything seems to be working? root@nas01:~# zpool status pool: zfs state: ONLINE scan: scrub repaired 0B in 00:48:59 with 0 errors on Mon Nov 27 01:49:00 2023 config: NAME STATE READ WRITE CKSUM zfs ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdg1 ONLINE 0 0 0 sdh1 ONLINE 0 0 0 sde1 ONLINE 0 0 0 sdf1 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 sdc1 ONLINE 0 0 0 sdl1 ONLINE 0 0 0 sdm1 ONLINE 0 0 0 sdn1 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 sdk1 ONLINE 0 0 0 sdd1 ONLINE 0 0 0 sdb1 ONLINE 0 0 0 cache sdj1 ONLINE 0 0 0 sdi1 ONLINE 0 0 0 errors: No known data errors I also see I/O on the pool: root@nas01:~# zpool iostat -v 1 capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- zfs 10.9T 21.8T 62 2 486K 27.9K raidz1-0 10.9T 3.69T 61 0 479K 8.92K sdg1 - - 15 0 126K 2.28K sdh1 - - 15 0 114K 2.23K sde1 - - 15 0 128K 2.27K sdf1 - - 15 0 112K 2.15K raidz1-1 48.8G 14.5T 0 0 4.79K 9.37K sdc1 - - 0 0 1.17K 2.36K sdl1 - - 0 0 1.26K 2.32K sdm1 - - 0 0 1.16K 2.40K sdn1 - - 0 0 1.21K 2.28K mirror-2 1004K 3.62T 0 0 1.55K 9.62K sdk1 - - 0 0 540 3.21K sdd1 - - 0 0 510 3.21K sdb1 - - 0 0 540 3.21K cache - - - - - - sdj1 23.9G 1.80T 35 0 146K 483 sdi1 23.8G 1.80T 33 0 142K 414 ---------- ----- ----- ----- ----- ----- ----- And I can access my shares via SMB. So do I just live with the GUI being inaccurate for now? I can monitor the shares fine otherwise (external monitoring, or from SSH): root@nas01:/var/log# df -h Filesystem Size Used Avail Use% Mounted on rootfs 32G 243M 32G 1% / tmpfs 32M 484K 32M 2% /run /dev/sda1 239G 448M 239G 1% /boot overlay 32G 243M 32G 1% /lib overlay 32G 243M 32G 1% /usr devtmpfs 8.0M 0 8.0M 0% /dev tmpfs 32G 0 32G 0% /dev/shm tmpfs 128M 2.6M 126M 2% /var/log tmpfs 1.0M 0 1.0M 0% /mnt/disks tmpfs 1.0M 0 1.0M 0% /mnt/remotes tmpfs 1.0M 0 1.0M 0% /mnt/addons tmpfs 1.0M 0 1.0M 0% /mnt/rootshare tmpfs 6.3G 0 6.3G 0% /run/user/0 zfs 17T 256K 17T 1% /mnt/zfs zfs/system 17T 499M 17T 1% /mnt/zfs/system zfs/isos 17T 256K 17T 1% /mnt/zfs/isos zfs/domains 17T 256K 17T 1% /mnt/zfs/domains zfs/x 25T 8.0T 17T 33% /mnt/zfs/x zfs/appdata 17T 10M 17T 1% /mnt/zfs/appdata /dev/md1p1 59G 453M 59G 1% /mnt/disk1 shfs 59G 453M 59G 1% /mnt/user0 shfs 59G 453M 59G 1% /mnt/user /dev/loop2 20G 589M 19G 3% /var/lib/docker /dev/loop3 1.0G 4.1M 905M 1% /etc/libvirt Docker seems to be happy as well, which I moved to this ZFS pool Very odd behavior Quote Link to comment
Solution JorgeB Posted November 29, 2023 Solution Share Posted November 29, 2023 49 minutes ago, ProfessionalIdiot said: I then tried adding 3 new devices to my pool via CLI: root@nas01:/var/log# zpool add zfs -f mirror sdk1 sdd1 sdb1 Unraid doesn't support mirror together with raidz in the same pool, also that's not recommend anyway, you should have all mirrors or all raidz of the same type and with, and that's all Unraid supports. Manual pool import will still work, but to get Unraid to import that pool you will need to backup, destroy and recreate it. Quote Link to comment
ProfessionalIdiot Posted November 29, 2023 Author Share Posted November 29, 2023 3 minutes ago, JorgeB said: Unraid doesn't support mirror together with raidz in the same pool, also that's not recommend anyway, you should have all mirrors or all raidz of the same type and with, and that's all Unraid supports. Manual pool import will still work, but to get Unraid to import that pool you will need to backup, destroy and recreate it. Got it - thank you. So for my understanding, when I go to fix this, is a valid config to create a raidz1 or z2 pool, with cache, and with spares added (via the instructions in your other post)? Adding spares would have been my next step if this all had worked as I (errantly) expected. Quote Link to comment
JorgeB Posted November 29, 2023 Share Posted November 29, 2023 48 minutes ago, ProfessionalIdiot said: when I go to fix this, is a valid config to create a raidz1 or z2 pool, with cache, and with spares added (via the instructions in your other post)? Yes, that's not a problem, the root (main data) vdevs must be all the same type and width, if you have a 4 device raidz2 vdev you can only add more 4 device raidz2 vdevs 1 Quote Link to comment
ProfessionalIdiot Posted November 29, 2023 Author Share Posted November 29, 2023 Great, thanks a bunch! 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.