November 19, 20178 yr This is not a rc issue, this server was on 6.3.5 when the problem started, I updated to latest rc but there was no change. I was doing some disk re-arrangements on 2 of my servers and I'm having a problem mounting 2 disks, both came from another server: Quote Filesystem Size Used Avail Use% Mounted on rootfs 3.9G 412M 3.5G 11% / tmpfs 32M 292K 32M 1% /run devtmpfs 3.9G 0 3.9G 0% /dev tmpfs 4.0G 0 4.0G 0% /dev/shm cgroup_root 8.0M 0 8.0M 0% /sys/fs/cgroup tmpfs 128M 1.8M 127M 2% /var/log /dev/sda1 7.4G 351M 7.0G 5% /boot /dev/loop0 6.8M 6.8M 0 100% /lib/modules /dev/loop1 4.4M 4.4M 0 100% /lib/firmware /dev/md1 2.8T 513G 2.3T 19% /mnt/disk1 /dev/md2 2.8T 17M 2.8T 1% /mnt/disk2 /dev/md3 2.8T 17M 2.8T 1% /mnt/disk3 /dev/md17 2.8T 2.7T 53G 99% /mnt/disk4 /dev/md16 2.8T 2.7T 55G 99% /mnt/disk5 /dev/md6 1.9T 1.8T 63G 97% /mnt/disk6 /dev/md7 1.9T 1.8T 57G 97% /mnt/disk7 /dev/md8 1.9T 1.8T 59G 97% /mnt/disk8 /dev/md9 1.9T 909G 954G 49% /mnt/disk9 /dev/md10 1.9T 1.8T 94G 96% /mnt/disk10 /dev/md11 2.8T 744G 2.1T 27% /mnt/disk11 /dev/md14 2.8T 2.2T 547G 81% /mnt/disk14 /dev/md15 2.8T 2.7T 30G 99% /mnt/disk15 /dev/md18 2.8T 2.4T 348G 88% /mnt/disk18 /dev/md19 2.8T 17M 2.8T 1% /mnt/disk19 /dev/md20 2.8T 2.6T 171G 94% /mnt/disk20 shfs 45T 30T 15T 67% /mnt/user As you can see md16 and md17 are mounting disks 5 and 4 respectively, so I can't access the data on disks 16 and 17. If I unassign both disks all looks correct. If I assign one the the problem disks to the cache slot or use the UD plugin it still mounts wrong, e.g.: Quote /dev/sdk1 2.8T 2.7T 55G 99% /mnt/disk5 PS: disk slots 12 and 13 are currently empty and unassigned. This happened before here, so it's not an isolated incident, might be worth investigating as it can cause confusion leading to data loss, I can see weird stuff on the syslog, but not the root cause: Quote Nov 19 13:43:59 Tower8 emhttpd: shcmd (317): mkdir -p /mnt/disk16 Nov 19 13:43:59 Tower8 emhttpd: shcmd (318): mount -t btrfs,xfs,reiserfs -o noatime,nodiratime /dev/md16 /mnt/disk16 Nov 19 13:43:59 Tower8 emhttpd: shcmd (319): btrfs filesystem resize max /mnt/disk16 Nov 19 13:43:59 Tower8 root: Resize '/mnt/disk16' of 'max' Nov 19 13:43:59 Tower8 kernel: BTRFS info (device md5): new size for /dev/md16 is 3000592928768 Nov 19 13:43:59 Tower8 emhttpd: shcmd (320): mkdir -p /mnt/disk17 Nov 19 13:43:59 Tower8 emhttpd: shcmd (321): mount -t btrfs,xfs,reiserfs -o noatime,nodiratime /dev/md17 /mnt/disk17 Nov 19 13:43:59 Tower8 emhttpd: shcmd (322): btrfs filesystem resize max /mnt/disk17 Nov 19 13:43:59 Tower8 kernel: BTRFS info (device md4): new size for /dev/md17 is 3000592928768 Nov 19 13:43:59 Tower8 root: Resize '/mnt/disk17' of 'max' Diags attached. tower8-diagnostics-20171119-1344.zip
November 19, 20178 yr This is very odd. Do you see directories /mnt/disk16 and /mnt/disk17? If you 'umount /mnt/disk4' and then type another 'df' does /mnt/disk4 now show as mounted on /dev/md4? (wondering double mount). Does 'df' and 'cat /proc/mounts' agree?
November 19, 20178 yr Author 27 minutes ago, limetech said: Do you see directories /mnt/disk16 and /mnt/disk17? Yes, but they show the contents of the disk4 and disk5 28 minutes ago, limetech said: If you 'umount /mnt/disk4' and then type another 'df' does /mnt/disk4 now show as mounted on /dev/md4? (wondering double mount). No, and if I remount the disk on the console it then shows correctly on df e.g.: /dev/md16 mounted on /mnt/disk16 but if I browse /mnt/disk16 it still shows the contents of /dev/md5 40 minutes ago, limetech said: Does 'df' and 'cat /proc/mounts' agree? Yes. /dev/md16 /mnt/disk5 btrfs rw,noatime,nodiratime,space_cache,subvolid=5,subvol=/ 0 0
November 19, 20178 yr 6 hours ago, johnnie.black said: I was doing some disk re-arrangements on 2 of my servers and I'm having a problem mounting 2 disks, both came from another server: In the server those devices came from, were they assigned as disk4 and disk5? Please post output of 'btrfs fi show' command.
November 21, 20178 yr Author In the server those devices came from, were they assigned as disk4 and disk5? Please post output of 'btrfs fi show' command. Sorry, somehow missed this reply. Was just going to post that found the problem, duplicated UUIDs, most likely these disks were once before rebuilt on one server then used on another, and now I was trying to use them together which is not possible with a duplicate UUID, just a shame that unlike e.g. xfs that gives a duplicate UUID mount error, btrfs error is no help at all, anyway, problem solved and sorry for wasting your time since it's got nothing to do with unRAID.
November 21, 20178 yr 1 hour ago, johnnie.black said: Sorry, somehow missed this reply. Was just going to post that found the problem, duplicated UUIDs, most likely these disks were once before rebuilt on one server then used on another, and now I was trying to use them together which is not possible with a duplicate UUID, just a shame that unlike e.g. xfs that gives a duplicate UUID mount error, btrfs error is no help at all, anyway, problem solved and sorry for wasting your time since it's got nothing to do with unRAID. I wonder if you could elaborate a bit? I think what you're saying is that disk5 and disk16 were at one time part of the same "pool", as were disk4 and disk16 (different pool). But now they are assigned and being used as individual devices, yet somehow during mount sequence btrfs still thinks they are joined together? What I saw in the system log was this: Nov 19 13:43:59 Tower8 emhttpd: shcmd (317): mkdir -p /mnt/disk16 Nov 19 13:43:59 Tower8 emhttpd: shcmd (318): mount -t btrfs,xfs,reiserfs -o noatime,nodiratime /dev/md16 /mnt/disk16 Nov 19 13:43:59 Tower8 emhttpd: shcmd (319): btrfs filesystem resize max /mnt/disk16 Nov 19 13:43:59 Tower8 root: Resize '/mnt/disk16' of 'max' Nov 19 13:43:59 Tower8 kernel: BTRFS info (device md5): new size for /dev/md16 is 3000592928768 Nov 19 13:43:59 Tower8 emhttpd: shcmd (320): mkdir -p /mnt/disk17 Nov 19 13:43:59 Tower8 emhttpd: shcmd (321): mount -t btrfs,xfs,reiserfs -o noatime,nodiratime /dev/md17 /mnt/disk17 Nov 19 13:43:59 Tower8 emhttpd: shcmd (322): btrfs filesystem resize max /mnt/disk17 Nov 19 13:43:59 Tower8 kernel: BTRFS info (device md4): new size for /dev/md17 is 3000592928768 Nov 19 13:43:59 Tower8 root: Resize '/mnt/disk17' of 'max' Notice the BTRFS info message identifies a device different than what is mounted at the mount point. When unRAID mounts a device, immediately following successful mount, it tries to grow the file system. This is to handle the case where someone has replaced a smaller device with a larger one. Most of the time of course, this is not the case but the attempt to grow the file system is benign. What I was going to do was create a special test release that does not execute the 'btrfs filesystem resize max' command, thinking this problem might be related to btrfs resizing code which historically has had some "quirks" in the past.
November 21, 20178 yr Author 1 hour ago, limetech said: I wonder if you could elaborate a bit? I think what you're saying is that disk5 and disk16 were at one time part of the same "pool", as were disk4 and disk16 (different pool). But now they are assigned and being used as individual devices, yet somehow during mount sequence btrfs still thinks they are joined together? No, they were always single array disks, what I think must have happened is this: -Disk 4 and 5 were once before used on another server as data disks, lets call it server A. -At some point in the past I replaced (rebuilt) those disks with new ones still on server A -The original disks then went to be used on this server, lets call it server B, I must have done a new config and kept the original filesystem, possibly just deleted or moved the files, I usually do that when it's a large change and I need to transfer data back and forth and only sync parity in the end. -Now this week, I again replaced those same disks on server A with larger ones this time, yeah I know I'm always tinkering with my servers , and then moved the originals to server B (now disks 16 and 17), again doing a new config and keeping the old filesystem, it's the only way I think that 2 pairs of disks could have the same UUID, chances of that being a coincidence I think are impossible. I still have this on the running SSH session, sdk is disk16, note the duplicate UUDI: root@Tower8:~# btrfs filesystem show /dev/md5 Label: none uuid: 9d7b3393-74c5-4988-baad-9f12a848fd6d Total devices 1 FS bytes used 2.67TiB devid 1 size 2.73TiB used 2.72TiB path /dev/md5 root@Tower8:~# btrfs filesystem show /dev/sdk1 Label: none uuid: 9d7b3393-74c5-4988-baad-9f12a848fd6d Total devices 1 FS bytes used 512.00KiB devid 1 size 2.73TiB used 1.78TiB path /dev/md5 Edited November 21, 20178 yr by johnnie.black
November 21, 20178 yr 48 minutes ago, johnnie.black said: -Disk 4 and 5 were once before used on another server as data disks, lets call it server A. -At some point in the past I replaced (rebuilt) those disks with new ones still on server A -The original disks then went to be used on this server, lets call it server B, I must have done a new config and kept the original filesystem, possibly just deleted or moved the files, I usually do that when it's a large change and I need to transfer data back and forth and only sync parity in the end. Ok that explains it. You ended up with two physical devices that were duplicates and thus had same UUID's. Actually I remember now this issue because I've run across it during testing, and honestly, I must have neglected to add it to my bug list because I don't see it there... This issue is mentioned here: https://unix.stackexchange.com/questions/246976/btrfs-subvolume-uuid-clash and https://btrfs.wiki.kernel.org/index.php/Gotchas#Block-level_copies_of_devices Personally I think this is a design flaw in btrfs. If I recall, that 'btrfstune -u' command can take a loooong time because the UUID is part of the metadata of every btrfs extent on the volume. Sadly, this is not an easy problem to solve. My solution was gong to be: detect mounting a volume that is being rebuilt, and then initiate that btrfstune operation first, and when finished, execute the mount. However this problem still exists: if you configure an array with one or two parity disks and only one data disk, it turns out all two (or three) disks are duplicates of each other (because of how the parity math works). That is, the array is a double or triple mirror. In this case you can't avoid having multiple physical disks with duplicate btrfs UUID's. Nasty problem, right?
November 21, 20178 yr Author 1 minute ago, limetech said: Personally I think this is a design flaw in btrfs. Agree, I would be happy with a warning on the syslog like with xfs : XFS (sdg1): Filesystem has duplicate UUID 822e2574-c2ee-4fcf-a00b-38a4acbc4865 - can't mount But obviously this would need to be done by the btrfs maintainers, not LT.
November 21, 20178 yr Does XFS generate that message if there are multiple devices with same FS uuid? If so, does it successfully mount the first one?
November 21, 20178 yr Author 1 minute ago, limetech said: Does XFS generate that message if there are multiple devices with same FS uuid? Yes. 1 minute ago, limetech said: If so, does it successfully mount the first one? I believe so but not sure, never tried with duplicate array disks, but you get that e.g. if you try to mount a duplicate with the UD plugin, happened before to users that rebuilt a disk and then want to copy or compare the data with the old one.
November 21, 20178 yr Just now, johnnie.black said: Yes. I believe so but not sure, never tried with duplicate array disks, but you get that e.g. if you try to mount a duplicate with the UD plugin, happened before to users that rebuilt a disk and then want to copy or compare the data with the old one. Ok there's our easy fix! unRAID can easily detect if there are duplicate UUID's among the btrfs devices. Let's run some experiments and see what XFS does. If there are 2 or more duplicate XFS volumes, does it let you mount the first one, but not subsequent ones, or does it not let any of them mount? However XFS handles it, we can make btrfs appear to do the same. For devices that don't mount, they would have status 'Unmountable' in the webGui which would permit the user to reformat or remove the device.
November 21, 20178 yr Thinking about it, why would XFS care about this? btrfs cares because it needs to know the volumes that comprise a multi-volume pool, but XFS does not support multi-volume.... things that make you go "hmmm"
November 21, 20178 yr Author 13 minutes ago, limetech said: Thinking about it, why would XFS care about this? btrfs cares because it needs to know the volumes that comprise a multi-volume pool, but XFS does not support multi-volume.... things that make you go "hmmm" Not sure, but it does, I'm doing a quick test on my work server and in about half an hour can confirm what happens when you use duplicate xfs UUIDs in the array, I'm pretty sure the 1st disk mounts but want to be sure.
November 21, 20178 yr Author 1 hour ago, limetech said: If there are 2 or more duplicate XFS volumes, does it let you mount the first one, but not subsequent ones, or does it not let any of them mount? Yep, it's like I thought, disk2 and 4 have the same UUID: Nov 21 18:16:50 Tower9 kernel: XFS (md2): Mounting V5 Filesystem Nov 21 18:16:51 Tower9 kernel: XFS (md2): Ending clean mount Nov 21 18:16:51 Tower9 emhttpd: shcmd (381): xfs_growfs /mnt/disk2 Nov 21 18:16:51 Tower9 root: meta-data=/dev/md2 isize=512 agcount=4, agsize=15262410 blks Nov 21 18:16:51 Tower9 root: = sectsz=512 attr=2, projid32bit=1 Nov 21 18:16:51 Tower9 root: = crc=1 finobt=1 spinodes=0 rmapbt=0 Nov 21 18:16:51 Tower9 root: = reflink=0 Nov 21 18:16:51 Tower9 root: data = bsize=4096 blocks=61049638, imaxpct=25 Nov 21 18:16:51 Tower9 root: = sunit=0 swidth=0 blks Nov 21 18:16:51 Tower9 root: naming =version 2 bsize=4096 ascii-ci=0 ftype=1 Nov 21 18:16:51 Tower9 root: log =internal bsize=4096 blocks=29809, version=2 Nov 21 18:16:51 Tower9 root: = sectsz=512 sunit=0 blks, lazy-count=1 Nov 21 18:16:51 Tower9 root: realtime =none extsz=4096 blocks=0, rtextents=0 ... ... Nov 21 18:16:53 Tower9 emhttpd: shcmd (385): mkdir -p /mnt/disk4 Nov 21 18:16:53 Tower9 emhttpd: shcmd (386): mount -t btrfs,xfs,reiserfs -o noatime,nodiratime /dev/md4 /mnt/disk4 Nov 21 18:16:53 Tower9 kernel: XFS (md4): Filesystem has duplicate UUID 902b7934-1750-45d4-aca1-9d850c52378b - can't mount Nov 21 18:16:53 Tower9 root: mount: /mnt/disk4: wrong fs type, bad option, bad superblock on /dev/md4, missing codepage or helper program, or other error. Nov 21 18:16:53 Tower9 emhttpd: shcmd (386): exit status: 32 Nov 21 18:16:53 Tower9 emhttpd: /mnt/disk4 mount error: No file system Nov 21 18:16:53 Tower9 emhttpd: shcmd (387): umount /mnt/disk4 Nov 21 18:16:53 Tower9 root: umount: /mnt/disk4: not mounted. Edited November 21, 20178 yr by johnnie.black
November 21, 20178 yr Author 1 hour ago, limetech said: Ok there's our easy fix! unRAID can easily detect if there are duplicate UUID's among the btrfs devices. Just one more thing, I don't know what you're going to use to compare the UUIDs but note that the duplicate filesystems don't show up on the initial btrfs scan, e.g.: Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid b7603644-cbea-4bef-b56a-8a88a272d534 devid 1 transid 3519 /dev/sdh1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 9d7b3393-74c5-4988-baad-9f12a848fd6d devid 1 transid 2923 /dev/sdg1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 21ed2ffb-a5a9-493c-bce3-592fc8fe0c54 devid 1 transid 966 /dev/sde1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 10162d6a-2259-4d1f-8349-8bf2d77b0c73 devid 1 transid 903 /dev/sds1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 12f896ba-ddb8-40c4-a606-7a9a2ea0ef62 devid 1 transid 5640 /dev/sdj1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 8ea0039e-8eb7-41db-8754-63db2bc2402c devid 1 transid 1176 /dev/sdq1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 74f51b72-4bdb-42c0-9621-c3aae3aba0aa devid 1 transid 1142 /dev/sdo1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 54f6605e-440f-44fb-b378-52c169e5bc96 devid 1 transid 732 /dev/sdr1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 40cfd27f-ce36-4e14-bd8c-8c68f7eabd8e devid 1 transid 6458 /dev/sdp1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 62cc453b-5bdb-4460-b14b-818347994bbd devid 1 transid 1054 /dev/sdn1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 7558a782-5392-46e9-80e2-9af369342887 devid 1 transid 362 /dev/sdd1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 23f56965-a58e-4a17-bcb6-c43221cc8ff8 devid 1 transid 821 /dev/sdc1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid bae8a481-f0ea-48a2-b8fb-5221c6f12718 devid 1 transid 7217 /dev/sdi1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 4bad0b13-d9c8-4599-9aaa-acefcb914b04 devid 1 transid 790 /dev/sdf1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 366a9fbe-ded6-4ae5-bc4e-7ab8df738a58 devid 1 transid 1463 /dev/sdb1 Nov 21 00:00:33 Tower8 kernel: BTRFS: device fsid 4382289a-2c31-4304-ba0f-5e90436b0d65 devid 1 transid 2982 /dev/sdm1 Both duplicate disks don't show up, disk16 is sdk and 17 is sdl, the only hint that something is wrong before the mount attempt is this: Nov 21 00:00:43 Tower8 emhttp: shcmd (52): /sbin/btrfs device scan |& logger Nov 21 00:00:43 Tower8 avahi-daemon[2791]: Server startup complete. Host name is Tower8.local. Local service cookie is 2960965990. Nov 21 00:00:44 Tower8 root: ERROR: device scan failed on '/dev/md17': File exists Nov 21 00:00:44 Tower8 root: ERROR: there are 1 errors while registering devices But it doesn't show up all the time (maybe only on v6.3.x), there was no similar error in the diagnostics I posted on this thread with v6.4, there was later with v6.3.5 and on the similar thread I linked in the OP.
November 21, 20178 yr How about the output of 'blkid' command? That's probably what I'll use to get the UUID's. Confirmed there is some funny business. Created 3-disk array: parity, parity2, disk1. Formatted disk1 as btrfs. Everything mounts and seems to work. Curiously the 'btrfs fi show' command only shows one of the devices as having a btrfs file system. Even after 'btrfs dev scan'. Now stop array, New Config, assign same three devices as disk1, disk2, disk3, and Start array. Now we have three btrfs file systems, all with identical data (well md5sum of a large file on them comes back good). Similar though, 'btrfs fi show' does not show all the volumes. 'blkid' shows all the volumes but with identical UUID's (as expected). System seems to work though just fine. But... all three mount points: /mnt/disk1, /mnt/disk2, /mnt/disk3 are referring to same device. Syslog completely clean. Well this is going to take some time to get right, maybe beyond scope of 6.4.0 release.
November 21, 20178 yr Author 10 minutes ago, limetech said: How about the output of 'blkid' command? That's probably what I'll use to get the UUID's. That's a good idea. 11 minutes ago, limetech said: Well this is going to take some time to get right, maybe beyond scope of 6.4.0 release. I wouldn't consider this an urgent fix, AFAIK it has only come up twice, still when it does it can be confusing and cause data loss, when it first happened to me, all looked normal on the main page, I was using the disk shares and disk16 was showing data I wasn't expecting to be there, I got confused and was just about to delete all data on it and if I had proceeded it would have deleted all data on disk5.
Archived
This topic is now archived and is closed to further replies.