March 6, 20224 yr I've just updated my server with 3 Toshiba MG09ACA18TE 18Tb drives. It's all working well from what I can see. Rebuilding parity. But I saw this error message on the console XFS (MD5): EXPERIMENTAL ONLINE SHRINK FEATURE IN USE. USE AT YOUR OWN RISK! Any idea what that is or why it's appeared? How much risk am I taking? Diagnostics attached tdm-diagnostics-20220306-2033.zip
March 6, 20224 yr Community Expert This warning started to appear to some users lately, looks like it happens mostly with large filesystems, curiously disk2 is the same size and looks to be using all the same options and is not showing the warning, you might want to ask in the XFS mailing list to see why the warning is there and if it's a concern.
March 6, 20224 yr Author 9 hours ago, JorgeB said: you might want to ask in the XFS mailing list to see why the warning is there and if it's a concern. Not sure how to do that, nor would I be able to help them if their questions got beyond uploading diagnostics. Googling around, this error message seems to be happening on unraid more than any other platform.
March 7, 20224 yr Community Expert 11 hours ago, dalben said: Not sure how to do that https://xfs.org/index.php/XFS_email_list_and_archives 11 hours ago, dalben said: nor would I be able to help them if their questions got beyond uploading diagnostics. Posting the kernel version and relevant syslog snippet should be enough to start, can also mention that it appears on disk1 but not in the similar disk2, they will ask for more info if needed. Linux version 5.14.15-Unraid Mar 6 19:59:21 tdm emhttpd: shcmd (81): mkdir -p /mnt/disk1 Mar 6 19:59:21 tdm emhttpd: shcmd (82): mount -t xfs -o noatime /dev/md1 /mnt/disk1 Mar 6 19:59:21 tdm kernel: SGI XFS with ACLs, security attributes, no debug enabled Mar 6 19:59:21 tdm kernel: XFS (md1): Mounting V5 Filesystem Mar 6 19:59:21 tdm kernel: XFS (md1): Ending clean mount Mar 6 19:59:21 tdm emhttpd: shcmd (83): xfs_growfs /mnt/disk1 Mar 6 19:59:21 tdm kernel: xfs filesystem being mounted at /mnt/disk1 supports timestamps until 2038 (0x7fffffff) Mar 6 19:59:21 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device Mar 6 19:59:21 tdm root: meta-data=/dev/md1 isize=512 agcount=32, agsize=137330687 blks Mar 6 19:59:21 tdm root: = sectsz=512 attr=2, projid32bit=1 Mar 6 19:59:21 tdm root: = crc=1 finobt=1, sparse=1, rmapbt=0 Mar 6 19:59:21 tdm root: = reflink=1 bigtime=0 inobtcount=0 Mar 6 19:59:21 tdm root: data = bsize=4096 blocks=4394581984, imaxpct=5 Mar 6 19:59:21 tdm root: = sunit=1 swidth=32 blks Mar 6 19:59:21 tdm root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Mar 6 19:59:21 tdm root: log =internal log bsize=4096 blocks=521728, version=2 Mar 6 19:59:21 tdm root: = sectsz=512 sunit=1 blks, lazy-count=1 Mar 6 19:59:21 tdm root: realtime =none extsz=4096 blocks=0, rtextents=0 Mar 6 19:59:21 tdm emhttpd: shcmd (83): exit status: 1 Mar 6 19:59:21 tdm emhttpd: shcmd (84): mkdir -p /mnt/disk2 Mar 6 19:59:21 tdm kernel: XFS (md1): EXPERIMENTAL online shrink feature in use. Use at your own risk! Mar 6 19:59:21 tdm emhttpd: shcmd (85): mount -t xfs -o noatime /dev/md2 /mnt/disk2 Mar 6 19:59:21 tdm kernel: XFS (md2): Mounting V5 Filesystem Mar 6 19:59:22 tdm kernel: XFS (md2): Ending clean mount Mar 6 19:59:22 tdm kernel: xfs filesystem being mounted at /mnt/disk2 supports timestamps until 2038 (0x7fffffff) Mar 6 19:59:22 tdm emhttpd: shcmd (86): xfs_growfs /mnt/disk2 Mar 6 19:59:22 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device Mar 6 19:59:22 tdm root: meta-data=/dev/md2 isize=512 agcount=32, agsize=137330687 blks Mar 6 19:59:22 tdm root: = sectsz=512 attr=2, projid32bit=1 Mar 6 19:59:22 tdm root: = crc=1 finobt=1, sparse=1, rmapbt=0 Mar 6 19:59:22 tdm root: = reflink=1 bigtime=0 inobtcount=0 Mar 6 19:59:22 tdm root: data = bsize=4096 blocks=4394581984, imaxpct=5 Mar 6 19:59:22 tdm root: = sunit=1 swidth=32 blks Mar 6 19:59:22 tdm root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Mar 6 19:59:22 tdm root: log =internal log bsize=4096 blocks=521728, version=2 Mar 6 19:59:22 tdm root: = sectsz=512 sunit=1 blks, lazy-count=1 Mar 6 19:59:22 tdm root: realtime =none extsz=4096 blocks=0, rtextents=0 Mar 6 19:59:22 tdm emhttpd: shcmd (86): exit status: 1
March 7, 20224 yr Could this have something to do with this issue? Mar 6 19:59:21 tdm emhttpd: shcmd (83): xfs_growfs /mnt/disk1 Mar 6 19:59:21 tdm kernel: xfs filesystem being mounted at /mnt/disk1 supports timestamps until 2038 (0x7fffffff) Mar 6 19:59:21 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device Mar 6 19:59:21 tdm root: meta-data=/dev/md1 isize=512 agcount=32, agsize=137330687 blks Mar 6 19:59:21 tdm root: = sectsz=512 attr=2, projid32bit=1 Mar 6 19:59:21 tdm root: = crc=1 finobt=1, sparse=1, rmapbt=0 Mar 6 19:59:21 tdm root: = reflink=1 bigtime=0 inobtcount=0 Mar 6 19:59:21 tdm root: data = bsize=4096 blocks=4394581984, imaxpct=5 Mar 6 19:59:21 tdm root: = sunit=1 swidth=32 blks Mar 6 19:59:21 tdm root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Mar 6 19:59:21 tdm root: log =internal log bsize=4096 blocks=521728, version=2 Mar 6 19:59:21 tdm root: = sectsz=512 sunit=1 blks, lazy-count=1 Mar 6 19:59:21 tdm root: realtime =none extsz=4096 blocks=0, rtextents=0 Mar 6 19:59:21 tdm emhttpd: shcmd (83): exit status: 1 Mar 6 19:59:21 tdm emhttpd: shcmd (84): mkdir -p /mnt/disk2 Mar 6 19:59:21 tdm kernel: XFS (md1): EXPERIMENTAL online shrink feature in use. Use at your own risk! It seems when the xfs_growfs doesn't have enough space on the disk, the shrink feature kicks in.
March 7, 20224 yr Author This is what I got back from Gao Xiang. All over my head. What can I tell him? @JorgeB? May I ask what is xfsprogs version used now? At the first glance, it seems that some old xfsprogs is used here, otherwise, it will show "[EXPERIMENTAL] try to shrink unused space" message together with the kernel message as well. I'm not sure what's sb_dblocks recorded in on-disk super block compared with new disk sizes. I guess the problem may be that the one new disk is larger than sb_dblocks and the other is smaller than sb_dblocks. But if some old xfsprogs is used, I'm still confused why old version xfsprogs didn't block it at the userspace in advance. Edited March 7, 20224 yr by dalben
March 7, 20224 yr Community Expert 1 hour ago, dlandon said: It seems when the xfs_growfs doesn't have enough space on the disk, the shrink feature kicks in. That is done by Unraid at every mount, if you look for example at disk2 same is done but there's no warning.
March 7, 20224 yr Community Expert 55 minutes ago, dalben said: May I ask what is xfsprogs version used now? You can get that with: xfs_repair -V 58 minutes ago, dalben said: I'm not sure what's sb_dblocks recorded in on-disk super block compared with new disk sizes. I don't know how to get that info, if it's not included in the snippet above you can ask him how to get it, you can also mention that the filesystem was expanded, possibly more than once, and if you remember the original size (when the filesystem was first created) also mention that. In any case I don't believe this is an issue, even if that experimental feature is enable Unraid never shrinks a filesystem, it only grows it.
March 7, 20224 yr I've had this error. I have multiple 16TB drives filled mostly up (100-200GB left). No idea why is there but I don't like the sounds of it either.
March 7, 20224 yr Community Expert This is normal: Mar 6 19:59:21 tdm emhttpd: shcmd (83): xfs_growfs /mnt/disk1 This is not normal: root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device Unraid tries to grow the filesystem at every mount, but usually there's no error when it can't be grown, and that's every time except after upgrading a disk. Even so the error doesn't always result in the warning, as seen for disks 1 and 2 above, and since Unraid never shrinks a filesystem I can't see how this would be an actual problem, but obviously if this can be done without the error (and warning) it would be better, for people seeing this warning, does it also happen with v6.9 or only v6.10-rc?
March 7, 20224 yr Author 5 hours ago, Ystebad said: I've had this error. I have multiple 16TB drives filled mostly up (100-200GB left). No idea why is there but I don't like the sounds of it either. I have 2 x 18tb with only ~6tb filled each. My sequence of events was I added the two 18tb disks. Unraid detected them and asked if I wanted them formatted, I said yes, it went and formatted them and away we went. Do you remember if you had the same sequence @Ystebad ? Edited March 7, 20224 yr by dalben
March 7, 20224 yr Author 4 hours ago, JorgeB said: Even so the error doesn't always result in the warning, as seen for disks 1 and 2 above, and since Unraid never shrinks a filesystem I can't see how this would be an actual problem, but obviously if this can be done without the error (and warning) it would be better, for people seeing this warning, does it also happen with v6.9 or only v6.10-rc? I can't help there, I have v6.10-rc and upgraded the disks this weekend. I don't fancy downgrading to test.
March 7, 20224 yr Author I just rebooted the server to see if the warning remains, and it did. Ultimately I can live with a warning message if there's confirmation that I am not really at risk. At the moment the xfs mailing list are unable to understand why it's happening to one and not the other. Uncertainty makes me uncertain... Mar 8 06:59:48 tdm emhttpd: shcmd (149): mount -t xfs -o noatime /dev/md1 /mnt/disk1 Mar 8 06:59:49 tdm kernel: SGI XFS with ACLs, security attributes, no debug enabled Mar 8 06:59:49 tdm kernel: XFS (md1): Mounting V5 Filesystem Mar 8 06:59:49 tdm kernel: XFS (md1): Ending clean mount Mar 8 06:59:49 tdm emhttpd: shcmd (150): xfs_growfs /mnt/disk1 Mar 8 06:59:49 tdm kernel: xfs filesystem being mounted at /mnt/disk1 supports timestamps until 2038 (0x7fffffff) Mar 8 06:59:49 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device Mar 8 06:59:49 tdm root: meta-data=/dev/md1 isize=512 agcount=32, agsize=137330687 blks Mar 8 06:59:49 tdm root: = sectsz=512 attr=2, projid32bit=1 Mar 8 06:59:49 tdm root: = crc=1 finobt=1, sparse=1, rmapbt=0 Mar 8 06:59:49 tdm root: = reflink=1 bigtime=0 inobtcount=0 Mar 8 06:59:49 tdm root: data = bsize=4096 blocks=4394581984, imaxpct=5 Mar 8 06:59:49 tdm root: = sunit=1 swidth=32 blks Mar 8 06:59:49 tdm root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Mar 8 06:59:49 tdm root: log =internal log bsize=4096 blocks=521728, version=2 Mar 8 06:59:49 tdm root: = sectsz=512 sunit=1 blks, lazy-count=1 Mar 8 06:59:49 tdm root: realtime =none extsz=4096 blocks=0, rtextents=0 Mar 8 06:59:49 tdm emhttpd: shcmd (150): exit status: 1 Mar 8 06:59:49 tdm kernel: XFS (md1): EXPERIMENTAL online shrink feature in use. Use at your own risk! Mar 8 06:59:49 tdm emhttpd: shcmd (151): mkdir -p /mnt/disk2 Mar 8 06:59:49 tdm emhttpd: shcmd (152): mount -t xfs -o noatime /dev/md2 /mnt/disk2 Mar 8 06:59:49 tdm kernel: XFS (md2): Mounting V5 Filesystem Mar 8 06:59:49 tdm kernel: XFS (md2): Ending clean mount Mar 8 06:59:49 tdm kernel: xfs filesystem being mounted at /mnt/disk2 supports timestamps until 2038 (0x7fffffff) Mar 8 06:59:49 tdm emhttpd: shcmd (153): xfs_growfs /mnt/disk2 Mar 8 06:59:49 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device Mar 8 06:59:49 tdm root: meta-data=/dev/md2 isize=512 agcount=32, agsize=137330687 blks Mar 8 06:59:49 tdm root: = sectsz=512 attr=2, projid32bit=1 Mar 8 06:59:49 tdm root: = crc=1 finobt=1, sparse=1, rmapbt=0 Mar 8 06:59:49 tdm root: = reflink=1 bigtime=0 inobtcount=0 Mar 8 06:59:49 tdm root: data = bsize=4096 blocks=4394581984, imaxpct=5 Mar 8 06:59:49 tdm root: = sunit=1 swidth=32 blks Mar 8 06:59:49 tdm root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Mar 8 06:59:49 tdm root: log =internal log bsize=4096 blocks=521728, version=2 Mar 8 06:59:49 tdm root: = sectsz=512 sunit=1 blks, lazy-count=1 Mar 8 06:59:49 tdm root: realtime =none extsz=4096 blocks=0, rtextents=0 Mar 8 06:59:49 tdm emhttpd: shcmd (153): exit status: 1 Mar 8 06:59:49 tdm emhttpd: shcmd (154): mkdir -p /mnt/disk3 Mar 8 06:59:49 tdm emhttpd: shcmd (155): mount -t xfs -o noatime /dev/md3 /mnt/disk3
March 8, 20224 yr Community Expert Since the warning appears to come from xfs_growfs, and that's part of xfsprogrs, it might go way once that's updated in a newer Unraid release.
March 8, 20224 yr Unraid OS never tries to shrink the size of a file system (well there is an exception). What we do upon every mount is to try to expand the size of each file system. For the devices in the unRAID array this handles the case where a smaller device has been replaced with a larger one. If the file system cannot be expanded because the device is the same, or a new device is the same size, then the operation will of course fail - and this is reported in the system log - entirely expected. Another case were we need to explicitly expand a file system is following replacement of a smaller device with a larger one in a btrfs pool. In this case a 'btrfs filesystem resize' is executed Now that I think of it, actually code does not prevent you from replacing a larger device with a smaller one in a btrfs pool, so technically that would be a case where a file system is shrunk.
March 8, 20224 yr Community Expert 59 minutes ago, limetech said: If the file system cannot be expanded because the device is the same, or a new device is the same size, then the operation will of course fail - and this is reported in the system log - entirely expected. Not most times, that xfs_growfs error only happens sometimes, and the warning appears to happen just in some of those times, e.g. from the diags above: disk1 has the error and the warning Mar 6 19:59:21 tdm emhttpd: shcmd (83): xfs_growfs /mnt/disk1 Mar 6 19:59:21 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device Mar 6 19:59:21 tdm kernel: XFS (md1): EXPERIMENTAL online shrink feature in use. Use at your own risk! disk2 has the error only Mar 6 19:59:22 tdm emhttpd: shcmd (86): xfs_growfs /mnt/disk2 Mar 6 19:59:22 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device disk3 no error or warning Mar 6 19:59:22 tdm emhttpd: shcmd (89): xfs_growfs /mnt/disk3 I do agree it's not an actual problem since the filesystem is never shrunk in the array, maybe a xfsprogs issue?
March 8, 20224 yr Community Expert After looking at a few more diags with this issue I think it's like this: With v6.10-rc2 any 1412TB or larger disk formatted with XFS gives the xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device the EXPERIMENTAL online shrink feature in use. Use at your own risk! appears for the first of these only, i.e., if you have a 10TB disk1, 12TB disk2, 14TB disk3 and 8TB disk4 there won't be any error for disk1, disk2 will have the error and the warning, disk3 only show the error and disk4 again no error or warning.
March 8, 20224 yr I looked at xfsprogs version: 5.13.0 source code and I do not see that "online shrink feature in use" error message text. If you type 'xfs_growfs -V' what version does it show?
March 8, 20224 yr Community Expert For rc2 it's 5.13.00, maybe despite xfs_growfs being part of xfsprogs the warning after the error comes from the kernel? Just a wild guess...
March 8, 20224 yr Community Expert In my mind the main question is why did the "xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device" started to appear with 1412TB or larger drives, I would bet that if that error went way so would the experimental warning.
March 8, 20224 yr I am on rc3 test version with a newer kernel, and upgraded to new 18 TB disks. Don’t have this message on any of them (12 disks in total).
March 8, 20224 yr Author Just a note on the warning/error. I probably wouldn't have noticed it if it was hidden in syslog but this error pops up on the console making it difficult to ignore.
March 8, 20224 yr Author 3 hours ago, limetech said: I looked at xfsprogs version: 5.13.0 source code and I do not see that "online shrink feature in use" error message text. If you type 'xfs_growfs -V' what version does it show? As @JorgeB confirmed, it's 5.13.0 root@tdm:~# xfs_growfs -V xfs_growfs version 5.13.0
March 9, 20224 yr Community Expert 13 hours ago, bonienl said: I am on rc3 test version with a newer kernel, and upgraded to new 18 TB disks. Don’t have this message on any of them (12 disks in total). That's good news, I assume you mean you don't get the xfs_growfs error and/or the experimental warning, but did you get it with rc2? Still not sure if the error always displays with any 1412TB or larger disk and rc2, but from the diags I've seen so far looks like it does.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.