XFS (MD5): EXPERIMENTAL ONLINE SHRINK FEATURE IN USE. USE AT YOUR OWN RISK!


Recommended Posts

This warning started to appear to some users lately, looks like it happens mostly with large filesystems, curiously disk2 is the same size and looks to be using all the same options and is not showing the warning, you might want to ask in the XFS mailing list to see why the warning is there and if it's a concern.

Link to comment
9 hours ago, JorgeB said:

 you might want to ask in the XFS mailing list to see why the warning is there and if it's a concern.

 

Not sure how to do that, nor would I be able to help them if their questions got beyond uploading diagnostics

 

Googling around, this error message seems to be happening on unraid more than any other platform. 

Link to comment
11 hours ago, dalben said:

Not sure how to do that

https://xfs.org/index.php/XFS_email_list_and_archives

 

11 hours ago, dalben said:

nor would I be able to help them if their questions got beyond uploading diagnostics

Posting the kernel version and relevant syslog snippet should be enough to start, can also mention that it appears on disk1 but not in the similar disk2, they will ask for more info if needed.

 

 

Linux version 5.14.15-Unraid

 

Mar  6 19:59:21 tdm emhttpd: shcmd (81): mkdir -p /mnt/disk1
Mar  6 19:59:21 tdm emhttpd: shcmd (82): mount -t xfs -o noatime /dev/md1 /mnt/disk1
Mar  6 19:59:21 tdm kernel: SGI XFS with ACLs, security attributes, no debug enabled
Mar  6 19:59:21 tdm kernel: XFS (md1): Mounting V5 Filesystem
Mar  6 19:59:21 tdm kernel: XFS (md1): Ending clean mount
Mar  6 19:59:21 tdm emhttpd: shcmd (83): xfs_growfs /mnt/disk1
Mar  6 19:59:21 tdm kernel: xfs filesystem being mounted at /mnt/disk1 supports timestamps until 2038 (0x7fffffff)
Mar  6 19:59:21 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device
Mar  6 19:59:21 tdm root: meta-data=/dev/md1               isize=512    agcount=32, agsize=137330687 blks
Mar  6 19:59:21 tdm root:          =                       sectsz=512   attr=2, projid32bit=1
Mar  6 19:59:21 tdm root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Mar  6 19:59:21 tdm root:          =                       reflink=1    bigtime=0 inobtcount=0
Mar  6 19:59:21 tdm root: data     =                       bsize=4096   blocks=4394581984, imaxpct=5
Mar  6 19:59:21 tdm root:          =                       sunit=1      swidth=32 blks
Mar  6 19:59:21 tdm root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Mar  6 19:59:21 tdm root: log      =internal log           bsize=4096   blocks=521728, version=2
Mar  6 19:59:21 tdm root:          =                       sectsz=512   sunit=1 blks, lazy-count=1
Mar  6 19:59:21 tdm root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Mar  6 19:59:21 tdm emhttpd: shcmd (83): exit status: 1
Mar  6 19:59:21 tdm emhttpd: shcmd (84): mkdir -p /mnt/disk2
Mar  6 19:59:21 tdm kernel: XFS (md1): EXPERIMENTAL online shrink feature in use. Use at your own risk!
Mar  6 19:59:21 tdm emhttpd: shcmd (85): mount -t xfs -o noatime /dev/md2 /mnt/disk2
Mar  6 19:59:21 tdm kernel: XFS (md2): Mounting V5 Filesystem
Mar  6 19:59:22 tdm kernel: XFS (md2): Ending clean mount
Mar  6 19:59:22 tdm kernel: xfs filesystem being mounted at /mnt/disk2 supports timestamps until 2038 (0x7fffffff)
Mar  6 19:59:22 tdm emhttpd: shcmd (86): xfs_growfs /mnt/disk2
Mar  6 19:59:22 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device
Mar  6 19:59:22 tdm root: meta-data=/dev/md2               isize=512    agcount=32, agsize=137330687 blks
Mar  6 19:59:22 tdm root:          =                       sectsz=512   attr=2, projid32bit=1
Mar  6 19:59:22 tdm root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Mar  6 19:59:22 tdm root:          =                       reflink=1    bigtime=0 inobtcount=0
Mar  6 19:59:22 tdm root: data     =                       bsize=4096   blocks=4394581984, imaxpct=5
Mar  6 19:59:22 tdm root:          =                       sunit=1      swidth=32 blks
Mar  6 19:59:22 tdm root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Mar  6 19:59:22 tdm root: log      =internal log           bsize=4096   blocks=521728, version=2
Mar  6 19:59:22 tdm root:          =                       sectsz=512   sunit=1 blks, lazy-count=1
Mar  6 19:59:22 tdm root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Mar  6 19:59:22 tdm emhttpd: shcmd (86): exit status: 1

 

 

  • Thanks 1
Link to comment

Could this have something to do with this issue?

Mar  6 19:59:21 tdm emhttpd: shcmd (83): xfs_growfs /mnt/disk1
Mar  6 19:59:21 tdm kernel: xfs filesystem being mounted at /mnt/disk1 supports timestamps until 2038 (0x7fffffff)
Mar  6 19:59:21 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device
Mar  6 19:59:21 tdm root: meta-data=/dev/md1               isize=512    agcount=32, agsize=137330687 blks
Mar  6 19:59:21 tdm root:          =                       sectsz=512   attr=2, projid32bit=1
Mar  6 19:59:21 tdm root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Mar  6 19:59:21 tdm root:          =                       reflink=1    bigtime=0 inobtcount=0
Mar  6 19:59:21 tdm root: data     =                       bsize=4096   blocks=4394581984, imaxpct=5
Mar  6 19:59:21 tdm root:          =                       sunit=1      swidth=32 blks
Mar  6 19:59:21 tdm root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Mar  6 19:59:21 tdm root: log      =internal log           bsize=4096   blocks=521728, version=2
Mar  6 19:59:21 tdm root:          =                       sectsz=512   sunit=1 blks, lazy-count=1
Mar  6 19:59:21 tdm root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Mar  6 19:59:21 tdm emhttpd: shcmd (83): exit status: 1
Mar  6 19:59:21 tdm emhttpd: shcmd (84): mkdir -p /mnt/disk2
Mar  6 19:59:21 tdm kernel: XFS (md1): EXPERIMENTAL online shrink feature in use. Use at your own risk!

 

It seems when the xfs_growfs doesn't have enough space on the disk, the shrink feature kicks in.

Link to comment

This is what I got back from Gao Xiang.  All over my head.  What can I tell him? @JorgeB?

 

May I ask what is xfsprogs version used now?

At the first glance, it seems that some old xfsprogs is used here,
otherwise, it will show "[EXPERIMENTAL] try to shrink unused space"
message together with the kernel message as well.

I'm not sure what's sb_dblocks recorded in on-disk super block
compared with new disk sizes.

I guess the problem may be that the one new disk is larger than
sb_dblocks and the other is smaller than sb_dblocks. But if some
old xfsprogs is used, I'm still confused why old version xfsprogs
didn't block it at the userspace in advance.

Edited by dalben
Link to comment
55 minutes ago, dalben said:

May I ask what is xfsprogs version used now?

You can get that with:

xfs_repair -V

 

 

58 minutes ago, dalben said:

I'm not sure what's sb_dblocks recorded in on-disk super block
compared with new disk sizes.

I don't know how to get that info, if it's not included in the snippet above you can ask him how to get it, you can also mention that the filesystem was expanded, possibly more than once, and if you remember the original size (when the filesystem was first created) also mention that.

 

In any case I don't believe this is an issue, even if that experimental feature is enable Unraid never shrinks a filesystem, it only grows it.

Link to comment

This is normal:

Mar 6 19:59:21 tdm emhttpd: shcmd (83): xfs_growfs /mnt/disk1

This is not normal:

root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device

Unraid tries to grow the filesystem at every mount, but usually there's no error when it can't be grown, and that's every time except after upgrading a disk.

 

Even so the error doesn't always result in the warning, as seen for disks 1 and 2 above, and since Unraid never shrinks a filesystem I can't see how  this would be an actual problem, but obviously if this can be done without the error (and warning) it would be better, for people seeing this warning, does it also happen with v6.9 or only v6.10-rc?

 

Link to comment
5 hours ago, Ystebad said:

I've had this error.  I have multiple 16TB drives filled mostly up (100-200GB left).  No idea why is there but I don't like the sounds of it either.

 

 

I have 2 x 18tb with only ~6tb filled each.

 

My sequence of events was I added the two 18tb disks.  Unraid detected them and asked if I wanted them formatted, I said yes, it went and formatted them and away we went.  Do you remember if you had the same sequence @Ystebad ?

Edited by dalben
Link to comment
4 hours ago, JorgeB said:

Even so the error doesn't always result in the warning, as seen for disks 1 and 2 above, and since Unraid never shrinks a filesystem I can't see how  this would be an actual problem, but obviously if this can be done without the error (and warning) it would be better, for people seeing this warning, does it also happen with v6.9 or only v6.10-rc?

 

 

I can't help there, I have v6.10-rc and upgraded the disks this weekend.  I don't fancy downgrading to test.

Link to comment

I just rebooted the server to see if the warning remains, and it did.

 

Ultimately I can live with a warning message if there's confirmation that I am not really at risk.  At the moment the xfs mailing list are unable to understand why it's happening to one and not the other.  Uncertainty makes me uncertain...

 

Mar  8 06:59:48 tdm emhttpd: shcmd (149): mount -t xfs -o noatime /dev/md1 /mnt/disk1
Mar  8 06:59:49 tdm kernel: SGI XFS with ACLs, security attributes, no debug enabled
Mar  8 06:59:49 tdm kernel: XFS (md1): Mounting V5 Filesystem
Mar  8 06:59:49 tdm kernel: XFS (md1): Ending clean mount
Mar  8 06:59:49 tdm emhttpd: shcmd (150): xfs_growfs /mnt/disk1
Mar  8 06:59:49 tdm kernel: xfs filesystem being mounted at /mnt/disk1 supports timestamps until 2038 (0x7fffffff)
Mar  8 06:59:49 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device
Mar  8 06:59:49 tdm root: meta-data=/dev/md1               isize=512    agcount=32, agsize=137330687 blks
Mar  8 06:59:49 tdm root:          =                       sectsz=512   attr=2, projid32bit=1
Mar  8 06:59:49 tdm root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Mar  8 06:59:49 tdm root:          =                       reflink=1    bigtime=0 inobtcount=0
Mar  8 06:59:49 tdm root: data     =                       bsize=4096   blocks=4394581984, imaxpct=5
Mar  8 06:59:49 tdm root:          =                       sunit=1      swidth=32 blks
Mar  8 06:59:49 tdm root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Mar  8 06:59:49 tdm root: log      =internal log           bsize=4096   blocks=521728, version=2
Mar  8 06:59:49 tdm root:          =                       sectsz=512   sunit=1 blks, lazy-count=1
Mar  8 06:59:49 tdm root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Mar  8 06:59:49 tdm emhttpd: shcmd (150): exit status: 1
Mar  8 06:59:49 tdm kernel: XFS (md1): EXPERIMENTAL online shrink feature in use. Use at your own risk!
Mar  8 06:59:49 tdm emhttpd: shcmd (151): mkdir -p /mnt/disk2
Mar  8 06:59:49 tdm emhttpd: shcmd (152): mount -t xfs -o noatime /dev/md2 /mnt/disk2
Mar  8 06:59:49 tdm kernel: XFS (md2): Mounting V5 Filesystem
Mar  8 06:59:49 tdm kernel: XFS (md2): Ending clean mount
Mar  8 06:59:49 tdm kernel: xfs filesystem being mounted at /mnt/disk2 supports timestamps until 2038 (0x7fffffff)
Mar  8 06:59:49 tdm emhttpd: shcmd (153): xfs_growfs /mnt/disk2
Mar  8 06:59:49 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device
Mar  8 06:59:49 tdm root: meta-data=/dev/md2               isize=512    agcount=32, agsize=137330687 blks
Mar  8 06:59:49 tdm root:          =                       sectsz=512   attr=2, projid32bit=1
Mar  8 06:59:49 tdm root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Mar  8 06:59:49 tdm root:          =                       reflink=1    bigtime=0 inobtcount=0
Mar  8 06:59:49 tdm root: data     =                       bsize=4096   blocks=4394581984, imaxpct=5
Mar  8 06:59:49 tdm root:          =                       sunit=1      swidth=32 blks
Mar  8 06:59:49 tdm root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Mar  8 06:59:49 tdm root: log      =internal log           bsize=4096   blocks=521728, version=2
Mar  8 06:59:49 tdm root:          =                       sectsz=512   sunit=1 blks, lazy-count=1
Mar  8 06:59:49 tdm root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Mar  8 06:59:49 tdm emhttpd: shcmd (153): exit status: 1
Mar  8 06:59:49 tdm emhttpd: shcmd (154): mkdir -p /mnt/disk3
Mar  8 06:59:49 tdm emhttpd: shcmd (155): mount -t xfs -o noatime /dev/md3 /mnt/disk3

 

Link to comment

Unraid OS never tries to shrink the size of a file system (well there is an exception).  What we do upon every mount is to try to expand the size of each file system.  For the devices in the unRAID array this handles the case where a smaller device has been replaced with a larger one.  If the file system cannot be expanded because the device is the same, or a new device is the same size, then the operation will of course fail - and this is reported in the system log - entirely expected.

 

Another case were we need to explicitly expand a file system is following replacement of a smaller device with a larger one in a btrfs pool.  In this case a 'btrfs filesystem resize' is executed

 

Now that I think of it, actually code does not prevent you from replacing a larger device with a smaller one in a btrfs pool, so technically that would be a case where a file system is shrunk.

  • Thanks 1
  • Upvote 1
Link to comment
59 minutes ago, limetech said:

If the file system cannot be expanded because the device is the same, or a new device is the same size, then the operation will of course fail - and this is reported in the system log - entirely expected.

 

Not most times, that xfs_growfs error only happens sometimes, and the warning appears to happen just in some of those times, e.g. from the diags above:

 

disk1 has the error and the warning

Mar  6 19:59:21 tdm emhttpd: shcmd (83): xfs_growfs /mnt/disk1
Mar  6 19:59:21 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device
Mar  6 19:59:21 tdm kernel: XFS (md1): EXPERIMENTAL online shrink feature in use. Use at your own risk!

 

disk2 has the error only

Mar  6 19:59:22 tdm emhttpd: shcmd (86): xfs_growfs /mnt/disk2
Mar  6 19:59:22 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device

 

disk3 no error or warning

Mar  6 19:59:22 tdm emhttpd: shcmd (89): xfs_growfs /mnt/disk3

 

I do agree it's not an actual problem since the filesystem is never shrunk in the array, maybe a xfsprogs issue?

  • Thanks 1
Link to comment

After looking at a few more diags with this issue I think it's like this:

 

With v6.10-rc2 any 1412TB or larger disk formatted with XFS gives the

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device

the

EXPERIMENTAL online shrink feature in use. Use at your own risk!

appears for the first of these only, i.e., if you have a 10TB disk1, 12TB disk2, 14TB disk3 and 8TB disk4 there won't be any error for disk1, disk2 will have the error and the warning, disk3 only show the error and disk4 again no error or warning.

Link to comment
3 hours ago, limetech said:

I looked at xfsprogs version: 5.13.0 source code and I do not see that "online shrink feature in use" error message text.

If you type 'xfs_growfs -V' what version does it show?

 

As @JorgeB confirmed, it's 5.13.0

 

root@tdm:~# xfs_growfs -V
xfs_growfs version 5.13.0

 

Link to comment
13 hours ago, bonienl said:

I am on rc3 test version with a newer kernel, and upgraded to new 18 TB disks. Don’t have this message on any of them (12 disks in total).

That's good news, I assume you mean you don't get the xfs_growfs error and/or the experimental warning, but did you get it with rc2? Still not sure if the error always displays with any 1412TB or larger disk and rc2, but from the diags I've seen so far looks like it does.

 

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.