JorgeB

After looking at a few more diags with this issue I think it's like this: With v6.10-rc2 any 1412TB or larger disk formatted with XFS gives the xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device the EXPERIMENTAL online shrink feature in use. Use at your own risk! appears for the first of these only, i.e., if you have a 10TB disk1, 12TB disk2, 14TB disk3 and 8TB disk4 there won't be any error for disk1, disk2 will have the error and the warning, disk3 only show the error and disk4 again no error or warning.

Not most times, that xfs_growfs error only happens sometimes, and the warning appears to happen just in some of those times, e.g. from the diags above: disk1 has the error and the warning Mar 6 19:59:21 tdm emhttpd: shcmd (83): xfs_growfs /mnt/disk1 Mar 6 19:59:21 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device Mar 6 19:59:21 tdm kernel: XFS (md1): EXPERIMENTAL online shrink feature in use. Use at your own risk! disk2 has the error only Mar 6 19:59:22 tdm emhttpd: shcmd (86): xfs_growfs /mnt/disk2 Mar 6 19:59:22 tdm root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device disk3 no error or warning Mar 6 19:59:22 tdm emhttpd: shcmd (89): xfs_growfs /mnt/disk3 I do agree it's not an actual problem since the filesystem is never shrunk in the array, maybe a xfsprogs issue?

UD mountpoint changed from /mnt/disks/Samsung_SSD_860_EVO_500GB_S59UNJ0N109855F to /mnt/disks/S59UNJ0N109855F, likely due to a UD update, change it back. P.S. there's filesystem corruption on disk2 and btrfs is detecting data corruption on cache, so good idea to run memtest.

Mar 6 12:44:08 Bamboo kernel: BTRFS info (device sde1): bdev /dev/sde1 errs: wr 0, rd 0, flush 0, corrupt 510952, gen 0 Mar 6 12:44:08 Bamboo kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 3707, gen 0 Lots of data corruption being detected on both pools, start by running memtest.

That's also what I suspect, diags would confirm.

That's metadata corruption, for this it's best to backup and re-format the pool.

Please post the diagnostics.

Yes.

Correct, like mentioned in the link, just set the correct power supply idle control, only if that option doesn't exist you should disable c-states. After a scrub the corrupt files(s) will be listed in the syslog, delete/replace from backups.

Checksum errors mean btrfs is detecting data corruption, make sure you're RAM is not overclocked since is a known source of data corruption with Ryzen/Threadripper and/or run memtest.

Rebooting should fix that, if it doesn't post new diags.

Since the warning appears to come from xfs_growfs, and that's part of xfsprogrs, it might go way once that's updated in a newer Unraid release.

Pool filesystem is corrupt, best bet is to backup, reformat and restore the data. There's also this: Mar 7 11:23:39 Orcrist kernel: macvlan_broadcast+0x116/0x144 [macvlan] Mar 7 11:23:39 Orcrist kernel: macvlan_process_broadcast+0xc7/0x10b [macvlan] Macvlan call traces are usually the result of having dockers with a custom IP address, upgrading to v6.10 and switching to ipvlan might fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enable, top right)), or see below for more info. https://forums.unraid.net/topic/70529-650-call-traces-when-assigning-ip-address-to-docker-containers/ See also here: https://forums.unraid.net/bug-reports/stable-releases/690691-kernel-panic-due-to-netfilter-nf_nat_setup_info-docker-static-ip-macvlan-r1356/

Script is known to be slow and/or not work properly with current releases, recommend doing it manually or using the remove drive and rebuild parity option.

It's normal for the stats not to be correct with that config, raid0 with different size and odd number of devices, you can change to single profile, they will be correct for that.

One more thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

This is normal: Mar 6 19:59:21 tdm emhttpd: shcmd (83): xfs_growfs /mnt/disk1 This is not normal: root: xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on device Unraid tries to grow the filesystem at every mount, but usually there's no error when it can't be grown, and that's every time except after upgrading a disk. Even so the error doesn't always result in the warning, as seen for disks 1 and 2 above, and since Unraid never shrinks a filesystem I can't see how this would be an actual problem, but obviously if this can be done without the error (and warning) it would be better, for people seeing this warning, does it also happen with v6.9 or only v6.10-rc?

https://wiki.unraid.net/Crossflashing_Controllers

Board/CPU/RAM would be the main suspects.

Unraid is seeing both devices with the same ID, they might work if you connect them to the onboard SATA ports (or flash the LSI to IT mode, which you should anyway, but will require a new config).

Please post the diagnostics.

I can't see because the syslog cuts before that due to all the disk issues, but possibly related to the USB drops, make sure you use a USB 2.0 port if available.

9300-8i works fine, have a couple myself, firmware is also recent, so it should be fine, a rebooting server is usually due to a hardware issue.

This is usually a flash drive problem, recreating it might fix it, this sometimes also helps.

That part of of the Fix Common Problems plugin, it's not stock.