JorgeB

Just replace all the bz* files on the flash drive and reboot.

Not totally clear for now, but I would guess local transfers are also at risk.

Most I saw have the same. Unlikely, especially if the same exact issue occurs with Ubuntu and zfs. Pass-through also appears to not be a factor, most of the users affected had vt-d enabled because it is by default, but some didn't even had any VMs.

Thank you both, unfortunately doesn't make things any clearer, trying to focus mostly on the Microserver gen8 since it's by far the most affected model but can't find anything in common why some are not affected: -jkpe's BIOS matches a lot of the affected models, quack75 has a different date but since the other matches it doesn't really matter. -some of the affected users have a bare stock server, without any extra card/controllers, others have an extra NIC, controller, NVMe device, etc -affected models all have Xeon CPUs, which is required for vt-d on these servers, and both Sandybridge and Ivybridge can be affected

Reboot to clear the logs and post new diags after array and VMs start.

Enable the syslog server and post that after a crash but if it's a hardware problem likely there won't be anything relevant logged.

Unfortunately there's nothing relevant logged, you can try is to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes could be a hardware problem, or some compatibly issue with the newer kernel, if it doesn't start turning on the other services one by one.

Cache filesystem is corrupt, best bet is to copy all important data elsewhere then re-format and restore the data.

Like mentioned diags show what appear to be power/connection issues, could also be a backplane issue.

Reformatting is never part of recovering data, with an array or pool drive. Please post the diagnostics.

Problem is that one member of the second pool, sdx, the last device, doesn't belong to that pool, and it's generating an error during device scan: May 31 12:54:31 farmer01 kernel: BTRFS: device fsid 31db1a82-5506-42f0-bffb-44f72cf077c3 devid 2 transid 62 /dev/sdu1 scanned by udevd (1562) May 31 12:54:31 farmer01 kernel: BTRFS: device fsid 31db1a82-5506-42f0-bffb-44f72cf077c3 devid 4 transid 62 /dev/sdv1 scanned by udevd (1601) May 31 12:54:31 farmer01 kernel: BTRFS: device fsid 31db1a82-5506-42f0-bffb-44f72cf077c3 devid 3 transid 62 /dev/sdw1 scanned by udevd (1507) May 31 12:54:31 farmer01 kernel: BTRFS: device fsid 31db1a82-5506-42f0-bffb-44f72cf077c3 devid 1 transid 62 /dev/sds1 scanned by udevd (1519) Fisrt 4 member are devices 1 to 4 from this pool, 5th member is device 3 from a different one, note the different fsid: May 31 12:54:31 farmer01 kernel: BTRFS: device fsid 06b8a25f-d7e5-4ab2-9e5b-cc1d3e77bfed devid 3 transid 57 /dev/sdx1 scanned by udevd (1507) This device is causing an error during device scan and it's what's preventing both pools to mount: May 31 13:20:47 farmer01 emhttpd: /mnt/cache ERROR: cannot scan /dev/sdx1: Input/output error Assuming there's no data there you just need to wipe that device wipefs -a /dev/sdx1 and unassign it from the second pool and both pools should now mount, you can then add it back.

As a reminder, so far I don't believe any Dell servers are affected by the corruption issue, even those that have NICs that use the tg3 driver, (though they are affected by the NIC being blacklisted with v6.10.2), if the NICs are the reason for the this problem I haven't found a single example of the IOMMU call traces or any signs of corruption in a Dell, actual issue affects mostly HP servers, about 5 or 6 different models so far, but there's also one IBM/Lenovo, so it's not just HPs, the identical Ubuntu bug report posted above also only mentions HP servers as of now.

Both @quack75and @jkpedo you mind confirming if you were using the onboard NIC, and not some add-on model? Also please post the BIOS you have, e.g.: HP ProLiant MicroServer Gen8, BIOS J06 04/04/2019 You can see that in first few lines of the syslog.

It would be one time thing, and even when it it's no longer needed it won't cause any issues leaving the file there.

Share needs to have an appropriately set minimum free space, usually we recommend setting that to twice the max file size you expect to copy to that share.

Enable the syslog server.

Possibly, I have some experience with SAS daisy chain but never did it with so many expanders, only 2 or 3, don't known if there's a limit, did you test to find out in which number of the chain does it stop working? You could also connect two enclosures per HBA, then would just need to daisy chain 5 units per connection, this would also be better for available bandwidth.

Yes.

It's working because you're back to v6.10.1, to upgrade to v6.10.2 you need to unblacklist the NIC, like linked above.

Yes.

Just repost there.

Didn't read the entire post, if you're getting an IP that's not the problem, please post the diagnostics.

You can do this to unblacklist the NICs: https://forums.unraid.net/bug-reports/stable-releases/after-i-upgraded-to-6102-my-network-interfaces-dissappeared-r1956/?do=findComment&comment=19584

Please use the existing plugin support thread:

Start by checking this: https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173