JorgeB

Start by checking network bandwidth by running a single stream iperf test.

This is usually a PSU problem.

Yep, so that it's moved to the array.

Remembered that while I don't have to hardware to test the real max for an x8 PCIe 3.0 slot, I can test with the HBA in an x4 slot, so same 9300-8i with 8 directly connected SSDs as above but this time I enabled bifurcation on the slot effectively turning it into an x4 slot: 02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) LnkSta: Speed 8GT/s (ok), Width x4 (downgraded) So an x8 slot should be able to do around 6800MB/s, possibly a little more with different hardware, still think 7000MB/s+ will be very difficult, but won't say it's impossible, though again don't think the PCIe bandwidth is what's limiting the OP's speed, IMHO and more likely it's because of the expander being used with non 12G devices.

Looks like it's not possible to just expose the disks individually with that enclosure, unless you create individual raids, so not ideal for Unraid: https://www.dell.com/community/PowerVault/Dell-MD3200-quot-Enhanced-JBOD-quot-IT-mode/td-p/3804722

This is usually a board/BIOS issue, the unclean shutdown part means you're timeouts are likely too low, you can analyze or post the diags saved on the flash drive at shutdown time to look for what is the problem.

For the amount of data you have probably best to backup pool to array, create new pool and restore, but if you prefer to keep dockers/VM online it's also possible to add the new device and then remove the other ones one by one, just note that direct device replacement is broken on v6.9.x and can't be used unless done manually.

It should be, another common issue with Ryzen is the RAM speed, good to check you're withing max official specs.

This is usually a flash drive problem, try recreating it or using a different one.

You can either run a correcting check then a non correcting one and the 2nd one should result in 0 errors, or alternatively run two non correcting checks and see if you get the same results, if the number of errors is the same but a low number good to check they are the same blocks.

No, if memtest didn't detect errors with both DIMMs very unlikely it will with just one, remove one DIMM and run two consecutive parity checks, if the second one still finds errors do the same with the other DIMM, if still errors with either one alone I would try a different board/CPU next.

You can, if you don't need a pool, but that's not the main issue, main issue is finding out what is corrupting data.

You can reboot before or after, it's just to clear the errors on the GUI, it won't affect anything else.

Forgot to mention, to clear the read errors reboot.

Test passed so disk is good for now, you need to run a correcting parity check because due to previous read errors parity won't be 100% in sync, then keep monitoring that disk.

It's not a device problem, it's a filesystem problem, corruption likely happened for the same reason you're getting sync errors. Most likley. Not that I can see, you could try with just one DIMM at a time and see if you don't get sync errors like that, note that the first check after the problem is fixed may still find errors.

No reason it shouldn't, SAS expanders are transparent to the OS. Depending on the number of disks you think you might need you may also consider the Intel RES2SV240 or the RES2CV360, both are SATA3 and can be found considerably cheaper on ebay, either one will give you a nice bandwidth bump, for more you'd need a PCIe 3.0 HBA.

Booting in safe mode disables plugins, so worth trying.

With 16 drives the bottleneck is going to be the expander, since it's SATA2 only it will be limited to 2200MB/s usable max, assuming dual link, that's around 140MB/s max per disk when used concurrently, you can see here for some more numbers, just upgrading the HBA won't help here, you'd need a SAS2/SATA3 expander and to improve even more also a PCIe 3.0 HBA to go above the PCIe 2.0 limit.

Yep, this usually suggests more a hardware issue, one thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

No, sorry, don't even know if it's possible, but you can google it, look for "lenovo 530-8i it mode"

RAID controller is failing to initialize: Jun 5 02:59:51 Taii kernel: megaraid_sas 0000:05:00.0: Ignore DCMD timeout: megasas_get_ctrl_info 5274 Jun 5 02:59:51 Taii kernel: megaraid_sas 0000:05:00.0: Could not get controller info. Fail from megasas_init_adapter_fusion 1865 Jun 5 02:59:51 Taii kernel: megaraid_sas 0000:05:00.0: Failed from megasas_init_fw 6406 Look for a firmware update, though we don't really recommend RAID controllers with Unraid, since it uses an LSI SAS3408 it might be flashable to IT mode, you can look for that, but it will likely require a new config and even after that it's not certain to correctly mount existing disks, one of the reason we don't recommend RAID.

Please post the diagnostics: Tools -> Diagnostics

OK, granted, but also keep in mind that, and again assuming the OP is using SATA devices, he will suffer from some performance loss due to the mentioned SAS wide link not being really 12G, and that is probably what's limiting his current bandwidth, and not the PCIe x8 slot. x4 SAS link has max bandwidth of 1200MB/s, 1100MB/s usable, x4 SAS2 link max is 2400MB/s, 2200MB/s usable, both of these I confirmed myself in the tests linked above and results are very consistent with a number of different devices, x4 SAS3 link max is 4800MB/s, I'm going to assume 4400MB/s usable, unfortunately I have no way of testing this since I'd need some SAS3 SSDs, but LSI points to the same values: Now if we accept those I can show the performance degradation of not using a real SAS3 link: Left side 8 SATA3 SSDs directly connected to an LSI 9300-8i, right side same but now with an expander in the middle, so while Databolt does a very good job, or max combined speed would only be 2.2GB/s, still about 10% degradation from what it should be with real SAS3 devices (4.4GB/s), and IMHO that's likely the limit the OP is hitting, and in that case getting a PCIe x16 HBA won't really help. I would say not possible with SATA devices because of the above, and very much doubt it's possible even with SAS devices, I believe that in optimal conditions around 6500MB/s would be possible, don't think 7000MB/s+ can be done, but happy to be proved wrong , just point to someone achieving those speeds in the real world with a HBA/expander, I would like to get my hands on some SAS3 SSDs to test myself, maybe some day if I can find some cheap ones.

You can try this, enable syslog mirror to flash then post that log after a crash.