JorgeB

It does but the last line is 2 minutes after boot, was it already without internet?

Bad RAM is usually the main suspect for unexpected filesystem corruption.

For now it's just a filesystem corruption problem, check filesystem

Forgot to mention, that disk is connected to a SATA port multiplier, those are not recommend, and that could be the problem.

Sep 19 16:10:54 Tower kernel: sd 14:2:0:0: [sdl] tag#25 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s Sep 19 16:10:54 Tower kernel: sd 14:2:0:0: [sdl] tag#25 Sense Key : 0x5 [current] Sep 19 16:10:54 Tower kernel: sd 14:2:0:0: [sdl] tag#25 ASC=0x21 ASCQ=0x4 Sep 19 16:10:54 Tower kernel: sd 14:2:0:0: [sdl] tag#25 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00 Sep 19 16:10:54 Tower kernel: I/O error, dev sdl, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0 Sep 19 16:10:54 Tower kernel: ata14: EH complete Sep 19 16:10:54 Tower kernel: BTRFS error (device sdi1): bdev /dev/sdl1 errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 This is a hardware issue, btrfs errors come after because it can't write to the filesystem, start by replacing/swapping cables. P.S. also need to check filesystem on the device below: Sep 19 16:10:04 Tower kernel: XFS (nvme0n1p1): Unmount and run xfs_repair

2TB 2.5" drives will be SMR, what brand/model? Some SMR models, particularity by Seagate performe decently, SMR drives by Toshiba or WD can have several minutes writing below 5MB/s, even when doing sequential writes.

AFAIK it requires re-formatting.

Formatted with type 1 protection You need to remove this protection from the disks, more info below: https://forums.unraid.net/topic/93432-parity-disk-read-errors/?do=findComment&comment=864078 https://forums.unraid.net/topic/110835-help-with-a-sas-drive/

Yes, you need a disk there, that disk won't be touched initially but needs to be assigned anyway for the procedure to work, and like mentioned it needs to be the same size or larger than the old disk. Because the initial start in maintenance mode doesn't mount the disk, it just records that a disk is there and it's capacity, once you unassign that disk it will be emulated based on parity and the remaining disks.

Correct, you also need to assign the new disk9 when you do the procedure, it must be same size or larger then old disk9.

Not quite following, you cannot start the array with 2 missing disks and single parity, it's also not possible to start the array in read-only mode, if there's something you need now you could mount the disk(s) individually with UD in read-only mode just to access the data, if you do a new config without the missing disks and start the array parity will be invalid and it would prevent future recovery if you fix one of the missing disks. Yes, you'd need to new another new config, add new disk(s) and re-sync parity. See here for a recent example: https://forums.unraid.net/topic/128492-multiple-disabled-disks/?do=findComment&comment=1170517

If the NVMe device is assigned to the array it's true, any writes will be much slower.

No, but you could mount the disks individually with UD in read-only mode. Yes, manually, but yes. Tools -> New config - create a new array without the missing disks and re-sync parity.

Still nothing relevant logged.

Disk dropped offline, check/replace cables and post new diags after array start.

This, most likely the problem.

Connect the disk to one of the onboard SATA ports and see if it's detected by the board BIOS, also make sure the disk is not affected by the 3.3v issue, that's mostly for WD/HGST disks but since it's a new model you never know, so use a molex to SATA adapter just in case.

Sep 19 12:25:53 Tower kernel: BTRFS error (device sdf1): block=2395218591744 write time tree block corruption detected This usually means bad RAM or other kernel memory corruption, Ryzen with overclocked RAM like you have is known to corrupt data, so you should fix that, then run memtest. Sep 17 16:59:52 Tower kernel: macvlan_broadcast+0x10e/0x13c [macvlan] Sep 17 16:59:52 Tower kernel: macvlan_process_broadcast+0xf8/0x143 [macvlan] Unrelated, macvlan call traces are usually the result of having dockers with a custom IP address and will end up crashing the server, upgrading to v6.10 and switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), or see below for more info.

btrfs check --repair should only be used if told to do so by a btrfs maintainer or as a last resort, it can do more harm then good, best bet is to backup and re-format the pool.

Only the onboard NIC is being detected on a hardware level, try a different PCIe slot if available.

Start by running a single stream iperf test and post the diagnostics it it's OK.

Run another scrub and post new diags when done.

Trim won't work for array devices, only pools.

Likely using most free as allocation method, that's not good for performance because parity writes will overlap.