JorgeB

Macvlan call traces are usually related to having dockers with a custom IP address, more info here:

New partition alignment is just one of the changes in -beta30 to help with the write amplification issue, not the only one, and not the one that makes the most difference, using space_cache=v2 with btrfs is what makes the most different, and you can do that now, or go with the beta, it's very stable for most.

Changed Status to Closed Changed Priority to Other

I'm going to close this for now, if you need more help with this issue please use the general support forum. P.S.: since you have identical HBAs and expanders it's easier to diagnose, e.g., swap HBA cables from one expander to the other, if same disks don't show it's likely an expander issue.

Aprox time is in the SMART report: Extended self-test routine recommended polling time: ( 976) minutes.

It's worth a try.

A dropped device won't come back online without at least a reboot, possibly a power cycle.

Is the dropped NVMe back online? Did you reboot?

Sometimes a newer kernel helps, like the one in -beta30, this can sometimes also help: Some NVMe devices have issues with power states on Linux, try this, on the main GUI page click on flash, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (on the top right) and add this to your default boot option, after "append" and before "initrd=/bzroot" nvme_core.default_ps_max_latency_us=0 Reboot and see if it makes a difference.

Crazy number of writes usually means a dropped device, and that was what happened here: Oct 13 16:09:58 floserver kernel: nvme nvme1: I/O 102 QID 8 timeout, aborting Oct 13 16:10:28 floserver kernel: nvme nvme1: I/O 102 QID 8 timeout, reset controller Oct 13 16:10:59 floserver kernel: nvme nvme1: I/O 28 QID 0 timeout, reset controller Oct 13 16:13:40 floserver kernel: nvme nvme1: Device not ready; aborting reset Oct 13 16:13:40 floserver kernel: nvme nvme1: Abort status: 0x7 Oct 13 16:13:40 floserver kernel: print_req_error: I/O error, dev nvme1n1, sector 41730296 Oct 13 16:15:48 floserver kernel: nvme nvme1: Device not ready; aborting reset Oct 13 16:15:48 floserver kernel: nvme nvme1: Removing after probe failure status: -19 Oct 13 16:17:56 floserver kernel: nvme nvme1: Device not ready; aborting reset Recommended reading this for better pool monitoring.

Log is being spammed with PCIe errors so not easy to analyze, but the NVMe devices are not part of the pool, try this: -reboot (mostly to clear the log) -stop array (if set to auto-start) -unassign both NVMe devices -start array, stop array -re-assign both NVMe devices to the pool -start array, if total/free space still doesn't change please post new diags.

Looks network related, if possible try to simplify your LAN config and test.

This.

Best to use the DOS or UEFI tools to do any crossflashing and/or other more advanced options, e.g., for some reason erasing the BIOS doesn't work with the Linux version, but if you just need to update firmware it can be easily done with Unraid. Download the MSDOS firmware package form Broadcom's support site (https://www.broadcom.com/support/download-search), for example for the 9300-8i it currently is this one: 9300_8i_Package_P16_IR_IT_FW_BIOS_for_MSDOS_Windows Note: Downloads for SAS2 HBAs like for example the 9211-8i are usually under "Legacy Products" and then "Legacy Host Bus Adapters". Now download the Linux Installer, also from the firmware section, again for the 9300-8i it currently is: Installer_P16_for_Linux From the DOS package we only need the firmware: 9300_8i_Package_P16_IR_IT_FW_BIOS_for_MSDOS_Windows\Firmware\SAS9300_8i_IT\SAS9300_8i_IT.bin Optional: if the HBA has a BIOS installed and we want to updated it also copy the BIOS file: 9300_8i_Package_P16_IR_IT_FW_BIOS_for_MSDOS_Windows\sasbios_rel\mptsas3.rom From the installer package we only need sas3flash (sas2flash for SAS2 models): Installer_P16_for_Linux\sas3flash_linux_x64_rel\sas3flash Copy both files to your Unraid server, you can copy them to flash first but still need to copy them elsewhere since the executable can't be run from the flash drive, we also don't want them in the array since it should be stopped before flashing. For this example I first copied both files to folder called "lsi" in the flash drive then copied them to a temp folder I created called /lsi (this folder will be stored in RAM and gone after a reboot), use the console and type: mkdir /lsi cp /boot/lsi/* /lsi After copying the files navigate to that directory, in this case: cd /lsi We now need to make sas3flash executable with: chmod +x sas3flash Before flashing we can confirm the adapter (or adapters) are being detected and check what is the firmware currently installed: ./sas3flash -listall Avago Technologies SAS3 Flash Utility Version 17.00.00.00 (2018.04.02) Copyright 2008-2018 Avago Technologies. All rights reserved. Adapter Selected is a Avago SAS: SAS3008(C0) Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr ---------------------------------------------------------------------------- 0 SAS3008(C0) 16.00.01.00 0e.01.00.07 08.37.00.00 00:01:00:00 1 SAS3008(C0) 16.00.01.00 0e.01.00.07 08.37.00.00 00:02:00:00 Make sure the array (and any UD devices using the HBA) is stopped and unmounted and now we can proceed with the upgrade, just use the standard sas3flash command, but remember that Linux is case sensitive, e.g.: ./sas3flash -o -f SAS9300_8i_IT.bin Optional: if you also want to flash the BIOS use ./sas3flash -o -f SAS9300_8i_IT.bin -b mptsas3.rom After it's done confirm the new firmware (and bios if also flashed) is installed: ./sas3flash -listall Avago Technologies SAS3 Flash Utility Version 17.00.00.00 (2018.04.02) Copyright 2008-2018 Avago Technologies. All rights reserved. Adapter Selected is a Avago SAS: SAS3008(C0) Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr ---------------------------------------------------------------------------- 0 SAS3008(C0) 16.00.10.00 0e.01.00.07 08.37.00.00 00:01:00:00 1 SAS3008(C0) 16.00.01.00 0e.01.00.07 08.37.00.00 00:02:00:00 If there are multiple adapters like in this case only the first one will be flashed, to flash the other one use -c to specify the adapter#, e.g.: ./sas3flash -c 1 -o -f SAS9300_8i_IT.bin ./sas3flash -listall Avago Technologies SAS3 Flash Utility Version 17.00.00.00 (2018.04.02) Copyright 2008-2018 Avago Technologies. All rights reserved. Adapter Selected is a Avago SAS: SAS3008(C0) Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr ---------------------------------------------------------------------------- 0 SAS3008(C0) 16.00.10.00 0e.01.00.07 08.37.00.00 00:01:00:00 1 SAS3008(C0) 16.00.10.00 0e.01.00.07 08.37.00.00 00:02:00:00 Finished Processing Commands Successfully. Exiting SAS3Flash. Now both are updated. P.S. to see more detailed info from an adapter we can use: ./sas3flash -list Avago Technologies SAS3 Flash Utility Version 17.00.00.00 (2018.04.02) Copyright 2008-2018 Avago Technologies. All rights reserved. Adapter Selected is a Avago SAS: SAS3008(C0) Controller Number : 1 Controller : SAS3008(C0) PCI Address : 00:02:00:00 SAS Address : 500605b-0-0b1c-2d00 NVDATA Version (Default) : 0e.01.00.07 NVDATA Version (Persistent) : 0e.01.00.07 Firmware Product ID : 0x2221 (IT) Firmware Version : 16.00.10.00 NVDATA Vendor : LSI NVDATA Product ID : SAS9300-8i BIOS Version : 08.37.00.00 UEFI BSD Version : 06.00.00.00 FCODE Version : N/A Board Name : SAS9300-8i Board Assembly : H3-25573-00H Board Tracer Number : SP60700521 Finished Processing Commands Successfully. If there's more than one we can again use -c, e.g.: ./sas3flash -c 1 -list

That's from the Fix Common Problems plugin, message tells you what to do, but any doubts please use the existing plugin support thread.

Thanks, already marked as SOLVED.

You can do it in the GUI, click on cache and scroll down to the balance section.

Disk looks healthy, most likely a power/connection issue, if you already replaced/checked them you can rebuild on top, but start the array and make sure the emulated disk mounts and contents looks correct before doing it. https://wiki.unraid.net/Troubleshooting#Re-enable_the_drive

I don't use it, best to ask here:

There's nothing logged about CPU overheating, did you clear the FCP error? I believe if you don't clear it it will keep warning.

Disk disable is normal, the question is if it came back online, post new diags.

Those diags are the same, or the disk is still offline, if you don't want to check connections you need to at least power cycle the server to see if it comes back online.

Disk2 dropped offline, so there's no SMART, check connections and post new diags.

That's what I though, you can check (or post) the syslog to see if those errors are on data or metadata, if it's data you can delete those files, if it's metadata you'll need to recreate the pool and restore data from backups.

BTW, if there's nothing else to delete you can delete the docker image, should be enough and it can easily be recreated later.