scs3jb

[2026-04-30 16:18:43] [ERR!] [guard_path] PERSIST_PATH outside allowed prefixes: /mnt/scratch_old/aicliagents [2026-04-30 16:18:43] [INFO] [MIGRATION] ERROR: Persistence path failed validation: /mnt/scratch_old/aicliagents [2026-04-30 16:21:37] [INFO] [AICliAgents] Initializing AICliAgents Plugin (v1.0)... [2026-04-30 16:21:38] [INFO] [StorageMountService] Mounting Home Stack for root [2026-04-30 16:21:38] [ERR!] [guard_path] PERSIST_PATH outside allowed prefixes: /mnt/scratch_old/aicliagents/persistence [2026-04-30 16:21:38] [ERR!] [MOUNT] Persistence path failed validation: /mnt/scratch_old/aicliagents/persistence [2026-04-30 16:21:38] [ERR!] [StorageMountService] Mount script FAILED for home root: [2026-04-30 16:21:38] [ERR!] [MOUNT] Persistence path failed validation: /mnt/scratch_old/aicliagents/persistence PERSIST_PATH outside of allowed prefixes a bug? From a fresh install: [2026-04-30 16:30:33] [ERR!] [guard_path] PERSIST_PATH outside allowed prefixes: /mnt/scratch_old/aicliagents/persistence [2026-04-30 16:30:33] [ERR!] [MOUNT] Persistence path failed validation: /mnt/scratch_old/aicliagents/persistence [2026-04-30 16:30:33] [ERR!] [StorageMountService] Mount script FAILED for agent gemini-cli: [2026-04-30 16:30:33] [ERR!] [MOUNT] Persistence path failed validation: /mnt/scratch_old/aicliagents/persistence [2026-04-30 16:30:33] [ERR!] [InstallerService] Failed to mount storage stack for gemini-cli [2026-04-30 16:30:33] [ERR!] [AICliAgents] Background Install Job FAILED for gemini-cli: Could not mount agent storage [2026-04-30 16:30:34] [INFO] [COMMIT] No changes to commit for home root [2026-04-30 16:30:34] [ERR!] [guard_path] PERSIST_PATH outside allowed prefixes: /mnt/scratch_old/aicliagents/persistence [2026-04-30 16:30:34] [ERR!] [MOUNT] Persistence path failed validation: /mnt/scratch_old/aicliagents/persistence [2026-04-30 16:30:34] [ERR!] [StorageMountService] Mount script FAILED for agent claude-code: [2026-04-30 16:30:34] [ERR!] [MOUNT] Persistence path failed validation: /mnt/scratch_old/aicliagents/persistence [2026-04-30 16:30:34] [ERR!] [InstallerService] Failed to mount storage stack for claude-code [2026-04-30 16:30:34] [ERR!] [AICliAgents] Background Install Job FAILED for claude-code: Could not mount agent storage Seems to try and force me to use the usb / fixed paths?

Anyone set up Forgejo CI Runner? I took a quick scan, looks like you will need to run a new container that will run either podman or docker in it? Anyone made anything that plugs straight into unraid? https://forgejo.org/docs/latest/admin/actions/runner-installation/ https://dbushell.com/2025/08/15/self-hosted-forgejo-actions-runner/

Thank you @JorgeB, dual link is now working with the firmware updated. Dual link in use, no more errors in syslog/dmesg and running a parity check is getting ~150 mb/s across 21 disks. I'm happy with that for sata drives on an expander. Marking as solution.

Let me write a miniguide for muppets like me: Download https://www.broadcom.com/site-search?q=Installer_P16_for_Linux and extract it to somewhere you can run things. Confirm the make, model and firmware with: root@beyonder-nas:/mnt/scratch_old/backups/storage# ./sas3flash -list Avago Technologies SAS3 Flash Utility Version 17.00.00.00 (2018.04.02) Copyright 2008-2018 Avago Technologies. All rights reserved. Adapter Selected is a Avago SAS: SAS3224(A1) Controller Number : 0 Controller : SAS3224(A1) PCI Address : 00:02:00:00 SAS Address : 500062b-2-0299-1611 NVDATA Version (Default) : 10.00.00.03 NVDATA Version (Persistent) : 10.00.00.03 Firmware Product ID : 0x2228 (IT) Firmware Version : 16.00.01.00 NVDATA Vendor : LSI NVDATA Product ID : SAS9305-24i BIOS Version : 08.37.00.00 UEFI BSD Version : 18.00.00.00 FCODE Version : N/A Board Name : SAS9305-24i Board Assembly : 03-25699-02004 Board Tracer Number : XW841903CD Finished Processing Commands Successfully. Exiting SAS3Flash. I have a 9305-24i, so need this https://docs.broadcom.com/docs/9305_24i_Pkg_P16.12_IT_FW_BIOS_for_MSDOS_Windows.zip . extract it and look for the firmware, it will be a .bin file. Now we can flash the firmware: root@beyonder-nas:/mnt/scratch_old/backups/storage# ./sas3flash -o -c 00 -f SAS9305_24i_IT_P.bin Avago Technologies SAS3 Flash Utility Version 17.00.00.00 (2018.04.02) Copyright 2008-2018 Avago Technologies. All rights reserved. Advanced Mode Set Adapter Selected is a Avago SAS: SAS3224(A1) Executing Operation: Flash Firmware Image Firmware Image has a Valid Checksum. Firmware Version 16.00.12.00 Firmware Image compatible with Controller. Valid NVDATA Image found. NVDATA Major Version 10.00 Checking for a compatible NVData image... NVDATA Device ID and Chip Revision match verified. NVDATA Versions Compatible. Valid Initialization Image verified. Valid BootLoader Image verified. Beginning Firmware Download... Firmware Download Successful. Verifying Download... Firmware Flash Successful. Resetting Adapter... Adapter Successfully Reset. NVDATA Version 10.00.00.03 Finished Processing Commands Successfully. Exiting SAS3Flash. Finally you can confirm it is flashed: root@beyonder-nas:/mnt/scratch_old/backups/storage# ./sas3flash -list Avago Technologies SAS3 Flash Utility Version 17.00.00.00 (2018.04.02) Copyright 2008-2018 Avago Technologies. All rights reserved. Adapter Selected is a Avago SAS: SAS3224(A1) Controller Number : 0 Controller : SAS3224(A1) PCI Address : 00:02:00:00 SAS Address : 500062b-2-0299-1611 NVDATA Version (Default) : 10.00.00.03 NVDATA Version (Persistent) : 10.00.00.03 Firmware Product ID : 0x2228 (IT) Firmware Version : 16.00.12.00 NVDATA Vendor : LSI NVDATA Product ID : SAS9305-24i BIOS Version : 08.37.00.00 UEFI BSD Version : 18.00.00.00 FCODE Version : N/A Board Name : SAS9305-24i Board Assembly : 03-25699-02004 Board Tracer Number : XW841903CD Finished Processing Commands Successfully. Exiting SAS3Flash. Now i recable and see if this solved my problem

Do you know where i could find this? I struggled to find the latest and broadcom site seems to no longer have download for anything but the latest. Edit: nevermind, i found what i need: https://www.broadcom.com/site-search?q=Installer_P16_for_Linux https://docs.broadcom.com/docs/9305_24i_Pkg_P16.12_IT_FW_BIOS_for_MSDOS_Windows.zip run sas3flash -list to confirm the version and for some reason its for MSDOS, you can change the model and 24i bit of the url to get the right package.

I have currently cabled my back-plane to single link as i found i received link resets and read/write errors (UDMA CRC errors) if i use dual link. I unfortunately do not have another SAS controller to eliminate it as the cause of the problems, but I have recabled. Currently I am saturating the single channel link when I do parity actions and it tops out at 2.2G. SAS Expander: Super Mei TOP24 Expander 05G451625012 REV CO BLACK 12Gb (there is a sticker saying FW 0a00 2521 V1.00) SAS Controller: Broadcom / LSI SAS3224 PCI-Express Fusion-MPT SAS-3 (rev 01) (during boot it says LSISAS3224: FWVersion(16.00.01.00), ChipRevision(0x01), BiosVersion(18.00.00.00) ) [1000:00c4] 02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3224 PCI-Express Fusion-MPT SAS-3 (rev 01) [10:0:20:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdaa 20.0TB [10:0:21:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdab 20.0TB [10:0:22:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdac 20.0TB [10:0:23:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdad 20.0TB [10:0:0:0] disk ATA WDC WD140EMFZ-11 0A81 /dev/sdc 14.0TB [10:0:1:0] disk ATA WDC WD140EMFZ-11 0A81 /dev/sdd 14.0TB [10:0:2:0] disk ATA WDC WD140EMFZ-11 0A81 /dev/sde 14.0TB [10:0:3:0] disk ATA ST12000VN0007-2G SC60 /dev/sdf 12.0TB [10:0:4:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdg 18.0TB [10:0:5:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdi 18.0TB [10:0:6:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdj 18.0TB [10:0:7:0] disk ATA WDC WD140EMFZ-11 0A81 /dev/sdn 14.0TB [10:0:8:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdo 18.0TB [10:0:9:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdp 18.0TB [10:0:10:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdq 18.0TB [10:0:11:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdr 18.0TB [10:0:12:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sds 20.0TB [10:0:13:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdt 20.0TB [10:0:14:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdu 20.0TB [10:0:15:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdv 20.0TB [10:0:16:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdw 20.0TB [10:0:17:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdx 20.0TB [10:0:18:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdy 20.0TB [10:0:19:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdz 20.0TB All of the diagnostics are from the single link setup, but perhaps there are some hints as to what is the problem. This link looks to be established fine, and you can see the firmware and versions of the controller. [ 36.314490] mpt3sas_cm0: iomem(0x0000000082500000), mapped(0x00000000430be8e9), size(65536) [ 36.316459] mpt3sas_cm0: ioport(0x0000000000004000), size(256) [ 36.341341] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 36.342958] mpt3sas_cm0: sending message unit reset !! [ 36.345780] mpt3sas_cm0: message unit reset: SUCCESS [ 36.348106] mpt3sas_cm0: scatter gather: sge_in_main_msg(9), sge_per_chain(15), sge_per_io(128), chains_per_io(8) [ 36.355313] mpt3sas_cm0: request pool(0x00000000d091fde2) - dma(0x139200000): depth(5852), frame_size(256), pool_size(1463 kB) [ 36.371186] mpt3sas_cm0: sense pool(0x00000000ebd46e7d) - dma(0x13a700000): depth(5791), element_size(96), pool_size (542 kB) [ 36.373700] mpt3sas_cm0: reply pool(0x000000004f53c277) - dma(0x13a800000): depth(5916), frame_size(128), pool_size(739 kB) [ 36.376178] mpt3sas_cm0: config page(0x00000000035a8032) - dma(0x13234d000): size(512) [ 36.378734] mpt3sas_cm0: Allocated physical memory: size(17295 kB) [ 36.381342] mpt3sas_cm0: Current Controller Queue Depth(5788),Max Controller Queue Depth(5824) [ 36.384083] mpt3sas_cm0: Scatter Gather Elements per IO(128) [ 36.525341] mpt3sas_cm0: _base_display_fwpkg_version: complete [ 36.525557] mpt3sas_cm0: overriding NVDATA EEDPTagMode setting [ 36.526162] mpt3sas_cm0: LSISAS3224: FWVersion(16.00.01.00), ChipRevision(0x01), BiosVersion(18.00.00.00) [ 36.526167] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) [ 36.530365] mpt3sas_cm0: sending port enable !! [ 36.534865] mpt3sas_cm0: hba_port entry: 0000000081440fd9, port: 255 is added to hba_port list [ 36.546212] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500062b202991611), phys(24) [ 36.553485] mpt3sas_cm0: expander_add: handle(0x0019), parent(0x0011), sas_addr(0xcc1be0310015463f), phys(37) [ 36.559900] expander-10:0: add: handle(0x0019), sas_addr(0xcc1be0310015463f) [ 36.559922] mpt3sas_cm0: port enable: SUCCESS [ 36.577354] mpt3sas_cm0: handle(0x1a) sas_address(0xcc1be03100154600) port_type(0x1) [ 36.590904] scsi 10:0:0:0: SATA: handle(0x001a), sas_addr(0xcc1be03100154600), phy(0), device_name(0x5000cca264d70f29) [ 36.644352] end_device-10:0:0: add: handle(0x001a), sas_addr(0xcc1be03100154600) [ 36.644814] mpt3sas_cm0: handle(0x1b) sas_address(0xcc1be03100154601) port_type(0x1) [ 36.651453] scsi 10:0:1:0: SATA: handle(0x001b), sas_addr(0xcc1be03100154601), phy(1), device_name(0x5000cca264df1ce0) [ 36.655977] end_device-10:0:1: add: handle(0x001b), sas_addr(0xcc1be03100154601) [ 36.656265] mpt3sas_cm0: handle(0x1c) sas_address(0xcc1be03100154602) port_type(0x1) [ 36.657287] scsi 10:0:2:0: SATA: handle(0x001c), sas_addr(0xcc1be03100154602), phy(2), device_name(0x5000cca28dc54e4b) [ 36.669233] end_device-10:0:2: add: handle(0x001c), sas_addr(0xcc1be03100154602) [ 36.678566] mpt3sas_cm0: handle(0x24) sas_address(0xcc1be03100154603) port_type(0x1) [ 36.702145] scsi 10:0:3:0: SATA: handle(0x0024), sas_addr(0xcc1be03100154603), phy(3), device_name(0x5000c500a392550d) [ 36.801469] end_device-10:0:3: add: handle(0x0024), sas_addr(0xcc1be03100154603) [ 36.811823] mpt3sas_cm0: handle(0x1d) sas_address(0xcc1be03100154604) port_type(0x1) [ 36.845256] scsi 10:0:4:0: SATA: handle(0x001d), sas_addr(0xcc1be03100154604), phy(4), device_name(0x5000cca2afc6dedc) [ 36.892815] end_device-10:0:4: add: handle(0x001d), sas_addr(0xcc1be03100154604) [ 36.902157] mpt3sas_cm0: handle(0x1e) sas_address(0xcc1be03100154605) port_type(0x1) [ 36.919262] scsi 10:0:5:0: SATA: handle(0x001e), sas_addr(0xcc1be03100154605), phy(5), device_name(0x5000cca284d5623f) [ 36.978479] end_device-10:0:5: add: handle(0x001e), sas_addr(0xcc1be03100154605) [ 37.003230] mpt3sas_cm0: handle(0x1f) sas_address(0xcc1be03100154606) port_type(0x1) [ 37.098613] scsi 10:0:6:0: SATA: handle(0x001f), sas_addr(0xcc1be03100154606), phy(6), device_name(0x5000cca2afc3c57c) [ 37.287258] end_device-10:0:6: add: handle(0x001f), sas_addr(0xcc1be03100154606) [ 37.295286] mpt3sas_cm0: handle(0x20) sas_address(0xcc1be03100154607) port_type(0x1) [ 37.305559] scsi 10:0:7:0: SATA: handle(0x0020), sas_addr(0xcc1be03100154607), phy(7), device_name(0x5000cca264dfc5ee) [ 37.321216] end_device-10:0:7: add: handle(0x0020), sas_addr(0xcc1be03100154607) [ 37.333591] mpt3sas_cm0: handle(0x22) sas_address(0xcc1be03100154608) port_type(0x1) [ 37.350951] scsi 10:0:8:0: SATA: handle(0x0022), sas_addr(0xcc1be03100154608), phy(8), device_name(0x5000cca284ef2699) [ 37.515223] end_device-10:0:8: add: handle(0x0022), sas_addr(0xcc1be03100154608) [ 37.520455] mpt3sas_cm0: handle(0x23) sas_address(0xcc1be03100154609) port_type(0x1) [ 37.544456] scsi 10:0:9:0: SATA: handle(0x0023), sas_addr(0xcc1be03100154609), phy(9), device_name(0x5000cca2afc510fb) [ 37.582514] end_device-10:0:9: add: handle(0x0023), sas_addr(0xcc1be03100154609) [ 37.591047] mpt3sas_cm0: handle(0x25) sas_address(0xcc1be0310015460a) port_type(0x1) [ 37.607455] scsi 10:0:10:0: SATA: handle(0x0025), sas_addr(0xcc1be0310015460a), phy(10), device_name(0x5000cca2afc41fea) [ 37.641779] end_device-10:0:10: add: handle(0x0025), sas_addr(0xcc1be0310015460a) [ 37.650665] mpt3sas_cm0: handle(0x26) sas_address(0xcc1be0310015460b) port_type(0x1) [ 37.667995] scsi 10:0:11:0: SATA: handle(0x0026), sas_addr(0xcc1be0310015460b), phy(11), device_name(0x5000cca284d4e2cf) [ 37.716350] end_device-10:0:11: add: handle(0x0026), sas_addr(0xcc1be0310015460b) [ 37.725195] mpt3sas_cm0: handle(0x27) sas_address(0xcc1be0310015460c) port_type(0x1) [ 37.747290] scsi 10:0:12:0: SATA: handle(0x0027), sas_addr(0xcc1be0310015460c), phy(12), device_name(0x5000cca2bec4ff5b) [ 37.849489] end_device-10:0:12: add: handle(0x0027), sas_addr(0xcc1be0310015460c) [ 37.849629] mpt3sas_cm0: handle(0x28) sas_address(0xcc1be0310015460d) port_type(0x1) [ 37.857081] scsi 10:0:13:0: SATA: handle(0x0028), sas_addr(0xcc1be0310015460d), phy(13), device_name(0x5000cca407e2e65a) [ 37.935803] end_device-10:0:13: add: handle(0x0028), sas_addr(0xcc1be0310015460d) [ 37.936005] mpt3sas_cm0: handle(0x29) sas_address(0xcc1be0310015460e) port_type(0x1) [ 37.943492] scsi 10:0:14:0: SATA: handle(0x0029), sas_addr(0xcc1be0310015460e), phy(14), device_name(0x5000cca2c0c512a2) [ 38.022516] end_device-10:0:14: add: handle(0x0029), sas_addr(0xcc1be0310015460e) [ 38.022733] mpt3sas_cm0: handle(0x2a) sas_address(0xcc1be0310015460f) port_type(0x1) [ 38.031128] scsi 10:0:15:0: SATA: handle(0x002a), sas_addr(0xcc1be0310015460f), phy(15), device_name(0x5000cca2bfc040d3) [ 38.115380] end_device-10:0:15: add: handle(0x002a), sas_addr(0xcc1be0310015460f) [ 38.116866] mpt3sas_cm0: handle(0x2b) sas_address(0xcc1be03100154610) port_type(0x1) [ 38.126201] scsi 10:0:16:0: SATA: handle(0x002b), sas_addr(0xcc1be03100154610), phy(16), device_name(0x5000cca2b3c380e6) [ 38.205629] end_device-10:0:16: add: handle(0x002b), sas_addr(0xcc1be03100154610) [ 38.206807] mpt3sas_cm0: handle(0x2c) sas_address(0xcc1be03100154611) port_type(0x1) [ 38.215695] scsi 10:0:17:0: SATA: handle(0x002c), sas_addr(0xcc1be03100154611), phy(17), device_name(0x5000cca2b3c34289) [ 38.293204] end_device-10:0:17: add: handle(0x002c), sas_addr(0xcc1be03100154611) [ 38.294389] mpt3sas_cm0: handle(0x2d) sas_address(0xcc1be03100154612) port_type(0x1) [ 38.304437] scsi 10:0:18:0: SATA: handle(0x002d), sas_addr(0xcc1be03100154612), phy(18), device_name(0x5000cca2becd9d1c) [ 38.389970] end_device-10:0:18: add: handle(0x002d), sas_addr(0xcc1be03100154612) [ 38.391154] mpt3sas_cm0: handle(0x2e) sas_address(0xcc1be03100154613) port_type(0x1) [ 38.401483] scsi 10:0:19:0: SATA: handle(0x002e), sas_addr(0xcc1be03100154613), phy(19), device_name(0x5000cca2bec8c3a3) [ 38.491219] end_device-10:0:19: add: handle(0x002e), sas_addr(0xcc1be03100154613) [ 38.492407] mpt3sas_cm0: handle(0x2f) sas_address(0xcc1be03100154614) port_type(0x1) [ 38.501773] scsi 10:0:20:0: SATA: handle(0x002f), sas_addr(0xcc1be03100154614), phy(20), device_name(0x5000cca2b3c4afc3) [ 38.583468] end_device-10:0:20: add: handle(0x002f), sas_addr(0xcc1be03100154614) [ 38.585420] mpt3sas_cm0: handle(0x30) sas_address(0xcc1be03100154615) port_type(0x1) [ 38.596504] scsi 10:0:21:0: SATA: handle(0x0030), sas_addr(0xcc1be03100154615), phy(21), device_name(0x5000cca2b3c35cc0) [ 38.679991] end_device-10:0:21: add: handle(0x0030), sas_addr(0xcc1be03100154615) [ 38.682711] mpt3sas_cm0: handle(0x31) sas_address(0xcc1be03100154616) port_type(0x1) [ 38.692968] scsi 10:0:22:0: SATA: handle(0x0031), sas_addr(0xcc1be03100154616), phy(22), device_name(0x5000cca2b3c35804) [ 38.773938] end_device-10:0:22: add: handle(0x0031), sas_addr(0xcc1be03100154616) [ 38.779631] mpt3sas_cm0: handle(0x32) sas_address(0xcc1be03100154617) port_type(0x1) [ 38.789334] scsi 10:0:23:0: SATA: handle(0x0032), sas_addr(0xcc1be03100154617), phy(23), device_name(0x5000cca2bec2f265) [ 38.880027] end_device-10:0:23: add: handle(0x0032), sas_addr(0xcc1be03100154617) [ 38.884685] mpt3sas_cm0: handle(0x21) sas_address(0xcc1be0310015463d) port_type(0x1) [ 38.901648] scsi 10:0:24:0: SES: handle(0x0021), sas_addr(0xcc1be0310015463d), phy(36), device_name(0xcc1be0310015463d) [ 38.919628] end_device-10:0:24: add: handle(0x0021), sas_addr(0xcc1be0310015463d) I was unable to find either the firmware for the expander nor the firmware for the controller. I am attaching diagnostics from the SAS controller in single link to the expander and running a parity check, as i didn't want to risk any more write errors. When it was in dual link mode i would get a flood of the following messages and an increase in UDMA CRC errors: mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) beyonder-nas-diagnostics-20250314-1045.zip

This is a great plugin, I had a dodgy cable and needed to pause a data rebuild. Saved me hours of time I didn't have to be able to pause, reboot and continue. This should definitely get incorporated into mainline unraid.

Switched out the 13900k for a 14900k, same power, same motherboard and all cores enabled. worked fine. 13900k was trash and had a broken core, returning it to amazon as Intel are horrible to deal with.

Okay for all the muppets out there, be aware firmware/bios updates and resetting actually does indeed reset the settings, including not just the stuff you are rabbit holing on fixing. tl;dr bios update reset the igpu, so it was set to auto and passing through ASPEED adapter instead of the actual intel iGPU

This used to work but suddenly stopped (along with plex hardware transcoding), any ideas? root@beyonder-nas:~# intel_gpu_top No device filter specified and no discrete/integrated i915 devices found root@beyonder-nas:~# uname -a Linux beyonder-nas 6.1.126-Unraid #1 SMP PREEMPT_DYNAMIC Sun Jan 19 15:51:34 PST 2025 x86_64 13th Gen Intel(R) Core(TM) i9-13900K GenuineIntel GNU/Linux root@beyonder-nas:~# cat /proc/cmdline BOOT_IMAGE=/bzimage i915.enable_fbc=1 i915.enable_guc=2 nvme_core.default_ps_max_latency_us=0 iommu=pt initrd=/bzroot root@beyonder-nas:~# cat /etc/modprobe.d/i915.conf options i915 enable_fbc=1 enable_guc=2 root@beyonder-nas:~# lspci 00:00.0 Host bridge: Intel Corporation Device a700 (rev 01) 00:01.0 PCI bridge: Intel Corporation Raptor Lake PCI Express 5.0 Graphics Port (PEG010) (rev 01) 00:01.1 PCI bridge: Intel Corporation Device a72d (rev 01) 00:06.0 PCI bridge: Intel Corporation Raptor Lake PCIe 4.0 Graphics Port (rev 01) 00:0a.0 Signal processing controller: Intel Corporation Raptor Lake Crashlog and Telemetry (rev 01) 00:14.0 USB controller: Intel Corporation Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller (rev 11) 00:14.2 RAM memory: Intel Corporation Alder Lake-S PCH Shared SRAM (rev 11) 00:15.0 Serial bus controller: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #0 (rev 11) 00:15.1 Serial bus controller: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #1 (rev 11) 00:16.0 Communication controller: Intel Corporation Alder Lake-S PCH HECI Controller #1 (rev 11) 00:16.3 Serial controller: Intel Corporation Alder Lake-S Keyboard and Text (KT) Redirection (rev 11) 00:17.0 SATA controller: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] (rev 11) 00:1a.0 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #25 (rev 11) 00:1b.0 PCI bridge: Intel Corporation Device 7ac0 (rev 11) 00:1b.4 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #21 (rev 11) 00:1c.0 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #1 (rev 11) 00:1c.1 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #2 (rev 11) 00:1c.3 PCI bridge: Intel Corporation Device 7abb (rev 11) 00:1c.4 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #5 (rev 11) 00:1d.0 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #9 (rev 11) 00:1f.0 ISA bridge: Intel Corporation Device 7a88 (rev 11) 00:1f.3 Audio device: Intel Corporation Alder Lake-S HD Audio Controller (rev 11) 00:1f.4 SMBus: Intel Corporation Alder Lake-S PCH SMBus Controller (rev 11) 00:1f.5 Serial bus controller: Intel Corporation Alder Lake-S PCH SPI Controller (rev 11) 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (17) I219-LM (rev 11) 01:00.0 RAID bus controller: Broadcom / LSI Fusion-MPT 24GSAS/PCIe SAS40xx/41xx (rev 01) 02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3224 PCI-Express Fusion-MPT SAS-3 (rev 01) 03:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN850X NVMe SSD (rev 01) 04:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN850X NVMe SSD (rev 01) 06:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN850X NVMe SSD (rev 01) 07:00.0 PCI bridge: Integrated Technology Express, Inc. IT8893E PCIe to PCI Bridge (rev 41) 09:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-LM (rev 03) 0a:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 06) 0b:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 52) 0c:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO 0d:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO

28% on a parity done with 2 p-cores and 4 e-cores so looking like a faulty core. I will see if this parity completes, then try increase the number of cores and repeat. RMA for intel I guess.

I disabled all but 2 P-cores and 4 E-cores and currently at 6.9% parity, which is further than its got with any other test. Lets see how it goes, but broken intel CPU core you think or still power?

So I'm at a loss: CPU is stable under duress (x265, ng-stress) Memory passes mem test, and is ECC so should be logging if there was bitflipping Bios/motherboard on multiple versions disk read/write seems to be fine... i'm able to transcode and play direct docker is fine, only the main unraid thread crashes parity is under less load than the CPU stress No power events in ipmi or kernel logs I was sure it would be the processor but seems only parity and its got 100% reproduce on that between 1-5% completed. Could it be LSI SAS controller, usb drive or something else and can they be isolated for testing without physical access to the server? Is there a parity like test i can run to simulate a failure?

No, all the original cables and into a sas backplane. The sas backplane isn't super good though. The read/writes to the drives seem to be fine. Unfortunately at 3.8%, its happened again: [ 4079.681264] ------------[ cut here ]------------ [ 4079.681274] kernel BUG at drivers/md/unraid.c:1617! [ 4079.681812] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 4079.682290] CPU: 8 PID: 15720 Comm: unraidd0 Tainted: P O 6.1.99-Unraid #1 [ 4079.682796] Hardware name: Supermicro Super Server/X13SAE-F, BIOS 4.1 10/01/2024 [ 4079.683290] RIP: 0010:unraidd+0x1051/0x1140 [md_mod] [ 4079.683791] Code: 00 83 3d 83 50 00 00 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 e3 7c a0 48 8b 73 20 e8 b2 b7 09 e1 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 10 41 c7 46 b0 00 10 00 00 49 8b 56 10 [ 4079.684820] RSP: 0018:ffffc900205efdf0 EFLAGS: 00010246 [ 4079.685333] RAX: 0000000000000000 RBX: ffff88814a680da8 RCX: 0000000000000000 [ 4079.685854] RDX: 0000000000000000 RSI: ffffffff829ee720 RDI: ffff888108624038 [ 4079.686377] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 [ 4079.686893] R10: 0000000000000000 R11: 0000000000000000 R12: ffff888148f62110 [ 4079.687398] R13: ffff88814a680fa0 R14: ffff88814a681018 R15: ffff8881049152d8 [ 4079.687899] FS: 0000000000000000(0000) GS:ffff889fff800000(0000) knlGS:0000000000000000 [ 4079.688404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4079.688903] CR2: 000015173aee8000 CR3: 000000000420a005 CR4: 0000000000770ee0 [ 4079.689404] PKRU: 55555554 [ 4079.689896] Call Trace: [ 4079.690382] <TASK> [ 4079.690860] ? __die_body+0x1a/0x5c [ 4079.691340] ? die+0x30/0x49 [ 4079.691806] ? do_trap+0x7b/0xfe [ 4079.692274] ? unraidd+0x1051/0x1140 [md_mod] [ 4079.692729] ? unraidd+0x1051/0x1140 [md_mod] [ 4079.693171] ? do_error_trap+0x6e/0x98 [ 4079.693607] ? unraidd+0x1051/0x1140 [md_mod] [ 4079.694044] ? exc_invalid_op+0x4c/0x60 [ 4079.694482] ? unraidd+0x1051/0x1140 [md_mod] [ 4079.694904] ? asm_exc_invalid_op+0x16/0x20 [ 4079.695323] ? unraidd+0x1051/0x1140 [md_mod] [ 4079.695731] md_thread+0xf4/0x122 [md_mod] [ 4079.696132] ? _raw_spin_rq_lock_irqsave+0x20/0x20 [ 4079.696531] ? signal_pending+0x1d/0x1d [md_mod] [ 4079.696922] kthread+0xe4/0xef [ 4079.697317] ? kthread_complete_and_exit+0x1b/0x1b [ 4079.697719] ret_from_fork+0x1f/0x30 [ 4079.698109] </TASK> [ 4079.698487] Modules linked in: xt_nat xt_tcpudp veth xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter ipmi_devintf md_mod xfs tcp_diag inet_diag i915 drm_buddy drm_display_helper intel_gtt iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs macvtap macvlan tap af_packet 8021q garp mrp bridge stp llc bonding tls igc e1000e intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp zfs(PO) kvm_intel kvm ast crct10dif_pclmul drm_vram_helper crc32_pclmul crc32c_intel i2c_algo_bit ghash_clmulni_intel drm_ttm_helper zunicode(PO) sha512_ssse3 sha256_ssse3 ttm zzstd(O) sha1_ssse3 drm_kms_helper zlua(O) aesni_intel zavl(PO) drm crypto_simd icp(PO) agpgart mei_pxp mei_hdcp syscopyarea i2c_i801 cryptd [ 4079.698517] zcommon(PO) rapl znvpair(PO) intel_cstate spl(O) ipmi_ssif wmi_bmof mpt3sas intel_uncore sysfillrect i2c_smbus mei_me nvme input_leds mpi3mr sysimgblt video ahci raid_class acpi_ipmi i2c_core joydev led_class fb_sys_fops mei nvme_core libahci scsi_transport_sas thermal fan wmi ipmi_si backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: igc] [ 4079.705481] ---[ end trace 0000000000000000 ]--- [ 4079.745331] RIP: 0010:unraidd+0x1051/0x1140 [md_mod] [ 4079.745970] Code: 00 83 3d 83 50 00 00 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 e3 7c a0 48 8b 73 20 e8 b2 b7 09 e1 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 10 41 c7 46 b0 00 10 00 00 49 8b 56 10 [ 4079.747220] RSP: 0018:ffffc900205efdf0 EFLAGS: 00010246 [ 4079.747846] RAX: 0000000000000000 RBX: ffff88814a680da8 RCX: 0000000000000000 [ 4079.748451] RDX: 0000000000000000 RSI: ffffffff829ee720 RDI: ffff888108624038 [ 4079.749071] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 [ 4079.749694] R10: 0000000000000000 R11: 0000000000000000 R12: ffff888148f62110 [ 4079.750313] R13: ffff88814a680fa0 R14: ffff88814a681018 R15: ffff8881049152d8 [ 4079.750899] FS: 0000000000000000(0000) GS:ffff889fff800000(0000) knlGS:0000000000000000 [ 4079.751505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4079.752120] CR2: 000015173aee8000 CR3: 000000000420a005 CR4: 0000000000770ee0 [ 4079.752723] PKRU: 55555554 [ 4079.753339] ------------[ cut here ]------------ [ 4079.753955] WARNING: CPU: 8 PID: 15720 at kernel/exit.c:816 do_exit+0x87/0x923 [ 4079.754582] Modules linked in: xt_nat xt_tcpudp veth xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter ipmi_devintf md_mod xfs tcp_diag inet_diag i915 drm_buddy drm_display_helper intel_gtt iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs macvtap macvlan tap af_packet 8021q garp mrp bridge stp llc bonding tls igc e1000e intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp zfs(PO) kvm_intel kvm ast crct10dif_pclmul drm_vram_helper crc32_pclmul crc32c_intel i2c_algo_bit ghash_clmulni_intel drm_ttm_helper zunicode(PO) sha512_ssse3 sha256_ssse3 ttm zzstd(O) sha1_ssse3 drm_kms_helper zlua(O) aesni_intel zavl(PO) drm crypto_simd icp(PO) agpgart mei_pxp mei_hdcp syscopyarea i2c_i801 cryptd [ 4079.754606] zcommon(PO) rapl znvpair(PO) intel_cstate spl(O) ipmi_ssif wmi_bmof mpt3sas intel_uncore sysfillrect i2c_smbus mei_me nvme input_leds mpi3mr sysimgblt video ahci raid_class acpi_ipmi i2c_core joydev led_class fb_sys_fops mei nvme_core libahci scsi_transport_sas thermal fan wmi ipmi_si backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: igc] [ 4079.763171] CPU: 8 PID: 15720 Comm: unraidd0 Tainted: P D O 6.1.99-Unraid #1 [ 4079.764063] Hardware name: Supermicro Super Server/X13SAE-F, BIOS 4.1 10/01/2024 [ 4079.764958] RIP: 0010:do_exit+0x87/0x923 [ 4079.765855] Code: 24 74 04 75 13 b8 01 00 00 00 41 89 6c 24 60 48 c1 e0 22 49 89 44 24 70 4c 89 ef e8 d1 2b 81 00 48 83 bb b0 07 00 00 00 74 02 <0f> 0b 48 8b bb d8 06 00 00 e8 d3 2a 81 00 48 8b 83 d0 06 00 00 83 [ 4079.767710] RSP: 0018:ffffc900205efee0 EFLAGS: 00010286 [ 4079.768635] RAX: 0000000000000000 RBX: ffff888109c4f000 RCX: 0000000000000000 [ 4079.769571] RDX: 0000000000000001 RSI: 0000000000002710 RDI: 00000000ffffffff [ 4079.770509] RBP: 000000000000000b R08: 0000000000000000 R09: ffffc900037e5020 [ 4079.771442] R10: 0000000000aaaaaa R11: 0000000000000001 R12: ffff888104636c00 [ 4079.772362] R13: ffff88814822dac0 R14: 0000000000000002 R15: ffffffff820b2ea5 [ 4079.773257] FS: 0000000000000000(0000) GS:ffff889fff800000(0000) knlGS:0000000000000000 [ 4079.774142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4079.775011] CR2: 000015173aee8000 CR3: 000000000420a005 CR4: 0000000000770ee0 [ 4079.775867] PKRU: 55555554 [ 4079.776700] Call Trace: [ 4079.777510] <TASK> [ 4079.778290] ? __warn+0xab/0x122 [ 4079.779051] ? report_bug+0x109/0x17e [ 4079.779793] ? do_exit+0x87/0x923 [ 4079.780513] ? handle_bug+0x41/0x6f [ 4079.781215] ? exc_invalid_op+0x13/0x60 [ 4079.781914] ? asm_exc_invalid_op+0x16/0x20 [ 4079.782601] ? do_exit+0x87/0x923 [ 4079.783272] make_task_dead+0x11c/0x11c [ 4079.783934] rewind_stack_and_make_dead+0x17/0x17 [ 4079.784598] RIP: 0000:0x0 [ 4079.785257] Code: Unable to access opcode bytes at 0xffffffffffffffd6. [ 4079.785925] RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX: 0000000000000000 [ 4079.786611] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 4079.787290] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 4079.787953] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 4079.788609] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 4079.789252] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 4079.789882] </TASK> 4.1 bios didn't help. No other errors but the parity is stuck/crashed and i can only reboot. The system hasn't hung/dockers are working fine.

I think the power draw is the same/higher with a x265 encode (~450w), but I guess a parity check would be more lines (LSI card, all the drives, cpu, etc.) and could be susceptible to that. Its a Corsair 1000w platinum so I'd pretty shuck with my view of corsair and their PSUs if its this, they've been really solid for me even in server builds. I do have a replacement PSU spare but don't have access to the server until march to switch it. I kicked off a new parity check with the updated bios, 4.1, and its got to 2.5% (not out of the woods by a long shot but further than it got the last two times). UPS load is currently ~ 360w and CPU isn't running as hot. Let's hope its a motherboard / bios issue with 3.3b. Really wish Supermicro would provide release notes