Whaler_99

Going to shut down later today when home from work, swap the controller with the GPU, as the GPU doesn't need x16 throughput and then test and see what happens. I also found the post on enabling the write cache, so will work on that as well.

Some weird numbers... Doing same test on the internal controller shows single and multi tests almost identical...

running disk speed tests now, shows all signaling at 6Gb on all drives. Curiously, I noticed this in the log: May 16 07:45:00 Tower root: Fix Common Problems: Warning: Docker Application binhex-plex has an update available for it May 16 07:45:00 Tower root: Fix Common Problems: Warning: Docker Application duckdns has an update available for it May 16 07:45:04 Tower root: Fix Common Problems: Warning: preclear.disk.plg Not Compatible with Unraid version 6.9.2 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk2 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk3 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk4 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk5 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk6 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk7 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk8 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk9 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk10 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk11 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk12 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk13 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk14 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk15 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk16 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk17 May 16 07:45:08 Tower root: Fix Common Problems: Other Warning: Background notifications not enabled This is coming up since I swapped to this new controller and three disks are not listed, the three on the internal controller. I wonder if not have the cache enabled on all these disks could be impacting the throughput.

OK, I'll wait the day+ and see if the check finishes and then try that

Huh... seems to be some sort of bandwidth block, the estimated speed never goes above the 40'ish MB/s mark... The only thing I can think of, since it's the LSI card and cables I changed, something is up with the settings on the card itself. Although when going through it, there isn't a lot of options even on the card to set.

Yes, I previously had dual LSI 8 port internal cards, which when doing a parity check, I'd get 100Mb/s.... So, dug a little deeper, looks like I ordered the wrong cables. https://www.amazon.ca/gp/product/B013G4EX9K/ref=ppx_yo_dt_b_asin_title_o01_s01?ie=UTF8&psc=1 SAS1 cables... not SAS2....

<sigh> frigging mobo manufacturers... checked manual, sure enough, then second x16 slot is limited to @x4 speeds... like why bother, lol? Looks like I might try and find another motherboard with two slots running at x8... Thanks for your help @JorgeB

It is in the second PCIe slot, so should be x8 on that board. Here are the results. 05:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02) Subsystem: Broadcom / LSI 9201-16e 6Gb/s SAS/SATA PCIe x8 External HBA Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 29 IOMMU group: 12 Region 0: I/O ports at d000 [size=256] Region 1: Memory at d0300000 (64-bit, non-prefetchable) [size=16K] Region 3: Memory at d0100000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at <ignored> [disabled] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend+ LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s (ok), Width x4 (downgraded) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [d0] Vital Product Data pcilib: sysfs_read_vpd: read failed: Input/output error Not readable Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable+ Count=15 Masked- Vector table: BAR=1 offset=00002000 PBA: BAR=1 offset=00003800 Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [138 v1] Power Budgeting <?> Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy- IOVSta: Migration- Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00 VF offset: 1, stride: 1, Device ID: 0064 Supported Page Size: 00000553, System Page Size: 00000001 Region 0: Memory at 00000000d0304000 (64-bit, non-prefetchable) Region 2: Memory at 00000000d0140000 (64-bit, non-prefetchable) VF Migration: offset: 00000000, BIR: 0 Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Kernel driver in use: mpt3sas Kernel modules: mpt3sas

Recently had some issues so I rebuilt my system with a new LSI SAS9201-16 e card. It is running the p20 firmware. I'm getting a throughput average of around 40MB/sec. I'm wondering if this is normal speed seen by others with this card or is this due to a dual disk rebuild? Yep, I run dual parity and two drives seemed to have failed, so it is rebuilding those now. Diag attached, thanks! tower-diagnostics-20220516-0740.zip

Quick update, after waiting for shipping, I finally got an LSI 16 port card that would work with the motherboard. Plugged that in, cabled everything up, booted up, ran parity and rebuild and everything is fine now.

Anyone know of a currently available, in Canada, motherboard that has two PCIe slots (either both x16 or x16 and x8 form factor) both running at x8 speed? Seems like SO many boards with multiple x16 slots, the second slots are wired for x4, or even x1, speed only... I know like the Asus Prime x570 does this, but its over $300... hoping to save the budget a bit. Thanks!

Thanks, and yes, I double checked power and data connections, even though when I initially swapped the disks, I didn't even open the case, they are all in 5x3 cages. Maybe get my power supply tester and test all the leads, who knows, maybe something is flaky there. Re: the SAS2LP cards, I knew about those. I previously bought LSI cards flashed to IT mode to replace but when I initially tried it, the mobo wouldn't recognize the cards. Think I will also try and get that resolved, I have another mobo I use...

System has been running fine. Most recent parity check, about two weeks ago, zero issues. Yesterday, decided to upgrade a 2TB drive to 4TB drive. Noted the S/N, unassigned, shut down system. I only have two 2TB disks left installed from among all the drives. So, pulled out one, wrong one, slotted back in, pull out other, replaced. Power up, assign, start rebuild. Now, after about 6% rebuild, the old 2TB disk starts generating millions of disk reads. I shut the server down, pulled the drive, make sure seated properly, check power and data cables, etc. But, nothing there should have changed. Power back up, go into maintenance mode, run XFS check, everything looks fine. Also did a SMART check, all clear. Start another rebuild, get to 20% or so this time, bam, millions of disk read errors on the 2TB unit. Not sure what the heck is going on with this disk suddenly. Attached DIAG if anyone can take a look and maybe point me in some direction. towerbackup-diagnostics-20220420-1127.zip

Not sure what you can get in EU, here in NA our options are limited as well as things are either EOL or out of stock. And yes, the manufacturers for the 3x5 cages are dwindling, just not enough of a market for it. Best options are looking at secondary markets, either local or EBay or such in your area. A year ago I found 4 of the istar 5x3 trayless units on kijiji, a secondary market site popular here in Canada.

I found that with a lot of the 3x5 units, the actual drive cage for each disk can be pretty flimsy...

No, I have not used it for a while. It was more of a gimmick if you have a bunch of laptop drives having around. Sure you could fit 6 laptop/2.5 drives in one slot, but it then ended up using 6 SATA ports. Performance isn't great on these drives and space... even now a 2TB laptop drive is big... Temps though were always fine for me. So, if you have 6 laptop drives and 6 spare SATA ports and no way to hook up another drive? Ok... but think, even with an adapter, you could put one 10TB 3.5 drive in the same bay, and would be better overall.

Yep, was two years out of date, updated it to the latest. Drive rebuild all complete and successful. Everything working ok again! Thanks @itimpi @trurl and @JorgeB

Thanks for the info! I'll get that done asap!

Thanks! That is what I am doing.

Yep, I replicate the entire array to a secondary system, just in case.

Started up, both drives red balled - Disabled, being emulated. So that's an improvement. New DIag attached, thanks for assistance @trurl! tower-diagnostics-20211107-1417.zip

Looks to have run ok. Here are results from Disk 15 after running repair and then another Check. Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 3 - agno = 4 - agno = 2 - agno = 5 - agno = 1 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 3 - agno = 4 - agno = 2 - agno = 5 - agno = 1 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. Also here are update DIags. Unfortunately I don't have extra 6TB drives. I have one 8TB as a spare for the Parity disks. tower-diagnostics-20211107-1409.zip

That is what I'm looking at now, thanks! Looks like there are issues.... Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x439496, xfs_agf block 0x7ffffff9/0x200 agf has bad CRC for ag 1 Metadata CRC error detected at 0x43cfad, xfs_bnobt block 0x80000000/0x1000 btree block 1/1 is suspect, error -74 Metadata CRC error detected at 0x43cfad, xfs_cntbt block 0x80000008/0x1000 btree block 1/2 is suspect, error -74 out-of-order cnt btree record 25 (195207574 483) block 1/2 out-of-order cnt btree record 26 (48588613 610) block 1/2 out-of-order cnt btree record 27 (22270440 653) block 1/2 out-of-order cnt btree record 28 (212725104 734) block 1/2 out-of-order cnt btree record 29 (68889253 763) block 1/2 out-of-order cnt btree record 30 (69842275 976) block 1/2 out-of-order cnt btree record 31 (408138 1065) block 1/2 out-of-order cnt btree record 32 (75921534 1140) block 1/2 out-of-order cnt btree record 33 (11907778 1249) block 1/2 out-of-order cnt btree record 34 (65323505 1402) block 1/2 out-of-order cnt btree record 35 (17823671 1554) block 1/2 out-of-order cnt btree record 36 (13061415 1664) block 1/2 out-of-order cnt btree record 37 (410865 1803) block 1/2 out-of-order cnt btree record 38 (14734129 1971) block 1/2 out-of-order cnt btree record 39 (71076551 2176) block 1/2 out-of-order cnt btree record 40 (55598632 2674) block 1/2 out-of-order cnt btree record 41 (66915612 2723) block 1/2 out-of-order cnt btree record 42 (15412807 2926) block 1/2 out-of-order cnt btree record 43 (34874023 3272) block 1/2 out-of-order cnt btree record 44 (15434037 3499) block 1/2 out-of-order cnt btree record 45 (21234912 3545) block 1/2 out-of-order cnt btree record 46 (96350505 3640) block 1/2 out-of-order cnt btree record 47 (41596365 4210) block 1/2 out-of-order cnt btree record 48 (254222786 4228) block 1/2 out-of-order cnt btree record 49 (56382107 4640) block 1/2 out-of-order cnt btree record 50 (116355071 4842) block 1/2 out-of-order cnt btree record 51 (67565316 5035) block 1/2 out-of-order cnt btree record 52 (196228737 5072) block 1/2 out-of-order cnt btree record 53 (48635751 5179) block 1/2 out-of-order cnt btree record 54 (59835462 5193) block 1/2 out-of-order cnt btree record 55 (63738351 5424) block 1/2 out-of-order cnt btree record 56 (16719899 5864) block 1/2 out-of-order cnt btree record 57 (35530273 6147) block 1/2 out-of-order cnt btree record 58 (268428864 6591) block 1/2 out-of-order cnt btree record 59 (37604543 6854) block 1/2 out-of-order cnt btree record 60 (69367424 6866) block 1/2 out-of-order cnt btree record 61 (64078359 7016) block 1/2 out-of-order cnt btree record 62 (69657714 7222) block 1/2 out-of-order cnt btree record 63 (199624535 7261) block 1/2 out-of-order cnt btree record 64 (16698834 7369) block 1/2 out-of-order cnt btree record 65 (65772445 7645) block 1/2 out-of-order cnt btree record 66 (34775739 7789) block 1/2 out-of-order cnt btree record 67 (18257967 8398) block 1/2 out-of-order cnt btree record 68 (19068571 8580) block 1/2 out-of-order cnt btree record 69 (392018 8644) block 1/2 out-of-order cnt btree record 70 (49514736 8874) block 1/2 out-of-order cnt btree record 71 (98503461 9154) block 1/2 out-of-order cnt btree record 72 (64967088 9377) block 1/2 out-of-order cnt btree record 73 (11450231 9611) block 1/2 out-of-order cnt btree record 74 (266755081 10222) block 1/2 out-of-order cnt btree record 75 (210009117 12488) block 1/2 out-of-order cnt btree record 76 (54254618 12559) block 1/2 out-of-order cnt btree record 77 (66681910 12934) block 1/2 out-of-order cnt btree record 78 (122412185 13163) block 1/2 out-of-order cnt btree record 79 (194016293 15256) block 1/2 out-of-order cnt btree record 80 (66762172 15968) block 1/2 out-of-order cnt btree record 81 (56180134 16629) block 1/2 out-of-order cnt btree record 82 (194582706 16801) block 1/2 out-of-order cnt btree record 83 (226786945 17666) block 1/2 out-of-order cnt btree record 84 (66461810 17865) block 1/2 out-of-order cnt btree record 85 (200768603 17936) block 1/2 out-of-order cnt btree record 86 (186525262 19021) block 1/2 out-of-order cnt btree record 87 (224943003 22652) block 1/2 out-of-order cnt btree record 88 (65862874 23072) block 1/2 out-of-order cnt btree record 89 (64823156 24092) block 1/2 out-of-order cnt btree record 90 (199479370 25987) block 1/2 out-of-order cnt btree record 91 (223888721 33049) block 1/2 out-of-order cnt btree record 92 (225972644 41700) block 1/2 agf_freeblks 27127499, counted 27153867 in ag 1 sb_fdblocks 374735719, counted 374919015 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 data fork in ino 2150484118 claims free block 297875666 data fork in ino 2150770520 claims free block 273513335 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... free space (1,11450231-11459841) only seen by one free space btree free space (1,19068114-19068569) only seen by one free space btree - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 3 - agno = 2 - agno = 4 - agno = 5 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. I'm running this from the GUI, I assume I now remove the -n option, run check again and it should repair things?

@itimpiPowered back up, those two drives now show "unmountable: not mounted" but everything else looks fine. Dockers back up, VM back up, etc... Latest Diag attached. tower-diagnostics-20211107-1338.zip

@itimpi System shut down, yep, those two drives and both SDD's (cache) are attached to the motherboard ports. Weirdly so are the parity drives, which seem fine. Also, all six of the drives are all in different cages and power connections, so definitely something with the mobo last night...