Whaler_99

I have used that one, similar to all the others, good build quality. I just not a big fan of trayless as the drives are a bit loose in the cage. I've had more drive issues with trayless cages than any others. As for the fan, unfortunately I cannot recall if you can swap it. There is a speed selector for it and it is a standard 80mm size, so there is that. You might just have to solder if it's a custom connector on the PCB.

Reboot and all fixed, thanks @JorgeB

OK, thanks. I'll wait till current rebuild is done and then reboot.

On last monthly parity check one of my disks threw a ton of errors and was disabled. I shut down today, swapped disks, booted up, assigned replacement, started rebuild / parity check. Now though the VM won't start and was working fine before I shut the server down. Attached the diag file, thanks for assistance! tower-diagnostics-20220704-1722.zip

where outside of toronto roughly? Just wondering of far I'd need to travel to pick up.

Fixed things up, enabled write cache on the disks, moved the controller into the top slot, much better looking now.

Going to shut down later today when home from work, swap the controller with the GPU, as the GPU doesn't need x16 throughput and then test and see what happens. I also found the post on enabling the write cache, so will work on that as well.

Some weird numbers... Doing same test on the internal controller shows single and multi tests almost identical...

running disk speed tests now, shows all signaling at 6Gb on all drives. Curiously, I noticed this in the log: May 16 07:45:00 Tower root: Fix Common Problems: Warning: Docker Application binhex-plex has an update available for it May 16 07:45:00 Tower root: Fix Common Problems: Warning: Docker Application duckdns has an update available for it May 16 07:45:04 Tower root: Fix Common Problems: Warning: preclear.disk.plg Not Compatible with Unraid version 6.9.2 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk2 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk3 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk4 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk5 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk6 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk7 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk8 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk9 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk10 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk11 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk12 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk13 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk14 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk15 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk16 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk17 May 16 07:45:08 Tower root: Fix Common Problems: Other Warning: Background notifications not enabled This is coming up since I swapped to this new controller and three disks are not listed, the three on the internal controller. I wonder if not have the cache enabled on all these disks could be impacting the throughput.

OK, I'll wait the day+ and see if the check finishes and then try that

Huh... seems to be some sort of bandwidth block, the estimated speed never goes above the 40'ish MB/s mark... The only thing I can think of, since it's the LSI card and cables I changed, something is up with the settings on the card itself. Although when going through it, there isn't a lot of options even on the card to set.

Yes, I previously had dual LSI 8 port internal cards, which when doing a parity check, I'd get 100Mb/s.... So, dug a little deeper, looks like I ordered the wrong cables. https://www.amazon.ca/gp/product/B013G4EX9K/ref=ppx_yo_dt_b_asin_title_o01_s01?ie=UTF8&psc=1 SAS1 cables... not SAS2....

<sigh> frigging mobo manufacturers... checked manual, sure enough, then second x16 slot is limited to @x4 speeds... like why bother, lol? Looks like I might try and find another motherboard with two slots running at x8... Thanks for your help @JorgeB

It is in the second PCIe slot, so should be x8 on that board. Here are the results. 05:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02) Subsystem: Broadcom / LSI 9201-16e 6Gb/s SAS/SATA PCIe x8 External HBA Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 29 IOMMU group: 12 Region 0: I/O ports at d000 [size=256] Region 1: Memory at d0300000 (64-bit, non-prefetchable) [size=16K] Region 3: Memory at d0100000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at <ignored> [disabled] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend+ LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s (ok), Width x4 (downgraded) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [d0] Vital Product Data pcilib: sysfs_read_vpd: read failed: Input/output error Not readable Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable+ Count=15 Masked- Vector table: BAR=1 offset=00002000 PBA: BAR=1 offset=00003800 Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [138 v1] Power Budgeting <?> Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy- IOVSta: Migration- Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00 VF offset: 1, stride: 1, Device ID: 0064 Supported Page Size: 00000553, System Page Size: 00000001 Region 0: Memory at 00000000d0304000 (64-bit, non-prefetchable) Region 2: Memory at 00000000d0140000 (64-bit, non-prefetchable) VF Migration: offset: 00000000, BIR: 0 Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Kernel driver in use: mpt3sas Kernel modules: mpt3sas

Recently had some issues so I rebuilt my system with a new LSI SAS9201-16 e card. It is running the p20 firmware. I'm getting a throughput average of around 40MB/sec. I'm wondering if this is normal speed seen by others with this card or is this due to a dual disk rebuild? Yep, I run dual parity and two drives seemed to have failed, so it is rebuilding those now. Diag attached, thanks! tower-diagnostics-20220516-0740.zip

Quick update, after waiting for shipping, I finally got an LSI 16 port card that would work with the motherboard. Plugged that in, cabled everything up, booted up, ran parity and rebuild and everything is fine now.

Anyone know of a currently available, in Canada, motherboard that has two PCIe slots (either both x16 or x16 and x8 form factor) both running at x8 speed? Seems like SO many boards with multiple x16 slots, the second slots are wired for x4, or even x1, speed only... I know like the Asus Prime x570 does this, but its over $300... hoping to save the budget a bit. Thanks!

Thanks, and yes, I double checked power and data connections, even though when I initially swapped the disks, I didn't even open the case, they are all in 5x3 cages. Maybe get my power supply tester and test all the leads, who knows, maybe something is flaky there. Re: the SAS2LP cards, I knew about those. I previously bought LSI cards flashed to IT mode to replace but when I initially tried it, the mobo wouldn't recognize the cards. Think I will also try and get that resolved, I have another mobo I use...

System has been running fine. Most recent parity check, about two weeks ago, zero issues. Yesterday, decided to upgrade a 2TB drive to 4TB drive. Noted the S/N, unassigned, shut down system. I only have two 2TB disks left installed from among all the drives. So, pulled out one, wrong one, slotted back in, pull out other, replaced. Power up, assign, start rebuild. Now, after about 6% rebuild, the old 2TB disk starts generating millions of disk reads. I shut the server down, pulled the drive, make sure seated properly, check power and data cables, etc. But, nothing there should have changed. Power back up, go into maintenance mode, run XFS check, everything looks fine. Also did a SMART check, all clear. Start another rebuild, get to 20% or so this time, bam, millions of disk read errors on the 2TB unit. Not sure what the heck is going on with this disk suddenly. Attached DIAG if anyone can take a look and maybe point me in some direction. towerbackup-diagnostics-20220420-1127.zip

Not sure what you can get in EU, here in NA our options are limited as well as things are either EOL or out of stock. And yes, the manufacturers for the 3x5 cages are dwindling, just not enough of a market for it. Best options are looking at secondary markets, either local or EBay or such in your area. A year ago I found 4 of the istar 5x3 trayless units on kijiji, a secondary market site popular here in Canada.

I found that with a lot of the 3x5 units, the actual drive cage for each disk can be pretty flimsy...

No, I have not used it for a while. It was more of a gimmick if you have a bunch of laptop drives having around. Sure you could fit 6 laptop/2.5 drives in one slot, but it then ended up using 6 SATA ports. Performance isn't great on these drives and space... even now a 2TB laptop drive is big... Temps though were always fine for me. So, if you have 6 laptop drives and 6 spare SATA ports and no way to hook up another drive? Ok... but think, even with an adapter, you could put one 10TB 3.5 drive in the same bay, and would be better overall.

Yep, was two years out of date, updated it to the latest. Drive rebuild all complete and successful. Everything working ok again! Thanks @itimpi @trurl and @JorgeB

Thanks for the info! I'll get that done asap!

Thanks! That is what I am doing.