GoChris

Members
  • Posts

    232
  • Joined

  • Last visited

Posts posted by GoChris

  1. Been having at least a week of this now I think, but every morning between 4 and 5am the server becomes unresponsive. Almost every time I can't ssh to it at all, and I have to hard reset.

     

    This morning I was able to SSH into it, the web ui only loaded the banner the first time I tried, but nothing after any subsequent tries. There was not much ram available, maybe ~600MB according to top. I did try to stop dockers, and most did, but they weren't using much ram according to docker status. Plex was using ~12GB which would not stop when using docker stop. The system has 64GB of ram. One time last week when I was able to ssh into it (about the only other time I could), top was showing kswapd using 100% cpu. On that day reboot never worked.

     

    Today /sbin/reboot did eventually reboot, however it was still considered an unclean shutdown. Before today I did increase some shutdown timeouts to ~5minutes. I did manage to get the diagnostics via ssh today before it rebooted, and I have attached. I would appreciate any insight. One last thing, I did upgrade to 6.11.5 last week and I think that was when I started having these problems so I downgraded, and still have the problems. I could be mistaken that the problems occurred before I upgraded.

    tower-diagnostics-20230227-0720.zip

  2. I'm also having a problem with the disk background color, it remains white when using the "black" theme no matter what I change the background colors too. I have tried it in firefox and chrome. What info can I provide to help troubleshoot? The information tab list also has a white background, making all the text basically unreadable.

  3. 1 minute ago, dlandon said:

    Try booting in safe mode.

    I did. That's how I eventually figured out what plugin was causing it. I removed all of them, booted up in regular mode, added a few plugins one at a time with reboots in between. After adding UD it will not boot fully. I hooked up a monitor and the last thing was on screen was something about starting samba, that was it.

  4. I did a scan at 25% intervals using the Disk Speed container. The slowest 4TB drive does indeed slow to ~80MB/s at the 4TB mark. As well as one 8TB that slows to ~88MB/s at it's end. So perhaps the slow performance of those at their edges are what's dropping the average down so much.

  5. 2 hours ago, JorgeB said:

    Something else is the problem, is 116.5MB/s the average speed or the starting speed? i.e., if you start a parity check a what speed does it start? Make sure nothing else is using the array.

     

    Alright in the first 5 minutes, it was between ~162 to 172MB/s, seemed pretty stable once it got going in the high 160s. I know drives slow down on the outer edges, so perhaps some of the older 4TB drives are quite slow? I will do some speed tests on all of them at different positions to see if anyone individually is slow. Average still seems low compared to the starting speed. I thought most dockers were stopped during the last rebuild, but I can't be certain now.

  6. 2 hours ago, JorgeB said:

    HBA links are OK, expander is also using dual link

     

    Something else is the problem, is 116.5MB/s the average speed or the starting speed? i.e., if you start a parity check a what speed does it start? Make sure nothing else is using the array.

    That was the avg speed from the history, so it would be faster at one point. I will do a test and report back after stopping any activity.

  7. Quote

    Check the HBAs link speed and with, you can do that with

    09:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
            Subsystem: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
            Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
            Latency: 0, Cache Line Size: 64 bytes
            Interrupt: pin A routed to IRQ 81
            IOMMU group: 17
            Region 0: I/O ports at e000 [size=256]
            Region 1: Memory at d04c0000 (64-bit, non-prefetchable) [size=16K]
            Region 3: Memory at d0080000 (64-bit, non-prefetchable) [size=256K]
            Expansion ROM at d0000000 [disabled] [size=512K]
            Capabilities: [50] Power Management version 3
                    Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                    Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
            Capabilities: [68] Express (v2) Endpoint, MSI 00
                    DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                    DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                            MaxPayload 512 bytes, MaxReadReq 512 bytes
                    DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                    LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns
                            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                    LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                    LnkSta: Speed 5GT/s (ok), Width x8 (ok)
                            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                    DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR-
                             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                             FRS- TPHComp- ExtTPHComp-
                             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                    DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                             AtomicOpsCtl: ReqEn-
                    LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                             Compliance De-emphasis: -6dB
                    LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                             Retimer- 2Retimers- CrosslinkRes: unsupported
            Capabilities: [d0] Vital Product Data
    pcilib: sysfs_read_vpd: read failed: No such device
                    Not readable
            Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
                    Address: 0000000000000000  Data: 0000
            Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
                    Vector table: BAR=1 offset=00002000
                    PBA: BAR=1 offset=00003800
            Capabilities: [100 v1] Advanced Error Reporting
                    UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                    UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                    UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                    CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                    CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                    AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                    HeaderLog: 00000000 00000000 00000000 00000000
            Capabilities: [138 v1] Power Budgeting <?>
            Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV)
                    IOVCap: Migration-, Interrupt Message Number: 000
                    IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
                    IOVSta: Migration-
                    Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
                    VF offset: 1, stride: 1, Device ID: 0072
                    Supported Page Size: 00000553, System Page Size: 00000001
                    Region 0: Memory at 00000000d04c4000 (64-bit, non-prefetchable)
                    Region 2: Memory at 00000000d00c0000 (64-bit, non-prefetchable)
                    VF Migration: offset: 00000000, BIR: 0
            Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
                    ARICap: MFVC- ACS-, Next Function: 0
                    ARICtl: MFVC- ACS-, Function Group: 0
            Kernel driver in use: mpt3sas
            Kernel modules: mpt3sas
    
    0a:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
            Subsystem: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
            Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
            Latency: 0, Cache Line Size: 64 bytes
            Interrupt: pin A routed to IRQ 89
            IOMMU group: 18
            Region 0: I/O ports at d000 [size=256]
            Region 1: Memory at d0ac0000 (64-bit, non-prefetchable) [size=16K]
            Region 3: Memory at d0680000 (64-bit, non-prefetchable) [size=256K]
            Expansion ROM at d0600000 [disabled] [size=512K]
            Capabilities: [50] Power Management version 3
                    Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                    Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
            Capabilities: [68] Express (v2) Endpoint, MSI 00
                    DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                    DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                            MaxPayload 512 bytes, MaxReadReq 512 bytes
                    DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                    LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns
                            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                    LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                    LnkSta: Speed 5GT/s (ok), Width x8 (ok)
                            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                    DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR-
                             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                             FRS- TPHComp- ExtTPHComp-
                             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                    DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                             AtomicOpsCtl: ReqEn-
                    LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                             Compliance De-emphasis: -6dB
                    LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                             Retimer- 2Retimers- CrosslinkRes: unsupported
            Capabilities: [d0] Vital Product Data
    pcilib: sysfs_read_vpd: read failed: No such device
                    Not readable
            Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
                    Address: 0000000000000000  Data: 0000
            Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
                    Vector table: BAR=1 offset=00002000
                    PBA: BAR=1 offset=00003800
            Capabilities: [100 v1] Advanced Error Reporting
                    UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                    UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                    UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                    CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                    CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                    AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                    HeaderLog: 00000000 00000000 00000000 00000000
            Capabilities: [138 v1] Power Budgeting <?>
            Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV)
                    IOVCap: Migration-, Interrupt Message Number: 000
                    IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
                    IOVSta: Migration-
                    Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
                    VF offset: 1, stride: 1, Device ID: 0072
                    Supported Page Size: 00000553, System Page Size: 00000001
                    Region 0: Memory at 00000000d0ac4000 (64-bit, non-prefetchable)
                    Region 2: Memory at 00000000d06c0000 (64-bit, non-prefetchable)
                    VF Migration: offset: 00000000, BIR: 0
            Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
                    ARICap: MFVC- ACS-, Next Function: 0
                    ARICtl: MFVC- ACS-, Function Group: 0
            Kernel driver in use: mpt3sas
            Kernel modules: mpt3sas

     

    Quote

    Also check that the expander is using dual link

     

    root@Tower:~# cat /sys/class/sas_host/host11/device/port-11\:0/sas_port/port-11\:0/num_phys
    8
    root@Tower:~# cat /sys/class/sas_host/host12/device/port-12\:0/sas_port/port-12\:0/num_phys
    1

     

    I have also attached diagnostics. I appreciate your help immensely.

    tower-diagnostics-20220829-1140.zip

  8. 5 minutes ago, JorgeB said:

    If you are getting lower speeds than that during rebuilds/parity check HBAs are not the problem.

    I am yes, alright I wonder what it could be. I rebuilt a 10TB a few days ago @ 116.5 MB/s. Good to know.

    CPU is an AMD Ryzen 7 3800X

  9. 18 minutes ago, JorgeB said:

    Those are PCIe 2.0, a PCIe 3.0 model connected to the expander with dual link would increase total bandwidth from the currently usable 3GB/s to 4.8GB/s, assuming of course the board/CPU supports PCIe 3.0.

    Yes, the board does, thanks for all the info so far.

     

    So in theory if all drives on the backplanes off the expander are populated (16), currently at 3GB/s would result in a max of 192MB/s per drive? If so, doesn't seem like an upgrade would net much with current HDD speeds.

  10. 6 hours ago, JorgeB said:

    This is SAS2, so having a SAS3 HBA won't change the link speed, it will still help if your board/CPU supports PCIe 3.0, same as a SAS2 HBA with PCIe 3.0 support would, like for example the 9207-8i.

    Both cards currently are LSI SAS2008 based.

  11. 6 hours ago, JorgeB said:

    Which Intel expander? Also what model disks?

     

    Intel RES2SV240 expander

     

    Drives are a combination of:

    HGST Deskstar NAS

    Seagate Ironwolf (NAS and Pros)

    And just some Seagate NAS and shucked desktop drives, might have a couple SMRs in there.

    Not deployed yet to upgrade some 4tb: WD Gold, Toshiba X300s (desktop performace they say)

     

    Edit: I do have a 2nd expander (same model) that's unused, unsure if it would help in any way.

  12. Wondering if upgrading my HBA SAS 6G cards will net me more speed in rebuilds and parity checks.

     

    Case: Norco 4224, 6 backplanes, I'm told they are SAS 3 capable. Drives on all backplanes.

    Drives: All 7200rpm

    Current HBAs: Two 2-port sas2 cards.

         Card 1: both ports -> intel expander -> 4 ports to backplanes

         Card 2: both ports to remaining 2 backplanes

     

    Will upgrading to a SAS3 12Gbps card actually increase the hdd speeds at all?

     

    Cheers

  13. On 6/10/2022 at 11:25 AM, je82 said:

    not to sure about this crazymax thing, no support thread, rutorrent actually doesn't work great, "remove" action in crazymax version deletes the torrent + the data and not just the torrent, not great but i'll try to recompile my own version.

    I just deleted 2 torrents, and it deleted my entire share, luckily it wasn't a big share, but about 1TB of data. It deletes the root folder of the torrent so if you change location that can be bad.

    So yes, deleting a torrent deletes the data, moving a torrent redownloads, it's just a bug fest of a container. do not recommend.