Random slowdowns with system

[2dumb2care]

So I have been having strange slowness issues with unraid ever since I decided to give it a try 4 months or so a go. Now since this was a second hand system from ebay I'm assuming not everything it 100%. My core issue the whole array becomes super slow out of no where for no reason. And when I say slow I mean it, anything from accessing my media share from a Windows computers, Docker Images freeze and lock up, the webUI takes ages to load content, parity syncs crawl (in the kbps), etc... This could last 2 minutes to 12 hours if I let it go that long and it seems every other reboot doesn't speed it up. I thought this was some issue with my cache drive and raid card not auto selecting the correct number of lanes so I tried hard setting them in the BIOS, so that they aren't defaulting to something low like x1. Not sure if that is really a thing that can happen but someone posted it here in another thread so gave it a shot.

 

I have tired booting into memtest however it doesn't seem to make any progress, not sure if its the quad socket board causing it to hang. I am meaning to make a different boot USB with the latest release of memtest to make sure its not some memory issue and I will update my post when I can.  I have also tried booting the system into save mode (no plugins) and that didn't seem to make a difference. I also tried disabling VM and docker all together, as a test since parity sync was expected to complete in 3 months. Early on, I tried moving my install to another USB as the one that used initially was old and I thought it as the issue. I am not the greatest person with linux, I live in the windows world.

 

System Details:

 

Primary Specs:

OS: Version: 6.7.2 

CPU: 4x Intel Xeon E5-4640 V1 8 Core 2.4Ghz (TOTAL 32 Cores)

RAM: 256GB 32x 8GB PC3-10600R ECC REG

Cache Drive: Crucial P1 500GB 3D NAND NVMe PCIe M.2 SSD - CT500P1SSD8

RAID Controller in HBA Mode: Microsemi Adaptec RAID 72405 Firmware Build 32118 - 2 May 2018

Supermicor 4U 24x 3.5" Drive Bays 1 Nodes
Server Chassis/ Case: CSE-848A-R1K62B
Motherboard: X9QRi-F+ Rev 1.20A
Backplane: BPN-SAS-846A 24-port 4U SAS 6Gbps direct-attached backplane,

 

Baseboard info from IPMI:

FRU Information

FRU Device ID: 0

Chassis Info:

Chassis Type: Other

Chassis Part Number: CSE-848TS-R1K62BP

Chassis Serial Number: C8480AC18M20112

Board Manufacturer: Supermicro

Board Serial Num: WM13CS000302

Board Part Num: X9QR7-TF+/X9QRi-F+

Manufacturer Name: Supermicro

Product PartNum: PIO-847R-TRF+-ST031

Product SerialNum: S12873024303117

Firmware Revision : 03.20

Firmware Build Time : 2014-01-03

Bios Version : 3.0a

 

 Plugins:

Screen Shot of Main Page for extra detail about setup:

 

tower-diagnostics-20190822-0136.zip

[JorgeB]

Try running a parity check after booting in safe mode and with all dockers/VMs disabled, if still slow post diags from that time.

[2dumb2care]

Currently loading into save mode no GUI with Docker and VM disabled. Parity sync is currently between 6 and 7 MB/s

tower-diagnostics-20190823-1253.zip

[JorgeB]

No errors, don't know how well that Adaptec controller works with Unraid, check disk and controller speeds with the diskspeed.

[2dumb2care]

Sorry for no update busy weekend and the 2 times I tried to kick off the test it seemed to just hang. Just rebooted the system and trying again will post log file when completed. 

[2dumb2care]

Sorry again for the delay, had to run the benchmarks in small batches or else it would hang on a random drive and would not save the results. I can try to run a fresh run again if need be since some of the drives were tested at different times. 

20190830_141351.tar.gz

[JorgeB]

5 minutes ago, 2dumb2care said:

Sorry again for the delay, had to run the benchmarks in small batches or else it would hang on a random drive and would not save the results. I can try to run a fresh run again if need be since some of the drives were tested at different times. 

Just post the graph, the one testing the controller, if you can't run the controller test without hanging there's likely some hardware issue.

[2dumb2care]

here is the most current graph

[JorgeB]

I wanted to see the controller test, where it tests each disk individually and then all together, click the highlighted part to run it:

 

[2dumb2care]

Here is the test you are looking for. Tried running twice and both died out with an error for disk 16. Going to try and reseat and possibly replace the drive to see if that helps the test complete fully.

 

 

Series 7 6G SAS/PCIe 3

Adaptec
RAID bus controller

Type: Onboard Controller
Current & Maximum Link Speed: 8GT/s width x8 (7.88 GB/s max throughput)
Capabilities: storage pm msi msix pciexpress bus_master cap_list rom emulated
 

Port 1: 

sdb 

2TB 

SEAGATE ST2000NM0001 Rev 0 Serial: Z1P42RH30000C314BRTK (Disk 12)

Port 2: 

sdc 

2TB 

SEAGATE ST2000NM0001 Rev 0 Serial: Z1P630A20000C401AWQR (Disk 18)

Port 3: 

sdd 

2TB 

SEAGATE ST2000NM0001 Rev 0 Serial: Z1P1EEQZ0000C232DREB (Disk 7)

Port 4: 

sde 

2TB 

SEAGATE ST2000NM0001 Rev 0 Serial: Z1P43BFS0000C314CB3Z (Disk 14)

Port 5: 

sdf 

2TB 

SEAGATE ST2000NM0001 Rev 0 Serial: Z1P430E30000C3148YAA (Disk 17)

Port 6: 

sdg 

2TB 

SEAGATE ST2000NM0001 Rev 0 Serial: Z1P42X6B0000C3148ZHK (Disk 13)

Port 7: 

sdh 

2TB 

SEAGATE ST2000NM0001 Rev 0 Serial: Z1P1ELEL0000C232EKGW (Disk 10)

Port 8: 

sdi 

2TB 

SEAGATE ST2000NM0001 Rev 0 Serial: Z1P421R70000C31469V1 (Disk 15)

Port 9: 

sdj 

2TB 

SEAGATE ST2000NM0001 Rev 0 Serial: Z1P422LR0000C31468YM (Disk 16)

Port 10: 

sdk 

2TB 

SEAGATE ST2000NM0001 Rev 0 Serial: Z1P1EKQ10000C232CPVX (Disk

Port 11: 

sdl 

5TB 

Seagate ST5000LM000 Rev 0001 Serial: WCJ04013 (Disk 4)

Port 12: 

sdm 

5TB 

Seagate ST5000LM000 Rev 0001 Serial: WCJ03ZBB (Disk 6)

Port 13: 

sdn 

5TB 

Seagate ST5000LM000 Rev 0001 Serial: WCJ048E9 (Disk 2)

Port 14: 

sdo 

5TB 

Seagate ST5000LM000 Rev 0001 Serial: WCJ04HZY (Disk 3)

Port 15: 

sdp 

2TB 

SEAGATE ST2000NM0001 Rev 0 Serial: Z1P1EKRG0000C232DPQA (Disk 9)

Port 16: 

sdq 

2TB 

SEAGATE ST2000NM0001 Rev 0 Serial: Z1P41PVD0000C3146NH9 (Disk 11)

Port 17: 

sdr 

5TB 

Seagate ST5000LM000 Rev 0001 Serial: WCJ03Z1W (Parity 2)

Port 18: 

sdt 

5TB 

Seagate ST5000LM000 Rev 0001 Serial: WCJ04ZBF (Disk 5)

Port 19: 

sdu 

5TB 

Seagate ST5000LM000 Rev 0001 Serial: WCJ04GMQ (Disk 1)

Port 20: 

sdv 

10TB 

Western Digital WD100EMAZ Rev 83.H0A83 Serial: JEGKR5AN (Parity)

 

Benchmarking Parity (sdv):  199.77 MB/Sec
Benchmarking Parity 2 (sdr):  142.30 MB/Sec
Benchmarking Disk 1 (sdu):  144.02 MB/Sec
Benchmarking Disk 2 (sdn):  141.38 MB/Sec
Benchmarking Disk 3 (sdo):  150.81 MB/Sec
Benchmarking Disk 4 (sdl):  142.26 MB/Sec
Benchmarking Disk 5 (sdt):  144.12 MB/Sec
Benchmarking Disk 6 (sdm):  144.91 MB/Sec
Benchmarking Disk 7 (sdd):  153.39 MB/Sec
Benchmarking Disk 8 (sdk):  153.46 MB/Sec
Benchmarking Disk 9 (sdp):  156.32 MB/Sec
Benchmarking Disk 10 (sdh):  150.07 MB/Sec
Benchmarking Disk 11 (sdq):  156.66 MB/Sec
Benchmarking Disk 12 (sdb):  150.03 MB/Sec
Benchmarking Disk 13 (sdg):  155.81 MB/Sec
Benchmarking Disk 14 (sde):  155.94 MB/Sec
Benchmarking Disk 15 (sdi):  152.14 MB/Sec
Benchmarking Disk 16 (sdj): 

Lucee 5.2.9.31 Error (application)

Messagetimeout [3060000 ms] expired while executing [/tmp/DiskSpeed/Instances/local/controller/bandwidth/128c72ecc72d3c0e3e7572b158b55646/sdj_benchmark.sh]

StacktraceThe Error Occurred in
/var/www/TestControllerBandwidth.cfm: line 165

163: "chmod 666 #ResultsFN#" & Chr(10)>
164: <CFFILE action="write" file="#BWDir#/#DriveID#_benchmark.sh" mode="766" output="#cmd#" addnewline="NO">
165: <CFEXECUTE name="#BWDir#/#DriveID#_benchmark.sh" timeout="3060" />
166: <CFSET Result=GetReadAvg(ResultsFN,Drive.Attrib.Configuration.RPM,2,MaxGap)>
167: <CFSET Avg=ListFirst(Result,"|")>
 

called from /var/www/DispController.cfm: line 97

95: </CFIF>
96: <CFELSE>
97: <CFINCLUDE template="TestControllerBandwidth.cfm">
98: <CFOUTPUT><br></CFOUTPUT>
99: </CFIF>
 

Java Stacktracelucee.runtime.exp.ApplicationException: timeout [3060000 ms] expired while executing [/tmp/DiskSpeed/Instances/local/controller/bandwidth/128c72ecc72d3c0e3e7572b158b55646/sdj_benchmark.sh]
  at lucee.runtime.tag.Execute._execute(Execute.java:241)
  at lucee.runtime.tag.Execute.doEndTag(Execute.java:252)
  at testcontrollerbandwidth_cfm$cf.call(/TestControllerBandwidth.cfm:165)
  at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:933)
  at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:823)
  at lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:805)
  at dispcontroller_cfm$cf.call(/DispController.cfm:97)
  at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:933)
  at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:823)
  at lucee.runtime.listener.ClassicAppListener._onRequest(ClassicAppListener.java:66)
  at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:45)
  at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2464)
  at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2454)
  at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2427)
  at lucee.runtime.engine.Request.exe(Request.java:44)
  at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1090)
  at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1038)
  at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:102)
  at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
  at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
  at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:684)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
  at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1152)
  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
  at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2527)
  at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2516)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
  at java.lang.Thread.run(Thread.java:748)
 

Timestamp9/2/19 5:23:55 PM EDT