FQs19

@Vr2Io With Global C-States set to Auto and IOMMU set to Enabled equals good performance threadripper19-diagnostics-20220712-1850.zip

I don't know which one is causing the issue. I'll go through and test each config and let you know. I just changed both to Auto cause at this point I just want it to work. HAHA

Sorry about that here are the two diagnostics. One from after changing the BIOS settings and the one after reverting the BIOS changes. threadripper19-diagnostics-20220712-1758.zip threadripper19-diagnostics-20220712-1810.zip

BIOS changes that reverted: -Global C-States to Auto from Enabled -IOMMU to Auto from Enabled Parity-Sync speeds back to normal 165~185MB/sec. I'll leave well enough alone now. Any issues with leaving those two settings to Auto? threadripper19-syslog-20220712-2109.zip

Just rebooted and started Parity-Sync after making those BIOS changes and what do you know, 10-14MB/sec writes! So there's obviously something wrong with the BIOS. I'm going to revert the BIOS changes I made and see if the performance comes back. Hopefully it does. I also just realized I attached the syslog, not the diagnostics to my previous post. Sorry. threadripper19-syslog-20220712-2109.zip

@JorgeB @Vr2Io So I decided to go back to basics and assume I messed up installing my components when I switched cases. I pulled my graphics card. I pulled the AIO cooler off and cleaned it. Previous thermal paste application was good. I pulled all the memory sticks. I pulled my Dimm.2 stick. I pulled the 24 pin power cable out. I made sure the two 8 pin power cables were seated all the way. They were. I pulled the 3960X CPU out of the carriage, after cleaning the thermal paste off. I verified all the pads were still in good shape. They were. I then took video of the motherboard CPU pins in the socket to verify nothing was damaged. No damage seen. I then re-inserted the CPU using the supplied torque wrench with the proper sequence of 1>2>3. I applied a liberal amount of thermal paste to the AIO heatsink. I then re-inserted my four memory sticks, my Dimm.2 stick, the 24 pin power cable, and my graphics card. I then plugged my PSU in (which is only 1000 watts and several years old. I am replacing this PSU with a new Seasonic PRIME 1300W 80+ Platinum one this week). I'm thinking my server could hit 1000 watts during high work loads and since the PSU is several years old I should replace it with a higher wattage one. I don't believe it was causing my errors, but if I'm going back to basics, this is something that should be replaced with a higher wattage PSU. I cleared the CMOS. I configured the BIOS, changing only the TPM to Firmware, the CPU speed to Auto from Default, the boot mode to CSM and Legacy USB boot, then saved and rebooted. Once in Unraid, I had to reconfigure the network settings since I left the Intel LAN controller enabled, then rebooted Unraid to save the changes. After getting back into Unraid and verifying the network changes worked, I added my two parity disks and started the array. I immediately saw normal performance, 165MB~185MB/sec!!! I don't see any high single thread CPU usage either!!! Every 1.0s: grep MHz /proc/cpuinfo Threadripper19: Tue Jul 12 17:16:26 2022 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 3800.000 cpu MHz : 3800.000 cpu MHz : 3800.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 3800.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 4335.127 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2387.163 cpu MHz : 2200.000 cpu MHz : 2800.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 3800.000 cpu MHz : 2200.000 cpu MHz : 2200.000 cpu MHz : 2200.000 After a while, the Fix Common Problems plugin through a warning saying it was Unable to communicate with Github. I had to go into my router, change the DNS provider to 8.8.8.8, then change it back to 1.1.1.1. Once I did that, the DNS error on Fix Common Problems cleared. There might be an issue with my SFP+ module. I'll deal with that annoyance later. I'm attaching my diagnostics to see if you can spot any difference after reseating my CPU, cooler, memory, and power cables. I'm hoping that my problems were just because I didn't have the CPU, cooler, or power cable seated properly after switching cases. It might also be that my PSU is just at its limit and with its age it couldn't keep up. I have two new TRX40 motherboards coming as well in case this Asus ROG Zenith II Extreme Alpha motherboard has a problem. I'm going to cancel this Parity-Sync, reboot the server, and go into my BIOS settings to change the following: -Global C-States to Disabled -IMMOU to Enabled -turn off the WiFi controller -set Fan speeds to what I want them at -make sure virtualization is enabled So what are your thoughts on all of this? Spot anything in the diagnostics that's different? Thank you both for the help along the way to my Unraid recovery. haha threadripper19-syslog-20220712-2109.zip

Will do. Thank you for the help once again.

@JorgeB I'm going to move my unraid to a Z390 motherboard. What's the easiest way for me to move my NVMEcache pool to my Arraycache pool, since I will only be able to use two M.2 NVME drives on that motherboard? I was just going to change the shares that use the NVMEcache to use the Array, then run Mover. After everything is on the array I'll change those same shares to use the Arraycache and run Mover again, reconfigure my settings to match the new assignments. I was also just considering using Midnight Commander, but I haven't used that very much. I'm sure I can figure it out, but just wanted to ask an expert like yourself before I try. I might have to lower the amount of disks I have as well, but that's easy to do with the Unbalance plugin.

Are you thinking that maybe the CPU is bad? Do you know of anything else I can try?

No point in trying to find what SATA port to connect my parity disks to since I can clearly see multiple issues. I'm throwing in the towel. I'll reach back out to Asus and see what they can do for me. I doubt anything really. They sure aren't going to give my $850 bucks back for the board. I won't accept a used motherboard in exchange for mine either. Really appreciate the help. Thank you.

@Vr2Io Testing old parity disks with one as Parity and one as a Data disk: No change. I did see 212MB/sec. for the first second, but it quickly fell off to ~80-100MB/sec. I'm definitely going to contact Asus and see how they can compensate me.

@Vr2Io I couldn't sleep without trying the 1603 BIOS. Performance is better, but not by much. Not sure if this speed is normal, but it's not the 180MB/sec. I'm expecting. This is one of my old parity disks ...KRD as parity and one of data disks as Disk 1: It was over 100MB/sec for a couple seconds after starting the array, but kept falling to ~80MB and leveled off there. Here's the CPU usage as well: Every 1.0s: grep MHz /proc/cpuinfo Threadripper19: Tue Jul 5 23:27:31 2022 cpu MHz : 2800.000 cpu MHz : 2845.614 cpu MHz : 2851.384 cpu MHz : 2945.021 cpu MHz : 2800.000 cpu MHz : 2200.000 cpu MHz : 2800.000 cpu MHz : 4241.914 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2462.802 cpu MHz : 3296.135 cpu MHz : 2800.000 cpu MHz : 2905.352 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 3259.315 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2967.138 cpu MHz : 2800.000 cpu MHz : 3800.000 cpu MHz : 2342.124 cpu MHz : 4240.152 cpu MHz : 2800.000 cpu MHz : 2889.171 cpu MHz : 2200.000 cpu MHz : 2800.000 cpu MHz : 2716.848 cpu MHz : 2800.000 cpu MHz : 2885.461 cpu MHz : 3800.000 cpu MHz : 2962.822 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 3227.638 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2200.000 cpu MHz : 2866.924 cpu MHz : 2794.674 cpu MHz : 2800.000 cpu MHz : 2892.098 cpu MHz : 2800.000 cpu MHz : 2347.624 cpu MHz : 2200.000 I'm going to try again with just the old parity disks like the first test you asked me to try.

I figured as much. It might be a couple days before I can try that, but I'll continue posting when I do. Thank you so much for the help.

I stopped the parity-sync and did a New Config, so this is just unraid sitting idle on the Main page. Every 1.0s: grep MHz /proc/cpuinfo Threadripper19: Tue Jul 5 22:59:38 2022 cpu MHz : 2918.511 cpu MHz : 2427.985 cpu MHz : 2200.000 cpu MHz : 2800.000 cpu MHz : 2775.257 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2863.275 cpu MHz : 2744.736 cpu MHz : 2200.000 cpu MHz : 3002.078 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 3369.383 cpu MHz : 2800.000 cpu MHz : 2514.198 cpu MHz : 2800.000 cpu MHz : 2798.982 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2785.893 cpu MHz : 3800.000 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2865.946 cpu MHz : 2800.000 cpu MHz : 2854.917 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2781.481 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 3066.897 cpu MHz : 2856.996 cpu MHz : 2800.000 cpu MHz : 2800.000 cpu MHz : 2929.347 cpu MHz : 2800.000 cpu MHz : 2800.000 If you want to see this while it's running the test, let me know.

@Vr2Io I swapped old parity disks and tried again. Same results, only getting ~60MB/sec. So disk ...KRD as parity 60MB/sec. and disk ...P4D as parity 60MB/sec.

@Vr2Io Same results as first test. ~60MB/sec. with occasional spikes to 80MB/sec. For the first second after starting the array, I did see a speed of 120MB/sec, but it never went that high again.

"Seek time Performance" for both are 15. I'll try again with first Parity as Parity and a Data disk as Disk 1.

Well it somewhat worked, though I'm only getting around 60MB/sec. It occasionally spikes up into the 80MB/sec range. CPU usage is like before: This still isn't anywhere near the 180MB/sec. I should be seeing. What are your thoughts on this?? threadripper19-diagnostics-20220705-2234.zip

I'll give it a try. Thanks.

I can try that. Do I need to disconnect all the other disks or can I just leave them as unassigned ones for this test? So New Config, one old parity disk as Parity and the other old parity disk as Data Disk 1. Also, what are the steps for getting back to my current assignment? I'm assuming just populate the slots with the screenshot I have now, keeping the parity empty. Is that correct?

So I cleared the CMOS and rebooted. No change. Still only getting 10MB/sec. I then moved my parity disks to the HBA. Still no change. Still only getting 10MB/sec, except this time my second parity disk through a UDMA CRC error, well actually two. It was zero now it's 2. This motherboard and/or CPU is toast. There's only one thing left to try and that's to re-flash the MB BIOS back to 1603 which gives me CPU Code 7 and TEST ERRORS when booting if I disable the Intel LAN and WiFi controllers. I'm currently running a special BIOS that Asus made for me to fix those errors; BIOS 9901. So if re-flashing back to 1603 allows parity to sync normally, I'll have to decide whether to continuing using BIOS 1603 dealing with the errors and whatever other anomalies there are or write back to Asus and tell them I can't run a parity sync. This is by far the worst computer I've ever built. I've never had such horrendous luck with pc components like this especially while they're still under warranty with no possibility to get new ones. Biggest waste of $2440 ever. threadripper19-diagnostics-20220705-2146.zip

No change with just 1 parity disk: I'm going to Clear my CMOS, start Unraid and try again. If that doesn't work, I'll move my parity disks to my HBA and try again. Thanks for the help. threadripper19-diagnostics-20220705-1554.zip

@JorgeB @Vr2Io I moved my parity drives to the ASMedia ports. So now I have 8 disks on the HBA, 2 disks on the MB SATA ports, and 2 parity disks on the ASMedia SATA ports. Speed increased to only 12MB/sec. CPU occasionally has 1 thread hitting 80%, but this is the average: Any other ideas? threadripper19-diagnostics-20220705-1524.zip

I'm thinking @Vr2Io might be right. I don't remember how I had my disks connected to my motherboard and HBA. It's been a minute. HAHA I thought I had 4 disks (including the parity disks) connected to my motherboard's 4 SATA ports with nothing connected to the ASMedia SATA ports and 8 disks connected to my HBA card. I'll try and move my parity disks to the HBA card and see what happens. @JorgeB @Vr2Io would either of you suggest maybe spreading the two parity disks over the motherboard ports? Like 1 on the motherboard SATA ports and the other on the ASMedia ports. How would either of you attach 14 drives to an 8 port HBA, 4 MB SATA ports, and 2 ASMedia ports?? I forget to mention, I moved my HBA to the first x8 slot from the second x16 slot. Would that make a difference? I currently have my Quadro RTX4000 on slot 1 (x16), the HBA on slot 2 (x8), slots 3 and 4 are empty. I will eventually put another video card in for a VM, but after I get all this sorted.

After rebuilding my server and setting everything up, I finally added my two WD Red Pro disks back as Parity. The first time I tried adding the two parity disks, it was running at 10MB/sec. I canceled that, rebooted my server and tried again. This time the performance is worse. It's currently running at 8MB/sec. I ran all disks through the DiskSpeed docker and all disks looked great. I know some disks are old, but I've been able to transfer data to my array at the normal 110-120MB/sec. I canceled the Parity-Sync so I could grab the DiskSpeed graph. After stopping my array, setting my two parity drives, and starting the array, the speed is still slow at 8.7MB/sec. Anyone have an idea of why the sync is so slow? On version 6.10.3. All plugins updated. Two parity drives are WD Red Pro's. Array disk are all WD with a mix of red, red plus, and pro. threadripper19-diagnostics-20220704-2135.zip