I'm relatively new to Unraid (and Linux) so be gentle. As a photographer and always in need of data storage, I decided to replace my 2 old Synology's with one big server. Might keep one just to have another backup together with one that is not in my home.

 

I upgraded my network with 2 Netgear 10GbE switches (both with 2x 10GbE ports). I am still on cat6 cables but plan to upgrade that to 6A, just to rule out problems there. My hardware is a Ryzen 5 3600, 16GB RAM, Asrock Rack X470D4U2-2T, a HBA from Broadcom 9400-16i, a Nvidia 1660GTX for the Plex transcoding, a 960 Pro 512GB as cache and for now 4x WD Gold 14TB's. So I might expect a pretty decent performing server, right?

 

When I test speeds with iPerf3 and Diskspeed everything seems normal. I get 3.9Gbit/s network speeds (not full full speed but decent enough considering the length and being cat6) and my disks seem normal with 255MB/s (with no drops) and 1.1Gbit/s for the cache drive.

 

When I move something from my computer (SMB shares) to the server (share with cache only) I'm averaging 50~70MB/s. The parity sync is at ~90MB/s. I get the same speeds when I pause the sync. When I move something to the array I get like 140MB/s in the beginning (first 30%) just to drop down to 50~70MB/s. Turbo write is on. I'm not expecting GB/s here but this just seem slow?

I have 2 10GbE ports and one Gbit port on the motherboard but I can only use one 10GbE for now. I'm not going to invest into a new very expensive 10GbE switch just to get "under Gbit" speeds. So I'm kinda out of ideas for now.

 

Not sure if there's something wrong in the hardware (RAM? Memtest seemed fine) or network settings (which I don't understand fully, bonding is on but I'm only running one port?) so I turn to the forum. I included the diagnosis file. 

marcel-diagnostics-20200323-1010.zip

Edited March 23, 20206 yr by jonasdegent

Wait for the parity sync to finish, even with turbo write performance isn't the same while parity disk is invalid.

also, any ideas why this is appearing on my dashboard?

3 minutes ago, jonasdegent said:

also, any ideas why this is appearing on my dashboard?

Are you running any kind of ad blocker? If so whitelist your Unraid GUI IP and server name.

Hmm, I do have Ad blocker disabled in Google Chrome but still the same error. No problems with it in Firefox and Edge.

Flush the Chrome cache and it should work

3 hours ago, dukiethecorgi said:

Flush the Chrome cache and it should work

That did the trick! Thanks!

Parity build just finished. Rebooted just to be sure. Same identical "slow" speeds. 😕 Turbo write is on.

Start a copy to the array and grab diags during it.

I presume you just do it the same way?

Made one during a copy to the share that suppose to be to the cache only and one during a copy to the array

to array marcel-diagnostics-20200324-1428.zip To cached marcel-diagnostics-20200324-1427.zip

Nothing jumps out, try enabling disk shares (Settings -> Global Share Settings), then transfer directly to a disk, e.g. \\tower\disk1, see if there's any difference.

9GB file. It was faster, a lot faster (240MB/s) and then it passes the 30% mark, it just drops dramatically to 55~80MB/s going up and down... Downloading from the array was fast all the way (230MB/s).

Could this be HBA related? Should I try to flash it to newer firmware? I don't know how but I'll probably figure it out.

 

The initial high speed is while it's being cache to RAM, limited by network/source device, then it's limited by the actual device speed, 55-80MB/s is still slow for those disks with turbo write enable, they should be handle around 200MB/s when empty, if you can connect the disks to the onboard SATA ports and repeat the test.

BTW, you can also test to \\tower\cache, this should also be much faster than before, because it's bypassing Unraid share system, but while it's normal to see a difference in performance here it's not normal to see such low speeds writing to the user shares.

Just attached the 4 drives to the onboard SATA ports (good thing I had a reverse breakout cable lying around. With that Corona quarantaine it ain't easy to get stuff) however Unraid won't let me start the array? There isn't an option to start it. I can see the drives in the BIOS and in Unraid

 

Nevermind, a reboot did the trick

Edited March 24, 20206 yr by jonasdegent

Copy via disk share to disk 1 and 2 and 3 is around 100MB/s average, to the cache drive via disk share it's 205MB/s. To an array share it's averaging 93MB/s. All copies were fast the first 40-ish percentages.

Edited March 24, 20206 yr by jonasdegent

Seems the user share overhead is higher than normal, though speeds still seam on the low side even when using disk shares, so likely something else at play here.

OK, wow... just switched to another backplane off my Inter-tech case. Speeds are what they should be (and even better than I expected). Sheeesh... I was about to give this thing up. Averaging 295MB/s to the array, 588MB/s to the cache. Maxing out my Cat6 cables. Still on onboard SATA though, so next test is the HBA.

OK, back to square one... seemed to work, now I'm back to slow speeds.

Pulled out the 2 DIMMs of RAM and replaced it by one DIMM of 16GB (tried multiple slots). The first time I booted up, it seemed to do the trick (I was still on SATA onboard). However, after one reboot, it stopped working again. Did manage to get it back fast again but I can't pinpoint where the problem is. It appears to be random. I'm now thinking either the processor or the board itself. When the transfers are fast like they should, the Plex transcoding from the 1660GTX also works like it should. When the speeds are slow, Plex has issues. So could this be related to PCI-E ports being faulty? Or processor? Maybe try to reseat the CPU now, but then I'm out of ideas.

 

Also booting from the USB stick takes a LONG time, usually stuck for a while at 'initramfs'

Edited March 24, 20206 yr by jonasdegent

Difficult to guess where the problem is with those symptoms, you'll need to swap some hardware around.

 

11 hours ago, jonasdegent said:

usually stuck for a while at 'initramfs'

This usually takes a few seconds.

I've got a "B2" on the Dr Debug which should be the GPU according to some sources, off course it's not even in the manual. Removed the 1660GTX but keeps failing. Too bad I can't test CPU, haven't got another one lying around.

Could someone think of a BIOS option that I could be overlooking? When I plug in 2 PCIe cards (NVidia card and the LSI), the server becomes slow and everything just takes a little longer. It works but far from ideal. If I remove one, it's mega fast. Super responsive. I already made an RMA for the motherboard but I just want to try one more time.

Looking into the IPMI the power supply status says Power Supply error. Is that because it isnt detected or is there really something wrong?

52 minutes ago, jonasdegent said:

Looking into the IPMI the power supply status says Power Supply error

Its a consumer grade p/s so it doesn't talk to the ipmi. That can be ignored. Do the other values look right? (12v rail, 5v rail, 3.3v rail, etc)

 

On 3/25/2020 at 8:15 AM, jonasdegent said:

I've got a "B2" on the Dr Debug

That's "b2", right? It could be related to "option rom" issues. Check if the bios is set to load option rom.  Maybe your video card needs a bios update.

 

You said you tried taking the video card out and it still wouldn't work? What about the HBA? Have you tried taking that out and seeing if you get the same "b2" code.