Hi all,

 

Having a really weird problem with my Unraid array that I just can't put my finger on, and I'm hoping someone will be able to help out.

 

The array runs perfectly for the first day or so, then begins slow down to a crawl, running at between 2 and 6mb/sec.

 

The syslog is showing drive errors, which I realise is Unraid being incapable of spinning down SAS drives, and another feature of the SAS controller that Unraid cannot support.

 

I was running FreeNAS when I first built the system hoping to use Bhyve for VMs, and it ran beautifully, no slowdowns, went for weeks at 100mb+/sec easy, but I switched to Unraid as the KVM VM is much better, and now I'm having a pretty bad time with the actual storage portion of the namesake's purpose. I am temporarily regretting the move, but I'm sure someone here will know the solution

 

In the meantime, I'm stuck stopping the array, rebooting Unraid, then starting the array again, every time this happens. A bit of a bummer.

 

I have tried just stopping the array and restarting it, but it does not fix the problem.

 

I have attached photos of the speed before and after rebooting Unraid.

 

The array is btrfs encrypted.

 

The server specs are:

Motherboard: ASUS Z9NA-D6

CPUs: 2 x Intel Xeon E5-2440

RAM: 24GB DDR3 ECC Registered

Cache: 400GB Seagate ST400FM0073 SAS SSD

Array: 3 x 3TB HP/Seagate MB3000FCWDH SAS - 1 in parity, 2 in array

SAS controller: ASUS PIKE 2008 (expansion card)

 

Any help would be greatly appreciated.

 

Thank you.

 

beemoth-diagnostics-20191024-0137.zip

Edited October 24, 20196 yr by Rubydoo

2 hours ago, Rubydoo said:

The syslog is showing drive errors, which I realise is Unraid being incapable of spinning down SAS drives, and another feature of the SAS controller that Unraid cannot support.

Try clicking on your SAS drives on Main and set 'Spin down delay' to Never.

 

It's not an issue with Unraid supporting SAS controllers - they work fine with SATA devices - it's SCSI device spin down which is not supported at this time.

Hi,

 

Did this, gave it time, still does exactly the same thing. Slowing to a complete crawl again after some time.

 

Can anyone offer some help?

 

Thanks.

Also here are the latest diagnostics.

beemoth-diagnostics-20191104-0558.zip

And after reboot it runs like this.

Just a hunch but could it be your cache ssd that is not trimmed untill you reboot ?

That will slow it down to a crawl.

You can run the fstrim command manualy agianst the /mnt/cache or schedule it with the usr scripts plugin. I run it nightly against my ssds.

 

Where are you doing the copying from and to? In other words,

• Where are the "FreeBSD...iso" and "highsierra.img" saved?
• Where are "Downloads" and "Linux" located?

You screenshot looks to be MacOS-based.

• Are you copying over wifi / wired network / br0 bridge?
• Are you able to test on a Windows-based machine?

Hi,

The first screenshots are from Unraid to the client (FreeBSD copied to the client machine), and the second screenshots are from the client to Unraid (highsierra.img copied to the Unraid share). Unraid is accessed via SMB in both cases.

The test is the same on Windows, and the clients are wired/lan.

The problem appears after the server has been on for a few days. It runs perfectly for about a day or so, then gradually slows to a crawl.

Thanks.

Hmm will try that Glenn, good idea. I didn't think of that. Will let you know how it goes. Thank you for the suggestion.

I have a Idea because i saw something similar. No solution, but maybe one step forward.

 

1. Install the stats plugin (if you don't have it yet).

2. Then reboot and directly open the stats page, switch to system stats and change the time to 10 minutes or so.

3. After that, start the copy like before.

 

Maybe its fast as long the data goes into the cached RAM. And as soon you RAM is full (only the cached RAM is full) the transnsfer speed drops.

 

Watch for the speed drop and make a screenshot from the stats.

Only a idea...

Ok, after doing fstrim speeds appeared good on unraid running locally (duplication copy), but network drives no change. But, after smb stop and start, and stopping and starting the array, then unmounting and mounting the drives again, back to normal, so fixed somewhere between that. Will wait for it to slow down again and then test it step by step to find exactly where it is.

 

Thanks all for the input, I will post the results in case someone else has the same issue.

Update: so the problem appears to be with smbd. I have just rebooted so I will need to let it run for a couple of days again to slow down. I will post my testing results and do some tweaking to see if I can't fix the problem. In the meantime, I'll just use nfs (media share) and afp (hackintosh workstation) for my primary needs. There is only one pc running smb and it's just for incremental backups, so bandwidth there isn't crucial. Will continue to update for the purposes of any user reading this who also encounters this same issue.