My unRAID array seems to have slowed down markedly over the last few weeks. Copy speeds are rarely over 30MB/s and sometimes down to under 1MB/s using MC (and this is copying within the array, not across a network). I always thought I had found a reason (Windows PC backing up to the array, Plex docker doing some indexing/updating, CrashPlan docker doing backup, etc.) but I'm not seeing any consistent cause. I've also tried stopping the various dockers, but write/read is still very slow.

 

I have 6 data disks (mostly 4TB) and 2 parity drives. I have a flashed Perc H310 which has been working fine. CPU is a Xeon D-1541 on a Supermicro X10SDV-TLN4F motherboard (but I use only gigabit ethernet) with 32GB RAM.

 

I upgraded to 6.3.5 today, but the array is just as slow. I think I must have something setup incorrectly, so I thought I would see if anyone could help out here. I've attached the diagnostics file.

tower-diagnostics-20170623-2259.zip

Edited June 26, 20179 yr by sonofdbn

1 hour ago, sonofdbn said:

Copy speeds are rarely over 30MB/s

Not unreasonable for parity writes. Have you tried Turbo Write?

 

 

Trurl, thanks for the suggestion. I hadn't thought about the impact of having to spin up the drives.

 

BUT: I didn't want to exaggerate the slowness or writing, so I stated 30MB/s based on what I had remembered. So as a test I'm now moving 28GB of files of 20 TV episodes from one user share to another - so everything is on the array, no network. I'm using Windows to do the move and looking at the progress window after about 10 minutes, the speed never reaches even 3MB/s. I suppose there is some Windows interaction overhead, but even so, this is very slow.

If you're using Windows then it is on the network. I usually use mc (Midnight Commander) from the unRAID command line. It is easy to learn and use. Others prefer Krusader docker as a file manager. Either method will keep the moves on unRAID and off the network.

I'm only initiating the move from my Windows PC. The files themselves are already on the array, so the network should not be a big issue (I believe). No VMs are running, no dockers running except CrashPlan, which is paused.

 

But OK, let's check this out: I'm now using MC, same files from one user share to another. After a few minutes, MC is reporting a speed of 1.5MB/s. (I don't think it's because doing it via Windows is faster; I think it's because the array is becoming increasingly unresponsive.)

 

 

Does slow only happen during writing, how about read speed ? i.e. file read, array check ?

Not sure how to check read speed. How would I go about doing that? I'd rather not do a parity check now - or is there some other array check that can be done?

Parity check should be a easy way to check and you can stop anytime. If you don't like that, you may try access file by any kind of network share to check speed normal or not.

Are you using the RFS filesystem? RFS writes will get very very slow as disks fill up.

Benson, thanks for the suggestion. I've started a parity check: it's S..L..O..W: I'm getting a speed of around 210KB/s (see attachment Parity_check.png). This compares with my parity check one month ago where the speed was 285 MB/s (see attachment Parity_check_previous.png). Now, I must admit I don't believe the previous speed, but it got me to thinking about my dual parity.

 

(Just saw the RFS question: no, I'm using XFS on all data drives, and no single drive has less than 140GB free.)

 

I've been running dual parity drives for a while, both 4TB. Shortly before the previous parity check, I changed one of the parity drives to an 8TB drive, and I'm reasonably sure the recent parity check was after the new parity drive was installed. So possibly the parity speed calculation was actually using 8TB when in fact there was only 4TB of parity being checked. (And the screenshot for the current parity check shows that Total size to be 8TB.) So "real" speed was perhaps half, which is still not bad, but more realistic.

 

So I'm wondering whether there's a problem caused by having two different sized parity drives. Is unRAID trying to do something extra because of the different sizes?

 

I'm leaving the parity check running for now because I can't think of what else to do.

 

From the Main tab in the GUI I noticed that there are a lot of writes being made to the parity drives. Not sure why that would be the case during a parity check. I have no dockers or VMs running, except for CrashPlan which is paused. I have the Dynamix File Integrity plugin, but that is set to not start any checks while parity checking is being done.

 

In the hope that someone can help, I've also uploaded the latest diagnostics, downloaded while the parity check was running.

 

tower-diagnostics-20170626-1026.zip

Edited June 26, 20179 yr by sonofdbn
Reply to RFS question

There's multiple read write activity on some disks during the parity check, so the speed will me severely impacted, that's normal.

OK, I thought parity checking involved only reading data and checking that against the parity. But in any case I realised that the read/write stats are cumulative; after clearing the level of activity is much lower.

 

But surely something is wrong with my setup. I'm 9 hours and 7 GB into parity checking, average speed is 226KB/s and estimated finish is in 402 days' time!

At the time the diags you-re saved there was heavy activity on disks 2, 3 and 5, stop that and after a minute parity check speed should return to a normal +100MB/s plus speed.

That makes sense. Is there a way of seeing what's causing all the activity? Using a top command and also looking at the Open Files plugin, it seems that most of the ongoing processes are sha256sum - and yes, there are files on at least disks 2 and 5. That might be related to the Dynamix File Integrity plug-in, but it's set to not start anything while parity-checking is going on. Perhaps these are slowly completing (but 9 hours!).

 

Is it safe to kill a process if under Open Files it shows that for that process there are 0 files that "may prevent shutdown"?

1 minute ago, sonofdbn said:

the ongoing processes are sha256sum - and yes, there are files on at least disks 2 and 5. That might be related to the Dynamix File Integrity plug-in

 

It's almost certainly that, cancel the check and wait for the checksums to finish or disable the plugin.

Thanks so much!

 

It was indeed the File Integrity plug-in! I tried various ways to suspend the plug-in and also tried killing the various sha256sum processes, but new ones kept on popping up. In the end I removed the plug-in and bingo! Speed immediately up and now at about 40MB/s.

 

Important note: I was probably not using the Dynamix File Integrity plug-in correctly, as I never spent any time setting it up carefully, so I certainly wouldn't generalise and claim in any way that the plug-in doesn't work properly.

2 hours ago, sonofdbn said:

OK, I thought parity checking involved only reading data and checking that against the parity.

Just a nit to pick here. Unraid's parity doesn't have any concept of files, or file systems, or data. It ONLY verifies drives as a whole, whether there is "data" on them or not. An unformatted disk in the array is as much a part of parity as one with TB's of data.

 

This is one of the core concepts of unraid, often misunderstood by people coming to it from other computing backgrounds.