(Solved) Terribly slow array access


Recommended Posts

My unRAID array seems to have slowed down markedly over the last few weeks. Copy speeds are rarely over 30MB/s and sometimes down to under 1MB/s using MC (and this is copying within the array, not across a network). I always thought I had found a reason (Windows PC backing up to the array, Plex docker doing some indexing/updating, CrashPlan docker doing backup, etc.) but I'm not seeing any consistent cause. I've also tried stopping the various dockers, but write/read is still very slow.

 

I have 6 data disks (mostly 4TB) and 2 parity drives. I have a flashed Perc H310 which has been working fine. CPU is a Xeon D-1541 on a Supermicro X10SDV-TLN4F motherboard (but I use only gigabit ethernet) with 32GB RAM.

 

I upgraded to 6.3.5 today, but the array is just as slow. I think I must have something setup incorrectly, so I thought I would see if anyone could help out here. I've attached the diagnostics file.

tower-diagnostics-20170623-2259.zip

Edited by sonofdbn
Link to comment

Trurl, thanks for the suggestion. I hadn't thought about the impact of having to spin up the drives.

 

BUT: I didn't want to exaggerate the slowness or writing, so I stated 30MB/s based on what I had remembered. So as a test I'm now moving 28GB of files of 20 TV episodes from one user share to another - so everything is on the array, no network. I'm using Windows to do the move and looking at the progress window after about 10 minutes, the speed never reaches even 3MB/s. I suppose there is some Windows interaction overhead, but even so, this is very slow.

Link to comment

I'm only initiating the move from my Windows PC. The files themselves are already on the array, so the network should not be a big issue (I believe). No VMs are running, no dockers running except CrashPlan, which is paused.

 

But OK, let's check this out: I'm now using MC, same files from one user share to another. After a few minutes, MC is reporting a speed of 1.5MB/s. (I don't think it's because doing it via Windows is faster; I think it's because the array is becoming increasingly unresponsive.)

 

 

Link to comment

Benson, thanks for the suggestion. I've started a parity check: it's S..L..O..W: I'm getting a speed of around 210KB/s (see attachment Parity_check.png). This compares with my parity check one month ago where the speed was 285 MB/s (see attachment Parity_check_previous.png). Now, I must admit I don't believe the previous speed, but it got me to thinking about my dual parity.

 

(Just saw the RFS question: no, I'm using XFS on all data drives, and no single drive has less than 140GB free.)

 

I've been running dual parity drives for a while, both 4TB. Shortly before the previous parity check, I changed one of the parity drives to an 8TB drive, and I'm reasonably sure the recent parity check was after the new parity drive was installed. So possibly the parity speed calculation was actually using 8TB when in fact there was only 4TB of parity being checked. (And the screenshot for the current parity check shows that Total size to be 8TB.) So "real" speed was perhaps half, which is still not bad, but more realistic.

 

So I'm wondering whether there's a problem caused by having two different sized parity drives. Is unRAID trying to do something extra because of the different sizes?

 

I'm leaving the parity check running for now because I can't think of what else to do.

 

From the Main tab in the GUI I noticed that there are a lot of writes being made to the parity drives. Not sure why that would be the case during a parity check. I have no dockers or VMs running, except for CrashPlan which is paused. I have the Dynamix File Integrity plugin, but that is set to not start any checks while parity checking is being done.

 

In the hope that someone can help, I've also uploaded the latest diagnostics, downloaded while the parity check was running.

 

Parity_check_previous.PNG

Parity_check.PNG

tower-diagnostics-20170626-1026.zip

Edited by sonofdbn
Reply to RFS question
Link to comment

OK, I thought parity checking involved only reading data and checking that against the parity. But in any case I realised that the read/write stats are cumulative; after clearing the level of activity is much lower.

 

But surely something is wrong with my setup. I'm 9 hours and 7 GB into parity checking, average speed is 226KB/s and estimated finish is in 402 days' time!

Link to comment

That makes sense. Is there a way of seeing what's causing all the activity? Using a top command and also looking at the Open Files plugin, it seems that most of the ongoing processes are sha256sum - and yes, there are files on at least disks 2 and 5. That might be related to the Dynamix File Integrity plug-in, but it's set to not start anything while parity-checking is going on. Perhaps these are slowly completing (but 9 hours!).

 

Is it safe to kill a process if under Open Files it shows that for that process there are 0 files that "may prevent shutdown"?

Link to comment
1 minute ago, sonofdbn said:

the ongoing processes are sha256sum - and yes, there are files on at least disks 2 and 5. That might be related to the Dynamix File Integrity plug-in

 

It's almost certainly that, cancel the check and wait for the checksums to finish or disable the plugin.

  • Upvote 1
Link to comment

Thanks so much!

 

It was indeed the File Integrity plug-in! I tried various ways to suspend the plug-in and also tried killing the various sha256sum processes, but new ones kept on popping up. In the end I removed the plug-in and bingo! Speed immediately up and now at about 40MB/s.

 

Important note: I was probably not using the Dynamix File Integrity plug-in correctly, as I never spent any time setting it up carefully, so I certainly wouldn't generalise and claim in any way that the plug-in doesn't work properly.

Link to comment
2 hours ago, sonofdbn said:

OK, I thought parity checking involved only reading data and checking that against the parity.

Just a nit to pick here. Unraid's parity doesn't have any concept of files, or file systems, or data. It ONLY verifies drives as a whole, whether there is "data" on them or not. An unformatted disk in the array is as much a part of parity as one with TB's of data.

 

This is one of the core concepts of unraid, often misunderstood by people coming to it from other computing backgrounds.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.