High shfs usage makes server unusable


Recommended Posts

I have an issue where Unraid is becoming unusable with extremely high load, caused by shfs spinning out of control. This seems to be due to radarr doing file operations, but I'm not 100% sure. What I'm seeing is the load on the server starts climbing (getting up to 150 at times), and all other actions that require disk access just hang. I can't even `ls` a directory. I've done all I can think of to handle it, but nothing has worked.

 

Things I've tried:

  • Convert cache from Raid1 to Raid0 - I have (2) 1TB nvme drives for cache. My Plex appdata folder takes up 700GB alone, so very little space was left for the data cache. I  converted to Raid0 so I have 2TB of cache, but that didn't help
  • Move NZBGet, Sonarr, and Radarr off the unraid server. When content is downloaded is when it's the worst. Especially if NZBGet has to repair a file. I thought by moving these service to another system, I could offload that work so Unraid wold only have to do be available to receive the copied file (over an NFS share)

 

These have not solved the problem. I still regularly see the load spike up. When that happens, I con't do anything with the server until the load drops. No file access, no Plex, nothing. It's truly infuriating.

 

I do have a lot of drives (21 including 2 parity drives), but I feel like this has gotten way worse recently. I thought maybe I have a failing drive that is causing the parity functions to hang, but I am seeing no SMART errors that I would expect to see.

 

I'm at my wits end and need help!

 

Unraid 6.8.3

21 spinning drives (19 data + 2 parity) 136TB capacity

2 1TB nvme cache drives in RAID0

storage-diagnostics-20201207-0935.zip

Link to comment

Nothing obvious, but diags show writing to 2 array disks at the same time, avoid that, also have you tried without encryption? There are a few isolated reports of bad performance when using it, another thing you can try is unpacking to cache using the disk share (/mnt/cache/share) instead of the user share.

Link to comment
51 minutes ago, JorgeB said:

also have you tried without encryption?

Are you asking if I've backed up 122TB of data, disabled encryption, and copied the data back to test disk performance? No. I have not done that.

 

Is there any way to test the read/write speeds of individual disks without disabling the array and testing each disk? I had thought about writing a test file to each `/mnt/disk#` but realized with the array started, I'm still getting hit with parity calculations, so even if I'm writing to 1 disk, I'm reading from all of them and writing to the 2 parity drives.

 

I'm wondering if anyone else has seen a performance hit when dealing with this many drives? Thinking about biting the bullet and ditching the 10 4TB drives in favor of 4 12TB drives. I'd get a capacity bump and get rid of 6 spinning drives which should help parity performance as well as heat and power draw. 

Link to comment
1 hour ago, timekiller said:

Are you asking if I've backed up 122TB of data, disabled encryption, and copied the data back to test disk performance?

No, you could start with just the cache pool, or cache plus one array disk and write to that one only for testing.

 

1 hour ago, timekiller said:

Is there any way to test the read/write speeds of individual disks without disabling the array and testing each disk?

If parity is assigned it will always be involved in any array disk write, but you can use turbo write for better write performance (it won't work if you write to multiple disks simultaneously).

 

1 hour ago, timekiller said:

I'm wondering if anyone else has seen a performance hit when dealing with this many drives?

I've had arrays with 30 drives without issues.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.