Jump to content

Extremely poor NVMe cache drive performance


Recommended Posts

Posted (edited)

I have a 500GB NVMe set up for my cache drive (btrfs) and I'm experiencing some extremely poor performance. 

 

I first noticed it when I set up a development share on the cache drive for a project. I have a Linux VM setup (on a separate SSD mounted with unassigned devices) that mounted the share and the performance on both serving a website and compiling node.js was abysmal.

 

For testing purposes I copied the data from the share to the VM's local filesystem and the compiling performance was approximately 20x faster. Night and day.

I've replaced the NVMe drive to see if it had something to do with faulty hardware but that was not the case.

 

Now I'm still very much a beginner when it comes to Unraid - so it's very possible I just have something configured incorrectly or I'm just going about this all wrong. 

Any help would be greatly appreciated.

Edited by Shobo
Posted

What brand of NVME drive?   Some of them are much slower than you might think :( 

 

You should attach your system diagnostics zip file (obtained via Tools >> Diagnostics) to your next post if you want any sort of informed feedback.

  • Thanks 1
Posted

I've done some additional tests and believe my issue doesn't have anything to do with the cache but instead of mounting shares within my VM.

 

First I tried various other shares and they all ran at approximately the same speed. 

 

Then I made a copy of my VM disk and moved it within a share that also had the project (on the array) and also set it to mount that same share.

Booted it up and tested compiling from the VM's filesystem - it was snappy as I'd expected.

I then tried compiling from mounted share (which again, is the same share the VM is running on) and it was upwards of 20x slower like all previous tests.

 

This makes me believe it isn't the cache at all but something with mounting Unraid shares on my VM that's causing the issue.

 

 

Posted

I've attached the output of bonnie++ on both the local filesystem of the VM as well as the mounted share.

The differences are quite large so something is definitely up with these mounted shares.

 

image.thumb.png.6f4efc8f6735d5ba14083078560cf0fb.png

Posted
15 hours ago, Shobo said:

I've done some additional tests and believe my issue doesn't have anything to do with the cache but instead of mounting shares within my VM.

What do you mean by "mounting shares"? That field in the VM template where you enter the /mnt/user/share and a mount name and then mount the mountname from within the OS?

If so that's a known issue for quite some time. It is theoretically possible to tune that (according to Red Hat) but nobody has ever managed to do it so performance is terrible.

 

You would have better performance mounting smb shares instead. The best performance with NVMe is to create custom smb config to access /mnt/cache/share (to by pass shfs).

Alternatively mounting the NVMe with Unassigned Device will also bypass shfs and give you best performance.

  • Thanks 1
Posted (edited)

Well then, that is great to know. 

I'm quite new to Unraid so wasn't sure there was any way to do this. 

 

1 hour ago, testdasi said:

You would have better performance mounting smb shares instead. The best performance with NVMe is to create custom smb config to access /mnt/cache/share (to by pass shfs).

Absolutely no idea to do this, but it'll give me a project to read into. 

 

1 hour ago, testdasi said:

Alternatively mounting the NVMe with Unassigned Device will also bypass shfs and give you best performance.

I can't do this because the NVMe is my cache drive.

 

Thanks a bunch.

Edited by Shobo

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...