Do CPU and memory impact SMB/NFS performance at all?

[Raident]

Essentially, I'm wondering if there's any quick and dirty way to speed up the transfer of large quantities of small files.

[Frank1940]

Yes! Two ways--- assuming that you are talking about writing files to the array.  .  

 

First, use a cache drive.

 

Second, go to Settings    >>>  Disk Settings     Then change the 'Tunable (md_write_method):' to "reconstruct write".

[trurl]

I would suggest not getting a cache involved until after the initial data load. When transferring a large volume of data to their server people often run into trouble if they are using cache because it has limited capacity and there are a lot of nuances with getting caching configured just right.

 

The second suggestion, also known as Turbo Write, is what I recommend.

[Raident]

First of all, thanks for the suggestions, Frank1940 and trurl.

 

To provide a bit of background on my setup, I'm already using Turbo Write and unfortunately I have no spare drive bays for a cache drive - the array was set up years ago, before the cache drive concept was introduced and at this point putting in a cache drive would require a pricey 3 drive (bigger parity + data + new SSD) upgrade. It's definitely something I'll seriously think about when the array reaches maximum capacity in about 2 years and I need to upgrade the array anyway, but for now I'd like to avoid spending money on new drives.

 

The reason I asked about CPU and memory is because this is a VM with only 1 vCPU and 2GB of RAM assigned to it, and those could be expanded very easily with just a few button clicks, if it would help at all.

[Frank1940]

There have been a few (or, perhaps, more than a few) persons who have used double sided tape (or Velcro) to secure an ssd to the inside of the case.   You do have to have a spare SATA port though.  

[Raident]

5 minutes ago, Frank1940 said:

There have been a few (or, perhaps, more than a few) persons who have used double sided tape (or Velcro) to secure an ssd to the inside of the case.   You do have to have a spare SATA port though.  

There are actually no SATA ports in the traditional sense - the drive bays are connected to a backplane, which in turn is connected to the mobo via some kind of proprietary (or maybe enterprise-grade?) connector.

 

NVMe via an adapter is theoretically possible as the PCIe x16 slot is open, but that gets really expensive, really quickly and also poses its own set of compatibility problems with VT-d passthrough, questions about whether it'll even be recognized by an older pre-Z97 system, etc.

[trurl]

5 hours ago, Raident said:

To provide a bit of background on my setup, I'm already using Turbo Write and unfortunately I have no spare drive bays for a cache drive - the array was set up years ago, before the cache drive concept was introduced and at this point putting in a cache drive would require a pricey 3 drive (bigger parity + data + new SSD) upgrade. It's definitely something I'll seriously think about when the array reaches maximum capacity in about 2 years and I need to upgrade the array anyway, but for now I'd like to avoid spending money on new drives.

I'm pretty sure cache drives have been part of Unraid as long as I have been using it. Cache pools were only introduced in V6 though.

 

Perhaps I am missing some implication that you have not made explicit, but how can adding a cache drive require a larger parity disk? The only thing that would require a larger parity disk is if you want to use an array data disk that is larger than your current parity.

[Raident]

3 hours ago, trurl said:

I'm pretty sure cache drives have been part of Unraid as long as I have been using it. Cache pools were only introduced in V6 though.

 

Perhaps I am missing some implication that you have not made explicit, but how can adding a cache drive require a larger parity disk? The only thing that would require a larger parity disk is if you want to use an array data disk that is larger than your current parity.

That is the problem. To add a cache drive, I would need to remove one of the 3 data disks to make space physically, which means that one of the 2 remaining disks would have to double in size just to keep the array at its current size, which in turn means that the parity disk needs to be doubled in size as well.

 

And needless to say, I can't just download more RAM hard drive space 😉

[trurl]

If you have a slot then you could add SATA ports that way, and as mentioned an SSD can just be stuck anywhere if you can also provide power to it.

 

But, cache isn't really critical and maybe just waiting until you outgrow this system and do a new build does make sense. If you are already doing Turbo Write then I guess you will just have to live with the speed.

 

We haven't really discussed how the transfer is being done though. Is this over the network?

[Raident]

There are no SATA, PCIe or even molex power cables. This is a prebuilt OEM system.

 

And yes, the transfer is being done over the network. In this case I was backing up my Steam library (hundreds of thousands of tiny configuration files along with a few huge archives containing the game assets) via Samba, but NFS under a similar scenario is similarly slow.

[Frank1940]

2 hours ago, Raident said:

hundreds of thousands of tiny configuration files along with a few huge archives containing the game assets

The problem is the file management overhead.  And that is further compounded by the time required for the physical movement of the read/write heads between the file allocation tables and the data area of the disk followed by the time delay involved with the disk rotation to bring the data area of the platter under the head.  I find that the transfer rate for backing up my data files is about half of that of writing a bluRay iso files even though I am using a cache drive.

[Raident]

21 hours ago, Frank1940 said:

The problem is the file management overhead.  And that is further compounded by the time required for the physical movement of the read/write heads between the file allocation tables and the data area of the disk followed by the time delay involved with the disk rotation to bring the data area of the platter under the head.  I find that the transfer rate for backing up my data files is about half of that of writing a bluRay iso files even though I am using a cache drive.

Short of creating an archive, I don't suppose there's some kind of way to get the sending and receiving sides to simply treat all of the small files as a contiguous block, is there?