Another performance problem

[thany]

I'm downloading a big file off my server. It started out at 270MB/s over the network. That's good.

 

Now, a few hours later it's gotten down to 50MB/s.

 

The CPU can do more, the disks can do more, the network can do more, the receiving end can do more. I don't understand what's holding up? What is making it slow, even though every component should be able to do more? Going off the read speed of the disks, it should be able to do 200MB/s at the very least.

 

Even if it were reading off the very end of the disk platters, it should still be capable of 100MB/s. It's just two gigantic files, so I have to assume they are not so heavily fragmented as to impact performance.

 

Why is it so slow?

 

/Edit

Very occasionally it's going at normal speed (250-280MB/s) but it also very quickly grinds itself down to what I'm reporting here. So it is ab-so-lu-tely capable of higher speeds, so something must be slowing it down. Something somewhere somehow. But what??

unraid-diagnostics-20221212-1207.zip

Edited December 12, 2022 by thany

[JorgeB]

You can first try reading the file locally to rule out any outside issues:

 

pv /path/to/file > /dev/null

 

[Frank1940]

Have you looked at the Dashboard and Main tabs to see what else might be going on?  (There is a toggle on the far right of the "Array Devices" section to switch between read/writes speeds and IO counts.) 

[thany]

On 12/12/2022 at 12:29 PM, JorgeB said:

You can first try reading the file locally to rule out any outside issues:

 

pv /path/to/file > /dev/null

 

 

But it's not just one file. It's any file. Okay, tried one, and it's happily doing 220MB/s.

 

First, another performance problem yet again. Write performance is also *terrible* atm. It started out fine(-ish) at ~60MB/s. Again, the disks can all do at least triple that, if not more. Now copying to the array is completely stuck.

 

On 12/12/2022 at 1:23 PM, Frank1940 said:

Have you looked at the Dashboard and Main tabs to see what else might be going on?  (There is a toggle on the far right of the "Array Devices" section to switch between read/writes speeds and IO counts.) 

 

Yeah, so the array shows a couple of MB/s read seemingly randomly, just intermittently, like nothing's going on. Meanwhile I'm trying to copy over some large files which has grinded to a complete halt, to the point that the application doing it, is frozen.

 

File Activity plugin shows *no files* being accessed. Where are those MBs per second coming from then??

 

So then, what is it reading, why is SMB access (or whatever is keeping it) being blocked, and why is it (whatever it needs to read) so terribly slow?

 

Attached diagnostics of the current situation. Hopefully some information pertaining to this problem, is in there.

unraid-diagnostics-20221218-2057.zip

Edited December 18, 2022 by thany
Tried `pv` command

[thany]

It feels to me like "sometimes" just "something" is locking things up. From the server's perspective, all's well & dandy, but then I can't do anything via SMB. Also those few MB/s while it should be completely idle, and File Activity plugin showing nothing. This really feels like something terribly spooky is going on.

 

Meanwhile it's going again.. ¯\_(ツ)_/¯

Edited December 18, 2022 by thany

[JorgeB]

13 hours ago, thany said:

But it's not just one file.

Yes, but you mentioned:

On 12/12/2022 at 11:11 AM, thany said:

It's just two gigantic files

so easy to test.

 

13 hours ago, thany said:

Write performance is also *terrible* atm. It started out fine(-ish) at ~60MB/s.

That's about right with default writing mode, see turbo write.

[thany]

I'll try turbo write.

 

I still don't understand why:

1) It is so slow by default. What is the holdup? I mean if *every* component can go faster, why isn't it?

2) Turbo write isn't the default setting, if it's clearly better.

 

If turbo must not become the default, then I wonder why anyone at all would bother putting a 10GbE card in their unRAID system... If 60MB/s "sounds about right", then 1GbE is more than anyone will ever need. Or am I mistaken?

Edited December 19, 2022 by thany

[Frank1940]

2 minutes ago, thany said:

If turbo must not become the default, then I wonder why anyone at all would bother putting a 10GbE card in their unRAID system... If 60MB/s "sounds about right", then 1GbE is more than anyone will ever need. Or am I mistaken

 

If you are writing directly to the array, you are right-- a 1Gb/s connection is adequate.  However the read speed can be (and often is) much faster than that depending on the read speed of the drive itself.  (Remember we will have file access time on the server and file creation overhead on the client to consider in the read case.)   Remember that Unraid's initial philosophy was that it was to be a storage system intended for applications where the primary use was write-once and read-many!    Write speed was not a primary consideration in either its concept or design.

 

What most of us have done to use fast writing SSD cache drives for transferring files to the array after the initial data loading.  (You could also use M.2 SSDs for even more speed!)  Then the files are moved off of the cache_drive/cache_array by 'mover' when the server is idle.  These can accept data much faster than a 1Gb/s Ethernet connection. 

 

BTW, if you are writing a large number of small files (say, <1MB), file creation overhead on Unraid will definitely impact the transfer speed.  

[thany]

Thanks for explaining. However, it still doesn't tell me why write speed is slower than the write speed of a single disk.

 

I cannot imagine for the life of me, what could possibly cause such a slowdown. Even if writing is quite badly designed, it should not cause a 60%+ overhead in performance.

 

To put it differently, if this isn't an intentional limitation, then what?

Edited December 19, 2022 by thany

[Frank1940]

49 minutes ago, thany said:

However, it still doesn't tell me why write speed is slower than the write speed of a single disk.

Read the first post in this thread:

         https://forums.unraid.net/topic/50397-turbo-write/

 

[itimpi]

15 hours ago, thany said:

Thanks for explaining. However, it still doesn't tell me why write speed is slower than the write speed of a single disk.

 

I cannot imagine for the life of me, what could possibly cause such a slowdown. Even if writing is quite badly designed, it should not cause a 60%+ overhead in performance.

 

To put it differently, if this isn't an intentional limitation, then what?

It is worth reading this section of the online documentation accessible via the ‘Manual’ link at the bottom of the GUI or the DOCS link at the top of each forum page.

 

it makes it clearer why you will never achieve the raw write speed of a drive when writing to a parity protected array in the default write mode.

[thany]

Alright thanks, I'll have a read through it.

 

In the mean time, "turbo write", as it seems to be coloquially called, makes *all* the difference. Before having read what it actually does under the bonnet, it feels like it should be the default. After all, other NAS systems also don't cap data transfers by setting an "incorrect" write mode internally. It's good to have this setting exposed for admins, but it might be better if it automagically did the best thing.

[thany]

I think I get the gist of it. Turbo write makes more sense to me. In a servery environment, it's likely that the disks are already spinning anyway. It might even make sense to keep the disks spinning forever, especially when using drives designed for NAS/server contexts.

 

However, it also made "cache" immediately click in my head. Since cache (for me at least) goes on an SSD, and doesn't get written to spinning rust straight away, I could use that on shares that typically take small files, and/or scheduled backups and still have the main array spun down during off hours. Feels like a win-win.