Poor USB 3 external drive and Unraid performance


Recommended Posts

Even when copying large 1+ GB files I'm only getting around 45 to 50 MB/sec copy rates using Krusader with an unassigned USB 3 external drive to my Unraid array of fast drives on fast hardware with lots of RAM. That's even a bit slower than I get over the network doing similar copies which are still really slow compared to most anything else. Is Krusader making things even worse?

 

A decent USB 3 external drive can generally match or outperform Gigabit ethernet connections but apparently not with Unraid? Why not?

 

With even a cheap low-end Qnap NAS, with a hugely less capable CPU and less RAM, I get more than double the rate copying from the same external USB drive using Qnap's "File Station" file manager which is similar to Krusader. Likewise just about any Linux , Mac, or Windows PC can also copy from the same drive at more than double the rate Unraid can manage on similar or even lower end hardware.

 

It seems like something is wrong? I know about cache, and one of my two Unraid servers has mirrored SSD cache, which Unraid 6.8.3 has been inexplicably trashing the SSDs with excessive writes, but that only helps to a point when doing large copies. There seems to be something in Unraid causing it to perform at around half the rate of most anything else? Is there perhaps a much faster way than Krusader to copy data from a USB 3 drive to Unraid? Or is this just mostly about how slow the Unraid parity implementation is compared to even a low power ARM CPU doing software RAID 5 in a low-end much more power efficient NAS?

 

Any help would be greatly appreciated so my large copies can take 4 hours instead of 8+ hours?

unraid speed.jpg

Edited by dev_guy
Link to comment
1 hour ago, dev_guy said:

I know about cache

But you don't mention if you are writing to a cached User Share.

 

1 hour ago, dev_guy said:

Unraid parity implementation is compared to ... RAID 5

Unraid IS NOT RAID. There is no striping. Unraid has lots of advantages over RAID, but it will never be as fast as striped RAID.

Link to comment
On 4/9/2021 at 3:44 PM, trurl said:

But you don't mention if you are writing to a cached User Share.

 

Unraid IS NOT RAID. There is no striping. Unraid has lots of advantages over RAID, but it will never be as fast as striped RAID.

There is no cache in the machine I'm referencing. My other Unraid system has a pair of mirrored 1TB Samsung SSDs for cache but not this one. Comparing Unraid write performance to RAID 5 is not an unfair comparison especially when Unraid is running on much higher performance hardware yet has less than half the write performance. As I understand it, RAID 5 has no performance advantage for writes as it has to compute parity roughly like Unraid does for writes.

 

Basically 40 MB/sec for a direct transfer from a USB 3 drive capable of almost 3 times that speed, to fast Sata 6 Unraid drives capable of more than 4X that speed, is really disappointing. So far nobody has explained why or how I might improve things besides perhaps adding cache which creates its own problems and won't help for transfers larger than the cache.

Link to comment
On 4/9/2021 at 3:11 PM, itwonttakelong said:

Well you made me fill good. I'm moving at 132.5 MiB/s using a 10 TB WD elements external drive on a USB 3.0 with old 8 GB of RAM, and I thought I was going slow. oh and running 6.9.2 maybe I'm just lucky.

Was that for large sustained copying? Are you using Krusader? For me, Krusader initially reports optimistic transfer rates but they quickly decline to less than one third what you're seeing. I think the initial high rate is from RAM cache.

Link to comment
36 minutes ago, dev_guy said:

As I understand it, RAID 5 has no performance advantage for writes as it has to compute parity roughly like Unraid does for writes.

The way parity is handled in the two cases and the resulting I/O is completely different.    A description of how unRaid actually writes with the resulting I/O can be found here in the online documentation accessible via the ‘Manual’ link at the bottom-m of the unRaid GUI.

Link to comment
1 hour ago, itimpi said:

The way parity is handled in the two cases and the resulting I/O is completely different.    A description of how unRaid actually writes with the resulting I/O can be found here in the online documentation accessible via the ‘Manual’ link at the bottom-m of the unRaid GUI.

They might be different but I don't think it's unfair to compare a low power NAS with similar drives in a protected array to Unraid in terms of write speeds. If I put say four 10TB drives in a commercial NAS running RAID 5 vs the same four drives into an Unraid server offering similar net storage and protection, I don't think it's unreasonable to expect at least somewhat similar performance especially when the Unraid box has several times the CPU performance, more RAM, etc. Obviously that's apparently not the case, but it's not like I'm comparing apples and oranges. The price of the drives far exceeds the price of the NAS or server hardware plus Unraid license. So they're also cost competitive.

Link to comment

On neither server type is the CPU likely to be the performance limiting.    UnRaid has some design objectives that are not there in RAID5 systems:

  • Not all drives need to be spinning when writing files to the array.  In fact on a typical system on the target drive and the parity drive need to be spinning. 
  • Disks of mixed sizes can be used, and all parity information is on a particular drive.
  • each disk has a self contained filing system.  This means any individual file must reside on a single drive and thus striping techniques cannot be used.  Any single file can never be read faster than a single drive can deliver.

the above list is not exhaustive but gives an idea that the objectives are different between the two approaches.


The link I gave describes what I/O is involved in writing to a an unRaid parity protected array and it is very different to the I/O pattern for a RAID5 array.   The net effect is that updating a sector on the unRaid main array involves more I/O operations plus a rotational delay that is not present on RAID5 systems meaning that performance will never get even close to a RAID5 system using the same disks.

 

If performance is your main objective then unRaid is not the best system to use.    However for most home users it’s other attributes mean it suits them well and performance is sufficient for their needs.

Link to comment
55 minutes ago, trurl said:

Looks like you've been using Unraid for several years. I don't understand why you are just now expecting it to perform like striped RAID.

To constructor and itimpi I have been using Unraid for several years but my original server has 1TB of SSD cache and I've never tried to do large transfers from an external USB 3 drive with it. The newer server does not have cache and was mainly intended to consolidate all the stuff I have on large portable hard drives.

 

I'm aware of the advantages with Unraid but some of those, such as mixing drive sizes, are now available on commercial NAS units if you want to go beyond RAID 5. But I didn't realize there's such a massive performance hit, without cache, even when you remove network performance from the equation with Unraid.

 

My original post wasn't so much a complaint but to ask if I'm doing something wrong? For example itwonttakelong is claiming 130+ MB/sec while I'm only getting around 40 MB/sec. To copy a USB 3 directly connected 8TB external drive to my Unraid server I'm looking at 50+ hours with large files and more like 100+ hours with smaller files. That's just really disappointing.

Link to comment
4 hours ago, dev_guy said:

For example itwonttakelong is claiming 130+ MB/sec while I'm only getting around 40 MB/sec. To copy a USB 3 directly connected 8TB external drive to my Unraid server I'm looking at 50+ hours with large files and more like 100+ hours with smaller files. That's just really disappointing.

Problem not relate USB 3.0, Unraid can transfer speed at max USB 3.0, people just telling you could do better.

Sometimes I will copy data from external USB disk directly to array, no speed issue. You need identify the bottleneck.

 

- Turbo write enable

- Any other loading in array

- Controller have enough bandwidth for all disk access in same time, diskspeed docker app could help you to identify.

 

It is true Unraid not always perform well, but it simple, you need understand why that slow and try your best to overcome.

Edited by Vr2Io
Link to comment

Thanks to all for your suggestions. I have, for example, defined a share with only 3 of 8 drives. Enabling reconstruct write for the md_write_method (which seems to be the equivalent of "turbo write" in 6.8.3?) causes not just the 3 drives in the share to spin up but all 8 drives to spin up even though the other 5 drives have nothing to do with the share. Why? Unless I'm missing something that not only seems like a bug, but it defeats one of the major advantages of Unraid. Please correct me if I'm wrong, but "turbo write" seems to be broken? And even with "turbo write" enabled speeds only roughly double to 80 to 90 MB/sec which is still well below the 120 to 130 MB/sec on a much lower power commercial NAS.

Link to comment
On 4/11/2021 at 8:28 PM, Vr2Io said:

Problem not relate USB 3.0, Unraid can transfer speed at max USB 3.0, people just telling you could do better.

Sometimes I will copy data from external USB disk directly to array, no speed issue. You need identify the bottleneck.

 

- Turbo write enable

- Any other loading in array

- Controller have enough bandwidth for all disk access in same time, diskspeed docker app could help you to identify.

 

It is true Unraid not always perform well, but it simple, you need understand why that slow and try your best to overcome.

Thanks for your reply. I know USB 3 can be very fast but not with Unraid. My 40 MB/sec performance is with no other connections to the server besides the Unraid and Krusader GUI web interfaces. And the controller is the same LSI/Avago 8 port SAS in IT mode so many Unraid builds use and is capable of much better performance under other operating systems. The controller, and drives, are not the bottle neck.

 

See my other post about turbo write. It needlessly spins up all the drives when only a few drives are assigned to the share in use. Even even with all drives spun up I still only get around 80 - 90 MB/sec which is still well below the 120 MB/sec a cheap NAS running RAID 5 can manage with the same external drive.

Link to comment
41 minutes ago, dev_guy said:

Thanks to all for your suggestions. I have, for example, defined a share with only 3 of 8 drives. Enabling reconstruct write for the md_write_method (which seems to be the equivalent of "turbo write" in 6.8.3?) causes not just the 3 drives in the share to spin up but all 8 drives to spin up even though the other 5 drives have nothing to do with the share. Why? Unless I'm missing something that not only seems like a bug, but it defeats one of the major advantages of Unraid. Please correct me if I'm wrong, but "turbo write" seems to be broken? And even with "turbo write" enabled speeds only roughly double to 80 to 90 MB/sec which is still well below the 120 to 130 MB/sec on a much lower power commercial NAS.


No - that is correct behaviour.   If you read this section of the online documentation accessible via the ‘Manual’ link at the bottom of the unRaid GUI it explains why.

 

basically ‘turbo write’ trades off the being able to keep drives spun down wherever possible against better performance.    However you still do not get some of the additional speed other RAID types can achieve using striping techniques because of the fact that files and parity are restricted to existing on single drives.  
 

No modern NAS is constrained by the CPU power as they all have more than enough to handle parity calculations.     They are nearly always constrained by the I/O rates the drives can support and the I/O pattern that arises from the mix of capabilities to be supported.

 

Link to comment
12 minutes ago, dev_guy said:

Even even with all drives spun up I still only get around 80 - 90 MB/sec

That only depends on the drives/controller used, like mentioned I can get 200MB/s+ in an optimal scenario.

 

14 minutes ago, dev_guy said:

well below the 120 MB/sec a cheap NAS running RAID 5 can manage

Unraid is not RAID, it doesn't stripe disks, it will never be as fast as those, but it has other advantages.

 

Link to comment
4 hours ago, itimpi said:


No - that is correct behaviour.   If you read this section of the online documentation accessible via the ‘Manual’ link at the bottom of the unRaid GUI it explains why.

 

basically ‘turbo write’ trades off the being able to keep drives spun down wherever possible against better performance.    However you still do not get some of the additional speed other RAID types can achieve using striping techniques because of the fact that files and parity are restricted to existing on single drives.  
 

No modern NAS is constrained by the CPU power as they all have more than enough to handle parity calculations.     They are nearly always constrained by the I/O rates the drives can support and the I/O pattern that arises from the mix of capabilities to be supported.

 

 

Thanks for the link but I still don't understand why ALL the drives have to be spun up for writes to a share that only uses a few drives? The other drives seem to be needlessly spun up as they're not part of the share and show no writes with their reported used space remaining the same even after copying a large amount of data to the share? I understand that "turbo write" is working as documented, but that doesn't mean it makes any sense?

 

Yes RAID 5 has some READ performance advantages but it's generally considered to be at a disadvantage during writes. So it doesn't have some unfair striped advantage over Unraid. See: http://rickardnobel.se/raid-5-write-penalty/

 

You might be correct that most modern CPUs can handle the parity calculations without it being a limiting factor.  But, to put this really simply, I can take the exact same drives, motherboard, and disk controller, running a different OS using software RAID 5, and get far better performance. So it's NOT the drives, controller, etc, that are constraining the performance. It's Unraid. Even with "turbo write" Unraid still takes 30+% longer to perform writes.

 

What I'm getting out of all of this is I need to just accept Unraid has really poor write performance and, short of using cache as a band aid, there's nothing that can be done?

Link to comment
15 minutes ago, dev_guy said:

Thanks for the link but I still don't understand why ALL the drives have to be spun up for writes to a share that only uses a few drives? The other drives seem to be needlessly spun up as they're not part of the share and show no writes with their reported used space remaining the same even after copying a large amount of data to the share? I understand that "turbo write" is working as documented, but that doesn't mean it makes any sense?

Because to calculate the contents of a particular sector on the parity drive, then the corresponding sector has to be read from ALL data drives to allow the contents of that sector on the parity drive to be calculated.

Link to comment
26 minutes ago, dev_guy said:

I can take the exact same drives, motherboard, and disk controller, running a different OS using software RAID 5, and get far better performance. So it's NOT the drives, controller, etc, that are constraining the performance. It's Unraid.

I don't think anyone has said anything here to dispute that.  We all know unRAID is not as performant as RAID 5 and other RAID levels. 

 

There are recognized and acknowledged differences between the two.  As the name suggest unRAID is not RAID but does offer some "RAID-like" capabilities in the form of parity protection against drive failure.  The tradeoff is slower read and write speeds for the ability to use any size disk to expand storage, the ability to read a drive independently outside of the array, and protection from losing ALL of your data if more drives than are protected by parity are lost.

 

40-60 MB/s is typical with real-time parity calculation and 80-110 MB/s is typical with Turbo Write which has the tradeoff of keeping all disks spinning.

 

Some speed issues come down to the controller and drives, but even in the best-case scenario, unRAID will always be slower than RAID. We accept that because of the advantages it provides in the way we utilize the server.

 

You have to decide if those advantages are worth the tradeoffs.

Link to comment

I am not sure I can entirely relate but I am preclearing 2 - 8tb drives via USB 3.0 (or maybe 3.1 - who can tell anymore).  Getting a steady 145 MB/S read.  I guess I should feel lucky although I know I am not constrained by CPU.  I have an i9-10850K and it's running about 12% with these being the only unRaid tasks being performed (no parities, array of two drives with no shares).  Per the Pre-Clear, my avg speed has been 175mb/s

image.png.ffc170de17f673f67f6149e9efe5388b.png

Edited by Hawkins12
Link to comment
2 hours ago, Hawkins12 said:

I am not sure I can entirely relate but I am preclearing 2 - 8tb drives via USB 3.0 (or maybe 3.1 - who can tell anymore).  Getting a steady 145 MB/S read.  I guess I should feel lucky although I know I am not constrained by CPU.  I have an i9-10850K and it's running about 12% with these being the only unRaid tasks being performed (no parities, array of two drives with no shares).  Per the Pre-Clear, my avg speed has been 175mb/s

image.png.ffc170de17f673f67f6149e9efe5388b.png

Preclearing is not the same as writing data. No parity is written for preclearing.

Link to comment

You seem to not be getting it, let me try one last time:

 

- Unraid is not RAID, data isn't stripped, it will never be as fast as other RAID solutions

- With reconstruct write enable (aka turbo write) you can still get decent speeds, if using fast disks and there aren't controller or other bottlenecks

- Reconstruct write is faster at the expense of spinning up all the array disks, doesn't matter what disks are used for that share

- Reconstruct write can only be as fast as the slowest disk in the array at that point, good way to see that is running a parity check, max writing speed will be similar to that at any point.

 

Here's an example of turbo write enable on a server with fast disks without controller bottlenecks:

 

1616904254_Screenshot2021-04-1409_07_28.png.2af9eeef24293088881df947a32e9d3f.png

 

Note also that speed will decreased as the disk fills up, since they are considerably faster in the outer tracks.

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.