About to Remove UNRAID - Any help would be wecome


9 posts in this topic Last Reply

Recommended Posts

Hello All 

 

Hope everyone is fine this Easter Weekend.. 

 

I am about at the end of my knowledge with UNRAID and got to the point where I am about to revert the server back to a Windows based OS (where it worked without issue for 3 years) but with UNRAID I am getting issues - and this is my hope in that someone could advise / help / suggest something. 

 

So I have a Dell Server, 100GB RAM, and 2 CPU's total over kill for a UNRAID server that is just sitting around and been a NAS - No VM's and a few docker images. 

I have connected at the moment 8 SATA HDD in servers backplane, and a new SSD connected as the cache drive. 

 

The server itself is still under Dell Warrantee and all hardware passes sell tests and I have also sent all hardware logs off to Dell and they have also confirmed no issue with any hardware 

 

The Issue

 

So the issue I am getting is when I copy, or move data around on the server all stars off good I get around 350MB/s write to the server and all looking great. Then about 10GB into the copy it drops off and goes down to around 60MB/s (screenshot below) 

image.png.2653781e097c95e8beff42687e5f6870.png
 

Whne I use Glances to take a look at whats going on I see that IOWait shoots up to 14% in the red and I get warnings or critical alerts showing up. 
 

image.png.375a00eed38ee5d4494177a5ba994404.png

 

image.png.c30e42de17423cf6b6f45ed6515aee39.png

UNRAID shows the CPU usage and while the copy is been done the CPU's are showing around 12% so its not like they are been pinned image.png.9595795437e97bd995d2eb98a26f84f9.png

 

After the file copy has compleated - around 2min later everything goes back to normal. 

 

I have also found I can trigger warnings, if I go into docker and select the check for updates for all dockers running - this will give me CPU_IOWAI errors in Glances. 

 

The CPU's in this server are 2 x Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz and are both 8 core each - so in how may high end CPU's but more than enough to run what I am doing, and work 100% under a Windows OS and even running a number of VM's in Hyper V with huge data throughput. 

So this is where I am totally confused and I dont know where to go from here. I have also attached my diagnostic file if someone could take a look I would be very grateful. 

In case its a config issue I would also be open to contracting someone with good standing on the forum to work to fix the issues as my knowledge of the underlying system in UNRAID isn't great. 

 

From what I read on the UNRAID forums - this issue isn't just me. Lots of others have reported the same issue - but never found a fix. Could it be something within UNRAID itself? Bug maybe? 

TIA 

 

diagnostics-20210402-1334.zip

Edited by IKWeb
Link to post

You should try setting Tunable (md_write_method) ( in disk setting ) from auto to reconstruct write, but this not help all time. Write to cache SSD then move back to array by mover is a common solution.

That is true for Unraid array is not a high performance storage, but you shouldn't compare it's performance in Windows when it is a single disk.

 

Most people will use Unassigned disk plugin, SSD/RAID cached pool, large RAM cache ... etc to improve the performance for different need. Last, lot of CPU core but with low frequency also a problem cause, array will also limit by its slowest member disks too.

 

Different hardware/config have large different performance result, so you will found some people have decent result but some just fair.

 

For example

 

A 1-data+1-parity disk setup ( Ryzen 1700 + 10Gb NIC ) will got 130MB/s ( power save mode @ 2Ghz ) to 250MB/s SMB transfer speed ( high performance @ 3.4Ghz ), but same CPU with large RAM cache approach, it can transfer large file in 1GB/s.

 

image.png.d46634473523b8bc31f035f3b0107d6a.png

 

 

Edited by Vr2Io
Link to post

Honestly, what you're seeing makes perfect sense and fits with the general capabilities.

unRAID doesn't seem to use system memory as a data cache, as aggressively as Windows does on similar hardware.

That 58MB/s data rate seems reasonable if you're reading/writing to the same disks, and have parity active; there's a lot of expensive work going on when doing that.

Reconstruct/Turbo write might help, but as @vr2gb mentioned, you'd get the highest rates doing segregated move from-to task. It might not be the shortest in terms of time.

If you have a LOT of data that needs to be moved between disks you might be better off disabling parity during the move(s), then rebuild it.

 

 

Link to post

Thanks both. 
 

So basically if you have more than say 10GB of data that you want to write to the server in one go. UNRAID is basically useless as it craps itself and takes what starts out at 300/400mbs and drops down to 50mbs 😮

 

Think it’s time to bin UNRAID for me and move to a solution that can take data that’s been written to it without crapping itself. 
 

thanks for taking the time tho for the above replies 👍

Link to post
8 minutes ago, IKWeb said:

 

So basically if you have more than say 10GB of data that you want to write to the server in one go

Most people do not have a parity drive implemented in this stage when they first set up the array which means data transfer can happen at the full speed of the individual disks. Parity is "turned on" after the initial data dump.  Calculating parity in real-time has a cost.

 

The volume added on a daily basis after the initial data transfer will determine if you are OK with real-time parity updates or if you prefer to use the cache disk method or enable Turbo Write.

Edited by Hoopster
Link to post
Posted (edited)
26 minutes ago, Hoopster said:

Most people do not have a parity drive implemented in this stage when they first set up the array which means data transfer can happen at the full speed of the individual disks. Parity is "turned on" after the initial data dump.  Calculating parity in real-time has a cost.

 

The volume added on a daily basis after the initial data transfer will determine if you are OK with real-time parity updates or if you prefer to use the cache disk method or enable Turbo Write.


@Hoopster so the share I am doing the above copies to is set to run from the cache. Then I assume move to the array over night. 
 

but those speeds writing to the cache is so bad. 
 

I would be happy to write to the cache at full speed then for it to take 50mb over night as the mover is invoked. 
 

but I though this was what happens when you tell the share to use the cache drive??

Edited by IKWeb
Link to post
1 hour ago, IKWeb said:

but those speeds writing to the cache is so bad. 
 

I would be happy to write to the cache at full speed then for it to take 50mb over night as the mover is invoked. 

You won't get the full speed of an SSD unless you are writing from SSD to SSD over a 10Gb network or internally in the server.  If you are writing from an HDD to an SSD over a 1Gb network you should be able to come close to saturating the network; however, there are always some overhead factors that will reduce that somewhat.

 

1 hour ago, IKWeb said:

but I though this was what happens when you tell the share to use the cache drive??

If you have the share set to "use cache: yes"  you are correct, this is how it should work.  Writes the share are first cached on the cache drive with no parity calculations involved as cache is not part of the parity-protected array.  When the mover runs at night, the data is moved from cache to the HDD(s) that are included in the share and parity is calculated and written then.

 

There are a few threads in the forums about improving write speed with SSDs, in particular Samsung SSDs.  This involves formatting the cache drive with 1MiB partition alignment.  See the unRAID 6.9.0 release notes for more information.

Link to post

Crushing a 1Gb (one) link with a modern spinning rust disk is child's play.  1Gb is 125MB raw, or roughly 100MB/s after overhead.  If you see 80MB/s you're doing great in my opinion, if you're going direct to the array.  If you're caching to an SSD first, you should be able to saturate a 1Gb link, but SSDs introduce another "X" factor: what is their TRIM status.  Are your sure your data from the first post wasn't a move to rust, or a move between rust on the same box?

 

I have a config not much different than yours (see sig), and I can say I've experienced what you're seeing in terms of speed, hence my assertions.  When I first built mine and began to populate it from over the network targets, 80MB/s was about my max speed (i'm on 1Gb here) WITHOUT parity active.  With parity turned on and without Reconstruct/Turbo Write active i would see about half that.

 

If you want the best short term transfer speeds, you need to make sure you're caching to SSD, and that it's cleaned up/TRIM'ed regularly (which it should be.)  Then you'll need to let the mover do it's thing.  Watching Dashboard or Main, you can see which disks are active. If it's just a data drive and the parity, it's going to hurt badly for overall throughput, since by design the parity drive is getting clobbered by both read AND write actions.  This is where Reconstruct/Turbo Write can help you, and you should get damn close to maximum rate on the array disks.  Otherwise, even with my near 200MB/s max speed drives, I can see write rates below 60MB/s (sometimes quite a bit lower, depending on the types and sizes of files)  if i'm just writing to a single drive plus parity.  unRAID doesn't have any significant intelligence about turning on/off Reconstruct/Turbo Write, in that it can't tell "you're moving 100s of GB of data on, we should turn everyone on and go TURBO!", so if you are going to drop a ton of data on the array, you should kick it on yourself.  There's actually a button on the Main page to help do that: "SPIN UP", assuming you have 'Tunable (md_write_method):' set to AUTO, you should see all the disks get used in terms of read/write activity.

 

Since you're apparently familiar with Windows, it's helpful to remember that unRAID is much more analogous to Storage Spaces (hurl!), than it is to an actual RAID array.  If you're throwing around lots of large files on a regular basis between disks, then it might not be the right choice.  It is, however, a fantastic choice for a glorified digital file cabinet, with drawers of potentially different sizes, with a little bit of added "security" in the form of a parity drive.

 

Finally, as I look again at your first post, that speed chart is from a Windows machine.  More details about how it came about would be beneficial. (VM? disk passthrough? network share? etc.)

Link to post
23 hours ago, IKWeb said:

Thanks both. 
 

So basically if you have more than say 10GB of data that you want to write to the server in one go. UNRAID is basically useless as it craps itself and takes what starts out at 300/400mbs and drops down to 50mbs 😮

 

Think it’s time to bin UNRAID for me and move to a solution that can take data that’s been written to it without crapping itself. 
 

thanks for taking the time tho for the above replies 👍

 

With the proper set up you can achieve excellent transfer speeds with Unraid.

 

I have set up a cache pool using nvme disks and below is an example of a file transfer of 80 GB from my PC to the server over a 10G link.

Speed hovers around 640 MB/s (or 5.1 Gb/s).  Note: to complete the remaining 29.4 GB it needs 60 seconds.

 

image.png.aa858f932f76b90726c73d79392b496f.png

 

image.png.936b1d69b523f618df3951c706d6e9dc.png

Edited by bonienl
Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.