How does Unraid spread files across the disks?


Arcaeus

Recommended Posts

2 hours ago, Arcaeus said:

Or is it slower due to a parity check

Do you mean you are doing a parity check at the same time? Parity check will slow mover down, and mover will slow parity check down. How could it be otherwise since they are both competing for use of the same disks. Parity check uses all disks.

2 hours ago, Arcaeus said:

And how often do you have your parity check set to?

Most people do a monthly parity check. Unraid parity is maintained realtime. Parity check isn't necessary to maintain parity. Any write operation (which includes what mover is doing) in the array updates parity at the same time. The check is just a check.

 

Link to comment
  • Replies 51
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

@johnnie.black Thank you for sending that little walkthrough over. Super helpful.

 

Ok so I got the docker image back and things seem like they're pretty much up and running. However, my cache drive is still not emptying out regularly and I don't want the dockers to lock up again due to it being full. I've manually ran the Mover, but it seems to complete without really moving anything. 

 

Why would this be? Do I need to pause my downloads in Deluge so the files aren't being used and it can move them?

Link to comment
1 hour ago, Arcaeus said:

Do I need to pause my downloads in Deluge so the files aren't being used and it can move them?

Probably simpler to just:

  1. shut down Deluge
  2. run mover
  3. Compute... Downloads on the Shares page to make sure it doesn't still have any files on cache
  4. set Downloads to cache-no so no more Downloads will be written to cache
  5. restart Deluge
Link to comment

It depends on how much you write between scheduled moves, and the capacity of cache. Caching writes is only one purpose of cache. It is also used for your dockers and VMs so they won't take a performance hit from parity updates, and so your dockers and VMs won't keep parity spunup.

 

And scheduling mover to run more often may not solve the problem. Mover works best if it is running during periods of inactivity. If you are writing a lot and trying to move at the same time you can expect things to slow down as different processes try to use the same disks.

 

You will have to work out for yourself which user shares, if any, you need cached. I don't cache my downloads. But if your downloads require some post-processing, then doing all that on the parity array will be slower. Everything is tradeoffs.

Link to comment
11 minutes ago, Arcaeus said:

Now the more disks you have in the system, the faster your overall transfer/load speed should be, correct? Doesn't Unraid use a Raid 5 kind of system to spread the data across the array?

No Unraid doesn't work like that at all. Each disk is an independent file system, and each file exists only on one disk. There is no striping. The only thing RAID like about Unraid is the fact that it has parity. But even the parity disk is independent.

 

The fact that each disk is independent is what makes it possible to mix different sized disks in the array. It also means that even if you lose more drives than parity can recover, you only lose those drives and the data on all other disks can still be read. In fact, any single Unraid data disk can be read by itself on any Linux.

 

But since it doesn't stripe, there is no speed gain from having multiple disks.

Link to comment

@trurl Ahh that makes more sense. I was trying to figure out how it could do RAID across different sized disks.

 

So now is there a way to decrease the loading times of the array (making it faster), outside of having everything on the cache disk? Or is that just what it is?

 

And in your opinion, what would you say Unraid really excels at? What is the use case where someone would choose Unraid over something else? Say, a Windows 10 computer with a bunch of drives in RAID 5/10?

Edited by Arcaeus
Link to comment

Unraid is easier to expand than traditional RAID. Not only can you mix drive sizes, you can add a disk without rebuilding the array. And as already mentioned, you can't lose everything unless all disks fail at the same time. With RAID, if you lose too many disks you lose everything.

 

For a media server, where most of the files are written once and read many times, write speed isn't that important, and read speed of a single disk is often sufficient for serving media.

 

You can get Unraid to write the array faster at the expense of making it spin all disks. See this explanation of the 2 methods of writing parity:

 

 

Link to comment
3 hours ago, Arcaeus said:

@trurl Ok cool, good to know.

 

So I'm still having issues with my cache disk filling up. I have cache = no, have stopperd and restarted the docker multiple times, but it keeps writing to that disk.

 

Do I need to delete all of the undownloaded torrent files out of Deluge and re-add them? or what?

Do you have a docker mapping to /mnt/cache/Download or something like that?

Link to comment
11 minutes ago, Arcaeus said:

Yes, I have SABnzbd's /data path mapped there. I also have been transferring files back and forth from my server to my main rig in order to rename the files properly. I think that may be adding to it too.

Then you have told SAB to write to the cache disk, not to a user share.

 

The user shares are simply the top level folders on cache or array with the same name. So, /mnt/cache/Downloads appears in the Downloads user share, and if you read the Downloads user share, those cache files are included in the read.

 

If you write to the Downloads user share, and it is set to cache-no, then those writes will not go to cache.

 

But, if you write to /mnt/cache/Downloads, you are specifically talking about the top level folder named Downloads on cache, so those writes will go to cache. And if the Downloads user share is cache-no, that is where they will stay.

 

Cache and array top level folders are always included when reading a user share. The user share settings only determine what disk is selected when writing a new file, and whether and how files get moved by mover.

Link to comment
33 minutes ago, Arcaeus said:

So it sounds like it should go to /mnt/user/Downloads then? And based on how small my cache drive is, I'm just hinking of setting it to cache-no so I don't have to keep checking on it and/or it locking up my dockers.

If you set it to cache-no then any files already on cache will stay there as already mentioned.

 

Stop that application, set the user share to cache-yes, run mover to get them moved from cache, then you can set it to cache-no.

 

Then before you restart the application, edit it so it writes to the user share instead of to cache.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.