Jump to content
Arcaeus

How does Unraid spread files across the disks?

52 posts in this topic Last Reply

Recommended Posts

2 hours ago, Arcaeus said:

Or is it slower due to a parity check

Do you mean you are doing a parity check at the same time? Parity check will slow mover down, and mover will slow parity check down. How could it be otherwise since they are both competing for use of the same disks. Parity check uses all disks.

2 hours ago, Arcaeus said:

And how often do you have your parity check set to?

Most people do a monthly parity check. Unraid parity is maintained realtime. Parity check isn't necessary to maintain parity. Any write operation (which includes what mover is doing) in the array updates parity at the same time. The check is just a check.

 

Share this post


Link to post

Ok, Mover completed. However, When I click on the Docker tab, it still says "Docker service failed to start". is there a terminal command to start that, or do I have to reboot the whole server?

Share this post


Link to post

@johnnie.black Thank you for sending that little walkthrough over. Super helpful.

 

Ok so I got the docker image back and things seem like they're pretty much up and running. However, my cache drive is still not emptying out regularly and I don't want the dockers to lock up again due to it being full. I've manually ran the Mover, but it seems to complete without really moving anything. 

 

Why would this be? Do I need to pause my downloads in Deluge so the files aren't being used and it can move them?

Share this post


Link to post

Turn every share except system, appdata and domains to YES. Make sure appdata, domains and system are set to PREFER. Run mover.

Share this post


Link to post

Ok, confirmed that all of those are set as instructed, with new diags on the way.

Share this post


Link to post
1 hour ago, Arcaeus said:

Do I need to pause my downloads in Deluge so the files aren't being used and it can move them?

Probably simpler to just:

  1. shut down Deluge
  2. run mover
  3. Compute... Downloads on the Shares page to make sure it doesn't still have any files on cache
  4. set Downloads to cache-no so no more Downloads will be written to cache
  5. restart Deluge

Share this post


Link to post

Alright, I'll work through this.

 

Isn't that what the cache drive is for? To write everything to before moving it out to the array? If I turn that off, doesn't that defeat the whole purpose of having the cache drive there?

Share this post


Link to post

It depends on how much you write between scheduled moves, and the capacity of cache. Caching writes is only one purpose of cache. It is also used for your dockers and VMs so they won't take a performance hit from parity updates, and so your dockers and VMs won't keep parity spunup.

 

And scheduling mover to run more often may not solve the problem. Mover works best if it is running during periods of inactivity. If you are writing a lot and trying to move at the same time you can expect things to slow down as different processes try to use the same disks.

 

You will have to work out for yourself which user shares, if any, you need cached. I don't cache my downloads. But if your downloads require some post-processing, then doing all that on the parity array will be slower. Everything is tradeoffs.

Share this post


Link to post

And mover probably will not be able to move any seeding torrents.

Share this post


Link to post

@trurl I understand, thank you for writing all of that out.

 

Now the more disks you have in the system, the faster your overall transfer/load speed should be, correct? Doesn't Unraid use a Raid 5 kind of system to spread the data across the array?

Share this post


Link to post
11 minutes ago, Arcaeus said:

Now the more disks you have in the system, the faster your overall transfer/load speed should be, correct? Doesn't Unraid use a Raid 5 kind of system to spread the data across the array?

No Unraid doesn't work like that at all. Each disk is an independent file system, and each file exists only on one disk. There is no striping. The only thing RAID like about Unraid is the fact that it has parity. But even the parity disk is independent.

 

The fact that each disk is independent is what makes it possible to mix different sized disks in the array. It also means that even if you lose more drives than parity can recover, you only lose those drives and the data on all other disks can still be read. In fact, any single Unraid data disk can be read by itself on any Linux.

 

But since it doesn't stripe, there is no speed gain from having multiple disks.

Share this post


Link to post

@trurl Ahh that makes more sense. I was trying to figure out how it could do RAID across different sized disks.

 

So now is there a way to decrease the loading times of the array (making it faster), outside of having everything on the cache disk? Or is that just what it is?

 

And in your opinion, what would you say Unraid really excels at? What is the use case where someone would choose Unraid over something else? Say, a Windows 10 computer with a bunch of drives in RAID 5/10?

Edited by Arcaeus

Share this post


Link to post

Unraid is easier to expand than traditional RAID. Not only can you mix drive sizes, you can add a disk without rebuilding the array. And as already mentioned, you can't lose everything unless all disks fail at the same time. With RAID, if you lose too many disks you lose everything.

 

For a media server, where most of the files are written once and read many times, write speed isn't that important, and read speed of a single disk is often sufficient for serving media.

 

You can get Unraid to write the array faster at the expense of making it spin all disks. See this explanation of the 2 methods of writing parity:

 

 

Share this post


Link to post

@trurl Ok cool, good to know.

 

So I'm still having issues with my cache disk filling up. I have cache = no, have stopperd and restarted the docker multiple times, but it keeps writing to that disk.

 

Do I need to delete all of the undownloaded torrent files out of Deluge and re-add them? or what?

Share this post


Link to post
3 hours ago, Arcaeus said:

@trurl Ok cool, good to know.

 

So I'm still having issues with my cache disk filling up. I have cache = no, have stopperd and restarted the docker multiple times, but it keeps writing to that disk.

 

Do I need to delete all of the undownloaded torrent files out of Deluge and re-add them? or what?

Do you have a docker mapping to /mnt/cache/Download or something like that?

Share this post


Link to post

Yes, I have SABnzbd's /data path mapped there. I also have been transferring files back and forth from my server to my main rig in order to rename the files properly. I think that may be adding to it too.

Share this post


Link to post
11 minutes ago, Arcaeus said:

Yes, I have SABnzbd's /data path mapped there. I also have been transferring files back and forth from my server to my main rig in order to rename the files properly. I think that may be adding to it too.

Then you have told SAB to write to the cache disk, not to a user share.

 

The user shares are simply the top level folders on cache or array with the same name. So, /mnt/cache/Downloads appears in the Downloads user share, and if you read the Downloads user share, those cache files are included in the read.

 

If you write to the Downloads user share, and it is set to cache-no, then those writes will not go to cache.

 

But, if you write to /mnt/cache/Downloads, you are specifically talking about the top level folder named Downloads on cache, so those writes will go to cache. And if the Downloads user share is cache-no, that is where they will stay.

 

Cache and array top level folders are always included when reading a user share. The user share settings only determine what disk is selected when writing a new file, and whether and how files get moved by mover.

Share this post


Link to post

So it sounds like it should go to /mnt/user/Downloads then? And based on how small my cache drive is, I'm just hinking of setting it to cache-no so I don't have to keep checking on it and/or it locking up my dockers.

Share this post


Link to post
33 minutes ago, Arcaeus said:

So it sounds like it should go to /mnt/user/Downloads then? And based on how small my cache drive is, I'm just hinking of setting it to cache-no so I don't have to keep checking on it and/or it locking up my dockers.

If you set it to cache-no then any files already on cache will stay there as already mentioned.

 

Stop that application, set the user share to cache-yes, run mover to get them moved from cache, then you can set it to cache-no.

 

Then before you restart the application, edit it so it writes to the user share instead of to cache.

Share this post


Link to post

Ok so i did that, but it looks like there is still a file on the cache disk in the downloads share. I tried running mover again, but it doesn't run.

Share this post


Link to post

Is your downloader (torrent) still working on a file? Wont move until its done.

Share this post


Link to post
5 minutes ago, Arcaeus said:

Ok so i did that, but it looks like there is still a file on the cache disk in the downloads share. I tried running mover again, but it doesn't run.

If mover logging is still on post diagnostics.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now