Jump to content

Best way to setup shares on unRaid to reduce Spin-up time?


Recommended Posts

Hello,

unRaid Noob here, need an opinion from someone with experience.

Say you have 10 HDD array, and you have shares created with default settings, meaning in the bottom options to Included Disk set to all, excluded disk set to none.

You also have HDD Spin-Down mode enabled for 1h.

You have Movies, TV Series, Game shares and they all mapped as network drives on windows PC.

Whats going to happen if say you decide to open the Movies share?

Will it spin-up ALL the hard drives in your array while you browse the share?

 

 

Because I assumed its going to spin up every HDD, I decided to manually configure shares and drives, like i give Movies share just this HDD and the TV series just 2 otehr HDD and the games just that HDD and the HDDs that i dont need right now permanently in spin down mode and nothing wakes them up

But i assume its not the correct way to manage unraid?

 

Whats the proper way that can both have unsused HDDs asleep and i wont have to manage space on hard drives?

 

Thanks

Edited by Hexenhammer
Link to comment
  • Hexenhammer changed the title to Best way to setup shares on unRaid to reduce Spin-up time?

What is your objective in this exercise?

 

I did a bit of playing with my servers. Let begin with some observations that I made.

  1. Unraid only spins up a disk when it needs to read information from the disk.  It does (appear) to use a caching system to store information from previous disk reads-- particularly directory information.  So leave your server on 24-7.
     
  2. You can actually see where which disks have information stored on your server for each share.  You can get to it on the Shares tab by clicking where shown below:
    image.thumb.png.a3320a573a164e7de70530854e3f97e6.png
    That will give you this:
    image.thumb.png.f95bdf88ec3988249c0dac2282e01652.png
    (I have the 'Dynamix File Manager' installed and this may well be a functionality provided by that plugin.)  You can see where  each directory has data stored and which disk each file is on.
     
  3. In     Settings    >>>>     Disk Settings   you will find the default spin down time for each disk.  
    image.png.d470732a2720f27a510c6e0f9500e247.png
    Set it for the value that you want. Be a bit careful here.  If you pause a movie and leave it paused longer than the spindown setting, the movie will usually restart replaying using the read-ahead RAM cache.  This can/usually results in second pause while the disk spins up when the cache empties.  Depending on the media player, this can be a problem.  (I have been bit by this one when I 'previewed' a movie on my PC and, much later, was watching it in my Home Theater!)
     
  4. Yes, you can restrict User Shares to certain disks.  But beware of two issues.  First, if doing this after the fact, a file in a share is not on one of the disks that you allocated, it will not be moved! (It will still be accessible.)  Second, you must track how full each disk is that you allocated to that share.  If all the disks become full, writing a new file to the share will fail!
     
  5. My experience is that disks will be spun up only if information on that disk has to be accessed.  So it is not a matter that all the disks will be spun up but rather how you have organized the share--  directories and sub-directories vs a 'flat' file system.  I use KODI and it has a database of the movie information that I have set up on a cache drive that is on an SSD that I never allow to spindown.  (How does one spin down an SSD but there is a setting to do so...)   This means I never have to spin up a data drive until I actually have selected a movie to watch.  

 

As far as I am concerned there is no 'perfect' solution to this problem UNLESS you are willing to never spindown your disks! 

 

Link to comment
3 hours ago, Frank1940 said:

What is your objective in this exercise?

 

I did a bit of playing with my servers. Let begin with some observations that I made.

  1. Unraid only spins up a disk when it needs to read information from the disk.  It does (appear) to use a caching system to store information from previous disk reads-- particularly directory information.  So leave your server on 24-7.
     
  2. You can actually see where which disks have information stored on your server for each share.  You can get to it on the Shares tab by clicking where shown below:
    image.thumb.png.a3320a573a164e7de70530854e3f97e6.png
    That will give you this:
    image.thumb.png.f95bdf88ec3988249c0dac2282e01652.png
    (I have the 'Dynamix File Manager' installed and this may well be a functionality provided by that plugin.)  You can see where  each directory has data stored and which disk each file is on.
     
  3. In     Settings    >>>>     Disk Settings   you will find the default spin down time for each disk.  
    image.png.d470732a2720f27a510c6e0f9500e247.png
    Set it for the value that you want. Be a bit careful here.  If you pause a movie and leave it paused longer than the spindown setting, the movie will usually restart replaying using the read-ahead RAM cache.  This can/usually results in second pause while the disk spins up when the cache empties.  Depending on the media player, this can be a problem.  (I have been bit by this one when I 'previewed' a movie on my PC and, much later, was watching it in my Home Theater!)
     
  4. Yes, you can restrict User Shares to certain disks.  But beware of two issues.  First, if doing this after the fact, a file in a share is not on one of the disks that you allocated, it will not be moved! (It will still be accessible.)  Second, you must track how full each disk is that you allocated to that share.  If all the disks become full, writing a new file to the share will fail!
     
  5. My experience is that disks will be spun up only if information on that disk has to be accessed.  So it is not a matter that all the disks will be spun up but rather how you have organized the share--  directories and sub-directories vs a 'flat' file system.  I use KODI and it has a database of the movie information that I have set up on a cache drive that is on an SSD that I never allow to spindown.  (How does one spin down an SSD but there is a setting to do so...)   This means I never have to spin up a data drive until I actually have selected a movie to watch.  

 

As far as I am concerned there is no 'perfect' solution to this problem UNLESS you are willing to never spindown your disks! 

 

 

I mainly want to keep unused drives in permanent off state, i know that turning them on and off is unhealthy, in this case i rather they keep spinning.

I build this unraid some days ago, first i have more drives than I ever need in near future so I had to decide if i rather use 50/50 14 and 16tb or load all 14tb except parity and when they eventually die swap them with 16.

Then i started creating shares [with no parity in the system] and i basically went the same way i have my HDDs now [I have like a metal HDD box and each HDD has label like SciFi TV, Movies etc]

So i created a share locked to some HDD, but honestly its not comfortable

 

Do you have cache dirs plugin or whatever its called? it supposed to cache dir structure to RAM and when you browse file system you browse this cached list

until you access the file

Haven't been able to test how it works

 

Do you have your HDDs in spin-down mode? can you check what happens when you browse on windows a share that split across many Hard drives? Does it spin them up?

 

 

Link to comment
44 minutes ago, Hexenhammer said:

Do you have cache dirs plugin or whatever its called? it supposed to cache dir structure to RAM and when you browse file system you browse this cached list

until you access the file

Haven't been able to test how it works

 

I have it installed on my servers BUT I only allow to cache my media share which is about 3000-4000 items.  If I try to do more shares involving tens of thousands of shares, there is a problem.  The RAM allocated for the cache can be needed for other purposes and cached directory information will get reallocated.  The disks get spun up to refill the cache-- either when the entries are required or when the RAM becomes available again.  It works best when you only cache small number of items for this reason. 

 

You can try it to see if it helps.  If you do, only cache those shares that you frequently access.   You will have to play with the settings to get things to work smoothly with a minimum of disk spin-ups to refresh the cache.  I do believe that support for this plugin is minimal as it has been more than two years since it was last updated.

 

53 minutes ago, Hexenhammer said:

Do you have your HDDs in spin-down mode? can you check what happens when you browse on windows a share that split across many Hard drives? Does it spin them up?

Yes, and I already provided you with that information-- See Number 4 above in my last post.  Example-- my movie files are under a sub directory called Movies under the Media share and there are sub-directories for the beginning letter of the movie's name---  0-9, A, B, C, D, etc.  This helps to minimize the number of disk being spun up as you don't have spin up all the disks when getting a listings for movies that begin with (say) "A". 

 

The scheme also prevents a secondary problem.  If you have a flat file system where you have all your movies in a single share directory (In my example-- the 3000-4000 movie items), the server has to send all of that file information over the network to the PC which then has to sort the information in the format that you want to view.  This time can be substantial especially if you have to spin up a large number of disks on the server. 

 

BTW, if you are using the 'High-water' algorithm to allocate files to the disk and large capacity data disks (say, >3TB) then when you write a large number of files are a single time (Like a back of your Windows data), then everything will most likely end up on a single disk.   In fact, using the High-water' allocation scheme, you could end up writing everything to a single disk for a month or two. (Using the 'Most-free' allocation will result in the most chaotic distribution of files against all the disks!)   In fact, I had a problem getting more than two disks of a possible four spun at once while browsing a share! 

 

 

  • Upvote 1
Link to comment

I've been using Dynamix Cache Directories.  Seems to work as advertised.  My drives do not spin up doing file directory type accesses.  Only seem to spin up when actually accessing/reading a file on that one drive.

 

As far as spinning up/down drives: It takes enough time to spin up a drive that some applications (talking about you, Plex) will error when trying to read a file on a spun down disk, and the application times out.  This can be an issue for users other than myself, especially when accessing media files off of my own LAN.

 

My goal is to keep as many drives as I can spun down until needed.  But my media files can be problematic if the drive is spun down.  My solution is to exclude the Media share from several of the drives, and set those excluded drives to a 15 minute spin down.  The drives that hold my media files never spin down, so I never see the timeout errors from Plex.  Seems to be the best compromise.

  • Upvote 1
Link to comment
5 hours ago, Frank1940 said:

 

Yes, and I already provided you with that information-- See Number 4 above in my last post.  Example-- my movie files are under a sub directory called Movies under the Media share and there are sub-directories for the beginning letter of the movie's name---  0-9, A, B, C, D, etc.  This helps to minimize the number of disk being spun up as you don't have spin up all the disks when getting a listings for movies that begin with (say) "A".

 

 

 

Maybe im missing something, but even if you sorted your movies in alphabet folders, from my limited understanding, unRaid can split folders across drives, so your A folder can be on Disk 2 and Disk 5?

 

Is that the recommended way to archive movies on unRaid?

 

 

Quote

I have it installed on my servers BUT I only allow to cache my media share which is about 3000-4000 items.  If I try to do more shares involving tens of thousands of shares, there is a problem.  The RAM allocated for the cache can be needed for other purposes and cached directory information will get reallocated.  The disks get spun up to refill the cache-- either when the entries are required or when the RAM becomes available again.  It works best when you only cache small number of items for this reason. 

 

 

Whats your RAM usage? If i plan to not have any VMs, just generic NAS + media streaming without encoding and maybe DNS server I had Pie4 and adguard DNS and if its available as plugin for unrauid  I would like to have it.

Right now t he system [12700 non K] has 32Gb of RAM, would you say its enough to have cacheDirs at max setting to cache as much as posible and have enough for system usage? [At worst case i can buy 32Gb more but i rather not]

Right now I installed a extra fast cache, 280Gb Optane U.2 SSD, i plan to use the swap file plugin and created 64GB swap file and see if it helps with Cache Dir, in theory it should, at least on windows it would be used, no idea if unraid will push the cached data to swap file.

I have no use for this optane drive so now it can be good for something, and to make it the  "system drive" make all the system folders go there and the swap file.

 

 

Quote

BTW, if you are using the 'High-water' algorithm to allocate files to the disk and large capacity data disks (say, >3TB) then when you write a large number of files are a single time (Like a back of your Windows data), then everything will most likely end up on a single disk.   In fact, using the High-water' allocation scheme, you could end up writing everything to a single disk for a month or two. (Using the 'Most-free' allocation will result in the most chaotic distribution of files against all the disks!)   In fact, I had a problem getting more than two disks of a possible four spun at once while browsing a share! 

 

Which algorithm would you recommend? for 14Tb drives

 

Thanks for help!

 

 

 

 

Link to comment
2 minutes ago, Hexenhammer said:

unRaid can split folders across drives, so your A folder can be on Disk 2 and Disk 5?

This is what Split Level is all about

 

3 minutes ago, Hexenhammer said:

Which algorithm would you recommend?

High-water is the default for good reasons already mentioned. It is a good compromise, using all disks eventually, without constantly switching disks.

  • Upvote 1
Link to comment
2 minutes ago, trurl said:

This is what Split Level is all about

 

High-water is the default for good reasons already mentioned. It is a good compromise, using all disks eventually, without constantly switching disks.

 

Thanks for replay!

trurl, do you know if CacheDir can be used with swap file plugin? Will unRaid swap the cachedir database to the swap file if there is one or it will purge it when RAM is needed?

I mean, is it good idea to add 64Gb Swap on Optane ssd? [no need to buy anything, i had it already from before]

If im missing something and you know about good uses for optane with unraid, please let me know, I want it used for what will have the best benefit for the unraid.

 

Thanks

 

Link to comment
4 hours ago, ConnerVT said:

I've been using Dynamix Cache Directories.  Seems to work as advertised.  My drives do not spin up doing file directory type accesses.  Only seem to spin up when actually accessing/reading a file on that one drive.

 

As far as spinning up/down drives: It takes enough time to spin up a drive that some applications (talking about you, Plex) will error when trying to read a file on a spun down disk, and the application times out.  This can be an issue for users other than myself, especially when accessing media files off of my own LAN.

 

My goal is to keep as many drives as I can spun down until needed.  But my media files can be problematic if the drive is spun down.  My solution is to exclude the Media share from several of the drives, and set those excluded drives to a 15 minute spin down.  The drives that hold my media files never spin down, so I never see the timeout errors from Plex.  Seems to be the best compromise.

 

 

Im the only user of this build, basically its "external HDD" of sorts.

 

When you say times out, it hangs or it lets you re-run the file? Eventually i plan to have all my media served back to me with Plex but only used as DLNA, no encoding, just to get movie/tv descriptions and genres.

Not everything will be on the array,some will be as unassigned device with iscsi back to PC, i like the idea of remote storage, i mean ignoring the parity and all that, just using iSCSI for existing drives that were uglyfying and heating up my gaming build sounds insanely good, i mean how i never even considered the idea?

I was thinking maybe I should keep one 18Tb HDD on my PC for emualtors [backed up to unraid], but now thinking i can just iscsi it from unraid and run emulators on it as if it was local storage, and run some plugin that will back up the content from unassigned HDD to the array

 

 

i just got a switch on sale, two 10gb ports and five 2.5gb and  im thinking maybe better to return it and connect the unraid directly to PC 10Gb to 10Gb and each connect to router with second Ethernet port, and this way on the 10Gb cards I could use Jumbo frames [which i cant use right now, at least from what i understand Jumbo frames they like MTU and I need 1492 MTU for PPPoe Fiber connection]

 

 

Link to comment
7 hours ago, Hexenhammer said:

When you say times out, it hangs or it lets you re-run the file? Eventually i plan to have all my media served back to me with Plex but only used as DLNA, no encoding, just to get movie/tv descriptions and genres.

 

Times out = When starting to play a movie in Plex, it somes back with an error saying it could not start playing the media.  Need to go back and press Play again, which then it plays without issue.  This is because the drive was still spinning up and the file unavailable longer than Plex is willing to wait for the file.

 

Dynamix Cache Directories doesn't consume a lot of memory for me.  I run 18 Dockers and a Home Assistant IO VM, with 52K media files using 11TB.  I have 16GB of DRAM which Unraid reports I use 35-40%.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...