I recently added 4 additional drives to my array and realize I have to re-org my media stuff.

I have been noticing that my drives are not spinning down as much as they should be.

After checking a bit more thoroughly...it looks like i'm one of the dummies that created a single "Media" folder with everything inside and did not setup the folder structure so that the split Levels would work correctly.

So pretty much everything is spread out over the existing 4x drives. So when accessing 1 movie it will bring up 1-4 drives depending on where my stupid files are.

 

So right now this is my structure:

media > movies > movie folder > movie files

media > kidsmovies > movie folder > movie files

media > tvshows > season folder > season files

media > anime > anime folder

 

So I will be creating 3 new user shares.  Setting each of the below shares with High Water.

tvshows

movies

anime

 

So once thats done I want to move the files to the new shares and get them redistributed correctly.

What would be the best recommendation?

Also what are the thoughts about assigning stuff to each disk? for example: Disk1/2 for anime only, Disk3/4 for movies only etc....

 

As for structure and split level...does the below sound correct for each share?

tvshows > Archer > Season 1 > episodes files

Split level 3

example:

/mnt/disk2/tvshows/archer/season 1/all files

/mnt/disk3/tvshows/archer/season 2/all files

 

movies > kidsmovie > Lego Batman > movie files

movies > HDMovies > John Wick > movie files

movies > SDMovies > The Last Dragon > movie files

Split level 2

Example:

/mnt/disk1/movies/kidsmovies/Lego Batman/all files

/mnt/disk2/movies/hdmovies/John Wick/all files

 

anime > Initial D > episode/movie files

Split level 2

example:

/mnt/disk1/anime/Initial D/all files

 

What i don't want is stuff like this (which is what they are right now):

/mnt/disk1/media/tvshows/archer/season 1/archer.s01e01.mkv

/mnt/disk2/media/tvshows/archer/season 1/archer.s01e02.mkv

/mnt/disk3/media/tvshows/archer/season 1/fanart/poster.jpg

 

Thanks 

Urlryn

Edited December 16, 20178 yr by Urlryn

3 hours ago, Urlryn said:

So once thats done I want to move the files to the new shares and get them redistributed correctly.

What would be the best recommendation?

Easiest way is using the consld8 script, it can do this automatically using this type of command:

 

https://forums.lime-technology.com/topic/34547-diskmv-a-set-of-utilities-to-move-files-between-disks/?do=findComment&comment=391785

 

Great!  I'll read up on that!  Should take care of the file transfers at least!

Thanks Johnnie!

 

I second Johnnie's recommendation!

18 hours ago, Urlryn said:

Also what are the thoughts about assigning stuff to each disk? for example: Disk1/2 for anime only, Disk3/4 for movies only etc....

 

I believe limiting shares to specific disks is a more organized way to manage user shares. Others like the idea of viewing unRAID as "one big disk", in which case you can have all of your drives containing all of your shares which also works, but I believe has some disadvantages. The major one being, in the event of a catastrophe in which multiple drives are lost, having specific disks contain specific shares will very frequently result in the surviving data being complete and useful. Otherwise, you could have only partial data from several shares, which my be useless in its partial state. And you might not even know if you have it all or not (which can be especially frustrating).

 

Depending on the type of files, this might pertain more to the data within a share being on a single disk, then the entire share being on a single disk. For example, you might rather have all of the episodes of show A survive, and lose the episodes from show B, than have some episodes from show A and some episodes from show B survive, but neither being complete. 

 

Unless all of your data in a share is in a very similar structure, and you have split level set up correctly, consider the allocation method ("fill up", "most fee", high water") carefully.  "Fill up" or "high water" will work better to keep related files (which are typically created at or near the same time) together on the same physical disk. Avoid "most free" in most cases, as if you have a bunch of disks in a share that have the same amount of free space, "most free" will tend to fan out larger files (higher risk of separating related files onto different disks), similar to dealing from a deck of cards. "High water" will deal half the deck to each disk, then go back to the first disk and deal half of the rest (1/4 of the deck), and go around all the disks again, then 1/8, etc., etc. So as the disks gets fuller and fuller, the risk of files that you'd prefer to stay on the same physical getting fanned out gets higher. I actually prefer "fill up". The disk will fill until it passes the min free boundary, at which time unRAID moves on to the next disk. So it's like dealing 99.9% of the deck and keeping just a few cards in reserve, and then moving on. Generally fill up does a good job of keeping files together regardless of the split level, although setting split level is important from something like TV shows which can come in incrementally over a long period of time.

 

I have never seen the fascination of filling the disks in a uniform way. I prefer to have each new disk fill in turn and do my best to keep related files together.

 

(#ssdindex)

Thanks SSD!

 

I was pretty much one of the "one big disk" state of mind when I originally set my server up.  Setup high water and let it go.

Kicking myself for not cleaning this up sooner! 

 

I do appreciate the information!  It makes a lot of sense and goes in line with what I was thinking when I asked the question.

 

Urlryn

I like to group data so quite similar to SSD:s ideas.

 

On one hand, I don't try to keep an even fill level on the drives. On the other hand, I like to have all photos on one drive, while specific drives may contain TV shows.

 

The above makes it much easier to consider backup recovery - with 100% of the photos lost from a storage disk, it's natural to restore a full backup. If the photos are splitted over 5 disks, then I need to spend time restoring just 20% - or have the restore process also restore the 80% photos that was still available on other drives.

 

Grouping the data also means that browsing the data can be done without spinning up every drive.

Thanks for the information and suggestions!  Really helps!

 

Now for the fun stuff...getting this all sorted and moved!  

 

Urlryn

I replaced my cache drive (larger one).  So following the directions...i changed the appdata share to cache:Yes

Once I finished adding the new drive in and updated the appdata share back to cache:prefer...then to cache:only.

 

I noticed (and this maybe normal but wanted to be sure)

 

The docker.img file.  It shows up in a couple locations.

 

/mnt/user/docker.img

/mnt/cache/docker.img

 

Is it supposed to still show in /mnt/user? if not...it can be deleted without affecting the one in cache?

Wasn't sure if its was the "same" file or if it was a copy and wasting additional space.

 

Thanks

Urlryn

4 minutes ago, Urlryn said:

Is it supposed to still show in /mnt/user?

It's the same file, /mnt/user includes /mnt/cache

Cool!....Thanks johnnie!

 

On 12/16/2017 at 4:42 PM, SSD said:

I believe limiting shares to specific disks is a more organized way to manage user shares.

Some of my data has gotten rather sloppy lately, I may give this a try.  I was going to run a tree every time I do a backup to figure out which files live on which disk in case of a catastrophic failure, but it really would be easier to just know what is on each disk.  I manage things actively enough that I don't need a "set it and forget it" high-watermark approach.  What combination of includes/excludes do you use?

@tdallen

 

I suggest includes. For smaller shares, limit to a single disk. For larger shares, list each included disk.

 

As the array grows, you may feel tempted to allow multiple shares to participate on the same new disk (otherwise you could have several near empty new disks, each slowing growing in its type of content). Although not optimal, I think this is ok (at least for a while) - but when you add another new disk, separate the content onto different disks and change the include settings. So when a disk is full, it contains only one type of content.

 

 

1 hour ago, SSD said:

@tdallen

 

I suggest includes. For smaller shares, limit to a single disk. For larger shares, list each included disk.

 

As the array grows, you may feel tempted to allow multiple shares to participate on the same new disk (otherwise you could have several near empty new disks, each slowing growing in its type of content). Although not optimal, I think this is ok (at least for a while) - but when you add another new disk, separate the content onto different disks and change the include settings. So when a disk is full, it contains only one type of content.

This is how I store my files on file server, on backup servers, on USB backup disks and potentially on backup CD/DVD/BD disks.

 

I have been doing it this way for over 20 years and for me it works very well.