Shares, discs and just general confusion!

[superloopy1]

Hi ... is there anything out there to explain disk or share mappings to me? I'm trying to set up a general htpc type setup including sab, nzb, deluge, openvpn etc that kind of thing but am having lots of trouble installing dockers to their right folders. There seems to be little consistency or explanation. A major problem i'm having is which order to onstall the dockers in as they appear to have dependencies on each other existing. Anyone able to help before i go nuts!!

 

Sent from my LG-D855 using Tapatalk

 

 

[BillClinton]

Have had a look at the great YouTube channel from [mention=67288]gridrunner[/mention].

 

https://m.youtube.com/channel/UCZDfnUn74N0WeAPvMqTOrtA

 

It’s a good place to start. He has videos on first setup and also on disk and share setup.

 

[Frank1940]

Did you look at the 'Stickies' here?   

 

       https://lime-technology.com/forums/forum/58-docker-engine/

 

[superloopy1]

Did you look at the 'Stickies' here?   
 
       https://lime-technology.com/forums/forum/58-docker-engine/
 

Thanks all, i'll follow them up. Any advice on the order in which its best to install as they all seem to be dependent on each other existing ahead of one another if that makes sense.

Sent from my LG-D855 using Tapatalk

[superloopy1]

Thanks all, i'll follow them up. Any advice on the order in which its best to install as they all seem to be dependent on each other existing ahead of one another if that makes sense.

 

Sent from my LG-D855 using Tapatalk

 

Right .. have had a quick look and some if thise videos look promising but its the mount points that i'm not clear on. Think i said disk and user shares earlier, i meant mount points now that i've checked out a couple of vids ...

 

Sent from my LG-D855 using Tapatalk

 

 

 

[trurl]

15 minutes ago, superloopy1 said:

Right .. have had a quick look and some if thise videos look promising but its the mount points that i'm not clear on. Think i said dusk and user shares earlier, i meant mount points now that i've checjed out a couple of vids ...

Sent from my LG-D855 using Tapatalk
 

The user shares are the folders in /mnt/user

[superloopy1]

The user shares are the folders in /mnt/user

Anything anywhere to explain what a 'user share' actually is? Are all docker mount points aimed at user shares? U'm confused over where 'config" and 'data' live.

Sent from my LG-D855 using Tapatalk

[trurl]

A user share is simply the aggregate of top level folders on cache and array with the same name. This is how unRAID is able to span disks.

 

Take a look at some of the links you have already been given. You may need to take some time understanding the basic NAS features of unRAID before some of the other things make sense. Go to the top left of this page and click on the Limetech logo. There is a lot of good introductory material on the website. There is also a wiki.

 

You may need to take things a little at a time. Try to understand and get one thing working before trying another one.

 

[rorton]

not sure how far you have got, and im no expert, but have set mine up so will try and throw some ideas about....

 

lets concentrate on your download apps, 

 

SabNZB or NZB Get connect to your usenet provider, and act as the 'client' app to fetch content

Sonar, Radar, Couch, Sick etc are all apps that let you manage your content, auto download, and they send their requests to SAB or NZG Get

 

you need to choose your download client like sab/nzb get first, install that, and get that running. 

 

From a disk perspective, think of it like this, 

 

you have 1 or 2 or 3+ physical disks in the server, and each of these added together in Unraid forms a 'logical' larger disk, so, 3 x 2tb disks can be seen as a single 4tb disk, (with 2tb used for parity) 

 

Disk shares are exactly what they say they are - like on a windows PC when you see your c: or d: drive, its the physical disk you are accessing and sharing on the network. 

User shares are a step further - user shares see the logical disk you have, so your 4tb logical disk, which is made of 2 x 2tb disks in this example - and you can make as many user shares as you want with diff names (media, tv, movies, backups, etc etc) and each user share will live on one or more physical disks. as you add more disks, the shares take this into account, and your storage 'pool' grows without you having to to much. 

 

Up to you how you want it setup, most do a 'set and forget' setup user shares, and don't really mind which disk the data sits on - others like to keep certain data on certain physical disks. 

 

If you have a cache drive its better in this example, as you can put your docker image, and directories for your docker apps on here, and it won't keep the array disks spun up all the time, not a dependency, but many have theirs setup like this. 

 

In my system, i have mine setup like this:

 

1 x 250gb SSD Cache Drive - i don't use it as a write cache, but i do have my docker container on this disk, and the 'appdata' directories for any dockers i have running. I also have a folder called 'downloads' on the cache disk, where sab downloads the raw files and extracts them, before moving them off the cache and onto the protected array - this is my 'config' if you like

 

In the array, i have

 

3tb Parity

3tb, and 2 x 1tb data disk, this gives me logically 5tb data

 

Then i have a number of user shares setup - 

 

Media (in here is movies and tv)

Backups

Audio (sonos connect here for mp3's etc)

Data (bit of a dumping ground of data, files and crap)

 

this bit is my 'data' bit

 

How far have you got with your setup - does any of the waffle make sense so far?

 

[Frank1940]

Quote

Disk shares are exactly what they say they are - like on a windows PC when you see your c: or d: drive, its the physical disk you are accessing and sharing on the network. 

 

Note to OP:  Disk Shares (they are designated as disk1 , disk2, etc in unRAID)  are normally not shared and are thus hidden from the casual user.  This is done on purpose for a VERY good reason.  If one copies from a Disk Share to a User Share (or the other way around), data  will soon lost!!!   Hence, do not make a Disk Share visible (By exporting the Disk as a Share) unless you really understand how to deal with them and the ramifications of doing so.  In fact, I can't think of a good reason to export any Disk Share except for the Flash Drive and the cache drive.

[rorton]

I will hold my hand up to sharing a disk. 

 

Ive set the global shares to exclude disk 3 in my array, the I have used disk 3 as an exported afp share for time machine to backup my mac

 

i read about issues using a user share with time machine, so when I reconfigured the system at the weekend with new disks, I made the time machine share disk only. 

[trurl]

22 minutes ago, Frank1940 said:

  In fact, I can't think of a good reason to export any Disk Share except for the Flash Drive and the cache drive.

I would say don't even share the cache drive, since it is a disk, and it is part of user shares, and thus subject to the User Share Copy Bug.

 

But I think this whole discussion of disk shares is just adding to the confusion.

 

OP, you don't need to and shouldn't share any disks on the network. User shares will let you access your files on the network, and also let you configure your dockers.

 

Forget we even mentioned disk shares.

[Frank1940]

14 hours ago, superloopy1 said:

Hi ... is there anything out there to explain disk or share mappings to me?

 

15 minutes ago, trurl said:

But I think this whole discussion of disk shares is just adding to the confusion.

 

As you can see, the OP brought up the subject in first post.   At that point (thought I), the cat is kinda of the bag, indicating that he was aware that they could be shared,  and I thought a word of caution was worth while.

[SSD]

I find it is best to think of user shares as the aggregate of top level directories on each disk. So if you have a folder on disk1 called Movies that contains 5 movies, and a folder of the same name on disk2 with 6 Movies, and a folder on disk3 also with the same name with 8 Movies, and then look at the Movies user share, you will see all 19 Movies in a single folder. UnRaid decides which physical disk to write new files to on the user share with some user share configuration settings.

 

User shares are the heart of unRaid and well worth understanding inside and out.

[superloopy1]

Ok guys, i'm getting there. I've unshared my disk shares which left me with just user shares visible. I still have to get my head around the concept of mount points. So, when a docker quotes 'config' its usually pointing to mnt/user/appdata? When it quotes anything else its usually mnt/user/xxxxx? Problem i have are those dockets which dont include the /user elent and just quote /mnt/data or /mnt/download. Are these an older version of the docker where we (unraid) now use /user to indicate where is best practice to place our containers?

I know, i'll go away soon and just have a play around but its better i understand it first or i'll just let it go.

Thanks for all the input so far.

Rorton... nzb or sab is first to install, then whats best order to avoid having to undo/redo configs?

Sent from my LG-D855 using Tapatalk

[rorton]

29 minutes ago, superloopy1 said:

Ok guys, i'm getting there. I've unshared my disk shares which left me with just user shares visible. I still have to get my head around the concept of mount points. So, when a docker quotes 'config' its usually pointing to mnt/user/appdata? When it quotes anything else its usually mnt/user/xxxxx? Problem i have are those dockets which dont include the /user elent and just quote /mnt/data or /mnt/download. Are these an older version of the docker where we (unraid) now use /user to indicate where is best practice to place our containers?

I know, i'll go away soon and just have a play around but its better i understand it first or i'll just let it go.

Thanks for all the input so far.

Rorton... nzb or sab is first to install, then whats best order to avoid having to undo/redo configs?

Sent from my LG-D855 using Tapatalk
 

Im still getting my head around the mount points, i'll bow down to someone who can explain that bit better than me...

 

With regards your apps - which do you want to include

 

Sab/NZBGet are the same thing really, so you want one of those first, and test it, make sure you can manually put in an nzb, it downloads to where it should do etc etc. 

Once happy with that, then go for something like sonaar/radaar (depends on what you are wanting to achieve) 

 

Sonaar/Radaar will need to point to Sab with an API link, so you setup Sonaar to look for a TV show for example, it finds the NZB, sends that to SAB, Sab d/loads it, tells Sonaar, it then moves it to your tv share. 

 

Ive been happy using all the dockers from Linuxserver.io, 

 

Here is how my mounts are set, reading above, i shouldn't have /cache set as my appdata directory, i read you should elsewhere

 

On my array, i have a user share called 'downloads'

then for the sab docker app, i have mounted what the app is asking for (downloads folder) and told it my download folder is at /mnt/user/downloads

then, when in the app, when the app references 'downloads' its actually looking at /mnt/user/downloads

 

 

My Sonaar app is similar

 

 

again, i have sonaar asking where my downloads directory is, so i have told it its on /mnt/user/downloads

i then have a user share on the array called tv and sonar wants to know where my tv content is, so i tell sonaar that tv is in /mnt/user/media/HD TV (your share name may be different, some ppl have a separate user share for movies, and a separate one for tv)

 

Key with the mount points is, the app is asking where your data is stored, or to be stored - and you are simply telling it where that share is, which is a sub directory of /mnt/user

 

 

 

[SSD]

Ok, what is a mount point.

 

Let's start with drive letters. When you add a drive to Windows, it gets a drive letter. The first disk is c:, then d:, etc.

 

Linux doesn't work like that. Each new drive will get a device name, like /dev/sde, and that can be used to partition and format the disk. But in order to use the drive to add new files or access existing, it has to be mounted. 

 

Since Linux has no drive letters, instead a disk needs to be mounted before you can access its files. You might think since the disk is /dev/sde, you could create a file called /dev/sde/my file.txt. No, doesn't work like that. The disk is a closed book until it is mounted.

 

To mount a disk, you create a folder, and then mount the disk there. So, for example, you could create a folder called /edrive. And mount /dev/sde there. Now by going to /edrive you can add new files to the disk and access existing files. The folder /edrive is its mount point. You could then unmount it, which closes the book so to speak, and then mount it some other place. Like /mnt/mydisk. 

 

The /mnt folder is by convention the location most disks get mounted. UnRaid mounts all your data disks there. E.g., /mnt/disk1.

 

Now disks are not the only things that can be mounted to a mount point. Suppose you are accessing a remote computer, and want to access its files. In Windows you might refer to a remote share as //mycomputer/sharename. But in Linux you would need to create a mount point and mount that remote share there. You might call it /mnt/sharename. Once mounted, you can access its files.

 

So what else can be mounted? UnRaid has this concept of user shares. They are not real disks, but they sort of look and act like them. So unRaid gives them a mount point and you access them by that mount point. E.g. /mnt/user/Movie. All user shares are in the /mnt/user folder.

 

So let's talk about Dockers for a second. Dockers are interesting little prepackaged units that contain Linux applications. A docker cannot see the disks you have mounted. In fact, a docker has a completely different "root file system" and cannot see anything on your unRaid server. So it can't be mischievous and access files it is not supposed to. But Dockers are built to expect the user will "map" folders from the server file system into the Docker's file system. So the docker can be built to expect its configuration files are in a folder called /config. And you can map your real folder called /user/appdata/appname to the Docker's /config folder. So whenever the docker accesses /config, it is really accessing your appdata user share for the docker. These mappings are called volume mappings. Don't confuse them with mount points although some of the concepts are similar.

 

So what if you didn't, and just left the config folder unmapped. What would happen? Well it would appear to work fine. All of the config data would be stored inside the docker container. But when the docker gets updated in a way that resets the container, all that configuration gets lost and you are back to ground zero. So anything you expect to persist needs to be mapped from locations on your server. The volume mappings vary from docker to docker, but most all have a /config mapping. Getting the mappings right is key to getting Dockers working correctly.

 

Hope that helps.

 

(#ssdindex - mount point basics)

[trurl]

3 hours ago, superloopy1 said:

Problem i have are those dockets which dont include the /user elent and just quote /mnt/data or /mnt/download.

Do you have a specific example of this that you can point to? Those mappings are guaranteed to produce a broken system.

[Frank1940]

About forty years ago, I had an introductory course to UNIX by a instructor who had received his training from Bell Labs.  One of the first things he told us was that there was no hardware in a UNIX operating system.  Everything was mounted as a file.  If a physical device was attached, it had a device driver that made it look like a directory which is simply a special type of file  in UNIX/Linux.  While that concept is difficult to get one's head around, it makes a lot of sense with unRAID.  User Shares are much easier to understand if you think of it being a collection of files inside of a directory/folder.  Again, there is simply a device driver that is behind the scenes to make a certain collection of files look like a directory.   And you mount (or attach) this device driver to to the larger file system at the appropriate point that you want it.  For an example of this, run the following two commands from the terminal window.

cd /mnt

ls -al

Look at what is mounted in this point in the file system.  We can see all of our data storing disks...

 

Now, run these two commands:

cd user0

ls -al

Wow, there are our User Shares.  All conveniently in a common file structure. So we have already seen a mixture of files and collection of files at one point.  If you want further proof,  run these commands:

cd ../

touch text.txt

ls -al

Looking at the attributes of this zero bite file, you can see it is an ordinary text file.  Think about what you have just done and try to explain it in any other manner that Linux is just a collection of files.

 

Let's get rid of that file before we finish.

 

rm text.txt

Quick description of the Linux commands used here:

 

cd              change directory

ls                list file names

touch         change file time stamps (one of its side effects is that it will create the file if it does not exist.)

rm              remove file of directory

 

 

[trurl]

8 minutes ago, Frank1940 said:

Now, run these two commands:

cd user0

ls -al

Wow, there are our User Shares.  All conveniently in a common file structure. So we have already seen a mixture of files and collection of files at one point.  If you want further proof,  run these commands:

 

Actually, our user shares are at /mnt/user. /mnt/user0 is the user shares excluding any files on cache. So, it's likely that any cache-only or cache-prefer user shares would not appear in that listing.

 

41 minutes ago, trurl said:

Do you have a specific example of this that you can point to? Those mappings are guaranteed to produce a broken system.

 

The reason I say those mappings produce a broken system is because there is nothing actually mounted there, so anything written to them would just fill up RAM.

[superloopy1]

Just what i needed, thanks! I've now installed sabnzbdvpn and delugevpn but my downloads are still going to the /config folders in /mnt/user/appdata and not the /data folders i'd expect /mnt/user/downloads even though i 'think' i've followed the video to the letter ... I watched and used SIO's youtube videos which alongwith the info here helped me a lot! I've checked the docker configs and they look as if they're set up ok. Where else can i look, somethings obviously not quite right?

 

One more thing ... i'm using PIA as vpn provider and seem unable to run both instances of delugevpn and sabnzbdvpn together? One, on startup, pulls down the other it seems. I needed to change port number 8118 to 8121 on sab as it conflicted with deluge just to get sab to run but now as i say both instances dont appear able to coexist ... have i done this in the right way or should i be looking elsewhere?

 

Sent from my LG-D855 using Tapatalk

 

 

 

 

 

 

[trurl]

3 hours ago, superloopy1 said:

I've checked the docker configs and they look as if they're set up ok. Where else can i look, somethings obviously not quite right?

If it's not the volume mappings then it's likely the way the application is setup to use the mapped folders.

[superloopy1]

Trurl ... you were right!

 

I'm slowly getting my head around these dockers. Anyone any idea why i cant run two vpn based instances together? And, was i right to change port numbers on sab or not? I'm no expert in any of this but it seems that unless my port 8118 is up and running via deluge then i cant use my pia vpn standalone with sab using the alternative port 8119. So, if both are running, no probs on either. Shutdown delugevpn and it takes sab (now on port 8119) with it. Looks like my vpn instance is being terminated by delugevpn, presumably because port 8118 is no longer available. Make sense? Not to me unfortunately. Maybe i should close this question down and move it to binhex for support??

 

Sent from my LG-D855 using Tapatalk

 

 

 

 

 

 

[trurl]

It is often necessary to change the host port since different dockers must use different host ports. You shouldn't need to change the container port since that is the port the application is designed to use.

[superloopy1]

It is often necessary to change the host port since different dockers must use different host ports. You shouldn't need to change the container port since that is the port the application is designed to use.

Both instances use port 8118 as vpn port though. That appears to be the problem. Closing the first app, in this case delugevpn terminates the vpn it seems and my sabnzbdvpn connection goes down?

Sent from my LG-D855 using Tapatalk