Need Help WIth Hardware, about to Press Buy It button

[Hexenhammer]

Guys, I just need a confirmation that NAS unRAID will work on Intel 9100/9350K + 16GB RAM without any negatives.

Like, Parity calculations do they benefit from more cores or faster single core?

 

unRAID for NAS, piHole, media casting no transcoding, one user for NAS, 2 users for media, maybe some dockers i dont know about.

But look at this project like Pre-build NAS with extra features, like QNAP has and it only runs on ARM or Celeron or cheap low powered AMD CPU

I just got TR3+ZenithII so i cant spare the money to buy expensive unRAID CPU and lots of RAM

[trurl]

25 minutes ago, Hexenhammer said:

Parity calculations do they benefit from more cores or faster single core

You can rest assured that parity calculations are only limited by disk speeds. Even dual parity unless you have a very old CPU.

[Hexenhammer]

2 hours ago, trurl said:

You can rest assured that parity calculations are only limited by disk speeds. Even dual parity unless you have a very old CPU.

 

Hi,

Is there any Benefit of using faster HDD's for Parity? My main drives will be 5400RPM WD Heliums with mix of old SMR segates until they replaced with WD, for parity I wanted to either shuck WD Black D10 12TB, which has Helium 7200RPM 250MB R/W HDD or to buy Segate Exos x16.

 

Thanks

[JonathanM]

1 hour ago, Hexenhammer said:

Is there any Benefit of using faster HDD's for Parity?

For a very limited set of conditions, yes.

 

Reading doesn't normally involve parity, so no benefit for read speeds. Writing is limited by the slower of either the target data disk or parity, whichever is slower, so no benefit for a single write. However, if you are trying to write to 2 different data disks at the same time, parity must shuffle back and forth to service both data disk writes, so in that case, a faster parity drive could make a small difference.

 

I wouldn't make the speed of the parity drive a deciding factor, other than to not be significantly slower. You would notice much more day to day performance difference by making your fastest drive be the data drive that's used most often, as read speeds are purely limited by the speed of that drive.

 

Much of the performance optimization discussions will be highly dependant on your exact use case, Unraid has an incredibly diverse array of possible uses that any single recommendation is not likely to be best for you personally.

 

If you lay out your exact daily interactions with Unraid you may get some good advice on best performance, but from your first post to this thread, I'm thinking you don't even know exactly how you plan on using it.

[Hexenhammer]

1 hour ago, jonathanm said:

...Unraid has an incredibly diverse array of possible uses that any single recommendation is not likely to be best for you personally.

 

If you lay out your exact daily interactions with Unraid you may get some good advice on best performance, but from your first post to this thread, I'm thinking you don't even know exactly how you plan on using it.

Hi,

I hope youll have time to read my small wall of text and replay, thanks in advance, Ben.....

 

Yes this is my first post but I planned my use-case for weeks and decided in favor of unRAID vs QNAP.

 

My usage is basically:

I want to move all the spinning rust out of my workstation and my the media HDD's that sit in a closet into single box.

A) 80% usage will be NAS-like function, for one user, I want HDD failure security [the parity option] and some sort of encryption [if unRAID supports Intel Encryption acceleration like on windows with Bitlocker that can do it on the fly].

 

B) Casting/Streaming 4K/HDR videos to either our two 4K TV's [no transcoding needed] or my PC, also no transcoding and maximum 2 users at once.

C) I forgot how its called, never used linux, docks, docklets? please correct me, I want: piHole and something like Sonarr, Radar.

And finally but I dont know if its possible, I want Backblaze backup software running on unRAID and slowly uploading everything I collected, all the TB to get real backup.

Thats it as far as my needs from unRAID for NAS.

 

I have basic understanding of unRAID, for example I dont know how unRAID spreads file load across HDD's, I know it doesn't do it like RAID.

Im sure it cant just fill one HDD, then move to next one, fill it and move to next one and so on, this will slow the used HDDs and unused will sit doing nothing.

 

Is there any setting that allows me to create folders on specific drives and then when i copy files to these folders they land on these drives?

If this possible its great! I can use the fastest HDD for games and create a games, emulators folder on it.

Is there any Tiering/Read Caching support in unRAID? something like providing an SSD drive and telling unRAID to always copy the most used files there?

Or even deeper, to always copy the most used files from specific folder [say my games folder]?

 

Regards

 

 

 

[trurl]

Each disk is an independent filesystem, so each file exists completely on a single disk. But, Unraid allows folders to span disks. This is known as User Shares. Each User Share has settings which control how it uses the disks.

 

21 minutes ago, Hexenhammer said:

Im sure it cant just fill one HDD, then move to next one, fill it and move to next one and so on, this will slow the used HDDs and unused will sit doing nothing.

This is the User Share setting called Allocation Method. There are 3 choices, High-water (default), Most Free, and Fill Up. High-water is a compromise for distributing files without constantly switching between disks.

 

24 minutes ago, Hexenhammer said:

Is there any setting that allows me to create folders on specific drives and then when i copy files to these folders they land on these drives?

Each User Share has settings which specify which disks to include, or which disks to exclude. Include means ONLY, Exclude means EXCEPT. You shouldn't set both include and exclude.

 

27 minutes ago, Hexenhammer said:

Is there any Tiering/Read Caching support in unRAID? something like providing an SSD drive and telling unRAID to always copy the most used files there?

Or even deeper, to always copy the most used files from specific folder [say my games folder]?

Unraid has a cache disk, but it isn't a read cache exactly. Its original purpose was for faster writes without parity, then moving those to the parity array at a later scheduled time.

 

Since then, cache functionality has grown to allow multiple disks in a redundant cache pool. People often use SSDs in the cache pool.

 

Each User Share has settings that control whether and how it uses cache. In addition to temporarily caching writes, some User Shares will usually be set to keep files permanently on cache for improved performance of docker and VMs, for example.

[Hexenhammer]

13 hours ago, trurl said:

Each disk is an independent filesystem, so each file exists completely on a single disk. But, Unraid allows folders to span disks. This is known as User Shares. Each User Share has settings which control how it uses the disks.

Thank you for replay, last question, about the user shares

Ill tell you what I want and you tell me if its possible:

my single reason to have unRAID, I want JBOD + Parity + Some folders to be on dedicated HDD's and all of this to be invisible for the workstation that only sees one big Share and some folders inside.

Can this be done?

Can I do virtual RAID 0 on top of unRAID? Virtual Drives that smaller then the drives they located on, one on each of the fastest HDD's that I have, merged into virtual RAID0, and I still get protection from unRAID+Parity since these are just files.

Edited March 5, 2020 by Hexenhammer

[Hoopster]

14 minutes ago, Hexenhammer said:

I want all my HDD's to be in single array and from my Workstation I want to see one huge single "disk", BUT I want some folders to be allocated on specific HDD's which will be transparent to the workstation

This all depends on how you setup user shares.  They can span all disks or they can be limited to a subset of disks or even to a single disk. 

 

In the shares setup you use the Include and Exclude disk functionality to determine exactly which disk(s) a share will use.  Note, you do not need to define includes AND excludes.  If you exclude disks 1, 3, and 5 the share will automatically use the rest.  If you include disks 1, 3, and 5 those are the only disks that will be used for the share and the rest will be automatically excluded.

 

These definitions are per share, so you can set up your user shares and disk assignments anyway you want.

 

unRAID will expose disks shares (individual disks by disk number) to client machines, but this is turned off by default and you should probably leave disk shares disabled.  Data loss can occur when you mix up disk shares with user shares and copy/move files between them.  Best to always work user share to user share or disk to disk and never mix the two.

[trurl]

33 minutes ago, Hexenhammer said:

workstation that only sees one big Share

You could do this, but this means you only have a single user share that all works the same way with the disks and with network access. Most people have multiple user shares so they can have different settings for different purposes.

36 minutes ago, Hexenhammer said:

Can I do virtual RAID 0 on top of unRAID? Virtual Drives that smaller then the drives they located on, one on each of the fastest HDD's that I have, merged into virtual RAID0, and I still get protection from unRAID+Parity since these are just files.

I will ignore the part about RAID0 since I think all you really mean by that is spanning disks, which has mostly been answered. Maybe this will answer more of that.

 

Each User Share can use any available space on any disks that it includes. The entire parity array has parity protection, but this has nothing to do with files.

 

Parity is just a bunch of bits and the parity calculation treats all disks as a bunch of bits. Parity doesn't contain any of your data. Parity PLUS ALL remaining disks allows the bits of a missing disk to be calculated. This is what parity always means wherever it is used. Parity is just an extra bit that allows a missing bit to be calculated from all the remaining bits.

[trurl]

50 minutes ago, Hexenhammer said:

workstation that only sees one big Share and some folders inside.

From the point of view of the workstation, you could think of the whole server as one big "thing" and the user shares on the server as folders within that, so I don't see any advantage to having everything in one big share and plenty of advantages to having multiple shares.

 

If you are used to mapping network shares to drive letters in Windows, then each share would have to be a separate mapping. But I don't see any good reason to map shares these days since applications can browse the network.

[Hexenhammer]

1 hour ago, trurl said:

From the point of view of the workstation, you could think of the whole server as one big "thing" and the user shares on the server as folders within that, so I don't see any advantage to having everything in one big share and plenty of advantages to having multiple shares.

 

If you are used to mapping network shares to drive letters in Windows, then each share would have to be a separate mapping. But I don't see any good reason to map shares these days since applications can browse the network.

 

Im Using Speed Commander to manage my files [if you never used it: Its like Norton Commander was for DOS but for windows ]

It has drop downs and split screen for folders so i need to MAP network shares for comfortable use and thats why I wanted single drive.

How would you create shares? Based on HDD type? I have 12 spaces for HDDs, 2 for parity 10 for data

12 & 10TB wd helium, one share

SMR second share for media, until i buy WD heliums to shuck

exos another share for games and emualtors

2TB Samsung 850 EVO as write cache [Can single SSD be used as write cache for 3 shares?]

512GBNVME for Dockers

and 1Tb Samsung 850 PRO as permanent cache for files from games share

So 3 extra drive letters

 

[Hexenhammer]

3 hours ago, Hoopster said:

This all depends on how you setup user shares.  They can span all disks or they can be limited to a subset of disks or even to a single disk. 

 

In the shares setup you use the Include and Exclude disk functionality to determine exactly which disk(s) a share will use.  Note, you do not need to define includes AND excludes.  If you exclude disks 1, 3, and 5 the share will automatically use the rest.  If you include disks 1, 3, and 5 those are the only disks that will be used for the share and the rest will be automatically excluded.

 

These definitions are per share, so you can set up your user shares and disk assignments anyway you want.

 

unRAID will expose disks shares (individual disks by disk number) to client machines, but this is turned off by default and you should probably leave disk shares disabled.  Data loss can occur when you mix up disk shares with user shares and copy/move files between them.  Best to always work user share to user share or disk to disk and never mix the two.

 

Thank you for replay, can you please tell me if such scenario is possible:

Single Share out of all hdds, but this share has folders that located on specific drives that are part of the share

For the workstation PC its all visible as Single Drive letter disk with pre-generated folders.

I can do something similar to this in windows by using links [I forgot the exact name of the feature], as example: I had HDD with folder 'Steam' and this folder in reality was an NVMe drive that wasn't even visible in windows.

 

[Hoopster]

16 minutes ago, Hexenhammer said:

Single Share out of all hdds, but this share has folders that located on specific drives that are part of the share

By definition a share is any top-level folder on any disk in the array.  Let's take the example of a "Media" share.  If you create a "Media" share in unRAID or if you create a folder on a disk called "Media" that is a top-level folder it is a user share in unRAID.  If you define the share to span multiple disks, there may eventually be a folder named "Media" on all of those disks.  When this folder gets created on a particular disk is determined by the allocation and split-level settings for a share.  The "Media" folder may exist only on one disk when it is first created but, as the amount of data stored gets larger and larger, it will spill over onto other disks included in the share following the allocation and split-level settings and a "Media" folder will be created on all disks as they are needed.

 

Sub folders of "Media" for example, Photos, Movies, Music, Videos, etc. are not shares.  You cannot create separate disk assignments for these sub folders.  They will utilize the same disks as their parent or top-level folder share, but, they will obey the allocation and split-level settings of the share in determining on which disks that are included in the share the contents of these folders will reside.  If you want to create separate disk assignments for Photos, Movies, Music, Videos, etc., they must be shares (top-level folders) and not merely sub-folders of a user share.

 

Here is the help text for these settings in the Shares tab which may help you better understand how shares work and how these settings influence the location of files (top-level and sub-folders) that belong to a particular share:

 

 

33 minutes ago, Hexenhammer said:

For the workstation PC its all visible as Single Drive letter disk with pre-generated folders.

unRAID itself has no concept of drive letters, but, of course you can map a share to a drive letter in Windows if you wish.  Any sub-folders that are created under that share will be visible under that drive letter in Windows.  You can create sub-folders as needed in any share.  Again, nothing is a "share" unless it is a top-level folder.

 

In summary, yes you can create a single share that spans all available disks if you wish, but, you will not have independent control over which disks any sub-folders of that share may span.  The only control you have is with allocation and split-level settings.  If you want an individual folder to be limited to a certain set of disks and you want a different folder to be limited to a different set of disks, these folders need to be top-level folder or user shares in unRAID.

[Hexenhammer]

45 minutes ago, Hoopster said:

 

 

 

Thank you for detailed replay and the screenshot, this is not what I expected but I can work with this if i have no alternative.

Ill have to create multi disk spanned shares based on Disk Type: SMR disks for media [until i replace them and merge the share with main share], 7200RPM for games/emulators and WD Helium everything else.

I want to clear one thing, I have Workstation for my use, unRAID PC will function as NAS, I may play with VM's and install Android but thats kinda it, ill use  dockers for pihole and usenet and BitTorrent, i need one with Backblaze software to backup [if its possible at all] and local network video casting without transcoding. Basic NAS functions.

 

When I had my homemade Windows 10 based server, I could map root folder of the installed disks to my workstation and on my workstation it was just like if I used a local drive, I never used user shares, i shared the whole drive.

Later on, on windows server I used Dynamic Volumes to create a single volume that spans across bunch of hard drives, and then on my workstation I maped it and used locally as single disk, so I expected unRAID to be like that but more advanced with parity to protect the drives and extra features.

 

 

Another question to sum this up, about Cache drive, can one cache SSD be assigned as write cache for number of shares?

Based on my needs ill have 3 shares, one spans 5 hdds, one spans 4 hdds and one with 1 fast disk [+2 parity drives and thats it 12 HDD bays filled up]

I have 2TB 850 EVO, can I use it as cache drive for all 3 shares?

I was also told that unraid [for now?] doesn't have automatic read cache or tiering to copy most used files from HDD's to SSD and ill have to do it manually.

I put aside 1TB 850PRO for this use, is this completely manual function, i have to say which I files i want on the cache and when i have new ones to move them manually and when old ones no longer needed i also have to do it manually

 

 

 

[trurl]

Whether you have one cache disk, or a cache pool of multiple disks, cache is treated as a single volume available to all user shares. In fact, cache is part of the user shares just like the disks in the parity array. Any top level folder on cache or parity array is a user share.

 

One of the settings for each user share is its Use cache setting. It tells Unraid how the user share will use cache, and whether it will move files from cache to array or array to cache. See here for more details on how this setting works:

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#comment-537383

 

You mention dockers and VMs. There are default user shares the system sets up to help manage these, and these are set to stay on cache for better performance. You would not normally store anything in these shares yourself.

 

Each user share has settings that control how it may be accessed over the network. You can even not share some of them, such as those "system" shares.

[Hexenhammer]

5 minutes ago, trurl said:

You mention dockers and VMs. There are default user shares the system sets up to help manage these, and these are set to stay on cache for better performance. You would not normally store anything in these shares yourself.

But this can be changed right? I have unused 500GB NVMe that I want to use for Dockers/VM's and the 2TB Samsung as write cache and the 1TB samsung as permanent cache for whatever I want to get fast.

 

P.S. Even parity disks are user shares, shouldn't they be hidden system drives, as far as I understand they hold important info that is unusable by end user anyway, parity data.

[Hoopster]

23 minutes ago, Hexenhammer said:

But this can be changed right? I have unused 500GB NVMe that I want to use for Dockers/VM's and the 2TB Samsung as write cache and the 1TB samsung as permanent cache for whatever I want to get fast.

 

P.S. Even parity disks are user shares, shouldn't they be hidden system drives, as far as I understand they hold important info that is unusable by end user anyway, parity data.

Parity disk(s) are never exposed to clients.  There are no shares (or even a filesystem) on a parity disk and they cannot be exported as disk shares.  They are just big buckets of bits.

 

I have one parity disk, four data disks and an SSD cache drive

 

I can choose to export the data disks and cache disk as disk shares if I wish to (only cache is publicly exported in my case).  There are also some more secure ways to export array disks. Parity is never visible.

 

 

The appdata, system and iso shares should generally be kept on the cache drive.  However, you can install VMs/virtual disks anywhere you want.  I also have an unassigned  1TB SSD (not part of the array) disk that I use for VMs and the landing place for downloads before I decide what to move to the array. In addition, I have a 256GB NVMe SSD that I am figuring out how to use best as either a "cache" drive or an unassigned device.

 

Docker containers should generally be in the appdata share on the SSD cache drive or pool, but, some have even opted to move those to an unassigned device.

Edited March 5, 2020 by Hoopster