Lot of questions


Recommended Posts

Hi Everyone.
After using synology NAS for many years, I tried to made my own synology (you know what i mean) for better Cost/performance ratio. (and because i can keep my array without a lot of effort)

I was not totally satisfy with this because a lot of thing:

- Having my main NAS on a hack ...

- Can't use nvme SSD

- Can't put the drive to sleep.

- Be ultra carefull before each minor version upgrade

 

Now i'm looking for a dedicated NAS OS (unraid, OMV, truenas)

 

HARDWARE

I want to keep my actual hardware (baby incoming in a few weeks)

MB : ga 170m d3h with I3 6100

RAM : 2x4 GB DDR3 Non-ecc

Case : Node 804

PSU : Old corsair CX 400W

HDD : 3x12TB (mybook desktop outside their case)+ 1x10Tb ironwolf pro

SSD : 2x 250gb NVME (1 on MB/ 1 inside https://www.amazon.fr/Adaptateur-Express-dissipateur-Chaleur-Ajustement/dp/B07S63RT67/ref=sr_1_33?__mk_fr_FR=ÅMÅŽÕÑ&dchild=1&keywords=carte%2Bradiateur%2Bm2&qid=1622452875&s=computers&sr=1-33&th=1)

 

MY NEED

What i do today.
- Store a lot of movie with plex  : Direct play from a shield TV 95% / 5 % transcoding 2 user max 1080p or 4k

- Store my music  . Direct play via Upnp from my stereo

- Docker : Jeedom, octoprint, adguard, bitwarden, watchtower, and maybe i forgot some other small ones.
- VM : Win 10 with some software i may need when i'm away from home.

- I want to move an old external blu ray inside the node 804 to rip my own movie because i keep and old computer only for that.

 

I will add more disk later because i can't stop buying 4k blu-ray and CDs

 

I just started reading about Unraid a few hours ago and i think understand that you can make only 1 big array or am i wrong ?

What I was thinking before reading about unraid :

- raid 1 of half the 2 ssd NVME for VM/Docker

- 2 x half of SSD as Cache for the main array

- SHR or equivalent  for the main array 

- Find 1 or 2 good cloud storage provider to back up the important file. (VM and docker conf + my personnal file)

 

Do you think my hardware will be enough (it was with xpen) ?

How will you configure my array regarding what i wrote before ?

Any other things I need to know/do before starting a trial  ? Like pre clear HDD because my MB is a gigabyte.

Do you know any large storage cloud where i can store all my data for a month during the switch ?

Thanks in advance

 

 

 

 

 

 

Edited by Hellboy42
Link to comment

Up .

Still looking how i will split my array for my "cold" storage (movie, music, documents) to stay IDLE  ou spin down and keeping full performance for VM and docker.

Still looking for a cloud storage to switch from xpen to unraid.
 

 

Edited by Hellboy42
Link to comment

Hello @Hellboy42, missed your message, your are right to update your thread.

 

On 6/1/2021 at 3:26 PM, Hellboy42 said:

Do you think my hardware will be enough (it was with xpen) ?

Should be OK for the NAS and docker parts. I do not use VMs, so I cannot be of much help on that part. However, the i3 6100 is only 2C/4T and might be quite limited for a W10 over the base Unraid. I'll let other users with more experience give their feedback.

 

On 6/1/2021 at 3:26 PM, Hellboy42 said:

How will you configure my array regarding what i wrote before ?

 

55 minutes ago, Hellboy42 said:

Still looking how i will split my array for my "cold" storage (movie, music, documents) to stay IDLE  ou spin down and keeping full performance for VM and docker.

I would use the Array/Pool(s) system to separate the data that is used often and the data rarely used. It works very well for me and my HDDs are spinned down 95% of the time.

You can select for each Share how to manage your files. Only on a disk pool, directly on the Array, etc.

This way, you can put all files for Dockers and VMs on a fast pool and don't spin the HDDs.

 

The question is what size of SSD pool you need. It will probably depend on the number of Docker/VMs and how much data the docker have to manage. It should be OK for starting.

 

On 6/1/2021 at 3:26 PM, Hellboy42 said:

Any other things I need to know/do before starting a trial  ? Like pre clear HDD because my MB is a gigabyte.

Not that I can think of, but I am not sure what you mean with Gigabyte.

 

1 hour ago, Hellboy42 said:

Still looking for a cloud storage to switch from xpen to unraid.

Never looked into that myself.

Link to comment

After reading several posts , i think the one array only is my major concern about my usage.
I don't want to run unraid inside an unraid VM and buy a second licence and usb stick. Maybe i should go with an OMV in VM for my entertainment storage in raid 5 by passing thru the disks but i don't know if it will work properly with hibernation when IDLE.
 

Link to comment
1 hour ago, Hellboy42 said:

After reading several posts , i think the one array only is my major concern about my usage.

 

...based on what feature (or lack thereof)?

Using the Array for media files is the main use-case and feature in unRaid.

unraid will spin down all HDDs in the array when not under access and will only spin-up the ones needed, when accessed.

As a file will not be striped accross HDDs, this is the most energy efficient way and "run as cold as you can get" in a live system.

 

1 hour ago, Hellboy42 said:

I don't want to run unraid inside an unraid VM and buy a second licence and usb stick. Maybe i should go with an OMV in VM for my entertainment storage in raid 5 by passing thru the disks but i don't know if it will work properly with hibernation when IDLE.

 

You should use the SSDs vor VMs, docker (and write cache) in a pool, separated from the array.

Link to comment
4 hours ago, Hellboy42 said:

Up .

Still looking how i will split my array for my "cold" storage (movie, music, documents) to stay IDLE  ou spin down and keeping full performance for VM and docker.

Still looking for a cloud storage to switch from xpen to unraid.
 

 



use the main array for cold storage of your media, and then create as many SSD pools as you need for cache/docker/VM purposes.     If the pools contain multiple devices then they can be made redundant and you can select any of the RAID profiles that BTRFS supports.

Link to comment

Ok, Ford Perfect and now iitimpi were quicker than I could.

 

I share his doubts. Why would a single Array be an issue ?

 

I think there is a misunderstanding somewhere. Either we are not clear enough to get our message through, or we do not understand what is the blocking point for you.

 

 

I can take my example, my Array is mainly media files and my drives are idle most of the time except when I :

  • read medias
  • I add / move files around

I keep all files docker related in my SSD cache pool (appdata & docker image). I could do the same if I was using VMs.

This provides quick access and does not spin up my HDDs.

 

You can now have different pools if you need to.

Link to comment

I think I don't understand properly how the array and the pool are working.

What i was planning was something like following (because of my synology experience)  but maybe i'm totally wrong.


Array 1 My cold storage (3x12Tb with futur expansion)  => I have no backup of this. Only one parity disk for the "security". I won't really lose anything in case of a major crash : I stil have all my CD/DVD/Bluray/4K. It will be weeks of hardwork but not risky. No need for caching because i don't need any performance if I can  read at the max bitrate of any movie.

This part is totally isolated from the working storage, i can remove all the disk and put them inside anorther computer with a fresh unraid install if more space neede in the futur.

 

Array 2 : Working storage. => 1x 10TB  + 2x250GB NVME

- 2x125Gb NVME + 1x125Gb from 10Tb HDD for docker, VM,ect  => in my head it was a 2 disk parity

- 2x125gb NVME Raid 0  for  write/read caching the rest of the 10Tb HDD

- Important files on the HDD are cloud sync + off site backup + external backup

 

Array 3 : optionnal (already on my old synology)

-2 WD Purple HDD for security camera

 

Can you tell me which disk is in the array and which one is in the pool because i'm totally lost.

I think i need to dig up my old HP N54L in my basement to made a few test will old HDD and SSD

 

      

 

 

 


 

Edited by Hellboy42
Link to comment

You can have an Array which is a parity protected grouping of devices. 

 

You can have pools which are either mirrored devices in either SSD, NVME or HDD, but not protected by a Parity drive. 

- You can have multiple pools which you are calling arrays up above.

 

Now if you want to have different folders say 

-TV

-Movies

-Security

 

Which are known as Shares and each share in the array will have its own security and access settings. 

 

You can also attach drives such as USB and use a Plugin called Unassigned.Devices so you can connect and disconnect when ever you need, but keep in mind they are not in the protected array. 

 

What I would seriously do is download a USB Trial copy and simply plug it into your PC. Disconnect all your drives of course and simply play with the interface. 

 

I have the following in my machine. 

 

Array

1 Parity drive

5 x 4TB drives

-Tv Shows

-Movies

-MP3

 

SSD Cache Pool for my dockers such as Plex, Handbrake, Filebot and others

SSD Cache Pool number 2 which I call ssdBackup which is basic backup of my dockers and such. 

Link to comment
1 hour ago, Hellboy42 said:

Can you tell me which disk is in the array and which one is in the pool because i'm totally lost.

I think i need to dig up my old HP N54L in my basement to made a few test will old HDD and SSD

That's a great idea. :) 

 

1 hour ago, Hellboy42 said:

Can you tell me which disk is in the array and which one is in the pool because i'm totally lost.

 

Your quote is not super clear about the SSDs, you first mention 2x250GB and then twice your write 2x125GB.

And I am not sure about the necessity of 10TB for just Docker and VMs. The docker image is generally 20GB and a VM image is probably 20 to 40 GB I guess ?

 

In your situation, I would do this (we can adjust the plan with your feedback ) ; an Array with :

  • Parity 1x12TB
  • Data :
    • 2x12TB (data1&2)
    • 1x10TB (data3)
    • 2x??TB WD Purple (data4&5)

One disk pool with your 2 250GB SSDs in RAID10.

 

Then If you really want to separated the data, you can set different shares with disk exclusion to limit the data in certain shares. For example :

media share with movies, music, etc limited to data1&2

security share limited to data4&5

data for vm limited to data3

appdata share set to stay on the cache pool

image share (for docker and VM images) set to stay on the cache pool

etc.

 

If think that some of the base concepts of Unraid are not super clear for you and that is perfectly understandable. :) 

 

You might want to look at the videos from Spaceinvaderone. There is a new series about 6.9 that cover most of the points, it starts at this one :

 

 

Link to comment
3 minutes ago, ChatNoir said:

Your quote is not super clear about the SSDs, you first mention 2x250GB and then twice your write 2x125GB.

And I am not sure about the necessity of 10TB for just Docker and VMs. The docker image is generally 20GB and a VM image is probably 20 to 40 GB I guess ?

I want half of the SSD for storage. The other half for caching the 10 TB disk

 

6 minutes ago, ChatNoir said:

And I am not sure about the necessity of 10TB for just Docker and VMs. The docker image is generally 20GB and a VM image is probably 20 to 40 GB I guess ?

only 1/10th of the 10TB for VM and docker . 

It will also be my "daily" working disk (that's why i need caching on this one) 

- my documents (sync in the cloud)

- blu ray ripper work in progress.

- torrenting

- ect...

 

Link to comment

I think you need to get your head around how user shares, pools and a cache(-pool) work together in unraid.

The concept is less complicated, once you know and accept these basics.

 

If you can use a test system, I'd strongly recommend to try that. A test licence is valid for 14 30 days and not limited in any way, except that the system needs internet access.

Edited by Ford Prefect
Link to comment

Made some digging in my basement: Hp N40L running

5 disk WD green  1TB ( 4 are OK , 1 with a warning)

1SSD OCZ vertex series 60gb // my very first SSD from 2009 , i think a green disk is faster today 😅

 

For now 2x1 Tb  + 1x1 Tb for parity ( in progress will take a few hour)

I will look further this week end because i can't do anything else during parity calculation with this dinosaure 😁

 

I understand where was the biggest part of my mistake.  On synology the cache is "invisible" it's  not a storage place : you can't put a share on it. 

 

After watching  the space invader part 3 video I still don't know how i should use my 10TB regarding what i was initially planing : Inside the array or use it like an unassigned device with share and use plugin to backup my VM/Docker on it ?

Can the second options  let me split easily in 2 differents rig without any parity re calculation  ? I put  the existing array inside another computer with fresh install. I made the 10TB my main array in another one ?

 

For example :

- One very low consommation and power server for the media storage;  maybe I can even keep my N40L for this  

- One war machine with full NVME SSD and why not put my quaddro P2000 inside of it to remove my CAD computer and only keep a Microsoft surface pro 7 and  a remote deskopt.

 

EDIT : still looking for a temporary cloud to transfer everything

Edited by Hellboy42
  • Like 1
Link to comment

...you wanted the 10TB disk used, partly for storage to/from VM and this share cached as well.

I'd actually leave it in the array, enable cache on it and use the share via SMB from inside the VM. This way, it will look like a network disk to the VM and unraid will take care of a transparent caching. Of course this will depend on what type of data will be written and how frequent.

 

You could as well define a third SSD as pool (and cache) just for this share on the 10TB.

 

Link to comment
6 hours ago, Hellboy42 said:

On synology the cache is "invisible" it's  not a storage place : you can't put a share on it. 

On Unraid cache isn't cache in the traditional sense. It's tiered storage.

 

Unraid doesn't have the option to use a disk as read cache. The files are read from the storage location, whether that's on the main parity array or on one of the defined pools. Existing files are updated in place. Only new files have the option to be written to one location and subsequently moved to another on schedule. If a file needs to be written and read at the speed of NVME, the file will need to stay there. You could manually set up a backup routine to make copies to a different location if you wish.

Link to comment

Used a part of my week end to test UNRAID a little more.

Question is : why did I keep a slow synology for so long ? Okay their is still thing that will take me a lot of time to set up because I'm not a linux power user, but choosing  my own hardware for my need is just so great.


I will keep the 2x250 NVME as raid 1 in the pool for VM and docker  with backup on the 10TB . Is it useless to use raid 1 with a backup ?

I will follow the general advice to keep the 10Tb inside the array with shares and disk exclusions as proposed by chatnoir. Like this I will have 2 disks runing (parity and 10tb) all the time ? 

I will try to dig up a Sata SSD 120 or 250 to cache the 10TB and use as a temp folder for some docker or VM (ripping for example)

Does someone know a good app/docker to rip CD to FLAC and blu ray to ISO or MKV ? Or will I need to use my actual software in my windows VM  ? 

I can't  add my blu ray drive to the actual setup (N40L) Will it be OK ? Does the blu ray drive will count as one drive for the licensing ?

 

Thanks in advance

 

Link to comment
4 hours ago, Hellboy42 said:

I will keep the 2x250 NVME as raid 1 in the pool for VM and docker  with backup on the 10TB . Is it useless to use raid 1 with a backup ?

Not useless in general, no...what makes it kind of useless is to keep it in the same physical location. But for this special case, like having a backup of a VM or Docker Config (you do not need to backup all dockers, as song as theses images are available somewhere on the internet, you know...just keep (and backup) the configuration, external to the container, which is normally stored unter /mnt/user/appdata/<your docker>) in fast access, this is not a wrong strategy.

 

4 hours ago, Hellboy42 said:

I will follow the general advice to keep the 10Tb inside the array with shares and disk exclusions as proposed by chatnoir. Like this I will have 2 disks runing (parity and 10tb) all the time ? 

Nope...only Disks under "constant/too frequent" access will stay up...Parity will only stay up if there is new data written to the array. If there is read-access only, on array disks, it will spin down.

 

4 hours ago, Hellboy42 said:

Does someone know a good app/docker to rip CD to FLAC and blu ray to ISO or MKV ?

there are Dockers for handbrake or makeMKV available.

Music, I cannot recall when I last needed one...there were too many of them to name. The last ones, when the kids were younger, and before the streaming area, I did with Windows media player, just for convenience....including mp3tag

 

4 hours ago, Hellboy42 said:

Or will I need to use my actual software in my windows VM  ? 

see above. But I gather you will need to passthrough the device physically, which would require a second SATA-Controller.

Therefore my strategy was to keep the drive in my Win desktop, create the ISOs from there onto the array and then start the batch processing with handbrake from unraid.

 

4 hours ago, Hellboy42 said:

I can't  add my blu ray drive to the actual setup (N40L) Will it be OK ? Does the blu ray drive will count as one drive for the licensing ?

I think it does not, however the tools I found were much better under Windows and I had a desktop anyway.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.