Preclearing Drives & Cache Drive Questions

[eldoonmad]

Hello Unraid community,

 

I'm new here and to Unraid. I'm about to switch my system over to Unraid from ESXi and add some shucked drives I just ordered. I've been doing a lot of research about setting everything up in Unraid properly but I could use a little support on a specific topic

I'm pretty much all set and ready to load up Unraid on the system with the flash drive, I'm just waiting for 4 10Tb drives I'm gonna shuck and put in. I started reading about preclearing the drives and I understand that, especially with these USB enclosed discs, it's a good safety measure to take. Other than the 4 drives, the only other storage media I have is a 500gb NVMe drive. Will this be efficient enough to Unraid for installing Unassigned Drives and Preclear so I can preclear the 4 drives? I'd like to plug them in via USB to do this before shucking them just to make a return easier if necessary. Then I'll install them in the system and build my drive pool.

 

Second question

I was thinking that I would utilize the 500gb NVMe drive I mentioned earlier to store Appdata, VMs and Dockers (I do plan on adding an additional NVMe at some point to mirror the data). Would it be advisable or not to use it as a cache drive as well? The 30Tb drive pool is mostly going to be for PLEX media so reading will be the bigger activity. Since I won't be doing a lot of writing to the drive pool except torrent downloads (and I suppose the initial transfer of data from my current storage; 15Tb), I'm wondering if that warrants the need for a cache drive.

 

Any suggestions/ clarification on these topics would be greatly appreciated.

[itimpi]

Q1:  

As long as you can boot Unraid you will be able to install the pre-clear plugin so that you can test those drives.   You can do it via USB but that will almost certainly make it slower.    Note that the preclear is not a fast process as it involves reading and writing every sector on the drive.   With 10TB drives it is going to take days for each drive.

 

There is then the question of whether the ‘preclear’ state will survive moving the drive from a USB enclosure to a SATA connection.    It depends on whether the USB enclosure presents the drive to the system identically to the way a SATA connection does.   In my experience most enclosure do, but not all.    However as your primary purpose at this point is to stress test the drives that is not important.   When you first set up the array whether a drive is pre-cleared or not makes no difference.    It is only later if you try to add a new drive to an existing parity protected array that this distinction matters at all, and even then it still works but just takes longer if the preclear state is not recognised.

 

Q2:

For your Use Case using the NVME drive as the cache seems a very sensible thing to do.    Note that whether ordinary file writes go via the cache is configurable at the share level so it is not uncommon for users to always write directly to the array bypassing the cache for media files and only use the cache for application/VM purposes.

 

during your initial data transfer you definitely do NOT want to go via the cache as it would not speed the process.    Also for the initial transfer if you do not mind the data being unprotected do the load without assigning the parity drive as having a parity drive significantly slows down writing to the array due to the way the system maintains parity in real time.    If you DO want parity protection from the outset then enable the “turbo” write feature which speeds up writing at the expense of keeping all drives spinning.    Parity can be added at any point so when to do it is up to you.

[eldoonmad]

Thank you very much for the response and the tips. I was able to get Unraid booted up and installed the preclear plugin. I decided to just plug the drives into another computer before shucking them and just making sure they were at least recognized and then pulled them from the enclosures and am running preclear on them all in the system. Doesn't seem too bad, got to 10% after an hour an a half but I did tell it to do it twice. 

 

Once this is done I'm going to build my array and create the shares so I can begin transferring data. I will not use cache for that process as you suggested. I do have a question about holding off on setting the parity drive.

Should I add all my drives to the array, create my share, transfer data and then enable one of the drives as parity?

or

Should I add all but one drive to the array, create my share, transfer data then add the last drive to the array and assign it as parity at that point?

 

Then I have a new question about how parity works with Unraid. I understand that it works differently than say RAID5, where it spans the data across all the disks and in Unraid it spans entire files across the discs. When one of the disks does die and all the data goes along with it, does the parity disk make up for that and still provide access to the data lost on that drive or does it only hold the instructions on how to rebuild the data when a new drive enters the array?

 

Again, thanks for the help.

[trurl]

8 minutes ago, eldoonmad said:

Should I add all my drives to the array, create my share, transfer data and then enable one of the drives as parity?

You can't do that because the disk would already have data on it and reassigning it as parity would overwrite that data.

9 minutes ago, eldoonmad said:

Should I add all but one drive to the array, create my share, transfer data then add the last drive to the array and assign it as parity at that point?

Yes

9 minutes ago, eldoonmad said:

Unraid it spans entire files across the discs.

Each disk in Unraid is an independent filesystem, and each file exists completely on a single disk. Folders can span disks. This is what user shares are about.

 

11 minutes ago, eldoonmad said:

When one of the disks does die and all the data goes along with it, does the parity disk make up for that and still provide access to the data lost on that drive or does it only hold the instructions on how to rebuild the data when a new drive enters the array?

Parity is basically the same concept wherever it is used in computers and communications, even with RAID5 or other RAID or Unraid. Parity is just an extra bit that allows a missing bit to be calculated from all the other bits. Parity PLUS ALL remaining disks allows the data for a failed disk to be calculated. That data can even be accessed from that calculation, for reading or writing, even before the disk is rebuilt.

[trurl]

16 minutes ago, eldoonmad said:

Once this is done I'm going to build my array and create the shares so I can begin transferring data. I will not use cache for that process as you suggested.

Don't cache the initial data load as already mentioned, but you should install cache before enabling dockers and VMs so those will get created on cache where they belong. If you allow them to be created on the array there will be a little work to do getting them moved to cache later.

[eldoonmad]

Fantastic. I appreciate the detailed responses, really helps me understand Unraid better.

 

Now, another question. I noticed that I'm not allowed to enable dockers yet because I don't have an array set up yet. I plan to install PLEX and Krusader for starters. Are dockers only stored on the array, I cannot set them up on another disk? I was thinking of having my NVMe set up as the cache disk and also storing app data, VMs and the dockers. Is this possible? Do I just need to create the array first, set that as cache and then it will be available for me to create a share on so I can map these items to it? Or is it better practice to store all of that on the array and keep the NVMe as a dedicated cache disk? Honestly the VMs are more to just play with so I can have various OS' to access and play with if I desire. They won't be heavily used.

 

Lastly, are plug-ins written to and stored on the flash drive Unraid is installed on?

[trurl]

12 minutes ago, eldoonmad said:

I was thinking of having my NVMe set up as the cache disk and also storing app data, VMs and the dockers. Is this possible?

That is usually how it is done, and in fact, was the reason I said this:

50 minutes ago, trurl said:

install cache before enabling dockers and VMs so those will get created on cache where they belong. If you allow them to be created on the array there will be a little work to do getting them moved to cache later.

[eldoonmad]

Got it, I missed that part earlier.

I was thinking of using Krusader to transfer the data. My thought is that will be quicker and more reliable than using Finder or Explorer in other OS'. Would you agree?

Since I need dockers enabled to install Krusader but don't want cache enabled before I do the initial file transfer. Would the best work flow be

Build array

Assign my cache drive

Enable Dockers and VMs

Install Krusader

Create share, but disable the cache in the share settings

Then once the initial transfer is done, enable cache on that share if I so desire

Add my parity drive

Edited May 2, 2020 by eldoonmad