Hi, as I have had a disk fail on me recently (and the PSU), which I'm trying to get back together (my disk arrives today), and I was playing around with it I saw something I hadn't before which was the default format... I am using unRaid 4.7

 

Seen a lot of posts but not really what I want to know in a concise way, so I'm gonna try to ask so that the people that really know can answer if possible, and without having to make them write a lot so yes/no questions when possible:

 

So, there are two formats, the unalligned and 4K, for AF disks. And what I want to know:

 

- Some disks are AF and some are not, and this kinda/mostly depends on when they were made, with most newer disks being AF, right? (anyway to verify if a disk is compatible or not to be 100% sure?)

 

- 4K is supposed to give a performance improvement (as long as the disk supports it)?

 

- My disks all seemd to be "unalligned", and I'm guesssing it's because that's the default and I never changed it?

 

- You can pick the format when you put a new disk in, specific for the disk, or if you had set the default to what you want, it will use that on new disks?

 

- Even if I were to switch the default format, this would have no effect on the disks that are currently installed and part of the array?

 

- If I install a new disk, and want it to have the 4k format, would that have any effect in it not playing nicely with the other disks that are unalligned..?

 

- If a disk already has a specific format (in my case unalligned) it is possible to change it? (albeit losing the data/rebuilding it I would guess). How?

 

 

Even though I've had this a while, I had always read the "quick start" guide, and didn't even know about pre-clear.  I did know the whole formatting thing unRaid did to my disk when I put a new one took forever, but that's not the same thing, is it? Pre-clear is very important?

 

Anything else I'm missing?

You can simply set the default format to 4K aligned and not worry about it.  It will work fine with all of your existing disks, and won't change any of them.

 

As for pre-clear => it serves two functions:

 

(1)  If you pre-clear a disk that you're going to ADD to the array, it will let the disk be added VERY quickly, requiring only a format, as UnRAID won't have to clear the disk to add it [This is necessary for parity to remain valid].

 

(2)  The pre-clear script does a VERY thorough test of the disk ... so it's a good way to be sure a new disk doesn't have any errors before you add it to your array.    Even if you're not adding it to the array -- for example, the new disk you're getting is going to replace a disk, which will require a rebuild ... so pre-clearing it won't save any time;  but WOULD thoroughly test the disk before you assigned it to replace your failed one and allowed the rebuild.

 

You can simply set the default format to 4K aligned and not worry about it.  It will work fine with all of your existing disks, and won't change any of them.

 

As for pre-clear => it serves two functions:

 

(1)  If you pre-clear a disk that you're going to ADD to the array, it will let the disk be added VERY quickly, requiring only a format, as UnRAID won't have to clear the disk to add it [This is necessary for parity to remain valid].

 

(2)  The pre-clear script does a VERY thorough test of the disk ... so it's a good way to be sure a new disk doesn't have any errors before you add it to your array.    Even if you're not adding it to the array -- for example, the new disk you're getting is going to replace a disk, which will require a rebuild ... so pre-clearing it won't save any time;  but WOULD thoroughly test the disk before you assigned it to replace your failed one and allowed the rebuild.

 

Thanks! So I can just set it to 4K, it will use that for my new disks, work just fine no worries. Great

 

However, if this is supposed to have a performance boost... wouldn't I *want* to change the existing disks? 

 

Regarding the pre-clear, I'm actually doing that now (replacing a disk) so I guess if it's recommended I would do the pre-clear on it. I'll look for info on that, since I don't want to clutter up this post in case it's useful for anyone with doubts on AF/mbr format like me

 

However, if this is supposed to have a performance boost... wouldn't I *want* to change the existing disks?

 

Not really.  Any disk that's not AF would make no difference.  On early AF disks there was a jumper that caused the disk to work fine on systems that didn't support AF ... if you have any of those early WD EADS disks, you most likely have the jumper installed.    If you have some of those without the jumper, you COULD copy off all the content;  do a New Config without that disk; pre-clear the disk; add it back to the array; and then copy the content back to it.    But this would take a few DAYS per disk ... and would give you very minimal improvement.  I would NOT bother.

 

 

 

Regarding the pre-clear ... I guess if it's recommended I would do the pre-clear on it. I'll look for info ..

 

As I noted before, it's not necessary, but it'd be a good way to test your disk before using it.

 

This thread has everything you need to know about pre-clear (well, almost everything  ... if you have any questions, just post back here):  http://lime-technology.com/forum/index.php?topic=2817.0

However, if this is supposed to have a performance boost... wouldn't I *want* to change the existing disks?

 

Note really.  Any disk that's not AF would make no difference.  On early AF disks there was a jumper that caused the disk to work fine on systems that didn't support AF ... if you have any of those early WD EADS disks, you most likely have the jumper installed.    If you have some of those without the jumper, you COULD copy off all the content;  do a New Config without that disk; pre-clear the disk; add it back to the array; and then copy the content back to it.    But this would take a few DAYS per disk ... and would give you very minimal improvement.  I would NOT bother.

 

 

 

Regarding the pre-clear ... I guess if it's recommended I would do the pre-clear on it. I'll look for info ..

 

As I noted before, it's not necessary, but it'd be a good way to test your disk before using it.

 

This thread has everything you need to know about pre-clear (well, almost everything  ... if you have any questions, just post back here):  http://lime-technology.com/forum/index.php?topic=2817.0

 

Thanks for the link! Will check it out right now

 

So bottom line regarding the format, is that the minimal improvement you get is not worth the trouble of converting unalligned disks, but since there is however an improvement, even if small, I should pick the 4k for any new disk I add to the array?

 

The only other thing that throws me off is AF disk, if its supported etc. Is the relation that 4K format is only possible on "AF disks"? What happens if you have it enabled and the disk is not AF? Will it not work/crash/just format unalligned? How can I know what my disks are? From my screen they all start with EARS, have no jumper installed and say unalligned.

 

 

So bottom line regarding the format, is that the minimal improvement you get is not worth the trouble of converting unalligned disks, but since there is however an improvement, even if small, I should pick the 4k for any new disk I add to the array?

 

Correct ... just set the standard format to 4K and don't be concerned -- everything will be just fine.

 

 

The only other thing that throws me off is AF disk, if its supported etc. Is the relation that 4K format is only possible on "AF disks"? What happens if you have it enabled and the disk is not AF? Will it not work/crash/just format unalligned? How can I know what my disks are? From my screen they all start with EARS, have no jumper installed and say unalligned.

 

Everything will be formatted correctly, whether it's AF or not.

 

If your disks are all EARS, then they're all AF disks.    But the very minor benefit of formatting them with 4K alignment is NOT worth the hassle of converting them all.    In fact, even unaligned, they're still far faster than you can transfer data in either direction to/from your UnRAID server      You would get a MINOR improvement in write speeds if they were all aligned ... but not enough that you'd notice it.    I'd just switch the default to 4K; leave everything alone;  and let your normal migration to larger drives take care of moving to all-aligned drives 

Lol, ok... Will do the drive replacement now, wish me luck!

 

In any case the replacement drive is WD10EZRX so hopefully no issues...

 

One other (unrelated?) thing I noticed... I used to have an 8 port Gb switch connecting everything (all wired of course), and I remember getting very good speeds, both reading and writing. At one point my switch broke, and I replaced it temporarily with a fast ethernet one. So with this i had speeds os 10MB/s, and said well, thats roughly Fast Ethernet, (optimal would be ~12). But then a few days ago I got the replacement, another 8 por Gb switch, and i find that for some reason while I may get initial bursts of 30-40 MB/s, it alwasy continues dropping until it's between 10-15 normally... Wtf?? I know enough that I won't really get 100-120MB/s, but shouldn't it be much higher than what I get? I remember it being much faster. Or am I missing something..?

I suspect you have been accessing the "failed" drive ... which is actually being emulated by UnRAID by reading ALL of the other drives and reconstructing the data.  You can both read and write to that drive even thought it's not really in the array while it's marked as failed 

 

... BUT that results in MUCH slower disk access, since, for example, a write that would normally require 4 I/O operations (a read and a write to both the data and parity disk) now requires 2 I/O operations to the parity disk, plus 2N operations (where N is the number of data disks you have) to emulate the read and write to the failed disk.

 

I suspect once you replace the failed disk and the rebuild competes you'll see the speeds you were anticipating with the Gb switch.

 

I suspect you have been accessing the "failed" drive ... which is actually being emulated by UnRAID by reading ALL of the other drives and reconstructing the data.  You can both read and write to that drive even thought it's not really in the array while it's marked as failed 

 

... BUT that results in MUCH slower disk access, since, for example, a write that would normally require 4 I/O operations (a read and a write to both the data and parity disk) now requires 2 I/O operations to the parity disk, plus 2N operations (where N is the number of data disks you have) to emulate the read and write to the failed disk.

 

I suspect once you replace the failed disk and the rebuild competes you'll see the speeds you were anticipating with the Gb switch.

 

Ok, hope so! I guess I'll know tomorrow after the process is complete

 

One "bad" thing is for some reason green drives were not readily available here, (and due to that, the bastards that did have them were selling them like at about 25% more..). So i finally said screw it and paid for a 7200 blue (and since the stores are in the city, they were even delivering it to me in a couple of hours).

Not 15 minutes later I get an email with the reply from a seller on our "ebay" that he did have the green in stock and it was at the same price..  

 

Well of course I bought it anway, so i'll be keeping my array green, and have an extra blue drive. Of course also my gaming rig is almost out of space for installing new games, so I guess it's almos sure what it's destination will be...

Well of course I bought it anway, so i'll be keeping my array green, and have an extra blue drive. Of course also my gaming rig is almost out of space for installing new games, so I guess it's almos sure what it's destination will be...

You could also preclear it and keep it as a cold spare and immediately replace any future failures of the same size drive or smaller.

Well of course I bought it anway, so i'll be keeping my array green, and have an extra blue drive. Of course also my gaming rig is almost out of space for installing new games, so I guess it's almos sure what it's destination will be...

You could also preclear it and keep it as a cold spare and immediately replace any future failures of the same size drive or smaller.

 

Yes, that would be the "sensible" thing to do right...?  if only I wasn't such an avid gamer... 

 

Looking at yoursg,any recommendatpn for a vm server? (Separatefrom my fileserver).

Looking at yoursg,any recommendatpn for a vm server? (Separatefrom my fileserver).

Not sure I'm understanding the question but hope the following will answer it anyway.

 

I have 2 ESXi 5.0 hyper-visor PCs.  Each has an unRAID server on it in a VM as well as a Win7 x64 VM each where I run a SageTV server to record from Satellite and OTA.  I also have a WHSv1 VM for backups on one of them and various other VMs on the other.  In particular one for Financial where I put tax return software and where I access my online financial accounts.  I also have a MySQL server that I setup to play around with anything needing a database but currently is unused.

Looking at yoursg,any recommendatpn for a vm server? (Separatefrom my fileserver).

Not sure I'm understanding the question but hope the following will answer it anyway.

 

I have 2 ESXi 5.0 hyper-visor PCs.  Each has an unRAID server on it in a VM as well as a Win7 x64 VM each where I run a SageTV server to record from Satellite and OTA.  I also have a WHSv1 VM for backups on one of them and various other VMs on the other.  In particular one for Financial where I put tax return software and where I access my online financial accounts.  I also have a MySQL server that I setup to play around with anything needing a database but currently is unused.

 

Wow.. That'll teach me to type on the ipad without double checking... Lately the crappiest text comes out... 

 

Anyway, you got my question.  What I want is a server I can put several VMs in, to stand up servers mostly for practice and certification (AD, DNS, certificate servers, exchange, sql, etc).

Looking at yoursg,any recommendatpn for a vm server? (Separatefrom my fileserver).

Not sure I'm understanding the question but hope the following will answer it anyway.

 

I have 2 ESXi 5.0 hyper-visor PCs.  Each has an unRAID server on it in a VM as well as a Win7 x64 VM each where I run a SageTV server to record from Satellite and OTA.  I also have a WHSv1 VM for backups on one of them and various other VMs on the other.  In particular one for Financial where I put tax return software and where I access my online financial accounts.  I also have a MySQL server that I setup to play around with anything needing a database but currently is unused.

 

Wow.. That'll teach me to type on the ipad without double checking... Lately the crappiest text comes out... 

 

Anyway, you got my question.  What I want is a server I can put several VMs in, to stand up servers mostly for practice and certification (AD, DNS, certificate servers, exchange, sql, etc).

Then any hyper-visor would work if unRAID will work acceptably.  I like VMWare ESXi because there is a free version, it is easy to setup and you can pass through PCI cards to VMs.  See the Atlas thread here for setup if you decide to go with ESXi.

Guys, quick (possibly stupid) question.

 

So I replaced the disk, booted up, array did not start, blue ball on the new disk (is it weird that both unRAID and mymain see the new disk in the slot but uumenu is still showing the old one???). 

 

So anyway, I was gonna go command line and do the pre-clear.

 

When that is complete, do I need to restart the server so that unRAID sees the change in the disk? Or is that not necessary?

 

Also, for much later, for the failed disk, would you recommend I put it on the system anyway (not to the array) and do a pre-clear to test it fully, or like you said before just run the wd tool on a separate machine?

Looking at yoursg,any recommendatpn for a vm server? (Separatefrom my fileserver).

Not sure I'm understanding the question but hope the following will answer it anyway.

 

I have 2 ESXi 5.0 hyper-visor PCs.  Each has an unRAID server on it in a VM as well as a Win7 x64 VM each where I run a SageTV server to record from Satellite and OTA.  I also have a WHSv1 VM for backups on one of them and various other VMs on the other.  In particular one for Financial where I put tax return software and where I access my online financial accounts.  I also have a MySQL server that I setup to play around with anything needing a database but currently is unused.

 

Wow.. That'll teach me to type on the ipad without double checking... Lately the crappiest text comes out... 

 

Anyway, you got my question.  What I want is a server I can put several VMs in, to stand up servers mostly for practice and certification (AD, DNS, certificate servers, exchange, sql, etc).

Then any hyper-visor would work if unRAID will work acceptably.  I like VMWare ESXi because there is a free version, it is easy to setup and you can pass through PCI cards to VMs.  See the Atlas thread here for setup if you decide to go with ESXi.

 

Yeah, that's more or less what I though of using, I'll go read up on the thread and think what's the best build for my needs. I assume decent cpu and lots of ram...

Guys, quick (possibly stupid) question.

 

So I replaced the disk, booted up, array did not start, blue ball on the new disk (is it weird that both unRAID and mymain see the new disk in the slot but uumenu is still showing the old one???). 

 

So anyway, I was gonna go command line and do the pre-clear.

 

When that is complete, do I need to restart the server so that unRAID sees the change in the disk? Or is that not necessary?

 

Also, for much later, for the failed disk, would you recommend I put it on the system anyway (not to the array) and do a pre-clear to test it fully, or like you said before just run the wd tool on a separate machine?

 

Hmm looking myself it seems even though unraid was fast toput the disk in the array in place of the old one, pre clear doesnt work on assigned disks 

 

So I un-assigned it, and doing the pre-clear now... Hope that's right.... 

Did you assign the new disk in place of the failed one?  (or in the same SATA port the old one is in ... which may have been why UnRAID automatically assume it was the replacement).

 

Hopefully you didn't change your config ... if you did, it's too late to do a rebuild -- and you just lost all of the data on the failed disk !!

 

If you're simply doing a pre-clear, and didn't change the config; the system will do a rebuild when you finish your pre-clear and assign the disk to the correct slot -- you can tell if you're okay by looking at the array ... if it shows a missing disk with a red ball you'll be okay.  If there's no missing disk, you've changed your config and no longer have the ability to rebuild.

 

In any event, since you've started the pre-clear, just let it finish and then see what the status is.

 

r.e. ESXi builds =>  there are several good threads on ESXi.    If I was building an ESXi server, I'd use a server board with at least ECC memory support ... and preferably buffered RAM (although that can get pricey).

 

The Atlas thread Bob pointed you to is a good start.

 

Did you assign the new disk in place of the failed one?  (or in the same SATA port the old one is in ... which may have been why UnRAID automatically assume it was the replacement).

 

Hopefully you didn't change your config ... if you did, it's too late to do a rebuild -- and you just lost all of the data on the failed disk !!

 

If you're simply doing a pre-clear, and didn't change the config; the system will do a rebuild when you finish your pre-clear and assign the disk to the correct slot -- you can tell if you're okay by looking at the array ... if it shows a missing disk with a red ball you'll be okay.  If there's no missing disk, you've changed your config and no longer have the ability to rebuild.

 

In any event, since you've started the pre-clear, just let it finish and then see what the status is.

 

When i turned the computer on i didnt manually assign anything, however, since i connected it to the exact same sata port I supposed unRAID picked up on this since it placed it on the same spot in the array, and the configuration was valid, with the red ball disk.

After i unassigned it, it showed up with a different color (which i dont remember now and not at home to check) but i *think* it even said like the disk had been replaced.

Left home this morning and the pre clear was still running (last phase i think) after about 7 hours.

 

The other question was if after the pre clear completes i need to reboot or if i can just start the array and unRAID will know it's been pre-cleared?

 

Edit: ok i remoted in to a computer at home and checked. Still says configuration valid, i could start the array if desired (unprotected) and it shows the failed disk red balled and as "missing". Still a picture is worth 1000 words, so attaching.

Everything looks good, right?

Everything looks good, right?

 

Yes, that looks exactly like it should with a failed disk.  Note that even with no disk installed (in the array), you can still read and write to disk2        That's what fault-tolerance does for you !!

UnRAID will emulate the disk by reading all the other disks plus your parity drive to reconstruct the data (for reads); or will write to the parity disk to maintain the correct parity (for writes).

 

When your pre-clear completes, you simply assign the new disk to replace disk2 and it will then do a rebuild of the disk [the fact that it's pre-cleared is irrelevant since you're not adding a new disk ... you're just doing that to test the disk].

 

Everything looks good, right?

 

Yes, that looks exactly like it should with a failed disk.  Note that even with no disk installed (in the array), you can still read and write to disk2        That's what fault-tolerance does for you !!

UnRAID will emulate the disk by reading all the other disks plus your parity drive to reconstruct the data (for reads); or will write to the parity disk to maintain the correct parity (for writes).

 

When your pre-clear completes, you simply assign the new disk to replace disk2 and it will then do a rebuild of the disk [the fact that it's pre-cleared is irrelevant since you're not adding a new disk ... you're just doing that to test the disk].

 

Wheew...

 

Yes, the last part I got, i win't be saving any time later being it's not a new disk to expand the array, and in fact the whole process takes longer, but i do have the advantage of having tested it.

Being I can log in remote, I guess i'll just keep an eye fir when it finishes and then launch the rebuild process while at work

 

Thanks!

One ither question just popped into my head...

 

You were just saying that the fault tolerance feature makes it so when i read, it will read all disks, so it will be like if the failed disk is there, etc, and when i write it will write the parity, etc.

 

Then before when i mentioned the network transfer speed, i was told it was probably due to the failed disk. But why?? I can understand it during a read, where instead of reading that disk it has to read them all, but for a write?? Sure, it writes to the parity drive, but isn't that the normal behaviour anyway? 

 

All of the disks are read to determine the missing value before a write can occur.

All of the disks are read to determine the missing value before a write can occur.

 

Hmmm I guess I need to learn more about the parity that unRAID uses.

 

I would have thougt (at the most basic level, ang thinking about 1s or 0s even/odd parity) that it would simply involve a "fake" write to the failed disk, and a write to the parity disk that would simply invert whatever is at that location...