I am looking to build a low power box to mix of two 256GB SATAIII non RAID SSD and 4 500GB 7.2K in RAID 5 or 10. I dont mind if its hardware or software RAID but if hardware i would have to buy the card.

 

The current planned architecture is to have 3 NICs in the storage box , 1 for general LAN connection and the other two directly connected to a dedicated NIC in each ESX. However i have the abilty to bond nics to the switch if this would be a better route.

 

I am not set in stone with this architecture but my main aim is to be able to saturate each 1Gb NIC if needed or rather get local HDD like performance via ethernet.

 

The problem is, as with all things ESX, there is just too much information out there. I was hoping there would be some relatively simple plug and play OS for this as i simply dont have the time to fettle to the nth degree.

 

The SAN would likely have a cheap i3 CPU/MB with 8GB of ram and the OS would be on a USB3 8GB key.

 

I will have < 10 vms and they are relatively light usage so I dont need blazing speed but equally i dont want rubbish.

 

If anyone has any pointers or links for me to read it would be very much appreciated.

....use a ZFS based solution, like OpenIndiana (OI) and export a pool for your ESX(i)s via NFS....use napp-it to get easy configuration option.via Web.

I can't make out a clear path from the disks you want to employ for having a datastore that would saturate at least 2 NICs at the same time.

Maybe you need to give it some more thought.

I doubt that OI will install and run safely on a 8GB USB3...>16MB is recommended.

 

Thats great. I didnt want to put in too much detail as I wasnt hoping for a detailed reply. Just suggesitons i can use to good and look at specific things.

 

Thanks i will go and research

Ironically, I have spent a better part of the weekend doing just this.

(post is a little long winded since i am still fighting this. I'll remind you that i am a Windowz admin and hardware guy. *nix is my weak point. This does not answer your question at all. but gives you some insight and sharing my weekend experience.)

 

I have an option of putting the datastore in an outboard NAS with NFS or iSCSI (this would be a great use for my spare HP Microserver). A separate box would also be helpful if you have multiple ESXi servers. While I do have 2 ESXi servers, I do not need a central SANS type datastore..  Instead, I chose to try and put the NFS array on the primary ESXi itself as a guest, then use the NFS share as the datastore.

 

The real reason i went with this route was bandwidth (plus there is that Tim Allen mentality that we can put more stuff into one thing). My new datastore will be 4x 2TB samsung F4's in NFS (old datastore is 1x 2TB and 2x1.5TB JBOD). I can get a good 400MB/s sequential read/write with the F4's in raidz. a single gigabit would be the bottleneck. I also plan to expand this out to a second vdev that will speed it up even more. I am still leaving my 2x 256GB SSD's as traditional datastores (one will have the NFS OS that hosts the rest of the guests.).

 

I started out trying OI as a baseline for napp-it. I honestly got stuck configuring it, broke it and had to reinstall.

I figured as long as I was reinstalling, I would also look into freeNAS since I have used it in the past for this purpose with good success (and without hacking it).

I was on a bit of a time crunch. [i only had one afternoon to get this up and running and my main datastore drive keeps making a horrid crash sound and going offline. with requires a reboot. I backed up most of that drive I thought. that drive died in in the restore. I need to get migrated before I lose it all... meh.. its just data.. had I more time, I would have kept up with the IO route.]

 

I actually got that to work quite well.. I had to do about 6 different hacks to it to get it to work in the end. It used to work out of the box for me.

apparently it has issues with the m1015 (works with a hack). i had issues with the ESX 10GB nic (the e1000 still works at 10GB I found out). the NFS reads fast but wont write with ZIL enabled (another hack). If i were to have used the iSCSI connector instead, I think it would have been much easier and just as fast. It also has ESXi tools built in so thats a big plus.

 

The NFS share is reporting about 408MB/s in a guest with CrystalDiskMark. Not bad considering my SSD boot drive on the same guest is reporting about 420MB/s.(it was 408MB read and 4.5MB write before I turned off ZIL)

 

So far... I feel pretty good about about running freeNAS for the datastore inside ESXi. But, I am not done testing yet.

I would not run it without a UPS though since i had to disable ZIL. A power loss could be a total disaster (my UPS does do a clean shutdown, i tested this). I need to look to see if I can either disable synchronous writes on the ESXi box (assuming this is the issue) or toss an SSD into the freeNAS for ZIL. I'll also try iSCSI (the route I should have probably went from the beginning) and see if this just solves the issues without a loss of speed. I'll do this once the migration is complete in a few days.

 

Right now where I stand is, I have freeNAS running as a guest and I am migrating my failing JBOD datastores to the NFS shares. Once that is complete. I'll try patching up and fixing the freeNAS issues listed above. I'll also try to install OI again next to the freeNAS once I have spare time. I will see what one I like better and use that. then I'll  remove the second array (migrating again if i need to).

 

 

 

 

The real reason i went with this route was bandwidth (plus there is that Tim Allen mentality that we can put more stuff into one thing). My new datastore will be 4x 2TB samsung F4's in NFS (old datastore is 1x 2TB and 2x1.5TB JBOD).

 

Johnm, do you expect a performance penalty in exposing the 4 drive zfs raidz as a nfs share ?

 

i remember you were using one of the guests in atlas as your usenet downloader, (sabnzbd maybe?). in any case, any usenet downloader writes a lot of small files and then does cpu/disk intensive parity repair ...

 

how do you think it would fare with shared disks on a zfs raid ?

In theory if sab is setup correctly and given more ram than the sum total parts of one file (usenet parts) sab should be quite efficient. Combine that with on the fly par2 and it should be fine.

 

Thats the theory in practice it would be nice to know how its handles it.

 

Johnm I am focusing my research on ZFS as we speak. I am truely surprised no one has created a distro just for this task there seems to be countless posts on refining nfs + zfs just for this purpose.

Johnm I am focusing my research on ZFS as we speak. I am truely surprised no one has created a distro just for this task there seems to be countless posts on refining nfs + zfs just for this purpose.

 

...the distro is an enterprise grade OS...it is all in OI or Solaris.

napp-it GUI will help...install is 1,2,3, finish..really.

Here are some docs to read:

• http://napp-it.org/doc/downloads/napp-it.pdf
  
• http://www.napp-it.org/doc/downloads/all-in-one.pdf
  

 

...you should also read some tipps regarding the advantage of using  ZFS snapshots with an ESXi datastore.

 

Thanks for the continued suggestions.

 

I am looking into what I can do with a HP microserver. This could be cheap low power hardware ideal to this task.

The real reason i went with this route was bandwidth (plus there is that Tim Allen mentality that we can put more stuff into one thing). My new datastore will be 4x 2TB samsung F4's in NFS (old datastore is 1x 2TB and 2x1.5TB JBOD).

 

Johnm, do you expect a performance penalty in exposing the 4 drive zfs raidz as a nfs share ?

 

i remember you were using one of the guests in atlas as your usenet downloader, (sabnzbd maybe?). in any case, any usenet downloader writes a lot of small files and then does cpu/disk intensive parity repair ...

 

how do you think it would fare with shared disks on a zfs raid ?

 

Anything that is not "native" to the OS will have some sort of penalty.

It seems to work faster then my previous setup so far.

to expand your thought. I am using newsbin pro for the downloader part.

It's dedicated VM split on 2 drives. a 30GB partition on one of my SSD's for its OS and database and a target disk that is now the NFS share.

 

I do not use the included auto par features as they are not efficient and do not allow fine tuning. Instead I use parbuddy.

I can fix a severely broken 8GB parset in about a minute or 2. with this setup. then have it move the completed set to it final destination. that way I can have it auto sort and move whole par sets instead of expanding them.

 

for your last Q

There seem to be Three schools of though on ESX(i) datastores for non-production uses.

1 use iSCSI targets on a SAN/NAS

2 use a NFS share on a SANS/NAS

3 use a local datastore (usually the home enthusiast, those companies with a single esxi box or low budget.)

All have their uses, pros and cons...

 

It really seems to be based on your budget, knowledge, the need for mobility or redundancy.

with the correct hardware. they could all preform about the same. it comes down to how much you want to spend.

 

 

 

...the distro is an enterprise grade OS...it is all in OI or Solaris.

napp-it GUI will help...install is 1,2,3, finish..really.

Here are some docs to read:

• http://napp-it.org/doc/downloads/napp-it.pdf
  
• http://www.napp-it.org/doc/downloads/all-in-one.pdf
  

 

...you should also read some tipps regarding the advantage of using  ZFS snapshots with an ESXi datastore.

 

I have seen this already. This is a good read for those that don't know it. I had started with this path before I was sidetracked with freeNAS (about when I crashed my OI). Now that i have freeNAS working. I am going to re-visit the OI/napp-it route and give it a fair shake.

 

I had decided on ZFS for thing like self healing, snapshots, and speed. It is what we use at work for our production ESX servers. with the exception is we use them as iSCSI targets on redundant SANS.

 

I was getting really sick of the free versions of ESXi taking hours to do simple file transfers and VM backups.

by moving the VM off the internal datastore, I regain the ability to manage my data store at full speed again.

 

for an external (or guest) datastore.. It seems that the OI/napp-it guys prefer NFS. the freeNAS guys prefer iSCSI, microsoft was NFS and now changed their mind to iSCSI if you use a windows NAS/SAN.

 

 

Thanks for the continued suggestions.

 

I am looking into what I can do with a HP microserver. This could be cheap low power hardware ideal to this task.

the HP microserver should do the task just fine. put 8GB of ram in it and should do just fine. you will not be using de-dupe so you need massive ram.

 

I think the bottleneck will be the 1GB nic. if you have a switch that is capable, add a dual or quad NIC and bind them.

 

 

 

Just a small update.

 

I am now looking at a slightly different approach with a virtualised SAN than uses one virtual machine on each of two ESX hosts. I have read that using 10gbit virtual nics and some clever setup can give serious speeds (essentially disk speed - some overhead rather than Ethernet speed of external SAN).

 

Some people have done this with nfs over drdb and theres eve a commercial product or two for it (e.g. SvSAN),

 

I like the concept and it doesnt preclude ZFS but what it does do is give me no single points of failure if i simply add a second switch.

Johnm I am focusing my research on ZFS as we speak. I am truely surprised no one has created a distro just for this task there seems to be countless posts on refining nfs + zfs just for this purpose.

 

...the distro is an enterprise grade OS...it is all in OI or Solaris.

napp-it GUI will help...install is 1,2,3, finish..really.

Here are some docs to read:

• http://napp-it.org/doc/downloads/napp-it.pdf
  
• http://www.napp-it.org/doc/downloads/all-in-one.pdf
  

 

...you should also read some tipps regarding the advantage of using  ZFS snapshots with an ESXi datastore.

 

I know this topic is old.  but I thought my comment had relevance.

 

There is a commercial product called Nexcentastor.  They offer a free version of of the product which I know works with ESXi.  I believe it uses a slightly older version of Solaris.  This version of Solaris is very reliable.  But then Solaris is what ZFS was developed under.  I had also heard that under Freenas and Linux in general that there are other problems that make the FreeNas version of ZFS not really a viable solution.  If your VMWare Datastore is under 18TB, then this would be an excellent alternative.  It has an excellent web interface and is supported through their forums.  Of course the commercial version costs 10's of thousands of dollars.  I think a 198TB raw datastore, with Gold support is roughly $50,000.

 

You can download a copy and give it a try here:  www.nexentastor.org/projects/1/wiki/CommunityEdition

 

On the other hand, if you were over the 18TB limitation, I have seen installations of Open Indiana and NappIT work very well.  I don't know how well under ESXi, but on its own it is very reliable.

 

Also, ZFS NEEDS LOTS of ram.  Dare I say it again?  LOTS of RAM.  Typically 1GB for every Terabyte of storage in the volumes. 

On the other hand, if you were over the 18TB limitation, I have seen installations of Open Indiana and NappIT work very well.  I don't know how well under ESXi, but on its own it is very reliable.

 

OpenIndiana and napp-it have been running great for me for a long time. At some point I am going to switch from OI to OmniOS though. It has a lot more developer support behind it.

Also, ZFS NEEDS LOTS of ram.  Dare I say it again?  LOTS of RAM.  Typically 1GB for every Terabyte of storage in the volumes.

 

Thought this requirement was for Dedupe not ZFS in general.

 

 

I'm also considering doing this, NFS datastore for ESX,  the approach I was going to use was 10gb Ethernet adaptors connected back to back using twinax.  Dual port on the NFS box and single port on the ESX servers.

Also, ZFS NEEDS LOTS of ram.  Dare I say it again?  LOTS of RAM.  Typically 1GB for every Terabyte of storage in the volumes.

 

Thought this requirement was for Dedupe not ZFS in general.

 

 

I'm also considering doing this, NFS datastore for ESX,  the approach I was going to use was 10gb Ethernet adaptors connected back to back using twinax.  Dual port on the NFS box and single port on the ESX servers.

 

I am fairly certain that you need 1GB of RAM for every TB of storage.  Its a caching thing.  The more ram the better your performance.  Also you could supplement your build by adding a SSD for ZIL and L2ARC which will speed thing up considerably.

 

This was pulled directly from a post on another forum about this exact same question:

 

The memory requirement is based on how ZFS itself operates. The writes are cached first into RAM because the system will wait for a certain amount of data is received before it can write to disk as it writes in 128k blocks, which also require a checksum to be created first before it can write it to disk. A lot of other data about the zpool is also kept in RAM which is used for keeping track of health and performance of the disks. And it is also used to perform background performance optimization such as data re-ordering (i.e. when it realizes that when you typically read data X, you will want to read data Y next).

 

Also recommended is a 64bit operating system to overcome the 32bit memory limitation of 4GB.  I think most of the distros are 64 bit now and ZFS was written for a 64bit OS anyway.

 

Also keep reading this post on AndandTech.com.  There is one poster that answers a series of questions that might help answer questions here.  Its like a "Things you want to know about ZFS but were afraid to ask sort of subject".  The link is here:  http://forums.anandtech.com/showthread.php?t=2251048

 

--Sideband Samurai

OpenIndiana and napp-it have been running great for me for a long time. At some point I am going to switch from OI to OmniOS though. It has a lot more developer support behind it.

 

Yea that is what I am exploring myself for a build I am about to start.

 

-- Sideband Samurai