ATLAS My Virtualized unRAID server


Recommended Posts

The reason for the decision to add a ZFS array  was the fact that i did not like a having single spinners for data. I have had to many fail in my lifetime. that and my disk IO was really the weakest link in my ESXi box. (also my single 2TB datastore was getting smart errors. It was time to replace it)

I was going to get a cheapo raid card like an arecca or a used HP. I had a few spare M1015's laying about, I decided why not try ZFS as a guest?..

 

The guest is pretty simple. just a vanilla ZFS raidz on a FreeNAS guest with and NFS guest. It has its own M1015 with 4x 2TB Samsung Green F4's. The array gets about 450MB/s.

 

My plan was to later add a second raidz vdev and then combine them into a single zpool for double the IO (think Raid 50).. in the end i decided my disk IO is fine and that i really don't need a 6TB data store. that, and I would have to split my server into a head/DAS configuration. (I might still do this in the future, I just don't know if I want 8 drives 24x7 spinning)

 

This post here compares the SATA3 SSD Vs. the Array speeds inside my usenet guest.

http://lime-technology.com/forum/index.php?topic=14695.msg182730;topicseen#msg182730

You can see the array is writing faster then the SSD and reading almost as fast.

In this image: the 60GB disk is SSD and the 500GB disk is the ZFS. both disks are also shared IO with other systems so there is a slight hit here.

qHGqUm.png

 

I have the usenet guest and my unraid cache drive on this array (along with several other guests). the array is reading and writing 24x7. It has gone about 6 months now without a single hiccup.

 

The build sort of was a PITA to get running at first. I had to tinker with it a bit. I had two or three big issues right off the bat. the first was the version of freenas I was running did not natively support the M1015 card. I also had add some "tunables".. I cant remember why. I believe those were because of the NFS overhead on top of ZFS brought the array to a crawl? I also remember haveing an issue getting the iSCSI to to be stable (my plan was iSCSI not NFS at first). I almost wish I made a writeup back then... If I had to rebuild the guest, I would be lost myself.

I had planned on switching to Open Indiana and not stick with FreeNAS (the OI guest is built and works). I just never got around to migrating the data.

 

To give you and idea of Atlas's stability. The FreeNAS guest is on 112 days uptime. The server itself has been up longer. I think my last hard reboot was when I replaced my dead expander. It is about time to down it for cleaning.. It is getting dusty inside.

 

PS.

*A nice advantage to the NFS datastore is Gigabit speed to the datastore for migrations or backups.

 

*if your unraid cache is on it. you can create a hybrid unraid server. Fast front end protected cache array that then migrates to the long term storage array.  You can create "cache only" protected shares for things that you need to be faster then standard unraid. Like lightroom data for example.

 

I setup a OI/napp-it guest (2 CPU 6 GB) and got ZFS up and running using with passthrough of an IBM1015.  I am a complete newbie with ZFS.  I was testing performance and I am a bit confused.  I have 6 WD 1TB Green drives.  I am planning on probably using 4 of them.  I setup multiple pools (mirrored vdevs with 4 or 6 drives, raidz with 4 or 6 drives, and a single basic drive) and I get the SAME read performance across each of them ~70 MB/s.  I tested with a Windows7 guest.  I included results for a local single drive datastore as well as an unraid guest and an unraid sever over the network.  From my unraid guest I see ~300 MB/s and from zfs I see ~70 MB/s, same as a single drive.

 

Here is zfs mirrored and raidz with 4 and 6 drives:

kMjSYU4l.jpg

 

Here is unraid over network, unraid guest, zfs single basic disc, and local single datastore drive:

YTo8FX4l.jpg

 

Any ideas why I get the same read performance as a single drive regardless of how I configure zfs with 4 or 6 drives?  What am I doing wrong?

Link to comment

Ok tryin to muster up the courage to build my esxi and unraid into 1 box. I'll probably need to dump $1600 at this build but want to know if you guys think I can just pull all my unraid drives into the new build and be ok? Of course I know nothing is perfect but are my chances pretty high that ill be ok?

 

About ready to start to order all the parts.  ;D

Thanks,

Jeff

Link to comment

JimmyJoe,

 

The ZFS outputs are quite similer for all tests, too similer if you ask me.

 

I dont think its the ZFS array setup, it is clearly something else.

 

I suggest checking the network cards first, are you passing one or using virtual nic?

I am using the virtual NIC in ESXi, not passing one through to any VMs.

Link to comment

I saw now that my CPU don't support VT-D

 

What is the downside?

 

//Peter

 

My MB and CPU shall support VT-d.... need to see how to enable VT-D in BIOS......

 

 

Spec from CPU:  Intel Core i5 3470 (3:e gen.) http://ark.intel.com/search/advanced/?s=t&VTD=true

 

Enhanced SpeedStep technology, integrerad minnesstyrenhet, Execute Disable Bit-funktion, Intel Virtualization Technology, Intel 64 Technology, Intel Trusted Execution Technology, streaming SIMD-utökningar 4.1, streaming SIMD-utökningar 4.2, Intel Fast Memory Access, Intel Flex Memory Access, Intel Turbo Boost Technology 2.0, Intel Anti-Theft Technology, Intel Advanced Vector Extensions (AVX), Intel AES New Instructions (AES-NI), Thermal Monitoring Technologies, Intel Virtualization Technology for Directed I/O (VT-d), Intel vPro Technology, Idle States, Intel VT-x with Extended Page Tables (EPT)

Link to comment

My MB and CPU shall support VT-d.... need to see how to enable VT-D in BIOS......

 

As well as a cpu feature your motherboard also has to support vt-d. It's not guaranteed it will allow you to enable it.

 

Consumer boards are much worse for this than enterprise boards (though they do exist) so it depends what you have. But be warned that having a cpu that supports it isn't, sadly, the whole story.

 

If it does support it enabling it should be quite obvious and straightforward.

Link to comment

Got some better performance numbers from ZFS now.  I changed from e1000 to vmxnet3 and switched from CrystalDiskMark to iometer.  With 4 WD 1TB green drives in raidz I am seeing 210 MB/s write and 260 MB/s read locally and from a Win7 guest about 132 MB/s write and 206 MB/s read.  Good enough for me.

 

Now that I have ZFS setup, how do I use it for a cache drive in unRAID?

Link to comment

Got some better performance numbers from ZFS now.  I changed from e1000 to vmxnet3 and switched from CrystalDiskMark to iometer.  With 4 WD 1TB green drives in raidz I am seeing 210 MB/s write and 260 MB/s read locally and from a Win7 guest about 132 MB/s write and 206 MB/s read.  Good enough for me.

 

Now that I have ZFS setup, how do I use it for a cache drive in unRAID?

 

 

I believe if the guest OS supports it using vmxnet3 is always the best choice since it's 10Gbit.

 

I'm also very interested in doing a setup like this to use as a cache drive. +1 on instructions how to do it.

Link to comment

I haven't done this myself - I still use a physical SSD for cache.. but I do have a ZFS pool set up as a datastore.  A loose set of instructions would look like this:

 

- Share out some space on the ZFS pool as either an iSCSI or NFS share.  (I use iSCSI with my setup).

- Add a VMware datastore using the newly-shared space.  (if it is iSCSI, you will first need to add an iSCSI software adapter to ESXi, bind it to a NIC and point it towards the IP address of the shared iSCSI/ZFS pool.. google will help if you're not sure with that bit.)

- Shut down, then edit your unRAID VM and create a new virtual disk (will have to be IDE) of the appropriate size for a cache drive and have it stored on the ZFS pool.

- Profit.

 

I'm probably going to give this a go myself at some point when I have time - maybe over the weekend.  I've already got the datastore, so should be fairly quick for me.

 

Link to comment

I haven't done this myself - I still use a physical SSD for cache.. but I do have a ZFS pool set up as a datastore.  A loose set of instructions would look like this:

 

- Share out some space on the ZFS pool as either an iSCSI or NFS share.  (I use iSCSI with my setup).

- Add a VMware datastore using the newly-shared space.  (if it is iSCSI, you will first need to add an iSCSI software adapter to ESXi, bind it to a NIC and point it towards the IP address of the shared iSCSI/ZFS pool.. google will help if you're not sure with that bit.)

- Shut down, then edit your unRAID VM and create a new virtual disk (will have to be IDE) of the appropriate size for a cache drive and have it stored on the ZFS pool.

- Profit.

 

I'm probably going to give this a go myself at some point when I have time - maybe over the weekend.  I've already got the datastore, so should be fairly quick for me.

Thanks, that worked for me.  I configured a 500GB virtual disk that is on my ZFS datastore and added that to my unRAID guest as my cache drive.  Unfortunately the performance is horrible.  :(

 

In my tests I have the following:

1.  ESXi guest for OI/ZFS/Napp-it configured with 4 WD 1TB Green drives in raidz

2.  ESXi Windows guest

3.  ESXi unRAID guest

4.  unRAID bare metal

 

Here's my write speeds when copying a 3.3GB file multiple times with consistent results:

~130 MB/s - Windows Guest -> zfs over SMB (I saw similar ~130MB/s with iometer tests)

~100 MB/s - Windows Guest -> unRAID bare metal (limited by 1GB network)

~80 MB/s - Window Guest -> unRAID guest using dedicated cache drive (limited by hard drive speed)

~20 MB/s - Windows Guest -> unRAID guest w/virtual disk for cache on zfs datastore

 

Why is this so slow when using a virtual disk on zfs for a cache drive in unRAID?  Why do I get ~20MB/s instead of ~130MB/s?  I would get better performance (~33 MB/s) if I didn't use the cache drive at all and took the parity hit in unRAID.  :'(

 

Link to comment

I'm now up and running with unRAID rc11 under ESXi 5.1 - pretty easy with the guide and all the help that is available here :-) Thanks a lot !!

 

I have some issues though:

  • vSphere-Client is showing my MV8 SATA controller as "unknown"(passthru)
  • I were not getting the PLOP installer window screen ("you can't save the settings"), hence I always have to select"USB" whenever I want to boot into unRAID

 

The second item is fixed, as I'm reading through all 76 pages and I discovered AaronD's earlier advise: "Also, the Plop boot manager zip file I downloaded contains two ISOs - the first is in the root of the zip file, the second is in the install folder. The first one is, I guess, a live version of Plop; the second is the installer. In my haste I grabbed the first one... perhaps you could mention this again as a note to save anyone else confusion."

 

 

Link to comment

I setup a cache through my ZFS Server.

 

The speed is slow, around 15 MB/s

 

I tried coping directly from unraid (midnight commander) got 15 MB/s

I tried coping from windows to the Cache share of unRaid, still 15MB.

 

When I go to the SMB share of the ZFS server I get about 140 MB/s

 

The ZFS shares it using NFS, I then created a datastore for it in ESXi, created an IDE vmdk and share to unraid.

 

I want to note that I tried coping to an unraid SHARE and directly to the Cache share... same result.

 

Any thoughts?

 

Link to comment

I setup a cache through my ZFS Server.

 

The speed is slow, around 15 MB/s

 

I tried coping directly from unraid (midnight commander) got 15 MB/s

I tried coping from windows to the Cache share of unRaid, still 15MB.

 

When I go to the SMB share of the ZFS server I get about 140 MB/s

 

The ZFS shares it using NFS, I then created a datastore for it in ESXi, created an IDE vmdk and share to unraid.

 

I want to note that I tried coping to an unraid SHARE and directly to the Cache share... same result.

 

Any thoughts?

 

 

Only that I had the exact same issue this time last year and it's never gotten any better - it appears to be a limitation of an IDE-based vmdk in combination with unRAID.  Not much you can do about it I don't think, unless Johnm did something in particular to address it.

 

I ended up sticking with an RDM'ed SSD for cache instead, which operates at full speed.

Link to comment

See this thread:  http://lime-technology.com/forum/index.php?topic=26649.0

 

In short, I am using a ThinkStation D20 (4158-AB7) MB with dual Xeon E5530 and 2x AOC-SASLP-MV8s.  This particular MB has a 5 port SAS/SATA controller that uses the same Marvell chipset as the MV8s.  So...you would think the same MV8 hack applies...or does it?  :)  I had so many issues, I decided to disable the onboard controller and try and flash a Dell Perc H310 (LSI chipset) with an IT firmware.  FAIL!!!  :(

 

In the end, I decided to give the onbaord controller another shot...here are my experiences (from the thread above)...

 

The saga continues...but in a good way! 

 

I have given up on trying to flash the Dell Perc H310 to IT Mode using an LSI firmware.  I just couldn't find a working combination.

 

So, I decided to try and work through the onboard SATA issue.  For those unfamiliar, I planned on running unRAID (RC11 at the time) in an ESXi VM with 2 AOC-SASLP-MV8 cards + the onboard SAS/SATA RAID controller (same exact Marvell chipset as the MV8s) to give me 20 drives...perfect for my Norco RPC-4220.

 

Anyway, I went through all of the normal routine required (Mv8 hack) to use the MV8s in passthrough mode in ESXi...but...I forgot to do it for the onboard controller.  To my surprise, unRAID seemed completely happy.  I let parity check run for about 30 minutes with no errors.

 

Then I remembered that I had not performed the MV8 hack on the onboard controller.  So, I stopped the parity check, stopped the array, powered down the unRAID VM and aplied the MV8 hack.  I started the unRAID VM again and almost immediately I started to see the "Disabling IRQ #" messages in the console!    Within 10 minutes, the webgui would become unresponsive.

 

So, I went "Hmmmm".  I decided shut down the unRAID VM and remove the MV8 hack for the onbaord controller from the VMX.  I started the unRAID VM and watched the console...no nasty IRQ messages or ESXi events!    I logged into the webgui and started a parity check.

 

The parity check is now 20% completed and runing @ 118MB/s!!!  W00t!!!  (fyi...I have never seen speeds as high as this).  I'm pretty certain that I would have seen the IRQ messages by now.  I won't think I am completely out of the woods until unRAID has been stable for a few days.  Beyond that, I still need to keep an eye out for ESXI PSODs.  From memory, these did not show up until about 30 - 45 minutes after ESXi was satrted without the hacks.  The fact that the onboard controller is not using the hack still worries me a bit in regards to PSODs.

 

Hope this helps someone else.

 

John

 

Link to comment

I setup a cache through my ZFS Server.

 

The speed is slow, around 15 MB/s

 

I tried coping directly from unraid (midnight commander) got 15 MB/s

I tried coping from windows to the Cache share of unRaid, still 15MB.

 

When I go to the SMB share of the ZFS server I get about 140 MB/s

 

The ZFS shares it using NFS, I then created a datastore for it in ESXi, created an IDE vmdk and share to unraid.

 

I want to note that I tried coping to an unraid SHARE and directly to the Cache share... same result.

 

Any thoughts?

 

 

Only that I had the exact same issue this time last year and it's never gotten any better - it appears to be a limitation of an IDE-based vmdk in combination with unRAID.  Not much you can do about it I don't think, unless Johnm did something in particular to address it.

 

I ended up sticking with an RDM'ed SSD for cache instead, which operates at full speed.

 

No other way?

 

Is it possible to mount a NFS share and set is as Cache...? it might not be through the WebGUI....

 

or, Can we export from ZFS using an iSCSI? will this work?

 

Link to comment

Do yourself a favour and ditch Plop :)  Try one of my pre-built boot .vmdk's:

 

http://lime-technology.com/forum/index.php?topic=26639.0

 

Thanks for the advise BetaQuasi. I have a new entry on the web console, marked in blue (sda, see overview picture).

 

I do hope that the screenshots of the settings can help to identify what I did wrong:

  • downloaded your VMDK file for 5.0rc12a
  • uploaded the whole folder into the datastore directory for ISO files
  • I wasn't asked to define any disk - I just remove the predefined disk and was adding a new one as: datastore1/ISO/unRAID-5.0rc12a/unRAID.vmdk

Where is my misunderstanding? Thanks for helping.

Link to comment

You've done nothing wrong - it's completely normal for the boot drive to appear at the bottom.  It's the correct size, matching the VMDK, and it can never be part of the actual array.

 

Now you've eliminated the need for plop, and your system should boot far faster, as the core of unRAID (bzroot/bzimage) is now extracting/booting off a virtual disk rather than plop mapping the USB drive through.

 

You weren't asked to define a disk as you were doing this with an existing VM - my instructions are for creating one from scratch. 

 

The only thing I would have done slightly differently is stored the .vmdk file in the actual unRAID virtual machine's folder.. but that's not overly important.  I just get a little OCD about those things!

Link to comment

In either situation, you still need to create some sort of 'physical' drive for unRAID to connect to via its web interface, which in a .vmdk's case, needs to be created with an IDE controller.  There is no real solution afaik.

 

 

Maybe we could get Tom to compile the pvscsi driver into the kernel.

I was going to attempt it myself to see how it works out.

Link to comment

The only thing I would have done slightly differently is stored the .vmdk file in the actual unRAID virtual machine's folder.. but that's not overly important.  I just get a little OCD about those things!

 

Great, great :-) that is really good to hear.

 

That means:

  • stop unraid and power down
  • move unRAID.vmdk from /datastore/ISO/unRAID-5.0rc12a (see pic-1) into /datastore/unRAID (see pic-2)
  • lastly change the path of the virtual disc drive (see pic-3)

I guess that these are the right steps?

 

{EDIT} Yes, that works as described :-)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.