ATLAS My Virtualized unRAID server

[BetaQuasi]

. After enabling passthrough on the 2 LSI 2008's the now show up in my unraid and Windows 7 installed drivers for the (something with Falcon). I had to ad 2 new PCI devices for that.

 

Eh?  The LSI's should only be passed through to the unRAID VM.. what does Win7 have to do with anything?  (Sorry, a little confusing! )

[dikkiedirk]

Sorry, in my enthousiasm I added them to my Windows7 VM too. Just to see how Windows 7 would deal with them. I kicked them out now. Sorry, sorry, sorry  . Sometimes I get a little carried away. Just glad everything seems to work now .

[BetaQuasi]

oh right! lol - glad to hear everything's working, enjoy!

[dikkiedirk]

I have now installed chilly's vib to enable the X9SCM's 2nd LAN port in ESXi.

 

Which port should be used for unraid? Is it neccessary to use it in passthrough?

 

Are there brackets for the onboard USB ports on the X9SCM?

[BetaQuasi]

Any port you like - yes it does need to be passed through.  Esxi will only show you valid USB sticks to pass through anyway so it's easy (add USB controller to vm then add USB device and it will show you the available sticks)

 

No brackets that I'm aware of - what's the need there?  My stick sits in there very securely.

 

 

[Johnm]

 

 

do not VTd your USB usb. just use the ESX usb passthrough method. (technically you can and some have, but see my warning about killing your exsi image and starting over)

 

Keep in mind that ESXi will not let you boot a guest from a USB stick at this time.

you have to use plop or a disk image of your flashdrive. both ways will boot and use the flash.

 

also keep in mind that the usb in esxi is painfully slow. it is just better then USB1.1.

if you boot unRAID from USB under ESXi, expect it to take twice as long.

[heredago]

I should and will probably create my own thread but in the meantime... 

 

First, I want to give you Johnm (and BetaQuasi and everybody else too ) a BIG BIG BIG thank you for your instructions. I, for sure, wouldn't have learned so much about building this box and how to install everything (it looks so simple once you "know" how to do it, but for a newb like me, all those instructions with the print-screens helped me so much Everything went better than expected! I was able to install ESXi and unRAID with Plop and everything.

 

I use a X9SCM-F-O mb (my build) with currently only 3 disks directly attached to the SATA ports to begin with (1xSSD for datastore, 1x3TB for parity, 1x3TB for Data (and this will grow when I transfer files from my current full HDs to the array when it's ready!)

 

BUT I get the following error when I try to create a RDM mapping in /vmfs/volumes/SSD1/RDMs:

Failed to create virtual disk: The destination file system does not support large files (12).

 

vmkfstools -r /vmfs/devices/disks/vml.01000000002020202020204d4a31333131594e4737384b5341486974616368 HTC3TB-01.vmdk -a lsilogic

 

As I'm using VMFS-5, there's no choice but to use 1MB block size which should be fine? When I research for this error, it always points to the block size but it's always for people running VMFS-3...  :'(

 

EDIT#1: I've just tried this with VMFS-3 and 8MB block size and I get the same error so I'm definitely doing something else wrong  :'(

 

Anybody has a suggestion!? 

[dikkiedirk]

 

 

do not VTd your USB usb. just use the ESX usb passthrough method. (technically you can and some have, but see my warning about killing your exsi image and starting over)

 

Keep in mind that ESXi will not let you boot a guest from a USB stick at this time.

you have to use plop or a disk image of your flashdrive. both ways will boot and use the flash.

 

also keep in mind that the usb in esxi is painfully slow. it is just better then USB1.1.

if you boot unRAID from USB under ESXi, expect it to take twice as long.

 

Yes, at the moment I use the plop method and it is slower when booting. How do i make an ISO of my USB-stick and how do I tell my unraid VM to boot from it. Is it still possible to install plugins this way?

[BetaQuasi]

You can boot the bzimage and bzroot (the main parts of unRAID) direct from vmdk, which is much faster.  See my build thread over here, I have uploaded a .vmdk of RC3 that you can use along with instructions (no screenshots as yet, I'm about to hit the sack.. maybe tomorrow if I can be bothered )

 

There's instructions for updating to a new build too.  The unRAID USB stick is still required (licensing) and plugins etc still live on this stick (i.e. \\tower\flash\extra etc).  Try it though, you'll see it's a lot faster.

 

In saying all that, once you're up and running, you'll rarely (if ever) reboot anyway.

[c3]

I should and will probably create my own thread but in the meantime... 

 

First, I want to give you Johnm (and BetaQuasi and everybody else too ) a BIG BIG BIG thank you for your instructions. I, for sure, wouldn't have learned so much about building this box and how to install everything (it looks so simple once you "know" how to do it, but for a newb like me, all those instructions with the print-screens helped me so much Everything went better than expected! I was able to install ESXi and unRAID with Plop and everything.

 

I use a X9SCM-F-O mb (my build) with currently only 3 disks directly attached to the SATA ports to begin with (1xSSD for datastore, 1x3TB for parity, 1x3TB for Data (and this will grow when I transfer files from my current full HDs to the array when it's ready!)

 

BUT I get the following error when I try to create a RDM mapping in /vmfs/volumes/SSD1/RDMs:

Failed to create virtual disk: The destination file system does not support large files (12).

 

vmkfstools -r /vmfs/devices/disks/vml.01000000002020202020204d4a31333131594e4737384b5341486974616368 HTC3TB-01.vmdk -a lsilogic

 

As I'm using VMFS-5, there's no choice but to use 1MB block size which should be fine? When I research for this error, it always points to the block size but it's always for people running VMFS-3...  :'(

 

EDIT#1: I've just tried this with VMFS-3 and 8MB block size and I get the same error so I'm definitely doing something else wrong  :'(

 

Anybody has a suggestion!? 

 

File size is limited to 2T, so no RDM for 3T drives. If you switch GPT you can fix it.

[Johnm]

 

 

do not VTd your USB usb. just use the ESX usb passthrough method. (technically you can and some have, but see my warning about killing your exsi image and starting over)

 

Keep in mind that ESXi will not let you boot a guest from a USB stick at this time.

you have to use plop or a disk image of your flashdrive. both ways will boot and use the flash.

 

also keep in mind that the usb in esxi is painfully slow. it is just better then USB1.1.

if you boot unRAID from USB under ESXi, expect it to take twice as long.

 

Yes, at the moment I use the plop method and it is slower when booting. How do i make an ISO of my USB-stick and how do I tell my unraid VM to boot from it. Is it still possible to install plugins this way?

 

Yes it is possible..

 

see this thread here. http://lime-technology.com/forum/index.php?topic=14695.msg149602#msg149602

it gives a general idea of how to do it.

I use option 3 myself now.

It sounds like you forgot to rename your flash before you imaged it, so it mounted the hard disk and it cant verify your key vs. the USB Id.

[Johnm]

Still playing with my NFS datastores.. I decided to try backing up the datastore to my unraid.

 

I started with my Datastore ISOs

 

interesting speeds.

 

 

 

Started out at 160MB/s dropped to 145MB/s after the buffer filled and then dropped to 126MB/s at the very tail end.

 

I can live with that...

 

EDIT:

The datastores themselfs are about the same speed to backup.

[c3]

Started out at 160MB/s dropped to 145MB/s after the buffer filled and then dropped to 126MB/s at the very tail end.

 

You do realize is was 125MB/sec (1Gbe wirespeed) the whole time. See MaxNL's charts.

 

What is the target NFS?

[Johnm]

Actually, you can "overspeed" a gigabit "virtual" card on ESXi.

 

everything was on the same ESXi box on the same VLAN.

 

I can humor the idea though and switch the E1000 to a vmxnet 3 and re-run the backup. I do not think it will change much.

 

the backup was from a freeNAS Raidz (raid5 4x 2TB) over CIFS (SMB) to my SSD Cache drive.

 

but yes. I never expected to get more then 1GiB speed this route.

 

After all of my reconfiguration. i had to rebuild my UPS shutdown and backup plan.

The test was to see if this was faster then the old crappy 10MB/s ESXi tools allowed.

[c3]

My ZFS regularly exceeds 1Gbe, but I think you exceeded the 10MB/sec.

[Talsit]

Johnm,

 

Are you doing NFS datastores from your unRaid?  I've been told that will be very slow.

 

I'm running datastores from two old 2.5" drives attached to the MB sata ports (datastore 1 and 2, not raided) and speeds are very acceptable.  I recently installed WHS 2011 as a VM so I can easily backup my Windows machines and it wants real hard drives to back up to (won't backup to vhd's assigned in the storage management section) so, ideally, I would like to have the VM running off datastore 1 and setup a datastore 3 via NFS on my unRaid that will have the required drives for data backup then assign them to the VM in Vsphere.

 

I've been trying to access NFS shares from VMs with no luck.  I'm using rc3 and I'm not sure why I can't get NFS working. 

[Johnm]

Johnm,

 

Are you doing NFS datastores from your unRaid?  I've been told that will be very slow.

 

I'm running datastores from two old 2.5" drives attached to the MB sata ports (datastore 1 and 2, not raided) and speeds are very acceptable.  I recently installed WHS 2011 as a VM so I can easily backup my Windows machines and it wants real hard drives to back up to (won't backup to vhd's assigned in the storage management section) so, ideally, I would like to have the VM running off datastore 1 and setup a datastore 3 via NFS on my unRaid that will have the required drives for data backup then assign them to the VM in Vsphere.

 

I've been trying to access NFS shares from VMs with no luck.  I'm using rc3 and I'm not sure why I can't get NFS working.

 

Negative.

 

Right now i have 3 datastores. 2 Local SSD's on the motherboards SATA6 ports. 1 Datastore on a NFS share that is hosted on a freeNAS VM for now. It is 4x 2TB RAIDz that autostarts with ESXi (surprisingly this works well).

 

I am backup my datastores to a HP microserver also running NFS on freeNAS. ultimately I will try and move the backup to another unraid box.

 

I also want to try NAPP-IT over freeNAS for the datastore.

 

I am sort of rebuilding my entire network slowly over the next month. I am not sure what my final setup will be. but I'm sure I'll post the outcome here. right now I am waiting on a crate of hard drives for temp storage for the migration/rebuild.

 

Interesting on the WHS2011 limitation. I am mid migration of my WHS2011 from an RMD drive to VMDK's. I'll have to test that part out when I get home before I get to much further in my migration. if that becomes a limitation, I'll just stick to RMD or add a PCIe controller for WHS2011.

[Talsit]

I'll be interested in that.  It may not be a limitation in WHS so much as my limitation...

[Johnm]

OH.. i think i am on a different page then you.. you meant to back the WHS2011 itself up to.. not the clients to the whs..

 

Yes! that is a limit. i ran into that a long time ago. I never actually back up my WHS. I keep thinking about it. but i see not point for me.

 

I use my unraid for my storage. my WHS is only for client backup and remote access.

 

the few files I have on my WHS are backed up to my unRAID with a robocopy script that is kicked off daily.

 

My big problem would have been if I had to restore a client and I found out my WHS had some sort of file corruption and cant restore. that is one reason I am moving my whs to a VMDK  on the NFS. that is raid5 with bit-rot protection...

[jbrodriguez]

the intel e3-1240 v2 (ivy bridge) is out, reasonably priced .. but there's a caveat ... to run it in a x9scm-f-o, you need to have bios 2 ... if you are doing a new build and don't have a previous e3-12xx or 2nd generation i3, then you're kind of out of luck, until supermicro ships the board with bios 2 ... ouch ! what to do ? what to do ?

[c3]

the intel e3-1240 v2 (ivy bridge) is out, reasonably priced .. but there's a caveat ... to run it in a x9scm-f-o, you need to have bios 2 ... if you are doing a new build and don't have a previous e3-12xx or 2nd generation i3, then you're kind of out of luck, until supermicro ships the board with bios 2 ... ouch ! what to do ? what to do ?

 

Awesome, the new chips are very nice, and priced only slightly more than the old ones. If the news that bios v2 is required is out, Supermicro will probably get it out soon. But it seems the Sandy Bridge models are still in good supply.

[dikkiedirk]

the intel e3-1240 v2 (ivy bridge) is out, reasonably priced .. but there's a caveat ... to run it in a x9scm-f-o, you need to have bios 2 ... if you are doing a new build and don't have a previous e3-12xx or 2nd generation i3, then you're kind of out of luck, until supermicro ships the board with bios 2 ... ouch ! what to do ? what to do ?

 

Get yourself a G440, for upgrading the BIOS. Or just stay with the Sandybridge e3-1240.

What benefit does the ivy bridge have anyway? Just a 22nm die-shrink of Sandy bridge with 5-15% increase in CPU-performance, and running hotter?

I'm still pleased with my e3-1240.

[c3]

Get yourself a G440, for upgrading the BIOS. Or just stay with the Sandybridge e3-1240.

What benefit does the ivy bridge have anyway? Just a 22nm die-shrink of Sandy bridge with 5-15% increase in CPU-performance, and running hotter?

I'm still pleased with my e3-1240.

 

I am pleased with my E3-1230s, but the newer version runs cooler, as in 11W lower, and as you say faster. To me, it is better to keep all the cluster the same, but if I was just starting, I might consider the newer version.

[heredago]

I have just added my MV8 to my X9SCM-F build and I'm trying to migrate from physical -z RDMs to just using them with VMDirectPath Hardware Passthough.

 

-added the card and plugged the drives to it

-added the card to VMDirectPath Passthough settings in ESXi

-added the PCI device in my Unraid VM settings & removed the old RDMs

-applied the MV8 hack requiring to edit the 2 files using winscp

-restarted ESXi and restarted the Unraid VM

 

But then when I access the interface, it's super slow and at some point, ESXi crashes like this:

 

 

I don't even have anything on my Unraid array hds and I could start all over but I'm just wondering if anybody has an idea?

[BetaQuasi]

What slot is that card in?  I've had no end of troubles running HBA's in slot #3 on an X9SCM.  From that screenshot, it appears you have nothing in any of the other slots?  I'd be trying it in both slot #1 and slot #4 and seeing if the result persists.