ATLAS My Virtualized unRAID server


Recommended Posts

  • TV card for the VDR (Mystique Satix Dual S2, PCI-e x1)

Unfortunately I have no experience at all with this card.  I use/have used the following tuners in ESXi pass through: AVer Media Duet and Hauppauge HVR-2250 which work on SuperMicro X9SCM and Tyan S5512 MBs also the Hauppauge HVR-1600 on the Tyan S5512 MB.  The AVer Media requires SandyBridge CPU and SandyBridge Only Bios (Ivy Bridge compatible bios on the X9SCM and S5512 MBs drop the AVer Media Duet).  Hopefully this year some time I will test an nVidia DualTV tuner.
Link to comment

I would spend a few bucks and get a rasberry PI

 

i'm running a raspberry pi as my main media centre. it is awesome for the money.

with the mpeg2 and vc1 codecs on it, it will play almost everything (including HD/1080p).

 

Using a VM on my ESXi, a USB Dual TV Tuner and a free program called ArgusTV and installing raspbmc on my Pi, i can steam live TV across my network (think you can also set recordings).

 

I record to my UnRaid, which shows up as recordings in the PVR area of XBMC on my Pi.

Link to comment

Thank you all for the swift feedback. I take the advise to leave the media box as seperate PC as it is today and only consolidate unRAID and the Windows 7 items under ESXi (as described in the first threat)

 

However to be as flexible as possible - let me ask the question again: what's the right SuperMicro mobo for the cards I mentioned? Btw. IPMI is also something that I would like to implement. And lastly, what't the "greenest" mobo  :D ?

 

The Fractal case compatibility is ATX, Micro ATX & Mini ITX motherboards.

Link to comment

Thank you all for the swift feedback. I take the advise to leave the media box as seperate PC as it is today and only consolidate unRAID and the Windows 7 items under ESXi (as described in the first threat)

 

However to be as flexible as possible - let me ask the question again: what's the right SuperMicro mobo for the cards I mentioned? Btw. IPMI is also something that I would like to implement. And lastly, what't the "greenest" mobo  :D ?

 

The Fractal case compatibility is ATX, Micro ATX & Mini ITX motherboards.

 

Honestly, I would probably still stick with my current build. It is tried and tested. and remains very stable for me.

If you need a 16x slot, then I would start looking at a Tyan or other SM boards.

For my desktops, I have migrated almost all of sandy' to ivys and Z77 chipsets. for servers, I am still on sandys because of the better compatibility with c202 and c204 motherboards.

I have yet to test a SM C216 based board. They might be the next best thing. I just hate beta testing for SM with my data. (although, when the Goliath thread started, it was built on a beta X9SCM board from Asia. that's still running)

As far as green, that's hard to get with tons of hard drives, but I'll say, the 1240 was not much more of a hog then the I3 chip. when idol, the newer XEONS are pretty energy efficient. I honestly forgot my power usage. I want to say it idols at about 35-40 with (everything spun down) and peaks at 200ish with 28 drives on it. I know it has been posted more then once in this thread.

 

Link to comment
  • 2 weeks later...

I woke up this morning and Atlas was locked up. A soft reboot failed.. and hard reboot resulted in a crash shortyly after the console comes up. no VMs will start...

 

I'm getting a generic error on the ESXi side before it totally dies.  just a minute or two after reboot. It looks like a license error. it could also be an esxi cache failure.

 

I am going to bet the flash drive is finally starting to give up..

 

It just so happens I am home this weekend. so i can work on it. (talk about  timing)....

 

I have also been putting off upgrading to 5.1, now i have that chance...

 

Installing to a new flash now.. I'll let you guys know the result..

 

Link to comment

I woke up this morning and Atlas was locked up. A soft reboot failed.. and hard reboot resulted in a crash shortyly after the console comes up. no VMs will start...

 

I'm getting a generic error on the ESXi side before it totally dies.  just a minute or two after reboot. It looks like a license error. it could also be an esxi cache failure.

 

I am going to bet the flash drive is finally starting to give up..

 

It just so happens I am home this weekend. so i can work on it. (talk about  timing)....

 

I have also been putting off upgrading to 5.1, now i have that chance...

 

Installing to a new flash now.. I'll let you guys know the result..

 

Ouch! My worst nightmare come true. I need to badly work on a backup system and the ability for my machine to power itself down when the UPS fires off. Need to move machine off my coffee table in the living room too! :D I too have put off 5.1 now that I know what I was doing wrong before, I think I need it to run Win8 in a VM and I still want to get OSX going.

 

Do please keep us posted, hopefully this won't be anything too bad for you to fix. <shiver>

Link to comment

I woke up this morning and Atlas was locked up. A soft reboot failed.. and hard reboot resulted in a crash shortyly after the console comes up. no VMs will start...

 

I'm getting a generic error on the ESXi side before it totally dies.  just a minute or two after reboot. It looks like a license error. it could also be an esxi cache failure.

 

I am going to bet the flash drive is finally starting to give up..

 

It just so happens I am home this weekend. so i can work on it. (talk about  timing)....

 

I have also been putting off upgrading to 5.1, now i have that chance...

 

Installing to a new flash now.. I'll let you guys know the result..

 

Ouch! My worst nightmare come true. I need to badly work on a backup system and the ability for my machine to power itself down when the UPS fires off. Need to move machine off my coffee table in the living room too! :D I too have put off 5.1 now that I know what I was doing wrong before, I think I need it to run Win8 in a VM and I still want to get OSX going.

 

Do please keep us posted, hopefully this won't be anything too bad for you to fix. <shiver>

 

I found the root really fast..

 

If anyone remembers the SSD  the I was having issues with a month or 2 ago?

Well that should have been a sign of things to come. I should have backed it up and did more testing on it.

unfortunately i was out town and there was not to much I could do..

 

Well that is the root of my issues.. I was a cache write error.. it so happens it is also the SSD my ZFS data store is on... so that locked up my ZFS. that in turn took out almost all of the rest of my VM's..

 

I had been keeping backups so nothing is lost.. other then maybe an SSD.

 

As of right now, any attempted writes to that SDD cause an error and ESXi drops it.  at least 5.1 does. 5.0 just left it online but frozen..

 

That SSD  is heavily abused.  That is the SSD my sql server and another 24x7 database run on and my usenet downloader.. so it sees more abuse and pounding then it should. in adition, it never goes idle for the garbage collection to do its job.  i know it was going to fry eventually from this.

 

You mentioned backups. i use both ghetto backup and veeam free. the veeam free is really damn nice for datastore migration. it goes as fast as your machines can go..

 

I did a final backup of the system at pretty much the speed gigabit to my windows server.. (pic attached)

veeam.png.239e7d821786f577e95156adaf546bc1.png

Link to comment

There has been a lot of talk in this thread about ZFS inside ESXi ..

 

this page has some good info on datastores on nix boxes inside ESXi

http://www.vmdamentals.com/?p=4465

 

he said the Nexenta was the only one to successfully rebuild in his testing.. I think i need to do a test myself once i get back up and running.  I wish he tried Open Indiana.. (it should be the same result as Nexenta)

Link to comment

OK.. Its baaaack......

 

I lost an SSD and lost a Flash Drive it looks like..

 

the flash might be fine. it did not want to boot back to the v5 flashdrive after i did a clean install of V5.1 on another Flash Drive.

At few bucks each, i'll just format it and put it to use elsewhere. its not worth my time to look at it. its a flash drive.. sheesh......

 

The SSD, It is still in the server for now.  I'll pull it out after my ZFS scrub and unraid Parity check pass. It might be fine after a reformat. it might not be. it might have  ton of burned out cells. ill format, smart test and SSDlife it.. see what i get.. its got years left on the warranty :) I might be calling on it..... I'll let you guys know..

 

 

I can not stress enough the importance of backups!!!

I thought I was up to date on my back ups. It turns out I was not. I had backups of most of my VM's (the ones that were on the ZFS array that I didnt need to restore) I did not have a good backup of my freeNAS build and I did not have a current backup of my usenet VM  (7 months old then it ran out of space and i didnt look at the logs.)

I was backing up the SSD's daily to the ZFS array to a share that was 100% full. then backing up the ZFS once a week.

(I always say, It is not a backup until you test the restore)

 

I also lost a few test VMs that i dont care about..

 

The usenet VM was not so bad because I do backup the database and the downloads are kept on a separate ZFS VMDK. restoring a 7 month old image and doing patches and the database restore was good enough. I only lost about 4 hours of data. the database backup happened right before the crash.

 

The FreeNAS rebuild was horrid and made me question why I use this product. i had to re apply all of the hacks. after that,  It would not re-import the array at all in the GUI. when I tried it from CLI, it failed the mounts and came back up as an empty array.. i decided to try an Export from CLI and then re-import from gui now that i could see it. This time it froze. i was pretty sure i had lost all of the data and would be doing a massive restore. I walked away and about 20 min later it unfroze and mounted the array. Apparently it was just doing its job while being unresponsive. Unforgiving I believe I called it before.. I still go with that.

I recommend those considering this route, consider a hardware RAID card or external storage array. it might just save you a headache..

 

So moral of the story.. backups... also, when your hardware starts giving errors.. fix it or replace it. I had warnings this was about to happen..

 

 

Bottom line.. what a waste of a weekend.. although most of it was just doing restores..  I'm just glad it happened before i left for several months, if that even makes any sense.

 

 

Link to comment

FYI: ESXI USB Flash Drive failed.

 

I received the message below after a power failure.  I got everything working again and wanted to leave a note for anyone else that encounters this problem.

 

Hypervisor Recovery

BANK 5: Invalid configuration

No HyperVisor Found

 

Long story short, the USB Flash Drive failed.  I had to re-install ESXi to a new USB Flash drive.  Brought up new install of ESXi using Atlas thread.  Make sure to disconnect all drives and usb sticks prior to installing ESXi, lets it erase everything!  After new install, re-connecting all drives and usb sticks (unraid) - all I had to do was locate the virtual machines (VMX files) on the data store drives (browse files / right click) and mount the virtual machines.  Then I had to add the MV8 hack, configure passthrough (per atlas guide) for the controllers and extra nic card.  After that all I had to do was remove the old pass through devices on the unRaid virtual machine and add them back in freshly.  Started up all the virtual machines, including unRaid - and everything restored perfectly.    I was careful that all the cables I removed were plugged back into their original slots.

 

 

 

Link to comment

I'm actually looking at cache drive setups at the moment.  But I'm not a 100% sure of the correct use of a cache drive. 

 

My understanding it is used for apps (inc downloads).  But for my setup I want to stream content to XBMC.  Is a cache drive usable as a data store as well (e.g. covers etc) or do I need another drive?

 

So one cache drive and one data drive for XBMC covers etc

Link to comment

I'm kind of itching to build one of these mega servers but I'm rather stuck at deciding what sort of motherboard to use.

 

The X9SCM-iiF seems like an obvious choice, but I worry about the Ivy Bridge issues people seem to have.

 

The Asus P8C WS is close to what I want, but no IPMI.

 

Are there any boards available in a form factor that will work in a 4224 with C216, IPMI, VT-d, etc? Maybe I'm just not looking hard enough >.<

 

Or am I barking up the wrong tree entirely with C216?

Link to comment

I'm actually looking at cache drive setups at the moment.  But I'm not a 100% sure of the correct use of a cache drive. 

 

My understanding it is used for apps (inc downloads).  But for my setup I want to stream content to XBMC.  Is a cache drive usable as a data store as well (e.g. covers etc) or do I need another drive?

 

So one cache drive and one data drive for XBMC covers etc

 

A cache drive helps with writes but does nothing for streaming apps that are mostly reads. If reading is your primary activity, and with XBMC it will be (I have a few of them myself) skip the cache. A cache drive IS helpful if you're doing lots of add-ons since programs can write data directly there but in this thread we're talking ESX servers and for things liek SQL server, SAB/Sick, etc. it's just easier to spin up a VM for them - in fact there's even an appliance that does Sick/Sab/Couch/Headphones/Maraschino out of the box for ESX...

Link to comment

I'm kind of itching to build one of these mega servers but I'm rather stuck at deciding what sort of motherboard to use.

 

The X9SCM-iiF seems like an obvious choice, but I worry about the Ivy Bridge issues people seem to have.

 

The Asus P8C WS is close to what I want, but no IPMI.

 

Are there any boards available in a form factor that will work in a 4224 with C216, IPMI, VT-d, etc? Maybe I'm just not looking hard enough >.<

 

Or am I barking up the wrong tree entirely with C216?

 

I'm running a different mobo in my setup -> http://lime-technology.com/forum/index.php?topic=23349.0 and it supports the features you're looking for. the Ivy issue appears to be mostly a passthru of certain kinds of hardware. I'm running a Sandy CPU due to those concerns but am very very tempted to upgrade and see if I run into issues as I'd like to be running an Ivy CPU myself. Sadly none of the boards at this level appear to support overclocking which is a real bummer :(

 

I've got an i7 board here from MSI that supports ESX with onboard NIC etc. that would make for a more budget build and would certainly fit in the room 4224 case - these cases are heyuge! It's a Z68MA-G4 board b3 rev. Sadly I paired it with a 2600K not knowing at the time that Intel neutered the K series CPU to prevent VTD passthru - grr. Thus when I built this system I went with the server oriented board posted above.

 

Lots of ways to slice this and I'm still learning, I may even take some VMware classes to learn more about this technology. I had the opportunity to pigenhole a VMware rep the other day and forgot to ask him about the Ivy issues some see too - grr. Anyway, this thread and others are huge helps so read them closely and ask questions, figure out what you need for your setup and fire away...

Link to comment

Yeah it's not about in terms of size, but rather form factor. Supermicro has a board that seem like they have the features I want (X9SPU-F) but it's UIO form factor >.<

 

Obviously a lot of people have success with their sandy bridge setups but if I'm going to put the kind of money needed into this kind of setup I'd rather get the newer tech (since there's no real price difference from what I've seen, unless I just missed someone having a great sale on 1240s vs 1240V2s).

 

I've read through (much) of this thread and many more like it and many on servethehome, etc. I've specced out the entire server *except* the motherboard due to the pesky ivy bridge thing. I'm only planning to have a single m1015 with expander, so it's a non-issue but I'd hate for it to bite me down the line if I put another card in there and expected it to pass through. :\

Link to comment

I'm running unRAID 5rc10 as a VM with 2GB RAM.

 

I tried to update to 4GB but received this error message:

 

Failed to start the virtual machine.

Module MemSched power on failed.

An error occurred while parsing scheduler-specific configuration parameters.

Invalid memory setting: memory reservation (sched.mem.min) should be equal to memsize(4096).

 

I tried with 3 GB and received a similar message. Tried back with 2GB, and it starts without issues.

 

Any idea what is this?

 

Link to comment

Yeah it's not about in terms of size, but rather form factor. Supermicro has a board that seem like they have the features I want (X9SPU-F) but it's UIO form factor >.<

 

Obviously a lot of people have success with their sandy bridge setups but if I'm going to put the kind of money needed into this kind of setup I'd rather get the newer tech (since there's no real price difference from what I've seen, unless I just missed someone having a great sale on 1240s vs 1240V2s).

 

I've read through (much) of this thread and many more like it and many on servethehome, etc. I've specced out the entire server *except* the motherboard due to the pesky ivy bridge thing. I'm only planning to have a single m1015 with expander, so it's a non-issue but I'd hate for it to bite me down the line if I put another card in there and expected it to pass through. :\

I believe the error with my AVerMedia Duet (isn't recognized with Ivy compatible bios on my X9SCM) is that it is a 1.0 PCIe x1 based card and the MBs bios only works with 2.0+ based cards.  Intel had the same problem on a H77 model MB and updated the bios to fix the problem.  It looks like (just checked) my specific model (DH77EB) has been updated as well so I'm going to try it tonight if I'm not recording from it.  So if Intel can fix it for their MBs the other makers should be able to fix their boards with a bios update.  ESXi 5.1 is required to recognize devices properly with an Ivy Bridge CPU for pass through - they show up with weird names.  Updating to Ivy isn't an option for me because even with the ESXi update to 5.1 that now allows PCI cards to pass through (PCIe works fine on 5.1 original if 2.0+ PCIe card) without pink screening it still doesn't pass through onboard USB controllers and I'm doing that as well.  It should be noted that even if the MB doesn't have a PCI slot some of the built in devices you might want to pass through are connected through PCI rather that PCIe.
Link to comment

Hrmph, I need to pass thru both an M1015 (x2) and USB for unRAID. I haven't flashed my board to the most recent BIOS, can that be done remotely without halting ESX or must everything be brought down? I believe I have a late enough BIOS that it already supports Ivy but since my specific BIOS isn't listed on Tyan's site I'm not 100% sure. <sigh> Looks like there are two flashes to be done too and not just the BIOS. I may wait until I have an Ivy BIOS in hand before playing with this.

Link to comment

Has any of you succeeded in integrating an UPS with ESXi? I downloaded APC Powerchute Network Shutdown but that doesn't seem to work with the free license of ESXi. Are there other tools that can make this work? Unraid is working fine with apcupsd. Or is it easier to install a guest OS (Windows XP or Linux) that can shutdown the server after all VM have shutdown?

Link to comment

Hrmph, I need to pass thru both an M1015 (x2) and USB for unRAID. I haven't flashed my board to the most recent BIOS, can that be done remotely without halting ESX or must everything be brought down? I believe I have a late enough BIOS that it already supports Ivy but since my specific BIOS isn't listed on Tyan's site I'm not 100% sure. <sigh> Looks like there are two flashes to be done too and not just the BIOS. I may wait until I have an Ivy BIOS in hand before playing with this.

On the Tyan S5512GM2NR - bios 1.05 (what I have) is Sandy Bridge only, 2.02 (I think it is .02 anyway) is Ivy compatible.  My ESXi info came from a blog post where the guy talked about PCI working now on the update to ESXi 5.1 but that a built in USB controller wouldn't pass through correctly.  If they have put out a 2nd update that may be fixed or it could be specific to his MB.  I just don't want to take a chance.  Sorry don't have the link handy or I would post it but I'll see what I can find.  I got the link when someone else posted a link to the blog on these forums.
Link to comment

Hrmph, I need to pass thru both an M1015 (x2) and USB for unRAID. I haven't flashed my board to the most recent BIOS, can that be done remotely without halting ESX or must everything be brought down? I believe I have a late enough BIOS that it already supports Ivy but since my specific BIOS isn't listed on Tyan's site I'm not 100% sure. <sigh> Looks like there are two flashes to be done too and not just the BIOS. I may wait until I have an Ivy BIOS in hand before playing with this.

On the Tyan S5512GM2NR - bios 1.05 (what I have) is Sandy Bridge only, 2.02 (I think it is .02 anyway) is Ivy compatible.  My ESXi info came from a blog post where the guy talked about PCI working now on the update to ESXi 5.1 but that a built in USB controller wouldn't pass through correctly.  If they have put out a 2nd update that may be fixed or it could be specific to his MB.  I just don't want to take a chance.  Sorry don't have the link handy or I would post it but I'll see what I can find.  I got the link when someone else posted a link to the blog on these forums.

I think this was the blog I read.  If you read the comments you can see where they say USB is still broken.  If you read through the VMware community post (I went to the end and backed up a page and didn't read whole thing) you can see that they have updated PCI to work now but built in USB controller doesn't pass through but USB card in PCIe slot does.
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.