ATLAS My Virtualized unRAID server

[hunter69]

I have sagetv installed and working the best I have ever seen as far as performance goes.  The issue is the server crashing.  I cannot tell you with confidence exactly when this started but I feel it started when I tried to pass the serial pci card and was forced to reinstall esxi.

Since that day, I have been having weird issue with the sagetv server.  This is the only server having the issue.

The first setup I had was a 250gig raptor rdm'd as the boot drive and a 1tb wd red rdm'd as the recording drive.  The server would randomly crash.  It would lose a drive.  I first thought it was the raptor (brand new drive).  I thought this because it was crashing, so I decided to make a datastore out of the raptor.  Well I lost the raptor datastore for no reason.  So I pulled the raptor and rebuilt the sage server.  I used my wd red for my boot and a 1 tb drive for my recording (rdm both).  Right now, approximately every 1.5-2 days I lose the recording drive.  I don't believe I have lost the boot drive.

As for loosing the recording drive if you pass through the controller it should solve the dropping problem there.  But otherwise I'm not sure what to tell you.  I would work on one VM at a time until you get them solid.

I appreciate all the help you have provided.  Right now (I am probably cursing myself), since I made the change with my sagetv server, it has been 3 days and everything has been functioning with no dropped discs.  I conducted a test, I had it begin 4 HD recording at the same time to see if it would cause a crash.  So far the server has not blinked.  I am going to let it run for 5 days then fire up my homeseer server to see if it remains stable.  If everything goes well (fingers crossed) the change that made a difference was removing the drive and then readding the rdm drives with the Independent Persistant options under the advanced options.

[jiovani]

Johnm,

 

Are you still using 2x M1015 and your Intel expander right now or are you using 3x M1015 and 1x MV8? If you aren't using the Intel expander do you plan to use it again when you do the Head+DAS setup? How do you have your drives currently connected to your HBAs?  By the way thank you so much for this thread!

 

-Shaun

[BobPhoenix]

I appreciate all the help you have provided.  Right now (I am probably cursing myself), since I made the change with my sagetv server, it has been 3 days and everything has been functioning with no dropped discs.  I conducted a test, I had it begin 4 HD recording at the same time to see if it would cause a crash.  So far the server has not blinked.  I am going to let it run for 5 days then fire up my homeseer server to see if it remains stable.  If everything goes well (fingers crossed) the change that made a difference was removing the drive and then readding the rdm drives with the Independent Persistant options under the advanced options.

Hope that works for you.  I've never tried it before.

[Johnm]

Johnm,

 

Are you still using 2x M1015 and your Intel expander right now or are you using 3x M1015 and 1x MV8? If you aren't using the Intel expander do you plan to use it again when you do the Head+DAS setup? How do you have your drives currently connected to your HBAs?  By the way thank you so much for this thread!

 

-Shaun

 

I am back on the expander. For some reason my expander had died. Intel was very quick in doing a cross-ship RMA for it.

 

 

For those that are unaware, I have 2 full 4224's with 2 unraid severs.

1 physical and one virtual.

 

I would virtualize both of them if I go to a DAS solution. I will need a second expander for my second unraid..

 

Current Build:

M1015 #1 = FreeNAS with Datastores

M1015 #2 + Expander = "Main unRAID"

2 free PCIe 4x slots (future storage servers in DAS?)

 

 

MY future build:

HEAD:

Main server, Datastores, ZFS array, parity&cache drives for unraid DAS boxes, small 2-3 drive unraid(free version) for TimeMachine and WHS backup Target OR migrate one of my 2008r2 8 drive hardware raids (I wont have enough PCIe slots for all 5 data servers).

 

DAS1: (on 24x7)

Expander, drives for "Main unRAID".

 

DAS2: (not on 24x7?)

Expander, Drives for "Backup unRAID".

 

DAS3:

Not likely, but you never know... I would like to move my 16 drive hardware raid here or a second media server.

This is only possible if I get a mobo with more PCIe slots.

 

It will be a while before I undertake this project. right now my spare home network funds are being funneled to a full desktop rebuild at home. I just built 4 new Z77 systems in mac cases. That and the server side is so stable, I hate to mess with it. I have not touched the system since I replaced the expander other then putting some more drives into it.

Once I run out of room on my main unRAID server, this will start to take priority.

[jiovani]

Thank you so much for the above reply Johnm! I have been following this thread for a long time now and plan to start getting the components for my initial build. I have a SM SC933T chassis that I will use first and eventually turn into my "Head" server then plan on getting a Norco after I fill up this case. Again thank you so much for the work you put into this great thread!  By the by you have 5 hardware raid Win2k8r2 servers? Your computer room/office must be a playground for people like us on this forum!  

 

Thanks!

 

-Shaun

[hunter69]

I appreciate all the help you have provided.  Right now (I am probably cursing myself), since I made the change with my sagetv server, it has been 3 days and everything has been functioning with no dropped discs.  I conducted a test, I had it begin 4 HD recording at the same time to see if it would cause a crash.  So far the server has not blinked.  I am going to let it run for 5 days then fire up my homeseer server to see if it remains stable.  If everything goes well (fingers crossed) the change that made a difference was removing the drive and then readding the rdm drives with the Independent Persistant options under the advanced options.

Well 5 days and rock solid  . The only issue was the sage service stopping once and typical collossus issues.  So I am moving to the next phase of testing.  I started my homeseer vm and will let it run for 3 days to see if sagetv starts to crash.  If that goes well then I will start my whs 2011 machine.

 

So far it seems as though making the following change was the solution:

Adding rdm drives with the Independent Persistant options under the advanced options

 

[siamsquare]

Yo,

 

I want to buy another Norco 4224 for my Unraid VM on an Intel expander card and was searching how to power up that second case on its' own PSU so I can turn it off when not used!

Besides shortcircuiting the black and green wire on the connector of the PSU and using it's ON/OFF switch I also found an Supermicro CSE-PTJBOD-CB2 power and fan controller where you connect your power connector,fans and wires from the case and be able to switch it on like a normal desktop!

It's a little more expensive then a paper clip....

 

[hunter69]

Well I started my homeseer server.  I immediately began noticing issues.  Here is what I am thinking the problems are.

 

1.  I need to delete my Homeseer server and recreate it.  I really hadn't done much with it so it will be a pain but I guess I have to do what I have to do.

 

2.  This is probably coincidental but I believe you guys are right on the drive going bad.  I think it might be done, I just hope I can access it to get my recordings back.  I am going to either add a new recording drive or reload my sageserver, utilizing my raptor and moving (my current boot drive) the wd red back to my recording drive.

 

or Iam thinking heavily of moving sagetv back to a hardware based server, not sure. 

 

I crashed esxi trying to pass that serial card to my homeseer server.  it affected sagetv (reload), I thought homeseer was fine because I got it to boot and run normally.  I think that due to this incident there is asome type of corruption or something tied to the vm itself. 

[jbrodriguez]

Yo,

 

I want to buy another Norco 4224 for my Unraid VM on an Intel expander card and was searching how to power up that second case on its' own PSU so I can turn it off when not used!

Besides shortcircuiting the black and green wire on the connector of the PSU and using it's ON/OFF switch I also found an Supermicro CSE-PTJBOD-CB2 power and fan controller where you connect your power connector,fans and wires from the case and be able to switch it on like a normal desktop!

It's a little more expensive then a paper clip....

 

looks interesting ... pls do report back on how it works for you.

[siamsquare]

Will do!

 

1 question though...how do you guys power-off the ESXi server? I usually log-in and stop the VMs and then select power-off in ESXi! But is it possible to just use the power button of my Norco?

 

thanks

[RockDawg]

I've been reading in this thread and others about the best way to implement automated UPS shutdown for VM's and ESXi.  I stumbled across this guys page and he says he found the perfect solution using CyberPower's PowerPanel Business Editions management software.  It is a standalone appliance software built on CentOS and requires no scripting or software to be installed on other VM's except for VMtools.  I watched his video and it seems to work just fine.  What do you guys think about it.  Is there a caveat I'm missing or does this indeed seem to be a great solution?

 

Link to the CyberPower software: http://www.cyberpowersystems.com/products/management-software/ppbe.html

 

Link to the guys Youtube video:

 

Link the guy's web page article:  http://tinkertry.com/configure-automated-shutdown-homelab-datacenter-15-minutes/

[kaiguy]

I've been reading in this thread and others about the best way to implement automated UPS shutdown for VM's and ESXi.  I stumbled across this guys page and he says he found the perfect solution using CyberPower's PowerPanel Business Editions management software.  It is a standalone appliance software built on CentOS and requires no scripting or software to be installed on other VM's except for VMtools.  I watched his video and it seems to work just fine.  What do you guys think about it.  Is there a caveat I'm missing or does this indeed seem to be a great solution?

 

Link to the CyberPower software: http://www.cyberpowersystems.com/products/management-software/ppbe.html

 

Link to the guys Youtube video:

 

Link the guy's web page article:  http://tinkertry.com/configure-automated-shutdown-homelab-datacenter-15-minutes/

This looks interesting.  I was wanting to upgrade my UPS to a pure sine wave device anyway.  Only downfall is taking up precious server resources with that VM (2 GB RAM, 80 GB HD).  Definitely interested to see how this works for someone.

[siamsquare]

Yo,

 

Need some help configuring resources on ESXi 5.1

Currently running a Xeon E1230 quadcore on X9SCM-F and 16GB ram. I have 2 VM's...one Unraid (4GB +2CPU) and one ZFS (8GB + 2CPU)

I've had time-out problems on ZFS when simultaniously uploading and downloading files and I can see in the Performance-tab that CPU power maxes out @100%

SO could use some help allocating the right amount of resources to the VMs!

 

thanks

 

EDIT1 even times out when not maxed out! What is wrong with my system,getting worse!

EDIT2 when I copy data internally from ZFS to Unraid i get the time-out problem but not when copying via PC Win7...VM switch error?

[dikkiedirk]

Right now I use a M1015 and Areca Arc1200 on my X9SCM-F motherboard with 2.0a BIOS. How are ur experiences with adding a 2nd or 3rd M1015 on this motherboard?

[Johnm]

Yo,

 

Need some help configuring resources on ESXi 5.1

Currently running a Xeon E1230 quadcore on X9SCM-F and 16GB ram. I have 2 VM's...one Unraid (4GB +2CPU) and one ZFS (8GB + 2CPU)

I've had time-out problems on ZFS when simultaniously uploading and downloading files and I can see in the Performance-tab that CPU power maxes out @100%

SO could use some help allocating the right amount of resources to the VMs!

 

thanks

 

EDIT1 even times out when not maxed out! What is wrong with my system,getting worse!

EDIT2 when I copy data internally from ZFS to Unraid i get the time-out problem but not when copying via PC Win7...VM switch error?

I do not have n answer for you. it sounds like a configuration issue. how many disks are you running in the ZFS?

My FreeNAS eats all available RAM (as expected) and hardly touches my CPU.

 

ZFS Server Performance

FreeNAS Usage Percentage
Latest: 7.25%
Max: 42.16%
Min: 1.87%
Average: 7.853%

This is 24x7 drive usage with almost 4 months of uptime. the heavy CPU cycles I believe are from when I do scrubs. the Average is only 7.853% CPU usage.

 

EDIT: added full server Performance

Atlas Usage Percentage
Latest: 30.65%
Max: 55.25%
Max: 20.21%
Average: 41.371%

 

There is a tutorial on [H]ard for installing OI on ESXi. I would look to that for configuration.

I do believe you need to assign it 2 cores,.. and no more... or gets unstable. (or was that freeNAS?)

 

Right now I use a M1015 and Areca Arc1200 on my X9SCM-F motherboard with 2.0a BIOS. How are ur experiences with adding a 2nd or 3rd M1015 on this motherboard?

I have had 3x M1015 and 1x arc1222 on this board at once. along with all possible combinations of 3xM1015 and 3x SASLP-MV8's (4 at a time obviously).

I only had issues when I first built the machine with an early Bios. after that i made some changes in the BIOS, it was stable (they are noted in the  install writeup).

 

Now the question I have for you, are you using a Sandybridge or Ivybridge CPU?

There are some people posting Ivybridge issues on this board due to most likely PCIe management with the IVY's

I do not have any IVY XEONs to test this.

 

 

[dikkiedirk]

I have a E3-1240. so Sandy Bridge.

[Johnm]

Then I do not believe that you should have an issue.

 

This tends to be a pretty stable board.

Of course, if you're tossing esxi on it first, you obviously have a new layer of issues to deal with.

 

The best thing is to test unRAID as bare metal first, then get it working under ESXi. this is not always an option in a production environment, but it might help with any troubleshooting if you do have issues.

[Johnm]

I just noticed i do have an issue with Atlas. I have been getting error log entries the last few days.

 

Device t10.ATA_____Corsair_Performance_Pro__
_______________1214810001001454001D 
performance has deteriorated. I/O latency 
increased from average value of 1194 
microseconds to 53369 microseconds.
warning
12/9/2012 8:10:33 PM
Atlas.

 

That is my main datastore that never gets a break. it has been 24x7 pounded for almost 4 months.

It has a couple of mission critical 24 guests that are always generating High IO.

That includes databases and temp files for my usenet downloader. It is also my ESXi host  cache.

 

I noticed it only has about 8% free.

I am going power off all my guests. Then offload some guests. Free up more space then let it sit for an hour or two.

Reboot and see if it flushes itself.

 

While I'm at it, I'll pull it from the rack, open it up and give it semi annual dusting with compressed air.

 

Ugh... downtime...

 

 

[RockDawg]

I have a E3-1240. so Sandy Bridge.

 

You should be good to go then.  I run 3-M1015's on my E3-1230, but I started out with a E3-1230 V2 which is Ivy Bridg and it wouldn't allow all three to work in ESXi.

[dikkiedirk]

I have a E3-1240. so Sandy Bridge.

 

You should be good to go then.  I run 3-M1015's on my E3-1230, but I started out with a E3-1230 V2 which is Ivy Bridg and it wouldn't allow all three to work in ESXi.

 

Which BIOS are you running on the mobo?

[Johnm]

ESXi boards:

prototype sample board has the original bios. (ESXi Dev)

production (retail) board has 2.0a  (Atlas)

 

Baremetal unRAID Boards:

Retail board Original shipping Bios still.

[RockDawg]

I'm pretty sure I'm runnning 2.0a, but I'll have to check when I get home.  I would've thought it would show in IPMI, but I'm not seeing it.

[dikkiedirk]

Thanks, I will add the M1015 soon. Got a few cheap of ebay. Too bad the RES2SV240 is still rather expensive here in Europe.

[BobPhoenix]

Thanks, I will add the M1015 soon. Got a few cheap of ebay. Too bad the RES2SV240 is still rather expensive here in Europe.

Starting with this post on RockDawg's thread are my bios and unraid log files when I was testing 3 M1015s in ESXi 5.0 on my X9SCM-F.

[siamsquare]

Thanks, I will add the M1015 soon. Got a few cheap of ebay. Too bad the RES2SV240 is still rather expensive here in Europe.

Starting with this post on RockDawg's thread are my bios and unraid log files when I was testing 3 M1015s in ESXi 5.0 on my X9SCM-F.

 

For more info on the bioses check this thread:

http://hardforum.com/showthread.php?t=1666761

 

Also need some help...how do you install vmtools on Unraid? My server isn't 24/7 on and would like to be able to shut it down without allways having to go to the Unraid menu and do it there!

My ZFS array shuts down without a problem but it has the tools installed!

 

thanks