ATLAS My Virtualized unRAID server

[AaronD]

I don't know your exact setup, would do a little more testing before you panic. there are to many factors to just take a guess.

 

My datastore is running on a 320GB laptop spinner, and the unraid disks are pased through from an LSI card - all green spinners. I have a cache drive also, but in this case I was reading from unRaid not writing to it. I have plans on doing something better with the datastore drives, but good things take time  

 

But you're right... I need to do more testing, familiarise myself with the system and how ESXi works, and work it through. I'm definately not panic-ing, more wondering how other people find performance. I know now that I have your 80MB/s as a bar to measure my performance against.

 

Cheers

Aaron

[WeeboTech]

A far as the SSD..

I find them to be incredibly faster then spinners. That is what I need for many of my IO intensive guests.

I have one guest that is  running a large database with thousands of writes per minute (or was it per second? i am not awake yet) 24x7. it would not be possible on a spinner without 16+ gigs of ram dedicated to it. The entire system was lagged on spinners, even when on bare metal.

 

It is not necessarily recommended to run a datastore on SSD.  ESXi does not support Trim so that's going to impact the life of the SSD.

I do state I am most likely going to burn out my SSD's. it should take a few years they way I am using them.. If it is sooner, so be it, I just hope not.

you could get the kingston SSD now+ with bios controlled trim and garbage collection.

In the next 2 years SSD's should be larger and cheaper (maybe faster) and I'll just pitch these and get improved models. these should make it until then. just make sure you leave lots of headroom on the drive.

For me, the cost is worth the reward.

---

 

 

From what I read, if you write 20GB a day, it will take 5 years before an SSD is unusable.

Quite frankly from what I'm reading in reviews, I think realisticly it is about 2 years.

 

One of my concerns is how do you back up the data store drive?

I'm unfamiliar with ESXi. I use workstation now and I've used GSX or vmware server.

 

I tried installing ESXi on a machine to realize that I would not no direct access to the drive if I wanted to copy files (at that time).

So how is it done in an automated fashion these days?

 

 

Also as far as laptop spinners, I would suggest looking at the momentus XT 500GB. One of the best laptop drives I've ever used.

 

[WeeboTech]

Curious if anyone has ever tried this.

 

1.  Ripping DVD's on a guest via XP or Windows (I use anydvd).

2.  Ripping CD's on a windows guest to unRAID. (I use EAC).

 

I can attest that it does not work so well with Workstation and USB or IDE.

I did get it to work with a SCSI configuration, but that was on a older dual xeon workstation.

I want to retire that puppy as it generates allot of heat and uses too much electricity for what I'm doing with it.

 

I built an itx machine using laptop processors to get by the issue of heat and power.

 

This is one of my main reasons to rebuild my unRAID Server.

I.E. Consolidate machines into an ESXi guest for ripping and transcoding media.

 

I just have way too many machines in my computer room.

[prostuff1]

Curious if anyone has ever tried this.

 

1.  Ripping DVD's on a guest via XP or Windows (I use anydvd).

Yes, I use AnyDVD and MakeMKV

 

2.  Ripping CD's on a windows guest to unRAID. (I use EAC).

 

I.E. Consolidate machines into an ESXi guest for ripping and transcoding media.

That is one of the reason I built my ESXi server.  I have an XP Guest set up on mine.  The server is in a 4220 with the bottom 4 rows attached to 2 SASLP cards.  Those 2 SASLP cards are passed through for use in unRAID.  The Motherboard ports are for my SSD Datastore, and the low profile BluRay drive.  I have the BlueRay drive set up in the XP VM and can use Any DVD or MakeMKV to rip my DVD collection without issue.  I would assume that I can do the same with a CD.

 

 

My ESXi build is pretty much done and where I want it.  The only thing I am thinking of doing is adding an ESATA card to the box.

[WeeboTech]

Curious if anyone has ever tried this.

 

1.  Ripping DVD's on a guest via XP or Windows (I use anydvd).

Yes, I use AnyDVD and MakeMKV

 

2.  Ripping CD's on a windows guest to unRAID. (I use EAC).

 

I.E. Consolidate machines into an ESXi guest for ripping and transcoding media.

That is one of the reason I built my ESXi server.  I have an XP Guest set up on mine.  The server is in a 4220 with the bottom 4 rows attached to 2 SASLP cards.  Those 2 SASLP cards are passed through for use in unRAID.  The Motherboard ports are for my SSD Datastore, and the low profile BluRay drive.  I have the BlueRay drive set up in the XP VM and can use Any DVD or MakeMKV to rip my DVD collection without issue.  I would assume that I can do the same with a CD.

 

 

My ESXi build is pretty much done and where I want it.  The only thing I am thinking of doing is adding an ESATA card to the box.

 

Have you ever tried EAC to rip audio tracks?

Is the blueray drive a SATA drive?

[prostuff1]

Have you ever tried EAC to rip audio tracks?

Nope, I have not

 

Is the blueray drive a SATA drive?

Yes, it is a laptop style SATA BluRay drive.  It was in an external enclosure USB enclosure and I pulled it out to put it in the slim-line optical bay of the 4220.

[Johnm]

 

From what I read, if you write 20GB a day, it will take 5 years before an SSD is unusable.

Quite frankly from what I'm reading in reviews, I think realisticly it is about 2 years.

 

One of my concerns is how do you back up the data store drive?

I'm unfamiliar with ESXi. I use workstation now and I've used GSX or vmware server.

 

I tried installing ESXi on a machine to realize that I would not no direct access to the drive if I wanted to copy files (at that time).

So how is it done in an automated fashion these days?

 

 

Also as far as laptop spinners, I would suggest looking at the momentus XT 500GB. One of the best laptop drives I've ever used.

 

 

I am doing an average of 200GB-250GB a day so lets see.. there are some tests where people are writing TB's a day and the drives are still going year+ later..

The cost is trivial for me vs the  lag.

 

for backups, one box is running GhettoVCB Backup and one is using the Free LAB version of Veeam.

 

Yes the momentus drives are nice... not sure how it would do as a datastore drive. but for a laptop, macmini or htpc i would use it.

 

I also rip in a guest. I am using MakeMKV and dvddecrypter(?) in combo with handbrake.

I use an internal SATA BR Drive that I have sitting outside my norco.

 

I'm using a special 2port esata bracket that has an external esata to sata cable with power. this allows the drive to sit outside the box with the internal plugs.

I got the bracket/wire combo with a gigabyte motherboard. I installed it in my ESXi box so i can plug drives in.

For now one port goes to the motherboard and i think later i will put the other port on a 2 port Sata card and pass it through to my WHS2011 and remove the RMD from the primary drive and use passthough

[WeeboTech]

Yes the momentus drives are nice... not sure how it would do as a datastore drive. but for a laptop, macmini or htpc i would use it.

 

I'm sure it would do as good if not better then a conventional spinner.

From all the benchmarks I've seen they rate mid point between conventional spinner and SSD.

 

Although nothing beats ssd, I see the hybrid technology as a major step.

There are lots of cool products out these days for it.

Thanks for the info, I'm getting closer to make a decision.

I may part out and sell off my other unraid servers and migrate to a new ESXi capable setup.

[cpshoemake]

Great! I've never heard of turnkey before, what a great resource. I would love to save my DB from unraid, and I do have the physical database files. Any tips on how I can import them into a new database on this appliance?

 

There is an import tab in the phpMyAdmin web interface. It's pretty self explanatory from there.

[Johnm]

OK,

 

As of this morning, I am officially having issues with my virtual unRAID inside Atlas.

First off, I am sure it is my own fault in messing with the server and not unraids fault...

 

To start off, I have been running beta 12 (not 12a) pretty much the entire time this box has been up.

I had briefly tested 13 and 14.

 

I am sure it is unrelated to my current problem, but ill mention it here.

I jumped on beta 14 5 min after Tom posted it  see if the LSI issue was solved.

It was NOT.

After going to beta 14 and rolling back to beta 12. i had some issues with drives.

when i downgraded from 14. i had 3 redballs caused by beta14's issue with LSI controllers.

This was beta 14's fault. avoid it if you have an LSI card

 

the way i down graded was to wipe my thumbdrive, copy the backup of the last know working config (right before i upgraded) and then redo the sys  bat.

 

I knew that test might be an issue beta by testing a production server.

While fixing the the redballs, I realized one drive (drive 4 i believe?) was read only and some of the data was possibly corrupted.

 

I ran the "reiserfsck --fix-fixable" and it thought it fixed it all up.

 

some of the data was still slightly mangled (12 or so BluRay rips reported as not the same binary file as the back up. they might have played fine).

Not a big deal since I have a back up unRAId that was current (and the original blurays if it came to that).

I just restored those files.

 

now while some of you might say "unraid lost data?" look at it this way, unRAID saved data. I was beta testing and stressing the beta. 3 red balls and it still switched the corrupted drive to read only and prevented any further data loss.

a half dozen corrupted file out of thousands of files? ill take that. we all know Windows you be quite happy scrambling up the rest of the drive and eating all the data.

 

 

Ok, so it was up and running just peachy for 2 weeks.  Now, 3 days in a row I have woken up to a different drive offline (redballed). It looks like it is a drive didn't wake up from sleep during the mover script..

"reiserfsck" found no errors and the drives have rebuilt just fine (about 7-8 hours per rebuild for those interested). No errors or data loss.

 

Right now i am humoring myself and running "reiserfsck" again on drive 6 (today's failure)

 

When I get home later, I'll most likely unrack the box and make some changes...

This box does have 2-M1015's and 1-MV8 again (I borrowed an MV8 for raid0 testing).

I know the beta12's have issues with the MV8's. but there was no kernel panic in the syslog.

I also know MV8 has no drives on it all (it might not even be wired to the backplane?).

Looking at the syslog, disk 6 just did not want to wake up in time to be written to. (someone please verify that)

The smart report is clean for this drive. no issues. the backplanes have both  power headers plugged in

the port is different each day so I'll rule out bad backplane.

 

after months of no issues. this sort of hit me like a hammer this week.

I'll admit i have monkeyed with the unraid on this box a ton and it most likely is my fault.

The only changes I think I have made in the last 2 weeks is to add 2 more 3TB hitachis when i did the downgrade.

1 or both of these drives were already in the server as "warm stand by" so i could swap them in remotely if a drive failed while i was out of town.

(there is a nice feature suggestion.. warm stand by support, auto rebuild of a bad drive)

I have no warm stand by drive plugged in right now.

 

 

Current unraid build:

Parity:

1x 3TB Hitachi LP

Data:

7x 3TB Hitachi LP

Cache

1x Mushkin 120GB SSD

Cold Standby Drives: (precleared in "unseated drive trays" ready to be added)

5x 3TB Hitachi LP

 

 

I am going to scandisk the flash drive remotely from a windows guest once the rebuild finishes.

Try irqpoll

If that all checks out as fine, I'll roll back to beta11 with little to no add ons.

I'll see if there is any change at all over the next week.

 

So much for my perfect little world..

 

Todays syslog is attached.

I am open to suggestions.

 

syslog-2011-12-08.zip

[Johnm]

A quick update..

 

I have not been home yet. but it is a hardware issue.

I have even rebooted unraid and it is refusing to rebuild the drive.

Every time I try and rebuild it. i am getting write errors and redballing again.

 

this will have to wait..

 

EDIT:

Home, It looks like drives 6 and 7 might not have been seated 100%?

 

it would not rebuild until I powered off the entire server and reseated all the drives in the backplanes...

those would have been the last 2 drives I installed..

 

I have mentioned before that the Hitachi drives do not seat perfectly in the norcos. they need an extra "shove" after the latch is locked...

 

I'm going to cross my fingers and hope it was this simple of a problem.

[delicatepc]

Hi Johnm,

 

Building an Atlas like clone.

 

Here is what we got ordered:

 

Case: Norco 4224 - 24 hotswap bays - $400

Power Supply: SeaSonic X-850 (SS-850KM Active PFC F3) 850W 80 PLUS GOLD Modular - $211

 

Mobo: SuperMicro MBD-X9SCM-F-O Intel 1155 C204 MicroATX Intel - IPMI/iKVM+ 2 Gig NICs - $200

CPU: Intel Xeon E3-1270 Sandy Bridge 3.4GHz 4 x 256KB L2 Cache 8MB L3 Cache LGA 1155 80W Quad-Core - $340

RAM: Kingston 16GB (2 of 2x4GB kits) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 (PC3 10600) - $170

OS SSD: Intel 311 Series Larsen Creek SLC SSD 20GB 2.5" SATA II - $115

 

RAID Controller Card: LSI Internal SATA/SAS 9211-8i 6Gb/s PCI-Express 2.0 - $240

Expander Card: Intel RAID Twenty-four port RES2SV240 - $280

Extra NIC: Intel Gigabit NIC - $28

 

Accessory - 1to7 Molex Connector: NORCO C-P1T7 4 Pin Molex 1 Male to 7 Female Power Extension Splitter Cable - $8

Accessory - Norco Rails: RL-26 Rails - $40

 

Server Rack: iStarUSA WN228 22U 800mm Depth Rack-mount Server Cabinet - $923

 

Total (too much): $2032 (not counting 1K toward rack or shipping). This is all part of a 6K order for a rack mounted experience.

 

My hope is to virtualize a NAS here, possibly using unRAID (but may go to OpenMediaVault) as well as several other services. ESxi is on the table as well as looking at ProxMox 2.0 beta.

 

Read about some of your troubles recently with unRaid. What I would love to see out of unRaid is a VM offering - something I can download and run in a VM (ESxi or KVM/QEMU).

 

~

dpc

[Johnm]

well first off, my server has not crashed since re-seating all the drives... so far.

 

@delicatepc

 

It looks pretty solid.

Few things that i noticed...

1. Whoa beefy CPU! you might not ever use all that in a home setting combined  with only 16 gig of ram.. but sweet.

 

2. if you have not ordered the ram yet, Kingston released the part # for the 8gig sticks. you might want to try to get that.

 

3. I am not sure if anyone here has used a true 9211 in unRAID yet. it should work just fine if you can flash it with IT firmware.

 

4. that SSD is pretty small. I am  running my SSDs for data stores. it would make a really good ESXi 5.0 cache drive. but I dont think that is what you had in mind?

otherwise you would need  another or larger drive for the datastores. I would run the OS off a flash drive. much cheaper and works fine.

 

5. expander.. nice touch.. I'd love to know your exact throughput. I have mine coming in January. I already sourced one from a server pull at a client site that's going down on Jan 1..

 

6. extra NIC, not a bad Idea. the second  NIC on this mobo now works in ESxi 5 with a minor hack.

 

7. The Norco 7 plug POS.. are really crap. it is more for fans then drives. they kind of scare me how the cheap they are. Yet, i have 2 in each of my boxs only because i got them free (I did pay for the first one).. i would recommend something like this http://www.svc.com/fc454-34.html and maybe this? http://www.svc.com/fc44-12bks.html . It might not be a bad idea to just go with two 1-in-3's http://www.svc.com/fc434-22.html. I have not used those yet, But I will be ordering a set next time i have my server apart.

 

8. Norco rails.. they make 2 versions with the same part number. one is ball bearing and one is friction. you do not want the friction ones. Norco said they were going to have some of the ball bearing versions available again only if you call them directly.

 

9. the server rack... see number #8. the rails might not work as expected.. otherwise.. nice.

 

It sounds like you will have your hands full and have some fun...

[AaronD]

 

2. if you have not ordered the ram yet, Kingston released the part # for the 8gig sticks. you might want to try to get that.

 

 

Damn, two weeks too late. What's the part number, and have you seen them out yet?

[Johnm]

 

2. if you have not ordered the ram yet, Kingston released the part # for the 8gig sticks. you might want to try to get that.

 

 

Damn, two weeks too late. What's the part number, and have you seen them out yet?

 

No I have not actually seen them in the wild. they are either just ariving or in pre-production.

A few websites list them as "special order". only one site has 9 of them in stock?

Last week they only had the 2pack kit number. It looks like this week they added the single stick part number.

 

I know we will be getting a ton of them for work after the new year. our budget is shot for this year.

 

Single: KVR1333D3E9S/8G 8GB 1333MHz DDR3 ECC CL9 DIMM

2 Pack: KVR1333D3E9SK2/16G 16GB 1333MHz DDR3 ECC CL9 DIMM (Kit of 2)

 

 

 

[SCSI]

I will start to virtualize my unraid to a new Esxi server next week. Can I use the current flash drive with Esxi already installed on it on the new esxi server with a different hardware?

 

Basically I will be moving all VMs from my old esxi server to a new server.

[znelbok]

 

6. extra NIC, not a bad Idea. the second  NIC on this mobo now works in ESxi 5 with a minor hack.

 

 

Can you elaborate on this more.  I going to need that extra NIC port for another VM I am going to add in.

 

Finally got my system booting after having to wait for ram and CPU.

 

I think I have the same Norco problem that a lot of others have with the backplane not working.  Can;t get ESXi to see any of the drives on the first two planes.  Plugged the drive in directly and it shows up.  Need to do a few more checks, but its not looking good so far.

[Johnm]

I will start to virtualize my unraid to a new Esxi server next week. Can I use the current flash drive with Esxi already installed on it on the new esxi server with a different hardware?

 

Basically I will be moving all VMs from my old esxi server to a new server.

 

Interesting question. I honestly do not know..

Since there are two versions embedded and installable. the embedded (usually an SD card) is server model specific.

 

The installable, is.. well.. installed.  I am not sure if it copies everything or just what it needs or if the config is hardware specific.

 

We usually just reinstall when we change boxes since it only takes a few min to re-import all the guests on the new build.

 

 

6. extra NIC, not a bad Idea. the second  NIC on this mobo now works in ESxi 5 with a minor hack.

 

 

Can you elaborate on this more.  I going to need that extra NIC port for another VM I am going to add in.

 

Finally got my system booting after having to wait for ram and CPU.

 

I think I have the same Norco problem that a lot of others have with the backplane not working.  Can;t get ESXi to see any of the drives on the first two planes.  Plugged the drive in directly and it shows up.  Need to do a few more checks, but its not looking good so far.

not cool, do you know what version backplanes you have?

 

For the 82579LM NIC, see the end of page 4 http://hardforum.com/showthread.php?t=1607992&page=4

There are 2 ways to get it up and running. Injecting it into the ISO before you start or adding it after as an update to an already built box.

It is still beta but reported working.

 

 

[znelbok]

The link had good info and I now have both adapters installed.

 

In summary, for a running system log in via SSH and run

esxcli software acceptance set --level=CommunitySupported

esxcli software vib install -v http://files.v-front.de/net-e1001e-1.0.0.x86_64.vib

(ref http://hardforum.com/showpost.php?p=1038028779&postcount=95)

 

Where is the version identification on these backplanes, only number I can see is 94V-0

 

Its only a month old and I had to wait a month for it to be manufacturered so hopefully the latest version

 

Does the SM SAS2-LP add on card ( the 6GB card) work with ESXi5, they seem to be missing and ESXi is not seeing any of the drives attached yet they are showing at bootup (different backplane to the one that appears to be not working).

[brian89gp]

I will start to virtualize my unraid to a new Esxi server next week. Can I use the current flash drive with Esxi already installed on it on the new esxi server with a different hardware?

 

Basically I will be moving all VMs from my old esxi server to a new server.

 

Interesting question. I honestly do not know..

Since there are two versions embedded and installable. the embedded (usually an SD card) is server model specific.

 

The installable, is.. well.. installed.  I am not sure if it copies everything or just what it needs or if the config is hardware specific.

 

We usually just reinstall when we change boxes since it only takes a few min to re-import all the guests on the new build.

 

 

It will work just fine.  ESXi installs all drivers, changing hardware has never been a problem for me as long as you stay on the HCL radar.

 

The network config though binds itself to NIC's by MAC address.  When you boot up on a different server you will have to go in through the console and enable the other (new) NIC's.

[Johnm]

It will work just fine.  ESXi installs all drivers, changing hardware has never been a problem for me as long as you stay on the HCL radar.

 

The network config though binds itself to NIC's by MAC address.  When you boot up on a different server you will have to go in through the console and enable the other (new) NIC's.

Thanks, That was What I had thought. But I have heard in the past that was not true.

I have only moved them to like hard ware.

 

I also assumed the embeded are server specific due to the SLIC's in the BIOS and not so much drivers.

 

 

[savestheday]

 

2. if you have not ordered the ram yet, Kingston released the part # for the 8gig sticks. you might want to try to get that.

 

 

Damn, two weeks too late. What's the part number, and have you seen them out yet?

 

No I have not actually seen them in the wild. they are either just ariving or in pre-production.

A few websites list them as "special order". only one site has 9 of them in stock?

Last week they only had the 2pack kit number. It looks like this week they added the single stick part number.

 

I know we will be getting a ton of them for work after the new year. our budget is shot for this year.

 

Single: KVR1333D3E9S/8G 8GB 1333MHz DDR3 ECC CL9 DIMM

2 Pack: KVR1333D3E9SK2/16G 16GB 1333MHz DDR3 ECC CL9 DIMM (Kit of 2)

 

 

 

 

John,

 

These are not the same as the single sticks, right?

 

http://www.thenerds.net/KINGSTON_MEMORY.Kingston_ValueRAM_8GB_DDR3_SDRAM_Memory_Module.KVR1333D3E9SK28G.html?affid=8

 

If I could get 16GB for under $200 I'd take it in a heartbeat.  Would bring me up to 24GB which I think is pretty usable for me.

[Johnm]

 

2. if you have not ordered the ram yet, Kingston released the part # for the 8gig sticks. you might want to try to get that.

 

 

Damn, two weeks too late. What's the part number, and have you seen them out yet?

 

No I have not actually seen them in the wild. they are either just ariving or in pre-production.

A few websites list them as "special order". only one site has 9 of them in stock?

Last week they only had the 2pack kit number. It looks like this week they added the single stick part number.

 

I know we will be getting a ton of them for work after the new year. our budget is shot for this year.

 

Single: KVR1333D3E9S/8G 8GB 1333MHz DDR3 ECC CL9 DIMM

2 Pack: KVR1333D3E9SK2/16G 16GB 1333MHz DDR3 ECC CL9 DIMM (Kit of 2)

 

 

 

 

John,

 

These are not the same as the single sticks, right?

 

http://www.thenerds.net/KINGSTON_MEMORY.Kingston_ValueRAM_8GB_DDR3_SDRAM_Memory_Module.KVR1333D3E9SK28G.html?affid=8

 

If I could get 16GB for under $200 I'd take it in a heartbeat.  Would bring me up to 24GB which I think is pretty usable for me.

Good question. especially since the 2 pack was $350 a week ago.

The part # is correct. the description is wrong. it is not SDRAM.

If you click the compare price. CDW and Wiredzone have correct description and similar price.

 

that is about the same price as 2 4GB sticks. It would be almost silly to buy 4 GB sticks anymore at that price...

 

It is worth testing it. as far as mixing with existing 4 GB sticks. SM boards can be touchy with mixed sizes on some boards. especially the Dual CPU boards.

I have a feeling this board might be OK. I have seen people post they have 12GB on the same board so they must have mixed ram

 

If you get them, please post your results. otherwise I'll order some after the holidays for myself. I want to 32 on my ESXi box and i have other X9SCM boards I can bump my old 4 gig sticks to..

[savestheday]

think I might pull the trigger on this when it becomes available:

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16820139979&Tpk=KVR1333D3E9SK2%2f16G

 

Just figured out that even though only one DIMM is pictured on that thenerds.net page, it's actually the 2x4 set I already have.

 

 

[c3]

 

 

No I have not actually seen them in the wild. they are either just ariving or in pre-production.

A few websites list them as "special order". only one site has 9 of them in stock?

Last week they only had the 2pack kit number. It looks like this week they added the single stick part number.

 

I know we will be getting a ton of them for work after the new year. our budget is shot for this year.

 

Single: KVR1333D3E9S/8G 8GB 1333MHz DDR3 ECC CL9 DIMM

2 Pack: KVR1333D3E9SK2/16G 16GB 1333MHz DDR3 ECC CL9 DIMM (Kit of 2)

 

 

 

 

John,

 

These are not the same as the single sticks, right?

 

http://www.thenerds.net/KINGSTON_MEMORY.Kingston_ValueRAM_8GB_DDR3_SDRAM_Memory_Module.KVR1333D3E9SK28G.html?affid=8

 

If I could get 16GB for under $200 I'd take it in a heartbeat.  Would bring me up to 24GB which I think is pretty usable for me.

 

TheNerds listing is for the (2)x4GB memory, read the part number.

K2=kit of 2

8G=8G total

 

The real 8G sticks are already available for less than Newegg's place holder.