ATLAS My Virtualized unRAID server

[dikkiedirk]

Can I make a new empty vmdk in Vsphere Client.  Do I need a VM for that? I must be overseeing something. Please give some pointers.

[PCRx]

Can I make a new empty vmdk in Vsphere Client.  Do I need a VM for that? I must be overseeing something. Please give some pointers.

 

From within the Vsphere Client, yes. Pick a VM and edit it's settings. Press the Add button. Select Hard Drive. Setup the new hard drive with the settings you want.

 

When I did mine I added it as a second drive on a XP guest. Started the guest, right clicked "My Computer -> Manage -> Storage". Setup the partition and formatted.

Install unRaid following the instructions given a few posts back.

 

When you're done turn off the guest and create your unRaid guest.

Browse to your datastore and copy the vmdk you just installed unRaid on over to the unRaid datastore directory. Check the file size to be sure you are copying the right one!.

Edit the unRaid VM and remove the current boot drive. Then add a new drive only this time select 'Reuse previously configured virtual disk. Point it at the VM's vmdk you just created and copied over.

Finish making any other changes you want and boot the unRaid VM.

 

Remember you need the USB key to be named UNRAID and the vmdk volume something else.

[dikkiedirk]

I'm running in trouble here. I have the syslog attached of my virtualized unraid server.

 

I let SABNZBd run over night but I couldn't access it this morning.

 

Please have a look at the syslog. I think SAB or python ran out of memory starting 05:23 last night. Might have to give unraid some more memory. I gave it 2 GB to start with.

 

What is the experts opinion on this? My guess is that 4 GB might do it.

syslog-2012-12-21.zip

[BobPhoenix]

I do see OOM-Killer entries in your log.  I would up the memory some your plugins must be using up your memory.

[dikkiedirk]

I tried to up the memory to 4 GB but it wont let me. I get an error like in the attachment.

 

The motherboard holds 16 GB memory, and no other VMs are running.

 

What can be wrong?

[BobPhoenix]

That means you need to go to the memory reservations and up the memory used to 4GB.  This happens automatically when first setup but after that you have to change it yourself.  If you are not using passthrough it doesn't happen.  Basically make the slider and the orange arrow match see graphic below.

[dikkiedirk]

Thanks, that worked.

 

I noticed some strange lines on the consol screen on unraid in Vspere, see the attachment please.

 

Do you know what it is?

[BobPhoenix]

Thanks, that worked.

 

I noticed some strange lines on the consol screen on unraid in Vspere, see the attachment please.

 

Do you know what it is?

I believe but am not positive that it is related to unRAID trying to get temperature data from a simulated drive.  Based on your other thread that I responded to I would say this is on your parity and cache drives since they are not real drives but composed on your ARC1200 controller.

[dikkiedirk]

I don't see it in a telnet session, only in Vsperes unraid console, weird. As long as it doesn't disrupt anything it's ok by me.

[BetaQuasi]

I have the same issue with a RDM'ed cache drive on 5.0rc8a.  Doesn't occur on earlier rc's... but then I had other issues with 8a, so am back on rc3 for the moment (which works great.)

[dikkiedirk]

I just went to RC8a, so it might be related to that. What other issues did you have?

 

I gave my virtualized unraid 4 GB now, have been cleaning out packages and plugins, and hammering the server a bit overnight: Parity check, downloading with SAB, mover kicking in at 03:40. And all went well. Also updated samba to 3.6.8 per Frank1940's instructions.

 

So far everything seems pretty stable.

[BetaQuasi]

My other issues were with Samba, even once on 3.6.8.  Even switching to NFS didn't resolve them, so it may have been some other hardware issue.  Parity checks were also taking several hours longer, and running slower.

 

In any case, rather than troubleshooting any of that (I really don't have the time these days), I just went back to the previous version I was on and everything has been fine since. 

[Johnm]

I am still way behind on RC's.. i have not been home more then a few days now and then in the last few months.

I have not dared to upgraded it then leave town in case of issues.

[dikkiedirk]

I just installed a Windows 7 VM. What would the fastest interface from it to my unraid shares for doing some repairing and unpacking of downloaded files and some remuxing. My guess is that the virtual network is the only way to go, or are there alternatives? What would be the best setup for this?

[dikkiedirk]

Would this be the right memory for use on the X9SCM-F:

 

32GB Kit 4X 8GB Samsung PC3L-10600R DDR3 PC3 1333 ECC REG M393B1K70CH0-YH9 10600

 

[Johnm]

Would this be the right memory for use on the X9SCM-F:

 

32GB Kit 4X 8GB Samsung PC3L-10600R DDR3 PC3 1333 ECC REG M393B1K70CH0-YH9 10600

 

No, You need unregistered memory.

 

[dikkiedirk]

Would this be the right memory for use on the X9SCM-F:

 

32GB Kit 4X 8GB Samsung PC3L-10600R DDR3 PC3 1333 ECC REG M393B1K70CH0-YH9 10600

 

No, You need unregistered memory.

 

Thanks John,

 

What would be the right Samsung part number?

[Johnm]

Dont have a samsung part number. I don't recall seeing it anywhere I get memory from.

I am pretty sure it on the Supermicro memory list. ( I had checked mine)

 

This is the same stuff http://www.superbiiz.com/detail.php?name=W1333EB8GS re-branded as super talent. (for half the price I paid 8 months ago)

 

I know you have asked before and i think you have gotten the answer before.

just search this thread for "samsung" you might even get the part number i ordered

 

you can also zoom in  on the pic in this post http://lime-technology.com/forum/index.php?topic=14695.msg175152#msg175152

 

 

 

[Johnm]

I just noticed i do have an issue with Atlas. I have been getting error log entries the last few days.

 

Device t10.ATA_____Corsair_Performance_Pro__
_______________1214810001001454001D 
performance has deteriorated. I/O latency 
increased from average value of 1194 
microseconds to 53369 microseconds.
warning
12/9/2012 8:10:33 PM
Atlas.

 

That is my main datastore that never gets a break. it has been 24x7 pounded for almost 4 months.

It has a couple of mission critical 24 guests that are always generating High IO.

That includes databases and temp files for my usenet downloader. It is also my ESXi host  cache.

 

I noticed it only has about 8% free.

I am going power off all my guests. Then offload some guests. Free up more space then let it sit for an hour or two.

Reboot and see if it flushes itself.

 

While I'm at it, I'll pull it from the rack, open it up and give it semi annual dusting with compressed air.

 

Ugh... downtime...

 

Just a follow up to this Error:

 

I did just what said I would.

1. I migrated some test guests to my ZFS datastore from the SSD. Thus freeing up about 1/3 of the drive (33% free).

2. I powered down all guests and went out shopping for a few hours.

3. When I got home I bounced entire server. When it came back up, it was self healed. Back to full speed.

 

The auto garbage seems to work just fine when you give it a chance to do its job....

[eaglepigjim]

Hi Johnm,

 

Thanks for the fantastic thread, I am still going through it and trying to get as much information as I can, before I figure out what I want to buy. I have a couple of questions:

 

1) Would you still use the Intel E3-1240 Xeon?

2) I saw a conversation in this thread about Sandy vs Ivy Bridge. Which would you recommend getting?

3) Do you still recommend the SUPERMICRO MBD-X9SCM-F-O?

4) What are your thoughts on a dual cpu motherboard with slower CPUs?

 

Thanks

[dheg]

I’m thinking of implementing FreeNAS as a fileserver for the datastores. Main reason is redundancy. Right now my server has only one SDD as a datastore, so if it fails I’m sc***  !

 

However, and although my main purpose is redundancy, I’m also very happy with the speed of my setup. I’m using a 256MB Plextor M3 SDD connected to one of the SATA III ports of the X9SCM-F-O. Last time I checked reading speeds were in excess of 500MB/s  . So ideally I’d like to get raid protection without sacrificing performance (and even improve it if possible).

 

My intention is to dedicate a M1015 for the FreeNAS fileserver with only 4 drives (1 dedicated 1U case), so the M1015 will be under used. And here is where I start having questions:

 

• Since I’ll be using only 4 drives, can I plug the M1015 in one of the PCIE x4 ports without sacrificing performance?
  
• I know close to nothing about FreeNAS, but I believe it’s a stripped raid. What kind of writing/reading performance improvement should I expect?
  
• Are SDDs really worth it for FreeNAS, or should I go with spinners? If using SSDs, what kind of speed should I expect? What about spinners? Please feel free to recommend parts  !
  
• When looking around I’ve seen many, reasonably priced, 1U SATA II enclosures for 4 drives. However, when looking for SATA III, prices really go up, especially here. I’m Spanish, so apart from the spelling mistakes (sorry!), my options to shop around aren’t many. If I go with SSDs, going with SATA III is a no brainer, but if using spinners, could I take SATA II?
  
• Anything else I’m missing?
  

 

I know Johnm has a similar setup, but does anyone else have/is planning, something like this? I wouldn’t like to overload Johm with so many questions; he has already contributed so much to this thread  !

 

[Johnm]

Answers in RED

 

Hi Johnm,

 

Thanks for the fantastic thread, I am still going through it and trying to get as much information as I can, before I figure out what I want to buy. I have a couple of questions:

 

1) Would you still use the Intel E3-1240 Xeon?

It really depends on the price and your budget. I have not maxed out my CPU yet in normal use (I did max it out while testing transcoding. in general I tend to transcode on my I7-3770k hack) I have no regrets so far.

 

Spending more might have been a waste for me. for you, it depends on your needs. for straight storage servers. the 1230 is fine. For multiple guests running at once, the 1240 might be a little more bang. For heavy transcoding, for say a handbrake guest, the bigger then better (until you max out your disk IO)

 

2) I saw a conversation in this thread about Sandy vs Ivy Bridge. Which would you recommend getting?

If you're going with a Sandybridge board, stick to a Sandybridge CPU. If you get an ivybridge board, stick to an IVYBridge CPU. there are just to many quirks going cross generation. some fatal.

 

3) Do you still recommend the SUPERMICRO MBD-X9SCM-F-O?

I have 3 I love them. Is there an upgrade? yes, I have honestly not tested them yet. we are still buying the X9SCM-F's at work to keep uniform. I'll need to upgrade eventually once I got with my DAS config. I'll need more PCIe slots.

 

4) What are your thoughts on a dual cpu motherboard with slower CPUs?

You can. it will use more electricity and generate more heat. especially if you are buying older XEON's. Newer ones that can be run in dual CPU config cost more. Keep in mind that Dual cpu boards usually have unique memory configs. you might have to buy 4 sticks right away.

 

The real showstopper is that the Free version of ESXi has the 32GB memory cap. Usually you have lots of extra CPU and be totally out of ram. a second CPU only adds to the is issue.

 

Again it depends on what your needs are. if you building a transcoding (or SQL/Exchange) guests, the more CPU, the better. for data-storage systems and workstation guests.. more ram (usually)..

 

Thanks

You're welcome. it is still a learning process for me also. lots of help from people on this forum for where this build is now. I just like to share my results with you guys.

 

[Johnm]

my answers in RED

I’m thinking of implementing FreeNAS as a fileserver for the datastores. Main reason is redundancy. Right now my server has only one SDD as a datastore, so if it fails I’m sc***  !

 

However, and although my main purpose is redundancy, I’m also very happy with the speed of my setup. I’m using a 256MB Plextor M3 SDD connected to one of the SATA III ports of the X9SCM-F-O. Last time I checked reading speeds were in excess of 500MB/s  . So ideally I’d like to get raid protection without sacrificing performance (and even improve it if possible).

 

My intention is to dedicate a M1015 for the FreeNAS fileserver with only 4 drives (1 dedicated 1U case), so the M1015 will be under used. And here is where I start having questions:

 

• Since I’ll be using only 4 drives, can I plug the M1015 in one of the PCIE x4 ports without sacrificing performance?
  Yes. you will be fine
   
   
   
  
• I know close to nothing about FreeNAS, but I believe it’s a stripped raid. What kind of writing/reading performance improvement should I expect?
  You have a few ways to set it up. you can do what is equivalent to a raid5 (RAIDZ) or a raid10 (Striped Mirrored Vdev’s). the Striped Mirrored Vdev’s will net you more speed at the cost of 2drives for parity. look back a few pages, SEE LINK Im actually getting better writes then my SATA3 Performance Pro2 and slightly less performance for the reads.
   
   
  
• Are SDDs really worth it for FreeNAS, or should I go with spinners? If using SSDs, what kind of speed should I expect? What about spinners? Please feel free to recommend parts  !
  Spinners are fine for this. the reason i say that is for space. with 4x 2TB Spinner Green Drives,  you will get around 6TB at 450MB/s read/write in RAIDZ. I would expect more in the 500-600 MB/s range with the new Seagate 7200 RPM 1TB platter drives.
   
  yes with 4x 250MB SSD, You will get 750MB at maybe 1000MB/s read and 700MB/s Write (am totally guessing here, I could be wrong. [you might even maxout the PCIe bus first. although 4x should be ok to handle it]) At a ridiculous cost. 
   
  When you weigh the size of array vs performance VS the cost. the spinners look like more bang for the buck.
  Spinner = 6TB @ 450MB/s for $360 vs SSD = 750MB @ Say 800MB/s (ish) for $750.
   
  Don't forget, you can put your unRAID cache drive on this array...
   
   
  
• When looking around I’ve seen many, reasonably priced, 1U SATA II enclosures for 4 drives. However, when looking for SATA III, prices really go up, especially here. I’m Spanish, so apart from the spelling mistakes (sorry!), my options to shop around aren’t many. If I go with SSDs, going with SATA III is a no brainer, but if using spinners, could I take SATA II?
  Sata2 should be fine for spinners. for SSD, youll want SATA3
  I guess i need to see what the item is you're looking at.
   
  
  
• Anything else I’m missing?
  Yes, if you are making a this a 1U standalone server, you're gigabit network will maxout before your NAS array.
  You will feel the slowness of the gigabit.
  consider making the freenas a guest ON the ESXi server itself. passthough the M1015 to the guest and use that. by using the ESXi VLAN, you can then use the 10GB virtual nics and get full NAS speed.
   
  Also, unless you are going full standalone freeNAS with iSCSI target (or NFS). you will still need 1 datastore drive local on the server.
   
  

 

I know Johnm has a similar setup, but does anyone else have/is planning, something like this? I wouldn’t like to overload Johm with so many questions; he has already contributed so much to this thread  !

 

 

EDIT:

I looked for some real world benchmarks for SSD's in raid5

http://www.storagereview.com/intel_ssd_510_raid_review

 

Pretty sad honestly.. about a 20% performance gain. so my guesses would be about correct with M3's, Performance PRO's or any other max IOP drives like the sandisk extreme.

 

 

[SidebandSamurai]

What about Visualizing ESXi servers? Have a dual processor board maxed out in memory,  Install ESXi, which will allow 32Gig, then setup two hosts, use one virtual ESXi host, maxed to 32Gig dedicated to one processor, and then set up a second virtual ESXi sever dedicated to the other CPU, with 32Gig?

 

I don't know if that would truly work, probably not, but worth a try.

 

--Sideband Samurai

[BLKMGK]

Couple of comments... Free version of ESX only allows for ONE physical CPU as well as the 32gig limit - don't use two.

 

If you setup FreeNAS or NAS4Free (my current project FWIW) with an M1015 you will run into a boot issue in ESX. Loader.conf will need to have two lines added to it as follows:

 

hw.pci.enable_msi="0" # Driver Interrupts problem SAS2008

hw.pci.enable_msix="0" # Driver Interrupts problem SAS2008

 

In NAS4Free these are found at /cf/boot/loader.conf and on FreeNAS they are under just /boot I think.

 

I'm running a Sandy CPU and I'll admit I want an Ivy. That said, only my Windows guests seem to use very much CPU and this is why I'm trying to dump them! For SAB/Sick/Couch/blah blah I want to run an appliance VM on Linux and ALMOST have it working. Almost... I want to add Plex in Linux too before I'm all done. My transcoding is done on a quite highly clocked 920 i7 that will soon be swapped to a 3770K like John has hehe. I just grabbed dual 27inch monitors for it and am sourcing the rest. Anyway, transcoding I'd do on a separate desktop. Run Linux and FreeBSD whereever you can and you can load up a TON of VMs without taxing things too badly.

 

One question - if you run a NAS for the storage volumes how will you boot it? Chicken/egg issue? Or also keep around say an SSD to boot the NAS and then boot the rest? Seems to be getting a bit complex but I'll admit I'm pondering this myself