Jump to content
kevinsyn

Blackhole 2.0 - 60TB v.Now with 100% more ESXi

62 posts in this topic Last Reply

Recommended Posts

Very nice! This case looks great with those cages. Did you take the whole case apart to remove those tabs?

Share this post


Link to post

Just the front cover which is easily removable and of course the side panels.

 

It did feel weird cutting up a case that cost almost $400 just minutes after getting it home...

Share this post


Link to post

Just the front cover which is easily removable and of course the side panels.

 

It did feel weird cutting up a case that cost almost $400 just minutes after getting it home...

 

I can understand that. Do you live in the USA, Canada or Australia. Is that case really close to $400. Amazon Germany now asks 260 Euro with free delivery.

Share this post


Link to post

I live in Australia, I think I paid $377. Because we are so far away from everything, importing something that size would be too costly. No expense was spared with this build, money wasn't really an issue.

 

Anyway I don't want to hi-jack the thread, good to see another A77 owner

Share this post


Link to post

I found solution to mounting SSDs in the Lian Li PC-A77FB case.  With a microATX motherboard in this case, there are 4 unused PCI slots towards the bottom of the case.  The PCI slot covers that come with the case have diagonal slots that line up perfectly with the side mounting holes on the SSD.  I mounted the SSD to the PCI slot cover and installed it in the case with the power and SATA connectors facing upward/outward.  (See attached pictures.)

 

I must admit i'm impressed with the SSD mount. That is an awesome idea. I dont have an SSD in there but that definitely works. is it stable since its only mounted on one side?

 

 

 

 

What speakers are in that picture in the OP? Can you share some info on your AV-setup?

 

Sure i'll post what i can.

 

 

Speakers:

KEF Q700 Floorstanding Speakers

KEF Q200c Center Speakers

KEF Q300 Rears

KEF Q400 Subs

Onkyo 809 Receiver

 

Unraid server serves as storage and download software (sab, sickbeard, couchpotato, etc). Media server is bitstreaming full HD (DTS-MA/DD-TrueHD) to receiver.

 

Not sure what else to post lol. Lemme know if you wanna know any more details.

 

 

 

 

 

I live in Australia, I think I paid $377. Because we are so far away from everything, importing something that size would be too costly. No expense was spared with this build, money wasn't really an issue.

 

Anyway I don't want to hi-jack the thread, good to see another A77 owner

 

 

I've built dozens of computers in my day to be honest the A77 was one of the nicest to work with. Highly recommend the case in every way and would love to see more people use it!

 

 

 

Share this post


Link to post

I found solution to mounting SSDs in the Lian Li PC-A77FB case.  With a microATX motherboard in this case, there are 4 unused PCI slots towards the bottom of the case.  The PCI slot covers that come with the case have diagonal slots that line up perfectly with the side mounting holes on the SSD.  I mounted the SSD to the PCI slot cover and installed it in the case with the power and SATA connectors facing upward/outward.  (See attached pictures.)

 

I must admit i'm impressed with the SSD mount. That is an awesome idea. I dont have an SSD in there but that definitely works. is it stable since its only mounted on one side?

 

Yes, since the SSDs are very light, mounting with screws on one side (via this method/case) yields stable attachment.  I have a Plextor model M3 Pro Series PX-256M3P 2.5" 256GB SATA III MLC and specs indicate it weighs 57 grams.

Share this post


Link to post

Very nice living room setting.

 

I see you are missing the WII :)

 

Never was a fan of the Wii... to be honest I dont even use the PS3 or Xbox much nowadays :( I just watch movies and TV Shows all day.

 

 

Looks nice however I would highly suggest putting your USB stick inside your machine opposed to the front of your machine. There are several adapters you can use to plug it in. Once kick or somebody wanting to annoy you by pulling it out could be a real pain in the butt later. ;)

 

The USB Stick that is protruding is the mouse/keyboard on the HTPC. I only leave them in front because I move around the keyboard and mouse  a lot around the house so its easier that way. If you are refering to the USB drive for the UnraidOS, that is inside the case directly attached to the motherboard so i'm not to worried about accidentally kicking it.

 

 

I have a similar build with the Lian-Li A77 and Super Micro cages. I replaced the Super Micro cage fans with Noctua NF-B9.

 

It is a nice setup and quiet enough to not intrude.

 

:-)

 

Alex

 

Thanks :) The A77 was way overkill and unnecessary but it looks beautiful and with the quiet fans my HTPC ends up being louder so im pretty satisfied thus far.

 

 

 

 

- I went a bit creative and attached the cache drive and the parity drive on a separate rack on top of the motherboard so it is not taking up any of the cages (23 drive capacity)

 

 

Can you please post a picture(s) of the "separate rack" and/or elaborate a bit of detail on how you mounted this inside the case?  I am in-process of building a very similar setup, (same MB, case and SM cages) and would like to find a creative way to mount the SSD cache drive inside the case without taking up one of the 20 slots in the 4x Super Micro 5-in-3 cages.

 

 

Sure thing... it wasn't anything fancy to be honest... actually its quite crude and I'm not sure how you would do it with another case due to one of the unique features of this case.

 

The A77 case comes with a Power Supply strap that mounts on the powersupply without the use of screws. Removing the hard drive cages in the A77F I ended up with 3x 4 harddrive cages. I just attached one the hard drive cages with the strap with a layer of foam to dampen it.

 

aexjm8.jpg

Clearer picture of the attached cage. As pictured the Cache and the Parity drive in there with room for 1 more as its' a 4x3 cage

 

Can you please explain how you attached the cage to the PSU strap? Did you use Velcro? Or is the fan bracket "clamped" by the 3 PCI-slot clamps? Is the drive cage clear from the motherboard by a few mm?

 

 

Share this post


Link to post

Sorry for late reply haven't checked back in a while!

 

I used double sided tape to bind the bottom and the PSU strap. I didnt want to drill through it as I might want to reposition it later. The cage is fairly stable when you put it against the PCI Clamp Slots so it doesnt move around much. HOWEVER, the trick is to space the drive cage from the motherboard slightly. If the cage touches the motherboard pins, it will not boot up so i put a spacer in between the cage and the motherboard.

 

 

Blackhole has been updated with new parts and ESXi!

Share this post


Link to post

Is it okay having a bunch of hard drives right next to that tower of speakers?

 

No problems thus far... maybe if they were closer to the sub but the sub is on the other side of the room...

Share this post


Link to post

What brand / model of speakers are those?

 

Speakers:

KEF Q700 Floorstanding Speakers

KEF Q200c Center Speakers

KEF Q300 Rears

KEF Q400 Subs

Share this post


Link to post

Hi, I am thinking of going your route with the X9SCM, actually I am halfway done getting esxi up and running before finding out that my i3 2120t will not be able to support passthru... So I am thinking of your processor.

 

It has no internal graphics, does that mean you have had to start using a videocard in the system ?  Or will it work "headless"? Or does the x9scm take care of graphics without the cpu needing to ?

 

Thanks in advance !

Share this post


Link to post

Hi, I am thinking of going your route with the X9SCM, actually I am halfway done getting esxi up and running before finding out that my i3 2120t will not be able to support passthru... So I am thinking of your processor.

 

It has no internal graphics, does that mean you have had to start using a videocard in the system ?  Or will it work "headless"? Or does the x9scm take care of graphics without the cpu needing to ?

 

Thanks in advance !

The X9SCM has built in video and it is recommended to NOT install a CPU with built in Video because of that.  The built in video comes complements of the IPMI hardware.

 

 

Edit: Opps! as kevinsyn points out below the IPMI is on the -F version

Share this post


Link to post

Hi, I am thinking of going your route with the X9SCM, actually I am halfway done getting esxi up and running before finding out that my i3 2120t will not be able to support passthru... So I am thinking of your processor.

 

It has no internal graphics, does that mean you have had to start using a videocard in the system ?  Or will it work "headless"? Or does the x9scm take care of graphics without the cpu needing to ?

 

Thanks in advance !

 

Yes unfortunately I dont think that has passthrough as welll. I ran into the same problem.

 

I decided to upgrade to the Ivy Bridge series (v2) mostly because of availability at the time of purchase. There are reports of a number of user issues with Ivy Bridge but thus far everything has gone flawlessly and haven't had any issues. If you are upgrading to an Ivy Bridge CPU, remember you have to flash your BIOS to 2.00b in order to support the CPU.

 

Motherboard has built in videocard so don't worry about it. And use IPMI if you have the X9SCM-F

Share this post


Link to post

I currently have an Ivy Bridge cpu and my parity checks run at less than 1mb/sec.  I am about to try to swap out to Sandy Bridge and see if that does it. 

 

At the same time - what settings are you using in the bios?  I have 2.0b.

 

From the Atlas thread - there is a step that states:

 

BIOS Settings:

In the advanced tab: PCIe/PCI/PnP Configuration

Set PCI ROM Priority to "EFI Compatible ROM"

(NOTE: for Ver 2.0a BIOS this is replaced with "Disable OPROM for slots 7&6" set them to "Disabled")

 

In my ver 2.0b bios - is nothing that looks like this.  It would be nice to make a bios change and get my system running as it should.

 

Other notes:

 

I am running ESXi 5.1, and unraid 5.0rc11.

 

I have tried all kinds of different settings in the VM - from multiple cpu with lots of memory to single cpu with just 2gb of ram.  Nothing has solved my issue of slow parity checks.  Also everything runs super slow when all the drives need to be accessed. 

 

 

Share this post


Link to post

Are you using vt-d passthru ?

 

I am passing through the entire controller cards.

 

I just tried disabling the VT-D Option in the bios.  Upon rebooting the unraid VM did not load because passthrough was not supported.  I enabled it again and passthrough is working again.

 

But parity checks and rebuilds are unbearably slow.    I have pulled ESXi out of the boot sequence and running unraid on the machine as if it were the only thing installed.

 

Rebuilding a failed drive now at 77mb/sec.    Will be done in just over 300 minutes.  Then I will be back to the drawing board on how to get unraid to work fast like this in ESXi.

 

Share this post


Link to post

Are you using vt-d passthru ?

 

I am passing through the entire controller cards.

 

I just tried disabling the VT-D Option in the bios.  Upon rebooting the unraid VM did not load because passthrough was not supported.  I enabled it again and passthrough is working again.

 

But parity checks and rebuilds are unbearably slow.    I have pulled ESXi out of the boot sequence and running unraid on the machine as if it were the only thing installed.

 

Rebuilding a failed drive now at 77mb/sec.    Will be done in just over 300 minutes.  Then I will be back to the drawing board on how to get unraid to work fast like this in ESXi.

What do you have for the memory being passed through to the VM?  If it is more than 4GB reduce it to that or smaller.  I have mine at 3GB memory for unRAID VM's.

Share this post


Link to post

I have 4gb ram and just one CPU setup.

 

I am currently switching over from 2 MV8 and 2 adaptec 1430 to 2 m1015 and one mv8.  Will see if that helps. 

Share this post


Link to post

Just as an FYI.  The reason I chose 3GB for unRAID was that when I passed through more than that to a Windows 32 bit VM it could only see 3GB the rest was reserved address space.  On other PCs not virtualized I could get it to about 3.5GB recognized.  So I figured that anything OVER 3GB  in a unRAID VM will probably only be accessed with PAE memory swapping.  Also since I was virtualizing I put any plugins in a different VM.  I only installed things like unMenu, monthly parity check, drive overheat, etc... Basically if it wasn't designed to keep the unRAID server healthy I didn't install it in the unRAID VM.  Any other plugins I would install in another VM with a Windows OS or a full Linux OS and keep unRAID as more of an appliance.

Share this post


Link to post

Its my idea to run a VM specifically for unraid in its totally naked and bare form,  I would only run cache_dirs on it as I think that is as close as something can be to a core component.. No further plugins.

 

And then run another VM wherein I run Sickbeard, couchpotatoe, sabnzbd, transmission and airvideo.

 

The media stuff would need to "talk" to the unraid filesystem through a virtual network adaptor keeping all the traffic in the same physical box.

 

The unraid VM in that form would probalby be ok with 2 gigs.. The otherone can get 4 or 6 leaving me with enough to play around.

 

Basically I will be running with the same hardware, so no more costs,with the exception of the fact I needed to swap my cpu (but found a nice way to do that with little cost).

Share this post


Link to post

Are you using vt-d passthru ?

 

I am passing through the entire controller cards.

 

I just tried disabling the VT-D Option in the bios.  Upon rebooting the unraid VM did not load because passthrough was not supported.  I enabled it again and passthrough is working again.

 

But parity checks and rebuilds are unbearably slow.    I have pulled ESXi out of the boot sequence and running unraid on the machine as if it were the only thing installed.

 

Rebuilding a failed drive now at 77mb/sec.    Will be done in just over 300 minutes.  Then I will be back to the drawing board on how to get unraid to work fast like this in ESXi.

 

To be honest I'm a little stumped on this one. I don't think there is a way to easily diagnose the problem unless you were to take all the parts out and insert them in one at a time and find your root cause. If you take EXSi stick out and throw boot unraid directly do the parity checks run as normal?

 

Just as an FYI.  The reason I chose 3GB for unRAID was that when I passed through more than that to a Windows 32 bit VM it could only see 3GB the rest was reserved address space.  On other PCs not virtualized I could get it to about 3.5GB recognized.  So I figured that anything OVER 3GB  in a unRAID VM will probably only be accessed with PAE memory swapping.  Also since I was virtualizing I put any plugins in a different VM.  I only installed things like unMenu, monthly parity check, drive overheat, etc... Basically if it wasn't designed to keep the unRAID server healthy I didn't install it in the unRAID VM.  Any other plugins I would install in another VM with a Windows OS or a full Linux OS and keep unRAID as more of an appliance.

 

I had a peculiar problem with my Unraid. I originally had 8GB before upgrading to EXSi. However when upgrading i moved all the plugins such as sickbeard/couchpotato/sab out to their own respective VM so i lowered the unraid VM RAM to 4GB. However I started noticing kernel panics whenever I did preclears. Since then changed it back to 8GB since i have room to spare at the moment. Anyways it kinda seems like a waste at 8GB since most of it is using as swap. Anyone else have this problem?

Share this post


Link to post

Well.. I do not have my esxi system up yet so I might be talking trash, but:

 

Why don't you set up a specific unraid (using only the free license) that you only use to preclear disks ?  That wat you can bring that VM up whenever you need to preclear (which is allmost never) and the rest of the time it will only use a little bit of diskspace...

 

I am planning on doing that.. Only thing I would need to do is switch the sata connector from mainboard to one of the expansion cards that are vt-d'd to the unraid VM ..

 

That way no chance of issues with the primary function of unraid..

Share this post


Link to post

Just as an FYI.  The reason I chose 3GB for unRAID was that when I passed through more than that to a Windows 32 bit VM it could only see 3GB the rest was reserved address space.  On other PCs not virtualized I could get it to about 3.5GB recognized.  So I figured that anything OVER 3GB  in a unRAID VM will probably only be accessed with PAE memory swapping.  Also since I was virtualizing I put any plugins in a different VM.  I only installed things like unMenu, monthly parity check, drive overheat, etc... Basically if it wasn't designed to keep the unRAID server healthy I didn't install it in the unRAID VM.  Any other plugins I would install in another VM with a Windows OS or a full Linux OS and keep unRAID as more of an appliance.

 

I had a peculiar problem with my Unraid. I originally had 8GB before upgrading to EXSi. However when upgrading i moved all the plugins such as sickbeard/couchpotato/sab out to their own respective VM so i lowered the unraid VM RAM to 4GB. However I started noticing kernel panics whenever I did preclears. Since then changed it back to 8GB since i have room to spare at the moment. Anyways it kinda seems like a waste at 8GB since most of it is using as swap. Anyone else have this problem?

You aren't using a SAS expander by any chance are you?  The reason I ask is that I have problems preclearing drives on my VM's and someone else pointed out to me that I needed to upgrade the firmware on my SAS Expander (see my sig for model).  I haven't had a chance to update the firmware and test again.  I should also upgrade to a newer firmware on my M1015's since I'm still using P11. 

 

So you might try those options if they apply.  I don't remember getting any panic's but the preclear's were failing saying they could NOT preclear the drives and taking them to my preclear station that isn't on a SAS expander or virtualized would clear the drives just fine.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.