ATLAS My Virtualized unRAID server


Recommended Posts

jesseasi Not sure where you are at now..

 

Few things.. the MV8's do need the hack.. i think you need to hack each one if you have more then one.. At least I think I did when I had 3 in my system. without the hack, it does pink screen when you try and access a drive on an MV8.

 

Also i had a similar issue with the latest RC and MV8's in passthough. drive(s) were dropping out on my MV8's I never had time to see what was the cause as i was out of town and remoting into the system. i just rolled back to an earlier version that was stable for me. No problems since.

 

I removed my dedicated unraid nic and put the unRAID on the 10GB ESXi virtual adapter and vlan. I found that most of my data transfer in my case is all inside the esxi box and it is much faster for me this way. that is just something to think about. your needs may be different.

 

as you have already learned. doing the updates from the command line inside esxi is much faster. I should probably put that on my first page. it is buried deep in this thread somewhere.

 

in the future for others.. when you have lots of hardware to passthough, it might be best to add one item at a time and test to help troubleshoot  where the system may have issues.

 

 

 

Link to comment

Well right now everything runs.  I pulled drive #10 that red balled and I am currently copying the data off of it on my windows machine using a third party app that allows me to read the unraid drive. This is going to take a couple of days at the speed it is running. 

 

The issue I have is when unraid needs to access all the drives.  Like parity check or when I try to copy data out of the failed drive.  It runs at less than 1mb per second.  And if I let it run a long time I am afraid a drive may fail.  (That is when drive 10 failed)

 

I really just want to sort this out so I can rebuild my drive then add in two 3tb drives I just bought.

 

I have the mv8 hack done for both cards. 

 

I would be happy to buy another mv8 and remove my two adaptec cards or buy three of the IBM M1015 cards.  If they are more stable.

 

Any ideas? 

 

I do have to reboot two or three times to get all the drives to show up.  And the ones missing look to be on the mv8 card. 

 

If this works it will be a great solution for me.

 

I appreciate the help.

 

Also. You mention.

 

Rolling back. What is it you rolled back?  Esxi?  Unraid?  (I started with ESXi 5.1 - so I don't know if rolling back means I would have to start all over.  Would my MV's work in 5.0 if I started over?)

 

If running M1015's will work and from what I understand give me a little speed boost - I will be happy to swap all my controllers out for new ones.

Link to comment

Getting anxious for a solution.

 

Anyone with any ides to my issue of crazy slow parity checks? 

 

Would I be better off buying one more MV8 and running three of them?  Or should I get 3 IBM M1015? 

 

Something is not right.  Should I consider installing ESXi 5.0?

 

Desperate to figure this out.

Link to comment

Getting anxious for a solution.

 

Anyone with any ides to my issue of crazy slow parity checks? 

 

Would I be better off buying one more MV8 and running three of them?  Or should I get 3 IBM M1015? 

 

Something is not right.  Should I consider installing ESXi 5.0?

 

Desperate to figure this out.

Let me start by saying I am no expert.  I am about to run ESXi for the first time myself too.  I have been doing a considerable amount of reading on various forums though.  I wanted to share my thoughts, and hopefully someone can correct me if I am wrong. 

 

The MV8's once hacked should be fine.  M1015's would be better.

 

1430sa (I also have two of these) I read somewhere that someone was having difficulty with these and ESXi.  Went back to try and find the link, but can't for the life of me find that thread.  The issue I read about was regarding passthrough of the 1430sa.  I don't remember the details, just remember that I didn't want to try it myself and plan on buying IBM M1015's instead.

 

CPU - I also read about people having various oddities with ESXi and an Ivy Bridge Xeon, which it looks like you have.  It seems that Ivy Bridge just isn't as stable as Sandy Bridge on the server platform.

 

I would want to validate the hardware, but wouldn't risk using any of my production disks (data that I care about) in doing so, especially since you already had one red ball.  I would recommend trying to reduce the components you have in order to identify the problem. 

 

Here's things I would try if I were you.

1.  Disconnect all drives that have data on them that you care about.  Make sure you have a good inventory list first.

2.  Remove the 1430sa's and one of the MV8 controllers.

3.  Test with passing through one MV8 with 3 disks (new ones, or old ones you don't care about the data).

4.  Install unRAID on a new fresh USB stick and use that.

5.  Try a Sandy Bridge CPU if you have one readily available.

6.  Try a few different versions of unRAID. - I think this is what Johnm meant with regards to the latest RC and rolling back due to drives dropping out with passthrough of the MV8.

 

Change one thing, test and verify.  If you start running stable in some configuration, start adding other components back in one at a time, test and verify.  Hopefully this will help identify the root cause of your issue.

Link to comment

jesseasi - Read through this thread:

http://lime-technology.com/forum/index.php?topic=22327.0

 

RockDawg was getting 4MB/s parity on an Ivy and 75 MB/s after switching to Sandy.  Based on that, you may want to try a Sandy Bridge CPU first.  I suspect that will correct your slow parity issue. 

 

The drives dropping out is probably a different issue, and hopefully an earlier unRAID RC will help you like it did Johnm.

 

 

Link to comment

Sounds good.  So the plan is to swap out the CPU for a Sandy Bridge ( Don't know what I am going to do with this Ivy Bridge CPU...ARGH!) and Replace all my controllers with three M1015's. 

 

I will report back once this is done.  May take a week or so.

 

What BIOS version are you using for Ivy Bridge? I've upgraded to Ivy Bridge and have no problems with parity checks, preclears through VM. Maybe upgrade your BIOS before throwing away (not literally of course) the cpu? Just a thought.

 

Only problem I've had thus far has been kernel panic when preclearing but I believe I found the problem with allocating too little ram to the VM.

Link to comment

Sounds good.  So the plan is to swap out the CPU for a Sandy Bridge ( Don't know what I am going to do with this Ivy Bridge CPU...ARGH!) and Replace all my controllers with three M1015's. 

 

I will report back once this is done.  May take a week or so.

 

What BIOS version are you using for Ivy Bridge? I've upgraded to Ivy Bridge and have no problems with parity checks, preclears through VM. Maybe upgrade your BIOS before throwing away (not literally of course) the cpu? Just a thought.

 

Only problem I've had thus far has been kernel panic when preclearing but I believe I found the problem with allocating too little ram to the VM.

 

 

Running Bios 2.0b

Build Date 09/17/2012

 

I don't see any newer bios on supermicro's website.  Right now I am about 2,500 miles away from my server.  I wonder if there is a way to update the bios using IPMI. 

Link to comment

I have three M1015's that I ordered off eBay on the way.  I believe they will need to flash these to "IT" mode.  However in looking at all the posts on the site related to flashing the M1015's, I am left being more confused. 

 

Is there a post somewhere that is the definitive instructions on how to flash the M1015's and with what firmware? 

 

I know this has to be covered somewhere.

 

Here is a outside link that I have found that seems to cover what is needed.  http://www.servethehome.com/ibm-serveraid-m1015-part-4/

 

But in the forums here I have seen discussions about not being able to power down drives and other newer versions of the firmware being available.  Also is "P11" all that is needed - there seems to be another version called "P15".  And maybe you need to flash to P11 before going to P15?  My head is spinning.

 

I guess I am looking for the most up to date firmware to be used and instructions on how to do it.....

 

Thanks!

Link to comment

I used fairly generic instructions to flash mine and am on Rev14 of the firmware with no issues spinning drives down.

 

Flashing - it was a 2-step process and I did have to use DOS. It wasn't hard, I did it one card at a time and the servethehome thread looks accurate although they're using older firmware. 14 was latest when I did mine, not sure what 15 has to offer if anything.

 

I'm not convinced you need to go Sandy FWIW but if you must unload that CPU I might be interested depending upon which it is and cost, I'm on Sandy myself and am curious about running an Ivy. :)

Link to comment

I have three M1015's that I ordered off eBay on the way.  I believe they will need to flash these to "IT" mode.  However in looking at all the posts on the site related to flashing the M1015's, I am left being more confused. 

 

Is there a post somewhere that is the definitive instructions on how to flash the M1015's and with what firmware? 

 

I know this has to be covered somewhere.

 

Here is a outside link that I have found that seems to cover what is needed.  http://www.servethehome.com/ibm-serveraid-m1015-part-4/

 

But in the forums here I have seen discussions about not being able to power down drives and other newer versions of the firmware being available.  Also is "P11" all that is needed - there seems to be another version called "P15".  And maybe you need to flash to P11 before going to P15?  My head is spinning.

 

I guess I am looking for the most up to date firmware to be used and instructions on how to do it.....

 

Thanks!

 

I think this is what you're after but not sure about the latest version: http://lime-technology.com/forum/index.php?topic=12767.0

Link to comment

Personally, I used these instructions and files to flash - it's not immediately obvious, but the blue SAS2008 link contains the files etc. and the instructions are quite specific for the various ways of flashing this card.

 

http://forums.laptopvideo2go.com/topic/29059-sas2008-lsi92409211-firmware-files/

 

The archive contains P14 I believe.. while not the latest, I've never had an issue with it.

Link to comment

As a slight aside - and out of interest - does anyone know the consequences (if any!) of mixing and matching firmware revisions (i.e P14, P15 etc) across different cards in the same chassis?

 

The correct answer would be to ensure they're all the same - but given the pain of flashing if you have a mixture does it matter? Will they all still 'merge' at the BIOS level as a single manageable instance?

Link to comment

As a slight aside - and out of interest - does anyone know the consequences (if any!) of mixing and matching firmware revisions (i.e P14, P15 etc) across different cards in the same chassis?

 

The correct answer would be to ensure they're all the same - but given the pain of flashing if you have a mixture does it matter? Will they all still 'merge' at the BIOS level as a single manageable instance?

 

I have had a few different version at the same time with no ill effect.  It shouldt really matter though.

 

I turn the post off on the card so they don't show up in the BIOS, but I did have a Dell external SAS HBA and an flashed M1015 post together and they are running very different firmware.

Link to comment

Have only just got around to ordering a RES2SV240 and have found they are discontinued  and can't find any in my country (uk)

 

I have an m1015 just wondering if anyone knows of an alternate or if I should try to find one in the US ?

 

Thanks

I bought it in amazon.es a few month ago

 

Sent from my GT-I9100 using Tapatalk 2

 

 

Link to comment
Raid Card Bios Settings

As the PC starts to post, watch for the Raid card BIOS.

When it starts detecting drives on the raid card, start pushing "ctrl m" (For mv8 anyways)

Controller Tab:

Disable INT 13h

 

Optional

Under staggered spin up: set spin up groups to lower the hit on your power at boot.

Exit and save.

 

Does anyone know the comparable settings that must be set for an m1015/RES2SV240?

 

BIOS Settings:

In the advanced tab: PCIe/PCI/PnP Configuration

Set PCI ROM Priority to "EFI Compatible ROM"

(NOTE: for Ver 2.0a BIOS this is replaced with "Disable OPROM for slots 7&6" set them to "Disabled")

 

My board came with 2.0b. I found all the other settings in the BIOS but don't see this one anywhere. Anyone know what the setting is in 2.0b?

 

Thanks.

Link to comment

Raid Card Bios Settings

As the PC starts to post, watch for the Raid card BIOS.

When it starts detecting drives on the raid card, start pushing "ctrl m" (For mv8 anyways)

Controller Tab:

Disable INT 13h

 

Optional

Under staggered spin up: set spin up groups to lower the hit on your power at boot.

Exit and save.

 

Does anyone know the comparable settings that must be set for an m1015/RES2SV240?

 

BIOS Settings:

In the advanced tab: PCIe/PCI/PnP Configuration

Set PCI ROM Priority to "EFI Compatible ROM"

(NOTE: for Ver 2.0a BIOS this is replaced with "Disable OPROM for slots 7&6" set them to "Disabled")

 

My board came with 2.0b. I found all the other settings in the BIOS but don't see this one anywhere. Anyone know what the setting is in 2.0b?

 

Thanks.

 

Same boat as you.  When my board with 2.0b, I feel like I might have missed something in the bios.  I am running ivy bridge cpu and my parity checks run at less then 1mb per second.  I wish/hoped it was something in my bios that I was overlooking.  I am about to install a Sandy Bridge CPU and possibly swap out all my cards for M1015.   

 

From what I have gathered - you don't need to do anything for the M1015 other than make sure the bios has been programmed to IT mode. 

 

I will am about to try all this - so I will be reporting back soon.

Link to comment

Well here is a quick update.

 

I swapped out my Ivy Bridge  3.2 CPU for a Sandy Bridge 3.4.  I took out the 8GB of ram and replaced it with 32GB. 

 

Started my parity check......and I am getting about 300kb/sec.  Estimated to take 103,000 Minutes to finish.  That is about 3.5 Months.  LOL.

 

Well that is just not going to work. 

 

Next is to swap out all my controller cards for the M1015's.  Time to start working on how to flash them.

 

 

Update - Appears I don't have a motherboard here that will flash the M1015's.  tried Intel D975XBX2, Intel DG451D and Asus P5N7A-V.

 

Maybe I need to try to downgrade EXSi or unraid.  Anyone have any ideas?

Link to comment

Well here is a quick update.

 

I swapped out my Ivy Bridge  3.2 CPU for a Sandy Bridge 3.4.  I took out the 8GB of ram and replaced it with 32GB. 

 

Started my parity check......and I am getting about 300kb/sec.  Estimated to take 103,000 Minutes to finish.  That is about 3.5 Months.  LOL.

 

Well that is just not going to work. 

 

Next is to swap out all my controller cards for the M1015's.  Time to start working on how to flash them.

 

 

Update - Appears I don't have a motherboard here that will flash the M1015's.  tried Intel D975XBX2, Intel DG451D and Asus P5N7A-V.

 

Maybe I need to try to downgrade EXSi or unraid.  Anyone have any ideas?

 

I have the X9SCM-F with a Xeon 1240V2 (Ivy Bridge) and it is running fine so far, though all I have been doing is preclearing. I was able to flash my M1015 on my friends ASRock Z77 board after getting the PAL error by using the UEFI shell. Maybe it would have worked on the X9SCM too? I thought that once you got the PAL error it was not possible on that board, but it worked. I followed the instruction from these two pages:

 

http://forums.laptopvideo2go.com/topic/29059-sas2008-lsi92409211-firmware-files/

http://lime-technology.com/forum/index.php?topic=25891.msg225711#msg225711

 

These two commands worked from DOS:

megarec -writesbr 0 sbrempty.bin
megarec -cleanflash 0

then I got the PAL error after rebooting and trying sas2flsh, so I followed newbie_dude's instructions at the 2nd link above and it worked. Good luck. Let me know if you get stuck anywhere.

Link to comment

Success.....

 

After who knows how many hours into this project. 

 

I was only able to flash 2 of the 3 M1015's that I got in - but I removed the two Adaptec 1430's and one MV8.  I am currently running two M1015's and one MV8.  For some reason this combination works. 

 

Rebuilding in my parity drive right now at 78MB/Sec. 

 

So far everything is looking good. 

 

I do want to find a way to flash my "bricked" M1015 and get then in my system.

 

Feels good to see it all working. 

 

Thank you everyone for all your help.  Some quick notes for anyone else that tries this.

 

Running

ESXi 5.1 & UnRaid 5.0RC11.

 

I had started with Ivy Bridge CPU.  In the tasks of troubleshooting - I switched to a Sandy Bridge.  In the end I don't think it matters.

Unraid does not need any more than 1 CPU and 4GB of ram.  Can probably run just as fast with just 2GB ram.  In the VM you need to reserve the full amount of memory to the unraid. 

The Supermicro Board can flash the M1015 using the UEFI shell.  Took me a long time to figure it out - but you can follow the steps here - http://lime-technology.com/forum/index.php?topic=20761.msg186485#msg186485

If you choose to use the MV8 controller - be sure to add the MV8 Hack found in the 1st page of this thread.  ESXi 5.1 needs the hack.  Also if you remove a pass through and then add it back later - you need to reapply the hack.  When a pass through device is removed the settings (and hack) is removed from the VMX file.

 

Well worth all the effort.  Thank you to everyone and to this thread which is a wealth of information.

 

Now if I can just fix my M1015 that would not flash.

 

Link to comment

jesseasi,

 

I'm looking to build an unRAID server similar to yours and others using the Supermicro board and either the m1015 or the MV8 cards.  I'd like to get a Xeon Ivy Bridge CPU since it's a little lower power, but I'm a little nervous after reading this board and the issues that people have had.  You said you don't think it matters.  If you ever get a chance, I'd like to hear whether the Ivy Bridge actually works, then I'll spec it for my build.

 

Thanks,

Jesse

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.