Jump to content
Talos

IBM ServerRaid Br10i FRU PCI-e 8x SAS HBA's

120 posts in this topic Last Reply

Recommended Posts

Filling in some additional details carried over from this thread: [ http://lime-technology.com/forum/index.php?topic=10927.0 ]

 

The two cards I have actually have 44E8690 part numbers on them. They are physically 8x PCI-Express. In order to physically fit in a 4x PCI-Express physical slot it would need to be open ended. I do not know if they will work at the electrical 4x PCI-Express level.

 

It is running the absolute latest firmware I could find and the "IT" variant of it [ http://lime-technology.com/forum/index.php?topic=7451.msg104148#msg104148 ]

 

It was working similarly even with earlier firmware.

 

The limitations in 5.0 beta 4 are no built in drive temp reports and no built-in spindown/spinup control, but using smartctl and hdparm are possible workarounds.

 

Share this post


Link to post

Hmmm ... Had my first setback.  I may not be running the absolute latest firmware.  I did upgrade to v6.30 dated 2009-11-12.  I think it is the newest from IBM's website. 

 

Smart report is not working for 2T drives (not sure it matters, but I was testing with a Hitachi 5K3000).  Works fine for 320G drive I was testing with previously.  I did get some action with the scsi option - but just the model and S/N and temperature.  No attributes.  Preclear wasn't happy that smart didn't work - and that was my goal, to preclear the drives.  I switched to motherboard ports for preclearing and it is humming away now.  Started out at 143MB/sec.  Fastest I've ever seen!

 

BRiT - can you confirm Smart is working with the latest BR10i firmware you installed with 2T drives?  Since I want to experiment with the RAID-1 cache drive, I would need the "IR" BIOS.  Can you point me in the right direction?

 

Thanks!

Share this post


Link to post

I'll flash over to the latest IR firmware from the general LSI SAS3081E support page included with this firmware package SAS3081ER P20 Supports SAS_1-5G_3G_6G and SATA_1-5G_3G_HDD.

 

On the "IT" firmware version my Slackware Current distro smart report works fine on my 2TB Seagate Lp drives by using: smartctl -A /dev/sdg or smartctl -A /dev/sdh. From what I can remember under Slackware Current distro I was getting smart reports under the v6.30 IR as well.

 

Firmware image's version is MPTFW-01.32.00.00-IT LSI Logic

x86 BIOS image's version is MPTBIOS-6.34.00.00 (2010.12.07)

 

I'll also boot into a pure unRAID 5.0 beta 4 environment and see if it works any differently than the full Slackware distro.

Share this post


Link to post

Flashed with 3081ERB3 and lsiutil displays:

 

Firmware image's version is MPTFW-01.32.00.00-IE LSI Logic

x86 BIOS image's version is MPTBIOS-6.34.00.00 (2010.12.07)

 

Under Slackware Current distro smart report works fine on the same 2TB Seagate Lp drives by using: smartctl -A /dev/sdg or smartctl -A /dev/sdh.

 

Under a pure unRAID 5.0 beta 4  smart report works fine on the same 2TB Seagate Lp drives by using: smartctl -A /dev/sdg or smartctl -A /dev/sdh.

 

Again, both of these drives are not assigned or associated with any volumes on the SAS card, but are part of the unRAID array.

 

 

Share this post


Link to post

Hmmm ... Had my first setback.  I may not be running the absolute latest firmware.  I did upgrade to v6.30 dated 2009-11-12.  I think it is the newest from IBM's website. 

 

Smart report is not working for 2T drives (not sure it matters, but I was testing with a Hitachi 5K3000).  Works fine for 320G drive I was testing with previously.  I did get some action with the scsi option - but just the model and S/N and temperature.  No attributes.  Preclear wasn't happy that smart didn't work - and that was my goal, to preclear the drives.  I switched to motherboard ports for preclearing and it is humming away now.  Started out at 143MB/sec.  Fastest I've ever seen!

 

BRiT - can you confirm Smart is working with the latest BR10i firmware you installed with 2T drives?  Since I want to experiment with the RAID-1 cache drive, I would need the "IR" BIOS.  Can you point me in the right direction?

 

Thanks!

 

I have the same combination. a BR10i and I just put a 5K3000 on it. Smart report is working fine as well as the pre-clear script using the -D option.

 

This is the firmware I flashed it with:

http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5077845

 

I believe that's the same as yours.

Share this post


Link to post

Not much I can do at the moment.  Preclears are running from the motherboard slots.  Once it is complete (tomorrow), I will reconnect the drives to the BR10i and try smartctl with the -A and -D parameters.  But the 300G did not need anything special, not sure why the larger disk would not work the same.

Share this post


Link to post

I just tried a 750GB F1 connected to the same BR10i. It needs the -D option as well.

 

UnRAID 5.0b4

preclear_disk.sh 1.6

Share this post


Link to post

I have moved my BR10i to its expected home in my C2SEE-O motherboard.  Strangely I cannot hit Ctrl-C and have it launch its BIOS-level configuration screens.  I was able to access them when the board was in another computer.  When I try in its current home, I get some message about it launching the configuration screen after the initialization is complete, but it them proceeds to boot to the USB stick.

 

I have configured the USB stick as a forced floppy / removable disk.  Not sure if that is somehow throwing it off.  I'm going to play around with it this weekend, but thought I'd post and see if anyone else was seeing this behavior.  Getting into the menu is necessary to create a RAID array, which I plan to do.  Not sure if I can set up the RAID array on another computer.  Hopefully either ...

 

- The new firmware will fix (I'm on 6.30 and need to upgrade to 6.34?)

- I'll find some BIOS options to make the boot menu come up

- Someone will have an idea to make it work

 

Any help or advice appreciated.

Share this post


Link to post

You're running out of ROM space. I have to temporarily disable the on-board SATA/RAID controller completely in order to free up enough space for the LSI bios to load. Depending on the board, you may to disable other stuff as well. It's a PITA getting into the LSI config, but luckily don't need to do it very often.

 

You might have to disable any boot options (such as the USB key) also. Mine behaves in a similar manner. If I hit Ctrl-C, I get the message that it will load after boot completes, but it seems as if the computer isn't smart enough to put the LSI BIOS before any boot process therefore the configuration never gets a chance to load before the boot devices.

Share this post


Link to post

I see the same behavior if I have it set to boot my USB flash drive. If I unplug the flash drive and force the computer to not have a valid boot device then the LSI BIOS will load. Odd behavior in my mind.

 

If you can run that lsiutil it looks like you can do all the BIOS items such as define new RAID Arrays and the like from it once you have a running system with /dev/mptctl support. The Linux Kernel option for this is CONFIG_FUSION_CTL under Device Drivers -> Fusion MPT device support -> Fusion MPT misc device (ioctl) driver. Unfortunately it doesn't look like unRAID has this configured.

Share this post


Link to post

You're right. The ioctl driver is not compiled in on unRAID. I was going to look at that and try to compile the module this weekend if I get time.

Sounds like you are using the same workarounds as I. My key is internally mounted in the case and not-so-convienent to remove etc so I just disable all booting in the motherboard BIOS instead. That along with disabling the on-board SATA allows the LSI BIOS to load properly.

Share this post


Link to post

I see the same behavior if I have it set to boot my USB flash drive. If I unplug the flash drive and force the computer to not have a valid boot device then the LSI BIOS will load. Odd behavior in my mind.

 

If you can run that lsiutil it looks like you can do all the BIOS items such as define new RAID Arrays and the like from it once you have a running system with /dev/mptctl support. The Linux Kernel option for this is CONFIG_FUSION_CTL under Device Drivers -> Fusion MPT device support -> Fusion MPT misc device (ioctl) driver. Unfortunately it doesn't look like unRAID has this configured.

 

Thanks!  I simply went in and set the Removable Boot Device to unassigned and the interface came up on the next boot.  I was able to confirm that the BIOS saw the 3T drive as a ~2T drive within the firmware.  That is good news (in a way).  If the firmware update allows the BR10i to see the 3T drives at full capacity, then unRAID may yet see them as 3T also.

 

I'm planning on playing after work today.  Hopefully will have some news later this evening.

Share this post


Link to post

Another utility worth exploring is cfggen.

 

It's a cli configuration app with support for the BR10i.

 

http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-63501

 

It too requires certain modules to be loaded which currently are not included in a default unRAID config.

 

Here is the output:

 

LSI Logic IR Configuration Utility 2.06.01
  cfggen <controller #> <command> <parameters> ...
    where <controller #> is:
      Number between 0 and 255
    where <command> is:
      AUTO       -  create an IR volume without specifying drives
      CREATE     -  create an IR volume
      DELETE     -  set controller configuration to factory defaults
      DISPLAY    -  display controller, volume and physical device info
      ENABLEIR   -  enable the IR capability of the controller
      DISABLEIR  -  disable the IR capability of the controller
      FORMAT     -  perform a low-level format of a SCSI hard disk drive
      HOTSPARE   -  make drive a hot spare
      LIST       -  lists all available LSI adapters. This command does not require <controller #>
      LOCATE     -  turn ON and OFF a drive's Identify LED
      REBUILD    -  start resyncing drives in volume without requiring a reboot
      STATUS     -  display current volume status info
      SETOFFLINE -  bring a harddisk offline (in IM/IME volumes)
      SETONLINE  -  bring a harddisk online
    where <parameters> are:
      Command specific values; enter "cfggen <controller #> <command>"
      to get command specific help

 

 

Share this post


Link to post

I am having trouble updating my firmware using the LSI sasflash program.  HELP!

 

****************************************************************************

  LSI Corporation SAS FLASH Utility.

 

  SASFlash Version 1.28.00.00 (2010.12.08)

 

  Copyright © 2006-2010 LSI Corporation. All rights reserved.

****************************************************************************

 

      Adapter Selected is a LSI SAS 1068E(B3):

 

      Executing Operation: Flash Firmware Image

 

              Checksum Passed: 00000000

 

              Verifying NVDATA Compatibility. . .

              NVDATA Versions Compatible

 

              ERROR: Product ID and Vendor ID match FAILED.

 

      Due to error remaining commands will not be executed.

      Unable to Process Commands.

      Exiting SASFlash.

 

 

Share this post


Link to post

I'll flash over to the latest IR firmware from the general LSI SAS3081E support page included with this firmware package SAS3081ER P20 Supports SAS_1-5G_3G_6G and SATA_1-5G_3G_HDD.

 

...

 

I followed this link and downladed the 1.5, 3.0 speed firmware.

 

I previously installed the IBM BIOS update following these instructions ...

 

1. Download the 2.72 fw from IBM site and execute it

2. Go trough the wizard till it tells you that it can't upgrade but don't click exit/close

3. Run cmd as administrator

4. cd %TEMP%

5. Locate folder with unpacked firmware there

6. cd <folder>/image

7. sasflash -list

8. lsiIRupdate ctlr-info.txt

9. sasflash -list

 

I tried this again to see if I could inject the updated firmware files and use this same process.  Am a bit nervious to brick my card, so haven't done it.  This is what I am seeing from the ./image directory and running sasflash -list

 

03/04/2011  07:35 PM    <DIR>          .
03/04/2011  07:35 PM    <DIR>          ..
10/14/2010  02:24 PM           194,048 bios_6.30.00.rom
10/14/2010  02:24 PM               246 ctlr-info.txt
10/14/2010  02:24 PM           143,872 hwdetect.exe
10/14/2010  02:24 PM             1,137 install.bat
10/14/2010  02:24 PM            57,344 lsiIRupdate.exe
10/14/2010  02:24 PM           118,784 mindrv.exe
10/14/2010  02:24 PM           277,976 mon_02d17_013000.fw
10/14/2010  02:24 PM             5,417 readme.txt
10/14/2010  02:24 PM           278,100 ruskin_b3_02d26_01.30.05.fw
10/14/2010  02:24 PM           277,840 ruskin_c0_02d26_01.30.05.fw
10/14/2010  02:24 PM           204,800 sasflash.exe
10/14/2010  02:24 PM           137,728 uefi_3.16.00.05.rom
10/14/2010  02:24 PM               288 winwrap.cfg
10/14/2010  02:24 PM           352,256 winwrap.exe
10/14/2010  02:24 PM               288 winwrap64.cfg

C:\Users\BJP~1\AppData\Local\Temp\ibm_ux_pkg_00000b48\image>sasflash -list

*************************************************************************
   LSI Corporation SAS FLASH Utility.

   SASFlash Version 1.24.00.02 (2010.02.15)

   Copyright (c) 2006-2007 LSI Corporation. All rights reserved.
*************************************************************************

       Adapter Selected is a LSI SAS 1068E(B3):

Controller Number:              1
Controller:                     1068E(B3)
PCI Address:                    00:01:00:00
SAS Address:                    500605b-0-0187-7950
NVDATA Version (Default):       0x2d26
NVDATA Version (Persistent):    0x2d26
Product ID:                     0x2704
Firmware Version:               01.30.05.00
NVDATA Vendor:                  IBM
NVDATA Product ID:              IBM ServeRAID BR
BIOS Version:                   06.30.00.00
BIOS Alternate Version:         N/A
EFI BSD Version:                03.16.00.05
FCODE Version:                  N/A

Finished Processing Commands Successfully.
       Exiting SASFlash.

 

Share this post


Link to post

You might need to use the '-o' parameter to force it directly using sasflash.

 

sasflash.exe -o -f 3081ERB3.fw -b mptsas.rom

Share this post


Link to post

http://lime-technology.com/forum/index.php?topic=7451.msg104148#msg104148

 

Alright, so I felt really adventurous tonight, so I'm trying out the newest firmware I could find. I even flashed it with "IT" mode.  :o

 

The firmware version is 01.32.00.00 (December 8, 2010). The BIOS version is 06.34.00.00 (December 7, 2010).

 

I downloaded from the general LSI SAS3081E support page this firmware package SAS3081ER P20 Supports SAS_1-5G_3G_6G and SATA_1-5G_3G_HDD. As the name indicates it only supports SATA 1.5G/S and 3G/S drives. There is another firmware package that supports only SATA 3G/S and 6G/S drives, but I opted not to use it since it does not support/negotiate with 1.5G/S drives.

 

Both firmware packages contain a dos-based flash utility (sasflash.exe). In order to flash the IBM BR10i with this firmware, you have to force it since the initial vendor id does not match. The sasflash option to force an update is -o and must be supplied with the -f and -b options. If asked if I wanted to continue a couple of times and responding with a "y" both times it then proceeded with the firmware update.

 

sasflash.exe -o -f 3081ETB3.fw -b mptsas.rom

 

If you want to remain with the IR firmware use this command:

 

sasflash.exe -o -f 3081ERB3.fw -b mptsas.rom

 

After this was done, I powered cycled the server and upon reboot, unraid detected the drives fine.

 

Most of the info was gathered from ServeTheHome (Article 1, Article 2, Forum Thread), but I did not see any clear indications the 3081-based firmware would work on the IBM BR10i since it is a 3082-based card.

 

I only suggest doing the update if you feel equally as adventurous.  ;D

 

bjp999, I'm using the 2.72 package from IBM, same as you. No issues. Are you having any?

 

Also, I got the mptctl module compiled and loaded. Both lsiutil and cfggen work great under unRAID.

Share this post


Link to post

I didn't have issues with that version, but opted for the newest versions available to see if it was able to cross-flash; yes, it was doable. I'll be toying around with some RAID-1 drive setups and unRAID.

 

The IBM 2.72 version seems to identify the 3TB as 2.2TB. We're hoping the newest version also supports 3TB drives.

Share this post


Link to post

Thanks BRiT for the "-o" option.  I should have reread the thread before asking the question.

 

I was able to successfully update my BIOS / firmware.

 

The results, however, are no different than with the prior firmware.

 

The BIOS screen, inside the controller setup, using fdisk -lu, and running cat /proc/partitions are showing the disk to be exactly 2 TiB (~2.2 TB).

 

hdparm -N is correctly showing the number of raw sectors as 5860533168 = 3 TB.

 

I am not sure what the OS is doing to compute the size, but it is not getting it from hdparm.

 

I can assign the drive to, for example, the cache slot.  It shows in unRAID as 2.2 TB.

 

Perhaps if the disk had a GPT partition in place, the OS would see it as a larger drive.

Share this post


Link to post

Look HERE.  The BR10i is the only controller I tested that acted this way!

Share this post


Link to post

Lets hope LSI will soon begin their testing with 3TB drives and provide updated firmware. It's likely some artificial cap on how many LBAs it would need for raw devices.

 

Out of curiosity, if you setup a RAID-0 of two 1.5TB or larger, does the controller report the proper size?

Share this post


Link to post

I sent a support email to LSI.  We'll see if they have anything to say.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.