So here's a weird one...i installed a LSI-9200-8e in my Supermicro X10DRC-LN4+ mobo.  (FYI: This card refuses to work on x8, only x4, even after I flashed the latest firmware.)  I also noticed the bios is missing from the card (...haven't had a chance to put that back, but google tells me that's ONLY for bootable devices to be seen; which obviously i do not need in unraid).  

First i plugged both SAS-e ---> mini SAS-e cables into my Dell SC400.  This contains x6 SAS disks in a ZFSz2 pool, that I created on my home server, filled with data, then moved to the main server enclosure.  Before I even started my array, I was getting double disks showing up in unraid.  One set in the assigned places, and the identical disks in UD, but with a different UID.  

I figured it was UNraid's lack of support for dual-link (I had hoped the SC400 acted like an expander and used all the bandwidth from both cables and not redundancy, but it seemed not to...im new to this).  So i did what anyone would do, I removed the second cable, and BAM no clones in UD...so i thought!  But, soon as i started up the array the clones were back!  

Ok, so everything showed normal in my array/cache pools otherwise, and i'm aware that SAS has 2 paths available, so cool...except nothing worked!  i cant stop the array, reboot, nothing...i can click the buttons, but its like they laugh at me when i do!  There was a bunch of IO errors in the disks (these disks had worked flawless, and without clones, in this same case from my home servers onboard HBA).
The only way to use my server again is to eliminate this from the server and it seems to be ok.  
As an added bonus, once everything booted again, i had no docker containers installed (thank God for backups!).
 

This is so annoying cuz i don't have any access to the settings of this HBA card.  I remember something about my onboard sas had a "dual-link" option that gave me clones as well, so how do i turn that feature off; or make UNRAID ignore dual link devices (till/if its supported)??

I am hoping someone has an easy answer i overlooked.  Unfortunately, I didn't take a diag file while this was happening as i couldn't do anything other then click on the tabs while the array was started.  And it was 6am, and i had started working on it at noon the day before.  But if we have to, i can make my unraid unstable again and take a diag before i start the array (with both and just the single SAS-e cable installed).

Edited January 1, 20242 yr by miicar

9 hours ago, miicar said:

I figured it was UNraid's lack of support for dual-link

Unraid supports dual link, it does not support SAS multipath.

 

 

4 minutes ago, JorgeB said:

Unraid supports dual link, it does not support SAS multipath.

 

 

Fair enough...i probably have my terminology (and understanding) wrong.  If i understand correctly, multipath is from more then one HBA card or server? 

The question remains, why the clones and crashing?

 

Edited January 1, 20242 yr by miicar

Dual link is for performance only, same HBA connected with two cables to the same expander, multipath is for redundancy, for example if you have two HBAs connected to the same expander, or one controller connected to two different expanders.

 

19 minutes ago, miicar said:

why the clones and crashing?

If you are still using multipath that is not supported, you need to correct that, if you see duplicates of the disks you will have issues.

Ok.  So i defiantly have dual link then.  With both cables, plugged into a single HBA, i get duplicates all the time, and with the one cable, im getting them ONLY when the array is started and the whole system grinds to a halt and needs a hard restart to bring back.

 

Just a note: This pool was made on a newer vs of unraid then i imported it too...don't think that matters tho?  

 

 

Edited January 1, 20242 yr by miicar

10 minutes ago, miicar said:

With both cables, plugged into a single HBA, i get duplicates all the time

This means it's not dual link, I'm not familiar with the Dell SC400 but you likely have the cables connected to two different expanders (IOM controllers), post diags and the output of "zpool import" with just a single cable connected.

3 hours ago, JorgeB said:

...you likely have the cables connected to two different expanders (IOM controllers)...

The SC400 has 2 removable banks; each with x4 mini-sas connections that act as expanders (i had x60 drives running at once when testing drives in multiple enclosures that were daisy-chained together).  I know the bottom bank is used for failover/multilink, so i have never used it in the current configuration. 
Regardless, i am only running one cable now. 
Being new years day, I will have to remote desktop into the site. So for now, i can only post diags with having one cable plugged in anyway.  I rather get that working at least, before i tackle dual link. 

To get this server stable for production, I have removed the pool from the array, but all the x7 3TB disks, in UD, are the ones connected to that external cabinet (x6 pool disks, one test disk). 
Something i didn't try the other day: After i took this diag, i tried mounting the pool within UD, and it seems to mount fine in UD, and read/write through SMB to a windows machine.  I did my quarterly updates, the only thing changed since the other day.  My staff have the day off today, so i am going to try officially adding this pool back into the mix, and i'll report back what happens.

Just so i'm clear on the proper import procedure:

- add new pool of 6 devices; named the same as it is on the disks
- add the disks to the pool (order shouldn't matter??)
- go into the pools settings and change to zfs, Rz2 (I'm an idiot and cannot follow instructions...leave the FS type to auto...i wonder if that is why i had this cloning problem, as i had manually set up that on the previous attempt)
- hit start!


elmstorage-diagnostics-20240101-0922.zip

3 hours ago, JorgeB said:

"zpool import"

~# zpool import
   pool: elmonsitebu
     id: 4775391284715079306
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        elmonsitebu  ONLINE
          raidz2-0   ONLINE
            sdaa1    ONLINE
            sdac1    ONLINE
            sdad1    ONLINE
            sdab1    ONLINE
            sdaf1    ONLINE
            sdae1    ONLINE

Edited January 1, 20242 yr by miicar
Fixed my import process mistake

47 minutes ago, miicar said:

- go into the pools settings and change to zfs, Rz2 (I'm an idiot and cannot follow instructions...leave the FS type to auto...i wonder if that is why i had this cloning problem, as i had manually set up that on the previous attempt)

So...Everything seems fine now!  The pool has imported and mounted properly; no clones, no spamming IO errors in the sys log, and no frozen GUI! 
So i ask myself:
- Was it (on a whim) changing the PCI-e lanes that HBA from x8x4x4 ---->x4x4x4x4 (i couldn't get it to run at its designed x8 lanes on any PCI-e slot...seems to be an issue that these v2.0 cards have.  Most people recommend running them at x4.  Its mainly used for backups, so speed isn't a concern)? 
- Did the second cable confuse the unraid config and it couldn't forget over multiple (hard) reboots? 

- Or simply (as i hope) b'cuz i edited the pool settings instead of leaving it to auto detect on import.  
- And furthermore, why did any of that mess up my dockers before?!?!?!  

I don't know if this thread should be left up for other people who might have similar issues or not (i haven't found many forums talking about duplicate drives listings).  when i have time, i think i'll set up some more simulation on another mobo/HBA/drive cab, and see if i can make it do this again. 
For this round, I will mark this as solved.  Thank you so much for nudging me in the right direction...i still feel like i solved it by rolling dice tho...not sure the actual cause!

Edited January 1, 20242 yr by miicar

well this is fun.  Woke up this morning to a restarted server and 3 of the 7 disks showing "emhttpd: device /dev/sd(x) has size zero"
Of course a reboot fixed it, but i have no idea why it happened in the first place...which is a big problem for me.
maybe I should go move into the bush and make friends with wolves...

UPDATE:  LSI/Broadcom called me back and informed me that there is not thermal protection on the 9200 HBA cards, so when i was running the backup, it probably got too hot and shut itself down; causing unraids kernel to panic and reboot (which is good i guess).  
After i rebooted the server, all the disks came back and are operating as expected (running a scrub atm just to be sure).

Edited January 2, 20242 yr by miicar