Any idea why my cache pool would keep crashing on me? After a reboot, one of the drives in my cache pool is now unmountable and all dockers unavailable. This is the second time in a week this has happened. I’ve been trying to add unassigned devices to my server to move my downloads and appdata folders to but haven’t made it that far because this keeps happening. Thoughts? Both SSD’s in the cache pool test fine in the SMART testing. Cables are a few months old and haven’t been touched. 

 

UPDATED: things keep getting worse... i rebuilt my cache pool from an appdata backup. docker were up and running immediately without me deleting my docker.img file. but, now everything stopped working. i then stopped docker services, deleted the docker.img, restarted docker and now the "the docker service failed to start."

 

then, 1 of my 4 unassigned devices (all ssd's) disappeared and then the rest of the drives were unmounted and cannot be remounted.

 

diagnostics attached

samson-diagnostics-20190831-0152.zip

Edited August 31, 20196 yr by lordmarqui
added more info

@johnnie.black would be best to answer, but it certainly looks to me like you've got bad connections to the controller or the drives.  Reseat everything

 

Since you've got 15+ drives, I would think the possibility exists that your P/S isn't up to snuff

Thanks for the reply... everything appears to be seated correctly but then again, all the issues began after I added these new ssd drives in. all drives are connected using a sata breakout cable on a 7pesh2 gigabyte server board. I have a pcie sata3 card that’ll arrive tomorrow which I plan to install to see if this it makes things better. 

 

Looking forward to @johnnie.black’s insight. Thanks again 

Could be cable/power issue, multiple devices are being disconnected on the HBA, also firmware on the HBA should be upgraded to latest as current one installed has known issues.

 

If possible connect the SSDs to the onboard SATA ports, or trim won't work.

thanks a lot for the reply...

 

the firmware on this motherboard has been flashed to IT mode and as far as i'm aware of is the most up to date, but i'm clearly no expert. (build guide: https://www.serverbuilds.net/anniversary and flashing to IT mode: https://www.serverbuilds.net/flashing-onboard-sas-2008)

 

i removed all ssd's that i recently installed (leaving my 2 cache drives)... got things back up and running... then i placed the ssd's into a 24-bay DAS unit that i built (similar build guide here: https://www.serverbuilds.net/16-bay-das) and then it all went down again...

 

could the issues be related to the specific drives i'm putting in? they're the same drives that are in my cache (https://www.amazon.com/Silicon-Power-Performance-Internal-SP512GBSS3A55S25/dp/B07997QV4Z?psc=1&SubscriptionId=AKIAIA3UEVTLIG7AIKFA&tag=serverbuilds-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=B07997QV4Z)... there seemed to be odd behavior when i tried to mount the ssd -- occasionally the ssd's would all mount fine (i was trying to add 4 new ssd's as unassigned devices). occasionally 1 would unmount and then become unmountable. occasinally i would lose 2 of the ssd's off the UI entirely and one would stay mounted and another unmount but the mount button would be grayed out.

 

does any of this make sense?

18 hours ago, lordmarqui said:

i'm aware of is the most up to date

It's using 20.00.04.00, latest is 20.00.07.00, and like mentioned that one has known issues.

 

18 hours ago, lordmarqui said:

could the issues be related to the specific drives i'm putting in?

Could be some compatibility issue with the HBA, but like mentioned all SSD shoudn't be on the HBA

i've done a few things (except for update the mobo firmware bc i'm afraid to brick my system).

 

I bought a 6 port pcie sata 3 card, installed it and got everything working late last night. I moved my incomplete and complete download folders to individual SSD’s (and placed 2 additional SSD's for use once this test passes). I then downloaded a bunch of stuff as I went to sleep since all the quick tests on the mounted SSD's looked great... Woke up this morning and noticed the 4 newly installed SSD’s unmounted about 1 hour after the downloads began. (my 2 separate cache drives are connected directly the mobo and are thankfully fine).

 

This is the same behavior as before. Is there something I’m doing wrong?

 

Thus far I’ve tried directly wiring to the mobo via sata breakout cables, mounting in my Das, and now via this pcie sata3 expansion card — each time the drives unmount. Any thoughts at all? diagnostics attached.

 

if this is truly a mobo firmware issue, do you have a 'hold-my-hand' tutorial so i don't destroy my setup, since this mobo is nowhere to be found anymore?

samson-diagnostics-20190903-1802.zip

Try connecting all the SSDs to the onboard STA ports, you have 4 unused Intel ports, those are the best ports for the SSD, despite those for being SATA2.

 

LSI firmware upgrade is usually very easy to do, just download the package from support and follow the instructions.

Thanks for the reply...

 

Just so I’m sure, this is the layout of my mobo. The 2 cache drives are in the sata 6gb/s (which I assume is sata3) which leaves me with 3 empty “mini sas cable connectors.” I have 2 sata breakout cables (which plug into one of these mini sas ports and had 6 outputs) — this gives me a total of 12 drives to run off of two ports. I know those cables work. Would it be reasonable to try one drive per cable?

i went ahead and placed 2 of my SSD's on 2 of the mobo's SAS ports... i started a few downloads and i'll report back soon.

i mounted 2 ssd's as i mentioned above, ran downloads for about 4 hours but then the complete_downloads SSD dropped off, but the incomplete_downloads SSD remained mounted.... diagnostics are attached.

 

thanks a lot for all the advice!

samson-diagnostics-20190904-0027.zip

11 hours ago, lordmarqui said:

which leaves me with 3 empty “mini sas cable connectors.”

The miniSAS connector closest to the 2 SATA ports are the remaining Intel SATA ports, use those for the SSDs.

 

ok great... i connected two of the ssd's to this port this morning and have downloaded about 400gb and no issues so far... i'll report back in a few days and mark this issue as resolved if no trouble.

 

thanks a lot!

assuming this solves my ssd woes, what do you think the problem was related to?

The firmware on the LSI, or some compatibility issue with those SSDs.