Cache mover makes disks throw I/O errors

[Abnorm]

Hey, 

so during the last few months i've been having issues copying files to my array, issue seems to come whenever the cache mover process is running on large transfers (+/- 2-500GB).  

Well, the thing is, as my internet sucks where i live, i've got a server located somewhere else where i physically move a usb drive back and forth, which is fine for my needs at the moment. Fibre is on it's way but it will be a while still. 

 

I bought a few seagate 4tb drives during the summer since the price was very good, I've normally ran WD Reds and have never had any issues. So a few weeks later my seagate drive started acting up, fine i thought, got a RMA and a new drive. While doing running the mover again the new one also died. Fair enough, unlucky I guess, so I RMA this one as well. 

Got a new one, same story, didn't find anything wrong, so i've later moved this disk to another server and it works fine. I got a WD Red as a new replacement in my current local server. While i changed the drive, i also switched sata cable just to be sure. But it happened again now.. 

 

So could anyone tell me what the heck is actually going on ? I'm getting a bit impatient running rebuilds that takes 2 days a few times a month. Server has been working fantastic up until the last 3-4 months. This seems to happen during the mover process from my ssd's to my array. A few other disks also reports issues, but they're never exluded from the array, only disk 11 (SDN) is. SMART never finds physical issues on either of these disks. 

 

Is there something I'm doing wrong ? Is there some config i need to be aware of ? 

 

attached is the diagnostics log dump. 

 

Please help, and thank you in advance. 

blackbox-diagnostics-20181125-0030.zip

[John_M]

I don't think the disks are the problem. Your Marvell-based controllers seem to be dropping them. It's a known issue with 64-bit Unraid.

[Abnorm]

Thanks for the reply John_M, much appreciated! 

 

I could see controllers might the issue, it would be pretty wierd if it was my drives after all, since I've replaced a bunch of them. It has never been an issue until the last few months. Good to know. 

I guess it's time to stop using my cheapo china sata controllers and get something worthwhile.. pretty evident it came back to haunt me.

Lesson learned

 

So, I've got 2x cheapo sata controllers, both got 4 ports, so I'm in the market for a 8 port pciex LSI sata controller it would seem. I cannot physically fit any more drives in my current box, so no need for any more ports.

 

Also, the idea is then to replace controller, boot server, array will fail, stop it, assign correct disks to the correct slot and restart array right ?

 

Any recommendations for what LSI controller to get ? 

 

[John_M]

35 minutes ago, Abnorm said:

Any recommendations for what LSI controller to get ?

I personally use Del H310s. They need to be crossflashed with IT mode firmware. They're available quite cheaply on Ebay if you're happy to take ones that have been pulled from servers.

[Abnorm]

On 11/25/2018 at 3:21 AM, John_M said:

I personally use Del H310s. They need to be crossflashed with IT mode firmware. They're available quite cheaply on Ebay if you're happy to take ones that have been pulled from servers.

Ok, sorry for the late reply,

Thank you for your recommendation  

I was browsing around a bit on ebay and found a IBM ServeRAID M1015, which is a LSI9211-8i controller really. Thisc an be crossflashed with a IT/HBA firmware so it should do the trick. Cost was £28 with free shipping, SFF cables was like £7  

[John_M]

Yes, that will be fine. It's another popular one. The mini-SAS connectors are not as conveniently positioned in my opinion, being on the top long edge of the board and near the front. It's just that I prefer the bundle of wires to come out of the rear, short edge, as they do on the Dell card.

[Abnorm]

Allright, stuff finally arrived from china in time before the holidays. So I've been messing with stuff today. 

For flashing IT mode i followed the following guide: https://www.servethehome.com/ibm-serveraid-m1015-part-4/

 

A quick note before the before note: Get long enough SFF cables! I bought 50cm, you should at minimum get 1m cables.  

 

A few notes (if anyone is doing the same):

The easiest way to do this is with a seperate computer, try to borrow from someone if possible. 

 

I flashed the card with IT mode without a Option ROM. This is to save some boot time as the option rom is not used. It usually is for configuring RAID (but we do not use a RAID for unraid.). 

To do this, the flash command looks like this: sas2flsh -o -f 2118it.bin 

On UEFI it would be: sas2flsh.efi -o -f 2118it.bin

 

I used freeDOS booted from USB for clearing the firmware, and ended up having to use UEFI for writing firmware and setting the SAS address. 

This varies on your hardware of course. I got a PAL error in freeDOS when trying to write the firmware, this is normal it seems. 

 

FreeDOS bootable USB can be created super easy in Rufus, just copy the files from the guide to USB afterwards, overwriting autoexec.bat is ok. it isn't needed for this. 

 

Another thing, do NOT mount your disk in UEFI, just do 'fl1:" <- depending on your device. I had one disk connected, but the devicelisting i UEFI cleary said 'USB' 

Using 'mount FL1' would not work in UEFI. 

 

Also, to flash in UEFI you will need the .efi version of the sas2flash tool. I found it inside Installer_P18_for_UEFI.zip and just googled it.  here: https://rebyte.me/en/avago/157482/file-1538648/

it seems antivirus does not like .efi files, so if it's gone, check your antivirus. I had no issue with windows defender, but it's really not all that strict on my client. 

 

If you've booted to UEFI, got it flashed etc the last step is to set the SAS address for the card.

The numbers will only matter if you have more than one card (think of the SAS address as a MAC-adress, having identical MAC-addresses will create a conflict and the network would not work, same with SAS). 

I'd of course recommend using the card's unique address, but any numbers should do. (16 hexadecimal numbers in total, including SAS prefix).  

 

Thanks for the tips, and hopefully this will help someone. If you have any questions, pm or mail me. 

Edited May 30, 2019 by Abnorm