Log spamming with Unknown results and random tags

[JuSnyder]

My Unraid server logs started filling up recently, and the logs aren't pointing me in any particular direction, so I'm hoping someone can help me out.  Google and searching previous forum posts, I can't find anything related.  The tag#'s are seemingly random - if you compare the first [sdi] tags with the next time [sdi] shows up (every 5 seconds), the tags are different and still show "UNKNOWN".  The server and all containers/components are running fine until the log fills up.  I rebooted yesterday and the log is already sitting at 9% full, so I'm looking at about a week of run time before I need to reboot again.  Any assistance is appreciated.

 

I'm on version 6.9.1 currently.

 

According to the logs, it looks like the only drives affected are:

sdc (SSD cache drive)

sdf - SAS drive - no data present

sdg - SAS drive - no data present

sdh - SAS drive - no data present

sdi - SAS drive - no data present

 

These are installed directly in the server using the same SAS card as "sdd" and "sde" - both of which have data, host all of my docker data, and are working fine.

 

sdj (parity), sdl, sdm, and sdk are connected via an eSAS card and are also all working as expected.

 

Below is sample of the log entries I'm dealing with.  Aside of the log bloat, I'm not noticing any negative experiences.  But once the log fills up, we all know what happens.  I've also attached the diagnostics before my logs fill up and it doesn't let me download it.

 

May 27 04:40:20 Tower kernel: sd 2:0:7:0: [sdi] tag#370 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:20 Tower kernel: sd 2:0:7:0: [sdi] tag#370 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:20 Tower kernel: sd 2:0:6:0: [sdh] tag#196 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:20 Tower kernel: sd 2:0:6:0: [sdh] tag#196 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:20 Tower kernel: sd 2:0:5:0: [sdg] tag#301 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:20 Tower kernel: sd 2:0:5:0: [sdg] tag#301 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:20 Tower kernel: sd 2:0:4:0: [sdf] tag#325 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:20 Tower kernel: sd 2:0:4:0: [sdf] tag#325 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:20 Tower kernel: sd 2:0:0:0: [sdc] tag#199 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:20 Tower kernel: sd 2:0:0:0: [sdc] tag#199 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:25 Tower kernel: sd 2:0:4:0: [sdf] tag#279 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:25 Tower kernel: sd 2:0:4:0: [sdf] tag#279 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:25 Tower kernel: sd 2:0:5:0: [sdg] tag#343 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:25 Tower kernel: sd 2:0:5:0: [sdg] tag#343 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:25 Tower kernel: sd 2:0:6:0: [sdh] tag#346 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:25 Tower kernel: sd 2:0:6:0: [sdh] tag#346 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:25 Tower kernel: sd 2:0:7:0: [sdi] tag#195 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:25 Tower kernel: sd 2:0:7:0: [sdi] tag#195 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:25 Tower kernel: sd 2:0:0:0: [sdc] tag#200 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:25 Tower kernel: sd 2:0:0:0: [sdc] tag#200 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:30 Tower kernel: sd 2:0:0:0: [sdc] tag#236 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:30 Tower kernel: sd 2:0:0:0: [sdc] tag#236 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:30 Tower kernel: sd 2:0:6:0: [sdh] tag#316 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:30 Tower kernel: sd 2:0:6:0: [sdh] tag#316 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:30 Tower kernel: sd 2:0:5:0: [sdg] tag#239 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:30 Tower kernel: sd 2:0:5:0: [sdg] tag#239 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 27 04:40:30 Tower kernel: sd 2:0:4:0: [sdf] tag#242 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
May 27 04:40:30 Tower kernel: sd 2:0:4:0: [sdf] tag#242 CDB: opcode=0x85 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
 

tower-diagnostics-20210528-0017.zip

[JorgeB]

Likely related to the megaRAID controller, see if it can be flashed to LSI IT mode but probably note, RAID controllers are not recommended.

[JuSnyder]

11 hours ago, JorgeB said:

Likely related to the megaRAID controller, see if it can be flashed to LSI IT mode but probably note, RAID controllers are not recommended.

Thanks for the reply.  Not only has this exact setup been working for well over a year without these errors, I have 2 other drives using that Raid controller (sdd and sde) that are not showing any errors in the logs, so that doesn't add up to me.  I will say in system devices though, that the 2 drives without errors show "SEAGATE" where the other 5 that are showing these unknown errors show "ATA".  I have an eSAS card for my 4 higher volume drives though and they also show ATA but work fine, so even that doesn't prove there's a problem with the hardware to me.

 

At the moment, I don't have any other way to connect these drives.  They're connected directly to the backplane of the server, so I'm not really sure what options I have.  I have been looking for an external SAS enclosure to use, but there's so many variables like supported drive size that I need to spend more time researching before I jump into something.  Is there something in the Diagnostics that specifically told you it could be that MegaRAID card?

[JorgeB]

2 minutes ago, JuSnyder said:

Is there something in the Diagnostics that specifically told you it could be that MegaRAID card?

No, but the megaRAID driver is not as well tested as the LSI IT driver (mpt3sas) and is much more likely to have some issues with specific models, of course it's just a suspicion, you'd need to test with an HBA to confirm.

[JuSnyder]

Is there any modifications I can make in the mean time to exclude these errors from the log?  As mentioned in the original post, the only negative effect I'm having right now is when the log fills up.  I wish there was an easy setting to disable duplicate/similar logs from being written in a set amount of time.  As far as what i've read, there's no way to clear the logs other than rebooting the server.

[JorgeB]

Just now, JuSnyder said:

Is there any modifications I can make in the mean time to exclude these errors from the log? 

Not that I'm aware of.

[JuSnyder]

I'm creeping around the iDRAC of my server, trying to find some kind of clue as to what's going on, and I noticed something interesting.  Even though 4 of the 5 drives giving errors have SAS connections, they're actually registering as SATA drives.  The SSD is also registering as a SATA drive, which would be correct, however the 2 drives that are working fine without any unraid errors are listed as actual SAS drives on the controller.  Months ago, I noticed log spamming because of a known spindown communication error between unraid and the sas controller and once I disabled all of the spin downs, everything was fine for a few months again, until i started getting the current errors. 

 

I'm still having a difficult time wrapping my head around why this would just recently start happening out of the blue, but I'm wondering if there's some sort of driver/communication problem with sata drives on a sas controller.  I'm limited on pcie ports in my server, so I suppose an esas disk shelf might need to happen sooner than later.  Thanks for your help JorgeB, I'm still hopeful that someone has some info on what may have caused this.