Constant 3ware errors spamming syslog obfuscating anything important and causing it to fill up


Recommended Posts

I have posted about this issue in the past but to no avail (see description below).  After upgrading to 6.4.0 the issue seems to have gotten worse.  There has to be something that can be done about this.  I am open to any suggestions.  Diagnostics are attached.  Would it help if I submit a bug report?

 

Problem:

I'm seeing errors like this in the syslog all the time, I think mostly when the drives are being accessed for something.

 

Jan 20 17:26:05 TowerMediaServ kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.

Jan 20 17:26:05 TowerMediaServ kernel: sd 1:0:9:0: [sdq] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08

Jan 20 17:26:05 TowerMediaServ kernel: sd 1:0:9:0: [sdq] tag#0 Sense Key : 0x5 [current]

Jan 20 17:26:05 TowerMediaServ kernel: sd 1:0:9:0: [sdq] tag#0 ASC=0x20 ASCQ=0x0

Jan 20 17:26:05 TowerMediaServ kernel: sd 1:0:9:0: [sdq] tag#0 CDB: opcode=0x85 85 06 20 00 00 00 00 00 00 00 00 00 00 40 e5 00

Jan 20 17:26:05 TowerMediaServ kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.

Jan 20 17:26:05 TowerMediaServ kernel: sd 1:0:9:0: [sdq] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08

Jan 20 17:26:05 TowerMediaServ kernel: sd 1:0:9:0: [sdq] tag#0 Sense Key : 0x5 [current]

Jan 20 17:26:05 TowerMediaServ kernel: sd 1:0:9:0: [sdq] tag#0 ASC=0x20 ASCQ=0x0

Jan 20 17:26:05 TowerMediaServ kernel: sd 1:0:9:0: [sdq] tag#0 CDB: opcode=0x85 85 06 20 00 00 00 00 00 00 00 00 00 00 40 98 00

Jan 20 17:26:05 TowerMediaServ kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.

Jan 20 17:26:05 TowerMediaServ kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.

Jan 20 17:26:05 TowerMediaServ kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.

Jan 20 17:26:05 TowerMediaServ kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.

Jan 20 17:26:05 TowerMediaServ kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.

Jan 20 17:26:05 TowerMediaServ kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.

Jan 20 17:26:05 TowerMediaServ kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.

Jan 20 17:26:05 TowerMediaServ kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.

Jan 20 17:31:10 TowerMediaServ kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.

Jan 20 17:31:10 TowerMediaServ kernel: scsi_io_completion: 8 callbacks suppressed

 

I found a handful of other threads and they all seem to say the error is benign - something relating to the kernel trying to access the drives directly for the temperature but the controller doesn't pass through the command.  Temps don't work in the GUI but I can get them via SSH and running the SMART commands manually so that's not huge deal.  I'm not experiencing any negative symptoms other than these annoying error messages in the syslog...depending on what the server is going I could see as much as 40-50 lines of this in the log every 5 minutes which means it's really easy for the noise to obscure any actual important information...and it's literally filling up the syslog.

 

unRAID version:

6.4.0

 

Hardware:

see signature

 

Docker containers:

binhex-delugevpn

cadvisor

Dolphin

Duckdns

Krusader

PlexMediaServer

PlexPy

 

VMs:

1x Win7-64

1x Win8

 

Plugins:

CA Auto Update Applications 2017.11.23 
CA Backup / Restore Appdata 2017.10.28 
CA Cleanup Appdata 2017.11.23 
Community Applications 2018.01.20b 
Dynamix Active Streams 2017.04.24a 
Dynamix Cache Directories 2016.08.26 
Dynamix File Integrity 2017.12.05 
Dynamix SCSI Devices 2017.06.05 
Dynamix SSD TRIM 2017.04.23a 
Dynamix System Buttons 2017.12.30 
Dynamix System Information 2017.11.18b 
Dynamix System Statistics 2018.01.20a 
Dynamix System Temperature 2017.12.06 
Enhanced Log Viewer 2017.12.16 
Fix Common Problems 2018.01.21 
Nerd Tools 2017.10.03a 
Open Files 2017.06.21 
Preclear Disks 2017.11.14 
Statistics 2017.09.22 
Tips and Tweaks 2018.01.04 
Unassigned Devices 2018.01.09b 
User Scripts 2017.11.18 

towermediaserv-diagnostics-20180125-2020.zip

Link to comment

Have a look at your smart reports for the disks that you have connected to that 3ware controller.  (They are in the smart directory of your diagnostics file.)  Notice that that controller is not passing through the information about any those disks--- no SMART reports, not even the serial numbers.  (That is probably the reason for all of those lines in the syslog!)   I saw a copy of the manual on line (Fee for downloading, so I didn't!).  I did determine that it was basically a hardware RAID controller but apparently has two  modes for that were not standard RAID configurations--- Single and JOBD.   You said that you had it set up as JOBD.  As I said, JOBD is not working properly with unRAID. I don't know if single will.  I would suggest that you do a bit googling to see what the 'single' mode is (and does) and, if that doesn't work, how the controller might be forced to behave in a more conventional JOBD manner.  

Link to comment
  • 1 year later...

I never managed to get "Single" mode running, so i'm also using JOBD mode. Getting the same errors as you do.

According to this post on serverfault, the error of the raid controller can safely be ignored, as it seems to be only related with not supported/forwarded status commands.

 

I receive the following two errors:

Oct 17 07:24:34 Winston kernel: 3w-sas: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.
Oct 17 07:24:34 Winston emhttpd: error: mdcmd, 2640: Invalid argument (22): write

What i did is to suppress those messages from getting written into the log.

To suppress those you need to write filters for rsyslogd into files in rsyslogs config dir, which are normally located under `/etc/rsyslog.d`.

I used the following two filters to get rid of this errors:

:msg,contains,"error: mdcmd, 2640: Invalid argument (22): write" stop
:msg,contains,"ERROR: (0x03:0x0101)" stop

So, whenever a message contains on of those contents, it will not get written into log.

You can f.e. edit the file `/etc/rsyslog.d/01-blocklist.conf` and add those lines. Then run `/etc/rc.d/rc.rsyslogd restart` to restart the service and check if that worked for you.

 

If you want to make that persistent you need to write those changes on each unraid start. For this edit `/boot/config/go` and add the following lines to it:

echo ":msg,contains,\"error: mdcmd, 2640: Invalid argument (22): write\" stop" >> /etc/rsyslog.d/01-blocklist.conf
echo ":msg,contains,\"ERROR: (0x03:0x0101)\" stop" >> /etc/rsyslog.d/01-blocklist.conf

 

Edited by martinjuhasz
Link to comment
  • 4 weeks later...
On 10/17/2019 at 12:26 PM, martinjuhasz said:

I never managed to get "Single" mode running, so i'm also using JOBD mode. Getting the same errors as you do.

According to this post on serverfault, the error of the raid controller can safely be ignored, as it seems to be only related with not supported/forwarded status commands.

 

I receive the following two errors:


Oct 17 07:24:34 Winston kernel: 3w-sas: scsi1: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.
Oct 17 07:24:34 Winston emhttpd: error: mdcmd, 2640: Invalid argument (22): write

What i did is to suppress those messages from getting written into the log.

To suppress those you need to write filters for rsyslogd into files in rsyslogs config dir, which are normally located under `/etc/rsyslog.d`.

I used the following two filters to get rid of this errors:


:msg,contains,"error: mdcmd, 2640: Invalid argument (22): write" stop
:msg,contains,"ERROR: (0x03:0x0101)" stop

So, whenever a message contains on of those contents, it will not get written into log.

You can f.e. edit the file `/etc/rsyslog.d/01-blocklist.conf` and add those lines. Then run `/etc/rc.d/rc.rsyslogd restart` to restart the service and check if that worked for you.

 

If you want to make that persistent you need to write those changes on each unraid start. For this edit `/boot/config/go` and add the following lines to it:


echo ":msg,contains,\"error: mdcmd, 2640: Invalid argument (22): write\" stop" >> /etc/rsyslog.d/01-blocklist.conf
echo ":msg,contains,\"ERROR: (0x03:0x0101)\" stop" >> /etc/rsyslog.d/01-blocklist.conf

 

I tried what you said but it's not working for me.  I'm able to edit the /etc/rsyslog.d/01-blocklist.conf` file and make the change persistent by adding the line to the GO file as well, and verified the changes are persistent after reboot, however I'm still getting the same error messages.  I'm running unRAID version 6.7.2...any idea why the error messages would still be appearing despite successfully adding the rsyslogd filters?

 

EDIT: it appears to be filtering properly but only for the Syslog Server log...not for just the normal syslog that you access by clicking the icon on the unRAID GUI toolbar.  Is there some way to filter them there as well?

 

Edited by tallguydirk
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.