sparklyballs Posted March 1, 2015 Share Posted March 1, 2015 Attached syslog, the block i'm really concerned about is begins at 1 mar 3:10 am. my scheduled parity check is running (began at 1 mar 00:00) and i was in a thread about stalls, so i thought i'd look at the log and see if there was anything like in mine when i saw some nasty sounding errors at the timestamp above. syslog.zip Link to comment
WeeboTech Posted March 1, 2015 Share Posted March 1, 2015 Feb 28 16:49:24 Unraid-Nas kernel: ata3.00: ATA-9: WDC WD40EZRX-00SPEB0, WD-WCC4ERYTPDHL, 80.00A80, max UDMA/133 Feb 28 16:49:24 Unraid-Nas kernel: ata3.00: 7814037168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA ... Feb 28 16:49:31 Unraid-Nas emhttp: WDC_WD40EZRX-00SPEB0_WD-WCC4ERYTPDHL (sdd) 3907018584 ... Mar 1 03:10:50 Unraid-Nas kernel: ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x280100 action 0x6 frozen Mar 1 03:10:50 Unraid-Nas kernel: ata3.00: irq_stat 0x08000000, interface fatal error Mar 1 03:10:50 Unraid-Nas kernel: ata3: SError: { UnrecovData 10B8B BadCRC } Mar 1 03:10:50 Unraid-Nas kernel: ata3.00: failed command: READ DMA EXT Mar 1 03:10:50 Unraid-Nas kernel: ata3.00: cmd 25/00:60:f8:14:ca/00:00:54:00:00/e0 tag 7 dma 49152 in Mar 1 03:10:50 Unraid-Nas kernel: res 50/00:00:f7:14:ca/00:00:54:00:00/e0 Emask 0x10 (ATA bus error) Mar 1 03:10:50 Unraid-Nas kernel: ata3.00: status: { DRDY } Mar 1 03:10:50 Unraid-Nas kernel: ata3: hard resetting link Mar 1 03:10:51 Unraid-Nas kernel: ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Mar 1 03:10:51 Unraid-Nas kernel: ata3.00: configured for UDMA/133 Mar 1 03:10:51 Unraid-Nas kernel: ata3: EH complete Mar 1 03:10:51 Unraid-Nas kernel: ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x280100 action 0x6 frozen Mar 1 03:10:51 Unraid-Nas kernel: ata3.00: irq_stat 0x08000000, interface fatal error Mar 1 03:10:51 Unraid-Nas kernel: ata3: SError: { UnrecovData 10B8B BadCRC } Mar 1 03:10:51 Unraid-Nas kernel: ata3.00: failed command: READ DMA EXT Mar 1 03:10:51 Unraid-Nas kernel: ata3.00: cmd 25/00:60:f8:14:ca/00:00:54:00:00/e0 tag 8 dma 49152 in Mar 1 03:10:51 Unraid-Nas kernel: res 50/00:02:00:00:00/00:00:00:00:00/a0 Emask 0x10 (ATA bus error) Mar 1 03:10:51 Unraid-Nas kernel: ata3.00: status: { DRDY } Mar 1 03:10:51 Unraid-Nas kernel: ata3: hard resetting link Mar 1 03:10:51 Unraid-Nas kernel: ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Mar 1 03:10:51 Unraid-Nas kernel: ata3.00: configured for UDMA/133 Mar 1 03:10:51 Unraid-Nas kernel: ata3: EH complete Mar 1 03:11:39 Unraid-Nas kernel: ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x280100 action 0x6 frozen Mar 1 03:11:39 Unraid-Nas kernel: ata3.00: irq_stat 0x08000000, interface fatal error Mar 1 03:11:39 Unraid-Nas kernel: ata3: SError: { UnrecovData 10B8B BadCRC } Mar 1 03:11:39 Unraid-Nas kernel: ata3.00: failed command: READ DMA EXT Mar 1 03:11:39 Unraid-Nas kernel: ata3.00: cmd 25/00:30:f0:7b:4d/00:00:55:00:00/e0 tag 12 dma 24576 in Mar 1 03:11:39 Unraid-Nas kernel: res 50/00:00:ef:7b:4d/00:00:55:00:00/e0 Emask 0x10 (ATA bus error) Mar 1 03:11:39 Unraid-Nas kernel: ata3.00: status: { DRDY } Mar 1 03:11:39 Unraid-Nas kernel: ata3: hard resetting link Mar 1 03:11:40 Unraid-Nas kernel: ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Mar 1 03:11:40 Unraid-Nas kernel: ata3.00: configured for UDMA/133 Mar 1 03:11:40 Unraid-Nas kernel: ata3: EH complete Mar 1 03:12:46 Unraid-Nas kernel: ata3: limiting SATA link speed to 3.0 Gbps Mar 1 03:12:46 Unraid-Nas kernel: ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x280100 action 0x6 frozen Mar 1 03:12:46 Unraid-Nas kernel: ata3.00: irq_stat 0x08000000, interface fatal error Mar 1 03:12:46 Unraid-Nas kernel: ata3: SError: { UnrecovData 10B8B BadCRC } Mar 1 03:12:46 Unraid-Nas kernel: ata3.00: failed command: READ DMA EXT Mar 1 03:12:46 Unraid-Nas kernel: ata3.00: cmd 25/00:30:48:5f:01/00:00:56:00:00/e0 tag 0 dma 24576 in Mar 1 03:12:46 Unraid-Nas kernel: res 50/00:00:47:5f:01/00:00:56:00:00/e0 Emask 0x10 (ATA bus error) Mar 1 03:12:46 Unraid-Nas kernel: ata3.00: status: { DRDY } Mar 1 03:12:46 Unraid-Nas kernel: ata3: hard resetting link Mar 1 03:12:46 Unraid-Nas kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 320) Mar 1 03:12:46 Unraid-Nas kernel: ata3.00: configured for UDMA/133 Mar 1 03:12:46 Unraid-Nas kernel: ata3: EH complete It could be an issue with the cable, but let's be sure. Attach a smartctl -a or smartctl -x report. You may want to do a long test, wait the recommended time and attach the report. Link to comment
sparklyballs Posted March 1, 2015 Author Share Posted March 1, 2015 smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.18.5-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: WDC WD40EZRX-00SPEB0 Serial Number: WD-WCC4ERYTPDHL LU WWN Device Id: 5 0014ee 26037daf5 Firmware Version: 80.00A80 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Mar 1 14:35:52 2015 GMT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (54000) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 540) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x7035) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 187 176 021 Pre-fail Always - 7650 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 659 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 3550 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 41 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 22 193 Load_Cycle_Count 0x0032 192 192 000 Old_age Always - 25796 194 Temperature_Celsius 0x0022 126 111 000 Old_age Always - 26 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 1 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 3549 - # 2 Short offline Aborted by host 10% 3549 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Link to comment
sparklyballs Posted March 1, 2015 Author Share Posted March 1, 2015 root@Unraid-Nas:~# smartctl -x /dev/sdd smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.18.5-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: WDC WD40EZRX-00SPEB0 Serial Number: WD-WCC4ERYTPDHL LU WWN Device Id: 5 0014ee 26037daf5 Firmware Version: 80.00A80 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Mar 1 14:37:49 2015 GMT SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM feature is: Unavailable Rd look-ahead is: Enabled Write cache is: Enabled ATA Security is: Disabled, frozen [sEC2] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (54000) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 540) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x7035) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0 3 Spin_Up_Time POS--K 187 176 021 - 7650 4 Start_Stop_Count -O--CK 100 100 000 - 659 5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0 7 Seek_Error_Rate -OSR-K 100 253 000 - 0 9 Power_On_Hours -O--CK 096 096 000 - 3551 10 Spin_Retry_Count -O--CK 100 100 000 - 0 11 Calibration_Retry_Count -O--CK 100 253 000 - 0 12 Power_Cycle_Count -O--CK 100 100 000 - 41 192 Power-Off_Retract_Count -O--CK 200 200 000 - 22 193 Load_Cycle_Count -O--CK 192 192 000 - 25796 194 Temperature_Celsius -O---K 127 111 000 - 25 196 Reallocated_Event_Count -O--CK 200 200 000 - 0 197 Current_Pending_Sector -O--CK 200 200 000 - 0 198 Offline_Uncorrectable ----CK 200 200 000 - 0 199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 1 200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x02 SL R/O 5 Comprehensive SMART error log 0x03 GPL R/O 6 Ext. Comprehensive SMART error log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 1 Extended self-test log 0x09 SL R/W 1 Selective self-test log 0x10 GPL R/O 1 NCQ Command Error log 0x11 GPL R/O 1 SATA Phy Event Counters 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xa0-0xa7 GPL,SL VS 16 Device vendor specific log 0xa8-0xb6 GPL,SL VS 1 Device vendor specific log 0xb7 GPL,SL VS 39 Device vendor specific log 0xbd GPL,SL VS 1 Device vendor specific log 0xc0 GPL,SL VS 1 Device vendor specific log 0xc1 GPL VS 93 Device vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer SMART Extended Comprehensive Error Log Version: 1 (6 sectors) No Errors Logged SMART Extended Self-test Log Version: 1 (1 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 3549 - # 2 Short offline Aborted by host 10% 3549 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Status Version: 3 SCT Version (vendor specific): 258 (0x0102) SCT Support Level: 1 Device State: Stand-by (1) Current Temperature: 25 Celsius Power Cycle Min/Max Temperature: 24/28 Celsius Lifetime Min/Max Temperature: 18/41 Celsius Under/Over Temperature Limit Count: 0/0 SCT Temperature History Version: 2 Temperature Sampling Period: 1 minute Temperature Logging Interval: 1 minute Min/Max recommended Temperature: 0/60 Celsius Min/Max Temperature Limit: -41/85 Celsius Temperature History Size (Index): 478 (381) Index Estimated Time Temperature Celsius 382 2015-03-01 06:40 34 *************** ... ..( 49 skipped). .. *************** 432 2015-03-01 07:30 34 *************** 433 2015-03-01 07:31 33 ************** ... ..( 11 skipped). .. ************** 445 2015-03-01 07:43 33 ************** 446 2015-03-01 07:44 34 *************** ... ..( 9 skipped). .. *************** 456 2015-03-01 07:54 34 *************** 457 2015-03-01 07:55 33 ************** 458 2015-03-01 07:56 33 ************** 459 2015-03-01 07:57 ? - 460 2015-03-01 07:58 24 ***** 461 2015-03-01 07:59 24 ***** 462 2015-03-01 08:00 25 ****** 463 2015-03-01 08:01 24 ***** 464 2015-03-01 08:02 24 ***** 465 2015-03-01 08:03 ? - 466 2015-03-01 08:04 25 ****** ... ..( 4 skipped). .. ****** 471 2015-03-01 08:09 25 ****** 472 2015-03-01 08:10 26 ******* ... ..( 7 skipped). .. ******* 2 2015-03-01 08:18 26 ******* 3 2015-03-01 08:19 27 ******** ... ..( 9 skipped). .. ******** 13 2015-03-01 08:29 27 ******** 14 2015-03-01 08:30 28 ********* ... ..( 21 skipped). .. ********* 36 2015-03-01 08:52 28 ********* 37 2015-03-01 08:53 27 ******** ... ..( 2 skipped). .. ******** 40 2015-03-01 08:56 27 ******** 41 2015-03-01 08:57 26 ******* ... ..( 10 skipped). .. ******* 52 2015-03-01 09:08 26 ******* 53 2015-03-01 09:09 25 ****** 54 2015-03-01 09:10 35 **************** ... ..( 17 skipped). .. **************** 72 2015-03-01 09:28 35 **************** 73 2015-03-01 09:29 36 ***************** ... ..( 68 skipped). .. ***************** 142 2015-03-01 10:38 36 ***************** 143 2015-03-01 10:39 35 **************** ... ..( 57 skipped). .. **************** 201 2015-03-01 11:37 35 **************** 202 2015-03-01 11:38 34 *************** ... ..( 31 skipped). .. *************** 234 2015-03-01 12:10 34 *************** 235 2015-03-01 12:11 35 **************** ... ..( 11 skipped). .. **************** 247 2015-03-01 12:23 35 **************** 248 2015-03-01 12:24 34 *************** ... ..(132 skipped). .. *************** 381 2015-03-01 14:37 34 *************** SCT Error Recovery Control command not supported Device Statistics (GP Log 0x04) not supported SATA Phy Event Counters (GP Log 0x11) ID Size Value Description 0x0001 2 0 Command failed due to ICRC error 0x0002 2 0 R_ERR response for data FIS 0x0003 2 0 R_ERR response for device-to-host data FIS 0x0004 2 0 R_ERR response for host-to-device data FIS 0x0005 2 0 R_ERR response for non-data FIS 0x0006 2 0 R_ERR response for device-to-host non-data FIS 0x0007 2 0 R_ERR response for host-to-device non-data FIS 0x0008 2 0 Device-to-host non-data FIS retries 0x0009 2 2 Transition from drive PhyRdy to drive PhyNRdy 0x000a 2 3 Device-to-host register FISes sent due to a COMRESET 0x000b 2 0 CRC errors within host-to-device FIS 0x000f 2 0 R_ERR response for host-to-device data FIS, CRC 0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC 0x8000 4 4208 Vendor specific Link to comment
sparklyballs Posted March 1, 2015 Author Share Posted March 1, 2015 The abort in the smartctl history was me getting click silly. Link to comment
sparklyballs Posted March 1, 2015 Author Share Posted March 1, 2015 UGH !!!! that's annoying, i initiated the extended test through the webui and the damn disk spun down after 30 minutes and killed the test. Link to comment
WeeboTech Posted March 1, 2015 Share Posted March 1, 2015 UGH !!!! that's annoying, i initiated the extended test through the webui and the damn disk spun down after 30 minutes and killed the test. Disable the spin down timer temporarily. It's always good to have a mark in the smart logs for a long history. From what I can see it's a data transfer error over the wire from drive to motherboard. Change, reposition and/or reseat the cable to the drive. make sure it is a tight fit. If it continues to occur change it, If it continues to occur there, it may be the controller on the drive itself. I didn't see anything jump out about surface defects. Link to comment
sparklyballs Posted March 2, 2015 Author Share Posted March 2, 2015 UGH !!!! that's annoying, i initiated the extended test through the webui and the damn disk spun down after 30 minutes and killed the test. Disable the spin down timer temporarily. It's always good to have a mark in the smart logs for a long history. From what I can see it's a data transfer error over the wire from drive to motherboard. Change, reposition and/or reseat the cable to the drive. make sure it is a tight fit. If it continues to occur change it, If it continues to occur there, it may be the controller on the drive itself. I didn't see anything jump out about surface defects. I ran the long test yesterday and it passed, i should add that prior to that i powered down and reaseated cables and drives bays for the whole server. Also i loaded up graylog server on a remote machine and routed log output from unraid to it. I've configured an alert to email me if this happens again. Link to comment
Squid Posted March 2, 2015 Share Posted March 2, 2015 Also i loaded up graylog server on a remote machine and routed log output from unraid to it. I've configured an alert to email me if this happens again. Awesome idea. Now that's going to be my next project - configure bash script to do the same and alert the user via the notification system. Link to comment
bonienl Posted March 2, 2015 Share Posted March 2, 2015 Also i loaded up graylog server on a remote machine and routed log output from unraid to it. I've configured an alert to email me if this happens again. Awesome idea. Now that's going to be my next project - configure bash script to do the same and alert the user via the notification system. The 'notify' script can be used to generate notifications, this should be the standard approach to work with notifications, below the help notify [-e "event"] [-s "subject"] [-d "description"] [-i "normal|warning|alert"] [-m "message"] [-x] [-t] [add] create a notification use -e to specify the event use -s to specify a subject use -d to specify a short description use -i to specify the severity use -m to specify a message (long description) use -x to create a single notification ticket use -t to force send email only (for testing) all options are optional notify init Initialize the notification subsystem. notify smtp-init Initialize sendmail configuration (ssmtp in our case). notify get Output a json-encoded list of all the unread notifications. notify archive file Move file from 'unread' state to 'archive' state. Link to comment
sparklyballs Posted March 2, 2015 Author Share Posted March 2, 2015 Also i loaded up graylog server on a remote machine and routed log output from unraid to it. I've configured an alert to email me if this happens again. Awesome idea. Now that's going to be my next project - configure bash script to do the same and alert the user via the notification system. The 'notify' script can be used to generate notifications, this should be the standard approach to work with notifications, below the help notify [-e "event"] [-s "subject"] [-d "description"] [-i "normal|warning|alert"] [-m "message"] [-x] [-t] [add] create a notification use -e to specify the event use -s to specify a subject use -d to specify a short description use -i to specify the severity use -m to specify a message (long description) use -x to create a single notification ticket use -t to force send email only (for testing) all options are optional notify init Initialize the notification subsystem. notify smtp-init Initialize sendmail configuration (ssmtp in our case). notify get Output a json-encoded list of all the unread notifications. notify archive file Move file from 'unread' state to 'archive' state. on the subject of notifications in general, i've only attempted it once and i was wondering whether it was a fluke or something that needs looking at. in the notifications settings pane i selected somewhere on my array to store the notifications and my system wouldn't reboot with it set like that. the log is somewhere on one of the annoucement threads, i'll go fetch it once i've found it. from here and the subsequent post. http://lime-technology.com/forum/index.php?topic=38409.msg356877#msg356877 Link to comment
itimpi Posted March 2, 2015 Share Posted March 2, 2015 in the notifications settings pane i selected somewhere on my array to store the notifications and my system wouldn't reboot with it set like that. the log is somewhere on one of the annoucement threads, i'll go fetch it once i've found it. Maybe it should not be possible to store notifications on the array or cache disk? I could see that causing problems as the system may want to generate notifications when they are not available. I would suggest that if you want them to be persistent then store them on the flash drive (/boot) is the obvious choice. The default is a RAM location which means they do not survive a reboot. Link to comment
bonienl Posted March 2, 2015 Share Posted March 2, 2015 Yeah, there can be a race condition when notifications are stored on the array, they may be generated before the array is ready or when the array goes off-line. The default setting is indeed a RAM location, which won't survive a system reboot as mentioned. Not sure yet what is wisdom here ... Link to comment
sparklyballs Posted March 2, 2015 Author Share Posted March 2, 2015 I don't know how many people would make use of it, but it would be nice to have the option in the gui hidden somewhere to output to a remote syslog. Link to comment
WeeboTech Posted March 2, 2015 Share Posted March 2, 2015 I don't know how many people would make use of it, but it would be nice to have the option in the gui hidden somewhere to output to a remote syslog. You can drop rsyslog .conf files in /etc/rsyslog.d directory. One use is for duplicate debug logging, another would be to send to a remote log server. Another alternative might be to pipe output to live process that takes certain messages and turns them into notifications. Link to comment
Squid Posted March 3, 2015 Share Posted March 3, 2015 I don't know how many people would make use of it, but it would be nice to have the option in the gui hidden somewhere to output to a remote syslog. You can drop rsyslog .conf files in /etc/rsyslog.d directory. One use is for duplicate debug logging, another would be to send to a remote log server. Another alternative might be to pipe output to live process that takes certain messages and turns them into notifications. Something like this: #!/bin/bash while : do tail -f -n 1 /var/log/syslog | while read LOG do if ( echo "$LOG" | egrep -i kernel | egrep -i error | egrep -v "error, sector=" > /dev/null 2>&1 ) || ( echo "$LOG" | egrep -i rsync | egrep -i error > /dev/null 2>&1 ) || ( echo "$LOG" | egrep -i "oom-killer" > /dev/nul$ then # Theres an error... Sleep for 10 seconds to try and get more information into the syslog before sending notification sleep 10 if echo "$LOG" | egrep -i kernel > /dev/null 2>&1 then ERROR_TYPE="Kernel Error" IMPORTANCE="alert" fi if echo "$LOG" | egrep -i rsync > /dev/null 2>&1 then ERROR_TYPE="Rsync (Mover) Error" IMPORTANCE="warning" fi if echo "$LOG" | egrep -i "oom-killer" > /dev/null 2>&1 then ERROR_TYPE="Out Of Memory Error" IMPORTANCE="alert" fi echo "$ERROR_TYPE Detected" > /tmp/emailbody.txt echo "" >> /tmp/emailbody.txt IPADDRESS=$(egrep -i ipaddr /var/local/emhttp/var.ini | sed 's/^[^"]*"\([^"]*\)".*/\1/') echo "To View complete syslog, click here: http://$IPADDRESS/Tools/Syslog" >> /tmp/emailbody.txt echo "" >> /tmp/emailbody.txt echo "Last 50 lines of syslog:" >>/tmp/emailbody.txt echo "" >> /tmp/emailbody.txt tail -n 50 /var/log/syslog >> /tmp/emailbody.txt notify -e "$ERROR_TYPE Detected" -s "$ERROR_TYPE" -d "$LOG" -i "$IMPORTANCE" -m "$(cat /tmp/emailbody.txt)" # alert sent out. Sleep for 10 minutes so that we're not innundating the user with multiple alerts sleep 600 # break out of inner loop so that any errors during the 10 minutes are ignored break fi done done Keeps scanning syslog for errors output by the kernel (excluding disk read errors output by md - dynamix should handle that I would think) (the errors listed in the OP would also be caught) , rsync errors (probably caused by mover), and oom-killer. After it detects an error, it waits for 10 seconds to let more information hit the syslog, then creates and sends out a notification, then pauses for 10 minutes so that the user doesn't get inundated with multiple errors. The email sent out has the last 50 lines of the syslog, and also a link to see the full syslog (only on the local network of course) Link to comment
sparklyballs Posted March 3, 2015 Author Share Posted March 3, 2015 Been having a play with the graylog install i set up on a remote server. I like the dashboard feature.. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.