Julez Posted December 25, 2013 Share Posted December 25, 2013 I've tried both methods of just using cp -R and also midnight commander to copy cache drive contents to an array drive temporarily mount the new cache drive and do the same copy procedure back to the new cache drive. Unfortunately every time I try to copy to the new drive I get read-only file system errors for several of the apps folders I have there such as Plex when trying to create the directories. I also get a few Input/Output errors for a few folders as well. I know I'm doing something wrong, but not sure what...I followed the midnight commander option step by step even when I tried that. Quote Link to comment
dgaschk Posted December 26, 2013 Share Posted December 26, 2013 Copy a terminal session showing the behavior. Enter "date" on the terminal and attach the syslog. Quote Link to comment
dirtysanchez Posted December 26, 2013 Share Posted December 26, 2013 Have you stopped all of your plugins before attempting these copies? Quote Link to comment
Julez Posted December 26, 2013 Author Share Posted December 26, 2013 Yeah any of those plugins using that cache drive should have been stopped. I actually removed all the plg files before I did the copy so they weren't even there to load on reboot. I'll try to get the terminal session here soon. Although I've been having some strange issues in general that I thought had been resolved from an earlier post where I had created a new config. All drives come back without any SMART errors every time I test them. But out of the blue I had a drive red ball during a parity check today. I also get some strange issues right now where when I stop the array it'll show all array drives as missing. I have to reboot the array in order for them to show back up. Because I've yet to see anything wrong with the actual drives that show up missing or are red balled etc... I did run reiserfsck on /dev/md1 and /dev/md2 yesterday. No problems on /dev/md2, but it wanted me to run reiserfsk --rebuild-tree on /dev/md1. I did that last night as well. Same issues occur though with the missing array drive. I guess that's what I get for dinking with a perfectly working array... Quote Link to comment
Julez Posted December 26, 2013 Author Share Posted December 26, 2013 SMART results from redballed drive: Statistics for /dev/sde ST3000DM001-1CH166_W1F1QC5N smartctl -a -d ata /dev/sde smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright © 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST3000DM001-1CH166 Serial Number: W1F1QC5N LU WWN Device Id: 5 000c50 05da2a276 Firmware Version: CC24 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Thu Dec 26 14:11:53 2013 CST ==> WARNING: A firmware update for this drive may be available, see the following Seagate web pages: http://knowledge.seagate.com/articles/en_US/FAQ/207931en http://knowledge.seagate.com/articles/en_US/FAQ/223651en SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 584) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 345) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x3085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 117 099 006 Pre-fail Always - 166684160 3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 306 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 064 060 030 Pre-fail Always - 3028077 9 Power_On_Hours 0x0032 093 093 000 Old_age Always - 6263 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 15 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 074 062 045 Old_age Always - 26 (Min/Max 21/30) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 10 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1808 194 Temperature_Celsius 0x0022 026 040 000 Old_age Always - 26 (0 19 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 306h+46m+12.040s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 8897003104 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 121752901140 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 6261 - # 2 Extended offline Interrupted (host reset) 00% 6261 - # 3 Short offline Completed without error 00% 259 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Quote Link to comment
Julez Posted December 26, 2013 Author Share Posted December 26, 2013 I'll post the copy logs etc... as soon as this parity rebuild is finished. Quote Link to comment
Julez Posted December 27, 2013 Author Share Posted December 27, 2013 So I did the copy once again the problem I was having before did not occur...instead the whole thing went south this time... Please note that I had just restarted the system just to make sure it was clear and ready to go. putty.zip syslog12262013.txt Quote Link to comment
Julez Posted December 27, 2013 Author Share Posted December 27, 2013 Doing a parity rebuild AGAIN last night because the drives showed up missing after the previous logs above. Drives came back on reboot, let parity rebuild overnight. This was the result half way through the night, it seems to be getting progressively worse. syslog12272013.zip Quote Link to comment
Julez Posted December 28, 2013 Author Share Posted December 28, 2013 So again I tried to rebuild the array. I reseated everything in the array and even took out the new card I had just put in, so the only thing different than before was that the SSD was still there as the cache drive rather than my old spinning disk. Although I also swapped out the SAS>SATA cable just to rule that out. The array was rebuilding and AGAIN a random drive redballed. This time it was the parity drive. Each drive in the array has redballed at some point, either during a parity rebuild, or after a data copy or something. Before letting it finish, I saw the parity rebuild was up to 60%, so I'm not sure where it failed. So I stopped the array and again all disks come back missing/invalid. Quote Link to comment
dgaschk Posted December 28, 2013 Share Posted December 28, 2013 What model PSU? Quote Link to comment
Julez Posted December 28, 2013 Author Share Posted December 28, 2013 700 Watt Thermaltake Toughpower http://www.newegg.com/Product/Product.aspx?Item=N82E16817153039 It does have some age on it though... Quote Link to comment
dgaschk Posted December 28, 2013 Share Posted December 28, 2013 18A for the disks is likely the issue. Find a good single rail PSU. See here: http://lime-technology.com/forum/index.php?topic=12219.0 Quote Link to comment
Julez Posted December 29, 2013 Author Share Posted December 29, 2013 Thanks for the help. Makes sense since I had added another PCIE card and the SSD when it started to fail. Prior to that it was just 5 7200rpm drives. And 2 USB drives. Plus a couple fans. One other thing I'm not sure if this is just an UnRAID bug currently or what, but when drives are spun down sometimes (also doesn't seem to happen all the time) the blinking green ball shows up as a broken image. Quote Link to comment
Julez Posted January 18, 2014 Author Share Posted January 18, 2014 Well I replaced the PSU with a good Seasonic variant. Updated all firmware on my stuff while I was at it. Updated to ESXi 5.5. Recreated the VM for UnRAID. Bought a new SuperMicro drive cage. Placed my 3 array drives into the new cage with brand new locking type SATA cables from my AOC-SAS2LP-MV8 Placed my 1 SSD cache drive in the old cage with the same cables connected to the AOC-SAS2LP-MV8 Placed my 2 VMFS datastore drive mirror in the old cage with the cache drive, but connected to my LSI SAS 9211-4 Rebuilt my UnRAID flash drive from scratch. Everything was going fine with no issues right up until I attempted to install CouchPotato again. It installed properly out of the gate. But upon a restart the plugin began to fail. I would restart the plugin and the plugin config on the unraid gui would lose the port number and data directory and put what was in the data directory into the install directory field. I need to get a syslog server going on my Synology or something since my syslogs get killed after a reboot. I've attached just a random syslog since the drive was red balled. After I removed the plugin, the next reboot resulted in all array drives missing again. And one more reboot resulted in a redballed drive, which I've ran SMART tests on and has results similar to what I posted earlier. Please help I'm nearly ready to jump ship...the sheer amount of time, effort and additional expense I've put in to this since my original failure above is insane. syslog1.zip Quote Link to comment
Julez Posted January 18, 2014 Author Share Posted January 18, 2014 This log was shortly after issuing a new config with just the two data drives without the parity mounted. I stopped the array and all disks were then missing. syslog.zip Quote Link to comment
Julez Posted January 18, 2014 Author Share Posted January 18, 2014 Probably the best log. I wiped out my flash drive again and fired up my syslog server on the synology and added the call to point to that via the go script on the UnRAID system. synosyslog.zip Quote Link to comment
dgaschk Posted January 18, 2014 Share Posted January 18, 2014 The last syslog is not helpful. Need the entire syslog in text format please. Quote Link to comment
Julez Posted January 18, 2014 Author Share Posted January 18, 2014 Ok I'll have to come up with another syslog server or something, that's the only export I can get off the synology's server. Quote Link to comment
dgaschk Posted January 18, 2014 Share Posted January 18, 2014 You can write it to a data disk. Set a cron job to trim occasionally. Quote Link to comment
Julez Posted January 19, 2014 Author Share Posted January 19, 2014 So I rebuilt the flash drive again and started from scratch. I was in the process of doing the new drive config, the only addon I had was unmenu. I was going to remove the directories for my apps on the SSD cache drive from my previous install since there really shouldn't have been anything in them. I was doing this while the parity check was running. At this point it started to become unstable. This log should be a little better for you. I'm starting to wonder if there's some issue with my MV8 or something. https://dl.dropboxusercontent.com/u/6473711/syslog1.zip Quote Link to comment
dgaschk Posted January 19, 2014 Share Posted January 19, 2014 Check for BIOS and firmware updates. See Check FileSystems in my sig to fix reiserfs errors unless you blank the disks. Quote Link to comment
Julez Posted January 19, 2014 Author Share Posted January 19, 2014 Bios and firmware were updated to latest and greatest on all cards and the motherboard about a week ago. I'll check the file systems. Quote Link to comment
Julez Posted January 19, 2014 Author Share Posted January 19, 2014 Ok so Disk 1 and Disk 2 of my array had no issues after doing the file system check. One thing I just realized I failed to do when adding the new cache drive though was to preclear that new SSD...so I'm actually doing that now. Since I cannot do a file system check on the parity drive. Just to be safe should I do anything with this drive? Quote Link to comment
dgaschk Posted January 20, 2014 Share Posted January 20, 2014 Do not pre-clear a SSD. There is no benefit. Quote Link to comment
Julez Posted January 20, 2014 Author Share Posted January 20, 2014 Ok, I went ahead and formatted the drive regardless. This evening I'll readd the cache drive and the parity drive and let the parity rebuild. All 3 array drives should be good to go by the SMART test as well as the file system check. The same can be said about the SSD cache drive. All firmware I've confirmed is updated yesterday evening. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.