March 2, 201214 yr Well, I've been having some *weird* issues lately. Drives showing up with write or read errors in the syslog and then I unmount and run reiserfsck which tells me there are no errors at all. I can initconfig and add the drives back to the array and they work fine, then, another day, another drive at a different point has the same error. It's culminated now in my disk12 showing as "unformatted". When I run reiserfsck, it does the semantic pass and show all the data on the disk, but unraid sees it as unformatted. I browsed the forum and tried the "unraid_partition_utility.sh" script (great util!) and it told me the partition was on 64 not 63. I followed the further instructions and "corrected" the partition. Then I ran reiserfsck --rebuild-tree (as --check told me to) and just for kicks, after it completed, I ran the util again. To my surprise it told me it was not suitable for unraid? It's like the partition changes were reverted somehow. Anyway, now it's running as "unformatted" and the array (after a reboot) is re-building parity. I think I'm pooched on this drive having any hope of recovering the data, should I just format it? Or should I get a new drive and "replace" it in the hopes it will rebuild disk12? Or is the current parity check wiping all that clean? I'm starting to give up... it used to be so great and now there are near daily errors. Any thoughts, oh great gurus? I will attempt to clarify anything (which I'm sure I left) lacking and am more than willing to try whatever is suggested. thanks in advance for any help you may have... Worried in Windsor. SYSLOG ATTACHED. syslog.zip
March 3, 201214 yr Author *BUMP* Anyone? I'm scared to do anything with this in case I lose the data that was/is/may be still on the drive... Should I stop the array, remove the disk and attach it to my windows machine to try to see what I can see? Km.
March 3, 201214 yr Author unRAID Version: unRAID Server Pro, Version 5.0-beta13 Motherboard: ASUSTeK - M5A97 Processor: AMD AthlonTM II X2 240 - 2.8 GHz Cache: L1 = 256 KB L2 = 2048 KB Memory: 5767169 kB - DIMM3 = 1333 MHz (0.8 ns) The power supply is a Thermaltake 750.
March 3, 201214 yr Are you absolutely certain the file system starts on sector 64? OK, let's try something to see if we can figure out where the file-system actually starts. Mar 1 12:00:08 Angband kernel: md: import disk12: [8,128] (sdi) WDC_WD20EARX-00PASB0_WD-WCAZAD207818 size: 1953514552 Assuming this line is telling the truth, disk12 is /dev/sdi type the following command on your server: dd if=/dev/sdi count=195 | od -c -A d | sed 30q The output should look a lot like this... Note where the string "R e I s E r 2 F s" appears.... Let's see if yours is at the same address, or one sector (512 bytes) further on sector 64. Output on one of my drives looks like this (it has a file-system starting on sector 63): 195+0 records in 195+0 records out 99840 bytes (100 kB) copied, 0.00169403 s, 58.9 MB/s 0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0000448 \0 \0 203 \0 \0 \0 ? \0 \0 \0 361 _ 8 : \0 \0 0000464 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0000496 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 U 252 0000512 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0097792 376 \v G \a e \a - \0 \a 212 267 001 022 \0 \0 \0 0097808 \0 \0 \0 \0 \0 \0 \0 \0 004 \0 \0 % 254 227 \ 0097824 204 003 \0 \0 036 \0 \0 \0 \0 \0 \0 \0 \0 020 314 003 0097840 220 \0 002 \0 R e I s E r 2 F s \0 \0 \0 0097856 003 \0 \0 \0 005 \0 217 016 002 \0 \0 \0 204 ] \0 \0 0097872 001 \0 \0 \0 353 300 256 263 242 347 N 347 264 362 315 364 0097888 345 V 253 366 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0097904 \0 \0 \0 \0 \a \0 036 \0 255 262 303 M \0 N 355 \0 0097920 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
March 4, 201214 yr Author Ok, now I'm stumped... apparently you can't get to the drive anymore. It was /dev/sdi in that syslog, but it's showing up as /dev/sdj now. whne I run that command I get the following: root@Angband:~# dd if=/dev/sdj count=195 | od -c -A d | sed 30q dd: reading `/dev/sdj': Input/output error 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000484026 s, 0.0 kB/s 0000000 root@Angband:~# ... and my syslog has the following: Mar 3 20:13:31 Angband kernel: Buffer I/O error on device sdj, logical block 0 So, assuming this means the disk is pooched, is it simply a matter of pulling it out and replacing it with a new disk (which will then rebuild the new drive from the "protected" data, ie, is it in "simultation" mode now?) or should I accept that the disk and it's contents are gone and move on Thanks for the assistance and answers, I appreciate it.
March 5, 201214 yr Replace the failed drive and assign the replacement to the disk12 slot. Disk12 will be rebuilt. It should be in simulation mode currently and a second failure will result in data loss. The sdX value does change between boots.
March 5, 201214 yr Author Replace the failed drive and assign the replacement to the disk12 slot. Disk12 will be rebuilt. It should be in simulation mode currently and a second failure will result in data loss. The sdX value does change between boots. Now I think there are more serious problems... take a look at the syslog now.... WTF? syslog-2012-03-04.zip
March 5, 201214 yr Mar 2 12:52:37 Angband vsftpd[16509]: connect from 173.193.6.215 (173.193.6.215) Mar 2 12:52:38 Angband vsftpd[16509]: [Administrator] FAIL LOGIN: Client "173.193.6.215" Mar 2 12:52:41 Angband last message repeated 2 times Mar 2 12:52:42 Angband vsftpd[16541]: connect from 173.193.6.215 (173.193.6.215) Mar 2 12:52:42 Angband vsftpd[16541]: [Administrator] FAIL LOGIN: Client "173.193.6.215" Mar 4 03:23:58 Angband vsftpd[17389]: connect from 136.199.199.81 (136.199.199.81) Mar 4 03:23:58 Angband vsftpd[17389]: [Administrator] FAIL LOGIN: Client "136.199.199.81" Mar 4 03:24:01 Angband last message repeated 2 times These appear to be break-in attempts. UnRAID is not secure. It must be behind a firewall! Mar 4 21:18:58 Angband kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 21312 does not match to the expected one 1 Mar 4 21:18:58 Angband kernel: REISERFS error (device md11): vs-5150 search_by_key: invalid format found in block 331939852. Fsck? This is a file system error. See here: http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems
March 5, 201214 yr Author Mar 2 12:52:37 Angband vsftpd[16509]: connect from 173.193.6.215 (173.193.6.215) Mar 2 12:52:38 Angband vsftpd[16509]: [Administrator] FAIL LOGIN: Client "173.193.6.215" Mar 2 12:52:41 Angband last message repeated 2 times Mar 2 12:52:42 Angband vsftpd[16541]: connect from 173.193.6.215 (173.193.6.215) Mar 2 12:52:42 Angband vsftpd[16541]: [Administrator] FAIL LOGIN: Client "173.193.6.215" Mar 4 03:23:58 Angband vsftpd[17389]: connect from 136.199.199.81 (136.199.199.81) Mar 4 03:23:58 Angband vsftpd[17389]: [Administrator] FAIL LOGIN: Client "136.199.199.81" Mar 4 03:24:01 Angband last message repeated 2 times These appear to be break-in attempts. UnRAID is not secure. It must be behind a firewall! Mar 4 21:18:58 Angband kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 21312 does not match to the expected one 1 Mar 4 21:18:58 Angband kernel: REISERFS error (device md11): vs-5150 search_by_key: invalid format found in block 331939852. Fsck? This is a file system error. See here: http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems As far as the FTP access goes, yeah, I had turned on FTP to try something out, it's normally completely firewalled and is now again. RE the file system errors, I figured I'd have to run reiserfsck. So are these not serious errors, like the disks are not failing imminently or anything? I mean I've had a few drives go "bad" in unraid, and only one of them was actually a bad drive.
March 5, 201214 yr Mar 4 21:16:54 Angband kernel: md: disk11 read error Mar 4 21:16:54 Angband kernel: handle_stripe read error: 2655518816/11, count: 1 Mar 4 21:16:54 Angband kernel: sd 7:0:2:0: [sdj] Device not ready Mar 4 21:16:54 Angband kernel: sd 7:0:2:0: [sdj] Result: hostbyte=0x00 driverbyte=0x08 Mar 4 21:16:54 Angband kernel: sd 7:0:2:0: [sdj] Sense Key : 0x2 [current] Mar 4 21:16:54 Angband kernel: sd 7:0:2:0: [sdj] ASC=0x4 ASCQ=0x2 Mar 4 21:16:54 Angband kernel: sd 7:0:2:0: [sdj] CDB: cdb[0]=0x28: 28 00 9e 48 00 9f 00 00 08 00 Mar 4 21:16:54 Angband kernel: end_request: I/O error, dev sdj, sector 2655518879 Mar 4 21:16:54 Angband kernel: sd 7:0:3:0: [sdk] Device not ready Mar 4 21:16:54 Angband kernel: sd 7:0:3:0: [sdk] Result: hostbyte=0x00 driverbyte=0x08 Mar 4 21:16:54 Angband kernel: sd 7:0:3:0: [sdk] Sense Key : 0x2 [current] Mar 4 21:16:54 Angband kernel: sd 7:0:3:0: [sdk] ASC=0x4 ASCQ=0x2 Mar 4 21:16:54 Angband kernel: sd 7:0:3:0: [sdk] CDB: cdb[0]=0x28: 28 00 9e 48 00 9f 00 00 08 00 Mar 4 21:16:54 Angband kernel: end_request: I/O error, dev sdk, sector 2655518879 Mar 4 21:16:54 Angband kernel: sd 7:0:4:0: [sdl] Device not ready Mar 4 21:16:54 Angband kernel: sd 7:0:4:0: [sdl] Result: hostbyte=0x00 driverbyte=0x08 Mar 4 21:16:54 Angband kernel: sd 7:0:4:0: [sdl] Sense Key : 0x2 [current] Mar 4 21:16:54 Angband kernel: sd 7:0:4:0: [sdl] ASC=0x4 ASCQ=0x2 Mar 4 21:16:54 Angband kernel: sd 7:0:4:0: [sdl] CDB: cdb[0]=0x28: 28 00 9e 48 00 9f 00 00 08 00 Mar 4 21:16:54 Angband kernel: end_request: I/O error, dev sdl, sector 2655518879 Mar 4 21:16:54 Angband kernel: md: disk2 read error Mar 4 21:16:54 Angband kernel: handle_stripe read error: 2655518816/2, count: 1 Mar 4 21:16:54 Angband kernel: md: disk8 read error Mar 4 21:16:54 Angband kernel: handle_stripe read error: 2655518816/8, count: 1 Mar 4 21:16:54 Angband kernel: md: disk12 read error Mar 4 21:16:54 Angband kernel: handle_stripe read error: 2655518816/12, count: 1 I just noticed multiple read errors. This usually indicates a power problem. Do these drives share a interface card?
March 5, 201214 yr Author Mar 4 21:16:54 Angband kernel: md: disk11 read error Mar 4 21:16:54 Angband kernel: handle_stripe read error: 2655518816/11, count: 1 Mar 4 21:16:54 Angband kernel: sd 7:0:2:0: [sdj] Device not ready Mar 4 21:16:54 Angband kernel: sd 7:0:2:0: [sdj] Result: hostbyte=0x00 driverbyte=0x08 Mar 4 21:16:54 Angband kernel: sd 7:0:2:0: [sdj] Sense Key : 0x2 [current] Mar 4 21:16:54 Angband kernel: sd 7:0:2:0: [sdj] ASC=0x4 ASCQ=0x2 Mar 4 21:16:54 Angband kernel: sd 7:0:2:0: [sdj] CDB: cdb[0]=0x28: 28 00 9e 48 00 9f 00 00 08 00 Mar 4 21:16:54 Angband kernel: end_request: I/O error, dev sdj, sector 2655518879 Mar 4 21:16:54 Angband kernel: sd 7:0:3:0: [sdk] Device not ready Mar 4 21:16:54 Angband kernel: sd 7:0:3:0: [sdk] Result: hostbyte=0x00 driverbyte=0x08 Mar 4 21:16:54 Angband kernel: sd 7:0:3:0: [sdk] Sense Key : 0x2 [current] Mar 4 21:16:54 Angband kernel: sd 7:0:3:0: [sdk] ASC=0x4 ASCQ=0x2 Mar 4 21:16:54 Angband kernel: sd 7:0:3:0: [sdk] CDB: cdb[0]=0x28: 28 00 9e 48 00 9f 00 00 08 00 Mar 4 21:16:54 Angband kernel: end_request: I/O error, dev sdk, sector 2655518879 Mar 4 21:16:54 Angband kernel: sd 7:0:4:0: [sdl] Device not ready Mar 4 21:16:54 Angband kernel: sd 7:0:4:0: [sdl] Result: hostbyte=0x00 driverbyte=0x08 Mar 4 21:16:54 Angband kernel: sd 7:0:4:0: [sdl] Sense Key : 0x2 [current] Mar 4 21:16:54 Angband kernel: sd 7:0:4:0: [sdl] ASC=0x4 ASCQ=0x2 Mar 4 21:16:54 Angband kernel: sd 7:0:4:0: [sdl] CDB: cdb[0]=0x28: 28 00 9e 48 00 9f 00 00 08 00 Mar 4 21:16:54 Angband kernel: end_request: I/O error, dev sdl, sector 2655518879 Mar 4 21:16:54 Angband kernel: md: disk2 read error Mar 4 21:16:54 Angband kernel: handle_stripe read error: 2655518816/2, count: 1 Mar 4 21:16:54 Angband kernel: md: disk8 read error Mar 4 21:16:54 Angband kernel: handle_stripe read error: 2655518816/8, count: 1 Mar 4 21:16:54 Angband kernel: md: disk12 read error Mar 4 21:16:54 Angband kernel: handle_stripe read error: 2655518816/12, count: 1 I just noticed multiple read errors. This usually indicates a power problem. Do these drives share a interface card? I have two Supermicro cards that supply 8 drives each and then there's the integrated 6-port SATA on the mobo. I'm not sure which drives are on which controllers... power issues, I have a Thermaltake 750W PSU. It has 56A on on rail for 12v, I thought that was enough for many drives? I supposed I could "stagger" the drives across the controllers but swapping some cables around?
March 5, 201214 yr It may be a problem with the interface card. Do disks 2, 8, 11, and 12 share an interface card?
March 5, 201214 yr Author It may be a problem with the interface card. Do disks 2, 8, 11, and 12 share an interface card? I'm not sure, will have to down the server and investigate later today...
March 8, 201214 yr Author Ok, so I replaced my disk 12 with a brand new disk. Here's the procedure I followed: 1) Stop Array 2) UnAssign old Disk from slot 12 3) Powerdown 4) Install new disk 5) Power on 6) Preclear new disk 7) Stop Array Assign new disk to "empty" slot 12 9) Allow the Data Rebuild Now, after the data rebuild is complete, it is showing the new disk 12 as "unformatted"?!?! I would have formatted it first, only it never asked me before the data-rebuild started. Not sure what to do now... Screenshot: http://imm.io/igFm Sys log attached. syslog-2012-03-08.zip
March 8, 201214 yr Ok, so I replaced my disk 12 with a brand new disk. Here's the procedure I followed: 1) Stop Array 2) UnAssign old Disk from slot 12 3) Powerdown 4) Install new disk 5) Power on 6) Preclear new disk 7) Stop Array Assign new disk to "empty" slot 12 9) Allow the Data Rebuild Now, after the data rebuild is complete, it is showing the new disk 12 as "unformatted"?!?! I would have formatted it first, only it never asked me before the data-rebuild started. Not sure what to do now... Screenshot: http://imm.io/igFm Sys log attached. DO NOT FORMAT IT. If it re-constructed it,, it should have re-constructed the formatting as well. You should not have had to do anything to format it. Contact Lime-Tech for guidance.
March 8, 201214 yr According to this in the syslog, Mar 8 13:54:58 Angband logger: mount: wrong fs type, bad option, bad superblock on /dev/md12, Mar 8 13:54:58 Angband logger: missing codepage or helper program, or other error Mar 8 13:54:58 Angband logger: In some cases useful info is found in syslog - try Mar 8 13:54:58 Angband logger: dmesg | tail or so Mar 8 13:54:58 Angband logger: it was unable to mount the disk... It might be as simple as running a file-system repair on the file-system on that disk. Or, it could be more. But since you are using the beta version of unRAID, best to get help from lime-tech. Send an e-mail to [email protected] Point him to this thread and your syslog. Joe L.
March 9, 201214 yr Author Thanks Joe; I've sent Tom an email. Km. According to this in the syslog, Mar 8 13:54:58 Angband logger: mount: wrong fs type, bad option, bad superblock on /dev/md12, Mar 8 13:54:58 Angband logger: missing codepage or helper program, or other error Mar 8 13:54:58 Angband logger: In some cases useful info is found in syslog - try Mar 8 13:54:58 Angband logger: dmesg | tail or so Mar 8 13:54:58 Angband logger: it was unable to mount the disk... It might be as simple as running a file-system repair on the file-system on that disk. Or, it could be more. But since you are using the beta version of unRAID, best to get help from lime-tech. Send an e-mail to [email protected] Point him to this thread and your syslog. Joe L.
March 9, 201214 yr You are using Beta 13 with one LSI controller. The support for the LSI controllers is broken big time in this release and also in Beta 14. Downgrade to Beta 12 or 12a and the disk problems will dissapear (but new ones related to realtek LAN and IRQs may pop up). Also you are using a motherboard with the initial factory released BIOS 0503. Upgrade to the newest one 0901, load the default configuration, save it, go back into the bios and disable any unused hardware feature (par. and serial ports, audio, floppy drive, firewire, even the IDE controller is you are not going to use any IDE drive), save this one and try again.
March 9, 201214 yr Author Now that you mention it, I seem to recall that these problems occurred when I switched to b14, so I downgraded to b13 in the hopes it would go away. I will go back to b12a, my buddy is running that and he's never had any issues... You are using Beta 13 with one LSI controller. The support for the LSI controllers is broken big time in this release and also in Beta 14. Downgrade to Beta 12 or 12a and the disk problems will dissapear (but new ones related to realtek LAN and IRQs may pop up). Also you are using a motherboard with the initial factory released BIOS 0503. Upgrade to the newest one 0901, load the default configuration, save it, go back into the bios and disable any unused hardware feature (par. and serial ports, audio, floppy drive, firewire, even the IDE controller is you are not going to use any IDE drive), save this one and try again.
March 9, 201214 yr Yeah, I have the same card (only one though) running beta12, and never have had any of these issues. seems to me, that beta12 is the most stable beta version, been running it since september, without issues. The only good thing about this, is that when Tophicles is rebuilding his server/array, he can still watch all my movies/tv on my unraid server (since we're on the same network )
March 9, 201214 yr Author Ok, so I went back to b12 and the disk still show as unformatted. I ran the unraid_partition_utility.sh and it showed that the drive was not partitioned for unraid properly (expected sector 63, saw sector 64). Now, I know that when I pre-cleared it, I did so with -A option, so it should be on 64. I ran the partition utility with the -A command and it told me the disk is now partitioned correctly. Upon reboot, however it appears to have reverted back.
March 9, 201214 yr Author Just for further information, I re-ran Joe L's SED command: root@Angband:~# dd if=/dev/sdj count=195 | od -c -A d | sed 30q 195+0 records in 195+0 records out 99840 bytes (100 kB) copied, 0.00294673 s, 33.9 MB/s 0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0000448 \0 \0 203 \0 \0 \0 @ \0 \0 \0 p 210 340 350 \0 \0 0000464 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0000496 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 U 252 0000512 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0066048 020 021 034 035 P U 243 \n 323 375 \f \0 022 \0 \0 \0 0066064 \0 \0 \0 \0 \0 \0 \0 \0 004 \0 \0 \0 \0 \0 \0 0066080 204 003 \0 \0 036 \0 \0 \0 \0 \0 \0 \0 \0 020 314 003 0066096 314 003 001 \0 R e I s E r 2 F s \0 \0 \0 0066112 003 \0 \0 \0 005 \0 9 : 002 \0 \0 \0 \0 \0 \0 \0 0066128 \0 \0 \0 \0 ( N 330 ^ 032 332 O 223 214 = 345 L 0066144 022 262 022 E \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0066160 \0 \0 \0 \0 001 \0 036 \0 036 031 P O \0 N 355 \0 0066176 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0066240 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 \0 \0 \0 0066256 0 \0 \0 \0 1 \0 \0 \0 W \0 \0 \0 X \0 \0 \0 0066272 204 \0 \0 \0 205 \0 \0 \0 206 \0 \0 \0 207 \0 \0 \0 0066288 245 \0 \0 \0 246 \0 \0 \0 257 \0 \0 \0 261 \0 \0 \0 0066304 273 \0 \0 \0 275 \0 \0 \0 276 \0 \0 \0 300 \0 \0 \0 0066320 301 \0 \0 \0 303 \0 \0 \0 327 \0 \0 \0 330 \0 \0 \0 0066336 353 \0 \0 \0 355 \0 \0 \0 357 \0 \0 \0 360 \0 \0 \0 0066352 362 \0 \0 \0 363 \0 \0 \0 365 \0 \0 \0 367 \0 \0 \0 0066368 370 \0 \0 \0 371 \0 \0 \0 377 \0 \0 \0 \0 001 \0 \0 0066384 001 001 \0 \0 002 001 \0 \0 003 001 \0 \0 005 001 \0 \0 0066400 \a 001 \0 \0 \b 001 \0 \0 \f 001 \0 \0 \r 001 \0 \0 0066416 016 001 \0 \0 017 001 \0 \0 023 001 \0 \0 024 001 \0 \0 root@Angband:~# ... and here's the output from the partition_utility script: root@Angband:/boot# unraid_partition_disk.sh -t /dev/sdj BLKRRPART: Device or resource busy This disk is currently in use. ######################################################################## terminate called after throwing an instance of 'int' Disk /dev/sdj: 2000.4 GB, 2000398934016 bytes 1 heads, 63 sectors/track, 62016336 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdj1 64 3907029167 1953514552 83 Linux Partition 1 does not end on cylinder boundary. ######################################################################## ============================================================================ == Disk /dev/sdj is NOT partitioned for unRAID properly. == expected start = 63, actual start = 64 == expected size = 3907029105, actual size = 3907029104 ============================================================================ root@Angband:/boot#
Archived
This topic is now archived and is closed to further replies.