February 18, 201313 yr Can anyone help? Here are the details. I have a unraid 4.7 server with 12 hard disks in it. One day, i noticed i wasn't able to access some content on it so I pulled up the manager I noticed that Drive 10 had red balled and Drive 11 was showing as unformatted. I don't know what happened as it was working and then suddenly stopped. Both drives are connected to a separate controller then the other drives in the array. Of course I went through the normal troubleshooting with checking the cables, etc. I switched the controller with a brand new one and had the same result. I have attached the syslog file. Can't run the smart reports. Any help would be greatly appreciated!!!! syslog-2013-02-18.zip
February 19, 201313 yr The red balled drive will be because of a write fail - looks like your issue started here: Feb 18 14:48:06 Tower init: Re-reading inittab Feb 18 14:48:13 Tower kernel: sas: command 0xc4370540, task 0xc41a2280, timed out: BLK_EH_NOT_HANDLED Feb 18 14:48:13 Tower kernel: sas: Enter sas_scsi_recover_host Feb 18 14:48:13 Tower kernel: sas: trying to find task 0xc41a2280 Feb 18 14:48:13 Tower kernel: sas: sas_scsi_find_task: aborting task 0xc41a2280 Feb 18 14:48:13 Tower kernel: /usr/src/sas/trunk/mvsas_tgt/mv_sas.c 1701:mvs_abort_task:rc= 5 Feb 18 14:48:13 Tower kernel: sas: sas_scsi_find_task: querying task 0xc41a2280 Feb 18 14:48:13 Tower kernel: /usr/src/sas/trunk/mvsas_tgt/mv_sas.c 1645:mvs_query_task:rc= 5 Feb 18 14:48:13 Tower kernel: sas: sas_scsi_find_task: task 0xc41a2280 failed to abort Feb 18 14:48:13 Tower kernel: sas: task 0xc41a2280 is not at LU: I_T recover Feb 18 14:48:13 Tower kernel: sas: I_T nexus reset for dev 0300000000000000 Feb 18 14:48:13 Tower kernel: sas: I_T 0300000000000000 recovered Feb 18 14:48:13 Tower kernel: sas: --- Exit sas_scsi_recover_host Feb 18 14:48:13 Tower kernel: ata2: translated ATA stat/err 0x41/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Feb 18 14:48:13 Tower kernel: ata2: status=0x41 { DriveReady Error } Feb 18 14:48:13 Tower kernel: ata2: error=0x04 { DriveStatusError } Feb 18 14:48:13 Tower kernel: ata2: translated ATA stat/err 0x41/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Feb 18 14:48:13 Tower kernel: ata2: status=0x41 { DriveReady Error } Feb 18 14:48:13 Tower kernel: ata2: error=0x04 { DriveStatusError } Feb 18 14:48:13 Tower kernel: ata2: translated ATA stat/err 0x41/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Feb 18 14:48:13 Tower kernel: ata2: status=0x41 { DriveReady Error } Feb 18 14:48:13 Tower kernel: ata2: error=0x04 { DriveStatusError } Feb 18 14:48:13 Tower kernel: ata2: translated ATA stat/err 0x41/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Feb 18 14:48:13 Tower kernel: ata2: status=0x41 { DriveReady Error } Feb 18 14:48:13 Tower kernel: ata2: error=0x04 { DriveStatusError } Feb 18 14:48:13 Tower logger: mount: wrong fs type, bad option, bad superblock on /dev/md11, Feb 18 14:48:13 Tower logger: missing codepage or helper program, or other error Feb 18 14:48:13 Tower logger: In some cases useful info is found in syslog - try Feb 18 14:48:13 Tower logger: dmesg | tail or so Feb 18 14:48:13 Tower logger: Feb 18 14:48:13 Tower emhttp: _shcmd: shcmd (41): exit status: 32 Feb 18 14:48:13 Tower emhttp: disk11 mount error: 32 Feb 18 14:48:13 Tower emhttp: shcmd (42): rmdir /mnt/disk11 But that all relates to disk 11 - as does : Feb 18 14:48:13 Tower kernel: sd 1:0:1:0: [sdc] Result: hostbyte=0x00 driverbyte=0x08 Feb 18 14:48:13 Tower kernel: sd 1:0:1:0: [sdc] Sense Key : 0xb [current] [descriptor] Feb 18 14:48:13 Tower kernel: Descriptor sense data with sense descriptors (in hex): Feb 18 14:48:13 Tower kernel: 72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 Feb 18 14:48:13 Tower kernel: 00 00 00 c6 Feb 18 14:48:13 Tower kernel: sd 1:0:1:0: [sdc] ASC=0x0 ASCQ=0x0 Feb 18 14:48:13 Tower kernel: sd 1:0:1:0: [sdc] CDB: cdb[0]=0x28: 28 00 b3 1a 43 c7 00 00 08 00 Feb 18 14:48:13 Tower kernel: end_request: I/O error, dev sdc, sector 3004842951 Feb 18 14:48:13 Tower kernel: md: disk11 read error Couldn't see anything relating to disk 10 - maybe that was before the start of this log? (You do say "one day, i noticed.) I'd say the obvious culprit would have been the controller. Unfortunately, I can't offer any more advice - hopefully someone more technically able will reply.
February 19, 201313 yr The red balled drive will be because of a write fail - looks like your issue started here: Couldn't see anything relating to disk 10 - maybe that was before the start of this log? (You do say "one day, i noticed.) I'd say the obvious culprit would have been the controller. Unfortunately, I can't offer any more advice - hopefully someone more technically able will reply. Or, one drive locked up the controller, and took the other it shares a controller with out by locking up the controller for both. Before you do ANYTHING, save a copy of the config directory. It will let you get back to the same stop-point in some situations. Then, try disconnecting the red-balled drive and rebooting, see if the array will start. Then try rebooting with just the other disk connected. In this case, unformatted is just a disk that failed to mount. DO NOT FORMAT IT!!!!!!! (unless you do not want the data on it)
February 19, 201313 yr Author Joe L, I took your advice. First I unplugged the red balled drive, Disk 10, and booted the server. It shows Drive 10 as "not installed" and Disk 11 as unformatted. The array starts Then I unplugged Disk 11 and connected the red ball drive and it shows as blue now and Disk 11 shows as missing and the array won't start... When I plug them both in, now it shows the drive that was shown as red balled as blue now, with a message that says "Disabled disk replaced.". I cannot tell what Disk 11 is doing because the array won't start. I don't want to start the array because it says it is going to start the rebuild process and I don't know what is going to happen with Disk 11. I have posted the lastest syslog file here. Thanks syslog-2013-02-19.txt
February 19, 201313 yr Joe L, I took your advice. First I unplugged the red balled drive, Disk 10, and booted the server. It shows Drive 10 as "not installed" and Disk 11 as unformatted. The array starts Then I unplugged Disk 11 and connected the red ball drive and it shows as blue now and Disk 11 shows as missing and the array won't start... When I plug them both in, now it shows the drive that was shown as red balled as blue now, with a message that says "Disabled disk replaced.". I cannot tell what Disk 11 is doing because the array won't start. I don't want to start the array because it says it is going to start the rebuild process and I don't know what is going to happen with Disk 11. I have posted the lastest syslog file here. Thanks Let's see if both disks are responding, then we can figure out why disk11 is not mounting (why it is showing as unformatted) You can try to get an hdparm report on the two drives. hdparm -i /dev/sda hdparm -i /dev/sdb and a smart report for each smartctl -a /dev/sda smartctl -a /dev/sdb Do you happen to remember if they were partitioned on sector 64 (4k-aligned) or sector 63 (un-alligned)? You can perform a file-system check on disk 11 by typing: reiserfsck --check /dev/sda1 (note the trailing "1" indicating the first partition) You need to give it the correct device letter in place of the "X" (in your case, I think the syslog shows sda). You can do this on each of the two data disks, even with the array not started. Do not attempt a repair just yet, regardless of what it tells you, just report back what is says to you. You can also get the output of fdisk -lu /dev/sda for disk11. Joe L.
February 20, 201313 yr Author Joe L, I used whatever is the default partion settings used in Unraid. I have run the test and included the results in the attached zip file. Thanks. fdisk_sda.zip
February 20, 201313 yr Only issue I see at all is one sector pending re-allocation on /dev/sda Let's ignore that for now. Both disks seem to be responding. You can try an experiment to see the actual error that occurs when disk11 mounts (again, assuming it is /dev/sda ). First, we make an empty directory mkdir /tmp/empty_mdizzle then, we attempt to mount (read-only) the first partition on that drive (note the trailing one on the device name) on that mount point (the empty directory) mount -o ro -t reiserfs /dev/sda1 /tmp/empty_mdizzle If unsuccessful, post the error, id successful, un-mount it. umount /tmp/empty_mdizzle (Note: correct command spelling is umount, not unmount)
February 20, 201313 yr Author I performed the three steps and they successfully completed without error.
February 20, 201313 yr I performed the three steps and they successfully completed without error. That is very good news. You can try them with the other disk /dev/sdb as well. It will let you know if it is working as expected. (remember, you must mount the first partition /dev/sdb1 for that disk) Then there is no reason I can think of why the disk should not mount normally, unless one of your plugins is attempting to use the disk before it gets mounted and ties up the default mount point. disable your plugins and add-ons, then reboot. Attache the resulting syslog. Joe L.
February 20, 201313 yr Author I was able to do all of those on sdb1 as well. I disabled as many plug-ins as I could think of using unmenu's tools and rebooted. Attached is the syslog. Thanks for all the help so far. syslog-2013-02-20.txt
February 21, 201313 yr I was able to do all of those on sdb1 as well. I disabled as many plug-ins as I could think of using unmenu's tools and rebooted. Attached is the syslog. Thanks for all the help so far. Does the disk still show as unformatted? It would be really nice if it mounted. Joe L.
February 21, 201313 yr I was able to do all of those on sdb1 as well. I disabled as many plug-ins as I could think of using unmenu's tools and rebooted. Attached is the syslog. Thanks for all the help so far. Does the disk still show as unformatted? Joe L. Since both disks are apparently responding, and have no file system errors, this is one of the few times where I would suggest in using the "trust" procedure to make unRAID trust that all the disks are to be trusted with their parity. To do that you would set a new disk configuration then tell unRAID that a non-existant disk is the one not valid. This procedure is for 4.7 ... It will not work the same on any of the 5.X series. Log in via telnet type initconfig answer to its response "Yes" (with a capital "Y" and lower case "es") Normally, this sets the parity disk as invalid (to be re-constructed based on the "new" config) Then, before doing anything else, type /root/mdcmd set invalidslot 99 The makes unRAID think parity is good.. (and non-existant disk 99 is not valid) If you refresh the main web-management page for unRAID, all the indicators will be blue. When you then start the array, a parity check will begin, but you can cancel it. You should see all the disks online... including the one that is currently reporting unformatted and the one that had red-balled. Joe L.
February 21, 201313 yr Author Joe L, Thanks so much. Everything "seems" to be working. I am going to run the parity check anyway. There are about 12TB on this thing so who knows what may have gotten lost, it'll be hard to tell with all that data. I'll search the forums to figure out a way to potentially catalog the files on each drive individually so I at least have some record so that I can potentially try to reget files that may be stored on local drives if I have a drive fail completely. This makes me very nervous but thank you for your advice. I REALLY appreciate it!!!!
February 21, 201313 yr If you set a new config, as I described and are back up running by trusting the array, you should now run a parity check... At least then you'll know you can recover back to this point. Joe L.
Archived
This topic is now archived and is closed to further replies.