September 16, 201015 yr My gosh, I'm getting the same type of problem as my previous issue when I added 2 new 2TB drives simultaneously previously (thread http://lime-technology.com/forum/index.php?topic=7126.0), but this time around, I added just 1 new 2TB and the web GUI seems to be stuck with "Starting..." in the Command Area. Last time I was using 4.5.4. Now I'm using 4.5.6 so apparently this bug has NOT been addressed. When I go to the console monitor, unRAID seems to be in a slow, perpetual loop of issuing "mdcmd (###): spindown #" commands (where ### appears to be an incrementing counter, and the single # appears to be the drive number). It seems to be issuing commands between two different drives two times, then it moves on to a different pair of drives: mdcmd (111): spindown 9 mdcmd (259): spindown 3 mdcmd (261): spindown 3 mdcmd (262): spindown 4 mdcmd (263): spindown 5 mdcmd (265): spindown 0 mdcmd (266): spindown 4 ...etc... I really don't want to go through this AGAIN. If I reset the computer, I will probably be stuck wasting another half day for unRAID to clear the new drive. I'm letting it run in circles for now, hoping SOMEONE can give me some advice as to how to resolve this issue. Since unRAID has not put up the command prompt, I don't think I can issue any commands to figure out what its doing. UPDATE: It appears its finished going through all the available drives with its spindown commands and I was able to get to the command prompt, however, the web GUI still is stuck at "Starting..." Now what?
September 16, 201015 yr Did you preclear the drive before adding it? If not, it has to write zeros to the entire 2 TB drive, a process that can take hours. And as you are noticing, the array is not available during this time. Does the machine show any hard drive activity?
September 16, 201015 yr Author No, I did not "preclear" the drive, tho I was under the impression that unRAID 4.5.x already does that as the moment I powered it up with a brand new drive (not used and re-allocated to the unRAID), it automatically started to "clear" the drive. From my previous experience, unRAID "clears" the drive (which took all day) before presenting the option to "format" the drive. For all intents and purposes, that sounds like "pre-clearing" to me. And no, there is no disk activity. Even the web GUI shows all drives spundown (flashing green dots), including the newly added drive.
September 16, 201015 yr No, I did not "preclear" the drive, tho I was under the impression that unRAID 4.5.x already does that as the moment I powered it up with a brand new drive (not used and re-allocated to the unRAID), it automatically started to "clear" the drive. No, it does not. It will clear a drive only after assigning it to the array and then subsequently pressing "Start" From my previous experience, unRAID "clears" the drive (which took all day) before presenting the option to "format" the drive. For all intents and purposes, that sounds like "pre-clearing" to me. It is clearing the drive prior to allowing the array to come up on-line ans then allowing you to format the drive. Formatting typically takes only a few minutes. The pre-clearing we are talking about is a script you can use on a drive that lets you add it to the array and have it recognize it as not needing the all-day clearing step where your server is off-line. And no, there is no disk activity. Even the web GUI shows all drives spundown (flashing green dots), including the newly added drive. That sounds suspicious. Please attach a syslog to your next post for analysis. (assuming you can log in via telnet to grab a copy) Joe L.
September 16, 201015 yr Author No, I did not "preclear" the drive, tho I was under the impression that unRAID 4.5.x already does that as the moment I powered it up with a brand new drive (not used and re-allocated to the unRAID), it automatically started to "clear" the drive. No, it does not. It will clear a drive only after assigning it to the array and then subsequently pressing "Start" Well, of course I pressed "Start" but I have no choice to avoid, pass, skip the "clearing" process, so in my book, that means it's "automatic" with no way to deviate from that path . If this is not the "pre-clear" that everyone is talking about, then what exactly is it "clearing" as it took roughly 8 hours to get to the point where it is now? From my previous experience, unRAID "clears" the drive (which took all day) before presenting the option to "format" the drive. For all intents and purposes, that sounds like "pre-clearing" to me. It is clearing the drive prior to allowing the array to come up on-line ans then allowing you to format the drive. Formatting typically takes only a few minutes. The pre-clearing we are talking about is a script you can use on a drive that lets you add it to the array and have it recognize it as not needing the all-day clearing step where your server is off-line. And no, there is no disk activity. Even the web GUI shows all drives spundown (flashing green dots), including the newly added drive. That sounds suspicious. Please attach a syslog to your next post for analysis. (assuming you can log in via telnet to grab a copy) Joe L. Syslog attached! syslog.txt
September 16, 201015 yr This is the cause of the lock-up. Sep 15 23:26:05 UnRAID kernel: BUG: unable to handle kernel paging request at f8e34000 Sep 15 23:26:05 UnRAID kernel: IP: [<f8e2a8e4>] md_cmd_proc_read+0x41/0x54 [md_mod] Sep 15 23:26:05 UnRAID kernel: *pdpt = 0000000001447001 *pde = 0000000035ecc067 *pte = 0000000000000000 Sep 15 23:26:05 UnRAID kernel: Oops: 0000 [#1] SMP Sep 15 23:26:05 UnRAID kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/host1/port-1:0/end_device-1:0/target1:0:0/1:0:0:0/block/sdb/stat Sep 15 23:26:05 UnRAID kernel: Modules linked in: md_mod xor i2c_i801 i2c_core ahci e1000e mvsas libsas scst scsi_transport_sas [last unloaded: xor] Sep 15 23:26:05 UnRAID kernel: Sep 15 23:26:05 UnRAID kernel: Pid: 3599, comm: emhttp Not tainted (2.6.32.9-unRAID #5) X7SPA-HF Sep 15 23:26:05 UnRAID kernel: EIP: 0060:[<f8e2a8e4>] EFLAGS: 00210202 CPU: 1 Sep 15 23:26:05 UnRAID kernel: EIP is at md_cmd_proc_read+0x41/0x54 [md_mod] Sep 15 23:26:05 UnRAID kernel: EAX: c37ddf38 EBX: fffff81c ECX: 3fffef72 EDX: fffff81c Sep 15 23:26:05 UnRAID kernel: ESI: f8e34000 EDI: d8964a54 EBP: c37ddf04 ESP: c37ddef4 Sep 15 23:26:05 UnRAID kernel: DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Sep 15 23:26:05 UnRAID kernel: Process emhttp (pid: 3599, ti=c37dc000 task=c3f24000 task.ti=c37dc000) Sep 15 23:26:05 UnRAID kernel: Stack: Sep 15 23:26:05 UnRAID kernel: c37ddf38 f8e2a8a3 00000000 d8961000 c37ddf4c c109d5f5 00000400 c37ddf3c Sep 15 23:26:05 UnRAID kernel: <0> 00000000 00000400 00000000 00000400 b779b000 f5fb4ae0 00000000 f5fb4ae0 Sep 15 23:26:05 UnRAID kernel: <0> 00000400 d8961000 00000001 f5fb4ae0 c109d4ea fffffffb c37ddf70 c1099f72 Sep 15 23:26:05 UnRAID kernel: Call Trace: Sep 15 23:26:05 UnRAID kernel: [<f8e2a8a3>] ? md_cmd_proc_read+0x0/0x54 [md_mod] Sep 15 23:26:05 UnRAID kernel: [<c109d5f5>] ? proc_file_read+0x10b/0x22d Sep 15 23:26:05 UnRAID kernel: [<c109d4ea>] ? proc_file_read+0x0/0x22d Sep 15 23:26:05 UnRAID kernel: [<c1099f72>] ? proc_reg_read+0x56/0x6a Sep 15 23:26:05 UnRAID kernel: [<c1099f1c>] ? proc_reg_read+0x0/0x6a Sep 15 23:26:05 UnRAID kernel: [<c106cbd8>] ? vfs_read+0x8a/0x114 Sep 15 23:26:05 UnRAID kernel: [<c106cf6f>] ? sys_read+0x3b/0x60 Sep 15 23:26:05 UnRAID kernel: [<c1002935>] ? syscall_call+0x7/0xb Sep 15 23:26:05 UnRAID kernel: Code: 55 f0 e8 6a c2 30 c8 8d 50 01 29 f2 39 d3 7c 0b 8b 45 0c 89 d3 c7 00 01 00 00 00 8b 45 f0 89 d9 81 c6 ac fd e2 f8 c1 e9 02 89 38 <f3> a5 89 d9 83 e1 03 74 02 f3 a4 5a 89 d8 5b 5e 5f 5d c3 55 89 Sep 15 23:26:05 UnRAID kernel: EIP: [<f8e2a8e4>] md_cmd_proc_read+0x41/0x54 [md_mod] SS:ESP 0068:c37ddef4 Sep 15 23:26:05 UnRAID kernel: CR2: 00000000f8e34000 Sep 15 23:26:05 UnRAID kernel: ---[ end trace 287aaf711b677649 ]--- Send an e-mail to [email protected] point them to this thread, ask they investigate a kernel bug in handling SMP processing when clearing a drive. In the interim, you'll need to reboot and hopefully the drive will be recognized as cleared. You should then be able to format it. Joe L.
September 16, 201015 yr Author Tanks! I knew something was amiss as this is twice it's happened to me... :'(
September 16, 201015 yr Author UGH! After rebooting, unRAID is CLEARING the drive again (after pressing Start, of course) Is there any way to bypass this step, as I'm sure I will meet the same fate as the other times! UPDATE: I went to the console and rebooted the machine, loaded up the preclear_disk.sh script onto the flash drive, unassigned the target drive from the array, and manually invoked the preclear script on it. Now another 8 hours will be wasted in this process just to "set a flag" that I hope will allow unRAID web GUI to go straight to formatting without invoking it's own "clearing" mechanism. There has GOT to be a better way... :'(
September 16, 201015 yr This is the cause of the lock-up. Sep 15 23:26:05 UnRAID kernel: BUG: unable to handle kernel paging request at f8e34000 Sep 15 23:26:05 UnRAID kernel: IP: [<f8e2a8e4>] md_cmd_proc_read+0x41/0x54 [md_mod] Sep 15 23:26:05 UnRAID kernel: *pdpt = 0000000001447001 *pde = 0000000035ecc067 *pte = 0000000000000000 Sep 15 23:26:05 UnRAID kernel: Oops: 0000 [#1] SMP Sep 15 23:26:05 UnRAID kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/host1/port-1:0/end_device-1:0/target1:0:0/1:0:0:0/block/sdb/stat Sep 15 23:26:05 UnRAID kernel: Modules linked in: md_mod xor i2c_i801 i2c_core ahci e1000e mvsas libsas scst scsi_transport_sas [last unloaded: xor] Sep 15 23:26:05 UnRAID kernel: Sep 15 23:26:05 UnRAID kernel: Pid: 3599, comm: emhttp Not tainted (2.6.32.9-unRAID #5) X7SPA-HF Sep 15 23:26:05 UnRAID kernel: EIP: 0060:[<f8e2a8e4>] EFLAGS: 00210202 CPU: 1 Sep 15 23:26:05 UnRAID kernel: EIP is at md_cmd_proc_read+0x41/0x54 [md_mod] Sep 15 23:26:05 UnRAID kernel: EAX: c37ddf38 EBX: fffff81c ECX: 3fffef72 EDX: fffff81c Sep 15 23:26:05 UnRAID kernel: ESI: f8e34000 EDI: d8964a54 EBP: c37ddf04 ESP: c37ddef4 Sep 15 23:26:05 UnRAID kernel: DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Sep 15 23:26:05 UnRAID kernel: Process emhttp (pid: 3599, ti=c37dc000 task=c3f24000 task.ti=c37dc000) Sep 15 23:26:05 UnRAID kernel: Stack: Sep 15 23:26:05 UnRAID kernel: c37ddf38 f8e2a8a3 00000000 d8961000 c37ddf4c c109d5f5 00000400 c37ddf3c Sep 15 23:26:05 UnRAID kernel: <0> 00000000 00000400 00000000 00000400 b779b000 f5fb4ae0 00000000 f5fb4ae0 Sep 15 23:26:05 UnRAID kernel: <0> 00000400 d8961000 00000001 f5fb4ae0 c109d4ea fffffffb c37ddf70 c1099f72 Sep 15 23:26:05 UnRAID kernel: Call Trace: Sep 15 23:26:05 UnRAID kernel: [<f8e2a8a3>] ? md_cmd_proc_read+0x0/0x54 [md_mod] Sep 15 23:26:05 UnRAID kernel: [<c109d5f5>] ? proc_file_read+0x10b/0x22d Sep 15 23:26:05 UnRAID kernel: [<c109d4ea>] ? proc_file_read+0x0/0x22d Sep 15 23:26:05 UnRAID kernel: [<c1099f72>] ? proc_reg_read+0x56/0x6a Sep 15 23:26:05 UnRAID kernel: [<c1099f1c>] ? proc_reg_read+0x0/0x6a Sep 15 23:26:05 UnRAID kernel: [<c106cbd8>] ? vfs_read+0x8a/0x114 Sep 15 23:26:05 UnRAID kernel: [<c106cf6f>] ? sys_read+0x3b/0x60 Sep 15 23:26:05 UnRAID kernel: [<c1002935>] ? syscall_call+0x7/0xb Sep 15 23:26:05 UnRAID kernel: Code: 55 f0 e8 6a c2 30 c8 8d 50 01 29 f2 39 d3 7c 0b 8b 45 0c 89 d3 c7 00 01 00 00 00 8b 45 f0 89 d9 81 c6 ac fd e2 f8 c1 e9 02 89 38 <f3> a5 89 d9 83 e1 03 74 02 f3 a4 5a 89 d8 5b 5e 5f 5d c3 55 89 Sep 15 23:26:05 UnRAID kernel: EIP: [<f8e2a8e4>] md_cmd_proc_read+0x41/0x54 [md_mod] SS:ESP 0068:c37ddef4 Sep 15 23:26:05 UnRAID kernel: CR2: 00000000f8e34000 Sep 15 23:26:05 UnRAID kernel: ---[ end trace 287aaf711b677649 ]--- Send an e-mail to [email protected] point them to this thread, ask they investigate a kernel bug in handling SMP processing when clearing a drive. In the interim, you'll need to reboot and hopefully the drive will be recognized as cleared. You should then be able to format it. Joe L. I think it might not relate to clearing a drive. this one and following two all have same "signature". i think they all have same root cause. and coincidentally , right before crash there is always disk spindown. http://lime-technology.com/forum/index.php?topic=7622.0 http://lime-technology.com/forum/index.php?topic=6302.0 --------------------------------------- signarture -------------------------------------------------- UnRAID kernel: IP: [<f8e2a8e4>] md_cmd_proc_read+0x41/0x54 [md_mod] UnRAID kernel: Call Trace: UnRAID kernel: [<f8e2a8a3>] ? md_cmd_proc_read+0x0/0x54 [md_mod] UnRAID kernel: [<c109d5f5>] ? proc_file_read+0x10b/0x22d UnRAID kernel: [<c109d4ea>] ? proc_file_read+0x0/0x22d UnRAID kernel: [<c1099f72>] ? proc_reg_read+0x56/0x6a UnRAID kernel: [<c1099f1c>] ? proc_reg_read+0x0/0x6a UnRAID kernel: [<c106cbd8>] ? vfs_read+0x8a/0x114 UnRAID kernel: [<c106cf6f>] ? sys_read+0x3b/0x60 UnRAID kernel: [<c1002935>] ? syscall_call+0x7/0xb
September 16, 201015 yr Author Is there ANY WAY to bypass the "clearing" process of unRAID or the "preclearing" process of the preclear_disk.sh script? unRAID successfully "cleared" the new drive but hung when trying to start up the array. Rebooting only got unRAID to start the clearing process all over again. SO, I tried preclear_disk.sh script. It finished it's clearing process after 8 hours then proceeded to write zero's. Well, misfortune comes in pairs for me as I had a momentary power outtage after the preclear script was 14 hours into its lengthy process, and now it wants to do this all over again. I do NOT want to waste another day on this as I spent 30 hours already. The drive PASSED all tests. It's just bugs and misfortune that keep biting me and I just want to get this new drive added to the array NOW. Is there a way to get to that final step and bypass all these "clearing" steps???
September 16, 201015 yr Is there ANY WAY to bypass the "clearing" process of unRAID or the "preclearing" process of the preclear_disk.sh script? unRAID successfully "cleared" the new drive but hung when trying to start up the array. Rebooting only got unRAID to start the clearing process all over again. SO, I tried preclear_disk.sh script. It finished it's clearing process after 8 hours then proceeded to write zero's. Well, misfortune comes in pairs for me as I had a momentary power outtage after the preclear script was 14 hours into its lengthy process, and now it wants to do this all over again. I do NOT want to waste another day on this as I spent 30 hours already. The drive PASSED all tests. It's just bugs and misfortune that keep biting me and I just want to get this new drive added to the array NOW. Is there a way to get to that final step and bypass all these "clearing" steps??? Short answer, once you have assigned a parity drive, all disks must be cleared. It is the only way to maintain parity protection. Longer answer, the preclear_disk script first READS the entire disk, then, in the second phase, WRITES the zeros to clear it, then in the third phase, verifies it can be READ back successfully. You got into the second phase, but it did not finish. If you wish to skip the pre-read and post-read that allow you to detect errors on the disk before you add it to your array, use the "-n" option. It will then ONLY write the zeros AND the appropriate signature. This will take 8 to 12 hours based on the write speed of your disk. Bad sectors will then be detected as you start to use the disk instead of before. Longer answer yet. Stop the array Assign your new drives Log in at the command prompt and type initconfig (This sets a new disk configuration and immediately invalidates the prior parity calculations based on the prior disk configuration. ) When you next start the array it will begin a full parity calculation. Expect this to take 8 or 9 hours (same as the first time you did it) During this time, the array does NOT have parity protection. If you can deal with that risk, then this might be your solution. You can use the array during this time, but are vulnerable to a disk crash.. (If they are empty, probably not much risk) Joe L.
September 16, 201015 yr Author Thanks Joe! I hope you are getting PAID for all this invaluable service you provide, yes? UPDATE: After roughly 60 hours from when I commenced this process, I was finally able to get through the clearing process via the pre-clear script and the -n switch. Apparently sometime during the wee hours it crashed yet again as I awoke today to find unRAID rebooted and the web GUI showing parity drive check in progress. Perhaps there is an issue with unRAID, my configuration and the pre-clear script, but no matter, I re-ran the script with the -n switch and it finished its zeroing and other processes after 7 hours, 22 minutes and 56 seconds. I've now been able to add it to the array WITHOUT seeing the "clearing" step, was able to get to the format stage and its now a happy member of my 9+1 array.
September 16, 201015 yr Thanks Joe! I hope you are getting PAID for all this invaluable service you provide, yes? I wish...
September 16, 201015 yr Thanks Joe! I hope you are getting PAID for all this invaluable service you provide, yes? He gets paid in gold stars and smiley face stickers next to his name. I think we have lapped the room a couple of times now...
September 18, 201015 yr I am a having a similar but different problem. During preclearing of a new drive, my array was off-line, and I could not start it. Every time I hit the start array button from the web console, it would hang, and refreshing would continue to state that the array was stopped. I re-booted after the preclear was finished, and still was unable to start the array. Finally, used the initconfig command. The new precleared drive has not been added to the array. Now, the array has started. Parity sync in progress. However - one of the previous existing drives is showing as unformatted. Never had a sign of failing drive before. Am I going to be able to recover this drive? Looks like I am going to have to reformat, and the parity sync is currently calculating without the drive in place.
September 18, 201015 yr I am a having a similar but different problem. During preclearing of a new drive, my array was off-line, and I could not start it. Every time I hit the start array button from the web console, it would hang, and refreshing would continue to state that the array was stopped. I re-booted after the preclear was finished, and still was unable to start the array. Finally, used the initconfig command. The new precleared drive has not been added to the array. Now, the array has started. Parity sync in progress. However - one of the previous existing drives is showing as unformatted. Never had a sign of failing drive before. Am I going to be able to recover this drive? Looks like I am going to have to reformat, and the parity sync is currently calculating without the drive in place. DO NOT REFORMAT. un-formatted simply indicates the drive could not be mounted. It might easily be fixed with a reiserfsck. use the procedure as described here in the wiki: http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems
Archived
This topic is now archived and is closed to further replies.