March 26, 200719 yr Tom, I ran into a major snag with 4.0-beta4. I wanted to test performance with a disk failure. I stopped the array, went to the disk page, unassigned disk6, and then re-started the array. It came up just fine, and it showed the one disk as missing. I was able to read files on the "missing" disk. I then stopped the array once more, re-assigned disk6 and then re-started the array once more. The screen then looked like this with most of the disks showing up as unformatted: It showed all but two of my disks as "unformatted".... ouch... It also showed it was rebuilding disk6. (I'm guessing from the native non-raid volumes /dev/hda,/dev/hdb,/dev/hdc,/dev/hde,etc.) I tried to stop and re-start the array. The drives still appeared as unformatted. I did an "ls -l /mnt/disk1" and the command came back with no files present. I suspect that the drive did not get mounted to the mount point. I stopped the array, put back my flash drive with the old 3.1 release, and the server came up cleanly with all drives present. Before stopping the array I made a copy of the syslog. It is attached. Perhaps it will give you a few clues. I'm now running back on the older release of unRaid on the other flash drive. Once I am satisfied all my disks are back I'll try the 4.0 release once more. Question: If 4.0 comes back up with all the disks unformatted and wanting to rebuild disk6, what is the best way to get it back sane again. (No, I did not make a "before' copy of the system.dat file, I'm sure that would have made recovery easier) I'm fairly sure I can take a copy of the 3.1 system.dat and put it in place and go through the same steps as I did originally, assigning the devices and having the array do a complete parity check as if this was the first time booting 4.0. Is there an easier way? In any case, I'll let you know. Right now I'm copying the files from disk6 to disk7 using the 3.1 version of unRaid... just in case something weird happens. Joe L.
March 26, 200719 yr Hi Joe, I had an "UNFORMATTED" scare today when going between v3.1beta2 back to v4.0beta4. My IDE HDDs showed as Unformatted under v3.1 which gave me a major panic, but having gone back to v4.0 I have all 6 of the IDE HDDs showing 4 errors each. Parity is rebuilding on another (Sata) HDD that I disabled & rebooted to check parity restoration today. I only put a couple of small files on the HDD as I naively though the rebuild wouldn't take very long! Mark.
March 26, 200719 yr Author Joe, are you certain you're running 4.0-beta4? Tom, Scroll the picture I posted at the start of this thread over to the right. It says 4.0-beta4. Another clue is this line from the attached log file: Mar 24 17:19:17 Tower emhttp[984]: unRAID System Management Utility version 4.0-beta4 So... to answer your quesstion... pretty sure I was on 4.0-beta4. I'm currently booted on the old flash drive with the old 3.1 versison of unraid. I'll swap back, but I think I might want to re-build parity on this version first. Your thoughts before I do anything? Right now I'm up, everything is online, and the array is fully online, but with 3.1. Joe L.
March 27, 200719 yr Joe, we have not yet been able to reproduce this problem. Please try 4.0-beta5. There is a little more debug information generated in this release (not documented in the change log) and output to syslog. If you get into the state where some drives appear 'Unformatted' after Start, it's because the mount failed. We did have this problem early on in development of 2.6 driver, but was solved. If it happens again, capture the syslog and then reboot.
March 27, 200719 yr Author Joe, we have not yet been able to reproduce this problem. Please try 4.0-beta5. There is a little more debug information generated in this release (not documented in the change log) and output to syslog. If you get into the state where some drives appear 'Unformatted' after Start, it's because the mount failed. We did have this problem early on in development of 2.6 driver, but was solved. If it happens again, capture the syslog and then reboot. My luck... figures it might be something different in my configuration than in yours, or you probably would have tracked it down before releasing it... I'm currently doing a parity check with 3.1. It is 40% done, has about 278 miniutes to go, and is moving alaong at a reported 17,867KB/Sec. When it is done I plan on swapping back to the new flash drive, with the newest 4.0-beta5. It has not written anything to the parity drive so far, so don't think anything is going to be out of sync Should I also copy the system.dat from the 3.1 flash drive to the 4.0 flash drive before rebooting on 4.0beta5 ?? (since the drives will all be in an OK status at that point) Or... should I use the existing system.dat file on the 4.0 flash drive that had most of my drives as invalid and unformatted?? When I reboot using 4.0-beta5, will unRaid see all the drives as formatted once again? Earlier this morning I did not reboot when marking a drive as unassigned and then assigned again, I simply stopped the array in between, unassigned/assigned and re-started the array. I understand about drives not being mounted... my disks spin up at different rates, some are up to speed much more quickly than others. Perhaps you are not waiting for them all to get up to speed? In the syslog I noticed you were sending file descriptor 2 to /dev/null on the mount commands... perhaps append the mount error output to the syslog instead of sendinig it to the bit-bucket. (If that is not what you did already) Joe L.
March 27, 200719 yr Author Tom, Something interesting I noticed when re-starting my array on version 3.1 of unraid. The only two disks that were (apparently) correctly mounted on version 4.0 beta also are the two fastest to be mounted. Here is a screen shot of my 3.1 flash drive starting the array. I pressed "refresh" to get a snapshot in time as the disks were being mounted. Notice that disk 2 and 3 are on-line, and the others are still "mounting" Could the issue with 4.0 be that it is not waiting long enough before deciding the disks could not be mounted? Joe L.
March 27, 200719 yr Author I upgraded my second flash drive with 4.0-beta5 and re-booted my server with it. I did not do anything to the system.dat file so it came up thinking it should rebuild disk6. It is in the process of rebuilding my disk6, all other disks were correctly mounted. None showed up as unformatted. Speed is estimated at roughly 18,000MB/Sec. Time to stress test the 4.0-beta5 unRaid server. Now, anything on disk6 must be re-constructed by reading all the other disks, so I decided to play an ISO image of a file stored on disk6 on my laptop. It is doing just fine Ok... lets try playing a different ISO from disk6 on unRaid on my Theater HTPC at the same time.... you guessed it, It too is doing fine. I then tried the MG-35 media player in my Den. It is ucLinux based and mounts the ISO image to play it off the unRaid server. I started a third movie. It is also doing just fine. I went up to my bedroom where I also have a MG-35 media player. Could unRaid do one more? I started yet another movie that I knew was located on the disk being reconstructed. This makes four different ISO images being served while the disk with them is being reconstructed. Now... how busy is the server... Below is an image of the "top" command showing how it is doing... Looks just under 70 percent idle, while reconstructing a replaced disk AND while serving up 4 separate ISO images that were on the disk being reconstructed. Before I started serving all the ISO images the reconstruction was going at just over 18,000KB/second. Serving 4 ISO images of DVDs have slowed the reconstruction rate to 10,000KB/Second. I'm not sure I'd recommend slowing the reconstruction when you have an actual failure but its nice to know unRaid 4.0-beta5 can handle it with ease. I'm impressed. Joe L. (I'm using an 8 port Netgear gigabit switch, but the media players are all 100Mbit, so it too is doing great)
March 27, 200719 yr That really is impressive! I've always left the server alone during parity recalcs & disk reconstruction, I hadn't appreciated that not only can I run "business as usual" from the other disks, but also from the failed one. You're braver than me Joe, but thanks for being so! Mark.
March 27, 200719 yr Author Recovery of disk6 completed. I'm back on line with all the disks. Later this evening I'll try my test unasssigning disk6 and then re-assigning it once more. Joe L.
March 28, 200719 yr Author OK... I repeated the same test I made yesterday... unfortunately, with the same failure in recovery. most of my drives appeared as "unformatted" :'( :'( :'( I performed the following steps... 1. All disks were in their spun down state, as the array had been unused for several hours. 2. using the web-interface, I stopped the array and waited for the web-page to refresh itself. This took quite a while, as the disks all were spun up first. 3. I logged in via telnet and copied the system.dat to a backup-copy-system.dat 4. I unassigned disk6 via the web-interface. 5. Using my laptop, I played a DVD ISO image that was on disk6. It was reconstructed from parity and the remaining drives. 6. I stopped the array once more 7. I re-assigned disk6 8. I started the array again... initially the web-interface said "mounting" for all the data drives. 9. I pressed the "refresh" button. This time, drives 2,3 and 7 appeared to mount correctly, and all the others apparently failed and reported 'unformatted' on the web-interface. (On my first test, only drives 2 and 3 appeared to mount correctly, this time drive 7 also mounted) Like last time, it said a "Data Rebuild" was in progress. 10. I stopped the array once more. 11. I made a copy of /var/log/syslog. It is attached to this post. 12. I Copied the backup system.dat to system.dat and re-started the array. unRaid correctly mounted the drives (all but disk6) and also reported it is rebuilding disk6. In 342.3 minutes it should be done rebuilding. Looking in the attached syslog, the disks that failed to mount all seemed to report error 32. Hope this helps Tom. Obviously there is still an issue. Perhaps the attached logfile will give you a clue how to fix the problem. Joe L.
March 28, 200719 yr "Error 32" simply means "mount error" - not much help. I put up another beta, 4.0-beta6 which will cause all mount error output to get sent to the console. Please re-run your test & tell me what shows up there.
March 28, 200719 yr Author Tom, Went through the same steps once more after installing 4.0-beta6. This time it worked as expected, and all the disks mounted fine. It is now rebuilding disk6 and in 329.3 minmutes I'll give it another try. I may install one of my 8 Gig drives to see if I can get more test cycles in a day. It would probably take less time to rebuild it than the 400Gig drive. I did find one thing. As I said, my array came up rebuilding the disk I had temporally unassigned. I wanted to see what would happen if I stopped the array, and then re-started it. The web-page looked like below. Notice the statement that a rebuild was occurring is gone, and it says the configuration is valid. There was no 'check box-I'm sure' to activate the "start" button. Disk6 was still shown in 'red' I was afraid to start the array, not knowing what it would do. Instead, I rebooted the server. It came up fine and again is re-building disk6. Joe L.
March 29, 200719 yr Author Tom, I tried my sequence of steps once more... Again unRaid failed to mount most of my disks, and they showed up as unformatted. I had no easy way to capture what was on the system console, but the error output of the 'mount' commands was there. I did however take a snapshot of it with the camera in my cell phone... see it below: I could only capture half of the screen, but for each of the drives that failed to mount it said basically the same thing... mount: special device /dev/md[1457] does not exist Looks like a timing issue of some kind... or perhaps some kind of lock on accesssing the MD devices that are slower. Hope this helps... I saved a copy of the syslog but the only difference is you are not sending fd2 of the mount commands to /dev/null Let me know if you need the syslog from beta6. Joe L.
March 29, 200719 yr Joe- your screenshot pointed to the problem, which should be fixed now in 4.0-beta7. Problem was a race condition. The unRAID driver creates a device for each disk, e.g., /dev/md1 for disk1, /dev/md2 for disk2, etc. Actually, the driver registers the device with the kernel, which generates a hotplug event, which triggers a daemon (udevd) to actually create the device node. Well the management utility "knows" the driver created the devices, so immediately after starting the driver, it proceeds to attempt to mount the /dev/md's. But it can happen that the device node is not present when we try to mount, and thus mount fails and device appears "unfomatted". Solution is to call 'udevsettle' before attempting mounts.
March 29, 200719 yr Author Tom, What a team!! (OK, you did most of the work, I just took pictures) Apparently my unRaid server is just perfect for uncovering things that work properly on your test machines. I think this the third bug we squashed working together. Thank you for finding and fixing the race condition. I'm now booted up on 4.0-beta7 and it worked as expected. All disks were mounted correctly, and none showed up as "unformatted" when I re-assigned a disk I had temporally unassigned. Now... don't forget about the web-interface screen shot two posts ago. Stopping an array that has a "Data Recovery" in progress resulted in a screen with a disk with a red "indicator", but no "I am sure" checkbox to restart the array, and no "Data Recovery will occur" text, but instead a "configuration valid" message was present. I was not brave enough to restart the array as I thought the web interface was confused, so I rebooted instead and it figured out it needed to perform the data recovery. Joe L.
March 29, 200719 yr Stopping an array that has a Data-Rebuild in progress results in a disk with a orange "indicator" (not red). When Stopped, it will say "Configuration valid", and the Start message will say "Start will bring array on-line". This is all correct. The reason it's different than the FIRST time system detects that a disabled disk has been replaced (where the "I'm sure I want to do this" checkbox is present), is because system wants to make sure user definitely won't mind this disk being written. If later you cancel the Rebuild, then Stop array (or just Stop array during Rebuild), system knows this is the same disk as before and it's ok to resume Rebuild on next Start. I guess it would be nice to include the verbage that "Start will start Rebuild of the new disk", but it's not necessary to include "I'm sure I want to do this". The "I'm sure I want to do this" checkbox only shows up where the action would result in something irreversible. -- In the process of testing this, I did find a bug however. Say you cancel a Rebuild then Stop the array. Now go to Devices page and unassign the disk (or power down and remove the disk), now array config will come up as "Too many missing/wrong disks" and won't let you Start array. This is a bug which is fixed in beta8 (not up yet though). If this state is detected, then when you Start the array, the missing disk just gets disabled again.
March 29, 200719 yr Author Stopping an array that has a Data-Rebuild in progress results in a disk with a orange "indicator" (not red). When Stopped, it will say "Configuration valid", and the Start message will say "Start will bring array on-line". This is all correct. The reason it's different than the FIRST time system detects that a disabled disk has been replaced (where the "I'm sure I want to do this" checkbox is present), is because system wants to make sure user definitely won't mind this disk being written. I understand, the user has already indicated the disk is OK to be written to... I guess it would be nice to include the verbage that "Start will start Rebuild of the new disk", but it's not necessary to include "I'm sure I want to do this". Yes, please do change the verbage to say "Start will start Rebuild of the new disk" in that situation. If I was confused and resorted to a re-boot because I thought the array was confused, then others less experienced will surely be confused. It is a minor change but will make it a lot easier when recovering from a true disk failure and it is stopped for some reason. I agree, you do not need the "I'm sure" checkbox at that time. In the process of testing this, I did find a bug however. Say you cancel a Rebuild then Stop the array. Now go to Devices page and unassign the disk (or power down and remove the disk), now array config will come up as "Too many missing/wrong disks" and won't let you Start array. This is a bug which is fixed in beta8 (not up yet though). If this state is detected, then when you Start the array, the missing disk just gets disabled again. Glad I could help you find another bug... Just think, fewer panic stricken e-mails pleading for help. ;D Joe L.
Archived
This topic is now archived and is closed to further replies.