January 17, 201016 yr I have a single drive in my array that s showing as unformatted, but it was formatted just fine yesterday. I added a hot spare drive to the array, just in case. When the system started, it formatted the spare drive and all of the data "appears" to be present, but the drive still shows as unformatted. I am running 4.5 Beta 13 and have attached a syslog. Any help would be appreciated. Also, what would cause this type of issue? I have seen a few other similar posts since this new(er) release. Does it have something to do with this newer release or do you think it might be hardware related? Thanks On last note: All of the drives appear green. syslog.txt
January 17, 201016 yr Author Any ideas? I have over 600 movies on the drives and I am really worried about losing the disk and the data. I do have backup (physical DVDs), but it will take a month to get all of the dvds back on to the system. I'm do not know linux at all and I do not know how to troubleshoot this issue on my own. thanks in advance.
January 17, 201016 yr Any ideas? I have over 600 movies on the drives and I am really worried about losing the disk and the data. I do have backup (physical DVDs), but it will take a month to get all of the dvds back on to the system. I'm do not know linux at all and I do not know how to troubleshoot this issue on my own. thanks in advance. Which drive is showing as unformatted? I'm trying to make sense of your syslog. I can see you added a disk and it went through the process of clearing it. Did that process complete? As far as "un-formatted" goes. It is an indication the file system could not be mounted. (The error is sort of misleading, since when it was coded I think it was expected the the only time it would fail would be is it was not yet formatted. ) So, a bit more description of what you did recently and how far things got will help. Joe L.
January 17, 201016 yr Author Sorry, I added to my first post, but I'll explain in more detail. All the drives showed as green when I checked the system, but one of the drives was showing unformatted and all of the data from that drive seemed to be missing from the share. When I checked the disk in windows explorer it was not present. I got a little panicked about this so I stopped the array and added in a spare 1.5T drive that I had sitting in the case. I brought the array back up and started it. The first thing it did was to format the new drive and that process did complete successfully. Now it appears that all of the data is there (although hard to tell exactly), the new drive is online, but the old drive is still showing as unformatted. Since I am very new to this I have no idea what this means or what to do next. I have attached a screen print from the management console if it will help.
January 18, 201016 yr Author Do you need any additional information? If you tell me what you need, I will be happy to provide it. Still have the array stopped so that I do not do any damage.
January 19, 201016 yr I'm sure one of the more knowledgeable guys will be by at some point. Almost everyone here is just a user like you so sometimes we need to be a bit patient to get help.
January 22, 201016 yr Do you need any additional information? If you tell me what you need, I will be happy to provide it. Still have the array stopped so that I do not do any damage. The reason unRAID is showing the disk as un-formatted is that it is unable to mount the drive. (There are other reasons a mount might fail, so the un-formatted label is misleading) In your syslog are these lines for disk7 Jan 16 19:47:16 Tower emhttp: shcmd (25): mkdir /mnt/disk7 Jan 16 19:47:16 Tower emhttp: shcmd (26): mount -t reiserfs -o noacl,nouser_xattr,noatime,nodiratime /dev/md7 /mnt/disk7 It shows the attempt to mount disk7. Those lines are followed by: Jan 16 19:47:25 Tower emhttp: disk7 mount error: 32 Jan 16 19:47:25 Tower emhttp: shcmd (28): rmdir /mnt/disk7 Jan 16 19:47:25 Tower kernel: REISERFS warning (device md7): sh-2021 reiserfs_fill_super: can not find reiserfs on md7 With those clues, we can assume that the file-system on disk7 is probably corrupted. You need to run a file system check on disk7. To do that, type reiserfsck -y /dev/md7 Report back with the output of the reiserfsck command. When you swapped the disks in your earlier actions, did you press the button labeled "restore" ?? or just "Start" ?? I'm just trying to understand if you've ever seen all the disks since adding your new one, and how/if unRAID might be reacting to your actions. Joe L.
January 22, 201016 yr Author I ran the command your specified and it states: ``````````` reiserfs_open: the reiserfs superblock cannot be found be found on /dev/md7. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb. ``````````` So, I am assuming from this that the file system is bad, but since only one disk is affected that I should not lose any data as the array should have protected it. I have two new 1.5T drives coming this afternoon so that I have additional hot spares (don't really need the space at the moment). Should I run the command as stated and is this the correct format? reiserfsck -y /dev/md7--rebuild-sb Also, to answer your other question, I simply pressed the Start button. Since I am new to linux and unraid I did not feel comfortable hitting the Restore button without some direction. Please let me know what steps I should take next and I will report back with the results. Thank you for helping me. I really appreciate it!!
January 22, 201016 yr I ran the command your specified and it states: ``````````` reiserfs_open: the reiserfs superblock cannot be found be found on /dev/md7. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb. ``````````` So, I am assuming from this that the file system is bad, but since only one disk is affected that I should not lose any data as the array should have protected it. I have two new 1.5T drives coming this afternoon so that I have additional hot spares (don't really need the space at the moment). Should I run the command as stated and is this the correct format? reiserfsck -y /dev/md7--rebuild-sb Also, to answer your other question, I simply pressed the Start button. Since I am new to linux and unraid I did not feel comfortable hitting the Restore button without some direction. Please let me know what steps I should take next and I will report back with the results. Thank you for helping me. I really appreciate it!! First... that is not the correct syntax for the rebuild command. So don't do it just yet. Next, it will prompt you for some information when(if) you do run it, and the answers must be perfect or you'll do more damage than anything else, and the "default" responses to the commands are not the correct choices. Before we go there, let's see if there is actually data that could be re-constructed in your existing parity calculations. Let's see what is actually could be re-constructed by parity before assuming it is correct. To do that, go to the devices page and un-assign disk7. Then, go back to the main page and "Start" the array. DO NOT use the button labeled as "restore" as it will immediately invalidate parity making reconstruction impossible. When you "Start" the array the contents of the un-assigned drive will be simulated, and if it is correct, and your files appear, then you will be able to rebuild it onto one of the replacement drives you have arriving shortly. If it is unable to mount the reconstucted/simulated image of the drive, then the "simulated" drive probably also has the same damaged superblock. Get back to me with that test first. In the mean-time, I'll find the post that shows how to rebuild the superblock so I can point you to it if we need to use that procedure. Joe L.
January 22, 201016 yr Author ok. I unassigned drive 7 and ran a few programs that normally seek out the data on the drives and report issues (MyMovies and iTunes) and all of the files appear to be intact on the simulated drive. The new drives should be here this afternoon so I will have some spares to play with. Now here is a REALLY dumb question...... I have a Norco 4220 case that holds 20 drives. The corrupted drive used to report as disk #5 and is now reporting as disk #7. Do you or anyone here know how I can determine which physical drive is actually being reported as drive #7. I have two internal 8-port SATA cards with reverse cables going down to 4 SAS connectors on 4 of the 5 drive planes. How do I tell which is the drive with the corruption? If I assume that something just happened and the drive itself is not the failure point, then this question may be moot, but if something happens in the future it would be nice to know. Just curious...... Thanks again.
January 22, 201016 yr ok. I unassigned drive 7 and ran a few programs that normally seek out the data on the drives and report issues (MyMovies and iTunes) and all of the files appear to be intact on the simulated drive. The new drives should be here this afternoon so I will have some spares to play with. Now here is a REALLY dumb question...... I have a Norco 4220 case that holds 20 drives. The corrupted drive used to report as disk #5 and is now reporting as disk #7. Do you or anyone here know how I can determine which physical drive is actually being reported as drive #7. I have two internal 8-port SATA cards with reverse cables going down to 4 SAS connectors on 4 of the 5 drive planes. How do I tell which is the drive with the corruption? If I assume that something just happened and the drive itself is not the failure point, then this question may be moot, but if something happens in the future it would be nice to know. Just curious...... Thanks again. You need to match up the serial numbers with those on the devices page. unRAID will use the "port" on the controller to select a specific drive. if you move a drive to a different port, it will probably show as a different "/dev/sdX device" Your best bet right now is to make a screen shot of the devices page so you can lable the drives when you power down to add the new drive. unRAID is NOT hot-plugable in any way, even if the hardware is. From what you've shared so far it sounds like you might have tried to plug in new drives while powered up. If you've done that in the past, forget it in the future. When the new drive arrives, you can put it in any physical slot in your case, and then assign it to slot7 in the array on the devices page. then press "Start" to begin the process of reconstruction onto it. Joe L.
January 23, 201016 yr Author I installed one of the new drives last night and after powering on the server it immediately went into a rebuild. The rebuild took quite a few hours (expected) and all of the drives are showing green. However, the new Disk 7 is showing as unformatted. Now I did not format the drive after installing it as it was doing the rebuild. Should I go ahead and format the drive now? ALso, even though it is showing as disk 7, it occupies a different slot in the case and the old disk 7 is still in its original location, but it is not in the array. Is this normal and should I simply format the new disk or should the rebuild have formatted it for me? Thanks
January 23, 201016 yr I installed one of the new drives last night and after powering on the server it immediately went into a rebuild. The rebuild took quite a few hours (expected) and all of the drives are showing green. However, the new Disk 7 is showing as unformatted. Now I did not format the drive after installing it as it was doing the rebuild. Should I go ahead and format the drive now? ALso, even though it is showing as disk 7, it occupies a different slot in the case and the old disk 7 is still in its original location, but it is not in the array. Is this normal and should I simply format the new disk or should the rebuild have formatted it for me? Thanks It sounds like all is OK, but you are confusing me a bit by your terminology. Post a syslog.
January 23, 201016 yr Author Here is the syslog and a screen image. The log is stating that disk 7 will not mount and you will see on the screen image the the new drive is showing as unformated. Syslog_012310.zip
January 23, 201016 yr Your earlier statement had led me to believe that disk7 had data on it. ok. I unassigned drive 7 and ran a few programs that normally seek out the data on the drives and report issues (MyMovies and iTunes) and all of the files appear to be intact on the simulated drive. Based on what has happened now, the "simulated" drive would have also been unformatted... I believe your files in the MyMovies and iTunes shares are on other disks... Now... do not confuse the physical slot in your rack with the assignment of "disk7" in your "devices" page. From what you now described, the "reconstructed" disk7 is not formatted, and apparently never was. Is this true? Did it ever have files on it? (do you ever look at the disk shares... hint: you should) Type od -x /dev/md7 | head -30 and let's see if it looks at all like it was formatted at all. If the "disk7" had just been added to the array, and never was formatted, then un-assigning it and rebuilding the un-formatted drive onto a new replacement drive will result in a new unformatted drive in the array. You still need to format the drive in the slot if the drive in the slot7 had never been formatted before. If all of your files are present, then to me it appears as if your files are all on the other disks in your array. Once you check, and verify all is present, then you can format the newly added drive. You started this thread with the statement: have a single drive in my array that s showing as unformatted, but it was formatted just fine yesterday. I added a hot spare drive to the array, just in case. When the system started, it formatted the spare drive and all of the data "appears" to be present, but the drive still shows as unformatted. Did you "hot-plug" the "hot-spare" drive in the array? If so, don't ever do it again. You confused unRAID and this whole process. The hardware may be hot-pluggable, and linux might be hot-pluggable, but unRAID is not hot-pluggable. Was the drive you added initially as a hot-spare pre-cleared? If not, I think the unRAID server started on a clearing process on the new disk, and never got to where you would press the "Format" button on it. unRAID never formats a new drive. (unless an old already formatted drive is reconstructed onto a replacement... in that case the formatting is already part of the data being reconstructed) Joe L.
January 23, 201016 yr Author Sorry for the confusion. I'll try to clear things up and answer your questions while I am waiting for a response from the server on your above command. First, I have not formatted the new disk. When I installed this new disk and powered on the server, it immediately went in to a rebuild and started reading and writing data. That finished around 3 am this morning. When I looked at the GUI screen it appeared to have finished and the new drive is showing as unformatted. Since I have not formatted the drive yet, I am assuming that this is correct, but what has the server been doing all this time with the rebuild? It looked like it was copying data to the new drive even though it was not yet formatted. I originally had 4 storage drives in the server plus a cache drive and a parity drive. The original drive I reported as drive #5 (the one showing up as unformatted) has been in the array for a few months now. I installed 6 total drives when I built the machine and added this disk a little while ago to increase storage. The drive was approximately half full (~700Mb out of 1.5Tb) and I formatted it when I installed the drive. When these first issues arose, I added in a disk that was sitting in the case, but had not been added to the array. I mistakenly called this a hot-spare as to me it was. In my ignorance I brought the array offline and then when to the Devices page and selected this existing "hot-spare" drive and added it to the array. I do not remember formatting it as things are getting a little convoluted at the moment. I have never plugged in or disconnected a drive while the machine had power, ever. I always shut the machine down as I read that unRaid was not a hot-swappable environment even though my server hardware is. It has been about 15 minutes now and the command that you gave me "od -x /dev/md7 | head -30" has not returned anything to my telnet screen at all. Not sure what it is supposed to be doing. Based on what I see, it appears that all of my data is being displayed in windows explorer. The file counts and directory counts seem to be correct.
January 23, 201016 yr Sorry for the confusion. I'll try to clear things up and answer your questions while I am waiting for a response from the server on your above command. While you wait, I'll try to answer a few of your questions. First, I have not formatted the new disk. When I installed this new disk and powered on the server, it immediately went in to a rebuild and started reading and writing data. That finished around 3 am this morning. When I looked at the GUI screen it appeared to have finished and the new drive is showing as unformatted. Since I have not formatted the drive yet, I am assuming that this is correct, but what has the server been doing all this time with the rebuild? It looked like it was copying data to the new drive even though it was not yet formatted. It was reading your parity disk and all the other data disks to reconstruct the data on the disk being rebuilt. Notice I said data, not files, not directories... That is because the "md" driver in unraid only knows the disk as a linear set of bits. It has absolutely no concept of what the bits on the "md" device represent. The bits might have files, might not. The bits might represent a formatted file system, they might represent a completely cleared drive and all the bits are zero. To the "md" driver, it is just bits. So... if the reconstructed drive had all zeros, you just wrote all zeros to the replacement drive. when you "assign" a physical disk to an unRAID slot in the array, all it does is point the slot in the array (the "md" device) to the disk controller port holding the disk being assigned. (Actually, it uses the major and minor device number to do this when starting the "md" kernel module, but it is based on the disk controller port.) You can see that you can have anything on a disk when you first connect it to a disk controller port. It can be completely all zeros, or have random bits set, or a test pattern from the disk manufacturer, or a foreign file-system from a previous use in another piece of equipment. Since the "md" device is connected to the disk, it will also have the same unknown contents when FIRST connected. If the unRAID server detects a special pre-cleared signature on the disk, it will skip a lengthy clearing step it needs to perform when you first attach a new disk to a previously un-assigned slot in the array. Otherwise, it MUST write all zeros to the drive before letting you use the array. This is to keep a valid parity calculation without having to read all the other drives. When first attached to an array, and written with zeros, the drive is "unformatted" It will stay un-formatted until you press the "Format" button on the management interface. I think that is where you are right now. The un-formatted drive will not mount as a reiser file-system, since it has no file-system (yet). I originally had 4 storage drives in the server plus a cache drive and a parity drive. The original drive I reported as drive #5 (the one showing up as unformatted) has been in the array for a few months now. I installed 6 total drives when I built the machine and added this disk a little while ago to increase storage. The drive was approximately half full (~700Mb out of 1.5Tb) and I formatted it when I installed the drive.Sounds quite normal. When these first issues arose, I added in a disk that was sitting in the case, but had not been added to the array. I mistakenly called this a hot-spare as to me it was.we sometimes call these a warm-spare. In my ignorance I brought the array offline and then when to the Devices page and selected this existing "hot-spare" drive and added it to the array. I do not remember formatting it as things are getting a little convoluted at the moment.If it had been pre-cleared, you would have had a "Format" button visible when you next started the array. The disk would show as un-formatted. If the disk was not pre-cleared, the array would first go through a lengthy clearing of the newly assigned drive, and then it would start the array and also show you the "Format" button. (and the disk would still show as un-formatted until you pressed the "Format" button) I have never plugged in or disconnected a drive while the machine had power, ever. I always shut the machine down as I read that unRaid was not a hot-swappable environment even though my server hardware is. Excellent. You have done your homework. I personally power down and also unplug the server from the wall, since the MB has some power on it even if just "powered down" I'd hate to brush some cable against something carrying voltage when just trying to plug in a new drive. Unplugging from the wall makes certain that can't happen. Wearing a wrist-grounding anti-static strap helps to ensure I don't zap something from static electricity. It has been about 15 minutes now and the command that you gave me "od -x /dev/md7 | head -30" has not returned anything to my telnet screen at all. Not sure what it is supposed to be doing. I'm guessing it will run for a few hours... let it run till it ends. It will confirm what I'm thinking is the case. Do not press the "Format" button until it finishes and we can evaluate what it prints. The "od" command does not print to its output repeating lines of zeros. I'm guessing the drive was cleared, but never formatted. If the drive had been formatted, the "od" command would have returned in a second or two. Based on what I see, it appears that all of my data is being displayed in windows explorer. The file counts and directory counts seem to be correct. also a good sign. You confused me originally stating it was one of your data drives that had failed. It was this statement that confused me: but one of the drives was showing unformatted and all of the data from that drive seemed to be missing from the share. When I checked the disk in windows explorer it was not present. Since you now see all your files and folders, I think you are well on the way to recovery and expansion of the usable space on your server. (Once you format the newly installed disk, it will be usable) We'll soon see if the "reconstruction" of the new disk simply wrote the zeros it had reconstructed from parity and the other data disks. Joe L.
January 23, 201016 yr Author It finished running and gave me the following: ~~~~~~~~~~~~~~~~~ 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 25652075360000 ~~~~~~~~~~~~~~~~~
January 23, 201016 yr Author OK. I'm gonna do it..... Joe.. Thanks for all of your help on this and educating a poor dumb lob like me. I really appreciate it. Hopefully I will only have one more post and that is to say that it worked. Go have some fun. It is Saturday after all!!
January 23, 201016 yr Author Done and done!! The format only took a few minutes and the drive is showing up in good health (green) and shows all 1.5T ready to use. Thanks again.. You're awesome!
January 24, 201016 yr OK. I'm gonna do it..... Joe.. Thanks for all of your help on this and educating a poor dumb lob like me. I really appreciate it. Hopefully I will only have one more post and that is to say that it worked. Go have some fun. It is Saturday after all!! Went out to a dinner-dance. Had some fun dancing with my wife. (I can program AND do the Waltz, Fox-Trot, Cha-Cha, Mambo, Tango, Hustle, Rumba, Swing, Samba, etc.... ) As you said, it was Saturday night. Happy to learn everything worked. Joe L.
Archived
This topic is now archived and is closed to further replies.