March 30, 200818 yr After leaving the parity sync to run overnight, it completed with 15 errors (assume it was the same 15 sync errors reported initially). The parity is now valid again. I have added only one drive and it is attached to the last Intel SATA connection (#6). The clearing process has begun and we will see how stable it is after that before I consider hooking up the final drive to the JMicron connection. Back to the sync errors, I assume that implies that when the array was operating with the 7 drives and I added new content, some data was written to the 2 new drives (and parity). When I removed the new drives, the parity drive had additional information. During the parity check process with only 5 drives, I assume the missing data was written back to those data drives. I am hoping that no information was lost... Regards, Peter
March 30, 200818 yr Those 15 errors are more likely to be the file system structures written to the disk when it was formatted. The 15 errors seem too low to be user data.
March 31, 200818 yr Rob, That seems to be the case as the last added content still seems valid. However, just to be safe, I am going to delete that recently added content and then add it back once I've determined that no more problems are occurring. Cool runner, Anyway, so far after the install of disk 6, clearing and formatting, it seems to be okay after a couple of reboots. I am running parity check once again to help me regain that warm and fuzzy feeling I used to have. Once that goes well, I will add the 7th drive via the JMicron port and run through the process one more time. Regards, Peter
March 31, 200818 yr Help! I added the 7th drive again to the internal JMicron port. After going through the clearing and formatting process again, all seems well until I reboot. When I do that, the status page shows that a new disk has been found again and wants to go through the clearing and formatting process again! I checked the bios settings (running latest non-beta bios 0903) and the JMicron port is set to AHCI. Does anyone know why I am stuck in this endless loop? Thanks in advance for any help. Regards, Peter p.s. Here are the vital stats: unRaid Server Pro 4.2.4 Asus P5B VM-DO, 2ghz Celeron 440, 1GB (2x512) ram Lexar Flash from Lime-Tech PC Power Cooling 610watt 7 WDC 1TB Greenpower drives (only 6 running now)
March 31, 200818 yr Author Peter, I guess the problem is indeed with the jmicron controller as I am stuck in the same loop. I do have a PCI controller in my server, but I had planned to use all the onboard ports before using it. Tom, Since you are selling boxes with this board, can you please test if this happens with your systems. If it is a driver issue, will moving to the 4.3beta possibly solve this since it is using the updated kernel?
April 2, 200818 yr Cool-runner, I gave up on these Jmicron port project for now. I was hoping that Tom would answer your question first before I proceeded to try anything else. Also, like you I have PCI SATA cards in reserve (Promise SATA TX4) so if need be I can get those going. Right now, the additional 1TB drive is not needed. I got it only because there was a good deal on it. Tom, Are you familiar with the problem we are having? Will 4.3 beta 4 solve our problem? I guess I am not quite ready to try the beta nor am I clear on the return path to 4.2.4 if I have any problems with 4.3. Regards, Peter
April 2, 200818 yr I guess I am not quite ready to try the beta nor am I clear on the return path to 4.2.4 if I have any problems with 4.3. Regards, Peter There is an easy way to test the new release and still revert back to your existing release if the boot fails: There is a file on your flash drive in the root folder named syslinux.cfg. It looks like this: default menu.c32 menu title Lime Technology LLC prompt 0 timeout 50 label unRAID OS menu default kernel bzimage append initrd=bzroot rootdelay=10 label Memtest86+ kernel memtest If you edit it and add three lines at its end, you can choose to boot on either the existing or new version of unRaid. To do this you would NOT replace the existing bzroot and bzimage files with the two from the unzipped download from line-technology, instead, you would download the beta version, unzip it in a folder on your PC, rename the bzroot and bzimage to new names (I use ones named after the new release, in this case bzroot434 and bzimage434 ) and then copy those newly named files to the root of my flash drive. When I re-boot, there is a third menu choice. unRaid NEW Version. If I choose it, I'm running the new release. If I choose the original menu choice, or let it time out, I'm running the old release. Once I'm happy everything is working, I can copy those two files to the normal bzroot and bzimage names so the default is the newer release. Shown below is a sample menu with the ability to boot the current or new release. Again, all you are adding are the three lines at the end. default menu.c32 menu title Lime Technology LLC prompt 0 timeout 50 label unRAID OS menu default kernel bzimage append initrd=bzroot rootdelay=10 label Memtest86+ kernel memtest label unRaid NEW Version kernel bzimage434 append initrd=bzroot434 rootdelay=10
April 3, 200818 yr Joe, That's slick! Thanks a lot for the help. I am getting a free Linux education here! I believe that I will do the following: 1. Make a backup of my current flash drive 2. Edit the cfg file per your example, boot into the beta, and play around in it reading files only (no writing yet) 3. Assuming all is well in #2, I will shut down, and add the 7th drive to the JMicron port, reboot, clear and format, and reboot a few times to see if the array remembers the drive (unlike 4.2.4) 4. Assuming all goes well in #3, write files to the array and verify operation 5. If #3 fails, then I will remove the 7th drive, deactivate the JMicron port, restore from my backup of the flash drive, and rerun parity check to correct the expected sync errors, and cry quietly in the corner... :'( Thanks again for your help. Regards, Peter
April 3, 200818 yr Peter, Do not "format" or run syslinux on your flash drive. It is already formatted, and already bootable. You only need to copy its config folder elsewhere as long as you do not reformat. To test your upgrade, you only need to copy the two renamed files bzroot434 and bzimage434, modify syslinux.cfg as described and then select the new menu choice when rebooting. If all goes well, you can stop, rename bzroot to bzroot421, bzimage to bzimage421, and then copy bzroot434 to bzroot, and bzimage434 to bzimage. Oh yes... don't cry. Joe L.
April 4, 200818 yr Well, I went straight to beta 5 as that was the latest available this morning. After booting and messing around with the array, everything seemed fine with the 6 drives attached. I then attached #7 to the JMicron port and activated it. I went through the normal clearing and formatting process. After several reboots it still seemed okay. At least it was better than 4.2.4 where every reboot resulted in drive 7 being identified as a new new drive over and over again. So I decided to copy some content over. The copy went well and I verified that it made it to the array okay and was accessible... I decided to run a parity check. :'( Immediately sync errors began to appear. I have left it running and will check it again tomorrow but this is not very promising. I can only see four possible explanations: 1. JMicron port on the P5B VM-DO is not compatible with unRAID - This seems unlikely as this is the stock board used for Lime-Techs builds 2. My JMicron port is defective - This seems unlikely as the coincidence of it occurring at the same time as Cool_runner is rather bizarre 3. The last WD 1TB drive is defective - I assumed that unRAID reads the S.M.A.R.T. data and reports issues. Also during boot, there are no errors reported by the BIOS as it reads the S.M.A.R.T. data. As soon as the parity finishes, I will run dedicated diags on the drive 4. I have a bad SATA cable - I will switch that out during the diags run Anyone have any other advice? Tom, Can you jump in here? Regards, Peter
April 4, 200818 yr The last parity check completed with 7,474 errors. I really do not know what to make of this. I am rerunning the parity check to see if any more sync errors will be generated. So far it has none. Once it completes, I will go through the diags mentioned above. Regards, Peter
April 6, 200818 yr Author Can someone please tell us what sync errors mean when running a parity check? Does it affect the data stored on the drives at all? I am planning to just move the drives to my PCI controller, but before I do this I would like to know what these errors are because I will most likely get them if I run parity check.
April 7, 200818 yr Can someone please tell us what sync errors mean when running a parity check? Does it affect the data stored on the drives at all? I am planning to just move the drives to my PCI controller, but before I do this I would like to know what these errors are because I will most likely get them if I run parity check. A sync error or 'Parity incorrect' error is just an indication that the parity calculation across all drives for the same bit position returned a non-zero value, which is wrong. That means that the calculation of parity at the Parity check has returned a different value than the calculation of parity when it was written to the parity drive. So something has changed, perhaps a drive error (bad data returned), or data was changed that was not reflected in the parity calculation, or a drive is no longer part of the parity calculations. A parity check or sync/rebuild does NOT affect the data drives, it only writes to the parity drive. A parity sync will rewrite the entire parity drive, a parity check will only correct wrong parity blocks. I don't understand why you think you "will most likely get them if I run parity check". As far as we can tell to this point, you and Peter have 2 very different problems. You seem to have a bad flash that won't update, which of course results in any added drives appearing as new every time. That's why I advised that adding the drives is a waste of time until the flash problem is fixed. Peter has said that he was able to reboot several times and the config was still okay, so his flash seems to be working correctly, but the drive is dropping out. We can't tell more than that because we haven't seen a syslog yet. The last parity check completed with 7,474 errors. I really do not know what to make of this. I am rerunning the parity check to see if any more sync errors will be generated. So far it has none. Once it completes, I will go through the diags mentioned above. Regards, Peter It would appear the drive attached to the JMB port may have dropped out again, which would cause a lot of 'sync errors'. I can't tell very much without a syslog. Would you capture and attach it? I wish there was a Troubleshooting tip somewhere prominent that if a user has a need for troubleshooting help, they should start by posting a syslog and summary of their hardware, along with the description of the problem and the exact error messages if any. I would love to see 'problem with unRAID' considered synonymous with 'capture the syslog'. Some problems don't need a syslog, but most do. It is our window into the internal workings of the Linux kernel and the unRAID driver, what it is seeing and what it is doing.
April 7, 200818 yr Rob, Thanks for the clarification on sync errors. Perhaps I did not look hard enough but I initially did not find information on how to access the syslog (or what it was for that matter). After figuring it out, I dumped a copy to a text file for review. There were no messages indicating errors, only that a new drive had been detected. Sadly as I understand it, the syslog is not archived between boots and the relevant log is now long gone. Also I was focussed on getting my array back up so I tried all the things mentioned in the posts above. At this point the array seems back to stable running 4.3beta6 with the 7th drive on the JMicron port. The only other thing I was able to test/change was the SATA cable. I was not successful in running the Western Digital diags. The diagnostic program would not load. I tried several times but nothing happened, just a black screen. Although I have continued to copy more content to the array, I have not yet deleted the source files as I am still not 100% confident that everything is back to normal. As mentioned a few posts back, I also have 2 of the Promise TX4 boards sitting unused in their boxes. I am contemplating installing one and moving the drive attached to the JMicron over to the Promise. As far as I can tell no data has been written to it as of yet. Cool_runner, Sorry for hijacking your thread. I misread it in that we were having the same problems (or at least very similar). Regards, Peter
April 7, 200818 yr The most effective way to diagnose a problem is to capture a syslog. The most detailed and articulate play-by-play description of your problem are not nearly as important as posting your syslog here. There are a TON of messages giving instructions for copying your syslog to your USB stick. Here is the basic command ... cp /var/log/syslog /boot/syslog.txt I usually date/time stamp my syslog copy ... something like .. cp /var/log/syslog /boot/syslog.040708.txt That way I will have several on the stick at the same time. BTW, it is actually a good idea to keep a "good" syslog on your boot stick - it may help later, if you do have a problem, to see what your log looked like when you didn't have a problem. For example, if you're seeing a suspicious line in your syslog, and you see the same line in your "good" syslog, it is likely not the culprit.
April 7, 200818 yr BTW, it is actually a good idea to keep a "good" syslog on your boot stick - it may help later, if you do have a problem, to see what your log looked like when you didn't have a problem. For example, if you're seeing a suspicious line in your syslog, and you see the same line in your "good" syslog, it is likely not the culprit. I wholeheartedly agree with this advice. If not kept on the flash drive, then keep a folder elsewhere with your old syslogs. As you have said, it is often very useful to have a baseline syslog to compare. Having read a lot of syslogs, there are many 'suspicious' looking lines. Probably the biggest key to understanding syslogs - is learning what to ignore.
April 7, 200818 yr Author I don't understand why you think you "will most likely get them if I run parity check". As far as we can tell to this point, you and Peter have 2 very different problems. You seem to have a bad flash that won't update, which of course results in any added drives appearing as new every time. That's why I advised that adding the drives is a waste of time until the flash problem is fixed. Peter has said that he was able to reboot several times and the config was still okay, so his flash seems to be working correctly, but the drive is dropping out. We can't tell more than that because we haven't seen a syslog yet. Rob, I believe we are experiencing the same problem. It is unfortunate that Tom can't test this because he sells systems based on this board, and if he experienced the same problem with drives on the jmicron ports then it is a driver issue. I will go ahead and attach the drives to my PCI controller, and add them to the array. If I am successful and reboots do not result in the drives being reported as new then it is a driver issue with the jmicron ports. BTW, I have emailed Tom (on Wed, Mar 26, 2008 at 9:49 AM) regarding my flash drive as he suggested in his post, and I have yet to here anything from him.
May 5, 200818 yr Author Well, I finally got a chance yesterday to work on my unRAID machine, and things did not turn out good. I first started by removing the new drives to get back to my old configuration. I then updated to 4.3beta6 and ran a parity check. It returned a whole buch of errors; over 180,000. I am trying to attached the syslog, but the only way I was able to get it down to less than the maximum allowed size was using 7-zip. Unfortunately, it is not a support attachment (*.7z). The zip file comes out to ~500kB, whereas the 7-zip file is only 32kB. Does anyone know how to attach large files?
May 5, 200818 yr Try putting the 7-zip file into a zip file. That should work. What others have done at times is find an online storage place like RapidShare. I'm not sure anyone wants to look at 180,000 errors though!
May 5, 200818 yr Author Try putting the 7-zip file into a zip file. That should work. What others have done at times is find an online storage place like RapidShare. I'm not sure anyone wants to look at 180,000 errors though! Doh! I should have thought of that. The reason I want to attach the whole thing is because there are some messages that appear between the parity errors. By the looks of it, it seems that these errors are only sync erros, but why so many?
May 5, 200818 yr The almost 181,000 'parity incorrect' errors are mainly in 3 series. They begin with every 24th block between 152 and 248, then jump around some with frequencies of every 8th or 16th or 24th, mostly every 8th. After 312, almost all are 8 apart, with an occasional 16 or 24. Later, its hard to find anything but 8 apart. This initial series (about 460 parity errors) carries through 4256, then there are a few normal messages from the unRAID startup, then parity incorrect for 65680 and 65688, then the last of the unRAID setup messages at 17:19:05, then a new series of about 90,700 'parity incorrect' errors picks up with block 262144 at 17:19:06, and occurs every 8 blocks to 1003080 at 17:19:22. Then there is a gap of 4 hours and 33 minutes, then another series (about 89,800 parity errors) picks up at 21:52:57 with block 1738276864, and occurs every 8 blocks through block 1739010896 at 21:53:14. At 22:30:46, the parity check is done, with a very good speed of 52.4MB/s. Summarizing... 17:19:04 checking parity 17:19:04 parity incorrect: 152 ... (about 460 parity errors) 17:19:04 parity incorrect: 4256 ... (further mounting and network setup messages) 17:19:05 parity incorrect: 65680 17:19:05 parity incorrect: 65688 ... (final setup messages) 17:19:07 parity incorrect: 262144 ... (about 90,700 parity errors) 17:19:22 parity incorrect: 1003080 ... long gap, no messages 21:52:57 parity incorrect: 1738276864 ... (about 89,800 parity errors) 21:53:14 parity incorrect: 1739010896 22:30:46 sync done. time=18614sec rate=52474K/sec That's it for the data gathering, now what to conclude, I don't know. It appears that something has written to either a data disk or the parity drive, to 3 areas only, and only to the first sector of each series of clusters, assuming a cluster size of 8 blocks, 16 sectors. The interspersed messages at the beginning are just the unRAID system still getting started. The parity check does not wait for everything to finish mounting. The fact that 4 or 5 transactions were replayed for each data drive, is the normal behavior after an improper shutdown on the previous run of unRAID. An entry in the syslog I thought I would mention, not sure of the significance. kernel: ACPI Warning (tbutils-0217): Incorrect checksum in table [OEMB] - 96, should be 8E [20070126] This *might* be an indication of a bit of corruption in the BIOS. You might want to try updating the BIOS, with the same or newer version, to see if this removes the warning. It did not occur in an earlier syslog, but perhaps the earlier kernel did not check it.
May 5, 200818 yr Author Thanks Rob. Before, when I added the two drives to the array, some writes were performed to the parity disk. Could these "errors" just be the result of that? kernel: ACPI Warning (tbutils-0217): Incorrect checksum in table [OEMB] - 96, should be 8E [20070126] Can someone else with an Asus P5B-VM DO running the latest beta check if they also get this message? Edit: Was any of my data affected by the parity check?
May 5, 200818 yr Before, when I added the two drives to the array, some writes were performed to the parity disk. Could these "errors" just be the result of that? I don't see how, but then I can't think of any process that would write such a pattern to the drive. Was any of my data affected by the parity check? No, a parity check does not touch any data. It does correct the parity information on the parity drive.
Archived
This topic is now archived and is closed to further replies.