February 9, 201016 yr Looks good. This time it ran to completion: # 1 Extended offline Completed without error 00% 11588 -
February 9, 201016 yr Author So does that mean the parity drive and it's cable are no longer suspect?
February 9, 201016 yr Drive /dev/hda has some errors though in your syslog Feb 8 15:52:05 Queeg kernel: hda: task_no_data_intr: status=0x51 { DriveReady SeekComplete Error } Feb 8 15:52:05 Queeg kernel: hda: task_no_data_intr: error=0x04 { DriveStatusError } Feb 8 15:52:05 Queeg kernel: hda: possibly failed opcode: 0xb0 Is it a 4Gig compact flash drive? As long as the errors are not too frequent you can probably ignore them if the drive is working otherwise.
February 9, 201016 yr So does that mean the parity drive and it's cable are no longer suspect? if /dev/sdc is your parity drive (currently unassigned), I'd say it looks like it is working.
February 9, 201016 yr Author Drive /dev/hda has some errors though in your syslog Feb 8 15:52:05 Queeg kernel: hda: task_no_data_intr: status=0x51 { DriveReady SeekComplete Error } Feb 8 15:52:05 Queeg kernel: hda: task_no_data_intr: error=0x04 { DriveStatusError } Feb 8 15:52:05 Queeg kernel: hda: possibly failed opcode: 0xb0 Is it a 4Gig compact flash drive? As long as the errors are not too frequent you can probably ignore them if the drive is working otherwise. Yes, is it a 4Gig compact flash drive. It is the boot drive.
February 9, 201016 yr Author So does that mean the parity drive and it's cable are no longer suspect? if /dev/sdc is your parity drive (currently unassigned), I'd say it looks like it is working. Ok, then. The RAM and the partiy drive have both checked out good. That leaves it to either one of the data drives or math co-processor/parity calc as the problem. How do I validate data drives without harming the data on them?
February 9, 201016 yr Drive /dev/hda has some errors though in your syslog Feb 8 15:52:05 Queeg kernel: hda: task_no_data_intr: status=0x51 { DriveReady SeekComplete Error } Feb 8 15:52:05 Queeg kernel: hda: task_no_data_intr: error=0x04 { DriveStatusError } Feb 8 15:52:05 Queeg kernel: hda: possibly failed opcode: 0xb0 Is it a 4Gig compact flash drive? As long as the errors are not too frequent you can probably ignore them if the drive is working otherwise. Yes, is it a 4Gig compact flash drive. It is the boot drive. apparently nobody told the linux kernel it does not know how to "seek" and give a "seek complete" response
February 9, 201016 yr A thread with that exact issue is here: http://lime-technology.com/forum/index.php?topic=5229.0 Basically, process of elimination. You've eliminated RAM and the parity drive itself. Next you can copy some large files and do md5sum comparisons on each of the disks in turn. Then if no single disk shows a glaring problem, assign only half of the disks and run some tests creating initial parity and then checking it, then the assign the other half and run tests... as I said, process of elimination. Read through the thread... that poster has just replaced an SATA controller. He ie reasonibly certain his older disk controller card is the cause of his errors (and we'll know more in a few more hours when he get to do a parity calc with all his drives and a subsequent parity check.) Joe L.
February 10, 201016 yr Author I ran reiserfsck to check the data drives. I ran the check multiple times to verify consistent results every run. /dev/sdb1 had two errors that were only required --fix-fixable. /dev/sda1 has 139 errors there will require --rebuild-tree So I decided to copy /mnt/disk1/* to /mnt/disk2/backup. I was then going to pre-clear sda1. But, now that I have copied onto sdb1 it's showing lots of errors that will require --rebuild-tree. The cables were recently replaced on both and the PCI sata adapter was also replaced recently to increase the sata ports. The parity drive is on the same sata adapter. I'm not certain how to proceed. sda1_check.zip
February 10, 201016 yr I ran reiserfsck to check the data drives. I ran the check multiple times to verify consistent results every run. /dev/sdb1 had two errors that were only required --fix-fixable. /dev/sda1 has 139 errors there will require --rebuild-tree So I decided to copy /mnt/disk1/* to /mnt/disk2/backup. I was then going to pre-clear sda1. But, now that I have copied onto sdb1 it's showing lots of errors that will require --rebuild-tree. The cables were recently replaced on both and the PCI sata adapter was also replaced recently to increase the sata ports. The parity drive is on the same sata adapter. I'm not certain how to proceed. Some of the reiserfsck options warn against running them more than once. I trust those you re-ran were not any of those. No knowing how you copied the files, I cannot give advice. It still sounds as if you have not started any process of elimination. The two attached files do not show any repair, they just show you have corrupted filesystems. If you copied files off of the disks while their file-systems were corrupt they might be good copies, or not... The files are named after physical disks... did you run reiserfsck on the /dev/sd?1 partitions, or the /dev/md? partitions? Joe L.
February 10, 201016 yr Author I only ran the --check multiiple times which is read only. cp /mnt/disk1/* /mnt/disk2/backup/ Also, I used BeyondCompare from a windows machine over the network to do some of the copying. The --check for /dev/sdb1 (disk2) showed no errors after doing -fix-fixxable. So I assumed it was checking out ok. It seemed reasonable to copy the data from disk1 over to it so I could do a pre-clear on disk1. Now, if the data on /dev/sda1 (disk1) was corrupt I might expect the copy to create files on disk2 that might not match the originals on disk1. However, I wouldn't expect it to corrupt the filesystem tree on disk2 drive. Let me know if you disagree. So with what appears now to be filesystem corruption on /dev/sdb1 (disk2) I'm not certain what to deduce from what has happened. More rambling: With the parity drive being attached to the same sata controller and having passed the rigorous tests I'm of the mind that the controller is working correctly. Since the disk2 drive had no serious errors to start with, that the cables are more suspect at this time. But the only reason I am leaning that way is the cables are both new from a single purchase. But the odds that two cables or two drives are experiencing the same problem is reaching. I'm going to run the --check several times and compare to see if the results are exactly the same every time.
February 10, 201016 yr I only ran the --check multiiple times which is read only.good. cp /mnt/disk1/* /mnt/disk2/backup/ Good. Also, I used BeyondCompare from a windows machine over the network to do some of the copying. The --check for /dev/sdb1 (disk2) showed no errors after doing -fix-fixxable. So I assumed it was checking out ok. It seemed reasonable to copy the data from disk1 over to it so I could do a pre-clear on disk1. Now, if the data on /dev/sda1 (disk1) was corrupt I might expect the copy to create files on disk2 that might not match the originals on disk1. However, I wouldn't expect it to corrupt the filesystem tree on disk2 drive. Let me know if you disagree. I agree with you, it should not cause corruption to the file-system on disk2.So with what appears now to be filesystem corruption on /dev/sdb1 (disk2) I'm not certain what to deduce from what has happened. I can deduce that something other than disk1 and the cabling to it is the issue. (It might have read errors, but that would not cause corruption to the file-system on disk1.) More rambling: With the parity drive being attached to the same sata controller and having passed the rigorous tests I'm of the mind that the controller is working correctly. I would not jump to that conclusion. See the other post I mentioned earlier, each of the two drives connected to the disk controller would work independently, but when both disks were accessed in a parity calc at the same time, random results occurred. Since the disk2 drive had no serious errors to start with, that the cables are more suspect at this time. But the only reason I am leaning that way is the cables are both new from a single purchase. But the odds that two cables or two drives are experiencing the same problem is reaching. I'm going to run the --check several times and compare to see if the results are exactly the same every time. I'm thinking it will be consistent, since you are only accessing one disk at a time. I suspect your problem is when you are accessing multiple disks at one time, and demands on the system are greater, and it could still be almost anything from your power supply, to your RAM (the disks were idle when it was being tested, so it could be affected by voltage variations when the disks are seeking, etc) to a specific disk controller card, as in the other thread. As I said, process of elimination is the only way to identify the issue. Unless you are seeing CRC errors, I would not suspect the SATA cables themselves. As I said earlier, un-assign some (half) or your data disks and see if you can create good/consistent parity on the remaining assigned drives. You'll need to press the button labeled "restore" to Set a new disk configuration and compute parity... Then check parity on the subset of drives. The file-system corruption to me indicates an issue in writing to the drives. Again, it could be memory or power supply issues or disk controller/Motherboard issues. Until you can get something (a subset of hardware) to be consistently good, you'll just be spinning your wheels and pulling out your hair. If you cannot assign a single data drive and a parity drive and get consistent results then try a different data drive on a different controller. If it works, suspect the first controller, or RAM, or the power supply, or power supply splitters (too large a voltage drop across poor connections) Joe L.
February 10, 201016 yr Author Is it valid to try the parity check with the subset of drives without first fixing the filesystem errors?
February 10, 201016 yr Is it valid to try the parity check with the subset of drives without first fixing the filesystem errors? Yes, it is perfectly OK. To parity, bits is bits. It has no concept of a file system , or of files... parity is just bit calculations. Joe L.
February 10, 201016 yr Author Ok. I have: 1. Unassighed disk2. 2. Restored the array. Now, only disk 1 is in the array. 3. Assigned the parity disk. 4. Started the array. Parity-Sync is running. This will take a while. Also, I had earlier set the drives to never spin down and didn't change that before starting the Parity-Sync.
February 11, 201016 yr Author The Parity-sync finished. So I pressed the button to do a parity check. The web page said it was waiting for something and became unresponsive. Five minutes later it would not refresh so I tried to telnet into the system but that failed. I pressed the shutdown button on the machine but it didn't respond to that either. So, it appears locked up for some reason. What now? Dave
February 11, 201016 yr The Parity-sync finished. So I pressed the button to do a parity check. The web page said it was waiting for something and became unresponsive. Five minutes later it would not refresh so I tried to telnet into the system but that failed. I pressed the shutdown button on the machine but it didn't respond to that either. So, it appears locked up for some reason. What now? Dave You probably eliminated the disk that was un-assigned from being your primary cause of problems. Can you log in via the system console? If you cannot shut down the server with the power button, then something very basic is wrong. Did you hold it down for 10 seconds or so? I still suspect RAM, power supply, or MB. Joe L.
February 11, 201016 yr Author Nope, system console is unresponsive. Holding down the button for 10 seconds (essentially power off) is the only thing that worked. I started it back up and upon browsing the page, the Parity check is in progress at .3%. It's like it didn't even know it shutdown. Anyway, it's doing the parity check now. At 1.2% done it has 30047 Synck errors so far. I have stopped it and am doing parity sync on the other drive (disk2). This will take a while.
February 11, 201016 yr Nope, system console is unresponsive. Holding down the button for 10 seconds (essentially power off) is the only thing that worked. I started it back up and upon browsing the page, the Parity check is in progress at .3%. It's like it didn't even know it shutdown. Anyway, it's doing the parity check now. At 1.2% done it has 30047 Synck errors so far. Should I stop it and do the Parity Sync on the other drive? Yes...if the parity errors are still showing, it sounds like your problem still exists... but you never finished calculating parity with that disk? did you? If so, there will be tons of parity errors. It needs to perform a full parity calc once before you can check it again. You might want to run a tail -f /var/log/syslog in another telnet session to see if you can catch any errors if/when the server crashes.
February 11, 201016 yr Author Parity-sync finished on Disk2. Parity check is running. As of 23% there are 54219 sync errors. Same problem both drives. I'd have to say that all the fun has just about been wrung out of this process.
February 11, 201016 yr Parity-sync finished on Disk2. Parity check is running. As of 23% there are 54219 sync errors. Same problem both drives. I'd have to say that all the fun has just about been wrung out of this process. You've eliminated each of the drives. That leaves memory, motherboard, disk controller, power-supply. If you have more than one memory strip, try running with only one, or only the other. (least expensive thing to try) Then, as I said, process of elimination. The previous poster that went through this found it to be the disk controller card.
Archived
This topic is now archived and is closed to further replies.