July 18, 200817 yr Author The question is, is there any test that can be done FAST, that can determine with some reliability, that the array/parity is healthy, or not? The basic crude/fast sanity check for the array is the status bit in the super.dat superblock file combined with the checksum of the superblock. If it does not indicate the array was shut down cleanly, then parity is suspect. If the bit shows the array was stopped cleanly, and the super.dat checksum is corrrect, then the assumption is parity is correct. Parity does not just get"out of sync" That test for super.dat sanity is already in place. I agree that by design parity should never get out of sync. But if you happened to add a drive while the clear disk bug was in place and your drive did not get cleared, parity could be out and you'd not know it without a parity check. The guys with 1000s of parity errors that filled up his ramdisk and crashed unRAID - what caused that? Users can go into a telenet prompt and accidently write something to the "sd?" device instead of the "md?" device and mess up parity. And don't forget Murphy, last I checked he was alive and well. The idea of a quick and dirty parity sanity check would likely find a number of users that thought they were protected but aren't - likely due to something screwy that they did. This is not nearly as important or useful as the scheduled parity check. But in the world of safety, a nice-to-have feature IMO.
July 18, 200817 yr Author Active thread, missed this one ... like disks not cleared and wrong parity disk assigned in the slot Here's what I don't understand, what does a disk not cleared have to do with invalid parity. When you add a new disk to the array, unRAID will write binary zeros to the entire disk and then add it to the array. Since the disk is full of zeros, it has no impact on parity and can just be added. (You knew that). But there was a bug in at least one (maybe several) unRAID versions where the filling with zeros was not happening and the disk was quickly added to the array. If you added a disk during this time your parity is not accurate as a result. Checking some random sectors later on in the disk (where nothing has been written yet) would very likely show parity errors.
July 21, 200817 yr Author Here's another one ... http://lime-technology.com/forum/index.php?topic=2264.msg17255#msg17255 How many of these are out there? If we had a parity sanity check, these would get detected!
July 21, 200817 yr At the very least, I think the FIRST instruction in all future releases should be to perform a full parity check. It will save a lot of people from having unexpected problems. Joe L.
July 24, 200817 yr Author Two more users are reporting getting thousands of parity errors. http://lime-technology.com/forum/index.php?topic=2273.0 It looks like hardware problems and not unRAID bugs, but still begs the question as to what is unRAID's responsibility in letting users know if there are catostrophic array integrity problems. The more user reported problems like this that are posted, the more important I think the sanity parity check and scheduled parity check features become. These 2 users know they have a problem and are working to resolve them. They are the "lucky ones". The people I'm more worried about are those that don't know. If parity is out of whack, and a bad sector develops, unRAID will dutifully reconstruct the bad sector (maybe 8 sectors=4K) using flawed parity and write bad data back to the bad sector location forcing a remap and supposedly repairing the problem. The result - small spurious data errors will be introduced into the array with almost no clue as to what or where. If a whole drive were to fail, the results would be more spectacularly bad - but at least more obvious. Not sure which is worse ... Best advice to everyone: Do a parity check today! If parity is good (which it is for a huge percentage of users) unRAID is wonderful, but if you have some hidden hardware or other problem that has caused parity to get out of whack, you are at high risk for disappointment when a sector or disk fails!
July 24, 200817 yr Two more users are reporting getting thousands of parity errors. http://lime-technology.com/forum/index.php?topic=2273.0 It looks like hardware problems and not unRAID bugs, but still begs the question as to what is unRAID's responsibility in letting users know if there are catostrophic array integrity problems. The more user reported problems like this that are posted, the more important I think the sanity parity check and scheduled parity check features become. I agree 1000% with this advice, and I did a full parity check last night. On my older, slower IDE based array, a full parity check runs for over 700 minutes, but it found zero errors. Most important is that it would have detected any issue if one existed, and that is the point that bjp999 is making. I think I put together a small script to do a automatic monthly parity check... (Tom, this would be a good addition to the "Feature Request" laundry list, with a configurable "cron" entry similar to the "mover" script. I would even make it enabled by default.) Joe L.
July 24, 200817 yr I put the following commands in a file named monthly_parity_check.sh I'll invoke it via the "go" script every time I reboot. It will perform a scheduled parity check once a month on the 1st at midnight. My server already has a script that monitors status, so that script will e-mail me hourly status reports when the parity check progresses. (It checks the array status once an hour) Joe L. #!/bin/sh crontab -l >/tmp/crontab grep -q "/root/mdcmd check" /tmp/crontab 1>/dev/null 2>&1 if [ "$?" = "1" ] then echo "# check parity on the first of every month at midnight:" >>/tmp/crontab echo "0 0 1 * * /root/mdcmd check NOCORRECT 1>/dev/null 2>&1" >>/tmp/crontab cp /tmp/crontab /var/spool/cron/crontabs/root- crontab /tmp/crontab fi Edit: added line to copy new /tmp/crontab to file in /var/spool/cron/crontabs read by unraid management interface. That way, changes made using it will not overwrite this entry for the monthly parity check. Edit: Added NOCORRECT key-word, so that the monthly check will not correct parity, but check it. This will prevent a bad data disk drive from overwriting parity with bad parity calculations because the data read from it is inconsistent.
August 1, 200817 yr I woke up this morning and saw a parity check running. My first instinct was to say "what happened?" but then remembered... it was the 1st of the month. The script I added, described in the prior post in this thread, works exactly as expected. My first scheduled and automatic monthly parity check will be complete in a few more hours. Joe L.
August 2, 200817 yr Mine worked (as expected) and finished before I got up. Good Job. Perhaps we should put a commented line in there for those who want it done weekly. Had I seen any errors, I would have switched it to weekly. For now it's good.
August 2, 200817 yr Thanks for the reminder guys - I don't have it automated, but I did a parity check for the first time since June 11 (when I did my last storage increase) ... all good! Cheers, Bill
August 18, 200817 yr Does the script give you a fatal error message on April 1st? I put the following commands in a file named monthly_parity_check.sh I'll invoke it via the "go" script every time I reboot. It will perform a scheduled parity check once a month on the 1st at midnight. My server already has a script that monitors status, so that script will e-mail me hourly status reports when the parity check progresses. (It checks the array status once an hour) Joe L. #!/bin/sh crontab -l >/tmp/crontab grep -q "/root/mdcmd check" /tmp/crontab 1>/dev/null 2>&1 if [ "$?" = "1" ] then echo "# check parity on the first of every month at midnight:" >>/tmp/crontab echo "0 0 1 * * /root/mdcmd check 1>/dev/null 2>&1" >>/tmp/crontab crontab /tmp/crontab fi
August 18, 200817 yr Does the script give you a fatal error message on April 1st? I'm sure that can be arranged. Bill
September 8, 200817 yr I put the following commands in a file named monthly_parity_check.sh I'll invoke it via the "go" script every time I reboot. It will perform a scheduled parity check once a month on the 1st at midnight. My server already has a script that monitors status, so that script will e-mail me hourly status reports when the parity check progresses. (It checks the array status once an hour) Joe L. #!/bin/sh crontab -l >/tmp/crontab grep -q "/root/mdcmd check" /tmp/crontab 1>/dev/null 2>&1 if [ "$?" = "1" ] then echo "# check parity on the first of every month at midnight:" >>/tmp/crontab echo "0 0 1 * * /root/mdcmd check 1>/dev/null 2>&1" >>/tmp/crontab crontab /tmp/crontab fi Sorry for being dense but is all I have to do is copy this to a .sh file like you say and then add a line to the "Go" script like this: before: #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & cp /boot/smarttest.sh /etc/cron.daily/ After: #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & cp /boot/smarttest.sh /etc/cron.daily/ /boot/monthly_parity_check.sh I dont need to copy this anywhere??? I can just run it right from the key?? Sorry, still learning
September 8, 200817 yr You can create the "monthly_parity_check.sh" file in the /boot folder (the top level folder on the flash drive) and yes, you can run it there by typing its name on the command line. When invoked, it will add an entry to the scheduled tasks for a monthly check of parity. that entry will be gone if you reboot the server. The additional line added to the end of the "go" script will run the same script and re-create the scheduled task every time you reboot since "go" is executed every time you reboot. Joe L.
September 8, 200817 yr Cool just wanted make sure I had the command correctly typed in the GO script. Thanks alot!!!!
September 8, 200817 yr Ok, sorry to be a pain here but I am having some trouble. I copied the code to a file called monthly_parity_check.sh using Editpad Lite. I copied that file to the flash drive (on the root). At the root@Tower:~# prompt I typed: /boot/monthly_parity_check.sh and got a /bin/sh^M :bad interpreter: No such file or directory. I also went directly to the boot folder to run just monthly_parity_check.sh at the prompt and got the same thing. I also ran a chmod +x and still have the same results. I know it is my very limited Linux skills that are holding me back....what am I doing wrong
September 8, 200817 yr Your file has embedded carriage-returns and newlines. Unix, unlike windows, uses just newlines. To fix, you need to get rid of the carriage returns (the control-M characters it is complaining about) You can do this fromdos <monthly_parity_chek.sh >mpc.sh mv mpc.sh monthly_parity_check.sh
October 2, 200817 yr I followed the instructions but my 1st of the month parity check didn't happen as scheduled (in the monthly cron folder). So I re-read this thread and checked everything again. The only strange thing I saw was that the monthly_parity_check.sh file looks "hidden" when I copy it over to my windows machine. I don't know if that means anything though. I"m on version 4.3.3 and here is a copy of my go file: #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & sleep 30 for i in /dev/md* do blockdev --setra 2048 $i done cp /boot/monthly_parity_check.sh /etc/cron.monthly/ For completeness I set up everything again - even running the fromdos <monthly_parity_check.sh >mpc.sh mv mpc.sh monthly_parity_check.sh to make sure the file is unix compatible. And I did the chmod +x monthly_parity_check.sh to make sure the permission is correct. But when I am in /boot and type monthly_parity_check.sh I get nothing - the prompt comes back immediately. Should this kick off a parity check or am I missing something? When I go to my webpage (after refreshing it) it does not show any parity check in progress as I was expecting. Here is the file as copied: #!/bin/sh crontab -l >/tmp/crontab grep -q "/root/mdcmd check" /tmp/crontab 1>/dev/null 2>&1 if [ "$?" = "1" ] then echo "# check parity on the first of every month at midnight:" >>/tmp/crontab echo "0 0 1 * * /root/mdcmd check 1>/dev/null 2>&1" >>/tmp/crontab crontab /tmp/crontab fi Can someone tell me how to check what is going on?
October 2, 200817 yr I followed the instructions but my 1st of the month parity check didn't happen as scheduled (in the monthly cron folder). So I re-read this thread and checked everything again. The only strange thing I saw was that the monthly_parity_check.sh file looks "hidden" when I copy it over to my windows machine. I don't know if that means anything though. I"m on version 4.3.3 and here is a copy of my go file: #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & sleep 30 for i in /dev/md* do blockdev --setra 2048 $i done cp /boot/monthly_parity_check.sh /etc/cron.monthly/ For completeness I set up everything again - even running the fromdos <monthly_parity_check.sh >mpc.sh mv mpc.sh monthly_parity_check.sh to make sure the file is unix compatible. And I did the chmod +x monthly_parity_check.sh to make sure the permission is correct. But when I am in /boot and type monthly_parity_check.sh I get nothing - the prompt comes back immediately. Should this kick off a parity check or am I missing something? When I go to my webpage (after refreshing it) it does not show any parity check in progress as I was expecting. Here is the file as copied: #!/bin/sh crontab -l >/tmp/crontab grep -q "/root/mdcmd check" /tmp/crontab 1>/dev/null 2>&1 if [ "$?" = "1" ] then echo "# check parity on the first of every month at midnight:" >>/tmp/crontab echo "0 0 1 * * /root/mdcmd check 1>/dev/null 2>&1" >>/tmp/crontab crontab /tmp/crontab fi Can someone tell me how to check what is going on? Type crontab -l You should see the two lines added by the script you created above. All the script does is add the two lines if they are not already there. It should return immediately to the prompt. To see it as it run interactively, type sh -xv monthly_parity_check.sh It will show you the commands and variables as it runs. I think the biggest issue is the line you added in your "go" script. The command you used copies the script, it does not run the script... I think it needs to. It does not need to copy the script to cron.monthly at all. I think you will get everything working if you change the line in your go script to /boot/monthly_parity_check.sh you do not need to copy it to the monthly cron folder. Since the "go" script only runs when you reboot, if you do not plan on rebooting in the next month, you might want to invoke /boot/monthly_parity_check.sh manually by typing ot on the command line. That will run it, and install the line for the task in the chronological scheduler. Joe L.
October 2, 200817 yr Joe L. thanks for the quick reply. I made the changes to my go as you suggested and went ahead and rebooted. I do see it listed when I do crontab -l as: "# check parity on the first of every month at midnight:" "0 0 1 * * /root/mdcmd check 1>/dev/null 2>&1" So I guess everything checks out - I'll just wait until next month and see what happens. Thanks for your help.
October 2, 200817 yr You could change the "0 0 1" in the script to "0 0 2", re-run it, and wait about 30 minutes. It will start at midnight on the 2nd of the month. Joe L.
Archived
This topic is now archived and is closed to further replies.