August 15, 200817 yr Author I felt the --fix-fixable was fairly safe, the rebuild-tree is much more drastic. In reality, neither should cause problems if run on a good file system, but I really did not want to put the rebuild-tree choice on the web-interface. Perhaps put the file system repair on its own page I think the fix-fixable is fairly safe, the --rebuild-tree takes a really long time. Any form of interruption can have a detrimental effect on your drive's data. There are warnings of this.
August 15, 200817 yr Where would you recommend this scripts sits in your user customisation structure WeeboTech?
August 15, 200817 yr Author Where would you recommend this scripts sits in your user customisation structure WeeboTech? I think package files usually end up in a usr/share/{packagename} directory. If they are modifiable and update over time then those files go in /var/{packagename} Unless there were someway to make it a single inline executable script file, then it would go in bin
August 15, 200817 yr I felt the --fix-fixable was fairly safe, the rebuild-tree is much more drastic. In reality, neither should cause problems if run on a good file system, but I really did not want to put the rebuild-tree choice on the web-interface. Perhaps put the file system repair on its own page I think the fix-fixable is fairly safe, the --rebuild-tree takes a really long time. Any form of interruption can have a detrimental effect on your drive's data. There are warnings of this. It might be best if the "Repair File System" option is removed. It should probably be done from the command line where you can see the output and not from a web-browser where it could be interrupted anyway. The Check option is read-only and will give the advice to run the appropriate fix option. Thoughts? Joe L.
August 15, 200817 yr Author When you run the check or repair is anything displayed on the screen? Keep in mind there is also the --log option. At the very least, the check option should stay. I think fix-fixable should be safe. (with a big warning not to interrupt it no matter what). I would not recommend --rebuild-tree. I ran this last night and it took over an hour or so on a 1tb drive.
August 15, 200817 yr When you run the check or repair is anything displayed on the screen? Keep in mind there is also the --log option. Nothing shows up on the screen until it is complete and the file system re-mounted and samba re-started. Then the output is visible. The output is also written to the end of the syslog. At the very least, the check option should stay. I agree there. It is a read-only operation, so no harm. I think fix-fixable should be safe. (with a big warning not to interrupt it no matter what). I would not recommend --rebuild-tree. I ran this last night and it took over an hour or so on a 1tb drive. I'm not sure if you can interrupt the fix-fixable from the web-browser. Once running, I think the only way would be to kill the awk process. The awk process is NOT multi-threaded, therefore cannot be listening to other requests from the browser while it is running reiserfsck. I'm thinking even if you close the browser the reiserfsck will continue, the next steps to re-mount the drive will then occur, and finally the awk web-server will write the results to the HttpService. If there is nothing there to listen, it will block there waiting for the next request. In any case, the rieserfsck will have finished. You can experiment with the "Check File System" option to prove my theory. It is exactly the same other than the option fed to feiserfsck. Joe L.
August 16, 200817 yr Whoever wrote the routine to convert the values to human readable ones must have had a marketing background ROFLMAO!
August 16, 200817 yr I am having an issue where the site comes up and works fine, but if I leave it up for a while (> ~15 minutes) it becomes unresponsive. Awk is running in an putty session. If I go there and press ^C, it closes and I can then restart it and it works fine. Is there anything I can do to troubleshoot this issue? I don't see anything in the syslog. The browser is plain ole IE6.
August 17, 200817 yr I am having an issue where the site comes up and works fine, but if I leave it up for a while (> ~15 minutes) it becomes unresponsive. Awk is running in an putty session. If I go there and press ^C, it closes and I can then restart it and it works fine. Is there anything I can do to troubleshoot this issue? I don't see anything in the syslog. The browser is plain ole IE6. I've been testing with Firefox and with IE7. I just tried with IE6. It does not seem to think the first page request was completed... odd. In any case, try this: On line 30, change print "HTTP/1.0", Status, Reason |& HttpService to print "HTTP/1.1", Status, Reason |& HttpService That seems to work here with both IE6 and IE7 as well as Firefox. (it seems to like HTTP 1.1 better than 1.0)
August 17, 200817 yr That was it! The problem with the "Dup files"page also seems to be gone. Thanks!
August 28, 200817 yr I love this tool and have been using it often. I was wondering if there is an update on the horizon?
August 28, 200817 yr I love this tool and have been using it often. I was wondering if there is an update on the horizon? I've made quite a few changes and enhancements to it since the last posted version. I was waiting to get a chance to finish up a few more before posting a newer version... Those changes should, if I can get it right, allow for binary objects to be sent to the browser, not just text. The real challenge is that awk is a text based utility... it was not designed to be used with binary data, but with strings of readable text. Fortunately, and with the aid of a LOT of google searching, there is a trick I can take advantage of to enable the feature I want to impliment. Since the last version I've added: A plug-in architecture, complete with the ability to define a replacement page for one of the built-in pages. I have a "syslog" plug-in that color codes specific lines. The built-in does not. I have moved all but the main page, the array management page, and the disk management page to plug-ins. It is VERY easy to write a plug-in. The main page will auto-refresh every 60 seconds as long as the array is in the STARTED state. (Tom has already indicated the "md" driver can get confused if you ask for status while it is expanding a drive, so I will let you do any refresh manuall in those abnormal statuses) I calculate my own total of disk space without rounding upward as the "du" command seems to do. This makes for a much more accurate space listing. I've added a lot more status information to the top of the main page. If you are doing a parity check, it will show you the progress... Under array management I've added Spin-Up and Spin-Down buttons. These hopefully will work on all drives. (spin up is done by reading a single random block of data from the target drive. Hopefully, the "random" block will not be in the buffer cache and the drive will need to spin up.) You can stop and start Samba from there too. I took the repair option off of the disk_management page, but added individual spin-up and spin-down buttons. As I mentioned, the syslog page is a plug-in. You can edit it to add your own color coding of lines you are interested in. Here is the entire code of the syslog plug-in. I currently highlight the emhttp log lines in blue, timeout and exceptions in orange, and errors in red. It is easy to add your own regex patterns. The few lines at the top of the plug-in (ADD_ON_URL, ADD_ON_MENU, etc) define how it will be handled by the main menu: BEGIN { #ADD_ON_URL=system_log #ADD_ON_MENU=Syslog #ADD_ON_STATUS=YES #ADD_ON_TYPE=awk theHTML = "<fieldset style=\"margin-top:10px;\"><legend><strong>System Log</strong></legend>" theHTML = theHTML GetSysLog(1000); theHTML = theHTML "</fieldset>" print theHTML } function GetSysLog(numlines, syslog, f) { nl=numlines cmd = "tail -" nl " /var/log/syslog" RS="\n" syslog = "<strong>Syslog (last " nl " lines)</strong>" syslog = syslog " <A href=" MyPrefix "/syslog>Click Here to Download Complete Syslog</A><hr>" while (( cmd | getline f ) > 0) { lower_f = tolower(f) # An example of how to highlight certain lines of output highlight = highlight_end = "" # if the line has the text "emhttp:" make it blue if ( f ~ "emhttp:" ) { highlight="<font color=\"blue\">" highlight_end="</font>" } if ( lower_f ~ "timeout|exception" ) { highlight="<font color=\"orange\">" highlight_end="</font>" } if ( lower_f ~ "error" ) { highlight="<font color=\"red\">" highlight_end="</font>" } syslog = syslog highlight f highlight_end "<br>" } close(cmd) return syslog } A much simpler plug-in now contains useful links: BEGIN { #ADD_ON_URL=links #ADD_ON_MENU=Useful Links #ADD_ON_STATUS=YES #ADD_ON_TYPE=awk theHTML = "<fieldset style=\"margin-top:10px;\"><legend><strong>Useful Links</strong></legend>" theHTML = theHTML "<table cellpadding=0 cellspacing=0>" theHTML = theHTML "<tr><td><a href=\"http://lime-technology.com/forum/index.php\">Lime Technology Support Forum</a></td></tr>" theHTML = theHTML "<tr><td><a href=\"http://lime-technology.com/wiki/index.php?title=UnRAID_Wiki\">Lime Technology Wiki</a></td></tr>" theHTML = theHTML "<tr><td><a href=\"http://lime-technology.com/wiki/index.php?title=Best_of_the_Forums\">Best Of The Forums</a></td></tr>" theHTML = theHTML "<tr><td><a href=\"http://lime-technology.com/wiki/index.php?title=Troubleshooting\">Unraid Troubleshooting</a></td></tr> " theHTML = theHTML "<tr><td><a href=\"http://lime-technology.com/wiki/index.php?title=FAQ\">Frequently Asked Questions</a></td></tr>" theHTML = theHTML "<tr><td><a href=\"http://www.lime-technology.com\">Lime Technology Home Page</a></td></tr>" theHTML = theHTML "</table></fieldset>" print theHTML } Because I am using some newer features in awk you must invoke it with an additional command line argument. It now must be invoked as follows: awk -W re-interval -f unmenu.awk the "-W re-interval" option allows me to use a more complicated regular expression that I'll need for the file-browser plug-in. The main page looks like this now: I've attached a newer version. Put the plug-in files in the same directory as the unmenu.awk program (I've just got them at /boot) There is a ScriptDirectory defined as "." but you can define it to be whatever you desire and put the plug-ins there. The number preface of the plug-in name determines the order in the top menu. The plug-in name must match NN-unmenu-whatever.awk or NN-unmenu-whatever.cgi if you want to write your own. I'm in the middle of making changes to the way HTTP headers are created and added. I want to be able to have a plug-in supply the HTTP headers instead of my main loop, and also I want to impliment "chunked transfer encoding" so we'll eventually be able to write a full file browser that allows you to click on a file name to access it from your browser. As it is, I'm learning far more than I ever really wanted to about HTTP headers and internals. Currently, we are still limited to GET requests, and to text and html objects. There is no "www" root, and all requests must match either a built-in or plug-in URL. Not too shabby for an "awk" script... in fact, pretty darned impressive. (try not to make too much fun of my code... ) Remember, invoke it as awk -W re-interval -f unmenu.awk Oh yes, it is single-threaded. It will NEVER be able to service more than one request at a time. If you are doing a lengthy operation (file-system-check) and waiting for results, do not expect to issue a second request from a different PC, or a different browser. They will go nowhere are there is nothing listening at that point in time from this simple awk-based server. Note: newer version attached further on in this thread. Joe L.
August 28, 200817 yr I just installed and ran this. I noticed that the the new webpage I get a message saying parity is invalid, but on Tom's page it says that it is valid. Who to believe? dave
August 28, 200817 yr I just installed and ran this. I noticed that the the new webpage I get a message saying parity is invalid, but on Tom's page it says that it is valid. Who to believe? dave Probably want to believe Tom's Please type the following command and post the output. It will help me to debug. /root/mdcmd status | strings The code is looking for an entry that in the mdcmd output that does not have an affiliated "md" device and where the status is not DISK_OK to flag that parity is not valid. Joe L.
August 28, 200817 yr Here is the section of output from mdcmd status related to my parity drive. Since the parity drive can be in any slot I determine which slot is the parity drive by looking at two lines... I am interested in any sections in the mdcmd output where the "diskName" is blank. My parity disk is in slot 0 of the array, yours may be different. I scan all the drives to find the one that does not have an "md" device. diskNumber.0=0 diskName.0= <--- I look for this being blank. On data disks it is md1, md2, etc. If blank, I assume it is the parity drive diskState.0=7 diskSize.0=976762552 diskModel.0=ST31000340AS diskSerial.0= 9QJ0JPJS diskNumReads.0=734150 diskNumWrites.0=6871 diskNumErrors.0=0 diskId.0=ata-ST31000340AS_9QJ0JPJS rdevNumber.0=0 rdevStatus.0=DISK_OK <--- I look for this to say DISK_OK. If not DISK_OK, I assume parity is bad. Perhaps a bad assumption?? rdevName.0=sdc rdevSize.0=976762552 rdevModel.0=ST31000340AS rdevSerial.0= 9QJ0JPJS rdevId.0=ata-ST31000340AS_9QJ0JPJS In any case, your command output will help me to learn what to look for. Edit: Do you have an empty slot in your array between two assigned slots? I'm not currently looking for that condition. If so, on line 353 of the script, in "function ArrayStateHTML" change this if ( disk_name[ i ] == "" ) { to this (adding the text in blue): if ( disk_name[ i ] == "" && disk_device[ i ] != "" ) { the resulting logic will then read as follows: parity_valid="" for ( i =0; i<numdisks; i++ ) { if ( disk_name[ i ] == "" && disk_device[ i ] != "" ) { if ( disk_status[ i ] ~ "DISK_OK" ) { parity_valid="Parity is Valid:" } else { parity_valid="<font color=\"red\">PARITY NOT VALID:</font>" } } } Joe L.
August 29, 200817 yr OK, I see a potential issue. I have my parity drive in slot 0, and it does show no name and a status of DISK_OK My slot 1 is empty and shows now name and a status of DISK_NP My slot 2 does have a drive and has a name. Basically I have a single drive on each ide channel so I don't have a master/slave as I have more ide channels than drives. Make sense? diskNumber.0=0 diskName.0= diskState.0=7 diskSize.0=312571192 diskModel.0=WDC WD3200JB-22KFA0 diskSerial.0=WD-WCAMR1939710 diskNumReads.0=1525444 diskNumWrites.0=1537749 diskNumErrors.0=0 diskId.0=ata-WDC_WD3200JB-22KFA0_WD-WCAMR1939710 rdevNumber.0=0 rdevStatus.0=DISK_OK rdevName.0=hda rdevSize.0=312571192 rdevModel.0=WDC WD3200JB-22KFA0 rdevSerial.0=WD-WCAMR1939710 rdevId.0=ata-WDC_WD3200JB-22KFA0_WD-WCAMR1939710 diskNumber.1=1 diskName.1= diskState.1=0 diskSize.1=0 diskModel.1= diskSerial.1= diskNumReads.1=0 diskNumWrites.1=0 diskNumErrors.1=0 diskId.1=ata-_ rdevNumber.1=1 rdevStatus.1=DISK_NP rdevName.1= rdevSize.1=0 rdevModel.1= rdevSerial.1= rdevId.1=ata-_ diskNumber.2=2 diskName.2=md2 diskState.2=7 diskSize.2=117220792 diskModel.2=ST3120026A diskSerial.2=3JT3FTTJ diskNumReads.2=1032027 diskNumWrites.2=98636 diskNumErrors.2=0 diskId.2=ata-ST3120026A_3JT3FTTJ rdevNumber.2=2 rdevStatus.2=DISK_OK rdevName.2=hdc rdevSize.2=117220792 rdevModel.2=ST3120026A rdevSerial.2=3JT3FTTJ rdevId.2=ata-ST3120026A_3JT3FTTJ
August 29, 200817 yr OK, I see a potential issue. I have my parity drive in slot 0, and it does show no name and a status of DISK_OK My slot 1 is empty and shows now name and a status of DISK_NP My slot 2 does have a drive and has a name. Basically I have a single drive on each ide channel so I don't have a master/slave as I have more ide channels than drives. Make sense? diskNumber.0=0 diskName.0= diskState.0=7 diskSize.0=312571192 diskModel.0=WDC WD3200JB-22KFA0 diskSerial.0=WD-WCAMR1939710 diskNumReads.0=1525444 diskNumWrites.0=1537749 diskNumErrors.0=0 diskId.0=ata-WDC_WD3200JB-22KFA0_WD-WCAMR1939710 rdevNumber.0=0 rdevStatus.0=DISK_OK rdevName.0=hda rdevSize.0=312571192 rdevModel.0=WDC WD3200JB-22KFA0 rdevSerial.0=WD-WCAMR1939710 rdevId.0=ata-WDC_WD3200JB-22KFA0_WD-WCAMR1939710 diskNumber.1=1 diskName.1= diskState.1=0 diskSize.1=0 diskModel.1= diskSerial.1= diskNumReads.1=0 diskNumWrites.1=0 diskNumErrors.1=0 diskId.1=ata-_ rdevNumber.1=1 rdevStatus.1=DISK_NP rdevName.1= rdevSize.1=0 rdevModel.1= rdevSerial.1= rdevId.1=ata-_ diskNumber.2=2 diskName.2=md2 diskState.2=7 diskSize.2=117220792 diskModel.2=ST3120026A diskSerial.2=3JT3FTTJ diskNumReads.2=1032027 diskNumWrites.2=98636 diskNumErrors.2=0 diskId.2=ata-ST3120026A_3JT3FTTJ rdevNumber.2=2 rdevStatus.2=DISK_OK rdevName.2=hdc rdevSize.2=117220792 rdevModel.2=ST3120026A rdevSerial.2=3JT3FTTJ rdevId.2=ata-ST3120026A_3JT3FTTJ The fix I described in my prior post should be the fix needed. It is as I predicted. You had a slot un-assigned. Now, when the loop gets to slot 1, it will not think it is the parity drive since the rdevName.1= is blank. (It will be populated in the parity drive) But, I think I also need to look for the DISK_NP value in the parity slot, just in case, and put up a different message.. Try something like this: parity_valid="" for ( i =0; i<numdisks; i++ ) { if ( disk_name[ i ] == "" && disk_device[ i ] != "" ) { if ( disk_status[ i ] ~ "DISK_OK" ) { parity_valid="Parity is Valid:" } else { if ( disk_status[ i ] ~ "DISK_NP" ) { parity_valid="Parity disk not configured:" } else { parity_valid="<font color=\"red\">PARITY NOT VALID:</font>" } } } } I'll zip up a new version and update the attachment... I'm going to include an item or two suggested by RobJ as well. Stay tuned... Joe L.
August 29, 200817 yr That did fix the invalid parity issue. On the Main page it still labels all the unassigned drives as parity. It looks like: Disk Status Status Disk Mounted Device Model/Serial Temp Reads Writes Errors Size Used % Used Free OK parity /dev/hda ata-WDC_WD3200JB-22KFA0_WD-WCAMR1939710 * 1525455 1537767 0 DISK_NP parity /dev/ ata-_ * 0 0 0 OK /dev/md2 /mnt/disk2 /dev/hdc ata-ST3120026A_3JT3FTTJ * 1032027 98636 0 120.03G 7.85G 7% 112.18G DISK_NP parity /dev/ ata-_ * 0 0 0 OK /dev/md4 /mnt/disk4 /dev/hde ata-WDC_WD3200JB-00KFA0_WD-WCAMR2056466 34°C 23376678 1439125 0 320.06G 257.50G 81% 62.56G DISK_NP parity /dev/ ata-_ * 0 0 0 OK /dev/md6 /mnt/disk6 /dev/hdg ata-WDC_WD3200JB-00KFA0_WD-WCAMR4003135 25°C 1147459 8 0 320.06G 316.85G 99% 3.22G
August 29, 200817 yr Looks great! But the problem with the page not fully loading is recurring. I tried changing the HTTP/1.0 to HTTP/1.1, but seemed to have no affect. Any ideas?
August 29, 200817 yr Looks great! But the problem with the page not fully loading is recurring. I tried changing the HTTP/1.0 to HTTP/1.1, but seemed to have no affect. Any ideas? The only line using HTTP/1.0 was the download of the syslog. Your change would not affect normal pages, might affect the download (don't know, never tried what you changed) I am re-writing the section creating the HTTP headers... stay tuned... (use Firefox for now. ) Joe L.
August 29, 200817 yr That did fix the invalid parity issue. On the Main page it still labels all the unassigned drives as parity. You've got to love beta testing... Thanks again, I've fixed that too. If you wish to fix yours, In function DiskStatus() there is a large "for" loop that builds the table for the display of disk information. The top few lines currently look like this: for ( i =0; i<numdisks; i++ ) { if ( disk_name[ i ] == "" ) { disk_name[ i ] = "parity"; } else { disk_name[ i ] = "/dev/" disk_name[ i ]; } Change to look like this (adding the lines in blue) for ( i =0; i<numdisks; i++ ) { if ( disk_status[ i ] == "DISK_NP" && disk_device[ i ] == "" ) { continue; } if ( disk_name[ i ] == "" ) { disk_name[ i ] = "parity"; } else { disk_name[ i ] = "/dev/" disk_name[ i ]; } Thanks for finding this... Joe L.
September 2, 200817 yr I'll zip up a new version and update the attachment... I'm going to include an item or two suggested by RobJ as well. Stay tuned... Joe L. I just realized it has been a while since I made this statement, but i had not posted any updated version. The code used to create HTTP headers is completely different. I now am using "chunked" encoding. I've tested with IE6 and IE7, as well as Firefox. It might be broken on some other browsers, but who knows... I'm not doing anything too unusual in the HTML itself. IT should eliminate the issue with some browsers trying to load more content after the content has already been sent. The main page will auto-refresh every 60 seconds as long as the array is in a STARTED status. I've put a lot of comments into place in the code. I now have the framework I need for the "file browser" plug-in written, and should be able to post it soon. With any luck, it will not need any changes to the main unmenu.awk program. Most of the work now to do is in the plug-ins. In any case, let me know of any issues you may find. there are so many different ways to configure unRAID, I can't test them all here as I only have one server. If you have modified my original code, be sure you move your versions out of the way before unzipping mine. You can always "diff" your changes against my newest version to see what changed. As I stated in the prior post, you must now invoke this with an additional parameter on the command line. To invoke awk -W re-interval -f unmenu.awk the "-W re-interval" option allows me to use a more complicated regular expression that I'll need for the file-browser plug-in. I've put a number of configurable items at the top of the unmenu.awk script to keep them organized. Most people will not have to change a thing. Who will be the first to post their own plug-in??? (I've already had people suggest modifications to the existing ones, and I incorporated almost all those suggestions) newest version is attached here: http://lime-technology.com/forum/index.php?topic=2595.msg20982#new Joe L.
September 3, 200817 yr Excellent! I like the color coded syslog. It has highlighted this particular line in my log in red ... ACPI Warning (tbutils-0217): Incorrect checksum in table [OEMB] - 66, should be 59 [20070126] Is this something that I should be worried about?
Archived
This topic is now archived and is closed to further replies.