Joe L.
-
Posts
19009 -
Joined
-
Last visited
-
Days Won
1
Content Type
Profiles
Forums
Downloads
Store
Gallery
Bug Reports
Documentation
Landing
Posts posted by Joe L.
-
-
unfortunately, google no longer operates code.google.com, therefore, the release list file cannot be accessed from them any longer.
A zip file of the entire unmenu source tree can be found at this link:
https://code.google.com/archive/p/unraid-unmenu/source/default/source
you can download the zip file, un-zip it, and have access to all the awk,shell files, and package definition config files within it.
Joe L.
-
Google no longer supports downloads of individual files from code.google.com.
unmenu cannot be installed using those instructions.
Joe L.
-
I only (very) recently put 6.2 beta on my server.The preclear_disk.sh script is failing on v6.2 betas because the sfdisk -R is no longer supported. Preclear_disk.sh is reporting the disk as busy and will not preclear it. It looks like 'blockdev --rereadpt' is the replacement according to the sfdisk man-pages here http://man7.org/linux/man-pages/man8/sfdisk.8.html.
" Since version 2.26 sfdisk no longer provides the -R or --re-read
option to force the kernel to reread the partition table. Use
blockdev --rereadpt instead."
EDIT: There is also an issue with reads failing. I changed the following:
read_entire_disk( ) { # Get the disk geometry (cylinders, heads, sectors) fgeometry=`fdisk -l $1 2>/dev/null` units=`echo "$fgeometry" | grep Units | awk '{ print $9 }'`to
read_entire_disk( ) { # Get the disk geometry (cylinders, heads, sectors) fgeometry=`fdisk -l $1 2>/dev/null` units=`echo "$fgeometry" | grep Units | awk '{ print $8 }'`and the reads will work.
Joe L. - Can we get an official fix and an update from you?
I did not have any issue pre-clearing the second parity disk I have just added to my array.
The fix will need to wait until I add/replace one of the existing disks with a larger one.
(Otherwise, I have no way to test the process. )
Whatever the fix might be, it must be backwards compatible with the older releases of unRAID.
In the interim, you can type this command to "patch" the preclear_disk.sh command
First change directory to the directory holding the preclear_disk.sh command. For most, it will be
cd /boot
then type (or copy from here and paste) the following:
sed -i -e "s/print \$9 /print \$8 /" -e "s/sfdisk -R /blockdev --rereadpt /" preclear_disk.sh
Your preclear disk script will be edited and should work with the two changes you mentioned. (actually, each occurs in two places, so there are a total of 4 lines changed)
Joe L.
-
1
-
1
-
-
You are both missing an important part of the equation.Many of us feel that running two or three preclear cycles will get the drive past the 'infant mortality' portion of the bathtub curve (google for further discussion). Uncovering an early HD failure before putting that drive into an array is much less stressful than finding a compromised array in the first week after introducing a new drive into the mix.
PS--- I could tell a story about how the concept of infant mortality came into general knowledge to the military during WWII but that would be completely out of topic...
I agree that there is value in stress testing the drive and checking to make sure nothing is failing after the first few writes.
That said, maybe this signals that a new plugin needs to be made that removes the clearing portion of the plugin and instead focuses entirely on stress testing. Leave the clearing entirely to the OS since that's not an issue anymore.
This should allow more cycles of stress testing without having to have that long post read cycle (that verifies the dive is zeroed) meaning you can do more cycles faster... I think.
I think you are missing a part of the equation. It is not only the stress introduced by the testing, the elapsed time is an integral part of the entire process.
Un-readable sectors are ONLY marked as un-readable when they are read. Therefore, unRAID's writing of zeros to the disk does absolutely nothing to ensure all the sectors on the disk can be read. (Brand new disks have no sectors marked as un-readable)
Sectors marked as un-readable are ONLY re-allocated when they are subsequently written to. It is the reason the preclear process I wrote first reads the entire disk and then writes zeros to it. (It allows it to identify un-readable sectors, and fix them where possible)
The entire reason for the post-read phase is because quite a number of disks failed when subsequently read after being written.
If you rely on unRAID to write zeros to the disk and then put it into service, the first time you'll learn of an un-readable sector error is when you go to read the disk after you've put your data on it. (or during a subsequent parity check)
The new feature in this release of unRAID will help some to avoid a lengthy un-anticipated outage of their server if they had not pre-cleared a disk, and for that it is a great improvement. This improvement in unRAID 6.2 does not however test the disks's reliability in any way, nor identify un-readable sectors (since it only writes them, and does not read them at all)
Additional discussion about the difference between the unRAID 6.2 initial zeroing of drives replacing the preclear process should continue in another thread... and not clutter up this thread in the announcement forum.
Joe L.
-
I personally would run it again, as I understand it, because unraid runs from RAM, logs don't survive a reboot (or loss of power) so there is no way to tell what the previous preclear indicated, if you really don't want to, you could run a SMART test and if everything in there is OK you will probably be fine, what you wont be able to tell is if there were any changes before the start of the preclear and after which can indicate possible problems.
*edit* looking at that smart test there is nothing that stands out to me, but I'm no expert, Current_Pending_Sector and Reallocated_Sector_Ct are 0 though so will probably be ok.
Actually, the preclear script logs its reports on the flash drive in
/boot/preclear_reports
You might look there. If the report is not there, then it finished the clearing step, but not the post-read phase to see if it was successfully zeroed.
Joe L.
-
no, not moved on... Just have precious free time to be as heavily involved as I was a few years ago. (when I was not working.)Yes, I think Joe L. has moved on. What changes have you made to the cache_dirs script and are there any incompatibilities to the original script?
EDIT: So far all I see is that you changed the 'B' to 'b' for disks busy.
Ha! I think you need glasses if that's all the change you see
But I totally get the need for a better description than the change-log.I don't have the time right now to go into details and I don't remember everything I did but this is what I wrote earlier:
I have added adaptive depth level, to prevent cache_dirs from thrashing disks when they are otherwise occupied and cache is evicted. I found the cache was often evicted with the number of files I had when system become occupied with other things.
I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consecutive scans, the depth is decreased, and future rescan is scheduled at higher depth.
If the file '/var/log/cache_dirs_lost_cache.log' exists then it will write a log that is easily imported into spreadsheet (excel) so its easier to check whether it trashes disks with current settings. I also added the kill I mentioned and some other quite minor bug-fixes.
If you need more let me know, and I might supply more detail over christmas. If you think it looks good and useful I might do a clean up run on the script. I havn't felt like spending more time on the script if nobody but me used it.
Best Alex
My servers are both built with out-dated hardware. I cannot contribute in the same way I did in the past. (One is an original server sold by Limetech, with IDE based drives, the second newer, but incapable to handle virtualization)
I do follow the threads... and respond occasionally...
Joe L.
-
Or, the SATA cables are picking up noise from adjacent cables. (adjacent power OR SATA cables)
This often occurs when a user attempts to make their server look neat by bundling all the SATA cables together.
When doing so, it is putting into place a situation where induced noise is very likely.
Therefore, cut the tie-wraps bundling cables together. Yes, it looks less neat, but... you'll see far fewer noise induced CRC errors.
Joe L.
-
If the errors are in different places each time, it is more likely to be a memory problem, disk controller problem, or a power supply problem.Thanks for the info, I had never seen this.
I am trying it now, although I think it is not the perfect choice for my case, since errors, appear in different places of the HDDs.
Very first thing to check is to run a memory test, preferably overnight (or at least several full passes). As often as not, a bad memory strip is the issue.
Joe L.
-
You are welcome.Thanks Joe => it's nice the fix is so simple ... despite all the newer "stuff" I still don't want to be without UnMenu
If you think about it, much of the newer (now native) features were originally implemented in unMENU, and unMENU was originally created to explore alternative and improved user-interfaces to unRAID. It has done exactly what it was created for. I am happy that it still offers substantial value after all these years.
No, unMENU does not offer "docker" or "virtual server" features... but it certainly holds its own with almost everything else.
(unRAID itself does not yet allow you to choose a vertical vs. horizontal menu bar... they still have some catching up to do)
-
Even though the pre-clear had failed (detected it had not filled the disk as expected), it could have written what looks to the BIOS as a valid master-boot-record to the hard-disk being cleared.Now the strange problem:
My initial or 1st Drive seemed to exhibit a similar problem so I was not particularly worried, albeit frustrated at the loss of time but... So I decided I would just re-boot (when the drive fails preclear it ceases to be seen) and low and behold the Windows machine would not boot off any USB stick regardless of location and/or Boot Selection attempt??? Very strange behavior, I finally re-attached my Windows HD and booted successfully, then totally cleared the 2nd 5TB Drive thinking that something was amiss. Still no change despite 3 different valid USB bootable sticks.
Finally this evening, I went into the Bios again and "Restored Default Settings" and low & behold UnRaid now Boots and I've re-started the PreClear Plug-In.
So the "burning" question is How Did PreClear Failure somehow write "code" to the BIOS! Is there anything that this will do to a system in the future?
First, no software (including Preclear) writes to the BIOS.
This is actually a common problem with many motherboards. Whenever you change the installed drives list for the system, the BIOS may decide to "help" you, and reorder the boot order so that the most likely hard drive will be booted, which is usually NOT the USB drive you had configured! You did the right thing by going into the BIOS and correcting the boot order, making sure the right drive is booted, not what the BIOS *thinks* is the right drive.
Thanks Rob - I agree in a sense, but I actually selected a "seen" USB Bootable Hard Drive and it/they still failed. Maybe the Bios still changed it to the Cleared (not PreCleared) hard drive as it showed "no Bootable disc found".
Still an interesting and "freaky" thing to witness. It worked fine until the PreClear "failed" then would not boot until it was reset.
Dave
In other words, as RobJ said, your bios was trying to "help" you by choosing one of your hard-disks to boot from that it thought had a valid master-boot-record, and since none contain actual code to boot from, nothing would boot until you set the bios back to boot from the correct usb-flash-drive.
-
even easier is to type:
if it moved, that would do it. As I said earlier, I'm not running the newer release, so have had no opportunity to look around.
We have the basic chicken vs. egg issue here.
Which is why I will NOT be upgrading to 6.1 unless this is made compatible or the features from this that I use are put into the unRAID GUI.Apparently unMENU is not compatible with V6.1-rc3. Any fixes in the works? I still find unMENU useful when the unRaid gui is not accessible. Thanks.
I do not typically upgrade my server to the latest version until it is out for a few days. Therefore, I have no way to test or make changes to unMENU.
From what I've read, the /root/mdcmd shell command no longer exists. unMENU uses it to get the status of the array.
/root/mdcmd was just a interface to /proc/mdcmd.
You could try putting it back into place and see if everything starts working once more
You can re-create it by typing:
echo 'if [ $1 == "status" ]; then' >> /root/mdcmd
echo ' cat /proc/mdcmd' >> /root/mdcmd
echo ' else' >> /root/mdcmd
echo ' echo $* >/proc/mdcmd' >> /root/mdcmd
echo 'fi' >> /root/mdcmd
chmod 755 /root/mdcmd
Ha. I was just looking at this too and see you responded.
I was thinking to create a file called /root/mdcmd that looks like:
#!/bin/bash /usr/local/sbin/mdcmd $*
That work also?
If so, there should be a command in the go file that if /root/mdcmd does not exist, to create that file. This would allow for the mdcmd command to be updated in a future release (not sure if ever would).
Anyway, just my $0.02.
ln -s /usr/local/sbin/mdcmd /root
I just upgraded to the latest release of unRAID 6.1-rc5, and it seems to be all that is needed to get unMENU working again.
You can add a line to the config/go file to perform that link command each time you reboot.
-
if it moved, that would do it. As I said earlier, I'm not running the newer release, so have had no opportunity to look around.
We have the basic chicken vs. egg issue here.
Which is why I will NOT be upgrading to 6.1 unless this is made compatible or the features from this that I use are put into the unRAID GUI.Apparently unMENU is not compatible with V6.1-rc3. Any fixes in the works? I still find unMENU useful when the unRaid gui is not accessible. Thanks.
I do not typically upgrade my server to the latest version until it is out for a few days. Therefore, I have no way to test or make changes to unMENU.
From what I've read, the /root/mdcmd shell command no longer exists. unMENU uses it to get the status of the array.
/root/mdcmd was just a interface to /proc/mdcmd.
You could try putting it back into place and see if everything starts working once more
You can re-create it by typing:
echo 'if [ $1 == "status" ]; then' >> /root/mdcmd
echo ' cat /proc/mdcmd' >> /root/mdcmd
echo ' else' >> /root/mdcmd
echo ' echo $* >/proc/mdcmd' >> /root/mdcmd
echo 'fi' >> /root/mdcmd
chmod 755 /root/mdcmd
Ha. I was just looking at this too and see you responded.
I was thinking to create a file called /root/mdcmd that looks like:
#!/bin/bash /usr/local/sbin/mdcmd $*
That work also?
If so, there should be a command in the go file that if /root/mdcmd does not exist, to create that file. This would allow for the mdcmd command to be updated in a future release (not sure if ever would).
Anyway, just my $0.02.
-
We have the basic chicken vs. egg issue here.
Which is why I will NOT be upgrading to 6.1 unless this is made compatible or the features from this that I use are put into the unRAID GUI.Apparently unMENU is not compatible with V6.1-rc3. Any fixes in the works? I still find unMENU useful when the unRaid gui is not accessible. Thanks.
I do not typically upgrade my server to the latest version until it is out for a few days. Therefore, I have no way to test or make changes to unMENU.
From what I've read, the /root/mdcmd shell command no longer exists. unMENU uses it to get the status of the array.
/root/mdcmd was just a interface to /proc/mdcmd.
You could try putting it back into place and see if everything starts working once more
You can re-create it by typing:
echo 'if [ $1 == "status" ]; then' > /root/mdcmd
echo ' cat /proc/mdcmd' >> /root/mdcmd
echo ' else' >> /root/mdcmd
echo ' echo $* >/proc/mdcmd' >> /root/mdcmd
echo 'fi' >> /root/mdcmd
chmod 755 /root/mdcmd
-
Since lime-tech is in release-candidate-2 of 6.1, I'd not expect new features, but instead just tiny bug-fixes so they can get to 6.1 final.Is NUT UPS support still planned for 6.1? Really looking forward to this.
(I can't speak for lime-tech, as I'm a customer, just like you, so it is always possible they would throw in something at the last moment... but I would look to a community plugin rather than something in 6.1 natively)
Joe L.
-
you can see all the available options by typing
preclear_disk.sh -?
In any case, you can specify multiple cycles on the command line easily, just use "-c N"
where N = the number of cycles desired.
example:
preclear_disk.sh -c 3 /dev/sdX
-
Yes,
I think ANY problem with the preclear is an issue./boot/preclear_bjp.sh -f -A -D /dev/sdz
Thanks,
Dave
The results of the preclear run should be stored on the flash drive in the preclear_reports folder.
Hi Guys,
From ~ 10 days ago, I wrote about my problem PreClearing a 5TB drive with a V6 key using a separate computer. The first drive got all the way through 2x but indicated that it could not Preclear the MBR, hence itimpi's comment.
I've checked, no log reports at all. After rebooting and adding the PreClear Plugin & Script, I started over several times but at some point 20+ hours into Preclear the drive/system disconnect (lose communication with the drive) rendering Preclear useless.
This has now happened to a new 2nd 5TB Drive (fresh out of the box). Anyone have a thought?
Dave
your drive is losing contact with the disk controller.
You are lucky you are discovering the issue before you start loading your data to the drive.
In many cases in the past, the issue was poor or intermittent SATA cabling to the drive, or an intermittent power splitter or power connection, or intermittent drive tray, or back-plane, or a power supply inadequate for the number of drives connected. Occasionally, it was traced to a flaky SATA controller port.
What exact power supply are you using? How many disks are being powered from it?
Do not get confused by the preclear report stating it could not clear the MBR. It must write a protective MBR to the drive, for older utilities that expect it to be there, even though the actual partition is located further up on the disk. Apparently, at the point where the MBR is being written the drive is already not communicating with the disk controller. (So no writes to the drive would work, regardless of what they were for)
Joe L.
-
one more line to change..
Change:
PACKAGE_INSTALLED /var/log/ds_store_cleanup
to
PACKAGE_INSTALLED /tmp/ds_store_cleanup
It is looking in /var/log for the file, but creating it in /tmp.
Having them match will make a huge difference.
Joe L.
-
well, if it does not install, it will not find the file it thinks should be there, thus the error message.
Many packages will not work with 6.0, but I would have guessed the ds_cleanup would have as it is a simple shell script.
I see the issue, the package is simply looking for a different version string than the one written.
PACKAGE_INSTALLATION echo "Version: 1.6" >/tmp/ds_store_cleanup
PACKAGE_VERSION_TEST grep "Version:" /tmp/ds_store_cleanup | awk '{print $2}'
PACKAGE_VERSION_STRING 1.5
Writing 1.6, looking for 1.5
Odds are it installed just fine, type
crontab -l
and you should see the expected entry.
You can edit the package file and change the 1.5 to 1.6 and also add a line
#AUTO_UPDATE=no
Joe L.
-
Sorry... My typo
ping -c 5 google.com
(I left out the number of pings to try in my prior example)
-
Not sure who wrote those instructions, but it sure was not me.I've been trying to install unMenu on v6.0.1 without any luck. I installed following these instructions, but can't connect via web browser. I set "My Host" to the IP address just like my previous version, but I've also tried tower:8080, tower.local:8080, brunnhilde:8080 and brunnhilde.local:8080. Honestly, I'm not sure if unMenu is running or not. Maybe this is not the proper way to install it in v6. I couldn't find any more up to date instructions though. Or maybe it's possible now to install packages without installing unMenu?
The instructions I wrote are here:
http://lime-technology.com/forum/index.php?topic=5568.0
and here:
https://code.google.com/p/unraid-unmenu/
Either should work. The ones you linked to are incomplete and do not describe creating the unmenu directory.
(it does refer you to a pre-boot section of the wiki. Did you perform those steps? Do you have the networking configured? What happens when you type:
ping -c google.com
what do you see when you type:
ls -l /boot/unmenu
-
That board is not in the server I just upgraded, so therefore, I cannot answer if it works, or not.I don't know for SURE, but if I recall correctly Joel's configuration includes a 1430SA ... and I know he just updated to v6 with no problem. If he noticed this, perhaps he'll confirm that he's using a 1430SA. [i'll send him a PM and a link to this comment.]
-
That was it. I'll update the wiki to make it more clear for the next "expert"I'll try removing those two files and let you know what happens when I reboot.
Joe L.
Joe L.
-
Yes, I know, and I followed its instructions specifically. (and followed the section for ADVANCED users who did not wish to re-format the flash drive.)There is this wiki: Upgrading to UnRAID v6
Tom...
root@Tower:~# ls -l /boot/extra
total 0
root@Tower:~# ls -l /boot/plugins
total 352
-rwxrwxrwx 1 root root 1510 Aug 10 2013 webGui-latest.plg*
-rwxrwxrwx 1 root root 333600 Aug 10 2013 webGui-latest.txz*
root@Tower:~# grep -v "#" /boot/config/go
/usr/local/sbin/emhttp &
root@Tower:~#
I'm guessing the files in the plugins directory should not be there. (I've never used unRAID plugins, so I'm guessing these were for the stock unRAID interface)
I did not install them specifically, I did copy the "plugins folder from the distribution to the flash drive, but that would have left previous contents.
I'll try removing those two files and let you know what happens when I reboot.
Joe L.
-
I did not. But clearing it made no difference. (I just tried)
Did you clear your browser cache?Upgrading from 5.0.6 to 6.0-rc4.
Which method did you use to upgrade? Did you format your flash drive, or manually move things around?
I did the whole format process moving from 5.0.6 a few weeks ago, and it went very smoothly.
I did not do a complete reformat. I did rename everything and disable everything, and never used dynamix ever previously.
But I totally get the need for a better description than the change-log.
Re: preclear_disk.sh - a new utility to burn-in and pre-clear disks for quick add
in User Customizations
Posted
Yes, and I fixed my original post. (and I've used "sed" for over 40 years)