Joe L.

It aborted the "post-read" when a read of 2000 blocks of data returned after reading less than 2000 blocks... So, 60% of the remaining blocks were not post-read. We do not know about the pre-read...It could have aborted early too. (I don't currently track if it got to the end, but, clearly I need to, as the display is overwritten by the next phase) Odds are as good as any that the pre-read aborted too, especially with the short total elapsed time. It appears to me as if the drive has already reallocated 15 sectors, and has 1 more pending re-allocation. The "High-Fly Writes" are not too good either. Hard to say what to do... I'd run a few more cycles of preclear before I decide what to do... It sure might be a candidate for return, but they might not take it if their utility does not indicate it is over their failure "threshold" In the mean-time, I'll see about a modification of the script to force it to continue reading past early "read" aborts of the drive. In the unRAID server, the read failure would have resulting in the same data block being reconstructed from parity and then re-written to the drive. That would have forced the sector reallocation. Since this "read failure" happened on the disk post-read is troubling, as all reallocation should have taken place in the zeroing phase. (assuming the pre-read completed, that is) At this point I'm guessing it did not complete either. I'm learning how drives and SMART firmware react to this script as you are... glad to have helped, at least to identify a possible flaky drive. Joe L.

Well... "a couple of minutes" plus 16 hours, 47 minutes pre-clearing time. ;D Glad to hear it all went smoothly. I guess you were in too much of a hurry to try 20 cycles... It would have only taken 14 days.... Joe L.

Does it pay to have a FORMAT option in the pre-clear script? No, it would not pay at all to have a FORMAT option. If the drive was formatted it would have a file-system type of "83" set in the 16 bytes that define the first partition in the MBR. It would then not have a valid pre-clear signature, and the unRAID software would go about clearing it when you assign it to the array. After it cleared the drive, you would still have to format it. For a large drive, you are facing 4 or more hours of down-time as the drive is cleared. The only other way to add a "formatted" drive to the array would be to use the button labeled "Restore", the array would be on line quickly, but you would lose parity protection for many hours as it would then start a full parity calculation. On my array, it takes over 12 hours... I'd rather keep the array protected, so this is not the best way to add a new drive for me. Joe L.

It took 10 hours, 2 minutes to pre-read/clear/post-read my 750Gig SATA drive. This is interesting in that it is almost exactly the same time it took for an IDE based drive of the same size. Read speeds averaged in the 75-80MB/s range. Write speeds averaged in the mid 60MB/s through mid 70MB/s. It shows the PCI bus can keep up with a single SATA drive and we are still mostly limited by the drive itself. I stopped my unRAID array, assigned the newly cleared 750 Gig SATA drive on the devices page to an empty slot, then went back to the main page. I was presented with a display showing a "blue" indicator for the new drive. The text alongside the "Start" button indicated it would be cleared if it was not already pre-cleared when the array was started. I checked the "checkbox" under the "Start" button (to enable it) and then clicked on the "Start" button to start the array. The screen indicated that it was starting. After I refreshed it, it showed the new disk as "Unformatted" and the array was up and running. My array was off-line for perhaps a minute as I assigned the new drive. Only a minute of down-time is a HUGE accomplishment, as in the past it took about 4 hours of down-time to add a 750 Gig drive while it was being cleared. In addition, I had some confidence in the drive as any marginal sectors would have been identified. I clicked on the "Format" button, and in a minute or two more I saw the new disk was available for data. Joe L.

I'll bet it will take a while... Please let us know how long it does take.... (I hope you only did one cycle, at least at first) I figure your disk is twice the capacity as mine, but probably twice as fast, so 10 hours or so for one cycle is my guess.... Joe L. Running for 13:20 hours now. Post-read @ 30% approx. 3 minutes per percent so i guess still 3,5 hours to go. I'm running a preclear_disk cycle on a 750Gig SATA drive I have plugged into a new 2-port PCI-Bus SATA controller. It is a very inexpensive controller card, and only rated at SATA 1.0 speeds, but I figure I am very limited by the PCI bus, so it really does not matter. I'm 9 hours, 20 minutes into the process, and 85% complete in the post-read process. Joe L.

I noticed that the 4.4final and 4.5beta releases do not have a working "smartctl" command. This will not prevent the preclear script from running, but you will be unable to learn if the disk SMART attributes change during the preclear process. To fix the smartctl program all you need to do is install the missing library it needs. It can be downloaded from: http://slackware.cs.utah.edu/pub/slackware/slackware-12.0/slackware/a/cxxlibs-6.0.8-i486-4.tgz Then, using file-explorer on your windows PC you can open up \\tower\flash and create a packages folder. You can copy or move the downloaded file there. From windows it will be at \\tower\flash\packages\cxxlibs-6.0.8-i486-4.tgz If you log in via the system console, or via telnet, the flash drive is mounted at /boot. The new directory you created is therefore at /boot/packages Your file will then be at /boot/packages/cxxlibs-6.0.8-i486-4.tgz Once downloaded, and saved as cxxlibs-6.0.8-i486-4.tgz you can install it by changing to the directory where you downloaded it (I have all my downloaded packages in the /boot/packages directory, so after logging in on the system console or via telnet To change directory I type cd /boot/packages and install it by typing : installpkg cxxlibs-6.0.8-i486-4.tgz As an alternative, if I did not want to change directory to where I put the file, I could just give the full path to the downloaded file like this: installpkg /boot/packages/cxxlibs-6.0.8-i486-4.tgz Once it is installed, the smartctl program will work until you reboot, at which time you will need to re-install it once more. Joe L.

I'll bet it will take a while... Please let us know how long it does take.... (I hope you only did one cycle, at least at first) I figure your disk is twice the capacity as mine, but probably twice as fast, so 10 hours or so for one cycle is my guess.... Joe L.

I just loaded the new 4.4 version of unRAID. Apparently, Tom has not included the "ncurses" package. This will break the display of the preclear_disk.sh script. To fix it, you can either install the "ncurses" package, or, a lot easier, change these few lines in the script from: clearscreen=`tput clear` goto_top=`tput cup 0 1` screen_line_three=`tput cup 3 1` bold=`tput smso` norm=`tput rmso` ul=`tput smul` noul=`tput rmul` to: if [ -x /usr/bin/tput ] then clearscreen=`tput clear` goto_top=`tput cup 0 1` screen_line_three=`tput cup 3 1` bold=`tput smso` norm=`tput rmso` ul=`tput smul` noul=`tput rmul` else clearscreen=`echo -n -e "\033[H\033[2J"` goto_top=`echo -n -e "\033[1;2H"` screen_line_three=`echo -n -e "\033[4;2H"` bold=`echo -n -e "\033[7m"` norm=`echo -n -e "\033[27m"` ul=`echo -n -e "\033[4m"` noul=`echo -n -e "\033[24m"` fi I'll post a new version of the preclear script shortly with these changes. Edit: updated version now attached to first post in this thread. Joe L.

I re-seated my existing disk controller cables and interface card in an attempt to diagnose the DMA errors I've been experiencing since attempting to expand my array into the last 4 empty slots. As I stated earlier, I was getting a DMA error that just locked up the server. Since these 4 slots are not yet assigned to anything in my array, my data is safe... I do need to test these slots by reading and writing to disks in them, and this preclear_disk.sh script is perfect for this. I can keep a disk far more active than otherwise, and at no risk to the overall parity protection of my array. Last night I re-ran a pre-clear cycle of my tiny small 8Gig test drive It is at the end connector of the first cable of the disk controller. It ran successfully in about 25 minutes. I then tried a pre-clear of a much larger 750Gig drive. It is on the end of the second cable off of the same Promise IDE controller. As you might have guessed, the 750Gig drive took quite a bit longer to pre-read/clear/post-read than my 8Gig drive. It took just under 10 hours for 1 cycle. It also experienced some changes to the SMART data. The preclear_disk.sh script is designed to take a SMART status report when it starts, and another at its end, and to show you any differences between them if they exist. In my example screen-shot, the Raw_Read_Error_Rate and See_Error_Rate are un-changed, but their "raw value" changed. (last value on the line) These are not likely to be problems. The Airflow_Temperature_Cel changed... also not likely to be a problem. There was an increase in the Hardware_ECC_Recovered counter. I'll need to keep an eye on that. It indicated the hardware in the disk corrected an error it detected in reading the disk. The unRAID OS never even knew anything as the error-correction-code in the drive's firmware handled the error. Makes you kind of wonder if all this is also happening on disks in our Windows PCs, and we are not notified by it until it fails to boot... Here is a screen shot of how it looked when it was done: I'm going to run this 750Gig drive through a few more pre-read/clear/post-read cycles to see if it changes any more, or if I get any more DMA errors. First, I'm going to save a copy of my syslog, as the SMART reports are logged there. That way, if I do have another DMA error lockup, the SMART report in the saved syslog will be available next time for comparison. Note: the SMART difference output is in "diff" format. The lines with a leading "<" are from the before SMART report, the lines with a leading ">" are from the after SMART report. Lines that are unchanged are not shown at all. Joe L.

Perhaps submit it to the background via batch. The process can write it's pid in /var/run and log file in /var/log. The browser interface section can refresh on the log and if need be use the pid file to send a kill to the process and it's children. Sounds like it could work... I'll put it on the list if somebody does not get to it first... I need to get my own array hardware working properly first. I've determined that one drive tray slot locks up the server when it has a disk installed. Not sure if it is the cable, or the controller, or the drive tray connector in the case... Time will tell. (I can't experiment in the evenings as much, as we are using the server to watch movies...) Joe L.

That is a long-term goal, although I would put it on its own plug-in page. The biggest issue is the display of progress as it performs the clear. I am taking advantage of a feature of the "dd" command to get its status when writing to the drive. I would need to re-write that section. Then, I would need a way to start, and stop the process from the web-browser, and a way to get the periodic update of status as it progresses. I know I don't want to submit a task and wait 200+ hours for the browser to return... First I just want to make sure it is doing the correct "pre-cleared" signature, and not creating a black-hole somewhere in the universe. Before I tackle anything else, I need to get the next version of unmenu.awk out published. bjp999 recently added a bit of logic and figured out how to get it to POST to a plug-in as well as GET. I've added that to the next version, along with a few fixes. (non-geek translation... unmenu.awk can now handle more complicated data entry forms) Glad you like unmenu.awk. It certainly is turning out to be interesting. Joe L.

How can we utilize that many discs, especially on the PCI-e bus, not to mention cases that would fit that many discs? I think that is the point... Most people will reach hardware limits LONG before reaching the software limit on the number of disks. I can't even imagine how long an all IDE array might take to compute parity with 30+ disks. (possibly days) It might not be practical even with an all SATA array An equally attractive possibility, for those who can handle the hardware, is multiple logical "arrays" in a single case. Each array having its own parity disk. At least with that you can compute parity on a smaller set of drives, and a single disk failure affects only needs to query half of the drives to reconstruct the missing data. The true issue for many of us is if the array, when in a degraded state, is still able to serve the media we have stored on it at a rate fast enough to be usable. I'm thinking you would have to have a very good set of hardware to be able to serve a full bitrate ISO image from a failed disk on an array with 30+ disks. Tom mentioned "multiple logical arrays" as a possibility long ago... Joe L.

You can kill the smbd process that has the file locked Log in via telnet, type: smbstatus It will type a listing of the locked files, looking similar to the listing below: Samba version 3.0.28a PID Username Group Machine ------------------------------------------------------------------- 2549 joe joe dell630 (192.168.2.10) Service pid machine Connected at ------------------------------------------------------- flash 2549 dell630 Tue Oct 7 22:54:23 2008 Locked files: Pid Uid DenyMode Access R/W Oplock SharePath Name Time -------------------------------------------------------------------------------------------------- 2549 0 DENY_NONE 0x100001 RDONLY NONE /boot . Tue Oct 7 22:55:43 2008 The "pid" of the smbd process holding the locked file is given in the output. In my case, /boot is locked by process ID 2549. I can then type ps -ef | grep smbd to see what process is holding the lock. (It will be an smbd process) ps -ef | grep smbd root 1790 1 0 Oct01 ? 00:00:01 /usr/sbin/smbd -D root 1791 1790 0 Oct01 ? 00:00:00 /usr/sbin/smbd -D root 2549 1790 0 Oct07 ? 00:03:19 /usr/sbin/smbd -D root 20032 22786 0 23:26 pts/1 00:00:00 grep smbd Lastly, you can kill the process holding the lock. Type (using your process ID) kill 2549 Here is what happened when I did that on my server [pre] root@Tower:/boot# kill 2549 root@Tower:/boot# smbstatus Samba version 3.0.28a PID Username Group Machine ------------------------------------------------------------------- Service pid machine Connected at ------------------------------------------------------- No locked files [/pre] The lock is gone... Have fun... Joe L.

That behavior is not that unusual... Just choose the flash drive from the end of the hard drive list. Many BIOS will "help you" by resetting the boot device every time you add a new hard disk. In our cases, we really don't want its help. If you cannot boot the flash drive from its entry in the Hard-Disk list, then use the HP formatting tool and try once more. It resets the geometry on the flash drive to look like a removable hard disk. More detailed instructions are in the Wiki: http://lime-technology.com/wiki/index.php?title=USB_Flash_Drive_Preparation Joe L.

I realize that most people today are putting together SATA based arrays, but mine is an original IDE based MD1200 in a Coolermaster Stacker case. I can't even take credit for the wiring. Tom at Lime-Technology pimped out my server when I purchased it from him in October 2005. It is fully cabled with flat IDE cables to the 12 Cremax MB123AK drive trays. It has an Intel D865GLCLK motherboard, 2.26GHz Celeron, and 512MB of RAM. I was very impressed with the way the cables were neatly arranged. I've seen PC's with only a few drives not look anywhere near as neat. Joe L.

Sure you can... Have each hdparm command write to its own output file when run in the background and use the "wait" command to pause until all have completed. Then concatenate their outputs to return to the user. Pseudo code follows: for i in /dev/md* do n=`basename $i` echo "spinning up $i >/tmp/spinup${n}.out hdparm -S242 $i >>/tmp/spinup${n}.out & #sleep 1 done wait cat /tmp/spinup*.out >/tmp/spincontrol.out #... send /tmp/spincontrol.out contents back to the user here ... All the drives will spin up in turn. If you want a slight delay between each drive to minimize the power needed from the power supply, add uncomment the "sleep 1" before the "done" When the last of the hdparm commands being run in the background completes, the "wait" will allow the continuation of the script to return the final results. Joe L.

You are correct. At the time I wrote how to make symbolic links to the other disks the "users shares" did not exist. Joe L.

If you create an endless loop in your folders, your media scans will take a really long (infinite) time. Now, it will be scan that loop of folders really fast with today's faster disks and processors, but it might still seem to take forever to finish

I've not gone through a single disk format/startup, I have added manny to my initial 2 drive config. When you initially add a disk it is first cleared (and this can take quite a while for a large disk. During this time the web-interface is not responsive. (One of my complaints too, as the whole unraid array is not available during this time) After clearing a filesystem is created on the disk, then the disk is mounted) 10 minutes or more is not unusual for this clearing process. In fact, with your 500Gig drive it could take an hour or more. I just did an experiment... I tried to read all of one of my 500Gig drives and send its contents to /dev/null. it has been going for an hour now and only half of the disk has been read. The total time to read it will probably be two hours. I think writing would take at least that time. now, you have SATA drives and they may be faster than my IDE drives... but odds are VERY high that 10 minutes is way too short a time to clear your 500Gig drive. Odds are you did not wait for the process to finish before deciding to reboot the unRaid server. If you see disk-activity lights you are probably observing the clearing process. If the unraid server has really locked up you will see NO disk activity and no response from the web-interface and odds are you will not be able to log into the server using the console. If you are seeing your disk in the dropdown list for assignment then odds are your motherboard settings are OK. Setting the BIOS for IDE emulation may kill your PATA performance. Try again, but be patient if the disk activity lights are blinking... Joe L.

I just did an experiment on my unRaid array. I normally have a parity drive plus 7 data drives. For this experiment I stopped the array, then I copied /boot/config/super.dat to a backup file. (I made the backup copy of super.day so I would not have to wait through a full parity re-calculation when I restore all the drives later) Next, I unassigned the parity drive and all of my data drives except for a single one. I then re-started the array. It came up just fine with a single data drive and no parity drive. I was still able to play an ISO image from that one data drive through its "Movies" user-share over the LAN on my laptop. I stopped the array (It is still considered an "array" when there is only one drive? ) I copied my backup copy of super.dat to /boot/config/super.dat I used the web-interface to re-assign all my drives to their prior slots in the array. (I did not try shuffling data drives around, but I think I could have. Most important is to put the parity drive back on the correct physical drive it used to be on) I re-started the array. It came up perfectly fine, all my disks were back where they were.) So... It sure looks like you can have just one data drive, and no parity drive, and still use unRaid as a network storage device. Joe L.

Aaron, Sure, the symbolic link command "ln -s" can be used to make a link to a directory at the top of a directory tree (as I showed in prior examples) or to individual files or subdirectories. So if you had /mnt/disk1/"Dirty Dancing.iso" and /mnt/disk1/"King Kong.iso" and /mnt/disk3/"The Haunting.iso" you could do this" mkdir /mnt/disk1/allmovies ln -s /mnt/disk1/"Dirty Dancing.iso" /mnt/disk1/allmovies/"Dirty Dancing.iso" ln -s /mnt/disk1/""King Kong.iso" /mnt/disk1/allmovies/""King Kong.iso" ln -s /mnt/disk3/"The Haunting.iso" /mnt/disk1/allmovies/"The Haunting.iso" Now you could browse the "allmovies" folder and see the three movies. Think of these "links" as being nearly identical to the "shortcuts" on windows. In this post http://lime-technology.com/forum/index.php?topic=112.msg606#msg606 Tom describes a batch program he uses to make windows shortcuts to all the movies on his server. A benefit of this is that the windows box does not have to wait for the unRaid server disks to spin up to read the list of available movies. Of course, once he selects a movie the disk it is on must spin up so it can play, but the other disks can stay spun down. Lots of ways to solve the same problem. Tom's solution works if you are using a media center PC to play your movies, the solution I described in this post might work if you are using a networked DVD player (where you cannot create "shortcuts") Joe L.

after running the commands you listed you would see the following as viewed from your windows machine You would still see network shares from the unRaid tower for disk1, disk2, disk3, disk4, disk5, etc... If you map //tower/disk1 to a drive on your windows PC, let's say drive "S" and then used windows file explorer to look in drive "S" you would see three folders. The three folders would be movies, tvshows, and music. In the movies folder would be three sub-folders, disk1, disk2, and disk3 In the tvshows folder would be three sub-folders, disk4, disk5, and disk7 In the music folder would be one sub-folder disk7. In each of the sub-folders you would see the files and folders on that disk. Yes, in the movies\disk1 folder you would find a movies folder and in it a disk1 folder and in it a movies folder and in it a disk1 folder, etc. This might confuse some software attempting to scan all the sub-folders. In this case, if it had no limit on the sub-directory tree depth it might scan forever. If this is the case you might need to leave disk1 out of the movies/tvshows/music folders (assuming they were created on disk1) to prevent from creating a loop in the file structure that might catch an unaware program on your pc doing a scan in an endless loop. Also in drive "S" you would see all the files stored on disk1. So if you stored movies on it they would show at the root of the "S" drive as well as in the movies\disk1 folder. To keep from seeing the other files at the root of the drive "S" I would do this cd /mnt/disk1 mkdir sharedfolders mkdir sharedfolders/movies mkdir sharedfolders/tvshows mkdir sharedfolders/music Then make the symbolic links to the others disks as you described: ln -s /mnt/disk2 sharedfolders/movies ln -s /mnt/disk3 sharedfolders/movies ln -s /mnt/disk4 sharedfolders/movies ln -s /mnt/disk5 sharedfolders/tvshows ln -s /mnt/disk7 sharedfolders/tvshows ln -s /mnt/disk8 sharedfolders/music Now, from windows map a new shared drive "S" from \\tower\disk1\sharedfolders It will now have in it the folders for movies, tvshows, and music and each of those in turn folders for disk2, disk3, disk4, etc (note this time I left out disk1 to keep from creating an endless loop in the directory sub-tree) Lots of possibilities exist. Just remember a link to a folder is just that a subsequent pointer to the same data stored on a disk. The data blocks holding your movie will be freed when the last link to them is removed. If you modify the data in a file through a link it will be modified exactly as if you modified it via the name it was created. The data blocks are the same. Joe L.