May 4, 200917 yr I must be doing something wrong as everyone else is reporting this utility is working great, but I can't seem to get it to work. I cannot use ls -R due to the sheer number of folders and files that are on the system, so I was ecstatic to see a find with a depth limiting feature version, but it doesn't seem to read any of the folders. When I run the command, none of my spun-down disks spin back up to try and read the folders and I am sure there are dozens of folders that are not cached. Below is a brief output of what I am seeing: root@Tower:/boot/utils# ./cache_dirs -F -d 4 Executed find -maxdepth 4 in 0.013799 seconds, avg=0.013799 seconds, now sleeping 5 seconds Executed find -maxdepth 4 in 0.013808 seconds, avg=0.013804 seconds, now sleeping 4 seconds Executed find -maxdepth 4 in 0.013591 seconds, avg=0.013733 seconds, now sleeping 5 seconds root@Tower:/boot/utils# ./cache_dirs -F -d 7 Executed find -maxdepth 7 in 0.014136 seconds, avg=0.014136 seconds, now sleeping 5 seconds Executed find -maxdepth 7 in 0.013572 seconds, avg=0.013854 seconds, now sleeping 6 seconds Executed find -maxdepth 7 in 0.013779 seconds, avg=0.013829 seconds, now sleeping 7 seconds Executed find -maxdepth 7 in 0.013557 seconds, avg=0.013761 seconds, now sleeping 8 seconds I have tried several different depths, but nothing seems to be adding to the cached dirs list. Am I doing something wrong? I want to cache the first few levels of all of my user shares. What makes you think the cached directory block is not already in memory, and therefore there is no need to spin up anything to read them. Your "diagnostic output seems to show that the "find -maxdepth 7" is occurring in an average of .013 seconds (in other words, all occurring in memory... the directory blocks are already in cache) Why do you think there are dozens of folders not scanned? Use the "Clear Statiistics button on the web-interface. Then to force the caches to be emptied, in a second telnet session, while you are running the cache_dirs in the foreground in the first (so you can see the effect) type the following: sync; echo 3 >/proc/sys/vm/drop_caches When you issue that command, anything not yet written to the disk will be written, and then all the cache'd memory freed. The next loop of the cache_dirs program will spin up the drives to read the directory data. Now, refresh the unRAID web-page to see how many "Reads" occurred. On my server, it needed to read 4 blocks from each of my disks to get all the directory information. Initially, all my disks had to spin up, since I had watched a movie or two while I did not have any "ls" or "find" running. My very first time through my cache_dirs loop took 83 seconds, the next took .29 seconds... then subsequently it averaged about .2 seconds. Now, I dump my cache with the command I gave above... and it still averaged about .2 seconds... That indicates to me the spinning disk takes hardly any time to scan at all and the loop overhead is greater than the time needed to read the disks. I can see the the number of blocks read (and interestingly, written) increment when I issue the "sync" and drop cache commands. In most cases, each disk on my array needs 4 reads to get all its directory data into cache... Joe L.
May 4, 200917 yr Interesting, this may just be my lack of experience in Linux talking then. I did as you said and saw all drives spin up and read between 5 and 10 blocks each. At the same time below is the output of the cache_dirs command: root@Tower:/boot/utils# ./cache_dirs -F -d 3 Executed find -maxdepth 3 in 0.013787 seconds, avg=0.013844 seconds, now sleeping 10 seconds Executed find -maxdepth 3 in 0.013576 seconds, avg=0.013814 seconds, now sleeping 10 seconds Executed find -maxdepth 3 in 0.013873 seconds, avg=0.013820 seconds, now sleeping 9 seconds Executed find -maxdepth 3 in 0.292074 seconds, avg=0.039116 seconds, now sleeping 8 seconds Executed find -maxdepth 3 in 0.013806 seconds, avg=0.037007 seconds, now sleeping 9 seconds Executed find -maxdepth 3 in 0.013685 seconds, avg=0.035213 seconds, now sleeping 10 seconds Executed find -maxdepth 3 in 0.013752 seconds, avg=0.033680 seconds, now sleeping 10 seconds I assumed it would take longer than a fraction of a second to gather the directory information, but I could be wrong. Is this also caching the file information within the folder, or just the folders themselves? After I ran it with a depth of 3, I aborted (ctrl-c) and ran with a depth of 4, but the read count on the drives did not increase. Why would this be the case? There are definitely folders with more than 4 depth in them.
May 5, 200917 yr Interesting, this may just be my lack of experience in Linux talking then. I did as you said and saw all drives spin up and read between 5 and 10 blocks each. At the same time below is the output of the cache_dirs command: root@Tower:/boot/utils# ./cache_dirs -F -d 3 Executed find -maxdepth 3 in 0.013787 seconds, avg=0.013844 seconds, now sleeping 10 seconds Executed find -maxdepth 3 in 0.013576 seconds, avg=0.013814 seconds, now sleeping 10 seconds Executed find -maxdepth 3 in 0.013873 seconds, avg=0.013820 seconds, now sleeping 9 seconds Executed find -maxdepth 3 in 0.292074 seconds, avg=0.039116 seconds, now sleeping 8 seconds Executed find -maxdepth 3 in 0.013806 seconds, avg=0.037007 seconds, now sleeping 9 seconds Executed find -maxdepth 3 in 0.013685 seconds, avg=0.035213 seconds, now sleeping 10 seconds Executed find -maxdepth 3 in 0.013752 seconds, avg=0.033680 seconds, now sleeping 10 seconds I assumed it would take longer than a fraction of a second to gather the directory information, but I could be wrong. Is this also caching the file information within the folder, or just the folders themselves? After I ran it with a depth of 3, I aborted (ctrl-c) and ran with a depth of 4, but the read count on the drives did not increase. Why would this be the case? There are definitely folders with more than 4 depth in them. Short answer is, a single "read" of the disk contains more than a single directory... It represents a whole part of the directory structure stored on the disk. There will not be a 1-to-1 correspondence. As far as the speeds... reading from ram is nearly instant. (as far as the times we are able to track) Reading from a spinning disk is only slight slower (we have to wait for a platter to spin around, but don't forget, the disk has its own 16 or 32 meg buffer in ram. It might not need to wait at all for the disk to spin, it might have the data requested in its own buffer. Although it does not work this way, If you had 10,000 files, and 16,000,000 bytes of ram in the disk as buffer, that is still 1,600 bytes per file in the buffer on the disk. Most file names are FAR shorter than that. Now you see why it may not need to spin up a disk. Joe L.
May 5, 200917 yr Understood. Thank you for the explanation and developing the tool Joe. I will test it over the next few days and let you know how it goes. It is great to have all your expertise in the community!
May 5, 200917 yr Thank you Joe L. for your another excellent tool. Just implemented and working great. Nevertheless, let me ask two questions: - Caching the cache drive was discussed earlier, but if I am correct it's not implemented yet. Are you planning to include this? - Why should one deciding not to cache a share? Like you, the Data share. Do we need to have any special care with it? Thank you again. It makes my whole network a lot more responsive! I wonder how could I live without this until now...
May 6, 200917 yr You might decide not to cache a dir if it has zillions of files you only access very rarely. There no reason to slow down the cahing process with stuff you dont need. Saying that I just cache everything i find it easier to "set and forget". I have been running this script for a while now and it works exactly as described. Have not seen any funnys at all.
May 6, 200917 yr You might decide not to cache a dir if it has zillions of files you only access very rarely. There no reason to slow down the cahing process with stuff you dont need. Saying that I just cache everything i find it easier to "set and forget". Exactly... I have one directory hierarchy in my "data" share that represents an entire windows PC... (It was done from a PC long ago... and has every file and folder present on that PC at that time... there was no other backup software other than dos xcopy. It was originally done to a USB drive. Other than the space it took, it did the job. When I eventually re-used that disk in my unRAID server, I transferred the files. # find /mnt/user/data | wc -l 1831113 I do not want to cache a bit over 1.8 million files, so I specifically need to exclude my "data" directory. I have been running this script for a while now and it works exactly as described. Have not seen any funnys at all. Nice to know. I'm still working on its successor. I'll post it once it is done. Joe L.
May 6, 200917 yr I'm still working on its successor. I'll post it once it is done. Joe L. Great news, although the "old" one is also working for me fine. But may I ask you again if you planned to include cache driver caching in the new version?
May 6, 200917 yr I do not want to cache a bit over 1.8 million files, so I specifically need to exclude my "data" directory. What happens if someone (like me) did try to cache millions of files? Will smoke pour out of my file server? Or will it just take a long time for the first scan and then I'll have fantastic performance afterwards?
May 6, 200917 yr I do not want to cache a bit over 1.8 million files, so I specifically need to exclude my "data" directory. What happens if someone (like me) did try to cache millions of files? Will smoke pour out of my file server? Or will it just take a long time for the first scan and then I'll have fantastic performance afterwards? You would probably end up reformatting the hard disks of every computer within a 100 mile radius... shortly followed by the creation of a tiny black hole...erasing all evidence of your server's parity drive. You will know it is imminent when you see the lights start to dim as all the nearby electrons slow due to the gravity gradient.... another clue will be the hard-drive lights blinking. Seriously, nothing will happen. The "find" will take a long time to run. If it has enough memory to complete, it will. If it allocates memory as it goes and does not have enough it can allocate, it might fail... but that would be an awful lot of files... especially since I just did it on my system with only 512 Meg of ram. The "fantastic" performance you described would only last until you played a movie... at some point, its blocks of data in the cache would all be newer than the results of the "find" command and the oldest unreferenced blocks will be re-used for the movie and your "fantastic" performance wouldn't be so good. Joe L.
May 6, 200917 yr This is not a performance tweak, but an operational improvement. The primary idea behind this script is to avoid having to spin up drives to access their directory entries. Trying to cache millions of files is not only pointless, but ineffective and probably counter-productive. There is only so much space allocated for the directory entries, so I assume it is possible that caching too many files could very well end up dumping the first files cached in a scan, to make room for the later ones. That would result in this script forcing drives to stay spun up! That is the opposite of what you are trying to achieve. You only want folders cached that are either being periodically scanned, such as media storage folders, or are part of a scanned group, such as within a User Share.
May 6, 200917 yr We have to be careful here not to make users scared of Joes excellent script. I have WADS of files including a silly amounts of tiny log files and the caching works as predicted. Drives spin down and stay down. Browsing is fast and doesn't bring up drives. I have 4GB of RAM though so I cant comment on 512MB users other than to say we know Joe has similar results to me with 512MB. Its not a perfect solution to the problem but i honestly believe it should be a part of unRAID proper now it improves the end user experience THAT much.
May 6, 200917 yr This is not a performance tweak, but an operational improvement. The primary idea behind this script is to avoid having to spin up drives to access their directory entries. Trying to cache millions of files is not only pointless, but ineffective and probably counter-productive. There is only so much space allocated for the directory entries, so I assume it is possible that caching too many files could very well end up dumping the first files cached in a scan, to make room for the later ones. That would result in this script forcing drives to stay spun up! That is the opposite of what you are trying to achieve. You only want folders cached that are either being periodically scanned, such as media storage folders, or are part of a scanned group, such as within a User Share. I understand. I'm trying to address the usuability issue of every time I enter a directory, I have to wait not only for the drive to spin up, but also (I think) for the list of files to be read from the hard drive, which can take a while for a directory with 100,000 or so files. I was hoping that with 4GB of RAM, I could keep an index in cache of the entire array and not only eliminate the spin-up delay but also eliminate having to read the directory from the HDD at all. I didn't understand that I couldn't use as much RAM as needed for that directory cache though
May 6, 200917 yr We have to be careful here not to make users scared of Joes excellent script. I have WADS of files including a silly amounts of tiny log files and the caching works as predicted. Drives spin down and stay down. Browsing is fast and doesn't bring up drives. I have 4GB of RAM though so I cant comment on 512MB users other than to say we know Joe has similar results to me with 512MB. Its not a perfect solution to the problem but i honestly believe it should be a part of unRAID proper now it improves the end user experience THAT much. The only possible downside I can think of, for somebody running it on older versions of unRAID, is that it might be attempting to scan the directory folders at exactly the same time the array is being stopped. It could possibly prevent the un-mount of a disk from occurring. From what I understand, the new version of unRAID is supposed to have improved the power-down sequence to kill any processes referencing the disks. I'll try an experiment to see if it works as described. I started a "find /mnt/user/data" and then pressed "Stop" on the management console. Oops... the extra logic might be there for a true powerdown, but it sure is not there for a "Stop" as proven by the following screen capture. Hey Tom, how about a bit of extra logic here... to not say "unformatted" but instead, un-mounted. Or, better yet, to re-mount the drives that could be un-mounted if unable to un-mount them all. (And give an error message stating they could not all be un-mounted, since one or more was busy) Then, the user could stop whatever they were doing, or, wait it out... (as I did) and press Stop once more. In any case, if the array "Stop" was un-successful, leaving you with a series of drives showing as "Unformatted" (but actually, just un-mounted) you would just press "Stop" once more and stop the array. And that's what I did do to recover... I pressed "Stop" a second time after the "find" had completed, and the array was stopped cleanly. Joe L.
May 6, 200917 yr Hey Tom, how about a bit of extra logic here... to not say "unformatted" but instead, un-mounted. Yes I will get to that.
May 6, 200917 yr This is not a performance tweak, but an operational improvement. The primary idea behind this script is to avoid having to spin up drives to access their directory entries. Trying to cache millions of files is not only pointless, but ineffective and probably counter-productive. There is only so much space allocated for the directory entries, so I assume it is possible that caching too many files could very well end up dumping the first files cached in a scan, to make room for the later ones. That would result in this script forcing drives to stay spun up! That is the opposite of what you are trying to achieve. You only want folders cached that are either being periodically scanned, such as media storage folders, or are part of a scanned group, such as within a User Share. I understand. I'm trying to address the usuability issue of every time I enter a directory, I have to wait not only for the drive to spin up, but also (I think) for the list of files to be read from the hard drive, which can take a while for a directory with 100,000 or so files. I was hoping that with 4GB of RAM, I could keep an index in cache of the entire array and not only eliminate the spin-up delay but also eliminate having to read the directory from the HDD at all. I didn't understand that I couldn't use as much RAM as needed for that directory cache though No matter what you do... it will take some time to send the file listing to your PC if you have 100,000 files in a single folder. The memory for disk cache is used both for directory entries, and also for recently accessed data... think of it being shared, although from what I've read, a portion of it is set aside for directory entries. There is no way in the Linux kernel to "pin" the directory entries so they are never freed. (we can tell it to favor freeing the data blocks, but it will still free the directory cache if they are least recently used. Unfortunately, playing an ISO image of multi-Gig size will probably use up all your cache memory as its blocks of data are cached, but never referenced any more (unless you rewind). All I can say is give it a try... with as much ram as your motherboard supports. It can't hurt, and you'll see if it helps in your situation. Joe L. Oh yes... I was kidding earlier about slower electrons... (its never been proven they slow down in the gravity gradient of a black-hole...)
May 6, 200917 yr Yep, that screenshot is exactly what happened to me... I had a lot of trouble getting the script running, cut and paste through windoze notepad was mangling something. I downloaded notepad++ and tried again, this time instead of syntax errors, the script seemed to "lockup". I thought that it might just be doing its thing and figured that it might even take hours to complete, but after the problems I had with notepad, I decided to reboot, make sure everything was squared away and then try again with a clean slate. I pressed the stop buttom, saw "unformatted" on all of my disks and had to go change my pants. All turned out OK with no damage to the array, but I decided to post here for advice before trying again.
May 6, 200917 yr Hey Tom, how about a bit of extra logic here... to not say "unformatted" but instead, un-mounted. Yes I will get to that. Wow... There is a God... We know at least one person accidentally pressed the "Format" button at that point. I don't remember if they lost some data. We also know there apparently are a lot of us who needed a change of underwear when we first encountered that screen appearance when trying to shut down. (and you can all tell the "data" folder is on disk3 in my array. It could not be un-mounted, so it still showed its contents. Those that could be successfully un-mounted all show incorrectly as unformatted. ) Joe L. (Now Tom might be God-Like, but he is not THE God... who obviously is listening to our prayers )
May 6, 200917 yr LOL - yes that can be "unsettling" to see "unformatted" Here's the reason why I haven't really addressed this yet: If you use only the "stock" unRAID distro, and never have command windows open, then this can't happen. The system has been designed to be entirely controlled via the webGui and external server access, e.g., via Windows Explorer. From the beginning I've wanted to "hide" underlying linux from the end-user. Hence the management of the system via webGui (ie, emhttp) can be view as a "closed system". The code is better and better at recognizing things that take place outside control of emhttp, for example, disk spin up/down; but, there are still places where this is not the case. (This being one of them.) Along these same lines, people have wanted "hooks" for a long time, eg, "array start", "array stop", etc. I have resisted this because of possibility of side-effects that emhttp doesn't know about. But release 5.0 will give me a chance to revisit all of this, since the goal of this release series is to make the server user-extensible. BTW, Joe: did -beta6 solve the ntp startup problem for you??
May 6, 200917 yr LOL - yes that can be "unsettling" to see "unformatted" Here's the reason why I haven't really addressed this yet: If you use only the "stock" unRAID distro, and never have command windows open, then this can't happen. ... With all due respect ... ya da... ya da... .... i could agree with you if telnet isnt enabled by default. But it is so .. cough.. em ... so well i dont agree Semantics really but Im bored so had to say something
May 6, 200917 yr LOL - yes that can be "unsettling" to see "unformatted" Here's the reason why I haven't really addressed this yet: If you use only the "stock" unRAID distro, and never have command windows open, then this can't happen. ... With all due respect ... ya da... ya da... .... i could agree with you if telnet isnt enabled by default. But it is so .. cough.. em ... so well i dont agree Semantics really but Im bored so had to say something Actually since rsync and ftp are installed.. a command window isn't the only process that can have a mount point with access. I would suggest just changing the unformatted status message change to "not unmounted!!" or "BUSY" if it is actually mounted and a stop condition was issued. You have to consider... if the drive is mounted.. it's most likley formatted! ;-)
May 6, 200917 yr LOL - yes that can be "unsettling" to see "unformatted" I undestand..., but these issues will continue... I know how you feel about the side-effects... That's why I suggested just attempting to re-mount the drives you had just un-mounted if the un-mount of all of them is impossible. Those that cannot be mounted might really be un-formatted. Those that can be re-mounted would then look normal and not be a surprise to the users. Along these same lines, people have wanted "hooks" for a long time, eg, "array start", "array stop", etc. I have resisted this because of possibility of side-effects that emhttp doesn't know about. But release 5.0 will give me a chance to revisit all of this, since the goal of this release series is to make the server user-extensible. Yes... they will need to be coded in a way that does not block emhttp from doing what it needs to do, or can recover from. Many things to consider. And, an informed "risk" for users who care to add after-array-start, and before-array-stop tasks. BTW, Joe: did -beta6 solve the ntp startup problem for you?? Yes, it did... I did run into one issue that is probably unrelated to the start of the NTP service, but possibly an effect of the new release of Linux... (but you'll know better if it could...) For the very first time ever, I loaded the new release, rebooted, and the array did not start. It said I had a disk missing. Looking in the syslog, it showed this series of lines: May 6 09:54:00 Tower kernel: ata5: SATA max UDMA/133 cmd 0xe800 ctl 0xe400 bmdma 0xd800 irq 18 May 6 09:54:00 Tower kernel: ata6: SATA max UDMA/133 cmd 0xe000 ctl 0xdc00 bmdma 0xd808 irq 18 May 6 09:54:00 Tower kernel: ata5: link is slow to respond, please be patient (ready=0) May 6 09:54:00 Tower kernel: ata5: device not ready (errno=-16), forcing hardreset May 6 09:54:00 Tower kernel: ata5: link is slow to respond, please be patient (ready=0) May 6 09:54:00 Tower kernel: ata5: SRST failed (errno=-16) May 6 09:54:00 Tower kernel: ata5: link is slow to respond, please be patient (ready=0) May 6 09:54:00 Tower kernel: ata5: SRST failed (errno=-16) May 6 09:54:00 Tower kernel: ata5: link is slow to respond, please be patient (ready=0) May 6 09:54:00 Tower kernel: ata5: SRST failed (errno=-16) May 6 09:54:00 Tower kernel: ata5: SRST failed (errno=-16) May 6 09:54:00 Tower kernel: ata5: reset failed, giving up Followed later by: May 6 10:00:12 Tower kernel: md: import disk0: [8,80] (sdf) ST31000340AS 9QJ0JPJS offset: 63 size: 976762552 May 6 10:00:12 Tower kernel: md: import disk1: [3,64] (hdb) HDS725050KLAT80 KRVA03ZAG3V5LD offset: 63 size: 488386552 May 6 10:00:12 Tower kernel: md: import disk2: [22,0] (hdc) ST3400620A 5QH00QPN offset: 63 size: 390711352 May 6 10:00:12 Tower kernel: md: import disk3: [22,64] (hdd) HDS725050KLAT80 KRVA03ZAG4V99D offset: 63 size: 488386552 May 6 10:00:12 Tower kernel: md: import disk4: [33,0] (hde) ST3400620A 5QH00PF4 offset: 63 size: 390711352 May 6 10:00:12 Tower kernel: md: import disk5: [33,64] (hdf) WDC WD5000AAKB-00YSA0 WD-WCAS88067391 offset: 63 size: 488386552 May 6 10:00:12 Tower kernel: md: import disk6: [34,0] (hdg) ST3400633A 3PM0BE0T offset: 63 size: 390711352 May 6 10:00:12 Tower kernel: md: import disk7: [34,64] (hdh) ST3500641A 3PM147P2 offset: 63 size: 488386552 May 6 10:00:12 Tower kernel: md: import disk8: [57,0] (hdk) ST3750640A 5QD29FXK offset: 63 size: 732574552 May 6 10:00:12 Tower kernel: md: import disk10: [3,0] (hda) ST3750640A 5QD2AX3G offset: 63 size: 732574552 May 6 10:00:12 Tower kernel: md: disk11 missing A reboot worked... The disk controller apparently responded the second time without timing-out. Joe L.
May 6, 200917 yr LOL - yes that can be "unsettling" to see "unformatted" Here's the reason why I haven't really addressed this yet: If you use only the "stock" unRAID distro, and never have command windows open, then this can't happen. ... With all due respect ... ya da... ya da... .... i could agree with you if telnet isnt enabled by default. But it is so .. cough.. em ... so well i dont agree Semantics really but Im bored so had to say something Actually since rsync and ftp are installed.. a command window isn't the only process that can have a mount point with access. I would suggest just changing the unformatted status message change to "not unmounted!!" or "BUSY" if it is actually mounted and a stop condition was issued. Half the time it is because I cd'd to a /mnt/disk* folder. It is not always an open file... I've seen many of these reported in the forum where logging off the tenlet session allowed the user to shut down cleanly. You have to consider... if the drive is mounted.. it's most likley formatted! ;-) Actually, if it was once mounted it is formatted..., when the error shown in my screenshot presents itself, it is because all the drives showing "unformatted" were un-mounted. The one(s) remaining were unable to be un-mounted.
May 6, 200917 yr Suppose I put a "fuser -k" in front of each un-mount, e.g., fuser -km /mnt/disk1 umount /mnt/disk1
May 6, 200917 yr Suppose I put a "fuser -k" in front of each un-mount, e.g., fuser -km /mnt/disk1 umount /mnt/disk1 I think that will suffice for now. root@unraid~ #cd /mnt/disk6 root@unraid /mnt/disk6 #fuser -km /mnt/disk6 /mnt/disk6: 862c Connection to unraid closed.
Archived
This topic is now archived and is closed to further replies.