Too slow when opening directories

[papnikol]

Hi everybody

My unraid server is quite slow but not when copying files (where i get speeds between 20-40Mbs with parity drive enabled). It is very slow when i browse files from my client computer. for example, i will go to a directory and when i left click or right click it may take about 10 to 40 seconds for the directory to open or the drop down menu to appear, respectively, even if the drives are not spun down.... During all that time, i have noticed that the server starts doing reads.

 

Now, I use my unraid server to store eveyfile i have, so there are hundreds of thousands of files stored. This is why i suspect it might have something to do with the filetree. The only solution i can think of is increasing the memory to 4Gb (i only have 500Mbs and using free -h i see that only about 5mbs are free) or upgrading to the new unmenu (thanks Joe) and installing a swap drive in a new USB drive. Will that help? Or is there another solution? Or is it possible that nothing can be done?

 

[Joe L.]

Hi everybody

My unraid server is quite slow but not when copying files (where i get speeds between 20-40Mbs with parity drive enabled). It is very slow when i browse files from my client computer. for example, i will go to a directory and when i left click or right click it may take about 10 to 40 seconds for the directory to open or the drop down menu to appear, respectively, even if the drives are not spun down.... During all that time, i have noticed that the server starts doing reads.

 

Now, I use my unraid server to store eveyfile i have, so there are hundreds of thousands of files stored. This is why i suspect it might have something to do with the filetree. The only solution i can think of is increasing the memory to 4Gb (i only have 500Mbs and using free -h i see that only about 5mbs are free) or upgrading to the new unmenu (thanks Joe) and installing a swap drive in a new USB drive. Will that help? Or is there another solution? Or is it possible that nothing can be done?

 

A swap file, regardless of where it is locate, will SLOW things down if it is used, not speed it up.  Putting a swap file on a flash drive is asking for it to wear out (fail) quickly, as they are rated for only a limited number of "writes" before they stop functioning.

 

If you are browsing a directory with thousands of files in it, it will take time to transmit all those to the client PC.  If your client PC only has a 100Mbit LAN connection, this will seem slow.  I have exactly this situation with my older media players.  My "Movies" folder has nearly 3500 files. (roughly 1000 Movies with affiliated thumbnails, fan-art, and info files)

 

To eliminate the server from having to read the physical disks you can add more RAM and install the cache_dirs program.  This will normally keep the disks from having to spin up at all as all the directory entries are already in memory. 

 

the cache_dirs program is attached to this thread describing it: http://lime-technology.com/forum/index.php?topic=4500.0

 

Once the directory entries are in memory, there is nothing that can access them faster.  You are basically limited by the speed of the LAN and the client PC.

 

If your client PC has a "thumbnail" view enabled, then the whole process will be much slower, as each file will be opened to read its contents to present the thumbnails to your file-browser..  (I recommend using a file-listing view instead if your directories are huge) cache_dirs does not keep the contents of all your files in memory.  It could not, as normally you have only a few Gigabytes of RAM, and typically thousands of Gigabytes of files.

 

Joe L.

[papnikol]

Joe, as usually, thanx for your response.

 

A swap file, regardless of where it is locate, will SLOW things down if it is used, not speed it up.  Putting a swap file on a flash drive is asking for it to wear out (fail) quickly, as they are rated for only a limited number of "writes" before they stop functioning.

 

well, if i use a $5-10 usb stick for swap, i wouldn't mind if it failed after 1-2 years. although, since you say it wouldn't help, there is no point of using it anyway

 

If you are browsing a directory with thousands of files in it, it will take time to transmit all those to the client PC.  If your client PC only has a 100Mbit LAN connection, this will seem slow.  I have exactly this situation with my older media players.  My "Movies" folder has nearly 3500 files. (roughly 1000 Movies with affiliated thumbnails, fan-art, and info files)

 

i actually have a gbit connection with a cat6 cable (i think i do at least). but even if it was 100 Mbit why would it take so long? if each file's length is even 100 bytes and i have 500.000 files, that would be 50Mbytes of browsing (and that is if all files were in the same directory). That wouldnt take more than one second to transmit. so i think the bottleneck is elsewhere (probably the memory).

 

To eliminate the server from having to read the physical disks you can add more RAM and install the cache_dirs program.  This will normally keep the disks from having to spin up at all as all the directory entries are already in memory. 

 

the cache_dirs program is attached to this thread describing it: http://lime-technology.com/forum/index.php?topic=4500.0

 

Once the directory entries are in memory, there is nothing that can access them faster.  You are basically limited by the speed of the LAN and the client PC.

 

i thought that cache_dirs stores only directory structure, not files. or does directory structure implies that files are included?

 

If your client PC has a "thumbnail" view enabled, then the whole process will be much slower, as each file will be opened to read its contents to present the thumbnails to your file-browser..  (I recommend using a file-listing view instead if your directories are huge) cache_dirs does not keep the contents of all your files in memory.  It could not, as normally you have only a few Gigabytes of RAM, and typically thousands of Gigabytes of files.

 

Joe L.

 

I have thumbnail view disabled

 

 

[Joe L.]

Joe, as usually, thanx for your response.

 

A swap file, regardless of where it is locate, will SLOW things down if it is used, not speed it up.   Putting a swap file on a flash drive is asking for it to wear out (fail) quickly, as they are rated for only a limited number of "writes" before they stop functioning.

True, but cached directory entries would not swap out of memory.  The only think that will use swap is memory images of processes that are sleeping, waiting on I/O or their turn to execute. 

 

well, if i use a $5-10 usb stick for swap, i wouldn't mind if it failed after 1-2 years. although, since you say it wouldn't help, there is no point of using it anyway

 

If you are browsing a directory with thousands of files in it, it will take time to transmit all those to the client PC.  If your client PC only has a 100Mbit LAN connection, this will seem slow.  I have exactly this situation with my older media players.  My "Movies" folder has nearly 3500 files. (roughly 1000 Movies with affiliated thumbnails, fan-art, and info files)

 

i actually have a gbit connection with a cat6 cable (i think i do at least). but even if it was 100 Mbit why would it take so long? if each file's length is even 100 bytes and i have 500.000 files, that would be 50Mbytes of browsing (and that is if all files were in the same directory). That wouldnt take more than one second to transmit. so i think the bottleneck is elsewhere (probably the memory).

There is a HUGE amount of overhead.  I'll bet the number of characters transferred is FAR higher.    For kicks, do a

ifconfig eth0

both before and after the directory listing to see the actual number of characters transferred.  It might be it is sending only a few directory items at a time, and waiting for an "ack" from the client PC before continuing with the next set of directory entries.

To eliminate the server from having to read the physical disks you can add more RAM and install the cache_dirs program.   This will normally keep the disks from having to spin up at all as all the directory entries are already in memory. 

 

the cache_dirs program is attached to this thread describing it: http://lime-technology.com/forum/index.php?topic=4500.0

 

Once the directory entries are in memory, there is nothing that can access them faster.  You are basically limited by the speed of the LAN and the client PC.

 

i thought that cache_dirs stores only directory structure, not files. or does directory structure implies that files are included?

You are correct.  It does not cache files, just whatever blocks read when doing a directory listing.

If your client PC has a "thumbnail" view enabled, then the whole process will be much slower, as each file will be opened to read its contents to present the thumbnails to your file-browser..  (I recommend using a file-listing view instead if your directories are huge) cache_dirs does not keep the contents of all your files in memory.  It could not, as normally you have only a few Gigabytes of RAM, and typically thousands of Gigabytes of files.

 

Joe L.

 

I have thumbnail view disabled

 

 

good.  or it would slow things down more.