tucansam Posted October 13, 2012 Share Posted October 13, 2012 I realize many folks use unraid for HTPC media storage, and may have a few hundred (or thousand) movies, lots of songs, etc. I am using my unraid server for that, but also as a primary file server for 30+ years of files that I've aquired over the years. I am also storing files for two professional photographers in the family, as well as 15 years worth of files from a family business, and easily have over a half a million files that I need safeguarded. The directory structure is not horribly complex but can reach 8-10 levels deep at times. Many individual directories may contain 2500 files or more. I am noticing extremely sluggish behavior when browsing files, and just noticed that this is effecting unraid's overall performance too (I went to the web interface and it took almost 30 seconds to respond and load). Windows explorer (arguably not the best program in the world) hangs (for minutes at a time) when browsing the file structure or moving and copying files, even one or two files at a time. I have a drive mapped, and if I open a command prompt and try to change to that drive in DOS, again, it hangs for a long time before it finally completes the command. The command "dir /s" on my main share ran slower than you'd expect on an old floppy disk. The unraid system is a dual core AthonX2 with 8GB RAM and all disks are SATA-II. It is connected via gig ethernet to my workstation, a 3GHZ quad core. I don't think hardware is whats slowing this system down. I checked the syslod and it looks like the Samba process is being killed and restarted periodically, and I have no idea why. Not a lot else looks amiss in the syslog. Performance was awesome last night as I began copying files. It wasn't until I got the share filled with most of my data that things started slowing down. Now I can barely browse the directory tree on the server in Windows. I'm wondering if the sheer number of files has unraid irritated. There isn't any sort of cache or indexing process running is there? Something that would continuously be reading the file structure for changes? If I copy a few hundred small files over, once Windows shows the process as being complete, the disk access light on the disk I'm copying to on the unraid server thrashes for minutes... After the copy process is complete. I am not yet running a parity drive, and the drive in use is a brand new WD 3TB that passed four preclear cycles. Simply trying to open a folder located on the server in explorer sometimes responds, sometimes takes a few seconds, sometimes takes 30 seconds... And usually explorer then hangs or closes itself. Again, things were working perfectly until I added massive data to the server. Any words of advice? Link to comment
tucansam Posted October 13, 2012 Author Share Posted October 13, 2012 As an addendum, if I telnet into the unraid server, cd to the mount point of my share, and run "ls" on a folder with a large number of files, it takes FOREVER to execute the command. At first I thought this was a problem with explorer.exe, and was expecting it to be fast if I'm interactively logged into the server. Alas, that is not the case. Link to comment
Frank1940 Posted October 13, 2012 Share Posted October 13, 2012 One thing which will speed up Windows Explorer (Win7) is to change the properties of the folder itself (Not a shortcut to it), choose 'Properties', click on the 'Customize' tab and change to 'optimize for document'. This will prevent windows from scanning all of the files in the folder to associate a program with the file extension. (It has helped me with folders with large numbers of files with mixed file types.) From another discussion recently, I learned Linux does cache files and directories. However, it requests unused memory from the pool to do this and this memory allocation can be lost when other processes need it. You didn't list how much memory you have in your unRAID server but I would certainly consider increasing it to the maximum that 32bit Linux will support. There is also the File Name Caching program that is a part of Simple features but I doubt if it can track a million files. Link to comment
tr0910 Posted October 13, 2012 Share Posted October 13, 2012 We use unRaid for our Photo Studio Repository and would have in excess of a million files at present, when you include all the other documents and website backups etc that are on the server. And this is only with 10 drives filled. Actually one 2tb drive has just under 1/2 million files, and then we bought the Canon 5d Mk II and since we alway shoot raw, the other drives each have about 1/4 million files. We are using mostly 3tb and some 2tb drives on v5 rc8a. Have 4 servers at present and 2 of them have only Amd 140 Semphron's with 1gb ram. The 2 Amd's are offsite backups to the main server, and the 4th one is a new work in process.... I make sure that I never have more than 95% full on any drive though. Might you have filled a drive completely? That can make unRaid sluggish. Link to comment
Simon Posted October 14, 2012 Share Posted October 14, 2012 Like tr0910, I see odd performance issues when one of the drives gets too full. Link to comment
tucansam Posted October 14, 2012 Author Share Posted October 14, 2012 Right now I'm at 700GB free on a 3TB disk and all other disks are totally empty. Link to comment
Simon Posted October 14, 2012 Share Posted October 14, 2012 Maybe post the syslog for one of the Unraid experts to check over too. When you telnet in, are you doing the ls on the /mnt/diskn name or /mnt/user/sharename ? Link to comment
Influencer Posted October 14, 2012 Share Posted October 14, 2012 If I had my test server set up I'd be interested to test this. Filling up the server with 1 byte files would be a good simulation to see if noticeable speed degradation presented itself. ...I really need to get on the ball with that thing! Link to comment
neilt0 Posted October 14, 2012 Share Posted October 14, 2012 See my topic about this for more info from WeeboTech: http://lime-technology.com/forum/index.php?topic=22906.0 Emptying a drive after it has had many drive write ops and re-filling it seems to help. I also use cache_dirs which obviously speeds up directory listings. Filling it using rsync may (my theory) help prevent directory fragmentation -- rsync writes directories before copying all the files. Link to comment
tucansam Posted October 15, 2012 Author Share Posted October 15, 2012 Emptying and re-filling? Like copying 3TB of data off the disk and then back? Example... I have a share. In that share are two folders -- "my documents" and "sort." If I click on "my documents" in Windows explorer, I get a second or two pause, and then the directory listing. According to Windows, "my documents" contains 433GB, 157,473 files, and 9,594 folders. "sort" contains 260GB, 130,912 files, and 9,926 folders. If I click on "sort," it takes a full two minutes before I get a directory listing. This information was obtained via right-click -> properties, and the process took over seven minutes total to count up both folders. During Windows' scan of the two folders, unraid looked like this: top - 16:24:55 up 21:08, 1 user, load average: 1.01, 0.70, 0.35 Tasks: 77 total, 2 running, 75 sleeping, 0 stopped, 0 zombie Cpu(s): 8.3%us, 8.6%sy, 0.0%ni, 52.8%id, 29.7%wa, 0.0%hi, 0.5%si, 0.0%st Mem: 8278288k total, 5331284k used, 2947004k free, 304312k buffers Swap: 0k total, 0k used, 0k free, 4638852k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6923 jason 20 0 23032 10m 8820 R 19 0.1 0:26.86 smbd 1639 root 20 0 138m 20m 556 S 13 0.3 3:57.48 shfs 1084 root 20 0 11900 1484 1056 S 0 0.0 0:02.42 emhttp 1466 root 20 0 0 0 0 S 0 0.0 0:13.60 unraidd 1 root 20 0 828 280 240 S 0 0.0 0:09.65 init 2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0 0.0 0:00.07 ksoftirqd/0 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0 7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1 9 root 20 0 0 0 0 S 0 0.0 0:00.04 ksoftirqd/1 10 root 20 0 0 0 0 S 0 0.0 0:01.43 kworker/0:1 11 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper 12 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:1 145 root 20 0 0 0 0 S 0 0.0 0:10.44 sync_supers 147 root 20 0 0 0 0 S 0 0.0 0:00.00 bdi-default 149 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd 256 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff SMART report on that drive looks like this: SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 176 176 021 Pre-fail Always - 6158 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 6 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 55 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 6 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 5 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 14 194 Temperature_Celsius 0x0022 120 115 000 Old_age Always - 30 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged This is over a gig network. I can't for the life of me figure out why basic file operations are so slow on this volume. Any suggestions? Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.