October 21, 201213 yr Well that sucks... I still have several hundred gigs of data in a pile in the garage to copy over to the unraid server, but at present already have 727,909 files in 35,919 folders, totaling 1.1TB. Its going to take me weeks to sort this as it is. But when I'm done sorting, I should be prepared to move and copy back each top level directory? Under linux, I assume (from the shell prompt), correct? Similar actions from a DOS command prompt won't work?
October 22, 201213 yr Well that sucks... I still have several hundred gigs of data in a pile in the garage to copy over to the unraid server, but at present already have 727,909 files in 35,919 folders, totaling 1.1TB. Its going to take me weeks to sort this as it is. But when I'm done sorting, I should be prepared to move and copy back each top level directory? Under linux, I assume (from the shell prompt), correct? Similar actions from a DOS command prompt won't work? You probably only need to compress directories that have many adds and deletes. if it's all adding, then you don't have to compress or rsync the directories, but if you want to defrag them, you may need to. I don't have all the answers with this. I do know that very large directories are usually inefficient in unix. There might be an option to use a BTREE strcture for the directories. I think ext3 had some option to do this a long time ago. Maybe reiserfs has something similar.
October 22, 201213 yr You probably only need to compress directories that have many adds and deletes. if it's all adding, then you don't have to compress or rsync the directories, but if you want to defrag them, you may need to. I don't have all the answers with this. I do know that very large directories are usually inefficient in unix. There might be an option to use a BTREE strcture for the directories. I think ext3 had some option to do this a long time ago. Maybe reiserfs has something similar. So I can use rsync to move files from one folder to another on the same server? And this will serve as the apparatus with which to defrag said folders? Can you advise on the syntax? I though rsync was supposed to be used from machine to machine.
October 22, 201213 yr rsync can be used as a local copy or a move (when you add --remove-sent-files) rsync lets you do a whole tree of files locally or remotely. You can also just do a mv source directory / destination directory The idea really is to create all directories first, then move files into them and let them grow as needed. If you still have 10,000 files in one directory, it's going to be a bit slow until that directory and it's related inodes get loaded into memory.
October 22, 201213 yr In looking at some reiserfs issues, it could also be that allot of similar names end up getting hash collisions for the directory tree. I don't know much about it and how to deal with it other then creating a deeper directory tree.
October 23, 201213 yr Use the cache dirs add-on. The only downside to this is when you start to have huge number of files, cache_dirs is not as effective as you would want. it can also cause OOM 'out of memory'. Even doing a find down a large tree, does not resolve the delay of adding a new file to a directory containing tons of files. In addition, I tried to increase the DENTRY table in the kernel to support more file entries via boot parameter. While it worked slightly, it chewed up low kernel ram, thus causing an OOM condition. In any case, I would suggest to try running it and see how your performance changes.
Archived
This topic is now archived and is closed to further replies.