Basically, with my current setup on my unraid box I get thousands (sometimes hundreds of thousands) of new files per day added to my cache drive. They're all really small, however, as they add up they come to be ~ 80GB (128GB SSD as my cache drive).

 

Issue is that the "Mover" script takes forever (12+ hours) to run when dealing with lots of small files, however, runs a lot faster with one large file (about 20 minutes for the same size, just with large files). Does anyone know of any way to increase the speed? I'm not too good with linux(/bash) programs/commands, so, I'm not really sure the best way of copying loads and loads of tiny files are.

This is the same behavior you'll see on any file system that doesn't cache directory writes.  The directory is being updated after each write -- so there's a LOT of activity and significant thrashing (and of course seeks are SLOW compared to the data transfers).    It's easy to demonstrate on a Windows box -- just uncheck the "Enable write caching" on a drive an copy a lot of small files to it;  then check it and do it again ... there'll be a HUGE difference in the times.

 

I don't know how to modify Linux's behavior to do the same effective thing.  I know there's a "cache pressure" parameter you can modify, but can't seem to find specifics on exactly how this works -- or whether it is effective with writes or only with reads.

 

Depending on how much you're writing at one time, you may actually find your performance is more to your liking if you simply eliminate the cache drive => writes will, of course, be slower -- but if the average writes are small files, the performance may not be as bad as you expect ... and you won't have 12 hours worth of writes queued up on your SSD. 

If what garycase said is true, then maybe it could be solved by enabling/installing cache_dirs?

 

I also wonder if you could rewrite mover so that it first rar/tar/zips all the files (saving the structure), then moves the one massive file to the head directory on one of the drives, and then unzips it all (since the structure is still in place inside the file, as long as all your drives are enabled for all your shares it should still look right I believe). There are some issues with this though:

1) as stated before, you have to have all your shares enabled on all drives else your mover script will have to be more complicated (you could do some recursive zipping creating one file per folder usershare instead of the entire structure)

2) you would also have to figure out how mover choses what drive to move files to to prevent problems since you're moving one large file it will go to a single drive each time, so it only spreads to other drives per each move.

3) It limits your cache space by half due to the need for the zip file before the move, so either run it more times per day or figure some other way.

 

I'm not a coder but that's the only idea I can come up with and I don't know what kind of coding problems you could run into, good luck and let us know what you work out.

I think that unRAID may have overheads when creating files on a user share as it tries to decide which disk any particular file should be allocated to.  This is acceptable for media files which tend to be comparatively large, but may be more of an issue with other file types.  This would mean that there be no answer to the time taken to move lots of small files from the cache to the user shares.

 

If I am right with my assumptions then the only way to get good performance for very large numbers of small files may be to write directly to the individual disk shares.

I think that unRAID may have overheads when creating files on a user share as it tries to decide which disk any particular file should be allocated to.  This is acceptable for media files which tend to be comparatively large, but may be more of an issue with other file types.  This would mean that there be no answer to the time taken to move lots of small files from the cache to the user shares.

 

If I am right with my assumptions then the only way to get good performance for very large numbers of small files may be to write directly to the individual disk shares.

 

a.k.a. there is no benefit to using the cache drive then and he should just disable cache for that share no? Even without the overhead, moving a lot of tiny files I don't think would benefit much from the speeds of a cache drive no?

I think that unRAID may have overheads when creating files on a user share as it tries to decide which disk any particular file should be allocated to.  This is acceptable for media files which tend to be comparatively large, but may be more of an issue with other file types.  This would mean that there be no answer to the time taken to move lots of small files from the cache to the user shares.

 

If I am right with my assumptions then the only way to get good performance for very large numbers of small files may be to write directly to the individual disk shares.

 

a.k.a. there is no benefit to using the cache drive then and he should just disable cache for that share no? Even without the overhead, moving a lot of tiny files I don't think would benefit much from the speeds of a cache drive no?

 

Not really the benefit of the speed, just more of the benefit of not having the data drives running all day.

 

Anyway, I guess I'll set it to not use a cache drive if that's the case, thanks.

FWIW, this is NOT an issue of UnRAID overhead in deciding where to write the files -- it is, as I noted earlier, simply the fact that for EACH file there's a directory update.

 

Cache_Dirs won't help at all ... that simple caches the existing directory structure (to eliminate the need to spin up disks to find a file) -- it doesn't eliminate the writes to update a directory after a file is saved.

 

And, finally, Zipping all the files, then unzipping them would make things even worse => (a) it would take the time to Zip the files on the cache drive;  (b) the write of the Zip file would indeed be pretty quick; but then © as you unzipped the file, you'd get a directory write for each file you unzipped (the same as you're seeing now).

 

The Mover operation would be MUCH faster if Linux would cache all of the directory writes;  but as I noted before, I'm not a "Linux guy", and have no idea if there's a parameter that can be set to force this.  [The disadvantage of this is a longer period where you're at risk of a corrupted file system in the event of a power failure before the cache is flushed ... basically not an issue as long as your system has a UPS]

 

But in the absence of that, I'd just disable the cache drive.  That will spread the writes out over the day;  and have the side benefit of all the data written to UnRAID will be fault tolerant (which is not the case right now until Mover has completed each night).

 

But in the absence of that, I'd just disable the cache drive.  That will spread the writes out over the day;  and have the side benefit of all the data written to UnRAID will be fault tolerant (which is not the case right now until Mover has completed each night).

 

Instead of disabling the cache disk entirely you could just create a share that doesn't use the cache disk and save these small files there. That way larger files like movies or TV shows can still take advantage of the cache drive.

 

That may have been what you meant, not sure.

That makes sense.    Since I don't use a cache drive (Once I write something to UnRAID, I want it to be fault-tolerant NOW), I had forgotten that you can enable/disable its use on a per-share basis.

 

But in the absence of that, I'd just disable the cache drive.  That will spread the writes out over the day;  and have the side benefit of all the data written to UnRAID will be fault tolerant (which is not the case right now until Mover has completed each night).

 

Instead of disabling the cache disk entirely you could just create a share that doesn't use the cache disk and save these small files there. That way larger files like movies or TV shows can still take advantage of the cache drive.

 

That may have been what you meant, not sure.

 

Yes, I did mean that I would only disable the cache drive for small files.