So we all know that unraid is great in that it can recover from a single drive failure. Also, if there is a 2nd drive failure, only the data on those two drives is lost and all the data on the other disks are fine. The issue is that I don't have my data well organized and split up onto disks. If I were to lose two disks, I would have no idea what data I actually lost. I'm suggesting a feature that keeps track of what files are on each disk. This can either be done periodically (cron job) or every time data is written to the array. A sqlite or mysql database may be easiest and more efficient than saving the output of a "find ." on each disk.

I have lots of ideas on this.

I've tested with .txt files, md5sum files, .gdbm files and sqlite tables.

Each works at different speeds and space requirements.

 

The .gdbm is the fastest for storing caching the stat[] block and providing space for a hash value.

This was my attempt at doing a replacement dir_cache that actually stored the stat blocks for other data.

 

During my experiments, after population, sweeping this file and file system were very fast (within 5 seconds) for approx 350,000 files as long as the data was already cached in memory.

 

If it wasn't cached, then it took approx 30 minutes to do the match/load.

Size was approx 80MB per 300,00 files, which could probably be compact more without hash value space.

 

With SQLite the same amount of files becomes over 550MB for half a million files.

The benefit of this approach is the whole array can be indexed like the slocate/mlocate programs.

A hash value can be maintained and a php/browser interface with emhttp can be used to access this data.

 

I had some other ideas about storing the drive's model/serial number as a label so that foreign disks could be mounted, cataloged and unmounted for locating files.

 

My issues with all of this so far as been, where do I store these tables?

If I use the ramdrive, it would run very fast but take ram away from other applications.

On an array drive, each database can be located for it's respective drive, however, that will require rsyncing or backing up that drive's database somewhere.

 

Frankly, I don't have a cache drive so I haven't considered how to use that for now.

As far as doing this in shell, it's pretty easy to use find to do this in cron.

There are examples of how to do this on the forum.

I've been running a nightly cron job to save the output of the find command to a file. As a comparison, my 756,000 files takes about 134 MB. This is only saving the file structure to a file and does not compute or save any md5sums.