April 11, 201016 yr Tapes are just not feasible any more. Were they ever, really ?? Yes, about 10 years ago I acquired some high density drives and tapes during a computer room move. With hundreds of free 20gb tapes and a SCSI LVD High speed drive, it was quite usable.
April 11, 201016 yr WeeboTech, do you have any time schedule for your checksum/par2 project? It sounds really good and the missing piece in unRAID!
April 11, 201016 yr WeeboTech, do you have any time schedule for your checksum/par2 project? It sounds really good and the missing piece in unRAID! I'm working on a couple laptop sales. After that, updates to the powerdown package. Then this "locate" and integrity verification package. I have not fully flushed out how the par2 verification package would be. For the integrity verification tool, It will be easy, a GDBM database or a SQLite Database, full filename as key, checksum and md5sum as data. The beauty is we'll have locate capability and verification. For par2, I need to build a set of directories as the source directory and store the par2 files there. I think it will have to be a different tool.
April 11, 201016 yr I too think that splitting them in two tools would be better, one - locate corruption only, the other - locate and correct corrpution. The second would require the first package to be installed though. Sounds great!
April 11, 201016 yr The second would require the first package to be installed though. Sounds great! They will be independent of one another. The first package is file based, The second is directory based. (but can operate on files)
April 12, 201016 yr WeeboTech, do you have any time schedule for your checksum/par2 project? It sounds really good and the missing piece in unRAID! I'm working on a couple laptop sales. After that, updates to the powerdown package. Then this "locate" and integrity verification package. I have not fully flushed out how the par2 verification package would be. For the integrity verification tool, It will be easy, a GDBM database or a SQLite Database, full filename as key, checksum and md5sum as data. The beauty is we'll have locate capability and verification. For par2, I need to build a set of directories as the source directory and store the par2 files there. I think it will have to be a different tool. Eagerly awaiting integrity checker and par2 verification packages. I did some experimenting with par2 (par2cmdline) on unRAID. Works very well. Only major problem is that par2 does not support subdirectories. All files must be in a single folder. It works well on almost full disks that will be treated as read-only. Here is an excerpt from a recent post (see THIS LINK for complete post / thread). If you wrote a script to create links to all files on a disk in a folder as described below, par2 would work and be easily automated with a script. My grand plan was to create links to all of the files on a drive in a single directory. The link names could be the fully qualified file names, the "/"s converted to periods or some other character. An automated script could be run to do this. You could then run PAR2 against that directory and have the same effect. I did an experiment and found that PAR2 worked with the links. But I never wrote the script. Most of my files are in a single directory and work fine with PAR2 without this technique.
April 12, 201016 yr What I was thinking of doing was borrowing some logic from the mover script. Traverse down a disks tree. Build the same directory structure on a designated par2 db disk. (Hidden directory on cache?). Now you have locations that can be traversed, paths can be derived and you have source file and destination storage directory. This is where all that extra horsepower some of us have purchased will come in handy. I think it would be important to have raid1 on cache for something like this.
April 12, 201016 yr So you'd have a separate set of PAR files per directory? Yes, I think that is a good way to go... Yes/No ?
April 12, 201016 yr Yes, I think that is a good way to go... Yes/No ? Sounds like a plan. Despite my comment about being paranoid earlier, I was actually also considering doing something similar for my files.
April 13, 201016 yr So you'd have a separate set of PAR files per directory? Yes, I think that is a good way to go... Yes/No ? It has pros and cons. On the pro side, it allows you to create par sets incrementally and not encounter a very lengthy re-PARing procedure if you add or delete some files. This would be very useful for drives that have some dynamic files that change frequently, as well as some other directories that are static. On the con side, the ability to recover from a significant data loss is drastically reduced. For example say that you lost a 1G .VOB file that was part of a 4G movie. You would not be able to recover this file unless you dedicated 25% of the source file space to creating pars (25% redundancy). On the other hand, if you had built a single par set on a 1T drive full of 4G movies (that would be 250 movies) and lost a single 1G .VOB file, you would only need to have used 1% redundancy in PAR files. That entire 1% could be used to recover any single file on the entire drive. If I had to choose, based on the way I use my array, I'd prefer it to create one giant PAR set for the entire drive. But better would be to have an option on a disk by disk basis. In that way you could create single par sets on drives that were full, and use the directory approach on more dynamic disks.
April 14, 201016 yr I did a quick (ok, not so quick) experiment last night. I created a single directory of symbolic links to a whole tree as an experiment. The idea was to consider one par set per drive. To create a directory of symlinks, I did a find down the tree, then used md5sum to create a hash out of the filename. Then did a symlink of ln -s $fullfilename $md5sumname This creates a huge directory of symlinks. With access to 500000 files, I did not have any collisions. But man, this took a long time to do in a shell. Doing a directory listing in that dir was just not practical. Just to give you an idea, this was the size of the directory list after. [root@hercules ~]# ls -ld --si /tmp/DIRLIST drwxr-xr-x 2 root root 30M Apr 14 07:02 /tmp/DIRLIST I have not tested a par2 set in that directory yet. I plan to try this on a smaller disk. It was just an experiment in feasibility. This was a real quick -n- dirty test, but here is what I used. First I did a find down a whole tree and piped that to > filelist find /mnt/disk7 -mount -type f -print > filelist Then I used this quick script to create one directory of symlnks to everything in filelist. The main purpose was to see if I would have collisions. #!/bin/bash TMPFILE=/tmp/filename.$$ trap "rm -f ${TMPFILE}" EXIT exec 3>filelist.md5sum exec 0<filelist mkdir /tmp/DIRLIST cd /tmp/DIRLIST while read filename do echo "${filename}" | md5sum > ${TMPFILE} read -a FIELDS < ${TMPFILE} if [ ! -L ${FIELDS[0]} ] then if ! ln -s "${filename}" "${FIELDS[0]}" then echo "error linking: '${filename}' to: '${FIELDS[0]}'" fi fi echo "${FIELDS[0]} ${filename}" >&3 done exec 3<- Please note, I ran this on my workstation against the whole root tree. It consisted of over 500,000 files. No collisions. I also tested the md5sum list by sorting it. Then running it through uniq, then comparing a wc -l on the 2 files. No entries were deleted. So no collisions were detected. (Which is pretty important). my /tmp folder is 2GB. If you do this on an unRAID system, you will want to redirect the tmp directory above. Possibly to a hidden folder on your cache drive. In comparison a 1.5tb drive with compressed movies came out to about 4000 files and ran in a better time.
April 14, 201016 yr I did a find down the tree, then used md5sum to create a hash out of the filename. Then did a symlink of ln -s $fullfilename $md5sumname If you did a md5sum of the file itself, instead of the file-name, your directory of symlinks would serve two purposes... The md5sum name could be a quick verification of file integrity. Yes, it will be slower... But then, you only need do it once. Almost no linux utilities can deal with directories of 500000 entries. You will reach many limits even trying to process the directory listing, forget about getting the listing via SAMBA any time this century
April 14, 201016 yr I did a find down the tree, then used md5sum to create a hash out of the filename. Then did a symlink of ln -s $fullfilename $md5sumname If you did a md5sum of the file itself, instead of the file-name, your directory of symlinks would serve two purposes... The md5sum name could be a quick verification of file integrity. Yes, it will be slower... But then, you only need do it once. I thought of this, but my consideration was to come up with a unique name in a directory that would always refer back to the full path of the file being linked. If I do an md5sum on the file itself, I'll get a hash. But if that file has changed, the hash will change. If the hash changes, what happens to the old hash? How do I resolve the variances? For now the filename to hash symlinking is really to make a whole tree accessible as a single directory for par2. Other then that I would store the md5sum hash of the data from the file into a database structure. Either GDBM or SQLite. then write shell tools to put/get the fields from these databases. It will just be so much cleaner that way.
April 14, 201016 yr A quick exercise... on one file, since testing on 500000 takes much longer... I created a test file. (much quicker to get checksum, and easier to fake a changed contents than one of my movie files) root@Tower:/mnt/disk1/Movies# echo "This is a test file" >JOE.ISO I calculated the md5 checksum on the file root@Tower:/mnt/disk1/Movies# md5sum JOE.ISO 5dd39cab1c53c2c77cd352983f9641e1 JOE.ISO I created a directory to hold all the symlinks root@Tower:/mnt/disk1/Movies# mkdir /tmp/md5sumdir I created a symbolic link from the file to the md5name. root@Tower:/mnt/disk1/Movies# cd /tmp/md5sumdir root@Tower:/tmp/md5sumdir# ln -s /mnt/disk1/Movies/JOE.ISO 5dd39cab1c53c2c77cd352983f9641e1 I can verify the file is intact by running md5sum on the md5filename root@Tower:/tmp/md5sumdir# md5sum 5dd39cab1c53c2c77cd352983f9641e1 5dd39cab1c53c2c77cd352983f9641e1 5dd39cab1c53c2c77cd352983f9641e1 The file name and checksum match. I can see the full path to the file... needed to know which file is corrupted if corruption is detected root@Tower:/tmp/md5sumdir# ls -l total 0 lrwxrwxrwx 1 root root 25 Apr 14 10:27 5dd39cab1c53c2c77cd352983f9641e1 -> /mnt/disk1/Movies/JOE.ISO I then changed the file, simulating corruption. root@Tower:/tmp/md5sumdir# echo "i am changing JOE.ISO" >>/mnt/disk1/Movies/JOE.ISO root@Tower:/tmp/md5sumdir# ls -l total 0 lrwxrwxrwx 1 root root 25 Apr 14 10:27 5dd39cab1c53c2c77cd352983f9641e1 -> /mnt/disk1/Movies/JOE.ISO Again, I can check if the file is corrupted. root@Tower:/tmp/md5sumdir# md5sum 5dd39cab1c53c2c77cd352983f9641e1 1956a7b7afd54a6fac7ddf1d8b4934dc 5dd39cab1c53c2c77cd352983f9641e1 This time the checksums do not match. I can still see the full path to the file with an "ls -l" command and if the file is completely removed I can still see where the symlink was pointing to. (it shows in "red" in the ls -l command as a broken symlink) root@Tower:/tmp/md5sumdir# rm /mnt/disk1/Movies/JOE.ISO root@Tower:/tmp/md5sumdir# ls -l total 0 lrwxrwxrwx 1 root root 25 Apr 14 10:27 5dd39cab1c53c2c77cd352983f9641e1 -> /mnt/disk1/Movies/JOE.ISO The directory of symlinks has a record of both the full paths to the files and a easy way to verify the file contents to know if it has changed. basically cd md5dir for filename file in `ls md5dir` do if [ $filename != `md5sum $filename` ] then echo "file `ls -l $filename` md5sum changed, it may be corrupted" fi done Your thoughts? Joe L.
April 14, 201016 yr This method is simpler and easier to work with at the shell level. Where this could have a problem is on duplicate files which may exist in different directories. Is this something to consider?
April 14, 201016 yr This method is simpler and easier to work with at the shell level. Where this could have a problem is on duplicate files which may exist in different directories. Is this something to consider? Yes, and I did not think of that... as identical files would have the same checksum, regardless of their names. (shows what happens when you only test with one file ) ok... how about the name of the symlink then being a combination of our ideas (and twice as long): fileMD5-filepathMD5 We can then still have unique names for identical files in different directories, and still determine if the contents changed (matching to the first part of the symlink name), and still know where it symlink's to. Joe L.
April 15, 201016 yr Please forgive me if my question is totally irrelevant / ignorant, but could you add a separate drive outside the unRAID array that is a duplicate of the parity drive within the array, and use those TWO copies to checksum against each other for corruption? Or does that not help with the double data drive loss issue?
April 15, 201016 yr Please forgive me if my question is totally irrelevant / ignorant, but could you add a separate drive outside the unRAID array that is a duplicate of the parity drive within the array, and use those TWO copies to checksum against each other for corruption? Or does that not help with the double data drive loss issue? I suppose the parity drive could be mirrored for redundancy, but I do not think it practical to protect against a double data drive loss issue. There is an assumption that the parity drive is the most important drive during a drive failure. In reality, EVERY drive in the array is critical during a drive failure. Every accessible drive, including parity, needs to be read, to simulate or virtualize the failed drive. No one drive is any more important then the other in this condition. If you loose the parity drive or any other data drive when the array is in redundancy mode, then the drive that is being handled virtually(simulated) will be lost. The RAID6 or Q-PARITY would help alleviate a double drive loss. The software approach we've been discussing probably would not handle a double drive loss. Buf it the drive came online or was "somewhat" accessible. Then the software tool approach could be used to determine which files are corrupt and may even possibly allow rebuilding them.
April 15, 201016 yr I suppose the parity drive could be mirrored for redundancy, but I do not think it practical to protect against a double data drive loss issue. I was thinking mirroring during write only operations, not true mirroring in terms of sync. That seems to me to represent a configuration less susceptible to random bit rot than true RAID 1. There is an assumption that the parity drive is the most important drive during a drive failure. In reality, EVERY drive in the array is critical during a drive failure. If you loose the parity drive or any other data drive when the array is in redundancy mode, then the drive that is being handled virtually(simulated) will be lost. Of course that makes sense. The parity drive itself is data designed to recreate the missing data of another drive. IMO they are all equally important. It's just the parity drive gets all the transactions. The RAID6 or Q-PARITY would help alleviate a double drive loss. The software approach we've been discussing probably would not handle a double drive loss. Buf it the drive came online or was "somewhat" accessible. Then the software tool approach could be used to determine which files are corrupt and may even possibly allow rebuilding them. So what you're working out is kind of a hybridized workaround that adds some level of intelligent scrubbing of the data (like ZFS does), although without the true 2 disk support. Where would all this extra data structure reside (I saw mention of db components, subfolder files, etc)?
April 15, 201016 yr I suppose the parity drive could be mirrored for redundancy, but I do not think it practical to protect against a double data drive loss issue. I was thinking mirroring during write only operations, not true mirroring in terms of sync. That seems to me to represent a configuration less susceptible to random bit rot than true RAID 1. neat idea, not practical. the only time the parity drive is used, is during write operations. (even to remove a file). You could always do a dd from one drive to another at a point in time. The point is, the moment you mount a hard drive, it gets written to. This updates the superblock, which is going to require a parity write. There is an assumption that the parity drive is the most important drive during a drive failure. In reality, EVERY drive in the array is critical during a drive failure. If you loose the parity drive or any other data drive when the array is in redundancy mode, then the drive that is being handled virtually(simulated) will be lost. Of course that makes sense. The parity drive itself is data designed to recreate the missing data of another drive. IMO they are all equally important. It's just the parity drive gets all the transactions. The parity gets all the "write" transactions and none of the read transactions unless you are in redundancy mode. If parity were striped, all drives would be used equally. Is there a correlation as to a drives usage and failure? Is it based on age, temperature and environment? Does the parity drive get used more then it should? I've seen various concepts communicated on the forum, that the parity drive is sort of a weak vulnerable point. In reality it's not. it does slow down performance, but it's not a weak link. The other concept is that of "thrashing' as an issue during parity build, sync and/or redundancy mode. So many systems are built on RAID5, do they have these concepts or fears? The redunancy/parity sync/parity generate is an issue to be concerned about when you are in one of those modes and you are doing significant writes. Other then that, the modes are fairly gentle on the drives (although not on the power supply and/or power usage). The RAID6 or Q-PARITY would help alleviate a double drive loss. The software approach we've been discussing probably would not handle a double drive loss. Buf it the drive came online or was "somewhat" accessible. Then the software tool approach could be used to determine which files are corrupt and may even possibly allow rebuilding them. So what you're working out is kind of a hybridized workaround that adds some level of intelligent scrubbing of the data (like ZFS does), although without the true 2 disk support. Where would all this extra data structure reside (I saw mention of db components, subfolder files, etc)? I almost don't want to say this... but think of a flex-raid type of data validation/protection on top of unRAID. It will be crude and rudimentary. I see many questions regarding possible and/or validated corruption at a level higher then unRAID that I think this could provide an extra level of "knowing".
April 15, 201016 yr As I said, I'm pretty ignorant with all this beyond the very broadest understanding of the subject, so thank you for taking the time to explain in more detail. I look forward to your progress.
Archived
This topic is now archived and is closed to further replies.