Jump to content
bonienl

Dynamix File Integrity plugin

784 posts in this topic Last Reply

Recommended Posts

I just installed this plugin. I think I've got everything set correctly, but when I try to "Build" I get the following error-

Jan 18 18:08:24 Brunnhilde bunker: error: no export of file: /mnt/disk1/TM-Mini/HT-Mini.sparsebundle/com.apple.TimeMachine.SnapshotHistory.plist

Jan 18 18:08:24 Brunnhilde bunker: exported 3 files from /mnt/disk1 with mask *. Duration: 00:00:00

Jan 18 18:15:59 Brunnhilde kernel: ata2.00: exception Emask 0x10 SAct 0x7fffffff SErr 0x280100 action 0x6 frozen

Jan 18 18:15:59 Brunnhilde kernel: ata2.00: irq_stat 0x08000000, interface fatal error

 

Only this line indicates an exception, the file wasn't exported because no checksum value is present in the extended attributes. Rerun Build for disk 1 to solve.

 

The kernel messages are not related.

 

 

Thanks. Started it again. Not sure what to think of this--

 

11jp5yf.jpg

 

That's a little over 10 years. Seems a little long?

 

Edit: Sorry. Should have waited a little longer before posting. Took it about 20 minutes or so but looks like it's moving. The ETA has dropped to 11:41:42.

Share this post


Link to post

I got another email today and I was using yesterdays latest version.

I upgraded to 17/01 version already.

 

Ok, I maybe on to something and think I solved the issue.

 

There is a new version 2016.01.18 available, can you try this and report back...

 

ok. i upgraded already and i will report tomorrow morning when i usually get those emails

 

thanks!

 

Same thing with 2016.01.18

 

Console and webGui login account <xxxxxxxxxxxx@gmail.com>

04:42 (6 hours ago)

 

to root, bcc: me

sed: -e expression #1, char 4: unknown command: `

'

sed: -e expression #1, char 4: unknown command: `

'

 

Should I delete the plugin plus any config files and reinstall or you want to try and find the cause?

 

Share this post


Link to post

I got another email today and I was using yesterdays latest version.

I upgraded to 17/01 version already.

 

Ok, I maybe on to something and think I solved the issue.

 

There is a new version 2016.01.18 available, can you try this and report back...

 

ok. i upgraded already and i will report tomorrow morning when i usually get those emails

 

thanks!

 

Same thing with 2016.01.18

 

Console and webGui login account <xxxxxxxxxxxx@gmail.com>

04:42 (6 hours ago)

 

to root, bcc: me

sed: -e expression #1, char 4: unknown command: `

'

sed: -e expression #1, char 4: unknown command: `

'

 

Should I delete the plugin plus any config files and reinstall or you want to try and find the cause?

 

A few more tries before I throw the towel...

 

Can you list the files in /boot/config/plugins/dynamix.file.integrity

# ls -l /boot/config/plugins/dynamix.file.integrity
total 77232
-rwxrwxrwx 1 root root 71350825 Jan 17 22:55 disk1.export.hash*
-rwxrwxrwx 1 root root   296467 Jan 17 22:54 disk10.export.hash*
-rwxrwxrwx 1 root root   123846 Jan 17 22:54 disk11.export.hash*
-rwxrwxrwx 1 root root   136238 Jan 17 22:54 disk2.export.hash*
-rwxrwxrwx 1 root root   290102 Jan 17 22:54 disk3.export.hash*
-rwxrwxrwx 1 root root  1894057 Jan 17 22:54 disk4.export.hash*
-rwxrwxrwx 1 root root  1023082 Jan 17 22:54 disk5.export.hash*
-rwxrwxrwx 1 root root   594672 Jan 17 22:54 disk6.export.hash*
-rwxrwxrwx 1 root root   902581 Jan 17 22:54 disk7.export.hash*
-rwxrwxrwx 1 root root   460207 Jan 17 22:54 disk8.export.hash*
-rwxrwxrwx 1 root root  1818157 Jan 17 22:54 disk9.export.hash*
-rwxrwxrwx 1 root root      211 Jan  9 12:56 disks.ini*
-rwxrwxrwx 1 root root      361 Jan 17 22:41 dynamix.file.integrity.cfg*
-rwxrwxrwx 1 root root    21184 Jan 17 22:43 dynamix.file.integrity.txz*
-rwxrwxrwx 1 root root      132 Jan 17 22:41 integrity-check.cron*
-rwxrwxrwx 1 root root     1884 Jan 17 22:41 integrity-check.sh*

Like to see if anything "unknown" is present.

 

Share this post


Link to post

In the File Integrity Control page, if I select a disk (say, disk3) and then Build, Export, Check all it good. If I move some files from disk3 to another disk, or if I simply delete them and then I Check disk3 again I get bunker warnings in my syslog that the files are missing, which is true. How then do I update the exported checksum file to remove the entries for the missing files? Re-running Build, Export only seems to add entries for new files - it doesn't delete entries for missing files. I moved over 10,000 from disk3 to disk8 and now I get over 10,000 missing file warnings each time I run a manual check on disk3. Do I really have to delete the disk3.export.hash file manually (followed by Build, Export to re-create it), or am I missing something? The Remove button removes the checksums stored in the extended attributes, not the exported hash files, according to the Help.

Share this post


Link to post

I did the following:

 

1. Move a folder with files from disk3 to disk11

2. Do an export of disk3

3. Do an export of disk11

 

- I verified that the new export file of disk3 is excluding the files I just moved to disk11

- I verified that the new export file of disk11 is including the files I just moved from disk3

 

Share this post


Link to post

I did the following:

 

1. Move a folder with files from disk3 to disk11

2. Do an export of disk3

3. Do an export of disk11

 

- I verified that the new export file of disk3 is excluding the files I just moved to disk11

- I verified that the new export file of disk11 is including the files I just moved from disk3

 

This is how it always worked for me.

Share this post


Link to post

I did the following:

 

1. Move a folder with files from disk3 to disk11

2. Do an export of disk3

3. Do an export of disk11

 

- I verified that the new export file of disk3 is excluding the files I just moved to disk11

- I verified that the new export file of disk11 is including the files I just moved from disk3

 

That works if disk3 contains one or more files. If it's completely empty then it doesn't update the exported hash file. In fact, it doesn't work if it contains just a single zero-length file (touch /mnt/disk3/test) either - there needs to be some actual content.

 

Share this post


Link to post

 

A few more tries before I throw the towel...

 

Can you list the files in /boot/config/plugins/dynamix.file.integrity

# ls -l /boot/config/plugins/dynamix.file.integrity
total 77232
-rwxrwxrwx 1 root root 71350825 Jan 17 22:55 disk1.export.hash*
-rwxrwxrwx 1 root root   296467 Jan 17 22:54 disk10.export.hash*
-rwxrwxrwx 1 root root   123846 Jan 17 22:54 disk11.export.hash*
-rwxrwxrwx 1 root root   136238 Jan 17 22:54 disk2.export.hash*
-rwxrwxrwx 1 root root   290102 Jan 17 22:54 disk3.export.hash*
-rwxrwxrwx 1 root root  1894057 Jan 17 22:54 disk4.export.hash*
-rwxrwxrwx 1 root root  1023082 Jan 17 22:54 disk5.export.hash*
-rwxrwxrwx 1 root root   594672 Jan 17 22:54 disk6.export.hash*
-rwxrwxrwx 1 root root   902581 Jan 17 22:54 disk7.export.hash*
-rwxrwxrwx 1 root root   460207 Jan 17 22:54 disk8.export.hash*
-rwxrwxrwx 1 root root  1818157 Jan 17 22:54 disk9.export.hash*
-rwxrwxrwx 1 root root      211 Jan  9 12:56 disks.ini*
-rwxrwxrwx 1 root root      361 Jan 17 22:41 dynamix.file.integrity.cfg*
-rwxrwxrwx 1 root root    21184 Jan 17 22:43 dynamix.file.integrity.txz*
-rwxrwxrwx 1 root root      132 Jan 17 22:41 integrity-check.cron*
-rwxrwxrwx 1 root root     1884 Jan 17 22:41 integrity-check.sh*

Like to see if anything "unknown" is present.

 

root@TOWER:~#  ls -l /boot/config/plugins/dynamix.file.integrity

total 15664

-rwxrwxrwx 1 root root    80732 Jan 19 04:40 disk1.export.hash*

-rwxrwxrwx 1 root root    32757 Jan 14 04:40 disk2.export.hash*

-rwxrwxrwx 1 root root 13644452 Jan 19 04:41 disk3.export.hash*

-rwxrwxrwx 1 root root      351 Jan 12 23:48 disks.export.20160112.bad.hash*

-rwxrwxrwx 1 root root    46799 Jan 13 23:56 disks.export.20160113.bad.hash*

-rwxrwxrwx 1 root root  840846 Jan 14 23:56 disks.export.20160114.bad.hash*

-rwxrwxrwx 1 root root  1081855 Jan 16 00:00 disks.export.20160115.bad.hash*

-rwxrwxrwx 1 root root  252745 Jan 16 18:19 disks.export.20160116.bad.hash*

-rwxrwxrwx 1 root root      12 Jan 19 04:42 disks.ini*

-rwxrwxrwx 1 root root      348 Jan 18 11:03 dynamix.file.integrity.cfg*

-rwxrwxrwx 1 root root    21184 Jan 18 15:44 dynamix.file.integrity.txz*

-rwxrwxrwx 1 root root      131 Jan 18 11:03 integrity-check.cron*

-rwxrwxrwx 1 root root      648 Jan 18 11:03 integrity-check.sh*

Share this post


Link to post

@John_M

 

You can use the scheduled verification tasks to do automatic checking of the files, this uses the extended attributes and is always up to date.

 

Share this post


Link to post

root@TOWER:~#  ls -l /boot/config/plugins/dynamix.file.integrity
total 15664
-rwxrwxrwx 1 root root    80732 Jan 19 04:40 disk1.export.hash*
-rwxrwxrwx 1 root root    32757 Jan 14 04:40 disk2.export.hash*
-rwxrwxrwx 1 root root 13644452 Jan 19 04:41 disk3.export.hash*
[color=red]-rwxrwxrwx 1 root root      351 Jan 12 23:48 disks.export.20160112.bad.hash*
-rwxrwxrwx 1 root root    46799 Jan 13 23:56 disks.export.20160113.bad.hash*
-rwxrwxrwx 1 root root   840846 Jan 14 23:56 disks.export.20160114.bad.hash*
-rwxrwxrwx 1 root root  1081855 Jan 16 00:00 disks.export.20160115.bad.hash*
-rwxrwxrwx 1 root root   252745 Jan 16 18:19 disks.export.20160116.bad.hash*[/color]
-rwxrwxrwx 1 root root       12 Jan 19 04:42 disks.ini*
-rwxrwxrwx 1 root root      348 Jan 18 11:03 dynamix.file.integrity.cfg*
-rwxrwxrwx 1 root root    21184 Jan 18 15:44 dynamix.file.integrity.txz*
-rwxrwxrwx 1 root root      131 Jan 18 11:03 integrity-check.cron*
-rwxrwxrwx 1 root root      648 Jan 18 11:03 integrity-check.sh*

 

Those log files in red are created when the utility finds a file which has its checksum changed but not its file modification time. This really should be an exception, but your log files are rather big. You may want to check these logs and see which disks/files are reported in here. I know this isn't the answer to the script issue, but it is worth checking the logs. Do you have notifications and/or error logging enabled?

 

 

Share this post


Link to post

@karateo

 

Another question: do you have excluded folders defined? If yes, are there special characters in their names?

 

Share this post


Link to post

I have all the folders excluded - excluded the Photos folder.

So I have only included the Photos folder which is at /mnt/disk3/Photos ONLY.

 

Here is a part of my syslog

Jan 18 03:47:42 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk1/Downloads/Completed/Home.2015.1080p.BluRay.x264-SECTOR7/home.2015.1080p.bluray.x264-sector7.r81 was modified

Jan 18 03:47:46 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk1/Downloads/Completed/Home.2015.1080p.BluRay.x264-SECTOR7/home.2015.1080p.bluray.x264-sector7.r05 was modified

Jan 18 03:47:49 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk1/Downloads/Completed/Home.2015.1080p.BluRay.x264-SECTOR7/home.2015.1080p.bluray.x264-sector7.r80 was modified

Jan 18 03:47:50 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk1/Downloads/Completed/Home.2015.1080p.BluRay.x264-SECTOR7/home.2015.1080p.bluray.x264-sector7.r18 was modified

Jan 18 03:47:53 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk1/Downloads/Completed/Home.2015.1080p.BluRay.x264-SECTOR7/home.2015.1080p.bluray.x264-sector7.r36 was modified

Jan 18 03:47:54 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk1/Downloads/Completed/Home.2015.1080p.BluRay.x264-SECTOR7/home.2015.1080p.bluray.x264-sector7.r61 was modified

Jan 18 03:57:41 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk1/Downloads/Completed/Hellboy.2004.DC.1080p.BluRay.x264.DTS-FGT/Hellboy.2004.DC.1080p.BluRay.x264.DTS-FGT.mkv was modified

Jan 18 03:57:46 TOWER bunker: verified 391 files from /mnt/disk1 with mask *. Found: 89 mismatches (updated), 0 corruptions. Duration: 01:27:45 Average speed: 21.6 MB/s

Jan 18 04:52:09 TOWER sSMTP[30877]: Creating SSL connection to host

Jan 18 04:52:09 TOWER sSMTP[30877]: SSL connection using ECDHE-RSA-AES128-GCM-SHA256

Jan 18 04:52:13 TOWER sSMTP[30877]: Sent mail for root@gmail.com (221 2.0.0 closing connection f205sm13599908wme.4 - gsmtp) uid=0 username=root outbytes=530

Jan 18 05:22:31 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.none.log.2 was modified

Jan 18 05:22:31 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.none.log.1 was modified

Jan 18 05:22:31 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.system.log.5 was modified

Jan 18 05:22:31 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.system.log.2 was modified

Jan 18 05:22:32 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.system.log.1 was modified

Jan 18 05:22:32 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.imdb.log.5 is corrupted

Jan 18 05:22:32 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.imdb.log.4 is corrupted

Jan 18 05:22:33 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.imdb.log.3 is corrupted

Jan 18 05:22:33 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.imdb.log.2 is corrupted

Jan 18 05:22:33 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.imdb.log.1 was modified

Jan 18 05:22:33 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.opensubtitles.log.5 is corrupted

Jan 18 05:22:33 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.opensubtitles.log.4 is corrupted

Jan 18 05:22:33 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.opensubtitles.log.3 is corrupted

Jan 18 05:22:33 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.opensubtitles.log.2 is corrupted

Jan 18 05:22:33 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.opensubtitles.log.1 was modified

Jan 18 05:22:33 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.themoviedb.log.5 is corrupted

Jan 18 05:22:33 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.themoviedb.log.4 is corrupted

Jan 18 05:22:33 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.themoviedb.log.3 is corrupted

Jan 18 05:22:33 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.themoviedb.log.2 is corrupted

Jan 18 05:22:33 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.themoviedb.log.1 was modified

Jan 18 05:22:34 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.localmedia.log.5 was modified

Jan 18 05:22:34 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.localmedia.log.4 was modified

Jan 18 05:22:34 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.localmedia.log.3 was modified

Jan 18 05:22:34 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.localmedia.log.2 was modified

Jan 18 05:22:34 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/Plex DLNA Server.log.1 was modified

Jan 18 05:22:35 TOWER bunker: error: BLAKE2 hash key mismatch, /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log.5 is corrupted

Jan 18 05:22:36 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log.4 was modified

Jan 18 05:22:38 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db-wal was modified

Jan 18 05:22:38 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db-shm was modified

Jan 18 05:22:39 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk3/appdata/plex/Library/Application Support/Plex Media Server/plexmediaserver.pid was modified

Jan 18 05:34:15 TOWER bunker: verified 69649 files from /mnt/disk3 with mask *. Found: 17 mismatches (updated), 13 corruptions. Duration: 03:04:14 Average speed: 49.6 MB/s

Jan 18 11:02:46 TOWER emhttp: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog

 

I don't know why it scans all the folders.

I ve run the plugin checking all disks but the excluded should be excluded!

 

BLAKE2

Excluded folders:

Documentaries , Downloads , DropboxEfi , DropboxTeo , Handbrake , Movies , Movies 3D , MoviesCouchpotato , Personal Documents , TV Series , Temp , VM ISOs , Virtualfiles , Websync , appdata , docker

Select/Unselect all Disk

1 Disk

Share this post


Link to post

Your situation is really a puzzle to me. I can't replicate the issue.

 

For me excluding disks and/or folders work as expected (also when using names with spaces).

 

May be a clean up and re-install will do the magic..

 

Here are the steps I propose:

 

1. Settings page: Uncheck all excluded folders (-> all folders are included)

2. Control page: check all disks and perform a "Remove" command

3. Uninstall the plugin (see plugin manager)

4. Delete or rename manually the folder /config/plugins/dynamix.file.integrity on your flash device

5. Reboot the system (optional but recommended)

6. Install the plugin using the URL:

https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg

7. Setttings page: configure your preferences, check all folders you want to exclude

8. Control page: perform a "Build" command (this should do only the folder 'Photos')

9. Control page: perform a "Export" command for all disks

10. Check the content of the export files and let me know.

 

Share this post


Link to post

@John_M

 

You can use the scheduled verification tasks to do automatic checking of the files, this uses the extended attributes and is always up to date.

 

Yes, thank you. I have verification scheduled but I stumbled upon the phenomenon I described while I was investigating the appearance of disks.export.<date>.bad.hash files as mentioned above and experienced by karateo.

 

The cause of my *.bad.hash files is the presence of Mac OS X related .AppleDB folders and .DS_Store files whose contents change without updating the timestamp. Is there a way to exclude them from the hashing process short of telling my Macs not to write them to network volumes?

 

I also have a problem with Mac OS X folder icons, which are stored in files with the name "Icon\r" - yes, that's a five character file name ending in a carriage return (ASCII 0x0D) character! Volume icons are not a problem because they are stored in .VolumeIcon.icns files, which is a manageable name and the contents are static. Reference: http://superuser.com/questions/298785/icon-file-on-os-x-desktop

 

Edit: I thought the Excluded Folders option might help me exclude the .AppleDB folders, which are present only in the root of each of my shares. I'd be looking to exclude /mnt/user/Public/.AppleDB and /mnt/user/Private/.AppleDB but the button just reads "None" and I can do nothing with it.

 

The .DS_Store files are more problematic because every folder could potentially contain one.

 

Share this post


Link to post

@John_M

 

You can use the scheduled verification tasks to do automatic checking of the files, this uses the extended attributes and is always up to date.

 

Yes, thank you. I have verification scheduled but I stumbled upon the phenomenon I described while I was investigating the appearance of disks.export.<date>.bad.hash files as mentioned above and experienced by karateo.

 

The cause of my *.bad.hash files is the presence of Mac OS X related .AppleDB folders and .DS_Store files whose contents change without updating the timestamp. Is there a way to exclude them from the hashing process short of telling my Macs not to write them to network volumes?

 

I also have a problem with Mac OS X folder icons, which are stored in files with the name "Icon\r" - yes, that's a five character file name ending in a carriage return (ASCII 0x0D) character! Volume icons are not a problem because they are stored in .VolumeIcon.icns files, which is a manageable name and the contents are static. Reference: http://superuser.com/questions/298785/icon-file-on-os-x-desktop

 

It is possible to add specific exclude rules and prevent checksum generation for these files. I can look into adding additional rules.

 

I am not an Apple user myself, so need a bit of guidance to understand what and where are folders and files created

 

1. .AppleDB is a single folder in the root of the disk, i.e. /mnt/diskX/.AppleDB or can appear randomly in sub-folders too?

2. .DS_store files appear in every folder or in one specific location?

3. Icon\r files (yaks why a CR in the name?) also appear in every folder or in one specific location?

4. Any other "special" cases?

5. Any settings on the Apple side which may change how and where things are created?

6. Is there no way to inhibit the creation of these folders/files (sounds as the first step to do)

 

Share this post


Link to post

Ok, after rereading the updated remarks of John_M, I made an exclusion for the Apple specifics: .AppleDB and .DS_Store.

 

For the moment these exclusions are hardcoded, I may change that to an option in the future.

 

There is a new version 2016.01.19 available, try it out and  let me know your results.

 

Share this post


Link to post

Your situation is really a puzzle to me. I can't replicate the issue.

 

For me excluding disks and/or folders work as expected (also when using names with spaces).

 

May be a clean up and re-install will do the magic..

 

Here are the steps I propose:

 

1. Settings page: Uncheck all excluded folders (-> all folders are included)

2. Control page: check all disks and perform a "Remove" command

3. Uninstall the plugin (see plugin manager)

4. Delete or rename manually the folder /config/plugins/dynamix.file.integrity on your flash device

5. Reboot the system (optional but recommended)

6. Install the plugin using the URL:

https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg

7. Setttings page: configure your preferences, check all folders you want to exclude

8. Control page: perform a "Build" command (this should do only the folder 'Photos')

9. Control page: perform a "Export" command for all disks

10. Check the content of the export files and let me know.

 

It looks promising!

I hope I don't receive any email tomorrow!

Disk 1

100%

Build Finished - added 0 files. Duration: 00:00:16

Disk 2

100%

Build Finished - added 0 files. Duration: 00:00:08

Disk 3

0.1%

Build Currently processing file 79 of 59911 [ETA: 07:24:52]

Share this post


Link to post

That looks good, I presume your photos folder is only on disk3.

 

Share this post


Link to post

That looks good, I presume your photos folder is only on disk3.

 

that's right

Disk 3

7.2%

Build Currently processing file 5878 of 59911 [ETA: 01:59:37]

 

I will let you know tomorrow. Thanks for your help.

 

By the way, which method do you use/suggest to fix any corruptions found?

 

Share this post


Link to post

By the way, which method do you use/suggest to fix any corruptions found?

 

The most straightforward approach is to keep a copy of your most critical files on a different system (or in the cloud). This will allow you to compare and restore if needed.

 

Share this post


Link to post

It is possible to add specific exclude rules and prevent checksum generation for these files. I can look into adding additional rules.

 

I am not an Apple user myself, so need a bit of guidance to understand what and where are folders and files created

 

1. .AppleDB is a single folder in the root of the disk, i.e. /mnt/diskX/.AppleDB or can appear randomly in sub-folders too?

2. .DS_store files appear in every folder or in one specific location?

3. Icon\r files (yaks why a CR in the name?) also appear in every folder or in one specific location?

4. Any other "special" cases?

5. Any settings on the Apple side which may change how and where things are created?

6. Is there no way to inhibit the creation of these folders/files (sounds as the first step to do)

 

1. The .AppleDB folder only appears in the root, as far as I've been able to discover.

2. .DS_Store files can appear in any (and potentially in every) folder.

3. An Icon\r file will appear in any folder to which the user has given a custom icon, otherwise they display as the default blue folder icon.

4. These are the only ones I've found. There are others documented here but some are very much pre OS X legacy items.

5. Only in as much as they can be disabled (see 6.)

6. Yes, there is. OS X can be configured not to write any of the dot-files to network storage but then you lose functionality. For example, one of the things stored in the .DS_Store file within a particular folder is information about the current view of that folder within the Finder - scroll position, icon size, icon position, etc. so that when you open the Finder window (equivalent to the Windows Explorer) you're presented with the same view as you had when you last closed it.

 

Edit: Ah! I'll give the updated version a try. Thank you.

 

Share this post


Link to post

Your situation is really a puzzle to me. I can't replicate the issue.

 

For me excluding disks and/or folders work as expected (also when using names with spaces).

 

May be a clean up and re-install will do the magic..

 

Here are the steps I propose:

 

1. Settings page: Uncheck all excluded folders (-> all folders are included)

2. Control page: check all disks and perform a "Remove" command

3. Uninstall the plugin (see plugin manager)

4. Delete or rename manually the folder /config/plugins/dynamix.file.integrity on your flash device

5. Reboot the system (optional but recommended)

6. Install the plugin using the URL:

https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg

7. Setttings page: configure your preferences, check all folders you want to exclude

8. Control page: perform a "Build" command (this should do only the folder 'Photos')

9. Control page: perform a "Export" command for all disks

10. Check the content of the export files and let me know.

 

Disk 1

100%

Export Finished - exported 3 files, skipped 0 files. Duration: 00:00:20

Disk 2

100%

Export Finished - exported 0 files, skipped 0 files. Duration: 00:00:08

Disk 3

100%

Export Finished - exported 59911 files, skipped 550 files. Duration: 00:00:38

 

disk1.export.hash (which should be empty)

235d93e628118ee91335b7c66ae7bc64e9034f953d8320c81c65055cfc99343b6bc3364c7be681387efb6c443629e02b062467c8aadf21b69cb2ca3a9a91ca78 */mnt/disk1/appdata/sonarr/logs/nzbdrone.txt

b33495cd90f3b318f9a8b2a1320f04fac870133f84464bb3ece9284f37ed71a66325e355f82bcf5d0556e8d519c5999a6b87ead9536dcbcc88cf8924350aae2b */mnt/disk1/appdata/sonarr/nzbdrone.db

74bef5b192cbe58d24807284f918b186fd501ffb813e5b20d153f4cae0f0ddd9788282320e426f3f12637707b94ef4b073013cc112f38eab1beda023b374abf4 */mnt/disk1/Downloads/Completed/33???.The.33.2015.HD1080P.X264.AAC.English.CHS-ENG.Mp4Ba/33???.The.33.2015.HD1080P.X264.AAC.English.CHS-ENG.Mp4Ba.mp4

 

disk2.export.hash missing which is expected

 

disk3.export.hash has 59910 which is expected

 

Another problem..

in syslog I see a lot of errors that shouldn't be there because

Excluded folders: Documentaries , Downloads , DropboxEfi , DropboxTeo , .....

BCIMROP.jpg

a part of it

Jan 20 10:50:38 TOWER bunker: exported 3 files from /mnt/disk1 with mask *. Duration: 00:00:20

Jan 20 10:50:56 TOWER bunker: error: no export of file: /mnt/disk3/DropboxTeo/Apps/SMSBackupRestore/sms.xsl

Jan 20 10:50:56 TOWER bunker: error: no export of file: /mnt/disk3/DropboxTeo/Apps/SMSBackupRestore/sms-2012-05-22 08-32-23.xml

Jan 20 10:50:56 TOWER bunker: error: no export of file: /mnt/disk3/DropboxTeo/Apps/tinyCam Monitor PRO/cameras.md5

Jan 20 10:50:56 TOWER bunker: error: no export of file: /mnt/disk3/DropboxTeo/Apps/tinyCam Monitor PRO/cameras.xml

Jan 20 10:50:56 TOWER bunker: error: no export of file: /mnt/disk3/DropboxTeo/Apps/ClockworkMod Carbon/Nexus 5 - 353490061763757/com.quizdom.json

 

You have already answered my question but I need to mention again because it's a drawback.

Now we can check only the excluded folders.

The problem is that every time someone creates a new folder, he has to remember to exclude it as well.

I believe it is more logical through GUI to check only the included folders.

I don't have any coding experience but I believe it shouldn't be that difficult.

 

You could excluded all except the folders the user wants to be included.

Share this post


Link to post

I think the files on disk1 were added because of the running protection scheme and before you removed the plugin. You can run a remove on disk1 and check that no files are added from this point on.

 

Bit puzzled about disk3, could be the same situation as disk1. Observe here if new files are reported (see syslog)

 

It isn't that easy to change from exclude to include, because the programs called in the background work only with an exclude list. I expect your situation is more the exception than the rule, maybe inconvenient but still workable.

 

Share this post


Link to post

What does this error mean after doing an export?

 

Jan 20 22:38:54 Tower bunker: error: no export of file: /mnt/disk1/Games/...

 

The build and export were both showing as up-to-date with green checkmarks both before and after running the export.

 

Also, I then tried running the build again because I thought maybe that file didn't have a hash recorded, but then it showed about 2100 files that needed to be updated. If thats the case, then why did the table show a green checkmark that the build was up to date? Executing the rebuild and re-export now shows 0 files skipped.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.