Thx And Bye Posted August 23, 2021 Share Posted August 23, 2021 9 hours ago, bonienl said: I have made an update which includes the hashing method in the name of the export file. After removing the old files and creating a new export it shows "related export file is not present": But the files are present on the flash: Quote Link to comment
Falcosc Posted August 23, 2021 Share Posted August 23, 2021 I thought about making the export files hash specific by myself, but after discovering all the places I did not dare to change the export file names, @bonienl did you forget some of them? I remember there were multiple places where these filenames are used. We have Export File creation, we have export file checks, export file status. And I remember we have an additional hash file definition in the exportrotate script and another one in the watcher script. At least for the export status, you find the error here: https://github.com/bergware/dynamix/blob/master/source/file-integrity/include/ProgressInfo.php#L57 Quote Link to comment
bastl Posted August 24, 2021 Share Posted August 24, 2021 There is another update showing for me right now. current version: 2021.08.23a new version: 2021.08.24 pressing the update button gives me the following error. Quote Link to comment
bonienl Posted August 24, 2021 Author Share Posted August 24, 2021 I made an update and version 2021.08.24 is available. This version addresses the regressions introduced by the different export file naming. Thx everyone for spotting 1 Quote Link to comment
bonienl Posted August 24, 2021 Author Share Posted August 24, 2021 (edited) 11 minutes ago, bastl said: pressing the update button gives me the following error. You caught me while upgrading the plugin. Didn't expect that. There is now version 24a to rectify the installation error. Hmm, something is not going right ... checking! Edited August 24, 2021 by bonienl 1 Quote Link to comment
bastl Posted August 24, 2021 Share Posted August 24, 2021 @bonienl 24a gives me the same error Quote Link to comment
bastl Posted August 24, 2021 Share Posted August 24, 2021 @bonienl 2021.08.24b fixed it. Thanks! Quote Link to comment
bonienl Posted August 24, 2021 Author Share Posted August 24, 2021 I pulled the update, because it isn't working properly. Please wait for a new update! Quote Link to comment
bonienl Posted August 24, 2021 Author Share Posted August 24, 2021 (edited) Please install latest version 24e. Thx Edited August 24, 2021 by bonienl 1 Quote Link to comment
Vr2Io Posted August 24, 2021 Share Posted August 24, 2021 (edited) 5 hours ago, bonienl said: Please install latest version 24e. Thx Thanks for the update. There are a minor bug, I have apply few "exclude folder" and "exclude file", during export, it always prompt 2 file skip in syslog. Finished - exported 148699 files, skipped 2 files. Duration: 00:00:45 Below are the two file attribute getfattr -d *.md5 # file: HASSIO.md5 user.DOSATTRIB=0sAAAEAAQAAAARAAAAIAAAAAAAAAAAAAAAQDLrLcJl1wE= # file: Hassio snapshots.md5 user.DOSATTRIB=0sAAAEAAQAAAARAAAAIAAAAAAAAAAAAAAA649GN8Jl1wE= I have try rename HASSIO.md5 to HASSIO.md to verify does because .md5 cause the problem, but it still show that two file skip and generate error log. Last, I remove the remain attribute in HASIO.md5, then FIP show skip file be 1 in the result. So, it seems extra attribute in file will wrongly count as skip. Finished - exported 148699 files, skipped 1 file. Duration: 00:00:45 Edited August 24, 2021 by Vr2Io Quote Link to comment
bonienl Posted August 24, 2021 Author Share Posted August 24, 2021 (edited) If you have updated the list of excluded folders or files, you need to run the "Clear" command on your disks to remove an existing hash from those excluded items. This will prevent skipped files. Edited August 24, 2021 by bonienl Quote Link to comment
Vr2Io Posted August 24, 2021 Share Posted August 24, 2021 (edited) 8 minutes ago, bonienl said: If you have updated the list of excluded folders or files, you need to run the "Clean" command on your disks to remove an existing hash. This will prevent skipped files. I have execute "clear", but those attribute not generate by FIP, so it won't clear ( also expect ). The name was "user.DOSATTRIB" Thanks Edited August 24, 2021 by Vr2Io Quote Link to comment
bastl Posted September 1, 2021 Share Posted September 1, 2021 Short report for something weird happened to me. Planned verify task for 2 disks started this morning, 1 disk finished and reported a corruption on disk 2. Verify task on disk 1 still running. The coruppted file in my case was the nextcloud.log file getting flooded. I cleared the logfile, verify on disk 1 still running. I selected disk 2 and started a export task and the following happened in the ui The ui shows as it looks like the full list of files beeing scanned on disk 1. Refreshing the page doesn't help. The counter for processed files is still counting up and constant read speeds on disk 1 showing that the process looks like it's still working. Only the page is messed up. Quote Link to comment
Falcosc Posted September 1, 2021 Share Posted September 1, 2021 (edited) 26 minutes ago, bastl said: Short report for something weird happened to me. Planned verify task for 2 disks started this morning, 1 disk finished and reported a corruption on disk 2. Verify task on disk 1 still running. The coruppted file in my case was the nextcloud.log file getting flooded. I cleared the logfile, verify on disk 1 still running. I selected disk 2 and started a export task and the following happened in the ui The ui shows as it looks like the full list of files beeing scanned on disk 1. Refreshing the page doesn't help. The counter for processed files is still counting up and constant read speeds on disk 1 showing that the process looks like it's still working. Only the page is messed up. Please share your /tmp/*disk2* files and /var/tmp/*disk2* files. I did add errors to them, and it looks like there is at least on layout function which does not ignore this new content. Is a disk2 process running? Because you should not see any disk2 stuff if no disk 2 process is running. After a process is finished, reload will remove your disk2 progress info. Edited September 1, 2021 by Falcosc Quote Link to comment
bastl Posted September 1, 2021 Share Posted September 1, 2021 @Falcosc I have no /tmp/disk2* files in that folder and I can't really share the /var/tmp/disk2* filelist with you. I have a lot of personal and work content on that disk, some named with my real name or the name of the company I work for. I can try to remove the personal stuff if it helps. 14 minutes ago, Falcosc said: Is a disk2 process running? I guess not. On the main tab I only see reads happen on disk one where the verify process is still active. Looks like export for disk2 is already finished. Timestamp for disk2 is short after the time a started the process for a new export. Quote Link to comment
Falcosc Posted September 1, 2021 Share Posted September 1, 2021 Yea, you can remove your personal stuff and save to see if the issue persists with your modified file Quote Link to comment
bastl Posted September 1, 2021 Share Posted September 1, 2021 @Falcosc Maybe it helps to note that I am from germany and some of the files include the german letters Ää Öö Üü and ß for example */mnt/disk2/company/tmp/Präsentation/industrie-medium.png also I have a folder where backups from my android phone are placed, some with the following naming included with an @ for example */mnt/disk2/nextcloud/user/files_versions/.SeedVaultAndroidBackup/1678426041580/kv/@pm@/Y2ibnQudnVXN1cn9tLZleQ.v1614280421 Quote Link to comment
Falcosc Posted September 1, 2021 Share Posted September 1, 2021 (edited) That is the reason why I would like to have the file. I test it once and didn't see any issue. Edit it as much as you like. In the best case, you can reproduce the issue with just normal characters and only 3-5 lines in your .end file Edited September 1, 2021 by Falcosc Quote Link to comment
bastl Posted September 1, 2021 Share Posted September 1, 2021 @Falcosc Verify on disk 1 is finished. The file integrity control panel page still shows the the files from disk2 like in the screenshot of my fist post. I have a lot of *.tmp.end files in /var/tmp and a disk2.tmp.end. Some are from august some from today. I renamed the disk2.tmp.end refreshed the page and disk2 export showed "canceled with error". Refreshed the page again. Error is gone. Started a new export of disk2. No errors. I'am kinda confused right now. Quote Link to comment
Falcosc Posted September 1, 2021 Share Posted September 1, 2021 It could be possible that we still need a broken disk2.tmp.end file in case the bug is not obvious. (I worry that the bug is not obvious because at least one test run with simulated corruptions didn't have this issue) Quote Link to comment
wildfire305 Posted September 8, 2021 Share Posted September 8, 2021 Wow, Blake 3 is very fast. How many threads is the plugin set to use? I have 24 threads, but my disks top out at 160 MB/s pulling one file. Which is what they did with MD5. Obviously the disks are the bottleneck. I can see it really rip through the tiny files though much faster than anything else. Quote Link to comment
Falcosc Posted September 8, 2021 Share Posted September 8, 2021 (edited) Actually you can't see how fast blake3 is because a single thread can handle 4-6GB/s which is way beyond spinning disk performance. So, we configured to run single threaded to be more efficient. The main goal off introducing blake3 was to reduce CPU load. I did benchmark memory and thread settings to get the best small file performance. The biggest issue was the amount of syscalls in the old script. But we had other issues which did slow down spinning disk IO as well. - added blake3 hash support for 2-6 times more hash rate than blake2 - reduced CPU load up to 4 times at same data rate when using blake3 - improve build, verify and check speed for all hash methods on small files - fixed stacking find commands avoiding clean shutdowns while watching control page - fixed starting multiple build status checks while watching control page - added monitor background verify processes and all manual bunker script executions - fixed rare process status detection bug "Operation aborted" - fixed file name truncation if key is broken - inline help for disk table and commands more accessible - fixed multi-language support for buttons - added watch error and warning messages of running or finished progress - added Disk status icon for running build status check Edited September 8, 2021 by Falcosc Quote Link to comment
wildfire305 Posted September 8, 2021 Share Posted September 8, 2021 Now that it has settled down, I'm seeing one thread per disk when it's running through my larger files. My disks are Very old 3TB that struggle to hit above 140MB for most of the disk surface. I like what you've done there with the plugin. Usually developers throw the whole system at every file and then it just sits there at 50 MB/s with 24 cores trying to shatter the platter and make dead the head. I always seem to have to reduce these things or tune them to my setup, but you guys did it right by default. If you're hashing an SSD more cores the better, but I'm rarely hashing my ssd's. Quote Link to comment
Falcosc Posted September 8, 2021 Share Posted September 8, 2021 (edited) It was a bit tricky to remove the bottleneck because it already was tuned to process sequentially with one thread per disk. The issue was around expensive binary calls with a lot of syscalls like file stat, get file attribute, unnecessary subshells for each file and so on. After fixing these I found all the other issues like unwanted parallel IO caused by monitoring. I did also experiment with offloading the really needed but still expensive syscalls into asynchronous subshells. But XFS isn't good enough to handle asynchronous file metadata actions without hurting spinning disk IO too much. If XFS metadata were completely cached, then you would reach nearly peak disk read performance on small files (did tests without Metadata, which did achieve this) But check export does skip all these things because it doesn't need to read or write file attributes if the exported hash matches. For that reason, this process is the fastest one. It is basically directly using the hash list cli argument and only starts custom error handling in case the cli response stream contains verification errors. Edited September 8, 2021 by Falcosc Quote Link to comment
Mr_4braham Posted October 9, 2021 Share Posted October 9, 2021 Would this plugin be of any good if I already have ECC RAM? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.