Falcosc

I thought about making the export files hash specific by myself, but after discovering all the places I did not dare to change the export file names, @bonienl did you forget some of them? I remember there were multiple places where these filenames are used. We have Export File creation, we have export file checks, export file status. And I remember we have an additional hash file definition in the exportrotate script and another one in the watcher script. At least for the export status, you find the error here: https://github.com/bergware/dynamix/blob/master/source/file-integrity/include/ProgressInfo.php#L57

You need to remove your SHA256 export file. In my last post, I told that you can keep your SHA hashes in your extended file attributes, but cannot use the "Check Export" button if you did configure blake3 and have SHA hashes in your export folder. If you want to keep your SHA exports, just rename them to avoid getting picked by the "Check Export" button.

Each hash method gets it's own extended file attribute, so build is enough if you want to keep your blake2 keys. But because the last scan date with file stats are shared, and we don't have a button to just clean the last scan date it would be better if you remove all attributes. Just wanted to make clear that you don't have too. On the other hand, shared scan date isn't tested very well because hash method changes is an uncommon usecase.

I would suggest adding all the small improvements to the change log, in case somebody doesn't follow this thread and want to know if a specific error was fixed: https://github.com/bergware/dynamix/pull/53/files

Yes, but it should be done in 0.4s real time and 3.5s user time if it scaled better. (user time is the sum of all thread times) Because single threaded was done in 3s user time. So, I thought maybe it is not a scaling issue, maybe it is a memory speed issue because for 60gb/s hashing rate you may need memory speed of 80-160gb/s which your platform doesn't have. But at these insane speeds, many things could be limiting the scaleability of the multithreaded execution. Your CPU is just too fast for blake3

I don't know how good hash calculations should scale, but spending more than twice the CPU time on multithreaded blake3 (8s multi vs 3s single) sounds a bit too wasteful. I mean, it is good that they at least give this possibility in case somebody would need the 23gb/s throughput, even if less efficient. But this just confirmed my feeling that we should keep it single threaded to improve efficiency. Maybe memory access time counts as user time? Because perfect scaling should result into 60gb/s which would be already the limit of your memory performance. We need more memory channels and DDR5 🤓 On huge files it would be most efficient to have single threaded but with mmap because on your 23gb file run no-mmap you spend additional 900ms sys time for all the little file access calls. I don't want b3sum to pop up in the memory usage graph as a big resource hugger to avoid wrong perceptions of this wonderful tool, so I would sacrifice the sys time saving for having an invisible memory footprint with no-mmap.

This ridiculous speed is the reason why the addition of blake3 is more about reducing CPU stress. But I managed to get a 2-3 Times speed improvement by reduce the number of expensive CLIs calls per file in the build, check and verification process. I did avoid writing numbers into the change log because the script speed improvement is only related to file count. With an average of 2,8mb/file (180.000 files at 500GB) I got from 65mb/s to 170mb/s on any hash method.

Remember, with AVX-512 you can calculate 8 64bit integer in one go. So basically, 8 times speed on a single thread. That's just cheating if correctly implemented

It depends on your CPU. Without AVX it uses SSE4.1 and is only 2 times faster than blake2. But with AVX it is just crazy: 1 thread 6gb/s. Because having more than 10-60gb/s speed per disk is really uncommon even in the far future of Unraid I did force b3sum to run in single thread mode and I did force it to not use mmap. mmap is an expensive syscall to put your file into userspace memory, users did already complain that having mmap does reduce the performance on small files too much because this syscall takes much longer then hashing the whole file. For that reason, the recent version of the b3sum CLI does skip mmap on files smaller than 16kb. Multithreading has overhead, too. So disabling multithreading does give even more small file performance. And if you use single threading, you don't benefit from mmap, you don't need to put your file into memory because you sequentially read it anyway. If you have multiple disks in your array, you don't need multithreading at all. So for the Unraid use case, it doesn't make sense to even make the parameters of b3sum configurable in the UI. I have a 2-Thread G4400T in temporary use, which is already faster than my array on blake3 despite being nearly the worst-case CPU for Unraid. On some CPUs with Intel SHA Extension, blake3 is even 2 times faster than Hardware accelerated if running single threaded https://github.com/xoofx/Blake3.NET#results-with-sha-cpu-extensions Running multithreaded is just unnecessary computing overhead, nobody has storage to benefit from 10 times more speed than Hardware accelerated SHA https://news.ycombinator.com/item?id=22236507 But you can just test it by yourself wget https://github.com/BLAKE3-team/BLAKE3/releases/download/1.0.0/b3sum_linux_x64_bin time b3sum_linux_x64_bin largefile time b3sum_linux_x64_bin --num-threads=1 --no-mmap largefile

Corruptions are not reported for non-existing files during check. Not before and not after this change. But to use the check button, you need to export the hashes first. - inline help for disk table and commands more accessible If you did delete the wrong files because you needed to guess them based on truncated file names, then it is related to usage of b2sum rarely leads to empty hashes. - fixed file name truncation if key is broken

The release got delayed a bit because the maintainer is busy with Unraid 6.10 preparations. Does anyone have questions about the change log before the release? - added blake3 hash support for 2-6 times more hash rate than blake2 (reduce CPU load up to 4 times at same data rate) - improve build, verify and check speed for all hash methods on small files - fixed full file system find commands for build status checks while watching control page - fixed starting multiple build status checks while watching control page - added monitor background verify processes and all manual bunker script executions - fixed rare process status detection bug "Operation aborted" - fixed file name truncation if key is broken - inline help for disk table and commands more accessible - fixed multi-language support for buttons - added watch error and warning messages of running or finished progress - added Disk status icon for running build status check

It is ok to have reasons why ASPM should be disabled. But it is an issue that there is no way to enable ASPM on unraid. How can I help to identify which part of Unraid is preventing ASPM to be enabled?

I did already test all of these 2 weeks ago: Step 7 and Step 8 Both commands did execute without error message and value was set. But both did not affect ASPM in Unraid. @mgutt Which part of your findings are not tested in the bugreport? At which configuration are you interested in? I'm not sure about CONFIG_PCIEASPM because ASPM is working in Unraid which means this can't be false. Only devices connected to my chipset don't get ASPM in Unraid. Ubunuto however did enable ASPM for all devices. @limetech I can do any test you like to have because I didn't clean up my test bench yet and still evaluating. New Unraid Builds, other Ubuntu Versions, Old Ubuntu with manual Kernel update. Just tell me what you need.

Good that you did report this before my 10 bug fixes and improvement got through test phase and pushed to the public. If you have told us this one day later, I would have thought that I broke it. You can ignore these, they do just tell you that you did not build a hash sum for your new or changed files. Your disk should get a yellow circle in the files integrity page, if not, then you may need to wait for the bug fixes. But don't stay to long of the file integrity status page, it has a bug which schedules an infinit amount of full file system indexing tasks. These are the fixes and improvements which are coming soon: add blake3 support https://github.com/bergware/dynamix/pull/42 speed up bash by reduce expensive calls https://github.com/bergware/dynamix/pull/43 use check list to speed up check button https://github.com/bergware/dynamix/pull/44 avoid indexing of filesystem while watching progress https://github.com/bergware/dynamix/pull/45 avoid terminal wrap during progress https://github.com/bergware/dynamix/pull/46 monitor verifications and all manual bunker calls https://github.com/bergware/dynamix/pull/47 fix truncated file names if key is broken https://github.com/bergware/dynamix/pull/48 fix inline help for disk table and commands https://github.com/bergware/dynamix/pull/49 show errors of progress https://github.com/bergware/dynamix/pull/50 Add Disk status refresh icon https://github.com/bergware/dynamix/pull/52 If you wonder why so much is in work, well, I did start to use the plugin and was unsatisfied with the performance and put 50 hours of developing and testing into it.

I did add Disk status check icons to fix the confusing presentation issue of our disk status. Maybe the Disk Status was broken, there was a regex which was unable to delete the build status if build was not the first entry. It scans only one disk at a time and take a while if you have many files. Disk 3 gets skipped because it has work to do. I thought about parallel execution, but rejected the Idea because it would hurt the array performance, so we keep it sequential. Add Disk status refresh icon https://github.com/bergware/dynamix/pull/52

I just found your bug. I wonder how anybody with more than 3 disks can actually watch the progress page without crashing the array performance One issue was that the currently running disk was disrupted by a full file system indexing in progress. I fixed that at the beginning of the week, I didn't have any more issues because disks in idle don't get checked every 2 minutes anyway. But today I had 3 disks online, and I really suffered from bad array performance. Turns out that the script with the find command doesn't check if it is already running. Which means if the total number of disks cannot finish a full file system find command in sequential order (one disk at a time) in 2 minutes, it will get a 2nd execution. After some time, you stack many find commands over each other that the whole array nearly stops working. Here is the fix https://github.com/bergware/dynamix/pull/45/commits/ebe6e770d1c46064064f9b3f4434dbc204b161e5

Not a bug, more an issue about presentation. Let me describe what is happening it shows disk1 as a circle, not green checkmark, even though it has just finished the build, and is up-to-date Problem is "just finished" the search for untagged files find command is only executed every 2 minutes this find is very expensive, so the 2 minutes repeat is already to much for my taste it is only executed every 2 minutes if you are looking at the page If you look at the page everything will go slower because the search for new files does slow down spinning disks, I did fix that problem: avoid indexing of filesystem while watching progress https://github.com/bergware/dynamix/pull/45 it shows disks 4, 5, 9 and 10 with a green checkmark, even though the builds are clearly still running and aren't finished maybe the find command did not finish because your build process is slowing the find command down to much it shows disks 7, 12, 17, 18, 19, 22, 23, 24, 26, 27 and 28 with a green checkmark, even though the build process has never been run on these disk... Depending on how many files you have there, the check is maybe still running Does this feature need to be improved?

This did bother me, too. You see it on the circle, but a progress bar is much better. So I added it: monitor verifications and all manual bunker calls https://github.com/bergware/dynamix/pull/47 At some point we lost our documentation, it was still available but kind of hidden. I add the help text toggle to the disc table and button bar to give new users a more easy start: fix inline help for disk table and commands https://github.com/bergware/dynamix/pull/49 Which process did not appear at dynamix.file.integrity/logs? For now, I did add a small non-downloadable watch errors popup which can be accessed during progress but only appears if there are at least 1 skip/warning/error: show errors of progress: https://github.com/bergware/dynamix/pull/50

I use 500GB test data with 180.000 files and I run through that about 20 times already to get all these improvements tested and benchmarked. I got 2 builds with 1 missing key each. And it was not the same file. So b2sum did just not return a key and I don't know the condition at which this happened. I don't terminate my test runs, so I was really unsure what is happening. But the user responses related to truncated files did prove that something is unstable. Did you notice that all truncated file reports did use blake2? Maybe just a coincidence, but maybe a clue.

I don't know why b2sum does not return a key in very rare cases because I did not log stderr and sigcode of all 800.000 b2sum calls. But it happened twice on my host and because b3sum is still without any calculation failure after 1.200.000 calls on 200.000 different files, I am confident that our b2sum binary is a bit unstable and may need an update. Testing such rare issue is complicated because my 1.200.000 hashes needed to be validated as well. If anybody want to still use the old blake2, we may have to check if the old b2sum cli got any kind of fixes. But blake3 is better than blake2 in performance anyway we don't take care of b2sum and if you need tools without blake3 support you should consider sha256 with hardware acceleration https://en.wikipedia.org/wiki/Intel_SHA_extensions It's cool that software calculated blake3 is even faster than current Intel SHA extensions That shows just how efficient blake3 really is.

I thought about it, but it is disruptive. Each hash has it's own field User.md5 User.blake2 and so on So for hashes we already on track But the following fields are shared: User.filesize (file size while hash creation) User.filedate (fie data during hash creation) User.scandate (last scan date) So if you change the hash method the plugin will get confused because the shared fields doesn't match to the hash creation. To solve it we would have to use this structure: User.blake2="hash filesize filedate scandate" I have tried this structure, it does improve creation time a bit because you need 3 syscalls less. But this change would invalidate existing attributes. I will consider this for a later release, for example by adding a migration script to avoid slow and complex bash script handling of backwards compatibility. For now, it is better to only use one hash method. By the way, b2sum is a bit unstable. It does return no key in one of 400.000 calls on my host. That's the reason why so many people have issues with truncated file names. If key is missing, the file name gets truncated by the length of the missing key. blake3 will fix this because it comes with a new CLI b3sum and I did already called b3sum 1.200.000 times without a single crash.

Community comes to the rescue! add blake3 support https://github.com/bergware/dynamix/pull/42 speed up bash by reduce expensive calls https://github.com/bergware/dynamix/pull/43 use check list to speed up check button https://github.com/bergware/dynamix/pull/44 avoid indexing of filesystem while watching progress https://github.com/bergware/dynamix/pull/45 avoid terminal wrap during progress https://github.com/bergware/dynamix/pull/46 monitor verifications and all manual bunker calls https://github.com/bergware/dynamix/pull/47 fix truncated file names if key is broken https://github.com/bergware/dynamix/pull/48 This weekend some or all of my pull requests get tested and some time later released I did these updates to speed up my process because the old tool can't handle small files: Old version: Average speed: 62 MB/s new Version: Average speed: 167 MB/s

I found a possible issue, something can get wrong during parsing the response from getting the extended file attributes. I can fix it if anybody of you is still around, you just need to execute the following commands to create something which can be manually checked: path=/mnt/disk1 hash=blake2 find "$path" -type f -name "*" -exec getfattr -n user.$hash --absolute-names "{}" 2>/dev/null 1>/tmp/debugfiles_raw.txt + sed -n "/^$/d;s/^# file: //;h;n;s/^user.$hash=\"//;s/\"$//;G;s/\n/ \*/;p" /tmp/debugfiles_raw.txt 1>/tmp/debugfiles_sed.txt In /tmp/debugfiles_sed.txt you will find your truncated line or an empty line or a line where the key is there but no file name after the key. I would like to see your result to fix it. I was unable to construct a broken file to get this to fail. Maybe there is an unexpected response from getfattr on stdout.

I can fix this issue. At which action did this happen?