February 13, 20242 yr nevermind I visited your github page... now I understand... I hope .... these are environment variables right?
February 13, 20242 yr Author 3 hours ago, doogle said: What do you mean by post args section? I made a spot in the template for these:
February 13, 20242 yr Thanx. I'll have to remember to click the advanced view slider. Wish it would just stay on but I guess not. That worked great.. you da man!
February 21, 20242 yr Hello, I am playing around with this, big TY to TQ (and Squid for the accompanying script). When testing the docker ClamAV I view <docker logs ClamAV> and get the scan summary with the tested directory as explained early in the thread. I see the correct number of files scanned. I set the container to scan a different dir and I get the same correct results as well. Everything seems to be working great. My question: Why is it that during the scan, the disk(s) on my array do not spin up for this activity? I am 100% sure the files are on certain HDDs and not cached. This is causing my brain to blue screen.
March 4, 20242 yr I recently deployed @Masterwishx script and have it setup correctly. however when it starts at the designated time it scan the shares but it takes less than a minute and its down and I know it should take longer than that. Below are the scanning results, i am not sure where to get the actual log of clamav starting and giving me the info on what it scanned Any help to figure out why i am doing wrong would be appreciated. clamavmaplog.txt clamavtargets.txt Edited March 4, 20242 yr by ijuarez
March 5, 20242 yr On 3/4/2024 at 9:10 AM, ijuarez said: I recently deployed @Masterwishx script and have it setup correctly. however when it starts at the designated time it scan the shares but it takes less than a minute and its down and I know it should take longer than that. Below are the scanning results, i am not sure where to get the actual log of clamav starting and giving me the info on what it scanned Any help to figure out why i am doing wrong would be appreciated. clamavmaplog.txt 370 B · 1 download clamavtargets.txt 92 B · 0 downloads this is what I get in the clamav docker logs no sure how to fix that
March 7, 20242 yr Author On 3/5/2024 at 10:57 AM, ijuarez said: this is what I get in the clamav docker logs no sure how to fix that You're not using my container.
March 7, 20242 yr 17 hours ago, TQ said: You're not using my container. Yep works much better with the correct container thanks
March 24, 20242 yr How can I exclude files or folders? Have the Script a excluded file/Option? For false positive founds.
March 26, 20242 yr On 3/24/2024 at 9:25 AM, Revan335 said: How can I exclude files or folders? Have the Script a excluded file/Option? For false positive founds. https://linux.die.net/man/1/clamscan edit docker template advance view under options for add option and path form docker point of view path should be (by default unraid /mnt/user is dockers /scan) --exclude=REGEX, --exclude-dir=REGEX Don't scan file/directory names matching regular expression. These options can be used multiple times. add option --exclude-dir=/scan/(directory you want to not scan.) or single file... Edited March 26, 20242 yr by bmartino1
October 22, 20241 yr Log showing a lot of this >> ...root: ClamAV: Could not download icon https://raw.githubusercontent.com/tquizzle/Docker-xml/master/img/clamav.png Advice?
October 28, 20241 yr On 8/21/2023 at 1:24 AM, IronBeardKnight said: For those of you that have setup the script to go with the ClamAV container but have noticed little to no activity coming from it when running "Docker Stats" this may be the fix to your issue. I don't believe that the container is setup to do a scan on startup so you may have to trigger it by adding this line to the scripts as seen below in the screen shot. I have also figured out how to get multithreading working although be warned when using multi you may want to schedual it for when your not using your server as it can be quite CPU and RAM hungry. Some thoughts for you before you proceed with multithreaded scans are to put a memory limit on your docker through extra parameters. Multi Thread: exec('docker exec ClamAV sh -c "find /scan -type f -print0 | xargs -0 -P $(nproc) clamscan"'); Single Thread: exec('docker exec ClamAV sh -c "clamscan"'); While diagnosing why my instance of ClamAV decided to not scan anymore I came across script edit and immediately needed to try it out. Ended up having to diagnose why my script kept exiting xargs for reason of "terminating by signal 9". I have no experience with xargs, so queue up reading montage. With a better understanding of how to use xargs to do things, I'm left with this question.... Isn't this "multithread hack" just running clamscan redundantly over the same files on all threads/cores? Someone correct me if I'm wrong, but honestly I don't want to scan the same book 16 times - I want to be able to scan multiple things at a time to speed up my scans. 140TB worth of Gutenburg content and ISOs is a lot to scan through, especially multiple times at the same time 😵💫
October 28, 20241 yr On 10/28/2024 at 10:55 AM, drkpu1se said: While diagnosing why my instance of ClamAV decided to not scan anymore I came across script edit and immediately needed to try it out. Ended up having to diagnose why my script kept exiting xargs for reason of "terminating by signal 9". I have no experience with xargs, so queue up reading montage. With a better understanding of how to use xargs to do things, I'm left with this question.... Isn't this "multithread hack" just running clamscan redundantly over the same files on all threads/cores? Someone correct me if I'm wrong, but honestly I don't want to scan the same book 16 times - I want to be able to scan multiple things at a time to speed up my scans. 140TB worth of Gutenburg content and ISOs is a lot to scan through, especially multiple times at the same time 😵💫 This is not to hyjack or to have useers use this over TQ docker... This was more a work around that I had made to get other things done.... As It sounds like you want concurrent scans...(which is happening by default...) So i would have you review the many options on calm AV man page... https://docs.clamav.net/manual/Usage/Scanning.html https://linux.die.net/man/1/clamscan As there are other options that would then be added to the extra parameters in the docker template... Sadly for me this took me down a docker configs and dockers compose rabbit hole. As I was unable to get TQ Dockers to work for me anymore... What they have is perfect and great for a unraid template and for the unraid community users base if you want to use a default calm av scan ... For my needs as I want to do a bit more... -edit some setting in freshclam/clamd or have my scan run with other advance options... https://serverfault.com/questions/745120/how-to-scan-multiple-directories-with-clamav So I made this github repo to be used on Unraid with the Docker Compose Plugin. https://github.com/bmartino1/ClamAV/tree/main ^- Ready for sharing... Lets use clamdscan over clamscan... --git clone... docker compose advance set gitclone folder. edit autoscan to fit your needs... I had a issues where the docker template was no longer calling my autoscan sh script but wanted the same settings that TQ has added to their docker image. I couldn't find the docker entry point they used to replace it to call my autoscan. So, decided to switch docker images... I used the ClamAV community alpine image and TQ docker image settings and configs... Many Thanks to TQ and there work on this docker image and there continued support on Unraid. Edited October 29, 20241 yr by bmartino1 github data fixed.
October 28, 20241 yr Author 2 hours ago, drkpu1se said: While diagnosing why my instance of ClamAV decided to not scan anymore I came across script edit and immediately needed to try it out. Ended up having to diagnose why my script kept exiting xargs for reason of "terminating by signal 9". I have no experience with xargs, so queue up reading montage. With a better understanding of how to use xargs to do things, I'm left with this question.... Isn't this "multithread hack" just running clamscan redundantly over the same files on all threads/cores? Someone correct me if I'm wrong, but honestly I don't want to scan the same book 16 times - I want to be able to scan multiple things at a time to speed up my scans. 140TB worth of Gutenburg content and ISOs is a lot to scan through, especially multiple times at the same time 😵💫 On my Github page, I've called this out. docker run -d --name=ClamAV \ --cpuset-cpus='0,1' \ -v /path/to/scan:/scan:ro \ -v /path/to/sig:/var/lib/clamav:rw \ tquinnelly/clamav-alpine -i --log=/var/lib/clamav/log.log --max-filesize=2048M
October 28, 20241 yr 54 minutes ago, bmartino1 said: This is not to hyjack or to have useers use this over TQ docker... This was more a work around that I had made to get other things done.... As It sounds like you want concurrent scans...(which is happening by default...) So i would have you review the many options on calm AV man page... https://docs.clamav.net/manual/Usage/Scanning.html https://linux.die.net/man/1/clamscan As there are other options that would then be added to the extra parameters in the docker template... Sadly for me this took me down a docker configs and dockers compose rabbit hole. As I was unable to get TQ Dockers to work for me anymore... What they have is perfect and great for a unraid tempate and for the unraid community users base if you want to use a default calm av scan ... For my needs as I want to do a bit more... -edit some setting in freshclam/clamd or have my scan run with other advance options... https://serverfault.com/questions/745120/how-to-scan-multiple-directories-with-clamav So I made this github repo to be used on Unraid with the Docker Compose Plugin. https://github.com/bmartino1/ClamAV/tree/main ^- WIP functional as you just need to edit docker compose and autoscan... --git clone... docker compose advance set gitclone folder. edit autoscan to fit your needs... I had a issues where the docker template was no longer calling my autoscan sh script but wanted the same settings that TQ has added to their docker image. I couldn't find the docker entry point they used to replace it to call my autoscan. So, decided to switch docker images... I used the ClamAV community alpine image and TQ docker image settings and configs... Many Thanks to TQ and there work on this docker image and there continued support on Unraid. I suppose I was confused at the wording differences between clamdscan and clamscan. When reading manpages of clamscan, there's no mention of multithread support whereas clamdscan mentions multithread support. When I brought up xargs issues, I was referring to the startup userscript line edit post made by @IronBeardKnight back in 2023. With my new understanding of xargs, I see that this line edit: exec('docker exec ClamAV sh -c "find /scan -type f -print0 | xargs -0 -P $(nproc) clamscan"'); essential attempts to spin up clamscan instances times the number of cores available. I apologize if my post sounded like it was directed towards you @bmartino1, that wasn't my intention. Thank you for clarifacation and additional reading material. 15 minutes ago, TQ said: On my Github page, I've called this out. docker run -d --name=ClamAV \ --cpuset-cpus='0,1' \ -v /path/to/scan:/scan:ro \ -v /path/to/sig:/var/lib/clamav:rw \ tquinnelly/clamav-alpine -i --log=/var/lib/clamav/log.log --max-filesize=2048M @TQ I see in your example you've pinned two cores, but that doesn't necessarily mean a given application is multithreaded. cpuset-cpus just tells docker to run the container on the specified cores. That said, I have played with the template to optimize what I'm going for. I believe I've set up my template and startup userscript about as good as I can get it. I understand clamav better now doing considerably more reading. Thank you TQ for your hard work! Thank you again as well bmartino for directing me in the right place to understand clamav better
October 28, 20241 yr Author 1 hour ago, drkpu1se said: @TQ I see in your example you've pinned two cores, but that doesn't necessarily mean a given application is multithreaded. cpuset-cpus just tells docker to run the container on the specified cores. That said, I have played with the template to optimize what I'm going for. I believe I've set up my template and startup userscript about as good as I can get it. I understand clamav better now doing considerably more reading. Thank you TQ for your hard work! Thank you again as well bmartino for directing me in the right place to understand clamav better Yes, I meant to call out the lack of multi-thread but the need for CPU pinning in the event that you have a large scan set and not a lot of cores. ClamAV will take what it can get, so limiting it to a sane set of defaults helps. Thanks for the kind words.
October 29, 20241 yr Thank You TQ! I've finalized the clamd.conf and autoscan.sh in my github... the docker compose variant is now finished and ready enough for sharing... Thank you for pointing me towards your github repo I was finally able to make unraid template edits to run my ash autoscan script once more (similar to the one in the docker compose variant) to work with your docker... as i can still use your docker image... In essence I had to replace your scan.sh script in the docker. Example of the 1 for 1 replacement echo not using TQ scan.sh lets run my own #cd /var/lib/clamav/ ash /var/lib/clamav/autoscan.sh to fix, edit and add additional stuff: https://github.com/tquizzle/clamav-alpine/blob/master/scan.sh such as running other scripts etc... example: this way I can start clamd and use a clamdscan Edited October 29, 20241 yr by bmartino1
November 4, 20241 yr My daily scan time has crept to 16 hours and i thought to do something to that. I made a modification to my script. The old script keeps scanning old files day after day. The modification makes it to concentrate to only new or modified files. The change selects folders which have modified files within a month, or directories modified within a month. A test run reduced scan time from 16 hours to one. Please note that this change is to my original script and @Masterwishx has a newer version. I have not compared the scripts and don't know if it fits directly into it. The snippet still creates same clamavtargets.txt file as before. But instead of listing the top-level folder it generates a list of subfolders under the top folder - modified within last month. When old script sent e.g. whole "/mnt/user/media" folder for scanning, now it may send e.g. "/mnt/user/media/movies2020s", "/mnt/user/media/SomeNewMusic" folders. Assuming there were folders like movies2010s, movies2000s and so on, the scan will skip a large part of (old) files. This modification is inefficient if you have all files in one huge directory. The snippet (not the whole script) ... echo Creating scan list: $FOLDERS rm $HOSTAPPDATA/clamavtargets.txt 2> /dev/null rm $HOSTAPPDATA/clamavmaplog.txt 2> /dev/null maperrors=0 for f in $FOLDERS do if [ -d "$HOSTSCANDIR/$f" ]; then # Find directories containing modified files. find "$HOSTSCANDIR/$f" -type f ! -name '*.log' -mtime -30 -exec dirname {} \; > $HOSTAPPDATA/clamavtargets_temp.txt # Find modified directories biut skip x top levels (mindepth). find "$HOSTSCANDIR/$f" -mindepth 2 -type d -mtime -30 -print >> $HOSTAPPDATA/clamavtargets_temp.txt # Relocate folder list into container cat $HOSTAPPDATA/clamavtargets_temp.txt | sort --unique | sed -e "s,$HOSTSCANDIR,/scan," >> $HOSTAPPDATA/clamavtargets.txt #echo "/scan/$f" >> $HOSTAPPDATA/clamavtargets.txt echo "Scanning container /scan/$f -> host $HOSTSCANDIR/$f" echo "Scanning container /scan/$f -> host $HOSTSCANDIR/$f" >> $HOSTAPPDATA/clamavmaplog.txt else echo "Warning: can't find share: $HOSTSCANDIR/$f" echo "Warning: can't find share: $HOSTSCANDIR/$f" >> $HOSTAPPDATA/clamavmaplog.txt maperrors=1 fi done # Remove children if higher level folder is on the list. cat $HOSTAPPDATA/clamavtargets.txt > $HOSTAPPDATA/clamavtargets_temp.txt while read f; do grep -v "$f/" $HOSTAPPDATA/clamavtargets_temp.txt > $HOSTAPPDATA/clamavtargets_temp2.txt; cat $HOSTAPPDATA/clamavtargets_temp2.txt > $HOSTAPPDATA/clamavtargets_temp.txt ; done < $HOSTAPPDATA/clamavtargets.txt cat $HOSTAPPDATA/clamavtargets_temp.txt > $HOSTAPPDATA/clamavtargets.txt echo "Scan list:" cat $HOSTAPPDATA/clamavtargets.txt echo "" ... Changing e.g. "-mtime -30" to "-mtime -7" reduces the time window from one month to a week. The "-mindepth 2" prevents top level folder and immediate children being added to the list if the directory is modified. Changing it to "-mindepth 1" allows including folders immediately below top folder to list. Note that even top level folder may be added if there are modified FILES right under it. The snippet ignores all *.log files. Remove "! -name '*.log' " if you want to include them. The snippet is released to public domain. Edited November 4, 20241 yr by Helediron
November 28, 20241 yr Hoi, why do i get these? `LibClamAV Warning: PNG: Unexpected early end-of-file.`
November 29, 20241 yr 14 hours ago, Duckers said: Hoi, why do i get these? `LibClamAV Warning: PNG: Unexpected early end-of-file.` The warning message LibClamAV Warning: PNG: Unexpected early end-of-file indicates that ClamAV's scanning library, LibClamAV, encountered a PNG file that appears to be truncated or corrupted, as it reached the end of the file unexpectedly during parsing. This situation can arise from genuinely corrupted files or from certain PNG files that don't strictly adhere to the PNG specification. Possible Causes: Corrupted PNG Files: The file might be incomplete or damaged, leading to an unexpected end-of-file during scanning. Non-Standard PNG Files: Some PNG files may have unconventional structures that, while possibly viewable in image viewers, don't conform to standard specifications, causing parsers like LibClamAV to flag them. Resolution Steps: Update ClamAV: Ensure you're using the latest version of ClamAV. Updates often include bug fixes and improvements that can resolve such warnings. For instance, ClamAV version 1.2.3 addressed this specific warning: Silenced a warning "Unexpected early end-of-file" that occurred when scanning some PNG files. as seen on the blog: https://blog.clamav.net/2024/04/clamav-131-123-106-patch-versions.html Identify the Problematic File: Increase Verbosity: Run clamscan with the -v (verbose) option to get more detailed output, which can help pinpoint the exact file causing the warning: add -v to extra parameter set as logging extra parameter --log or > log.txt Review the Log: Examine the log to locate the file associated with the warning. Examine the File: Check Integrity: Open the identified PNG file with an image viewer to see if it displays correctly. Replace if Necessary: If the file is indeed corrupted, consider replacing it with a valid version. Additional Considerations: False Positives: If the PNG files open correctly and aren't corrupted, the warning might be a false positive due to strict parsing by LibClamAV. Suppressing Warnings: If the warnings are numerous and pertain to non-malicious files, you might choose to suppress them by updating ClamAV, as newer versions have addressed this issue. Edited November 29, 20241 yr by bmartino1
December 1, 20241 yr On 11/29/2024 at 5:43 AM, bmartino1 said: The warning message LibClamAV Warning: PNG: Unexpected early end-of-file indicates that ClamAV's scanning library, LibClamAV, encountered a PNG file that appears to be truncated or corrupted, as it reached the end of the file unexpectedly during parsing. This situation can arise from genuinely corrupted files or from certain PNG files that don't strictly adhere to the PNG specification. Gotcha! Will run verbose next time! Also, how can i make it use more cpu performance? As it appears to only use 1 core, or less? Or does it have a argument to do multiple disks at once to multithread up the scan that way if the little cpu it uses is due to spinning rust's mechanical limitation? Cause it's been over 4 or 5 days scanning now, and it has by the looks of it, barely started lol. And oddly my last post only had 1 error even if i pasted a bunch Here's the other errors i need explained as well. LibClamAV Warning: file_bytes is not valid unicode: invalid utf-8 sequence of 1 bytes from index 112 LibClamAV Warning: Bytecode run timed out in interpreter after 662625000 opcodes LibClamAV Warning: Bytecode 'BC.Legacy.Exploit.Andr-2.{Extra_Field,Master_Key}' (id: 35) failed to run: Exceeded time limit Gonna assume the 2 below was due to too big file? And what's the command to increase the size? LibClamAV info: scancws: Error decompressing SWF file. Scanning what was decompressed. LibClamAV Warning: SWF: declared output length != inflated stream size, 14879 != 4916 Edited December 1, 20241 yr by Duckers
December 1, 20241 yr I would recommend you use the clamav-clamdscan docker image template. Its why I made it. Using the same setting as TQ. *I don't use the extra parameter I instead use a script in the fresh clam DB files location. you can turn off privileged. I have UUID and PUIG running as root to help with file access issues. *This is why I have privileged enabled in the template as a warning. So what's the difference that TQ ClamAV has... The Difference is in what calls a script in the db folder made to scan using clam daemon. The clam daemon calls multiple clamscan to scan a file using the full cpu performance. For me a 48Hour scan with TQs clamAV on scanning 6+TB of data using clamdscan became a 8-hour scan. the caveat. clam daemon revolves around the clamd.conf so if you have a folder/file to exclude (has to be regex in clamd.conf) or if you misconfigure clamd.conf then clamav-clamdscan will not run.... This will be noticeable in the log. So I tried to set up and fix as much as possible. For recovery, I have set it up where one can remove the dockers /etc/clamd.conf file and it will redownload the default again. By default, it is set up to scan the system with a sh script that is found in the fresh clam db. For recovery, you can remove the clamdscan.sh file and the docker will re download the default. If you make edits to either, it will not replace... By default, a fresh download of the docker will scan /mnt/user you can edit the script file to add other multiple folders as docker /scan then becomes host /mnt/user/ so if i want to scan the appdata folder only i edit the script and have /scan/appdata [or edit the docker template to be /mnt/user/appdata] if I want to exclude something, I have to edit the clamd.conf at the very bottom of the file (IT IS LONG!)... to add a regex value to script that. Since editing the clamd scan can bork the clam scan (seen in unraid by going to the docker log...) I added a recovery function. TQ is more robust as it calls one clamscan with options. Where as clamdscan calls multiple clamscans based on the clamd.conf... Information to edit the /scan in TQ docker is available earlier in the form. I would still recommend running TQ image. Edited December 28, 20241 yr by bmartino1 Data - typo
December 2, 20241 yr On 12/1/2024 at 7:52 PM, bmartino1 said: I would recommend you use the clamav-clamdscan docker image template. Its why I made it. Using the same setting as TQ. *I don't use the extra parameter I instead use a script in the fresh clam DB files location. Didn't find it in apps section. As i don't know how to install dockers via docker image.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.