mgutt

No need to test iperf. I enabled the FTP server, opened Filezilla, set parallel connections to 5 and choosed 5 huge files from 5 different disks: By that I was able to reach 900 MB/s in total: Similar test, this time I started multiple downloads through Windows Explorer (SMB): This time my wife viewed a movie through Plex, so results could be a little bit slower than possible, but Filezilla was still able to download >700 MB/s so shouldn't be a huge difference. So whats up with SMB? I checked the used SMB version through Windows PowerShell and it returns 3.1.1

Did you solve the issue? My transfer speed isn't as low as yours, but I think it should be better. What I did: a) If found in this thread a hint how to install iostat. So I did the following to install it: cd /mnt/user/Marc wget http://slackware.cs.utah.edu/pub/slackware/slackware64-14.2/slackware64/ap/sysstat-11.2.1.1-x86_64-1.txz upgradepkg --install-new sysstat-11.2.1.1-x86_64-1.txz b) Started iostat as follows: watch -t -n 0.1 iostat -d -t -y 5 1 c) I downloaded through windows a huge file that is located on my SSD cache and as we can see its loaded from the NVMe as expected: d) Then I downloaded a smaller file to test the RAM cache. The first download was delivered through the NVMe: e) The second transfer shows nothing (file was loaded from RAM / SMB RAM cache): This leaves some questions: 1.) Why is the SSD read speed 80 MB/s slower than the RAM although its able to transfer much more than 1 GB/s? 2.) Why is the maximum around 500 MB/s? Note: My PCs network status, the Unraid dashboard and my switch show all 10G as link speed.

I did the same now with Lubuntu and Google Chrome Remote Desktop. Here is a little tutorial (in German), too:

My script contains a loop of different short running processes (mkdir, openssl, touch) and not one long running single process. If I close f.e. the "Run Script" window it is killed completely (as expected).

Released Version 0.2:

CA User Scripts does not kill scripts!

Maybe someone needs a script for file encryption:

Description filecryptor is a shell script that reads all files in a source directory and copies them encrypted through AES-256 in a target directory. The passphrase must be added through a file named "filecryptor.key" that has to be located in the root of the source directory. Your password is salted with the file's timestamp and hashed with PBKDF2 10.000x times. This is the default value of filecryptor. You can change it, but you need to remind the exact value or you are not able to decrypt the files anymore! Feel free to upload your files to a cloud after encryption has been finished! You can execute this script by using the CA User Scripts plugin. Features - encrypt files of a source to a target directory with AES-256 by your password - encryption is hardened by iterating your password through PBKDF2 10.000x times which slows down bruteforce attacks - encryption is hardened by salting your password through the file's timestamp leaving no chance to rainbow table attacks - decrypt files of a source folder - resume if last script execution has been interrupted (delete filecryptor.last in target dir to fully restart) - skip_files_last_seconds allows skipping files that are (partially) written at the moment - dry_run allows testing filecryptor - already existing files with a newer or same timestamp will be skipped #!/bin/sh # ##################################### # filecryptor # Version: 0.3 # Author: Marc Gutt # # Description: # Copies and encrypts files from a source to a target dir. This scripts uses # AES encryption with the file modification time as salt. By that re-encrypting # produces identical files making it easier for rsync/rclone to skip already # transfered files. # # How-to encrypt: # 1.) Add a text file with the name "filecryptor.key" and your encryption password as content in the source directory. # 2.) Set your source and target directories # 3.) Execute this script # # How-to decrypt: # 1.) Backup your encrypted files # 2.) Add a text file with the name "filecryptor.key" and your encryption password as content in the source directory. # 3.) Set ONLY the source directory (encrypted files will be overwritten through their decrypted version!) # 4.) Execute this script # # Notes: # - A public salt should be safe (https://crypto.stackexchange.com/a/59180/41422 & https://stackoverflow.com/a/3850335/318765) # - After encrypting you could use rsync with "--remove-source-files" to move the encrypted files to a final target directory # - rclone has a "move" mode to move the encrypted files to a cloud # - You can set the PBKDF2 iterations through "iter". A higher number of iterations adds more security but is slower # (https://en.wikipedia.org/wiki/PBKDF2#Purpose_and_operation & https://security.stackexchange.com/a/3993/2296) # # Changelog: # 0.3 # - set your own PBKDF2 iteration count through "iter" setting # - salt is padded with zeros to avoid "hex string is too short, padding with zero bytes to length" message on openssl output # - decryption now supports resume # - empty openssl files are deleted on decryption fails # - encryption of already existing files in the target will be skipped as long the source files aren't newer # 0.2 # - skip dirs on resume (not only files) # - bug fix: endless looping empty dirs # 0.1 # - first release # # Todo: # - optional filename encryption # - use a different filename if decrypted filename "${file}.filecryptor" already exists # - add "overwrite=true" setting and check file existence before executing openssl # ##################################### # settings source="/mnt/disks/THOTH_Photo" target="/mnt/user/photo" skip_files_last_seconds=false dry_run=false iter=10000 # remind this number or you are not able to decrypt your files anymore! # check settings source=$([[ "${source: -1}" == "/" ]] && echo "${source%?}" || echo "$source") target=$([[ "${target: -1}" == "/" ]] && echo "${target%?}" || echo "$target") target=$([[ $target == 0 ]] && echo "false" || echo "$target") target=$([[ $target == "" ]] && echo "false" || echo "$target") skip_files_last_seconds=$([[ $skip_files_last_seconds == 0 ]] && echo "false" || echo "$skip_files_last_seconds") dry_run=$([[ $dry_run == 0 ]] && echo "false" || echo "$dry_run") dry_run=$([[ $dry_run == 1 ]] && echo "true" || echo "$dry_run") # defaults pwfile="${source}/filecryptor.key" resume="${target}/filecryptor.last" # check if passphrase exists if [[ ! -f $pwfile ]]; then echo "Error! filecryptor did not found ${source}/filecryptor.key" exit 1 fi # check if we have a starting point if [[ -f $resume ]]; then last_file=$( < $resume) fi function filecryptor() { path=$1 echo "Parsing $path ..." for file in "$path"/*; do # regular file if [ -f "$file" ]; then # skip passphrase file if [[ $file == $pwfile ]]; then echo "Skip $pwfile" continue fi # skip files until we reach our starting point if [[ -n $last_file ]]; then if [[ $file != $last_file ]]; then echo "Skip $file" else echo "Found the last processed file $last_file" last_file="" fi continue fi file_time=$(stat -c %Y "$file") # file modification time # decrypt file if [[ $target == "false" ]]; then echo "Decrypt file ${file}" if [[ $dry_run != "true" ]]; then if openssl aes-256-cbc -d -iter 10000 -in "$file" -out "${file}.filecryptor" -pass file:"${pwfile}"; then rm "$file" mv "${file}.filecryptor" "$file" touch --date=@${file_time} "${file}" else rm "${file}.filecryptor" # cleanup on fail fi fi # remember this file as starting point for the next execution (if interrupted) if [[ $dry_run != "true" ]]; then echo "$file" > "${resume}" fi continue fi # skip new files if [[ "$skip_files_older_seconds" =~ ^[0-9]+$ ]]; then compare_time=$(($file_time + $skip_files_older_seconds)) current_time=$(date +%s) # is the file old enough? if [[ $compare_time -gt $current_time ]]; then continue fi fi dirname=$(dirname "$file") dirname="${dirname/$source/}" # remove source path from dirname file_basename=$(basename -- "$file") # skip already existing files with same timestamp if [ -f "${target}${dirname}/${file_basename}" ]; then target_file_time=$(stat -c %Y "$file") # file modification time if [[ $target_file_time -ge $file_time ]];then echo "Skipped ${file} as it already exists in target" continue fi fi # create parent dirs echo "Create parent dirs ${target}${dirname}" if [[ $dry_run != "true" ]]; then mkdir -p "${target}${dirname}" fi # encrypt file echo "Create encrypted file ${target}${dirname}/${file_basename}" if [[ $dry_run != "true" ]]; then salt="${file_time}0000000000000000" salt=${salt:0:16} openssl aes-256-cbc -iter $iter -in "$file" -out "${target}${dirname}/${file_basename}" -S $salt -pass file:"${pwfile}" fi # modification time echo "Set original file modification time" if [[ $dry_run != "true" ]]; then touch --date=@${file_time} "${target}${dirname}/${file_basename}" # https://unix.stackexchange.com/a/36765/101920 fi # remember this file as starting point for the next execution (if interrupted) if [[ $dry_run != "true" ]]; then echo "$file" > "${resume}" fi # dir elif [ -d "$file" ]; then # skip dir until we reach our starting point if [[ -n $last_file ]]; then if [[ $last_file != "${file}"* ]]; then echo "Skip $file" continue else echo "Found the last processed dir $file" fi fi filecryptor "$file" fi done } filecryptor "$source" # clean up if [[ $dry_run != "true" ]]; then rm "${resume}" fi exit # encrypt example if file_time=$(stat -c %Y "file.txt"); then openssl aes-256-cbc -iter 10000 -in "file.txt" -out "file.enc" -S $file_time -pass file:"filecryptor.key" touch --date=@${file_time} "file.enc" fi # decrypt example if file_time=$(stat -c %Y "file.enc") && openssl aes-256-cbc -d -iter $iter -in "file.enc" -out "file.enc.filecryptor" -pass file:"filecryptor.key"; then rm "file.enc" mv "file.enc.filecryptor" "file.txt" touch --date=@${file_time} "file.txt" fi

Hi, I have two different problems with this plugin: 1.) When I choose "Power Save" as my CPU Scaling Governor, my Plex Docker Container buffers all streams every 10-15 seconds. Do I need an other driver or is the "ACPI" correct for my Atom C3758? Testing with <watch grep \"cpu MHz\" /proc/cpuinfo> returned 800 Mhz if no process was running. Now its around 1.7 Ghz (with On Demand). 2.) By default, the NIC settings contained "eth0" in the list, but as long this value was part of the settings my Unraid server was not able to connect the internet anymore! I do not know which of these settings caused this problem. My suggestion is to add "N/A" to the different options to be able to reset all NIC settings. While the server was running I was able to reset them by re-applying the unraid network settings (removed and added the second DNs server ip ^^).

I had the same problem (no ping to 8.8.8.8 possible, but local access works). Finally I found out that the Tips and Tweaks Plugin killed my network settings. I do not know why, but after removing the "eth0" of the list and rebooting the machine, the internet was accessible again: I think the plugin is overwriting my settings on boot causing this error. I will post this in the plugin's thread, too. EDIT: Sometimes even the local network did not work. If this happens it helped to login through the server itself (IPMI or physical) and after log in with "root" and my password, I used this command to re-apply the network settings: /etc/rc.d/rc.inet1 restart After that I was able to open the webclient again.

I tested the "Power Save" setting for my Intel Atom C3758. By that the CPU was throttled down to 800 Mhz. First it looked good, but then I found out that all content that was streamed through Plex, started to buffer every 10-15 seconds. After choosing the default "On Demand" settiing everything went back to normal. Tips and Tricks shows "Driver: ACPI CPU Freq" as the driver. Could this be the reason (as it did not use the intel pstate driver)?

I have two unraid servers. One uses an Atom C3758 CPU. After installing the Tips and Tweaks Plugin and changing from On Demand to Power Save it downclocks to 800 Mhz as expected. The other server uses a Pentium J5005. The dashboard returns most of the time a usage of 0-2% for all cores: By using this command in the web terminal: watch grep \"cpu MHz\" /proc/cpuinfo it returns 2.3 Ghz: Every 2.0s: grep "cpu MHz" /proc/cpuinfo Black: Mon Jun 15 00:03:51 2020 cpu MHz : 2399.270 cpu MHz : 2315.888 cpu MHz : 2088.474 cpu MHz : 2333.949 After disabling the Intel Turbo it returns 1.3 GHz: Every 2.0s: grep "cpu MHz" /proc/cpuinfo Black: Mon Jun 15 00:07:02 2020 cpu MHz : 1318.495 cpu MHz : 1342.343 cpu MHz : 1245.570 cpu MHz : 1369.103 It clocks now under the baseclock (1.5 Ghz) of this CPU?! But the real strange thing is the power consumption as its nearly the same for all settings (all five disks spun down): Performance + Turbo: 18.59W Power Save + Turbo: 18.09W Power Save: 18.09W Power Save + Closed all browser windows: 18.02W Power measurement was done with a good consumer power meter (<5W +/- 100mW and >5W +/- 2%). EDIT: Hmm I restarted the server and altough the Tips and Tricks plugin displays "Power Save" its in "Performance" mode (as all cores stay at 2.6GHz).

Yes, that's correct. Sorry, I thought it was a file because mv returned a file error instead of "Directory not empty" error for directories. mv is not able to merge dirs. cp is able to do that. Example: cp -al source/folder destination rm -r source/folder The -l option causes to create hardlinks instead of copying the files. But cp does not contain an exclude flag so it does not solve your problem. Instead use rsync "--link-dest=". By that it creates hardlinks and does NOT copy the files and after that "--remove-source-files" removes the initial hardlink of a file: rsync -a --remove-source-files --exclude=".sync*" --link-dest=/mnt/disks/DELUGETorrents/Seedbox_Downloads/ /mnt/disks/DELUGETorrents/Seedbox_Downloads/ /mnt/disks/DELUGETorrents/Sonarr_Pickup/ Sadly it does not remove the empty dirs in the source. This needs a separate command: https://unix.stackexchange.com/a/46326/101920

Do you maybe aren't allowed to delete the target file? In this manual it says that the target file is removed before the local file is moved: https://unix.stackexchange.com/a/236676/101920 You should test it in a script by using rm: rm '/mnt/disks/DELUGETorrents/Sonarr_Pickup/The 100 Season 3 [1080p] WEB-DL H264'

Hmm I wonder why this does not work. But you can instead use rsync with the --remove-source-files option. This will delete the source after it has been transfered to the target. Example: rsync -a --remove-source-files --exclude=".sync*" /mnt/disks/DELUGETorrents/Seedbox_Downloads/ /mnt/disks/DELUGETorrents/Sonarr_Pickup/ Explanation: -a = -rlptgoD = recursive, symbolic links, preserve all auth, preserve date --remove-soure-files = delete source file after transfer --exclude=".sync*" = ignore files starting with ".sync Another useful setting if you want to use rsync to sync two folders: --delete = delete files in the target that are not present in the source (not useful in combination with "--remove-soure-files")

The [-f] in the documentation means that -f is optional. You must not type the brackets in your script to use it. And you made a second mistake. You did not add a white space, so mv tried to find the path "[-f]/mnt...". That's the reason why it returned the "no such directory" error.

Solution in Windows without using RDP or other VNC software: 1. ) Install AutoHotKey 2.) Right click on desktop -> create AutoHotKey script 3.) Edit script and paste: ^+v::Send {Raw}%Clipboard% 4.) Doube-click the script 5.) Now you can use CTRL+SHIFT+V to paste anything in the noVNC browser window (its typing the chars that are in the clipboard) Paste the script in the windows startup folder so its automatically executed on reboot: https://www.autohotkey.com/docs/FAQ.htm#Startup

Solution? https://stackoverflow.com/a/43099210/318765

This is how I check the status and start a container through one of my scripts: # check if mkvtoolnix container exists if [[ ! "$(docker ps -q -f name=mkvtoolnix_mkv2sub)" ]]; then # https://stackoverflow.com/a/38576401/318765 # check for blocking container if [[ "$(docker ps -aq -f status=exited -f name=mkvtoolnix_mkv2sub)" ]]; then docker rm mkvtoolnix_mkv2sub fi echo "mkvtoolnix container needs to be started" # start mkvtoolnix container docker_options=( run -d --name=mkvtoolnix_mkv2sub -e TZ=Europe/Berlin -v "${docker_config_path}mkvtoolnix_mkv2sub:/config:rw" -v "${movies_path}:/storage:rw" jlesage/mkvtoolnix ) echo "docker ${docker_options[@]}" docker "${docker_options[@]}" fi Found here: https://stackoverflow.com/a/38576401/318765 But this checks only if the container is already existing or its status is "exited". I tried to check if the container is not used anymore by its cpu usage, but this produced problems with parallel running scripts so I disabled it: # check if container exists if [[ -x "$(command -v docker)" ]] && [[ "$(docker ps -q -f name=mkvtoolnix_mkv2sub)" ]]; then # stop container only if its not in use (by other shell script) mkvtoolnix_cpu_usage="$(docker stats mkvtoolnix_mkv2sub --no-stream --format "{{.CPUPerc}}")" # if [[ ${mkvtoolnix_cpu_usage%.*} -lt 1 ]]; then # we do not stop the container as our script is not race-condition safe! # echo "Stop mkvtoolnix container" # docker stop mkvtoolnix_mkv2sub # docker rm mkvtoolnix_mkv2sub # fi fi Maybe I will set in the future a random container name and clean them up after they become too old. I'm not sure.

You need to add "shopt -s extglob" to allow this command: https://askubuntu.com/a/1228370/227119

Then try !(.syncthing.*) I think this should work.

If your source files have all the same extension you could use this: /mnt/disks/DELUGETorrents/Seedbox_Downloads/*.mp4 Or this one should work, too: /mnt/disks/DELUGETorrents/Seedbox_Downloads/!(*.synch)

Ok, if you need hourly scheduling, CA user scripts is the correct way. Edit: instead of physically copying all files you could hardlink them in a second folder. By that the "copy process" would be much faster and it will not waste your storage space. Edit2: This should work: rsync -a --delete --link-dest=/mnt/disks/DELUGETorrents/Seedbox_Downloads/ /mnt/disks/DELUGETorrents/Seedbox_Downloads/ /mnt/disks/DELUGETorrents/Sonarr_Pickup/ "--link-dest" forces rsync to create hard links instead if copying the complete file. "--delete" means it removes all files from Sonarr_Pickup/ if they do not exist in Seedbox_Downloads/ anymore. Before executing this command you should empty the destination folder or use a different one. Edit3: Ok "cp" supports that by using the "l"-flag too: https://superuser.com/a/1360635/129262

Do you really want to execute this multiple times? It sounds you only want to execute it one time. Then you could use the Terminal:

If realized in JS f.e. through a Pop-up, it could only fill the field. By that it would be still possible to manually edit it. Could you send me the source code of the list page? I'll try to realize that.