comet424

so i dunno if the community apps has an app that does what i looking for what i want is since DropBox only lets you download 1 or 2gb databasee the free and i download to it from another site it syncs to it.. and will sync to my dropbox on my desktop and i manually move it to the array but i looking for an app maybe that when files are sync to the drive that it moves the files automaticlly.. so when files show up just move it to the specific folder if i have to make seperate automations for each folder. as i dont wanna move the folder just the contacts in said folder.. and i guess id install dropbox on unraid but looking for something like how TDARR works when files show up it process's so i looking for if files show up in the dropbox folder on the array move it to specific folder on the array so it clears up the dropbox so i can download more to it

also if i try to move files from one of my pools to array with cache drive first is slow and if i try to watch a movie streaming off the same server... the file is choppy but shouldt i be able to stream from the server no issue especially with 10g networking shoulldnt have any issues streaming from the array but i do.. and i using asus tuf x570 motherboard for my main server.. its like i cant do multiple things at once

so i now copy from Desktop to a Array Share that uses Cache Drive first it was running at 300+ but then drops down to 12mbs or so test copy folder is Array Folder With Cache First test folder is Folder on the Desktop with the 990 Samsung so copying from the Array Folder with CAche First it started off 500 then dropped down below 200 then droped to 0 for 5 seconds then started up and finishes copying at 500 in test 4 image

so i did a copy of that test file over the network so from the UD share i have to my desktop pc.. my desktop has a Samsung 990 Pro M2. 2Tb i cant figure how to get a share of the Cache Drive to copy directly from it but here is screen shot it stayed around that speed so read from the WD was over 500 and copying from the desktop to the UD stuck around 150

i can shove the UD drive in my desktop PC the PC and unraid both on 10g networking is there a pv command i can go from windows to unraid to do the tests that way through the smb shares?

the cache drive and UD drive just sit there idle now i cant remember if they are both on the SATA on the motherboard or the LSI HBA card and the samsung 980 drives they host the appdata isos vms, system folders and the backup files folder not linked to cache drive first.. so then its hard to narrow down the issue then

and if the sata cache drive WD isnt fast would a samsung sata cache drive be better? also same for UD drive or do i need a m2 samsung or wd drive to be better copying since i had terrible speeds through SMB to SMB (UD to Cache drive or through Krusader where i was getting 1mb speeds or less from WD Sata to WD Sata and or have a PICE card that has M.2 for cache drive and are the sata 1tb drives failing slowing down ? even though they are both empty reality

so i did the tests i added extra tests too so transport_data = 1TB WD Blue Sata UD Drive cache_drive = 1TB WD Blue Sata Cache Drive unraid_files = 2TB Samsung 980 Pro M.2 x2 in mirror Mode disk1 = 14Tb Seagate Red Pro 7200 RPM

opps i forgot the > i redoing the test sorry about my dislexia missed it

for copying files in Krusader i used /mnt/user for the shares to array and to copy to cache and the UD drive i used /disks/disks for the UD drive /disks/cache_drive for the cache drive is that pv command supposed to show scrolling scrambled characters like hex of a file do i let it just scroll the entire file

how do you get benchmark cache drive mounting /mnt /mnt/cache_drive doesnt let you scan it

which one the cache or the unassigned drive one.. they both the same 1 tb wd blue so i ran diskspeed app on the UD for the 500mb/s i getting nearly 5000mb/s test it says didnt know it could go that fast but its not able to test the cache drive

which one would be the cultuprate or is it something unraid doing it.. are there other tests i can do.. to narrow things down etc

test is copying 4 files of 34.9gb combined smallest file 1.8gb 4.7gb 7.7gb biggest 20.7gb so copying from array to unassigned drive SSD it was copying like 90-140mib/s average 130-135mib/s array to cache drive SSD it was copying at 140-220mib/s average 180-190mib/s but drops down to 50-60mib for a bit then back up then drops down to 30mib unassigned drive to array (avoiding a cache) started at 300mibs then dropped then dropped to 5mibs then goes up to 300mib then to like 50 and up to 340mibs t9hen averaging around 50 then drops to 25ish then back up to 340 then down to 30 then back up to 300 ish jumps to 394mibs and then drops to like 20 and then 4mib then jumps up etc test unassigned to temp folder on array with cache drive first... copying started at 300 then dropped to 25-35mib average is 30mib drops down to 5 -18 but then jump back to roughly 30mib drops down to 1-5mib/s then jumps back up to 25-30mib/s average then drops to average of 12-16mib/s cache to the unnassigned drive 30-150mib/s averaging 150mib/s to 130-147mib./s on 20.7gb file array to cache drive average 18mib/s then drops to average 4 copying the 20.7gb file average out at like 20-30mibs/ i used krusader as it was easier to do if i need to do like 1 big file and do the tests over again i can i didnt bother doing all the files i was doing as i didnt wanna be stuck for days... but it feels like i copying over a usb flash drive and not a sata connection i dunno just seems i getting piss poor speeds

i really think something wrong with my unraid when copying files to a ssd cache drive.. ive asked before but never really get diffintive answer always its probably small files well right now i trying to copy 150gb of 77 files ranging 1 2 gb files to like 5gib files.. i only can transfer like 2 3mb or under a mb telling me in krusader its going to take 2 days to move files... there is definity something wrong i even did a reboot... is there a test can be done.. its just ridiculus i cant be copying at 500mbs instead im at less then 1mbs both are Blue SSD WD at 6gb connection... i even shut off all VMS. dockers, and it still wont go faster. so i copying from my Unassigned Drive 1TB drive to the CAche drive.. its also slow if i copy it through windows SMB share there has to be something i can do to fix it... its slower the a spinning hard drive and i copying it to my Temp folder on my array which is cache drive first so i frustrated tardis-diagnostics-20250714-1932.zip

id like to see an option when you download Unraid update it will roll out and update all the unraid servers for people that have 5+ unraid servers.. and poor internet... i do it by downloading but wish unraid offered resume which is a pain alot of time.. when u have bad internet... but once downloaded it should roll out update like windows does for updates you can do i currentlly do this by copying the bz files to a folder on each server then copy it to the /boot and then i force a reboot.. if i dont it glitches the unraid server... as sometimes after the copy of the bz files and ar eboot the unraid glitchs and resets to Tower name blows out the banner erases the background color to white.. so it buggers things up so i looking for a more stable multiple update roll out it works and doesnt work at times.. so its not fool proof.. but be nice if the connect offered it to roll out... also allow unraid downloads be able to resume instead of downloading from scratch if you download 500 meg and it fails you start at 0 and not at 500 when you have a 3mbps internet and its flaky at times be nice to have a resume and also roll out updates and if not possible allow unraid downloads to use the LanCache Docker so the Unraid app can be lan cached so that make unraid servers download the new update from the lancache

what is the largest file you wanna use the rsync --parital etc before you loose performance? from the above results... the smallest file was 274meg and largest was 4Gb out of the 221 files it reported.. pok ill install filezilla linuserver and ill install sftp fail to 2ban ill look them up and then you do it by scripts? to upload like upload resume on its own and --delete option or do you run rsync to --delete if you wanna delete a file later and that script rsync i guess that be ok when you wanna use documents and project files as they be smaller.. and can you do the same with sftp like backup rsycn script does.. always learning something new just trying to do the best performance and best backup solutions.. as who wants to learn 20 different backup ways lol.. and then will it all work with user scripts? as right now how i do it is like 1 script and i have like rsync sections 1 documents, 1 family files, 1 music, 1 videos, and the largest videos in the videos is like 30gb so that partial will never end i bet lol.. and then i have a 2 and 3 user script that does the same pointing to 2 other servers.. and then i pull up dated data from my sisters 1 unraid server to backup to my main server.. and then it backs up that data to the other 3 servers.. so thats where i like the best performance.. cuz all it be doing is backing up lol.. but let me install the filezillas and the sftp and get it configured and you can help me the next steps... do you need to see the user scripts i use

oh ya and here is the results of using rsync -avzue ssh --partial --append-verify --checksum --exclude first time no files located in the location Number of files: 222 (reg: 201, dir: 21) Number of created files: 222 (reg: 201, dir: 21) Number of deleted files: 0 Number of regular files transferred: 201 Total file size: 153,367,530,462 bytes Total transferred file size: 153,367,530,462 bytes Literal data: 153,367,530,462 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 153,261,702,565 Total bytes received: 3,999 sent 153,261,702,565 bytes received 3,999 bytes 32,857,049.32 bytes/sec total size is 153,367,530,462 speedup is 1.00 now the 2nd time i ran it so you know would just do a check.. it was slow as mollases several hours it took to just do a check of files sending incremental file list Number of files: 222 (reg: 201, dir: 21) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 153,367,530,462 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 14,723 Total bytes received: 37 sent 14,723 bytes received 37 bytes 2.14 bytes/sec total size is 153,367,530,462 speedup is 10,390,754.10

@mgutt so for what i ment .. is when it rsync'd it transfered the timestamp and size of the file before it fully copied the file.. i wasnt using --partial so when it rsync'd like the options i showed above i normally use.. it was in the middle of a transfer.. but rsync had set that the file was complete at the start... this was an issue where the drive on the 2nd comp lost connection and rsync though it was completely copied when it wasnt... so when rsync transfered the filename timestamp and file size.. it only copied 10% of the file.. and when i re ran the rsync rsync -avzue ssh it skips that file thinking its completed yet it wasnt completed... as it only copied 10% or 20% of the file as this is what i normally use rsync -avzue ssh --exclude-from='/boot/config/plugins/user.scripts/scripts/exclude1.txt' -s --stats --numeric-ids --progress 'source' root@$serverip:'destination' --delete >> /mnt/user/Temp/logs/cronlogs/certain log file.log @bmartino1 so i been trying rsync -avzue ssh --partial --append-verify --checksum --exclude-from='/boot/config/plugins/user.scripts/scripts/exclude1.txt' -s --stats --numeric-ids --progress 'source folder' [email protected]:'destination folder' --delete >> /mnt/user/Temp/logs/cronlogs/certain log file.log oh my god so damn slow.. so i tested just 1 folder.. of 222 files of 150gig for the folder.. the first time it took more then several hour.. it sat at incremental list for 40 min or so and when it finally completed.. i re ran to see if it go faster.. no its stuck on incremental list... so this no good files are completed under rsync i using.. as if i do like /mnt/user/Videos and say you have 40TB in there i guess it incrementals the list makes it unaccessable i noticed but it takes a day and still nothing.. and thats even with 10Gig networking.... so that is unusable.. using Krusdar was faster as when it does copyy i only get 26mb/s not like krusader at times 240mb/s so i guess the prital append is too slow so from the link yo sent about the incremental rsync thats only good for small files not for big files so good for documents,pictures but not for like video backups.. so then what is the best way to do it then if i wanna backup documents, project files, videos ,etc like i have 3 Unraid Boxes on the local network 10Gig Networking between them 2 Unraid Boxes off site using a 3mbp/s connection as its slow since rsync -avzue ssh --partial --append-verify --checksum takes days and days it looks like when u wanna start over.. and even when its complete and u just run it again for updated files... its hours... as of writting this the 2nd round of a completed folder is still going on stuck on sending incremental file list.. and its been like 20 min writting this and it was already like 20 min prior.. so this not best solution when you wanna make sure like 70 TB is backup of family files documents etc even on 10gig networking its super slow.. so the rsync -avzue ssh is good if no network failure or drive failure so it but the rsync -avzue ssh --partial --append-verify --checksum is so slow its going to take a month and if it fails network connection or what not and you restart it it start back from scratch... i did look at restic but that didnt work you need a password file and u cant even run it to create one and for whatever reason my internet was down and i couldnt install syncthing so i havent tried it... as how i do it is everynight at like 12am it runs the rsync all 4 backup servers.. and if any files need adding it rysncs and then shuts down so sometimes 5 min and done.. but now the partial i guess its checking the file... the partial didnt work if the file was already corrupted i found out so like that one file 10% when i did the --partial --apend etc it rsynced butr the file was still corrupted... so i blew out my backup server.. and started fresh with the --partal --eppend-verify --checksum and that works but takes several hours to do 222 files at 150gb in the folder been like an hour to make this reply and the --partal --eppend-verify --checksum is still stuck on sending file incremental on the folder it completed transfering so its way too slow to do vertification it transfered a true copy... what is the best for 3 10 gig servers. on the local network... and you mentioned about sftp that would work as you can use zmodem or what not to upload to to the internet the other 2 remote servers... but i like the idea of this snapshots so you can rollback like a walmart commerical lol.. so for now i just krusader copying. i guess that is safe that just uses copy but i want it all unattended at like 12am every night or once a week etc

so running rsync -avzue ssh --checksum --progress user@server:/source/ /destination/ --delete even when the file already copied did nothing i guess i gotta blow the backups and then re run it rsync -avzue ssh --partial --append-verify --checksum --progress user@server:/source/ /destination/

ok ill have to re look into it all what you mentioned... you use the sftp to send.. can you do that in a user script?? as i set them in a user script. and i also use the --delete to delete and files to change.. but i guess thats no good for the reduncy backups is there one better then the other? so i can learn another backup solution.. ive played with the syncthing but i never liked it.. so i just stuck with the rsync but found the issues of the file problem problem why i found some videos messed up on the backup server... so running it like this for now rsync -avzue ssh --partial --append-verify --checksum --progress user@server:/source/ /destination/ will that always check the file. i did try rsync -avzue ssh --checksum --progress user@server:/source/ /destination/ --delete it did go slow but i dont know if it repairs that file that is only 10% copied yet still retians its fully done.. when you do rsync -avzue ssh --partial --append-verify --checksum --progress user@server:/source/ /destination/ and it finishes it when you re run it again will it go slow or will it go fast the next time like without the checksum? like does it keep a record that it fully copied the first time wth the checksum? so that the 2nd time it will just zip through.. as i also backup files to my sisters unraid and uses a 3mbp/s connection. so like i found issues i mentioned over the network and then i also update over slow internet as offsite... so i just trying to find the best of both worlds.. is 1 better over then the other? like i know the snapshots but thats really only on truenas and i dont use it cuz i wanted to be able to add hard drives as i go not need all the hard drives same size at once... ill look into borgbackup and restic... is one better then the other?

is there a Rsync option when copying files to make sure the file is copied correctly? the situation i found is ... if rsyncing script and if i copying a file from 1 server to another and when it hasnt reached that folder yet.. and the file is only transfered say 10% of the file.. then rsync sees it copys the 10% of the file but copys the time and file stamp and pretends the file is completed... so when you re run rsync and the file is completed... rsync skips over that file thinking its done even though its a false flag that file is only 10% done with a stamp of complete... my question is is there a option that verifies all the file bits are actually sent over like a true file integrity check? i currently use ``` rsync -avzue ssh ``` and if say rsync isnt a good file backup solution what is a better file server backup does everyone recommend as i rsync 80tb+ as backups but who knows now if some files never got replicated correctly.. and i looking for a solution if i make a file and it backup and i made a new change a few times like 3 times and backups but i want that first change is there a way to get backup copy of the 3 changes or i need 3 serveers 1 for daily weekly monthy so it has like different backups?

my problem i noticed is i rsync from one server to backup server.. now on the backup server 1 drive failed i had to run the xfs_repair -L after googling. but i didnt know drive lost its partition so its been rysncing to the rest of the drives and filled it up.. as i noticed the server on too long and the rysncing showed out of disk space.. as what i backing up is only 43TB total array is 56TB and when i fixed the drive its 1TB free and its 55TB used not 43TB used... i deleted the lost+found folder but it deleted 2gb.. i ran the rysnc --delete but i guessing duplicated files are now on the other drives cuz of the 1 failure... so say this is my movie.mp4 was on the failed drive but now its on the other drives... buy the rysnc --delete doesnt delete the 2 copies now is there a way to force rysync to delete the duplicate copies.. i tried looking at ubalance but it didnt have any feature like that.. or do you just format all the drives and resync over to get back to 43tb instead.. i guess thats what they call symb links as it points the file to a certain drive but now i have 2 copies of same file... and is there a way to Unraid when a drive fails.. to text me to Send to HOME ASSISTANT... or an email or so.. or is there a way to not allow Rsync to run on the host side when the remote side a Drive failed... is there any options like that out there?

my main server is a ryzen 5800x 8 core cpu and 32gig ecc ram so my drives are like this in the pic.. but i still find is slow as molasses even if you play a video off the array it can be slow even if the drives are spinning and not sleeping so i have 1 drive for Cache sata, 1 drive for Lan Cache, as my internet slow less then 5megbit so i use a lanche.. a download drive for misc downloads and then i move to array games drive i was playing with for network share so you only have to install game 1 location but have a dual boot windows 10 11 and point to it was playing with it and then i have dual 2tb nvms for my unraid files.. iso, vms, appdata as i found plex was always slow and still slow at times... i using a Asus ROG STRIX X570-E motherboard so only has 2 nvmes on there your setup i take it ur using 4 nvmes? and if so how do you do it... and do you know if the iperf test is getting 10 gig cuz i confused about 1.25 and what i get is 1 or its the 9... and ya its so annoying when you copy a file it will even stop the video or it wont even allow a video to start in vlc it will just keep running that yellow line for loading and it may stay stuck 5 min... till the copying is stuck.. is that all due to the copying straight to the array.. i guess truenas doesnt have this issue? but being free wasnt fezable since i needed to buy 1 drive at a time and ended up buying 5 unraid licenses just so i can go 1 2 drives when they affordable and now western digital has 32TB drives could use a couple of those and just use my 14tb drives for something else lol and would mirror 2 sata ssds improve things? or 4 or i seen you can put 2 nvme in 1 single sata slot with the enclosure would that improve things or still capped at 500mbs

ok i just have 1 1TB sata SSD for cache drive is that good enough or do you need to have like 2 to mirror them.. and will that speed up copying too if your copying from a pool drive like a download or a cctv drive pool to the array... as i try to speed up things like with the 10gig nic but i find if i move files 2 moving large files and trying to stream 1 video on the same desktop i didnt think there be any slow downs as i have 10gig nics so it would max out the hard drive speeds.. so when copying files to the array is taxing the system on IO is that the same for like TrueNas with there raid stuff does it get taxed the same or is it different? not like i cant run truenas anyways as i cant afford 3 or 4 large hard drives in one shot so that i cant really test just curious.. so if 2 streams are accessing 1 hard drive do you see chunking? and is there anyway to fix that... do you guys recommend like moving Media Files to a 2nd server just for watching movies tv shows.. and the main server keeps a back up of the videos does the backups of kids files, documents photos etc.. and i tried googling. above about the 10gig stuff i think its saying my 10gig nick only operating at 1gb as the 9Gbit/s is should be least 90Gbits to get close to the 10Gig network so how do you 2 guys setup your media server to get the best out of the system... like all in one i have or like 2 seperate... as i have 5 unraid servers i bought 3 are home servers and 2 are off site at sisters house. this way i have off site backup too.. but always looking to get best speed as i can.. without spending a fortune on needing the same hard drives like with truenas