Hoopster Posted January 29, 2021 Share Posted January 29, 2021 2 minutes ago, stingray060 said: How do you have the email portion done? Here is my backup script set to run as a cron job once a week via the User Scripts plugin: #!/bin/bash #description=This script backs up shares on MediaNAS to BackupNAS #arrayStarted=true echo "Starting Sync to BackupNAS" echo "Starting Sync $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log # Power On BackupNAS ipmitool -I lan -H 192.168.1.16 -U admin -P xxxxxxxx chassis power on # Wait for 3 minutes # echo "Waiting for BackupNAS to power up..." sleep 3m echo "Host is up" sleep 10s # Set up email header echo To: [email protected] >> /boot/logs/cronlogs/BackupNAS_Summary.log echo From: [email protected] >> /boot/logs/cronlogs/BackupNAS_Summary.log echo Subject: MediaNAS to BackupNAS rsync summary >> /boot/logs/cronlogs/BackupNAS_Summary.log echo >> /boot/logs/cronlogs/BackupNAS_Summary.log # Backup Pictures Share echo "Copying new files to Pictures share ===== $(date)" echo "Copying new files to Pictures share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to Pictures share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Pictures.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/Pictures/ [email protected]:/mnt/user/Pictures/ >> /boot/logs/cronlogs/BackupNAS_Pictures.log # Backup Videos Share echo "Copying new files to Videos share ===== $(date)" echo "Copying new files to Videos share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to Videos share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Videos.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/Videos/ [email protected]:/mnt/user/Videos/ >> /boot/logs/cronlogs/BackupNAS_Videos.log # Backup Movies Share echo "Copying new files to Movies share ===== $(date)" echo "Copying new files to Movies share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to Movies share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Movies.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/Movies/ [email protected]:/mnt/user/Movies/ >> /boot/logs/cronlogs/BackupNAS_Movies.log # Backup TVShows Share echo "Copying new files to TVShows share ===== $(date)" echo "Copying new files to TVShows share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to TVShows share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_TVShows.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/TVShows/ [email protected]:/mnt/user/TVShows/ >> /boot/logs/cronlogs/BackupNAS_TVShows.log # Backup OtherVids Share echo "Copying new files to OtherVids share ===== $(date)" echo "Copying new files to OtherVids share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to OtherVids share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_OtherVids.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/OtherVids/ [email protected]:/mnt/user/OtherVids/ >> /boot/logs/cronlogs/BackupNAS_OtherVids.log # Backup FamVideos Share echo "Copying new files to FamVideos share ===== $(date)" echo "Copying new files to FamVideos share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to FamVideos share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_FamVideos.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/FamVideos/ [email protected]:/mnt/user/FamVideos/ >> /boot/logs/cronlogs/BackupNAS_FamVideos.log # Backup Documents Share echo "Copying new files to Documents share ===== $(date)" echo "Copying new files to Documents share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log echo "Copying new files to Documents share ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Documents.log rsync -avu --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x" /mnt/user/Documents/ [email protected]:/mnt/user/Documents/ >> /boot/logs/cronlogs/BackupNAS_Documents.log echo "moving to end ===== $(date)" echo "moving to end ===== $(date)" >> /boot/logs/cronlogs/BackupNAS_Summary.log # Add in the summaries cd /boot/logs/cronlogs/ echo ===== > Pictures.log echo ===== > Videos.log echo ===== > Movies.log echo ===== > TVShows.log echo ===== > OtherVids.log echo ===== > FamVideos.log echo ===== > Documents.log echo Pictures >> Pictures.log echo Videos >> Videos.log echo Movies >> Movies.log echo TVShows >> TVShows.log echo OtherVids >> OtherVids.log echo FamVideos >> FamVideos.log echo Documents >> Documents.log tac BackupNAS_Pictures.log | sed '/^Number of files: /q' | tac >> Pictures.log tac BackupNAS_Videos.log | sed '/^Number of files: /q' | tac >> Videos.log tac BackupNAS_Movies.log | sed '/^Number of files: /q' | tac >> Movies.log tac BackupNAS_TVShows.log | sed '/^Number of files: /q' | tac >> TVShows.log tac BackupNAS_OtherVids.log | sed '/^Number of files: /q' | tac >> OtherVids.log tac BackupNAS_FamVideos.log | sed '/^Number of files: /q' | tac >> FamVideos.log tac BackupNAS_Documents.log | sed '/^Number of files: /q' | tac >> Documents.log # now add all the other logs to the end of this email summary cat BackupNAS_Summary.log Pictures.log Videos.log Movies.log TVShows.log OtherVids.log FamVideos.log Documents.log > allshares.log zip BackupNAS BackupNAS_*.log # Send email of summary of results ssmtp [email protected] < /boot/logs/cronlogs/allshares.log cd /boot/logs/cronlogs mv BackupNAS.zip "$(date +%Y%m%d_%H%M)_BackupNAS.zip" rm *.log #Power off BackupNAS gracefully sleep 30s ipmitool -I lan -H 192.168.1.16 -U admin -P xxxxxxxx chassis power soft I receive an email every Monday morning with the backup summary. Here is what the email summary looks like: Copying new files to Pictures share ===== Mon Jan 25 01:03:12 MST 2021 Copying new files to Videos share ===== Mon Jan 25 01:04:21 MST 2021 Copying new files to Movies share ===== Mon Jan 25 01:04:29 MST 2021 Copying new files to TVShows share ===== Mon Jan 25 01:04:30 MST 2021 Copying new files to OtherVids share ===== Mon Jan 25 01:11:32 MST 2021 Copying new files to FamVideos share ===== Mon Jan 25 01:11:33 MST 2021 Copying new files to Documents share ===== Mon Jan 25 01:11:35 MST 2021 moving to end ===== Mon Jan 25 01:11:41 MST 2021 ===== Pictures Number of files: 211,602 (reg: 211,023, dir: 579) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 1,560,962,991,176 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 131,068 File list generation time: 0.006 seconds File list transfer time: 0.000 seconds Total bytes sent: 3,406,741 Total bytes received: 1,488 sent 3,406,741 bytes received 1,488 bytes 49,039.27 bytes/sec total size is 1,560,962,991,176 speedup is 457,998.27 ===== Videos Number of files: 20,976 (reg: 20,788, dir: 188) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 4,403,940,401,907 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.002 seconds File list transfer time: 0.000 seconds Total bytes sent: 367,020 Total bytes received: 413 sent 367,020 bytes received 413 bytes 43,227.41 bytes/sec total size is 4,403,940,401,907 speedup is 11,985,696.45 ===== Movies Number of files: 1,368 (reg: 872, dir: 496) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 2,576,720,412,648 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.004 seconds File list transfer time: 0.000 seconds Total bytes sent: 64,327 Total bytes received: 651 sent 64,327 bytes received 651 bytes 43,318.67 bytes/sec total size is 2,576,720,412,648 speedup is 39,655,274.29 ===== TVShows Number of files: 4,161 (reg: 3,946, dir: 215) Number of created files: 40 (reg: 39, dir: 1) Number of deleted files: 0 Number of regular files transferred: 39 Total file size: 3,543,239,095,970 bytes Total transferred file size: 45,301,852,400 bytes Literal data: 45,301,852,400 bytes Matched data: 0 bytes File list size: 65,536 File list generation time: 0.009 seconds File list transfer time: 0.000 seconds Total bytes sent: 45,313,076,007 Total bytes received: 1,202 sent 45,313,076,007 bytes received 1,202 bytes 107,504,335.02 bytes/sec total size is 3,543,239,095,970 speedup is 78.19 ===== OtherVids Number of files: 63 (reg: 36, dir: 27) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 83,461,506,455 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.007 seconds File list transfer time: 0.000 seconds Total bytes sent: 2,743 Total bytes received: 38 sent 2,743 bytes received 38 bytes 1,854.00 bytes/sec total size is 83,461,506,455 speedup is 30,011,329.18 ===== FamVideos Number of files: 69 (reg: 34, dir: 35) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 139,897,682,244 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.005 seconds File list transfer time: 0.000 seconds Total bytes sent: 2,849 Total bytes received: 94 sent 2,849 bytes received 94 bytes 1,177.20 bytes/sec total size is 139,897,682,244 speedup is 47,535,739.80 ===== Documents Number of files: 448 (reg: 439, dir: 9) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 933,993,046 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 18,023 Total bytes received: 20 sent 18,023 bytes received 20 bytes 2,775.85 bytes/sec total size is 933,993,046 speedup is 51,764.84 This one is not very exciting. It only had some new recorded TV shows to backup. Quote Link to comment
Hoopster Posted January 29, 2021 Share Posted January 29, 2021 Below is the forum thread that discusses how to set up SSH between the servers so logging in to the backup server does not require any manual intervention for password entry. Perhaps on a local network, you might just eliminate the password on the backup server and not worry about it; however, I keep threatening to move the backup server offsite to my son's house and do the backup over the Internet since we both have lots of bandwidth. In that case, I definitely want it to be SSH protected. Quote Link to comment
stingray060 Posted January 29, 2021 Author Share Posted January 29, 2021 (edited) That is fantastic, Looks beyond my capability, but it would be awesome to have. Thanks for sharing, Edited January 29, 2021 by stingray060 Quote Link to comment
stingray060 Posted January 29, 2021 Author Share Posted January 29, 2021 Strange issue if either of you know the solution, I made a new server connected it all up etc. Its giving me Bond0 not found or something, then giving me IpV4 of 169.254.200.89 which is way out of my range... It is connected and showing transferring on the LAN Led's! Quote Link to comment
Hoopster Posted January 29, 2021 Share Posted January 29, 2021 8 minutes ago, stingray060 said: That is fantastic, Looks beyond my capability, but it would be awesome to have. If the backup server does not have a password, you don't need all the SSH stuff if you feel that is beyond your reach. It took me several days of study and experimentation to figure it out as well, but it has been running for three years as regular as clockwork. Quote Link to comment
Hoopster Posted January 29, 2021 Share Posted January 29, 2021 1 minute ago, stingray060 said: Strange issue if either of you know the solution, I made a new server connected it all up etc. Its giving me Bond0 not found or something, then giving me IpV4 of 169.254.200.89 which is way out of my range... It is connected and showing transferring on the LAN Led's! In Network Settings is bonding enabled? If so, set it to No. The 169..... address it what gets assigned when the server can not get a DHCP address. In this case because of network issues, it appears. Quote Link to comment
stingray060 Posted January 29, 2021 Author Share Posted January 29, 2021 1 minute ago, Hoopster said: In Network Settings is bonding enabled? If so, set it to No. The 169..... address it what gets assigned when the server can not get a DHCP address. In this case because of network issues, it appears. Cant even get onto webui as i cant get it to connect to network. Error on logs is Cannot find device "Bond0" Quote Link to comment
Hoopster Posted January 29, 2021 Share Posted January 29, 2021 Just now, stingray060 said: Cant even get onto webui as i cant get it to connect to network. Error on logs is Cannot find device "Bond0" Duh! Edit network.cfg directly on the flash drive. It is in the config folder. Quote Link to comment
Hoopster Posted January 29, 2021 Share Posted January 29, 2021 Another option is to just delete network.cfg and unRAID will recreate it on boot with defaults which is with bonding disabled. Quote Link to comment
stingray060 Posted January 29, 2021 Author Share Posted January 29, 2021 4 minutes ago, Hoopster said: Another option is to just delete network.cfg and unRAID will recreate it on boot with defaults which is with bonding disabled. I opened up the network config and it had the defaults down, i set bonding to "no" and tried again, still gives the same error. Quote Link to comment
Hoopster Posted January 29, 2021 Share Posted January 29, 2021 5 minutes ago, stingray060 said: I opened up the network config and it had the defaults down, i set bonding to "no" and tried again, still gives the same error. Do you have a keyboard and monitor connected to the server? Do you ever get a command line? If so, type 'diagnostics' at the command line and post them here. If diagnostics cannot be written to the flash drive, your flash drive has problems and could be the cause of the problems. Quote Link to comment
stingray060 Posted January 29, 2021 Author Share Posted January 29, 2021 So, referring back the my current system with all the data loss. I have 4TB of empty space and data on google cloud i want to download onto it. What is the best way of accessing a command line to shift it back over? Im familiar with the gsutil commands, just not sure how i go about getting the command line to enter them. Quote Link to comment
stingray060 Posted January 29, 2021 Author Share Posted January 29, 2021 22 minutes ago, Hoopster said: Do you have a keyboard and monitor connected to the server? Do you ever get a command line? If so, type 'diagnostics' at the command line and post them here. If diagnostics cannot be written to the flash drive, your flash drive has problems and could be the cause of the problems. Just a keyboard, I can get to diagnostics, and it says generated successfully, but when i plug back into pc, there is nothing there, Over the last couple of minutes i tried using another USB, but it said it was not bootable, the USB's are brand new, so not sure why that is. I put my old one back in that was generating the error to get the diagnostics, but failed Quote Link to comment
Hoopster Posted January 29, 2021 Share Posted January 29, 2021 4 minutes ago, stingray060 said: Just a keyboard, I can get to diagnostics, and it says generated successfully, but when i plug back into pc, there is nothing there, Over the last couple of minutes i tried using another USB, but it said it was not bootable, the USB's are brand new, so not sure why that is. I put my old one back in that was generating the error to get the diagnostics, but failed How are you creating the flash drive? USB Creator or manually? On the unbootable USB stick try running the appropriate make_bootable for your OS (make_bootable.bat for Windows). If on Windows, run it as administrator. Are you inserting the USB flash drive in a USB 2 port? They have fewer problems than with USB 3 ports with some flash drives. I doubt this is the issue, just a question for another data point. Quote Link to comment
trurl Posted January 29, 2021 Share Posted January 29, 2021 18 minutes ago, stingray060 said: gsutil Don't know anything about that, but according to your diagnostics you should be able to see those in /mnt/disks, which is the same place Unassigned Devices mounts things. Can you see anything like that in Main - Unassigned Devices? I always just use Midnight Commander to work with files mounted on the server. It is a built-in textGUI you can launch from the command line by entering mc. It's pretty easy to use and more can be found about it with Google. Quote Link to comment
stingray060 Posted January 29, 2021 Author Share Posted January 29, 2021 2 minutes ago, Hoopster said: How are you creating the flash drive? USB Creator or manually? On the unbootable USB stick try running the appropriate make_bootable for your OS (make_bootable.bat for Windows). If on Windows, run it as administrator. Are you inserting the USB flash drive in a USB 2 port? They have fewer problems than with USB 3 ports with some flash drives. I doubt this is the issue, just a question for another data point. I have gone back to my original USB which i can boot from, just not get bond0. Yes on all devices, i was using the usb creation tool, and the make bootable .bat. I did try a manual file install, still no luck. Not sure which port is which, i think i have tried it in them all now. Quote Link to comment
Hoopster Posted January 29, 2021 Share Posted January 29, 2021 1 hour ago, stingray060 said: I have gone back to my original USB which i can boot from, just not get bond0. Yes on all devices, i was using the usb creation tool, and the make bootable .bat. I did try a manual file install, still no luck. Not sure which port is which, i think i have tried it in them all now. What version of unRAID are you using and what is the motherboard and NIC? Diagnostics would tell me that but you can't get them. Some people get similar errors with newer hardware and stable unRAID (6.8.3) but running the 6.9.0 RC works. Others get the issue because of flash drive problems. (sorry for the delayed response. It was early afternoon here and I had a work meeting even though I mostly just pretend to work 😁) Quote Link to comment
stingray060 Posted January 29, 2021 Author Share Posted January 29, 2021 2 minutes ago, Hoopster said: What version of unRAID are you using and what is the motherboard and NIC? Diagnostics would tell me that but you can't get them. Some people get similar errors with newer hardware and stable unRAID (6.8.3) but running the 6.9.0 RC works. Others get the issue because of flash drive problems. (sorry for the delayed response. It was early afternoon here and I had a work meeting even though I mostly just pretend to work 😁) Gigabyte GA-G81M-S2H. no NIC. You'll never believe how shocking my day has just got. I didnt realise my server was left running over night, in maintenance mode, so my rclone script continued to run, matching my source files with the google cloud files. Deleting all my cloud backups. Fucking Great......... Quote Link to comment
Hoopster Posted January 29, 2021 Share Posted January 29, 2021 7 minutes ago, stingray060 said: Gigabyte GA-G81M-S2H OK, that is an older board with a built-in Realtek NIC. It should work as Realtek drivers for Linus/unRAID have improved a lot lately. The problem is that without diagnostics (or at least the syslog) we can't see what is happening when unRAID tries to initialize the NIC and load drivers. If you happen to have a PCIe NIC laying around you could put in the server (preferably with an Intel chipset) we could at least determine if the NIC is the problem. Sorry about the cloud data loss. That sucks. I always pull the Ethernet cable when I suspect data issues just for that reason. CrashPlan backup is constantly running in the background and I don't want to think about it so I just pull the cable. That way it can't get propagated to the backup server either even though that is only once a week. Quote Link to comment
stingray060 Posted January 29, 2021 Author Share Posted January 29, 2021 1 minute ago, Hoopster said: OK, that is an older board with a built-in Realtek NIC. It should work as Realtek drivers for Linus/unRAID have improved a lot lately. The problem is that without diagnostics (or at least the syslog) we can't see what is happening when unRAID tries to initialize the NIC and load drivers. If you happen to have a PCIe NIC laying around you could put in the server (preferably with an Intel chipset) we could at least determine if the NIC is the problem. Sorry about the cloud data loss. That sucks. I always pull the Ethernet cable when I suspect data issues just for that reason. CrashPlan backup is constantly running in the background and I don't want to think about it so I just pull the cable. That way it can't get propagated to the backup server either even though that is only once a week. I'm having a nightmare, i thought i had fixed my new server to start repairing the lost files, but i clearly haven't, it is running 6.9.0-beta46. Is this a bad idea? Sorry to go around in circles with you, but i thought i had this damn thing fixed, which is the most important thing, give me a minute and il see if i can send you a diagnostic. Quote Link to comment
Hoopster Posted January 29, 2021 Share Posted January 29, 2021 1 minute ago, stingray060 said: I'm having a nightmare, i thought i had fixed my new server to start repairing the lost files, but i clearly haven't, it is running 6.9.0-beta46. Is this a bad idea? Sorry to go around in circles with you, but i thought i had this damn thing fixed, which is the most important thing, give me a minute and il see if i can send you a diagnostic. Take your time, no hurry. Get your first issue resolved and then we can talk about the second server boot issue. I should pretend to do some more work anyway. Quote Link to comment
stingray060 Posted January 29, 2021 Author Share Posted January 29, 2021 4 minutes ago, Hoopster said: Take your time, no hurry. Get your first issue resolved and then we can talk about the second server boot issue. I should pretend to do some more work anyway. Im just re-installing my unraid server USB, along with going back to 6.8.3, not sure why i was on beta, but not taking any risks from now on. Fresh start, is the plan. If you have any advice in regards to first setup settings, im not really aware of anything important. Quote Link to comment
trurl Posted January 29, 2021 Share Posted January 29, 2021 1 hour ago, stingray060 said: it is running 6.9.0-beta46 There has never been a version named that. Looks like you fat-fingered one key off. Your previous diagnostics said beta35. Quote Link to comment
stingray060 Posted January 30, 2021 Author Share Posted January 30, 2021 8 minutes ago, trurl said: There has never been a version named that. Looks like you fat-fingered one key off. Your previous diagnostics said beta35. Just goes to show how tired i am!! Got home at 17:30 been trying to fix it ever since, its 12pm now time for bed!!! Quote Link to comment
stingray060 Posted January 30, 2021 Author Share Posted January 30, 2021 @trurl How do i get data of drives if they refuse to allow me to mount? http://olii.co.uk/i/k50.png Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.