tr0910 Posted December 9, 2016 Share Posted December 9, 2016 (edited) Suppose that you have more than one server, and that you want to keep a backup of your unRaid server. Suppose that you have tried the sync options, but find that they are a bit finicky and unreliable. This is a totally reliable bullet proof / stable way to keep your backup server fresh. All you need is 2 unRaid servers connected to the internet, and we use secure keyed ssh to make the rsync commands secure and optimized for transiting the internet. What I am using is a unRaid 6.2.4 server at another site with IPMI support. This server starts up via IPMI accepts the backup, then shuts down via IPMI. I added a raspberry PI at the destination to act as an always on computer to accept the IPMI commands and to start the unRaid backup server. I also added pivpn.io to this unit as well. Very easy to do and now I have a secure OpenVPN server installed without needing to fight with a limited AT&T router. Some ports are forwarded in the router for the SSH and the VPN. SSH is set up for secure keyfile access and a different port to keep the script kiddies shooed away. The raspberry pi and the IPMI is strictly optional and add some nice touches, but are in no ways required. This is an attempt to document the process of setting up a secure way to move data efficiently across town or across the world. I have used rsync locally for years and it has moved petabytes of data reliably for me. But we need to have offsite backups, and with Google Fiber I should be able to quit doing the sneakernet dance.... 1. First we need to set up SSH-Keys which permit rsync to be scripted and secure over the internet. Thanks to this site. http://troy.jdmz.net/rsync/index.html I have modified his instructions to make them unRaid friendly so you can just follow along To start with, we have 2 unRaid 6.2 or 6.3 servers on the same network to get the ssh stuff set up securely. (Having the servers on the same network is not needed for advanced users. But in order to keep it more simple, I have assumed this to simplify the process for those following along.) First, I'll define some variables. My server is called Tower and my remote server has a local ip address of 192.168.13.84 In my explanation, I will be synchronizing files (copying only new or changed files), and I will be starting this process from the host I want to copy things to. In other words, I will be syncing files from /remote/dir/ on remotehost (192.168.13.84), as remoteuser (root), to /this/dir/ on thishost, as thisuser. I want to make sure that 'rsync' over 'ssh' works at all before I begin to automate the process, so I test it first as thisuser: $ rsync -avz -e ssh [email protected]:/remote/dir /this/dir/ The authenticity of host '192.168.13.221 (192.168.13.84)' can't be established. ECDSA key fingerprint is SHA256:30e/zRmjYVJLB4+5cBc2GjMqEpP7O3sUFcYlzQNYPtY. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.13.84' (ECDSA) to the list of known hosts. [email protected]'s password: and type in root@remotehost's password when prompted. 2. Configuring thishost If that all worked out, or I eventually made it work, I am ready for the next step. I need to generate a private/public pair of keys to allow a 'ssh' connection without asking for a password. I generate the key I will use on thishost (as thisuser): $ ssh-keygen -t rsa -b 2048 -f /root/.ssh/Tower-rsync-key Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): [press enter here] Enter same passphrase again: [press enter here] I use scp to get the file over to remotehost: $ scp /root/.ssh/Tower-rsync-key.pub [email protected]:/root/ Just get the file over there some way if you don't like scp. 3. Configuring remotehost I 'ssh' over to remotehost: $ ssh [email protected] root@remotehost's password: [type correct password here] $ echo I am now $USER at $HOSTNAME I am now root at remotehost and need to do some work. I need to make sure I have the directory and files I need to authorize connections with this key [3]: $ if [ ! -d .ssh ]; then mkdir .ssh ; chmod 700 .ssh ; fi $ mv Tower-rsync-key.pub .ssh/ $ cd .ssh/ $ if [ ! -f authorized_keys ]; then touch authorized_keys ; chmod 600 authorized_keys ; fi $ cat Tower-rsync-key.pub >> authorized_keys Now the key can be used to make connections to this host, but these connections can be from anywhere (that the ssh daemon on remotehost allows connections from) and they can do anything (that root can do). We can change this later Edit I didn't have the above permissions correct. Make sure that the files in the .ssh directory have the exact permissions below, and the .ssh directory has 700 permissions chmod 700 .ssh/ chmod 600 .ssh/Tower-rsync-key.pub chmod 644 .ssh/authorized_keys 4. Troubleshooting Now that I have the key with no password in place and configured, I need to test it out. I exit from the ssh session to remotehost and try: $ rsync -avz -e "ssh -i /root/.ssh/Tower-rsync-key" /boot/config/ [email protected]:/boot/fromTower/ Or we can move data the other direction like this rsync -avz -e "ssh -i /root/.ssh/Tower-rsync-key" [email protected]:/boot/config/ /boot/fromRemoteHost/ If this works, you are finished the first step. Clean up your flash drives on both servers and remove the garbage you just copied there to test that it works. Next we need to make these changes persistent across reboots, and then make it work through firewalls and finally set it all up in cron securely. 5a.Make the changes persistent across reboots First you need to save your newly created keys from /root/.ssh on the local and remote server to your flash drive (I used /boot/config/sshroot so they will survive a reboot. Use mc or commandline as follows cp /root/.ssh/* /boot/config/sshroot/ Then you need to modify your /boot/config/go file as follows # Copy files back to /root/.ssh folder and set permissions for key files and known_hosts and authorized_keys cd /root/.ssh cp /boot/config/sshroot/* /root/.ssh/ chmod 600 * This copies the files back to their correct location after a reboot. 5b. Add the SSH config plugin for improving security. (Optional, can be done later) Install the ssh config plugin from community application. Even after doing the above, I still needed to do more to secure my ssh login. Via the plugin, I shut down password logins via ssh, change the port of the ssh server to something other than 22 to cut down on the hackers trying to get in and keep my log file from exploding. This simplifies the configuration of the ssh server. 6. Are we golden?? Now unplug the server destined for remote use and move it to it's final destination. Find out it's WAN ip address, and forward the SSH port (22) or (other port you changed to in the SSH config plugin) in your router to the unRaid local ip address. For simplicity sake the following example assumes that you are still using port 22. To test if you have a connection try: ssh -i /root/.ssh/Tower-rsync-key [email protected] uptime 20:04:23 up 1:30, 1 user, load average: 0.05, 0.05, 0.00 If you get a response without a password challenge you are golden and ready for an rsync test. rsync -avz -e "ssh -i /root/.ssh/Tower-rsync-key" /mnt/disk2/Documents/ [email protected]:/mnt/disk1/fromTower/ sent 2,105,771,960 bytes received 4,899 bytes 3,443,625.28 bytes/sec total size is 2,196,999,197 speedup is 1.04 It's working. Over gbit lines you can see that the we aren't getting anywhere the full 100 mb/sec transfer rates that we are looking for. Further tests showed this to be a worst possible case. Single ISO files would tranfer as fast at 25mb/sec, but there is significant variability to the sustained speed. More troublesome is the speed at which is goes when there is nothing to transfer. If everything is up to date, it has big problems and never completes the sync. This kept me puzzling for a few days. I needed it to reliably complete sync even if nothing needed to be copied. From this site we have a faster way. 200,000 files synced in seconds not hours. (assuming the files already match, nothing to copy) https://gist.github.com/KartikTalwar/4393116 This is the process we end up using after a bit of testing. It reliably uses the bandwidth of the internet and it just works. rsync -avu --numeric-ids --progress -e "ssh -i /root/.ssh/Tower-rsync-key -T -o Compression=no -x " /mnt/disk1/ [email protected]:/mnt/disk1/ Edited May 11, 2017 by tr0910 Fixed errors in permissions 4 1 2 Quote Link to comment
tr0910 Posted December 9, 2016 Author Share Posted December 9, 2016 I didn't want to leave this offsite backup server running 24x7 as it only needs to wake up at most once per day. The problem was that the offsite location was using the brain dead AT&T router and once the unRaid server was shut down, I would have to ask somebody to turn it on to accept a backup. This wasn't real cool Enter a raspberry pi running piVPN. http://www.pivpn.io/ In less than 2 hours from my first experience with the raspberry it was running as a dedicated vpn at this offsite location. Highly recommended... Duckdns is handling the handling the fact that AT&T changes the IP at the destination. Edit: After using the VPN for a number of days, I finally just went pure SSH by: ssh -i /root/.ssh/pi_key [email protected] -p 56754 ipmitool -I lan -H 192.168.1.178 -U user -P password chassis power on . . bunch of rsync commands over ssh . . ssh -i /root/.ssh/pi_key [email protected] -p 56754 ipmitool -I lan -H 192.168.1.178 -U user -P password chassis power soft IPMI on this Supermicro X9SCM is able to shut unRaid down cleanly via this method. The VPN is still nice to have and is used to troubleshoot any problems. 1 1 Quote Link to comment
tr0910 Posted December 10, 2016 Author Share Posted December 10, 2016 UDT and UDR transport utilities promise to speed up rsync immensely. Has anyone tried this? http://smartaleksolutions.com/blog/2015/10/5/udt-and-udr-transport-utilities Does anyone have the ability to create a binary of UDT for slackware? https://github.com/LabAdvComp/UDR http://udt.sourceforge.net/ https://lists.samba.org/archive/rsync/2013-February/028178.html Another option might be Tsunami, but Tsunami UDP transfers only files and doesn’t do directories/ subdirectories, we need to tar them all up as one single tar file (additional storage capacity needs to be taken into consideration). https://bluepiit.com/blog/tsunami-udp-protocol-installation-setup-and-limitations/ Update, no solution was found here, but for now I don't really need it. Quote Link to comment
tr0910 Posted January 12, 2017 Author Share Posted January 12, 2017 Followup, this is now working quite satisfactorily. The backup is from disk to disk, and for disks with no activity, there is about 60 seconds of activity, then it moves on to another disk. Given the 15mbit AT&T Uverse internet limitation, the throughput is about 3mbyte/second during off time, with much lower speed noted during peak time. I have tried to create some problems by sending a large backup over while someone is trying to do some gaming and internet browsing at the same time. Surprisingly, the gamers and the browsers did not notice the fact that their internet was slow. This was without any attempt to throttle the rsync during the transfer. I have a raspberry pi that creates a vpn into the second location. There is AT&T residential Uverse internet as this receiving location and Google Fiber on the sending location. The VPN is only used to send an IPMI signal to the Supermicro X9SCM motherboard that it is time to turn on. After that, the rsync via SSH as outlined above is all that is required. The openvpn plugin of Peter's is used on the unRaid side, and piVPN is the build on the raspberry pi. Quote Link to comment
tr0910 Posted January 31, 2017 Author Share Posted January 31, 2017 Pure SSH now is being used for the IPMI portion as well. ssh -i /root/.ssh/pi_key [email protected] -p 56754 ipmitool -I lan -H 192.168.1.178 -U user -P password chassis power on This is getting real smooth and reliable..... Quote Link to comment
kizer Posted February 2, 2017 Share Posted February 2, 2017 Thanks for the step by step. I'm starting to journey down this ssh/rsync world myself. Quote Link to comment
skulker Posted February 22, 2017 Share Posted February 22, 2017 Thank you for the step-by-step. I think I'm going to give this a try. Quote Link to comment
teamhood Posted February 26, 2017 Share Posted February 26, 2017 This looks great and is on my to-do list. Following! Quote Link to comment
tr0910 Posted February 26, 2017 Author Share Posted February 26, 2017 (edited) On 2/25/2017 at 8:03 PM, teamhood said: This looks great and is on my to-do list. Following! Looking forward to your refinements... So far I've only got this working in USA where CG-NAT isn't messing things up. Some countries in Europe and Asia hide your IP behind several levels of NAT, and getting rsync through this nasty ipv4 abortion is more complex. What I have had to do to get around CGNAT is change the machine I am rsyncing from. I have to sync from behind the great firewall of China where the IP addresses are being CGNATified, and rsync to a destination where the IP address is more stable, (such as the USA). Because rsync is flexible you can initiate the connection from either end and successfully get the transfer to work. rsync -avu --stats --log-file=/boot/logs/cronlogs/rsync.log -e "ssh -T -o Compression=no -x -p 51842" /mnt/user/Documents/ [email protected]:/mnt/user/Documents/ This combination of flags seems to work with with transferring multimegabyte incompressible files across the world So what do you have to do if both of your IP addresses are CGNAT? I don't have a good answer yet for that. Edited March 10, 2017 by tr0910 Quote Link to comment
skulker Posted March 6, 2017 Share Posted March 6, 2017 Please - I need a little help. I had everything working as expected up to; On 12/9/2016 at 1:30 PM, tr0910 said: 5a.Make the changes persistent across reboots - the hard way Now I'm lost. When I make the change to the go file it creates the directory but does not copy the files. I am not a linux guru but I think there are some ;'s missing and the syntax is not quite correct. But - as I say - I am very low on the linux command line learning curve. I have copied the files manually but I am now being asked for a password. So I tried the easy way... I am also very new to unRAID v6. As a matter of fact I just upgraded from v5 so I could try this. So this plugin stuff is a mystery to me. I did install the ssh plugin but I have no idea how to configure it properly to make it persistent to reboots. Would someone be able to guide me please? Thanks Quote Link to comment
tr0910 Posted March 7, 2017 Author Share Posted March 7, 2017 If using the plugin, is automatic reinstalled at reboot. And just make sure your key files are in the right folder /config/ssh then on reboot they get copied to the respective dir on the server. via mobile Quote Link to comment
skulker Posted March 7, 2017 Share Posted March 7, 2017 17 hours ago, tr0910 said: If using the plugin, is automatic reinstalled at reboot. And just make sure your key files are in the right folder /config/ssh then on reboot they get copied to the respective dir on the server. Perhaps I am just plain stupid; Everything is working perfectly until I reboot. I have installed the ssh plugin and copied the files from /root/.ssh/ to /boot/config/ssh/ After a reboot /root/.ssh/ is empty and a password is required. Thanks for any guidance on what I am doing wrong. Quote Link to comment
tr0910 Posted March 7, 2017 Author Share Posted March 7, 2017 The first and main thing is that it works. The copying the keys into the right place at boot is minor. Are you using the SSH config plugin?via mobile Quote Link to comment
skulker Posted March 7, 2017 Share Posted March 7, 2017 3 hours ago, tr0910 said: Are you using the SSH config plugin? Yes. But when I copy the keys manually this no longer seems to work and asks for a password. So I am doing something incorrectly. It's very frustrating. Quote Link to comment
ken-ji Posted March 8, 2017 Share Posted March 8, 2017 SSH keys have a very strict file and directory permission. Are you ensuring the correct permissions? chmod 700 .ssh/ chmod 600 .ssh/id_rsa chmod 644 .ssh/authorized_keys Quote Link to comment
skulker Posted March 8, 2017 Share Posted March 8, 2017 35 minutes ago, ken-ji said: SSH keys have a very strict file and directory permission. Are you ensuring the correct permissions? Thank you for the reply ken-ji No - I am not ensuring anything. that is for two reasons: 1. I know very little LINUX (I grew up on OpenVMS if you know what that is) and 2. tr0910 wrote the following; 23 hours ago, tr0910 said: If using the plugin, is automatic reinstalled at reboot. And just make sure your key files are in the right folder /config/ssh then on reboot they get copied to the respective dir on the server. By "automatic" I assume these things were taken care of. Perhaps I should not assume. In any event the .ssh/ folder is empty after a reboot. And yes I have the key files in the /config/ssh folder. <sigh> I have so much to learn - and I am an old dog trying new tricks Quote Link to comment
skulker Posted March 8, 2017 Share Posted March 8, 2017 I think I got it. I was reading a post from ken-ji in another topic https://forums.lime-technology.com/topic/50038-ssh-support-in-v6/ Then it hit me. I should not be using /root/.ssh/ but instead /etc/ssh/ - no? On 12/9/2016 at 1:30 PM, tr0910 said: ssh -i /root/.ssh/Tower-rsync-key [email protected] uptime requires a password because /root/.ssh/ is empty But ssh -i /etc/ssh/Tower-rsync-key [email protected] uptime does not require a password - even after reboots. Do I have this correct? Quote Link to comment
tr0910 Posted March 8, 2017 Author Share Posted March 8, 2017 Yes that is where the plugin puts it on reboot. Sounds like you got it Once you have it solid you can use the plugin to change the SSH port, and dis allow password login to keep the script kiddies and the CIA out... I was getting several login attempts per second showing in my logs, until I did this. The plugin helps manage this. via mobile Quote Link to comment
ken-ji Posted March 8, 2017 Share Posted March 8, 2017 (edited) Well your post mentioned copying the keys manually, so I pointed out the permissions are probably wrong. The USB has no concept of permissions since its using the "universal" FAT/FAT32 filesystem. When copied to unRAID RAM disk (or any Linux filesystem), it gets the equivalent allow anybody r/w access, which is a big no no for SSH keys (you don't want other people coipying or changing your keys) Also you can setup a default config file called ssh_config (/boot/config/ssh - so it gets copied in) IdentityFile /etc/ssh/myprivatekey So you don't have to always specify the keyfile (If you are only using one key file) or Host server1 Hostname 11.22.33.44 IdentityFile /etc/ssh/server1.key Host server2 Hostname 55.66.77.88 IdentityFile /etc/ssh/server2.key if you have more than one server Edited March 8, 2017 by ken-ji Quote Link to comment
tr0910 Posted March 10, 2017 Author Share Posted March 10, 2017 Let me know if the original post needs updated to fix errors or make it easier to understand. Quote Link to comment
skulker Posted March 10, 2017 Share Posted March 10, 2017 1 hour ago, tr0910 said: Let me know if the original post needs updated to fix errors or make it easier to understand. I hit another snag and just haven't had time to do anymore. I thought I had it working. I could ssh and rsync from the backup server using the keys and no password. But then I decided to try to reboot my primary (the one I will backup). Now no keys work. Not from the primary nor the backup server. Back to the drawing board and I will post when life lets me try some things. Thanks Quote Link to comment
skulker Posted March 10, 2017 Share Posted March 10, 2017 I have a few free moments to try again - from scratch. I deleted the .ssh/ folder and all my created files from /config/ssh/ and /etc/ssh/ I followed the steps and after... On 12/9/2016 at 1:30 PM, tr0910 said: 4. Troubleshooting ...everything was working as expected. I have the SSH Config app installed so I copied all files from /root/.ssh/ to /boot/config/ssh on both servers After I reboot "thishost" (as described above) everything work fine. I rebooted a couple of times and everything is good. After I reboot "remotehost" it no longer works. Yes I am still entering the commands on thishost. So - what the heck am I doing wrong? Quote Link to comment
skulker Posted March 10, 2017 Share Posted March 10, 2017 (edited) Got it. I suspect that tl0910 hasn't rebooted remotehost yet - or I am doing something wrong. After reading this from ken-ji On 3/7/2017 at 7:38 PM, ken-ji said: SSH keys have a very strict file and directory permission. Are you ensuring the correct permissions? I did a bit of experimenting and found the answer. My go file now looks like this on remotehost (no changes on the go file on thishost): # umask for root .ssh setup with known hosts and authorized keys umask 077 # Copy the files and set the permission cp /etc/ssh/Tower-rsync-key.pub /root/.ssh/ cp /etc/ssh/authorized_keys /root/.ssh/ chmod 700 /root/.ssh/ #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & I have now successfully rebooted remotehost and thishost and things work. Next I am going to congiure ssh for another and disallow root logins - but one step at a time. When I finish I'llpm tl0910 with what I did and he can edit his original post if I am doing this correctly. Thanks again ken-ji and tl0910 - I continue to learn. Edited March 10, 2017 by skulker wrong pub key file name Quote Link to comment
tr0910 Posted March 11, 2017 Author Share Posted March 11, 2017 (edited) 2 hours ago, skulker said: Got it. I suspect that tl0910 hasn't rebooted remotehost yet - or I am doing something wrong. After reading this from ken-ji Next I am going to congiure ssh for another and disallow root logins - but one step at a time. When I finish I'llpm tl0910 with what I did and he can edit his original post if I am doing this correctly. Thanks again ken-ji and tl0910 - I continue to learn. My remotehost gets started via ipmi then once it is up, we start the the rsync process. So I know that my remotehost is restarting fine. I likely have other things wrong but that is why we are all testing each others scripts. Here is my script run from ThisHost with the backup destination RemoteHost across the city or halfway around the world. This script adds a RaspBerry pi at the RemotePlace to SSH into before the RemoteHost is turned on. The only requirement is that the RaspBerry pi being on at all times. Everything else is controlled by the script. This code is very much beginners work cut and pasted from various places on the net. The scripting gurus could likely cut 75% of this code. #!/bin/bash ################################################################### # We are using an always on Raspberry Pi at the RemoteHost Place # The remote host is turned off when the script begins and we use the pi to turn on # the RemoteHost via IPMI. Why don't we just forward the port of the IPMI to the WWW? # Because it isn't as secure as coming via keyed SSH to the pi and then connecting to the # IPMI interface. # # Once the RemoteHost is awake and ready to rock and roll, the rsync begins #################################################################### echo "Starting Sync to RemoteHost at RemoteHost's Place" #echo "Starting Sync `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log # Start up RemoteHost via ipmi note IPMI address is different from RemoteHost address as this is the IPMI address of remotehost ssh -i /root/.ssh/RemoteHost_pi_key [email protected] -p 26754 ipmitool -I lan -H 192.168.1.178 -U ADMIN -P ADMIN chassis power on # Wait for a few minutes - try 5 times echo "waiting for RemoteHost to come online..." count=1 sleep 180 # Check if host online RemoteHost is 179 HOSTS="192.168.1.179" for myHost in $HOSTS do while [ $count -le 5 ] do ssh -i /root/.ssh/RemoteHost_pi_key [email protected] -p 26754 ping -c 1 "$myHost" > /dev/null if [ "$?" -eq 0 ] ; then echo "Hosts up" #============================================================================== sleep 30 # Set up email header echo To: [email protected] >> /boot/logs/cronlogs/RemoteHostS1summary.log echo From: [email protected] >> /boot/logs/cronlogs/RemoteHostS1summary.log echo Subject: RemoteHost rsync summary >> /boot/logs/cronlogs/RemoteHostS1summary.log echo >> /boot/logs/cronlogs/RemoteHostS1summary.log # Backup Disk 1 getting files from yesterday echo "moving to Disk1 ===== `date`" echo "moving to Disk1 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log echo "moving to Disk1 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1disk1.log # The first rsync group of files have dates in the file names and are selected based on yesterdays date # after this we simply rsync disk2 to disk2, disk3 to disk3 etc... dte=$(date -d "yesterday 13:00 " '+%Y-%m-%d') src="/mnt/disk1/downloads/ftp_dump/usa_newtheme/backup_"$dte"*" dest="[email protected]:/mnt/disk10/downloads/ftp_dump/usa_newtheme/" rsync -avu --stats -e "ssh -i /root/.ssh/Kim-rsync-key -T -o Compression=no -x -p31798" $src $dest >> /boot/logs/cronlogs/RemoteHostS1disk1.log # Backup Disk 2 echo "moving to Disk2 ===== `date`" echo "moving to Disk2 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log echo "moving to Disk2 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1disk2.log rsync -avu --stats --exclude=Z_K1zq* --exclude=.Recyc* -e "ssh -i /root/.ssh/Kim-rsync-key -T -o Compression=no -x -p 31798" /mnt/disk2/ [email protected]:/mnt/disk2/ >> /boot/logs/cronlogs/RemoteHostS1disk2.log # Backup Disk 3 echo "moving to Disk3 ===== `date`" echo "moving to Disk3 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log echo "moving to Disk3 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1disk3.log #rsync -avu --stats --progress /mnt/disk3/ /mnt/RemoteHostS1disk3/ >> /boot/logs/cronlogs/RemoteHostS1disk3.log # Backup Disk 4 echo "moving to Disk4 ===== `date`" echo "moving to Disk4 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log echo "moving to Disk4 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1disk4.log rsync -avu --stats --exclude=.Recyc* -e "ssh -i /root/.ssh/Kim-rsync-key -T -o Compression=no -x -p 31798" /mnt/disk4/ [email protected]:/mnt/disk4/ >> /boot/logs/cronlogs/RemoteHostS1disk4.log # Backup Disk 5 echo "moving to Disk5 ===== `date`" echo "moving to Disk5 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log echo "moving to Disk5 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1disk5.log rsync -avu --stats --exclude=.Recyc* -e "ssh -i /root/.ssh/Kim-rsync-key -T -o Compression=no -x -p 31798" /mnt/disk5/ [email protected]:/mnt/disk5/ >> /boot/logs/cronlogs/RemoteHostS1disk5.log # Backup Disk 6 echo "moving to Disk6 ===== `date`" echo "moving to Disk6 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log echo "moving to Disk6 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1disk6.log rsync -avu --stats --exclude=.Recyc* -e "ssh -i /root/.ssh/Kim-rsync-key -T -o Compression=no -x -p 31798" /mnt/disk6/ [email protected]:/mnt/disk6/ >> /boot/logs/cronlogs/RemoteHostS1disk6.log # Backup Disk 7 echo "moving to Disk7 ===== `date`" echo "moving to Disk7 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log echo "moving to Disk7 ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1disk7.log rsync -avu --stats --exclude=.Recyc* -e "ssh -i /root/.ssh/Kim-rsync-key -T -o Compression=no -x -p 31798" /mnt/disk7/ [email protected]:/mnt/disk7/ >> /boot/logs/cronlogs/RemoteHostS1disk7.log echo "moving to end ===== `date`" echo "moving to end ===== `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log # Add in the summaries cd /boot/logs/cronlogs/ echo = > disk1.log echo = > disk1.log echo ===== > disk2.log echo ===== > disk3.log echo ===== > disk4.log echo ===== > disk5.log echo ===== > disk6.log echo ===== > disk7.log echo Disk1 >> disk1.log echo Disk2 >> disk2.log echo Disk3 >> disk3.log echo Disk4 >> disk4.log echo Disk5 >> disk5.log echo Disk6 >> disk6.log echo Disk7 >> disk7.log tac RemoteHostS1disk1.log | sed '/^Number of files: /q' | tac >> disk1.log tac RemoteHostS1disk2.log | sed '/^Number of files: /q' | tac >> disk2.log tac RemoteHostS1disk3.log | sed '/^Number of files: /q' | tac >> disk3.log tac RemoteHostS1disk4.log | sed '/^Number of files: /q' | tac >> disk4.log tac RemoteHostS1disk5.log | sed '/^Number of files: /q' | tac >> disk5.log tac RemoteHostS1disk6.log | sed '/^Number of files: /q' | tac >> disk6.log tac RemoteHostS1disk7.log | sed '/^Number of files: /q' | tac >> disk7.log # Check to see if server shuts down # Shut down RemoteHost via IPMI using a soft shutdown ssh -i /root/.ssh/RemoteHost_pi_key [email protected] -p 56754 ipmitool -I lan -H 192.168.1.178 -U ADMIN -P ADMIN chassis power soft thisString="something is on" # We are looking for this "off" string to come from the IPMI interface searchString="off" for i in {1..30} do echo "Checking to see if Server is down $i times" ssh -i /root/.ssh/RemoteHost_pi_key [email protected] -p 26754 ipmitool -I lan -H 192.168.1.178 -U ADMIN -P ADMIN chassis power status > /root/status.txt while read newstring; do thisString=$newstring echo $thisString after $i times >> RemoteHostS1summary.log echo $thisString # thisString should have "Chassis power is off" once shut down done < /root/status.txt case $thisString in # match exact string "$searchString") echo yep, it matches exactly;; # match start of string "$searchString"*) echo yep, it matches at the start ;; # match end of string *"$searchString") break ;; # searchString can be anywhere in thisString *"$searchString"*) echo yep, it matches in the middle somewhere ;; *) echo Nope still running ;; esac sleep 60 done # now add all the other logs to the end of this email summary cat RemoteHostS1summary.log disk1.log disk2.log disk3.log disk4.log disk5.log disk6.log disk7.log > diskall.log zip -m RemoteHostS1 RemoteHostS1*.log # Send email of summary of results ssmtp [email protected] < /boot/logs/cronlogs/diskall.log cd /boot/logs/cronlogs mv RemoteHostS1.zip "`date +%Y%m%d_%H%M`_RemoteHostS1.zip" rm disk*.log # Wait for 5 minutes #sleep 300 count=6 else if [ $count -eq 5 ] then echo To: [email protected] >> /boot/logs/cronlogs/RemoteHostS1summary.log echo From: [email protected] >> /boot/logs/cronlogs/RemoteHostS1summary.log echo Subject: RemoteHost failed to start >> /boot/logs/cronlogs/RemoteHostS1summary.log echo >> /boot/logs/cronlogs/RemoteHostS1summary.log printf "Number of failed connections to RemoteHost $count\n" >> /boot/logs/cronlogs/RemoteHostS1summary.log echo "Host down 192.168.1.179" echo "Host down 192.168.1.179" >> /boot/logs/cronlogs/RemoteHostS1summary.log echo "Tried 5 times" echo "Tried 5 times" >> /boot/logs/cronlogs/RemoteHostS1summary.log ssh -i /root/.ssh/RemoteHost_pi_key [email protected] -p 26754 ipmitool -I lan -H 192.168.1.178 -U ADMIN -P ADMIN chassis power status >> /boot/logs/cronlogs/RemoteHostS1summary.log sleep 30 echo "Now trying to shut the power down" >> /boot/logs/cronlogs/RemoteHostS1summary.log date >> /boot/logs/cronlogs/RemoteHostS1summary.log # Shut down RemoteHost via IPMI using a soft shutdown ssh -i /root/.ssh/RemoteHost_pi_key [email protected] -p 26754 ipmitool -I lan -H 192.168.1.178 -U ADMIN -P ADMIN chassis power soft sleep 60 echo "Should be down now" >> /boot/logs/cronlogs/RemoteHostS1summary.log ssh -i /root/.ssh/RemoteHost_pi_key [email protected] -p 26754 ipmitool -I lan -H 192.168.1.178 -U ADMIN -P ADMIN chassis power status >> /boot/logs/cronlogs/RemoteHostS1summary.log # Send email of summary of results ssmtp [email protected] < /boot/logs/cronlogs/RemoteHostS1summary.log cd /boot/logs/cronlogs mv RemoteHostS1summary.log "`date +%Y%m%d_%H%M`_RemoteHostS1summary.logarchive" fi sleep 60 printf "Number of failed connections to RemoteHost $count\n" ((count++)) fi done done Edited March 11, 2017 by tr0910 Quote Link to comment
ken-ji Posted March 11, 2017 Share Posted March 11, 2017 quick points to check: myaccount.duckdns.org - is this your public ip that's doing NAT to your remote host? because your rsync dest is [email protected]:/mnt/disk10/downloads/ftp_dump/usa_newtheme/ it should be a public ip:port that maps to your remotehost (since the remote host has in LAN ip: 192.168.1.179). Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.