Syncronize servers using rsync over SSH next door or across the world


Recommended Posts

Suppose that you have more than one server, and that you want to keep a backup of your unRaid server.  Suppose that you have tried the sync options, but find that they are a bit finicky and unreliable.  This is a totally reliable bullet proof / stable way to keep your backup server fresh.  All you need is 2 unRaid servers connected to the internet, and we use secure keyed ssh to make the rsync commands secure and optimized for transiting the internet.

 

What I am using is a unRaid 6.2.4 server at another site with IPMI support.  This server starts up via IPMI accepts the backup, then shuts down via IPMI.

I added a raspberry PI at the destination to act as an always on computer to accept the IPMI commands and to start the unRaid backup server.

I also added pivpn.io to this unit as well.  Very easy to do and now I have a secure OpenVPN server installed without needing to fight with a limited AT&T router.  Some ports are forwarded in the router for the SSH and the VPN.  SSH is set up for secure keyfile access and a different port to keep the script kiddies shooed away.  The raspberry pi and the IPMI is strictly optional and add some nice touches, but are in no ways required.

 

This is an attempt to document the process of setting up a secure way to move data efficiently across town or across the world.  I have used rsync locally for years and it has moved petabytes of data reliably for me.  But we need to have offsite backups, and with Google Fiber I should be able to quit doing the sneakernet dance.... 

 

1. First we need to set up SSH-Keys which permit rsync to be scripted and secure over the internet.

 

Thanks to this site.

 

http://troy.jdmz.net/rsync/index.html

 

I have modified his instructions to make them unRaid friendly so you can just follow along

 

To start with, we have 2 unRaid 6.2 or 6.3 servers on the same network to get the ssh stuff set up securely.  (Having the servers on the same network is not needed for advanced users.  But in order to keep it more simple, I have assumed this to simplify the process for those following along.)

 

First, I'll define some variables. My server is called Tower and my remote server has a local ip address of 192.168.13.84

 

In my explanation, I will be synchronizing files (copying only new or changed files), and I will be starting this process from the host I want to copy things to. In other words, I will be syncing files from /remote/dir/ on remotehost (192.168.13.84), as remoteuser (root), to /this/dir/ on thishost, as thisuser.

I want to make sure that 'rsync' over 'ssh' works at all before I begin to automate the process, so I test it first as thisuser:

$ rsync -avz -e ssh [email protected]:/remote/dir /this/dir/ 
The authenticity of host '192.168.13.221 (192.168.13.84)' can't be established.
ECDSA key fingerprint is SHA256:30e/zRmjYVJLB4+5cBc2GjMqEpP7O3sUFcYlzQNYPtY.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.13.84' (ECDSA) to the list of known hosts.
[email protected]'s password:

 

and type in root@remotehost's password when prompted.

 

2. Configuring thishost

 

If that all worked out, or I eventually made it work, I am ready for the next step. I need to generate a private/public pair of keys to allow a 'ssh' connection without asking for a password.  I generate the key I will use on thishost (as thisuser):

$ ssh-keygen -t rsa -b 2048 -f /root/.ssh/Tower-rsync-key 
Generating public/private rsa key pair. 
Enter passphrase (empty for no passphrase): [press enter here] 
Enter same passphrase again: [press enter here] 
 

 

I use scp to get the file over to remotehost:

$  scp /root/.ssh/Tower-rsync-key.pub [email protected]:/root/
Just get the file over there some way if you don't like scp.

 

3. Configuring remotehost

 

I 'ssh' over to remotehost:
$ ssh [email protected]
root@remotehost's password: [type correct password here]
$ echo I am now $USER at $HOSTNAME
 

I am now root at remotehost and need to do some work.

I need to make sure I have the directory and files I need to authorize connections with this key [3]:

$ if [ ! -d .ssh ]; then mkdir .ssh ; chmod 700 .ssh ; fi 
$ mv Tower-rsync-key.pub .ssh/ 
$ cd .ssh/ 
$ if [ ! -f authorized_keys ]; then touch authorized_keys ; chmod 600 authorized_keys ; fi 
$ cat Tower-rsync-key.pub >> authorized_keys 
 

 

Now the key can be used to make connections to this host, but these connections can be from anywhere (that the ssh daemon on remotehost allows connections from) and they can do anything (that root can do).  We can change this later

 

Edit I didn't have the above permissions correct.  Make sure that the files in the .ssh directory have the exact permissions below, and the .ssh directory has 700 permissions

chmod 700 .ssh/
chmod 600 .ssh/Tower-rsync-key.pub
chmod 644 .ssh/authorized_keys

 

4. Troubleshooting

 

Now that I have the key with no password in place and configured, I need to test it out.  I exit from the ssh session to remotehost and try:

 

$ rsync -avz -e "ssh -i /root/.ssh/Tower-rsync-key" /boot/config/ [email protected]:/boot/fromTower/
 

 

Or we can move data the other direction like this

 

 rsync -avz -e "ssh -i /root/.ssh/Tower-rsync-key" [email protected]:/boot/config/ /boot/fromRemoteHost/
 

 

If this works, you are finished the first step.  Clean up your flash drives on both servers and remove the garbage you just copied there to test that it works.

 

Next we need to make these changes persistent across reboots, and then make it work through firewalls and finally set it all up in cron securely.

 

5a.Make the changes persistent across reboots

 

First you need to save your newly created keys from /root/.ssh on the local and remote server to your flash drive (I used /boot/config/sshroot so they will survive a reboot.  Use mc or commandline as follows

 

cp /root/.ssh/* /boot/config/sshroot/
 

 

Then you need to modify your /boot/config/go file as follows

 

# Copy files back to /root/.ssh folder and set permissions for key files and known_hosts and authorized_keys 

cd /root/.ssh
cp /boot/config/sshroot/* /root/.ssh/
chmod 600 *
 

 

This copies the files back to their correct location after a reboot.

 

 

5b. Add the SSH config plugin for improving security. (Optional, can be done later)

 

Install the ssh config plugin from community application.  Even after doing the above, I still needed to do more to secure my ssh login.

 

Via the plugin, I shut down password logins via ssh, change the port of the ssh server to something other than 22 to cut down on the hackers trying to get in and keep my log file from exploding.  This simplifies the configuration of the ssh server.

 

6. Are we golden??

 

Now unplug the server destined for remote use and move it to it's final destination.  Find out it's WAN ip address, and forward the SSH port (22) or (other port you changed to in the SSH config plugin) in your router to the unRaid local ip address.  For simplicity sake the following example assumes that you are still using port 22.  To test if you have a connection try:

 

ssh -i /root/.ssh/Tower-rsync-key [email protected] uptime
20:04:23 up  1:30,  1 user,  load average: 0.05, 0.05, 0.00
 

 

If you get a response without a password challenge you are golden and ready for an rsync test.

 

rsync -avz -e "ssh -i /root/.ssh/Tower-rsync-key" /mnt/disk2/Documents/ [email protected]:/mnt/disk1/fromTower/
sent 2,105,771,960 bytes  received 4,899 bytes  3,443,625.28 bytes/sec
total size is 2,196,999,197  speedup is 1.04
 

 

It's working.  Over gbit lines you can see that the we aren't getting anywhere the full 100 mb/sec transfer rates that we are looking for.    Further tests showed this to be a worst possible case.  Single ISO files would tranfer as fast at 25mb/sec, but there is significant variability to the sustained speed.  More troublesome is the speed at which is goes when there is nothing to transfer.  If everything is up to date, it has big problems and never completes the sync.  This kept me puzzling for a few days.  I needed it to reliably complete sync even if nothing needed to be copied.

 

From this site we have a faster way.  200,000 files synced in seconds not hours.  (assuming the files already match, nothing to copy)

 

https://gist.github.com/KartikTalwar/4393116

 

This is the process we end up using after a bit of testing.  It reliably uses the bandwidth of the internet and it just works.

 

rsync -avu --numeric-ids --progress  -e "ssh -i /root/.ssh/Tower-rsync-key  -T  -o Compression=no -x "  /mnt/disk1/  [email protected]:/mnt/disk1/
 
Edited by tr0910
Fixed errors in permissions
  • Like 4
  • Thanks 1
  • Upvote 2
Link to comment

I didn't want to leave this offsite backup server running 24x7 as it only needs to wake up at most once per day.  The problem was that the offsite location was using the brain dead AT&T router and once the unRaid server was shut down, I would have to ask somebody to turn it on to accept a backup.  This wasn't real cool

 

Enter a raspberry pi running piVPN.

 

http://www.pivpn.io/

 

In less than 2 hours from my first experience with the raspberry it was running as a dedicated vpn at this offsite location.  Highly recommended...

 

Duckdns is handling the handling the fact that AT&T changes the IP at the destination.

 

Edit: After using the VPN for a number of days, I finally just went pure SSH by:

 

ssh -i /root/.ssh/pi_key  [email protected] -p 56754 ipmitool -I lan -H 192.168.1.178 -U user -P password chassis power on
.
.
bunch of rsync commands over ssh
.
.
ssh -i /root/.ssh/pi_key  [email protected] -p 56754 ipmitool -I lan -H 192.168.1.178 -U user -P password chassis power soft

 

IPMI on this Supermicro X9SCM is able to shut unRaid down cleanly via this method.

 

The VPN is still nice to have and is used to troubleshoot any problems.

  • Like 1
  • Upvote 1
Link to comment

UDT and UDR transport utilities promise to speed up rsync immensely.  Has anyone tried this?

 

http://smartaleksolutions.com/blog/2015/10/5/udt-and-udr-transport-utilities

 

Does anyone have the ability to create a binary of UDT for slackware?

 

https://github.com/LabAdvComp/UDR

http://udt.sourceforge.net/

https://lists.samba.org/archive/rsync/2013-February/028178.html

 

Another option might be Tsunami, but Tsunami UDP transfers only files and doesn’t do directories/ subdirectories, we need to tar them all up as one single tar file (additional storage capacity needs to be taken into consideration). 

 

https://bluepiit.com/blog/tsunami-udp-protocol-installation-setup-and-limitations/

 

Update, no solution was found here, but for now I don't really need it.

 

Link to comment
  • 1 month later...

Followup, this is now working quite satisfactorily.

 

The backup is from disk to disk, and for disks with no activity, there is about 60 seconds of activity, then it moves on to another disk.  Given the 15mbit AT&T Uverse  internet limitation, the throughput is about 3mbyte/second during off time, with much lower speed noted during peak time.  I have tried to create some problems by sending a large backup over while someone is trying to do some gaming and internet browsing at the same time.  Surprisingly, the gamers and the browsers did not notice the fact that their internet was slow.  This was without any attempt to throttle the rsync during the transfer.

 

I have a raspberry pi that creates a vpn into the second location.  There is AT&T residential Uverse internet as this receiving location and Google Fiber on the sending location.

 

The VPN is only used to send an IPMI signal to the Supermicro X9SCM motherboard that it is time to turn on.  After that, the rsync via SSH as outlined above is all that is required.  The openvpn plugin of Peter's is used on the unRaid side, and piVPN is the build on the raspberry pi.

Link to comment
  • 3 weeks later...
  • 3 weeks later...
On 2/25/2017 at 8:03 PM, teamhood said:

This looks great and is on my to-do list. Following!

 

Looking forward to your refinements...  So far I've only got this working in USA where CG-NAT isn't messing things up.  Some countries in Europe and Asia hide your IP behind several levels of NAT, and getting rsync through this nasty ipv4 abortion is more complex. 

 

What I have had to do to get around CGNAT is change the machine I am rsyncing from.  I have to sync from behind the great firewall of China where the IP addresses are being CGNATified, and rsync to a destination where the IP address is more stable, (such as the USA).  Because rsync is flexible you can initiate the connection from either end and successfully get the transfer to work.

 rsync -avu --stats --log-file=/boot/logs/cronlogs/rsync.log  -e "ssh  -T  -o Compression=no -x -p 51842"  /mnt/user/Documents/ [email protected]:/mnt/user/Documents/  

This combination of flags seems to work with with transferring multimegabyte incompressible files across the world

 

So what do you have to do if both of your IP addresses are CGNAT?  I don't have a good answer yet for that.

Edited by tr0910
Link to comment
  • 2 weeks later...

 

Please - I need a little help.

 

I had everything working as expected up to;

 

On 12/9/2016 at 1:30 PM, tr0910 said:

5a.Make the changes persistent across reboots - the hard way

 

Now I'm lost. When I make the change to the go file it creates the directory but does not copy the files. I am not a linux guru but I think there are some ;'s missing and the syntax is not quite correct. But - as I say - I am very low on the linux command line learning curve.

 

I have copied the files manually but I am now being asked for a password.

 

So I tried the easy way...

 

I am also very new to unRAID v6. As a matter of fact I just upgraded from v5 so I could try this. So this plugin stuff is a mystery to me.

 

I did install the ssh plugin but I have no idea how to configure it properly to make it persistent to reboots.

 

Would someone be able to guide me please?

 

Thanks

 

Link to comment
17 hours ago, tr0910 said:

If using the plugin, is automatic reinstalled at reboot. And just make sure your key files are in the right folder /config/ssh then on reboot they get copied to the respective dir on the server.

 

Perhaps I am just plain stupid;

 

Everything is working perfectly until I reboot.

 

I have installed the ssh plugin and copied the files from /root/.ssh/ to /boot/config/ssh/

 

After a reboot /root/.ssh/ is empty and a password is required.

 

Thanks for any guidance on what I am doing wrong.

Link to comment
35 minutes ago, ken-ji said:

SSH keys have a very strict file and directory permission.

Are you ensuring the correct permissions?

 

Thank you for the reply ken-ji

 

No - I am not ensuring anything. that is for two reasons:

 

1. I know very little LINUX (I grew up on OpenVMS if you know what that is) and

2. tr0910 wrote the following;

 

23 hours ago, tr0910 said:

If using the plugin, is automatic reinstalled at reboot. And just make sure your key files are in the right folder /config/ssh then on reboot they get copied to the respective dir on the server.

 

By "automatic" I assume these things were taken care of.

 

Perhaps I should not assume.

 

In any event the .ssh/ folder is empty after a reboot. And yes I have the key files in the /config/ssh folder.

 

<sigh> I have so much to learn - and I am an old dog trying new tricks ;-)

 

 

Link to comment

I think I got it.

 

I was reading a post from ken-ji in another topic

 

https://forums.lime-technology.com/topic/50038-ssh-support-in-v6/

 

Then it hit me. I should not be using /root/.ssh/ but instead /etc/ssh/ - no?

 

On 12/9/2016 at 1:30 PM, tr0910 said:

ssh -i /root/.ssh/Tower-rsync-key [email protected] uptime

 

requires a password because /root/.ssh/ is empty

 

But 

 

ssh -i /etc/ssh/Tower-rsync-key [email protected] uptime

 

does not require a password - even after reboots.

 

Do I have this correct?

Link to comment

Yes that is where the plugin puts it on reboot. Sounds like you got it

 

Once you have it solid you can use the plugin to change the SSH port, and dis allow password login to keep the script kiddies and the CIA out...

 

I was getting several login attempts per second showing in my logs, until I did this. The plugin helps manage this.

 

via mobile

 

 

 

 

 

 

Link to comment

Well your post mentioned copying the keys manually, so I pointed out the permissions are probably wrong. The USB has no concept

of permissions since its using the "universal" FAT/FAT32 filesystem. When copied to unRAID RAM disk (or any Linux filesystem), it gets the equivalent allow anybody r/w access, which is a big no no for SSH keys (you don't want other people coipying or changing your keys)

 

Also you can setup a default config file called ssh_config (/boot/config/ssh - so it gets copied in)

IdentityFile /etc/ssh/myprivatekey

So you don't have to always specify the keyfile (If you are only using one key file) or

Host server1
  Hostname 11.22.33.44
  IdentityFile /etc/ssh/server1.key
Host server2
  Hostname 55.66.77.88
  IdentityFile /etc/ssh/server2.key

if you have more than one server

Edited by ken-ji
Link to comment
1 hour ago, tr0910 said:

Let me know if the original post needs updated to fix errors or make it easier to understand.

 

I hit another snag and just haven't had time to do anymore.

 

I thought I had it working. I could ssh and rsync from the backup server using the keys and no password.

 

But then I decided to try to reboot my primary (the one I will backup). Now no keys work. Not from the primary nor the backup server.

 

Back to the drawing board and I will post when life lets me try some things.

 

Thanks

Link to comment

I have a few free moments to try again - from scratch.

 

I deleted the .ssh/ folder and all my created files from /config/ssh/ and /etc/ssh/

 

I followed the steps and after...

 

On 12/9/2016 at 1:30 PM, tr0910 said:

4. Troubleshooting

 

...everything was working as expected.

 

I have the SSH Config app installed so I copied all files from /root/.ssh/ to /boot/config/ssh on both servers

 

After I reboot "thishost" (as described above) everything work fine. I rebooted a couple of times and everything is good.

 

After I reboot "remotehost" it no longer works. Yes I am still entering the commands on thishost.

 

So - what the heck am I doing wrong?

Link to comment

Got it.

 

I suspect that tl0910 hasn't rebooted remotehost yet - or I am doing something wrong.

 

After reading this from ken-ji

 

On 3/7/2017 at 7:38 PM, ken-ji said:

SSH keys have a very strict file and directory permission.

Are you ensuring the correct permissions?

 

I did a bit of experimenting and found the answer. My go file now looks like this on remotehost (no changes on the go file on thishost):

 

# umask for root .ssh setup with known hosts and authorized keys 
umask 077

# Copy the files and set the permission
cp /etc/ssh/Tower-rsync-key.pub /root/.ssh/
cp /etc/ssh/authorized_keys /root/.ssh/
chmod 700 /root/.ssh/

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &

 

I have now successfully rebooted remotehost and thishost and things work.

 

Next I am going to congiure ssh for another and disallow root logins - but one step at a time. When I finish I'llpm tl0910 with what I did and he can edit his original post if I am doing this correctly.

 

Thanks again ken-ji and tl0910 - I continue to learn.

Edited by skulker
wrong pub key file name
Link to comment
2 hours ago, skulker said:

Got it.

 

I suspect that tl0910 hasn't rebooted remotehost yet - or I am doing something wrong.

 

After reading this from ken-ji

 

Next I am going to congiure ssh for another and disallow root logins - but one step at a time. When I finish I'llpm tl0910 with what I did and he can edit his original post if I am doing this correctly.

 

Thanks again ken-ji and tl0910 - I continue to learn.

My remotehost gets started via ipmi then once it is up, we start the the rsync process.  So I know that my remotehost is restarting fine.  I likely have other things wrong but that is why we are all testing each others scripts. 

 

Here is my script run from ThisHost with the backup destination RemoteHost across the city or halfway around the world.  This script adds a RaspBerry pi at the RemotePlace to SSH into before the RemoteHost is turned on.  The only requirement is that the RaspBerry pi being on at all times.  Everything else is controlled by the script.

 

This code is very much beginners work cut and pasted from various places on the net.  The scripting gurus could likely cut 75% of this code.

 

#!/bin/bash
###################################################################
# We are using an always on Raspberry Pi at the RemoteHost Place
# The remote host is turned off when the script begins and we use the pi to turn on 
# the RemoteHost via IPMI.  Why don't we just forward the port of the IPMI to the WWW?
# Because it isn't as secure as coming via keyed SSH to the pi and then connecting to the 
# IPMI interface.  
#
# Once the RemoteHost is awake and ready to rock and roll, the rsync begins
####################################################################

echo "Starting Sync to RemoteHost at RemoteHost's Place"
#echo "Starting Sync `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log

# Start up RemoteHost via ipmi note IPMI address is different from RemoteHost address as this is the IPMI address of remotehost
ssh -i /root/.ssh/RemoteHost_pi_key  [email protected] -p 26754 ipmitool -I lan -H 192.168.1.178 -U ADMIN -P ADMIN chassis power on


# Wait for a few minutes - try 5 times
echo "waiting for RemoteHost to come online..."
count=1
sleep 180
# Check if host online RemoteHost is 179

HOSTS="192.168.1.179"
for myHost in $HOSTS


do
  while [ $count -le 5 ]
  do
	ssh -i /root/.ssh/RemoteHost_pi_key  [email protected] -p 26754 ping -c 1 "$myHost" > /dev/null

	if [ "$?" -eq 0 ] ; then
		echo "Hosts up"
		#==============================================================================
		sleep 30

		# Set up email header

		echo To: [email protected] >> /boot/logs/cronlogs/RemoteHostS1summary.log
		echo From: [email protected] >> /boot/logs/cronlogs/RemoteHostS1summary.log
		echo Subject: RemoteHost rsync summary >> /boot/logs/cronlogs/RemoteHostS1summary.log
		echo   >> /boot/logs/cronlogs/RemoteHostS1summary.log

		# Backup Disk 1 getting files from yesterday
		echo "moving to Disk1 =====  `date`"
		echo "moving to Disk1 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log
		echo "moving to Disk1 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1disk1.log

		# The first rsync group of files have dates in the file names and are selected based on yesterdays date
		# after this we simply rsync disk2 to disk2, disk3 to disk3 etc...
		
		dte=$(date -d "yesterday 13:00 " '+%Y-%m-%d')
		src="/mnt/disk1/downloads/ftp_dump/usa_newtheme/backup_"$dte"*"
		dest="[email protected]:/mnt/disk10/downloads/ftp_dump/usa_newtheme/"
		rsync -avu --stats   -e "ssh -i /root/.ssh/Kim-rsync-key  -T  -o Compression=no -x -p31798"  $src $dest  >> /boot/logs/cronlogs/RemoteHostS1disk1.log
		
		# Backup Disk 2
		echo "moving to Disk2 =====  `date`"
		echo "moving to Disk2 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log
		echo "moving to Disk2 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1disk2.log

		rsync -avu --stats --exclude=Z_K1zq* --exclude=.Recyc*  -e "ssh -i /root/.ssh/Kim-rsync-key  -T  -o Compression=no -x -p 31798"  /mnt/disk2/ [email protected]:/mnt/disk2/  >> /boot/logs/cronlogs/RemoteHostS1disk2.log
		
		# Backup Disk 3
		echo "moving to Disk3 =====  `date`"
		echo "moving to Disk3 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log
		echo "moving to Disk3 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1disk3.log

		#rsync -avu --stats --progress /mnt/disk3/ /mnt/RemoteHostS1disk3/  >> /boot/logs/cronlogs/RemoteHostS1disk3.log

		# Backup Disk 4
		echo "moving to Disk4 =====  `date`"
		echo "moving to Disk4 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log
		echo "moving to Disk4 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1disk4.log

		rsync -avu --stats --exclude=.Recyc*  -e "ssh -i /root/.ssh/Kim-rsync-key  -T  -o Compression=no -x -p 31798"  /mnt/disk4/ [email protected]:/mnt/disk4/  >> /boot/logs/cronlogs/RemoteHostS1disk4.log

		# Backup Disk 5
		echo "moving to Disk5 =====  `date`"
		echo "moving to Disk5 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log
		echo "moving to Disk5 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1disk5.log

		rsync -avu --stats --exclude=.Recyc*  -e "ssh -i /root/.ssh/Kim-rsync-key  -T  -o Compression=no -x -p 31798"  /mnt/disk5/ [email protected]:/mnt/disk5/  >> /boot/logs/cronlogs/RemoteHostS1disk5.log

		# Backup Disk 6
		echo "moving to Disk6 =====  `date`"
		echo "moving to Disk6 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log
		echo "moving to Disk6 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1disk6.log

		rsync -avu --stats --exclude=.Recyc*  -e "ssh -i /root/.ssh/Kim-rsync-key  -T  -o Compression=no -x -p 31798"  /mnt/disk6/ [email protected]:/mnt/disk6/  >> /boot/logs/cronlogs/RemoteHostS1disk6.log

		# Backup Disk 7
		echo "moving to Disk7 =====  `date`"
		echo "moving to Disk7 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log
		echo "moving to Disk7 =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1disk7.log

		rsync -avu --stats --exclude=.Recyc*  -e "ssh -i /root/.ssh/Kim-rsync-key  -T  -o Compression=no -x -p 31798"  /mnt/disk7/ [email protected]:/mnt/disk7/  >> /boot/logs/cronlogs/RemoteHostS1disk7.log

		echo "moving to end =====  `date`"
		echo "moving to end =====  `date`" >> /boot/logs/cronlogs/RemoteHostS1summary.log

	
		# Add in the summaries
		cd /boot/logs/cronlogs/
		echo = > disk1.log
		echo = > disk1.log
		echo ===== > disk2.log
		echo ===== > disk3.log
		echo ===== > disk4.log
		echo ===== > disk5.log
		echo ===== > disk6.log
		echo ===== > disk7.log
		echo Disk1 >> disk1.log
		echo Disk2 >> disk2.log
		echo Disk3 >> disk3.log
		echo Disk4 >> disk4.log
		echo Disk5 >> disk5.log
		echo Disk6 >> disk6.log
		echo Disk7 >> disk7.log
		tac RemoteHostS1disk1.log | sed '/^Number of files: /q' | tac >> disk1.log
		tac RemoteHostS1disk2.log | sed '/^Number of files: /q' | tac >> disk2.log
		tac RemoteHostS1disk3.log | sed '/^Number of files: /q' | tac >> disk3.log
		tac RemoteHostS1disk4.log | sed '/^Number of files: /q' | tac >> disk4.log
		tac RemoteHostS1disk5.log | sed '/^Number of files: /q' | tac >> disk5.log
		tac RemoteHostS1disk6.log | sed '/^Number of files: /q' | tac >> disk6.log
		tac RemoteHostS1disk7.log | sed '/^Number of files: /q' | tac >> disk7.log

		# Check to see if server shuts down
		# Shut down RemoteHost via IPMI using a soft shutdown
		ssh -i /root/.ssh/RemoteHost_pi_key  [email protected] -p 56754 ipmitool -I lan -H 192.168.1.178 -U ADMIN -P ADMIN chassis power soft

		thisString="something is on"
		# We are looking for this "off" string to come from the IPMI interface
		searchString="off"
		for i in {1..30}
		do
			echo "Checking to see if Server is down $i times"
			ssh -i /root/.ssh/RemoteHost_pi_key  [email protected] -p 26754 ipmitool -I lan -H 192.168.1.178 -U ADMIN -P ADMIN chassis power status  > /root/status.txt
			while read newstring; do
				thisString=$newstring
				echo $thisString after $i times >> RemoteHostS1summary.log
				echo $thisString
				# thisString should have "Chassis power is off" once shut down
			done < /root/status.txt

			case $thisString in
			# match exact string
			"$searchString") echo yep, it matches exactly;;
			# match start of string
			"$searchString"*) echo yep, it matches at the start ;;
			# match end of string
			*"$searchString") break ;;
			# searchString can be anywhere in thisString
			*"$searchString"*) echo yep, it matches in the middle somewhere ;;
  
			*) echo Nope still running ;;
			esac
			sleep 60
		done

		# now add all the other logs to the end of this email summary
		cat RemoteHostS1summary.log disk1.log disk2.log disk3.log disk4.log disk5.log disk6.log disk7.log > diskall.log
		zip -m RemoteHostS1 RemoteHostS1*.log 
		# Send email of summary of results
		ssmtp [email protected] < /boot/logs/cronlogs/diskall.log
		cd /boot/logs/cronlogs  
		mv RemoteHostS1.zip "`date +%Y%m%d_%H%M`_RemoteHostS1.zip"
		rm disk*.log

		# Wait for 5 minutes
		#sleep 300

		count=6


	else
		if [ $count -eq 5 ]
			then
			

			echo To: [email protected] >> /boot/logs/cronlogs/RemoteHostS1summary.log
			echo From: [email protected] >> /boot/logs/cronlogs/RemoteHostS1summary.log
			echo Subject: RemoteHost failed to start >> /boot/logs/cronlogs/RemoteHostS1summary.log
			echo   >> /boot/logs/cronlogs/RemoteHostS1summary.log
			printf "Number of failed connections to RemoteHost $count\n"  >> /boot/logs/cronlogs/RemoteHostS1summary.log

			echo "Host down 192.168.1.179"
			echo "Host down 192.168.1.179" >> /boot/logs/cronlogs/RemoteHostS1summary.log
			echo "Tried 5 times"
			echo "Tried 5 times" >> /boot/logs/cronlogs/RemoteHostS1summary.log
			ssh -i /root/.ssh/RemoteHost_pi_key  [email protected] -p 26754 ipmitool -I lan -H 192.168.1.178 -U ADMIN -P ADMIN chassis power status >> /boot/logs/cronlogs/RemoteHostS1summary.log

			sleep 30
			echo "Now trying to shut the power down" >> /boot/logs/cronlogs/RemoteHostS1summary.log
			date >> /boot/logs/cronlogs/RemoteHostS1summary.log
			# Shut down RemoteHost via IPMI using a soft shutdown
			ssh -i /root/.ssh/RemoteHost_pi_key  [email protected] -p 26754 ipmitool -I lan -H 192.168.1.178 -U ADMIN -P ADMIN chassis power soft

			sleep 60
			echo "Should be down now" >> /boot/logs/cronlogs/RemoteHostS1summary.log
			ssh -i /root/.ssh/RemoteHost_pi_key  [email protected] -p 26754 ipmitool -I lan -H 192.168.1.178 -U ADMIN -P ADMIN chassis power status >> /boot/logs/cronlogs/RemoteHostS1summary.log
		
			# Send email of summary of results
			ssmtp [email protected] < /boot/logs/cronlogs/RemoteHostS1summary.log
			cd /boot/logs/cronlogs  
			mv RemoteHostS1summary.log "`date +%Y%m%d_%H%M`_RemoteHostS1summary.logarchive"
		
		fi
		sleep 60
	    printf "Number of failed connections to RemoteHost $count\n"

		((count++))

	fi

  done

done

 

Edited by tr0910
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.