[GUIDE] Scheduled backup your ESXi VMs to unRAID with ghettoVCB


Recommended Posts

HOW TO Backup your ESXi VMs to unRAID on a scheduled basis with ghettoVCB

 

Background

I recently experienced a catastrophic datastore failure when migrating my ESXi server into a Norco RPC-4224 chassis--one of the backplanes fried my datastore drives.  In an attempt to speed up any future recovery process (I lost 3 VMs outright), I decided to institute a scheduled backup solution.

 

Disclaimer

I am in no way an expert with ESXi.  This solution works for me, using the primary ghettoVCB informational document as a guide (https://communities.vmware.com/docs/DOC-8760).  Keep in mind this is a fairly simple backup, and not a fault-tolerant solution that many on here are using (ZFS array over NFS, for example).  I am basing this off of ESXi 5.5, but should be identical for 5.1.  Any other versions you should take a look at the above referenced document for variations in configuration.

 

Also note, I could not get this to backup unRAID without powering down the VM. In my configuration, I keep my VMs spinning, and the script takes advantage of snapshots to make the backup.  I assume there is some issue with snapshotting unRAID, be it in my configuration or in general.  YMMV.  As such, I exclude unRAID from the VM backups

 

What you'll need

 

1. Enable NFS, if you haven't already.

We will need to create an NFS share to store the backups on your unRAID array.  I have one array disk dedicated to backups, so I exported that disk with NFS.  A user share should work too, but I personally haven't tried it.  If you want to enable the user share, just go into the share settings and export the share via NFS.

 

In my case, I'm exporting disk10.

 

2. Extract and configure ghettoVCB

There seems to be two options to configure ghettoVCB.  Either editing the script itself or via a global configuration file.  We're going to go with the latter.

 

Edit ghettoVCB.conf with a Linux-friendly text editor (notepad++, TextMate, etc.).  The items I changed are as follows:

 

VM_BACKUP_VOLUME=/vmfs/volumes/nfsbackup/ESXi

I can't tell that this made a difference at all, aside from the backup not being saved under an "ESXi" directory.

VM_BACKUP_ROTATION_COUNT=2

I don't need more than 2 copies.  Default is 3.

ENABLE_NON_PERSISTENT_NFS=1
UNMOUNT_NFS=1
NFS_SERVER=192.168.0.10
NFS_MOUNT=/mnt/disk10
NFS_LOCAL_NAME=nfsbackup
NFS_VM_BACKUP_DIR=Backup

This enables a non-persistent NFS share to be mounted, which is then unmounted after the backup process completes.  The NFS_SERVER should be your unRAID IP.  Make sure this is static!  If you are using a disk share, it will be /mnt/diskX.  If you are using a user share, it should be /mnt/user/usersharename.  For NFS_VM_BACKUP_DIR, I use "Backup", since I also have a user share named Backup (if I ever add another drive).  So this essentially is putting the VM backups in the root of the Backup user share.  If you're connecting to a user share, you should be able to have this be whatever you'd like.  I am uncertain if you can specify a deeper path.

 

Everything else in ghettoVCB.conf remained as default.

 

3. Create a list of VMs to backup

Since I don't want to power down my VMs to backup, I opted to exclude unRAID.  We could use an exclude file, but I'd rather specify what VMs I want backed up, since I spin up ones pretty regularly to test things out.  So we'll create an include file.  I named mine backup_vm and saved it in the ghettoVCB folder I'm working in.

 

Here's where things can be tricky.  Whatever names you have for your VMs, as displayed within the vSphere client, must not have brackets or parentheses.  This will cause your backups to fail (I had to remove parenthesis from mine).  If you have other symbols, you may run into a problem as well.  Spaces are fine.  In your include file, list the names exactly as they appear in the client.  For example,

Windows 7-pandora
Windows XP-hyperion
Ubuntu-telesto
CentOS-janus

Have you figured out my VM naming convention?  My server's name is "saturn"... Anywhoo.

 

4. Transfer the ghettoVCB folder to a datastore

I'm assuming here you have enough knowledge of how to do this part.  If not, drop me a post and I'll expand upon this section.  Here's my path: /vmfs/volumes/2TBRED/ghettoVCB.  Once it was transfered, the two .sh files already had execute privileges.  If yours doesn't, or you just want to be safe, go to your ghettoVCB directory via SSH and run these commands:

chmod +x ghettoVCB.sh
chmod +x ghettoVCB-restore.sh

 

5. Let's do a dry run!

Ok, so you've configured your .conf file, you've created a list of VMs to backup, and you've copied everything over to an ESXi datastore.  Time to see if this bad boy works.

 

SSH to the ghettoVCB folder, and run this command:

./ghettoVCB.sh -f backup_vm -g ghettoVCB.conf -d dryrun

This is telling the script to (-f) backup VMs in our backup_vm file, (-g) using ghettoVCB.conf as the config file, and (-d) doing it as a dry run to see if we have errors.

 

Hopefully it'll look something like this:

 

 # ./ghettoVCB.sh -f backup_vm -g ghettoVCB.conf -d dryrun
Logging output to "/tmp/ghettoVCB-2014-02-24_08-01-52-42917.log" ...
2014-02-24 08:01:52 -- info: ============================== ghettoVCB LOG START ==============================

2014-02-24 08:01:52 -- info: CONFIG - USING GLOBAL GHETTOVCB CONFIGURATION FILE = ghettoVCB.conf
2014-02-24 08:01:52 -- info: CONFIG - VERSION = 2013_26_11_2
2014-02-24 08:01:52 -- info: CONFIG - GHETTOVCB_PID = 42917
2014-02-24 08:01:52 -- info: CONFIG - VM_BACKUP_VOLUME = /vmfs/volumes/nfsbackup/Backup
2014-02-24 08:01:52 -- info: CONFIG - ENABLE_NON_PERSISTENT_NFS = 1
2014-02-24 08:01:52 -- info: CONFIG - UNMOUNT_NFS = 1
2014-02-24 08:01:52 -- info: CONFIG - NFS_SERVER = 192.168.0.10
2014-02-24 08:01:52 -- info: CONFIG - NFS_VERSION = nfs
2014-02-24 08:01:52 -- info: CONFIG - NFS_MOUNT = /mnt/disk10
2014-02-24 08:01:52 -- info: CONFIG - VM_BACKUP_ROTATION_COUNT = 2
2014-02-24 08:01:52 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2014-02-24_08-01-52
2014-02-24 08:01:52 -- info: CONFIG - DISK_BACKUP_FORMAT = thin
2014-02-24 08:01:52 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = 0
2014-02-24 08:01:52 -- info: CONFIG - ENABLE_HARD_POWER_OFF = 0
2014-02-24 08:01:52 -- info: CONFIG - ITER_TO_WAIT_SHUTDOWN = 3
2014-02-24 08:01:52 -- info: CONFIG - POWER_DOWN_TIMEOUT = 5
2014-02-24 08:01:52 -- info: CONFIG - SNAPSHOT_TIMEOUT = 15
2014-02-24 08:01:52 -- info: CONFIG - LOG_LEVEL = dryrun
2014-02-24 08:01:52 -- info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCB-2014-02-24_08-01-52-42917.log
2014-02-24 08:01:52 -- info: CONFIG - ENABLE_COMPRESSION = 0
2014-02-24 08:01:52 -- info: CONFIG - VM_SNAPSHOT_MEMORY = 0
2014-02-24 08:01:52 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = 0
2014-02-24 08:01:52 -- info: CONFIG - ALLOW_VMS_WITH_SNAPSHOTS_TO_BE_BACKEDUP = 0
2014-02-24 08:01:52 -- info: CONFIG - VMDK_FILES_TO_BACKUP = all
2014-02-24 08:01:52 -- info: CONFIG - VM_SHUTDOWN_ORDER =
2014-02-24 08:01:52 -- info: CONFIG - VM_STARTUP_ORDER =
2014-02-24 08:01:52 -- info: CONFIG - RSYNC_LINK = 0
2014-02-24 08:01:52 -- info: CONFIG - EMAIL_LOG = 0
2014-02-24 08:01:52 -- info:
2014-02-24 08:01:53 -- dryrun: ###############################################
2014-02-24 08:01:53 -- dryrun: Virtual Machine: Windows 7-pandora
2014-02-24 08:01:53 -- dryrun: VM_ID: 3
2014-02-24 08:01:53 -- dryrun: VMX_PATH: /vmfs/volumes/SSD1/Win7Pro/Win7Pro.vmx
2014-02-24 08:01:53 -- dryrun: VMX_DIR: /vmfs/volumes/SSD1/Win7Pro
2014-02-24 08:01:53 -- dryrun: VMX_CONF: Win7Pro/Win7Pro.vmx
2014-02-24 08:01:53 -- dryrun: VMFS_VOLUME: SSD1
2014-02-24 08:01:53 -- dryrun: VMDK(s):
2014-02-24 08:01:53 -- dryrun: 	/vmfs/volumes/blahblahblahblahblahblahblahblah/Win7Pro/Win7Pro_2.vmdk	150 GB
2014-02-24 08:01:53 -- dryrun: 	Win7Pro.vmdk	55 GB
2014-02-24 08:01:53 -- dryrun: INDEPENDENT VMDK(s):
2014-02-24 08:01:53 -- dryrun: TOTAL_VM_SIZE_TO_BACKUP: 205 GB
2014-02-24 08:01:53 -- dryrun: ###############################################

2014-02-24 08:01:53 -- dryrun: ###############################################
2014-02-24 08:01:53 -- dryrun: Virtual Machine: Windows XP-hyperion
2014-02-24 08:01:53 -- dryrun: VM_ID: 22
2014-02-24 08:01:53 -- dryrun: VMX_PATH: /vmfs/volumes/2TBRED/Windows XP/Windows XP.vmx
2014-02-24 08:01:53 -- dryrun: VMX_DIR: /vmfs/volumes/2TBRED/Windows XP
2014-02-24 08:01:53 -- dryrun: VMX_CONF: Windows XP/Windows XP.vmx
2014-02-24 08:01:53 -- dryrun: VMFS_VOLUME: 2TBRED
2014-02-24 08:01:53 -- dryrun: VMDK(s):
2014-02-24 08:01:53 -- dryrun: 	Windows XP.vmdk	20 GB
2014-02-24 08:01:53 -- dryrun: INDEPENDENT VMDK(s):
2014-02-24 08:01:53 -- dryrun: TOTAL_VM_SIZE_TO_BACKUP: 20 GB
2014-02-24 08:01:53 -- dryrun: ###############################################

2014-02-24 08:01:53 -- dryrun: ###############################################
2014-02-24 08:01:53 -- dryrun: Virtual Machine: Ubuntu-telesto
2014-02-24 08:01:53 -- dryrun: VM_ID: 21
2014-02-24 08:01:53 -- dryrun: VMX_PATH: /vmfs/volumes/2TBRED/Ubuntu/Ubuntu.vmx
2014-02-24 08:01:53 -- dryrun: VMX_DIR: /vmfs/volumes/2TBRED/Ubuntu
2014-02-24 08:01:53 -- dryrun: VMX_CONF: Ubuntu/Ubuntu.vmx
2014-02-24 08:01:53 -- dryrun: VMFS_VOLUME: 2TBRED
2014-02-24 08:01:53 -- dryrun: VMDK(s):
2014-02-24 08:01:53 -- dryrun: 	Ubuntu.vmdk	250 GB
2014-02-24 08:01:53 -- dryrun: INDEPENDENT VMDK(s):
2014-02-24 08:01:53 -- dryrun: TOTAL_VM_SIZE_TO_BACKUP: 250 GB
2014-02-24 08:01:53 -- dryrun: ###############################################

2014-02-24 08:01:53 -- info: ###### Final status: OK, only a dryrun. ######

2014-02-24 08:01:53 -- info: ============================== ghettoVCB LOG END ================================

If the final status is OK, it's time to run this for real.

 

6. Perform the backup, using full paths in the command

So let's take our test command:

./ghettoVCB.sh -f backup_vm -g ghettoVCB.conf -d dryrun

Get rid of the dryrun portion, and use the full paths to the files.

/vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB.sh -f /vmfs/volumes/2TBRED/ghettoVCB/backup_vm -g /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB.conf > /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB-backup-$(date +\%Y%m%d%H%M%S).log

Notice we also added a bit to export the results into a dated log file.  Adjust the datastore name accordingly.

 

Hopefully that worked!  My NFS share amazingly connected on the first try, and unmounted automatically as intended.  Not sure if the dry runs actually create the mount or not, but you'll definitely know at this stage if it did. 

 

If all went well, it's time to setup a cronjob.  To the next post...

Link to comment

Setup crontab to schedule the backup

 

The dry run worked, and the manual backup worked.  Now let's schedule this bad boy.

 

7. Decide when you want your backup to run

A few things to keep in mind here.  ESXi servers are always running on UTC time.  Even when you set up an NTP server within the vSphere client, and it properly displays your timezone, the server itself is UTC.  So remember this point when deciding your backup time.  I wanted mine to run at 2:05am every Tuesday morning, so I'll have to tell ESXi to run it at 10:05am, since I'm in California (UTC minus 8).  If you'd like to read up on cron format, google cron, or just check out this page.

 

8. Add the crontab entry

WARNING: These next parts are specific for ESXi 5.1/5.5.  If you are on another version, please refer to the ghettoVCB instructions.

 

The file we will be adding our entry to is /var/spool/cron/crontabs/root.  And here's what I added:

5 10 * * 2 /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB.sh -f /vmfs/volumes/2TBRED/ghettoVCB/backup_vm -g /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB.conf > /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB-backup-$(date +\%Y%m%d%H%M%S).log

Its the same command we used above, but adding the 5 bits to tell the cronjob what time it is.

 

I edited it directly from Transmit and it worked fine.  Whatever you do, just make sure the method it's Linux-friendly (Windows users have been warned).

 

Kill cron and restart:

kill $(cat /var/run/crond.pid)
crond

 

9. Make sure ESXi retains this change on every boot

Edit the file /etc/rc.local.d/local.sh and add the following before the "exit 0"

/bin/kill $(cat /var/run/crond.pid)
/bin/echo "5 10 * * 2 /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB.sh -f /vmfs/volumes/2TBRED/ghettoVCB/backup_vm -g /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB.conf > /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB-backup-\$(date +\\%Y%m%d%H%M%S).log" >> /var/spool/cron/crontabs/root
crond

Again, you'll need to change the volume to match your setup.  Also note, the format around the date portion is different from the command we've executed in the past (and added to the cron job).  It's formatted in such a way for the "echo" to enter the command in the cron properly.  Please make sure this portion (ghettoVCB-backup-\$(date +\\%Y%m%d%H%M%S).log) is properly entered.

 

And finally, save the ESXi configuration with this command:

/sbin/auto-backup.sh

 

10. Profit

If you've followed these steps, you should feel confident that your backup will run on the schedule you've set.  I rebooted my system just to make sure the cronjob indeed survived the reboot.  It did. 

 

Hope this helps some people!  If you know of a better method to go about any of these parts, please let me know and I will update this post accordingly.

 

How to restore a backed up VM

 

What if you need to restore a backup?  Well, there are a few ways to go about doing this.  ghettoVCB comes with an experimentally supported script called ghettoVCB-restore.sh.  However, unRAID user joshpond has run into issues getting it to work.  While the script would likely be the fastest way to get back up and running (if it worked), I have another solution that I have confirmed to work.

 

1. Create an NFS datastore, if non-persistent in your backup configuration

In the vSphere client, go to Configuration | Storage | Add Storage and select Network File System.  Follow the same guidelines I explained for backup in the NFS section above (Server will be your unRAID IP address, Folder will depend on if using a disk or user share).  You can check the box to mount read-only to be on the safe side, if you so desire. Give the datastore any name you wish.

 

In keeping with my example,

Server: 192.168.0.10
Folder: /mnt/disk10/Backup
Mount NFS read only: Unchecked
Datastore Name: nfsrestore

 

2. Browse the datastore

Right-click on your newly mounted NFS datastore, and select Browse Datastore.  Go into the folder of the VM that you'd like to restore, then select the backup folder you want (the folder with the most recent date, I would assume).  Right-click on the VM's backup folder and select Copy.

 

Close out the datastore browser and select the datastore you wish to restore to.  Again, right-click and select Browse Datastore.  Right-click in the empty space and select Paste.  Depending on the size of the backup, it should take quite some time, so grab a beer, eat some food, and watch a movie (e.g., joshpond reported a 350GB VM took 4 hours from his unRAID array to a RAID 0 datastore via this method).

 

3. Add the restored VM to the inventory

Once the copy completes, right-click the folder (should contain some sort of date format) and rename it to the VM name.  It should probably be as simple as wiping out the numerical portion.

 

Double click on the newly renamed folder, right click on the .vmk file, and select Add to Inventory.  When prompted if you Copied or Moved the VM, select Moved (if you are truly restoring the VM--it will maintain the old MAC address and some other variables).

 

For housekeeping, you may want to delete the file STATUS.ok from the restored directory, and delete the unRAID NFS restore mount in the Storage configuration of the vSphere client..

 

 

Changes to guide

02/27/2014 - Added a real-world restore time example.

02/25/2014 - Changed the date formatting for logs to be more descriptive (previous version was %s, seconds since 1970-01-01 00:00:00 UTC, which is not meaningful, IMHO); Added guide to restore VMs.

Link to comment

Also note, I could not get this to backup unRAID without powering down the VM. In my configuration, I keep my VMs spinning, and the script takes advantage of snapshots to make the backup.  I assume there is some issue with snapshotting unRAID, be it in my configuration or in general.  YMMV.  As such, I exclude unRAID from the VM backups.

Very nice guide.  Thank you.  I will have to look into doing this.

 

For the quote above: If I had to guess I would say you are using pass through on your unRAID VM.  Probably for your HDD controller.  Your other VMs that work you probably don't have pass through on PCI/PCIe cards.  The documentation for pass through and personal experience with pass through leads me to this conclusion since my Windows VMs that I pass through tuner cards are not snapshot-able either - the option is greyed out in the GUI client.  But other VMs where I have no pass through PCI/PCIe devices the snapshot option is available in the GUI client.

Link to comment

You are absolutely right.  I knew I remembered reading something about snapshots and passthroughs a while back.

 

ghettoVCB has the ability to shut down and restart the VMs during the backup process, which negates the need for snapshots.  However, I don't know how that would interfere with the NFS mount.  Observing how it currently takes place, all the VM cloning happens on the mount, as it is treated as a datastore.  If the unRAID VM has to be shut down, I think it will break this process.  I'm fine with the unRAID VM not bein backed up, since all of the real data is being stored on the flash drive.  I manually make images of it from time to time.

 

The shutdown would probably work fine for other VMs with passhtrough, however.  ghettoVCB offers the ability to define the backup policies per VM using the ghettoVCB-vm_backup_configuration_template file, in the event you don't want to shut down all the VMs you're backing up.

Link to comment
  • 1 month later...

@kaigauy are you backing up to unRaid protected array or elsewhere?

 

Just a possible heads up for others:

 

I get some vmdk errors with thin VM backing up as thin to unRaid server and it stops.

https://communities.vmware.com/thread/329611

This seems to suggest if the NFS datastore can't support thin provisioning then backup may fail.

 

If I backup the thin VM as zeroedthick it seems to work with this note popping up:

DISKLIB-FLAT : Unable to reserve the space: The specified feature is not supported by this version. But the disk will be created: "/vmfs/volumes................VMNAME-flat.vmdk".

It then keeps going along and you end up with the three files

.vmx

.vmdk

-flat.vmdk (which is the big file)

 

I haven't tried restoring from this but so far this is the only way to get a backup that runs through.

I'll add more as I test more.

 

Josh

Link to comment
  • 7 months later...
  • 8 years later...

I also have ghettoVCB write to an unRAID shre via NFS. This worked fine until last december. Since december ghettoVCB runs into timeouts when deleting older snapshots. Since the ESXi and ghettoVCB have not been changed, but I installed the unRAID OS updates, I suspect that it has something to do with the unRAID NFS daemon. I googled for solutions and the only proposed direction was to increase the timeout ESXi is using for NFS file removal. This didn't help in my case.

 

Anybody else ran into this problem? Were you able to solve it and how?

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.