kaiguy

I know I'm eagerly awaiting the beta of this bad boy!

@joshpond, I'm backing up as thin. The VMs are a mix of thin and thick.

This is going to be awesome! I'd like to echo DW's thoughts, as those are my primary 3 items that I look for. I also will take a look at the syslog just to make sure nothing funky is going on. Perhaps a quick "tail" option of the log would be a nice feature.

You are absolutely right. I knew I remembered reading something about snapshots and passthroughs a while back. ghettoVCB has the ability to shut down and restart the VMs during the backup process, which negates the need for snapshots. However, I don't know how that would interfere with the NFS mount. Observing how it currently takes place, all the VM cloning happens on the mount, as it is treated as a datastore. If the unRAID VM has to be shut down, I think it will break this process. I'm fine with the unRAID VM not bein backed up, since all of the real data is being stored on the flash drive. I manually make images of it from time to time. The shutdown would probably work fine for other VMs with passhtrough, however. ghettoVCB offers the ability to define the backup policies per VM using the ghettoVCB-vm_backup_configuration_template file, in the event you don't want to shut down all the VMs you're backing up.

Setup crontab to schedule the backup The dry run worked, and the manual backup worked. Now let's schedule this bad boy. 7. Decide when you want your backup to run A few things to keep in mind here. ESXi servers are always running on UTC time. Even when you set up an NTP server within the vSphere client, and it properly displays your timezone, the server itself is UTC. So remember this point when deciding your backup time. I wanted mine to run at 2:05am every Tuesday morning, so I'll have to tell ESXi to run it at 10:05am, since I'm in California (UTC minus . If you'd like to read up on cron format, google cron, or just check out this page. 8. Add the crontab entry WARNING: These next parts are specific for ESXi 5.1/5.5. If you are on another version, please refer to the ghettoVCB instructions. The file we will be adding our entry to is /var/spool/cron/crontabs/root. And here's what I added: 5 10 * * 2 /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB.sh -f /vmfs/volumes/2TBRED/ghettoVCB/backup_vm -g /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB.conf > /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB-backup-$(date +\%Y%m%d%H%M%S).log Its the same command we used above, but adding the 5 bits to tell the cronjob what time it is. I edited it directly from Transmit and it worked fine. Whatever you do, just make sure the method it's Linux-friendly (Windows users have been warned). Kill cron and restart: kill $(cat /var/run/crond.pid) crond 9. Make sure ESXi retains this change on every boot Edit the file /etc/rc.local.d/local.sh and add the following before the "exit 0" /bin/kill $(cat /var/run/crond.pid) /bin/echo "5 10 * * 2 /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB.sh -f /vmfs/volumes/2TBRED/ghettoVCB/backup_vm -g /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB.conf > /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB-backup-\$(date +\\%Y%m%d%H%M%S).log" >> /var/spool/cron/crontabs/root crond Again, you'll need to change the volume to match your setup. Also note, the format around the date portion is different from the command we've executed in the past (and added to the cron job). It's formatted in such a way for the "echo" to enter the command in the cron properly. Please make sure this portion (ghettoVCB-backup-\$(date +\\%Y%m%d%H%M%S).log) is properly entered. And finally, save the ESXi configuration with this command: /sbin/auto-backup.sh 10. Profit If you've followed these steps, you should feel confident that your backup will run on the schedule you've set. I rebooted my system just to make sure the cronjob indeed survived the reboot. It did. Hope this helps some people! If you know of a better method to go about any of these parts, please let me know and I will update this post accordingly. How to restore a backed up VM What if you need to restore a backup? Well, there are a few ways to go about doing this. ghettoVCB comes with an experimentally supported script called ghettoVCB-restore.sh. However, unRAID user joshpond has run into issues getting it to work. While the script would likely be the fastest way to get back up and running (if it worked), I have another solution that I have confirmed to work. 1. Create an NFS datastore, if non-persistent in your backup configuration In the vSphere client, go to Configuration | Storage | Add Storage and select Network File System. Follow the same guidelines I explained for backup in the NFS section above (Server will be your unRAID IP address, Folder will depend on if using a disk or user share). You can check the box to mount read-only to be on the safe side, if you so desire. Give the datastore any name you wish. In keeping with my example, Server: 192.168.0.10 Folder: /mnt/disk10/Backup Mount NFS read only: Unchecked Datastore Name: nfsrestore 2. Browse the datastore Right-click on your newly mounted NFS datastore, and select Browse Datastore. Go into the folder of the VM that you'd like to restore, then select the backup folder you want (the folder with the most recent date, I would assume). Right-click on the VM's backup folder and select Copy. Close out the datastore browser and select the datastore you wish to restore to. Again, right-click and select Browse Datastore. Right-click in the empty space and select Paste. Depending on the size of the backup, it should take quite some time, so grab a beer, eat some food, and watch a movie (e.g., joshpond reported a 350GB VM took 4 hours from his unRAID array to a RAID 0 datastore via this method). 3. Add the restored VM to the inventory Once the copy completes, right-click the folder (should contain some sort of date format) and rename it to the VM name. It should probably be as simple as wiping out the numerical portion. Double click on the newly renamed folder, right click on the .vmk file, and select Add to Inventory. When prompted if you Copied or Moved the VM, select Moved (if you are truly restoring the VM--it will maintain the old MAC address and some other variables). For housekeeping, you may want to delete the file STATUS.ok from the restored directory, and delete the unRAID NFS restore mount in the Storage configuration of the vSphere client.. Changes to guide 02/27/2014 - Added a real-world restore time example. 02/25/2014 - Changed the date formatting for logs to be more descriptive (previous version was %s, seconds since 1970-01-01 00:00:00 UTC, which is not meaningful, IMHO); Added guide to restore VMs.

HOW TO Backup your ESXi VMs to unRAID on a scheduled basis with ghettoVCB Background I recently experienced a catastrophic datastore failure when migrating my ESXi server into a Norco RPC-4224 chassis--one of the backplanes fried my datastore drives. In an attempt to speed up any future recovery process (I lost 3 VMs outright), I decided to institute a scheduled backup solution. Disclaimer I am in no way an expert with ESXi. This solution works for me, using the primary ghettoVCB informational document as a guide (https://communities.vmware.com/docs/DOC-8760). Keep in mind this is a fairly simple backup, and not a fault-tolerant solution that many on here are using (ZFS array over NFS, for example). I am basing this off of ESXi 5.5, but should be identical for 5.1. Any other versions you should take a look at the above referenced document for variations in configuration. Also note, I could not get this to backup unRAID without powering down the VM. In my configuration, I keep my VMs spinning, and the script takes advantage of snapshots to make the backup. I assume there is some issue with snapshotting unRAID, be it in my configuration or in general. YMMV. As such, I exclude unRAID from the VM backups. What you'll need The most recent version of ghettoVCB at github https://github.com/lamw/ghettoVCB/downloads WinSCP or Transmit or some other SFTP client A working unRAID array 1. Enable NFS, if you haven't already. We will need to create an NFS share to store the backups on your unRAID array. I have one array disk dedicated to backups, so I exported that disk with NFS. A user share should work too, but I personally haven't tried it. If you want to enable the user share, just go into the share settings and export the share via NFS. In my case, I'm exporting disk10. 2. Extract and configure ghettoVCB There seems to be two options to configure ghettoVCB. Either editing the script itself or via a global configuration file. We're going to go with the latter. Edit ghettoVCB.conf with a Linux-friendly text editor (notepad++, TextMate, etc.). The items I changed are as follows: VM_BACKUP_VOLUME=/vmfs/volumes/nfsbackup/ESXi I can't tell that this made a difference at all, aside from the backup not being saved under an "ESXi" directory. VM_BACKUP_ROTATION_COUNT=2 I don't need more than 2 copies. Default is 3. ENABLE_NON_PERSISTENT_NFS=1 UNMOUNT_NFS=1 NFS_SERVER=192.168.0.10 NFS_MOUNT=/mnt/disk10 NFS_LOCAL_NAME=nfsbackup NFS_VM_BACKUP_DIR=Backup This enables a non-persistent NFS share to be mounted, which is then unmounted after the backup process completes. The NFS_SERVER should be your unRAID IP. Make sure this is static! If you are using a disk share, it will be /mnt/diskX. If you are using a user share, it should be /mnt/user/usersharename. For NFS_VM_BACKUP_DIR, I use "Backup", since I also have a user share named Backup (if I ever add another drive). So this essentially is putting the VM backups in the root of the Backup user share. If you're connecting to a user share, you should be able to have this be whatever you'd like. I am uncertain if you can specify a deeper path. Everything else in ghettoVCB.conf remained as default. 3. Create a list of VMs to backup Since I don't want to power down my VMs to backup, I opted to exclude unRAID. We could use an exclude file, but I'd rather specify what VMs I want backed up, since I spin up ones pretty regularly to test things out. So we'll create an include file. I named mine backup_vm and saved it in the ghettoVCB folder I'm working in. Here's where things can be tricky. Whatever names you have for your VMs, as displayed within the vSphere client, must not have brackets or parentheses. This will cause your backups to fail (I had to remove parenthesis from mine). If you have other symbols, you may run into a problem as well. Spaces are fine. In your include file, list the names exactly as they appear in the client. For example, Windows 7-pandora Windows XP-hyperion Ubuntu-telesto CentOS-janus Have you figured out my VM naming convention? My server's name is "saturn"... Anywhoo. 4. Transfer the ghettoVCB folder to a datastore I'm assuming here you have enough knowledge of how to do this part. If not, drop me a post and I'll expand upon this section. Here's my path: /vmfs/volumes/2TBRED/ghettoVCB. Once it was transfered, the two .sh files already had execute privileges. If yours doesn't, or you just want to be safe, go to your ghettoVCB directory via SSH and run these commands: chmod +x ghettoVCB.sh chmod +x ghettoVCB-restore.sh 5. Let's do a dry run! Ok, so you've configured your .conf file, you've created a list of VMs to backup, and you've copied everything over to an ESXi datastore. Time to see if this bad boy works. SSH to the ghettoVCB folder, and run this command: ./ghettoVCB.sh -f backup_vm -g ghettoVCB.conf -d dryrun This is telling the script to (-f) backup VMs in our backup_vm file, (-g) using ghettoVCB.conf as the config file, and (-d) doing it as a dry run to see if we have errors. Hopefully it'll look something like this: # ./ghettoVCB.sh -f backup_vm -g ghettoVCB.conf -d dryrun Logging output to "/tmp/ghettoVCB-2014-02-24_08-01-52-42917.log" ... 2014-02-24 08:01:52 -- info: ============================== ghettoVCB LOG START ============================== 2014-02-24 08:01:52 -- info: CONFIG - USING GLOBAL GHETTOVCB CONFIGURATION FILE = ghettoVCB.conf 2014-02-24 08:01:52 -- info: CONFIG - VERSION = 2013_26_11_2 2014-02-24 08:01:52 -- info: CONFIG - GHETTOVCB_PID = 42917 2014-02-24 08:01:52 -- info: CONFIG - VM_BACKUP_VOLUME = /vmfs/volumes/nfsbackup/Backup 2014-02-24 08:01:52 -- info: CONFIG - ENABLE_NON_PERSISTENT_NFS = 1 2014-02-24 08:01:52 -- info: CONFIG - UNMOUNT_NFS = 1 2014-02-24 08:01:52 -- info: CONFIG - NFS_SERVER = 192.168.0.10 2014-02-24 08:01:52 -- info: CONFIG - NFS_VERSION = nfs 2014-02-24 08:01:52 -- info: CONFIG - NFS_MOUNT = /mnt/disk10 2014-02-24 08:01:52 -- info: CONFIG - VM_BACKUP_ROTATION_COUNT = 2 2014-02-24 08:01:52 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2014-02-24_08-01-52 2014-02-24 08:01:52 -- info: CONFIG - DISK_BACKUP_FORMAT = thin 2014-02-24 08:01:52 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = 0 2014-02-24 08:01:52 -- info: CONFIG - ENABLE_HARD_POWER_OFF = 0 2014-02-24 08:01:52 -- info: CONFIG - ITER_TO_WAIT_SHUTDOWN = 3 2014-02-24 08:01:52 -- info: CONFIG - POWER_DOWN_TIMEOUT = 5 2014-02-24 08:01:52 -- info: CONFIG - SNAPSHOT_TIMEOUT = 15 2014-02-24 08:01:52 -- info: CONFIG - LOG_LEVEL = dryrun 2014-02-24 08:01:52 -- info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCB-2014-02-24_08-01-52-42917.log 2014-02-24 08:01:52 -- info: CONFIG - ENABLE_COMPRESSION = 0 2014-02-24 08:01:52 -- info: CONFIG - VM_SNAPSHOT_MEMORY = 0 2014-02-24 08:01:52 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = 0 2014-02-24 08:01:52 -- info: CONFIG - ALLOW_VMS_WITH_SNAPSHOTS_TO_BE_BACKEDUP = 0 2014-02-24 08:01:52 -- info: CONFIG - VMDK_FILES_TO_BACKUP = all 2014-02-24 08:01:52 -- info: CONFIG - VM_SHUTDOWN_ORDER = 2014-02-24 08:01:52 -- info: CONFIG - VM_STARTUP_ORDER = 2014-02-24 08:01:52 -- info: CONFIG - RSYNC_LINK = 0 2014-02-24 08:01:52 -- info: CONFIG - EMAIL_LOG = 0 2014-02-24 08:01:52 -- info: 2014-02-24 08:01:53 -- dryrun: ############################################### 2014-02-24 08:01:53 -- dryrun: Virtual Machine: Windows 7-pandora 2014-02-24 08:01:53 -- dryrun: VM_ID: 3 2014-02-24 08:01:53 -- dryrun: VMX_PATH: /vmfs/volumes/SSD1/Win7Pro/Win7Pro.vmx 2014-02-24 08:01:53 -- dryrun: VMX_DIR: /vmfs/volumes/SSD1/Win7Pro 2014-02-24 08:01:53 -- dryrun: VMX_CONF: Win7Pro/Win7Pro.vmx 2014-02-24 08:01:53 -- dryrun: VMFS_VOLUME: SSD1 2014-02-24 08:01:53 -- dryrun: VMDK(s): 2014-02-24 08:01:53 -- dryrun: /vmfs/volumes/blahblahblahblahblahblahblahblah/Win7Pro/Win7Pro_2.vmdk 150 GB 2014-02-24 08:01:53 -- dryrun: Win7Pro.vmdk 55 GB 2014-02-24 08:01:53 -- dryrun: INDEPENDENT VMDK(s): 2014-02-24 08:01:53 -- dryrun: TOTAL_VM_SIZE_TO_BACKUP: 205 GB 2014-02-24 08:01:53 -- dryrun: ############################################### 2014-02-24 08:01:53 -- dryrun: ############################################### 2014-02-24 08:01:53 -- dryrun: Virtual Machine: Windows XP-hyperion 2014-02-24 08:01:53 -- dryrun: VM_ID: 22 2014-02-24 08:01:53 -- dryrun: VMX_PATH: /vmfs/volumes/2TBRED/Windows XP/Windows XP.vmx 2014-02-24 08:01:53 -- dryrun: VMX_DIR: /vmfs/volumes/2TBRED/Windows XP 2014-02-24 08:01:53 -- dryrun: VMX_CONF: Windows XP/Windows XP.vmx 2014-02-24 08:01:53 -- dryrun: VMFS_VOLUME: 2TBRED 2014-02-24 08:01:53 -- dryrun: VMDK(s): 2014-02-24 08:01:53 -- dryrun: Windows XP.vmdk 20 GB 2014-02-24 08:01:53 -- dryrun: INDEPENDENT VMDK(s): 2014-02-24 08:01:53 -- dryrun: TOTAL_VM_SIZE_TO_BACKUP: 20 GB 2014-02-24 08:01:53 -- dryrun: ############################################### 2014-02-24 08:01:53 -- dryrun: ############################################### 2014-02-24 08:01:53 -- dryrun: Virtual Machine: Ubuntu-telesto 2014-02-24 08:01:53 -- dryrun: VM_ID: 21 2014-02-24 08:01:53 -- dryrun: VMX_PATH: /vmfs/volumes/2TBRED/Ubuntu/Ubuntu.vmx 2014-02-24 08:01:53 -- dryrun: VMX_DIR: /vmfs/volumes/2TBRED/Ubuntu 2014-02-24 08:01:53 -- dryrun: VMX_CONF: Ubuntu/Ubuntu.vmx 2014-02-24 08:01:53 -- dryrun: VMFS_VOLUME: 2TBRED 2014-02-24 08:01:53 -- dryrun: VMDK(s): 2014-02-24 08:01:53 -- dryrun: Ubuntu.vmdk 250 GB 2014-02-24 08:01:53 -- dryrun: INDEPENDENT VMDK(s): 2014-02-24 08:01:53 -- dryrun: TOTAL_VM_SIZE_TO_BACKUP: 250 GB 2014-02-24 08:01:53 -- dryrun: ############################################### 2014-02-24 08:01:53 -- info: ###### Final status: OK, only a dryrun. ###### 2014-02-24 08:01:53 -- info: ============================== ghettoVCB LOG END ================================ If the final status is OK, it's time to run this for real. 6. Perform the backup, using full paths in the command So let's take our test command: ./ghettoVCB.sh -f backup_vm -g ghettoVCB.conf -d dryrun Get rid of the dryrun portion, and use the full paths to the files. /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB.sh -f /vmfs/volumes/2TBRED/ghettoVCB/backup_vm -g /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB.conf > /vmfs/volumes/2TBRED/ghettoVCB/ghettoVCB-backup-$(date +\%Y%m%d%H%M%S).log Notice we also added a bit to export the results into a dated log file. Adjust the datastore name accordingly. Hopefully that worked! My NFS share amazingly connected on the first try, and unmounted automatically as intended. Not sure if the dry runs actually create the mount or not, but you'll definitely know at this stage if it did. If all went well, it's time to setup a cronjob. To the next post...

Super helpful thread and tools. Finally got all my firmwares up to date. Thanks BetaQuasi!

Is 12a the recommended beta for use with an LSI controller at the moment? Any drawbacks to using 12a coming from 14 (I seem to recall it was fine when I was running that version)? Just got my m1015 and want to give it a whirl.

Looks to me that the rear fan is blowing outward, not inward.

roastdawgg, those are awesome pictures. Really clean build! I don't know how you guys get your cables looking so neat--I can't do them with just zipties. On a different note, I noticed your RAM is installed in different colored slots/directly next to each other. If you'd like to run it in dual channel mode (which I think you would considering you have a matched pair), you're going to want to install them in the A1 and B1 slots.