UnRAID on VMWare ESXi with Raw Device Mapping

[heffe2001]

Have you tried it with ESXi managing the SAS card directly, and using RDM to pass the drives over to the Unraid VM (using the LSI SAS emulated controller) like in the first post of this thread?  If ESXi supports the controller directly that should work.  I'm not sure if there would be a performance hit or not (I'm not doing that, I'm basically doing what you're attempting, but with the LSI card mentioned in this thread).

 

I would think if the ESXi software sees your controller and allows you to set up RDM with the drives connected to it, it would work in a VM Unraid.

 

 

[heffe2001]

Can you post your syslog for the native mode (not under ESXi) and your hardware config if it is not a big trouble.  

 

I would have to pull out the ESXi boot drive, and reconfigure the drives again, but that shouldn't be a huge problem.  

 

As for hardware I'm using:

 

ASUS M4A89TD PRO/USB3 AM3 AMD 890FX SATA 6Gb/s USB 3.0 ATX AMD Motherboard

AMD Phenom II X4 810 Deneb 2.6GHz Socket AM3 95W Quad-Core Processor

G.SKILL NS 4GB (2 x 2GB) 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10666) Desktop Memory (I REALLY need another set of these to go up to 8g)

3 x Western Digital Caviar Green WD20EADS 2TB SATA 3.0Gb/s 3.5" Internal Hard Drive

LSI Logic SAS3081E-S 8 Port SAS Controller PCIE 8X (on passthrough to the unraid VM)

30gb OCZ Core v2 SSD (for ESXi Boot drive & unraid VM)

160gb Seagate Sata (for datastore2, using it for my Win7 VM)

1gb no-name USB key

 

 

I just checked my logs under ESXi, and I did notice a good many:

 

mdcmd (8914): spindown1

md: disk1: ATA_OP_STANDBYNOW1 ioctl error: -22

 

When I had it set up native to my controller, I don't think the system had any time to spin down (was dumping data to the 2 data drives as fast as they'd take it), so I'm not 100% sure it wasn't doing this before the VM entered into it...  I disabled spindown for now and everything appears to be working correctly.  Since I'm going to a lower drive count machine, I don't mind them spinning all the time.  Not sure if the versions above were supposed to fix that problem or not though.

 

 

[Ford Prefect]

Have you tried it with ESXi managing the SAS card directly, and using RDM to pass the drives over to the Unraid VM (using the LSI SAS emulated controller) like in the first post of this thread?  If ESXi supports the controller directly that should work.  I'm not sure if there would be a performance hit or not (I'm not doing that, I'm basically doing what you're attempting, but with the LSI card mentioned in this thread).

 

I would think if the ESXi software sees your controller and allows you to set up RDM with the drives connected to it, it would work in a VM Unraid.

 

 

No I did not try that.

I am sure that it will work, though.

That kind of setup is my last resort. Reads from my external unRAID to the

Win7-VM - using teracopy - maxed out at 37Mbyte/sec (using a e1000 inside the VM).

That's about 50-60% of the native average performance I usually get.

But this I can also get with KVM where I do not need a Win-Box to manage all that.

[SK]

I just checked my logs under ESXi, and I did notice a good many:

 

mdcmd (8914): spindown1

md: disk1: ATA_OP_STANDBYNOW1 ioctl error: -22

 

When I had it set up native to my controller, I don't think the system had any time to spin down (was dumping data to the 2 data drives as fast as they'd take it), so I'm not 100% sure it wasn't doing this before the VM entered into it...  I disabled spindown for now and everything appears to be working correctly.  Since I'm going to a lower drive count machine, I don't mind them spinning all the time.  Not sure if the versions above were supposed to fix that problem or not though.

 

the error means that LSI card doesn't support ATA spindown, if sg_start (from sg3_utils) works and really put those drives to standby mode then my mod may help.

 

 

[SK]

Fresh version of unRAID-VM ISO  based on unRAID 4.6 and stable 2.6.35.9 kernel available at http://www.mediafire.com/?2710vppr8ne43

 

check README for details

[SK]

FYI - under ESXi for VMs backup I start using script from http://communities.vmware.com/docs/DOC-8760, which pretty smart and skip physical RDMs (if someone want to backup unRAID VM). It even support gzip compression for small VMs

[queeg]

FYI - under ESXi for VMs backup I start using script from http://communities.vmware.com/docs/DOC-8760, which pretty smart and skip physical RDMs (if someone want to backup unRAID VM). It even support gzip compression for small VMs

 

I'm getting set up with this script now also.  My ESXi server and unRAID server are separate however.  Currently I'm having ESXi create NFS datastore that points to one of my unRAID shares and that is working.  

 

But I'm not sure how to get the email part working.  The ghettoVCD script doesn't take parameters for SMTP authentication and it's using nc internally and I don't know if it can handle auth either.

Any suggestions or help would sure be appreciated.

[SK]

FYI - under ESXi for VMs backup I start using script from http://communities.vmware.com/docs/DOC-8760, which pretty smart and skip physical RDMs (if someone want to backup unRAID VM). It even support gzip compression for small VMs

 

I'm getting set up with this script now also.  My ESXi server and unRAID server are separate however.  Currently I'm having ESXi create NFS datastore that points to one of my unRAID shares and that is working.  

 

But I'm not sure how to get the email part working.  The ghettoVCD script doesn't take parameters for SMTP authentication and it's using nc internally and I don't know if it can handle auth either.

Any suggestions or help would sure be appreciated.

 

I have this working on single physical server, during backup window ESXi host mounts NFS backup share as datastore from unraid VM, perform hot backup (by doing VM snapshot and then clone) and then unmount it. There are some issue with enabling experimental gzip based compression - it did not work for me on large VMs, but works fine without compression.

 

I have not look into email notification though, but don't think netcat can handle SMTP authentication. Perhaps using local SMTP proxy on another VM/host without authentication can help with this, something like sSTMP - http://wiki.debian.org/sSMTP

 

[queeg]

What I've done is make a NFS datastore in ESXi where the backups are copied to which works great and put the ghettoVCD scripts onto the NFS share as well.  

 

I also tried MKSBackup which is a wrapper for ghettoVCD and it runs from a linux or Windows machine to run the backup.  It handles the email correctly but it's method to do the backup is intended to be pscp.exe which isn't needed and also fails with error.

MKSBackup wants to be configured so that ghettoVCD backs up the VM to a ESXi local datastore and then MKSBackup has pscp.exe copy that to the remote destination.  I don't need it to do that because the NFS link already looks like a local datastore to ghettoVCD.

 

EDIT:  After more reading I believe it's may be possible to disable the pscp copy in MKSBackup so I'm going to see if I can set that up that way.  

 

I'll look into the sSMTP utility.

 

BTW, what did you do to set up your NFS connection?  Is it just from settings in the ghettoVCD.conf?

[SK]

What I've done is make a NFS datastore in ESXi where the backups are copied to which works great and put the ghettoVCD scripts onto the NFS share as well.  

 

I also tried MKSBackup which is a wrapper for ghettoVCD and it runs from a linux or Windows machine to run the backup.  It handles the email correctly but it's method to do the backup is intended to be pscp.exe which isn't needed and also fails with error.

MKSBackup wants to be configured so that ghettoVCD backs up the VM to a ESXi local datastore and then MKSBackup has pscp.exe copy that to the remote destination.  I don't need it to do that because the NFS link already looks like a local datastore to ghettoVCD.

 

EDIT:  After more reading I believe it's may be possible to disable the pscp copy in MKSBackup so I'm going to see if I can set that up that way.  

 

I'll look into the sSMTP utility.

 

BTW, what did you do to set up your NFS connection?  Is it just from settings in the ghettoVCD.conf?

 

just vhettoVCB settings related to non persistent nfs connection, when enabled (ENABLE_NON_PERSISTENT_NFS, etc) it will setup nfs datastore on fly just for backup duration

 

[queeg]

just vhettoVCB settings related to non persistent nfs connection, when enabled (ENABLE_NON_PERSISTENT_NFS, etc) it will setup nfs datastore on fly just for backup duration

 

 

Good to know. 

 

 

I've given the email a little more thought.  While the MKSbackup wrapper does send email I don't think getting an email for each backup is that useful.  I'm thinking a very concise log that is monitored by some process and reports problems might be better. 

I'm also thinking that having and unRAID cron job start the backup would be optimum.  The way mine is set up with a persistant NFS link it would be possible for unRAID to connect to ESXi and run the job off the NFS link which itself points back to unRAID.  That way, with the scheduler, the backup script, the logs and the backups themselves on unRAID everything is under parity protection. 

But how sturdy is the persistant NFS link?  I'm a complete newbie to NFS so I don't know how it behaves if one of the sides goes offline for a reboot or something?

 

 

[heffe2001]

Just wanted to reply and let you guys know with the latest supplied version (the one based on 4.6), it DOES spin down my drives when running with the card passed through.  Still no temps, but that isn't a problem at all for me.

 

This is under ESXi, with my 1068 card passed through to the VM...

 

 

[SK]

I've given the email a little more thought.  While the MKSbackup wrapper does send email I don't think getting an email for each backup is that useful.  I'm thinking a very concise log that is monitored by some process and reports problems might be better. 

I'm also thinking that having and unRAID cron job start the backup would be optimum.  The way mine is set up with a persistant NFS link it would be possible for unRAID to connect to ESXi and run the job off the NFS link which itself points back to unRAID.  That way, with the scheduler, the backup script, the logs and the backups themselves on unRAID everything is under parity protection. 

But how sturdy is the persistant NFS link?  I'm a complete newbie to NFS so I don't know how it behaves if one of the sides goes offline for a reboot or something?

 

Nfs client reboot is fine as soon as it remounts when gets back online, nfs server reboot - likely require remount on client(s) side to avoid stalled mount points.

 

in my single server config I have logs and backup on NFS unRAID datastore, backup script is cron scheduled on ESXi which run off compact flash.

 

 

[queeg]

I've given the email a little more thought.  While the MKSbackup wrapper does send email I don't think getting an email for each backup is that useful.  I'm thinking a very concise log that is monitored by some process and reports problems might be better.  

I'm also thinking that having and unRAID cron job start the backup would be optimum.  The way mine is set up with a persistant NFS link it would be possible for unRAID to connect to ESXi and run the job off the NFS link which itself points back to unRAID.  That way, with the scheduler, the backup script, the logs and the backups themselves on unRAID everything is under parity protection.  

But how sturdy is the persistant NFS link?  I'm a complete newbie to NFS so I don't know how it behaves if one of the sides goes offline for a reboot or something?

 

Nfs client reboot is fine as soon as it remounts when gets back online, nfs server reboot - likely require remount on client(s) side to avoid stalled mount points.

 

in my single server config I have logs and backup on NFS unRAID datastore, backup script is cron scheduled on ESXi which run off compact flash.

 

Well, I changed from the persistant connection to allowing ghettoVCB mount and unmount the NFS connection.

 

I saw errors in the system log about the onboard Realtek nic so I decided to disable it in bios.  After some rebooting and fiddling all network connections were lost and based on help from the ESXi forum I ended up resetting the configuration   and had to set up the networking from scratch and add the VM's back to inventory.  Grumble.

 

EDIT:  1/2/2011

I've changed to doing non-persistant NFS connections with ghettoVCB and that's workiing perfectly.  Also, in case you haven't tried the Veeam Monitor application I'd just encourage you to try it out.  It's amazing.

 

 

 

[SK]

I've given the email a little more thought.  While the MKSbackup wrapper does send email I don't think getting an email for each backup is that useful.  I'm thinking a very concise log that is monitored by some process and reports problems might be better.  

I'm also thinking that having and unRAID cron job start the backup would be optimum.  The way mine is set up with a persistant NFS link it would be possible for unRAID to connect to ESXi and run the job off the NFS link which itself points back to unRAID.  That way, with the scheduler, the backup script, the logs and the backups themselves on unRAID everything is under parity protection.  

But how sturdy is the persistant NFS link?  I'm a complete newbie to NFS so I don't know how it behaves if one of the sides goes offline for a reboot or something?

 

Nfs client reboot is fine as soon as it remounts when gets back online, nfs server reboot - likely require remount on client(s) side to avoid stalled mount points.

 

in my single server config I have logs and backup on NFS unRAID datastore, backup script is cron scheduled on ESXi which run off compact flash.

 

Well, I changed from the persistant connection to allowing ghettoVCB mount and unmount the NFS connection.

 

I saw errors in the system log about the onboard Realtek nic so I decided to disable it in bios.  After some rebooting and fiddling all network connections were lost and based on help from the ESXi forum I ended up resetting the configuration   and had to set up the networking from scratch and add the VM's back to inventory.  Grumble.

 

EDIT:  1/2/2011

I've changed to doing non-persistant NFS connections with ghettoVCB and that's workiing perfectly.  Also, in case you haven't tried the Veeam Monitor application I'd just encourage you to try it out.  It's amazing.

 

thanks - will get a try with Veeam Monitor free version, preso on their site looks promising

 

[ChaOConnor]

Hi!  Read all 11 pages and I had a question.  Would it be possible to go the other way?  If I setup unRaid, can I then later put ESXi on and import a VM of the unRaid USB?

 

OR

 

If I follow these instructions and I can't quite get unRaid to work under ESXi, can I bypass ESXi and just boot off the unRaid USB and have a functioning unRaid server?

 

Thanks!

 

- Cha

[ChaOConnor]

So i have unRaid setup and working. Can i make an image of the usb and just import that into esxi? Thanks!

[fade23]

I couldn't find a good answer by searching. Do SK's patches support MPT2SAS for LSI SAS2008 chipsets?

[heffe2001]

I've been running the 4.6 version of this download for about a week on my production machine, and had a couple issues..  I forgot to disable the spindown timer, and it appears that with it on, it makes unraid throw errors on the drives connected to my LSI controller.  The main unraid status page didn't show any errors, but when I checked out the unmenu page, it was showing around 600k errors on all 8 drives connected to this controller.  I'm guessing it was after they spun down, it for whatever reason wouldn't spin back up.  Had the problem again today with one drive spinning down, and not coming back up, but unraid marked it disk_dsbl, although smart status shows no problems at all.  It's currently rebuilding parity on that particular drive, and I remembered to disable the spindown this time...

 

Hopefully that's what the problem was, I can live without spindown for the short term, but I think I'm going to end up swapping that controller out for a MV8, can't afford the system going down when I'm out of town, the wife would kill me, lol.

 

 

[SK]

I've been running the 4.6 version of this download for about a week on my production machine, and had a couple issues..  I forgot to disable the spindown timer, and it appears that with it on, it makes unraid throw errors on the drives connected to my LSI controller.  The main unraid status page didn't show any errors, but when I checked out the unmenu page, it was showing around 600k errors on all 8 drives connected to this controller.  I'm guessing it was after they spun down, it for whatever reason wouldn't spin back up.  Had the problem again today with one drive spinning down, and not coming back up, but unraid marked it disk_dsbl, although smart status shows no problems at all.  It's currently rebuilding parity on that particular drive, and I remembered to disable the spindown this time...

 

Hopefully that's what the problem was, I can live without spindown for the short term, but I think I'm going to end up swapping that controller out for a MV8, can't afford the system going down when I'm out of town, the wife would kill me, lol.

 

 

For drives spindown to work it should be supported by controller (such LSI) in first place. For instance LSI1068 based cards I dealt with had no correct spindown support.

 

To clarify a few things:

 

1) the patched version (unraid-VM distro) is intended for unraid in Vmware ESXi VM only, list of deviations from original one is in README file. Virtualization layer between hardware and unraid distro brings challenges partially addressed by the patch to open-sourced unraid driver. Other challenges are in management interface and needs to be addressed by Lime given closed source of unraid management code.

 

2) there are no additional code to support any unsupported by standard unraid version controllers other than additional handling of drives discovery. Also patched version based on newer linux kernel.

 

3) Running unraid-VM distro on physical server should be ok, but if controller doesn't handle spindown/spinup correctly then spindown needs to be disabled.

 

 

[heffe2001]

For drives spindown to work it should be supported by controller (such LSI) in first place. For instance LSI1068 based cards I dealt with had no correct spindown support.

 

To clarify a few things:

 

1) the patched version (unraid-VM distro) is intended for unraid in Vmware ESXi VM only, list of deviations from original one is in README file. Virtualization layer between hardware and unraid distro brings challenges partially addressed by the patch to open-sourced unraid driver. Other challenges are in management interface and needs to be addressed by Lime given closed source of unraid management code.

 

2) there are no additional code to support any unsupported by standard unraid version controllers other than additional handling of drives discovery. Also patched version based on newer linux kernel.

 

3) Running unraid-VM distro on physical server should be ok, but if controller doesn't handle spindown/spinup correctly then spindown needs to be disabled.

 

 

Turning off Spindown definitely fixed the issue, but of course with the hit in power consumption.  I need to put the system on my Kill-a-watt to see what the draw is when they are all spinning just to see how bad it is.  Most of my drives are green WD's, but not all of them (I do have a couple blues and a black).  Like I said, I can live with it as it is for now, but I'm still considering swapping out the LSI controller for one of the MV8's, just not 100% sure I want to make any changes on the hardware as it is for now.  Anybody know if the mv8 can be used with passthrough in ESXi?  I'm still considering using this server as a multi-host machine..

 

[Ford Prefect]

Anybody know if the mv8 can be used with passthrough in ESXi?  I'm still considering using this server as a multi-host machine..

 

..the AOC-SASL-MV8 (8Port PCI-e) did not work with ESXi and vmdirectpath.

[SilntBob]

Anybody know if the mv8 can be used with passthrough in ESXi?  I'm still considering using this server as a multi-host machine..

 

..the AOC-SASL-MV8 (8Port PCI-e) did not work with ESXi and vmdirectpath.

 

Which devices, at this point, have been confirmed to work with unraid with vmdirectpath passthrough?

[SilntBob]

So, at this point I'm just testing an UnRAID installation on my ESXi server before I consider migrating any data over to it....but I've run into an interesting problem/situation.

 

I had unRAID configured to use an old 500GB drive that I had around here + a 2TB "green" drive + another 2TB "green" drive for the parity drive.

 

Some data had already been copied to the 2TB drive from a previous UnRAID test, so I configured a new array and the parity check started.

 

While the parity check was running I threw some other trash/test data that I had onto the 500GB drive.

 

In the process of completing the parity check the 500GB drive finally died on me (it had been running hot, and I had had some issues with it, that is why I decided to use it as part of this test)

 

What was interesting to me is that after the drive died I was having problems accessing unRAID via the web interface or SMB share the connection would just hang when trying to connect.  Telnet connections were still working fine, so it probably isn't a network stack issue.

 

looking at the syslog data (which I don't have saved from *that* iteration) there were some complaints about some corruption on the filesystem of the USB device.

 

I spent some time trying to get the server to become accessible again before giving up and "starting over" by formatting a new USB stick with a default unRAID install.

After reformatting I created a new array using just the 2 2TB green drives, one as parity and the other as a data drive.

 

After reconfiguring the array came up without issue and a parity check was started.

 

I had done pretty much zero customization of the settings from a default install.

 

I let that run over night.  The last time I refreshed the progress of the parity check it had about 181 mins left.

After more than that amount of time had passed I decided to refresh the progress window again and it was hung again.....

 

I have attached my syslog from this current cycle.

Reviewing it I notice some interesting/concerning activity.

 

Throughout the syslog I see events like:

Jan 24 14:55:24 Tower in.telnetd[3461]: connect from 41.239.223.22 (41.239.223.22)
Jan 24 14:55:27 Tower telnetd[3461]: ttloop: peer died: EOF
Jan 24 15:20:24 Tower in.telnetd[3562]: connect from 125.237.40.101 (125.237.40.101)
Jan 24 15:20:24 Tower telnetd[3562]: ttloop: peer died: EOF
Jan 24 15:37:17 Tower in.telnetd[3629]: connect from 92.99.132.165 (92.99.132.165)
Jan 24 15:37:17 Tower telnetd[3629]: ttloop: peer died: EOF
Jan 24 16:19:27 Tower in.telnetd[3802]: connect from 92.96.200.166 (92.96.200.166)
Jan 24 16:19:27 Tower telnetd[3802]: ttloop: peer died: EOF
Jan 24 16:25:56 Tower in.telnetd[3829]: connect from 94.59.250.245 (94.59.250.245)
Jan 24 16:25:56 Tower telnetd[3829]: ttloop: peer died: EOF
Jan 24 16:37:33 Tower in.telnetd[3876]: connect from 81.213.114.110 (81.213.114.110)
Jan 24 16:37:34 Tower telnetd[3876]: ttloop: peer died: EOF
Jan 24 16:43:43 Tower in.telnetd[3901]: connect from 88.228.34.124 (88.228.34.124)
Jan 24 16:43:44 Tower telnetd[3901]: ttloop: peer died: EOF
Jan 24 16:43:53 Tower in.telnetd[3904]: connect from 85.107.32.163 (85.107.32.163)
Jan 24 16:43:54 Tower telnetd[3904]: ttloop: peer died: EOF
Jan 24 16:43:57 Tower in.telnetd[3905]: connect from 78.179.184.243 (78.179.184.243)
Jan 24 17:01:11 Tower telnetd[3905]: ttloop: read: Connection timed out
Jan 24 17:07:04 Tower in.telnetd[4000]: connect from 78.80.91.38 (78.80.91.38)
Jan 24 17:07:05 Tower telnetd[4000]: ttloop: peer died: EOF

 

This server is set up with the default configuration (no name change, no root password change, etc)

Network address is being issued by DHCP and it is being provided a NATed address that has NO port-forwarding enabled.

Those IP addresses are from places like Nigeria, UAE, Egypt, Turkey ....That is HIGHLY suspicious to me, does anyone have ANY idea why I would be seeing these connections?

 

I'm using the custom ISO image that was posted in this thread with the filename unraid_4.6-vm_1.0.6.iso and MD5 hash 82962143AAC52908D34052FD74D73CF1

can anyone else confirm that that MD5 ISO hash is "safe" ?

 

Since I'm just working with disposable, non-sensitive test data I'm obviously not overly concerned with security at this point, but I *am* concerned as to why I'm seeing unexpected IP connections to/from suspicious foreign IP addresses.

 

Looking at the rest of the syslog does anyone have any clue why I am now unable to access the unraid server via the web interface?

 

What should my next steps be?

 

Any help is greatly appreciated...thanks!

UnRaidsyslog.zip

[smino]

Yes, the power use is high, but i'm saving money with this mobo and ESXi. Prior to this I had 5 devices in my rack at home. Now I have 1 ESXi running the following VM's

 

unRAID 5 (much less power use than WHS and its inefficient use of HD space)

pfSense (Firewall/Router)

Mythbuntu (DVR/TV backend)

HomeSeer (inside Windows 7 for home automation)

Milestone (inside Windows 2008r2 for security cameras)

PBXinaFlash (VOIP/Asterisk/FreePBX)

 

Not to mention its much quieter in the basement now I'm only running one box and associated fans.

 

Jon

 

Noj, I am looking to do exactly what you did there.

Can you post all the details of your setup?

Does your mythtv play any 1080P source without any problems?