JimmyJoe

I am using the virtual NIC in ESXi, not passing one through to any VMs.

I setup a OI/napp-it guest (2 CPU 6 GB) and got ZFS up and running using with passthrough of an IBM1015. I am a complete newbie with ZFS. I was testing performance and I am a bit confused. I have 6 WD 1TB Green drives. I am planning on probably using 4 of them. I setup multiple pools (mirrored vdevs with 4 or 6 drives, raidz with 4 or 6 drives, and a single basic drive) and I get the SAME read performance across each of them ~70 MB/s. I tested with a Windows7 guest. I included results for a local single drive datastore as well as an unraid guest and an unraid sever over the network. From my unraid guest I see ~300 MB/s and from zfs I see ~70 MB/s, same as a single drive. Here is zfs mirrored and raidz with 4 and 6 drives: Here is unraid over network, unraid guest, zfs single basic disc, and local single datastore drive: Any ideas why I get the same read performance as a single drive regardless of how I configure zfs with 4 or 6 drives? What am I doing wrong?

Great price, I may have to get some more of these!

I am using 4TB drives with 5.0rc11 without issue. Works great.

Cool, sounds good. Looking forward to see how it works for you.

That's a good question, but I didn't really do any apples to apples comparisons. I will say that I haven't noticed any performance hit at all for running unRAID in a VM. It works very well.

Well... turns out neither. I think. I pulled the ASRock MB and put in my C2SEA and was still having problems. I figured out that I could reproduce a sata problem by spinning up/down all drives. So I just replaced my power supply and so far problems went away. I can spin up/down without issue. So I think my Corsair TX-750 power supply was the problem. I installed a new Corsair HX-750 and looks good so far, but I just put it in 30 minutes ago. Gonna let it run for a few days and then decide if I want to go back to the ASRock MB and use the on-board sata controller with the monoprice cables. Update - I went ahead and put the monoprice cables back in and updated the first post. I am going to leave the C2SEA in the box, that was really my long term plan anyway. So far so good! I think the ASRock MB would work fine, it did before I swapped in the "bad" power supply.

Thanks for confirming this! I'm going to proceed with pre-clearing a second 4 TB drive so I can add it as a data drive. BTW I'm not sure what you mean by "disk properties". I clicked on the link in the shortened serial number in mymain but I don't see a "disk properties" setting. And in the "disk settings" on the Settings page in Main, I don't see it either. Not terribly important as it's only cosmetic, just would be nice to hide it. Thanks for easing my mind. Sure. Sorry I wasn't clear. If you go to the default menu in MyMain you can click on the serial ID for a disk, and that will bring up the disk configuration. You can then add an attribute for hpa_ok and set it to 1. That tells MyMain that you are confirming there is no HPA on the disk, even though it isn't a know size that MyMain recognizes. Here's a picture of what one of mine looks like:

3,907,018,532 is correct. Mymain hasn't been updated with this value yet. You can go into disk properties and set: hpa_ok = 1 as a workaround until Mymain is updated.

I was getting some sata errors so I installed 2 Adaptec 1430sa controllers and changed the sata cables. I am currently using older 18" (red, straight end, non-locking) cables that came with some older sata controllers. That was four days ago, and I haven't seen a sata error since I swapped them out. So I suspect the problem I am having is either with my MB sata or with the 18" (black, straight end, locking) sata cables I bought from monoprice. I am going to let it run for a few more days and then swap stuff around again to try and determine if my problem is the MB or monoprice cables. OR maybe I'll just leave it alone since it's not broken anymore.

I wouldn't hesitate to buy it again for an 8 drive system (expandable to 13 with a 5in3 bay). I really like the case a lot. Too bad they don't make one with 12 drives and 3 fans in front. The removable fans are great and make it so you can remove a drive without any tools and only need to disconnect the sata and power cables from the one drive you want to swap out.

Only the basic version. The Plus and Pro versions need the flash drive passed through to the VM. You can also do a hybrid install. Most everything loads from the virtual disk (fast) and the unraid mounts and uses the flash.

I am using a Samsung 500GB drive that is passed through on the M1015 as my cache drive. I haven't tried to use a virtual disk. At some point, I am going to setup ZFS and probably move my cache drive to that.

Yep, the bottom pops off fairly easily without breaking anything or removing the drive. And here's a picture of the dock removed: http://lime-technology.com/forum/index.php?topic=25675.msg224226#msg224226 I've been able to remove the dock from the drives with no problem exposing the SATA and power connectors. However, when connected to power and SATA on my Server, none of the HDD's are recognized. I've tried a couple of computers with different power supplies, different cables, etc. and they don't show up in BIOS or O/S. I've tried connecting another 2TB WD20EARS, I have laying around in the same way and it's fine. Anyone have any ideas? I connected about 10 of these while still in the case. For me, the cables I had wouldn't fit very well inside the housing. I used a knife to shave off part of the plastic for the power and the sata cable in order to be able to connect securely to the drive. Are you sure your cables are connected and fully seated to the drive and not just snug in the plastic case?

I just set this up the first time recently. I connected 2 network cables from my switch to the motherboard, one to the dedicated IPMI port and one for network. I then installed IPMI View on a Win7 box and had it scan for devices. It found my IPMI on my MB, which automatically was assigned an IP via DHCP. I was using a different supermico MB, but the process should be the same I would think.

Problem Solved - see Updates below. I am having a hard time getting ssmtp to work and could use some help. I was previously using unraid_notify and had no problem sending email but want to try the email packages that are part of unmenu. This is on a test system. To troubleshoot the problem and remove any variables from my old installation I did a fresh install. 1. Format USB stick 2. Install 5.0rc11 and make bootable 3. Install unmenu 1.5 Revision: 246 4. Use unmenu Pkg Manager to install gcc-4.2.4-i486-1.tgz 5. Use unmenu Pkg Manager to install ssmtp_2.64.orig.tar.bz2 Before setting config I tried to use ssmtp, knowing it wouldn't work: root@Tower:/boot/unmenu# echo -e "Subject: a test\n\nThis is the email body." | ssmtp -d root ssmtp: Cannot open your_host:0 I then set the config and hit "Re-Install Now with Edited Variables" and tried again: root@Tower:/boot/unmenu# echo -e "Subject: a test\n\nThis is the email body." | ssmtp -d root ssmtp: Connection lost in middle of processing After 10 minutes I get the message: "ssmtp: Connection lost in middle of processing". I assume it is timing out. root@Tower:/boot/unmenu# which ssmtp /usr/sbin/ssmtp root@Tower:/boot/unmenu# ssmtp -? ssmtp: No recipients supplied - mail will not be sent root@Tower:/boot/unmenu# ssmtp -V sSMTP 2.64 (Not sendmail at all) root@Tower:/boot/unmenu# net lookup smtp.gmail.com 173.194.79.108 Syslog entries: Mar 7 07:55:08 Tower sSMTP[9968]: Set Root="your_id@your_email_host.com" Mar 7 07:55:08 Tower sSMTP[9968]: Set MailHub="your_host" Mar 7 07:55:08 Tower sSMTP[9968]: Set RemotePort="0" Mar 7 07:55:08 Tower sSMTP[9968]: Set HostName="localhost" Mar 7 07:55:08 Tower sSMTP[9968]: Set UseSTARTTLS="False" Mar 7 07:55:08 Tower sSMTP[9968]: Set AuthUser="your_id@your_email_host.com" Mar 7 07:55:08 Tower sSMTP[9968]: Set AuthPass="your_password" Mar 7 07:55:08 Tower sSMTP[9968]: Set FromLineOverride="True" Mar 7 07:55:08 Tower sSMTP[9968]: Unable to locate your_host Mar 7 07:55:08 Tower sSMTP[9968]: Cannot open your_host:0 Mar 7 07:55:35 Tower sSMTP[10209]: Set Root="[email protected]" Mar 7 07:55:35 Tower sSMTP[10209]: Set MailHub="smtp.gmail.com" Mar 7 07:55:35 Tower sSMTP[10209]: Set RemotePort="465" Mar 7 07:55:35 Tower sSMTP[10209]: Set HostName="[email protected]" Mar 7 07:55:35 Tower sSMTP[10209]: Set UseSTARTTLS="True" Mar 7 07:55:35 Tower sSMTP[10209]: Set AuthUser="myemail" Mar 7 07:55:35 Tower sSMTP[10209]: Set AuthPass="mypassword" Mar 7 07:55:35 Tower sSMTP[10209]: Set FromLineOverride="True" Mar 7 07:55:35 Tower sSMTP[10209]: Creating SSL connection to host Mar 7 08:05:35 Tower sSMTP[10209]: Connection lost in middle of processing My problem looks very similar to the one posted here: http://lime-technology.com/forum/index.php?topic=5568.msg61670#msg61670 Any ideas? Update 1: OK, I must be doing something stupid with my gmail account setup. I tried it with another email (not gmail) and it worked fine. Do I have something set incorrectly to use gmail? Thanks! Update 2: Yup, I was doing something stupid I guess. unraid_notify works for me with port 465, ssmtp doesn't. I had to change it to port 587 and now it works fine with gmail. Hopefully this post will help someone else that may have the same problem as me.

The back: Close up of the SATA cables routed in a half circle through those slots. Half on top, half on bottom:

I couldn't agree more on the case, I too would love to have another 4 drive slots and a 3rd removable 120mm fan. That would rock! I'll try covering the vent holes in the bottom of my case too, thanks for the tip. I can't get a picture of the other side of the case right now, but I will post later. The cable routing is not as neat on that side. The ASRock MB has the sata slots horizontal with the MB, so they point towards the front of the case. I routed my sata cables through the slots between the drive cages and the other side of the case. The slot is not much wider than the width of the sata cable. So I basically made a big U loop with each one, and then the next one is right on top of the last. This seemed like the best way to not put pressure on the MB sata connectors, since each cable is going straight into the connector with no bend or pressure from turning the cable. I know a picture is worth a thousand words but hope that helps until I can get the pic up.

I don't know if any of the listed MB will work. For me, these didn't work: ASRock Z77 Extreme 4 (Intel Z77) ASUS A8N-SLI Premium (NVIDIA nForce4) This one did: Gigabyte P55M-UD4 (Intel P55)

I needed the HX-750 power supply for another build, so I swapped it out with a TX-750 that I already had. I guess "If it ain't broke don't fix it" comes to mind. Since I swapped the PS I am getting some sata errors across multiple drives. Ugh! The TX-750 powered my older norco 4020 build for years without issue. Mar 3 00:07:56 voyager kernel: ata6: exception Emask 0x10 SAct 0x0 SErr 0x4090000 action 0xe frozen (Errors) Mar 3 00:07:56 voyager kernel: ata6: irq_stat 0x00400040, connection status changed (Drive related) Mar 3 00:07:56 voyager kernel: ata6: SError: { PHYRdyChg 10B8B DevExch } (Errors) Mar 3 00:07:56 voyager kernel: ata6: hard resetting link (Minor Issues) Mar 3 13:01:53 voyager kernel: ata8: exception Emask 0x10 SAct 0x0 SErr 0x90002 action 0xe frozen (Errors) Mar 3 13:01:53 voyager kernel: ata8: irq_stat 0x00400000, PHY RDY changed (Drive related) Mar 3 13:01:53 voyager kernel: ata8: SError: { RecovComm PHYRdyChg 10B8B } (Errors) Mar 3 13:01:53 voyager kernel: ata8: hard resetting link (Minor Issues) Mar 3 15:43:15 voyager kernel: ata8: exception Emask 0x10 SAct 0x0 SErr 0x90002 action 0xe frozen (Errors) Mar 3 15:43:15 voyager kernel: ata8: irq_stat 0x00400000, PHY RDY changed (Drive related) Mar 3 15:43:15 voyager kernel: ata8: SError: { RecovComm PHYRdyChg 10B8B } (Errors) Mar 3 15:43:15 voyager kernel: ata8: hard resetting link (Minor Issues) Mar 5 20:07:41 voyager kernel: ata8: exception Emask 0x10 SAct 0x0 SErr 0x90002 action 0xe frozen (Errors) Mar 5 20:07:41 voyager kernel: ata8: irq_stat 0x00400000, PHY RDY changed (Drive related) Mar 5 20:07:41 voyager kernel: ata8: SError: { RecovComm PHYRdyChg 10B8B } (Errors) Mar 5 20:07:41 voyager kernel: ata8: hard resetting link (Minor Issues) Mar 6 00:05:18 voyager kernel: ata3: exception Emask 0x10 SAct 0x0 SErr 0x4090000 action 0xe frozen (Errors) Mar 6 00:05:18 voyager kernel: ata3: irq_stat 0x00400040, connection status changed (Drive related) Mar 6 00:05:18 voyager kernel: ata3: SError: { PHYRdyChg 10B8B DevExch } (Errors) Mar 6 00:05:18 voyager kernel: ata3: hard resetting link (Minor Issues) Mar 6 00:05:18 voyager kernel: ata4: exception Emask 0x50 SAct 0x0 SErr 0x4090800 action 0xe frozen (Errors) Mar 6 00:05:18 voyager kernel: ata4: irq_stat 0x00400040, connection status changed (Drive related) Mar 6 00:05:18 voyager kernel: ata4: SError: { HostInt PHYRdyChg 10B8B DevExch } (Errors) Mar 6 00:05:18 voyager kernel: ata4: hard resetting link (Minor Issues) So I'll do the following to try and fix this, if I get another error, I'll proceed down the list: 1. Maybe cables aren't seated properly - Check all cables and make sure they are snug (power and sata). 2. Maybe it's the ASrock MB on-board sata controller - Add 2 Adaptec 1430sa controllers (already have on hand) instead of MB sata 3. Maybe it's the new monoprice sata cables - Replace the sata cables. 4. Maybe it's the ASrock MB - Replace with Supermicro C2SEA (already have on hand) Update #1 - Checked all the cables, nothing appeared loose. I don't think it was a cable not being seated properly, especially since I saw the error on 4 of the installed 7 drives. I did re-route some of the power connections to the fans that I had on the back side of the case, since they were close to some of the sata cables. Who knows. I also went ahead and updated the bios for good measure. One great feature of the MB is that you can go into the bios and tell it to grab the latest firmware from the internet and apply it. No searching, downloading, or copying to some portable media.... just do it. Now I wait to see if the errors come back. Update #2 - Everything looked good for 2 days, then got another SATA error. Mar 8 09:15:31 voyager kernel: ata6: exception Emask 0x10 SAct 0x0 SErr 0x4090000 action 0xe frozen Mar 8 09:15:31 voyager kernel: ata6: irq_stat 0x00400040, connection status changed Mar 8 09:15:31 voyager kernel: ata6: SError: { PHYRdyChg 10B8B DevExch } Mar 8 09:15:31 voyager kernel: ata6: hard resetting link Mar 8 09:15:36 voyager kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Mar 8 09:15:36 voyager kernel: ata6.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES) succeeded Mar 8 09:15:36 voyager kernel: ata6.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out Mar 8 09:15:36 voyager kernel: ata6.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out Mar 8 09:15:36 voyager kernel: ata6.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES) succeeded Mar 8 09:15:36 voyager kernel: ata6.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out Mar 8 09:15:36 voyager kernel: ata6.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out Mar 8 09:15:36 voyager kernel: ata6.00: configured for UDMA/133 Mar 8 09:15:36 voyager kernel: ata6: EH complete I went ahead and installed both of my 1430sa controllers and used different cables (since it was easier that re-routing my existing cables). Now I wait to see if the errors come back. Update #3 Went for 4 days with no issues. So I decided to add a couple of drives and play with the array a little. Moved around about 500GB of data with no issue. Then when I was streaming something to my XBMC I lost the stream. Checked unRAID and it was VERY unhappy. Lot's of sata errors, LOTS of read/write errors, basically hosed. This was at the same time that several of the drives spun down. So I rebooted and played with spinning drives up/down. That basically caused the array to implode with sata errors, even sometimes rebooting itself. Very Ugly. Soooo... I swapped out the ASRock MB with my Supermicro C2SEA booted up and SAME problem. Well, all that was left to swap out was the PS or drives. So I removed the TX-750 PS and installed a new HX-750. Booted up, no errors and can spin up/down drives at will. Gonna let it run for a couple days and see what happens, but pretty sure my TX-750 was the problem.

Nevermind, I think I got the backups all figured out. Got the scripts stored in a datastore and have added the job to cron. I think it will persist a reboot, we'll see. I followed instructions here: http://blog.waldrondigital.com/2012/09/14/modifying-ghettovcb-to-run-on-vmware-5-1/ UPDATE - Yay, it worked. Cron entry persisted through a reboot so my backups are now good to go!

Well, if you're not going to use IPMI, then don't pay for it. Personally, I just setup my first server this week with IPMI and I'll never go back. It's awesome. I'm running ESXi though and have unRAID running in a VM. It's very cool to run headless. I would pay $100 more just for this feature, but I am using it. Even if I was using it just for unRAID, I would still use IPMI. For example, I always preclear from the console, using different TTY windows when running more than one at a time. That works great over IPMI.

Hey Kegler - Where did you put your backup script so that it would persist? I made a directory on /tools but that didn't survive a reboot. Right now I have the backup script in one of my datastores. Also, I noticed that there is no cron configured by default, do you have your backups automatically scheduled? Thanks!

I am still on ESXi 5.0U1 and my Health Status has been working fine. This may be a 5.1 issue. A quick search turned this up. http://communities.vmware.com/message/2113736 Wow. Feeling kind of dumb now. I actually read that earlier, but only made it to the 4th post and thought that was the end of the thread. Duh!

I downloaded and started using Notepad++, it's cool, thanks for that. Backups - Played around with ghettoVCB and got some backups working to my other array via NFS. Windows VM's will backup hot, but unRAID fails after 15 min on snapshot creation, it will backup cold. I'll play with that later. It's not really a big deal. Boot - I also setup a vmdk to boot unRAID from instead of plop so that makes it boot much faster. Patching - I went ahead and applied the latest ESXi 5.1 patch too. Pretty painless. Health Status Issue - I don't see anything in the vSphere client for Health Status. I want to see CPU/MB temps but everything is listed as "? Unknown". Anyone have any ideas for this? Why doesn't anything show? Any other way to get this information? Thanks! Datastore Performance - I posted the below info in the Atlas thread, but I am posting it here too. There isn't really a good forum to post ESXi related questions in. Some things are build related, but not specific to any kind of hardware. Wasn't sure where to put it. Maybe Mods could start an ESXi sub-forum? ESXi datastore performance testing - What's a good way to test and measure performance? I currently have 2 datastores: M1015 Raid1 - 2x Seagate Barracuda LP 2 TB 5900RPM SATA 3 GB/s 32 MB Cache NFS over 1GB network to unRAID with a cache disk I ran a couple of quick tests using dd like this: dd if=/dev/zero bs=1024 count=1000000 of=<datastore>/testfile dd if=<datastore>/testfile of=/dev/null bs=1024k My quick results for a 1GB file: Datastore Write MB/s Read MB/s Local Raid1 2.9 5.2 NFS share 5.5 12.2 I want to test some other datastore configurations like a local drive connected to the MB, a SSD, and will probably setup ZFS/OI/napp-it with 4 drives. What is a good way to test datastore performance? Is dd good or is there something better? Thanks!