mjorud Posted August 4, 2012 Share Posted August 4, 2012 Hi all, I'm going to move from a big house to a smaller apartment and to make thing more practical I want to consolidate my servers to one server. Today unRAID is running on Intel DG43NB, Intel E1400 and Sil3124 in a Norco 4020. I was running WHS v1 until 4 months ago when I converted to unRAID. I also have a PowerEdge 2850 running ESXi v4.1 with a few VMs. Both servers are rather noisy (understatement) so they are located in the garage. The new server will run unRAID, pfSense, Ubuntu Server (Plex Media Server, SABnzbd, CrashPlan, rsync, MySQL, Transmission/uTorrent, AUPCUPSD, ) and WinXP/Win7 (BlueIris). Motherboard: 1x Tyan S5512GM4NR ATX Server Motherboard (http://www.newegg.com/Product/Product.aspx?Item=N82E16813151246) Case: 1x Norco 4020 (I have this) CPU: 1x Intel Xeon E3-1240 (http://www.newegg.com/Product/Product.aspx?Item=N82E16819115082) RAM: 2x Kingston 8GB DDR3 SDRAM DDR3 1333 ECC (http://www.newegg.com/Product/Product.aspx?Item=N82E16820239135) PSU: 1x Corsair HX Series HX750 (http://www.newegg.com/Product/Product.aspx?Item=N82E16817139010) HBA/RAID 1x IBM ServeRAID M1015 + RES2SV240 (http://www.newegg.com/Product/Product.aspx?Item=N82E16816117207) Cables: 1x SFF-8087 to SFF-8087 : 5x SFF-8087 to SATA (fan out) I have a few questions though. My Norco 4020 only have SATA connectors on the backplanes. If I have not misunderstood completely I can still use M1015/RES2SV240 combo using one SFF-8087 to SFF-8087 between M1015 and RES2SV240, one SFF-8087 to SATA 4 (http://www.newegg.com/Product/Product.aspx?Item=N82E16816115064) from M1015 to one of the backplanes (parity disk, cache disk) and four of the same cable to the rest of the backplanes, right? The Samsung Spinpoint F1 HD322HJ 320GB 7200 RPM might be not be sufficient for a datastore drive but I since I already have it I might aswell use it. If I find that the datastore drive will be a bottleneck I might opt for two 2.5" spinners in RAID. 2.5" because the Norco 4020 has two internal hdd bays. Data being produced on the different VMs will be stored on unRAID (SABnzbd and Transmission on a non-parity drive. CrashPlan and BlueIris on the array). Tyan S5512GM4NR has four NIC. At least two are used by pfSense (WAN and LAN). I want one of the VMs to be on a different subnet than the rest and because of that I need to use the third NIC for OP1/DMZ (in pfSense). The last NIC would be used as vSwitch for VM management and the rest of the VMs. Is it recommended to have unRAID on it's own NIC? What do you guys think? Thanks in advance. Quote Link to comment
BetaQuasi Posted August 5, 2012 Share Posted August 5, 2012 No need for unRaid on its own NIC. In my build, unRaid and all the other VM's share the LAN vSwitch that pfSense is connected to (which is actually 4 ports on an Intel NIC.. overkill yes but geek factor is good ) That Tyan motherboard may require an SSI-compliant power supply, so you may want to double check that the HX series have that feature. I'm pretty sure the AX's do.. or just get a Seasonic X-series instead to be sure. If it requires SSI, it won't even power up without the right PSU. Re: the cables - yes that should be correct, though Newegg use the term 'fanout' for some reason. Typically, you have 'forward breakout' and 'reverse breakout' versions of these cables. You need to make sure you get 'forward breakout' in your case. If you wanted to be 100% sure, either find someone that's used those cables from Newegg, or just get these and save a few $$ in the process! 0.5m: http://www.monoprice.com/products/product.asp?c_id=102&cp_id=10254&cs_id=1025406&p_id=8186&seq=1&format=2 0.75m: http://www.monoprice.com/products/product.asp?c_id=102&cp_id=10254&cs_id=1025406&p_id=8187&seq=1&format=2 Quote Link to comment
marcusone Posted August 5, 2012 Share Posted August 5, 2012 Sounds like a great rig. Post pics when done. Sent from my SGH-I727R using Tapatalk 2 Quote Link to comment
Johnm Posted August 5, 2012 Share Posted August 5, 2012 The build itself looks good. PSU... I am pretty sure the HX is just a TX with modular cables. it "should" be fine. but i can not 100% guarantee it (the OEM on that should be seasonic) it is the 750M's that has issues for sure. as far as the NICS... PFsence likes 3 nics for my setup. but you might be able to use one virtual.. as far as unRAID haveing a dedicated nic. I did that at first and decided It was better for my need to use a virtual 10GB nic due to the fact that most of my data transfers came from inside the ESXi box and that the single shared nic was faster then what unRAID could use. basicly, i didnt have a bottleneck by not giving it a dedicated nic. instead I increased performance in most of my scenarios of data use. The cables you need are standard or "forward breakout cables". you will need 5 of them. one for each backplane. a 6th might get used if you put an sdd or 2.5" cache drive in the 2.5" slot for unraid. If you go the expander route, you will need a single 8088to 8088 cable (comes with expander unless your get the NC version [no cables]) as far as the expander. it is not really needed. you could go with 3x m1015's for less $$ is my guess. But you will use 3 PCIe slots if you do go with 3 HBA's The F1 will most likely work fine for a 2-3 low drive IO VM's (pfsence, AD controller, script/monitoring type VM's for example) for something like a torrent downloader or a workstation, you will most likely need more speed. burn that bridge when you get there... Quote Link to comment
mjorud Posted August 10, 2012 Author Share Posted August 10, 2012 Hi all, Thank you for all your feedback. I have decided to go for 2x M1015 (will add one more when needed) because of performance and price... although they are not as cheap as they used to be. On eBay they go for ~130 USD each. I live in Norway so with shipping it gets a bit expensive. Tyan (or Supermicro for that matter) is almost impossible to get hold of in Norway. I found one dealer in Norway that sells Tyan S5512GM4NR... for hefty 540 USD! Quote Link to comment
mjorud Posted September 4, 2012 Author Share Posted September 4, 2012 Parts are finally starting to arrive. I tested the M1015's in a Dell OptiPlex and the controllers BIOS tells me that the adapter name is SAS92010-8i with FW revision 14.00.00.00-IT. I guess the gentleman from eBay did flash them for me. I opted for the Tyan MB with only 2 NICs because of availability and price... and for some reason the Norwegian Customs like it very much because they have been playing with it for over two weeks now. My plan is to rebuild the server completely. New harddrives are preclearing for the third time as we speak and I want to use the new harddrives (the old smaller drives are going in a more portable unRAID/XBMC box) to build a new array from scratch. Would there be a problem to mount the drives that is now currently in the array outside the array later to copy files to the new array? Quote Link to comment
mjorud Posted September 10, 2012 Author Share Posted September 10, 2012 The ESXI server is now up and running. The hardware installation was pretty straight forward... yank out the old parts and put in the new ones. I had three Intel Pro/1000 GT in the old PE2850. Those cards worked just fine with ESXI v4 so I was a bit disappointed when I found out that ESXI v5.0 did not support them. My Tyan motherboard only have two NICs so I'm using one NIC for pfSense's WAN and the second one for all VMs and Management Network. Using the same NIC on VMs and Management Network is not advisable (not quite sure why but that's what I read). I'm going to get an Intel Pro/1000 PT Dual NIC. The installation of ESXI v5.0 Update went without any problems and so did the various VMs. Getting unRAID up and running was no problem thanks to the Atlas thread. Both M1015s are configured as DirectPath, and the USB-stick as passthrough. I never got Plop to work so I downloaded the .vmdk, updated it from Windows XP and it's running flawlessly. I'm now transferring data from the old unRAID to the new system by manually mounting drives from the old array then using rsync (--stats --progress -av . mnt/...). I have seen a great improvement in parity check speeds. The old system averaged at 20 MB/sec but my new baby averages at 110 MB/sec. I have not yet run Memtest but will do that this weekend when I'm away. I have read about another utility to test stability but I don't remember the name. Anyone (edit: was it Prime95?) Quote Link to comment
mjorud Posted September 15, 2012 Author Share Posted September 15, 2012 Hi all, I'm currently rsync'ing files from an external harddrive to the array (without cache drive) and preclearing a disk. All of a sudden I noticed that the transfer rate of the rsync went gron 30 MB/s to 200 kb/s. Preclearing is OK. I took a look at the syslog and here is an excerpt. Sep 15 09:22:30 unraid kernel: smbd: page allocation failure: order:2, mode:0x4020 Sep 15 09:22:30 unraid kernel: Pid: 10007, comm: smbd Not tainted 3.4.4-unRAID #2 Sep 15 09:22:30 unraid kernel: Call Trace: Sep 15 09:22:30 unraid kernel: [<c1062e3a>] warn_alloc_failed+0xbd/0xcf Sep 15 09:22:30 unraid kernel: [<c10636b2>] __alloc_pages_nodemask+0x47c/0x4a5 Sep 15 09:22:30 unraid kernel: [<c1063731>] __get_free_pages+0xf/0x21 Sep 15 09:22:30 unraid kernel: [<c10820a8>] __kmalloc+0x2c/0xf0 Sep 15 09:22:30 unraid kernel: [<c129f625>] pskb_expand_head+0xc1/0x224 Sep 15 09:22:30 unraid kernel: [<c103e45c>] ? try_to_wake_up+0x1a6/0x1b0 Sep 15 09:22:30 unraid kernel: [<c129fafe>] __pskb_pull_tail+0x41/0x21e Sep 15 09:22:30 unraid kernel: [<c12a39d5>] ? netif_skb_features+0x84/0x8e Sep 15 09:22:30 unraid kernel: [<c12a6e7e>] dev_hard_start_xmit+0x24b/0x353 Sep 15 09:22:30 unraid kernel: [<c12b56f0>] sch_direct_xmit+0x54/0x13f Sep 15 09:22:30 unraid kernel: [<c12a7084>] dev_queue_xmit+0xfe/0x286 Sep 15 09:22:30 unraid kernel: [<c12c29d1>] ip_finish_output+0x220/0x258 Sep 15 09:22:30 unraid kernel: [<c12c2a9c>] ip_output+0x93/0x9a Sep 15 09:22:30 unraid kernel: [<c12c1f34>] ip_local_out+0x1b/0x1e Sep 15 09:22:30 unraid kernel: [<c12c2418>] ip_queue_xmit+0x2ad/0x2ee Sep 15 09:22:30 unraid kernel: [<c129dacd>] ? __skb_clone+0x22/0xbf Sep 15 09:22:30 unraid kernel: [<c12d1c86>] tcp_transmit_skb+0x4d3/0x508 Sep 15 09:22:30 unraid kernel: [<c12d3e63>] tcp_write_xmit+0x2f2/0x3ec Sep 15 09:22:30 unraid kernel: [<c12d3f85>] tcp_push_one+0x28/0x2c Sep 15 09:22:30 unraid kernel: [<c12c95e8>] do_tcp_sendpages+0x34f/0x490 Sep 15 09:22:30 unraid kernel: [<c12c977a>] tcp_sendpage+0x51/0x66 Sep 15 09:22:30 unraid kernel: [<c12c9729>] ? do_tcp_sendpages+0x490/0x490 Sep 15 09:22:30 unraid kernel: [<c12e15bb>] inet_sendpage+0x82/0x9c Sep 15 09:22:30 unraid kernel: [<c12e1539>] ? inet_dgram_connect+0x5e/0x5e Sep 15 09:22:30 unraid kernel: [<c1298b4e>] kernel_sendpage+0x1a/0x2d Sep 15 09:22:30 unraid kernel: [<c1298b88>] sock_sendpage+0x27/0x2c Sep 15 09:22:30 unraid kernel: [<c109e672>] pipe_to_sendpage+0x5a/0x6c Sep 15 09:22:30 unraid kernel: [<c1298b61>] ? kernel_sendpage+0x2d/0x2d Sep 15 09:22:30 unraid kernel: [<c109e6d8>] splice_from_pipe_feed+0x54/0xc4 Sep 15 09:22:30 unraid kernel: [<c109e618>] ? splice_from_pipe_begin+0x10/0x10 Sep 15 09:22:30 unraid kernel: [<c109ed45>] __splice_from_pipe+0x36/0x55 Sep 15 09:22:30 unraid kernel: [<c109e618>] ? splice_from_pipe_begin+0x10/0x10 Sep 15 09:22:30 unraid kernel: [<c109eeac>] splice_from_pipe+0x51/0x64 Sep 15 09:22:30 unraid kernel: [<c109eeeb>] ? default_file_splice_write+0x2c/0x2c Sep 15 09:22:30 unraid kernel: [<c109eefe>] generic_splice_sendpage+0x13/0x15 Sep 15 09:22:30 unraid kernel: [<c109e618>] ? splice_from_pipe_begin+0x10/0x10 Sep 15 09:22:30 unraid kernel: [<c109f040>] do_splice_from+0x57/0x61 Sep 15 09:22:30 unraid kernel: [<c109f061>] direct_splice_actor+0x17/0x1c Sep 15 09:22:30 unraid kernel: [<c109f609>] splice_direct_to_actor+0xbe/0x16a Sep 15 09:22:30 unraid kernel: [<c109f04a>] ? do_splice_from+0x61/0x61 Sep 15 09:22:30 unraid kernel: [<c109f700>] do_splice_direct+0x4b/0x62 Sep 15 09:22:30 unraid kernel: [<c1084b62>] do_sendfile+0x157/0x19f Sep 15 09:22:30 unraid kernel: [<c1084be6>] sys_sendfile64+0x3c/0x7c Sep 15 09:22:30 unraid kernel: [<c131f01d>] syscall_call+0x7/0xb Sep 15 09:22:30 unraid kernel: Mem-Info: Sep 15 09:22:30 unraid kernel: DMA per-cpu: Sep 15 09:22:30 unraid kernel: CPU 0: hi: 0, btch: 1 usd: 0 Sep 15 09:22:30 unraid kernel: Normal per-cpu: Sep 15 09:22:30 unraid kernel: CPU 0: hi: 186, btch: 31 usd: 214 Sep 15 09:22:30 unraid kernel: HighMem per-cpu: Sep 15 09:22:30 unraid kernel: CPU 0: hi: 186, btch: 31 usd: 14 Sep 15 09:22:30 unraid kernel: active_anon:5822 inactive_anon:32 isolated_anon:0 Sep 15 09:22:30 unraid kernel: active_file:91894 inactive_file:212444 isolated_file:0 Sep 15 09:22:30 unraid kernel: unevictable:43853 dirty:1021 writeback:0 unstable:0 Sep 15 09:22:30 unraid kernel: free:134025 slab_reclaimable:13737 slab_unreclaimable:2735 Sep 15 09:22:30 unraid kernel: mapped:2458 shmem:46 pagetables:238 bounce:0 Sep 15 09:22:30 unraid kernel: DMA free:3548kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:7960kB inactive_file:1096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15804kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3152kB slab_unreclaimable:96kB kernel_stack:64kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Sep 15 09:22:30 unraid kernel: lowmem_reserve[]: 0 869 2016 2016 Sep 15 09:22:30 unraid kernel: Normal free:5468kB min:3736kB low:4668kB high:5604kB active_anon:1128kB inactive_anon:0kB active_file:238692kB inactive_file:518252kB unevictable:328kB isolated(anon):0kB isolated(file):0kB present:890008kB mlocked:0kB dirty:52kB writeback:0kB mapped:184kB shmem:0kB slab_reclaimable:51796kB slab_unreclaimable:10844kB kernel_stack:680kB pagetables:112kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Sep 15 09:22:30 unraid kernel: lowmem_reserve[]: 0 0 9175 9175 Sep 15 09:22:30 unraid kernel: HighMem free:527084kB min:512kB low:1744kB high:2976kB active_anon:22160kB inactive_anon:128kB active_file:120924kB inactive_file:330428kB unevictable:175084kB isolated(anon):0kB isolated(file):0kB present:1174436kB mlocked:0kB dirty:4032kB writeback:0kB mapped:9648kB shmem:184kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:840kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Sep 15 09:22:30 unraid kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 09:22:30 unraid kernel: DMA: 543*4kB 170*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3548kB Sep 15 09:22:30 unraid kernel: Normal: 1367*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5468kB Sep 15 09:22:30 unraid kernel: HighMem: 27*4kB 4458*8kB 3881*16kB 6517*32kB 2650*64kB 387*128kB 6*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 527084kB Sep 15 09:22:30 unraid kernel: 348259 total pagecache pages Sep 15 09:22:30 unraid kernel: 0 pages in swap cache Sep 15 09:22:30 unraid kernel: Swap cache stats: add 0, delete 0, find 0/0 Sep 15 09:22:30 unraid kernel: Free swap = 0kB Sep 15 09:22:30 unraid kernel: Total swap = 0kB Sep 15 09:22:30 unraid kernel: 524272 pages RAM Sep 15 09:22:30 unraid kernel: 295938 pages HighMem Sep 15 09:22:30 unraid kernel: 5604 pages reserved Sep 15 09:22:30 unraid kernel: 243990 pages shared Sep 15 09:22:30 unraid kernel: 147713 pages non-shared Sep 15 09:22:30 unraid kernel: smbd: page allocation failure: order:2, mode:0x4020 Sep 15 09:22:30 unraid kernel: Pid: 10007, comm: smbd Not tainted 3.4.4-unRAID #2 Sep 15 09:22:30 unraid kernel: Call Trace: Sep 15 09:22:30 unraid kernel: [<c1062e3a>] warn_alloc_failed+0xbd/0xcf Sep 15 09:22:30 unraid kernel: [<c10636b2>] __alloc_pages_nodemask+0x47c/0x4a5 I'm running the 5.0-rc6-r8168-test2 without any addons (not even VMTools). Any ideas what this is? Thanks in advance. syslog.zip Quote Link to comment
mjorud Posted March 30, 2013 Author Share Posted March 30, 2013 I have been running my ESXi server (I named him Nansen) for almost six months and it has been running flawlessly. I also built an off-site baremetal unRAID server (which I named Amundsen ) for backup. VPN tunnel between them and using rsync for transfer. I have replaced the stock fans of my Norco with Noctua fans (a bit more pleasant) and also upped the RAM from 16 MB to 32 MB. I'm running unRAID with only VMTools and Powerdown script. Sabnzdb, Deluge, APCUPSd and PMS is running on a Ubuntu Server guest. pfSense guest is acting as my router. MySQL for XBMC is running on the second Ubuntu Server. Tvheadend is running on the third Ubuntu Server. As for now this is connected to a DM and XBMC, but I'm looking into getting a cable DVB-tuner (USB) but not sure which. I also have a WinXP guest for file management. Recently I have been playing with a SBS 2011 guest trying to set up Exchange but I don't understand a thing of that. And I thought SBS was intuitive and easy to set up. I was wrong. Now my biggest problem is that I really don't have anything to play around with anymore because everything just works. Anyone have an idea for what to do next? Quote Link to comment
Rick Sanchez Posted March 31, 2013 Share Posted March 31, 2013 I'm currently rsync'ing files from an external harddrive to the array. How? Quote Link to comment
mjorud Posted April 1, 2013 Author Share Posted April 1, 2013 I'm currently rsync'ing files from an external harddrive to the array. How? I mount the external drive (or non-array SATA drive in my case). mount -t reiserfs /dev/disk/by-id/scsi-SATA_SAMSUNG_HD103UJS13PJDWS152308-part1 /mnt/temp I have this in my go file as I use this hard drive as my temporary download disk outside the array (don't use cache drive). How I figured out the disk/by-id I can't remember but it's here on the forum somwhere. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.