The Monolith - 17-drive Fractal Design XL R2 build [initial build complete]


Recommended Posts

Likes, Dislikes, Add-ons, Future plans, Cost, and Current Status in OP updated.

 

I'm currently migrating my files to the array. I ran into a pretty serious bug in Linux Mint and/or unRAID: stopping file copy mid-stream results in incomplete files. I wrote a post describing my approach to mitigate the risk of incomplete or corrupt files, along with a newbie-friendly guide for running md5deep to compare files between the source and unRAID array, which might be useful for others.

Link to comment

Nice work -- especially without a Dremel.  I've done this several times, but have always used a Dremel with the proper cutting bit.  Grinding it away worked fine for you -- and as long as you're VERY cautious not to grind any of the actual connections it clearly doesn't matter.

 

You ARE, I presume, well aware that only providing 1/4th of the bandwidth the card is designed for significantly degrades your performance -- but as I noted earlier, this isn't really an issue for "normal" array operation ... only for rebuilds, parity checks, and the initial parity sync.

 

Link to comment

You ARE, I presume, well aware that only providing 1/4th of the bandwidth the card is designed for significantly degrades your performance -- but as I noted earlier, this isn't really an issue for "normal" array operation ... only for rebuilds, parity checks, and the initial parity sync.

 

Yes, I'm aware :) That's actually another reason why I didn't want to change the cards to x1: when/if I get a new mobo with x4 slots, I can get the benefit of the higher bw right away.

 

I read some threads here, and someone was using up to six drives on a plain PCI bus IIRC. More than that and it started choking up pretty badly.

Link to comment

I've used 4-port PCI cards in systems in the past;  and they work fine as long as you're only accessing one of the attached drives at a time.    A single PCIe lane has notably better bandwidth than PCI, but will still be a bottleneck with modern 1TB/platter drives if you're accessing more than 2 at once.    But even accessing 8 at a time should WORK -- it will be be significantly throttled in the per/disk bandwidth.

 

It'll be interesting to see what your parity check times are  :)

Link to comment
  • 3 weeks later...

Well, that didn't take long: I had my first failed drive. It was an ages old 1TB drive with a cool 100 reallocated sectors, so it was a matter of sooner rather than later.

 

Unfortunately it failed - or I noticed it - just a few hours after I had added two 2TB drives to the array. I was hoping that I could just use one of those drives, but since they were already in the array that wasn't an option. After consulting the forums, the safest, quickest and easiest approach was to buy a new one.

 

I ran two pre-clear cycles on the new drive at around 40 hours total, and rebuild lasted around 18 hours, and everything is working again.

 

Due to the rather long time my array was unprotected, I've re-considered having a cache disk. I'm adding one to work mainly as a warm spare, which will hasten recovery dramatically. As I'm not even near the max capacity of the array data-wise or physically, I'm not making any compromise on size, either.

 

I'm also building a fan bracket to the drive cage, as the two drives in it are running at just above 40 degrees. I knew the box would need the bracket, and as I'm closing the end of the initial build now is a good time to do it.

 

Finally I have to investigate what's going on with my speeds. I never really checked what transfer speeds to expect, and I've been transferring data to the array at around 9.5MB/sec. This is apparently half the minimum expected speeds. The Monolith, my desktop and router have gigabit ethernet so that shouldn't be the bottleneck. I suspect/hope that it's my really long network cable which has seen better days.

Link to comment

Your 9.5MB/s data rate is suspiciously close to the max rate for a 100Mb network connection.  If your Ethernet cable has any incontinuity, or if one of the connectors has a poor connection on one of the wires, that's exactly what I'd expect to see.

 

I doubt the length of your cable is an issue;  but it certainly may be a connector or a break in one of the wires.  I'd definitely try new cables.

 

Link to comment

Your 9.5MB/s data rate is suspiciously close to the max rate for a 100Mb network connection.  If your Ethernet cable has any incontinuity, or if one of the connectors has a poor connection on one of the wires, that's exactly what I'd expect to see.

 

I doubt the length of your cable is an issue;  but it certainly may be a connector or a break in one of the wires.  I'd definitely try new cables.

 

Yep, as I suspected, it was the cable. But it wasn't the one I was suspicious of, the one leading from my router to the array, but the cable between the router and my desktop.

 

Changed the cable, which tripled (!) my transfer speed, which now top at around 30MB/s to the array. Much better! Too bad I'm almost done with seeding the box :P

 

I hadn't noticed anything strange before, since my internet doesn't reach 30MB or even 10MB per sec. Never thought a basic cable would make such a difference.

Link to comment

I'm not the least bit surprised -- as I noted, the symptoms were exactly what I'd suspect with a discontinuity in the cabling.    You always need to check EVERY cable involved.    I found this issue once on a friend's system in the 18" cable he had between his UPS and the server [you DO have a UPS -- right?  ... and it DOES protect your network cable, right?  :) ]    He had shut down the server for an annual "blow out the dust" cleaning ... and apparently the movement had stressed the cable where it connected to the system.

 

Link to comment

I'm not the least bit surprised -- as I noted, the symptoms were exactly what I'd suspect with a discontinuity in the cabling.    You always need to check EVERY cable involved.    I found this issue once on a friend's system in the 18" cable he had between his UPS and the server [you DO have a UPS -- right?  ... and it DOES protect your network cable, right?  :) ]    He had shut down the server for an annual "blow out the dust" cleaning ... and apparently the movement had stressed the cable where it connected to the system.

 

Ooh, good point about the UPS network cable, that's another potential weak link. I'm buying a new set of cat6 cables today, so thanks for the reminder!

Link to comment

Glad you've got your speeds up-to-par now  :)

 

Anytime someone complains about speeds only hitting 9-10Mb even with Gb NICs, routers, and switches I immediately think about a discontinuity in the cables.    That will cause the network to slow down to 100Mb speed -- exactly what you were seeing.

 

After yesterday it got... interesting. I bought a whole bunch of cat5 and cat6 cables. Moved my box to its final place, and noticed speeds dropped again.

 

After a lot of testing and headscratching and tire kicking, I finally tracked the culprit. The cables were ok. It is my UPSs which are to blame! Even if I use only one UPS (yes, I have two), speeds drop to 10MB/sec. Same cabling, speeds go up to 40+MB/sec.

 

The UPS manual clearly states 10/100Base-T, no mention of gigabit ethernet, so they are in spec. Therefore I'm SOL and stuck with them. I've had one for years, and my internet speeds aren't fast enough that the the UPS was a bottleneck. Now they are. Argh.

 

The electricity where I live is shit, I've lost a mobo, a GPU and several (!) routers until I bought the UPS. Not sure how unreliable internet cabling is, though.

 

Another regrouping to figure out what to do next  :o

Link to comment

Interesting.    A UPS is generally just a pass-through with surge protection ... not an active circuit.  So the speed shouldn't matter.    None of mine have any issue with Gb speeds ... a Belkin, a couple of CyberPower units, and 3 APC units.

 

I suspect the issue is your UPS connectors are defective, and aren't passing all 8 lines through.

 

Unfortunately, that's not something you can repair.    You could either skip the network surge protection (not recommended) or buy a separate surge protection unit for the network cabling and not use the UPS ports for that.    This would work well:  http://www.newegg.com/Product/Product.aspx?Item=9SIA1N80N45133

 

 

Link to comment

Interesting.    A UPS is generally just a pass-through with surge protection ... not an active circuit.  So the speed shouldn't matter.    None of mine have any issue with Gb speeds ... a Belkin, a couple of CyberPower units, and 3 APC units.

 

I suspect the issue is your UPS connectors are defective, and aren't passing all 8 lines through.

 

That could be it. I'll check with APC whether they should work with gigabit or not.

 

Unfortunately, that's not something you can repair.    You could either skip the network surge protection (not recommended) or buy a separate surge protection unit for the network cabling and not use the UPS ports for that.    This would work well:  http://www.newegg.com/Product/Product.aspx?Item=9SIA1N80N45133

 

Thanks for the tip! Yes, that would be much cheaper than replacing both UPSs with higher-tier models.

 

edit: ended up getting APC ProtectNet.

Link to comment
  • 3 weeks later...

Initial build complete. OP fully updated, partly re-written. Added sections on a DIY fan bracket, cache drive, and ethernet speed woes which was covered in previous posts in the thread. Images were gone, changed the free hosting provider, added some new ones.

 

Running into the 20k character space limit in the OP :o

Link to comment

The re-org of the first post looks good.    Nice to have a bit of discussion r.e. the problems & pitfalls you can encounter ... may very well keep someone else from fighting those same issues.

 

That's one of the main reasons why I started this thread. Building an unRAID box yourself can be pretty intimidating, so hopefully this will take away some of the fear, and my bumbling will re-assure that problems can be overcome :)

 

For your SAS card, email Supermicro.

 

They sent me out a new full height bracket for free! YMMV.  ;D

 

Thanks for the tip!

Link to comment
  • 2 weeks later...

Ran into the 2000 character limit in OP, will be updating in replies to the post.

 

UPDATE 2014-03-16 Installed additional exhaust fan, HDD cage, and fan controller

 

NEW EXHAUST FAN

 

The top HDD cage was running quite hot even with the intake fan pushing air through it, described in an earlier update. Temps of disk 5 which was the hottest climbed to 45 in normal usage, even higher in parity checks.

 

I installed an exhaust fan to the top of the cage, which pulls air from right behind the cage. I did several readings of before and after, running the HDDs at idle overnight. Installiing the exhaust fan had a huge impact on top cage HDD temps: Disk 5 which was running at 23C above ambient went to 15C. Even the average of all HDD temps also went from 14C to 11-12.

 

ADDITIONAL HDD CAGE

 

After confirming with Fractal Design tech support that an additional HDD cage fits in the case, I ordered one from their spare parts center.

 

Disappointingly and misleadingly, the cage didn't come with HDD trays although the picture has trays in it. I have contacted Fractal Design to clarify this. Fortunately I had extra trays from my desktop, and had ordered two extra ones just in case. The cage also doesn't come with mounting screws. You can use the screws that come with the case which are used to secure the trays to the cage, though.

 

edit: Fractal Design sent me the trays for free after I contacted them and pointed to the picture which had trays on it. No mounting screws, so order them separately if you're running out as they are not standard screws. Subsequently they took down the picture, and added a note saying the trays are not included. This is for the EU site, apparently the cage comes with trays on the US site. Further detail in following posts. /edit

 

Installation is easy: you mount the cage to the bottom of the case with four screws. This is in the same position as the bottom fan. Unfortunately installing HDDs into this cage is not so easy. There is no opening at the back of the case so you can't reach the drives from there, and although you can mount the cage sideways, there is no position where you could access the power and SATA ports and replace the HDDs. Therefore you have to remove the entire cage whenever you need to replace a drive or change cabling. Cabling in general gets quite messy, as there is not much space behind the cage.

 

Nevertheless, it is a workable solution, and this brings the case to its max capacity of 17 HDD slots. I am currently at 13. Attached picture shows the cage.

 

I'll have to wait and see how hot this cage runs. There is a space for a side exhaust or intake fan, which is close to the cage, so that could be an option if the HDDs run too hot.

 

h3168227-144358.jpg

 

FAN CONTROLLER

 

I felt a bit limited with the three-position fan controller on the case. I bought NZXT LXE fan controller. It is an external controller, so it doesn't take a precious 5.25" slot. It does use a PCI slot, but it does not attach to the actual slot - it just occupies the back panel space. You control the fans with an external console, which is a very nice-looking piece of geeky equipment.

 

The manual is rudimentary, and I'm being generous: it merely goes through the simple installation in a few terse but clear steps. I attached the temp probes to HDDs and fan cables, and taped the probe and fan assignments on a post-it on the back of the external console for reference (pic below). I didn't bother to attach anything to the CPU as I don't do transcoding; it runs around 40C at all times. Power is via molex cable.

 

One of my Silent Wings fans was the 4-pin PWM version, and the fan controller cables are only 3-pin. You could cut 3-pin cable to accept the 4-pin cable, but fortunately I had ordered a bunch of Sharkoon fan extension and y-cables, and the extension cable doubled as a 4-to-3 pin adapter.

 

The controller has two modes, automatic and manual. You can set temp warnings (audio and visual), which I set to 40C. Currently running in automatic mode, which spins the five attached fans at 600-800 rpm. Temps using the external probes are surprisingly close to the ones reported by the drive, within 1-2 degrees.

 

Picture attached above shows the mess of cabling. Thankfully both the sensor and fan cables are clearly numbered.

 

h3168229-144359.jpg

Link to comment

The extra cages is slick -- it fits very nicely in the case.  I'll have to remember that, as I have several friends who have used this cage (I built the systems for them) ... and if they want additional drives that's a good approach.    Thanks for the heads-up that trays don't come with the cage -- I'm surprised at that; but at least I'd now know to order the trays as well  :)

Link to comment

I've just taken delivery of a Define XL R2, and I have the extra lower drive cage ordered as well. Disappointed to hear it doesn't come with trays for the drives - from the pic on the website I assumed it did.

 

Looking again at the spare parts site the trays are pretty cheap, but the delivery to the UK is quite high. I've emailed to ask if my order can be amended to include trays, but in case it can't - are there any other trays that would be compatible? Or is there another (UK based) supplier?

Link to comment

The extra cages is slick -- it fits very nicely in the case.  I'll have to remember that, as I have several friends who have used this cage (I built the systems for them) ... and if they want additional drives that's a good approach.    Thanks for the heads-up that trays don't come with the cage -- I'm surprised at that; but at least I'd now know to order the trays as well  :)

 

Indeed it fits well, and has very solid construction, as everything else with the Fractal Design. I like how they offer the flexibility of putting a fan in that space, moving another HDD cage to that slot to fit a radiator in the front (I believe that's the use for it), or an additional cage like I did. It's clearly not optimal, keeping in mind the caveats I mentioned earlier, but is a perfectly workable solution for drives you don't swap out all the time - which I would imagine is the case for most unRAID boxes.

 

I'll update on temp levels when I have more HDDs in the cage, but that will take months. Initial results look ok: the single drive in the cage runs at 37C while doing pre-clearing, all fans at 500rpm (lowest setting available with the NZXT fan controller), which is similar to what I got with other drives which were in other cages.

 

BTW, make sure you order the right cage. I linked to the bottom cage in my previous post, which looks very similar to the upper cage in the pictures, but the SKU is different so it probably doesn't fit.

 

Also, it's again in short supply. It was in same status in early Jan and went out of stock shortly thereafter, and only became available 2-3 weeks ago.

Link to comment

I've just taken delivery of a Define XL R2, and I have the extra lower drive cage ordered as well. Disappointed to hear it doesn't come with trays for the drives - from the pic on the website I assumed it did.

 

Looking again at the spare parts site the trays are pretty cheap, but the delivery to the UK is quite high. I've emailed to ask if my order can be amended to include trays, but in case it can't - are there any other trays that would be compatible? Or is there another (UK based) supplier?

 

I was rather annoyed that the cage didn't come with the trays as well. They are pictured on the website, so it looks like false advertising to me. I've gently requested them to send me the trays separately, let's see if they do.

 

I pulled two trays from my Define R4 desktop as I don't use them there. It  uses the same trays. I also had ordered two "extras" just in case.

 

The spare parts center ships by courier which explains the high cost - maybe you can request them to send by post via email for savings? I'm not familiar with UK retailers, based in NL here. Given most retailers and online shops are box movers these days, your best bet is the spare parts center - but it doesn't hurt asking around from more customer-service oriented shops if they could order them for you.

Link to comment

Out of interest which trays did you order from the spare parts shop?

 

A search for "tray" brings back several, many of which look the same but are different prices.

 

Also, which contact email are you using to communicate with them? I just replied to the order confirmation email that I had when ordering the lower cage.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.