bobkart

I measured the depth of the case and it's more like 15.5" as opposed to the 17" in the specs (corrected in initial post). Maybe they measured the handles too? I just wrote a 39GiB file to the server and captured the progress. There is a fair bit of "wavering" starting at 10% into the copy, but the speeds still stay well above disk-write speeds (including parity overhead) until 90% into the copy, where it finally drops to the mid-40MB/s range.

Well, it's up to you of course, but with 64GB of memory, I'm pretty sure if you increase your dirty_ratio from 20 to even just 50, you'll see a dramatic increase in how much of that 31GB files gets copied before the speed drops to drive speeds (including parity overhead). I.e 20% of 64GB is 12.8GB, whereas 50% of 64GB is 32GB. So the operating system will allow up to 32GB of memory to be filled with "yet to be written to disk" data before it slows down the incoming data, versus just 12.8GB. Granted if you have other stuff using memory, there might not be that much available to use as cache.

This curve wasn't nearly as flat as the one I got for the 18GiB write, but it should serve the desired purpose. That's a 24GiB write, getting 60% through at pretty close to maximum GbE speed, then it starts to waver, not really sure what that's about, but it's definitely not due to running out of cache before the end of the write. Here's something of a writeup on the Linux VM caching parameters: https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/ And now that I look more closely at that information, the wavering I sometimes see could be due to a lower-than-desirable vm.dirty_expire_centisecs setting . . . stock is 30 seconds, whereas I could (possibly) benefit from something more like 300 seconds (while acknowledging the increased risk of data loss such a high setting creates). The server *is* powered through a decent UPS so that's one big source of concern pretty well eliminated. Short of something on the mainboard going south, power supply failure would seem to be the next obvious potential problem point (and where redundant power supplies come in handy!). Will be experimenting with that other VM setting next, although the current performance is quite acceptable.

My write speeds started and stopped right at full GbE speeds (~112 Mb/s), on the 18GiB file I used to confirm that my "RAM cache" idea was working (previously it had dropped to ~60 Mb/s early into the copy, as you report). Haven't heard of speedtest.sh, and a search didn't reveal anything. I'll get you a screenshot of a 20GiB file write later tonight. Probably 30Gib would just push past my dirty_ratio. I suspect you haven't increased your dirty_ratio past the 20% that it comes at stock. I'm using 75%.

Looks like 32GB ECC memory plus a dirty_ratio of 75 will work fine (for me anyway) as a substitute for a cache drive. I can write 20GB files at full GbE speeds. I just switched to using this server as primary and my previous primary as "primary backup". Rsyncing more frequently of course until I'm satisfied with the reliability of this one.

Good to know, thanks. I wish they made a 200-watt version of the power supply I used for this build. I just added a second 16GB ECC UDIMM, and idle power consumption only bumped up by 2-3 tenths of a watt. Next I'll turn on cache_dirs and see how much of the 32GB that uses (under 10% without it).

I typically see drive temperatures in the low-to-mid-30Cs during a Parity check. An earlier version of the build used quieter, lower-powered Fractal Design fans, but drive temperatures started to approach 40C during a Parity Check, so I took a step up on the fans. I actually had the eight-core version of this board in the build for a while, having received it by accident. Unfortunately I had to return it due to random crashes/reboots. And you're right, the difference was only 2-3 watts at idle (and about $100). It's the most efficient "traditional" PSU at those power levels that I've found yet . . . not that I've tried very many; most of my other builds use a picoPSU (the 80-watt version typically), which are very efficient at low power levels. I tried an M4-ATX (250 watts) with this build and it didn't quite beat the Seasonic (about 2 watts difference). Which reminds me of another data point asked for in the UCD Guidelines: peak power consumption at boot. Hard to know for sure what this is, due to the numbers flying by so quickly, but I've watched the meter many times and have yet to see it hit 140 watts (edited into initial post). With 2-3 more drives being added at some point, that ought to climb by 40-60 watts, so I expect it would still be under 200 watts peak.

Feel free to post your power consumption numbers here (along with protected capacities). I'm especially interested to see who has gotten under one watt idle per protected terabyte, and what they did to achieve it. And thanks in advance to any who might offer feedback/criticism of my build. In that regard, right off I'll say that I understand the additional risk that using such large drives incurs (with over 15 hours for a data or parity rebuild). This is where a dual-parity solution would shine. I do have multiple backup servers, and will typically rsync to them weekly. And if/when a rebuild is needed, I'd likely rsync again before firing it off. EDIT: the wattmeter is now showing 29.0 watts, with the occasional dip to 28.9 watts.

Inside of server.

Rear view of server.

Top-front picture of server (and wattmeter).

Wattmeter showing idle power consumption.

(Title has been updated from 40TB under 30 watts to 48TB under 35 watts.) (Update: motherboard replacement saved over 10 watts at idle, to under 24.) Greetings fellow unRAIDers. I've built a number of unRAID servers over the years, but this is the first one that doesn't have the drives in separate enclosures from the mainboard. Most of those other builds, including my current primary server, use the SGI SE3016 3U 16-bay SAS enclosure. This build puts low idle power consumption as a top priority. My current primary server (44TB protected) draws a bit over 60 watts at idle. I realized I'd have to lose the external enclosures to hit my goal of less than one watt (at idle) per terabyte (protected). There were a few challenges involved with this build, which I'll go into later, but first, the build details. unRAID Version: 6.1.8 Mainboard / Processor: ASRock Rack C2550D4I http://www.newegg.com/Product/Product.aspx?Item=N82E16813157419 Memory: 1x 16GB ECC UDIMM (DDR3-1600), brand unknown http://www.ebay.com/itm/Single-16GB-ECC-UDIMM-DDR3-1600MHz-for-Xeon-E3-xxxx-V3-Atom-C2000-based-only-/191591469461 Chassis: iStarUSA D-214-MATX http://www.newegg.com/Product/Product.aspx?Item=N82E16811165572 Drive Cages: 2x three-drive cages from Ark IPS-2U235 chassis http://www.arktechinc.net/server-case/2u-rackmount/2u-ipc-2u235.html Power Supply: Seasonic SS-400FL2 http://www.newegg.com/Product/Product.aspx?Item=N82E16817151097 Drives: 6x Seagate 8TB Archive v2 (ST8000AS0002) http://www.newegg.com/Product/Product.aspx?Item=N82E16822178748 Fans: 2x Xigmatek XSF-F8252 http://www.newegg.com/Product/Product.aspx?Item=N82E16835233083 Cables: 6x StarTech 12-Inch Latching Round SATA Cable (LSATARND12) http://www.amazon.com/gp/product/B006K260T2 2x Monoprice 18inch Molex to 3x SATA Power Cable http://www.amazon.com/gp/product/B002HR8XE4 Admittedly, the memory was expensive. A pair of 8GB ECC UDIMMs could be had for more like $100. I already had the 16GB ECC UDIMM (four actually). All drives were purchased for $175 - $200 each. I also got a good price on the mainboard. So not including the memory and the drives (and fans/cables): $200 - mainboard $120 - power supply $60 - chassis -------- $380 - TOTAL There were three main difficulties with this build. First, I wanted ECC memory, because I won't be using a cache drive, but will instead achieve a similar effect with "lots of memory" (and bumping up dirty_ratio). Without that requirement, there are many more mainboard options. Second difficulty was wanting to use a shallow-depth chassis. Most all of my computer hardware is rackmount, but the racks themselves are only 16" - 18" deep. I have a few 20" - 26" deep chassis installed, and could have easily found a workable 2U chassis within that depth range. I ended up with one with a depth of 17" (just checked, it's more like 15.5" deep), and yes it is on the tight side. The final difficulty involved the Seagate 8TB Archive drives, and those missing center side mounting points. Most drive cages want to use that mounting point; I had to dig into my spare parts to find some that could use just the two end points. Here are the numbers. Power Consumption: idle (all drives spun down): 29.1 watts reading (only one drive spun up): 35.0 watts writing (only two drives spun up): 46.5 watts all drives spun up: 58.2 watts peak at boot up: <140 watts I haven't run a Parity Check since switching to the latest power supply, but with the previous one I saw numbers ranging from just under 80 watts at the beginning to just over 70 watts at the end. After a fair bit of Tunables testing, I arrived at disk settings that yielded this Parity Check time and speed: Duration: 15 hours, 6 minutes, 38 seconds. Average speed: 147.1 MB/s Drive temperatures during a Parity Check get into the low-to-mid-30Cs. I just checked write speeds to the one drive that's only 5/8 full and got pretty consistent speeds around 60MB/s (after the memory filled up). Probably for a near-empty drive it'd be closer to 80MB/s, and I'd guess about 50MB/s for a near-full drive. Future directions . . . I'll bump up the memory to at least 32GB, then see how well that works as a cache-drive substitute, going all the way to 64GB if needed. And, I have a couple more 8TB drives; I plan to install a 3-bay hotswap cage in the two ODD bays, then add at least one more data drive, and probably a "warm spare" (leaving the third bay open for "guest" drives). To avoid needing to use hdparm to get the spare drive to spin down, I'm going to try just making it the cache drive (but not actually enabling caching), then it should spin down 15 minutes after booting and stay that way. This sever will be on 24x365, only being taken down for maintenance (hence the priority on low idle power consumption). I'll post some pictures in the next post . . . is that 800-pixel maximum width still in force? Also, even at 768 pixels wide, my pictures are over 100KB each, so to avoid the 192KB-per-post maximum, I guess I'll be making a new post for each picture.

My tunables testing on 6.0-beta10 yielded a time of 15 hours 5 minutes (147.3 MB/s). Switching to 6.1.6 with the same settings gave me a time within 10 seconds of that one. Now I just hope these settings work on the ASRock C2550D4I board I just got. Also, I'm still getting the no parity-check entries logged problem . . . tried the things in that thread (remove parity-check log file, switch back and forth to dashboard), no joy.

The latest run came in at 15.9 hours, 140.7 MB/s. I think there's still room for improvement, because I had some earlier results (with just three drives instead of six) in a SAS enclosure (SE3016) that came in just under 15 hours (148.9 MB/s). I guess I'll keep playing with it. One thing I noticed, on two 6.1.6 servers I'm experimenting with right now, that after each Parity Check, where the duration is normally reported, it says unavailable (no parity-check entries logged). Some entries from last month are in the History, but nothing recent. The only thing I can think of is that I Disabled Page Updates around when it seems to have started not logging them. A search of this forum didn't turn up any similar complaints, am I the first to encounter this problem?

My understanding is that you can write to your array while parity is being checked (I've done it many times). Write speed will of course be reduced in that situation.

Ah, okay. I'll keep your numbers in mind during my testing. I have a run going now on 6.1.6 with settings based on what I used for the baseline run (on 6.0-beta10a): md_num_stripes = 40960 md_sync_window = 12288 md_sync_thresh = 12280 So far (~15%) it's yielding speeds comparable to the baseline run. Only 3% memory used (I've got way more RAM than I need on this motherboard!). I may go back to 6.0-beta10a to try the tunables tester . . . in the past when I've used it, it revealed only minor improvements, but I was using SAS HBAs then and I'm straight to the motherboard now. But this isn't likely to be the motherboard I (eventually) use for this build. I just ordered that one (ASRock Rack C2550D4I). RAM should move straight across from the board I'm using now as that's a Supermicro A1SAM-C2750F. Although I might back it down from 64GB to 32GB. I do notice that my over-the-network writes finish more quickly with lots of RAM (Linux buffering).

I see. So you bumped md_sync_window up from the default of 384 to 1280, then settled on an md_sync_threshold of 1024, leaving md_num_stripes at the default setting of 1280. Let me know if I have that right . . . once my baseline Parity Check finishes (on 6.0-beta10a), I'll try those/similar settings on 6.1.6. Is it just coincidence that you're using 1280 for both md_sync_window and md_num_stripes? Or is there a connection. Thanks again JB.

Got it, thanks again JB. Any way to read out what these are set to? The 'mdcmd status' command isn't showing any of them.

Hi JB, thanks for that information. I found this mention of md_sync_thresh: http://lime-technology.com/forum/index.php?topic=44952.msg429454#msg429454 So it sounds like you're suggesting that my problem *could* be due to md_sync_thresh being (now by default) half of md_sync_window compared to (in the past) one less than md_sync_window. Your average speed and duration data points look JUST like mine: I get ~15 hours when it's working (beta10a) and more like mid-16s when not (6.1.6). Looks like you're running the same drives as I am (Seagate 8TB). What are your settings of md_sync_thresh between your two run? I'm guessing the slower run is for a setting of half of md_sync_window, is the faster run for a setting of one less than md_sync_window? And to answer your question, no I've not adjusted the two tunables that *are* still pinned out to the GUI. EDIT: I still would like to know how to reveal the values of tunables that *aren't* accessible via the GUI.

Looks like you *can* share an unassigned drive: http://lime-technology.com/forum/index.php?topic=45798.0

My understanding is that onboard (motherboard) RAID and controller card (HBA) RAID are both hardware RAID. Software RAID is what an OS might provide.

Clunky yes, but how about df /mnt/cache | tail -1 | cut --delimiter=' ' --fields=1 I don't have a cache drive on any of my servers, but when I run this for disk1 I get /dev/md1

My first unRAID server was an eight-"drive" array where each "drive" was really a five-drive RAID5 array. To unRAID each RAID5 array looked like a single disk. I realize there are reasons not to do it that way, and I don't do it that way any more. But as far as I know, there's nothing preventing you from setting up your four-drive RAID10 array (in the BIOS), and having unRAID see that as a single drive. You could then use that drive as either a data disk, parity disk, cache disk, or "none of the above", i.e. mount it separately from the disks that unRAID manages. As Aaron points out, unRAID does not do iSCSI. And I've never shared a "none of the above" disk in unRAID so I can't say if that can be done. But if you were to make your four-drive RAID10 "disk" a data disk or a cache disk, *that* can be shared easily enough. I've considered doing something like that for the parity disk, to make it less likely to be the bottleneck when writing to more than one data disk at the same time. But right now I'm pushing towards low power consumption on my latest build (less than one watt per terabyte at idle), and using more drives that way would go against that goal.

I started a new server build, and decided to try unRAID 6 official, as opposed to beta12 as I've been using up until now (and still am on my primary server). I believe I'm getting the same problem as I reported with beta12 (compared to beta10a, reported here: http://lime-technology.com/forum/index.php?topic=38290.0 ). The workaround arrived at there was to set a longer poll_spindown time (100 instead of 10). In trying to apply that to the official release, I see poll_attributes instead, and it's already set quite long (30 minutes). I searched around a bit and found this brief discussion: http://lime-technology.com/forum/index.php?topic=38295.msg356200#msg356200 The relevant bit is "poll_spindown remains a variable, just not tunable from the webGui." Apparently the mdcmd command is used to adjust these tunables from the command line. What I'm not finding is a way to ask for the value of a tunable. There's 'mdcmd status', which dumps /proc/mdcmd, but I don't see poll_(anything) in that output. I could just start trying to set poll_spindown to different values to see if it fixes my problem, but ideally I could at least see what it's set to now, and also see that I'm actually changing it when I do try to adjust it. (Just FYI, how I suspect this or something similar to be the problem is that Parity Checks/Syncs proceed much more quickly when I try them with 6.0-beta10 compared to 6.1.6. I've tried all the obvious stuff like disabling page updates during parity operations, and even completely disabling page updates. I still see markedly slower speeds, both peak and average, with *nothing else changed* other than the bzroot/bzimage files. Peak speeds can be over 200 MB/s in the "good" situation versus not much more than 160 MB/s for the problematic one. That's how I know it's not drives/cables/SATA ports/motherboard/CPU/RAM/...) Thanks for your help.