Seagate’s first shingled hard drives now shipping: 8TB for just $260

[garycase]

A few thoughts ...

 

=>  I think it's safe to say that if you do a write of perhaps 200,000 files with an average size of 250k and that doesn't cause the system to "hit the wall" of shingled band rewrites, then there's nothing to worry about with using these drives.  The key thing is be sure the files are smaller than the shingled bands, so they don't automatically result in full band writes instead of utilizing the persistent cache.    Based on the reverse engineering done by others, the band size has been determined to be about 40MB on the outer cylinders, decreasing as the drive moves towards the inner cylinders -- so file sizes of 250k are fine, as they're well below this.

 

HOWEVER ... to really know how the SMR drives are performing, this same test needs to be done on a traditional PMR setup, because with a large number of such small files the write speeds are going to be significantly slower than with larger files due to the directory updates.    I'm not home at the moment -- and won't be for over a week [typing this on my laptop from a resort at Lake Travis while sipping a Margarita  ], so I can't do that, but perhaps someone else could to provide a baseline.  Ideally that baseline should be on a system using drives with 1TB/platter areal density (at least for the parity drive and the drive being written to), so the only variable is the shingled technology.

 

Note that if these writes are using a good bit of the persistent cache, the average will still be slower than with a PMR drive, as there's an additional latency of ~ 20ms/write caused by the drive updating the cache map when it writes to the persistent cache.  This isn't enough additional overhead to make much difference when you're writing large files that largely bypass the persistent cache; but with a lot of small files it will likely be more apparent IF they use the persistent cache => which they absolutely would if the writes were all random (but likely won't in the scenario you're going to test, since I suspect they'll be sequential writes that largely bypass the need for the cache).

 

 

 

 

[danioj]

A few thoughts ...

 

=>  I think it's safe to say that if you do a write of perhaps 200,000 files with an average size of 250k and that doesn't cause the system to "hit the wall" of shingled band rewrites, then there's nothing to worry about with using these drives.  The key thing is be sure the files are smaller than the shingled bands, so they don't automatically result in full band writes instead of utilizing the persistent cache.    Based on the reverse engineering done by others, the band size has been determined to be about 40MB on the outer cylinders, decreasing as the drive moves towards the inner cylinders -- so file sizes of 250k are fine, as they're well below this.

 

HOWEVER ... to really know how the SMR drives are performing, this same test needs to be done on a traditional PMR setup, because with a large number of such small files the write speeds are going to be significantly slower than with larger files due to the directory updates.    I'm not home at the moment -- and won't be for over a week [typing this on my laptop from a resort at Lake Travis while sipping a Margarita  ], so I can't do that, but perhaps someone else could to provide a baseline.  Ideally that baseline should be on a system using drives with 1TB/platter areal density (at least for the parity drive and the drive being written to), so the only variable is the shingled technology.

 

Note that if these writes are using a good bit of the persistent cache, the average will still be slower than with a PMR drive, as there's an additional latency of ~ 20ms/write caused by the drive updating the cache map when it writes to the persistent cache.  This isn't enough additional overhead to make much difference when you're writing large files that largely bypass the persistent cache; but with a lot of small files it will likely be more apparent IF they use the persistent cache => which they absolutely would if the writes were all random (but likely won't in the scenario you're going to test, since I suspect they'll be sequential writes that largely bypass the need for the cache).

 

 

Lake Travis while sipping a Margarita 

 

I am sooo jelous!

 

I could provide the baseline data too! I can execute two copies side by side from my W7 VM on my iMac. One to the Main Server (made up of WD Red 3TB's) and the Backup Server (made of of the Seagates).

 

I have spoke with c3 about some more advanced tests (as I interpret them) BUT I am also interested in just doing a real world test and this seems like it would fit the bill.

 

Ill generate a set of data with files of ~215k and about 50GB of them and ill distribute them randomly over a directory structure about 20 deep. Then I'll run the copy to both servers simultaneously!

 

Should do it I think!

[garycase]

Excellent -- that  should definitely do the trick.

 

There are several more advanced tests that have been run by organizations with some advanced testing capabilities to reverse-engineer the characteristics of these drives.    One that can fairly easily force the drive to have major write issues (due to a full persistent cache and ongoing random writes) is a continuous set of random 4k writes scattered throughout the disk.  But no file system is going to cause this kind of behavior.    With this very non-real-world scenario, you can effectively trash the performance in about 20 minutes of continuous writing => at which point the drive effectively freezes until it is able to clear some of the persistent cache.  At that point, writes are under 1MB/s for a few hours !!  [To clear each 4k block from the cache requires reading en entire band and rewriting it ... an operation that takes 1-2 seconds -- and there are a lot of 4k blocks in 25GB  ]    And it will be even slower if there is still ongoing write activity being attempted (or even reads, which would cause a lot of thrashing of the heads).

 

 

[danioj]

Excellent -- that  should definitely do the trick.

 

There are several more advanced tests that have been run by organizations with some advanced testing capabilities to reverse-engineer the characteristics of these drives.    One that can fairly easily force the drive to have major write issues (due to a full persistent cache and ongoing random writes) is a continuous set of random 4k writes scattered throughout the disk.  But no file system is going to cause this kind of behavior.    With this very non-real-world scenario, you can effectively trash the performance in about 20 minutes of continuous writing => at which point the drive effectively freezes until it is able to clear some of the persistent cache.  At that point, writes are under 1MB/s for a few hours !!  [To clear each 4k block from the cache requires reading en entire band and rewriting it ... an operation that takes 1-2 seconds -- and there are a lot of 4k blocks in 25GB  ]    And it will be even slower if there is still ongoing write activity being attempted (or even reads, which would cause a lot of thrashing of the heads).

 

OK - I have started preparing. For everyones benefit here is a screenshot of how I am preparing the test. Basically I am running a script that is generating files of 128KB. 10 Files per folder and 10 subfolders per folder and down 6 levels. I'll cancel the script when it hits 50GB and that will be the test data set.

 

[garycase]

Looks good.

 

One thought:  I see from your configuration that your main server has a cache drive.    Clearly you do NOT want that cache to be involved in the test writes.    I'd create a new non-cached share [e.g. "TestShare"] and use it as the destination for your copies to this server.    If you set that share to use "Most free" allocation and allow it to use all of your data drives, that will also keep the writes as far out on the drives as possible ... which will maximize the write speed.

 

[JimPhreak]

Did these drives just recently go up in price?  I could l have sworn I see them for $260ish last week but now I can't find them for under $300.

[danioj]

 

Looks good.

 

One thought:  I see from your configuration that your main server has a cache drive.    Clearly you do NOT want that cache to be involved in the test writes.    I'd create a new non-cached share [e.g. "TestShare"] and use it as the destination for your copies to this server.    If you set that share to use "Most free" allocation and allow it to use all of your data drives, that will also keep the writes as far out on the drives as possible ... which will maximize the write speed.

 

Good catch! I'll apply the settings and do as you suggest.

 

I'm travelling again - this time to ADL. Back Wednesday night!

 

When I got this morning the script had generated 32GB of data so still some way off 50GB. So I hit Ctrl-Z and paused the thing.

 

I'll resume it when I get back and start the copy. Better to be there to monitor and take screen shots anyway!!

 

Hope you're enjoying the lake!!

[danioj]

 

Did these drives just recently go up in price?  I could l have sworn I see them for $260ish last week but now I can't find them for under $300.

 

You're observations match my own.

 

My view is that vendors are inflating the price. My view is that they see the other 6TB drives so close in price and are like - woah - this is an 8TB drive the average consumer won't know the difference in applied technologies (eg SMR over PMR) so we can still charge more! So they do.

[garycase]

Agree.  I saw them for $259 just a couple weeks ago, but now they're $289 and up.    I suspect it's simply a supply and demand thing --- they've been pretty popular, and are now in short supply, so the vendors can charge more.    I'd expect them to be down in the $250 range within 2-3 months.

 

Daniel => Agree r.e. monitoring the process ... just do it when you get back.

 

 

[johnsanc]

Good ole supply and demand.

 

http://camelcamelcamel.com/Archive-3-5-Internal-Hard-Drive/product/B00QX0ZGO6

[JimPhreak]

I'm glad I have some time before I need to upgrade my parity drives.  Paying $600 to gain an extra 3TB of usable space is ROUGH!

[garycase]

I'm glad I have some time before I need to upgrade my parity drives.  Paying $600 to gain an extra 3TB of usable space is ROUGH!

 

??  If you buy two of these and swap out a 3TB parity you'll gain 11TB ... 3TB from the old parity drive and 8TB from the 2nd 8TB drive.

 

[JimPhreak]

I'm glad I have some time before I need to upgrade my parity drives.  Paying $600 to gain an extra 3TB of usable space is ROUGH!

 

??  If you buy two of these and swap out a 3TB parity you'll gain 11TB ... 3TB from the old parity drive and 8TB from the 2nd 8TB drive.

 

1 for each of my servers (main server and backup).  Everytime I upgrade/replace a drive I have to double up.  So effectively I'm only gaining 3TB of space .

[garycase]

Ahh ... the light bulb went on and I realized what you're doing => replacing your 3TB parity drive in two different servers.  So you won't gain 11TB.  However, you ARE gaining 6TB of space for your $600 => 3TB on each of 2 servers 

 

Still a bit "rough" => but not as bad as if you were only gaining 3TB in total.

 

[JimPhreak]

Ahh ... the light bulb went on and I realized what you're doing => replacing your 3TB parity drive in two different servers.  So you won't gain 11TB.  However, you ARE going 6TB of space for your $600 => 3TB on each of 2 servers 

 

Still a bit "rough" => but not as bad as if you were only gaining 3TB in total.

 

Right but since I'm mirroring both servers, I'm effectively only gaining 3TB of usable space.  Just the reality of redundancy.

[c3]

Right but since I'm mirroring both servers, I'm effectively only gaining 3TB of usable space.  Just the reality of redundancy.

 

So, buying the lowest $/TB drives really makes sense. Currently that would be the $120 5TB.

[JimPhreak]

Right but since I'm mirroring both servers, I'm effectively only gaining 3TB of usable space.  Just the reality of redundancy.

 

So, buying the lowest $/TB drives really makes sense. Currently that would be the $120 5TB.

 

That certainly is the most cost effective way for me to go based on the $/TB ratio.  However since my plan is most likely to populate my entire server with these 8TB shingled drives over the next few years, it makes sense for me to go big with regard to upgrading my parity drives.  I'm limited to slots for data disks (not including parity and cache) per server so I can't simply add more drives when I need more space as the space fills up.

[garycase]

Yes, when you back everything up there's indeed a cost -- you have to double-up on everything.  I'd still say you're getting 6TB of extra space for your $600, however => it just happens to be split into 3TB on each of your 2 servers.  Same thing is true if you decide to add more space later => as you add additional 8TB drives you'll need to do it pairs -- doesn't mean you're paying $600/drive, however ... just means you have to buy them 2-at-a-time, for $300 each.

 

[JimPhreak]

Yes, when you back everything up there's indeed a cost -- you have to double-up on everything.  I'd still say you're getting 6TB of extra space for your $600, however => it just happens to be split into 3TB on each of your 2 servers.  Same thing is true if you decide to add more space later => as you add additional 8TB drives you'll need to do it pairs -- doesn't mean you're paying $600/drive, however ... just means you have to buy them 2-at-a-time, for $300 each.

 

I agree with you 100%.  It's just that my initial thought when doing an upgrade always first goes to "how many more files can I store on my server" and that answer is 3TB worth.  While you're correct with your assessment and I understand it fully, it's just a mind game that my head plays with me .

[c3]

That certainly is the most cost effective way for me to go based on the $/TB ratio.  However since my plan is most likely to populate my entire server with these 8TB shingled drives over the next few years, it makes sense for me to go big with regard to upgrading my parity drives.  I'm limited to slots for data disks (not including parity and cache) per server so I can't simply add more drives when I need more space as the space fills up.

 

If you are only buying 8TB drives in a few years, the 5TB is probably a good choice now. The $/TB is far better then the $600/3TB. As you become slot full, you might be pressed to go beyond the 8TB. The 8TB is not going to last the year as the largest unRAID usable drive (10TB Helio is not).

[JimPhreak]

That certainly is the most cost effective way for me to go based on the $/TB ratio.  However since my plan is most likely to populate my entire server with these 8TB shingled drives over the next few years, it makes sense for me to go big with regard to upgrading my parity drives.  I'm limited to slots for data disks (not including parity and cache) per server so I can't simply add more drives when I need more space as the space fills up.

 

If you are only buying 8TB drives in a few years, the 5TB is probably a good choice now. The $/TB is far better then the $600/3TB. As you become slot full, you might be pressed to go beyond the 8TB. The 8TB is not going to last the year as the largest unRAID usable drive (10TB Helio is not).

 

Not a terrible idea.  We'll see where the prices are in 2-3 months when I plan to upgrade and I'll go from there.  But I can't argue with your logic.

[danioj]

Post edited as it was incomplete.

 

Please refer to the following post later in the thread: http://lime-technology.com/forum/index.php?topic=36749.msg373164#msg373164

[garycase]

As I noted before, copying a lot of very small files WILL be slow, because of the intervening directory writes.

 

But there may be another issue here => doing both copies at the same time may be causing some notable slowdown in your VM on the imac.    Not sure if that's the case or not, but you if you want to test it just abort the copies; then do the test to your main server by itself for a brief time (an hour is plenty) and see if the speed is appreciably different.    The reality is you don't actually need to run that entire test anyway -- the test to the main server is mainly to just establish a baseline for what kind of speeds you'll see with all the small files.  An hour or two is plenty to determine that.    Then you could just stop that copy and do the full copy to the backup server with the shingled drives to see if (a) you get similar speeds and (b) if it completely hangs at some point (which is the real thing we're trying to determine here).  ["completely hangs" isn't entirely accurate -- it will, eventually recover and continue if that happens ... but you get the idea]

 

[c3]

 

Here are screen shots of the Main Server (Containing ONLY 3TB WD Red's (x5 incl 1 Parity) with a tempshare set to MostFree and no Cache) and share settings for the test:

 

 

 

Here are screen shots of the Backup Server (Containing ONLY 8TB Seagate Archives (x3 incl 1 Parity) with a tempshare set to MostFree and no Cache)  and share settings for the test:

 

 

Seemed to have repeated an image there.

 

I created the data set on my iMac as the screenshots indicate and then shared the folder via VirtualBoxGuest Additions with the W7 VM (so it shows as a Network Drive in W7 even though the data exists on the same physical disk) and then executed a simultaneous copy to the two shares from the W7VM to the Main Server and the Backup Server.

 

Screen Shot 1: Note the copy started sloooooooow!: 1% (~ 1MB/s)

 

 

Screen Shot 2: As time went on still slooooooooow!: 3% (~ 500KB/s)

 

 

I will check in over the next few hours and show how it goes!

 

INTERESTINGLY for me anyway - they are both copying slow! With no real difference! Will be interesting in an hour or so!! 

 

Which is which? You show two copy processes in Screen Shot 2, but can we tell which is to which target (ie is the copy to 8TB the upper process or lower)?

 

From here, it smells like the source is not memory and likely a bottleneck.

[opentoe]

Agreed! It would be much worse if it was Unraid.

 

I went into the data centre at work and asked the tech in there and he might have solved it for me. He said that terracopy has a memory leak issue that he has experienced which means it keeps consuming until it crashes the OS when copying large volumes of data. He said that the issue was identified by the terracopy team and addressed in their latest v3 alpha. Suggested I try that (I downloaded latest "stable"). I'll give it one more go with this suggestion before I switch to Rsync.

 

The gods don't want me to do this do they!??

 

I know this is from a little bit ago but I'm glad to read this. I remember ALWAYS having trouble with Teracopy and wanting to use it but it just kept failing. Going to check it out now.