Ways to benchmark your unraid server in file transfer speeds

[unraided]

I've never really gauged what my write and read speeds are to my server, how would I go about doing this? Thanks.

[purko]

I've never really gauged what my write and read speeds are to my server, how would I go about doing this? Thanks.

Simple.  Look at your watch, copy a large file, when its done look at your watch again.   

 

[unraided]

Yeah, that is a way I should of been more specific, I meant to say in data transfer, like in megabytes per second, both in read and write benchmarks. Is their something that could be run in unraids command line or a Windows utility which could assist in these benchmarks? Thanks.

[purko]

I meant to say in data transfer, like in megabytes per second

Well, that method gives you megabytes per second, if you know how many megabytes you copied, and how many seconds it took.

 

[gabbott]

Yeah, that is a way I should of been more specific, I meant to say in data transfer, like in megabytes per second, both in read and write benchmarks. Is their something that could be run in unraids command line or a Windows utility which could assist in these benchmarks? Thanks.

 

If you are running windows and doing the file transfers, it should give you that info in the transfer window.

[purko]

If you are running windows and doing the file transfers, it should give you that info in the transfer window.

Windows 2k/XP don't give you any meaningful info about the file transfers.

(I don't care to learn whether windows 7 does)

 

If you need to have good measurements, get yourself TeraCopy.

Added bonus: TeraCopy will seriously speed up your transfers.

Another bonus: You can set it up to check the CRCs after it completes all the copying.

 

[unraided]

Sounds like TeraCopy is the answer to my original and second question i was going to ask. Another question, would you get faster transfers speeds with Windows XP or Windows 7? I know Windows 7 has methods to improve file transfer speeds, but others say it can slower speeds too. Thoughts? Thanks.

[purko]

I know Windows 7 has methods to improve file transfer speeds

Yes, windows 7 personally rushes all your stuff to the server, bypassing cables disks and everything.

Just wait till you see windows 8 teleport technology!

 

[unraided]

Good point 'purko'. I'm using a mix of Win XP and 7 PCs. I'll try TeraCopy and see what I get in terms on megabytes per second. Thanks.

[TrimbleEpic]

Just throwing this out there...

 

It might also be useful to benchmark - two computers copying two files to the server, each file being saved to a different HDD.  (benchmark the controller and bus speed)

[vwalien]

I just switched to TeraCopy today, now I'm getting 95MB/s xfer rate, this is up from the 4MB/s I was getting with ftp (filezilla/vsftpd) on my 1Gbps Lan.

 

I'm totally impressed with this software over TC or FTP or SMB.

 

Edit, sorry this information is not 100% correct, I did see the 95MB/s rate when TeraCopy was performing the CRC verification, this could be the *read* speed coming from the unraid server.

I actually get a file copy/write speed of anywhere between 21MB/s all the way down to about 6MB/s depending on the file size.

Also, I make sure that my computer is not doing any other harddrive activity, and my unraid server was not doing anything else.

 

[jimwhite]

I just switched to TeraCopy today, now I'm getting 95MB/s xfer rate, this is up from the 4MB/s I was getting with ftp (filezilla/vsftpd) on my 1Gbps Lan.

 

I'm totally impressed with this software over TC or FTP or SMB.

 

It would be interesting to know HOW this is happening... maybe some of it can be used to Tune unRAID itself??

[vwalien]

I just switched to TeraCopy today, now I'm getting 95MB/s xfer rate, this is up from the 4MB/s I was getting with ftp (filezilla/vsftpd) on my 1Gbps Lan.

 

I'm totally impressed with this software over TC or FTP or SMB.

 

It would be interesting to know HOW this is happening... maybe some of it can be used to Tune unRAID itself??

 

sorry I reported my information too early (didn't realize what TeraCopy was doing)..  

 

So Here's the numbers I've seen after using it for 8 hours..

 

CRC Check, the test it does at the end, after copying the was running at 95MB/s

File copy of large files (ISO's etc) was running at 19MB/s (on average)

File copy of many small files was running at 6MB/s (on average)

 

I get these messages from the title bar in TeraCopy, it reports the speed there. But I also look at the network utilization in the TaskManager and it was up to 16% usage. Before utilization was only 1%.

 

My unraid is not optimized in any way, I would need to replace my motherboard/cpu/ram to bypass the IDE bus. I have a mixed bag of ide/sata-I/sata-II drives, all connected to the IDE bus through IDE connectors and PCI Sata adapter cards.

 

But still, this is a huge improvement for me.

[purko]

I get these messages from the title bar in TeraCopy, it reports the speed there.

Those are current speeds (which may vary a lot), and not the average speeds.

Click on button "More", and then in the bottom of that window there's a pop-down with the complete log, averages, times, etc..

Check that log after the copy finishes.

 

[Yorgo]

Is there a benefit to using TeraCopy Pro with unRAID or are all the tests you guys made with the freeware version?

 

[purko]

Is there a benefit to using TeraCopy Pro with unRAID or are all the tests you guys made with the freeware version?

I don't think there's any speed benefit with TeraCopy Pro.

 

[unraided]

I've used TeraCopy before and it has improved file trasfer speeds, not by a huge improvement, but it depends on the files your copying too. So the best way to transfer files to your server via a ethernet connection is to copy the files straight to either the disk share directly or to your or share simply just simply using two windows explorer windows? Thanks.

[havix]

For testing network file transfers I use Robocopy.  This will give you probably the most accurate measurement plus it's a Microsoft tool.  Teracopy I've had mixed results with over the years and I would personally not recommend using it as a replacement to Windows Explorer's file copy method. 

[purko]

For testing network file transfers I use Robocopy.  This will give you probably the most accurate measurement plus it's a Microsoft tool.

That's one reason for me NOT to want Robocopy.

There's a whole bunch of bugs in windows7 when copying to unRAID that we are trying to work around.

 

Microsoft dropped Robocopy many years ago. Now they decided to include it in windows 2008, vista, and windows7.

 

One notable thing about Robocopy is that it will silently fail to copy open files that are in use by other users or applications. (!!!)

Say, I've just ripped a few movies with DVDShrink to my local disk. Then I copy all that stuff to my server with Robocopy. The copying finishes, and I happily move on to other things, blissfully unaware that some of the .VOBs were never really copied, just because the DVDShrink app happened to still be open on my laptop... I will find out about this months later, when I try watching that movie. So, thanks, but no thanks!

 

[havix]

Robocopy is a must have tool in the Windows Enterprise environment.  I've used it for large file server migrations to maintain permissions, timestamps, and to mirror file systems up until the point of the migration. It is generally excepted as a time proven professional tool for file copying.  To use this at home might be considered overkill but it's one of those things that once you know how to use it there isn't anything else out there like it with the same features and functionality.  To see why it's considered a great tool take a look at the things you can do with it "robocopy /?".    I hate to argue too much about this so I'll just copy in from the Robocopy Wiki on SOME of the reasons why you would use this over other methods.  

 

# Ability to tolerate network outages and resume copying where it previously left off (incomplete files are noted with a date stamp corresponding to 1980-01-01 and contain a recovery record so Robocopy knows from where to continue).

# Ability to correctly copy file data, attributes and file timestamps, as well as NTFS ACLs, owner information, and auditing information using command line switches (/COPYALL or /COPY:); copying folder timestamps is also possible in later versions (/DCOPY:T).

# Ability to assert the Windows NT "backup right" (/B) so an administrator may copy an entire directory, including files denied readability to the administrator.

# Persistence by default, with a programmable number of automatic retries if a file cannot be opened.

# A "mirror" mode, which keeps trees in sync by optionally deleting files out of the destination that are no longer present in the source.

# Ability to copy large numbers of files that would otherwise crash the built-in XCOPY utility.

# A progress indicator on the command line that updates continuously.

# Ability to copy long file and folder names exceeding 256 characters — up to a theoretical 32,000 characters — without errors.[1]

# Supports multithreaded copying (Windows 7 only).

 

[purko]

Havix, I am sure robocopy is very useful to you.

It's just that when I see the words microsoft and great used together, that's when you lose me.

 

[havix]

purko, up until I used Unraid I would have said "It's just that when I see the words linux and great used together, that's when you lose me". 

[purko]

purko, up until I used Unraid I would have said "It's just that when I see the words linux and great used together, that's when you lose me". 

 

Right.  To rephrase your statement, up until then you were unfamiliar with Linux.  Understandable.

 

I can't say that I'm unfamiliar with microsoft though. 

Most of the hair that's missing from my head, that's microsoft's fault.

 

[havix]

My mistake was trying to use Linux as my desktop OS.  It seems like every year or so I load up a couple different distros of linux and run them on my laptop as my main OS for a couple weeks before I give up.  I even ran Solaris 10 for about 6 months.  In a server environment, mobile environment and appliance market Linux certainly has something to offer but as a desktop OS the apps just aren't at the same level.

 

 

[terrastrife]

you know, teracopy and supercopier both rely on robocopy, thats why supercopier took so long to get its update for 7, and why teracopy was/is so buggy

 

robocopy isnt bad, but with a frontend its even better