Hi guys. The other day, I was transferring about 1TB worth of data from an 1TB External USB2 drive (attached to a Windows 7 client), while copying to my unraid server and I noticed that both the parity and data disk which was receiving the content paused for about 5 seconds (other their about's).

The copy took about 7 ~ 8 hours and during this time, I had noticed that pause occur on several occasions (say about 5 or so times). The copy from the External Disk to my unraid server went fine, no errors in Windows as well. I'm using Windows 7, no copy-aided applications (ie: Like TeraCopy) and I'm running unraid server 4.5.1 Pro. My conclusion is maybe unraid had built up a cache collection in RAM and was re-caching the next load of copied content from the Windows client to unraid, or something like that. No real bad drama, but was curious as to why this occurs and was wanting to know whether others had picked up on some similar behavior with their rigs as well. Thanks.

I've seen similar behaviour with Vista, where transfer speeds appeared to be dependent on the volume of data being copied.

 

I documented some of my observations (using unRAID 4.4) in:

"Curious performance in copying to unRAID" - http://lime-technology.com/forum/index.php?topic=4541.msg41562#msg41562

 

That thread didn't provide any definitive answers, however some of the theories floated were: re-caching; how the Raiser file system creates files;

 

For me it was also a curiosity...

Hi guys. The other day, I was transferring about 1TB worth of data from an 1TB External USB2 drive (attached to a Windows 7 client), while copying to my unraid server and I noticed that both the parity and data disk which was receiving the content paused for about 5 seconds (other their about's).

The copy took about 7 ~ 8 hours and during this time, I had noticed that pause occur on several occasions (say about 5 or so times). The copy from the External Disk to my unraid server went fine, no errors in Windows as well. I'm using Windows 7, no copy-aided applications (ie: Like TeraCopy) and I'm running unraid server 4.5.1 Pro. My conclusion is maybe unraid had built up a cache collection in RAM and was re-caching the next load of copied content from the Windows client to unraid, or something like that. No real bad drama, but was curious as to why this occurs and was wanting to know whether others had picked up on some similar behavior with their rigs as well. Thanks.

 

 

My observations are

 

(a) using tool like Teracopy is better than Windows's build-in copy-and-paste.

 

(b) Large file is better than small file. You usually can get better overall throughput in large file than small one. For example, if you have 100GB

    data need to move, moving one 100GB file you will have better sustainable throughput than moving 100 of 1GB files. because those file

    open/close and file system updates overhead will interrupt data copy flow. Also large file tend to cluster together in data blocks while small

    file might have higher fragmentation in data blocks in file system. more fragmentation more disk seek time.

Hi guys. The other day, I was transferring about 1TB worth of data from an 1TB External USB2 drive (attached to a Windows 7 client), while copying to my unraid server and I noticed that both the parity and data disk which was receiving the content paused for about 5 seconds (other their about's).

The copy took about 7 ~ 8 hours and during this time, I had noticed that pause occur on several occasions (say about 5 or so times). The copy from the External Disk to my unraid server went fine, no errors in Windows as well. I'm using Windows 7, no copy-aided applications (ie: Like TeraCopy) and I'm running unraid server 4.5.1 Pro. My conclusion is maybe unraid had built up a cache collection in RAM and was re-caching the next load of copied content from the Windows client to unraid, or something like that. No real bad drama, but was curious as to why this occurs and was wanting to know whether others had picked up on some similar behavior with their rigs as well. Thanks.

See the possible explanation for the pauses here:

http://lime-technology.com/forum/index.php?topic=4782.0

Well actually I use to be a strong believer of TeraCopy, but I just haven't bothered with it these days, though it is highly customizable in its copying functions and can queue up copy jobs, which is great. During this time I noticed these pauses, I was transferring files which averaged size was from 700MB to 10GB, but I didn't take notice of which exact sizes of files that had being transferred when the pauses occurred.

It isn't a show stopper nor does it cause any problems or errors either. WeeboTech made a good point in one of the links above about unraid flushing its contents in RAM before it takes on another load of data. I only have 2GB DDR2 in my machine, and with the prices of RAM these days, it might be viable to just buy another 2GB and see if it reduces this. Again, this pausing might increase a 6 hour copy to 6 and 1 minute, so again, no real show stopper, but thanks for the information.  

You might want to give TeraCopy another try.  The CRC checks alone make it worth it (especially for large transfers like the one you just did).

WeeboTech made a good point in one of the links above about unraid flushing its contents in RAM before it takes on another load of data. I only have 2GB DDR2 in my machine, and with the prices of RAM these days, it might be viable to just buy another 2GB and see if it reduces this.

 

(a) File system I/O usually is buffer-I/O, when file is closing, system will flush all buffers related to this file.

(b) When buffer pool is full, system will kick oldest or less-used buffers (depends on implementation) to disk, hence flushing.

© Even you add more memory, your buffer pool is still limited compared with data you are going to write to unRAID, so sooner or later

    your buffer pool will be fulled, then (b) will be kicked in as well.

(d) Some system if using lazy-write (aka delay write) will have a daemon to do periodically flush in order to make sure data will not be sitting in buffers indefinitely.

(f) If you want to flush buffer manually use command "sync" from console.

Thanks for the detailed explanation GK20, some handy information to know. Hey Rajahal, funny you mention that, I had to do a heap of copying the other day and I needed to queue up these copies as well, so I decided to install TeraCopy , which made that task a lot easier. Thanks for the tips guys.