Copying Huge File from iMac to unRAID

[jdag]

I just built my 1st unRAID server and am getting a feel for the system.  I've copied a variety of files over to user shares already, but this 1 particular file is driving me nuts.

 

It is an Aperture Library file, which is a Mac container of thousands of files, and the file is a huge 215gb.  I've tried a number of different methods, copying it with cache on and off.  But the pace is painfully slow, with the iMac showing an estimate at about 6 days to complete.

 

I've transferred roughly 250gb from that same iMac to the unRAID server already, and those transfers, albeit smaller individual files, were speedy.

 

Any ideas on how I can get this moving along?

 

Thanks in advance, John

[Squid]

I just built my 1st unRAID server and am getting a feel for the system.  I've copied a variety of files over to user shares already, but this 1 particular file is driving me nuts.

 

It is an Aperture Library file, which is a Mac container of thousands of files, and the file is a huge 215gb.  I've tried a number of different methods, copying it with cache on and off.  But the pace is painfully slow, with the iMac showing an estimate at about 6 days to complete.

 

I've transferred roughly 250gb from that same iMac to the unRAID server already, and those transfers, albeit smaller individual files, were speedy.

 

Any ideas on how I can get this moving along?

 

Thanks in advance, John

If it winds up transferring thousands and thousands of little files, then I'm not completely surprised due to the overhead involved with any transfer of that many file. 

 

In that case, you could zip it transfer it, then unzip it.

[John_M]

That will be thousands and thousands of photos, so a few megabytes each, I should think.

 

You could try opening the Aperture Library package (navigate to it within the Pictures folder, which is within your user's home folder) by right clicking it and choosing Show Package Contents. You can then copy the contents in more manageable chunks into the Aperture Library folder that's stored on your unRAID server. You don't say how you connect your Mac to the server - AFP or SMB?

 

[jdag]

Thanks for the prompt replies, all.  It appears to be moving along better.  It took about 90 minutes for the first 1Gb or so.  Then about another hour for the next 2Gb.  But all of a sudden it "released" and is up to about 23Gb, with the latest 20Gb moving at a rate of about 1Gb per minute.

 

Aperture keeps tons and tons of thumbnails in addition to the photos.  I bet that was it.  Now here's hoping it will continue at this pace.  I would much rather not mess around with the file.

[jdag]

It finished...took about 5 hours.  But really, the 1st 2-1/2 hours were for just 2-3Gb...the remaining 210Gb or so was about 2-1/2 hours.

[ijuarez]

Also helps if you connect to a gigabit ethernet

 

Sent from my SAMSUNG-SM-N920A using Tapatalk

 

 

[John_M]

Also helps if you connect to a gigabit ethernet

 

Sent from my SAMSUNG-SM-N920A using Tapatalk

 

Yes, it certainly does, but what makes you think that wasn't the case? To transfer 210 GB in 2.5 hours needs a gigabit connection - it works out at a little less than 200 Mb/s - so he clearly wasn't connected at 100 Mb/s.

 

[ijuarez]

Also helps if you connect to a gigabit ethernet

 

Sent from my SAMSUNG-SM-N920A using Tapatalk

 

Yes, it certainly does, but what makes you think that wasn't the case? To transfer 210 GB in 2.5 hours needs a gigabit connection - it works out at a little less than 200 Mb/s - so he clearly wasn't connected at 100 Mb/s.

I was more making a statement for others to make sure they used gigabit but I guess the tone was not correct.  [emoji1]

 

Sent from my SAMSUNG-SM-N920A using Tapatalk

 

 

[andreidelait]

Hi,

 

I can confirm this scenario as well.

And YES I'm using an AFP share and a gigabit connection.

For some reason for any transfer it just starts slow at the beginning and gets at the normal speed after some time.

I know it's well known that Macs are slow with SMB Shares but I am not familiar with AFP.

I'm very new to unRAID (i have the server for one month) and also new using AFP.

I'm not sure if this is a normal behavior but it also bothered me since the first transfer.

 

Good luck.

[John_M]

Try copying a few large files so that you get the feel of it. A folder containing ten 1 GB files should take much less time than a folder containing ten thousand 1 MB files.

[jedimstr]

Have you experimented with using an SMB share with your Mac instead?

 

I know it's "common wisdom" that Mac's are slower with SMB than AFP, but in the last few versions of OSX SMB is actually the primary share format and AFP is deprecated. Apple considers AFP a "Legacy Service". From Mavericks and higher, OSX tries to connect to shares via SMB first before falling back to AFP.

 

On my 10GbE fiber connections, my old Mac Pro 1,1 on Yosemite transfers upwards of 600MB/s to an SMB share which compares favorably to my Windows 10 PC which transfers around 800MB/s on average to the same share.  Bottleneck for my Mac Pro is probably the old PCIe 1 bus that the SFP+ card is connected to rather than what SMB on OSX can do.  I keep AFP off now.

[andreidelait]

I understand that it has to be slow when you copy a big chunk of small files.

That works the same on any file system AFAIK.

When I copy from a Mac to an AFP share starts slow even with a single large file.

It does not bother me because it catches fast and works well.

 

Last time I tried a SMB with Yosemite and it was slow. VERY SLOW.

I will give it a try when I get home to see the difference between SMB and AFP with El Capitan.

 

 

Late edit:

 

So i gave it a try with two ISO files in total of 6.6 GB. I know is very far for a good test but I don't have the time or the knowledge to do a proper test.

Actually is no difference between AFP and SMB in this scenario.

copy to afp 01:37

copy to smb 01:35

copy from afp 01:01

copy from smb 01:04

 

I'm sure it happened to me a few days ago, to start with slow speeds when I was copying to AFP share. But to be honest I didn't try with SMB the same transfer.

Anyway if it happens again and I can reproduce the scenario I'll keep you posted.