Hello,

 

What would be your suggestion to download very large (>10GB) files from remote server over Internet? I used to use ftp, but because of the size of the files, there is a high risk for CRC errors and corrupted files. Not sure it does matter, but I can have VPN connection to the server.

 

Thank you for your suggestions in advance.

One possible way is to create a split archive using a program like 7zip or Rar. Either program provides for CRC checking of individual parts. I know rar can provide for some means of repairability depending on how the archive is created. Both will also compress the files if possible to minimize the data that needs to be moved.

 

Though you might be able to use TeraCopy and it's ability to resume to copy the files directly.

 

 

Thanks, but that is what I meant on "comfortable".

 

Since I am downloading files like this regularly, I can't afford compressing/splitting all those files one by one on the server than do the opposite on the other side after downloads.

 

I can't beleive that torrenting is the only way to ensure data consistency with larger files, and there is no other way of transferring with on the fly CRC.

 

I am looking into TeraCopy if it is doing something like this, thanks for the pointer (however it is not just about resuming. FTP is also able to resume, but not on a reliable way).

I think, at least in the free version, Teracopy cannot do on the fly crc checks, but will do a crc check after copying.

 

Why not try par2 files. Create a minimum set of par2 files on the source server, transfer with ftp the files and a par file. Then run a check, if there are errors then transfer over the minimum amount of par files to repair.

 

You probably could batch script this up for easy use.

I think, at least in the free version, Teracopy cannot do on the fly crc checks, but will do a crc check after copying.

 

Why not try par2 files. Create a minimum set of par2 files on the source server, transfer with ftp the files and a par file. Then run a check, if there are errors then transfer over the minimum amount of par files to repair.

 

You probably could batch script this up for easy use.

Perhaps this:

http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp

 

unRAID includes the rsync program.  You have to set up a configuration file and start the daemon process on the unRAID server, but it can certainly handle large files and does verification of the transfer.

 

Joe L.

Thank you for the suggestions.

 

@jamerson9

It would again the same pain as with rar or 7zip and it would also require time consuming and uncomfortabe terminal operations for a daily task.

 

To be more clear what I am looking for:

I've a server where I am doing torrent downloads (for example a few HD Movies which are 10-20GB). I am looking for a solution to download those movies from this server to unRAID. In the past most of the movies were rared, so wasn't a problem to download via ftp. Lately most of them are just the pure file and that gets corrupted via ftp.

 

@Joe L.

Is rsync something which I should look at? I would prefer a GUI based solution, but it would be even better if some automatic download could be set up around this. My problem with rsync is that it doesn't provide some GUI based overview about what's going on (what is in progress, what is completed, etc) and control.

Deltacopy seems to be a win32 based application. Actually I have Linux on both the server and on unRAID (obviously). I could have a win32 machine in vmware to install Deltacopy, this case I suppose I should setup the rsync daemon on the server.

 

Nevertheless if I end up with a manual solution as previously with ftp, then I want to find something which is the same simple: I connect with some GUI, select stuff to download, then sit back and see them downloading and have the possibility to check the status.

I've not downloaded or used DeltaCopy.

 

From the page I linked to:

In technical terms, DeltaCopy is a "Windows Friendly" wrapper around the Rsync program, currently maintained by Wayne Davison. "rsync" is primarily designed for Unix/Linux/BSD systems. Although ports are available for Windows, they typically require downloading Cygwin libraries and manual configuration.

 

Unlike "rsync", DeltaCopy is a only available for Windows and is tightly integrated with services available only on Microsoft platforms (NT4, XP, 2000 & 2003). Here is a list of features

 

    * Incremental backup - Copies part of the file that is actually modified

    * Task scheduler - Profiles in DeltaCopy can run based on a schedule

    * Email notification - Administrators can receive email confirmation on successful as well as failed transfers

    * One-click restore - Backed up files can be easily restored.

    * Windows friendly environment - No need to manually modify configuration files or play around with command line options.

 

You do need to create a rsync.conf file on the unRAID server and start the rsyncd process.

 

Starting the rsyncd process can be done with a line like this in your "go" script:

rsync --daemon --config=/boot/config/rsyncd.conf

 

The config file can be anywhere.  Mine looks like this:

cat /boot/config/rsyncd.conf

gid = root

uid = root

log file = /var/log/resyncd.log

 

[user]

        path = /mnt/user

        comment = all of user

        read only = no

        list = yes

 

I think it will provide the GUI interface you are looking for.

I presume you are telling us you have a seedbox and need to retrieve data off of it.

 

I have a similar situation and what I do is to use SFTP (SSH file transfer Protocol). Because the line is encrypted, I have not had any of the corruption problems that I had with normal ftp. I have had great success using filezilla to do the transfers. Some other things that you can do is to do a recheck of the folder after downloading using the original .torrent file. So for very large torrents, I have a local install of Utorrent that I point to the same folder that I just downloaded over ftp and have it to a force recheck. This will allow the torrent's native checksum (which is sha I believe) do the integrity checking.

 

Something you can also do is to create a checksum file on the server such as a .sfv file with sha/md5/crc32. This will create checksums for all the file(s) that you can compare with the downloaded versions.