Is copying direct to share correct?

[jkBuckethead]

I am brand new to UnRAID.  I just fired up version 6.6.0 for the first time yesterday.  At the moment my initial parity sync is 64% complete.  I am transitioning from an Amahi server.  Amahi worked fine, but the only way it provides data security is by replicating shares on multiple disks.  My media collection is large so this means an awful lot of extra hard drive space.  UnRAID's parity may not be foolproof, but it is good enough for my non-critical movies and television programs.  My plan is to move my files and drives from my Amahi server.  I started UnRAID with three new drives (1 parity, 2 storage), and I will add drives from the Amahi box as files are transferred thus freeing the drive.  After files are transferred I will also be installing Plex, sabnzbd, radarr, and sonarr.  These apps currently run on a separate windows PC.

 

Since I am coming from Amahi, I am used to some specific rules regarding file transfers.  Amahi uses "landing zones" for each share that are similar (but different) to the UnRAID cache.  Files are first written to the landing zone, then moved to the storage pool discs by a background process.  When moved the file is replaced by a symlink pointing to the new location.  For this reason, you could really screw up a share if you messed with files directly on the storage pool drives.

 

My plan for moving files to UnRAID is to copy them directly from the existing network share to UnRAID using Krusader.  I have figured out how to open the existing network share using the New Net Connection tool.  I have also found the new UnRAID share in /Root/media/.  I just need to know if it is safe to move/copy files from the network share directly to /Root/media/sharename/.  This appears to be the right place, but I just want to be sure.  I have moved a few test files, and everything looks good.  I'd just like to be sure before I pull the trigger on several dozen TB of files.

 

Thanks in advance.

[trurl]

Without more information about how you have Krusader volume mapping configured can't say for sure. 

[jkBuckethead]

I'm not exactly sure how to provide more information about Krusader's setup.  I do know that I used all defaults when I installed it.  If there is a log or something, please point me to it, and I'll post it.

 

I didn't do anything fancy.  I just connected to my existing server on one side, and navigated to /Root/media/sharename/ on the other side.  I'm just want to know if this is the correct way to write files to the share.  So far the test files show up just fine when I open the share in windows so I think I am correct.  Just hoping for confirmation.

[JonathanM]

31 minutes ago, jkBuckethead said:

I'm not exactly sure how to provide more information about Krusader's setup.

Post a screenshot of your krusader docker edit screen.

[trurl]

40 minutes ago, jkBuckethead said:

So far the test files show up just fine when I open the share in windows so I think I am correct.  Just hoping for confirmation.

If they are showing up where expected then it must be OK.

 

To your other point about cache. The cache disk is considered part of the user shares. Each user share has settings that control whether and how it uses cache.

 

User shares are the way Unraid allows folders to span disks. A file will never span a disk since each disk is an independent filesystem and a file must be completely on a single disk. But folders can span disks.

 

User shares are simply the aggregate of top level folders on cache and array. The name of the user share is the same as the name of the top level folder. And any top level folder is automatically a user share with the same name as the folder. Any top level folders with the same names on cache and array are part of the same user share. So for example, if you have a top level folder on disk1 named media and a top level folder on disk2 named media, all files/folders in the media folder on both disks are in the media user share.

 

And as I mentioned, cache is also part of the user shares in this sense, but cache has some additional functionality. This link gives a pretty good explanation of the different cache settings each user share can have:

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?page=2#comment-537383

 

 

[jkBuckethead]

2 hours ago, trurl said:

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?page=2#comment-537383

 

Thanks, the link helped me better understand how unraid's cache and data pool interact.  It certainly seems that copying to the share folder directly and letting the system manage the process is the best way.  

 

Now starts the long task of moving 60+TB of data.  I wish I could move the physical disks and copy files directly for the obvious speed advantage.  Unfortunately in the past I have noticed that Amahi deletes symlinks without deleting the actual files.  If I copy directly from the disks I would end up with a lot of old files that have been deleted from the share.

[xman111]

good luck.. i only had about 10 tb and it was a nightmare.

[trurl]

21 hours ago, jkBuckethead said:

Now starts the long task of moving 60+TB of data. 

For the initial data load, you might be better off not caching anything and not running parity.

 

Cache will typically not have enough space for this sort of transfer so it winds up just getting in the way since you have to make it move more frequently, and those moves will impact performance. And if you write to uncached user shares, then of course it goes directly to the array so nothing to move.

 

If you don't have a parity disk, there will be no parity to update as these disks get written, so it will go faster. And since you have another copy of the data on the source there is little risk running without parity for the initial load.

[FreeMan]

18 hours ago, jkBuckethead said:

I wish I could move the physical disks and copy files directly for the obvious speed advantage.

Assuming (with all inherent dangers) that the Amahi server is similar to unRAID in that a complete file is stored on a single disk (i.e. no striping or other data spreading), you could physically pull a disk, connect it to the unRAID server via a SATA->USB adapter and mount it using the Unassigned Disk plugin to eliminate the network transfer. For even faster transfer speed, you could physically mount the disk inside the unRAID case, plug it into a SATA port and mount it via UD to do the same.

 

Both methods will leave the data on the former Amahi server disk until you've confirmed that all is good, but, of course, it would start to break your logical assignments on the old server.

 

It's a trade-off for you to decide - faster transfer speed vs being able to access the files from other devices while the transfers are happening.

[JonathanM]

1 hour ago, FreeMan said:

connect it to the unRAID server via a SATA->USB adapter and mount it using the Unassigned Disk plugin to eliminate the network transfer. For even faster transfer speed, you could physically mount the disk inside the unRAID case, plug it into a SATA port and mount it via UD to do the same.

A healthy 1GB network will transfer faster than a USB2.0 connection. SATA connection might be a little faster, depending on the disks in question. I'd benchmark the current method, then determine if the extra hassle and risk is worth the small bump in speed.

 

Remember, most drive issues are connection related, I'd be reluctant to take down a working machine for a small speed increase.

[jkBuckethead]

Thanks for the suggestions.  I have a 1 Gb connection to Amahi, and a 2x1Gb bonded connection to the unraid server. I know bonding doesn't really help me for a single client, but it's there none the less.  I can't confirm the speed I am getting because if there is a network speed plugin I don't yet have it loaded. 

 

With this network setup and no cache or parity, I am transferring at around 45 MBps so there is definitely speed to gain if I physically move disks and USB 3 or SATA III.  Unfortunately, this would mean taking the amahi server offline because it will not boot with a drive missing.  This is one of the things I dislike about amahi.  Even though the files are stored intact (like unraid) you don't just lose the files on the one disk, you lose the whole server.  

 

Unfortunately, this all may be a moot point at the moment.  For some strange reason I cannot access the unraid server from the PC that is currently running plex, sonarr, radarr and sabnzbd.  I've posted about this elsewhere, so not going to hijack this thread, but I'll have to abandon unraid if I can't get it working with all my machines.