October 1, 20187 yr Hi all, I am pretty new with Unraid. I've been trying this out on some older hardware to try and utilize for backup storage. There are something like 10 or so 3+4TB drives with another 4TB as parity. In this example I've setup rysync on unraid which allows my other NAS to connect to the unraid one. I've noticed that my backup process will stop way to early and it appears that only 1 drive contains data. Even if I try and resume the transfer after it stops it does not work. I've tried this twice now one having rsync transfer to: /mnt/disk1/mypath and the other time /mnt/user/mypath Not really sure what the difference is between disk# and user. As I'm not really to sure how unraid setup the array I am wondering if I am transferring to the wrong path? IE is ZFS mapped somewhere else? Or if that should be fine would there be something else I can be checking as to why my disks are not all being used? Thoughts? Thanks!
October 1, 20187 yr Community Expert Not clear what any of this has to do with ZFS. Unraid isn't ZFS. The User Shares are in /mnt/user. User shares are how Unraid allows folders to span disks. /mnt/disk1 is just disk1. I think your problem is probably due to rsync creating all the folders in advance of transferring any of the files. Unraid is probably creating all those folders on disk1 since it has plenty of space, then rsync fills those with files and it runs out of space. I know others have had issues with large rsync transfers for this reason. Could you tell us more details about your rsync command? Also, go to Tools - Diagnostics and post the complete diagnostics zip.
October 1, 20187 yr Community Expert 8 minutes ago, trurl said: Unraid is probably creating all those folders on disk1 since it has plenty of space This is a possibility, if split level > 0 is being used, another possibility is no min free space is configured (or an inadequate value is set) for that share.
October 1, 20187 yr Author Hi all, Thanks for your quick replies. I will try and answer your questions. My apologies I mixed up Z and X. The file system is XFS. :) This would explain why my first transfer did not work out as it was to the disk. I don't believe I can give you the rsync command as this is done via a backup software on my NAS. The other NAS is Synology w/ Hyper Backup software. Within the software it supports Rsync for non-Synology NAS. When I look at the file system under unraid it does appear that the backup is split up. Looks something like this: command: du -h /mnt/user/mybackup/backupfile.hbk/Pool/0/0 /mnt/user/mybackup/backupfile.hbk/Pool/0/1 /mnt/user/mybackup/backupfile.hbk/Pool/0/2 /mnt/user/mybackup/backupfile.hbk/Pool/0/3 ... ... All about 100G each. However interesting again checking the actual disk (df -h) disk1 is only 33% used (all others are 1%) In terms of settings I believe I just used defaults. Checking under the share I see: Allocation method: High-Water Minimum free space: 0KB Split level: Automatically split any directory as required Included disks: all Use cache disk: Prefer Is the something recommended to change these to? Cheers!
October 1, 20187 yr 4 minutes ago, qwaven said: Allocation method: High-Water Toggle the help in the GUI and read the description for each of the allocation methods.
October 1, 20187 yr Community Expert 5 minutes ago, qwaven said: However interesting again checking the actual disk (df -h) disk1 is only 33% used (all others are 1%) Then it's working correctly, with High water it only starting writing to another when disk1 reaches 50% (if they are all the same size), still you should set minimum free space to twice the largest file size you expect to copy, or you could get ENOSPC when the disks reach full capacity.
October 1, 20187 yr Author 5 minutes ago, jonathanm said: Toggle the help in the GUI and read the description for each of the allocation methods. Yes under help it says High Water will choose the lowest numbered disk with free space still above the high water mark (which is 2TB then) But if no disk has that than its divided again. I'm not clear why this would stop my other drives from being used. The data I'm transferring should be closer to 30TB which well exceeds the capacity of any individual drive. Thanks!
October 1, 20187 yr Author 8 minutes ago, johnnie.black said: Then it's working correctly, with High water it only starting writing to another when disk1 reaches 50% (if they are all the same size), still you should set minimum free space to twice the largest file size you expect to copy, or you could get ENOSPC when the disks reach full capacity. I'm not clear what ENOSPC means. Given that it looks like Synology is transferring in chunks of 100G, than you are recommending I change the minimum to 200G free space? Thanks
October 1, 20187 yr Community Expert 16 minutes ago, qwaven said: Given that it looks like Synology is transferring in chunks of 100G, than you are recommending I change the minimum to 200G free space? If the largest file will always be 100GB, no need to set 200, set it at 120 or 150GB max.
October 1, 20187 yr 41 minutes ago, qwaven said: I'm not clear why this would stop my other drives from being used. The data I'm transferring should be closer to 30TB which well exceeds the capacity of any individual drive. https://wiki.unraid.net/Un-Official_UnRAID_Manual#High_Water Does that help? Unraid doesn't have any way of knowing what your total transfer will be, each file is evaluated for placement when it is opened for writing.
October 1, 20187 yr Author 20 minutes ago, johnnie.black said: If the largest file will always be 100GB, no need to set 200, set it at 120 or 150GB max. I have done this and tried to resume but it stopped again. I do not see any attempt to write to any other disk. They are all spun-down except for the one disk in the array. :( Any ideas? Is there anyway to just make this not care about the file sizes being used and span files across the disks like a normal raid does?
October 1, 20187 yr Community Expert Just now, qwaven said: I have done this and tried to resume but it stopped again. It shouldn't stop, please post your diagnostics, maybe something visible there: Tools -> Diagnostics
October 1, 20187 yr Community Expert You should set Use cache disk to No or possibly Yes. Prefer is definitely not what you want. You should set Minimum Free to larger than the largest file you expect to write to that User Share. Further explanation: Unraid has no way to know how large a file will become when it begins to write it. If there is less than Minimum Free it will choose a different disk to begin the write. If there is more than Minimum Free, it can choose the disk and if the file is too large to fit it will error. Based on what you said, you have a User Share named "mybackup". I assume the share settings you gave us are for that share. 1 hour ago, qwaven said: Use cache disk: Prefer This is definitely wrong. Prefer means prefer to store files on cache permanently if there is room. And if there is not room overflow to the array temporarily but move to cache when space is available. And the way it determines if it needs to overflow is if there is less than Minimum Free for cache. This is a separate setting in Global Share Settings. If you have a cache disk you should set Minimum Free for cache. Not sure it will offer this setting if you don't have cache. So, it is trying to put everything on cache and probably filling it up until it errors because of the Minimum Free setting for cache. Or maybe you don't even have a cache disk. In that case it would pick an array disk instead and later try to move the files to cache. And you haven't set Minimum Free for that User Share, so it could also fill up an array disk until it errors instead of choosing another disk. Anyway, the diagnostics we have asked for will clear up some of these questions.
October 1, 20187 yr Author Hi all, So I've tried changing the cache setting to no but this also did not seem to change anything. I also noticed that if I dig deeper into the file structure the files are actually more like 50MB in size. 200K backup.hbk/Pool/0/0/52.index.2 51M backup.hbk/Pool/0/0/65.bucket.2 200K backup.hbk/Pool/0/0/65.index.2 51M backup.hbk/Pool/0/0/66.bucket.2 200K backup.hbk/Pool/0/0/66.index.2 51M backup.hbk/Pool/0/0/54.bucket.2 200K backup.hbk/Pool/0/0/54.index.2 51M backup.hbk/Pool/0/0/58.bucket.2 Not sure if this helps or makes it worse. I've also attached the diagnostic files. diagnostics-20181001-1659.zip
October 1, 20187 yr Community Expert You're using the rsync daemon, I'm not experienced with that but your syslog is full of errors like: Oct 1 10:00:23 FFDFS003 rsyncd[17827]: Badly formed boolean in configuration file: "4". Though I don't know if that's a problem and why it's aborting, but if nothing else it's spamming your log, did you try rsync over SSH?
October 1, 20187 yr Community Expert 2 hours ago, qwaven said: So I've tried changing the cache setting to no but this also did not seem to change anything. As I suspected you don't have cache so no effect from that setting. But you should pay attention to it in the future in case you add cache. Also, since the default for that setting is No then you must have set it to Prefer yourself. Most things in the webUI have help text if you just click on the label for the setting. The Use cache setting trips a lot of people up so if you do add cache be sure to check out the FAQ for more details on that setting. The Unraid FAQ thread is pinned near the top of this same subforum. An alternative way you might try transferring your data would be to use Unassigned Devices on the Unraid side to mount your other NAS over the network and work with the files from the Unraid side of things. Might be a bit of a learning curve though.
October 1, 20187 yr Community Expert 1 hour ago, johnnie.black said: did you try rsync over SSH? If you can work at the command line you should be able to do this from either server.
October 2, 20187 yr Author Hi all, Thanks for your comments. I believe the backup software is already doing rsync over ssh. I suppose I can try without as well. I have not used rsync via command line as I was hoping to use the backup software to manage the backup since it incorporates more than just a 1:1 copy. I do notice that my backup has this .hbk on the end. I am wondering if it is actually a single file for transport purpose which can be browsed or something after. I'm not sure how to determine this as it seems like files when I look at it. I'm curious though. With unraid is there no way to just span a single file across multiple disks? Example lets say I had a large file over 4TB in size what would happen? Thanks
October 2, 20187 yr Community Expert 1 minute ago, qwaven said: I believe the backup software is already doing rsync over ssh. It likely using SSH but it's using the rsync daemon module, it's not direct rsync /source /destination. 3 minutes ago, qwaven said: With unraid is there no way to just span a single file across multiple disks? No, since each disk has an individual filesystem, not possible for the same file to cross filesystems.
October 3, 20187 yr Author Hi all, Thanks for your help with this. I've done some more digging and turns out there is a better rsync option within the backup software. This seems to have resolved my issue. /dev/md1 3.7T 1.9T 1.9T 51% /mnt/disk1 /dev/md2 2.8T 736G 2.1T 27% /mnt/disk2 /dev/md3 2.8T 2.9G 2.8T 1% /mnt/disk3 ... Transfer seems to have moved from the first drive onto the next. However seems to have done this about half full. Cheers!
October 3, 20187 yr Community Expert 42 minutes ago, qwaven said: However seems to have done this about half full. That's how high water works, though because of the different disk sizes it will move to disk3 once disk2 is at 33%.
October 4, 20187 yr Author 18 hours ago, johnnie.black said: That's how high water works, though because of the different disk sizes it will move to disk3 once disk2 is at 33%. I assume it will eventually come back to fill the rest? Transfer is going quite a bit slower than I had hoped. Its just about at 5TB transferred now. Going to be a while before it's done. :P Was going at about 45MB/s
October 4, 20187 yr Community Expert Just now, qwaven said: I assume it will eventually come back to fill the rest? Yes: https://wiki.unraid.net/Un-Official_UnRAID_Manual#High_Water 1 minute ago, qwaven said: Transfer is going quite a bit slower than I had hoped. Its just about at 5TB transferred now. Going to be a while before it's done. Was going at about 45MB/s Try enabling turbo write
October 4, 20187 yr Author 4 hours ago, johnnie.black said: Yes: https://wiki.unraid.net/Un-Official_UnRAID_Manual#High_Water Try enabling turbo write I have now enabled this but unfortunately it does not appear to do a ton. Got a little increase but nothing drastic. Except now all the drives are on except just the one required. Will leave it for now and see if it gets better. Anyway thanks for your help.
October 4, 20187 yr Community Expert 20 minutes ago, qwaven said: I have now enabled this but unfortunately it does not appear to do a ton. Got a little increase but nothing drastic. Except now all the drives are on except just the one required. Will leave it for now and see if it gets better. Anyway thanks for your help. Yes, all disks will be spun up, but would should get considerable more speed than normal write mode, 100MB/s+, as long as using relatively recent disks, no controller bottlenecks and copying mostly large files, problem might also be the source...
Archived
This topic is now archived and is closed to further replies.