June 21, 200917 yr Hello! I am getting my new unRAID server ready to begin using as a backup server. I have found that when I use Microsoft SyncToy (http://www.microsoft.com/downloads/details.aspx?FamilyID=c26efa36-98e0-4ee9-a7c5-98d0592d8c52&DisplayLang=en), the photos that are copied do not respect the split level setting that I have used. Below are two paths of some of my folders on the unRAID server. "Photos" is a user share using the "most-free" allocation method. \\tower\Photos\2009\Family\01-01 New Year 0 1 2 3 4 \\tower\Photos\2009\Family\02-01 Birthday 0 1 2 3 4 I have set the split level to 3 so that any folder and its contents under level 3 ("01-01 New Year" and "02-01 Birthday" in this case) are copied on to the same drive. When I use Windows Explorer to copy the folder "Family" from my primary server to unRAID, "01-01 New Year" and "02-01 Birthday" correctly respect the split level setting. So: "01-01 New Year" is created on Disk1 as "\\disk1\Photos\2009\Family\01-01 New Year", and "02-01 Birthday" is created on Disk2 as "\\disk2\Photos\2009\Family\02-01 Birthday" However, when I use SyncToy to copy the "Family" folder, unRAID appears to act as if the split level is set to something very high since unRAID begins to split the contents of "01-01 New Year" and "02-01 Birthday" across all disks so that for example "01-01 New Year\1.jpg" is on disk1 and "01-01 New Year\2.jpg" is on disk2, etc., even though the split level is 3. I tried to lower the split level down to 2 and got the same results with SyncToy. When I lowered the split level down to 1, files were no longer spread across any of my drives as if the split level was set to 0. I am using unRAID Server Pro v4.5-beta6. I don't see any reason why the software I use to copy the files would make a difference since unRAID should be managing what is copied to it. Is this a bug, or am I missing something? Does anyone have any suggestions as to how I can get around this problem? Any explanation of why this occurs would also be appreciated. Thanks.
June 24, 200917 yr Anyone have any ideas about this? If you copy the files using Windows Explorer is it working differently than with synctoy? My guess is that there is an issue with the user share configuration and not something unique about synctoy.
June 24, 200917 yr Author Thanks for responding bjp999! Correct - Explorer is producing a different result than SyncToy. When you say that there "is an issue with the user share configuration", do you mean with the configuration I assigned or with the unRAID software itself? If it is with the configuration I assigned, can you please let me know what setting I need to adjust to correct this? If it is the unRAID software itself, do I need to post this bug somewhere for Tom to add to his Things To Do list? Thanks.
June 24, 200917 yr My thought was it was likely related to the configuration of your user shares (split level, disks included\excluded, etc.). But if Windows Explorer is working as expected, that now seems unlikely. Is it possible that synctoy is configred to copy to a disk share instead of a user share? UnRaid does not know or care what is copying files so it is very unlikely that this is a bug - but I've learned to never say never so we'll have to see how it plays out.
June 25, 200917 yr Author Here are my user share settings: Share name: Photos Allocation method: Most-free Min. free space: (blank) Split level: 3 Included disk(s): (blank) Excluded disk(s): (blank) Export (SMB): export read/write Export (NFS): (blank) I verified that I was indeed copying to a user share (\\tower\Photos) and not a disk share (like \\tower\disk1). Another confirmation of using the user share vs the disk share is that the files in the folders are being separated on to individual disk drives - if I was incorrectly using the disk share, then the files would only show up on the same disk (please correct me if my understanding is wrong here). Do I need to do anything differently than simply keep this thread to notify Tom of the issue? Thanks!
June 25, 200917 yr UnRaid does not know or care what is copying files so it is very unlikely that this is a bug - but I've learned to never say never so we'll have to see how it plays out. I completely agree with this. It is very hard for me to see how this could be a bug, partly because this is the very first time it has ever been reported, and partly because I can't see a connection between app and unRAID. I know that elsewhere some have said that there may be a difference in where a file is created based on how much space it required. (By the way, you have never mentioned how much space is free on these drives, and that is a critical part of the decision process, as to whether to 'split' to another drive or not.) It has been said that if the file is opened with zero bytes and then grows, unRAID may put it on a drive without enough space, because it can't know ahead of time how much will be required; it was also said (something like) that if the file was created with a large size, then unRAID would know to place it on a drive with sufficient room. If that is true, then that is a mechanism whereby different apps might cause unRAID to place files differently, based on how they 'created' the file. But I personally can't confirm that such a thing actually happens, because I have never heard of a file create command that included a file size. Of course, that does not mean it does not exist, especially since I know nothing about Samba, and perhaps a network aware program can use some special Samba call to open/create a file handle with a specified file size. If it does not exist, then the only way that could happen is that unRAID creates the file handle on the first drive, then if a subsequent file seeks/extends to a huge value, beyond the usable drive space (disk size minus configured free space), unRAID closes the handle and re-opens it on another drive, and then executes the file size expansion, something I have never heard of unRAID doing, but is not impossible. Another point to make here, once a second drive contains an unwanted copy of the folder name, then where a file is placed is *undefined* currently! That is, if the split level would normally inhibit creating "01-01 New Year" on multiple drives, but one has been created any way (or exists for any reason), then it is not necessarily a mistake if unRAID saves files to both. Technically, I believe the Split Level setting is designed to govern the decision whether to use an existing folder or create a new one on another drive, not to govern where files are saved if there are multiple *existing* folders (whether they should be there or not). So for your testing of apps, you would need to make sure that after a test, you consolidate all of the files in one folder, and remove all other copies of that folder. Also, have you tested Explorer *after* SyncToy, or only before it? Would you mind reporting your free space on the relevant drives, then do a little more testing? Consolidate to one folder, then try Explorer, note the results, then consolidate again (deleting added files so same disk space and folder structure), then try SyncToy, note the results, then consolidate and restore same setup, then try Explorer again ... And you might want to try these tests also with Split Level = 2, as well as 3. By the way, one other possibility, suggested very smartly by another in a different thread, is that there may be file system corruption, that affects the true disk space free. Try running reiserfsck on these data drives (NOT the parity drive!). See Check Disk File systems.
June 26, 200917 yr Author Thanks for responding RobJ. My config is slightly different than it was when I originally experienced the problem: 1. I have added a new hard drive 2. I have copied data to it using Explorer (I needed to clear up some space on one of my other servers). When I originally experienced the problem, I had 3 data drives + 1 parity drive. All drives are 1TB Samsung drives. No data was on the server yet. So, I basically had all of the space available (close to 3TB). When I did the original copy using Synctoy, none of the discs contained the "2009" folder. So, the problem you mentioned that can occur when a folder already exists didn't apply in my case. Yes, I copied using Explorer after the Synctoy operation had completed and I don't believe that I had cleared the files that Synctoy had copied. You may be on to something here with this. I just tested using SyncToy now (with data on the drives) and the SyncToy operation correctly did not split the contents "01-01 New Year". I'm wondering if the problem is due to starting with empty drives. Let me find some space to move the files from the unRAID server so I can delete everything and start from scratch to reproduce the results so we can troubleshoot from there. I'll get back to you on this. Thanks.
July 8, 200916 yr Author Hello again! Apologies for the delay - I had a family emergency. I removed the 2 new drives so that the server now has 3 data drives + 1 data drive (all Samsung HD103UJ 1TB drives). I ran reiserfsck on all three data drives and it reported that no corruptions were found on any of the drives. I tried running SyncToy when the drives were empty and also when each of the 3 drives had a few files on them - the files still were not split correctly. I ran the tests you requested and have posted the results below. I have also attached a file listing of the original files and the files that appear after I copy them using SyncToy and manually using Explorer so you can more easily see the problem. 1. Empty disks size free disk1 ata-SAMSUNG_HD103UJ_S13PJ1CS301518 976,762,552 976,699,896 disk2 ata-SAMSUNG_HD103UJ_S13PJ1CS302055 976,762,552 976,699,896 disk3 ata-SAMSUNG_HD103UJ_S13PJ1CS301519 976,762,552 976,699,896 2. "Photos" user share created - Empty "Photos" folder created on Disk1 3. Files/Folders being copied to server File listing exported to "Original.txt" 4. After manual copy using explorer (files correctly "split") File listing exported to "ExplorerManual.txt" size free disk1 ata-SAMSUNG_HD103UJ_S13PJ1CS301518 976,762,552 976,554,120 disk2 ata-SAMSUNG_HD103UJ_S13PJ1CS302055 976,762,552 976,698,212 disk3 ata-SAMSUNG_HD103UJ_S13PJ1CS301519 976,762,552 976,699,896 4. Deleted all files and folders from all individual disks (same as #1 - Empty Disks) size free disk1 ata-SAMSUNG_HD103UJ_S13PJ1CS301518 976,762,552 976,699,896 disk2 ata-SAMSUNG_HD103UJ_S13PJ1CS302055 976,762,552 976,699,896 disk3 ata-SAMSUNG_HD103UJ_S13PJ1CS301519 976,762,552 976,699,896 5. Removed and recreated "Photos" user share - Empty "Photos" folder created on Disk1 6. After Synctoy operation (files are split incorrectly) File listing exported to "SyncToy.txt" size free disk1 ata-SAMSUNG_HD103UJ_S13PJ1CS301518 976,762,552 976,651,288 disk2 ata-SAMSUNG_HD103UJ_S13PJ1CS302055 976,762,552 976,649,748 disk3 ata-SAMSUNG_HD103UJ_S13PJ1CS301519 976,762,552 976,651,180 Please let me know if you need any additional information. Thanks!
July 14, 200916 yr That's interesting! It appears to me as if Explorer is using a strictly sequential copy thread, using a list of the files based on the same directory sorting you have set for display. SyncToy is using its own list, possibly a hashed list, of the files to copy, and it's my guess that it is using a multi-threaded scheme. Whereas Explorer is doing one file operation, awaiting completion and status, then the next file operation, it appears to me as if SyncToy had simultaneous file operations going. Just speculation here, but I suspect there is a hole in the logic, that when 2 near simultaneous requests to create a file arrive, and unRAID has not yet created the lower level folders that would constrain files by the Split Level setting, then possibly each is getting shunted to a different drive, since you are using Most Free, which is the mode most likely to spread the copies evenly to all available drives. Explorer asked unRAID to create the first file and its path, which set up Sub1 on Disk 1, and saved the file there, and subsequently saved all of the Sub1 files there. SyncToy may have had at least 3 threads asking unRAID to create the Sub1 path, and initially since it did not exist elsewhere, and it was Most Free mode, each thread caused a Sub1 path to be created on all 3 data drives. From then on, all 3 drives were legitimate targets for the files. I suspect Tom never foresaw something like this happening. If you are sure you want Most Free, then I would drop the Split Level to 2, AND pre-create Photos/2002/Sub1/Name on Disk1, prior to the SyncToy copies. Try testing that, and variations of that. For the record, with Split Level perhaps one too high, SyncToy and unRAID did an amazing job of evenly spreading the photos across the 3 drives, about 50MB each. With each photo save, unRAID in Most Free mode kept choosing the next drive, Disk1 then Disk2 then Disk3 then Disk1 then Disk2 then Disk3 ...
July 16, 200916 yr Author RobJ: Thanks for responding. The multiple thread theory makes sense. I tried what you asked and pre-created Photos/2002/Sub1/Name on Disk1 with split level=2. But, this still didn't work. I also tried to pre-create with split level=3 (what I was using) and this still didn't work. Below is the reason I am interested in using the Most Free method: The photos are categorized as follows: |--Photos (root folder) |--Year (like 2008, 2009) |--Main Category (like Birthday, Family) |--Date and name (like 01-01 John Doe) From my reading of unRAID, I can lose any 1 of the drives and successfully recover. However, if I lose two of the drives, then at least one disk of information is lost. To minimize the loss of a chunk of a large span of time of photos, I opted for Most-Free. This is because if I selected High Water, then I may potentially have four months or so of photos on one drive until the High Water point is reached and then the next disk is used. In the event of losing two or more disks, I would lose about four months of photos. On the other hand, with the Most Free method, by selecting a split level of 3, each event (like 01-01 John Doe) is all placed on one disk, and the next event is placed on the next disk, etc (assuming each new event is about the same size). In this case, in the event of losing two or more disks, I would potentially only lose a few events in a given timeframe, and even fewer if the disks were not that full. Also, I get decent performance since if I am browsing an event's photos, they are all on one drive (due to the split level) and hence I do not need to spin up other drives to read the photos and prevent stuttering. If you have any suggestions to accomplish this goal, I would appreciate anyone's input. I'll try another sync tool to see if I can avoid this problem. In any case, I recently discovered that SyncToy 2.0 has file corruption issues with NAS devices. I don't know whether unRAID is affected (the photos I copied using SyncToy for these tests were not corrupted), but combined with the split level problem, I will need to consider a different solution. Here are some links to the SyncToy/NAS issues in case you are interested: http://social.microsoft.com/forums/en-US/synctoy/thread/8cd6481c-1e96-43aa-aca6-ee9f95a6b50f http://www.microsoft.com/downloads/details.aspx?FamilyID=c26efa36-98e0-4ee9-a7c5-98d0592d8c52&DisplayLang=en (read the first section in the "Overview").
July 16, 200916 yr RobJ: Thanks for responding. The multiple thread theory makes sense. I tried what you asked and pre-created Photos/2002/Sub1/Name on Disk1 with split level=2. But, this still didn't work. I also tried to pre-create with split level=3 (what I was using) and this still didn't work. Below is the reason I am interested in using the Most Free method: The photos are categorized as follows: |--Photos (root folder) |--Year (like 2008, 2009) |--Main Category (like Birthday, Family) |--Date and name (like 01-01 John Doe) From my reading of unRAID, I can lose any 1 of the drives and successfully recover. However, if I lose two of the drives, then at least one disk of information is lost. To minimize the loss of a chunk of a large span of time of photos, I opted for Most-Free. This is because if I selected High Water, then I may potentially have four months or so of photos on one drive until the High Water point is reached and then the next disk is used. In the event of losing two or more disks, I would lose about four months of photos. On the other hand, with the Most Free method, by selecting a split level of 3, each event (like 01-01 John Doe) is all placed on one disk, and the next event is placed on the next disk, etc (assuming each new event is about the same size). In this case, in the event of losing two or more disks, I would potentially only lose a few events in a given timeframe, and even fewer if the disks were not that full. Also, I get decent performance since if I am browsing an event's photos, they are all on one drive (due to the split level) and hence I do not need to spin up other drives to read the photos and prevent stuttering. If you have any suggestions to accomplish this goal, I would appreciate anyone's input. I'll try another sync tool to see if I can avoid this problem. In any case, I recently discovered that SyncToy 2.0 has file corruption issues with NAS devices. I don't know whether unRAID is affected (the photos I copied using SyncToy for these tests were not corrupted), but combined with the split level problem, I will need to consider a different solution. Here are some links to the SyncToy/NAS issues in case you are interested: http://social.microsoft.com/forums/en-US/synctoy/thread/8cd6481c-1e96-43aa-aca6-ee9f95a6b50f http://www.microsoft.com/downloads/details.aspx?FamilyID=c26efa36-98e0-4ee9-a7c5-98d0592d8c52&DisplayLang=en (read the first section in the "Overview"). Personally, until Microsoft fixes SyncToy, I'd not rely on it for the file copy. It sounds like they have a serious bug... and are tied up in red-tape in their ability to provide a fix. As far as split level, you can always copy the files to the /mnt/diskX disk and control exactly which one then end up on. Lastly, having the photos spread across two disks in an alternate month manner might help to prevent data loss, but FAR more important is the ability to identify and quickly replace a failed drive. If the files are critical to you, you should A. Perform periodic parity checks to ensure your drives are all working as expected. B. Install some form of e-mail alert if your array fails in any way. C. Keep a spare disk on hand, to act as a replacement when one in the array fails. (Don't worry, one will, eventually, it is only a matter of time. The odds of a second concurrent failure increase the longer the first failed drive remains in the array. If you can replace the first failed drive faster, the odds of a second concurrent failure are smaller.) D. Install a UPS. E. Run smartctl reports on your drives regularly. This might alert you to a drive with problems before it fails entirely. F. Read this post: (10 ways to lose data with unRAID) http://lime-technology.com/forum/index.php?topic=2601.msg21033#msg21033 G. There is an 11th way to lose data, it is thinking that unRAID is the same as a "backup" of your data. It is not. All it would take is a single lightning strike/tornado/fire/flood and your server and the drives in it might all be made worthless. The ONLY backup is one physically separate from the unRAID server. Off-site is best. With 1.5 and 2TB drives available today, I'd use a USB style docking adapter, and copy critical files to it periodically, then store it elsewhere.. Or, if you have a DVD writer on your PC, copy critical files to a DVD, store it elsewhere... (A bank safe--deposit box can hold a LOT of dvd's, or probably a few hard-disks) H. Make sure you back up your data regularly. Backup copies are useless if not made. Joe L.
July 16, 200916 yr Author Thanks for tips I'll have to work on implementing those in the coming weeks. Thanks again!
Archived
This topic is now archived and is closed to further replies.