luca Posted May 17, 2015 Share Posted May 17, 2015 I've started moving my backup server to XFS, following to the letter bjp999's instructions on the first page. I'm on step 7, and the first disk is being copied to the destination, and I noticed something odd. While most of the file are getting copied to the t folder, a few very recent files (along with new files that are being created) are going to a separate folder on the root: /mnt/disk21/t/ <most of the files are going here> /mnt/disk21/folder <a few recent files are going here> folder is a hidden share Also odd, a /mnt/disk21/t/folder has been created, but Windows tells me I have no rights to access it when I try to browse the content. Any ideas? disk21 is included in the folder share and something is writing to the share. I don't understand. The source drive is disk2. Shouldn't anything on disk2 have ended up on /mnt/disk21/t ? Now that cp is done, I can access /mnt/disk21/folder from Windows, all the files appear to be there. Do you think I can move everything to the root of disk21? Quote Link to comment
trurl Posted May 17, 2015 Share Posted May 17, 2015 I've started moving my backup server to XFS, following to the letter bjp999's instructions on the first page. I'm on step 7, and the first disk is being copied to the destination, and I noticed something odd. While most of the file are getting copied to the t folder, a few very recent files (along with new files that are being created) are going to a separate folder on the root: /mnt/disk21/t/ <most of the files are going here> /mnt/disk21/folder <a few recent files are going here> folder is a hidden share Also odd, a /mnt/disk21/t/folder has been created, but Windows tells me I have no rights to access it when I try to browse the content. Any ideas? disk21 is included in the folder share and something is writing to the share. I don't understand. The source drive is disk2. Shouldn't anything on disk2 have ended up on /mnt/disk21/t ? Now that cp is done, I can access /mnt/disk21/folder from Windows, all the files appear to be there. Do you think I can move everything to the root of disk21? By something, I don't mean you copying from disk2 to disk21. I mean an app or something writing to the share and it ending up on disk21 because disk21 is part of the share. Quote Link to comment
JohnO Posted May 18, 2015 Share Posted May 18, 2015 Thanks to all the help, I replaced two drives and re-formatted my array to XFS successfully! John Quote Link to comment
luca Posted May 20, 2015 Share Posted May 20, 2015 By something, I don't mean you copying from disk2 to disk21. I mean an app or something writing to the share and it ending up on disk21 because disk21 is part of the share. Right, I should have figured that out. 3 disks done (out of 11), one in progress. I'm averaging 36 MB/s with the copying so far. Quote Link to comment
luca Posted May 22, 2015 Share Posted May 22, 2015 I am planning on removing one of the smallest HDs (disk6) from the array, once the change from RFS to XFS is completed. To this end I kept disk6 as the last disk to get copied. Is the best way to accomplish removing a drive, short of rebuilding parity (stop the array, un-assign the drive, then run tools / new config)? Should I delete all data from disk6 before un-assigning. would that make any difference? Quote Link to comment
trurl Posted May 22, 2015 Share Posted May 22, 2015 I am planning on removing one of the smallest HDs (disk6) from the array, once the change from RFS to XFS is completed. To this end I kept disk6 as the last disk to get copied. Is the best way to accomplish removing a drive, short of rebuilding parity (stop the array, un-assign the drive, then run tools / new config)? Should I delete all data from disk6 before un-assigning. would that make any difference? Deleting data will make no difference. You will still have to rebuild parity. The only way to accomplish this without rebuilding parity is to zero the disk while still in the array. Then you can do new config with trust parity. There has been a few threads about this. However, it will require using the command line and will be very unforgiving if you make a mistake. Quote Link to comment
JohnO Posted May 22, 2015 Share Posted May 22, 2015 Deleting data will make no difference. You will still have to rebuild parity. This is the part that took me a few days to get through my head. I had come up with great ways to empty a disk, thinking it would save time, but it really doesn't fit. Thankfully, the steps can be separated, and many can be (have to be) completed with the array up and available, so there is no reason to force it into a single outage window. Quote Link to comment
JonathanM Posted May 22, 2015 Share Posted May 22, 2015 Deleting data will make no difference. You will still have to rebuild parity. This is the part that took me a few days to get through my head. I had come up with great ways to empty a disk, thinking it would save time, but it really doesn't fit. Thankfully, the steps can be separated, and many can be (have to be) completed with the array up and available, so there is no reason to force it into a single outage window. The server is still available while parity is being built. It may be a little slower, but still usable. The only time the server would be completely down for an extended period of time is if you add a new drive that hasn't been precleared, and the array is offline until the clearing process is complete. Quote Link to comment
pacload Posted May 26, 2015 Share Posted May 26, 2015 Could one simply copy paste files from windows or should I specifically do it from a screen session with line commands? Thank You Quote Link to comment
trurl Posted May 26, 2015 Share Posted May 26, 2015 Could one simply copy paste files from windows or should I specifically do it from a screen session with line commands? Thank You However you do it make sure you only use the disks and not the user shares. If you want to use Windows some have recommended Teracopy. I used mc in a screen session. Many have done it from the command line in various ways. There is a docker named unBalance that might be useful. Read the thread for more ideas. Quote Link to comment
pacload Posted May 26, 2015 Share Posted May 26, 2015 Thanks for the suggestions trurl, i'm quite liking the Teracopy idea and I'll give it a try Quote Link to comment
garycase Posted May 26, 2015 Share Posted May 26, 2015 +1 for TeraCopy Used it for years, and really like both the interface and the features. I always use the automatic verification -- almost doubles the time to copy, but ensure the copies are good. And unlike Windows Explorer, if it encounters errors, you'll see exactly which files had problems AND the copy doesn't abort -- the rest of the data is still copied. Quote Link to comment
Hoopster Posted June 6, 2015 Share Posted June 6, 2015 OK, I have read through the instructions several times and it all makes perfect sense. I have five data disks in my array and a parity disk. Disk 5 was recently added (formatted as RFS) and is empty. All five data disks are 3TB WD Reds. I have formatted Disk 5 (it was empty as I had not added it to a user share yet) with XFS and am copying the contents of Disk 1 to Disk 5 as per instructions. My planned migration is like this since all disks are identical in size: Disk 1 ---> Disk 5 Format disk 1 XFS Disk 2 --> Disk 1 Format disk 2 XFS Disk 3 --> Disk 2 Format disk 3 XFS Disk 4 --> Disk 3 Format Disk 4 XFS Disk 4 then becomes the extra disk I can add to user shares as needed. My question is what does this do to user shares where I have the current disk 1 ... disk 4 assigned to the shares? Do I need to unassign the disks and reassign the "new Disk 1 (Disk 5)" as Disk 1, etc. after the changes? As you can see, I am not 100% certain about how physical disks, array disk assignments and user shares that use these disk are all related after making these changes. Of course, initial array configuration and share creation all makes perfect sense, but, I am unclear on what happens when contents are moved from one disk to another after changing file system. Quote Link to comment
itimpi Posted June 6, 2015 Share Posted June 6, 2015 If you have allowed the User Shares to use all disks, then you do not have to do anything else. If you have restricted and shares to only use specific disks, then afterwards you should change the share settings to match your new layout. Quote Link to comment
Hoopster Posted June 6, 2015 Share Posted June 6, 2015 If you have allowed the User Shares to use all disks, then you do not have to do anything else. If you have restricted and shares to only use specific disks, then afterwards you should change the share settings to match your new layout. OK, thanks. I have several shares restricted to certain disks and one that uses all current data disks. For example, I have a Pictures share that uses all the current data disks (disk1 - disk4). After the changes, would I need to set that to disk1, disk2, disk3, disk 5? I have a Videos share that uses disk2, disk3, disk4 so that would require a change to disk1, disk2, disk3, correct? And so on with all shares where the contents have moved to different physical disks? EDIT: Perhaps the easiest thing to do would be to change all shares to use all disks (at least the larger ones that already span multiple disks). Quote Link to comment
RobJ Posted June 6, 2015 Share Posted June 6, 2015 There's one other option, do a New Config when you are all done converting, and re-assign all drives to their original slots, then you can either rebuild parity (optionally without the empty drive), or mark it as already valid (but good to run a parity check anyway). The actual data will be on physically different drives, but the same disk numbers, so share configuration will still be correct. Your choice, but the quickest would be what you added by your edit. Quote Link to comment
SSD Posted June 7, 2015 Share Posted June 7, 2015 It's important to understand that user shares only loosely obey their configurations. For example, if you had a share using disks 1-3, and during migration those drives became disk5, disk1, disk2 (without changing the setting), the files on disk5 would continue to be shown as part of the share, but would not receive any new files, disk3 could start receiving new files, and disks 1 and 2 would work as expected. Quote Link to comment
Just Me Posted June 11, 2015 Share Posted June 11, 2015 Hey! Okay, it seems I'm too stupid for this I've tried to copy the content from disk1 to the new disk10. I've used this command to copy all files: cp -rpv /mnt/disk1/* /mnt/disk10/t This seemed to work, all files are in /disk10/t Now I want to check if all files are okay so I type this command in the command line: rsync -nrcv /mnt/disk1 /mnt/disk10/t >/boot/verify.txt But all it does was a lot of reading on disk1, no reading on disk10 so disk10 went to sleep after a while. After the process has finished I have a verify.txt file. It seems to contain a list of all files that are an disk1. e.g. sending incremental file list disk1/ disk1/Dokus/ disk1/Dokus/SDTV/ disk1/Dokus/SDTV/Ice Road Truckers/ disk1/Dokus/SDTV/Ice Road Truckers/Season 1/ disk1/Dokus/SDTV/Ice Road Truckers/Season 1/Ice Road Truckers - Season1 Disc1.dvd disk1/Dokus/SDTV/Ice Road Truckers/Season 1/Ice Road Truckers - Season1 Disc1.iso . . . sent 111,485 bytes received 8,188 bytes 5.49 bytes/sec total size is 2,964,339,119,913 speedup is 24,770,325.14 (DRY RUN) What I'm doing wrong? I thought the verify.txt should only contain files that are not okay but rsync never read any files an disk10. I'm confused, any help? Quote Link to comment
Freddie Posted June 11, 2015 Share Posted June 11, 2015 Now I want to check if all files are okay so I type this command in the command line: rsync -nrcv /mnt/disk1 /mnt/disk10/t >/boot/verify.txt But all it does was a lot of reading on disk1, no reading on disk10 so disk10 went to sleep after a while. After the process has finished I have a verify.txt file. It seems to contain a list of all files that are an disk1. A single slash can make a big difference. It should be: rsync -nrcv /mnt/disk1/ /mnt/disk10/t >/boot/verify.txt Without the slash, rsync would copy the directory. With the slash rsync would copy the contents of the directory. @bjp999: Perhaps your instruction post should be edited to include the slash: (edited) From a screen session, run this command: rsync -nrcv /mnt/[source]/ /mnt[dest]/t >/boot/verify.txt Quote Link to comment
Just Me Posted June 11, 2015 Share Posted June 11, 2015 Okay I've tried rsync -nrcv /mnt/disk1/ /mnt/disk10/t >/boot/verify.txt but rsync still only reads disk1, disk10 ist still spun down. So I guess it is not working either? Quote Link to comment
Hoopster Posted June 12, 2015 Share Posted June 12, 2015 I recently went through the reiserfs ---> xfs transition on my array as well. I got the same results you did, just a list of the contents of the first disk in verify.txt. To "verify" the contents, I ended up doing a directory compare on each disk from Windows to verify that the number of files and size counts were exactly equal. I also spot checked several files in each directory by comparing them, opening them (pictures, videos, movies and TV shows). The array has been in use for several days now since the transition and I have not yet encountered a problem file. Quote Link to comment
Just Me Posted June 13, 2015 Share Posted June 13, 2015 Okay I've tried rsync -nrcv /mnt/disk1/ /mnt/disk10/t >/boot/verify.txt but rsync still only reads disk1, disk10 ist still spun down. So I guess it is not working either? I was wrong, this works. Rsync just reads the whole source disk first, which takes hours, and then starts to compare the checksum with the destination disk. So it seems everything is fine. Thank you Freddie for your help Quote Link to comment
spuddo Posted June 19, 2015 Share Posted June 19, 2015 Hi thanks for this. So easy to follow. I am about to start converting. Could you please help me understand the necessity for creating the "T" folder if I am copying from Disk to Disk. I do have user shares? Regards Quote Link to comment
SSD Posted June 19, 2015 Share Posted June 19, 2015 Hi thanks for this. So easy to follow. I am about to start converting. Could you please help me understand the necessity for creating the "T" folder if I am copying from Disk to Disk. I do have user shares? Regards I do not think it is necessary. As explained, I did it to try to avoid unRAID detecting a very large number of duplicate files in user shares. Creating the "t" directory is painless, and many users have followed the guide, you might just want to do it the same way. But it is up to you. Quote Link to comment
RobJ Posted June 20, 2015 Share Posted June 20, 2015 Hi thanks for this. So easy to follow. I am about to start converting. Could you please help me understand the necessity for creating the "T" folder if I am copying from Disk to Disk. I do have user shares? I do not think it is necessary. As explained, I did it to try to avoid unRAID detecting a very large number of duplicate files in user shares. Creating the "t" directory is painless, and many users have followed the guide, you might just want to do it the same way. But it is up to you. Which reminds me, I've wanted to ask, has *anyone* done it without using the extra T folder, and did you have any problems? Did file duplicates appear, and if they did, were they a serious problem, or just a temporary nuisance? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.