tunetyme

I want to be sure that I get this one right the first time. S is my video parent directory on all video disks In MC Left Panel (physical disk) /mnt/disk3/disk3 highlight S Right Panel /mnt/user0/disk3/S Move from the rigth panel to the left panel?? I need to clarify this, as I read your directions it appears that I go to the physical disk on both left and right panels and move it to itself /mnt/disk3/disk3/S to /mnt/disk3/disk3. I would have thought that I would move files from /mnt/user0/disk3/S to /mnt/disk3/disk3 which I would need to swap panels to move file from left to right.

I have a lot to learn. It is apparent that I need to learn all the different ways to skin this cat. RE: disk3 share I opened mc and there it is under User 0 is disk3 share. To delete, it I will need to delete the entire disk and restore from a backup. Is there and easier way?

I did preclear the disk because I could not get the format button to appear with the rfs drive after it was copied. As I have said I don't mind preclearing 6 year old drives to make sure they are healthy and still have reliable life in them. I know that formating does not write zeros to the drive but perhaps it should.

I think it is implied when you format the disk. In effect it is all zeros and a new previous prepared drive is also all zeros. ???? And the difference is? Back to one of my original questions is you "can you upgrade drives (plural) and convert to xfs." I have said many times that, if I can't maintain parity, it is no big deal. I copy an old drive to a new drive. Repeat until the new drives are install and each one has the correct files on each disk and the are in their designated slot. Format the old drives to xfs and repeat the process until all the rfs drives have been replaced . Build parity. Done. Copying should take a lot less time since I am not updating parity. As I have asked before is maintaining parity "mission critical" for all users. If you don't have a backup, maybe. Storage is cheap and Icy Dock 4 in 3 are inexpensive and makes life real easy. What else are you going to do with the drives after you do your upgrades that still have life in them? Buy a few more on eBay. Now perhaps you understand why I thought the process was convoluted. What you are saying is you can only introduce one new drive. Copy format repeat reusing the drive rfs drive and going through your whole array. My method would be preclear all the new drives and format them xfs. Let's use 2 new 4TB drives. Unassign parity drive. Copy one of the old 2TB drives to a new 4TB drive, repeat for second drive. Swap places for the drives and format or preclear and format the drives if you want make sure they are still reliable and repeat the process of copying until you have worked through your whole array. Build parity. Done. I guess that's about 4 cents worth.

If you follow through the process, after you swap the new drive with all the files copied on it then you format the old RFS drive as XFS (formatting would destroy parity) or you could put another drive in that is already formated XFS (Like a larger drive) and repeat the process. Read the last several steps.

Nope I got those steps right. It was the warning that the parity drive would be erased. That is what started this discussion. See page 24 5th post down

This is the procedure that I have been doing: File System Conversion If you have not run a successful Parity Check recently, do so now. You want to be certain that the array is perfect before you start Prepare a strategy for the order of drive conversion. Because you can't replace a larger drive with a smaller drive (unless the total file space used will fit on the smaller drive), you will have to order the conversions so that your largest data drive is first, then the next largest, then the next, with the smallest data drive being last. Obviously, the order doesn't matter for drives that are the same size. If your empty swap drive is not already installed and assigned, install it, and with the array stopped, assign it to the next empty drive slot (for our example, we will assign it to Disk 11) Click on the disk name of your swap drive (e.g. 'Disk 11') and change the format to XFS if it isn't already, then click Apply and Done If you have enabled User Shares (and most users have), go to Settings -> Global Share Settings and add your swap disk to ' Excluded disk(s) ' (for our example, we would put disk11) Start the array; your empty swap drive should show as 'Unmountable', and a Format button will be present Click the check box for formatting, then click the Format button; it takes a few minutes, says it's formatting; when done, array should show an additional drive, almost completely empty, formatted with XFS At the console or within a screen session, copy all data from your drive to be converted to the empty swap drive; use an rsync command based on the following, except change the drive numbers as appropriate for your system; type it exactly with the same slashes, upper and lower case matter; this command will take a long time but parity will be fully preserved; when complete, prompt should return with no errors showing; your array now has 2 drives that are identical except for their format, their file system (one of them is excluded from shares) rsync -avPX /mnt/disk10/ /mnt/disk11/ (in our example, we are copying our large disk10 to the new and empty swap drive) This step is optional, as the previous rsync automatically checksums each transfer. But if you would like to verify that the end-to-end transfer was perfect, perform the next rsync command below; it will take a long time, and probably nothing will be copied unless the drive has been updated (see warning below!) since the full copy above; there's no progress info, it's over when the prompt returns rsync -rcvPX /mnt/disk10/ /mnt/disk11/ Stop the array; we are now going to swap the drive assignments Click on Tools, then New Config, then Retain current configuration:, then select All, then check Yes I want to do this, click Apply then Done Important Warning! Doing a New Config will reset the file system type for all disks to Auto! While usually this is not a problem, especially with the latest unRAID, in some circumstances this can lead to unmountable disk(s). If that happens, then you need to select the correct file system for those disk(s). If in doubt, ask for help! Go back to the Main page and click on the dropdown for the swap drive (e.g. Disk 11) and unassign it (click on "unassigned" or "no device") Click on the dropdown for the other drive (the one being converted, e.g. Disk 10 to start), and reassign it as the physical drive of the swap drive, the drive that was empty (e.g Disk 11) Click on the dropdown for the slot of the swap drive (e.g. Disk 11) and reassign it to the physical drive that was being converted (e.g. Disk 10) Important! Click on each drive name (e.g. Disk 10 and Disk 11) and swap the file system format of the drive - if it's ReiserFS change it to XFS, if it's XFS change it to ReiserFS; it's important to swap the disk formats as well as the physical drive assignments At this point, you have now swapped the 2 drives, which is fine as they are identical (except for file system format); parity remains valid because the same drives are assigned, their slot does not matter; however if you have a second parity drive, it's now invalid! You should see all array disks with a blue icon, a warning that the parity disk will be erased, and a check box for Parity is already valid; IMPORTANT! click the check box, make sure it's checked to indicate that Parity is already valid or your Parity disk will be rebuilt! then click the Start button to start the array; it should start up without issue and look almost identical to what it looked like before the swap, with no parity check needed; however the XFS disk is now online and its files are now being shared as they normally would; check it all if you are in doubt If you are sure it's all fine, stop the array and click the empty swap disk slot (e.g. still Disk 11), and change the format to XFS, then click Apply and Done Start the array; the Format button should be available, format it now; when done, your empty disk slot now has a fresh and empty disk formatted with XFS and ready to fill again; your data drive has completed the conversion process and is already back online, with all files and shares intact, but formatted with XFS You are now ready to convert the next drive, so circle back to Step 8 and repeat these steps (Step 8 through Step 18), substituting your next drive to be converted; the empty and excluded swap disk slot will always be the same (e.g. always Disk 11 in our example), the other will change as you convert different data drives When done, you have an empty XFS drive appended to your system, probably your smallest drive, and still excluded. It's up to you what you want to do with it. You can leave it as is, or you can unassign it and rebuild parity, or you can use the parity preserving remove-a-drive procedure, instructions elsewhere. Remember, it's probably still globally excluded from shares. I do recommend that if you are going to try this procedure, you read through the steps and notes carefully until you fully understand them, and understand the importance of each detail. Missing a step or typing the wrong disk number could be disastrous! If you wish, you can perform parity checks at any point during and after. I don't believe they are necessary, I only did one before starting, and I believe I did another after it was all done. Warning! If you run the verification copy in Step 9, and it actually copies files, then it is likely you have a process still changing the drive! These newly copied files were not there for the Step 8 copy! You need to determine what process (Docker, plugin, VM, an external backup, or the Mover) made the changes to this drive, and stop it. Then you may need to run Step 9 again, because the process may have made even more changes to folders, after the Step 9 rsync process had moved past those folders. In summary, if the Step 9 copy actually copies any files, then you should probably repeat Step 9 until nothing is copied. I've checked the above pretty carefully, if you see any errors, PLEASE let me know ASAP! I'm sure it can be improved. Steps 17 and 18 are a repeat of 4, 6, and 7, but it seemed safer to do it this way. Please let us know of any issues or suggestions! ____________________________________________________________________________________________________________________________________________________________ Step 8 Everything was going according to the the above method until I missed a /. rsync -avPX /mnt/disk10/ /mnt/disk11/ this is the proper command I missed the / slash after disk10. This created another share someplace that we can't find. I did steps 10 -16 where I became confused about parity. RobJ has updated this section as above to help with the confusion (see previous 2 pages). Step 16 was the start of the confusion about parity.

I may need to dig a little deeper to comprehend this. Still haven't been able to solve the disk3 share issue does anyone have a way for me to address this?

I do understand how parity itself works and I understand how it works across drives basically odd or even. What I was trying to express is once I have made a duplicate set of files (rsync) on two different size disks with different formats (RFS v XFS) I am now able to remove the old disk and replace it with the new disc (same slot) and parity is still valid. By removing the old disk did I just remove one of the bits being counted? By swapping the new disk with the old and removing the old disk (changing slots and removing the old disk completely) how is parity maintained? I have just swapped and removed one disks that was being counted in the odd even for each bit of data. I used to deal with parity issues when dealing with serial and parallel data communications. As I have tried to explain I tend to think in terms of an absolute address on each disk where a bit is counted as odd or even. any changes in the bit being counted or the quantity of bits being counted (number of disks) changes parity. As I have said I need to think about what relative addressing means.

I think in terms of absolute addresses not relative. I need to think about that for awhile. I guess I need to throw some rocks in the pond and contemplate relativity along with my naval. I guess I'll go hitch up the buggy and go to town... So rsync doesn't do anything more than copy files. I had the impression that it does much more on a lower level.

OK, let me see if I can explain my dilemma in a different way. Using bits At address x on a 2TB drive RFS (not that the format is essential) =1 and address y = 0 I use rsync to copy the data to a 4TB drive xfs I am assuming that both formats use the same address scheme you are saying that address x = 1 and address y = 0 I guess I am thinking like Windows where if I copy a file from drive a to drive b it also defragments the file when it is copied.. Are files ever fragmented under Linux?

there is a user named disk3

When I compute free space for disk3 it shows 10.9TB ????

index of /mnt/user/disk 3 Type Name Size Location Last Modified Parent Directory S Disk 3 2017-01-22 15:05 1 object: 1 directory, 0 files

Not on Cache, nor disk3 I've looked down 3 levels When I look under share info under include it says all and exclude says none. I can't change these.

Any ideas on correcting the disk3 share? It shows up in shares but I can not find it on any disk in my system. The data does not appear in my primary video Share all the other disks do.

Are you saying when you use rsync and copy a file drive one (2TB rfs to 4TB xfs) to drive two that each bit ends up at the exact same address? If so, that's amazing and who ever wrote that command didn't have a life for quit awhile. Again, if you are old school like me then "show me". That's why If I do verify parity before starting the process and then substitute the 4TB drive for the 2TB drive in the array I would want to see that parity is still valid. If I do it one time then I will trust it. Remember I started back in the day of punch cards,then Decwriters (tons of greenbar) and finally CRT's. We were taught to be skeptical...GIGO (garbage in garbage out). I remember when they were trying to get the code rock solid for basic things like a keyboard. It may be that kids these days don't have to contend with this anymore but this was my training. Still think the step 11 when newconfig turns format type back to auto is a mistake. If I unassign the drive, I can quickly see the format type so I am confident the info is there. Auto was most likely a short cut.

Those of us who are old school think more along the lines of a parity bit used in each byte (9th bit) or data communications. I will studied the references. I didn't read them before because I thought I already understood how parity works.

I think most users think parity is static in other words when I think of data stored on my drive it is 8 bits which is a group of ones and zeros These are at a very particular location on my drive when I make a copy it does not necessarily land at that exact same spot on the new drive it could be in any other sector. So without testing and seeing that is at the same exact location then there is a leap of faith that you are asking for that I have a hard time making. That's why I am suggesting that there is a method of doing a verify option to prove to the new user that it in fact works.

I think you missed my point...The fact that you have two different size drives and different format the question in my mind is how can parity be valid when I swap out a 2TB RFS drive with 4TB xfs drive. I know it works out now but at the time it was a strech to comprehend this.

Ahhhh I didn't see that slash. I've checked disk7 (now it is disk3) and no folder called disk3 The old disk 3 has been precleared and formated xfs

Trurl I have checked every disk on my system (can't see parity) there is no file folder named disk3. The rsync command I used was rsync -avPX /mnt/disk3 /mnt/disk7

RobJ I might add something at the end that would reassure the user if they don't click the check box. "If you forget to click the Parity already valid and start the system it will rebuild parity and it will take awhile but no harm is done." It is difficult for me as a user to have the confidence that everything on the new 4TB drive is identical to the old 2TB drive with a different format. So I wouldn't click that button unless I was absolutely sure that parity is valid. For me, if I could just verify that parity is correct one time then I could use this method with confidence. In new users minds a command is absolute, get it wrong and you're screwed. All this changes over time after you use commands and see how they actually work. One gets over their fears of mass destruction. I know I have hand ripped well over 5,000 CD most required me to type in all the song info. So when I lost that hard disk and all the data I was upset to say the least. (Hence, I keep 2 backups) Now it's a breeze compared to what I went through 20 years ago.) I am willing to try any command to learn about it and how it works on my data as long as I have the confidence that I am not going to destroy the data and I know the consequences if I get it wrong and how to fix it. From what I've read on the forum many questions are the result of some action and a surprise result then the help! question shows up on the forum. Let me know if this is helpful. As I have said I would be happy to try to help with the Wiki if you think this kind of perspective would be helpful.

Share disk3 is identically named to a disk share This is usually caused by moving the contents of one disk to another (XFS Conversion?) and an improperly placed slash. The solution is to move the contents of the user share named disk3 to be placed within a validly named share. Ask for assistance on the forums for guidance on doing this. How can I correct this?

As I have said, I have had my head in the sand for the last 6 years with the attitude that if it ain't broke don't fix it. (I won't make that mistake again). I am aware that I have just enough knowledge to do some real damage to my server. In hindsight, I think I should have asked for advice at the beginning of this process of upgrading and converting drives. I didn't realize that I needed help until I had finished installing the new drives. The replacement process went well with a few bumps in the road. I have 8 drives and I have replaced 6 of the 8. I replaced the Parity and cache first as they both require opening the server case and I could move both drives into the 2 open Icy Dock 4 in 3 slots. Cache was there but was blank but I keep it in the overall configuration. I was able to verify parity, so I unassigned the original parity drive rebuilt parity on the new 4TB drive, then formated the cache drive. Parity was maintained. The cache drive formated btrfs as the default while the global settings were set to xfs, so I assumed that was correct. Copy method was as follows: Parity 2TB to 4TB new Cache 1TB to 1TB new btrfs disk1 2Tb to 2TB new xfs (mv command) disk2 2TB to 4TB new xfs rsync disk3 2TB rfs to xfs disk4 2TB to 4TB new xfs (rebuild) disk5 2 TB rfs to xfs disk6 2Tb to 2TB new xfs (mv command) My logic was to do the parity and cache first, then the 2 2TB NAS drive (disk 1 and 6). (these drives won't exceed 2 TB of data for several more years) the data on the drives is organized by genre and disk 6 was an overflow for the other 5. I have all drives set to fillup each drive before going to the overflow. It has worked well and this allows me to do NTFS backups by genre on a regular basis. This make it easy to find something on a backup as I can look at one disk not 6 to find anything. The process was to do the data drives requiring the least amount of time first then the big drives. Converting the rfs drives to xfs drives was an after thought. The events are as follows: the 2 2TB drives I used the mv command from the console. This went smoothly. the first 4TB drive (disk 2) I used the rsync command per the wiki this went smoothly Parity was maintained. This is were the mistakes began on my part. I thought that perhaps it would be faster if I allow the system to rebuild a drive from parity instead of copying and this would be faster than rsync. No speed improvements but the new drive was now rfs instead of xfs. I understand why. At the time I became frustrated because I had erroneously reasoned that parity only tracked the actual data. As a result, I started down the path of preclearing one of the 2TB drives and formating it to xfs then copying the data using rsync then I may as well do the other 2 drives. I have repeatedly tried to just format the drives (still had data on them) to xfs. I spent hours trying to get the format button to pop up. So, I reasoned that I needed to preclear the drives to be able to format them. I am not really upset about this because it tells me that the drives are still in good condition and in my mind it was questionable before. As I write this the 4TB and the 2 TB drives are completing the preclear process. I thought that I was going to have this all done last week (actually mid February). I started this process back in early January with upgrading software and here we are looking for completion of this stage in early April (I hope). In terms of time I think I would have been much better off replacing all the drives one at a time. This is what really confused me and caused my frustration. How would a user know to tell the system that parity is correct? I would and will always error to the system knows if it is correct or not. When I am trying something new and it doesn't turn out the way expected then I go back to the things I know work. I do think that this could be fixed. As for plugins and dockers I will need to upgrade the rest of my system before I can leave the stone age and strive to deal with future shock. I want to thank everyone for their patience in dealing with me and I really appreciate your help. Barry