tunetyme

Nope I got those steps right. It was the warning that the parity drive would be erased. That is what started this discussion. See page 24 5th post down

This is the procedure that I have been doing: File System Conversion If you have not run a successful Parity Check recently, do so now. You want to be certain that the array is perfect before you start Prepare a strategy for the order of drive conversion. Because you can't replace a larger drive with a smaller drive (unless the total file space used will fit on the smaller drive), you will have to order the conversions so that your largest data drive is first, then the next largest, then the next, with the smallest data drive being last. Obviously, the order doesn't matter for drives that are the same size. If your empty swap drive is not already installed and assigned, install it, and with the array stopped, assign it to the next empty drive slot (for our example, we will assign it to Disk 11) Click on the disk name of your swap drive (e.g. 'Disk 11') and change the format to XFS if it isn't already, then click Apply and Done If you have enabled User Shares (and most users have), go to Settings -> Global Share Settings and add your swap disk to ' Excluded disk(s) ' (for our example, we would put disk11) Start the array; your empty swap drive should show as 'Unmountable', and a Format button will be present Click the check box for formatting, then click the Format button; it takes a few minutes, says it's formatting; when done, array should show an additional drive, almost completely empty, formatted with XFS At the console or within a screen session, copy all data from your drive to be converted to the empty swap drive; use an rsync command based on the following, except change the drive numbers as appropriate for your system; type it exactly with the same slashes, upper and lower case matter; this command will take a long time but parity will be fully preserved; when complete, prompt should return with no errors showing; your array now has 2 drives that are identical except for their format, their file system (one of them is excluded from shares) rsync -avPX /mnt/disk10/ /mnt/disk11/ (in our example, we are copying our large disk10 to the new and empty swap drive) This step is optional, as the previous rsync automatically checksums each transfer. But if you would like to verify that the end-to-end transfer was perfect, perform the next rsync command below; it will take a long time, and probably nothing will be copied unless the drive has been updated (see warning below!) since the full copy above; there's no progress info, it's over when the prompt returns rsync -rcvPX /mnt/disk10/ /mnt/disk11/ Stop the array; we are now going to swap the drive assignments Click on Tools, then New Config, then Retain current configuration:, then select All, then check Yes I want to do this, click Apply then Done Important Warning! Doing a New Config will reset the file system type for all disks to Auto! While usually this is not a problem, especially with the latest unRAID, in some circumstances this can lead to unmountable disk(s). If that happens, then you need to select the correct file system for those disk(s). If in doubt, ask for help! Go back to the Main page and click on the dropdown for the swap drive (e.g. Disk 11) and unassign it (click on "unassigned" or "no device") Click on the dropdown for the other drive (the one being converted, e.g. Disk 10 to start), and reassign it as the physical drive of the swap drive, the drive that was empty (e.g Disk 11) Click on the dropdown for the slot of the swap drive (e.g. Disk 11) and reassign it to the physical drive that was being converted (e.g. Disk 10) Important! Click on each drive name (e.g. Disk 10 and Disk 11) and swap the file system format of the drive - if it's ReiserFS change it to XFS, if it's XFS change it to ReiserFS; it's important to swap the disk formats as well as the physical drive assignments At this point, you have now swapped the 2 drives, which is fine as they are identical (except for file system format); parity remains valid because the same drives are assigned, their slot does not matter; however if you have a second parity drive, it's now invalid! You should see all array disks with a blue icon, a warning that the parity disk will be erased, and a check box for Parity is already valid; IMPORTANT! click the check box, make sure it's checked to indicate that Parity is already valid or your Parity disk will be rebuilt! then click the Start button to start the array; it should start up without issue and look almost identical to what it looked like before the swap, with no parity check needed; however the XFS disk is now online and its files are now being shared as they normally would; check it all if you are in doubt If you are sure it's all fine, stop the array and click the empty swap disk slot (e.g. still Disk 11), and change the format to XFS, then click Apply and Done Start the array; the Format button should be available, format it now; when done, your empty disk slot now has a fresh and empty disk formatted with XFS and ready to fill again; your data drive has completed the conversion process and is already back online, with all files and shares intact, but formatted with XFS You are now ready to convert the next drive, so circle back to Step 8 and repeat these steps (Step 8 through Step 18), substituting your next drive to be converted; the empty and excluded swap disk slot will always be the same (e.g. always Disk 11 in our example), the other will change as you convert different data drives When done, you have an empty XFS drive appended to your system, probably your smallest drive, and still excluded. It's up to you what you want to do with it. You can leave it as is, or you can unassign it and rebuild parity, or you can use the parity preserving remove-a-drive procedure, instructions elsewhere. Remember, it's probably still globally excluded from shares. I do recommend that if you are going to try this procedure, you read through the steps and notes carefully until you fully understand them, and understand the importance of each detail. Missing a step or typing the wrong disk number could be disastrous! If you wish, you can perform parity checks at any point during and after. I don't believe they are necessary, I only did one before starting, and I believe I did another after it was all done. Warning! If you run the verification copy in Step 9, and it actually copies files, then it is likely you have a process still changing the drive! These newly copied files were not there for the Step 8 copy! You need to determine what process (Docker, plugin, VM, an external backup, or the Mover) made the changes to this drive, and stop it. Then you may need to run Step 9 again, because the process may have made even more changes to folders, after the Step 9 rsync process had moved past those folders. In summary, if the Step 9 copy actually copies any files, then you should probably repeat Step 9 until nothing is copied. I've checked the above pretty carefully, if you see any errors, PLEASE let me know ASAP! I'm sure it can be improved. Steps 17 and 18 are a repeat of 4, 6, and 7, but it seemed safer to do it this way. Please let us know of any issues or suggestions! ____________________________________________________________________________________________________________________________________________________________ Step 8 Everything was going according to the the above method until I missed a /. rsync -avPX /mnt/disk10/ /mnt/disk11/ this is the proper command I missed the / slash after disk10. This created another share someplace that we can't find. I did steps 10 -16 where I became confused about parity. RobJ has updated this section as above to help with the confusion (see previous 2 pages). Step 16 was the start of the confusion about parity.

I may need to dig a little deeper to comprehend this. Still haven't been able to solve the disk3 share issue does anyone have a way for me to address this?

I do understand how parity itself works and I understand how it works across drives basically odd or even. What I was trying to express is once I have made a duplicate set of files (rsync) on two different size disks with different formats (RFS v XFS) I am now able to remove the old disk and replace it with the new disc (same slot) and parity is still valid. By removing the old disk did I just remove one of the bits being counted? By swapping the new disk with the old and removing the old disk (changing slots and removing the old disk completely) how is parity maintained? I have just swapped and removed one disks that was being counted in the odd even for each bit of data. I used to deal with parity issues when dealing with serial and parallel data communications. As I have tried to explain I tend to think in terms of an absolute address on each disk where a bit is counted as odd or even. any changes in the bit being counted or the quantity of bits being counted (number of disks) changes parity. As I have said I need to think about what relative addressing means.

I think in terms of absolute addresses not relative. I need to think about that for awhile. I guess I need to throw some rocks in the pond and contemplate relativity along with my naval. I guess I'll go hitch up the buggy and go to town... So rsync doesn't do anything more than copy files. I had the impression that it does much more on a lower level.

OK, let me see if I can explain my dilemma in a different way. Using bits At address x on a 2TB drive RFS (not that the format is essential) =1 and address y = 0 I use rsync to copy the data to a 4TB drive xfs I am assuming that both formats use the same address scheme you are saying that address x = 1 and address y = 0 I guess I am thinking like Windows where if I copy a file from drive a to drive b it also defragments the file when it is copied.. Are files ever fragmented under Linux?

there is a user named disk3

When I compute free space for disk3 it shows 10.9TB ????

index of /mnt/user/disk 3 Type Name Size Location Last Modified Parent Directory S Disk 3 2017-01-22 15:05 1 object: 1 directory, 0 files

Not on Cache, nor disk3 I've looked down 3 levels When I look under share info under include it says all and exclude says none. I can't change these.

Any ideas on correcting the disk3 share? It shows up in shares but I can not find it on any disk in my system. The data does not appear in my primary video Share all the other disks do.

Are you saying when you use rsync and copy a file drive one (2TB rfs to 4TB xfs) to drive two that each bit ends up at the exact same address? If so, that's amazing and who ever wrote that command didn't have a life for quit awhile. Again, if you are old school like me then "show me". That's why If I do verify parity before starting the process and then substitute the 4TB drive for the 2TB drive in the array I would want to see that parity is still valid. If I do it one time then I will trust it. Remember I started back in the day of punch cards,then Decwriters (tons of greenbar) and finally CRT's. We were taught to be skeptical...GIGO (garbage in garbage out). I remember when they were trying to get the code rock solid for basic things like a keyboard. It may be that kids these days don't have to contend with this anymore but this was my training. Still think the step 11 when newconfig turns format type back to auto is a mistake. If I unassign the drive, I can quickly see the format type so I am confident the info is there. Auto was most likely a short cut.

Those of us who are old school think more along the lines of a parity bit used in each byte (9th bit) or data communications. I will studied the references. I didn't read them before because I thought I already understood how parity works.

I think most users think parity is static in other words when I think of data stored on my drive it is 8 bits which is a group of ones and zeros These are at a very particular location on my drive when I make a copy it does not necessarily land at that exact same spot on the new drive it could be in any other sector. So without testing and seeing that is at the same exact location then there is a leap of faith that you are asking for that I have a hard time making. That's why I am suggesting that there is a method of doing a verify option to prove to the new user that it in fact works.

I think you missed my point...The fact that you have two different size drives and different format the question in my mind is how can parity be valid when I swap out a 2TB RFS drive with 4TB xfs drive. I know it works out now but at the time it was a strech to comprehend this.

Ahhhh I didn't see that slash. I've checked disk7 (now it is disk3) and no folder called disk3 The old disk 3 has been precleared and formated xfs

Trurl I have checked every disk on my system (can't see parity) there is no file folder named disk3. The rsync command I used was rsync -avPX /mnt/disk3 /mnt/disk7

RobJ I might add something at the end that would reassure the user if they don't click the check box. "If you forget to click the Parity already valid and start the system it will rebuild parity and it will take awhile but no harm is done." It is difficult for me as a user to have the confidence that everything on the new 4TB drive is identical to the old 2TB drive with a different format. So I wouldn't click that button unless I was absolutely sure that parity is valid. For me, if I could just verify that parity is correct one time then I could use this method with confidence. In new users minds a command is absolute, get it wrong and you're screwed. All this changes over time after you use commands and see how they actually work. One gets over their fears of mass destruction. I know I have hand ripped well over 5,000 CD most required me to type in all the song info. So when I lost that hard disk and all the data I was upset to say the least. (Hence, I keep 2 backups) Now it's a breeze compared to what I went through 20 years ago.) I am willing to try any command to learn about it and how it works on my data as long as I have the confidence that I am not going to destroy the data and I know the consequences if I get it wrong and how to fix it. From what I've read on the forum many questions are the result of some action and a surprise result then the help! question shows up on the forum. Let me know if this is helpful. As I have said I would be happy to try to help with the Wiki if you think this kind of perspective would be helpful.

Share disk3 is identically named to a disk share This is usually caused by moving the contents of one disk to another (XFS Conversion?) and an improperly placed slash. The solution is to move the contents of the user share named disk3 to be placed within a validly named share. Ask for assistance on the forums for guidance on doing this. How can I correct this?

As I have said, I have had my head in the sand for the last 6 years with the attitude that if it ain't broke don't fix it. (I won't make that mistake again). I am aware that I have just enough knowledge to do some real damage to my server. In hindsight, I think I should have asked for advice at the beginning of this process of upgrading and converting drives. I didn't realize that I needed help until I had finished installing the new drives. The replacement process went well with a few bumps in the road. I have 8 drives and I have replaced 6 of the 8. I replaced the Parity and cache first as they both require opening the server case and I could move both drives into the 2 open Icy Dock 4 in 3 slots. Cache was there but was blank but I keep it in the overall configuration. I was able to verify parity, so I unassigned the original parity drive rebuilt parity on the new 4TB drive, then formated the cache drive. Parity was maintained. The cache drive formated btrfs as the default while the global settings were set to xfs, so I assumed that was correct. Copy method was as follows: Parity 2TB to 4TB new Cache 1TB to 1TB new btrfs disk1 2Tb to 2TB new xfs (mv command) disk2 2TB to 4TB new xfs rsync disk3 2TB rfs to xfs disk4 2TB to 4TB new xfs (rebuild) disk5 2 TB rfs to xfs disk6 2Tb to 2TB new xfs (mv command) My logic was to do the parity and cache first, then the 2 2TB NAS drive (disk 1 and 6). (these drives won't exceed 2 TB of data for several more years) the data on the drives is organized by genre and disk 6 was an overflow for the other 5. I have all drives set to fillup each drive before going to the overflow. It has worked well and this allows me to do NTFS backups by genre on a regular basis. This make it easy to find something on a backup as I can look at one disk not 6 to find anything. The process was to do the data drives requiring the least amount of time first then the big drives. Converting the rfs drives to xfs drives was an after thought. The events are as follows: the 2 2TB drives I used the mv command from the console. This went smoothly. the first 4TB drive (disk 2) I used the rsync command per the wiki this went smoothly Parity was maintained. This is were the mistakes began on my part. I thought that perhaps it would be faster if I allow the system to rebuild a drive from parity instead of copying and this would be faster than rsync. No speed improvements but the new drive was now rfs instead of xfs. I understand why. At the time I became frustrated because I had erroneously reasoned that parity only tracked the actual data. As a result, I started down the path of preclearing one of the 2TB drives and formating it to xfs then copying the data using rsync then I may as well do the other 2 drives. I have repeatedly tried to just format the drives (still had data on them) to xfs. I spent hours trying to get the format button to pop up. So, I reasoned that I needed to preclear the drives to be able to format them. I am not really upset about this because it tells me that the drives are still in good condition and in my mind it was questionable before. As I write this the 4TB and the 2 TB drives are completing the preclear process. I thought that I was going to have this all done last week (actually mid February). I started this process back in early January with upgrading software and here we are looking for completion of this stage in early April (I hope). In terms of time I think I would have been much better off replacing all the drives one at a time. This is what really confused me and caused my frustration. How would a user know to tell the system that parity is correct? I would and will always error to the system knows if it is correct or not. When I am trying something new and it doesn't turn out the way expected then I go back to the things I know work. I do think that this could be fixed. As for plugins and dockers I will need to upgrade the rest of my system before I can leave the stone age and strive to deal with future shock. I want to thank everyone for their patience in dealing with me and I really appreciate your help. Barry

Trurl: I was just reading Johnathanm comment regarding legacy issues. I had two Shares that was ans is set up one for music (P) and one for movies (S). I did this when 4.7 was set up and I initially had a few small drives. RobJ: I was not trying to be critical. I followed that step and I took the information literally. I am determined to learn all the ins and outs of unRaid and Linux over the coming years. I remember reading the Unix command descriptions (in the 80's while working with AT&T haven't used them 25 years) that were cryptic at best for a new user. When an event or message appears that I do not expect, nor a caveat what to do if such and such happens, then my first question is what did I screw up in the process. If it appears that I did things correctly and now have some additional steps (fumbling around trying things) then it appears to be convoluted. I think the question for the process describe is does parity need to be maintained or not? I used this for media NAS for the last 6 years under 4.7. I see that what you have been creating while I sat quite happy with my system cranking along without any hiccups until I needed to increase the size of my disk drives. Back to the issue: If parity needs to be maintained then follow method A: If parity does not need to be maintained then method B: Please remember when learning a new system it is the unexpected items that pop up causing end users to do stupid things. If you it would benefit you and other users I will gladly help with trying things and help finding items that may need further explanation or a reference to a detailed description. It kind of like that old TV commercial "Let Mikey eat it he hates everything" suddenly Mikey is eating the cereal. I could benefit greatly by becoming more knowledgeable and an asset to the forum and hopefully reduce some of the misunderstandings by users that create a lot of questions. OK, and I thought that the plugins were the new thing too. I think that the steps I need to do is first finish straightening out my data, make sure my basic NAS sytem is working with JRiver media center, (I would like to talk with you during this process to be sure I get everything setup to step into the brave new world of unRaid) then upgrade my mobo, cpu and ram. At least I will know what kind of an upgrade I will need to do. I have looked into the new AMD Ryzen (low end) CPU that uses 65 watts, Asus mobo and DDR4 RAM 16MB and it turns out to be surprisingly affordable. I actually will not need to do some of the other hardware upgrades like power supply, high end cpu cooler etc. that brings the cost in-line with a FX-8350 (125 watts) based rig. Either way I will be out of pocket about $600 but If I wait a few months there may be some new less expensive options that may provide a more cost effective solution.

Sorry about the venting. I tried following the guide to the letter and when I exchanged the 4TB drive for the 2TB parity was no longer valid put me over the edge. I realize (or hope) this will be the only time I have to go through this process. I only have two more data drives to change out of 6 and then I think I would like to change the cache drive to xfs. I still think the NewConfig command could be improved by keeping each of the disks format identified. I am not sure "auto" is beneficial. I am preclearing 2 previous written to data drives now so that will take a couple of days. I should have done the conversion as I replaced the drives and spread out the task over a longer period of time. I won't move it. On the plus side I have wanted to get some exposure to the rsync command. I have seen a number of references to it and I would like to find an in depth tutorial on it's use. While I complain sometimes about the process I am more frustrated with my lack of knowledge more than anything else. I have wanted to rearrange my disks for a number of years so I am going to take this opportunity to do it. While I have an easy time setting up disks for preclearing I seem to have difficulty having the format button show up????? I have had it show up on occasion but it wouldn't execute the command when I clicked on it. Can't tell you how I got there as I was trying everything. I did not know about the legacy stuff on /mnt/diskx vs /mnt/user locations. Is there something in the WIKI?? Is there any other legacy issues I am unaware of since I have jumped from 4.7 to 6.3.2? Thanks for your help.

Trurl: My 2 cents is that there must be a way to stream line this process. I followed the directions in the Wiki reference step by step. It finished the rsync sometime late last night, moved the drives around as directed now it says "All data on the parity drive will be erased when array is started" Every possible combination of configurations ended up with the same message. Considering all the time it takes to do this one disk at a time, I would suggest that a backup copy of each disk be made using rsync, then preclear all the data drives again and format them xfs. In my case it would save about a 3 -4 days time per disk. Another option would be to buy a new disk that is the same size or larger. Preclear and format dive XFS. Unassign the disk being replaced. Put new disk in slot. Somehow prevent the rebuild, then use rsync to copy from the old RFS disk to the new XFS. No offense, but this is a convoluted process because now I have to preclear the old RFS drive then format it and I find that I don't get the format button and reboot and use trial and error until it pops up. I have no idea what I've done or not done to get this accomplished. At this point, since I have backups of all my drives on NTFS format, I think the best way to deal with this is to unassign the parity drive and go through the lengthy process of doing one drive at a time then rebuild parity. If a file is damaged, I have a backup or I can get the original back out and rip it again. I don't see all this as mission critical stuff and frankly it shouldn't be this hard and time consuming. It has taken me 1 day+ to preclear and format a 2TB drive 1.75 days to copy using rsync 1.7TB of data. I have 2 more drives to do one 4TB (double the time) and one 2TB. At least I can preclear both of them at the same time. Thankfully, I have Icy Dock data drive bays. I can't begin to imagine what people do when they have to climb inside their case every time. BTW I think it is a significant flaw when you use new config that drive format goes to auto. Major source of frustration to go through and change everything back. It seems to me that there be a program that allows you to use the same drives in the same slots add one disk that files are copied to, Copy file then reformat drive (hopefully this can be done without preclearing the drive again) move files back to drive. Repeat until all disks are converted. At that point rebuild parity or if there is a way to maintain parity then do so. The only downside is you have to copy the files twice and at least on my rig, that takes a long time. Well in another week or two this nightmare will be over. (I hope) I won't have to mess with this again until it is time to install the other two 4TB drives in a few months.

I think it was inevitable that I would end up here. I have recently upgraded from 4.7 to 6.3.2. Now the challenge is to get all my drives converted to XFS. My configuration: Disk1 new 2TB XFS blank This will stay as a 2TB drive 0% used Disk 2 new 4TB XFS copied from disk1 no new data is on drive 49% used Disk 3 new 4TB RFS was formated XPS but tried to do a rebuild vs copy no new data is on drive Still have the original Disk 3 (just removed from the array) 47% used Disk 4 old 2TB RFS keep on a 2 TB drive until I replace with 4 TB drive 95% used Disk 5 old 2TB RFS keep on a 2 TB drive until I replace with 4 TB drive 78% used Disk 6 new 2TB RFS keep on a 2 TB drive until I replace with 4 TB drive 88% used Parity new 4TB Cache new 1TB BTRFS If this could be XFS I would prefer to have everything with the same format 0% I have 2 open slots available for for drives in my ICY Dock trayless Additional drives: I have the original 2TB disk3 untouched I have the 2TB parity drive I have the original 2TB disk 2 (needs to be precleared and formated for use I have complete backups of each drive on NTFS drives so I am not worried about losing anything. I am looking for the most efficient way to change each data drive to XFS and if advised to do so convert the cache drive. The server is in what I call maintenance mode so I will not make any changes to the data on any drive but will move wherever it needs to go. When I did the copy vs rebuild on the 4TB drives, it took about the same length of time. Disk one could be used as a temporary disk as this is a different share than all the other drives. 1. What is the best method to convert to XFS? 2. Should I convert the cache drive too? 3. I have fumbled around to preclear and format my drives XFS that have been previously used. What is the best method to do this? For example disk 3 was converted to RFS (formated XFS) because I used the rebuild method. Do I need to preclear the drive again? What steps do I need to do to convert any of my old drives to XFS and begin replacing the RFS drives? 4. I would like to learn how to use rsync command to move the files if that is advisable. It appears to be quite useful when one learns how to use it properly. 5. If I need to mount one of the NTFS drives and copy what is the best method and how do I do it? Geezzz, I need to go back to school. I think they teach this stuff in first grade. It would be so embarrassing trying to sit in one of those little chairs with my knees next to my ears.

I've spent enough time on the bleeding edge of technology. No need to do that now.