RobJ

The procedure in this post is only for releases of unRAID v6.0 and v6.1. For unRAID v6.2 or later, go to File System Conversion on the wiki. A few days ago, I finally began the process of converting old drives to XFS, and I found a few ideas that make the process simpler and less of a disruption. Parity protection is always preserved, and normal server operation can continue at almost all times, except for brief moments of swapping drive assignments. I use the rsync command with the -avPX options, as it accurately performs copies of all data, metadata, and extended attributes of all files and folders. In addition, from statements online, I understand that rsync ALWAYS checksums every bit of data transferred all the way to its reconstruction at the destination. That makes the additional lengthy checksum verification almost redundant, so I include that verification only as an option, for the most paranoid. I've come to feel that an rsync transfer is the only option most users need, and that speeds this process up. Then I perform a simple swap procedure, that preserves all shares and data, and doesn't require any lengthy parity checks or builds. You can if you wish perform parity checks at any time, but they are optional, if you are sure there has been a successful recent one before you start. I exclude the swapping drive from User Shares, so you never have to worry about files duplicated within shares, and therefore don't need a temporary copy location, and never need to move any files. Important Update! The swap trick below does not work in unRAID v6.2] (steps 10 through 13). This is really unfortunate as the swap trick is a key part, makes the procedure much simpler and more convenient to perform. I've left this post unchanged, but it is only for releases of unRAID v6.0 and v6.1. For v6.2 or later, go to File System Conversion on the wiki. Update and Important Warning! My system is static, and my array only changes when I manually copy files and make changes. Which is why I completely forgot that other user's arrays are dynamic, constantly changing in the background due to Dockers, plugins, VM's, scheduled backups from other networked machines, and the Mover copying and moving files around. That's a big problem when you are converting a drive, because you cannot allow any changes to it once the copying begins. As the copy sweeps through the folders of the original drive, any new files added to folders already processed will be lost! It is your responsibility to make sure that NOTHING can make changes to a drive being copied and converted! That may mean stopping your Dockers, plugins, VM's, and the Mover, if any of those could possibly add files or make changes to the drive being converted. And you should check for and temporarily disable any scheduled backups from other machines. Yes, this can require some careful planning! You may want to consider rebooting into Safe Mode (stops plugins from loading), and disabling any VM's and the Docker service and the Mover. Procedure to convert drives to XFS format ================================ System assumptions before you begin: - You have a given number of data drives you wish to convert (we will use an example of 10 data drives) - You have prepared a new empty drive that is as large or larger than the largest data drive; preferably you have Precleared it or otherwise tested it - Note: the new drive should not be formatted with XFS; if it is, then you will need to clear it or format it to something else first 1. If you haven't recently run a successful Parity Check, do so now; you want to be sure the array is perfect before you start 2. Prepare a strategy for the order of drive conversion. Because you can't replace a larger drive with a smaller drive (unless the total file space used will fit on the smaller drive), you will have to order the conversions so that your largest data drive is first, then the next largest, then the next, with the smallest data drive being last. Obviously, it doesn't matter for drives that are the same size. 3. With your empty drive installed and array stopped, assign it to the next empty drive slot (for our example, we will assign it to Disk 11) 4. Click on the disk name of your empty drive (e.g. 'Disk 11') and change the format to XFS if it isn't already, then click Apply and Done 5. If you have enabled User Shares (and most users have), go to Settings -> Global Share Settings and add your empty disk to 'Excluded disk(s)' (for our example, we would put disk11) 6. Start the array; your empty drive should show as 'Unmountable', and a Format button will be present 7. Click the check box for formatting, then click the Format button; it takes a few minutes, says it's formatting; when done, array should show an additional drive, almost completely empty, formatted with XFS 8. At the console or within a screen session, copy all data from your drive to be converted to the new and empty drive; use an rsync command based on the following, except change the drive numbers as appropriate for your system; type it exactly with the same slashes, upper and lower case matter; this command will take a long time but parity will be fully preserved; when complete, prompt should return with no errors showing; your array now has 2 drives that are identical except for format (one is excluded from shares) rsync -avPX /mnt/disk10/ /mnt/disk11/ (using our example, copying our large disk10 to the new empty drive) 9. This step is optional, as the previous rsync automatically checksums each transfer. But if you would like to verify that the end-to-end transfer was perfect, perform the next rsync command below; it will take a long time, and probably nothing will be copied unless the drive has been updated (see warning below!) since the full copy above; there's no progress info, it's over when the prompt returns rsync -rcvPX /mnt/disk10/ /mnt/disk11/ 10. Stop the array; we are now going to swap their assignments; when stopped, click on the dropdown for the new drive (e.g. Disk 11) and unassign it; it should notify you of 'Missing drive' 11. Click on the dropdown for the other drive (e.g. Disk 10), the one being converted, and reassign it as the physical drive that was just added, the new drive that was empty; you will be notified of 'Wrong drive' 12. Click on the dropdown for the slot of the new drive (e.g. Disk 11) and reassign it to the physical drive that was being converted (e.g. Disk 10); you will get more notifications of wrong or missing drives; you have just swapped the 2 drives, which is fine as they are identical (except for file system format) 13. Important! Click on each drive name (e.g. Disk 10 and Disk 11) and change the format of the drive; if it's ReiserFS change it to XFS, if it's XFS change it to ReiserFS; it's important to swap the disk formats as well as the physical drive assignments 14. You should see both drives listed with errors, possibly as unmountable, and a check box; click it and click the Start button to start the array; it should start up without issue and look almost identical to what it looked like before the swap, with no parity check needed; however the XFS disk is now online and its files are now being shared as they normally would; check it all if you are in doubt 15. If you are sure it's all fine, stop the array and click the empty disk slot (e.g. still Disk 11), and change the format to XFS, then click Apply and Done 16. Start the array; the Format button should be available, format it now; when done, your empty disk slot now has a fresh and empty disk formatted with XFS and ready to fill again; your data drive has completed the conversion process and is already back online, with all files and shares intact, but formatted with XFS 17. You are now ready to convert the next drive, so circle back to Step 8 and repeat these steps (Step 8 through Step 16), substituting your next drive to be converted; the empty and excluded disk slot will always be the same (e.g. always Disk 11 in our example), the other will change as you convert different data drives When done, you have an empty XFS drive appended to your system, probably your smallest drive, and still excluded. It's up to you what you want to do with it. You can leave it as is, or you can unassign it and rebuild parity, or you can use the parity preserving remove-a-drive procedure, instructions elsewhere. Remember, it's probably still globally excluded from shares. I do recommend that if you are going to try this procedure, you read through them carefully until you fully understand them, and understand the importance of each detail. Missing a step or typing the wrong disk number could be disastrous! If you wish, you can perform parity checks at any point during and after. I don't believe they are necessary, I only did one before starting, and I may do one after the last. Warning! If you run the verification copy in Step 9, and it actually copies files, then it is likely you have a process still changing the drive! These newly copied files were not there for the Step 8 copy! You need to determine what process (Docker, plugin, VM, an external backup, or the Mover) made the changes to this drive, and stop it. Then you may need to run Step 9 again, because the process may have made even more changes to folders, after the Step 9 rsync process had moved past those folders. In summary, if the Step 9 copy actually copies any files, then you should probably repeat Step 9 until nothing is copied. I've checked the above pretty carefully, if you see any errors, PLEASE let me know ASAP! I'm sure it can be improved. Steps 15 and 16 are a repeat of 4, 6, and 7, but it seemed safer this way.

One of the nice features of v6 is all the notifications possible. I now get an array health report emailed every day, which just happens to include a nice screen-shot-like listing of the complete array and its drives, with models and serial numbers. Just as good as a screen shot (and easier), and much easier than notes.

In your case, these are a few of the obvious one: Much improved GUI. Built-in APCUPSD support E-mail Notifications Better support for plugins In the future, you will find that user-based support in this forum will decline as more and more people convert to version 6.X. That alone should provide you with all the reason to convert. I would add that v4 and v5 are no longer supported, and you are increasingly vulnerable to attacks from infected stations on your own network. Just as you should keep your Windows and other desktop stations updated, your unRAID server should be kept updated to the latest version. LimeTech has been doing a good job lately in patching security vulnerabilities as they arrive. While unRAID servers are still not ready for full exposure to the Internet, they are protected from known attacks (Shell Shock, Heartbleed, etc) by infected stations and routers on your local network, IF you are keeping unRAID updated to the latest (currently v6.1.8 ).

Well, it was installed when I ran sensors-detect, but it seems Perl is no longer present after rebooting. I just ran `installpkg /boot/packages/perl-5.18.1-x86_64-1.txz` again, uninstalled and re-installed the plugin, and now the aforementioned Detect button behaves as expected. Other than putting the Perl archive at /boot/packages/perl-5.18.1-x86_64-1.txz, is there something else that I need to do to ensure Perl is always available after reboots? Perl is only needed temporarily when you want to run sensors-detect or use the Detect button, not needed after that, so there's no reason to keep it installed. Besides it's big, wastes memory. Yeah, that wiki page probably needs some updating ...

You are positive that Perl is installed?

I've never tried this, so can't guarantee anything. Manually edit your sensors.conf (/boot/config/plugins/dynamix.system.temp/sensors.conf), and add the line "compute temp9 (@-30),(@+30)" just below the line with the faulty temp. Replace temp9 with your raw temp name. I don't know, but you may need to reboot. For example, # sensors chip "acpitz-virtual-0" label "temp1" "CPU Temp" compute temp1 (@-30),(@+30) chip "k8temp-pci-00c3" label "temp1" "MB Temp" For more info, see the following links - http://linux.die.net/man/5/sensors.conf https://www.opennet.ru/man.shtml?topic=sensors.conf&category=5

I don't want to ruffle any feelings, there's enough of that already, but I do want to comment on the discussion above. (Just my opinion of course ...) Apart from the attitudes, there's a lot of good information here, useful and helpful. Our community is changing, widening, gathering interest from many sources lately, partly from the VM hype and successes (Linus Tech Tips a big example), and partly from the age and experience of this community and unRAID itself, and the respect it has slowly been gaining. It's amazing how often when searching Google for an unknown error, unRAID forum links are very prominently listed. It's hard to search for a disk drive error and not find an unRAID link near the top. For these reasons, I'd like to see us more welcoming to all, even if they aren't necessarily unRAID users, as I think it adds to our community. There's good info in the posts above, and it's to our benefit that it's located in our forum not elsewhere. Users from outside our community have experience we can learn from, and I believe a welcoming hand creates a wider mutual respect, benefiting unRAID in the long run. We *want* to be a welcoming community, where everyone is treated with respect. We do ask that newcomers treat us with respect too, and certainly, expressions like "where do you get off" are not acceptable here. Users from outside often do have more 'attitude', not used to the respect we try to show everyone. But treated with respect, I expect them to soften. Yes, sometimes we moderators have to deal with users who have proven to be incorrigible, even after warnings, but I hope we are never heavy handed. The discussion above does diverge some, from 5TB Seagates, but it seems too entangled to separate into multiple threads. Take someone wiser than me to do it.

I understand there are 2 different versions of P20, a bad buggy P20 that ends in .0.0, and a good P20 that ends in .4.0.

Bad memory can be so embarrassing at times! I commented from memory and it failed me, again. It looks fine, nothing else needed!

Interesting tool, nicely done! Thank you. One minor request... In the syslog, I noticed you log the start and finish of a speed test, but not the result. Would it be possible to add one more line with a short summary of the test results? Hmmm... since I'm making requests - bonienl's Parity Check History button is really nice! Any chance of something similar here, with timestamps, so we could compare bandwidth speeds at various times? My memory doesn't work so well, and I'm too lazy to write the numbers down (terrible excuse I know!). It's all there, bad memory strikes again!

I can't remember for sure, but I thought that interim results were saved to the flash drive, somewhere. If not, perhaps gfjardim can intercept them at the end of each pass, and copy them to the flash drive. Seems like the initial SMART reports and all interim reports *should* be available on the flash drive, no matter what interruptions happen.

I've suggested elsewhere in the past that CacheDirs should ALWAYS be used with specified Includes. Bonienl provides a very convenient dropdown, to select only those folders you really need cached, and no others. Performance and RAM usage are best doing it this way. And if you only specify the exact folders you want included, you should never have to worry about other disks being scanned. However, that was all before Unassigned disks could be in the system, and there have been code changes, so excluding unwanted drives may be necessary now (don't know for sure). Well technically CacheDirs doesn't 'un-cache' anything, but if a user wants too many items cached and doesn't have sufficient buffer space for them, then CacheDirs is going to fill up all available cache room, then the kernel is going to dump the earliest entries (from the first drives and folders). Then seconds later on the next pass, CacheDirs is going to have to go to disk to reload those, keeping those drives spinning and setting up constant disk thrashing. Again, you should only cache what you absolutely need cached, and nothing more. Almost a year ago, I started rewriting CacheDirs into a plugin friendly version, event driven. I made some slow progress, cutting out a lot of the old, then discovered what a great job bonienl did in wrapping and controlling it from his Dynamix plugin. I stopped, since it wasn't needed any more. And then I believe you provided additional modifications and fixes, to modernize it and make it more plugin compatible. One thing I was going to add was more control over what is cached, by adding the option to specify absolute paths. Currently, paths are relative to /mnt/user, but if you allow the user to specify paths beginning with a slash, you can assume they are absolute, and provide more pinpoint specification of sub-folders they may want cached. Should be simple for one of you to add, and be helpful for certain users.

The NerdPack has been a supplementary collection of modules that not every one needs. It helps to keep the installed core leaner. Your request is a different usage, a way to provide newer versions of the installed modules, whenever users feel LimeTech is not moving quickly enough. That's possibly valid, but I think I would prefer a separate collection. With each unRAID release, you'll have to decide what to change with it, what's no longer necessary to install because it's now included in the core. That makes this new collection quite volatile, much more than the NerdPack has been. There's good value in the NerdPack, in keeping our core systems leaner. I can see in the future Tom and Eric getting together and moving more modules from the installed unRAID core into NerdPack (or its built-in successor). It would be especially attractive to those users who only wanted a 'lean and mean' NAS. They would uncheck all VM support and uncheck all Docker support, and any other feature support they don't plan to use. Some things can't be separated, have to be compiled into the kernel core, but others should be (libvirt?).

I can see how that could happen, which makes it a little dangerous. The Linux kernel does support hot-swapping, but I don't think the unRAID modules are completely in sync with the kernel. That is, unRAID does not always know what has been re-assigned. Occasionally, the kernel will reassign a previously used drive symbol (sdj, sdm, etc), once it has been dropped and available. I believe unRAID will assume that that drive symbol still refers to the drive it knew about. I would definitely avoid doing that.

That's a 2007 BIOS, upgrade it if at all possible. Virtualization support has come a long way.

I think it only displays in the header if your server IS the local master. Perhaps bonienl might consider changing the display for more consistency, show Local Master: yes/no always. Or show Local Master: Tower or Local Master: Jupiter or Local Master: MyWinServer or Local Master: ASUSrouter, etc.

This may or may not be, but it sounds like a sync behavior. If there are unwritten writes to a device, then a sync will require spinning up that device to flush them. Off-topic, but I've never understood why Linux has write caching, but allows the writes to be delayed essentially forever. 5 seconds should be more than long enough. I'm probably not understanding something fundamental though. I also don't understand why sync can't be limited to a single device, not everything.

I don't have any ideas myself, but I do suggest simplifying the process first - try running a simple Preclear from the command line, the old way.

Both very good points. Rather than repeat myself then, I just linked to my first post, but I probably should have quoted myself. Dan's points are quite valid, something I didn't think of. Hopefully someone like Dan will step up again when this happens next. We have to remember though that in this case he stepped up because this plugin was very important to him. This problem isn't going away, and I suspect it will take time, but there are going to be more unfortunate situations, without easy resolutions. After a bad loss, some users may become 'gun shy', and begin rightly or wrongly flocking more toward linuxserver.io options over binhex options (to give one example), without at least an informal statement up front that *someone* is willing to take over if needed. Perhaps the most concerning are all the PhAzE plugins. At the same time, I do recognize it's possible it may *never* become a serious problem. Shutting up now, that's as far as I'll push on this.

I can't help with your first question, but this one is familiar, should probably be part of a FAQ somewhere. CacheDirs forces the caching of the directory entries, which is all that is needed for simple folder browsing. But user workstation tools often want more, associated metadata such as thumbnails. Since that's still only on disk, it has to spin up. You'll have to wait for a Mac user to provide you with the appropriate instructions to avoid requiring extraneous metadata from networked folders and files.

Well since you asked... I've been thinking about something I think is important, part of the maturation of the unRAID community and its many development partners. I brought it up first over here. unRAID and the numerous addon options have come a long way, and it's time to think about risk management, not just for our data but also for the tools. The community is growing, and depending more and more on so many plugins and containers. Yet for the most part, these plugins and containers have a single author, and that's a single failure point. Think of what would happen to so many users if something happened to PhAzE, binhex, bungy, etc, with so many plugins and containers and so many dependent users. As we're all reminded now and then, life happens, and it rarely comes with advance notifications. Businesses have to have disaster plans, for hurricanes, earthquakes, tornadoes, fires, etc, and so do we. Rudimentary perhaps, but something that users can count on. It's important that users be able to select tools with a known backup plan, a succession plan if something happens to the original author. In a way, I'm an outsider in the addon development world here, so I don't want to say what should happen. Perhaps as simple as a field in CA for 'Backup author/group'? But I'm hoping this can get the ball rolling, and you Squid and the many authors can decide on a mechanism, that provides at least a minimum of a succession plan. I know you have already worked out blacklisting processes, and that's good, but we really need a way to make sure important plugins and containers are carried on, not lost and blacklisted. Once something is set up, then we can all nudge the authors to make sure they have someone they can get to cover for them, if the unexpected happens... However, I can imagine some authors being resistant. That's fine, it's their right. What is a backup author? Basically someone that is able and willing. At a minimum, they need to be able to manage a repository, the original or their own; they need to be able to access the source, and make needed corrections; and they need to be able to package up an update. More than that is gravy.

So great to see you back, gfjardim! I fully understand setbacks, just out of the hospital myself, life happens! I think credit is due everywhere here, definitely not blame. This plugin was the result of some great work, and had become so important to many users, that when life intervened with the author and maintenance was required, another user stepped in. His work was so successful, a stream of additional requests began, and time passed. Yes, with the passing of time, the distance between the forks necessarily widened, but it's not insurmountable. And I think we all agree we don't want another situation like the past. This brings up something I've been thinking about for awhile - the real need (a critical need in my opinion) for a stated succession plan or backup author for ALL plugins and Docker containers. The backup person does not have to be as gifted, just capable and willing to carry on in a limited way at least, if something should happen to the original author. Right now, the Linuxserver.io group have an important advantage over others, a built-in succession plan. I find it worrying to consider what would happen if something happened to PhAzE or binhex or a few other authors, with the number of plugins and containers and so many users depending on them. I hope they will consider finding someone they can assign, 'just in case'. I hope that Squid will consider the possibility of adding another field for this, so that when users evaluate equivalent options, the one with a backup plan has a significant advantage. I understand not every plugin or container needs a backup person or plan. In some cases, you should be able to remove one container, load a similar one, and carry on with little change, and no loss. But that's certainly not true in many cases, especially important plugins like this one.

I'd say it's nearly impossible to detect bitrot in constantly changing files, without added file monitoring and constant check-summing. Obviously, it's impossible to determine what is bitrot during file changes, so the only period you can usefully monitor is the idle period between changes. You would need monitoring software to detect when the file is closed after modification, and immediately initiate a fresh checksum, then be able to detect and pause any file modifying software, so that a checksum can be recalculated and compared. I doubt that TimeMachine has a way to pause itself while you re-verify the checksum, so the only alternative is constant re-verification, perhaps on 5 minute intervals. That way, you could detect bitrot *after* a file is modified, up until some time in the last 5 minutes before it's modified again. This does not seem feasible, especially for numerous files like this. I would say that any constantly changing file should be excluded from check-summing. Side note: not changing the file modification timestamp is just plain wrong! Not that it matters in this case...

Fred, it sounds like you still have the v4 user and password files (passwd, smbpasswd, and shadow), which were supposed to be deleted and re-entered in ALL v4 to v5 upgrades, and of course before upgrading v4 to v6. That means you may not have upgraded to v6 correctly, and may still have other remnants of older versions, which are going to cause you more trouble. I recommend reviewing the instructions in the Upgrade or Clean Install section, and checking your boot drive files against Files on flash drive. There were only a few v4 config files that were safe to keep.

I didn't realize there was a workaround! Great! I must have missed it. I've added a note in blue to the Marvell bug thread, based on your answer. Can you check whether it's worded correctly, and do you have any further suggestions?