Pauven

I would highly recommend at least starting with the original slots where the replacement drives are being detected, remove the replacements and install the originals there. For now, don't touch any of the other "good" drives, as that could be compounding the problem, especially if you start to lose track of which drives are which. Keep it simple.

Okay, slow down. Up to this point, most of the advice has been either about doing tests or hypothesizing options. Anytime you think you're ready to take action, please post here your planned steps for review and approval. Anytime you take an action, you're one step closer to losing data if it is the wrong action. I believe your data is still intact, so don't give up hope. But slow down and work with the guys here, don't do anything that's not reviewed and approved. Did you follow this guidance to backup the current flash drive first, before restoring from backup? From a planning perspective, we need to know what options remain.

Hey guys, I'm just chiming in here as jkwaterman is a friend of mine, and we've already been chatting via email - I sent him here for expert advice. I'm super happy to see JorgeB, trurl and Frank1940 are helping out - you guys are sharp so I know he's in good hands. I read through everything, and I do have a few thoughts. Everything you guys are suggesting is pretty much a match for what I've advised via email as well, so we're all already on the same page. Restoring the super.dat from his Apr 2023 backup is a great idea, but I think that only applies if he didn't change any drives between the backup and before the drives failed out. If we send him down this path, I think he first needs to confirm he didn't upgrade/swap/add any drives post backup, and also he should have a new backup of his current (bad) config, in case this goes sideways and he wants to get back to the current state. I wanted to point this out since I didn't see anyone ask this particular question. I also strongly agree with trying to use the original failed drives, and that he should perform SMART tests to validate the drives are okay before re-using them. One thing I'm not sure about is if he uses the old drives, should he use the Trust Parity feature (I assume that's still a feature, been a decade since I last did this). I'm imagining that he's got two paths forward with the old drives. He could recreate the array config using all the original drives, and do a Trust Parity so it won't be rebuilt, and then immediately swap out the two suspect drives and rebuild onto the replacements. Basically, with this approach he's using the GUI to recreate the pre-failed drive config state, and then manually failing/upgrading the drives. Otherwise, he could again recreate the array config using all the original drives, but don't Trust Parity and instead rebuild new Parity via the data on the suspect drives. This second approach sounds slightly riskier, as we're trusting the suspect drives to survive the parity rebuild, and unfortunately we don't know the nature of the errors that started this whole fiasco. I know for a fact that he has started the array numerous times in disk emulation mode, so data could have been written to the array. Additionally we are both users of the My Movies software, which has a habit of updating local movie data from online web contributions that other users continually submit, and this metadata in turn gets written to the array. It's probably safe to assume that My Movies was running at some point during disk emulation mode, so that the current parity data no longer matches the data on the failed drives. I just wanted to point this out, so that we all know to only either trust the parity data, or trust the suspect drive data, but expect the two data sources to be slightly out of sync with each other. Note that the updates from My Movies are trivial and will automatically be reapplied if he reverts to the old drive data, so no risk of data loss there if he reverts to them. One question I had myself is: Is it possible to manually fix the drive config, via text editing, so that the parity drives are re-added to the array in a trusted state, but the 2 failed drives are still shown as missing/wrong/replaced? I was thinking there was a way to accomplish this via text file edits, but I really don't know. I helped with his server build. This power supply has 62A on +12V if I'm not mistaken. Thanks for helping jkwaterman out, guys, I know we both really appreciate it!!!

Thanks Rysz. Actually, it's my signature that's really outdated, hah! But I was still on 6.9.2, and I had to upgrade to 6.10+ even to use the URL method. I'm on the latest 6.12 now, and I was able to install from URL. I assume it's the same MergerFS release as the CA version. I like MergerFS, it's working as I hoped. But it's not perfect. The "Create" option is static for files vs. directories, and I was finding that it would create a directory, write some files to it, drop below the minimum free space, and then create a new directory on a different branch. Considering that I'm backing up uncompressed blu-rays, typically around 45 GB in size, I need the min free space for creating a directory to be at least 45 GB higher than the min free space for creating files. To solve this, I customized the mirror.sh script someone else wrote (which is used to create each directory right before files are written to it, rather than creating all empty directories first and then copying files). I changed it to have it create directories based upon 100 GB min free space, and to evaluate my MergerFS branches in a particular sequence. I then was able to configure MergerFS with a much lower 4 GB min free space, which only applies to files since my script creates the directories. When used with MergerFS's "ep" Existing Path option, I now have MergerFS writing the backup files to where my backup script creates the directories. This allows me to keep my blu-ray disc directories whole on a single drive, and all my MergerFS branches fill up one-by-one. I'm in backup nirvana!!!

A year ago I created an easy, affordable backup solution for my Unraid server. Essentially just a stack of external USB drives that I mounted with Unassigned Devices and joined together in a BTRFS JBOD style pool. With 5x 16TB drives, this gave me a single 80TB storage volume. At the time, this solution seemed perfect. I had a backup script that used RSYNC to copy my files to the single mount point, and I thought that BTRFS filled up each drive one-by-one. Since my Unraid data is basically already a backup of my physical data, having this portable backup volume that could be stored offsite was more than I needed, even without any built-in redundancy. This week, while adding a new 20TB drive to expand this pool up to 100TB, I learned I made several mistakes in my backup solution. First, when adding the new drive I made a few mistakes and ended up corrupting the BTRFS pool. And since my pool had no redundancy, BTRFS prohibits mounting it in RW mode to fix it, so the only option was to start over, recreate the entire pool, and re-backup the original 80TB of data. That was painful enough. But in redoing all this, I discovered that BTRFS is automatically balancing, writing to the drive with the most free space for each file. With the nature of the data I'm storing, losing a single drive would now make the entire backup worthless as I need each directory to remain whole on a single drive, and can't lose any files inside each directory. While my BTRFS backup pool is better than nothing, this is way too fragile for me to continue using it. While researching solutions, I came across MergeFS and eventually this thread. This sounds like the right type of solution. My core requirements are to plug in my USB drives, mount them as a single filesystem, and run a backup script to copy any new/altered data to my backup pool, with data filling up each drive, one-by-one, before moving on to the next drive. That way, if I lose a drive, I only lose the data backed up to that one drive, plus any directories that happened to be spanning the transition between drives. Sorry for the long lead-in. Now to my questions: Is the plugin on CA yet? I searched and can't find it, so I'm assuming I have to install it via URL. Can someone help me with the configuration? I read through the MergerFS github page, and there's tons of options and the examples don't seem to apply to my use case. I'm a bit overwhelmed. I need commands for configuring, mounting, unmounting, and expanding the pool. Thanks! -Paul

I did see that, but then you appended with your edit and I thought you were changing your answer, hence my confusion. I currently have 78.6 TB of data backed up in this pool, as-is. If I follow those steps, is there any risk I could lose that data and have to repopulate the back-up? It was over a week of copying, I don't want to have to do that again. If I'm understanding you correctly, I can remove the 5 disks, delete the history, then 1 at a time insert the disk and rename it to the same pool name, delete my history again just to make sure, and then the next time I bring in all 5 drives at the same time, they will appear as a single pool. Does that sound right?

So does that mean this isn't possible? Sorry, I got confused. Since the mount point has to be the disk label, and it won't let me rename to an existing value, that makes it impossible to do the solution you offered, right?

I just tried changing the Mount Point to all be the same, and it won't let me. It reports "Fail". I think it is because it's changing the disk label and the mount point at the same time. Is there a trick to doing this? Errors in the log: Apr 10 17:24:51 Tower unassigned.devices: Error: Device '/dev/sdx1' mount point 'Frankenstore' - name is reserved, used in the array or by an unassigned device.

Thanks dlandon.

Hey dlandon. Thanks so much for this awesome tool. I've been using it for years, and it's been a big help for certain tasks. One of the things I occasionally use it for is for a removable btrfs JBOD drive pool of 5 USB HDD's. It's so easy to plug it in, mount it, run my rsync backup job, then put it back in offline storage when I'm done. I love the fact I don't have to stop/start the array to use it, that I don't get warnings when I unplug it, and that I don't get any fix common problems warnings for duplicated data on a cache disk. I was recently sharing my solution with some fellow users, and I discovered that the tutorial for how to create the btrfs drive pool for UD was removed. I reached out to JorgeB and he restored that post so now I have those instructions again. While working with these other users on how to do this hot-pluggable backup pool, and comparing with how it works using stock Unraid pools, a few things cropped up that I wanted to ask you about. After all, UD is the best tool for creating hot-pluggable drive pools that are normally stored offline, but there are a couple things Unraid pools do a bit better. First, when mounting the 1st pool device, the buttons to mount the other devices remain enabled. One of my fellow users got confused, and clicked mount on all devices, and then saw the pool was mounted multiple times. Would it be possible to both make it more obvious that all the drives in the pool are now mounted, and to disable/hide the mount button on the other drives? Currently the only indication is the partition size on the mounted drive. Perhaps even the other drives that got mounted in the pool can be inset to the right, beneath the parent, to better indicate what is going on. Second, would it be possible to add a feature in the GUI to add a partition to an existing pool? I believe that Unraid pools let you do this, but in UD you have to go out to the command line and do the btrfs dev add... command to add the partition to a mount point. I know it's a pretty easy command line, but some users are very uncomfortable with the cmd line and prefer the GUI approach. I know most people seem to think that Unraid pools are the only game in town now, even your own documentation states to use them. But for hot-pluggable, removable drive pools, UD is so much better, I hope you continue to support and enhance this capability. Thanks!!! Paul

Awesome, thank you Johnnie (should I still call you Johnnie, or Jorge, or something else?), that's exactly what I needed. I was surprisingly close in my recreation of the steps based upon my research, but was full of doubt. You're extremely helpful as always. 😊 Another user did a test and discovered he was able to mount a pool created in Unraid using UD, no big surprise I guess since these are just standard btrfs pools. So for some users it might be easier to create the pool using Unraid, remove it, delete the definition, and then use UD from then on for hot-plugging. I would definitely use the Unraid pools feature if it more gracefully handled hot-pluggable backup pools, and didn't require the stop/start. I'm not complaining, though, since UD does this extremely well.

Hey Johnnie/@JorgeB, I could use some help on this. Side note, your new username and logo had me all confused, I couldn't figure out how you seemed to have been here for years/decades, yet I didn't recognize the name. I finally figured out your provenance, though I'm still baffled by the user name change. Anyway, to my issue. I created a portable backup drive pool, as described above, with Unassigned Devices back when I was running 6.8, using the directions you linked to above. I plug it in 2-4 times a year and do a backup, it's fantastic. Those instructions (which I think you wrote) have since been deleted, since the preferred way is to use the multiple drive pools feature in 6.9. But the functionality in 6.9 is not the same. If you create a drive pool and then unplug it, Unraid is unhappy about the missing drives. You can make the warnings go away if you delete the pool, but then you have to make sure you add recreate the pool with all the drives back in the correct order before doing your next incremental. You also have to stop/start the array to do any changes to the pool. Unassigned Devices did this particular task so much better. No warnings, don't have to delete the config, just plug it in and mount it, don't have to stop the array. While I can understand that the preferred method is to use Unraid for multiple permanent drive pools, I don't understand why the documentation for doing it with UD was deleted, as that still serves a niche. I'm trying to help some other users get up and going with the solution I'm using, and since I can't find the documentation I can't fully help them. I think there were some command lines I used when setting up the btrfs pool as jbod, possibly related to formatting but I don't recall. I also need to expand my UD backup drive pool soon, almost ran out of space on my last backup so I need a 6th drive, and I'm worried I won't be able to do this correctly without the instructions. Even the UD support thread points to the now deleted instructions, and the internet archive doesn't have any successful copies of the FAQ. Is this something you can help with, or point me to someone who can? Thanks! Paul

Big thanks to @Cessquill for the write-up, and for all the other contributors to this thread! I just successfully fixed this on my server. I finally took the plunge after waiting half a year, as I started having major compatibility issues with Unassigned Devices on 6.8.3, and couldn't put off upgrading anymore. I followed all the steps that Cessquill outlined, and disabled EPC on my four Seagate ST8000NM0055 drives. I used the latest SeaChest Utility files from Seagate's website, downloaded yesterday. They appear to have changed again, and this time I used the files from path: \Linux\Non-RAID\centos-7-x86_64\ Oddly different than @optiman's experience with his ST8000NM0055's, mine all had Low Current Spinup disabled, so I didn't mess with that. I'm also running on a Marvel based controller, which probably creates a unique data point that this issue doesn't just affect LSI controllers. Last time I upgraded to 6.9.x, I had major issues, and could not get beyond 66GB of my parity rebuild, which is why I rolled back to 6.8.3. After applying the EPC fix and upgrading to 6.9.2, I've done multiple drive spindown's/spinup's with no issues, and a full parity check which completed in record time. It's perhaps too early to celebrate, but it does seem like the issue is resolved on my setup. I also have two pre-cleared ST8000VN0022's that are not in my array. They had both EPC and Low Current Spinup (Ultra Low) enabled. I decided to leave Low Current Spinup alone, but went ahead and disabled EPC for both of these drives. These will migrate into my array in the coming months, so I don't know yet how they'll behave. I don't even know if I would have had issues with them, but since other users here mentioned them I decided to play it safe. I also used SeaChest_Info to examine my other non-Seagate drives (surprisingly it works), and found that EPC exists and is enabled on my HGST_HUH728080ALE drives, but those don't cause any problems. I kinda hate that these Seagate Exos 8TB drives are such a good value, as they've become my chosen upgrade path, so now I'll have to remember to disable EPC on all new drives going forward. While I do like the HGST drives better, the price premium is just too much for a server this large. Thanks again!!! -Paul

If this is true, then why does the problem only appear after upgrading to Unraid 6.9.x? I'd been running on 6.8.3 for a long time without issues. Last spring I upgraded to 6.9.3 and bam! the issues hit immediately. I never did the EPC fix. The problem was incessant on 6.9.x, and I didn't want to risk loosing data playing around with drive setting as I had 2 drives out and was already risking data loss, so I rolled back to 6.8.3, and the problem went away. Half a year later and it's been smooth sailing on 6.8.3. I stayed on 6.8.3 because it works and there wasn't anything in the 6.9.x branch I'm needing. Even the very first post here mentions that the problems started with 6.9.0, which 100% matches my experience. Perhaps what you are saying is that the problem lies in the Linux kernel or one of the various drivers that were upgraded in the 6.9.x releases, and the issue is not in any of LimeTech's Unraid code. That may be true, though I'm not sure I've seen it clearly detailed in this thread exactly where the problem lies, so I would appreciate pointers to any additional information I may have missed. It certainly seems reasonable to me that since a change in 6.9.x broke this, another change in 6.10.x could fix it, so I'm not inclined to give up hope entirely. And there have been many times LimeTech has chased down bugs in other components on behalf of their users - and this issue has been reported to them in more than one ticket so they should be aware of it, though disappointingly I've never seen them weigh in on the topic.

I can confirm that ST8000NM0055 drives are most definitely affected by this issue. This bit me hard when I upgraded to v6.9.2 back in April. I had to roll back to 6.8.3 to recover from a dual-drive "failure" and inability to rebuild on 6.9.2, and never attempted any of the fixes posted here. I felt extremely lucky to escape without losing data, and I'm still running 6.8.3. optiman, glad to read this worked for you. Since it has been a couple weeks, is your system still okay? I'm starting to feel a little trapped on 6.8.3, so I'll probably have to apply this fix. Since we both have ST8000NM0055 drives, your results matter most to me. I was hopeful that this was a bug in 6.9.x that would be fixed in 6.10, and that I wouldn't need to do the drive fix. Came here to see if anyone had tested this on 6.10 without applying these fixes, but no dice. Paul