A week or two ago I upgraded to 6.6.7, I was on a previous version 6.something don't remember what exactly.

 

This past week I realized how low on space I am so I decided to increase my space. I decided to replace one of my two 3TB parity drives with a 4TB drive, and then replace one of my 1TB data drives with the old 3TB parity drive. I followed the directions outlined here: https://wiki.unraid.net/The_parity_swap_procedure

This is what I did:

1. Stop the array

2. Unassign the old drive - This is the 1TB data drive

3. Stop the array again

4. Power down

5. Pull the old drive - I removed the 1TB drive and installed the 4TB in it's spot

6. Power on

7. Stop the array

8. Unassign the parity drive - I unassigned one of the 3TB parity drives

9. Assign the new drive in the parity slot - I assigned the 4TB drive to the parity spot just vacated

10. Assign the old parity drive in the slot of the old data drive being replaced - I assigned the 3TB to the old 1TB data drive spot

11. Go to the Main -> Array Operation section

You should now have a Copy button, with a statement indicating "Copy will copy the parity information to the new parity disk".

Put a check in the Yes I want to do this checkbox (older versions: Yes, I'm sure), and click the Copy button

Now patiently watch the copy progress, takes a long time. All of the contents of the old parity drive are being copied onto the new drive, then the remainder of the new parity drive will be zeroed.

 

I started the copy and left it. 16 hours later it had copied 1% of the old parity drive to the new one. I checked the syslog and there were a bunch of errors.  Thinking I might not have put the new drive in it's slot firmly enough I tried to stop the copy but the system didn't respond at all to that request. Finally I just shutdown the machine and reseated the drive.  When I brought it back up it had both 3TB parity drives in their original spot and the 1TB data drive unassigned. At this point I was quite nervous about the whole thing. I decide to try something else. I have in the array a 3TB unassigned spare drive for emergencies. I decided to rebuild the missing 1TB drive onto that, so I assigned that drive to the slot (didn't do anything with the parity or the new 4TB drive I had installed) and started a rebuild. It got to about 90% before I went to bed. When I woke up the machine was unresponsive. I checked the monitor connected to it and found a Kernel panic and out of memory error. I have 4GB of RAM in the machine. I know it's not much compared to some builds but it's always been enough. I have rebuild drives in the past and rebuild parity in the past and that has always been enough memory. During the rebuild the memory usage I saw never rose above 30%.

 

 

After the kernel panic I had no choice but to reboot the machine and it came back up with just the one data drive missing (the previous 1TB one). Freaking out at this point I figured I'd just return my system to the way it was before I started on this misadventure. I shutdown the machine took out the 4TB drive (which was to be the new parity drive) and put back the 1TB data drive (that had no issue other than it was one of my smallest drives). I booted up the machine, as I expected since I unassigned the 1TB data drive, it showed up as missing. When I reassigned the 1TB drive back into it's spot it wanted to rebuild it. Considering the problems with the last rebuild and knowing the data on the drive was fine I didn't want to do that. I took a gamble, I used the New Config tool to accept all the data drives as good (I had completed a parity check just a week prior so I was confident the data was good) knowing I would have to rebuild the two parity drives. I did that.  All data drives were back in their places and parity was rebuilding. The two parity drives were rebuilding at an okay pace, the cpu was maxed but the memory usage was not so I had hopes it would work. The estimate was 11 hours so I went to bed again. The next morning (this morning) it was still running, memory usage was fine. I left the house to run some errands and hoped to come back to a good (if crowded) array. I came back to an unresponsive (page not found) system and this on the monitor:

 

Unless I am missing something it feels like at the end of a disk rebuild or parity rebuild there is something that makes a huge memory grab? I have ordered 2x8GB sticks for my machine, it's the most it can handle, but I have a scary thought that it might not help.

 

I am seriously losing it, going out of my mind with panic. Please help. Please.

 

With nothing else to do I am attempting another parity rebuild:

 

 

I am however sure it will fail in the same way.

 

Help me Obi Wan, you're my only hope.

 

tower-diagnostics-20190504-0945.zip

Edited May 7, 20197 yr by technomancer
Mark solved

No reason to attempt a parity swap in this situation. You should have just replaced parity with a larger disk and let it rebuild, then replaced the data drive with the old parity and let it rebuild. Parity swap is only needed when you want to replace a failing data disk while upsizing parity. Since you didn't have any failing disk you could have just taken the simpler replace each one at a time.

 

Haven't got time to take a longer look at this right now. Will come back later if nobody else has.

I think your way (which I agree is better) would have resulted in the same error I am facing now on the rebuild.

I agree your problem is not due to attempting parity swap.

 

I think where you are now looks hopeful. The screenshot is showing all disks mounted and mostly full. You should consider getting your disks converted from ReiserFS after you get things square and have some extra capacity.

 

DIsk7 needs to be replaced as it has quite a few pending sectors. Possibly this is causing some of your problems. No idea what might be causing the OOM though, rebuilding shouldn't really consume any memory.

 

Since your diagnostics are after the reboot can't see what might have led up to the dump. See this sticky for some ideas on how you might get us more information from before the (next) crash:

 

https://forums.unraid.net/topic/37579-need-help-read-me-first/

 

Thanks 😊 I installed the Fix Common Problems plugin. When I run it it sits at scanning forever... I think it's because all my cpu is taken up by the current parity rebuild.

 

Should I stop the rebuild, run the plugin, then start the (doomed to fail) parity rebuild?

 

I am desperate, thanks for the help!

 

Wait: it looks like it did finish scanning. What should I do with it to capture more info before the next panic?

 

NM: I found and started trouble shooting mode. Sorry for being a spaz, just panicking.

Edited May 4, 20197 yr by technomancer
Update

You can stop it if you don't want to wait. FCP Troubleshooting Mode is one good way to get syslogs from before a crash.

 

16 minutes ago, trurl said:

DIsk7 needs to be replaced

Curious why this isn't the disk you intended to replace. You should see SMART warnings for that disk on the Dashboard. You shouldn't ignore these. All bits of all disks must be reliably read in order to reliably rebuild a disk.

 

If you just didn't notice this disk problem, then you must not have Notifications configured. You must configure Notifications to notify you immediately by email or other agent as soon as Unraid detects a problem.

 

If you had asked our advice before doing anything, we probably would have actually recommended the parity swap, but for disk7 instead of one of your 1TB disks.

Okay. Will do. Turn on notifications, replace disk 7 first.

Attached is the syslog tail from Fix Common Problems. I hope this helps.

FCPsyslog_tail.txt

Still not sure of the cause.  Uninstalled Dynamix Cache Dirs plugin and am running a parity rebuild. Who knows...

Did not work. Later syslog tail attached. Please any help, please.

FCPsyslog_tail2.txt

You only have 4GB RAM, but very few plugins, no dockers or VMs, so I don't understand what is causing the OOM.

 

Maybe try rebooting in SAFE mode, New Config without the suspect disk7, and see if you can rebuild parity. Then if you can get the array stable that way we can see about getting more capacity and try to copy the data from that disk with it out of the array.

Rebooted in safe mode with gui. Doing a parity rebuild now. Super desperate.

1 hour ago, technomancer said:

Rebooted in safe mode with gui. Doing a parity rebuild now. Super desperate.

Did you unassign disk7?

 

I don't think your situation is that bad. All your other disks with their data should be fine, and probably most if not all data from disk7 will be OK too.

 

21 hours ago, technomancer said:

I upgraded to 6.6.7, I was on a previous version 6.something don't remember what exactly.

Can you take a guess at how long ago you installed the previous version?

Do you have backups of anything important and irreplaceable?

Thank you for sticking with me.

 

I did not unassign disk 7, I want to do this one step at a time. First try and rebuild parity in safe mode. I'll look at removing disk 7 if this fails.

 

I am sorry, I let my version get a little olde before I updated it. I think it was on a 6.5 something when I went to 6.6.7.

 

2 hours ago, trurl said:

Do you have backups of anything important and irreplaceable?

 

Unraid is not a backup solution, we should all know that. There are no family photos on my array. That being said I have put thousands of hours into my data collection and would be devastated to lose it. The simple truth is I can't afford a backup solution of the size required to backup all my stuff. I don't think I am the only person in that situation. I use unraid as it is more fault tolerant than individual drives by themself. It's the best that I can do.

 

Back to the problem at hand.  Does anyone have a thorough understanding of the parity or drive rebuild process? I have watched the parity rebuild run just fine for 8 of the estimated 11 hours without issue. The OOM problem does not happen at the beginning of the rebuild, it does not happen at the middle of the rebuild. Something changes near or at the end of the process, maybe a different service is called, or a calculation is run at or near the end of the rebuild that causes this OOM error.

Edited May 5, 20197 yr by technomancer
Update

Shouldn't be anything sufficiently different at the end vs earlier to cause OOM. The parity calculation is very simple and doesn't change throughout.

 

Your last syslog tail showed the dump occurring a few minutes after midnight. Do you have anything scheduled?

Only the mover at the default time, 3:40 I think.

 

Should I go to a previous version of unraid? I'm not sure how to do that... I never had this problem with previous versions...

I doubt this is due to the version. I was just trying to determine if you were previously using a version before 6.4 because there are some additional things that may need to be taken care of when upgrading from an older version.

 

How is the rebuild going now?

3 hours left...

 

Thanks again for your help with this. I really do appreciate it.

Woohoo!!!!!!!

 

Finally after 3 days I have valid parity and all my data disks in the array!

 

So it seems the problem is a plugin. What do I do now? Do I just run in safe mode? Is there a way to do a clean install of the plugins I have or something?

If you put your flash in your PC and delete a plugin's .plg file from config/plugins, then the plugin won't be installed when booting.

So delete them all and boot into normal mode?

Or you could just rename the config/plugins folder and none would load.

 

I don't know which plugin might be to blame though. You don't have many and those you do have are used by a lot of people without problems.

 

If you are only running a NAS though you probably don't need any.

Thanks. Thanks so much.