Messed up big time, tried adding a 2nd Parity Drive.

[RobinG]

I had one parity drive and decided to add another one, Stopped the array - new config and accidently selected a drive with data as the 2nd parity drive instead of the one that was empty. Now what?

[trurl]

New Config isn't even necessary to add parity2.

 

Since disk1 is unmountable, there must be more to this than you have told us. Please tell us more about what you did to get to the state in your screenshot.

[trurl]

And attach Diagnostics to your NEXT post in this thread.

[RobinG]

It was a disk that was in the array that I tried to move to parity.

skynet-diagnostics-20240128-1608.zip

[trurl]

Let me see if I can read between your lines to get the details I'm looking for.

 

The disk currently assigned as disk1, was originally assigned as disk1, but you accidentally assigned it as parity2.

 

Then, realizing your mistake, you stopped and corrected the disk assignments, so now, the disks are assigned correctly, but disk1 had been partially overwritten by parity2 when it was incorrectly assigned.

 

All that I just said above is an example of the kind of detail I am asking for. Maybe it's even correct.

 

If it is correct, say so.

 

If it isn't correct, please provide similar details that explains exactly what you did.

[RobinG]

My original array had one paity drive a 12tb, I wanted to add another parity dive the 12tb ending in MM. I moved a couple files off hard drive MM so I can use it as a parity drive. I created a new config and must have selected the incorrect old parity drive which must have been the 12tb ending in 76. I'm about 4 tbs of data lost now. Sorry if i'm not explaining well i'm freaking out a bit.

[trurl]

Just going through the days of syslog trying to piece this together. First thing I notice is it looks like newcache S62ANJ0RA16190B is going bad and has been for at least as far back as your last reboot on Jan 20. We can deal with that later.

 

But it does make me wonder about the health of your other disks. Which of your disks have SMART ( 👎 )warnings on the Dashboard page?

 

[RobinG]

i did this within the last hour, i did notice my bad nvme drive

[RobinG]

Just two drives both of the 500gb dirves in newcache pool

[trurl]

Before New Config

Jan 28 15:38:38 SkyNet kernel: md: import disk0: (sdj) ST12000VN0008-2YS101_ZV701678 size: 11718885324 
Jan 28 15:38:38 SkyNet kernel: md: import disk1: (sdb) ST12000VN0008-2YS101_ZRT08392 size: 11718885324 
Jan 28 15:38:38 SkyNet kernel: md: import disk2: (sdf) ST10000VN0008-2PJ103_ZS512LS4 size: 9766436812 
Jan 28 15:38:38 SkyNet kernel: md: import disk3: (sdl) ST12000VN0008-2YS101_ZR70R3MM size: 11718885324 
Jan 28 15:38:38 SkyNet kernel: md: import disk4: (sdk) ST4000VN008-2DR166_ZGY9GDA7 size: 3907018532 
Jan 28 15:38:38 SkyNet kernel: md: import disk5: (sdi) ST10000NE0008-2PL103_ZS50R8V6 size: 9766436812 
Jan 28 15:38:38 SkyNet kernel: md: import disk6: (sdc) ST8000VN0022-2EL112_ZA17XPEY size: 7814026532 

 

After New Config

Jan 28 15:42:56 SkyNet kernel: md: import disk0: (sdb) ST12000VN0008-2YS101_ZRT08392 size: 11718885324 
Jan 28 15:42:56 SkyNet kernel: md: import disk1: (sdj) ST12000VN0008-2YS101_ZV701678 size: 11718885324 
Jan 28 15:42:56 SkyNet kernel: md: import disk2: (sdi) ST10000NE0008-2PL103_ZS50R8V6 size: 9766436812 
Jan 28 15:42:56 SkyNet kernel: md: import disk3: (sdf) ST10000VN0008-2PJ103_ZS512LS4 size: 9766436812 
Jan 28 15:42:56 SkyNet kernel: md: import disk4: (sdc) ST8000VN0022-2EL112_ZA17XPEY size: 7814026532 
Jan 28 15:42:56 SkyNet kernel: md: import disk5: (sdk) ST4000VN008-2DR166_ZGY9GDA7 size: 3907018532 
Jan 28 15:42:56 SkyNet kernel: md: import disk29: (sdl) ST12000VN0008-2YS101_ZR70R3MM size: 11718885324 

 

Parity is disk0, parity2 is disk29

So these are the changed assignments at that point

disk0 -> disk1

disk1 -> disk0

disk2 -> disk3

disk3 -> disk29

disk4 -> disk5

disk5 -> disk2

disk6 -> disk4

 

Then parity sync began, overwriting disk0 (was disk1) and disk29 (was disk3)

 

To be continued...

 

[trurl]

Parity sync began

Jan 28 15:46:34 SkyNet kernel: mdcmd (36): check 
Jan 28 15:46:34 SkyNet kernel: md: recovery thread: recon P Q ...

and was cancelled 2:19 later

Jan 28 15:48:53 SkyNet kernel: mdcmd (37): nocheck cancel
Jan 28 15:48:53 SkyNet kernel: md: recovery thread: exit status: -4

 

To be continued....

[trurl]

Another New Config

Jan 28 15:50:46 SkyNet kernel: md: import disk0: (sdj) ST12000VN0008-2YS101_ZV701678 size: 11718885324 
Jan 28 15:50:46 SkyNet kernel: md: import disk1: (sdb) ST12000VN0008-2YS101_ZRT08392 size: 11718885324 
Jan 28 15:50:46 SkyNet kernel: md: import disk2: (sdi) ST10000NE0008-2PL103_ZS50R8V6 size: 9766436812 
Jan 28 15:50:46 SkyNet kernel: md: import disk3: (sdf) ST10000VN0008-2PJ103_ZS512LS4 size: 9766436812 
Jan 28 15:50:46 SkyNet kernel: md: import disk4: (sdc) ST8000VN0022-2EL112_ZA17XPEY size: 7814026532 
Jan 28 15:50:46 SkyNet kernel: md: import disk5: (sdk) ST4000VN008-2DR166_ZGY9GDA7 size: 3907018532 
Jan 28 15:50:46 SkyNet kernel: md: import disk29: (sdl) ST12000VN0008-2YS101_ZR70R3MM size: 11718885324 

So

disk0 <- disk1 <- disk0

disk1 <- disk0 <- disk1

disk2 <-  <- disk5

disk3 <-  <- disk2

disk4 <-  <- disk6

disk5 <-  <- disk4

disk29 <- <- disk3

 

If I understand correctly, disk3 being reused as parity2 was intended.

 

So, a little shuffled, but data disks in data slots and parity disks in parity slots. Except...

 

Disk1 spent some time being overwritten as parity and became unmountable.

 

Does all that sound correct?

 

[RobinG]

this all sound correct

[trurl]

appdata has files on both pools and all over the array, probably including disk1

domains share has files on the array and cache pool

system share all on cache pool

 

You should disable Docker and VM Manager until everything is fixed, including your newcache pool.

 

Do you have another copy of anything important and irreplaceable?

 

I think the only thing you can do at this point is continue with parity sync, then try to repair filesystem on disk1 and hope for the best.

 

Post new diagnostics when parity sync completes and we can work on the other things.

[RobinG]

i have no copies, i will update you after parity sync.

[trurl]

5 minutes ago, trurl said:

I think the only thing you can do at this point is continue with parity sync, then try to repair filesystem on disk1 and hope for the best.

In any case, continuing with parity sync won't make anything worse, since parity sync will only write parity and only read data disks. Might be a good idea if you don't write anything to your server.

 

4 minutes ago, RobinG said:

i have no copies

You must always have another copy of anything important and irreplaceable. You get to decide what qualifies.

 

Parity is not a substitute for backup. Plenty of ways to lose data besides failed disks, including user error.

[trurl]

Somewhat ironic that you were trying to increase security by adding another parity. It's debatable whether you even need parity2 with so few disks. I don't have parity2 on either of my servers.

[trurl]

Just now, trurl said:

It's debatable whether you even need parity2 with so few disks.

That extra disk might be better used as an Unassigned Device for backup of important and irreplaceable files.

[RobinG]

good idea!

[RobinG]

Got about two hours left on the parity-sync. I moved all the disks back to their previous slots.

[trurl]

3 hours ago, RobinG said:

moved all the disks back to their previous slots.

Do you plan to leave disk3 in the array now?

[RobinG]

yes, you are right i don't need a 2nd parity drive.

[RobinG]

xfs-repair output

[trurl]

No that's the wrong command. You have to use the md device or you will invalidate parity.

 

Do it from the webUI, it will use the correct command.

 

Try again.

[trurl]

Just in case you were looking at old documentation, here is the link: check filesystem on disk1

 

Unmountable disk3 had no contents, correct?