[SOLVED] already repaired xfs on unmountable disks but physical screen still shows xfs corruption detected


Recommended Posts

2 hours ago, trurl said:

If any error appears that test is done, no need to let it complete.

Thanks for the tip!

I found the offending ram stick, and moved on to the next one.

Test seems to be going a bit faster this way.

Memtest already shows “pass complete, no errors, press esc to exit” but it’s still running.

Do I need to wait? Seems like it’s starting another round of the tests.

743237FB-4BA1-404A-B318-A71A6816076C.jpeg

Link to comment
1 hour ago, itimpi said:


just to point out that you can get scenarios where all sticks test fine individually, but you still get failures when all sticks are installed at the same time.

Oh wow ok but I’m still testing the individual sticks tho. I’ve already found 1 with errors. I want to be sure the others are good.

 

I’m so surprised I had faulty ram my server was running 24x7 so well for months at a time. Well… I’ve had the occasional disk problem but I always assumed it was just my old hardware and drives. 
 

it should be ok to run with just 3 stick right? Just temporary until the ones I ordered arrives.

Link to comment
15 minutes ago, JorgeB said:

It will keep running until you stop, if one pass found the error it should be enough, at least for now.

Yeah I stopped it and testing the next one now.

 

Aaaand as my luck would have it, I just noticed my bios wasn’t detecting one of my disks. I think it’s dead…😭😭 I had been shutting down and powering back up to test the ram. I guess it couldn’t take it.

The timing couldn’t have been worse 😭😭

I’m going to have to shrink my array for now until I can buy another drive…

 

Once memtest for the last stick is done I’ll scatter the contents from the (emulated) missing drive to the remaining drives, remove it and rebuild the parity.

 

should be quite safe to do since I’ve already found the bad ram stick, right?

Link to comment

with my current streak of misfortune i'd give that a high possibility....

one problem just led to another and another and another. i'm literally crying right now.

 

ok, my server is running on 3 legs now (1 faulty ram stick, 3 were ok).

missing disk as expected. it suffered from the click of death. it's a goner.

that weird xfs error message still scrolling on the screen.

unbalance doesn't seem to work, would've been great if it did. probably doesn't work when there's a missing disk?

so i'm moving folders from the emulated disk using krusader.

it seems much slower than normal... its progressing but the estimated time remaining doesn't seem to get shorter. transfer speeds ranging between 15MiB/s - 20MiB/s so its definitely going to be a while.

after transfer is done i'll rebuild parity... that'll take another day...

all this while that weird xfs error scrolls on the screen.. ughhh the stress...

i'm calling it a night for now.

 

thanks you guys for all your advice. just knowing that you guys are around here is SOOO reassuring. 

Link to comment

parity rebuild completed, so it was time to try xfs_repair with the latest version xfsprogs.

i stopped array and started in maintenance mode, and ran xfs_repair -V (just to confirm it was the new version and it returned xfs_repair version 5.12.0)

the command i used was xfs_repair -v /dev/md1

it ran quite quickly and didn't appear to show any problems.

 

when i started the array back up again then those XFS (md1): Internal error rval errors started appearing again.

so it appears latest xfs_repair wasn't able to fix that.

 

it's very puzzling... disks and shares seem ok but my logs are getting flooded with this error.

 

I wanted to attach the diagnostics but unraid is still stuck at "downloading". i've restarted the browser and even tried private browsing.

Link to comment

i've been thinking...

 

formatting the disk should clear up the xfs errors, right?

would it work if do this:

1. unassigned the disk

2. start array so unraid emulates the missing disk

3. format the emulated disk

4. use unassigned devices to copy everything back to the now empty emulated disk

5. re-assign the disk and unraid rebuild the disk

I've only ever seen the option to format an unmountable disk, so i don't know if this would work.

 

another idea i have would require a new spare disk:

1. assign new disk into the array

2. move all the data from disk1 over to new disk

3. new config - unassign new disk, assign new disk into disk1 slot

4. start the array and let parity rebuild (again)

 

Link to comment
1 minute ago, limawaken said:

so i guess i'll go with the reformatting method. would my ideas work?

Yes, you need to format the emulated disk before mounting the other one, it will be slow and the array will be unprotected during the procedure but you should have backups of anything important anyway so it's a way to avoid buying a new disk.

  • Thanks 1
Link to comment

I forgot that since some time ago you can't change the fs on emulated disks, this was done to avoid users thinking that they can use that to change the filesystem of a disk while keeping the data, you can do this instead, start the array in maintenance mode then type:

 

wipefs -a /dev/md1

 

Then re-start array in normal mode and format the disk.

  • Thanks 1
Link to comment

Ah thanks.
 

I’ve actually unassigned disk1. Now it’s being emulated.

Then I thought to try mount using unassigned devices to make sure it could be accessed before I start formatting.

 

Is it normal that I can’t mount the disk with unassigned devices if I haven’t yet formatted the emulated disk? because it isn’t able to mount and the disk log shows something about file system has duplicate uuid then wrong fs type, bad option, bad superblock, codepage helper program or other error

 

is it because it’s because it is clashing with the emulated disk?

Link to comment
3 minutes ago, limawaken said:

I should be able to mount it after I wipe and format the emulated disk1?

Yes,

 

3 hours ago, JorgeB said:

you need to format the emulated disk before mounting the other one

You can't have the same filesystem mounted twice, after it's wiped/formatted it's no longer the same filesystem, UUID will be different.

  • Thanks 1
Link to comment

ok that worked.

at first unraid wasn't able to format the emulated disk. unraid still showed unmountable disk and the option to format was still there even after several tries to format it.

 

i had to reboot then after that format was ok and now the emulated disk is empty.

 

then it just like you said - after wipefs and format unassigned devices was able to mount the disk.

 

no more "XFS (md1): Internal error rval..." errors!

 

then its weird because it suddenly appeared again ONCE but this time it was XFS (sdc1): Internal error rval....

but it only appeared just once... and i think it was after i started up dockers. (i re-enabled dockers after it looked like there were no more errors and the logs looked clean)

that was around half an hour ago, so far it hasn't reappeared.

 

what's sdc1? does this error have something to do with the dockers?

 

i've attached the diagnostics.

 

silometalico-diagnostics-20210620-2019.zip

Link to comment

 

35 minutes ago, JorgeB said:

That's expected since sdc is the old disk1 that is now mounted with UD

ah i see... well it only appeared once, not like before.

so I guess we can say that xfs corruption error has been fixed!

 

now all the data has been copied from the old disk to the emulated disk and i'm now ready to assign the old disk back into its slot.

 

is there a correlation between all these disk errors and the installation of myservers plugin or unraid 6.9.2?

i've noticed several forum topics with similar issues. 

 

anyway thanks @JorgeB you're a true hero. cheers🍻

 

Link to comment
  • JorgeB changed the title to [SOLVED] already repaired xfs on unmountable disks but physical screen still shows xfs corruption detected

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.