[SOLVED] already repaired xfs on unmountable disks but physical screen still shows xfs corruption detected


53 posts in this topic Last Reply

Recommended Posts

2 hours ago, trurl said:

If any error appears that test is done, no need to let it complete.

Thanks for the tip!

I found the offending ram stick, and moved on to the next one.

Test seems to be going a bit faster this way.

Memtest already shows “pass complete, no errors, press esc to exit” but it’s still running.

Do I need to wait? Seems like it’s starting another round of the tests.

743237FB-4BA1-404A-B318-A71A6816076C.jpeg

Link to post
  • Replies 52
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Yes, only 0 errors are acceptable, and even that not a guarantee there's aren't issues, but any errors is a guarantee there are.   Looks like you need an updated GLIBC, download this one, al

If any error appears that test is done, no need to let it complete.

just to point out that you can get scenarios where all sticks test fine individually, but you still get failures when all sticks are installed at the same time.

Posted Images

1 hour ago, itimpi said:


just to point out that you can get scenarios where all sticks test fine individually, but you still get failures when all sticks are installed at the same time.

Oh wow ok but I’m still testing the individual sticks tho. I’ve already found 1 with errors. I want to be sure the others are good.

 

I’m so surprised I had faulty ram my server was running 24x7 so well for months at a time. Well… I’ve had the occasional disk problem but I always assumed it was just my old hardware and drives. 
 

it should be ok to run with just 3 stick right? Just temporary until the ones I ordered arrives.

Link to post
15 minutes ago, JorgeB said:

It will keep running until you stop, if one pass found the error it should be enough, at least for now.

Yeah I stopped it and testing the next one now.

 

Aaaand as my luck would have it, I just noticed my bios wasn’t detecting one of my disks. I think it’s dead…😭😭 I had been shutting down and powering back up to test the ram. I guess it couldn’t take it.

The timing couldn’t have been worse 😭😭

I’m going to have to shrink my array for now until I can buy another drive…

 

Once memtest for the last stick is done I’ll scatter the contents from the (emulated) missing drive to the remaining drives, remove it and rebuild the parity.

 

should be quite safe to do since I’ve already found the bad ram stick, right?

Link to post
12 minutes ago, limawaken said:

should be quite safe to do since I’ve already found the bad ram stick, right?

*Should*, difficult to have more than 1 bad DIMM at the same time, but of course not impossible.

Link to post

with my current streak of misfortune i'd give that a high possibility....

one problem just led to another and another and another. i'm literally crying right now.

 

ok, my server is running on 3 legs now (1 faulty ram stick, 3 were ok).

missing disk as expected. it suffered from the click of death. it's a goner.

that weird xfs error message still scrolling on the screen.

unbalance doesn't seem to work, would've been great if it did. probably doesn't work when there's a missing disk?

so i'm moving folders from the emulated disk using krusader.

it seems much slower than normal... its progressing but the estimated time remaining doesn't seem to get shorter. transfer speeds ranging between 15MiB/s - 20MiB/s so its definitely going to be a while.

after transfer is done i'll rebuild parity... that'll take another day...

all this while that weird xfs error scrolls on the screen.. ughhh the stress...

i'm calling it a night for now.

 

thanks you guys for all your advice. just knowing that you guys are around here is SOOO reassuring. 

Link to post

parity rebuild completed, so it was time to try xfs_repair with the latest version xfsprogs.

i stopped array and started in maintenance mode, and ran xfs_repair -V (just to confirm it was the new version and it returned xfs_repair version 5.12.0)

the command i used was xfs_repair -v /dev/md1

it ran quite quickly and didn't appear to show any problems.

 

when i started the array back up again then those XFS (md1): Internal error rval errors started appearing again.

so it appears latest xfs_repair wasn't able to fix that.

 

it's very puzzling... disks and shares seem ok but my logs are getting flooded with this error.

 

I wanted to attach the diagnostics but unraid is still stuck at "downloading". i've restarted the browser and even tried private browsing.

Link to post
4 hours ago, limawaken said:

Internal error rval errors started appearing again.

Those look more like a xfs bug to me, you'd need to post in the xfs mailing list (or re-create that filesystem by formatting the disk).

Link to post

i've been thinking...

 

formatting the disk should clear up the xfs errors, right?

would it work if do this:

1. unassigned the disk

2. start array so unraid emulates the missing disk

3. format the emulated disk

4. use unassigned devices to copy everything back to the now empty emulated disk

5. re-assign the disk and unraid rebuild the disk

I've only ever seen the option to format an unmountable disk, so i don't know if this would work.

 

another idea i have would require a new spare disk:

1. assign new disk into the array

2. move all the data from disk1 over to new disk

3. new config - unassign new disk, assign new disk into disk1 slot

4. start the array and let parity rebuild (again)

 

Link to post

sorry @jorgeB i posted the last one just as you replied...

 

so i guess i'll go with the reformatting method. would my ideas work?

 

getting a spare disk looks like the better method, but would require to buy a new disk.

 

Link to post
1 minute ago, limawaken said:

so i guess i'll go with the reformatting method. would my ideas work?

Yes, you need to format the emulated disk before mounting the other one, it will be slow and the array will be unprotected during the procedure but you should have backups of anything important anyway so it's a way to avoid buying a new disk.

Link to post
1 hour ago, limawaken said:

@JorgeB where would i find the command to format the disk? since the gui doesn't detect the disk as unmountable, it doesn't give me the format option.

This is covered here in the online documentation that can be accessed via the Manual link at the bottom of the Unraid GUI.

Link to post

I forgot that since some time ago you can't change the fs on emulated disks, this was done to avoid users thinking that they can use that to change the filesystem of a disk while keeping the data, you can do this instead, start the array in maintenance mode then type:

 

wipefs -a /dev/md1

 

Then re-start array in normal mode and format the disk.

Link to post

Ah thanks.
 

I’ve actually unassigned disk1. Now it’s being emulated.

Then I thought to try mount using unassigned devices to make sure it could be accessed before I start formatting.

 

Is it normal that I can’t mount the disk with unassigned devices if I haven’t yet formatted the emulated disk? because it isn’t able to mount and the disk log shows something about file system has duplicate uuid then wrong fs type, bad option, bad superblock, codepage helper program or other error

 

is it because it’s because it is clashing with the emulated disk?

Link to post
23 minutes ago, limawaken said:

Is it normal that I can’t mount the disk with unassigned devices if I haven’t yet formatted the emulated disk?

You can't mount both at the same time, you can stop the array and mount the UD disk.

Link to post
1 minute ago, JorgeB said:

You can't mount both at the same time, you can stop the array and mount the UD disk.

I should be able to mount it after I wipe and format the emulated disk1?


Or am I only able to mount it when array is stopped?

Link to post
3 minutes ago, limawaken said:

I should be able to mount it after I wipe and format the emulated disk1?

Yes,

 

3 hours ago, JorgeB said:

you need to format the emulated disk before mounting the other one

You can't have the same filesystem mounted twice, after it's wiped/formatted it's no longer the same filesystem, UUID will be different.

Link to post

Sorry for asking so many questions. I must be such a pain right now…

I’m just trying to think how to copy the data back to the formatted emulated disk.

I can’t think of how I’d be able to do it if I can’t mount the disk when the array is started.

Link to post

ok that worked.

at first unraid wasn't able to format the emulated disk. unraid still showed unmountable disk and the option to format was still there even after several tries to format it.

 

i had to reboot then after that format was ok and now the emulated disk is empty.

 

then it just like you said - after wipefs and format unassigned devices was able to mount the disk.

 

no more "XFS (md1): Internal error rval..." errors!

 

then its weird because it suddenly appeared again ONCE but this time it was XFS (sdc1): Internal error rval....

but it only appeared just once... and i think it was after i started up dockers. (i re-enabled dockers after it looked like there were no more errors and the logs looked clean)

that was around half an hour ago, so far it hasn't reappeared.

 

what's sdc1? does this error have something to do with the dockers?

 

i've attached the diagnostics.

 

silometalico-diagnostics-20210620-2019.zip

Link to post
20 hours ago, limawaken said:

ut this time it was XFS (sdc1): Internal error rval....

That's expected since sdc is the old disk1 that is now mounted with UD.

Link to post

 

35 minutes ago, JorgeB said:

That's expected since sdc is the old disk1 that is now mounted with UD

ah i see... well it only appeared once, not like before.

so I guess we can say that xfs corruption error has been fixed!

 

now all the data has been copied from the old disk to the emulated disk and i'm now ready to assign the old disk back into its slot.

 

is there a correlation between all these disk errors and the installation of myservers plugin or unraid 6.9.2?

i've noticed several forum topics with similar issues. 

 

anyway thanks @JorgeB you're a true hero. cheers🍻

 

Link to post
1 minute ago, limawaken said:

is there a correlation between all these disk errors and the installation of myservers plugin or unraid 6.9.2?

Doubt it.

Link to post
  • JorgeB changed the title to [SOLVED] already repaired xfs on unmountable disks but physical screen still shows xfs corruption detected

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.