Disk error, what do i do now?

[JorgeB]

If xfs_repair doesn't work, and like it won't if it didn't by now, there's one last thing you can try, and you need to do that to run a file recovery util anyway, rebuild the disk as is, then run xfs_repair on the sd device (xfs_repair -v /dev/sdx1), the result will likely be the same but it's just to rule out any md driver issues, there have been a couple of times where it did make a difference.

[nmills3]

ok, i'll let xfs repair finish anyway (can't hurt to let it try) and if it doesn't work i'll put the existing drive into a different slot and do what you've said. i'm assuming i need to replace /dev/sdx1 with whatever the drive got assigned with a 1 on the end

[JorgeB]

Just now, nmills3 said:

i'm assuming i need to replace /dev/sdx1 with whatever the drive got assigned with a 1 on the end

Correct, don't forget the 1 in the end or it won't work.

[nmills3]

Sorry, could not find valid secondary superblock

[nmills3]

ok, the rebuild actually seems to be doing something. so after this is done should i stop the array and restart it in maintenance mode for the xfs repair or should i just do it in normal started mode?

[nmills3]

so i just woke up and have a very confusing message

[nmills3]

i think the notification was just wrong because in the rebuild history it shows this 

[JorgeB]

Post new diags please.

[nmills3]

tower-diagnostics-20221201-1059.zip

[nmills3]

the repair tool yielded the same result on a normal drive so i'm currently running UFS professional on it

[JorgeB]

6 minutes ago, nmills3 said:

the repair tool yielded the same result on a normal drive

You mean running xfs_repair on /dev/sdx1 correct? Unfortunately that was the most likely result but still worth a try, the strange part is that clearly there's an XFS filesystem on that disk:

 

Nov 30 20:35:46 Tower kernel: XFS (md3): Mounting V5 Filesystem
Nov 30 20:35:47 Tower kernel: XFS (md3): Metadata CRC error detected at xfs_inobt_read_verify+0x12/0x5a [xfs], xfs_inobt block 0x18
Nov 30 20:35:47 Tower kernel: XFS (md3): Unmount and run xfs_repair
Nov 30 20:35:47 Tower kernel: XFS (md3): First 128 bytes of corrupted metadata buffer:
Nov 30 20:35:47 Tower kernel: 00000000: 11 00 04 7f 00 00 00 fc 57 de 24 68 e1 01 a6 03  ........W.$h....
Nov 30 20:35:47 Tower kernel: 00000010: 80 45 1a e6 46 a0 a4 53 00 00 00 01 00 0a 40 9d  .E..F..S......@.
Nov 30 20:35:47 Tower kernel: 00000020: f0 ac fe df 81 ba bd 46 8d 48 33 94 94 df 43 5d  .......F.H3...C]
Nov 30 20:35:47 Tower kernel: 00000030: 00 00 00 0b d3 5d 74 2d ff ff ff 7f ff ff bf ff  .....]t-........
Nov 30 20:35:47 Tower kernel: 00000040: ff ff ff ff ff ff ff ff ff ff fa 7f ff ff bf ff  ................
Nov 30 20:35:47 Tower kernel: 00000050: ff ff ff ff ff ff ff ff ff ff fa 3f ff ff bf ff  ...........?....
Nov 30 20:35:47 Tower kernel: 00000060: ff ff ff ff ff ff ff ff ff ff f9 ff ff ff bf ff  ................
Nov 30 20:35:47 Tower kernel: 00000070: ff ff ff ff ff ff ff ff ff ff f0 bf ff ff bf ff  ................
Nov 30 20:35:47 Tower kernel: XFS (md3): metadata I/O error in "xfs_btree_read_buf_block.constprop.0+0x7a/0xc7 [xfs]" at daddr 0x18 len 8 error 74
Nov 30 20:35:47 Tower kernel: XFS (md3): Failed to read root inode 0x80, error 117
Nov 30 20:35:47 Tower root: mount: /mnt/disk3: mount(2) system call failed: Structure needs cleaning.

 

But it's not the first time this happens, I guess it might depend on what's damaged or the kind of damage, UFS explorer is probably the best bet to try and recover some data, other option would be contacting the XFS mailing list to see if they can revive the fs.

[nmills3]

well i'll update here with the results of my file recovery when that finishes in about 6 hours

[nmills3]

also for the future, is there anything (other than the obvious of having backups) to try and stop this kind of issue happening again?

[JorgeB]

Not really, if you can't have a backup of everything try at least to have of the most important stuff.

[nmills3]

do you know of any good apps to keep a log of what is on each drive? because my other issue is that i don't even know what half the stuff on this drive is. I know what some of the folders/files i lost were, but i have no idea what most of the files on it were and probably won't know until i try and find something and it's missing. I need something that just writes a file that says what file paths are on each disk like once a week

[nmills3]

so I ran ufs and it seems to have found all the files. but the version i used won't let me copy files bigger than 768KB without getting a licence that costs over £500. should i be using a different version?

[nmills3]

for now i'm going to run the standard version of the ufs software on the drive and see if that can find all the same files. if it can, i'll just buy a licence for that version

[nmills3]

ok, UFS standard seems to be recovering either all or most of the files. so now that the lost data part is sorted, what do i need to get disk 3 in my array to a working state? should i just start the array and use the format unmountable drives button?

[JorgeB]

Yes, when the data recovery is finished just format the disk.

[itimpi]

11 hours ago, nmills3 said:

for now i'm going to run the standard version of the ufs software

For future reference of anyone else following this thread I believe that you would only need the (very expensive) pro version of the software if you are using encrypted drives.   The standard version should work for all non-encrypted drives.

[nmills3]

On 12/1/2022 at 7:31 PM, nmills3 said:

do you know of any good apps to keep a log of what is on each drive? because my other issue is that i don't even know what half the stuff on this drive is. I know what some of the folders/files i lost were, but i have no idea what most of the files on it were and probably won't know until i try and find something and it's missing. I need something that just writes a file that says what file paths are on each disk like once a week

Does anyone know if something like this exists

[Kilrah]

Some people have just been running the tree command dumping it to a file in a scheduled user script.

Edited December 2, 2022 by Kilrah

[nmills3]

does anyone have suggestions for tools that can work out what i file type a file is meant to be? i have a few files that lost their names and i have no idea what they are. i know they're not images or pdf files and video players don't seem to work on them so i don't think it's that

[JorgeB]

Try this:

https://linux.die.net/man/1/file

[nmills3]

so i got my server working again and i've been moving the files back over to it. but i just noticed that my log is filling up with lots of errors like this one

Dec 4 12:14:45 Tower smbd[21808]: [2022/12/04 12:14:45.456081, 0] ../../source3/smbd/files.c:1199(synthetic_pathref) Dec 4 12:14:45 Tower smbd[21808]: synthetic_pathref: opening [Clone Hero/songs/RB/Rock Band 1 DLC/Metallica - Blackened (2x Bass Pedal Expert+)/rhythm.ogg] failed

would that be related to my current issue or should i make a new thread fir help with that

tower-diagnostics-20221204-1216.zip