I just built a new Unraid server.  Moved all the drives from the old server to the new one.  I used the unbalance plugin to move files around so I could convert the file system to XFS.

 

When that was all complete, I had an error on disk 3, so I replaced it and rebuilt it.

Now, my Media share shows as empty:

 

However, if I check the individual disks, they all show the data from the Media share:

 

I see an error in the log file:

Dec 30 15:05:33 Tower kernel: XFS (md3): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x156c69f68 len 32 error 117
Dec 30 15:05:33 Tower kernel: XFS (md3): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
Dec 30 15:05:33 Tower kernel: XFS (md3): Metadata corruption detected at xfs_buf_ioend+0x4c/0x95 [xfs], xfs_inode block 0x156c69f68 xfs_inode_buf_verify
Dec 30 15:05:33 Tower kernel: XFS (md3): Unmount and run xfs_repair
Dec 30 15:05:33 Tower kernel: XFS (md3): First 128 bytes of corrupted metadata buffer:
Dec 30 15:05:33 Tower kernel: 00000000b764e26b: be fa cc 07 c4 c8 71 c1 d6 3e 51 3e a0 cb af dc ......q..>Q>....
Dec 30 15:05:33 Tower kernel: 00000000ee47918b: d1 56 aa 91 0b c1 90 9f 95 2f af 7d 54 ee d1 a9 .V......./.}T...
Dec 30 15:05:33 Tower kernel: 000000003f63270c: 1d 5d 78 9e 52 83 f6 b8 30 a5 de 46 6c 4d 1e f3 .]x.R...0..FlM..
Dec 30 15:05:33 Tower kernel: 00000000dcf7f7c5: 2a 1d 6a 11 a3 33 f4 1a 4f cf bd 60 8c 3b 09 e6 *.j..3..O..`.;..
Dec 30 15:05:33 Tower kernel: 000000009db33fd9: d4 38 5a e6 42 bf bf 8b 10 25 73 6a f1 58 5a f5 .8Z.B....%sj.XZ.
Dec 30 15:05:33 Tower kernel: 000000002d170146: 93 98 cb 16 97 71 05 c2 a0 08 32 6b 5a 4e ab 26 .....q....2kZN.&
Dec 30 15:05:33 Tower kernel: 00000000b756f334: 4e 32 80 f0 e3 72 0a 79 b8 0d 1a 64 68 c6 95 70 N2...r.y...dh..p
Dec 30 15:05:33 Tower kernel: 00000000a507f273: 82 bb 13 a9 e6 a8 de 65 f7 b2 8f 14 6e 55 f4 ad .......e....nU..

 

I have attached my diagnostics files:

tower-diagnostics-20181230-1506.zip

 

 

Any help would be greatly appreciated

Edited December 31, 20187 yr by biiiink

Filesystem corruption on disk3.

 

Here is the relevant link from the FAQ for Unraid V6 "sticky" pinned near the top of this same subforum:

 

https://wiki.unraid.net/index.php/Check_Disk_Filesystems

What was the error you had on disk3 that caused you to replace it anyway? Could be there was nothing wrong with the disk itself. Connection issues are much more common than bad disks.

 

Possibly you already had filesystem corruption before the rebuild. And rebuilding from parity will just rebuild the corrupt filesystem.

im running xfs_repair right now.

 

i dont quite remember what the error was, but googling it let me to information saying the drive was failing.  Replacing the drive wasn't such a big deal as I have a few spares kicking around.

So the xfs_repair is still running.  Been running for nearly 24 hrs.  Is this normal?  The disk is 3GB and would be probably 80% full.

 

Most recent thing i see in the status window is:

Phase 7 - verify and correct link counts...
resetting inode 131 nlinks from 9 to 8
resetting inode 4427235023 nlinks from 5 to 4
resetting inode 222 nlinks from 5 to 4
resetting inode 2147483776 nlinks from 421 to 417
resetting inode 2147483866 nlinks from 5 to 4

 

1 hour ago, biiiink said:

Been running for nearly 24 hrs.  Is this normal? 

Not really, any disk activity?

1 hour ago, johnnie.black said:

Not really, any disk activity?

Doesn't look like it, judging by the number of reads

I cancelled the check, but I'm not sure where to go from here.  I've tried googling the errors in the log but i'm getting nowhere.  

 

Any help would be greatly appreciated.

Start xfs_repair again

5 hours ago, johnnie.black said:

Start xfs_repair again

I've done that through the Gui and the terminal, but it always gets stuck at the same spot

Try mounting the disk once and then running xfs_repair again, if it doesn't help or the disk doesn't mount best bet is to ask for help on the xfs mailing list.

I've done that too.  same result.

 

To be clear, the disk mounts just fine, and shows all the files just fine.  However, the share that the files belong to is completely empty

35 minutes ago, biiiink said:

I've done that too.  same result.

 

To be clear, the disk mounts just fine, and shows all the files just fine.  However, the share that the files belong to is completely empty

If that disk has the files but the user share doesn't then maybe that disk isn't the problem. Other things can break user shares, and other things can cause a user to report an empty share when user shares aren't broken.

 

What specifically are you looking at that makes you think the share is empty?

 

Go to Main - Array Devices, click on the folder icon at the far right on the same line as the disk, and post a screenshot.

 

Go to User Shares, click on the Compute... link next to the share in question, and after it gives the result, post a screenshot.

23 minutes ago, trurl said:

If that disk has the files but the user share doesn't then maybe that disk isn't the problem. Other things can break user shares, and other things can cause a user to report an empty share when user shares aren't broken.

 

What specifically are you looking at that makes you think the share is empty?

 

Go to Main - Array Devices, click on the folder icon at the far right on the same line as the disk, and post a screenshot.

 

Go to User Shares, click on the Compute... link next to the share in question, and after it gives the result, post a screenshot.

Both of those screenshots are in my original post.  Also, if i connect to the server and check the Media share, it shows as empty as well.

 

I also notice that this error shows up twice when i go into the media share in the web Gui: (It also shows up, but only once, when i go into the Media folder on disk 3, but not on any other disk)

Jan 2 06:49:25 Tower kernel: XFS (md3): Metadata corruption detected at xfs_buf_ioend+0x4c/0x95 [xfs], xfs_inode block 0x156c69f68 xfs_inode_buf_verify
Jan 2 06:49:25 Tower kernel: XFS (md3): Unmount and run xfs_repair
Jan 2 06:49:25 Tower kernel: XFS (md3): First 128 bytes of corrupted metadata buffer:
Jan 2 06:49:25 Tower kernel: 000000005491da45: be fa cc 07 c4 c8 71 c1 d6 3e 51 3e a0 cb af dc ......q..>Q>....
Jan 2 06:49:25 Tower kernel: 0000000033c25846: d1 56 aa 91 0b c1 90 9f 95 2f af 7d 54 ee d1 a9 .V......./.}T...
Jan 2 06:49:25 Tower kernel: 0000000040618539: 1d 5d 78 9e 52 83 f6 b8 30 a5 de 46 6c 4d 1e f3 .]x.R...0..FlM..
Jan 2 06:49:25 Tower kernel: 000000009e9334aa: 2a 1d 6a 11 a3 33 f4 1a 4f cf bd 60 8c 3b 09 e6 *.j..3..O..`.;..
Jan 2 06:49:25 Tower kernel: 000000009881fdfb: d4 38 5a e6 42 bf bf 8b 10 25 73 6a f1 58 5a f5 .8Z.B....%sj.XZ.
Jan 2 06:49:25 Tower kernel: 000000004113fee1: 93 98 cb 16 97 71 05 c2 a0 08 32 6b 5a 4e ab 26 .....q....2kZN.&
Jan 2 06:49:25 Tower kernel: 0000000049cb44db: 4e 32 80 f0 e3 72 0a 79 b8 0d 1a 64 68 c6 95 70 N2...r.y...dh..p
Jan 2 06:49:25 Tower kernel: 00000000cc661843: 82 bb 13 a9 e6 a8 de 65 f7 b2 8f 14 6e 55 f4 ad .......e....nU..
Jan 2 06:49:25 Tower kernel: XFS (md3): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x156c69f68 len 32 error 117
Jan 2 06:49:25 Tower kernel: XFS (md3): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
Jan 2 06:49:25 Tower kernel: XFS (md3): Metadata corruption detected at xfs_buf_ioend+0x4c/0x95 [xfs], xfs_inode block 0x156c69f68 xfs_inode_buf_verify
Jan 2 06:49:25 Tower kernel: XFS (md3): Unmount and run xfs_repair
Jan 2 06:49:25 Tower kernel: XFS (md3): First 128 bytes of corrupted metadata buffer:
Jan 2 06:49:25 Tower kernel: 00000000aa3c7d84: be fa cc 07 c4 c8 71 c1 d6 3e 51 3e a0 cb af dc ......q..>Q>....
Jan 2 06:49:25 Tower kernel: 00000000c13b8090: d1 56 aa 91 0b c1 90 9f 95 2f af 7d 54 ee d1 a9 .V......./.}T...
Jan 2 06:49:25 Tower kernel: 00000000cb0ed316: 1d 5d 78 9e 52 83 f6 b8 30 a5 de 46 6c 4d 1e f3 .]x.R...0..FlM..
Jan 2 06:49:25 Tower kernel: 00000000de4d9b94: 2a 1d 6a 11 a3 33 f4 1a 4f cf bd 60 8c 3b 09 e6 *.j..3..O..`.;..
Jan 2 06:49:25 Tower kernel: 0000000021fff399: d4 38 5a e6 42 bf bf 8b 10 25 73 6a f1 58 5a f5 .8Z.B....%sj.XZ.
Jan 2 06:49:25 Tower kernel: 000000008e981e48: 93 98 cb 16 97 71 05 c2 a0 08 32 6b 5a 4e ab 26 .....q....2kZN.&
Jan 2 06:49:25 Tower kernel: 00000000808a9410: 4e 32 80 f0 e3 72 0a 79 b8 0d 1a 64 68 c6 95 70 N2...r.y...dh..p
Jan 2 06:49:25 Tower kernel: 00000000b2bf6473: 82 bb 13 a9 e6 a8 de 65 f7 b2 8f 14 6e 55 f4 ad .......e....nU..
Jan 2 06:49:25 Tower kernel: XFS (md3): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x156c69f68 len 32 error 117
Jan 2 06:49:25 Tower kernel: XFS (md3): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.

 

Edited January 2, 20197 yr by biiiink

That certainly looks like corruption on disk3. And corruption on any disk has been known to break user shares.

 

Maybe you could just try to copy whatever you can from that disk, have Unraid reformat it, and then see if the share comes back.

12 minutes ago, trurl said:

Maybe you could just try to copy whatever you can from that disk, have Unraid reformat it,

If xfs_repair doesn't finish this is your best bet.

 

1 hour ago, trurl said:

Maybe you could just try to copy whatever you can from that disk, have Unraid reformat it, and then see if the share comes back.

I attempted this through unbalance, but it came back with this:

 

So im going to do it manually.  I will report back when its complete

So im trying to copy the files to an empty drive, but i get an error:

cp: cannot stat 'Downloads/incomplete': Structure needs cleaning

This error comes up for every folder im trying to copy.

 

Googling it leads me back to running xfs_repair.

 

So now im not sure what else to do

by the way, the command im running is:

xfs_repair -L /dev/md3

I've attached the entire log in case there is something im missing

 

xfs_repair.txt

Edited January 2, 20197 yr by biiiink

That looks like a very corrupt filesystem, still xfs_repair should fix would more or less data loss, so you could try asking for help on the xfs mailing list.

Just to update this thread:

 

It was basically determined that nothing would fix the filesystem.  It was just too corrupt.  However, I could still mount the drive and see all the files.  So I mounted the drive using the unassigned devices plugin, and manually copied all the folders to an empty drive.  After checking the drive to ensure that there were no issues, i used the "new config" tool to create a new drive configuration.  Reassigned all the drives, and I now have my Media share back.

 

Thank you to everyone who offered their help/advice/opinion

Edited January 9, 20197 yr by biiiink