Griminal Posted January 28 Share Posted January 28 I had a disk become disabled so I did the procedure to remove it from the array and add it back. SMART data was good. I rebooted the server and saw the screenshot I have attached. So I removed the offending drive again, rebooted, and did an XFS_repair while the disk was out of the array. I left the drive out and am going to purchase a replacement drive, just to have a spare. I then noticed a lot of data missing. I see a lot of data in my lost+found. It seems like my emulated data must have been wrong. I'm not sure. Files and folders I've added for the past month or so seem to be missing data. Maybe I have BTRFS issues? I hope?? hyde-diagnostics-20240127-2321.zip Quote Link to comment
trurl Posted January 28 Share Posted January 28 5 minutes ago, Griminal said: disk become disabled so I did the procedure to remove it from the array and add it back Not entirely clear. Do you mean you rebuilt the disk? Did the rebuild complete? 2 minutes ago, Griminal said: I removed the offending drive again, rebooted, and did an XFS_repair while the disk was out of the array. Do you mean you did XFS_repair on the physical disk while it was not in the array? If so, you invalidated parity. Or do you mean you did XFS_repair on the emulated/disabled disk in the array, with the physical disk not in the array? Quote Link to comment
trurl Posted January 28 Share Posted January 28 10 minutes ago, Griminal said: Files and folders I've added for the past month or so seem to be missing data. Maybe I have BTRFS issues? I hope?? Looks like disk1 is still missing/emulated. Are you looking at the contents of physical disk1? Or are you looking at the contents of the emulated disk1? Physical disk1 will not have any files written while the disk was emulated. Don't see what BTRFS has to do with it, cache is the only btrfs Quote Link to comment
Griminal Posted January 28 Author Share Posted January 28 Trurl... thannks for the responses. Unfortunately, I think I screwed up at step 4. The instructions don't say explicitly say "allow the array to rebuild"... I stopped it at like .3% and moved on to step 4. So I think I'm hosed. Oddly, Plex hasn't reflected some of these files being removed. Normally, as soon as the file system is changed, it reflects it. Any options for me or am I out of luck? I don't see the missing files in emulated or on the mounted, non-member data disk. 1. Stop array 2. Un-assign disabled disk 3. Start array so the missing disk is registered 4. Important: If the drive to be rebuilt is a data drive then check that the emulated drive is showing the content you expect to be there as the rebuild process simply makes the physical drive match the emulated one. If this is not the case then you may want to ask in forums for advice on the best way to proceed. 5. Stop array 6. Reassign disabled disk Quote Link to comment
trurl Posted January 28 Share Posted January 28 After step 3, nothing happens. I don't understand what you mean by you "stopped it at like .3%". There is nothing to stop, and nothing that would indicate any progress. Perhaps you mean you started a read check after starting the array with the disk unassigned? That is not part of the instructions, but will not hurt anything. Rebuild doesn't start until step 6. Still unclear about the xfs_repair, but I guess nothing we can do about that at this point. We will work from where we are. Have you done anything else since the diagnostics you posted earlier? Quote Link to comment
trurl Posted January 28 Share Posted January 28 Since lost+found is showing in your emulated disk1, then either you repaired emulated disk1, or a repair had been done on disk1 on some earlier occasion. Check filesystem on emulated disk1. Be sure to run it from the webUI, not the command line. Do it with -n for now. Capture the output and post it. Also, do the same on the "non-member" disk, ZL2LCCS2, which I assume was previously assigned as disk1. Capture the output and post it. Quote Link to comment
Griminal Posted January 28 Author Share Posted January 28 Will do. Thanks for the help. No changes to diagnostics. I ordered a few new 16TB and I'll wait for those before I make massive file changes. Disk1 emulated: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 5 - agno = 7 - agno = 13 - agno = 4 - agno = 6 - agno = 8 - agno = 11 - agno = 1 - agno = 9 - agno = 10 - agno = 12 - agno = 15 - agno = 14 - agno = 2 - agno = 3 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. Non-member: FS: xfs Executing file system check: /sbin/xfs_repair -n '/dev/sdg1' 2>&1 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 5 - agno = 9 - agno = 12 - agno = 15 - agno = 4 - agno = 7 - agno = 6 - agno = 8 - agno = 10 - agno = 11 - agno = 13 - agno = 14 - agno = 2 - agno = 1 - agno = 3 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. No file system corruption detected! Quote Link to comment
trurl Posted January 28 Share Posted January 28 According to that, ZL2LCCS2 has no corruption. But it won't have anything that was written to disk1 while it was emulated. Do the check again on the emulated disk, this time without -n. If it asks for it, use -L. Post the results. Quote Link to comment
trurl Posted January 28 Share Posted January 28 Also, mount ZL2LCCS2 as an Unassigned Device, and see if it has lost+found. Quote Link to comment
Griminal Posted January 28 Author Share Posted January 28 Mounted ZL2LCCS2 has lost+found. Some data is there. Doing a search for files larger than 2GB yields no results. I have a few ISOs that went missing. Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 5 - agno = 11 - agno = 2 - agno = 4 - agno = 1 - agno = 8 - agno = 7 - agno = 10 - agno = 6 - agno = 12 - agno = 9 - agno = 13 - agno = 14 - agno = 15 - agno = 3 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done Quote Link to comment
Solution trurl Posted January 28 Solution Share Posted January 28 Looks like more might have been added to the lost+found on the emulated disk1. If you rebuild disk1, the result will be exactly what the emulated disk shows. If you rebuild to a new disk and keep the original as is, you will have both to work with to recover files. The linux "file" command can help determine the type of data a particular file might contain. Quote Link to comment
Griminal Posted January 28 Author Share Posted January 28 Yep. Ordered a replacement drive so I could do just that. I learned a lot of lessons this time. Thanks for the help. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.