January 3, 20242 yr Trying to do some cleanup of files, and when I tried deleting a folder with 1000s of files in it, I started getting I/O errors: Quote rm: cannot remove 'Backups/Files/more folders/IMG_2373.JPG': Structure needs cleaning rm: cannot remove 'Backups/Files/more folders/IMG_2472.JPG': Input/output error <dozens more I/O error listings> After that, I couldn't access the folder above the one I tried to delete, or the share that the folder was mounted in. I then found that I couldn't even get into one of the disks in my array from the terminal: Quote root@MyServer:/mnt/disk3# ls /bin/ls: cannot open directory '.': Input/output error I stopped the array, started it in maintenance mode and tried running a disk check with no options. It told me there were outstanding logs. So I restarted the array fully, stopped it again, started in maintenance mode, and ran the disk check again with no options. Below is the log from that check disk run (edited to remove redundancy): Quote Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 3 - agno = 5 - agno = 2 - agno = 1 - agno = 7 - agno = 6 - agno = 4 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... Metadata CRC error detected at 0x463fa8, xfs_dir3_free block 0x3a3812a40/0x1000 free block 16777216 for directory inode 15511924341 bad header rebuilding directory inode 15511924341 Metadata CRC error detected at 0x463fa8, xfs_dir3_free block 0x3a3812a38/0x1000 free block 16777216 for directory inode 15511924372 bad header rebuilding directory inode 15511924372 Metadata CRC error detected at 0x4619e8, xfs_dir3_leaf1 block 0x3a3812a30/0x1000 leaf block 8388608 for directory inode 15514174026 bad CRC rebuilding directory inode 15514174026 Warning: recursive buffer locking at block 15628053040 detected Metadata CRC error detected at 0x4619e8, xfs_dir3_leaf1 block 0x3a3812a28/0x1000 leaf block 8388608 for directory inode 15608717125 bad CRC rebuilding directory inode 15608717125 Warning: recursive buffer locking at block 15628053032 detected Warning: recursive buffer locking at block 15628053040 detected Warning: recursive buffer locking at block 15628053040 detected (Repeated 50 times) Warning: recursive buffer locking at block 15628053040 detected Metadata CRC error detected at 0x4619e8, xfs_dir3_leaf1 block 0x3a3812a20/0x1000 leaf block 8388608 for directory inode 15608782647 bad CRC rebuilding directory inode 15608782647 Warning: recursive buffer locking at block 15628053024 detected Warning: recursive buffer locking at block 15628053032 detected Warning: recursive buffer locking at block 15628053032 detected (Repeated 65 times) Warning: recursive buffer locking at block 15628053032 detected Metadata CRC error detected at 0x4619e8, xfs_dir3_leaf1 block 0x3a3812a18/0x1000 leaf block 8388608 for directory inode 15608955416 bad CRC rebuilding directory inode 15608955416 Warning: recursive buffer locking at block 15628053016 detected Warning: recursive buffer locking at block 15628053024 detected Warning: recursive buffer locking at block 15628053024 detected (Repeated 64 times) Warning: recursive buffer locking at block 15628053024 detected - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... cache_purge: shake on cache 0x509600 left 4 nodes!? cache_purge: shake on cache 0x509600 left 4 nodes!? cache_zero_check: refcount is 1, not zero (node=0x152714237a10) cache_zero_check: refcount is 1, not zero (node=0x15271422c610) cache_zero_check: refcount is 1, not zero (node=0x152733386410) cache_zero_check: refcount is 1, not zero (node=0x152720b5f810) done I've started the array back up and things seem functional. I can access data on my other disks without issue, and data outside the folder I was trying to delete seems to be intact. Are there any suggestions on how I should proceed?
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.