January 22, 20224 yr Hello all, Back in september i had to stop my server for moving. When i restarted it i had problems with some disks. I finally decided to give it a chance. I bought 2 wd mybook 6Tb to shuck drives. Right now one is in the preclearing process. So the situation is: Normally my array has 2 6Tb parity disks 4 data drives: disks 1, 2 and 3 are 4Tb disks. Disk 4 is 6Tb. Now disk 1 and 2 are ok disk 3 is disabled. i can read emulated content. I tried xfs repair -L without success. The disk can't be mounted. disk 4 is not detected and i can't read emulated disk. What can i do to recover disk 3&4 content and have server back on line. For now array is started, vm and docker are disabled and a preclear is running. I attached diags. godzilla-diagnostics-20220122-1520.zip
January 23, 20224 yr Community Expert Start the array in maintenance and post the output of: xfs_repair -v /dev/md3 xfs_repair -v /dev/md4
January 23, 20224 yr Author thank you for your answer. Will do that when preclear ends. If i understood correctly: md3 is fine as it is mounted. Disk3 should be able to be rebuilt. md4 should have file system errors and if xfs_repair correct errors i should be able to rebuild disk4. so once i have md3 and md4 without error i can rebuild simultanously disk3 and disk4. Actual disk 4 (wd60EFRX) is not seen by controller. It makes some click noises at startup.
January 23, 20224 yr Community Expert 21 hours ago, caplam said: The disk can't be mounted. 4 minutes ago, caplam said: md3 is fine as it is mounted. Which is it?
January 23, 20224 yr Community Expert I see that later in the diags disk3 shows is mounted, since you mentioned it didn't and the diags showed that initially I didn't check until the end, in that case post the output of xfs_repair just for disk4.
January 23, 20224 yr Author perhaps i don't understand correctly. When i write disk3 i mean sdg the physical disk. md3 is the "logical disk". As disk3 is disabled, md3 is emulated. Am i correct?
January 23, 20224 yr Author xfs_repair -v /dev/md4 Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... writing modified primary superblock - block cache size set to 6137384 entries sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... zero_log: head block 205153 tail block 205149 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. with the gui i can't run it; nothing happens
January 23, 20224 yr Community Expert Only things you care for now are the emulated disks, emulated disk3 is mounting now so it can be rebuilt. Just now, caplam said: then use the -L option Use -L and post output.
January 23, 20224 yr Author xfs_repair -L /dev/md4 Phase 1 - find and verify superblock... sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... sb_icount 0, counted 40128 sb_ifree 0, counted 349 sb_fdblocks 1464608875, counted 440547342 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 3 - agno = 5 - agno = 2 - agno = 4 - agno = 1 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (4:205143) is ahead of log (1:2). Format log to cycle 7. done
January 23, 20224 yr Author i'm stopping array but it's pretty long. I will restart it in normal mode to see if md4 can be mounted. If yes i suppose the next step is rebuilding disk3 and 4 (for that i have to wait preclear ends)
January 23, 20224 yr Community Expert 1 minute ago, caplam said: If yes i suppose the next step is rebuilding disk3 and 4 (for that i have to wait preclear ends) Yes, for disk3 make sure contents look correct, also look for a lost+found folder, before rebuilding on top of the old disk.
January 23, 20224 yr Author i'll use another disk and keep actual disk3 apart. Start and stop of the array are very long. I'm waiting for the start to finish to see if md4 is mounted.
January 23, 20224 yr Author start is not finished but log is showing that. So i guess it should be good. an 23 13:01:10 godzilla emhttpd: shcmd (849): mkdir -p /mnt/disk4 Jan 23 13:01:10 godzilla emhttpd: shcmd (850): mount -t xfs -o noatime /dev/md4 /mnt/disk4 Jan 23 13:01:10 godzilla kernel: XFS (md4): Mounting V5 Filesystem Jan 23 13:01:10 godzilla kernel: XFS (md4): Ending clean mount Jan 23 13:01:11 godzilla kernel: xfs filesystem being mounted at /mnt/disk4 supports timestamps until 2038 (0x7fffffff) Jan 23 13:01:11 godzilla emhttpd: shcmd (851): xfs_growfs /mnt/disk4 Jan 23 13:01:11 godzilla root: meta-data=/dev/md4 isize=512 agcount=6, agsize=268435455 blks Jan 23 13:01:11 godzilla root: = sectsz=512 attr=2, projid32bit=1 Jan 23 13:01:11 godzilla root: = crc=1 finobt=1, sparse=1, rmapbt=0 Jan 23 13:01:11 godzilla root: = reflink=0 Jan 23 13:01:11 godzilla root: data = bsize=4096 blocks=1465130633, imaxpct=5 Jan 23 13:01:11 godzilla root: = sunit=0 swidth=0 blks Jan 23 13:01:11 godzilla root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Jan 23 13:01:11 godzilla root: log =internal log bsize=4096 blocks=521728, version=2 Jan 23 13:01:11 godzilla root: = sectsz=512 sunit=0 blks, lazy-count=1 Jan 23 13:01:11 godzilla root: realtime =none extsz=4096 blocks=0, rtextents=0
January 23, 20224 yr Author finally array is started, md4 is mounted. Impossible to tell if all files are here. I have not seen any lost+found directory. Can i rebuild disk3 and 4 simultanously ? is it preferable to copy the content of md4 to an external disk ?
January 23, 20224 yr Community Expert 6 minutes ago, caplam said: Can i rebuild disk3 and 4 simultanously ? Yes. 6 minutes ago, caplam said: is it preferable to copy the content of md4 to an external disk ? Don't see much point in doing that instead.
January 23, 20224 yr Author ok thank you. Preclear should end in 4 or 5 hours. Rebuild should take at least 3 days.
January 23, 20224 yr Community Expert 3 hours ago, caplam said: Rebuild should take at least 3 days Why?
January 23, 20224 yr Author I don't know, didn't do the math but 1 pass preclear on 6Tb is 24H. 2 disks to rebuild (4 and 6Tb). If it can rebuild 2 disks simultanously at 100MB/s i guess it could be done in 17 hours. For now i'm waiting preclear end on 6Tb replacement drive. 4 tb just finished 1 hour ago. edit: rebuild has started for 2 disks. Edited January 23, 20224 yr by caplam
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.