caplam Posted January 22, 2022 Share Posted January 22, 2022 Hello all, Back in september i had to stop my server for moving. When i restarted it i had problems with some disks. I finally decided to give it a chance. I bought 2 wd mybook 6Tb to shuck drives. Right now one is in the preclearing process. So the situation is: Normally my array has 2 6Tb parity disks 4 data drives: disks 1, 2 and 3 are 4Tb disks. Disk 4 is 6Tb. Now disk 1 and 2 are ok disk 3 is disabled. i can read emulated content. I tried xfs repair -L without success. The disk can't be mounted. disk 4 is not detected and i can't read emulated disk. What can i do to recover disk 3&4 content and have server back on line. For now array is started, vm and docker are disabled and a preclear is running. I attached diags. godzilla-diagnostics-20220122-1520.zip Quote Link to comment
JorgeB Posted January 23, 2022 Share Posted January 23, 2022 Start the array in maintenance and post the output of: xfs_repair -v /dev/md3 xfs_repair -v /dev/md4 Quote Link to comment
caplam Posted January 23, 2022 Author Share Posted January 23, 2022 thank you for your answer. Will do that when preclear ends. If i understood correctly: md3 is fine as it is mounted. Disk3 should be able to be rebuilt. md4 should have file system errors and if xfs_repair correct errors i should be able to rebuild disk4. so once i have md3 and md4 without error i can rebuild simultanously disk3 and disk4. Actual disk 4 (wd60EFRX) is not seen by controller. It makes some click noises at startup. Quote Link to comment
JorgeB Posted January 23, 2022 Share Posted January 23, 2022 21 hours ago, caplam said: The disk can't be mounted. 4 minutes ago, caplam said: md3 is fine as it is mounted. Which is it? Quote Link to comment
JorgeB Posted January 23, 2022 Share Posted January 23, 2022 I see that later in the diags disk3 shows is mounted, since you mentioned it didn't and the diags showed that initially I didn't check until the end, in that case post the output of xfs_repair just for disk4. Quote Link to comment
caplam Posted January 23, 2022 Author Share Posted January 23, 2022 perhaps i don't understand correctly. When i write disk3 i mean sdg the physical disk. md3 is the "logical disk". As disk3 is disabled, md3 is emulated. Am i correct? Quote Link to comment
caplam Posted January 23, 2022 Author Share Posted January 23, 2022 xfs_repair -v /dev/md4 Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... writing modified primary superblock - block cache size set to 6137384 entries sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... zero_log: head block 205153 tail block 205149 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. with the gui i can't run it; nothing happens Quote Link to comment
JorgeB Posted January 23, 2022 Share Posted January 23, 2022 Only things you care for now are the emulated disks, emulated disk3 is mounting now so it can be rebuilt. Just now, caplam said: then use the -L option Use -L and post output. Quote Link to comment
caplam Posted January 23, 2022 Author Share Posted January 23, 2022 xfs_repair -L /dev/md4 Phase 1 - find and verify superblock... sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... sb_icount 0, counted 40128 sb_ifree 0, counted 349 sb_fdblocks 1464608875, counted 440547342 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 3 - agno = 5 - agno = 2 - agno = 4 - agno = 1 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (4:205143) is ahead of log (1:2). Format log to cycle 7. done Quote Link to comment
caplam Posted January 23, 2022 Author Share Posted January 23, 2022 i'm stopping array but it's pretty long. I will restart it in normal mode to see if md4 can be mounted. If yes i suppose the next step is rebuilding disk3 and 4 (for that i have to wait preclear ends) Quote Link to comment
JorgeB Posted January 23, 2022 Share Posted January 23, 2022 1 minute ago, caplam said: If yes i suppose the next step is rebuilding disk3 and 4 (for that i have to wait preclear ends) Yes, for disk3 make sure contents look correct, also look for a lost+found folder, before rebuilding on top of the old disk. Quote Link to comment
caplam Posted January 23, 2022 Author Share Posted January 23, 2022 i'll use another disk and keep actual disk3 apart. Start and stop of the array are very long. I'm waiting for the start to finish to see if md4 is mounted. Quote Link to comment
caplam Posted January 23, 2022 Author Share Posted January 23, 2022 start is not finished but log is showing that. So i guess it should be good. an 23 13:01:10 godzilla emhttpd: shcmd (849): mkdir -p /mnt/disk4 Jan 23 13:01:10 godzilla emhttpd: shcmd (850): mount -t xfs -o noatime /dev/md4 /mnt/disk4 Jan 23 13:01:10 godzilla kernel: XFS (md4): Mounting V5 Filesystem Jan 23 13:01:10 godzilla kernel: XFS (md4): Ending clean mount Jan 23 13:01:11 godzilla kernel: xfs filesystem being mounted at /mnt/disk4 supports timestamps until 2038 (0x7fffffff) Jan 23 13:01:11 godzilla emhttpd: shcmd (851): xfs_growfs /mnt/disk4 Jan 23 13:01:11 godzilla root: meta-data=/dev/md4 isize=512 agcount=6, agsize=268435455 blks Jan 23 13:01:11 godzilla root: = sectsz=512 attr=2, projid32bit=1 Jan 23 13:01:11 godzilla root: = crc=1 finobt=1, sparse=1, rmapbt=0 Jan 23 13:01:11 godzilla root: = reflink=0 Jan 23 13:01:11 godzilla root: data = bsize=4096 blocks=1465130633, imaxpct=5 Jan 23 13:01:11 godzilla root: = sunit=0 swidth=0 blks Jan 23 13:01:11 godzilla root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Jan 23 13:01:11 godzilla root: log =internal log bsize=4096 blocks=521728, version=2 Jan 23 13:01:11 godzilla root: = sectsz=512 sunit=0 blks, lazy-count=1 Jan 23 13:01:11 godzilla root: realtime =none extsz=4096 blocks=0, rtextents=0 Quote Link to comment
caplam Posted January 23, 2022 Author Share Posted January 23, 2022 finally array is started, md4 is mounted. Impossible to tell if all files are here. I have not seen any lost+found directory. Can i rebuild disk3 and 4 simultanously ? is it preferable to copy the content of md4 to an external disk ? Quote Link to comment
JorgeB Posted January 23, 2022 Share Posted January 23, 2022 6 minutes ago, caplam said: Can i rebuild disk3 and 4 simultanously ? Yes. 6 minutes ago, caplam said: is it preferable to copy the content of md4 to an external disk ? Don't see much point in doing that instead. Quote Link to comment
caplam Posted January 23, 2022 Author Share Posted January 23, 2022 ok thank you. Preclear should end in 4 or 5 hours. Rebuild should take at least 3 days. Quote Link to comment
trurl Posted January 23, 2022 Share Posted January 23, 2022 3 hours ago, caplam said: Rebuild should take at least 3 days Why? Quote Link to comment
caplam Posted January 23, 2022 Author Share Posted January 23, 2022 (edited) I don't know, didn't do the math but 1 pass preclear on 6Tb is 24H. 2 disks to rebuild (4 and 6Tb). If it can rebuild 2 disks simultanously at 100MB/s i guess it could be done in 17 hours. For now i'm waiting preclear end on 6Tb replacement drive. 4 tb just finished 1 hour ago. edit: rebuild has started for 2 disks. Edited January 23, 2022 by caplam Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.