saving my server


caplam

Recommended Posts

Hello all,

 

Back in september i had to stop my server for moving. When i restarted it i had problems with some disks.

I finally decided to give it a chance. I bought 2 wd mybook 6Tb to shuck drives.

Right now one is in the preclearing process.

 

So the situation is:

Normally my array has 2 6Tb parity disks

4 data drives: disks 1, 2 and 3 are 4Tb disks. Disk 4 is 6Tb.

Now disk 1 and 2 are ok

disk 3 is disabled. i can read emulated content. I tried xfs repair -L without success. The disk can't be mounted.

disk 4 is not detected and i can't read emulated disk.

What can i do to recover disk 3&4 content and have server back on line. For now array is started, vm and docker are disabled and a preclear is running.

I attached diags.

godzilla-diagnostics-20220122-1520.zip

Link to comment

thank you for your answer.

Will do that when preclear ends.

If i understood correctly:

md3 is fine as it is mounted. Disk3 should be able to be rebuilt.

md4 should have file system errors and if xfs_repair correct errors i should be able to rebuild disk4.

so once i have md3 and md4 without error i can rebuild simultanously disk3 and disk4.

Actual disk 4 (wd60EFRX) is not seen by controller. It makes some click noises at startup.

Link to comment
xfs_repair -v /dev/md4
Phase 1 - find and verify superblock...
bad primary superblock - bad CRC in superblock !!!

attempting to find secondary superblock...
.found candidate secondary superblock...
verified secondary superblock...
writing modified primary superblock
        - block cache size set to 6137384 entries
sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128
resetting superblock root inode pointer to 128
sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129
resetting superblock realtime bitmap inode pointer to 129
sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130
resetting superblock realtime summary inode pointer to 130
Phase 2 - using internal log
        - zero log...
zero_log: head block 205153 tail block 205149
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

 

with the gui i can't run it; nothing happens

Link to comment
xfs_repair -L /dev/md4
Phase 1 - find and verify superblock...
sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128
resetting superblock root inode pointer to 128
sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129
resetting superblock realtime bitmap inode pointer to 129
sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130
resetting superblock realtime summary inode pointer to 130
Phase 2 - using internal log
        - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
destroyed because the -L option was used.
        - scan filesystem freespace and inode maps...
sb_icount 0, counted 40128
sb_ifree 0, counted 349
sb_fdblocks 1464608875, counted 440547342
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 3
        - agno = 5
        - agno = 2
        - agno = 4
        - agno = 1
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
Maximum metadata LSN (4:205143) is ahead of log (1:2).
Format log to cycle 7.
done

 

Link to comment

start is not finished but log is showing that. So i guess it should be good.

an 23 13:01:10 godzilla emhttpd: shcmd (849): mkdir -p /mnt/disk4
Jan 23 13:01:10 godzilla emhttpd: shcmd (850): mount -t xfs -o noatime /dev/md4 /mnt/disk4
Jan 23 13:01:10 godzilla kernel: XFS (md4): Mounting V5 Filesystem
Jan 23 13:01:10 godzilla kernel: XFS (md4): Ending clean mount
Jan 23 13:01:11 godzilla kernel: xfs filesystem being mounted at /mnt/disk4 supports timestamps until 2038 (0x7fffffff)
Jan 23 13:01:11 godzilla emhttpd: shcmd (851): xfs_growfs /mnt/disk4
Jan 23 13:01:11 godzilla root: meta-data=/dev/md4               isize=512    agcount=6, agsize=268435455 blks
Jan 23 13:01:11 godzilla root:          =                       sectsz=512   attr=2, projid32bit=1
Jan 23 13:01:11 godzilla root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Jan 23 13:01:11 godzilla root:          =                       reflink=0
Jan 23 13:01:11 godzilla root: data     =                       bsize=4096   blocks=1465130633, imaxpct=5
Jan 23 13:01:11 godzilla root:          =                       sunit=0      swidth=0 blks
Jan 23 13:01:11 godzilla root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Jan 23 13:01:11 godzilla root: log      =internal log           bsize=4096   blocks=521728, version=2
Jan 23 13:01:11 godzilla root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Jan 23 13:01:11 godzilla root: realtime =none                   extsz=4096   blocks=0, rtextents=0

 

Link to comment

I don't know, didn't do the math but 1 pass preclear on 6Tb is 24H.

2 disks to rebuild (4 and 6Tb). 

If it can rebuild 2 disks simultanously at 100MB/s i guess it could be done in 17 hours.

For now i'm waiting preclear end on 6Tb replacement drive. 4 tb just finished 1 hour ago.

 

edit: rebuild has started for 2 disks.

Edited by caplam
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.