June 30, 20179 yr started a parity check. found 1000's of errors, check aborted by itself now one of my drives is unmountable and the files that are on it seem lost as I can't see them any more even if the array is online. Drive is still showing up in the list of drives on unraid so it appears it can still see it. XFS file system un 30 14:43:05 Tower kernel: XFS (md11): Mounting V5 Filesystem Jun 30 14:43:05 Tower kernel: XFS (md11): Log inconsistent or not a log (last==0, first!=1) Jun 30 14:43:05 Tower kernel: XFS (md11): empty log check failed Jun 30 14:43:05 Tower kernel: XFS (md11): log mount/recovery failed: error -22 Jun 30 14:43:05 Tower kernel: XFS (md11): log mount failed Jun 30 14:43:05 Tower root: mount: wrong fs type, bad option, bad superblock on /dev/md11, Jun 30 14:43:05 Tower root: missing codepage or helper program, or other error Jun 30 14:43:05 Tower root: Jun 30 14:43:05 Tower root: In some cases useful info is found in syslog - try Jun 30 14:43:05 Tower root: dmesg | tail or so. Jun 30 14:43:05 Tower emhttp: err: shcmd: shcmd (147): exit status: 32 Jun 30 14:43:05 Tower emhttp: mount error: No file system (32) Jun 30 14:43:05 Tower emhttp: shcmd (148): umount /mnt/disk11 |& logger Jun 30 14:43:05 Tower root: umount: /mnt/disk11: not mounted Jun 30 14:43:05 Tower emhttp: shcmd (149): rmdir /mnt/disk11 Jun 30 14:43:05 Tower emhttp: shcmd (150): sync Jun 30 14:43:06 Tower emhttp: shcmd (151): mkdir /mnt/user Jun 30 14:43:06 Tower emhttp: shcmd (152): /usr/local/sbin/shfs /mnt/user -disks 4094 -o noatime,big_writes,allow_other -o remember=330 |& logger Jun 30 14:43:06 Tower emhttp: shcmd (153): rm -f /boot/config/plugins/dynamix/mover.cro
June 30, 20179 yr Community Expert Fix filesytem on the emulated disk11 (md11): https://wiki.lime-technology.com/Check_Disk_Filesystems#Drives_formatted_with_XFS Disk11 has pending sectors, replace it with a new one and rebuild.
June 30, 20179 yr Author xfs_repair -v /dev/md11 -bash: root@Tower:~#: command not found - block cache size set to 676816 entries root@Tower:~# Phase 1 - find and verify superblock... Phase 2 - using internal log -bash: Phase: command not found - zero log... root@Tower:~# - block cache size set to 676816 entries Log inconsistent or not a log (last==0, first!=1) empty log check failed zero_log: cannot find log head/tail (xlog_find_tail=22) -bash: -: command not found fatal error -- ERROR: The log head and/or tail cannot be discovered. Attempt to mount the filesystem to replay the log or use the -L option to destroy the log and attempt a repair.
June 30, 20179 yr Author sorry this should be the correct copy - more or less the same thing: xfs_repair -v /dev/md11 Phase 1 - find and verify superblock... - block cache size set to 676816 entries Phase 2 - using internal log - zero log... Log inconsistent or not a log (last==0, first!=1) empty log check failed zero_log: cannot find log head/tail (xlog_find_tail=22) fatal error -- ERROR: The log head and/or tail cannot be discovered. Attempt to mount the filesystem to replay the log or use the -L option to destroy the log and attempt a repair. Should I try a repair on the physical disk - sdi or destory the log?
June 30, 20179 yr Author inode identifier 0 mismatch on inode 7297414737 bad CRC for inode 7297414738 bad magic number 0x0 on inode 7297414738 bad version number 0x0 on inode 7297414738 inode identifier 0 mismatch on inode 7297414738 bad CRC for inode 7297414739 bad magic number 0x0 on inode 7297414739 bad version number 0x0 on inode 7297414739 inode identifier 0 mismatch on inode 7297414739 bad CRC for inode 7297414740 bad magic number 0x0 on inode 7297414740 bad version number 0x0 on inode 7297414740 inode identifier 0 mismatch on inode 7297414740 bad CRC for inode 7297414741 bad magic number 0x0 on inode 7297414741 bad version number 0x0 on inode 7297414741 inode identifier 0 mismatch on inode 7297414741 bad CRC for inode 7297414742 bad magic number 0x0 on inode 7297414742 bad version number 0x0 on inode 7297414742 inode identifier 0 mismatch on inode 7297414742 bad CRC for inode 7297414743 bad magic number 0x0 on inode 7297414743 bad version number 0x0 on inode 7297414743 inode identifier 0 mismatch on inode 7297414743 bad CRC for inode 7297414744 bad magic number 0x0 on inode 7297414744 bad version number 0x0 on inode 7297414744 inode identifier 0 mismatch on inode 7297414744 bad CRC for inode 7297414745 bad magic number 0x0 on inode 7297414745 bad version number 0x0 on inode 7297414745 inode identifier 0 mismatch on inode 7297414745 bad CRC for inode 7297414746 bad magic number 0x0 on inode 7297414746 bad version number 0x0 on inode 7297414746 inode identifier 0 mismatch on inode 7297414746 bad CRC for inode 7297414747 bad magic number 0x0 on inode 7297414747 bad version number 0x0 on inode 7297414747 inode identifier 0 mismatch on inode 7297414747 bad CRC for inode 7297414748 bad magic number 0x0 on inode 7297414748 bad version number 0x0 on inode 7297414748 inode identifier 0 mismatch on inode 7297414748 bad CRC for inode 7297414749 bad magic number 0x0 on inode 7297414749 bad version number 0x0 on inode 7297414749 inode identifier 0 mismatch on inode 7297414749 bad CRC for inode 7297414750 bad magic number 0x0 on inode 7297414750 bad version number 0x0 on inode 7297414750 inode identifier 0 mismatch on inode 7297414750 bad CRC for inode 7297414751 bad magic number 0x0 on inode 7297414751 bad version number 0x0 on inode 7297414751 inode identifier 0 mismatch on inode 7297414751 bad CRC for inode 7297414752 bad magic number 0x0 on inode 7297414752 bad version number 0x0 on inode 7297414752 inode identifier 0 mismatch on inode 7297414752 bad CRC for inode 7297414753 bad magic number 0x0 on inode 7297414753 bad version number 0x0 on inode 7297414753 inode identifier 0 mismatch on inode 7297414753 bad CRC for inode 7297414754 bad magic number 0x0 on inode 7297414754 bad version number 0x0 on inode 7297414754 inode identifier 0 mismatch on inode 7297414754 bad CRC for inode 7297414755 bad magic number 0x0 on inode 7297414755 bad version number 0x0 on inode 7297414755 inode identifier 0 mismatch on inode 7297414755 bad CRC for inode 7297414756 bad magic number 0x0 on inode 7297414756 bad version number 0x0 on inode 7297414756 inode identifier 0 mismatch on inode 7297414756 bad CRC for inode 7297414757 bad magic number 0x0 on inode 7297414757 bad version number 0x0 on inode 7297414757 inode identifier 0 mismatch on inode 7297414757 bad CRC for inode 7297414758 bad magic number 0x0 on inode 7297414758 bad version number 0x0 on inode 7297414758 inode identifier 0 mismatch on inode 7297414758 bad CRC for inode 7297414759 bad magic number 0x0 on inode 7297414759 bad version number 0x0 on inode 7297414759 inode identifier 0 mismatch on inode 7297414759 bad CRC for inode 7297414760 bad magic number 0x0 on inode 7297414760 bad version number 0x0 on inode 7297414760 inode identifier 0 mismatch on inode 7297414760 bad CRC for inode 7297414761 bad magic number 0x0 on inode 7297414761 bad version number 0x0 on inode 7297414761 inode identifier 0 mismatch on inode 7297414761 bad CRC for inode 7297414762 bad magic number 0x0 on inode 7297414762 bad version number 0x0 on inode 7297414762 inode identifier 0 mismatch on inode 7297414762 bad CRC for inode 7297414763 bad magic number 0x0 on inode 7297414763 bad version number 0x0 on inode 7297414763 inode identifier 0 mismatch on inode 7297414763 bad CRC for inode 7297414764 bad magic number 0x0 on inode 7297414764 bad version number 0x0 on inode 7297414764 inode identifier 0 mismatch on inode 7297414764 bad CRC for inode 7297414765 bad magic number 0x0 on inode 7297414765 bad version number 0x0 on inode 7297414765 inode identifier 0 mismatch on inode 7297414765 bad CRC for inode 7297414766 bad magic number 0x0 on inode 7297414766 bad version number 0x0 on inode 7297414766 inode identifier 0 mismatch on inode 7297414766 bad CRC for inode 7297414767 bad magic number 0x0 on inode 7297414767 bad version number 0x0 on inode 7297414767 inode identifier 0 mismatch on inode 7297414767 bad CRC for inode 7297414768 bad magic number 0x0 on inode 7297414768 bad version number 0x0 on inode 7297414768 inode identifier 0 mismatch on inode 7297414768 bad CRC for inode 7297414769 bad magic number 0x0 on inode 7297414769 bad version number 0x0 on inode 7297414769 inode identifier 0 mismatch on inode 7297414769 bad CRC for inode 7297414770 bad magic number 0x0 on inode 7297414770 bad version number 0x0 on inode 7297414770 inode identifier 0 mismatch on inode 7297414770 bad CRC for inode 7297414771 bad magic number 0x0 on inode 7297414771 bad version number 0x0 on inode 7297414771 inode identifier 0 mismatch on inode 7297414771 bad CRC for inode 7297414772 bad magic number 0x0 on inode 7297414772 bad version number 0x0 on inode 7297414772 inode identifier 0 mismatch on inode 7297414772 bad CRC for inode 7297414773 bad magic number 0x0 on inode 7297414773 bad version number 0x0 on inode 7297414773 inode identifier 0 mismatch on inode 7297414773 bad CRC for inode 7297414774 bad magic number 0x0 on inode 7297414774 bad version number 0x0 on inode 7297414774 inode identifier 0 mismatch on inode 7297414774 bad CRC for inode 7297414775 bad magic number 0x0 on inode 7297414775 bad version number 0x0 on inode 7297414775 inode identifier 0 mismatch on inode 7297414775 bad CRC for inode 7297414776 bad magic number 0x0 on inode 7297414776 bad version number 0x0 on inode 7297414776 inode identifier 0 mismatch on inode 7297414776 bad CRC for inode 7297414777 bad magic number 0x0 on inode 7297414777 bad version number 0x0 on inode 7297414777 inode identifier 0 mismatch on inode 7297414777 bad CRC for inode 7297414778 bad magic number 0x0 on inode 7297414778 bad version number 0x0 on inode 7297414778 inode identifier 0 mismatch on inode 7297414778 bad CRC for inode 7297414779 bad magic number 0x0 on inode 7297414779 bad version number 0x0 on inode 7297414779 inode identifier 0 mismatch on inode 7297414779 bad CRC for inode 7297414780 bad magic number 0x0 on inode 7297414780 bad version number 0x0 on inode 7297414780 inode identifier 0 mismatch on inode 7297414780 bad CRC for inode 7297414781 bad magic number 0x0 on inode 7297414781 bad version number 0x0 on inode 7297414781 inode identifier 0 mismatch on inode 7297414781 bad CRC for inode 7297414782 bad magic number 0x0 on inode 7297414782 bad version number 0x0 on inode 7297414782 inode identifier 0 mismatch on inode 7297414782 bad CRC for inode 7297414783 bad magic number 0x0 on inode 7297414783 bad version number 0x0 on inode 7297414783 inode identifier 0 mismatch on inode 7297414783 - process newly discovered inodes... 2abfd65e7ec0: Badness in key lookup (length) bp=(bno 0xe93a6fd0, len 8192 bytes) key=(bno 0xe93a6fd0, len 4096 bytes) 2abfd65e7ec0: Badness in key lookup (length) bp=(bno 0xe93a6fe0, len 8192 bytes) key=(bno 0xe93a6fe0, len 4096 bytes) 2abfd65e7ec0: Badness in key lookup (length) bp=(bno 0xe93a6ff0, len 8192 bytes) key=(bno 0xe93a6ff0, len 4096 bytes) 2abfd65e7ec0: Badness in key lookup (length) bp=(bno 0xe93a7000, len 8192 bytes) key=(bno 0xe93a7000, len 4096 bytes) Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 1 - agno = 0 entry "logs" in shortform directory 2147483744 references free inode 4299589491 junking entry "logs" in directory inode 2147483744 corrected i8 count in directory 2147483744, was 3, now 2 entry "backups" in shortform directory 96 references non-existent inode 6443893447 junking entry "backups" in directory inode 96 data fork in ino 2147483748 claims dup extent, off - 0, start - 302323328, cnt 240 correcting nextents for inode 2147483748 bad data fork in inode 2147483748 cleared inode 2147483748 corrected i8 count in directory 96, was 1, now 0 entry "Series" in shortform directory 2147483751 references free inode 4298782819 junking entry "Series" in directory inode 2147483751 corrected i8 count in directory 2147483751, was 1, now 0 corrected directory 2147483751 size, was 49, now 37 bogus .. inode number (0) in directory inode 2147483751, clearing inode number corrected directory 96 size, was 64, now 48 bogus .. inode number (0) in directory inode 96, clearing inode number xfs_repair: dir2.c:1419: process_dir2: Assertion `(ino != mp->m_sb.sb_rootino && ino != *parent) || (ino == mp->m_sb.sb_rootino && (ino == *parent || need_root_dotdot == 1))' failed. Aborted root@Tower:~#
June 30, 20179 yr Author running xfs_Repair on the drive - looks like it is trying to find a backup of the superblock. If I can recover that drive, how do I mount it to get data off of it?
June 30, 20179 yr Community Expert Can't see because the logs are after reboot but you were probably running a correcting check when disk11 failed, it may have corrupted parity making the emulated disk corrupt also. If xfs_repair can't fix the emulated disk, best bet is to see if the old disk mounts with the UD plugin and if so try to copy everything you can to the array or other PC, start with the most important data first.
June 30, 20179 yr Author never mind figured it out - unassigned device plugin from community plugins. not showing up under there, but the XFS_Repair on it is still running.. see what happens. Thanks for the fast help. last file I have on the array is from september 2016 - almost a year of data on that drive. should of replaced it sooner.
June 30, 20179 yr Community Expert 31 minutes ago, heyme said: yes that is what happened Parity check should always be non-correct unless sync errors are expect, like after an unclean shutdown. 19 minutes ago, heyme said: but the XFS_Repair on it is still running You may wait for xfs_repair to finish, but it will take several hours and probably fail since there appears to be metadata corruption. Then, you'll need to unassign the disk from the array for it to be available as an unassigned device: -stop the array -unassign disk11 (select no device) -start the array Old disk11 should now be available in the plugin.
July 1, 20179 yr Author Since my parity is now corrupted, how do i force it to rebuild based on good disks?
July 1, 20179 yr Community Expert You can rebuild disk11 with a new disk and format it, this will make parity in sync again.
July 1, 20179 yr Author If I rebuild onto that disk won't it rebuild the corrupted MD11 image on it? how do I format a disk once it is in the array?
July 1, 20179 yr Community Expert 1 minute ago, heyme said: won't it rebuild the corrupted MD11 image on it? Yes, but since it's going to be formatted doesn't matter and it will keep parity in sync, because the disk is unmountable there will be an option to format it next to the start array button. Other option is to do new config with a new disk11 and re-sync parity.
July 1, 20179 yr Author okay. since it is a 4TB drive, to wait for a rebuild just to format it again, then copy the data back to it will just take days. Probaby do a new config. Thanks for the help
Archived
This topic is now archived and is closed to further replies.