VACInc

Hitting this same issue unfortunately. I'll be rolling back for the time being as well @JorgeB Any idea when 7.0.1 will be on the table? or a beta for 7.0.1?

Looks related to

Looks like it's picking up the USB and starting the boot process, but then suddenly stops with: mount: proc mounted on /proc mount: sysfs mounted on /sys. mount: /sys/firmware/efi/efivars: unknown filesystem type 'efivarfs'. dmesg(1) may have more information after failed mount system call. mount: tmpfs mounted on /run. mount: none mounted on /sys/kernel/debug. waiting up to 30 seconds for device with label UNRAID to come online ... umount: /: not mounted. not found - press ENTER key to reboot... I verified the USB drive is labeled properly: sdw 29.3G disk └─sdw1 UNRAID 29.2G part

THANK YOU @trurl and @JorgeB! I have one final question to complete my journey here. Now that the data has been moved from the missing drive (as well as any other drive I want to remove from the array, 4 total); I should follow the steps here: https://docs.unraid.net/legacy/FAQ/shrink-array/#the-remove-drives-then-rebuild-parity-method That said, I also want to replace the parity drives -- can I follow the steps here but use new parity drives in the new config (remove the current ones) while also getting rid of the disks I want to remove? Or would I have to create a new config first including the original parity drives, and only after that replace the parity drives?

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x469f20, xfs_agi block 0x2/0x200 agi has bad CRC for ag 0 clearing needsrepair flag and regenerating metadata Metadata CRC error detected at 0x47191d, xfs_inobt block 0x18/0x1000 btree block 0/3 is suspect, error -74 Metadata CRC error detected at 0x47191d, xfs_finobt block 0x20/0x1000 btree block 0/4 is suspect, error -74 undiscovered finobt record, ino 128 (0/128) undiscovered finobt record, ino 169789760 (0/169789760) undiscovered finobt record, ino 212608960 (0/212608960) undiscovered finobt record, ino 222122432 (0/222122432) undiscovered finobt record, ino 236685632 (0/236685632) undiscovered finobt record, ino 237556416 (0/237556416) undiscovered finobt record, ino 277155008 (0/277155008) undiscovered finobt record, ino 291104832 (0/291104832) undiscovered finobt record, ino 296201280 (0/296201280) sb_fdblocks 4273345783, counted 4307396050 root inode chunk not found Phase 3 - for each AG... - scan and clear agi unlinked lists... found inodes not in the inode allocation tree found inodes not in the inode allocation tree - process known inodes and perform inode discovery... - agno = 0 15156cc4c680: Badness in key lookup (length) bp=(bno 0xa1ec940, len 4096 bytes) key=(bno 0xa1ec940, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0xa1ec960, len 4096 bytes) key=(bno 0xa1ec960, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0xcac27c0, len 4096 bytes) key=(bno 0xcac27c0, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0xcac27e0, len 4096 bytes) key=(bno 0xcac27e0, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0xcbae4c0, len 4096 bytes) key=(bno 0xcbae4c0, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0xcbae4e0, len 4096 bytes) key=(bno 0xcbae4e0, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0xd3d51c0, len 4096 bytes) key=(bno 0xd3d51c0, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0xd3d51e0, len 4096 bytes) key=(bno 0xd3d51e0, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0xe1b8940, len 4096 bytes) key=(bno 0xe1b8940, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0xe1b8960, len 4096 bytes) key=(bno 0xe1b8960, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0xe28d2c0, len 4096 bytes) key=(bno 0xe28d2c0, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0xe28d2e0, len 4096 bytes) key=(bno 0xe28d2e0, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0x10850cc0, len 4096 bytes) key=(bno 0x10850cc0, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0x10850ce0, len 4096 bytes) key=(bno 0x10850ce0, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0x1159e840, len 4096 bytes) key=(bno 0x1159e840, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0x1159e860, len 4096 bytes) key=(bno 0x1159e860, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0x11a7ac40, len 4096 bytes) key=(bno 0x11a7ac40, len 16384 bytes) 15156cc4c680: Badness in key lookup (length) bp=(bno 0x11a7ac60, len 4096 bytes) key=(bno 0x11a7ac60, len 16384 bytes) bad CRC for inode 128 bad CRC for inode 131 bad CRC for inode 132 bad CRC for inode 133 bad CRC for inode 128, will rewrite Bad mtime nsec 3332600597 on inode 128, resetting to zero Bad ctime nsec 1212846252 on inode 128, resetting to zero cleared root inode 128 bad CRC for inode 131, will rewrite Bad atime nsec 3628754445 on inode 131, resetting to zero Bad mtime nsec 3823352658 on inode 131, resetting to zero Bad ctime nsec 4201589378 on inode 131, resetting to zero Bad crtime nsec 4033325188 on inode 131, resetting to zero correcting imap cleared inode 131 bad CRC for inode 132, will rewrite correcting imap cleared inode 132 bad CRC for inode 133, will rewrite Bad atime nsec 1946158217 on inode 133, resetting to zero Bad mtime nsec 1991492530 on inode 133, resetting to zero Bad ctime nsec 1991492530 on inode 133, resetting to zero Bad crtime nsec 1946158217 on inode 133, resetting to zero correcting imap cleared inode 133 correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap imap claims a free inode 237556416 is in use, correcting imap and clearing inode cleared inode 237556416 correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap - agno = 1 bad CRC for inode 2147483777 bad CRC for inode 2147483777, will rewrite free inode 2147483777 contains errors, corrected - agno = 2 - agno = 3 bad CRC for inode 6442451073 bad CRC for inode 6442451073, will rewrite cleared inode 6442451073 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - agno = 16 - agno = 17 - agno = 18 - agno = 19 - agno = 20 - agno = 21 - agno = 22 - agno = 23 - agno = 24 - agno = 25 - agno = 26 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 4 - agno = 6 - agno = 12 - agno = 5 - agno = 17 - agno = 18 - agno = 8 - agno = 22 - agno = 11 - agno = 24 - agno = 25 - agno = 26 - agno = 13 - agno = 15 - agno = 14 - agno = 2 - agno = 16 - agno = 7 - agno = 19 - agno = 20 - agno = 21 - agno = 23 - agno = 10 - agno = 9 - agno = 3 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... reinitializing root directory - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... disconnected dir inode 131, moving to lost+found Phase 7 - verify and correct link counts... resetting inode 139 nlinks from 2 to 3 Maximum metadata LSN (1:1368614) is ahead of log (1:2). Format log to cycle 4. done

@trurl Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

@JorgeB it was a parity sync restoring parity1 @trurl Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... would write modified primary superblock Primary superblock would have been modified. Cannot proceed further in no_modify mode. Exiting now.

@JorgeB Attached! Again, thanks for your patience with this normandy-diagnostics-20250113-1406.zip

@JorgeB It's not showing it's emulated though just faulty/not installed. Can emulation be forced?

@JorgeB Even if I get the second parity drive healthy again? There is no way around needing a drive in "dish 9"?

@JorgeB Disk9 is lost. But there was data on it. I assumed that data still wouldn't be actually lost though because there was still a parity drive functioning, am I correct?

Okay, I've stabilized things for the time being by using a separate JBOD and controller for the "extra" drives. That said, one drive had to be taken out of the equation so I'll have 2 parity drives fully healthy and still a missing "disk 9" (out of 14 data disks). The missing drive cannot be returned as it seems to be faulty (scanned on a separate system, needs to be returned). I don't need that extra space so can I create a new config without disk 9 and without any additional empty drives and move on with life? I assume no data is lost at this point correct? Thanks again for your help everyone

Though I'm just realizing now; that if that is part of the issue it also means the LSI HBA handles the issues externally poorly for the other connected JBOD (because the parity drives are in the SA120, and the rest of the drives with read errors are not). Not unimaginable; but still surprising, I would think it would be more robust and less prone to stability issues.

That would also be interesting...potentially a power issue between because of differentials. Most the the drives are powered via the SA120 (12 drives -- never had an issue), and then 4 are external and just running off a consumer NZXT C1200. They're also using a generic (and sketchy?) SFF-8088 to 4x SATA cable, which is what my brain keeps thinking is potentially problematic. I have a QNAS 4x mini JBOD showing up tomorrow, that'll allow me to switch both power and the cable out; hopefully that gets us to the destination.

Correct; but what is very weird is they only appear when doing a parity sync -- reading from those drives are perfectly fine otherwise (zero read errors when reading from when not doing a sync). This is what lead me to believe there's an issue somewhere in between the disks and the motherboard (either the controller or the cabling) -- I have a new one coming soon. In the meantime, based on that, I assume my data should all still be present correct (assuming no additional drive commits harakiri)