October 30, 20241 yr Hi all, luck struck me twice the last two days, at first one of my data drives suddenly failed. Today the replacement arrived, after following the procedure to replace the drive i could not start the machine back up. Turns out the flash drive also died some time in the last month. Of course i did not think of backing up the config folder, but i took a screenshot of the drive list before going in to replace the drive. Im now on a new usb drive, i have all the drives back in there old slots and there is a disc clear in process for the new drive. When starting the array i selected that parity is already valid in order for the parity drive to not be deleted. Now to my question, will the new drive take the place of the failed drive even though i lost my previous configuration? Thanks Edited October 30, 20241 yr by nolibert
October 31, 20241 yr Community Expert You should not have started the array, but it may still be possible, try this: -Stop array -Unassign the disk you want to rebuild -Start array and post the diagnostics.
October 31, 20241 yr Author Hi, ok that sounds bad, here are the diagnostics. Thanks epic-diagnostics-20241031-1005.zip
October 31, 20241 yr Community Expert That won't work because disk3 was still not part of the array, do this: -Tools -> New Config -> Retain current configuration: All -> Apply -Check all assignments and assign any missing disk(s) if needed, including the new disk3, replacement disk should be same size or larger than the old one -IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked) -Stop array -Unassign disk3 -Start array (in normal mode now) and post new diags.
October 31, 20241 yr Author Hi, went through all the described steps, i attached the new diagnostics. disk3 now says not installed, but no data is beeing emulated it says content is beeing emulated Thanks for your help! epic-diagnostics-20241031-1050.zip Edited October 31, 20241 yr by nolibert
October 31, 20241 yr Community Expert Solution Post the output from: xfs_repair -v /dev/md3p1 P.S. there appears to be a container constantly restarting spamming the log, you should be able to find out which one is it by looking at their up times.
October 31, 20241 yr Author Hi, thanks, there is a immich microservice which someshow explodes all the time, i disabled it Output from above: Phase 1 - find and verify superblock... bad primary superblock - bad magic number !!! attempting to find secondary superblock.. does not look like its finding anything
October 31, 20241 yr Community Expert I'm afraid that parity is no longer able to emulate the disk, if the old disk is not completely dead you can try to recover some data from there.
October 31, 20241 yr Author i just went through all the steps again and taking a second look at my drive configuration, my parity drive was mounted as a second parity drive since changing it out for a bigger one some time ago. i redid the steps you described and also redid the ssh command. I dont have full output since the webui cuts the output but here are the last infos i have after it found a corrupted superblock: bp=(bno 0x24919560, len 16384 bytes) key=(bno 0x24919560, len 4096 bytes) Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 entry "wheel-0.36.2.dist-info" at block 0 offset 376 in directory inode 2232600014 references non-existent inode 613037589 clearing inode number in entry at offset 376... entry "certifi" at block 0 offset 456 in directory inode 2232600052 references non-existent inode 613037376 clearing inode number in entry at offset 456... entry "colorama" at block 0 offset 504 in directory inode 2232600052 references non-existent inode 613037384 clearing inode number in entry at offset 504... entry "html5lib" at block 0 offset 552 in directory inode 2232600052 references non-existent inode 613037391 clearing inode number in entry at offset 552... entry "packaging" at block 0 offset 616 in directory inode 2232600052 references non-existent inode 613037420 clearing inode number in entry at offset 616... entry "pep517" at block 0 offset 640 in directory inode 2232600052 references non-existent inode 613037432 clearing inode number in entry at offset 640... entry "progress" at block 0 offset 696 in directory inode 2232600052 references non-existent inode 613037442 clearing inode number in entry at offset 696... entry "requests" at block 0 offset 720 in directory inode 2232600052 references non-existent inode 613037447 clearing inode number in entry at offset 720... entry "tenacity" at block 0 offset 768 in directory inode 2232600052 references non-existent inode 613037466 clearing inode number in entry at offset 768... entry "toml" at block 0 offset 792 in directory inode 2232600052 references non-existent inode 613037479 clearing inode number in entry at offset 792... entry "webencodings" at block 0 offset 832 in directory inode 2232600052 references non-existent inode 613037569 clearing inode number in entry at offset 832... entry "metadata" at block 0 offset 2792 in directory inode 2232600059 references non-existent inode 613037381 clearing inode number in entry at offset 2792... entry "tests" in shortform directory 2232631694 references non-existent inode 613037575 junking entry "tests" in directory inode 2232631694 entry "extern" at block 0 offset 2216 in directory inode 2232631698 references non-existent inode 613037597 clearing inode number in entry at offset 2216... entry "_distutils" at block 0 offset 2240 in directory inode 2232631698 references non-existent inode 613037599 clearing inode number in entry at offset 2240... entry ".." at block 0 offset 80 in directory inode 2232631733 references non-existent inode 613037599 entry "packages" at block 0 offset 792 in directory inode 2232631871 references non-existent inode 613037489 clearing inode number in entry at offset 792... entry "util" at block 0 offset 816 in directory inode 2232631871 references non-existent inode 613037492 clearing inode number in entry at offset 816... entry "__pycache__" at block 0 offset 1264 in directory inode 2270779264 references non-existent inode 613518622 clearing inode number in entry at offset 1264... entry "colorama-0.4.6.dist-info" at block 0 offset 672 in directory inode 613514397 references non-existent inode 613518600 clearing inode number in entry at offset 672... entry "distutils-precedence.pth" at block 0 offset 760 in directory inode 613514397 references non-existent inode 613518605 clearing inode number in entry at offset 760... entry "exceptiongroup-1.1.1.dist-info" at block 0 offset 832 in directory inode 613514397 references non-existent inode 613518612 clearing inode number in entry at offset 832... entry "idna" at block 0 offset 1016 in directory inode 613514397 references non-existent inode 613520725 clearing inode number in entry at offset 1016... entry "itsdangerous-2.1.2.dist-info" at block 0 offset 1088 in directory inode 613514397 references non-existent inode 613520753 clearing inode number in entry at offset 1088... entry "Jinja2-3.1.2.dist-info" at block 0 offset 1152 in directory inode 613514397 references non-existent inode 613520760 clearing inode number in entry at offset 1152... entry "__init__.py" in shortform directory 613517178 references non-existent inode 613518592 junking entry "__init__.py" in directory inode 613517178 entry "__pycache__" in shortform directory 613517178 references non-existent inode 613518593 junking entry "__pycache__" in directory inode 613517178 entry "operations" at block 0 offset 872 in directory inode 1322853918 references non-existent inode 613036808 clearing inode number in entry at offset 872... entry "req" at block 0 offset 896 in directory inode 1322853918 references non-existent inode 613036813 clearing inode number in entry at offset 896... entry "resolution" at block 0 offset 912 in directory inode 1322853918 references non-existent inode 613036821 clearing inode number in entry at offset 912... entry "utils" at block 0 offset 936 in directory inode 1322853918 references non-existent inode 613036824 clearing inode number in entry at offset 936... entry "vcs" at block 0 offset 960 in directory inode 1322853918 references non-existent inode 613036853 clearing inode number in entry at offset 960... entry "auth.py" in shortform directory 1322853951 references non-existent inode 613036800 junking entry "auth.py" in directory inode 1322853951 entry "cache.py" in shortform directory 1322853951 references non-existent inode 613036801 junking entry "cache.py" in directory inode 1322853951 entry "download.py" in shortform directory 1322853951 references non-existent inode 613036802 junking entry "download.py" in directory inode 1322853951 entry "lazy_wheel.py" in shortform directory 1322853951 references non-existent inode 613036803 junking entry "lazy_wheel.py" in directory inode 1322853951 entry "session.py" in shortform directory 1322853951 references non-existent inode 613036804 junking entry "session.py" in directory inode 1322853951 entry "utils.py" in shortform directory 1322853951 references non-existent inode 613036805 junking entry "utils.py" in directory inode 1322853951 entry "xmlrpc.py" in shortform directory 1322853951 references non-existent inode 613036806 junking entry "xmlrpc.py" in directory inode 1322853951 entry "__init__.py" in shortform directory 1322853951 references non-existent inode 613036807 junking entry "__init__.py" in directory inode 1322853951 entry "caches" in shortform directory 4773197967 references non-existent inode 613036860 junking entry "caches" in directory inode 4773197967 entry "_securetransport" in shortform directory 4773198006 references non-existent inode 613037485 junking entry "_securetransport" in directory inode 4773198006 entry "packaging" in shortform directory 4773317763 references non-existent inode 613037577 junking entry "packaging" in directory inode 4773317763 entry "more_itertools" in shortform directory 4773317793 references non-existent inode 613037630 junking entry "more_itertools" in directory inode 4773317793 entry "__pycache__" in shortform directory 4774899489 references non-existent inode 613518606 junking entry "__pycache__" in directory inode 4774899489 entry "__pycache__" in shortform directory 4774899496 references non-existent inode 613518618 junking entry "__pycache__" in directory inode 4774899496 entry "protobuf" in shortform directory 4774899500 references non-existent inode 613518642 junking entry "protobuf" in directory inode 4774899500 entry ".." at block 0 offset 80 in directory inode 4774899501 references non-existent inode 613518642 entry "__pycache__" at block 0 offset 1704 in directory inode 4774899501 references non-existent inode 613518655 clearing inode number in entry at offset 1704... entry "__pycache__" in shortform directory 4774900163 references non-existent inode 613520615 junking entry "__pycache__" in directory inode 4774900163 entry "data" at block 0 offset 96 in directory inode 4774900167 references non-existent inode 613520711 clearing inode number in entry at offset 96... entry "__pycache__" at block 0 offset 872 in directory inode 4774900167 references non-existent inode 613520713 clearing inode number in entry at offset 872... entry "__pycache__" in shortform directory 4774900186 references non-existent inode 613520744 junking entry "__pycache__" in directory inode 4774900186 Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - reset superblock... Phase 6 - check inode connectivity... reinitializing root directory - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 bad hash table for directory inode 613514397 (no data entry): rebuilding rebuilding directory inode 613514397 bad hash table for directory inode 1322853918 (no data entry): rebuilding rebuilding directory inode 1322853918 - agno = 1 bad hash table for directory inode 2232600014 (no data entry): rebuilding rebuilding directory inode 2232600014 bad hash table for directory inode 2232600052 (no data entry): rebuilding rebuilding directory inode 2232600052 bad hash table for directory inode 2232600059 (no data entry): rebuilding rebuilding directory inode 2232600059 bad hash table for directory inode 2232631698 (no data entry): rebuilding rebuilding directory inode 2232631698 entry ".." in directory inode 2232631733 points to non-existent inode 613037599, marking entry to be junked bad hash table for directory inode 2232631733 (no data entry): rebuilding rebuilding directory inode 2232631733 bad hash table for directory inode 2232631871 (no data entry): rebuilding rebuilding directory inode 2232631871 bad hash table for directory inode 2270779264 (no data entry): rebuilding rebuilding directory inode 2270779264 - agno = 2 entry ".." in directory inode 4774899501 points to non-existent inode 613518642, marking entry to be junked bad hash table for directory inode 4774899501 (no data entry): rebuilding rebuilding directory inode 4774899501 bad hash table for directory inode 4774900167 (no data entry): rebuilding rebuilding directory inode 4774900167 - agno = 3 - traversal finished ... - moving disconnected inodes to lost+found ... disconnected dir inode 132, moving to lost+found disconnected dir inode 66784781, moving to lost+found disconnected dir inode 148725908, moving to lost+found disconnected inode 613041536, moving to lost+found disconnected inode 613041537, moving to lost+found disconnected dir inode 623393521, moving to lost+found disconnected dir inode 1322853905, moving to lost+found disconnected dir inode 2232600043, moving to lost+found disconnected dir inode 2232600049, moving to lost+found disconnected dir inode 2232631691, moving to lost+found disconnected dir inode 2232631696, moving to lost+found disconnected dir inode 2232631733, moving to lost+found disconnected dir inode 2232631849, moving to lost+found disconnected dir inode 2232631862, moving to lost+found disconnected dir inode 2270777534, moving to lost+found disconnected dir inode 2270779294, moving to lost+found disconnected dir inode 2270779297, moving to lost+found disconnected dir inode 4298782848, moving to lost+found disconnected dir inode 4715809071, moving to lost+found disconnected dir inode 4773197856, moving to lost+found disconnected dir inode 4773197952, moving to lost+found disconnected dir inode 4773197957, moving to lost+found disconnected dir inode 4773197988, moving to lost+found disconnected dir inode 4773198014, moving to lost+found disconnected dir inode 4774899501, moving to lost+found disconnected dir inode 4774900163, moving to lost+found disconnected dir inode 6685468803, moving to lost+found disconnected dir inode 7009614330, moving to lost+found Phase 7 - verify and correct link counts... resetting inode 139 nlinks from 78 to 1 resetting inode 175 nlinks from 208 to 1 resetting inode 65472941 nlinks from 2 to 28 resetting inode 2232600014 nlinks from 14 to 13 resetting inode 2232600052 nlinks from 20 to 10 resetting inode 2232600059 nlinks from 4 to 3 resetting inode 2232631694 nlinks from 5 to 4 resetting inode 2232631698 nlinks from 6 to 4 resetting inode 2232631871 nlinks from 5 to 3 resetting inode 2270779264 nlinks from 4 to 3 resetting inode 4773197967 nlinks from 3 to 2 resetting inode 613514397 nlinks from 76 to 71 resetting inode 4773198006 nlinks from 3 to 2 resetting inode 613517178 nlinks from 4 to 3 resetting inode 4773317763 nlinks from 3 to 2 resetting inode 4773317793 nlinks from 4 to 3 resetting inode 4774899489 nlinks from 3 to 2 resetting inode 4774899496 nlinks from 3 to 2 resetting inode 4774899500 nlinks from 3 to 2 resetting inode 4774899501 nlinks from 3 to 2 resetting inode 4774900163 nlinks from 3 to 2 resetting inode 4774900167 nlinks from 4 to 2 resetting inode 4774900186 nlinks from 3 to 2 resetting inode 1322853918 nlinks from 15 to 10 Note - stripe unit (0) and width (0) were copied from a backup superblock. Please reset with mount -o sunit=<value>,swidth=<value> if necessary Maximum metadata LSN (1:8879305) is ahead of log (1:1664872). Format log to cycle 4. XFS_REPAIR Summary Thu Oct 31 05:26:05 2024 Phase Start End Duration Phase 1: 10/31 05:22:32 10/31 05:22:33 1 second Phase 2: 10/31 05:22:33 10/31 05:22:35 2 seconds Phase 3: 10/31 05:22:35 10/31 05:23:44 1 minute, 9 seconds Phase 4: 10/31 05:23:44 10/31 05:23:45 1 second Phase 5: 10/31 05:23:45 10/31 05:23:46 1 second Phase 6: 10/31 05:23:46 10/31 05:24:38 52 seconds Phase 7: 10/31 05:24:38 10/31 05:24:38 Total run time: 2 minutes, 6 seconds done does this look good? UPDATE: Restarted the array and now i have a lot of lost and found folders that include parts of my files, looks like there is a lot of stuff im missing here. Is there a easy way to get this back in its original place or should i simply move it by hand? Thanks again btw. you saved me quite some headaches Edited October 31, 20241 yr by nolibert
October 31, 20241 yr Community Expert 25 minutes ago, nolibert said: my parity drive was mounted as a second parity drive since changing it out for a bigger one some time ago Yep, in that case it wouldn't work as parity1, I did find strange at least some remnants of a filesystem not being found. 25 minutes ago, nolibert said: Is there a easy way to get this back in its original place or should i simply move it by hand? I'm afraid that it will have to be done by hand.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.