kahlif

Uploading diags, but history: Had 8 x 20TB array. About 4 months ago, added 4 x 24TB, 2 upped to parity then the other 2 added to array. Everything was running well, but now 3 drives reporting issues. The 2 24TB drives (disks 9 and 10) disabled, then when I try to do the recovery, disk 6 (one of the 20TB drives) is reporting read errors like crazy. Extended smart tests on all drives passed without issues, and original setup ran all tests fine. Now, luckily, 9 & 10 were empty still (I use spillover) but 6 has data. When trying xfs_repair on all 3 disks, I get the same: Phase 1 - find and verify superblock... superblock read failed, offset 0, size 524288, ag 0, rval -1 fatal error -- Input/output errorIdeas? akanas-safemode-diagnostics-20260103-2158.zip

looks like adding the containers back 1 at a time using the profiles has everything saved, so we are all good!

One thing - the list of docker containers is empty... Any way to get that config back?

Holy cow!!! Did the same to the others, unmounted then remounted, and it looks like its all back!!! TYSVM!!!!!

Looking good! Created a new DOS (MBR) disklabel with disk identifier 0x50d4d50d. Created a new partition 1 of type 'Linux' and of size 953.9 GiB. Partition #1 contains a btrfs signature. Do you want to remove the signature? [Y]es/[N]o: n /dev/nvme0n1p1 : 2048 2000409263 (953.9G) Linux /dev/nvme0n1p2: write New situation: Disklabel type: dos Disk identifier: 0x50d4d50d Device Boot Start End Sectors Size Id Type /dev/nvme0n1p1 2048 2000409263 2000407216 953.9G 83 Linux The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks. root@akanas:~# btrfs fi show Label: none uuid: 444cab5a-d33c-4a5c-810d-80753362c808 Total devices 1 FS bytes used 372.00KiB devid 1 size 20.00GiB used 536.00MiB path /dev/loop2 Label: none uuid: b6a29e09-4796-492e-81fd-fa22626f0335 Total devices 1 FS bytes used 416.00KiB devid 1 size 1.00GiB used 126.38MiB path /dev/loop3 Label: none uuid: fa55b70a-6dca-4a0e-86e3-229e4f7755ae Total devices 1 FS bytes used 669.72GiB devid 1 size 953.87GiB used 674.02GiB path /dev/nvme0n1p1

root@akanas:~# sfdisk /dev/nvme0n1 Welcome to sfdisk (util-linux 2.41). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Checking that no-one is using this disk right now ... OK Disk /dev/nvme0n1: 953.87 GiB, 1024209543168 bytes, 2000409264 sectors Disk model: SP010TIMEC3M5EV0 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes sfdisk is going to create a new 'dos' disk label. Use 'label: <name>' before you define a first partition to override the default. Type 'help' to get more information. >>> 2048 Created a new DOS (MBR) disklabel with disk identifier 0x1d27bfe1. Created a new partition 1 of type 'Linux' and of size 953.9 GiB. Partition #1 contains a btrfs signature. Do you want to remove the signature? [Y]es/[N]o: I chose no for now...

Absolutely not - went to bed with nothing happening. That is the most painful part...

Nothing, no output, no error. EDIT: I had started the array in maintenance mode with the above. When started normally, I get: root@akanas:~# btrfs fi show Label: none uuid: 444cab5a-d33c-4a5c-810d-80753362c808 Total devices 1 FS bytes used 372.00KiB devid 1 size 20.00GiB used 536.00MiB path /dev/loop2 Label: none uuid: b6a29e09-4796-492e-81fd-fa22626f0335 Total devices 1 FS bytes used 416.00KiB devid 1 size 1.00GiB used 126.38MiB path /dev/loop3

Went to bed, all was fine. Woke to this. Cache is used only for main array, containers is where I have appdata and such (so filling cache wouldn't take down VMs, as happened once before. Logs attached. I tried a reboot just to see, but nothing changed. akanas-diagnostics-20250706-0945.zip Edit: Steps taken reading forums: root@akanas:~# blkid /dev/sda1: LABEL_FATBOOT="UNRAID" LABEL="UNRAID" UUID="16FC-3C51" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="95171951-01" /dev/loop1: BLOCK_SIZE="131072" TYPE="squashfs" /dev/loop0: BLOCK_SIZE="131072" TYPE="squashfs" /dev/md2p1: UUID="845e86ed-3072-42dd-911a-578457dbc4c8" BLOCK_SIZE="512" TYPE="xfs" /dev/md5p1: UUID="50699b8d-2d8f-4f93-b588-df0c3870669e" BLOCK_SIZE="512" TYPE="xfs" /dev/md1p1: UUID="e90ecf46-6718-41f5-9f09-e914d82d994c" BLOCK_SIZE="512" TYPE="xfs" /dev/md4p1: UUID="40718d4f-2b92-49cf-a2a3-ac5a965e7cbd" BLOCK_SIZE="512" TYPE="xfs" /dev/md3p1: UUID="7c513fa3-3022-4f47-88d3-4433cd2f62af" BLOCK_SIZE="512" TYPE="xfs" /dev/md6p1: UUID="3d5988f7-656e-449e-acca-d010b0bdf7fa" BLOCK_SIZE="512" TYPE="xfs" Running filesystem check on any of the disks: ERROR: not a regular file or block device: /dev/ ERROR: cannot open file system Opening filesystem to check... Worried to try anything else... EDIT 2: SMART extended self-tests on all 3 drives result in 0 errors. No commands are returning a filesystem. Am I hosed? This is the 3rd time I would have to rebuild the cache, first when an SSD suddenly died on me and corrupted everything, and second the aforementioned disk full issue...