First, I'll explain what exactly happened. I hot-added a drive to the system, not to the array. I don't intent to add it at all. Everything seemed fine, except in Unassigned Devices, one o the main array drives showed up. I didn't touch it, because the array seemed to be intact. The drive I added also showed up an I mounted it. Then. I added another drive, which also showed up in Unassigned Devices. Perfect.

 

I wanted to see what's up with one of my drives showing in UD, so I attempted to stop the array. Nothing happened. Huh. So I manually stopped my VMs, and released files still locked on my pc. Stop the array. Still nothing. Alrightythen, a reboot it is. After about 3 minutes, the system came back up again, and this time, that one disk that showed up in UD, doesn't anymore. Instead, it shows in the array as "Device is disabled, contents emulated."

 

I guess emulated contents means it's using parity to serve the missing data. Doesn't matter, let's first solve the actual problem.

 

In the log I was able to find:

 

Jun 28 00:35:37 unraid kernel: XFS (md1p1): Corruption warning: Metadata has LSN (1:155054) ahead of current LSN (1:149811). Please unmount and run xfs_repair (>= v4.3) to resolve.
Jun 28 00:35:37 unraid kernel: XFS (md1p1): log mount/recovery failed: error -22
Jun 28 00:35:37 unraid kernel: XFS (md1p1): log mount failed
Jun 28 00:35:37 unraid root: mount: /mnt/disk1: wrong fs type, bad option, bad superblock on /dev/md1p1, missing codepage or helper program, or other error.
Jun 28 00:35:37 unraid root:        dmesg(1) may have more information after failed mount system call.
Jun 28 00:35:37 unraid emhttpd: shcmd (38): exit status: 32
Jun 28 00:35:37 unraid emhttpd: /mnt/disk1 mount error: Unsupported or no file system

 

So I stopped the array again, so that at least things weren't going to try and use a filesystem that can't be mounted.

 

Then, I started the array in maintenance mode, with the intent to do a filesystem check. The check when down fairly quickly, and the result is as follows:

 

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
ignored because the -n option was used.  Expect spurious inconsistencies
which may be resolved by first mounting the filesystem to replay the log.
        - scan filesystem freespace and inode maps...
sb_fdblocks 2596075000, counted 2283514869
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 3
        - agno = 2
        - agno = 4
        - agno = 5
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 6
        - agno = 12
        - agno = 13
        - agno = 1
        - agno = 7
        - agno = 14
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.

 

I'm not sure how to interpret this. It feels positive, but that's only because I'm not seeing anything that is yelling at me. Only thing that concerns me is that "replay the log" sentence. I have no idea how to do that.

 

For good measure, let's also have the clarity of mind to supply the output of `blkid`:

 

root@unraid:~# blkid
/dev/sda1: LABEL_FATBOOT="UNRAID" LABEL="UNRAID" UUID="272B-4CE1" BLOCK_SIZE="512" TYPE="vfat"
/dev/loop1: TYPE="squashfs"
/dev/sdf9: PARTUUID="c1c50c9c-48d1-1044-a906-d698d6bb5fba"
/dev/sdf1: LABEL="ssd" UUID="17022178612354483592" UUID_SUB="10143665043133095078" BLOCK_SIZE="512" TYPE="zfs_member" PARTLABEL="zfs-6a86c4c98cead86a" PARTUUID="610d4be7-859e-c04c-ac9c-64c88c2b549e"
/dev/sdd9: PARTUUID="8863e506-6f9e-c548-a0d6-4dd5402ca53d"
/dev/sdd1: LABEL="ssd" UUID="17022178612354483592" UUID_SUB="16910454766111439637" BLOCK_SIZE="512" TYPE="zfs_member" PARTLABEL="zfs-59bc099f2ed51ee4" PARTUUID="bb6c7dd3-52f9-1d41-ba9b-56fc0cb6b006"
/dev/md2p1: UUID="3d4faa0b-eb20-42d8-9635-98fa3154070c" BLOCK_SIZE="512" TYPE="xfs"
/dev/sdb1: UUID="c4c67326-2450-46d6-b0bb-8f99c9cf535c" BLOCK_SIZE="512" TYPE="xfs"
/dev/sdk1: PARTLABEL="Microsoft reserved partition" PARTUUID="0eddd9a7-6a13-486b-9084-8d635e560bc1"
/dev/sdk2: LABEL="Archive" UUID="32BB-D30F" BLOCK_SIZE="512" TYPE="exfat" PARTLABEL="Basic data partition" PARTUUID="68c0d2c6-c5c8-4f3f-918a-d00c7cb1b3bf"
/dev/sdi9: PARTUUID="e95523c3-3274-9e41-beea-18bed5fe8502"
/dev/sdi1: LABEL="ssd" UUID="17022178612354483592" UUID_SUB="4254412105164419679" BLOCK_SIZE="512" TYPE="zfs_member" PARTLABEL="zfs-b465be5a5071c8f5" PARTUUID="360de15b-cd6f-1444-ba23-05d364b8f2f5"
/dev/md1p1: UUID="10f79463-d4ab-4d1b-84bd-5a04a4934de4" BLOCK_SIZE="512" TYPE="xfs"
/dev/sdg1: UUID="10f79463-d4ab-4d1b-84bd-5a04a4934de4" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="9d6ced82-21c9-43ba-9f30-1151fe599def"
/dev/loop0: TYPE="squashfs"
/dev/sde1: UUID="3d4faa0b-eb20-42d8-9635-98fa3154070c" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="7dbb0b26-8bca-4cb1-9b99-05271e644bef"
/dev/sdc9: PARTUUID="e324a5e9-e84b-7e43-9241-0e58097c03e8"
/dev/sdc1: LABEL="ssd" UUID="17022178612354483592" UUID_SUB="4258666910141786191" BLOCK_SIZE="512" TYPE="zfs_member" PARTLABEL="zfs-2ca269b663a3f652" PARTUUID="a8895a61-69b4-d948-b0b5-4f13f1389da0"
/dev/sdl1: PARTUUID="fa7a5bb3-1e71-263c-49de-d9d5243e7b88"
/dev/sdj1: UUID="3d0fba2a-df71-412b-9794-e386cef8f9b9" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="eef81965-07a6-441c-939e-04e281dc176d"
/dev/md3p1: UUID="10b78442-e0fa-4ee8-851c-21785b3fb351" BLOCK_SIZE="512" TYPE="xfs"
/dev/sdh1: UUID="10b78442-e0fa-4ee8-851c-21785b3fb351" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="fe020cb4-b7dd-46c7-8039-32cd9adca459"

 

There's that /mnt/md1p1 again: it detects an xfs filesystem, which seems positively correct. So I guess the disk is physically fine after all this.

 

I shall also attach a diagnostics zipfile. I think it might be best to pull those two drives for now, since the trouble all started when I added the first one. Maybe this will even fix the problem magically.

 

Currently I just want to get the array up and running again. That's priority number 1. After that, and *only* after that, I would like to do some evaluation to how this could've happened, what was the actual problem, and how can this be prevented in the future. And perhaps even come up with a plan to bake protection against this failure into unRAID itself, is at all possible.

 

But for now, what do I do to get the array up and running again please?

unraid-diagnostics-20230628-0109.zip

Edited June 27, 20233 yr by thany

For now, I watched this video:

 

I decided to go with his last approach, that is allegedly the safest:

 

1. Stop the array

2. Unassign the faulty device

3. Start the array in maintenance mode

4. Repair the XFS filesystem with the -L option (not sure if this is even neccesary, because of next steps)

5. Stop the array once more

6. Assign the drive back into the array where it belonged

7. Start the array

 

The array is now rebuilding. This feels like a good sign to me, but let's see where it goes.

Edited June 28, 20233 yr by thany

Thank you so much Thany. I have been getting drives dropping off for sometime. I fumbled through your instructions, I think I was meant to go command line but got lost in the video. But found I could click on the drive, around step 4. even though 'unassigned' I was able to check the disk and repair.

Now rebuilding so fingers crossed.