It all started when I wanted to remove my cache drive, the system had been unstable after a ram upgrade and the addition of my first cache drive. So, I shut down and removed the ram. Booted up and everything was good. Did tools>new config and cleared the config and marked the cache drive and not installed. Rebooted to an offline array that wanted me to reassign devices. This is where the problem begins... my parity drive and one of my 2TB data drives are 1 digit different in the serial number and I reversed them and rebooted. System restarted and started a parity check, which I canceled because I thought it was odd. Looked at the disk assignments are realized I had really screwed up. I reassigned the drives and rebooted with parity valid checked to try and avert more damage. Realizing what I had done, I unassigned the parity disk and moved it to the array and started the array. It now tells me that the 2 WD drives are unreadable, I somewhat expected that since it likely over wrote part of one or the other. The cache drive that started all of this was on a PCI controller, I removed the controller and moved the empty cache drive to one of my mobo sata controller ports.
So, how screwed am I? I do not have a backup since getting to an offsite backup at 30Mbps is painful. The loss of years worth of movies and all of my digitized cd collection will be painful.
I am in this state now... parity drive in array and array not starting, but it knows about my shares.
The system now thinks my array is 70% full, was 49% before this adventure.
Interesting stuff from the logs... disk 3 sees corruption, was the disk swapped in place of the parity drive. Disk 5 is showing unsupported file system (real parity drive).
Dec 18 20:02:36 Tower emhttpd: shcmd (154): mkdir -p /mnt/disk1
Dec 18 20:02:36 Tower emhttpd: shcmd (155): mount -t xfs -o noatime,nouuid /dev/md1 /mnt/disk1
Dec 18 20:02:36 Tower kernel: SGI XFS with ACLs, security attributes, no debug enabled
Dec 18 20:02:36 Tower kernel: XFS (md1): Mounting V5 Filesystem
Dec 18 20:02:36 Tower kernel: XFS (md1): Ending clean mount
Dec 18 20:02:36 Tower emhttpd: shcmd (156): xfs_growfs /mnt/disk1
Dec 18 20:02:36 Tower root: meta-data=/dev/md1 isize=512 agcount=4, agsize=91571160 blks
Dec 18 20:02:36 Tower root: = sectsz=512 attr=2, projid32bit=1
Dec 18 20:02:36 Tower root: = crc=1 finobt=1, sparse=1, rmapbt=0
Dec 18 20:02:36 Tower root: = reflink=1 bigtime=1 inobtcount=1
Dec 18 20:02:36 Tower root: data = bsize=4096 blocks=366284638, imaxpct=5
Dec 18 20:02:36 Tower root: = sunit=0 swidth=0 blks
Dec 18 20:02:36 Tower root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1
Dec 18 20:02:36 Tower root: log =internal log bsize=4096 blocks=178849, version=2
Dec 18 20:02:36 Tower root: = sectsz=512 sunit=0 blks, lazy-count=1
Dec 18 20:02:36 Tower root: realtime =none extsz=4096 blocks=0, rtextents=0
Dec 18 20:02:36 Tower emhttpd: shcmd (157): mkdir -p /mnt/disk2
Dec 18 20:02:36 Tower emhttpd: shcmd (158): mount -t xfs -o noatime,nouuid /dev/md2 /mnt/disk2
Dec 18 20:02:36 Tower kernel: XFS (md2): Mounting V5 Filesystem
Dec 18 20:02:37 Tower kernel: XFS (md2): Ending clean mount
Dec 18 20:02:37 Tower emhttpd: shcmd (159): xfs_growfs /mnt/disk2
Dec 18 20:02:37 Tower root: meta-data=/dev/md2 isize=512 agcount=4, agsize=91571160 blks
Dec 18 20:02:37 Tower root: = sectsz=512 attr=2, projid32bit=1
Dec 18 20:02:37 Tower root: = crc=1 finobt=1, sparse=1, rmapbt=0
Dec 18 20:02:37 Tower root: = reflink=1 bigtime=1 inobtcount=1
Dec 18 20:02:37 Tower root: data = bsize=4096 blocks=366284638, imaxpct=5
Dec 18 20:02:37 Tower root: = sunit=0 swidth=0 blks
Dec 18 20:02:37 Tower root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1
Dec 18 20:02:37 Tower root: log =internal log bsize=4096 blocks=178849, version=2
Dec 18 20:02:37 Tower root: = sectsz=512 sunit=0 blks, lazy-count=1
Dec 18 20:02:37 Tower root: realtime =none extsz=4096 blocks=0, rtextents=0
Dec 18 20:02:37 Tower emhttpd: shcmd (160): mkdir -p /mnt/disk3
Dec 18 20:02:37 Tower emhttpd: shcmd (161): blkid -t TYPE='xfs' /dev/md3 &> /dev/null
Dec 18 20:02:37 Tower emhttpd: shcmd (162): mount -t xfs -o noatime,nouuid /dev/md3 /mnt/disk3
Dec 18 20:02:37 Tower kernel: XFS (md3): Mounting V5 Filesystem
Dec 18 20:02:37 Tower kernel: XFS (md3): Corruption warning: Metadata has LSN (1:115531) ahead of current LSN (1:71954). Please unmount and run xfs_repair (>= v4.3) to resolve.
Dec 18 20:02:37 Tower kernel: XFS (md3): log mount/recovery failed: error -22
Dec 18 20:02:37 Tower kernel: XFS (md3): log mount failed
Dec 18 20:02:37 Tower root: mount: /mnt/disk3: wrong fs type, bad option, bad superblock on /dev/md3, missing codepage or helper program, or other error.
Dec 18 20:02:37 Tower root: dmesg(1) may have more information after failed mount system call.
Dec 18 20:02:37 Tower emhttpd: shcmd (162): exit status: 32
Dec 18 20:02:37 Tower emhttpd: /mnt/disk3 mount error: Wrong or no file system
Dec 18 20:02:37 Tower emhttpd: shcmd (163): umount /mnt/disk3
Dec 18 20:02:37 Tower root: umount: /mnt/disk3: not mounted.
Dec 18 20:02:37 Tower emhttpd: shcmd (163): exit status: 32
Dec 18 20:02:37 Tower emhttpd: shcmd (164): rmdir /mnt/disk3
Dec 18 20:02:37 Tower emhttpd: shcmd (165): mkdir -p /mnt/disk4
Dec 18 20:02:37 Tower emhttpd: shcmd (166): mount -t xfs -o noatime,nouuid /dev/md4 /mnt/disk4
Dec 18 20:02:37 Tower kernel: XFS (md4): Mounting V5 Filesystem
Dec 18 20:02:37 Tower kernel: XFS (md4): Ending clean mount
Dec 18 20:02:37 Tower emhttpd: shcmd (167): xfs_growfs /mnt/disk4
Dec 18 20:02:37 Tower root: meta-data=/dev/md4 isize=512 agcount=4, agsize=122094660 blks
Dec 18 20:02:37 Tower root: = sectsz=512 attr=2, projid32bit=1
Dec 18 20:02:37 Tower root: = crc=1 finobt=1, sparse=1, rmapbt=0
Dec 18 20:02:37 Tower root: = reflink=1 bigtime=1 inobtcount=1
Dec 18 20:02:37 Tower root: data = bsize=4096 blocks=488378638, imaxpct=5
Dec 18 20:02:37 Tower root: = sunit=0 swidth=0 blks
Dec 18 20:02:37 Tower root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1
Dec 18 20:02:37 Tower root: log =internal log bsize=4096 blocks=238466, version=2
Dec 18 20:02:37 Tower root: = sectsz=512 sunit=0 blks, lazy-count=1
Dec 18 20:02:37 Tower root: realtime =none extsz=4096 blocks=0, rtextents=0
Dec 18 20:02:37 Tower emhttpd: shcmd (168): mkdir -p /mnt/disk5
Dec 18 20:02:37 Tower emhttpd: shcmd (169): blkid -t TYPE='xfs' /dev/md5 &> /dev/null
Dec 18 20:02:37 Tower emhttpd: shcmd (169): exit status: 2
Dec 18 20:02:37 Tower emhttpd: shcmd (170): blkid -t TYPE='btrfs' /dev/md5 &> /dev/null
Dec 18 20:02:37 Tower emhttpd: shcmd (170): exit status: 2
Dec 18 20:02:37 Tower emhttpd: shcmd (171): blkid -t TYPE='reiserfs' /dev/md5 &> /dev/null
Dec 18 20:02:37 Tower emhttpd: shcmd (171): exit status: 2
Dec 18 20:02:37 Tower emhttpd: /mnt/disk5 mount error: Unsupported or no file system
Dec 18 20:02:37 Tower emhttpd: shcmd (172): umount /mnt/disk5
Dec 18 20:02:38 Tower root: umount: /mnt/disk5: not mounted.
Dec 18 20:02:38 Tower emhttpd: shcmd (172): exit status: 32
Dec 18 20:02:38 Tower emhttpd: shcmd (173): rmdir /mnt/disk5
Dec 18 20:02:38 Tower emhttpd: shcmd (174): sync
Dec 18 20:02:38 Tower emhttpd: shcmd (175): mkdir /mnt/user0
Dec 18 20:02:38 Tower emhttpd: shcmd (176): /usr/local/sbin/shfs /mnt/user0 -disks 30 -o default_permissions,allow_other,noatime |& logger
Dec 18 20:02:38 Tower emhttpd: shcmd (177): mkdir /mnt/user
Dec 18 20:02:38 Tower emhttpd: shcmd (178): /usr/local/sbin/shfs /mnt/user -disks 31 -o default_permissions,allow_other,noatime -o remember=0 |& logger
Dec 18 20:02:38 Tower emhttpd: shcmd (180): /usr/local/sbin/update_cron
Fix common problems is also reporting this for every share on the drive, somewhat expected i think.