After working fine for some time, sdl / disk10 suddenly showed up as unformatted in my array.

 

SMART report shows no evidence of any issue, and I have valid parity so I decided to try and reformat the disk.  This silently fails in the GUI, and the log shows as below. I can provide full logs if required, but the issue seems to be that unRAID thinks the filesystem type is "lu# Generated settings:"   (see line 5 of log below)

 

I've rebooted / removed plugins etc in attempt to fix this issue so unfortunately I don't have any logs from time that the disk started being misidentified in this way.

 

This shows up in the GUI as well:

https://imgur.com/a/YBYy8

 

Nov 23 17:47:50 media kernel: usb 1-1: reset high-speed USB device number 2 using ehci-pci
Nov 23 17:48:02 media emhttpd: req (18): startState=STARTED&file=&cmdFormat=Format&unmountable_mask=1024&confirmFormat=OFF&optionCorrect=correct&csrf_token=****************
Nov 23 17:48:04 media emhttpd: shcmd (551): /sbin/wipefs -a /dev/md10
Nov 23 17:48:05 media emhttpd: shcmd (552): mkdir -p /mnt/disk10
Nov 23 17:48:05 media emhttpd: shcmd (553): mount -t lu# Generated settings: -o noatime,nodiratime /dev/md10 /mnt/disk10
Nov 23 17:48:05 media root:
Nov 23 17:48:05 media root: Usage:
Nov 23 17:48:05 media root: mount [-lhV]
Nov 23 17:48:05 media root: mount -a [options]
Nov 23 17:48:05 media root: mount [options] [--source] <source> | [--target] <directory>
Nov 23 17:48:05 media root: mount [options] <source> <directory>
Nov 23 17:48:05 media root: mount <operation> <mountpoint> [<target>]
Nov 23 17:48:05 media root:
Nov 23 17:48:05 media root: Mount a filesystem.
Nov 23 17:48:05 media root:
Nov 23 17:48:05 media root: Options:
Nov 23 17:48:05 media root: -a, --all mount all filesystems mentioned in fstab
Nov 23 17:48:05 media root: -c, --no-canonicalize don't canonicalize paths
Nov 23 17:48:05 media root: -f, --fake dry run; skip the mount(2) syscall
Nov 23 17:48:05 media root: -F, --fork fork off for each device (use with -a)
Nov 23 17:48:05 media root: -T, --fstab <path> alternative file to /etc/fstab
Nov 23 17:48:05 media root: -i, --internal-only don't call the mount.<type> helpers
Nov 23 17:48:05 media root: -l, --show-labels show also filesystem labels
Nov 23 17:48:05 media root: -n, --no-mtab don't write to /etc/mtab
Nov 23 17:48:05 media root: -o, --options <list> comma-separated list of mount options
Nov 23 17:48:05 media root: -O, --test-opts <list> limit the set of filesystems (use with -a)
Nov 23 17:48:05 media root: -r, --read-only mount the filesystem read-only (same as -o ro)
Nov 23 17:48:05 media root: -t, --types <list> limit the set of filesystem types
Nov 23 17:48:05 media root: --source <src> explicitly specifies source (path, label, uuid)
Nov 23 17:48:05 media root: --target <target> explicitly specifies mountpoint
Nov 23 17:48:05 media root: -v, --verbose say what is being done
Nov 23 17:48:05 media root: -w, --rw, --read-write mount the filesystem read-write (default)
Nov 23 17:48:05 media root:
Nov 23 17:48:05 media root: -h, --help display this help and exit
Nov 23 17:48:05 media root: -V, --version output version information and exit
Nov 23 17:48:05 media root:
Nov 23 17:48:05 media root: Source:
Nov 23 17:48:05 media root: -L, --label <label> synonym for LABEL=<label>
Nov 23 17:48:05 media root: -U, --uuid <uuid> synonym for UUID=<uuid>
Nov 23 17:48:05 media root: LABEL=<label> specifies device by filesystem label
Nov 23 17:48:05 media root: UUID=<uuid> specifies device by filesystem UUID
Nov 23 17:48:05 media root: PARTLABEL=<label> specifies device by partition label
Nov 23 17:48:05 media root: PARTUUID=<uuid> specifies device by partition UUID
Nov 23 17:48:05 media root: <device> specifies device by path
Nov 23 17:48:05 media root: <directory> mountpoint for bind mounts (see --bind/rbind)
Nov 23 17:48:05 media root: <file> regular file for loopdev setup
Nov 23 17:48:05 media root:
Nov 23 17:48:05 media root: Operations:
Nov 23 17:48:05 media root: -B, --bind mount a subtree somewhere else (same as -o bind)
Nov 23 17:48:05 media root: -M, --move move a subtree to some other place
Nov 23 17:48:05 media root: -R, --rbind mount a subtree and all submounts somewhere else
Nov 23 17:48:05 media root: --make-shared mark a subtree as shared
Nov 23 17:48:05 media root: --make-slave mark a subtree as slave
Nov 23 17:48:05 media root: --make-private mark a subtree as private
Nov 23 17:48:05 media root: --make-unbindable mark a subtree as unbindable
Nov 23 17:48:05 media root: --make-rshared recursively mark a whole subtree as shared
Nov 23 17:48:05 media root: --make-rslave recursively mark a whole subtree as slave
Nov 23 17:48:05 media root: --make-rprivate recursively mark a whole subtree as private
Nov 23 17:48:05 media root: --make-runbindable recursively mark a whole subtree as unbindable
Nov 23 17:48:05 media root:
Nov 23 17:48:05 media root: For more details see mount(8).
Nov 23 17:48:05 media emhttpd: shcmd (553): exit status: 1
Nov 23 17:48:05 media emhttpd: /mnt/disk10 mount error: No file system
Nov 23 17:48:05 media emhttpd: shcmd (554): umount /mnt/disk10
Nov 23 17:48:05 media root: umount: /mnt/disk10: not mounted.
Nov 23 17:48:05 media emhttpd: shcmd (554): exit status: 32
Nov 23 17:48:05 media emhttpd: shcmd (555): rmdir /mnt/disk10
Nov 23 17:48:05 media emhttpd: Starting services...

 

Edited November 23, 20178 yr by ConnectivIT

Please post diags.

There are some plugin errors logged as I just renamed the plugins folder and rebooted to make sure the issue wasn't plugin-related.  Seems like that isn't the proper way to remove plugins any more!

 

 

Edited November 23, 20178 yr by ConnectivIT

Thanks for the diags.

 

The problem is that a setting is corrupted in a file on the flash: config/disk.cfg.

 

You can Stop array and then edit that file.  If via telnet/ssh/console, you would type:

 

nano /boot/config/disk.cfg

 

Find the line that reads:

diskFsType.10="lu# Generated settings:"

and change to:

diskFsType.10="luks:xfs"

 

Now you can Start array.

 

How did this happen?  Have no clue, but will look into it.

 

 

and I have valid parity so I decided to try and reformat the disk.

Not sure this is what you mean but note that if you format a disk parity will be updated and there's no way to recover that data using it.

 

 

Interesting... I made the change and refreshed the browser.  Still no padlock on disk10 (it was an encrypted disk)

https://imgur.com/a/4DBPr

 

Verified that my change to /boot/config/disk.cfg had saved correctly - yep.

 

Rebooted to see if this would change anything - still no padlock.  Checked disk.cfg and it has reverted to diskFsType.10="lu# Generated settings:"

 

Very odd!  This is running as VMWare guest under ESXI 6.5 incase that's relevant.  Has been running b9 for some time without issue, upgraded to b13.... about a week ago?

 

 

disk.cfg

6 minutes ago, ConnectivIT said:

Interesting... I made the change and refreshed the browser.  Still no padlock on disk10 (it was an encrypted disk)

https://imgur.com/a/4DBPr

 

Verified that my change to /boot/config/disk.cfg had saved correctly - yep.

 

Rebooted to see if this would change anything - still no padlock.  Checked disk.cfg and it has reverted to diskFsType.10="lu# Generated settings:"

 

Very odd!  This is running as VMWare guest under ESXI 6.5 incase that's relevant.  Has been running b9 for some time without issue, upgraded to b13.... about a week ago?

 

 

disk.cfg

 

Guess you're passing the usb flash through somehow?  Something could be broken with that (speculation).

 

Suggest you boot in 'safe mode' (to disable plugins) and see if issue persists.

OK so disk10 only had two of the configuration lines available,

 

I copied everything from disk9, now I get a padlock:

https://imgur.com/a/8kJ0u

 

Unfortunately it would mount:

https://imgur.com/a/zLdyj

 

This unRAID is just a backup currently, so losing a disk is not a problem.

Edited November 23, 20178 yr by ConnectivIT

>>Not sure this is what you mean but note that if you format a disk parity will be updated and there's no way to recover that data using it.

 

I guess you are right... I never issued a new disk configuration, so I assumed I had good parity, but that would only be true until the moment unRAID mounted everything without disk10....

9 minutes ago, ConnectivIT said:

OK so disk10 only had two of the configuration lines available,

 

Are you referring to corruption in disk.cfg file?  If so, getting down to the bottom of this is important.

10 minutes ago, limetech said:

 

Guess you're passing the usb flash through somehow?  Something could be broken with that (speculation).

 

Suggest you boot in 'safe mode' (to disable plugins) and see if issue persists.

 

Yep, the specific USB device is being passed through to the VM.  I use plop to boot directly from it:

https://www.pendrivelinux.com/boot-from-usb-without-bios-support-via-plop-cd/

 

Has worked well for many years!

 

I have had issues with the built-in upgrade process though - upgrading to b12 and b13 both failed and I had to reimage the USB device on Windows and copy over my config.

1 minute ago, limetech said:

 

Are you referring to corruption in disk.cfg file?  If so, getting down to the bottom of this is important.

 

Yes, the reason changing the fsType didn't stick is because it wasn't a complete disk configuration - it only had two lines compared to disk9 that had 17 lines:

diskComment.9=""
diskFsType.9="luks:xfs"
diskSpindownDelay.9="-1"
diskSpinupGroup.9=""
diskExport.9="e"
diskFruit.9="no"
diskSecurity.9="public"
diskReadList.9=""
diskWriteList.9=""
diskExportNFS.9="-"
diskSecurityNFS.9="public"
diskHostListNFS.9=""
diskExportAFP.9="-"
diskSecurityAFP.9="public"
diskReadListAFP.9=""
diskWriteListAFP.9=""
diskVolsizelimitAFP.9=""
diskVoldbpathAFP.9=""
diskComment.10=""
diskFsType.10="lu# Generated settings:
startArray="no"
spindownDelay="30"
queueDepth="auto"

I will re-enable my plugins and send you a new diagnostics file.

 

This issue has nothing to do with encryption.  Maybe your usb flash device is bad or corrupted.  Maybe plug into a windows pc and check it.  Maybe try a different flash device.

1 hour ago, limetech said:

This issue has nothing to do with encryption.  Maybe your usb flash device is bad or corrupted.  Maybe plug into a windows pc and check it.  Maybe try a different flash device.

 

I did a full scan of the drive when I had issues with built in upgrade (no issues) but I will check the flash drive again.  This is from syslog:

 

Nov 23 20:27:06 media kernel: sdl: sdl1
Nov 23 20:27:06 media kernel: sd 33:0:10:0: [sdl] Attached SCSI disk
Nov 23 20:27:06 media kernel: GPT:disk_guids don't match.
Nov 23 20:27:06 media kernel: GPT:partition_entry_array_crc32 values don't match: 0x9ad9c074 != 0xab54d286
Nov 23 20:27:06 media kernel: GPT: Use GNU Parted to correct GPT errors.

 

 

For context - this happened again:

https://imgur.com/a/Rx0hP

Edited November 23, 20178 yr by ConnectivIT

Flash drive is fine - Windows scan and full scan with dlgdiag, no issues.

 

Not sure what happened to the disk, seems unfixable, but fixing disk.cfg has allowed format to work successfully:

 

 

I'm actually not convinced I wasn't without parity as maybe something happened to the disk that didn't get written to parity?  I should have tried bringing the array up with a replacement disk.  Too late now though - sorry.

Edited November 23, 20178 yr by ConnectivIT

Problem is that we've never seen any of these kinds of errors before, and you're running unRAID OS in a VMWare guest which is not officially supported.  The problem could be anywhere.

7 hours ago, limetech said:

you're running unRAID OS in a VMWare guest which is not officially supported.  The problem could be anywhere.

 

Yes, completely understand that. 

 

7 hours ago, limetech said:

Problem is that we've never seen any of these kinds of errors before

 

Me too!  unRAID has been stable for me as a VMWare guest for about 7 years, so not sure if this was some kind of hardware issue / anomaly or some compatibility issue with the latest beta running as a VMWare guest.

 

Array is up and running again, thanks for looking in to it.  Will post if I see any more issues.