Description:

I have a problem where disks fail to mount and instead displays as Unmountable: wrong file system. This happens whenever I try to start the array.

I can start and stop the array several times and each time a different drive/s will fail to mount until eventually, if I am lucky, all drives will mount successfully, but it might take me 10 times. Restarting the server has no significant effect. This can happen to any of the disk; array disks no matter the file system or type of disk; pool SSDs, cache SSD...

I was advised to update to Unraid 7.1.3 because of the "Encryption passphrase sometimes incorrect"-fix. So I did update. In fact I just made a completely new flash drive with 7.1.3- This did not make a difference unfortunately.

This problem has been bugging me since I started using Unraid and I have tried to remove HBA card, remove a disk that gave me errors, switched out cables, I have made several new flash keys, switch position on the drives and so on.

Steps to Reproduce:

1. Make a new USB flash key with Unraid 7.1.3

2. Connect the flash drive and start the server.

3. Enter the passphrase for the encrypted drives.

4. Start the array.

5. Random disks will fail to mount.

6. Stop the array.

7. Start the array.

8. Some other random disks will fail to mount.

9. Continue this process until eventually all drives will mount.

Environment:

OS: Unraid 7.1.3

Unraid USB Creator: 1.0.1

Hardware:

CPU: Intel Core i9-14900K

RAM: 128 GB (Kingston 4x32GB ECC Registered DDR5 4800 PC5 38400)

Motherboard: ASUS PRO WS W680-ACE IPMI

GPU: Intel Arc B580

Unraid USB flash disk: Samsung FIT Plus 3.1

PSU: Corsair RM850i

Additional Context:

• All my drives are encrypted.

• I do have some SMART errors (UDMA CRC errors on some drives) but I am fairly certain this problem occurred before I had any such errors.

Edited July 2, 20251 yr by Bob-omb

So the issue is this:

Jun 14 08:44:33 Tower emhttpd: shcmd (379): /usr/sbin/cryptsetup luksOpen /dev/md1p1 md1p1

Jun 14 08:44:35 Tower emhttpd: disk1: Encrypted volume present

Jun 14 08:44:35 Tower emhttpd: shcmd (382): /usr/sbin/cryptsetup luksOpen /dev/md2p1 md2p1

Jun 14 08:44:38 Tower emhttpd: shcmd (385): /usr/sbin/cryptsetup luksOpen /dev/md3p1 md3p1

Jun 14 08:44:40 Tower emhttpd: disk3: Encrypted volume present

Jun 14 08:44:40 Tower emhttpd: shcmd (388): /usr/sbin/cryptsetup luksOpen /dev/md4p1 md4p1

Jun 14 08:44:43 Tower emhttpd: disk4: Encrypted volume present

Jun 14 08:44:43 Tower emhttpd: shcmd (391): /usr/sbin/cryptsetup luksOpen /dev/md5p1 md5p1

Jun 14 08:44:45 Tower emhttpd: disk5: Encrypted volume present

Jun 14 08:44:45 Tower emhttpd: shcmd (394): /usr/sbin/cryptsetup luksOpen /dev/nvme0n1p1 nvme0n1p1 --allow-discards

Jun 14 08:44:47 Tower emhttpd: cache: Encrypted volume present

Jun 14 08:44:47 Tower emhttpd: shcmd (397): /usr/sbin/cryptsetup luksOpen /dev/nvme1n1p1 nvme1n1p1 --allow-discards

Jun 14 08:44:49 Tower emhttpd: shcmd (400): /usr/sbin/cryptsetup luksOpen /dev/nvme3n1p1 nvme3n1p1 --allow-discards

Jun 14 08:44:51 Tower emhttpd: downloads2: Encrypted volume present

Jun 14 08:44:51 Tower emhttpd: shcmd (403): /usr/sbin/cryptsetup luksOpen /dev/nvme2n1p1 nvme2n1p1 --allow-discards

Jun 14 08:44:54 Tower emhttpd: epsilon: Encrypted volume present

Jun 14 08:44:54 Tower emhttpd: shcmd (406): /usr/sbin/cryptsetup luksOpen /dev/sdh1 sdh1 --allow-discards

Jun 14 08:44:56 Tower emhttpd: ssd: Encrypted volume present

After an encrypted device is open, Unraid should detect a new encrypted volume, e.g., emhttpd: disk4: Encrypted volume present,if you look at the output above, this time, it worked correctly for all devices expect disk2, hence why it was then unmountable at array start, at the next start they were all detected currently, including disk2:

Jun 14 08:45:54 Tower emhttpd: shcmd (552): /usr/sbin/cryptsetup luksOpen /dev/md1p1 md1p1

Jun 14 08:45:56 Tower emhttpd: disk1: Encrypted volume present

Jun 14 08:45:56 Tower emhttpd: shcmd (555): /usr/sbin/cryptsetup luksOpen /dev/md2p1 md2p1

Jun 14 08:45:59 Tower emhttpd: disk2: Encrypted volume present

Jun 14 08:45:59 Tower emhttpd: shcmd (558): /usr/sbin/cryptsetup luksOpen /dev/md3p1 md3p1

Jun 14 08:46:01 Tower emhttpd: disk3: Encrypted volume present

Jun 14 08:46:01 Tower emhttpd: shcmd (561): /usr/sbin/cryptsetup luksOpen /dev/md4p1 md4p1

Jun 14 08:46:04 Tower emhttpd: disk4: Encrypted volume present

Jun 14 08:46:04 Tower emhttpd: shcmd (564): /usr/sbin/cryptsetup luksOpen /dev/md5p1 md5p1

Jun 14 08:46:06 Tower emhttpd: disk5: Encrypted volume present

Jun 14 08:46:06 Tower emhttpd: shcmd (567): /usr/sbin/cryptsetup luksOpen /dev/nvme0n1p1 nvme0n1p1 --allow-discards

Jun 14 08:46:08 Tower emhttpd: cache: Encrypted volume present

Jun 14 08:46:08 Tower emhttpd: shcmd (570): /usr/sbin/cryptsetup luksOpen /dev/nvme1n1p1 nvme1n1p1 --allow-discards

Jun 14 08:46:10 Tower emhttpd: downloads: Encrypted volume present

Jun 14 08:46:10 Tower emhttpd: shcmd (573): /usr/sbin/cryptsetup luksOpen /dev/nvme3n1p1 nvme3n1p1 --allow-discards

Jun 14 08:46:12 Tower emhttpd: downloads2: Encrypted volume present

Jun 14 08:46:12 Tower emhttpd: shcmd (576): /usr/sbin/cryptsetup luksOpen /dev/nvme2n1p1 nvme2n1p1 --allow-discards

Jun 14 08:46:15 Tower emhttpd: epsilon: Encrypted volume present

Jun 14 08:46:15 Tower emhttpd: shcmd (579): /usr/sbin/cryptsetup luksOpen /dev/sdh1 sdh1 --allow-discards

Jun 14 08:46:17 Tower emhttpd: ssd: Encrypted volume present

The fact that this happens to you with multiple Unraid versions, and that I've never seen any other similar cases reported, with any release, suggests to me, it's something specific with your hardware/config, other than this issue, do you encounter any other issues, like the server crashing/hanging, etc.?

In any case, I will report the issue to LT to see if they think something can be done on their side.

Thank you so much for taking the time to help out @JorgeB I know you don't have to spend your time helping with this, so I really appreciate it! 🙏

This has been driving me nuts.

After an encrypted device is open, Unraid should detect a new encrypted volume

Well it seems like the drive is detected, it just won't mount:

Jun 14 08:44:57 Tower emhttpd: mounting /mnt/disk2
Jun 14 08:44:57 Tower emhttpd: shcmd (417): mkdir -m 0666 -p /mnt/disk2
Jun 14 08:44:57 Tower emhttpd: /sbin/blkid /dev/md2p1 2>&1
Jun 14 08:44:57 Tower emhttpd: /dev/md2p1: UUID="bfbc647c-28b8-4ea0-bf42-2da9a2153d85" TYPE="crypto_LUKS"
Jun 14 08:44:57 Tower emhttpd: shcmd (418): rmdir /mnt/disk2
Jun 14 08:44:57 Tower emhttpd: disk2: mount error: wrong file system
Jun 14 08:44:57 Tower emhttpd: mounting /mnt/disk3

It is tempting to assume this is hardware related. If so it can be a bad motherboard, CPU or RAM. Not sure where to begin. I might try a process of elimination by taking out RAM sticks.

I did a memtest which came out good. I used the Intel Processor Diagnostic Tool to check the CPU for errors. No issues were found. I have tried a brand new flash drive with Unraid 7.1.3 with no configs or any changes.

The fact that this happens to you with multiple Unraid versions, and that I've never seen any other similar cases reported, with any release, suggests to me, it's something specific with your hardware/config, other than this issue, do you encounter any other issues, like the server crashing/hanging, etc.?

I don't want to sound counterproductive but, to be honest I have had nothing but problems since I started using Unraid a year ago. Since I am new to Linux and Unraid I don't know what to expect.

Some hours ago I had some problems where a docker app (plex) would not shutdown and the server would not shut down. Feel free to have a look at the logs:

Edited July 2, 20251 yr by Bob-omb

I missed that there are some errors logged that typically mean a bad flash drive, but can also be RAM/CPU related:

Jun 14 08:35:30 Tower kernel: SQUASHFS error: xz decompression failed, data probably corrupt

Jun 14 08:35:30 Tower kernel: SQUASHFS error: Failed to read block 0xa4a2524: -5

Jun 14 08:35:31 Tower kernel: SQUASHFS error: xz decompression failed, data probably corrupt

Jun 14 08:35:31 Tower kernel: SQUASHFS error: Failed to read block 0xebc7b20: -5

So replace the flash drive with a different one, and if it still happens post new diags.

I ran the server on one RAM module at a time to see if the problem was related to bad RAM sticks. I switched out one after the other, trying to startup the array. All four RAM modules produced the same problem.

This leads me to believe the problem is either the motherboard or the CPU, but I don't have any direct evidence. I am fairly new to Unraid and Linux so I will need some help describing the underlying issue if I am going to do an RMA on the CPU or motherboard. Basically I have no clue what is going on under the hood.

Regarding bad flash drive I have tried three of them now and I always get the SQUASHFS error: xz decompression failed error. Although I should say all of the flash drives are the same type (Samsung FIT Plus 3.1).

Edited July 2, 20251 yr by Bob-omb

26 minutes ago, Bob-omb said:

I always get the SQUASHFS error: xz decompression failed error.

Try using a different brand/model flash drive first, just to make sure it's not that, if the same, it could also be the CPU, since the 14900K is one of the most affect models by the Intel 13/14 Gen issue, BIOS update can sometimes help, if the CPU is not too far gone.

22 hours ago, JorgeB said:

Try using a different brand/model flash drive

Here is what I did:

• Made a new Unraid flash disk on a Kingston DataTraveler 50 USB 3.1 Gen 1.

• Connected the flash disk to a USB 2.0 port on the MB.

• Took out all NVMe disks and GPU.

  • Connected devices are now: IPMI card, flash disk, 6 x 3,5" drives and one 2,5", LAN, power.

• Loaded Optimized Defaults in the BIOS (although I had to change SlimSAS to SATA mode).

• Started Unraid and signed up for a 30 day trial.

• Setup an array with the exact same drive assignment, checked "Parity is already valid" and entered the passphrase. Started array.

• Array started fine around Jun 16 00:56 (clock not adjusted).

• I get these errors again:

Jun 16 00:58:20 Tower kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Jun 16 00:58:20 Tower kernel: SQUASHFS error: Failed to read block 0x1c2a290: -5
Jun 16 00:58:20 Tower kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Jun 16 00:58:20 Tower kernel: SQUASHFS error: Failed to read block 0x177274c: -5

• At Jun 16 01:00:07 I start the array again. Loaded OK. No issues.

• At Jun 16 01:01:47 I start the array again. Loaded OK. No issues.

• At Jun 16 01:03:47 I start the array again. Loaded OK. No issues.

• At Jun 16 01:04:59 I start the array again. This time I get the Unmountable: wrong file system on Disk 1.

• At Jun 16 01:07:57 I start the array again. This time I get the Unmountable: wrong file system on Disk 3.

Summary:

Even after making a fourth flash drive (this time a different brand) I still get the xz decompression failed, data probably corrupt error.

Disks in the array continues to fail mounting when starting the array although it works sometimes.

Hypothesis:

1. Bad CPU.

2. Bad motherboard.

3. Bad Unraid.

It would be very helpful to have someone from Lime take a look at the logs before I do an RMA on the motherboard and CPU.

Note:

The Intel 13th and 14th Gen CPU Stability "Vmin Bug/voltage-shift bug"is mostly visible during gaming and heavy load, but this guy shows evidence that decompression can trigger the instability issue. Perhaps this can explain why we get a SQUASHFS error: xz decompression failed.

Edited July 2, 20251 yr by Bob-omb

3 hours ago, Bob-omb said:

Even after making a fourth flash drive (this time a different brand) I still get the xz decompression failed, data probably corrupt error.

CPU would be my main suspect.

Yes, that or the motherboard. I have just sent the CPU as an RMA. Let's see how it goes.

Got a replacement CPU and everything works like a dream again. I am so happy! 🥳

Edited July 3, 20251 yr by Bob-omb

Update:

I am having mounting issues again. Not only that but for the first time ever the entire server completely freezed with the CPU stuck at 99 degrees C. Nothing worked. Docker containers, SSH, GUI everything was unresponsive. I shut the server down through KVM and restarted. The first time I restarted the GUI was unresponsive and would not let me see the drives. It was basically just loading forever. The second time I restarted I got the good old `Unmountable: wrong file system` error again.

Everything ran smooth for a two weeks. I had an uptime of 13 days, but now I have the same issue again. I kind of expected that this could happen since the Intel stability issues usually increase over time, but I really hoped for the best. At this point I am fairly certain that the CPU is the culprit since everything was working super smooth for the first days after I switched out the CPU. Although, in theory it might be caused by bad MB or PSU.

So now what?

Should I RMA again? I mean if this continues I am tempted to just ask for a different CPU or get my money back.

nova-diagnostics-20250722-1256.zip

Jul 22 12:50:10 nova emhttpd: disk2: Wrong encryption key

Jul 22 12:50:13 nova emhttpd: disk3: Wrong encryption key

Jul 22 12:50:15 nova emhttpd: disk4: Wrong encryption key

Jul 22 12:50:17 nova emhttpd: disk5: Wrong encryption key

This is obviously wrong since the key is correct for disk1, and it's typically a sign of bad RAM, but it could also be a bad CPU once more. If you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.