tl;dr

• Disk 1 showed up as unmountable – ✅ fixed

• Replacement Disk 1 shows up as unmountable as well – ❓reboot solves the problem

• Cache SSD 1 disappeared – ✅ replaced

• Cache SSD 2 reports SMART 187 ❗

• Unraid refused to work ❗

• Unraid refuses to replace SSD 2 ❗

• How should I proceed to replace the Cache SSD 2?

• Do these various problems have a common root cause?

  • extended SMART self-test passed ✅

  • memtes86 passed ✅

  • what else could I check ❓

Help!

I am experiencing various problems with different storage devices which really frustrate me, and by now I'm scared to even touch the system.

Disk 1 showed up as unmountable

It started with an unmountable hard drive which, as it turned out, did not actually have a problem.

Replacement Disk 1 showes up as unmountable as well

Since then, the problem reoccurred. The replacement disk was also unmountable twice, but another reboot fixed the problem.

Cache SSD 1 disappeared

Then there are two SSDs in the cache pool.

SSD 1 (Intenso SATA III High, 240 GB) simply disappeared from the array from time to time. As it was still under warranty, Intenso replaced the device without further ado.

 

As a precaution, I also replaced the SATA cables and the controller to which SSD 1 was connected.

Cache SSD 2 reports SMART error 187

Soon after I inserted the new drive, SSD 2 (SanDisk SSD Plus 240 GB) started reporting SMART error 187:

---

Event: Unraid Cache 2 SMART health [187]

Subject: Warning - reported uncorrect is 145

Description: SanDisk_SSD_PLUS_240GB (sde)

Importance: warning

---

The error count increased quite fast in the first few days and then slowed down to round 4 per day – I did not realize immediately, had accidentally disabled the email notifications :(

Unraid refused to work

I was about to submit an RMA, but then unraid reported that it could not write to the cache:

---

Event: Fix Common Problems - Rumo

Subject: Errors have been found with your server (Rumo).

Description: Investigate at Settings / User Utilities / Fix Common Problems

Importance: alert

 

* **Unable to write to cache\*\*

---

In the disk view, the SSD 2 was shown as “mounted read-only”. Unfortunately, the system failed to work and stopped responding before I could gather diagnostics :(

Unraid refuses to replace SSD 2

I have tried to replace SSD 2 with a 500 GB spare hard drive but unraid refused to start the array with the reason “Wrong Pool State / cache - too many missing/wrong devices”, so I re-added the SSD 2 and got the message “pool BTRFS too many profiles (You can ignore this warning when a pool balance operation is in progress)” – I think this was the case, right?

How should I proceed with cache pool?

I no longer really believe that the SSD 2 itself has a problem, but I would still like to replace it. How should I proceed?
Shall I move all data from the cache to another disk and rebuild the cache pool or is there a trick how I can convince unraid that it should use the new hard drive and mirror SSD 1 to that?

Do these various problems have a common root cause?

Even though this may be a coincidence of different errors, I would still like to narrow down a possible root cause.

✅ I have run memtes86 twice on the system and both runs ended without errors

✅ Also I performed an extended SMART self-test on SSD 2 and it completed without errors

✅ The output of the power supply should be more than sufficient, although I cannot measure the real power output.

Do you see any information in the diagnostics that could point to a cause that led to all these storage problems?

What else could I test?

 

The one thing all failing disks have in common is BTRFS.

Should I reformat the disks to get rid of it like stated here?

https://www.reddit.com/r/unRAID/comments/dnksmn/comment/f5e08gp/

Also I wonder if upgrading to unraid 7 would make it better or worse, but the idea of upgrading an unstable system does not make me feel very confident.

Thanks for your time reading this longish post!

I appreciate any ideas :)

diagnostics-after-reboot-20250520-1723.zip diagnostics-first-smart-error-187-20250508-0912.zip

Run an extended SMART test on cache2 and post new diags.

Done :)

rumo-diagnostics-20250520-2053.zip

Test passed, now scrub the pool and post the results.

# btrfs scrub start /mnt/cache

scrub started on /mnt/cache, fsid 5bc1ef98-0e8a-4204-95cf-5489396b3058 (pid=30559)

# ERROR: there are uncorrectable errors

# btrfs scrub status /mnt/cache/

UUID: 5bc1ef98-0e8a-4204-95cf-5489396b3058

Scrub started: Wed May 21 11:37:16 2025

Status: finished

Duration: 0:06:32

Total to scrub: 115.46GiB

Rate: 301.59MiB/s

Error summary: read=72

Corrected: 0

Uncorrectable: 72

Unverified: 0

During the operation, unraid reported a rapid increase in error 187 from 154 that night to 347.

Also I rsynced the cache to a backup disk and got one I/O error:
rsync: [sender] read errors mapping "/mnt/cache/.../huge.tif": Input/output error (5)

Shall I discard this file and run the scrub command again?

To make matters worse, i had an out of memory error this morning. New Diagnostics attached.

rumo-diagnostics-20250521-1043.zip

Edited May 21, 20251 yr by cheatinger
Formatting

The device appears to be failing despite just passing the SMART test, since the pool has dual data profiles and the other device is no good, I would recommend copying what you can to another device/pool or the array, and then creating a new pool with a new device.

I have backed up the cache partition and removed the defective tiff.

Now scrub completed without any errors.

/mnt# btrfs scrub start -B /mnt/cache/

scrub done for 5bc1ef98-0e8a-4204-95cf-5489396b3058

Scrub started: Wed May 21 14:28:27 2025

Status: finished

Duration: 0:06:54

Total to scrub: 112.74GiB

Rate: 278.85MiB/s

Error summary: no errors found

Shall I still replace the drive?

Any Ideas what causes the mount problem of Disk 1 or the out of memory error?

55 minutes ago, cheatinger said:

Shall I still replace the drive?

I would recommend it.

56 minutes ago, cheatinger said:

Any Ideas what causes the mount problem of Disk 1

It's mounted on the diags posted, is this an intermittent issue?

57 minutes ago, cheatinger said:

or the out of memory error?

It appears to be caused by Immich, check its config or limit its RAM usage

On 5/21/2025 at 3:52 PM, JorgeB said:

It's mounted on the diags posted, is this an intermittent issue?

Yes, like described in the opening post, I had the problem before and replaced the disk.

Since then, the problem has reappeared a few times.

Now it happened again after a reboot. Diagnostics attached.

You see - various drive issues:

Cache drives fail, Disk 1 repeatedly un-mountable, The Disk are connected to different controllers.

rumo-diagnostics-20250522-1713.zip

If the issue is intermittent, no much point in posting diags when the issue is not occurring, but I see the error now:

May 22 17:11:58 Rumo kernel: BTRFS error (device md1p1): superblock metadata_uuid doesn't match metadata uuid of fs_devices: 91bbebad-2d42-4c7b-8289-b13bb979f588 != 00000000-0000-0000-0000-000000000000

May 22 17:11:58 Rumo kernel: BTRFS error (device md1p1): dev_item UUID does not match metadata fsid: 00000000-0000-0000-0000-000000000000 != 91bbebad-2d42-4c7b-8289-b13bb979f588

May 22 17:11:58 Rumo kernel: BTRFS error (device md1p1): superblock contains fatal errors

This suggests a filesystem issue, replacing the disk will not help, I would recommend backing up and reformatting that disk.

6 minutes ago, JorgeB said:

This suggests a filesystem issue, replacing the disk will not help, I would recommend backing up and reformatting that disk.

So the fs issue has been replicated to the replacement disk?

If parity is 100% in sync, any fs issue it replicated to the rebuilt disk, parity is fs agnostic, it's just bits.

So, I have backed up the data.
Next steps will now be

1. Stop the array

2. Set disk 1 to unassigned

3. Tools -> New Config -> Preserve current assignments: all

4. Start the array - ensure 'parity is already valid' is uncheckt

5. Preclear the disk

6. Assign the disk

7. Restore the data from backup

8. Rebuild Parity

Right?
Alternatively I could move the data on disk 2 with unbalanced, right?

If you have a backup of disk1, you could just reformat it and restore the data from the backup.

I stopped the array, changed the fs-type to xfs to make sure that the disk is really formatted anew and started the array.

Disk 1 is now labeled as "Unmountable: Unsupported or no file system" and data rebuild is running.

I think this is not what you intended?

Update - This is how it looks:

After formatting the disk:

I see no option to prevent the system from rebuilding the disk from parity.

Started the array and got this:

Went through the whole process again, now the data-rebuild is running.
Will this result in btrfs formatted disk with corrupted super blocks like before, while unraid expects an xfs-formatted disk?

Edited May 26, 20251 yr by cheatinger
details added

Post new diags.

Attached.

Data-rebuild has finished and there are a few directories on the disk.

rumo-diagnostics-20250527-0918.zip

I only see disk1 being formatted xfs, and it was successful, don't see any attempt to then format back to btrfs.

After another day the backup is restored and the server is back in service.

Next step will be rebuilding the pool with another disk.

Thank you very much for your support so far!

Also I still wonder if all the problems have a common cause.
The defective super block, two different defects on two cache ssds, the unability to replace the cache disk and - what worries me the most - the newly formatted replacement drive was also shown as unmountable on the first attempt.

Is this all coincidence?

Is there something wrong in my configuration?

What other hardware tests could I do?

Edited May 28, 20251 yr by cheatinger

If you haven't yet, it's always good to run memtest.

I have run memtes86 twice on the system and both runs ended without errors.

Just keep monitoring then, hopefully the issues are resolved.

I will, thanks again for your support!