After years of unraid running with only minor hiccups, my server is suddenly having a bunch of issues.

 

The server was unresponsive, so I did a sudo reboot from the CLI.

 

One drive is showing as missing, the docker service has failed to start, and it now shows no VMs installed.

 

Diagnostics attached.

 

Thanks in advance.

 

tower-diagnostics-20230102-2311.zip

Edited January 3, 20233 yr by yyc321

You cache pool is completely full, free up some space and reboot.

None of your pools have a Minimum Free Space value set.    You should set that appropriately to stop pools getting completely full.

Disable Docker and VM Manager in Settings until you get this cleaned up. Not only cache, but several of your disks are too full. Full disks don't perform as well. More importantly, filesystem repair needs some free space to work in if that ever becomes necessary on a disk. And, in the case of btrfs (cache), overfilling has been known to cause corruption.

 

In addition to setting Minimum Free for each of your pools such as cache, you should also set Minimum Free for each of your user shares. You need to set pool Minimum Free to larger than the largest file you expect to write to the pool, and set each user share Minimum Free to larger than the largest file you expect to write to the share.

 

If I were you, I would try to keep at least 50G free on each array disk.

 

What is the purpose of this cache:prefer share?

s---m                             shareUseCache="prefer"  # Share exists on cache

 

You need to add capacity, another array disk, or upsizing smaller array disks.

7 hours ago, yyc321 said:

One drive is showing as missing

vars shows missing disk12 disabled. df shows emulated disk12 mounted, so that's good.

 

No disabled disk in SMART. I see an unassigned disk in SMART, but it isn't large enough to have been disk12.

 

Since you rebooted before getting diagnostics not clear why disk12 went missing. Looks like you started a correcting parity check with a missing disk. That's not going to help anything.

 

First, disable Docker and VM Manager, then

 

Shutdown, check connections to see if you can get disk12 to show up again. Can you see disk12 in BIOS?

 

After you get your array stable again you can work on capacity. If disk12 has failed, you could get some more capacity by using a larger replacement for it. Your parity disks will allow 16TB replacement.

 

You guys are great!  Thank you so much.

 

Disabling docker and VMs and cleaning up the array did the trick.

I actually had a sub-folder in one of the shares that was accumulating all sorts of junk data, so it was really easy to free up about 8TB of data with a couple of clicks.

 

Not sure why the shares and cache didn't have a minimum free space.  I must of changed them to 0 at some point, but I have no idea why I would have done that.

 

Still trying to track down the source of the missing drive.  

I've been wondering for a while if my HBA (9305-24i) is overheating.  I had that happen in a different chassis, and it started dropping drives.  I'll check the cooling, and the cables.

Thanks again!

Edited January 3, 20233 yr by yyc321

3 hours ago, yyc321 said:

Not sure why the shares and cache didn't have a minimum free space

zero is the default setting. Some debate about that, but whatever default might be chosen would be a compromise and would still require explanation to the user.

 

3 hours ago, yyc321 said:

Still trying to track down the source of the missing drive.

Did you get it to show up again?

10 hours ago, trurl said:

Can you see disk12 in BIOS?

 

Post new diagnostics

After several reboots to deal with the VM and Docker issues, the drive has reappeared in the list. 

 

Not sure if this was a drive issue, something with the HBA, or an OS issue.

 

I've re-added the drive to the array, and am completing a rebuild now.  If it drops offline again, I'll post an update.

So, the drive dropped offline again.  The UI was unresponsive, so I rebooted from the CLI.

 

I've attached the diagnostics, however they were taken after the reboot - not sure if this clears the info that you would need to diagnose the problem?

 

Is there a way to create anonymized diagnostics from the terminal?

tower-diagnostics-20230106-1924.zip

Disk12 missing. How is this disk powered?

 

13 minutes ago, yyc321 said:

taken after the reboot

Diagnostics includes the current syslog, which is only since boot. So can't tell us anything about what happened before.

 

Setup syslog server.

It's a supermicro chassis, with a supermicro backplane powering all of the drives.

 

I'll get the syslog server setup.