I had an I/O error on transmission.  I tried to restart that docker and then the whole Docker now fails.  I'm guessing I probably need to replace my SSDs?  I just thought even thought they were showing bad SMART stats, I didn't think I could wear the SSD out before filling up 100TB.  Usually they can do like 200TB right?

 

Just want to confirm before I place the order, I had already been looking to replace them just in case.

 

tower-diagnostics-20210510-2057.zip

Syslog is showing a problem with cache2, looks more like a connection problem, replace cables and post new diags after array start.

On 5/11/2021 at 4:16 AM, JorgeB said:

Syslog is showing a problem with cache2, looks more like a connection problem, replace cables and post new diags after array start.

I restarted, it's asking me to wipe the drive?  That doesn't sound good.

 

I'm almost positive I haven't written more than 200TB across both drives.  They've been definitely used hard, so if it really is broken, no problem.  I've probably written 1.5-1.8x my current size since I replace data and it has to go through the SSD. So maximum, I think I've written 130-150TB since I only have 80TB of data.

tower-diagnostics-20210512-1731.zip

Edited May 13, 20215 yr by tential

Try this, if Docker/VM services are using the cache pool disable them, unassign all cache devices, start array to make Unraid "forget" current cache config, stop array, reassign all cache devices (there can't be an "All existing data on this device will be OVERWRITTEN when array is Started" warning for any cache device), re-enable Docker/VMs if needed, start array, post new diags after array start.

I tried as suggested, it's showing it's still unmountable without a format?  It says no file system on first drive too?  

tower-diagnostics-20210513-0732.zip

Pool is in a very bad state, there are some recovery options here you can try, but also make sure to see this for better pool monitoring for the future.

The more likely reason for your filesystem corruption isn't bad disks. Have you ever filled cache?

You should disable docker until you get cache fixed. Your System and Appdata shares have been recreated on disk16 since cache wasn't available.

 

You will want to delete those or move them back to cache before enabling docker again. Probably deleting would be more appropriate since you will want to get your appdata from backup when you start over with docker.

7 hours ago, trurl said:

The more likely reason for your filesystem corruption isn't bad disks. Have you ever filled cache?

Yes, I've filled it previously, multiple times.  I've run out of space a bunch previously.  I haven't run out of space recently though.  I probably dipped below 20 free GB when this happened, but I try not to go below 50-20GB remaining now.

 

I disabled docker, but it wasn't running/saying no dockers were installed.

 

What step should I take next to reenable cache?

I misunderstood your post and thought the data was safe on disk 16.  I'm guessing filing the drive wasn't too good for it?

Edited May 13, 20215 yr by tential

1 minute ago, tential said:

What step should I take next to reenable cache?

If you already recovered any important data, or there isn't important data there, just re-format the pool.

trurl says the data is on disk 16 already? (Did not know this was something that even happened.  Would love an explanation of how/why it backed it up like this?  Not complaining, cool! Just want to learn lol).

 

So it's ok for me to reformat, and just move the data from disk16, back to my cache, then restart my docker?

trurl mentioned that since cache wasn't available new docker/VM images were created on the array, but those are new, they won't have any of your data.

3 hours ago, JorgeB said:

trurl mentioned that since cache wasn't available new docker/VM images were created on the array, but those are new, they won't have any of your data.

If there is no easy way to reenable the pool so that it works again, or easy way to recover the data, and I have to delete the data, so be it.  Just didn't want to jump to that conclusion.  And didn't want to keep using the drives if they had to be retired, but doesn't sound like that yet from your responses.

13 hours ago, tential said:

If there is no easy way to reenable the pool so that it works again, or easy way to recover the data

 

18 hours ago, JorgeB said:

Pool is in a very bad state, there are some recovery options here you can try, but also make sure to see this for better pool monitoring for the future.

 

On 5/14/2021 at 2:38 AM, JorgeB said:

 

 

I see, so I gave it a shot and I think I got it? I'm just not sure exactly how to start the copy and what settings to use.

I got the /x directory mounted and I want to move it to my study materials folder on Disk 17, the only disk with enough space + extra.

 

I selected the Alt+5 to copy and hit the Dive into sub dirs check list.

 

Now do I just hit OK?  Or did I choose a bad copy place (copy to a mount vs array).  I wasn't able to find array places, only these mounts?

 

Also, I have no shares anymore?  Should I be worried?

Edited May 15, 20215 yr by tential

8 hours ago, tential said:

Now do I just hit OK? 

Yes, but you can also browse /x and only copy just what you need.

On 5/16/2021 at 12:04 AM, JorgeB said:

Yes, but you can also browse /x and only copy just what you need.

I got a lot of errors doing that, so guessing that's not valid? 

 

I tried BTRFS repair as a last resort, but it's telling me the cache pool is still mounted, I haven't even started the array yet.

 

You need to unmount the filesystem first:

umount /x

 

I don't think I actually did this step correctly, how do I do this step where I use the mdX but it's for a cache pool instead of a disk?  I don't want to use md1 to restore from disk 1 accidently instead of cache pool?

 

mdX would be to restore from an array disk, to restore from a cache disk you always use sdX.

12 hours ago, JorgeB said:

mdX would be to restore from an array disk, to restore from a cache disk you always use sdX.

 

Thanks for the clarification.  I tried, still can't get it to recognize the second disk is part of the pool, oh well.  Guess I need to start from scratch.

 

What steps do I take to reset it now that it for some reason sees both ssds, and wants me to load them, but it won't work?  So I need to wipe/rebuild the pool.  How should I do that?

Edited May 25, 20215 yr by tential

Wipe the SSDs with

blkdiscard -f /dev/sdX

Then add to pool and re-format.

8 hours ago, JorgeB said:

Wipe the SSDs with

blkdiscard -f /dev/sdX

Then add to pool and re-format.

 

It's telling me -f is an invalid option?

 

Depends on the Unraid version, do it without -f, v6.9.x requires -f

Ok, I was  able to reset, and then follow some basic instructions to get everything back to what it used to be.  Setting up my dockers again is going to be a task.  I had my cache drive setup incorrectly I guess since I never ever used Mover, hence why my SSDs got full in the first place. 

 

I always read you can't move data between shares on unraid.  However, I always did that?  Am I misunderstanding?  I used to have a downloads share, that was on my cache only.  When a DL completed, I manually moved it over to the correct share it should be on.  I was thinking of creating a Completed Downloads share instead, that would use the cache drive to store complete downloads that then would use mover to go to the array.  Then copy from the Completed Downloads share to the correct share.  This is basically how I've always done it, just I'd be using the mover to free up space.  Is this going to cause me issues?  It's always worked in the past with no issues so not sure I'm understanding this correctly.