Copy data from a failing cache drive

[propman07]

Hello

 

Running Unraid 6.9.2

 

I'm getting this error on my docker page for several of my dockers:

 

Warning: mkdir(): Input/output error in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 331

 

Tried to update plugins, but that doesn't seem to have helped the issue any.

 

Looking for some troubleshooting guidance. Thanks.

[trurl]

attach diagnostics to your NEXT post in this thread

[propman07]

diagnostic file attached.

tower-diagnostics-20220728-1447.zip

[trurl]

Your cache drive has failed

SMART overall-health self-assessment test result: FAILED!

and corrupted docker.img, which is readonly

[propman07]

well that's not good. I checked the SMART log for the drive....it has failed.

 

Any help in next steps to recover would be appreciated. Looking at replacement drives now so I can work on getting back up and running. So far, the system is still working as some of the dockers are still running....Plex is one that I use a lot....any advice on whether or not I should shut the system down to avoid further damage?

 

Thanks.

[propman07]

Hi-

 

I've been having issues with my Unraid server, and I've come to learn that my cache drive is failing. I can still read some data off of it, and I've ordered a replacement drive. Is there a way to copy / clone the failing cache drive to the new cache drive? I'm using Unraid 6.9.2 and the failing cache drive is a Samsung_SSD_970_EVO_500GB.

 

Thanks.

[ChatNoir]

@propman07 as stated in the first post of the FAQ, this is not a place where to post questions.

 

On 4/19/2016 at 4:17 AM, RobJ said:

Please do not ask for support here

 

I've splitted your question in it's own thread.

[JorgeB]

Please post the diagnostics. 

[propman07]

RobJ- Not sure where else to go for support....thought that was the intent of the forums.

ChatNoir- Thanks for splitting my question to it's own thread....not sure where to find it....

JorgeB- I'd love to post the diagnostics....but don't want to break any more rules, and can't seem to find the new thread where asking for support / posting diagnostic log would be acceptable....so now I'm kind of stuck...

 

Here is a link to another thread (*moderator edit* the threads have been merged.) that I started about an error I was getting in the unRAID GUI which I was informed that the cache drive had failed, and that the docker.img file was set to read only:

 

https://forums.unraid.net/topic/126552-warning-mkdir-inputoutput-error-in-dockerclientphp-on-line-331/#comment-1153112

[trurl]

On 7/28/2022 at 6:13 PM, propman07 said:

I checked the SMART log for the drive....it has failed

  So did I and showed that information in the post just before yours.

 

  

1 hour ago, propman07 said:

RobJ- Not sure where else to go for support

RobJ hasn't visited the forum in over 5 years.

 

1 hour ago, propman07 said:

can't seem to find the new thread

You are posting in it. You can get to your post history by clicking on your username next to any of your posts.

 

1 hour ago, propman07 said:

link to another thread that I started

Would have been better kept to that thread. I have merged it all into this thread since you know where this thread is so nobody will have to go looking for it.

 

Diagnostics already posted above, but might as well give us new diagnostics so everything is up-to-date.

 

attach diagnostics to your NEXT post in this thread.

 

 

 

 

 

 

[propman07]

Trurl-

 

Thanks for helping to get things cleaned up, from a thread perspective. I was going to try to get the individual app settings from the docker page earlier today, but I tried to get to that page, and it is telling me that no dockers are installed. I went ahead and stopped the docker process. Also noted that I can access some of the shares on the unRAID server, but they appear to be read-only.

 

Anyway....here is the attached diagnostic file. I plan on shutting down the server so that I can pull the failed drive (nvme) and install a new one which arrived earlier today.

 

Thanks.

tower-diagnostics-20220729-2037.zip

[JorgeB]

Docker image is corrupt, but since the log rotated and we can't see the initial boot please reboot and post new diags after array start.

[propman07]

11 hours ago, JorgeB said:

Docker image is corrupt, but since the log rotated and we can't see the initial boot please reboot and post new diags after array start.

JorgeB-

 

Thanks for the help. I performed a reboot and started the array. It's currently going through a parity check, but here is the log as requested.

dlvtower-diagnostics-20220730-1307.zip

[JorgeB]

The NVMe device is failing and needs to be replaced.

 

Not really surprising considering the total writes, wow!

 

 

Data Units Written:                 6,252,778,690 [3.20 PB]

 

[propman07]

JorgeB- Thanks....figured that was part of the issue. I'm working on installing a replacement. Obviously, I've got something incorrect with my config. I've noticed that some of the youtube videos on setting up the "rrs" have changed since I put the system together back in 2019.

 

Since the appdata folder was on this drive, I take it that I'll have to re-configure all of my dockers once I get a new image created? I was running the CA Backup docker, but it looks like the last time I ran it was back in 2021. Maybe that's as good of a place to start as any.

 

Thanks again for the reply.

[JorgeB]

Yes, use the last backup and then reconfigure the remaining.

[trurl]

18 hours ago, propman07 said:

re-configure all of my dockers

Disable Docker and VM Manager in Settings until you get your new device installed. Restore appdata from backup.

 

Your docker.img is probably larger than necessary if you have things setup correctly, and making it that large will not fix things, it will only make it take longer to fill.

 

After your new device is installed, create docker.img, 20G is often more than enough.

 

https://wiki.unraid.net/Manual/Docker_Management#Re-Create_the_Docker_image_file

 

Previous Apps on the Apps page will let you reinstall your docker using the same docker settings you installed them with.

 

https://wiki.unraid.net/Manual/Docker_Management#Re-Installing_Docker_Applications

 

But within each application, settings and anything else the application needs to keep track of, such as plex library, would come from your appdata backup.

[propman07]

Here is where I am at.

 

Installed new cache drive. Booted server. At login screen used username/password combination. Screen said they were invalid? Made a few more attempts only to now be locked out? I went through the My Servers page, and it shows the server is online, but I can't seem to login, even though I did not change my username/password settings prior to the cache drive failing.

[JorgeB]

Use the recover password procedure:

 

https://wiki.unraid.net/Manual/Troubleshooting#Lost_root_Password

[propman07]

JorgeB-

 

Thanks for the reply. I ran through that yesterday. I don't want to lose the other passwords, so I may try the second method shown to remove the characters in between the two colon characters.

 

Thanks.

[propman07]

Update:

 

Finally able to get the server back up and running. I installed the replacement nvme drive, and was finally able to get things back up and running. It looks like I probably lost my VMs, but I can re-create those. I did have some trouble logging into the server once I had the hardware replaced. It looks like the GUI does not like special characters in the password field. I had a "$" in my password, and I was unable to get it to save with that character in the password field. I removed it, and that seemed to be the fix. I'm running about 17 dockers for a total docker.img size of 17.6 GB. Thanks again to all that replied.

[trurl]

3 minutes ago, propman07 said:

running about 17 dockers for a total docker.img size of 17.6 GB

Keep an eye on it to make sure it doesn't grow. You can see % used on the Dashboard page in the Memory section.