Hi, 

 

I woke up this morning this this error in the Docker Tab, wanted to let you know providing diagnostics before restarting it all.

 

Warning: stream_socket_client(): unable to connect to unix:///var/run/docker.sock (Connection refused) in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 682
Couldn't create socket: [111] Connection refused
Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 866

Warning: stream_socket_client(): unable to connect to unix:///var/run/docker.sock (Connection refused) in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 682
Couldn't create socket: [111] Connection refused
Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 932

untheran-diagnostics-20210201-0813.zip

I just saw that the docker file was not  found. Then my appdata share was not there anymore ...  But My cache drive is still in my main tab, no erreors ... Strying to stop the array to start it back and it takes forever.

Cache device dropped offline:

 

Feb  1 02:19:26 untheran kernel: nvme nvme0: I/O 98 QID 4 timeout, aborting
Feb  1 02:19:26 untheran kernel: nvme nvme0: I/O 99 QID 4 timeout, aborting
Feb  1 02:19:30 untheran kernel: nvme nvme0: I/O 192 QID 5 timeout, aborting
Feb  1 02:19:30 untheran kernel: nvme nvme0: I/O 193 QID 5 timeout, aborting
Feb  1 02:19:30 untheran kernel: nvme nvme0: I/O 194 QID 5 timeout, aborting
Feb  1 02:19:56 untheran kernel: nvme nvme0: I/O 14 QID 0 timeout, reset controller
Feb  1 02:19:56 untheran kernel: nvme nvme0: I/O 98 QID 4 timeout, reset controller
Feb  1 02:22:59 untheran kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Feb  1 02:22:59 untheran kernel: nvme nvme0: Abort status: 0x371
### [PREVIOUS LINE REPEATED 4 TIMES] ###
Feb  1 02:25:00 untheran kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Feb  1 02:25:00 untheran kernel: nvme nvme0: Removing after probe failure status: -19
Feb  1 02:27:00 untheran kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1

 

Try power cycling the serve to see if it comes back.

I just saw that the docker file was not  found. Then my appdata share was not there anymore ...  But My cache drive is still in my main tab, no errors ... Strying to stop the array to start it back and it takes forever. Forces a reboot. The cache drive wasn'T there anymore ....

lsblk, no 2TB Drive ...
Went to bios, not there.
Shutdown the server, shut off the PSU, reslotted the M.2 drive, And now it'S back !

Very strange, I'll keep an eye on it for the next days.

2 hours ago, Altheran said:

Forces a reboot.

 

3 hours ago, JorgeB said:

Try power cycling the serve to see if it comes back.

 

A reboot doesn't usually help.

I've been experiencing the same issue over the last couple of days with the very same behavior. As soon as I put load on the drive, it goes offline and even a reboot doesn't help.

 

I had the same problem a couple of weeks ago back when the drive wasn't housing my Docker files and just acted as a secondary cache drive and iirc I actually managed to fix this, but for the life of me I can't remember what I did...back then it happened when trying to remove unnecessary streams from mkv files - the drive just collapsed. After I changed something though it worked without issue.

 

EDIT:

Same issues described here:

 

 

 

EDIT2:

 

I found my fix in my backed up config files: Adding "nvme_core.default_ps_max_latency_us=0" to "Unraid OS" in the syslinux configuration made the issues disappear.

Edited February 6, 20215 yr by Pillendreher

Thanks for your fix. I haven't run into the same problem again, and have added a second NVMe drive to the cache pool. but just responding so I have something to reference if it happens again

I'd just given up and moved everything off the cache until a real fix was found. Applied the fix last night and so far everything looks good, I'll keep an eye on it over the next week and post back if anything happens.

 

Cheers!

Hi Awesome Community, related to this initial issue. I have been facing the same issue (see screen shots) everytime i clean shut down and restart the unraid server. This has begun to grate (5-6 times) as each time i must restore all docker images from previous downloads and restore vm backups. Previous posts has pointed at the images getting corrupted but i am unsure as to why and what the resolution is. Attached is diagnostics, i invite comment.

 

**diagnostics deleted for system security**

 

Edited October 11, 20223 yr by Be-Art

The pool dockervmcache is having some issues.  Not a btrfs guy, so wait for @JorgeB

Pool should be backed up and reformatted. 

@Squid & @JorgeB Thanks for your responses. I have formatted the pool again and restored appdata and vm images. I have been able to shutdown and start up again without corruption. Will monitor.