Strange Docker problems?

[casperse]

Hi All

 

After replacing my cache with a large drive I have had problems with my dockers

I then did a rebuild - deleted my old docker-file and downloaded and reinstalled all my Dockers

But afterwards I see strange behaviour Dockers that will not start but seem to be running (Green arrow)

Dockers that just stops? or Emby that was running perfect before just stops and give errors? (Didn't change anything)

 

 

Here is my Diagnostic files I hope someone can help me!

Your help is much appreciated this is so strange?

Gone from a stabile system to a "GHOST IN THE MACHINE" 

 

My container size is 40G no where near that:

 

plexzone-diagnostics-20200224-2209.zip

Edited February 24, 2020 by casperse

[casperse]

This really scares the shit out of me......

 

How can the user path be gone?

 

 

Should I reboot the system?

Seems my User shares disappeared?

 

Starting and stopping the array didn't help....

 

Rebooting and hooping that will get my user shares back....

 

Edited February 24, 2020 by casperse

[testdasi]

Do a disk check on your USB stick.

Stick dropping offline / corrupt can cause weird problems.

[casperse]

8 hours ago, testdasi said:

Do a disk check on your USB stick.

Stick dropping offline / corrupt can cause weird problems.

You might be right... just got this new error

I think the Nvidia files is on the flash so that would make sense!

 

How to check in Unraid a check command from the command line?

 

 

 

[casperse]

How to run this manually? or see the log?

I currently have a very strange problem and it might be the USB?

 

I had the user shares disappear! - I did manage to get a Diagnostic before rebooting if anyone would take a look I would be really happy

 

 

 

[casperse]

Update: Following the error leading to the Nvidia plugin which apparently also located on the USB?

I re-installed the plugin and rebooted the server, sofar it looks like everything is running again?

 

Could this cause the loss of User shares? (I didn't know that the Nvidia plugin was such a big deal of the Unraid server, if this is the case?)

 

[itimpi]

Just now, casperse said:

Update: Following the error leading to the Nvidia plugin which apparently also located on the USB?

I re-installed the plugin and rebooted the server, sofar it looks like everything is running again?

 

Could this cause the loss of User shares? (I didn't know that the Nvidia plugin was such a big deal of the Unraid server, if this is the case?)

 

It is unlikely to be the Nvidia plugin as all that does is control whether the Unraid OS files on the USB stick  that are the standard version should be replaced by the version that includes the Nvidia drivers.     This replacement will only happen at the time you load a new version via the Nvidia settings - it will not have any affect the rest of the time as once a version is loaded onto the USB stick it boots just like the standard Unraid.

 

As was mentioned it is likely that the USB stick going offline for some reason caused the problem and rebooting fixed this.   I guess there could be an inherent fault in the version of Unraid OS that the Nvidia plugin has installed, but this seems unlikely or more people would be having the same problem.

[trurl]

7 hours ago, casperse said:

I did manage to get a Diagnostic before rebooting if anyone would take a look

Post them. I'll likely split this into its own thread since it is a support request not a feature request.

[casperse]

9 hours ago, itimpi said:

It is unlikely to be the Nvidia plugin as all that does is control whether the Unraid OS files on the USB stick  that are the standard version should be replaced by the version that includes the Nvidia drivers.     This replacement will only happen at the time you load a new version via the Nvidia settings - it will not have any affect the rest of the time as once a version is loaded onto the USB stick it boots just like the standard Unraid.

 

As was mentioned it is likely that the USB stick going offline for some reason caused the problem and rebooting fixed this.   I guess there could be an inherent fault in the version of Unraid OS that the Nvidia plugin has installed, but this seems unlikely or more people would be having the same problem.

My USB was placed inside a rackmount server in the middle of the motherboard - Safe place but hard to get to

I did run several test and all was clean?

 

My hope is that the logs above will give a hint to what might caused this....

 

[casperse]

3 hours ago, trurl said:

Post them. I'll likely split this into its own thread since it is a support request not a feature request.

Thanks @trurl I added my logs to the first post in this thread:

BTW: Just took my server apart to get the USB out and I did several disk check in windows all was fine no errors

[trurl]

Why do you have disks assigned in 2-7 and 10-22 but none assigned to 1,8,9?

[jonp]

I took a look at your system log and did see some messages that I shouldn't have.  Specifically I saw a kernel message relating to python:

 

Feb 24 10:26:30 PLEXZONE kernel: python[5234]: segfault at 6e6f68747988 ip 000014bccd18db98 sp 000014bcc6979c68 error 6 in ld-musl-x86_64.so.1[14bccd180000+46000]
Feb 24 10:26:30 PLEXZONE kernel: Code: 48 8b 47 10 48 39 47 18 75 14 89 f1 48 c7 c0 fe ff ff ff 48 d3 c0 f0 48 21 05 f4 ee 06 00 48 8b 57 18 48 8b 47 10 48 89 42 10 <48> 89 50 18 48 8b 47 08 48 89 c2 48 83 e0 fe 48 83 ca 01 48 89 57

 

I see you have a LOT of plugins installed.  Please try booting in safe mode, disable all plugins, and see if things go back to normal.

[casperse]

10 hours ago, jonp said:

I took a look at your system log and did see some messages that I shouldn't have.  Specifically I saw a kernel message relating to python:

 

Feb 24 10:26:30 PLEXZONE kernel: python[5234]: segfault at 6e6f68747988 ip 000014bccd18db98 sp 000014bcc6979c68 error 6 in ld-musl-x86_64.so.1[14bccd180000+46000]
Feb 24 10:26:30 PLEXZONE kernel: Code: 48 8b 47 10 48 39 47 18 75 14 89 f1 48 c7 c0 fe ff ff ff 48 d3 c0 f0 48 21 05 f4 ee 06 00 48 8b 57 18 48 8b 47 10 48 89 42 10 <48> 89 50 18 48 8b 47 08 48 89 c2 48 83 e0 fe 48 83 ca 01 48 89 57

 

I see you have a LOT of plugins installed.  Please try booting in safe mode, disable all plugins, and see if things go back to normal.

@jonp Yes I am "like many others" a Space invader fan 😊 - and moving from Synology I found many great plugin that helped me accomplish this

But you are right I need to remove the non essential plugins! - problem is that many are now "mandatory"

I guess if possible it would be better to move to dockers if I can find a similar application to replace the plugin

 

I am now running in safe mode and have attached a new Diag file - hope the error is gone now!

 

What to do next? - retracing and uninstalling every plugin, starting from the latest ones?

(If you tell me where to look for this python error I can search for it after each reboot?)

 

Again your help is much appreciated - hope we can find the source of this problem and kill it

 

@trurl I have all my drive numbers in Unraid assigned to the corresponding numbers of my servers HD cases (Make it much simpler to find a failing drive)

Drive one is a Un-Assigned Drive - so I dont spin up the array - was inspired by this post about seeding and ended up with this solution - another forum post ,(Drive 8 and 9 is temporary empty because I am slowly replacing all my 3TB drives with larger ones - So drive 8 and 9 was moved to a larger drive 6 and so on its slow but it works and it requires $ so it's a slow process 😉 )  

 

plexzone-diagnostics-20200226-0809.zip

[casperse]

Update:

This might not be related?

 

But all this started after upgrading to a new bigger M.2 cache drive, from a 1TB to a 2TB drive

Could this somehow be the problem? (I formatted it like the old one "btrfs encrypted")

Recreated the docker image and re-installed all the dockers!

Only the Apdata is copied back to the new cache drive

 

Below removed this had nothing to do with the error!

Focusing on testing Plex for my stability testing...

I just did a small test by copying the Plex Apdata folder to a UAD SSD and did a install with the same user "docker profile" but pointing to the SSD for Apdata

and then Plex started to work like it did before (Exactly the same files) now HW encoding for TV works it doesn't when running from the cache drive?

- This is so weird, it's like all logic is out the window..... Again this might not be related and just me grasping at some point of error origin 

 

I have so far removed the latest added plugins:

Custom Tab

Dynamix File Integrity

Dynamix Active Streams

Server Layout

Edited February 28, 2020 by casperse

[trurl]

8 hours ago, casperse said:

Only the Apdata is copied back to the new cache drive

This is usually spelled "appdata". If you actually used "Apdata" then you may have created some problems.

[casperse]

On 2/27/2020 at 2:58 AM, trurl said:

This is usually spelled "appdata". If you actually used "Apdata" then you may have created some problems.

No its ok "appdata" just a misspelling here....

[trurl]

On 2/26/2020 at 12:19 PM, casperse said:

Recreated the docker image and re-installed all the dockers!

Your system share has some files on the disk7. Maybe libvirt instead of docker image though. You might just check the contents of disk7 to make sure your docker image is on cache now.

 

 

[casperse]

48 minutes ago, trurl said:

Your system share has some files on the disk7. Maybe libvirt instead of docker image though. You might just check the contents of disk7 to make sure your docker image is on cache now.

 

 

Yes I can see that this is the libvirt

I have disabled VM and I am running mover now, shouldn't that move the file to cache? (I have system prefer cache setting)

[trurl]

4 minutes ago, casperse said:

I have disabled VM and I am running mover now, shouldn't that move the file to cache? (I have system prefer cache setting)

If you disabled it in Settings - VM Manager then that should work.

[itimpi]

5 minutes ago, casperse said:

Yes I can see that this is the libvirt

I have disabled VM and I am running mover now, shouldn't that move the file to cache? (I have system prefer cache setting)

If it still will not move then check that there is not already a copy on the cache drive.    Mover will not do anything if the file already exists on the target.

[casperse]

36 minutes ago, itimpi said:

If it still will not move then check that there is not already a copy on the cache drive.    Mover will not do anything if the file already exists on the target.

Yes its on both? - which one should I delete

Update I deleted the oldest file not updated on drive 7

Edited February 28, 2020 by casperse

[casperse]

Ups didnt like that getting a Libvirt does not start error now....

Edited February 28, 2020 by casperse