Unraid becomes unresponsive during parity-check, mover operations

[dziewuliz]

Recently installed a new i5 cpu, in order to add a second cache disk to have a zfs pool, for redudancy.

Since then, I noticed my Unraid server would become unresponsive, during scheduled mover operations, or manual parity check operations -  inaccessible via GUI or ssh. The server power is still on and running.

 

Attaching diagnostics and syslog (stops responsing post 6am).

 

tower-diagnostics-20231111-1129.zip syslog.zip

Edited November 11, 2023 by dziewuliz

[itimpi]

Have you tried running without Docker started?    I noticed that there appeared to be a container continually restarting although this should not stop everything from running.

 

I also noticed that the 'system' share had files on disk1 as well as cache.   if this is the docker.img file then it would be expected that performance would be badly degraded while the operations you mention are running.   Your current settings for this share do not allow mover to transfer files from array->cache.  In addition since all the files are not on the cache you will not be able to enable Exclusive mode on the share to get better performance.

 

You also have a share anonymized as 'd-------s' that has files or folders on the cache but the settings do not allow mover to transfer the files off the cache to the array - is this intended?

 

One thing that occurs to me is there any chance that you have a power related issue?   An operation like Parity Check starting would put maximum load on the power supply.   It can also sometimes cause an issue if you make use of power splitters and connect too many drives to a single power supply line.

[dziewuliz]

Thanks helping with the troubleshooting, @itimpi

 

On 11/11/2023 at 6:26 PM, itimpi said:

Have you tried running without Docker started?    I noticed that there appeared to be a container continually restarting although this should not stop everything from running.

 

Stopped docker, and ran the parity check successfully (with some 170 sync-errors)

 

On 11/11/2023 at 6:26 PM, itimpi said:

I also noticed that the 'system' share had files on disk1 as well as cache.   if this is the docker.img file then it would be expected that performance would be badly degraded while the operations you mention are running.   Your current settings for this share do not allow mover to transfer files from array->cache.  In addition since all the files are not on the cache you will not be able to enable Exclusive mode on the share to get better performance.

 

There was an empty system folder on disk1, deleted.

 

On 11/11/2023 at 6:26 PM, itimpi said:

You also have a share anonymized as 'd-------s' that has files or folders on the cache but the settings do not allow mover to transfer the files off the cache to the array - is this intended?

 

Stored syslog there.

 

On 11/11/2023 at 6:26 PM, itimpi said:

One thing that occurs to me is there any chance that you have a power related issue?   An operation like Parity Check starting would put maximum load on the power supply.   It can also sometimes cause an issue if you make use of power splitters and connect too many drives to a single power supply line.

This was my concern as well - I have a UPS (500w), Corsair PSU (450W) and 10x 3.5HDD's, some 6 of which were connected through power splitter cable. 

 

Now, since I was able to complete parity check without issues, would it be safe to assume this is Docker related issue? I will resume Docker, and continue turning on one at a time, to see if it triggers the same issue. Any other suggestions?

 

Thanks a lot.

[dziewuliz]

Same result - became unresponsive over night. Attaching syslog and diagnostics:

tower-diagnostics-20231113-1340.zip

syslog.zip

[itimpi]

The last thing before the crash appears to be a macvlan related crash and such crashes seem to eventually crash the whole server.  Either switch docker to using ipvlan or make sure you have macvlan set up as indicated in the 6.12.4 release notes.

 

[dziewuliz]

Thanks, this was indeed the case. Both suggested solutions worked.