Jump to content
We're Hiring! Full Stack Developer ×

Sporadic System Freeze/GUI Unavailability


Recommended Posts

Hey all. 

 

Long-time listener, first-time caller. I have a home server (E5-2667 v2 / X9DRL-iF) running Unraid 6.12.4. I have a single SSD cache (Samsung 850 Evo 500GB) and a variety of disks (8 total, including parity). Sporadically, though seemingly correlated to high-IO situations (copying large files, running Mover, etc) the system will be unavailable via GUI. Docker applications will function for some time but eventually begin throwing errors (query timeouts, etc). I followed instructions to log in via SSH and run the "diagnostics" command, the output of which I'm attaching here.

 

One of the error messages in the logs that sticks out to me is:

time="2024-01-17T23:44:01.928430881-07:00" level=warning msg="error copying stderr" runtime=io.containerd.runc.v2

 

I saw an older thread mentioning that this could be a cache issue. Any thoughts? Thanks in advance for your time.

tower-diagnostics-20240117-2345.zip

Link to comment
4 minutes ago, trurl said:

Probably unrelated to your issue, but your appdata and system shares have files on the array.

 

 

That time stamp does not appear in your diagnostics. Post new diagnostics.

 

Not familiar enough to understand the significance of the first part.

 

Regarding the second part, I checked the zip archive contents again, and that timestamp/error definitely exists. There are also multiple occurrences, not just at that timestamp, found inside docker.txt.

Link to comment
10 minutes ago, cdcoulson said:

found inside docker.txt

I see, don't know what it means. Doesn't seem like it should affect the host though. Lots of free RAM, rootfs isn't full, nothing else obviously wrong.

 

You do have a rather long uptime, though that shouldn't be a problem either. Most diagnostics I get to see are from people who have unfortunately already rebooted before getting diagnostics.

 

11 minutes ago, cdcoulson said:

significance of the first part

Ideally, Docker/VM related shares (appdata, domains, system) would have all files on cache or other fast pool with nothing on the array, so Docker/VM performance won't be impacted by slower array, and so array disks can spin down since these files are always open.

 

 

Link to comment
2 minutes ago, trurl said:

I see, don't know what it means. Doesn't seem like it should affect the host though. Lots of free RAM, rootfs isn't full, nothing else obviously wrong.

 

You do have a rather long uptime, though that shouldn't be a problem either. Most diagnostics I get to see are from people who have unfortunately already rebooted before getting diagnostics.

 

Ideally, Docker/VM related shares (appdata, domains, system) would have all files on cache or other fast pool with nothing on the array, so Docker/VM performance won't be impacted by slower array, and so array disks can spin down since these files are always open.

 

 

Ah, understood. Yep, during impact, I'm able to log in via SSH and run commands without issue. When doing so, I typically do see load average spike (> 40.0) fairly high. I forgot about this part: using a CLI client on my phone, I logged in via SSH and started looking at IO stats for the disks and noticed that sdj (cache) had a relatively high counter for "kB_dscd" -- is this possibly a separate/another indicator of an issue with the cache disk?

 

https://imgur.com/a/4Cg4B1Y (Apologies for the formatting -- tough to correlate values to headers, but it's definitely discarded KB to sdj -- cache)

 

I see. I just checked, and appdata is set to "Cache" for primary storage with no secondary storage selected; system is set to "Cache" for primary storage, "Array" for secondary storage (High Water/Automatically split only the top level directory as required/All/None) with Mover action set to "Array -> Cache". Is it preferred to set system share to only use Cache (i.e. unset secondary storage option), then run Mover?

Link to comment

Mover ignores shares with nothing set a Secondary since they don't have a Mover action.

 

Nothing can move open files. You have to Disable Docker and VM Manager in Settings before these can be moved.

 

Mover won't replace files so if something exists in both places you will have to decide which to keep.

 

You can see how much of each disk or pool is used by each user share by clicking Compute... for the share on the user shares page.

 

Dynamix File Manager plugin will let you work with folders and files directly on the server.

Link to comment
36 minutes ago, trurl said:

Mover ignores shares with nothing set a Secondary since they don't have a Mover action.

 

Nothing can move open files. You have to Disable Docker and VM Manager in Settings before these can be moved.

 

Mover won't replace files so if something exists in both places you will have to decide which to keep.

 

You can see how much of each disk or pool is used by each user share by clicking Compute... for the share on the user shares page.

 

Dynamix File Manager plugin will let you work with folders and files directly on the server.

Thanks.

 

For appdata, I have data on cache (19GB), disk 1 (1GB) and disk 2 (1GB) despite it being set to Cache for Primary and nothing to Secondary. I enabled Array as Secondary, then "Array -> Cache" to allow me to use the Mover with the goal of shifting all appdata files from Disks 1/2 to Cache. That's running right now.

 

(EDIT: I followed instructions and disabled Docker/VM Manager prior to making any changes.)

 

For system, I have data on cache (1GB) and disk 1 (33GB). Again, settings are Cache primary & Array secondary with Array -> Cache. I did not change any of these settings. Will Mover also handle these files? If not, how can I tell from CLI which files reside on cache/array?

 

I'm pretty familiar with the command line -- don't really need a file manager, unless that can somehow show me which files/directories are on cache versus array.

 

(EDIT: Apologies, realized I could simply navigate to /mnt/cache & /mnt/diskX to compare and remove files as-needed. Thanks!)

 

Thanks!

Edited by cdcoulson
Link to comment

Okay, more targeted question after my previous edits/discoveries.

 

I sorted out my appdata situation -- not sure how duplicate directories were created off-cache, but it's sorted, now.

 

For system, though, the issue still remains: I have data on cache (1GB) and disk 1 (33GB). Again, settings are Cache primary & Array secondary with Array -> Cache. I did not change any of these settings recently.

 

How can I move the 33GB system folder on disk 1 to cache? Should I?

 

(EDIT: Docker disabled, VM Manager disabled, settings above for system: [Cache primary, Array secondary, Array -> Cache], and when invoking Mover, nothing happens )

 

 

Edited by cdcoulson
Link to comment
6 minutes ago, cdcoulson said:

How can I move the 33GB on disk 1 to cache? Should I?

Nothing can move open files. And you shouldn't move if a file exists both places unless you decide to keep the one you are moving.

 

You can use the command line if you want. But be careful with that. User Shares are just different paths to the disk files, but Linux doesn't know that. Don't mix disks and user shares when moving or copying files. Linux won't know if the source and destination are the same file, and will try to overwrite what it is trying to read.

 

Dynamix File Manager won't allow you to make this mistake.

Link to comment
1 minute ago, trurl said:

Nothing can move open files. And you shouldn't move if a file exists both places unless you decide to keep the one you are moving.

 

You can use the command line if you want. But be careful with that. User Shares are just different paths to the disk files, but Linux doesn't know that. Don't mix disks and user shares when moving or copying files. Linux won't know if the source and destination are the same file, and will try to overwrite what it is trying to read.

 

Dynamix File Manager won't allow you to make this mistake.

 

Understood on the first sentence.

 

The problem is: the docker directory on cache is empty. Docker directory on disk1 (array) is 31G. If Docker/VM Manager are disabled, and the system share is set to cache primary/array secondary and Array -> Cache, why won't the Mover move the contents of the docker directory from disk 1 (array) to cache? They're still in-use (how, if Docker is disabled?)

 

Unless Mover won't move the docker directory because the docker directory on cache already exists (despite being empty)?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...