Hi friends,

 

It has been quite an eventful morning. First, thank you for reading my post. There is a long story of troubleshooting that precedes this current issue, included at the bottom of the post. I am concerned about the default shares available under the 'shares' tab, which appear to have attempted to default to a new Cache pool that was created - 'cachefast' - instead of remaining on the standard 'cache' pool/drive. The intention of the new drive is to act as a dedicated share for VMs/appdata while the old would serve for the array cache and downloads. However, I hadn't made any changes to the shares configs at this point.

 

Thankfully, my non-default Data shares appear normal and remain set to Cache (primary) and Array (secondary). The 'Fix Common Problems' plugin is alerting to this issue, with the following messages: 

 

"Share appdata set to use pool cachefast, but files / folders exist on the cache pool".

 

Okay, so I understand that it is telling me to move the files or reconfigure the share. I tried that with the 'isos' share, but may have done something wrong? Now it is showing the opposite message when the share is set to Cache (primary) and Array (secondary). Should that be changed to Cachefast (secondary)?

 

"Share isos set to use pool cache, but files / folders exist on the cachefast pool"

 

When I go to 'Docker', it showed No Containers Installed. I verified that the data still exists in user/appdata, and try to re-install one of the containers. I reinstalled FreshRSS and it picked up right where things left off. Same with Observium and Plex, thankfully - so the containers may just need to be reinstalled?

 

I would really appreciate some assistance in reviewing this issue and ways to address. Software troubleshooting is not my strong point. I am happy to post any screenshots or diagnostic information that would be helpful, but to be quite blunt, I am terrified of restarting the server at this point. There is little confidence that it will boot cleanly - I've gotten it up and online, and I would like to stop walking back to that room for now.

 

Background:

This is very truncated, but all started with a hardware upgrade yesterday, moving from an AMD 5600g/32gb/MSI B550 to an Intel 14500/64gb/Asus Z790 and installing the new SSD (cachefast). I assemble the new parts outside of the system, and it boots without issue once the BIOS is updated. However, the memory became unseated when installing the board in the chassis and I noticed that the server only showed 32gb when online. This was after multiple reboot attempts and periods of memory training, very stressful. Once the array is started and things appear normal with the new hardware, I decide to leave the memory issue until this morning. 

 

This morning, I take flash and appdata backups, and shutdown the system. I found that both sticks were not fully seated and correct that issue. However, the board takes a long time and multiple reboots again before showing any post output. Once I am able to get in to unraid after ~30m of troubleshooting, I can see 64gb and attempt to start the array. However, it does not accept my normal password. I re-enter the same pw and it is accepted, the array starts, but two of the Data disks show unmountable/no file system. My heart sinks and the server kernel panics shortly after. I am able to get back to the BIOS eventually with one stick installed, and boot back up in to unraid. The system again rejects my password then accepts, and now shows 8 of 9 Data drives as 'unmountable'. Another reboot, and I launch the Memtest86 from unraid. The memory stick almost immediately fails. I swap it for the other stick - installed in to a different slot - and the machine takes some time but eventually boots again.  After 1.5hr of testing with no issue, I stop Memtest and try to boot in to unraid. Not clean, another round of memory testing and anxiously waiting for POST output. Finally, the dang thing starts and I'm back in unraid. I successfully start the array with no password error or missing disks. That is what lead to now, where I noticed the Docker containers missing. For some reason, the plugin Unraid Connect was also removed - but that is reinstalled now.

 

Thank you again for reading my novel and any suggestions you can offer,

Outlaws

 

Attach Diagnostics to your NEXT post in this thread.

Here, apologies - Diagnostics output

unii-diagnostics-20250223-1143.zip

13 minutes ago, Outlaws said:

new Cache pool that was created - 'cachefast' - instead of remaining on the standard 'cache' pool/drive. The intention of the new drive is to act as a dedicated share for VMs/appdata

The relevant shares

appdata  # Share exists on cachefast, cache
domains  # Share exists on cache
isos     # Share exists on disk1, disk6, disk7, cachefast
system   # Share exists on cachefast, cache

Set appdata, domains, system to

Primary:cachefast; Secondary:cache; Mover action:cache->cachefast

 

Doesn't really matter where you put isos, but for consistency, you could do

Primary:cachefast; Secondary:array; Mover action:array->cachefast

 

Nothing can move open files. Disable Docker and VM Manager in Settings

 

Then run mover, wait for it to complete, post new diagnostics.

 

 

Thank you for the quick response! I have tried that suggestion with the 'domains' share as a test, since there should be 1 VM but I don't really care about it like I do the appdata information.

 

I set the mover action as suggested and invoked the mover, and it moved about 13gb over. However, the VM manager is not starting - shows the error "Libvirt Service failed to start." and no logs. I disabled the VM Manager again and captured Diagnostics - attached. Let me know if I should rerun with the VM manager set to enabled. Could this be because I did not touch the system share - where I see libvirt.img?

 

I see now that 'system' is set to primary:cachefast and secondary:array with Mover being cachefast -> array. I can't recall if this was the setting was done before the first diag was taken, if you could please compare and check that. I am guessing the best move would be to have it set to Primary:cachefast; Secondary:cache; Mover action:cache->cachefast and see if the VM manager can start?

 

When I look in the File Manager, I can see the 3x docker containers I re-installed now show as having data on 'cachefast' and 'cache'. All the others are cache only.

 

At this point, I want to 1. copy the appdata information off of cache/cachefast to the array as an additional backup, just in case it gets overwritten in a mover event - should this be  possible via the built in File Manager or something like Krusader? I am hoping to just copy the files from the appdata share, not try to move the share to Array as primary/secondary. Then 2. I would like to try restoring the containers on the 'Cache' pool and have 'Cachefast' back to blank/unused.

 

Eventually, I see that the new 'cachefast' drive was formatted without encryption. I would prefer to reformat it as btrf - encrypted to match the other cache drive and other array devices being encrypted, and because it was formatted with the bad memory installed in the system. But it looks like this involves stopping the array, so I will probably save that for another day.

unii-diagnostics-20250223-1213.zip

Okay - I compared the 1143 diag file vs the 1213 file, and they both show the same config for the system share - so not sure when I set that to have Array as secondary. But I have updated system to now use what you recommended, Primary:cachefast; Secondary:cache; Mover action:cache->cachefast -- but I have not invoked the mover just yet. Wanted to correct that before it tried to split the docker.img and libvirt.img across all three pools.

29 minutes ago, Outlaws said:

copy the appdata information off of cache/cachefast to the array as an additional backup

Appdata Backup plugin.

 

If you intend to reformat cachefast as encrypted better to do it now instead of later.

 

While you have Docker and VM Manager disabled, you could just set Mover action for all those to cachefast->cache to get cachefast emptied and ready to reformat.

 

Okay, that sounds like an action plan. I already have the Appdata Backup plugin files from before this morning and before the HW upgrade yesterday - just wondering about backups for the backups lol. 

 

I did see on their Restore page where it notes that it cannot rebuild the Docker Containers, which I think is the part that is malfunctioning here?  When I rebuild the docker container via installing one from the App store, it can detect the previous configs from appdata. It shows those 3x apps as having data on cachefast and cache, so I'm guessing it installed the new container on cachefast. What would be the best way to get Docker to recognize the previous Docker containers on 'cache', or is it better to reinstall them all? Do you think disabling Docker and setting appdata to Primary:cache; Secondary:cachefast; Mover action:cachefast->cache, then invoking mover would be the best step? 

 

I really appreciate the help on a Sunday afternoon 🥲

 

 

The entries from the Add (or Update) Container page are stored as xml templates on the flash drive. Previous Apps on the Apps page will use those templates to reinstall.

 

Containers that are already installed should find their appdata, assuming your containers were setup to reference user share appdata (/mnt/user/appdata) instead of cache directly (/mnt/cache/appdata). It won't matter which pool appdata is actually on, since all pools and array disks are part of user shares.

trurl, thanks again for your support and suggestions yesterday! The tech world exists because of people like you, kindly sharing their knowledge. As I had just recently upgraded my license key (in order to add the new cache drive), I have created a support case with Limetech so that they can review this issue and behavior as well. If the container XMLs are stored on the flash drive, I'm hopeful that they can be restored or pulled from one of the flash backups that I've taken recently.

 

I have paused the mover for now, and will update here once support is able to review and offer a suggestion.

2 hours ago, Outlaws said:

offer a suggestion

What exactly are you wanting a suggestion about? I have already given several, but maybe I missed the specific thing in the midst of all the other text.

19 minutes ago, trurl said:

What exactly are you wanting a suggestion about? I have already given several, but maybe I missed the specific thing in the midst of all the other text.

 

You're right, you have been a tremendous resource and sanity check - and apologies that I type as if I'm paid by the word. Support was able to share why this issue may have occurred:

 

the syslog shows that the issue resulted from the old cache not mounting initially, this was likely caused by the bad RAM issue, and since the default shares didn't exist, Unraid created new ones on the other pool, since that one mounted.

 

Right now, I believe that I understand the plan for getting files back on to the 'cache' pool. I also think that explains why it looks like I have duplicate filespace consumed up on both drives, if it recreated the docker and libvirt.img on both 'cache' and 'cachefast'. 

 

However, before I take any further mover actions or stop the array, I want to know if it is possible to get any reassurance that it will remount the array without similar errors that I saw with the faulty RAM installed (where password was initially rejected, and some Data drives showed Unmountable - no file system). Would there be any way to validate the file system on the drives or the password integrity? Or am I stuck with having to stop the array and finding that out when I try to remount?

 

After that, I will revert to trying to restore the docker containers.

Edited February 24, 20251 yr by Outlaws

11 minutes ago, Outlaws said:

I want to know if it is possible to get any reassurance that it will remount the array without similar errors that I saw with the faulty RAM installed (where password was initially rejected, and some Data drives showed Unmountable - no file system

While it is possible those were caused by bad RAM, those problems are not specific to bad RAM and we have seen them for other reasons on systems where RAM was OK.

1 hour ago, Outlaws said:

where password was initially rejected, and some Data drives showed Unmountable - no file system

This is a typical symptom of bad RAM, it will fail to decrypt the passphrase/key for some drives and work for others, so most likely that was the issue, but we cannot say it was that for sure, I would say 95% that was the problem, so if the issue is resolved, it should not happen again.

26 minutes ago, JorgeB said:

This is a typical symptom of bad RAM, it will fail to decrypt the passphrase/key for some drives and work for others, so most likely that was the issue, but we cannot say it was that for sure, I would say 95% that was the problem, so if the issue is resolved, it should not happen again.

Hi Jorge - love the circuit profile picture. Thanks for this note of reassurance, it makes sense that the ram is involved in crypto processes. Obviously, my biggest worry is that I will stop the array and be unable to re-mount - are there any additional checks that I can perform with the array mounted via gui or terminal to check the file system & hopefully address these concerns? Sorry if it feels like I am asking the same questions here, I really appreciate both of your input!

22 minutes ago, Outlaws said:

it makes sense that the ram is involved

The RAM is involved in EVERYTHING. The CPU can't do anything with anything until it is loaded into RAM.

 

23 minutes ago, Outlaws said:

are there any additional checks that I can perform with the array mounted via gui or terminal to check the file system & hopefully address these concerns?

We will see what needs to be done after you post new diagnostics with the array started.

Hi trurl,

 

I think the only thing that will make me trust this machine again is to remove the new hardware (motherboard/processor/memory) - except the new SSD, since that can be disabled via software. I am going to revert back to a known good combination of hardware hopefully Wednesday, once a part arrives overnight and I can spend some time testing it outside of the unraid box.

 

When I looked at the Log from GUI, I can see that it reports a kernel panic overnight at 1:15am. This looks similar to the kernel panic that caused the machine to hang & reboot the morning prior. If you have any interest, I've attached a current diag + log + a capture of the original kernel panic.

 

One thing I am still wondering - why is Fix Common Problems alerting for this? Could it not be recognizing both pools as cache for some reason?

 

appdata: Primary: Cachefast, Secondary: Cache 

 

Feb 23 12:58:34 Unii root: Fix Common Problems: Error: Default docker appdata location is not a cache-only share

unii-diagnostics-20250224-1540.zip unii-log.txt

Feb 24 01:15:06 Unii kernel: ------------[ cut here ]------------
Feb 24 01:15:06 Unii kernel: Can't encode file handler for inotify: 255

 

The panic in the syslog is harmless, it's a kernel issue, everyone gets it once per boot.

3 hours ago, JorgeB said:

Feb 24 01:15:06 Unii kernel: ------------[ cut here ]------------
Feb 24 01:15:06 Unii kernel: Can't encode file handler for inotify: 255

 

The panic in the syslog is harmless, it's a kernel issue, everyone gets it once per boot.

Jorge, thank you for confirming that these logs are not indicating another issue on my side!! I am still most likely going to revert the hardware change so I can regain some trust in the system, and that will allow me to test that new hardware further outside the chassis.

 

My main question at this point is what settings I should configure for the appdata/system shares prior to shutting down? I am most likely going to remove the new 'cachefast' SSD after all, and repurpose that for another system. For that scenario, would it be best to 1. configure the shares as 'cache' only, not pointed to 'cachefast', and then 2. restart and mount the array without the 'cachefast' pool? 

 

Should I avoid invoking the mover since the original containers/data should still exist on 'cache', and it may try to overwrite them with the new files on /cachefast/? I can see via the File System that it created a docker.img file in /cachefast/system which is the same size as the one in /cache/system

 

I have attached a backup of the config/shares folder from before any of these changes if that would be helpful to compare vs the current settings.

 

Thank you again for all the assistance so far!

Thomas/Outlaws

original shares.zip

I would not touch the settings for now, just stop the array, remove the new pool, start the array and post new diags.

 

Mover will not overwrite existing files, but will also fail to move them, but the old pool should still have all the files that you need.

Understood. I will probably take more actions tomorrow after I can finish testing the old equipment that I want to reinstall. So the current plan from my side would be to:

 

1. Take diag/flash backup

2. Not change share settings/mover actions

3. Disable Docker service

4. Stop the array

5. Remove the new SSD from the new 'cachefast' pool

6. Remove the new 'cachefast' pool

7. Start the array and pray all the disks show healthy

8. Start the Docker service, and see if containers have restored

9. Take diag/flash backup

10. Pause if things appear to have returned to normal, and consider if I still want to proceed with hardware backout

 

Does that sound right, or would you modify any steps?

Sorry for the bump - a few quick edits to the plan above before I move forwards tomorrow morning. I think the best action would be to have the shares configured correctly before stopping and restarting the array - so they know exactly where to look for their original files. I plan to disable docker, flip them back to primary:cache, but not invoke the mover.

 

Fingers crossed & I will let you know how it all goes!

Looks OK to me, you can skip #5

Hi All,

 

Thank you for the help!!! I have mostly good news to report. I was able to set the shares back to Primary:Cache, stop the array, remove the new pool, and restart the array without issue. Once I started Docker again, the original containers re-appeared.

 

Most of the containers seem to be good - however, the most important container in binhex-plexpass seems to have become broken? I've attached the output of the GUI and container log to this post, but recognize it may be best to start a new thread or post in the Binhex thread for further support with this specific issue.

 

I don't see any specific error in the container log except a usb error, and I can verify the container has network connectivity from it's console.

 

 

binhex-plexpass container log.txt

Apologies, here is a new Diag as well.

unii-diagnostics-20250226-0642.zip