geekypenguin

Would trying a bios update be worthwhile before I go spending money?

Sorry it's taken a few days to respond, there's been a lot to work through and while I still don't know the cause, I reached a point where I had to stop and revert to known good. Firstly I had the macvlan kernel issue that's known in 6.12 which frustrated things. With both sticks of ram on their own, I was getting data corruption errors, always in the same disk. I was also getting corruption of my docker.img which was causing docker's to crash without the cache going read only. As the nvme drives were new, I got a warranty replacement on the nvme drive with all the errors and attempted to rebuild the cache pool onto the second drive, but was flooded with "nvme frozen state error detected, reset controller" etc messages for the replacement drive. I read in a few bug reports to add ```nvme_core.default_ps_max_latency_us=0 pcie_aspm=off``` to the boot config but this didn't help either. This is unfortunately where I had to stop. I've removed the second cache drive and reverted to single drive mode for my cache which has been working fine for a few days now with all the ram re-installed. Not sure where else to go from this to be honest. I can stay like this with no redundancy on my cache I suppose but would like to get to the bottom of it.

Thanks for you help. Removed one ram stick and reset the stats and configured the userscript as suggested. I'll let you know how it gets on

Ok First two passes of memtest have returned zero errors. I'll keep it running a bit longer to be sure none materialise

Thanks, I'll run a memtest now. I saw those messages about nvme1 which is what made me suspect a bad ssd? I also see messages about multiple uncorrected fatal error received, frozen state error detected, and device recovery successful.

A second diagnostics download immediately after rebooting if that's of any use: lisa-diagnostics-20231124-1015.zip

I replaced my cache drive about a week ago and also added a second drive as a raid1 pool. Since then, approximately every 2 days the dockers and VMs lock up and trying to write to the cache drive returns a message that the file system is read only. I've tried running a balance and a scrub and the scrub returns no errors, yet the problem keeps recurring. The only way to bring it back to life is to reboot, but it soon happens again. What have I missed? Or could the new SSDs just be faulty? Diagnostics attached. lisa-diagnostics-20231124-0959.zip

Yes, sorta. No

Good point, I completely missed that people were using the LTS tag and not the latest tag

There's your problem When I said export and import I didn't mean backing up your app data folder. If you restore the old appears it will push you back to the old version. From unifi itself there is an export config option. Use this. Still take a backup of your appdata incase everything brakes as a fallback, but that's not the process for upgrading

Have a look on unifi's website for the update path from the version you're at to v5.13. you can't jump from 5.6 to 5.13 in one go. If memory serves there is at least one to go to before changing to the latest .13 tag. Alternatively, backup your config, bin the docker and fire up a new one with latest, before importing your old config

Ok so another update, now been running for over 3 days with the most current docker container, no ram cap and DPI turned off, and my ram is sitting just below 800MB used.

I haven't set a limit yet on mine but turning off DPI has worked well for me. There was still a memory leak somewhere but it's not as bad, I got to 950MB after 5 days. However, I noticed another update yesterday and it's at 705MB after 12 hours running (it started at 709mb) so curious to know what's changed as unifinis still reporting the same controller version (5.13.29) Edit: another hour later and the ram has fallen further to 682MB!?!

I haven't limited mine to 4GB yet, I'll set a cap and turn on DPI later today

Currently up to 765MB so it has grown but would usually be over 1.5GB by now with 31 hours up-time. I'm keeping the DPI off on mine as it's clearly helping but I think it's too soon to call the issue fixed for version 5.13.29 And I'm just using a user script: docker restart unifi

Yeah I get that. Longer term I'll set it higher, I was more meaning for testing to see what it does when it "runs out" of ram. It's not really a workable solution for me to put a cap on the memory of it's going to start causing networking issues and I'll stick with the daily Cron that restarts the docker instead. I'm hoping the DPI has fixed it though as memory has actually gone down in the last hour

I was thinking around 1.5GB as its always been around 1.3GB when I reset it every day

Thanks, I'll set one in the morning if it grows any further over night. I'll try setting it much lower just to see what it does

Nope, don't know what it would do when the caps reached. Also, don't know how to...

See if it stabilises around there? Mine grew quickly to start with but now it's fluctuating between 716 and 722 so I'm going to keep it like this for a few days and see what happens. Not good that it killed the webui though

Looks like it was added in 5.9. Mine shows beta in the unifi app. Not checked the web UI because I can never remember my password

That's what I'm wondering, is it a problem with the USG rather than the controller. I don't remember turning on DPI but I must have done, and being a Beta feature I wouldn't be surprised if this was having issues. Anyone know when DPI was added to unifi? Was it after 5.11?

I'm on 5.13.29, uptime of 13 hours so far, and it's gone from 660ish to now being 720ish MB used. I have also turned off the deep packet inspection (beta) to see if that's the cause. It's still growing but would be at 1.3GB after 24 hours so we'll see how we go.

Thanks, I'll look into what involved in changing back to .11. I'm not totally adverse to wiping my config as I want to make some big changes anyway. I'm on 5.12.72 at the mo

I've just noticed the same. I was wondering why I couldn't start a VM then noticed this docker was using 15GB ram. Applied the update that was pending and we're already back over a GB ram used in a minute. Any luck on the unifi pages?