seecs2011

Of course, as I submit, I make a small change and its up...carry on

Hello - in the process of updating hardware/troubleshooting another issue I lost my HomeAssistant VM. I tried modifying it and it just disappeared from my VM page in unraid. I have the VMDK file and cannot for the life of me get it to boot. I tried changing the bios to SeaBIOS per a different thread and that just hangs at booting from disk when I VNC to it. I was getting a screen similar to this before bios change in setup. Unfortunately, the backups for this HA were only being written within itself from what I can tell so I can' import a backup. Please help with getting a vmdk running again adn does anyone know why it just disappeared from the UI when trying to edit the vm in the UI?

Problem with that is that I replaced every component

Now it crashed with an Ubuntu live disk...what the hell? I can't figure out what is wrong with this thing

Something broke in this process - I reinstalled the old hardware and am getting the same behavior

Worth noting, safe mode also crashes, only stability I've had for more than a minute or two has been the memtest I've been running just to make sure the system can stay up apart from UnRAID OS for a bit...

Welp - I've now replaced all components and it is rebooting within a minute of the web portal becoming available now

Follow up here - if you have nvme's pooled for cache, is there any concern about just moving those over to a new board? Like, will UnRaid recognize that they are the same pool devices in the same way as the array drives? Will having the array drives on an HBA change anything? I'm currently doing troubleshooting replacements so curious if I should plan any extra contingencies in that case...

as previously mentioned, yes that was done.

Cool - well here is the latest. Replaced PSU - no dice RAM - no dice Docker turned off - finally made it through full parity check, crashed after parity was done. Checked bios - no issues to report - all config as it should be Disks are fine minus a random error on my original disks that just won't clear - the error id translates to 1 errors in 65538 operations. (its the 188 error that is ignored) I'm in the danger zone at this point - I can't finish parity checks most of the time and because it crashes when parity finishes or is cancelled, I cannot invoke the mover and disks are starting to get into an undesirable state. Please advise. EDIT: Crash occurs within roughly 7 min of parity finishing or being cancelled

Anyone want to give me a sanity check on this pre-boot stuff?

Welp. Got the new psu installed yesterday and the crashing actually got slightly worse if anything. Ruling that out... Noticing again that the writes to flash are ticking up... Can't figure out what is causing that though as the git logs are not showing anything being updated...

New update, I am noticing a lot of writes to flash drive again. I checked what files are being updated the most and they are under the .git/objects folder. Its adding 5-10 every few minutes. I had 300 writes an hour ago and am at 600 now. Can't find anything else changing. Not sure why git is updating like crazy...

Thinking more and more it is power related - just had it crash after a 16 hour uptime randomly. Thought I should cancel parity check since it keeps trying to do it on reboot and then it immediately crashed again which I missed. Just cancelled that parity check and am waiting to see. I tried to up the idle load via bios but yeah. PSU gets here Friday and I'm off that day so I know what I'll be working on...

lol - in like 20 chrome tabs! I do have a bootable that I could try and have a psu on its way. Something to try while I wait for sure, although as we speak I'm on my longest uptime in more than two weeks at 12 hours even

I have been leaning more and more that direction. I don't have a spare, unfortunately but am watching a bit to see over the next couple days if some of the changes I've done help. Based on the other things I've tested, I'm at its either PSU or UnRAID OS. You might have solidified my choice to grab a spare PSU to have around for spare anyway...

I've run extensive testing there and am about 95% sure it isn't RAM. Interestingly - I saw that the writes to my USB were kinda high, sometimes dozens of writes at a time. For example that pic was after like an hour of runtime. As stated in my previous reply I'm thinking there may be something to that based on the syslog around the crashes. This led me to another thread about a high number of writes and there talked about uninstalling unassigned devices preclear and myservers from plugins. I can't find myservers plugin though (even though that log file references it) I removed preclear as I had that installed and now my USB interactions look like this after 8-9 hours: That also means I have doubled my uptime average. It does seem that there was one overnight crash after around 9 or 10 hours instead of what had been 3-5... So we're getting somewhere, I think. Any other thoughts on root cause based on that information?

The segfault hasn't happened in the last calendar year though and there have been dozens of crashes in the last few weeks. Also not many crashes around those segfault messages. Some of the crashes seem to happen after several attempts of this Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update Any processes for hardware troubleshooting that doesn't involve rip and replace and not being able to return used items for the test? Or could that flash_backup be an issue?

Should note - syslog has been next to useless but here is the file - note - the first log of the reboot is always "Tower root: Delaying execution of fix common problems scan for 10 minutes" so you can see from those timestamps how often it crashes now syslog-192.168.1.124.log

Over the last few years I've had issues off and on with my unraid server crashing. At first it was a c-state issue that I fixed. Then a year later, my mobo re-enabled all that stuff somehow and I made changes to c-states again as well as how the machine delivered power at idle. Both times I got stability for a few months. A couple weeks ago, crashes started speeding up again. I found some time to check BIOS assuming that the changes got made magically again but they hadn't. I've run memtest for about 15 hours and no errors were found. What else should I check for root cause of crashes? I disconnected my UPS because I had heard that that could be buggy and cause random shutdowns, but this is a hard reboot. I am at the point now where it is happening every 3-5 hours give or take. Happens regardless of if parity check is happening (last time, it was consistently after parity finished). SMART data is all good except for a couple one off command errors but I checked seagate and they are not the issue (one of the 188 attribute codes). Thoughts at this point are, in order: PSU CPU MOBO Not sure what else it could be. Setup is: MOBO: Gigabyte Technology Co., Ltd. B550M DS3H , Version Default string American Megatrends International, LLC., Version F19 BIOS dated: Fri 22 Mar 2024 12:00:00 AM CDT CPU: AMD Ryzen 7 5700G with Radeon Graphics @ 3800 MHz RAM: 64 GB Gskill PSU: Corsair RM650x Diagnostic from yesterday attached. tower-diagnostics-20241028-0821.zip

I'm planning to check this weekend on that - I had disabled them before so not sure how they would have gotten re-enabled unless bios auto-updated (which I didn't realize could happen until my gaming pc did that a few months ago). But the behavior is certainly pointing that direction (again).

Interesting troubleshooting find - system seems to stay up as long as a parity check is happening. If I cancel the check, I get a crash withing roughly the next hour--2hours. If I leave it running, crashes occur in the same window after finishing (hence the daily-ish crashes - my parity check takes roughly 22-25 hours. Combined with the disk on sde being a common thread, I'm wondering if that disk is going bad contrary to smart data reports or if somehow idle states are creeping back in as an issue.

Since its an easy test, I may start with thermal re-application. I have a PSU tester and didn't see any red flags there. Of note, I had the dashboard pulled up last time it crashed and was able to see that all the reported information there looked normal.

Thanks for catching that, probably not a huge deal given its use-case, but I deleted that part of the post to be safe.

Well, I made it a little over 1 day and 5 hours this time. Really tired of this system. I'm out of ideas. Got a neat new error on this boot though May 8 22:38:30 TOWER root: error: /webGui/include/DashboardApps.php: wrong csrf_token Shows about 30 times in a row. Same context around the crash/reboot again as well May 8 19:12:01 TOWER sSMTP[29961]: Creating SSL connection to host May 8 19:12:02 TOWER sSMTP[29961]: SSL connection using TLS_AES_256_GCM_SHA384 May 8 19:12:02 TOWER sSMTP[29961]: Authorization failed (535 5.7.8 https://support.google.com/mail/?p=BadCredentials 6a1803df08f44-6a15f17963csm1182386d6.5 - gsmtp) May 8 19:27:44 TOWER emhttpd: spinning down /dev/sde May 8 19:27:49 TOWER emhttpd: read SMART /dev/sde May 8 19:43:50 TOWER emhttpd: spinning down /dev/sde May 8 19:43:56 TOWER emhttpd: read SMART /dev/sde May 8 19:59:57 TOWER emhttpd: spinning down /dev/sde May 8 20:00:03 TOWER emhttpd: read SMART /dev/sde May 8 20:16:04 TOWER emhttpd: spinning down /dev/sde May 8 20:16:10 TOWER emhttpd: read SMART /dev/sde May 8 20:32:11 TOWER emhttpd: spinning down /dev/sde May 8 20:32:16 TOWER emhttpd: read SMART /dev/sde May 8 20:48:18 TOWER emhttpd: spinning down /dev/sde May 8 20:48:20 TOWER emhttpd: read SMART /dev/sde May 8 21:04:24 TOWER emhttpd: spinning down /dev/sde May 8 21:04:30 TOWER emhttpd: read SMART /dev/sde May 8 21:20:31 TOWER emhttpd: spinning down /dev/sde May 8 21:20:36 TOWER emhttpd: read SMART /dev/sde May 8 21:36:37 TOWER emhttpd: spinning down /dev/sde May 8 21:36:43 TOWER emhttpd: read SMART /dev/sde May 8 21:52:44 TOWER emhttpd: spinning down /dev/sde May 8 21:52:49 TOWER emhttpd: read SMART /dev/sde May 8 22:08:51 TOWER emhttpd: spinning down /dev/sde May 8 22:08:56 TOWER emhttpd: read SMART /dev/sde May 8 22:24:57 TOWER emhttpd: spinning down /dev/sde May 8 22:25:03 TOWER emhttpd: read SMART /dev/sde May 8 22:31:14 TOWER root: Delaying execution of fix common problems scan for 10 minutes May 8 22:31:14 TOWER unassigned.devices: Mounting 'Auto Mount' Devices... May 8 22:31:14 TOWER emhttpd: Starting services... May 8 22:31:14 TOWER emhttpd: shcmd (75): /etc/rc.d/rc.samba restart May 8 22:31:14 TOWER wsdd2[9158]: 'Terminated' signal received. May 8 22:31:14 TOWER nmbd[9134]: [2024/05/08 22:31:14.344902, 0] ../../source3/nmbd/nmbd.c:59(terminate) May 8 22:31:14 TOWER nmbd[9134]: Got SIGTERM: going down... May 8 22:31:14 TOWER winbindd[9172]: [2024/05/08 22:31:14.344934, 0] ../../source3/winbindd/winbindd_dual.c:1950(winbindd_sig_term_handler) May 8 22:31:14 TOWER winbindd[9170]: [2024/05/08 22:31:14.344938, 0] ../../source3/winbindd/winbindd_dual.c:1950(winbindd_sig_term_handler) May 8 22:31:14 TOWER winbindd[9172]: Got sig[15] terminate (is_parent=0) May 8 22:31:14 TOWER winbindd[9170]: Got sig[15] terminate (is_parent=1) May 8 22:31:14 TOWER wsdd2[9158]: terminating.