daemian

Hey All, I tried doing some searching and was finding posts with similar behaviors, but none of the proposed solutions I found have worked. We had a power outage recently and it resulted in an unclean shutdown (either my UPS is not lasting long enough for the shutdown to complete or the automatic shutdown isnt working properly - i have to check on that after I get things back in working order). When i started the array back up it reported an unclean shutdown was detected (as epected). I started the array, and the parity check started. It was then that i noticed that only 1 of my user shares was listed on the shares page, and my Docker service would not start (presumably because of the shares issue as Settings > Docker is indicating that my vDisk location does not exist. Its on a cache-only share, which is missing). I let the parity check finish, and am now trying to stop the array (seems like a reboot was a common solution i found). But its getting stuck on "Retry unmounting user share(s). Log is spamming this over and over: Jul 15 08:18:17 ur01 emhttpd: shcmd (521060): rm -f /boot/config/plugins/dynamix/mover.cron Jul 15 08:18:17 ur01 emhttpd: shcmd (521061): /usr/local/sbin/update_cron Jul 15 08:18:17 ur01 emhttpd: Retry unmounting user share(s)... Jul 15 08:18:22 ur01 emhttpd: shcmd (521062): /usr/sbin/zfs unmount -a Jul 15 08:18:22 ur01 emhttpd: shcmd (521063): umount /mnt/user Jul 15 08:18:22 ur01 root: umount: /mnt/user: not mounted. Jul 15 08:18:22 ur01 emhttpd: shcmd (521063): exit status: 32 Jul 15 08:18:22 ur01 emhttpd: shcmd (521064): rmdir /mnt/user Jul 15 08:18:22 ur01 root: rmdir: failed to remove '/mnt/user': Directory not empty Jul 15 08:18:22 ur01 emhttpd: shcmd (521064): exit status: 1 lsof | grep /mnt/user comes back blank (nothing) ls /mnt/user shows a folder for the one share that was showing up ("icons") mount | grep /mnt/user also comes back blank I tried to collect a diagnostics, but the page did not finish loading. I found the folder it had created in the root folder, so i zipped it up and scp'ed it off (attached). How should i proceed? Thanks! ur01.marx.local-diagnostics-20240715-0838.zip

thanks as always JorgeB, I ran it w/ -L and here was the output Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_fdblocks 458152498, counted 459133020 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 inode 2584779900 - bad extent starting block number 4503567558284715, offset 0 correcting nextents for inode 2584779900 bad data fork in inode 2584779900 cleared inode 2584779900 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 1 - agno = 0 - agno = 3 - agno = 2 entry "4096-4096-max.png" in shortform directory 2584779899 references free inode 2584779900 junking entry "4096-4096-max.png" in directory inode 2584779899 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (15:874858) is ahead of log (1:2). Format log to cycle 18. done Out of curiosity i ran it again now w/ -n Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. Looks good to my eye. Started the array successfully now and things seem to be working 🤞 Appreciate the help as always, I will report back if i have more trouble.

So i have been having some issues w/ my unraid recently (crashing, likely becasue of macvlan). I *think* we may have gotten to the bottom of that. I started a new thread as I dont think this issue is related to that (though i could be wrong). I started my array yesterday, it was running through the parity check and all seemed to be well. Fast forward to this morning and suddenly in the main page I see: Unmountable disk present: Disk 2 • HGST_HDN726040ALE614_N8H1H2EZ (sde) This disk is not disabled (no red icon). I am doing my best to follow these instructions in the docs. I tried to stop the array, and it got stuck on syncing filesystem. Tried several things, eventually had to hard power it down. Brought it back up and started it in maintenance mode and ran xfs_repair on disk 2 with the -n flag and got: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being ignored because the -n option was used. Expect spurious inconsistencies which may be resolved by first mounting the filesystem to replay the log. - scan filesystem freespace and inode maps... sb_fdblocks 458152498, counted 459133020 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 inode 2584779900 - bad extent starting block number 4503567558284715, offset 0 correcting nextents for inode 2584779900 bad data fork in inode 2584779900 would have cleared inode 2584779900 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 entry "4096-4096-max.png" in shortform directory 2584779899 references free inode 2584779900 would have junked entry "4096-4096-max.png" in directory inode 2584779899 inode 2584779900 - bad extent starting block number 4503567558284715, offset 0 correcting nextents for inode 2584779900 bad data fork in inode 2584779900 would have cleared inode 2584779900 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... entry "4096-4096-max.png" in shortform directory inode 2584779899 points to free inode 2584779900 would junk entry - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. Then tried running it again without -n and got: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. Not sure if i should try the -L option. I googled and saw another post that asked about the smart report for the drive as it may be failing, so I am attaching that here as well. Looking for any advice on how to proceed. I don't know if this is related to the recent crashed, or purely coincidental. Perhaps one of the hard crashed caused the issue and it a took a bit to manifest - I don't know. Thank you for your help. Great appreciated! HGST_HDN726040ALE614_N8H1H2EZ-20240521-1119.txt

Thanks JorgeB, I followed the process to delete the docker image and recreate it, i am now seeing ipvlan in the output. This are now running again, ill report back as time goes by and we see if its stable or the problem persists. root@ur01:~# docker network ls NETWORK ID NAME DRIVER SCOPE ac212ad490b6 br0 ipvlan local 86ec9a039786 bridge bridge local db1ff75c54d7 host host local fbfefb6d9d64 none null local Thanks!

root@ur01:~# docker network ls NETWORK ID NAME DRIVER SCOPE 26b57be4530c br0 macvlan local 202e09e3a3ce bridge bridge local a30031a6206f host host local 91776a234133 none null local It does seems to still be using macvlan for br0. For reference, here is my settings page:

Thanks JorgeB, I made the chagne to ipvlan, saved, rebooted and started the array. A few hours it crashed again the same as before. I verified it its still set to ipvlan after bringing it back up. another syslog and diagnostics is attached. Thank you for your help! ur01.marx.local-diagnostics-20240518-2259.zip syslog

Good Afternoon, Have an Unraid install that has been going strong for many many years. Have run into issues a few times here and there, but typically its been related to a specific docker (most recently it was Frigate). It has been solid for 9 months or so, until this past Sunday night when it crashed without warning. I brought it back up, and its been crashing ever since. I had to re-add all of my dockers (the page ended up empty), which i never completely finished as its continued to have issues. I tried not running Frigate as that was the culprit in the past, but its still crashing. I also tried safemode, and it also crashed. I ram a memory test, and it passed. I was on 6.12.6 and at one point in this process decided to upgrade to 6.12.10.s Same issue with both Here is the diagnostics and last 2 syslog files (both from crashes). Any ideas what might be going on? Thanks in Advanced! syslog syslog-previous ur01.marx.local-diagnostics-20240517-1543.zip

Thanks for the assistance. I ran unraid in safe mode for several days without issues, so i rebooted into "regular" mode and on a hunch started everything but my Frigate docker. Low and behold it ran for a week without issue. Yesterday i started Frigate back up and it crashed in a few hours. So clearly its that docker causing the issue and I can try to get assistance with that specific application for root cause. Thinking about it though and I have a followup questions - is there anything i can do to prevent that docker from crashing the entire system? For instance, can I throttle its access to resources so it cannot choke everything out (at least thats what I assume is happening)? Or are there are prevention/protection measures i can put in place?

i did also switch from macvlan to ipvlan a while back (last time i had stability issues) and found it made a massive difference. I did double check that i am still set to ipvlan. Syslog is attached from the latest crash. Thanks, syslog

Hello, I have been running unraid for many years. I have had issues at times, but for the last 1-2 years its been extremely stable. Then, seemingly out of nowhere, I stated having stability issues. I cannot think of any notable changes. The server will run fine for a few hours, but then the dockers crash and the GUI goes unreachable. I am not able to connect via SSH either. If i hit the local console it responds initially, and I am able to enter a username, but then i never get to enter the password (it times out after 60 sec and goes back to the username prompt. After it started happening, I did try upgrading to 6.12.3, but that did not help, so i went back down to 16.12.1 Attached is a diagnostics taken after the most recent crash. Hoping someone can take a look and help me figure out what is causing this. Its driving me crazy! TIA! ur01.marx.local-diagnostics-20230812-2205.zip

Just wanted to chime in to say I am running into the same issue w/ my Raven scanner and unraid (which had been working for quite some time). Hoping you get a resolution from their support - let us know!

just to provide an update. I made the suggested change and so far its been almost 7 days without issue. I will post back if it occurs again - otherwise future readers can assume that did it. Thanks for your help Jorge!

ok so the server has crashed a few times since the last post. I did try downgrading back to 6.9.2 - but the problem persisted. Attached is the syslog file from the latest crash time frame (Aug 2 around 11:30am). TIA! syslog-192.168.166.2 (2).log

ok I have enabled the syslog server. Thanks for coaching me along here. I will report back when i have data after a crash.

Hi all, Hoping someone smarter than I can help me figure out whats going on. I have been running unraid for many years. I have been on the current hardware for over a year and its been very stable. Recently, I began experiencing crashes pretty regularly (uptime always less than 24 hours). This started after 2 changes: 1. I began to really use Frigate (docker). I had run a test setup w/o a TPU for a few weeks with just one camera for a few weeks without issue. However, once my Coral arrived I passed it through to my docker and added my 2 other cameras. 2. I did also upgrade to 6.10.3 at that same time. Now I am experiencing the crashes. It's almost certain the Frigate setup or update is to blame as that is what changed when I started having issues. Hoping someone can look at the diagnostics and help me figure out what the specific issue is. Thanks you in advanced ur01-diagnostics-20220722-1359.zip