Jambo

Twice today my unraid system has become unresponsive to the WebUI and SSH, with many docker containers going offline but some mysteriously living (Homeassistant thankfully!) Diags attached and the last lines before the last "crash" below: Jul 24 14:43:38 HomeServer kernel: eth0: renamed from veth6e7a151 Jul 24 14:43:38 HomeServer kernel: docker0: port 1(veth67d1762) entered blocking state Jul 24 14:43:38 HomeServer kernel: docker0: port 1(veth67d1762) entered forwarding state Jul 24 14:43:38 HomeServer kernel: docker0: port 1(veth67d1762) entered disabled state Jul 24 14:43:38 HomeServer kernel: veth6e7a151: renamed from eth0 Jul 24 14:43:38 HomeServer kernel: docker0: port 1(veth67d1762) entered disabled state Jul 24 14:43:38 HomeServer kernel: device veth67d1762 left promiscuous mode Jul 24 14:43:38 HomeServer kernel: docker0: port 1(veth67d1762) entered disabled state Jul 24 14:44:07 HomeServer kernel: overlayfs: upper fs does not support RENAME_WHITEOUT. Jul 24 14:44:07 HomeServer kernel: overlayfs: upper fs does not support RENAME_WHITEOUT. Jul 24 14:44:07 HomeServer kernel: overlayfs: upper fs does not support RENAME_WHITEOUT. Jul 24 14:44:07 HomeServer kernel: docker0: port 1(veth12c3dc0) entered blocking state Jul 24 14:44:07 HomeServer kernel: docker0: port 1(veth12c3dc0) entered disabled state Jul 24 14:44:07 HomeServer kernel: device veth12c3dc0 entered promiscuous mode Jul 24 14:44:08 HomeServer kernel: eth0: renamed from veth11600ae Jul 24 14:44:08 HomeServer kernel: docker0: port 1(veth12c3dc0) entered blocking state Jul 24 14:44:08 HomeServer kernel: docker0: port 1(veth12c3dc0) entered forwarding state Jul 24 14:44:08 HomeServer kernel: docker0: port 1(veth12c3dc0) entered disabled state Jul 24 14:44:08 HomeServer kernel: veth11600ae: renamed from eth0 Jul 24 14:44:08 HomeServer kernel: docker0: port 1(veth12c3dc0) entered disabled state Jul 24 14:44:08 HomeServer kernel: device veth12c3dc0 left promiscuous mode Jul 24 14:44:08 HomeServer kernel: docker0: port 1(veth12c3dc0) entered disabled state Jul 24 14:44:37 HomeServer kernel: overlayfs: upper fs does not support RENAME_WHITEOUT. Jul 24 14:44:37 HomeServer kernel: overlayfs: upper fs does not support RENAME_WHITEOUT. Jul 24 14:44:37 HomeServer kernel: overlayfs: upper fs does not support RENAME_WHITEOUT. Jul 24 14:44:37 HomeServer kernel: docker0: port 1(vethd39a47e) entered blocking state Jul 24 14:44:37 HomeServer kernel: docker0: port 1(vethd39a47e) entered disabled state Jul 24 14:44:37 HomeServer kernel: device vethd39a47e entered promiscuous mode Jul 24 14:44:38 HomeServer kernel: eth0: renamed from veth6181726 Jul 24 14:44:38 HomeServer kernel: docker0: port 1(vethd39a47e) entered blocking state Jul 24 14:44:38 HomeServer kernel: docker0: port 1(vethd39a47e) entered forwarding state Jul 24 14:44:38 HomeServer kernel: docker0: port 1(vethd39a47e) entered disabled state Jul 24 14:44:38 HomeServer kernel: veth6181726: renamed from eth0 Jul 24 14:44:38 HomeServer kernel: docker0: port 1(vethd39a47e) entered disabled state Jul 24 14:44:38 HomeServer kernel: device vethd39a47e left promiscuous mode Jul 24 14:44:38 HomeServer kernel: docker0: port 1(vethd39a47e) entered disabled state Jul 24 14:45:01 HomeServer kernel: sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=1s Jul 24 14:45:01 HomeServer kernel: sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 00 58 10 90 00 00 f0 00 Jul 24 14:45:01 HomeServer kernel: I/O error, dev sdb, sector 5771408 op 0x0:(READ) flags 0x84700 phys_seg 30 prio class 2 Jul 24 14:45:01 HomeServer kernel: usb 3-5: USB disconnect, device number 5 Jul 24 14:45:01 HomeServer kernel: sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Jul 24 14:45:01 HomeServer kernel: sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 00 58 11 80 00 00 10 00 Jul 24 14:45:01 HomeServer kernel: I/O error, dev sdb, sector 5771648 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 Jul 24 14:45:01 HomeServer kernel: device offline error, dev sdb, sector 6593224 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 Jul 24 14:45:01 HomeServer kernel: device offline error, dev sdb, sector 6593224 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 Jul 24 14:45:01 HomeServer kernel: I/O error, dev loop0, sector 129768 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 2 homeserver-diagnostics-20240724-1545.zip

So I've got it running and working again, Selectively and slowly putting the plugins back has narrowed it down to either: Dynamix SSD Trim intel-gpu-top No idea which but whatever. Can't test now as it's doing a 16tb disk clear Thanks mate - didn't see the option the first time I ventured in there!

I'm genuinely shocked that the answer is to start from scratch and rebuild to solve this issue. Also, the Unraid connect backup is not accessible if you don't have a working server? What's the point of it? So the key files to start with are: /config/super.dat /config/Plus.key /config/pools/ /config/plugins/dockerMan I'll also change the disk.cfg to prevent auto-start and just check the UI before continuing. I'm guessing after that, copy shares and all of plugins over? Would you go as far as copying over all the config after that?

Yep, that works. I assume it's no help but diags below just in case. How do I get to the unraid connect flash backup without the server GUI? Or do I have to find the issue in the config? tower-diagnostics-20230909-0229.zip

Thanks for the interest, new diags below. GUI Mode shows blank white screen on Unraid's own firefox. homeserver-diagnostics-20230908-2320.zip

Looking through the syslog I can see: Sep 8 11:29:16 HomeServer crond[1925]: exit status 255 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null ### [PREVIOUS LINE REPEATED 1 TIMES] ### Sep 8 11:30:32 HomeServer emhttpd: error: cmdStart: missing csrf_token Sep 8 11:31:16 HomeServer crond[1925]: exit status 255 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null Not sure if they're relevant?

homeserver-diagnostics-20230908-1132.zip Diagnostics attached. I set the "Autostart Array on Startup" to false so I could add a new drive and immediately the UI crashed and only responded with error 500. I restarted the system and on startup, the array did not start, and the UI is still responding error 500. I've tried the following commands to no avail: /etc/rc.d/rc.nginx reload /etc/rc.d/rc.php-fpm restart Any help? Logging in via the SAFE Mode with GUI, it also fails, with just a white screen showing in Firefox.

I had the same issue and also downgraded to a previous version. I had to "edit" each docker container, resave it (Delete one character and add it back to unlock save button) and the container worked again. Everything works fine after downgrading to 6.11.4. Also found 6.12.1 to be a nightmare.

I downgraded back to 6.11.4 and that seems to have solved it. No issues since

Just froze once again. Took out half the RAM and still froze, put the other half in and froze again. Both of those freezes were extremely fast too, happening within 3 hours of turning on. All four sticks of RAM seems to last the longest but certainly still failing. syslog attached showing last two power cycles, freeze happened at 13:14 local time, line 3551. syslog (1)

Still crashing after disabling C-States and on supported speeds for it's capacity and rank. Tried swapping out RAM and no improvement either.

Oh thanks, not seen that before. Will certainly disable C-States (although I think it's disabled already) and test lower speeds.

Thanks for the response - I just tried changing from macvlan to ipvlan as that had a mention of instability but no improvement as it froze again last night. The quote in the first post is from the preserved syslog, the line before the dash being the last line before freezing, the line after being the first line after starting up again. Diagnostics attached and starting filesystem check now. Log from last night's crash: Jul 6 14:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 6 14:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 Jul 6 15:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 6 15:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 Jul 6 16:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 6 16:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 Jul 6 17:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 6 17:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 Jul 6 18:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 6 18:00:02 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 Jul 6 19:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 6 19:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 Jul 6 20:00:02 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 6 20:00:02 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 Jul 6 21:00:02 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 6 21:00:02 HomeServer kernel: BTRFS info (device nvme0n1p1): relocating block group 3588542693376 flags data|raid1 Jul 6 21:00:03 HomeServer kernel: BTRFS info (device nvme0n1p1): found 23 extents, stage: move data extents Jul 6 21:00:03 HomeServer kernel: BTRFS info (device nvme0n1p1): found 20 extents, stage: update data pointers Jul 6 21:00:03 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 Jul 6 22:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 6 22:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 --------------------------------------------------------------------------------------------------------------------- Jul 6 22:32:43 HomeServer kernel: Linux version 6.1.34-Unraid (root@Develop) (gcc (GCC) 12.2.0, GNU ld version 2.40-slack151) #1 SMP PREEMPT_DYNAMIC Fri Jun 16 11:48:38 PDT 2023 Jul 6 22:32:43 HomeServer kernel: Command line: BOOT_IMAGE=/bzimage initrd=/bzroot i915.alpha_support=1 Jul 6 22:32:43 HomeServer kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' Jul 6 22:32:43 HomeServer kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' Jul 6 22:32:43 HomeServer kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' Jul 6 22:32:43 HomeServer kernel: x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers' Jul 6 22:32:43 HomeServer kernel: x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 Jul 6 22:32:43 HomeServer kernel: x86/fpu: xstate_offset[9]: 832, xstate_sizes[9]: 8 Jul 6 22:32:43 HomeServer kernel: x86/fpu: Enabled xstate features 0x207, context size is 840 bytes, using 'compacted' format. Jul 6 22:32:43 HomeServer kernel: signal: max sigframe size: 3376 Jul 6 22:32:43 HomeServer kernel: BIOS-provided physical RAM map: Jul 6 22:32:43 HomeServer kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009d3ff] usable Jul 6 22:32:43 HomeServer kernel: BIOS-e820: [mem 0x000000000009d400-0x000000000009ffff] reserved Jul 6 22:32:43 HomeServer kernel: BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved Jul 6 22:32:43 HomeServer kernel: BIOS-e820: [mem 0x0000000000100000-0x0000000009e01fff] usable Jul 6 22:32:43 HomeServer kernel: BIOS-e820: [mem 0x0000000009e02000-0x0000000009ffffff] reserved Again, nothing obvious homeserver-diagnostics-20230707-0926.zip

Hey all, Since updating to 6.12.1 on Monday 3rd, the server has been crashing. One on Monday night, about 12 hours after update. Once again on Tuesday night, and just a few hours ago on Wednesday. The latest one I caught on a persistent syslog but nothing obvious to my eye... Any ideas? Jul 5 14:10:14 HomeServer kernel: XFS (md2p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x182390d3e dinode Jul 5 14:10:14 HomeServer kernel: XFS (md2p1): Unmount and run xfs_repair Jul 5 14:10:14 HomeServer kernel: XFS (md2p1): First 128 bytes of corrupted metadata buffer: Jul 5 14:10:14 HomeServer kernel: 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00 IN.............. Jul 5 14:10:14 HomeServer kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ Jul 5 14:10:14 HomeServer kernel: 00000020: 64 9d bf 76 28 76 78 cd 64 9d bf 76 2d a5 f7 45 d..v(vx.d..v-..E Jul 5 14:10:14 HomeServer kernel: 00000030: 64 9d bf 76 2d a5 f7 45 00 00 00 00 00 a1 09 7d d..v-..E.......} Jul 5 14:10:14 HomeServer kernel: 00000040: 00 00 00 00 00 00 0a 11 00 00 00 00 00 00 00 01 ................ Jul 5 14:10:14 HomeServer kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 e2 4a c9 ef .............J.. Jul 5 14:10:14 HomeServer kernel: 00000060: ff ff ff ff 94 26 61 31 00 00 00 00 00 00 00 0a .....&a1........ Jul 5 14:10:14 HomeServer kernel: 00000070: 00 00 00 1a 00 1d cb b7 00 00 00 00 00 00 00 00 ................ Jul 5 14:10:49 HomeServer kernel: XFS (md2p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x182390d3e dinode Jul 5 14:10:49 HomeServer kernel: XFS (md2p1): Unmount and run xfs_repair Jul 5 14:10:49 HomeServer kernel: XFS (md2p1): First 128 bytes of corrupted metadata buffer: Jul 5 14:10:49 HomeServer kernel: 00000000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00 IN.............. Jul 5 14:10:49 HomeServer kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ Jul 5 14:10:49 HomeServer kernel: 00000020: 64 9d bf 76 28 76 78 cd 64 9d bf 76 2d a5 f7 45 d..v(vx.d..v-..E Jul 5 14:10:49 HomeServer kernel: 00000030: 64 9d bf 76 2d a5 f7 45 00 00 00 00 00 a1 09 7d d..v-..E.......} Jul 5 14:10:49 HomeServer kernel: 00000040: 00 00 00 00 00 00 0a 11 00 00 00 00 00 00 00 01 ................ Jul 5 14:10:49 HomeServer kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 e2 4a c9 ef .............J.. Jul 5 14:10:49 HomeServer kernel: 00000060: ff ff ff ff 94 26 61 31 00 00 00 00 00 00 00 0a .....&a1........ Jul 5 14:10:49 HomeServer kernel: 00000070: 00 00 00 1a 00 1d cb b7 00 00 00 00 00 00 00 00 ................ Jul 5 15:00:02 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 5 15:00:02 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 Jul 5 15:05:25 HomeServer nginx: 2023/07/05 15:05:25 [error] 10087#10087: *216480 open() "/usr/local/emhttp/_proxy_auth" failed (2: No such file or directory) while sending to client, client: 172.17.0.4, server: , request: "GET /_proxy_auth?acs_token=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..cC7cWBc3B4nEiqoPrbsR9w.rA5wILyQLA7Bu2EN2tKVGCHzbCxnwkQrajB-2Ol83Ao9T3FML9sS2_VNC-Cw-JET5hrZFslPe8ZAo_nhvI7oYG5kFe7pUXXLCNkNyb4O-qCmoDvx9oyt9TB5W7hzMnpGjgrBBWfpGD2bs78qHO_5WM-Sz6vlw7MsSTgpv6lpFrgR5Ql98sZDtjSu9sl4EKJ0.GJjxAfouN62G4Zd79X5bzQ HTTP/1.1", host: "unraid.james.am", referrer: "https://login.microsoftonline.com/" Jul 5 15:06:20 HomeServer flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update Jul 5 16:00:02 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 5 16:00:02 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 Jul 5 16:36:29 HomeServer webGUI: Successful login user root from 172.17.0.4 Jul 5 17:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 5 17:00:01 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 Jul 5 18:00:02 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: start -dusage=50 Jul 5 18:00:02 HomeServer kernel: BTRFS info (device nvme0n1p1): balance: ended with status: 0 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Jul 5 19:07:09 HomeServer kernel: Linux version 6.1.34-Unraid (root@Develop) (gcc (GCC) 12.2.0, GNU ld version 2.40-slack151) #1 SMP PREEMPT_DYNAMIC Fri Jun 16 11:48:38 PDT 2023 Jul 5 19:07:09 HomeServer kernel: Command line: BOOT_IMAGE=/bzimage initrd=/bzroot i915.alpha_support=1 Jul 5 19:07:09 HomeServer kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' Jul 5 19:07:09 HomeServer kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' Jul 5 19:07:09 HomeServer kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' Jul 5 19:07:09 HomeServer kernel: x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers' Jul 5 19:07:09 HomeServer kernel: x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 Jul 5 19:07:09 HomeServer kernel: x86/fpu: xstate_offset[9]: 832, xstate_sizes[9]: 8 Jul 5 19:07:09 HomeServer kernel: x86/fpu: Enabled xstate features 0x207, context size is 840 bytes, using 'compacted' format. Jul 5 19:07:09 HomeServer kernel: signal: max sigframe size: 3376 Jul 5 19:07:09 HomeServer kernel: BIOS-provided physical RAM map: Dashed line where lockup was noticed. Disk LED and power LED were still on, hard drives making activity noises but otherwise nothing happening. VMs, docker and webUI all failing to connect.

Seems to be disconnecting now every few seconds - Looking at the github issue it's affecting others: https://github.com/Darkside138/DiscordSoundboard/issues/173