stultus

6.9.1 was beautifully stable with 6+ month uptimes for me. I upgraded to 6.10 around July 4th, got a lot of PCIe errors, and so moved to 6.10.3 to solve that known bug. Ever since then my system dies every week or too. This is the last thing I see on the monitor, always related to nf_conntrack which appears to be a docker/network issue. Last weekend I changed from macvlan to ipvlan but that didn't solve my issue. Any ideas where to start poking around? Aug 3 01:54:07 Sophos kernel: md: recovery thread: check P Q ... Aug 3 01:54:56 Sophos flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update Aug 3 02:25:19 Sophos kernel: ------------[ cut here ]------------ Aug 3 02:25:19 Sophos kernel: WARNING: CPU: 1 PID: 14065 at net/netfilter/nf_conntrack_core.c:1192 __nf_conntrack_confirm+0xb8/0x254 [nf_conntrack] Aug 3 02:25:19 Sophos kernel: Modules linked in: tun nvidia_modeset(PO) nvidia_uvm(PO) veth macvlan xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xfs md_mod nvidia(PO) drm backlight nct6775 hwmon_vid efivarfs iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables btusb btrtl btbcm edac_mce_amd kvm_amd kvm crct10dif_pclmul wmi_bmof mxm_wmi btintel crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd bluetooth igb rapl input_leds i2c_piix4 i2c_algo_bit ccp ecdh_generic hid_lg_g15 k10temp i2c_core ahci led_class ecc libahci wmi button acpi_cpufreq Aug 3 02:25:19 Sophos kernel: CPU: 1 PID: 14065 Comm: kworker/1:1 Tainted: P O 5.15.46-Unraid #1 Aug 3 02:25:19 Sophos kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X370 Taichi, BIOS P6.40 07/28/2020 Aug 3 02:25:19 Sophos kernel: Workqueue: events macvlan_process_broadcast [macvlan] Aug 3 02:25:19 Sophos kernel: RIP: 0010:__nf_conntrack_confirm+0xb8/0x254 [nf_conntrack] Aug 3 02:25:19 Sophos kernel: Code: 89 c6 48 89 44 24 18 e8 53 e4 ff ff 44 89 f2 44 89 ef 89 c6 89 c5 e8 2c e8 ff ff 84 c0 75 9f 49 8b 87 80 00 00 00 a8 08 74 19 <0f> 0b 89 ee 44 89 ef 45 31 e4 e8 dc df ff ff e8 ae e4 ff ff e9 71 Aug 3 02:25:19 Sophos kernel: RSP: 0018:ffffc900001dcdb0 EFLAGS: 00010202 Aug 3 02:25:19 Sophos kernel: RAX: 0000000000000188 RBX: ffffffff828e1500 RCX: 0000000000000000 Aug 3 02:25:19 Sophos kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa02c1648 Aug 3 02:25:19 Sophos kernel: RBP: 000000000000fca9 R08: 3705f11d51bb3b46 R09: 9409ee1e63995df3 Aug 3 02:25:19 Sophos kernel: R10: aef9ca04f70f31cc R11: 8711e21c188c2656 R12: 0000000000005102 Aug 3 02:25:19 Sophos kernel: R13: 0000000000005102 R14: 0000000000000000 R15: ffff88814e840dc0 Aug 3 02:25:19 Sophos kernel: FS: 0000000000000000(0000) GS:ffff8887fe840000(0000) knlGS:0000000000000000 Aug 3 02:25:19 Sophos kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 3 02:25:19 Sophos kernel: CR2: 000000c00014d000 CR3: 0000000148b04000 CR4: 0000000000350ee0 Aug 3 02:25:19 Sophos kernel: Call Trace: Aug 3 02:25:19 Sophos kernel: <IRQ> Aug 3 02:25:19 Sophos kernel: nf_conntrack_confirm+0x26/0x3f [nf_conntrack] Aug 3 02:25:19 Sophos kernel: nf_hook_slow+0x3e/0x93 Aug 3 02:25:19 Sophos kernel: ? ip_protocol_deliver_rcu+0x135/0x135 Aug 3 02:25:19 Sophos kernel: NF_HOOK.constprop.0+0x73/0xce Aug 3 02:25:19 Sophos kernel: ? ip_protocol_deliver_rcu+0x135/0x135 Aug 3 02:25:19 Sophos kernel: __netif_receive_skb_one_core+0x79/0x9a Aug 3 02:25:19 Sophos kernel: process_backlog+0xab/0x143 Aug 3 02:25:19 Sophos kernel: __napi_poll.constprop.0+0x2a/0x114 Aug 3 02:25:19 Sophos kernel: net_rx_action+0xe8/0x1f2 Aug 3 02:25:19 Sophos kernel: __do_softirq+0xef/0x218 Aug 3 02:25:19 Sophos kernel: do_softirq+0x50/0x68 Aug 3 02:25:19 Sophos kernel: </IRQ> Aug 3 02:25:19 Sophos kernel: <TASK> Aug 3 02:25:19 Sophos kernel: netif_rx_ni+0x53/0x85 Aug 3 02:25:19 Sophos kernel: macvlan_broadcast+0x116/0x144 [macvlan] Aug 3 02:25:19 Sophos kernel: macvlan_process_broadcast+0xc7/0x110 [macvlan] Aug 3 02:25:19 Sophos kernel: process_one_work+0x198/0x27a Aug 3 02:25:19 Sophos kernel: worker_thread+0x19c/0x240 Aug 3 02:25:19 Sophos kernel: ? rescuer_thread+0x28b/0x28b Aug 3 02:25:19 Sophos kernel: kthread+0xde/0xe3 Aug 3 02:25:19 Sophos kernel: ? set_kthread_struct+0x32/0x32 Aug 3 02:25:19 Sophos kernel: ret_from_fork+0x22/0x30 Aug 3 02:25:19 Sophos kernel: </TASK> Aug 3 02:25:19 Sophos kernel: ---[ end trace f391be142f2ac721 ]--- sophos-diagnostics-20220812-1638.zip

I uninstalled plugin and then reinstalled which forced me to remap path. Now I'm backing up /mnt/user/appdata and it's humming along again!

I've just noticed that this plugin stopped my weekly backups approx 2 months ago. Any suggestions -- I'm running 6.9.1 and was just verifying my backup status before upgrading to 10.x. Running it manually displays the below. Edit: It hasn't been backing up since May 2021. Ugh. Backup / Restore Status: Not Running Backing Up appData from /mnt/cache/appdata/ to /mnt/user/backups/appdatabackup/[email protected] Executing tar: /usr/bin/rsync -avXHq --delete --log-file="/var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log" "/mnt/user/system/libvirt/libvirt.img" "/mnt/user/backups/libvertbackup/" > /dev/null 2>&1 2022/07/03 07:23:06 [10626] building file list 2022/07/03 07:23:06 [10626] sent 75 bytes received 19 bytes 188.00 bytes/sec 2022/07/03 07:23:06 [10626] total size is 1,073,741,824 speedup is 11,422,785.36 Verifying Backup Using command: cd '/mnt/cache/appdata/' && /usr/bin/tar --diff -C '/mnt/cache/appdata/' -af '/mnt/user/backups/appdatabackup/[email protected]/CA_backup.tar.gz' > /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log & echo $! > /tmp/ca.backup2/tempFiles/verifyInProgress Searching for updates to docker applications Backup/Restore Complete. tar Return Value: 0 Backup / Restore Completed

Thanks for the response. I have managed to forget my AP and USG login/passwords and was rather hoping not to rebuild everything 😞

I'm trying to modernize my install due to the recent vulnerability. Previously my approach was: if it works, don't break it. I'm pulling linuxserver/unifi-controller:LTS and the controller software is version 5.6.42 (Build: atag_5.6.42_10376). This seems very old? I can't upgrade my USG/APs because, presumably, the controller software predates the firmware releases. They are running firmwares circa summer 2019. Can anyone confirm if the older controller version is my problem, and confirm if I need to modernize beyond long term stable to patch this vulnerability? (It looks like 5.14.23 is recommended and the method is So, to summarize, to those who wish to move from LTS (5.6.42) to 5.14.23-ls76, you must first use 5.10.24-ls21 to upgrade the database.)

Thanks, that was the 2nd error that happened last night - a first ever SSD drive dropping off. A reboot brought it right back up. I'm running an extended SMART test on the spinning drive I originally posted about. That'll take a full day.

I'm having the same drive flake out in the middle of the night, once every couple months. Yesterday I reseated the SATA cabling on drive and motherboard, and it died again in the night. Usually I just reboot and rebuild from parity since it's a newer drive. Can someone take a peek at the attached logs and give me any indication on the cause/solution? TIA Some snippets: Jun 21 00:00:51 Sophos kernel: ata9.00: exception Emask 0x0 SAct 0xffffffff SErr 0x0 action 0x6 frozen Jun 21 00:00:51 Sophos kernel: ata9.00: failed command: WRITE FPDMA QUEUED Jun 21 00:00:51 Sophos kernel: ata9.00: cmd 61/20:00:20:3c:a4/00:00:00:00:00/40 tag 0 ncq dma 16384 out Jun 21 00:00:51 Sophos kernel: res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Jun 21 00:00:51 Sophos kernel: ata9.00: status: { DRDY } Jun 21 00:00:51 Sophos kernel: ata9.00: failed command: WRITE FPDMA QUEUED Jun 21 00:00:51 Sophos kernel: ata9.00: cmd 61/80:08:20:82:a4/00:00:00:00:00/40 tag 1 ncq dma 65536 out Jun 21 00:00:51 Sophos kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jun 21 00:01:30 Sophos kernel: ata9: hard resetting link Jun 21 00:01:30 Sophos kernel: ata9: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Jun 21 00:01:30 Sophos kernel: ata9.00: supports DRM functions and may not be fully accessible Jun 21 00:01:45 Sophos kernel: ata9.00: qc timeout (cmd 0xef) Jun 21 00:01:45 Sophos kernel: ata9.00: failed to set xfermode (err_mask=0x4) Jun 21 00:01:45 Sophos kernel: ata9: hard resetting link Jun 21 00:01:55 Sophos kernel: ata9: softreset failed (1st FIS failed) Jun 21 00:01:55 Sophos kernel: ata9: hard resetting link Jun 21 00:02:05 Sophos kernel: ata9: softreset failed (1st FIS failed) Jun 21 00:02:05 Sophos kernel: ata9: hard resetting link Jun 21 00:02:40 Sophos kernel: ata9: softreset failed (1st FIS failed) Jun 21 00:02:40 Sophos kernel: ata9: limiting SATA link speed to 3.0 Gbps Jun 21 00:02:40 Sophos kernel: ata9: hard resetting link Jun 21 00:02:45 Sophos kernel: ata9: softreset failed (1st FIS failed) Jun 21 00:02:45 Sophos kernel: ata9: reset failed, giving up Jun 21 00:02:45 Sophos kernel: ata9.00: disabled Jun 21 00:02:45 Sophos kernel: ata9: EH complete Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#12 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=149s Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#18 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#18 CDB: opcode=0x2a 2a 00 0b e0 15 10 00 00 30 00 Jun 21 00:02:45 Sophos kernel: blk_update_request: I/O error, dev sdj, sector 199234832 op 0x1:(WRITE) flags 0x8800 phys_seg 6 prio class 0 Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#12 CDB: opcode=0x93 93 08 00 00 00 00 00 00 30 00 00 00 00 20 00 00 Jun 21 00:02:45 Sophos kernel: blk_update_request: I/O error, dev sdj, sector 12288 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 0 Jun 21 00:02:45 Sophos kernel: BTRFS error (device sdj1): bdev /dev/sdj1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 Jun 21 00:02:45 Sophos kernel: BTRFS warning (device sdj1): direct IO failed ino 265 rw 1,34817 sector 0xbe01528 len 0 err no 10 Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=0x00 Jun 21 00:02:45 Sophos kernel: BTRFS error (device sdj1): bdev /dev/sdj1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0 Jun 21 00:02:45 Sophos kernel: BTRFS warning (device sdj1): direct IO failed ino 265 rw 1,34817 sector 0xbe01540 len 0 err no 10 Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] Sense not available. Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#19 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#19 CDB: opcode=0x2a 2a 00 0b e0 14 88 00 00 08 00 Jun 21 00:02:45 Sophos kernel: blk_update_request: I/O error, dev sdj, sector 199234696 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 Jun 21 00:02:45 Sophos kernel: BTRFS error (device sdj1): bdev /dev/sdj1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0 Jun 21 00:02:45 Sophos kernel: BTRFS warning (device sdj1): direct IO failed ino 265 rw 1,34817 sector 0xbe01490 len 0 err no 10 Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#20 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=106s Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#20 CDB: opcode=0x2a 2a 00 0b e0 14 70 00 00 18 00 Jun 21 00:02:45 Sophos kernel: BTRFS warning (device sdj1): direct IO failed ino 265 rw 1,34817 sector 0xdfbdfc0 len 0 err no 10 Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#18 access beyond end of device Jun 21 00:02:45 Sophos kernel: BTRFS warning (device sdj1): direct IO failed ino 265 rw 1,34817 sector 0xdfbdfe0 len 0 err no 10 Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#19 access beyond end of device Jun 21 00:02:45 Sophos kernel: BTRFS warning (device sdj1): direct IO failed ino 265 rw 1,34817 sector 0xdfbe000 len 0 err no 10 sophos-diagnostics-20210621-0733 (1).zip

Happy birthday!

Still works in 2019.

Thanks so much for spending the time to respond. I have tried to troubleshoot the docker image sizing previously but did not identify any dockers taking up huge amounts of space. It seems to be stable at its current size. Since my out of memory errors seemed to be concomitant with the “find problems” plugin, and since these errors were related to the Unraid version not being up to date, I said a quick prayer and updated to 6.6.6 last night. Everything seems to be fine, and I did not get an error message/out of memory email last night. If I get more memory errors in the future, I’ll post back in this thread. Again, thanks!

Just got this error two nights ago. Got the same error last night. Any guidance? It looks like some erros are due to me not upgrading from 6.5.3, but I don't know if that's mandatory. sophos-diagnostics-20190117-1726.zip

Also having issues with non-functional graphs here. I'm running pihole on the server IP address on default ports, if that helps. (I can run the old alpine version with no problems, beyond that version being out of date!)

I had an issue but solved it by moving to port 80. Wish I could delete this post

Works! Thank you!

What I meant was, do you recommend a newer version of unraid for new users?