shaunsund

Lost my USB drive Sunday sometime. System still was running, Plex worked but the /boot dir was non-existent. The old drive won't show up as a USB device and due to some misconfigurations with my backup system and the Flash backup, I only had 3 month-old backup of my config/ dir. Making lemonade out of lemons, I decided to start from scratch (my config dir had become a mess of old plugins and other detritus ) and throwing caution to the wind I went with a fresh install of 6.12.0-rc3 on a new USB drive. Working on bringing back some Dockers and then VMs, but currently I seem to be having some odd issues: The Apps tab (limited to Safari I think) fails to load all the CSS at times and has misinformed page items (large app logos, wrong fonts, etc). I also get "Did not parse stylesheet at <server address>/plugins/community.applications/skins/Narrow/css.php?v=1678545693 because non CSS MIME types are note allowed in strict mode." and an "avatar FailedToLoad error' on the browser console. I get past this by toggling the "Disable content blockers" setting for the page - until it happens again. Haven't figured out why or what causes it again. I also have seen times where I'l be on a Settings page, for example, and the UI will refresh and take me to the Main page. Got some random times where I can't pull images and the UI is slow to load. I could ping my router from unraid but couldn't ping 1.1.1.1. Yet other devices could ping 1.1.1.1 so it wasn't a loss of my network service. Then, randomly, I can pull images and things work. I doubt I am the first to have issues like these, but haven't found a reason or solution here with Google. the -2108 is the diag from when I lost the drive. -1330 is today after getting some things working. fractal-diagnostics-20230416-2108.zip fractal-diagnostics-20230417-1330.zip

I've wiped my app data several times, removed the images and re-downloaed several times, and even rebooted!. Still does it. You ask about Nerd Tools... I'm not sure how that would affect a container, but I do have libffi, libsodium, python3, vim, unrar, iftop, iotop and tmux innstalled.

Yes. 6.11.5

Cleared the cached docker config, redownloaded from the App Store, deleted the appdata folder for this container and tried again, but still got a segmentation fault.

I did find a dump log from MegaSync. It seems to crash after gathering a file list from Mega. 56886e1e-a417-80e0-2834909c-50631d7e.dmp

It's unusable. The container restarts.

Megasync client segfaults and the container restarts. Basically can't sync.

I have suddenly started to get `Error for "xrdb -query" command: "/bin/bash: line 1: xrdb: command not found\n" It will happen everytime it syncs a file list from mega. I have redownloaded and configured the docker several times.

A few scripts that run hourly, daily or weekly. Only 2 that run at the start of the array. I have disabled them both and restarted. Still have that strange output.

Here ya go! fractal-diagnostics-20230211-2131.zip

So I got an odd message coming up on my screen on every reboot lately: The lines starting with "sh:" are errors from what appears to be a disk report that somehow is running as a script upon a boot. I must have coped the output to a script or something dumb. Here's the odd thing; I can't find any file in /boot matching any of the terms. I've ran: find /boot -type f -exec grep -l "ATA8-ACS" {} \; find /boot -type f -exec grep -l "Firmware" {} \; The server is working fine. Other than this odd output every time it boots! find and grep seem to be failing me. Can anyone think of where to look or what to do? Thanks in advance.

I'll add my thoughts on the extended tests: I found the search for duplicate files handy - sometimes in the past a Move operation would do weird things. Would be nice to verify my FS.

As the attached diagnostics should show, every now and then, /var/log fills up and I have to reboot. I can't narrow it down to a certain docker image or plug-in. Normally, I couldn't catch it filling up until I made a user script to check the space. I did notice that the Dashboard won't show CPU usage and the log viewer page's close button is labeled 'undefined' I can change this script I made to grab certain info if anyone has some suggestions. fractal-diagnostics-20210413-1257.zip

I see your points. Still don't know why it did it.

well if a drive disappeared then the calculation for disk free would change -- although I would expect that to cause a warning before disk usage. I did check while writing the first post, every disk's utilization warning and critical are equal to the global. Also, even if they were different, why did they alarm for 2 disks and report ok for 6 disks a minute later? Like JorgeB mentioned it seems to be "an elusive issue and difficult to replicate"

Wow! I should go buy a lottery ticket.

So this is odd. I have 8 data disks and all except #1 are nearly full so I set my global warning at 93% and critical is 97% usage so that everything is OK utilization wise. Today was the second time I got the below messages: and also one for disk 8. Yet, a minute later I get the following: also for disk 3, 4, 5, 7, and 8 I got the Warning at 5:19pm and then the Notice emails at 5:20pm. Disks are practically idle during this time. I don't know why they would hiccup these utilization errors. If I had a disk or cache drive drop out then maybe the totals might change but there is no indication in syslog as to a cause for these warnings. fractal-diagnostics-20201104-1804.zip

Has anyone else been getting Out of memory errors when it comes to qbittorent? I've had the max memory set to 2G for the longest time but within the last few weeks I always get a OOM error from Unraid. Even upping the memory to 2.5G still gives OOM. Seems like there is now a memory leak. Oct 29 21:10:58 fractal kernel: Memory cgroup out of memory: Kill process 24455 (qbittorrent-nox) score 984 or sacrifice child

I've got 2 other drives on the same controller in addition to the SSDs. No problems with those. Why the other 2 drives didn't also complain is curious. Actually, I didn't get enough sleep. ~4hrs. Confused the Free space with the FS size. Smart reports are OK on each drive. btrfs reports OK. It must be the motherboard. Which, seeing how its out of warranty by 3 months seems even more likely.

Last night I noticed many errors from my cache drives: FS went RO, Docker not responsive and the logs started doing: Oct 12 23:46:00 fractal kernel: ata8.00: exception Emask 0x0 SAct 0xfffff03f SErr 0x0 action 0x6 frozen Oct 12 23:46:00 fractal kernel: ata8.00: failed command: WRITE FPDMA QUEUED Oct 12 23:46:00 fractal kernel: ata8.00: cmd 61/18:00:a8:16:ff/00:00:0b:00:00/40 tag 0 ncq dma 12288 out Oct 12 23:46:00 fractal kernel: res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Oct 12 23:46:00 fractal kernel: ata8.00: status: { DRDY } Oct 12 23:46:00 fractal kernel: ata8.00: failed command: WRITE FPDMA QUEUED Oct 12 23:46:00 fractal kernel: ata8.00: cmd 61/20:08:c0:16:ff/00:00:0b:00:00/40 tag 1 ncq dma 16384 out Oct 12 23:46:00 fractal kernel: res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) and then Oct 13 00:27:29 fractal rsyslogd: action 'action-3-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1908.0 try https://www.rsyslog.com/e/2027 ] Oct 13 00:27:29 fractal rsyslogd: file '/mnt/user/meta/syslog-10.10.10.10.log'[2] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: Read-only file system [v8.1908.0 try https://www.rsyslog.com/e/2027 ] Oct 13 00:27:29 fractal rsyslogd: action 'action-3-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1908.0 try https://www.rsyslog.com/e/2027 ] Oct 13 00:27:29 fractal kernel: scsi_io_completion_action: 127 callbacks suppressed Oct 13 00:27:29 fractal kernel: sd 7:0:0:0: [sdd] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Oct 13 00:27:29 fractal kernel: sd 7:0:0:0: [sdd] tag#5 CDB: opcode=0x28 28 00 03 61 18 e0 00 00 20 00 Oct 13 00:27:29 fractal kernel: print_req_error: 133 callbacks suppressed Oct 13 00:27:29 fractal kernel: print_req_error: I/O error, dev sdd, sector 56695008 Oct 13 00:27:29 fractal kernel: btrfs_dev_stat_print_on_error: 127 callbacks suppressed Oct 13 00:27:29 fractal kernel: BTRFS error (device dm-8): bdev /dev/mapper/sdd1 errs: wr 74, rd 10858, flush 0, corrupt 0, gen 0 Oct 13 00:27:29 fractal kernel: sd 8:0:0:0: [sde] tag#25 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Oct 13 00:27:29 fractal kernel: sd 8:0:0:0: [sde] tag#25 CDB: opcode=0x28 28 00 0e 20 18 e0 00 00 20 00 Oct 13 00:27:29 fractal kernel: print_req_error: I/O error, dev sde, sector 236984544 Oct 13 00:27:29 fractal kernel: BTRFS error (device dm-8): bdev /dev/mapper/sde1 errs: wr 417, rd 9685, flush 0, corrupt 0, gen 0 Oct 13 00:27:29 fractal kernel: sd 7:0:0:0: [sdd] tag#3 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 Oct 13 00:27:29 fractal kernel: sd 7:0:0:0: [sdd] tag#3 CDB: opcode=0x28 28 00 11 86 70 70 00 00 08 00 I was able to shutdown after grabbing diagnostics and went to bed. Woke up this morning to a working array but the cache drive was missing 173G. From 500G to 327G. Does anyone know what happened to my drives? I fear that I am a victim of the excessive cache write bug and it has killed my SSDs although I would have expected them just to die rather than loose space. I've included the diags from last night and this morning after a successful boot. I'm going to look into new SSDs (any suggestions?) My motherboard can also support 2x M.2 drives. Thanks! fractal-diagnostics-20201013-0027.zip fractal-diagnostics-20201013-0658.zip

I've had /var/log hit 100% 2x in the last week, Was able to get some error and syslog files off as, until I ran sudo /etc/rc.d/rc.nginx restart and then was able to pull diagnostics. In the logs I found thousands of line like: Aug 17 04:40:15 fractal nginx: 2020/08/17 04:40:15 [error] 31737#31737: *955988 nchan: error publishing message (HTTP status code 507), client: unix:, server: , request: "POST /pub/cpuload?buffer_length=1 HTTP/1.1", host: "localhost" Aug 17 04:40:15 fractal nginx: 2020/08/17 04:40:15 [crit] 31737#31737: ngx_slab_alloc() failed: no memory Aug 17 04:40:15 fractal nginx: 2020/08/17 04:40:15 [error] 31737#31737: shpool alloc failed Aug 17 04:40:15 fractal nginx: 2020/08/17 04:40:15 [error] 31737#31737: nchan: Out of shared memory while allocating channel /disks. Increase nchan_max_reserved_memory. Aug 17 04:40:15 fractal nginx: 2020/08/17 04:40:15 [error] 31737#31737: *955989 nchan: error publishing message (HTTP status code 507), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost" Aug 17 04:40:16 fractal nginx: 2020/08/17 04:40:16 [crit] 31737#31737: ngx_slab_alloc() failed: no memory Aug 17 04:40:16 fractal nginx: 2020/08/17 04:40:16 [error] 31737#31737: shpool alloc failed Aug 17 04:40:16 fractal nginx: 2020/08/17 04:40:16 [error] 31737#31737: nchan: Out of shared memory while allocating channel /cpuload. Increase nchan_max_reserved_memory. Aug 17 04:40:16 fractal nginx: 2020/08/17 04:40:16 [error] 31737#31737: *955990 nchan: error publishing message (HTTP status code 507), client: unix:, server: , request: "POST /pub/cpuload?buffer_length=1 HTTP/1.1", host: "localhost" Aug 17 04:40:16 fractal nginx: 2020/08/17 04:40:16 [crit] 31737#31737: ngx_slab_alloc() failed: no memory Aug 17 04:40:16 fractal nginx: 2020/08/17 04:40:16 [error] 31737#31737: shpool alloc failed Aug 17 04:40:16 fractal nginx: 2020/08/17 04:40:16 [error] 31737#31737: nchan: Out of shared memory while allocating channel /var. Increase nchan_max_reserved_memory. Aug 17 04:40:16 fractal nginx: 2020/08/17 04:40:16 [alert] 31737#31737: *955991 header already sent while keepalive, client: 10.10.10.14, server: 0.0.0.0:443 Aug 17 04:40:16 fractal kernel: nginx[31737]: segfault at 0 ip 0000000000000000 sp 00007ffea6b931f8 error 14 in nginx[400000+21000] Aug 17 04:40:16 fractal kernel: Code: Bad RIP value. Aug 17 04:40:16 fractal nginx: 2020/08/17 04:40:16 [alert] 17122#17122: worker process 31737 exited on signal 11 Another thing I noticed was that on the Dashboard page, the bars for the cores weren't showing activity. From what I have read, this can be from 'other' browsers. I use Opera and Chromium. fractal-diagnostics-20200817-1654-PLUS.zip

Loving this plugin, but did notice something odd: Image named '0' with wrong Icon for the image name. It can't be a orphaned image; I have a User Script that removes those.

Eureka moment! I have a raspberry pi I can use. Thanks!

I'll post this incase someone has experienced this or can come up with something. Was copying files over nfs to another unraid server. after about 4+ hours the GUI wasn't loading, my load in htop was high 30s. Was able to login via ssh and kill some dockers but the load never got better than 25. diagnostics command wasn't responsive. Had to power off the hard way. Brought it back up and of course Parity check starts, but did get a message on startup about one of my cache drives: Warning [FRACTAL] - Cache pool BTRFS missing device(s)Samsung_SSD_850_EVO_500GB_S3PTNB0JC12576E (sdh) But as it checks parity all seems to be working fine. I find it hard to troubleshoot instances when the system is unresponsive to a point where gathering diagnostics is impossible. Does anyone have tips to get some information or 'refresh' the GUI so that a sluggish system can be recovered without a power reset? The attached diag is after the reboot. fractal-smart-20200625-2333.zip