February 3, 20251 yr Hello there, recently I have upgraded my hardware, reassigned disks3-5 to disks4-6, and added a new disk to pool as disk3. Here was my old config: Quote MB: Supermicro X9SCM-F | CPU: Intel® Xeon® E3-1230 v2 | RAM: Micron 4*8G 2RX8 PC3L-12800E My new parts are in the signature. After upgrading, I experience multiple random freezing, meaning that I cannot login to the webUI, SSH has no response (I typed the username (root) and pressed enter, it responded timout after 60s, without prompting for password). VM & dockers cannot be accessed. The only way I can do is to force shutdown, and turn it on again. After each restart, there will be a parity check kicks in, so far each time is successful with 0 error. I am using ipvlan for docker. Would anyone please help to check my diags and see any potential reason, appreciate it thanks 🙏 Edited February 9, 20251 yr by PzrrL removing diag
February 3, 20251 yr Community Expert Solution Unfortunately, there's nothing relevant logged, this can also be a hardware issue, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the individual docker containers. Additionally, it may be good to run memtest, also, look in the BIOS for a "Global C-States" or similar setting and disable that to retest, it's been known to be a problem with some boards, with both Intel and AMD CPUs.
February 4, 20251 yr Author Thanks @JorgeB for providing some directions. I have ran latest version of memtest86 over night snd it shows all passed. I am suspecting it is s C-state problem, as the parity checks take around 26 hours to complete and it frozen 2-3days after restarting. This seems matching the timeline of frozen. I have made the changes in BIOS as attached and see if it helps. If not, I will go with the route of running in safe mode. May I know does it matter to have safe mode with or without GUI? Quote if it doesn't start turning on the other services one by one, including the individual docker containers. Precisely do you mean booting as normal (non safe mode) and try to start dockers one by one to find the possible culprit? thanks!
February 4, 20251 yr Community Expert 3 hours ago, PzrrL said: May I know does it matter to have safe mode with or without GUI? Should be the same 3 hours ago, PzrrL said: Precisely do you mean booting as normal (non safe mode) and try to start dockers one by one to find the possible culprit? Yes, if it doesn't crash with them disabled.
February 9, 20251 yr Author looks like "Global C-States" in the BIOS is the culprit. My uptime is now 5 days 13 hours 31 minutes and still counting. However, I have run a parity check since restarting, not sure if there is any potential variables here.
February 11, 20251 yr Author It happened again seems c state is not the culprit. I notice the log file is very full like 90% before it died. The next time i tried to access the dashboard, the system is not responding already. Not sure if this is related.
February 12, 20251 yr Author Thanks, I have enabled the syslog, and it frozen again during parity check (died with 24 hours after restarting) Side note, the reason why the log file full was due to the flooding of the following log: Feb 8 12:57:30 UnCRATE nginx: 2025/02/08 12:57:30 [crit] 11115#11115: ngx_slab_alloc() failed: no memory Feb 8 12:57:30 UnCRATE nginx: 2025/02/08 12:57:30 [error] 11115#11115: shpool alloc failed Feb 8 12:57:30 UnCRATE nginx: 2025/02/08 12:57:30 [error] 11115#11115: nchan: Out of shared memory while allocating message of size 16131. Increase nchan_max_reserved_memory. Feb 8 12:57:30 UnCRATE nginx: 2025/02/08 12:57:30 [error] 11115#11115: *1571842 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost" Feb 8 12:57:30 UnCRATE nginx: 2025/02/08 12:57:30 [error] 11115#11115: MEMSTORE:00: can't create shared message for channel /devices I remember reading some posts and this was due to the self signed SSL certificate, and I have removed it from `/boot/config/ssl/certs`, not sure if it is stil flooding now, will observe. I believe the cert was there because I enabled "Use SSL/TLS" under `Management Access` before. Even disabling it had this log flooding. Back to the freezing topic, I remember I experienced freezing long time ago due to some kernel incompatibility issue in the docker image from binhex/arch-qbittorrentvpn, and had to stick with a specific version to avoid the error. I am using latest image now, but not sure if the freezing is related. I am currently still having c-state disabled in the BIOS Please kindly advise, and thank you for the help 🙏 Edited February 18, 20251 yr by PzrrL
February 12, 20251 yr Community Expert Log is full of spam making difficult to analyze, do you know the time code when it last crashed?
February 12, 20251 yr Author 6 hours ago, JorgeB said: Log is full of spam making difficult to analyze, do you know the time code when it last crashed? Sorry about that, here is the list of timestamps right after restart (so logs before each timestamp should be logs before freezing), and for each restart below, there is an email notification about "Unclean shutdown detected": Feb 3 23:43:29 Feb 4 12:04:17 Feb 11 23:44:35 Feb 12 15:14:29 It seems to me that for each restart, the first line of log would contain "UnCRATE cache_dirs: Arguments=".
February 12, 20251 yr Community Expert Nothing that I would consider really relevant logged, there are issues with a device before one of the crashes: Feb 11 17:58:37 UnCRATE kernel: I/O error, dev sdb, sector 3059892 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 Feb 11 18:44:28 UnCRATE kernel: usb 1-13.1: reset high-speed USB device number 7 using xhci_hcd Feb 11 18:45:22 UnCRATE kernel: sd 1:0:0:0: [sdb] Unaligned partial completion (resid=15104, sector_sz=512) Feb 11 18:45:48 UnCRATE kernel: sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 00 25 6a 84 00 00 20 00 Feb 11 18:46:05 UnCRATE kernel: sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Feb 11 18:46:31 UnCRATE kernel: sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 00 25 6a 84 00 00 20 00 Feb 11 18:46:58 UnCRATE kernel: I/O error, dev sdb, sector 2452100 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 2 And issues that suggest memory close to being exhausted before another one, at least in my experience: Feb 12 14:54:43 UnCRATE php-fpm[10812]: [WARNING] [pool www] child 18628 exited on signal 9 (SIGKILL) after 48.912090 seconds from start Feb 12 14:55:34 UnCRATE php-fpm[10812]: [WARNING] [pool www] child 18648 exited on signal 9 (SIGKILL) after 52.886776 seconds from start Feb 12 14:55:58 UnCRATE php-fpm[10812]: [WARNING] [pool www] child 18656 exited on signal 9 (SIGKILL) after 62.276534 seconds from start Feb 12 14:57:00 UnCRATE php-fpm[10812]: [WARNING] [pool www] child 18962 exited on signal 9 (SIGKILL) after 63.779147 seconds from start Feb 12 14:57:49 UnCRATE php-fpm[10812]: [WARNING] [pool www] child 18968 exited on signal 9 (SIGKILL) after 77.758153 seconds from start Either by itself should not crash the server, but they should still be resolved, other than that, my best advice is still the same I wrote above: On 2/3/2025 at 6:12 PM, JorgeB said: boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the individual docker containers.
February 12, 20251 yr Author Feb 11 17:58:37 UnCRATE kernel: I/O error, dev sdb, sector 3059892 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 Feb 11 18:44:28 UnCRATE kernel: usb 1-13.1: reset high-speed USB device number 7 using xhci_hcd Feb 11 18:45:22 UnCRATE kernel: sd 1:0:0:0: [sdb] Unaligned partial completion (resid=15104, sector_sz=512) Feb 11 18:45:48 UnCRATE kernel: sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 00 25 6a 84 00 00 20 00 Feb 11 18:46:05 UnCRATE kernel: sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Feb 11 18:46:31 UnCRATE kernel: sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 00 25 6a 84 00 00 20 00 Feb 11 18:46:58 UnCRATE kernel: I/O error, dev sdb, sector 2452100 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 2 This sdb is the USB thumbdrive, I guess it is dying and need replacement? 1 hour ago, JorgeB said: And issues that suggest memory close to being exhausted before another one, at least in my experience: Feb 12 14:54:43 UnCRATE php-fpm[10812]: [WARNING] [pool www] child 18628 exited on signal 9 (SIGKILL) after 48.912090 seconds from start Feb 12 14:55:34 UnCRATE php-fpm[10812]: [WARNING] [pool www] child 18648 exited on signal 9 (SIGKILL) after 52.886776 seconds from start Feb 12 14:55:58 UnCRATE php-fpm[10812]: [WARNING] [pool www] child 18656 exited on signal 9 (SIGKILL) after 62.276534 seconds from start Feb 12 14:57:00 UnCRATE php-fpm[10812]: [WARNING] [pool www] child 18962 exited on signal 9 (SIGKILL) after 63.779147 seconds from start Feb 12 14:57:49 UnCRATE php-fpm[10812]: [WARNING] [pool www] child 18968 exited on signal 9 (SIGKILL) after 77.758153 seconds from start Sorry I don't quite understand here (esp "before another one", which one are you referring to?), can you elaborate more on this please? 1 hour ago, JorgeB said: Either by itself should not crash the server, but they should still be resolved, other than that, my best advice is still the same I wrote above: saly a painful process, I will do it and see how, thanks for the advice 🙏 Edited February 12, 20251 yr by PzrrL add more description
February 13, 20251 yr Community Expert If sdb is the flash drive, you can try using a different USB port, if the issues continue, replace it. 12 hours ago, PzrrL said: Sorry I don't quite understand here (esp "before another one", which one are you referring to?), can you elaborate more on this please? Before another crash event, this one: 14 hours ago, PzrrL said: Feb 12 15:14:29
February 18, 20251 yr Author On 2/4/2025 at 2:12 AM, JorgeB said: one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, I have restarted and boot in safe mode. However, the array is stopped with the following in the status bar: Quote Array Stopped•Autostart disabled (safe mode specified) System running in safe mode To let it run as a basic NAS, should I start up the array? Sorry for asking dumb question, but if the array is not started i guess it is just a powered-on machine with zero functionalty lol
February 24, 20251 yr Author On 2/4/2025 at 2:12 AM, JorgeB said: if it doesn't start turning on the other services one by one, including the individual docker containers. Seems no hardware issue with safe mode enabled (only start array, no docker/vm turned on) as I didnt experience any freezing 6 days straight. Do I restart without safe mode first? Or just keep it in safe mode, then start enabling docker service, and start up one docker at a time? besides, I dont see much docs related to safe mode. What does it exactly differ from the normal mode? Thanks Edited February 24, 20251 yr by PzrrL
February 24, 20251 yr Community Expert Safe mode disabled the plugins, next step would be to try normal mode with services still disabled, or if you prefer, safe mode and services enabled.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.