May 2, 20251 yr So I'm not sure what is going on and I'm looking for ideas. Right this moment in time: tower.local or IP loads the login gui. Once I login, I get a 502 bad gateway nginx error screen. I'm streaming local plex stuff to the Apple TV with no issue. Ping responds SSH says "connection refused" (ssh root@<ip>) myconnect docker shows connect ECONNREFUSED /var/run/docker.sock I can connect to docker containers locally that are running (except the notifiarr one) externally I cannot connect to anyone via nginx reverse proxy It appears as though the parity check that has been running for about a day (happened last night as well and I dirty did it) is done as the HDD lights are no longer flickering. I don't want to do a dirty shutdown to grab the logs if I don't have to, but I think I'll have to anyway unless something clears itself. Any ideas?
May 2, 20251 yr Author I can’t SSH into the server or pull up the web interface. How else could I get them? EDIT: I’ll try with a monitor attached and see. Edited May 2, 20251 yr by DrDiggity
May 2, 20251 yr Author Bus error when I run diagnostics from CLI There is a steady php-fpm[7969]: [WARNING] [pool www] child 2122280 exited on signal 7 (SIGBUS) after 11.019540 seconds from start about 15 times in a row coming across the CLI and then crond[1832]: exit status 135 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null This just repeats over and over..
May 2, 20251 yr Author There is also this: crond [1832]: exit status 135 from user root /usr/local/emhttp/plugins/parity.check.tuning/parity.check.tuning.php "monitor" &>/dev/null Pressing the power button at the CLI gives me: sh INIT: switching to runlevel: 0 /etc/initscript: line 22: /etc/rc.d/rc.0: Input/output error /etc/initscript: line 22: /etc/rc.d/rc.0: Success And now I have no prompt anymore, but the same errors continue. I did enable the syslog server at some point so I am just going to bite the bullet and dirty dog this server and see what I can grab.
May 2, 20251 yr Author 1d7h of parity checking to go... Attached is the syslog file. syslog-192.168.13.227.log
May 2, 20251 yr Community Expert Nothing jumps out in the log other than some USB errors, was this when it crashed? May 1 20:13:37 Tower kernel: usb 1-3: reset high-speed USB device number 3 using xhci_hcd May 1 20:13:43 Tower kernel: usb 1-3: device descriptor read/64, error -110 May 1 20:13:59 Tower kernel: usb 1-3: device descriptor read/64, error -110 May 1 20:13:59 Tower kernel: usb 1-3: reset high-speed USB device number 3 using xhci_hcd May 1 20:14:04 Tower kernel: usb 1-3: device descriptor read/64, error -110 May 1 20:14:20 Tower kernel: usb 1-3: device descriptor read/64, error -110 May 1 20:14:20 Tower kernel: usb 1-3: reset high-speed USB device number 3 using xhci_hcd May 1 20:14:26 Tower kernel: usb 1-3: device descriptor read/8, error -110 May 1 20:14:41 Tower kernel: usb 1-3: device descriptor read/8, error -110 May 1 20:14:41 Tower kernel: usb 1-3: reset high-speed USB device number 3 using xhci_hcd May 1 20:14:47 Tower kernel: usb 1-3: device descriptor read/8, error -110 May 1 20:15:02 Tower kernel: usb 1-3: device descriptor read/8, error -110
May 2, 20251 yr Author I'd guess about then is when I lost web GUI and SSH access. I couldn't say for sure though. Could that be a bad USB drive?
May 2, 20251 yr Community Expert If that device was the flash drive, I would expect to see other errors logged, post the output from lsusb -vt
May 2, 20251 yr Author /: Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/12p, 480M ID 1d6b:0002 Linux Foundation 2.0 root hub |__ Port 001: Dev 002, If 0, Class=Human Interface Device, Driver=usbhid, 12M ID 0764:0501 Cyber Power System, Inc. CP1500 AVR UPS |__ Port 003: Dev 003, If 0, Class=Mass Storage, Driver=usb-storage, 480M ID 0781:5571 SanDisk Corp. Cruzer Fit /: Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/4p, 10000M ID 1d6b:0003 Linux Foundation 3.0 root hub
May 2, 20251 yr Author I just tried to create a diagnostics as things started slowing down and GUI would not refresh. Starting diagnostics collection... sh: line 1: /usr/bin/todos: Input/output error tail: cannot open '/boot/bzfirmware.sha256' for reading: No such file or directory tail: cannot open '/boot/bzimage.sha256' for reading: No such file or directory tail: cannot open '/boot/bzmodules.sha256' for reading: No such file or directory tail: cannot open '/boot/bzroot-gui.sha256' for reading: No such file or directory tail: cannot open '/boot/bzroot.sha256' for reading: No such file or directory timeout: failed to run command ‘uptime’: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error grep: write error: Broken pipe sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sort: fflush failed: 'standard output': Broken pipe sort: write error timeout: failed to run command ‘todos’: Input/output error timeout: failed to run command ‘echo’: Input/output error sh: line 3: /usr/bin/todos: Input/output error ls: write error: Broken pipe timeout: failed to run command ‘echo’: Input/output error sh: line 3: /usr/bin/todos: Input/output error ls: write error: Broken pipe timeout: failed to run command ‘echo’: Input/output error timeout: failed to run command ‘echo’: Input/output error timeout: failed to run command ‘echo’: Input/output error sh: line 3: /usr/bin/todos: Input/output error ls: write error: Broken pipe timeout: failed to run command ‘echo’: Input/output error sh: line 3: /usr/bin/todos: Input/output error ls: write error: Broken pipe timeout: failed to run command ‘echo’: Input/output error timeout: failed to run command ‘echo’: Input/output error sh: line 3: /usr/bin/todos: Input/output error ls: write error: Broken pipe timeout: failed to run command ‘echo’: Input/output error sh: line 3: /usr/bin/todos: Input/output error ls: write error: Broken pipe timeout: failed to run command ‘echo’: Input/output error sh: line 3: /usr/bin/todos: Input/output error ls: write error: Broken pipe sed: can't read /tower-diagnostics-20250502-1051/config/go.txt: No such file or directory xargs: echo: Input/output error timeout: failed to run command ‘todos’: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error timeout: failed to run command ‘echo’: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error timeout: failed to run command ‘todos’: Input/output error awk: fatal: cannot open source file `inplace' for reading: Input/output error timeout: failed to run command ‘todos’: Input/output error timeout: failed to run command ‘todos’: Input/output error timeout: failed to run command ‘todos’: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error sh: line 1: /usr/bin/todos: Input/output error done. ZIP file '/boot/logs/tower-diagnostics-20250502-1051.zip' created. Once it completed the GUI refreshed with an error at the top that said the flash drive was corrupted. lsusb -vt /: Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/12p, 480M ID 1d6b:0002 Linux Foundation 2.0 root hub |__ Port 001: Dev 002, If 0, Class=Human Interface Device, Driver=usbhid, 12M ID 0764:0501 Cyber Power System, Inc. CP1500 AVR UPS /: Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/4p, 10000M ID 1d6b:0003 Linux Foundation 3.0 root hub
May 3, 20251 yr Author New drive is installed and up and running. So far so good. normally within 24 hours things are not working so I'll see how things play out and let others know. Hopefully the issue was the USB drive and this can be closed. After the weekend is over I'll run the parity check again to make sure we are good and go from there.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.