djvj Posted January 25, 2018 Share Posted January 25, 2018 I've been running 6.3.2 for almost a year w/o a reboot, then upgraded to 6.3.5 for about a month, also no problems. Just upgraded to 6.4.0 just over a day ago and now my docker tab is not responding when I click it. I see this in the log: Jan 25 00:37:26 Storinator kernel: r750:HIM_EVENT_DEVICE_TIMEOUT vd ffff880458459000 Jan 25 00:37:26 Storinator kernel: r750:[ ] Cdb [88, 0, 0, 0, 0, 0, 0,1f, ed,d8, 0, 0, 0, 8, 0, 0]. Jan 25 00:37:26 Storinator kernel: r750:[ ] H2D FIS(Slot:09): 00258227 401fedd8 00000000 00000008 Jan 25 00:37:28 Storinator kernel: r750:[01:00 P6] Asyn Notification Received Jan 25 00:37:28 Storinator kernel: r750:[01:00 P6] GSCR changed Jan 25 00:37:32 Storinator kernel: r750:[01:00 P6] Asyn Notification Received Jan 25 00:37:32 Storinator kernel: r750:[01:00 P6] GSCR changed Jan 25 00:37:36 Storinator kernel: r750:[01:00 P6] Asyn Notification Received Jan 25 00:37:36 Storinator kernel: r750:[01:00 P6] GSCR changed Jan 25 00:37:37 Storinator kernel: r750:[01:00 28] Start Soft Reset for 2/0 Jan 25 00:37:38 Storinator kernel: r750:[01:00 30] Start Soft Reset for 2/2 Jan 25 00:37:38 Storinator kernel: r750:[01:00 38] Start Soft Reset for 2/4 Jan 25 00:37:39 Storinator kernel: r750:[01:00 30] Start Soft Reset for 2/2 Jan 25 00:37:39 Storinator kernel: r750:[01:00 38] Start Soft Reset for 2/4 Jan 25 01:00:01 Storinator Plugin Auto Update: Checking for available plugin updates Jan 25 01:00:08 Storinator Plugin Auto Update: Community Applications Plugin Auto Update finished Jan 25 01:09:12 Storinator kernel: r750:[01:00 30] Device request(0) timeout. Jan 25 01:09:12 Storinator kernel: r750:HIM_EVENT_DEVICE_TIMEOUT vd ffff880458459000 Jan 25 01:09:12 Storinator kernel: r750:[ ] Cdb [88, 0, 0, 0, 0, 0, 0, 0, 8,60, 0, 0, 0,20, 0, 0]. Jan 25 01:09:12 Storinator kernel: r750:[ ] H2D FIS(Slot:00): 00258227 40000860 00000000 00000020 Jan 25 01:09:14 Storinator kernel: r750:[01:00 P6] Asyn Notification Received Jan 25 01:09:14 Storinator kernel: r750:[01:00 P6] GSCR changed Jan 25 01:09:18 Storinator kernel: r750:[01:00 P6] Asyn Notification Received Jan 25 01:09:18 Storinator kernel: r750:[01:00 P6] GSCR changed Jan 25 01:09:22 Storinator kernel: r750:[01:00 P6] Asyn Notification Received Jan 25 01:09:22 Storinator kernel: r750:[01:00 P6] GSCR changed Jan 25 01:09:23 Storinator kernel: r750:[01:00 28] Start Soft Reset for 2/0 Jan 25 01:09:24 Storinator kernel: r750:[01:00 30] Start Soft Reset for 2/2 Jan 25 01:09:24 Storinator kernel: r750:[01:00 38] Start Soft Reset for 2/4 Jan 25 01:09:25 Storinator kernel: r750:[01:00 30] Start Soft Reset for 2/2 Jan 25 01:09:25 Storinator kernel: r750:[01:00 38] Start Soft Reset for 2/4 Jan 25 01:10:52 Storinator nginx: 2018/01/25 01:10:52 [error] 9507#9507: *245694 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.1.145, server: , request: "GET /Docker HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.3", referrer: "http://192.168.1.3/Docker" Jan 25 01:13:56 Storinator kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO Jan 25 01:13:56 Storinator kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO Jan 25 01:13:56 Storinator kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO Jan 25 01:13:56 Storinator kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO Jan 25 01:13:56 Storinator kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO Jan 25 01:16:36 Storinator nginx: 2018/01/25 01:16:36 [error] 9507#9507: *246654 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.1.145, server: , request: "GET /Docker HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.3", referrer: "http://192.168.1.3/Main" I also noticed my krusader docker once wasn't loading complete directories, like only some folders/files showed up. Restarted that docker to fix. Then now my rutorrent docker was reporting disk full when there is 720GB free on the 3TB disk it has access to.... Some very funny things happening that never happened before. I wanted to restart it as well but now the docker tab won't load. Not sure what to do next. Link to comment
JorgeB Posted January 25, 2018 Share Posted January 25, 2018 Please post your diagnostics: Tools -> Diagnostics Link to comment
djvj Posted January 25, 2018 Author Share Posted January 25, 2018 Oops, forgot about that, here it goes storinator-diagnostics-20180125-0232.zip Link to comment
JorgeB Posted January 25, 2018 Share Posted January 25, 2018 You're getting constant out of memory errors, reboot and shutdown all docker and VMs, then start turning them on one by one and testing for a while to try and find if one of them is the culprit. Link to comment
djvj Posted January 25, 2018 Author Share Posted January 25, 2018 ok thanks I'll try that. Link to comment
djvj Posted January 25, 2018 Author Share Posted January 25, 2018 So I noticed I had a ton of postgres processes. I killed a few of them, then my dockers started responding and I was able to view the docker tab. I shutdown all the dockers one by one, gitlab-ce freed up 35% of my ram, the others were small chunks, not more than a few each. After all were done, I still had 55% ram used with nothing running, not even the docker service. Something was stuck eating up my memory. Rebooted and started at 6% memory usage. Enabled the array, and jumped to 30%. sabnzbd - 1% couchpotato - no change dropbox - 2% Leaving it here for now as I goto sleep, want to see if it creeps up over the next 8 hours. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.