July 2, 20179 yr Community Expert Hello, My webUI tower/Main is not responding and neither are my SMB shares. But I can access the server via terminal and via unMenu - in the unmenu log I cannot see any errors or issues reported below are the last few lines of the log which is related to unmenu I believe (8080) - this is not there in the earlier part of the log only after unmenu was started. This has happened a few times in the last few weeks and I had to hard reboot the server to get access to the shares again. All my docker apps and plugins seems to be running well and can be accessed via the WebGUI but looks like I can't play stuff on Plex. I have 16GB of RAM installed so it can't be running out of memory, I have uninstalled most application that were hogging the memory like Crashplan. Can anyone help me understand why this is happening please and how to fix it with out a hard reboot/prevent it from happening again? Jul 2 10:39:27 Tower kernel: TCP: request_sock_TCP: Possible SYN flooding on port 8080. Sending cookies. Check SNMP counters. Jul 2 10:39:35 Tower kernel: TCP: request_sock_TCP: Possible SYN flooding on port 8080. Sending cookies. Check SNMP counters. Jul 2 10:54:47 Tower kernel: TCP: request_sock_TCP: Possible SYN flooding on port 8080. Sending cookies. Check SNMP counters. Jul 2 10:57:06 Tower kernel: TCP: request_sock_TCP: Possible SYN flooding on port 8080. Sending cookies. Check SNMP counters. Thanks.
July 2, 20179 yr Community Expert You might want to read this post (found by googling): https://access.redhat.com/solutions/30453 If seems to indicate that there is a possibility that you have exposed your server on WAN (The Internet). If this is not the case, something else is going on inside of your LAN network. You might want to shutdown the Plex clients or anything else that could be attempting to communicate with your server. Also, post up your diagnostics file. 'Tools' >>> 'Diagnostics' with your next post.
July 3, 20179 yr Author Community Expert 5 hours ago, Frank1940 said: You might want to read this post (found by googling): https://access.redhat.com/solutions/30453 If seems to indicate that there is a possibility that you have exposed your server on WAN (The Internet). If this is not the case, something else is going on inside of your LAN network. You might want to shutdown the Plex clients or anything else that could be attempting to communicate with your server. Also, post up your diagnostics file. 'Tools' >>> 'Diagnostics' with your next post. Thanks Frank! I had googled SYN Flooding and read that before posting here - but the funny thing is that the SYN flooding does not seem to be happening at any point before I started uu - so I'm not sure if it is some malicious attack causing it. I do have some ports forwarded on the server so that is definitely something I am concerned about.Attached is the most recent log if you still need the other diagnostics I will post that as well. I looked for any processes with X or Z - and found a plex related PID with a Z status - tried killing it but was not successful and had to hard reboot. So it is most probably something internal and related to Plex I believe. Appreciate the help as always. log.zip
July 3, 20179 yr Author Community Expert What is the reason you are using unmenu ? I normally don't use it, especially after 6.x.x. But used it after a long time yesterday because the main GUI was not coming up but terminal was and uu worked, was trying to download and look at the log mainly with uu but download did not work from there either. I even tried to copy the log over to the boot drive through terminal before launching uu and that also did not work. Pointed out that in my OP since I saw that in the log after uu was launched those were literally the next few lines after it was launched. My post was mainly to solve for the recurring gui freeze and share inaccessible issue. Sorry for the confusion. Sent from my Pixel XL using Tapatalk
July 6, 20178 yr Author Community Expert Is this disk failing (smart report attached) - some of the content in this disk is inaccessible and is causing the shares to become unresponsive if some application tries to access that folder/file? ST3000DM001-1CH166_W1F14EBG-20170706-1432.txt
July 6, 20178 yr Community Expert Nothing in the SMART report on this disk indicates a problem with the disk itself. (The #199 errors are usually related to cabling or disk controller. Plus, there are so few of them for the number of hours you have on the disk, and they could have easily occurred more than a year or two ago.)
July 6, 20178 yr Community Expert 23 minutes ago, abhi.ko said: causing the shares to become unresponsive if some application tries to access that folder/file? Filesystem corruption is more likely to cause this, check filesystem on that disk or post the diagnostics after that happens.
July 7, 20178 yr Author Community Expert 10 hours ago, Frank1940 said: Nothing in the SMART report on this disk indicates a problem with the disk itself. (The #199 errors are usually related to cabling or disk controller. Plus, there are so few of them for the number of hours you have on the disk, and they could have easily occurred more than a year or two ago.) 9 hours ago, johnnie.black said: Filesystem corruption is more likely to cause this, check filesystem on that disk or post the diagnostics after that happens. Thank you both. Running the reiserfsck now. Will post the output once it is done. I did run an extended SMART test as well and the results for that is attached. Edit: Here is the file system check results - no corruptions found. So I have no idea what is causing this. Any suggestions or troubleshooting tips please? reiserfsck 3.6.24 Will read-only check consistency of the filesystem on /dev/md6 Will put log info to 'stdout' ########### reiserfsck --check started at Thu Jul 6 22:52:40 2017 ########### Replaying journal: Trans replayed: mountid 129, transid 53575, desc 3102, len 1, commit 3104, next trans offset 3087 Replaying journal: | | 0.2% 1 trans Trans replayed: mountid 129, transid 53576, desc 3105, len 1, commit 3107, next trans offset 3090 Trans replayed: mountid 129, transid 53577, desc 3108, len 1, commit 3110, next trans offset 3093 Trans replayed: mountid 129, transid 53578, desc 3111, len 1, commit 3113, next trans offset 3096 Replaying journal: | / 0.7% 4 trans Trans replayed: mountid 129, transid 53579, desc 3114, len 1, commit 3116, next trans offset 3099 Trans replayed: mountid 129, transid 53580, desc 3117, len 1, commit 3119, next trans offset 3102 Replaying journal: | - 1.0% 6 trans Trans replayed: mountid 129, transid 53581, desc 3120, len 18, commit 3139, next trans offset 3122 Replaying journal: | \ 1.2% 7 trans Trans replayed: mountid 129, transid 53582, desc 3140, len 12, commit 3153, next trans offset 3136 Replaying journal: |= | 1.3% 8 trans Trans replayed: mountid 129, transid 53583, desc 3154, len 14, commit 3169, next trans offset 3152 Replaying journal: |= / 1.5% 9 trans Replaying journal: Done. Reiserfs journal '/dev/md6' in blocks [18..8211]: 9 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 348111 Internal nodes 2289 Directories 1820 Other files 16353 Data block pointers 350620110 (0 of them are zero) Safe links 0 ########### reiserfsck finished at Thu Jul 6 23:24:04 2017 ########### I ran the diagnostics as well.Here are the files. tower-diagnostics-20170707-0019.zip Edited July 7, 20178 yr by abhi.ko Added Info
July 7, 20178 yr Author Community Expert Now there is a new problem after restarting the array after the file system checks were done and the diagnostics collected. Tower is saying no apps/docker containers installed when i do have them in the image and the image is saved in the cache and the settings have docker enabled pointing to the correct image location. I tried rebooting but no luck. Shares are all mounted and working fine so are plugins. Edited July 7, 20178 yr by abhi.ko
July 7, 20178 yr Community Expert 29 minutes ago, abhi.ko said: Tower is saying no apps/docker containers installed when i do have them in the image and the image is saved in the cache and the settings have docker enabled pointing to the correct image location. Post new diags
July 7, 20178 yr Community Expert Docker is starting correctly (though you have a very large docker image), if there are no dockers you'll need to re-add them using the previous templates.
July 7, 20178 yr Author Community Expert Just now, johnnie.black said: Docker is starting correctly (though you have a very large docker image), if there are no dockers you'll need to re-add them using the previous templates. Thanks - is there a way I can increase the size of the docker image or figure out why it is so huge and clean up?
July 7, 20178 yr Community Expert Delete the image and create a new smaller one before adding the templates
July 7, 20178 yr Author Community Expert 5 minutes ago, johnnie.black said: Delete the image and create a new smaller one before adding the templates Thanks again. How do I create a new image after deleting? Is there an option somewhere in the settings?
July 7, 20178 yr 14 hours ago, abhi.ko said: Thanks again. How do I create a new image after deleting? Is there an option somewhere in the settings? Just restart the service
July 7, 20178 yr Author Community Expert On 7/7/2017 at 1:40 AM, abhi.ko said: here you go. tower-diagnostics-20170707-0138.zip Thanks Squid! I'm still having the instability issues with the system - Tower is accessible via telnet but WebGUI and shares are inaccessible. Edited July 8, 20178 yr by abhi.ko
July 8, 20178 yr Author Community Expert Hi All, Sorry for posting again. But I am trying to desperately find the root cause for this issue and a fix - my linux tech abilities are a great limitation, hence soliciting your help. Any and all help appreciated. Here I am posting the latest diagnostics file. What I have noticed so far: Certain files and folders are causing the problem. When you try to access the said files/folders via the shares (mostly) or directly via telnet - the system freezes and shares crash. I have left the log running both on the WebGUI and on telnet (tail -f) and tried to see if it throws an error of some sort when the shares and WebGUI go down - but nothing is captured there. Telnet session continues to run as normal even when the shares are inaccessible and the system stops responding unless you try and cd into a problem directory or try to access a problem file. I have tried to reboot, reboot -f and powerdown from telnet but none of those work - I have left it alone to try and shutdown properly for hours and still nothing. It reponds with a message that shutdown is initiated but nothing happens. Only way I can get it back up and running is a physical reboot or power down (pressing the switch on the system). Have checked the parity a few times including the last being yesterday and it completed without any issues - no errors. I haven't yet resized the docker image since I am not sure if these issues are related (I don't believe they are) - I have a 1TB cache SSD, so 50GB for docker image is something I can live with. I did recently add 2 sticks of RAM - making the total RAM available 16GB - however this issue has been happening before that. I have run the file system checks on 2 disks which had one of these problem directories and that found no issues. One result is shared above. Neither did the extended SMART test find anything. Please take a look at what might be the issue and help me out. Thanks in advance. If there is more information or diagnostics you need, do let me know. tower-diagnostics-20170708-1053.zip
July 8, 20178 yr 1 minute ago, abhi.ko said: When you try to access the said files/folders via the shares (mostly) or directly via telnet - the system freezes and shares crash. What are the exact names of the files? Any weird punctuation, etc contained within them?
July 8, 20178 yr Author Community Expert 2 minutes ago, Squid said: What are the exact names of the files? Any weird punctuation, etc contained within them? No nothing weird and these are old files that have worked well in the past. Most recent example of one that made it crash yesterday is given below. I noticed yesterday that this movie was a duplicate and tried to delete the first folder (the one without the MKV) and that is when it crashed. But if you look at the date they were created and last modified - it is in 2015.
July 8, 20178 yr Delete it via the command line or via midnight commander (mc). Something within is possibly screwing with SMB / Windows causing issues. But, you may also have file system corruption on whatever disk that folder is contained on.
July 10, 20178 yr Author Community Expert On 7/8/2017 at 0:40 PM, Squid said: Delete it via the command line or via midnight commander (mc). Something within is possibly screwing with SMB / Windows causing issues. But, you may also have file system corruption on whatever disk that folder is contained on. Thanks - that worked for deleting the directory. But the problem is still not solved it It has started occurring much more frequently. Every day now. The only way I can get it solved is by physically rebooting the system. I am planning on running file system check on all drives one by one - unless there is an easier way were I can queue/schedule it together.
July 10, 20178 yr Community Expert You can check several at the same time, just open a new SSH connection for each one. You may also want to consider converting all your disks to xfs, reiserfs can the source of stability issues for some users with unRAID v6.
Archived
This topic is now archived and is closed to further replies.