• [6.8-RC1] kernel: tun: unexpected GSO


    Outcasst
    • Retest Minor
    Oct 12 03:35:17 Storage kernel: tun: unexpected GSO type: 0x0, gso_size 1357, hdr_len 1411
    Oct 12 03:35:17 Storage kernel: tun: 13 e4 3d f7 10 86 b8 9e 87 b1 5f 81 d9 7a 98 c9 ..=......._..z..
    Oct 12 03:35:17 Storage kernel: tun: 26 fa 2d 78 50 03 f2 b2 22 55 bc 68 29 75 83 46 &.-xP..."U.h)u.F
    Oct 12 03:35:17 Storage kernel: tun: 04 35 d4 e4 71 d8 5c 04 e3 e2 a2 6d 4e 1f 22 9d .5..q.\....mN.".
    Oct 12 03:35:17 Storage kernel: tun: 6f 97 72 60 c9 63 2b dc f4 ec c7 4f 68 60 66 9e o.r`.c+....Oh`f.

    Getting the above message repeated over and over again in the log whenever a docker tries to access the NIC.

    storage-diagnostics-20191012-0237.zip

    • Thanks 3



    User Feedback

    Recommended Comments



    I'm also getting this error.

     

    Seems to be okay right up until the point of starting a VM and then the log fills up. I have a good half a dozen dockers with static IPs running at the time of starting the VM

    Link to comment

    I dont get this when I have my dockers running. I only get the below when i have my win10 vm running. Any thoughts?

     

    Oct 17 09:10:36 eMu-Unraid kernel: tun: unexpected GSO type: 0x0, gso_size 1308, hdr_len 1374
    Oct 17 09:10:36 eMu-Unraid kernel: tun: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
    Oct 17 09:10:36 eMu-Unraid kernel: tun: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
    Oct 17 09:10:36 eMu-Unraid kernel: tun: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
    Oct 17 09:10:36 eMu-Unraid kernel: tun: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
    Oct 17 09:10:36 eMu-Unraid kernel: tun: unexpected GSO type: 0x0, gso_size 1308, hdr_len 1374
    Oct 17 09:10:36 eMu-Unraid kernel: tun: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
    Oct 17 09:10:36 eMu-Unraid kernel: tun: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
    Oct 17 09:10:36 eMu-Unraid kernel: tun: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
    Oct 17 09:10:36 eMu-Unraid kernel: tun: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
    Oct 17 09:10:36 eMu-Unraid kernel: tun: unexpected GSO type: 0x0, gso_size 1308, hdr_len 1374
    Oct 17 09:10:36 eMu-Unraid kernel: tun: e4 11 5f aa 32 28 bd eb 33 81 72 78 b4 79 b9 06 .._.2(..3.rx.y..
    Oct 17 09:10:36 eMu-Unraid kernel: tun: 94 79 67 11 20 f1 22 7b e0 ea 93 9f 33 8f 1a 91 .yg. ."{....3...
    Oct 17 09:10:36 eMu-Unraid kernel: tun: c7 1c 0b 59 f7 4b d5 b4 e8 dc 34 2a 1b 91 aa bf ...Y.K....4*....
    Oct 17 09:10:36 eMu-Unraid kernel: tun: cf 91 3d ae 3f 28 d2 bb 4a 6a dc 06 e9 5a 6f 8a ..=.?(..Jj...Zo.
    Oct 17 09:10:36 eMu-Unraid kernel: tun: unexpected GSO type: 0x0, gso_size 1308, hdr_len 1374
    Oct 17 09:10:36 eMu-Unraid kernel: tun: 07 49 c0 cc 1b 77 a9 be 60 db 20 5d 2e c3 85 4f .I...w..`. ]...O
    Oct 17 09:10:36 eMu-Unraid kernel: tun: f7 b1 67 e9 64 1f 74 81 ff b5 f9 88 d1 1f e9 99 ..g.d.t.........
    Oct 17 09:10:36 eMu-Unraid kernel: tun: 5d b9 fc 77 c7 c1 37 11 71 43 62 e9 e7 fc bf 59 ]..w..7.qCb....Y
    Oct 17 09:10:36 eMu-Unraid kernel: tun: 4d ed a4 17 8f 13 59 ca 18 12 7d 8e 27 99 0b f3 M.....Y...}.'...
    Oct 17 09:10:37 eMu-Unraid kernel: tun: unexpected GSO type: 0x0, gso_size 1308, hdr_len 1374
    Oct 17 09:10:37 eMu-Unraid kernel: tun: cb 46 69 df aa 1c 9f bf f7 20 d8 ab 75 59 ad 16 .Fi...... ..uY..
    Oct 17 09:10:37 eMu-Unraid kernel: tun: 5b 15 b6 8e 54 71 d2 ee 3a 45 0f 29 62 5e 6e 78 [...Tq..:E.)b^nx
    Oct 17 09:10:37 eMu-Unraid kernel: tun: fc 9f 1d 3d a2 ea 75 5a 9b 34 2e ce d0 7f 8b 1c ...=..uZ.4......
    Oct 17 09:10:37 eMu-Unraid kernel: tun: 7e 92 83 41 60 ce d2 06 13 c3 d2 5e 11 28 99 07 ~..A`......^.(..
    Oct 17 09:10:37 eMu-Unraid kernel: tun: unexpected GSO type: 0x0, gso_size 1308, hdr_len 1374
    Oct 17 09:10:37 eMu-Unraid kernel: tun: 13 c1 5b 4f 70 36 60 1c 09 18 ed 9e 44 19 3b a4 ..[Op6`.....D.;.
    Oct 17 09:10:37 eMu-Unraid kernel: tun: ed 5a 9d a7 94 e2 4c 15 3f e6 3e 4d cb c7 f8 f9 .Z....L.?.>M....
    Oct 17 09:10:37 eMu-Unraid kernel: tun: 89 2f 87 0f 0f 3c f7 5a 96 ff 1a da 04 e1 99 df ./...<.Z........
    Oct 17 09:10:37 eMu-Unraid kernel: tun: c4 0b 35 cb a7 e0 23 c4 d7 5d f2 18 72 ce c1 3a ..5...#..]..r..:
    Oct 17 09:10:38 eMu-Unraid kernel: tun: unexpected GSO type: 0x0, gso_size 1308, hdr_len 1374
    Oct 17 09:10:38 eMu-Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Oct 17 09:10:38 eMu-Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Oct 17 09:10:38 eMu-Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Oct 17 09:10:38 eMu-Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Oct 17 09:10:38 eMu-Unraid kernel: tun: unexpected GSO type: 0x0, gso_size 1308, hdr_len 1374
    Oct 17 09:10:38 eMu-Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Oct 17 09:10:38 eMu-Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Oct 17 09:10:38 eMu-Unraid kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

     

    Link to comment

    I think this has something to do with VMs and dockers sharing the same bridge interface.

     

    Setting my VM to Vibr0 fixes this, but this isn't a solution as I can no longer RDP in to it.

     

    Setting dockers to bridge/host isn't an option either since I need them to have static custom IP addresses.

    Link to comment

    If someone can point me in the direction of setting up the Vibr0 to pass to my br0 traffic i can live with a different ip range. I know enough advanced networking to be dangerous... lol

     

    Otherwise I will have to go get another NIC :(

    Link to comment
    7 hours ago, emuhack said:

    If someone can point me in the direction of setting up the Vibr0 to pass to my br0 traffic

    That's not possible.

    7 hours ago, emuhack said:

    Otherwise I will have to go get another NIC

    That's the easiest way to solve the issue.

    You should configure the 2nd NIC without IP address under network settings, and assign it to VMs

    • Like 1
    Link to comment

    I can confirm the same behaviour on RC4. I do indeed have VMs and dockers with assigned IP addresses.

     

    Update: my log fills up to 100% after one day with a flood of these errors. Is there a way to pinpoint which container/VM is responsible?

    Edited by szymon
    update
    Link to comment

    I too am getting this on one of my unraid server on every RC including RC5 luckily this is a server with a quad port nic as I run a multitude of dockers with static IP using br0 an 3 vm's VLAN's is not an option for me as the VM's need to talk another unraid server and my Unifi gear is already segregating my smart home devices to a dedicated locked down VLAN. having said that I will stop docker and VM manager when i get home and configure each of the 3 additional ethernet port with bridging and set my VM's to br1, br2 and br3 ill have to fish up some ethernet and make some room on my switch but I should be able to do it. otherwise I would really like this bug to be squashed I am constantly filling /var/log and need to remove them since the syslog server setting is not working correctly (currently have mirror to flash off and to store on a share on my cache drive but its not storing there at all syslog just keeps filling up at /var/log

    • Thanks 1
    Link to comment

    Update running RC5 and wow what a pain it was to configure and get up and running here is what i had to do. (I blame working all night then missing a couple settings due to being up well over 24 hours at that point)

     

    I did all the work with the array stopped.


    First the additional 3 ports were Port downed in Network settings, i upped them and connected ethernet

    The mac addresses for eth2 and 3 were reversed to swap them i had to disable docker and vm manager and had to reboot, but didn't as i wanted to set it all up first. 

    updated the eth1,eth2,eth3 settings (names and DHCP to auto) for them then rebooted

     

    I was getting an /dev read/24 Error 110 on boot and not getting an ip on eth0 (this is apparently a power issue to the USB port as per some earching on these forums, however its been working fine for well over a year on that port) tried multiple USB ports same issue, blew network.cfg away in my laptop same issue. I found an internal USB port on the motherboard (Acer AR380F2) and can boot from that..saw bonding enabled on eth0 for eth1,2,3, disabled it, saved and rebooted, so far so good.

    Fixed mac addressing to the ports (eth2 and eth3 macs were reversed), rebooted,

     

    Labeled and set one port up at a time applying each port change.

     

    then went started docker service and VM manager and updated the network for each VM to br1, 2 and 3 (only running 3 VM's)

    set the mac address to each VM to the same as the eth ports that was an issue and causing a ton of flooded alarms in the syslog. and the VM's were not getting an IP.

    I wanted DHCP reserved IP's to each VM (for my port forwards) so I figured i would turn off DHCP on the eth1,2 and 3 ports under network settings and use the random Mac Address creator in the VM settings and assign those to my Firewall

    booted one VM to see...success
    booted second...success
    booted third....success
    those alarms from the OP are gone as well and syslog is normal once again (and my unRAID server and VM's each have a dedicated 1 gbps link to the LAN now. I am thinking of changing this to 2 links VM for one link as one is a very light weight fedora server i use to SSH tunnel home to from my office


    Moral of the story, don't do this when you're tired and if a single change requires a reboot don't change anything else until after the reboot.

    Link to comment

    Same RC 5 10 Dockers 4 vms br0 for all, ever since rc5 I've noticed one of my br0 Dockers would "stall out" become unreachable, I'm assuming this issue is why. Error does not appear until a VM is started 

     

     

    tower-diagnostics-20191108-1356.zip

    Edited by Fiservedpi
    New info has come to light
    Link to comment
    4 hours ago, Fiservedpi said:

    I'm assuming this issue is why.

    We need diagnostics to confirm, your assumptions doesn't prove anything.

    Link to comment

    Still happening:

    Nov 22 23:59:23 MediaServer kernel: tun: unexpected GSO type: 0x0, gso_size 125, hdr_len 179
    Nov 22 23:59:23 MediaServer kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
    Nov 22 23:59:23 MediaServer kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
    Nov 22 23:59:23 MediaServer kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
    Nov 22 23:59:23 MediaServer kernel: tun: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

    I have two windows VMs running with static ip addresses.  I have nine dockers running and when I set one of the dockers to static ip, the errors start.

    Link to comment
    14 hours ago, limetech said:

    Changed Status to Retest

    Just to add (I sent this directly to LT as well) but for other users who are troubleshooting - I can cause the errors to start by starting a Linux (ubuntu) VM on my VLAN as well, so while the issue is connected to VM's (at least it appears to be in my case) it's not the OS on the VM itself.

    • Like 1
    Link to comment
    21 hours ago, Chunks said:

    Just to add (I sent this directly to LT as well) but for other users who are troubleshooting - I can cause the errors to start by starting a Linux (ubuntu) VM on my VLAN as well, so while the issue is connected to VM's (at least it appears to be in my case) it's not the OS on the VM itself.

    It has to do with docker and VMs are in the same LAN (?) thats probably why VLANs help.

    Edited by nuhll
    Link to comment

    This situation can be quite confusing.  It is not necessarily a VM or Docker specific issue.  It seems to relate to static ip addresses in VMs and Dockers.  Everything appears to work, but the logging is extreme.  I ran into this situation when I was testing the beta 6.8.  I was able to get around it by not using static ip addresses in Dockers.  Others have been able to solve the issue with an added NIC or setting up VLANS.

     

    When I first ran into this situation, I did a little research and the error comes from 'tun'.  The developers implemented the log message because they felt that the network error that causes this message was important and the underlying error should be resolved, rather than just being ignored by turning off the logging.

     

    This issue came up for me on a specific version of 6.8 beta.  LT is researching this issue and what changed between the version that worked for me and the version where I ran into the problem  LT will eventually get it resolved.  Just turning off the logging is probably not be right answer.  Be patient while they work on it.  They are not ignoring it!  Don't get frustrated if LT does not respond to every post or PM.  I'd rather they work on a solution and not spend a lot of time discussing the issue over and over.

     

    There are several options you have if you have this issue:

    • Do not use this release candidate and wait for a resolution.
    • Use another NIC for Dockers or VMs so they are separated.
    • Set up VLANs to isolate Dockers and VMs.
    • Remove static ip addresses from Dockers and/or VMs until the error logging stops.

     

     

    • Like 2
    • Thanks 1
    Link to comment

    Another simple workaround can be to stop either VMs or containers, while using the other.

    E.g. when a VM is needed to run, stop the container(s) which may interfere.

    Link to comment
    22 hours ago, nuhll said:

    It has to do with docker and VMs are in the same LAN

    No, it is about VM and Docker custom network (macvlan) sharing the same interface at the same time.

    Both setup a virtual interface to communicate with the physical interface, and this seems to lead to a conflict.

     

    See the posts above for all the possible workarounds until the issue gets resolved.

    Link to comment

    only way to fix it (if i read correct)

     

    - VLAN (cant do)

    - or start dockers / vm seperate (cant do)

    - dont use br0, which i cant do

     

    1.) What about virtual NICs its pretty easy to add them to linux, cant we do vlan with it? I tried adding a virtual nic, but unraid didnt behave like i expected (it was just adding a new route, without the option to chose the new nic in settings)

     

    2.) It seems like only problem is spamming log full, cant we just remove that spam out of the log, until the root of the problem is fixed?

    Edited by nuhll
    Link to comment

    There are three more options to consider:

     

    1. Reconfigure your docker container to use either host or bridge network

    2. Use a second interface and use this for either VMs or containers solely

    3. Use a stable version of Unraid

     

    I don't know if virtual NICs are going to work with all the different set ups possible with Unraid.

    But if you find another suitable workaround using this approach, please post, as it may benefit others

    Edited by bonienl
    Link to comment

    For those reporting this issue: what version of Unraid OS were you running that did not exhibit this issue or similar issue reported here:

     

    Link to comment

    The first sign of this GSO error for me was Rc4, had zero issues on RC4 OR before all started with first boot of RC5 and starting docker and VM's if only start docker no error once you hit start on a VM it appears

    Edited by Fiservedpi
    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.