• [SOLVED] [6.9.0-beta25] through [6.9.0-RC1] Syslog Server Broken, Error in Logs.


    nblom
    • Solved Minor

    Hello all,

    I was attempting to have my unraid server log to itself so that the logs persisted.

    I did the following:

    1. enabled the local syslog server
    2. set it log to a new share that I previously created
    3. enabled rotation
    4. set it to 50mb
    5. 4 files
    6. Set my remote server to my private unraid ip
    7. left the ports alone (UDP 514)
    8. Mirror to flash set to No

     

    When I click apply, I get an error in my System Logs that says the configuration failed to parse. Nothing gets put in the share I created.

     

    syslog_settings.thumb.png.fda2c5dcfe58d15e196400cf73bbf853.png

     

    The error:

    syslog_error.thumb.png.8045b4e1a82175ca7437b1aabfcd33be.png

     

    Text of the error:

    Jul 19 08:03:18 XXX ool www[24597]: /usr/local/emhttp/plugins/dynamix/scripts/rsyslog_config
    Jul 19 08:03:20 XXX rsyslogd:  Could not find template 1 'remote' - action disabled [v8.2002.0 try https://www.rsyslog.com/e/3003 ]
    Jul 19 08:03:20 XXX rsyslogd: error during parsing file /etc/rsyslog.conf, on or before line 121: errors occured in file '/etc/rsyslog.conf' around line 121 [v8.2002.0 try https://www.rsyslog.com/e/2207 ]
    Jul 19 08:03:20 XXX rsyslogd: [origin software="rsyslogd" swVersion="8.2002.0" x-pid="26836" x-info="https://www.rsyslog.com"] start

     

    fenrir-diagnostics-20200719-0823.zip

    • Thanks 1



    User Feedback

    Recommended Comments



    Hello all,

    I have exactly the same error messages in 6.9.0 beta 25 ("Could not find template 1 'remote' (resp. 'flash') - action disabled ...") whatever the settings are, either remote logging or mirror syslog to flash.

    Link to comment

    Still not working, but I was able to get the errors to go away; still nothing is showing up in my syslogs share:

    $template remote,"%msg%" # added manually
    *.* ?remote
    *.* /mnt/user/syslogs/sys.log;remote # added manually

    I am sure I am doing something wrong.

    I initially got the error message to go away with just adding the $template line. This is what seems to have been missing from the configuration generated from the UI.

     

    Please note: In order for the new lines to stick in the file, you canNOT use the UI. The UI will overwrite the changes and break the configuration again.

     

    To manually restart the rsyslogd daemon:

    $ /etc/rc.d/rc.rsyslogd restart

     

    Edited by nblom
    typo
    Link to comment

    The same error happens in 6.9.0-beta29 as in 6.9.0-beta25.

     

    image.thumb.png.cb555b63335ef30a7867f67df68e9c51.png

     

    Screenshot of my settings:

    image.png.f25c38bd1d757ad53d7b6d60155c83c4.png

     

    Error:

    Oct  1 14:20:01 Fenrir ool www[27683]: /usr/local/emhttp/plugins/dynamix/scripts/rsyslog_config
    Oct  1 14:20:03 Fenrir rsyslogd:  Could not find template 0 'remote' - action disabled [v8.2002.0 try https://www.rsyslog.com/e/3003 ]
    Oct  1 14:20:03 Fenrir rsyslogd: error during parsing file /etc/rsyslog.conf, on or before line 116: errors occured in file '/etc/rsyslog.conf' around line 116 [v8.2002.0 try https://www.rsyslog.com/e/2207 ]
    Oct  1 14:20:03 Fenrir rsyslogd:  Could not find template 1 'remote' - action disabled [v8.2002.0 try https://www.rsyslog.com/e/3003 ]
    Oct  1 14:20:03 Fenrir rsyslogd: error during parsing file /etc/rsyslog.conf, on or before line 122: errors occured in file '/etc/rsyslog.conf' around line 122 [v8.2002.0 try https://www.rsyslog.com/e/2207 ]
    Oct  1 14:20:03 Fenrir rsyslogd: [origin software="rsyslogd" swVersion="8.2002.0" x-pid="29297" x-info="https://www.rsyslog.com"] start

     

    Link to comment

    Same here - has been working in the past but I am doing a new system and not working - tried deleting /boot/config/rsyslog.conf but comes back and is broken

     

    Link to comment

    Still preset with beta30

     

    This isn't really a minor bug as suggetive by the priority tag - it completely prevents diagnosing when the server crashes ( I have no idea how to figure out what goes wrong with my server, and I don't know how to fix this)

    Link to comment

    I bumped this to urgent since @Dataone made a good point. It is technically a data loss bug (not an array problem). The logs are lost which makes it very difficult to diagnose server issues.

     

    @limetech Not sure if you are the one to notify about this, but this has been happening since at least 6.9.0beta25. I can't really revert to earlier versions right now to test.

    • Thanks 2
    Link to comment

    Just to confirm the misconfiguration of rsyslogd is still present in 6.9.0-beta35. Whatever the settings are (mirror to flash, activation of the local syslog server, forward to a remote syslog server), rsyslogd crashes when parsing rsyslog.conf with the same error again and again :

    "rsyslogd: Could not find template 1 'remote' - action disabled"

    I do agree with @nblom and @Dataone that this is now an urgent matter. I do enjoy testing beta releases, but it doesn't make much sense if I'm not given the possibility to provide the developers with a syslog in case of an unexpected crash ...

     

    So @limetech, a simple acknowledgment of this issue and a confirmation that you don't imagine a candidate release with such a blackhole in the management capabilities would be more than welcome. I know your "lack of immediate reply does not mean (our) report is being ignored.", as stated in your pinned post, but more than four months after the OP, with three beta releases (29, 30, and 35) in the meantime, I'm sure you will understand we wonder whether this post has ever been read and/or taken into account.

     

    Thanks in advance.

    Edited by Gnomuz
    Link to comment

    @nblom

    As the original poster, I suppose you are able to change the post title. May I suggest something like "[6.9.0-beta25]->[6.9.0-beta35] Major syslog server issue" so that the post is not considered as only related to beta 25 and thus ignored by the developers.

    Thanks

    Link to comment

    Great, let's hope it will draw attention ! Really, we can't insist enough that a way to have persistent logs is a must in case of a crash, that's, among others, why rsyslog has been created so long ago ;)

    Link to comment
    1 hour ago, limetech said:

    Fixed in next release.  Guys this is not Urgent

    Let's say that on July 19th 2020 (date of the OP), it may have been considered as "Minor", but on November 23rd 2020, more than 4 months later, without any reaction or acknowledgement in the meantime, I'm sure you understand the "Urgent" tag was a kind of desperate "message in a bottle" ...

    Moreover, as @Dataonestated above, and I completely agree with him, I hardly would qualify as "Minor" a bug which prevents from preserving syslogs in case of a crash, and thus prevents any serious diagnostic on the root cause, especially on beta versions we are happy to test and debug with you.

    Anyway, glad to see it has been taken into account and we'll be able to retest in the next beta release !

    Link to comment
    48 minutes ago, Gnomuz said:

    Let's say that on July 19th 2020 (date of the OP), it may have been considered as "Minor", but on November 23rd 2020, more than 4 months later, without any reaction or acknowledgement in the meantime, I'm sure you understand the "Urgent" tag was a kind of desperate "message in a bottle" ...

    Moreover, as @Dataonestated above, and I completely agree with him, I hardly would qualify as "Minor" a bug which prevents from preserving syslogs in case of a crash, and thus prevents any serious diagnostic on the root cause, especially on beta versions we are happy to test and debug with you.

    Anyway, glad to see it has been taken into account and we'll be able to retest in the next beta release !

    There are other ways of capturing logs:

    • direct to a syslog server on another machine
    • have the webgui Log window open
    • have a terminal window open running 'tail -f /var/log/syslog'

     

    I get some people think this what they consider "urgent" but sorry I have to be somewhat strict or else ever single bug report will be marked urgent (because it's always "urgent" to the reporter).

    Link to comment

    Thanks for letting us know.

    I look forward to the fix in the next release.

     

    I understand the need for prioritization. Since it is scheduled for release, I don't think we need anymore debate.

     

    I will close this ticket after I test it in the next release.

     

    Thanks again.

    Link to comment

    I've been reporting bugs and bugs have also been reported to me for more than 30 years in large IT teams, so be sure I understand your doomed to failure attempt to sort between urgent, important, minor and -above all- problems with the chair-to-keyboard interface ...

    But in that case, you may have noticed from the reports that redirection to a remote syslog server is also faulty, as rsyslogd crashes immediately each time it is launched, whatever the settings are. As for the webgui log window, the terminal window running "tail -f", or the IPMI remote KVM for me, you and I have for sure learned the hard way over the years that they are rarely open in front of you, especially in the case of an unexpected crash in the middle of the night or during the weekend !

    If ever I were to experiment a crash with beta35 right now, I would report it, but would also be totally unable to provide you with any relevant pre-crash log, which for sure would not ease the diag or fix on your side.

    I think that's why we have taken the liberty of insisting on this issue, as it is key for the debugging and validation process of our OS in beta phases, and may thus help you deliver a stable and exciting 6.9.0 final release soon(er)™ to this great community 😉

    Link to comment
    1 hour ago, Gnomuz said:

    they are rarely open in front of you, especially in the case of an unexpected crash in the middle of the night or during the weekend

    Debugging relies very greatly on being able to repeat the problem.  If someone experiences a crash and they just say, "hey it crashed", well nothing much to do about that.  But if they create the same conditions that led to the crash before and, anticipating a crash might soon happen, have a terminal window open, then that might be able to capture enough output to be useful.  That was my point.

    Link to comment
    1 hour ago, Stupifier said:

    Ouch...just stumbled across this while I was in the process of troubleshooting a crash on Beta. This stinks.....

    If you have the ability, you can run a syslog server on another machine and capture the logs. I am using Kiwi syslog server until this issue is fixed.

    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.