Export Moogsoft Service Logs and Tomcat Logs to JSON

Log files are useful for troubleshooting and ensuring everything is running smoothly. Converting the log files to JSON gives you a local copy of them. This document guides you in exporting Moogsoft Enterprise and Tomcat logs to JSON.

Exporting Tomcat Logs

You can refer to the official doc to help you migrate in logging from java.utils.logging to log4j for tomcat 8.5 and later. However, there is no direct download link for the required and dependent libraries.

Before you begin

Before you start exporting Tomcat Logs:

Configure logging for Tomcat

  1. Create a set of directories in Catalina home (in your case /usr/share/apache-tomcat) named "log4j2/lib" and "log4j2/conf".

  2. Create the following directories: usr/share/log4j2/lib/ and usr/share/log4j2/conf.

  3. Copy log4j2-api-2.13.0.jar, log4j2-core-2.13.0.jar, and log4j2-appserver-2.13.0.jar to the log4j2/lib directory.

  4. Create the following file: /usr/share/apache-tomcat/log4j2/conf/$CALTALINA_HOME/log4j2/lib/logrj2-tomcat.xml.

  5. Edit log4j2-tomcat.xml as follows:

    <?xml version="1.0" encoding="UTF-8"?>
    <Configuration status="info">
        <Appenders>
            <Console name="Console" target="SYSTEM_OUT">
                <JSONLayout compact="true" eventEol="true" properties="true"/>
            </Console>
        </Appenders>
        <Loggers>
            <Logger name="*" level="trace">
                <AppenderRef ref="Console"/>
            </Logger>
            <Root level="trace">
                <AppenderRef ref="Console"/>
            </Root>
        </Loggers>
    </Configuration>
  6. Create or modify setenv.sh in the tomcat bin directory to include CLASSPATH=$CATALINA_HOME/log4j2/lib/*:$CATALINA_HOME/log4j2/conf.

  7. Restart Tomcat. The following is an example the Tomcat log catalina.out in JSON format:

    {"thread":"Catalina-utility-2","level":"TRACE","loggerName":"org.apache.jasper.compiler.JspRuntimeContext","message":"Checking JSPs for unload in context [/manager], JSP count: [0] queue length: [-1]","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.appserver.tomcat.TomcatLogger","threadId":42,"contextMap":{},"instant":{"epochSecond":1576835560,"nanoOfSecond":539000000},"threadPriority":1}
    {"thread":"Catalina-utility-2","level":"TRACE","loggerName":"org.apache.jasper.compiler.JspRuntimeContext","message":"Checking JSPs for unload in context [/docs], JSP count: [0] queue length: [-1]","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.appserver.tomcat.TomcatLogger","threadId":42,"contextMap":{},"instant":{"epochSecond":1576835560,"nanoOfSecond":539000000},"threadPriority":1}
    {"thread":"ajp-nio-8009-ClientPoller","level":"TRACE","loggerName":"org.apache.tomcat.util.net.NioEndpoint","message":"timeout completed: keys processed=0; now=1576835560571; nextExpiration=1576835560569; keyCount=0; hasEvents=false; eval=false","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.appserver.tomcat.TomcatLogger","threadId":39,"contextMap":{},"instant":{"epochSecond":1576835560,"nanoOfSecond":571000000},"threadPriority":5}

Exporting JSON logs for all Moogsoft Enterprise services

Moogsoft Enterprise logs use standard Log4J configuration. Edit the PatternLayout and JsonLayout to configure the logs for JSON format. You can change the Moogsoft Services configuration files, the Layout, and 3-4 supporting parameters under $MOOGSOFT_HOME/config/logging/servlets and $MOOGSOFT_HOME/config/logging.

In the following example, you are adding a Rolling File appender that outputs the json log through a secondary file.

  1. Add a new appender in the integration.log.json file to output a new log (json-integration.log) file that contains all the logs related to Moogsoft integrations in JSON format.

    Append below config in the “appenders” section:

    "RollingFile": {
            "name": "myappender",
            "fileName": "/tmp/json-integration.log",
            "filePattern": "json-integration-%d{MM-dd-yy}-%i.gz",
            "JsonLayout": {
              "compact": true,
              "eventEol": true,
              "locationInfo": true,
              "properties": true
            },
            "Policies": {
              "SizeBasedTriggeringPolicy": {
                "size": "500M"
              }
            },
            "DefaultRolloverStrategy": {
              "max": "40"
            }
          }
        }
  2. Add the appender reference in the logger section:

    "loggers": {
          "Logger": [
            {
              "name": "com.moogsoft",
              "additivity": false,
              "AppenderRef": [
                {
                  "ref": "${sys:MoogsoftLogAppender}"
                }
              ],
              "level": "warn"
            },
            {                             ---> Add this block for the appender reference
              "name": "com.moogsoft",
              "additivity": false,
              "AppenderRef": [
                {
                  "ref": "myappender"  ->> **** this the appender reference for the rolling File appender that we added. *****
                }
              ],
              "level": "warn"
            },
          ]
        }
  3. The output in the new log file should be as follows:

    {"thread":"0:HeartbeatThread","level":"WARN","loggerName":"com.moogsoft.mooms.CMoomsMessageSender","message":"Problem sending message id : [36edbc91-b0b0-4169-ad40-45dda0d9418f]","thrown":{"commonElementCount":0,"name":"java.lang.NullPointerException","extendedStackTrace":[{"class":"com.moogsoft.mooms.CMoomsMessageSender","method":"basicPublish","file":"CMoomsMessageSender.java","line":751,"exact":true,"location":"mooms-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.mooms.CMoomsMessageSender","method":"sendNoWait","file":"CMoomsMessageSender.java","line":560,"exact":true,"location":"mooms-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.mooms.CMoomsMessageSender","method":"send","file":"CMoomsMessageSender.java","line":544,"exact":true,"location":"mooms-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.mooms.CMoomsMessageSender","method":"send","file":"CMoomsMessageSender.java","line":198,"exact":true,"location":"mooms-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.mooms.CMoomsMessageSenderPool","method":"send","file":"CMoomsMessageSenderPool.java","line":278,"exact":true,"location":"mooms-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.mooms.CAbstractMoomsHandler","method":"send","file":"CAbstractMoomsHandler.java","line":235,"exact":true,"location":"mooms-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.health.CHeartbeatThread","method":"run","file":"CHeartbeatThread.java","line":308,"exact":true,"location":"health-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.health.CHaThread$CHaRunnable","method":"run","file":"CHaThread.java","line":200,"exact":true,"location":"health-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","file":"ThreadPoolExecutor.java","line":1128,"exact":true,"location":"?","version":"?"},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","file":"ThreadPoolExecutor.java","line":628,"exact":true,"location":"?","version":"?"},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":834,"exact":true,"location":"?","version":"?"}]},"endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","instant":{"epochSecond":1576762647,"nanoOfSecond":459534000},"contextMap":{},"threadId":21,"threadPriority":5,"source":{"class":"com.moogsoft.mooms.CMoomsMessageSender","method":"basicPublish","file":"CMoomsMessageSender.java","line":760,"classLoaderName":"app"}}
  4. Put the preceding rolling file appender in all of the following files (/usr/share/moogsoft/config/logging and /usr/share/moogsoft/config/logging/servlets) to make the services output the json logs:

    broker.log.json
    integrations.log.json
    controller.log.json
    moog_farmd.log.json
    moogsoft.log.json
    graze.log.json  
    moogpoller.log.json  
    moogsvr.log.json  
    situation_similarity.log.json  
    toolrunner.log.json

    For JSON layout options: https://logging.apache.org/log4j/2.x/log4j-core/apidocs/org/apache/logging/log4j/core/layout/JsonLayout.html

Other example logging formats

More than one appender type in json:

"RollingFile": [
                {
                    "name": "FILE",
                    "fileName": "${sys:MoogsoftLogFilename}",
                    "filePattern": "${sys:MoogsoftLogFilename}-%d{MM-dd-yy}-%i.gz",
                    "PatternLayout": {
                        "header": "${sys:MoogsoftLogHeader}",
                        "pattern": "{{LOG_PATTERN}}"
                    },
                    "Policies": {
                        "SizeBasedTriggeringPolicy": {
                            "size": "500M"
                        }
                    },
                    "DefaultRolloverStrategy": {
                        "max": "40"
                    }
                },
                {
                    "name": "ES_FILE",
                    "fileName": "/var/log/moogsoft/moog_farmd-ES.log",
                    "filePattern": "/var/log/moogsoft/moog_farmd-ES.log-%d{MM-dd-yy}-%i.gz",
                    "JsonLayout": {
                        "compact": true,
                        "eventEol": true,
                        "properties": true,
                        "stacktraceAsString": true,
                        "objectMessageAsJsonObject": true
                    },
                    "Policies": {
                        "SizeBasedTriggeringPolicy": {
                            "size": "500M"
                        }
                    },
                    "DefaultRolloverStrategy": {
                        "max": "5"
                    }
                }
            ],

Use a logger that is referencing more than one appender:

"Async" : {
                "name": "Async",
                "AppenderRef": [{
                    "ref": "FILE"
                }, {
                    "ref": "ES_FILE"
                }]
            }
        },
        "loggers": {
            "Logger": {
                "name": "com.moogsoft",
                "additivity": false,
                "AppenderRef": [{
                    "ref": "Async"
                }],
                "level": "warn"
            }
        }