Export Moogsoft Service Logs and Tomcat Logs to JSON

Log files are useful for troubleshooting and ensuring everything is running smoothly. Converting the log files to JSON gives you a local copy of them. This document guides you in exporting Moogsoft Enterprise and Tomcat logs to JSON.

Exporting Tomcat Logs

You can refer to the official doc to help you migrate in logging from java.utils.logging to log4j for tomcat 8.5 and later. However, there is no direct download link for the required and dependent libraries.

Before you begin

Before you start exporting Tomcat Logs:

Make sure you have ssh access to the servers where you are doing this procedure.
If the product is installed or is running under root, ensure you have root access.
Download the necessary Jar files: https://www.apache.org/dyn/closer.lua/logging/log4j/2.13.0/apache-log4j-2.13.0-bin.tar.gz
Download the necessary Jackson Jar files: https://download.jar-download.com/cache_jars/com.fasterxml.jackson.core/jackson-databind/2.10.0.pr3/jar_files.zip

Configure logging for Tomcat

Create a set of directories in Catalina home (in your case /usr/share/apache-tomcat) named "log4j2/lib" and "log4j2/conf".
Create the following directories: usr/share/log4j2/lib/ and usr/share/log4j2/conf.
Copy log4j2-api-2.13.0.jar, log4j2-core-2.13.0.jar, and log4j2-appserver-2.13.0.jar to the log4j2/lib directory.
Create the following file: /usr/share/apache-tomcat/log4j2/conf/$CALTALINA_HOME/log4j2/lib/logrj2-tomcat.xml.

Edit log4j2-tomcat.xml as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="info">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <JSONLayout compact="true" eventEol="true" properties="true"/>
        </Console>
    </Appenders>
    <Loggers>
        <Logger name="*" level="trace">
            <AppenderRef ref="Console"/>
        </Logger>
        <Root level="trace">
            <AppenderRef ref="Console"/>
        </Root>
    </Loggers>
</Configuration>

Create or modify setenv.sh in the tomcat bin directory to include CLASSPATH=$CATALINA_HOME/log4j2/lib/*:$CATALINA_HOME/log4j2/conf.

Restart Tomcat. The following is an example the Tomcat log catalina.out in JSON format:

{"thread":"Catalina-utility-2","level":"TRACE","loggerName":"org.apache.jasper.compiler.JspRuntimeContext","message":"Checking JSPs for unload in context [/manager], JSP count: [0] queue length: [-1]","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.appserver.tomcat.TomcatLogger","threadId":42,"contextMap":{},"instant":{"epochSecond":1576835560,"nanoOfSecond":539000000},"threadPriority":1}
{"thread":"Catalina-utility-2","level":"TRACE","loggerName":"org.apache.jasper.compiler.JspRuntimeContext","message":"Checking JSPs for unload in context [/docs], JSP count: [0] queue length: [-1]","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.appserver.tomcat.TomcatLogger","threadId":42,"contextMap":{},"instant":{"epochSecond":1576835560,"nanoOfSecond":539000000},"threadPriority":1}
{"thread":"ajp-nio-8009-ClientPoller","level":"TRACE","loggerName":"org.apache.tomcat.util.net.NioEndpoint","message":"timeout completed: keys processed=0; now=1576835560571; nextExpiration=1576835560569; keyCount=0; hasEvents=false; eval=false","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.appserver.tomcat.TomcatLogger","threadId":39,"contextMap":{},"instant":{"epochSecond":1576835560,"nanoOfSecond":571000000},"threadPriority":5}

Exporting JSON logs for all Moogsoft Enterprise services

Moogsoft Enterprise logs use standard Log4J configuration. Edit the PatternLayout and JsonLayout to configure the logs for JSON format. You can change the Moogsoft Services configuration files, the Layout, and 3-4 supporting parameters under $MOOGSOFT_HOME/config/logging/servlets and $MOOGSOFT_HOME/config/logging.

In the following example, you are adding a Rolling File appender that outputs the json log through a secondary file.

Add a new appender in the integration.log.json file to output a new log (json-integration.log) file that contains all the logs related to Moogsoft integrations in JSON format.

Append below config in the “appenders” section:

"RollingFile": {
        "name": "myappender",
        "fileName": "/tmp/json-integration.log",
        "filePattern": "json-integration-%d{MM-dd-yy}-%i.gz",
        "JsonLayout": {
          "compact": true,
          "eventEol": true,
          "locationInfo": true,
          "properties": true
        },
        "Policies": {
          "SizeBasedTriggeringPolicy": {
            "size": "500M"
          }
        },
        "DefaultRolloverStrategy": {
          "max": "40"
        }
      }
    }

Add the appender reference in the logger section:

"loggers": {
      "Logger": [
        {
          "name": "com.moogsoft",
          "additivity": false,
          "AppenderRef": [
            {
              "ref": "${sys:MoogsoftLogAppender}"
            }
          ],
          "level": "warn"
        },
        {                             ---> Add this block for the appender reference
          "name": "com.moogsoft",
          "additivity": false,
          "AppenderRef": [
            {
              "ref": "myappender"  ->> **** this the appender reference for the rolling File appender that we added. *****
            }
          ],
          "level": "warn"
        },
      ]
    }

The output in the new log file should be as follows:

{"thread":"0:HeartbeatThread","level":"WARN","loggerName":"com.moogsoft.mooms.CMoomsMessageSender","message":"Problem sending message id : [36edbc91-b0b0-4169-ad40-45dda0d9418f]","thrown":{"commonElementCount":0,"name":"java.lang.NullPointerException","extendedStackTrace":[{"class":"com.moogsoft.mooms.CMoomsMessageSender","method":"basicPublish","file":"CMoomsMessageSender.java","line":751,"exact":true,"location":"mooms-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.mooms.CMoomsMessageSender","method":"sendNoWait","file":"CMoomsMessageSender.java","line":560,"exact":true,"location":"mooms-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.mooms.CMoomsMessageSender","method":"send","file":"CMoomsMessageSender.java","line":544,"exact":true,"location":"mooms-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.mooms.CMoomsMessageSender","method":"send","file":"CMoomsMessageSender.java","line":198,"exact":true,"location":"mooms-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.mooms.CMoomsMessageSenderPool","method":"send","file":"CMoomsMessageSenderPool.java","line":278,"exact":true,"location":"mooms-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.mooms.CAbstractMoomsHandler","method":"send","file":"CAbstractMoomsHandler.java","line":235,"exact":true,"location":"mooms-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.health.CHeartbeatThread","method":"run","file":"CHeartbeatThread.java","line":308,"exact":true,"location":"health-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"com.moogsoft.health.CHaThread$CHaRunnable","method":"run","file":"CHaThread.java","line":200,"exact":true,"location":"health-7.3.0.1.jar","version":"7.3.0.1 (c194c6f)"},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","file":"ThreadPoolExecutor.java","line":1128,"exact":true,"location":"?","version":"?"},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","file":"ThreadPoolExecutor.java","line":628,"exact":true,"location":"?","version":"?"},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":834,"exact":true,"location":"?","version":"?"}]},"endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","instant":{"epochSecond":1576762647,"nanoOfSecond":459534000},"contextMap":{},"threadId":21,"threadPriority":5,"source":{"class":"com.moogsoft.mooms.CMoomsMessageSender","method":"basicPublish","file":"CMoomsMessageSender.java","line":760,"classLoaderName":"app"}}

Put the preceding rolling file appender in all of the following files (/usr/share/moogsoft/config/logging and /usr/share/moogsoft/config/logging/servlets) to make the services output the json logs:
```
broker.log.json
integrations.log.json
controller.log.json
moog_farmd.log.json
moogsoft.log.json
graze.log.json  
moogpoller.log.json  
moogsvr.log.json  
situation_similarity.log.json  
toolrunner.log.json
```
For JSON layout options: https://logging.apache.org/log4j/2.x/log4j-core/apidocs/org/apache/logging/log4j/core/layout/JsonLayout.html

Other example logging formats

More than one appender type in json:

"RollingFile": [
                {
                    "name": "FILE",
                    "fileName": "${sys:MoogsoftLogFilename}",
                    "filePattern": "${sys:MoogsoftLogFilename}-%d{MM-dd-yy}-%i.gz",
                    "PatternLayout": {
                        "header": "${sys:MoogsoftLogHeader}",
                        "pattern": "{{LOG_PATTERN}}"
                    },
                    "Policies": {
                        "SizeBasedTriggeringPolicy": {
                            "size": "500M"
                        }
                    },
                    "DefaultRolloverStrategy": {
                        "max": "40"
                    }
                },
                {
                    "name": "ES_FILE",
                    "fileName": "/var/log/moogsoft/moog_farmd-ES.log",
                    "filePattern": "/var/log/moogsoft/moog_farmd-ES.log-%d{MM-dd-yy}-%i.gz",
                    "JsonLayout": {
                        "compact": true,
                        "eventEol": true,
                        "properties": true,
                        "stacktraceAsString": true,
                        "objectMessageAsJsonObject": true
                    },
                    "Policies": {
                        "SizeBasedTriggeringPolicy": {
                            "size": "500M"
                        }
                    },
                    "DefaultRolloverStrategy": {
                        "max": "5"
                    }
                }
            ],

Use a logger that is referencing more than one appender:

"Async" : {
                "name": "Async",
                "AppenderRef": [{
                    "ref": "FILE"
                }, {
                    "ref": "ES_FILE"
                }]
            }
        },
        "loggers": {
            "Logger": {
                "name": "com.moogsoft",
                "additivity": false,
                "AppenderRef": [{
                    "ref": "Async"
                }],
                "level": "warn"
            }
        }

In this section: