Moogsoft Docs

Configure Logging

Moogsoft AIOps components generate log files to report their activity. As a Moogsoft AIOps administrator, you can refer to the logs to audit system usage or diagnose issues. In certain cases you may want to change logging levels based upon your specific environment or needs. See the Log Levels Reference for details.

Moogsoft AIOps uses Apache Log4j for logging. See the Log4j configuration documentation for more information.

You can edit the log configuration files at  $MOOGSOFT_HOME/config/logging/  There is a configuration file for every component or servlet in Moogsoft AIOps . These files can be found in $MOOGSOFT_HOME/config/logging/servlets/  and follow the naming convention  <servlet_name>.log.json  . These configuration files control the logs for the following:

•  events.log.json  : Logs for the proxy LAM.
•  graze.log.json  : Graze request logs.
•  moogpoller.log.json  : Moogpoller logs.
•  moogsvr.log.json  : Logs relating to SAML/LDAP authentication and internal API calls.
•  situation_similarity.log.json  : Situation Similarity servlet logs.
•  toolrunner.log.json  : Toolrunner servlet logs.

The other default configuration files include:

•  moog_farmd.log. json:  Configures logs for Moogfarmd process.
•  moogsoft.log.json  : Configures logs for all of the utilities.
•  integrations.log.json  : Configures logs for LAMs and integrations.

You can change log levels and make other configuration changes to components while they are running. Moogsoft AIOps reads any changes and applies them every two seconds.

You can configure these files to meet your requirements. Refer to the Log4j documentation to see the available properties or see Log Configuration File Examples .

Log Files by Component

The following reference provides information about the log files for the various Moogsoft AIOps components.

Apache Tomcat

Log location:  /usr/share/apache-tomcat/logs 

Primary log file:  catalina.out 

To change the logging level for the Moogsoft AIOps servlets which run in Tomcat, e dit the relevant files in  $MOOGSOFT_HOME/config/logging/servlets  . Nginx Log location:  /var/log/nginx  Primary log file:  error.log  To change the logging level for Nginx: 1. Edit  /etc/nginx/nginx.conf  . 2. Set the  LogLevel  property. For example to enable debug logging: LogLevel debug 3. Restart Ngnix. Moogfarmd By default Moogfarmd writes logs into a log file stored in  /var/log/moogsoft  if you have write permissions for this directory. Otherwise, the logs are written to $MOOGSOFT_HOME/log  . By default the log file takes the name of the HA address of the process. For example, MOO.moog_farmd.farmd_instance1.log.

MOO is the default HA cluster name in  $MOOGSOFT_HOME/config/system.conf  . If you change it the Moogfarmd log file path changes accordingly. Restart Moogfarmd after making any of the following configuration changes. To use a custom log configuration file for Moogfarmd: 1. Make a copy of the default Moogfarmd log configuration file and rename it, for example: cd$MOOGSOFT_HOME/config/logging
cp moog_farmd.log.json mymoog_farmd.log.json
2. Edit the new file according to your Moogfarmd logging requirements.
3. Edit the  configuration_file  property in the  log_config  section of  moog_farmd.conf  to point to the new file. For example:

log_config:
{
configuration_file: "mymoogfarmd.log.json"
}

To change the logging level for Moogfarmd:

1. Edit  /etc/init.d/moogfarmd  .
2. Set the  LogLevel  property. For available log levels see Log Levels Reference .  For example to enable logging at the debug level:

LOG_LEVEL=DEBUG

Alternatively, edit the level in  $MOOGSOFT_HOME/config/logging/moog_farmd.log.json  . For example: "configuration": { "ThresholdFilter": { "level": "trace" }, } To save Moogfarmd logs to a different location and/or filename, edit the Moogfarmd log configuration file located at $MOOGSOFT_HOME/config/logging/moog_farmd.log.json. 

For example:

"RollingFile":
{
"name"		: "FILE",
"fileName"	: "/var/log/moogsoft/Moogfarmd_test.log"
}

LAMs and Integrations

LAMs and integrations log their processing and data ingestion to two types of log files, process and capture.

Process Logs

LAMs and integrations record their activities as they ingest raw data. By default these process logs are written to a log file stored in  /var/log/moogsoft  if the user running the LAM has write permissions for this directory. Otherwise, the logs are written to  $MOOGSOFT_HOME/log  . By default the log file takes the name of the LAM or integration. For example, MOO.solarwinds_lam.log. The configuration of LAM process logs is specified in a file located at $MOOGSOFT_HOME/config/logging/integrations.log.json  .

To specify the log configuration for a particular LAM:

1. Make a copy of the default LAM log configuration file and rename it with the name of the LAM, for example:

cd $MOOGSOFT_HOME/config/logging cp integrations.log.json solarwinds_lam.log.json 2. Edit the file according to your LAM logging requirements. 3. Edit the  configuration_file  property in the  log_config  section of the LAM configuration file to point to the new file. For example: log_config: { configuration_file: "$MOOGSOFT_HOME/config/logging/solarwinds_lam.log.json"
}

If a polling integration or LAM fails to connect to the target system using the connection details in the UI or configuration file, Moogsoft AIOps creates an alert with critical severity and writes the details to the process log. The following example shows a log file entry for a failed Zabbix Polling integration with an invalid URL:

WARN : [target1][20190117 13:03:33.942 +0000] [CZabbixPollingTask.java:129] +|40001: An error response received
from Zabbix REST server: [Invalid URL provided [http://zabbixserver1/zabbix/api_jsonrpc.php] for User Login request]|+

The following error code raises a Moogsoft AIOps alert. The alert details are listed below:

External ID Type Class Severity Example Alert Description
40001 Internal Integrations Error Failed Connection Attempt Critical Failed Connection Attempt for target [target1] and destination [ http://zabbixserver1/zabbix/api_jsonrpc.php ]. This is attempt [1] out of [infinite].

If the integration or LAM polls successfully on the next attempt, the alert is cleared. If the integration or LAM is restarted to resolve the connection issue the alert is not cleared and must be handled manually.

Capture Logs

In addition to process logs, all LAMs except the Logfile LAM allow you to capture the raw data they receive. This feature is disabled by default. To enable it, edit the LAM's configuration file and uncomment the  capture_log  property in the agent section. The default path to the capture log files is  $MOOGSOFT_HOME/log/data-capture/<lam_name>.log  . An example agent section in a LAM configuration file is as follows: agent: { name : "SolarWinds", capture_log : "$MOOGSOFT_HOME/log/data-capture/solarwinds_lam.log"
}

MySQL

Log location:  /var/log/mysqld.log 

MySQL logging defaults to the highest level. To remove warnings from the MySQL log:

1. Edit  /etc/my.cnf  .

log_warnings = 0
3. Restart the MySQL service .

RabbitMQ

Log location:  /var/log/rabbitmq 

Refer to the RabbitMQ documentation for information on how to configure RabbitMQ.

Elasticsearch

Log location:  /var/log/elasticsearch/elasticsearch.log   

Refer to the Elasticsearch documentation for information on how to configure Elasticsearch.

Log Rotation

Moogfarmd, LAMs and integrations use a Java-based logging utility that automatically runs at startup to prevent log files becoming unmanageably large. The utility also prevents the loss of log data when you restart Moogsoft AIOps.

The utility compresses each rotated log into gzip (.gz) format and appends the filename with a date stamp. Rotated log files are retained for 40 days before they are purged.

The logging utility rotates the logs when the file size reaches 500MB by default. It rotates up to 40 files by default. This is controlled in by two properties under  RollingFile  and  Policies  in  $MOOGSOFT_HOME/config/logging/<component_log_file_name>.log.json  size The size limit of the log file in megabytes that triggers a log rotation. Type : Integer Default : 500M max The maximum number of files that Moogsoft AIOps can rotate. Type : Integer Default : 40 The default logger configuration appears in $MOOGSOFT_HOME/ config/logging/<component_log_file_name>.log.json  as follows:

"Policies":
{
"SizeBasedTriggeringPolicy":
{
"size": "500M"
}
},
"DefaultRolloverStrategy":
{
"max": "40"
}