Configure Logging
Moogsoft Enterprise components generate log files to report their activity. As a Moogsoft Enterprise administrator, you can refer to the logs to audit system usage or diagnose issues. In certain cases you may want to change logging levels based upon your specific environment or needs. See the Log Levels Reference for details.
Moogsoft Enterpriseuses Apache Log4j for logging. See the Log4j configuration documentation for more information.
Configure your log files
You can edit the log configuration files at $MOOGSOFT_HOME/config/logging/
There is a configuration file for every component or servlet in Moogsoft Enterprise. These files can be found in $MOOGSOFT_HOME/config/logging/servlets/
and follow the naming convention <servlet_name>.log.json
. These configuration files control the logs for the following:
events.log.json
: Logs for the proxy LAM.graze.log.json
: Graze request logs.moogpoller.log.json
: Moogpoller logs.moogsoft_bridge.log.json
: Adjusts the logging level for Moogsoft Bridgemoogsvr.log.json
: Logs relating to SAML/LDAP authentication and internal API calls.situation_similarity.log.json
: Situation Similarity servlet logs.toolrunner.log.json
: Toolrunner servlet logs.
The other default configuration files include:
moog_farmd.log.json:
Configures logs for Moogfarmd process.moogsoft.log.json
: Configures logs for all of the utilities.integrations.log.json
: Configures logs for LAMs and integrations.
You can change log levels and make other configuration changes to components while they are running. Moogsoft Enterprise reads any changes and applies them every two seconds.
You can configure these files to meet your requirements. Refer to the Log4j documentation to see the available properties or see Log Configuration File Examples.
Log files by component
The following reference provides information about the log files for the various Moogsoft Enterprise components.
Apache Tomcat
Log location: /usr/share/apache-tomcat/logs
RPM:
/usr/share/apache-tomcat/logs/catalina.out
Tarball:
$MOOGSOFT_HOME/cots/apache-tomcat/logs/catalina.out
Primary log file: catalina.out
To change the logging level for the Moogsoft Enterprise servlets which run in Tomcat, edit the relevant files in $MOOGSOFT_HOME/config/logging/servlets
.
vi $MOOGSOFT_HOME/config/logging/servlets/moogsvr.log.json ... "loggers": { "Logger": { "name": "com.moogsoft", "additivity": false, "AppenderRef": [{ "ref": "STDOUT" }], "level": "info" } } ...
Nginx
Log location: /var/log/nginx
Primary log file: error.log
To change the logging level for Nginx:
Edit
/etc/nginx/nginx.conf
.Set the
LogLevel
property. For example to enable debug logging:LogLevel debug
Restart Ngnix.
Moogfarmd
By default Moogfarmd and Ticketing integrations write logs into a log file stored in /var/log/moogsoft
if you have write permissions for this directory. Otherwise, the logs are written to $MOOGSOFT_HOME/log
. By default the log file takes the name of the HA address of the process. For example, MOO.moog_farmd.farmd_instance1.log.
MOO is the default HA cluster name in $MOOGSOFT_HOME/config/system.conf
. If you change it the Moogfarmd log file path changes accordingly.
Restart Moogfarmd after making any of the following configuration changes.
To use a custom log configuration file for Moogfarmd:
Make a copy of the default Moogfarmd log configuration file and rename it, for example:
cd $MOOGSOFT_HOME/config/logging cp moog_farmd.log.json mymoog_farmd.log.json
Edit the new file according to your Moogfarmd logging requirements.
Edit the
configuration_file
property in thelog_config
section ofmoog_farmd.conf
to point to the new file. For example:log_config: { configuration_file: "mymoogfarmd.log.json" }
To change the logging level for Moogfarmd, edit the file $MOOGSOFT_HOME/config/logging/moog_farmd.log.json
. For example:
"configuration": { "ThresholdFilter": { "level": "trace" }, }
You can also modify the log level using moog_farmd --loglevel
. There are two options:
Use
farmd_cntl --loglevel <level>
A runtime change; restarting farmd resets log level to match logger config.
Adjust the logger configuration
Adjusting moogfarmd log level (moog_farmd.log.json):
vi $MOOGSOFT_HOME/config/logging/moog_farmd.log.json ... "Logger": [ { "name": "com.moogsoft", "additivity": false, "AppenderRef": [{ "ref":"${sys:MoogsoftLogAppender}" }], "level": "info" } ...
See Moogfarmd Reference for more information.
To save Moogfarmd logs to a different location and/or filename, edit the Moogfarmd log configuration file located at $MOOGSOFT_HOME/config/logging/moog_farmd.log.json
. For example:
"RollingFile": { "name" : "FILE", "fileName" : "/var/log/moogsoft/Moogfarmd_test.log" }
LAMs and integrations
LAMs and monitoring integrations log their processing and data ingestion to two types of log files, process and capture. Ticketing integrations do not have dedicated log files, and instead log their processing and data to var/log/moogsoft/MOO.moog_farmd.log
. For more information, refer to the preceding section on Moogfarmd.
Process logs
LAMs and integrations record their activities as they ingest raw data. By default these process logs are written to a log file stored in /var/log/moogsoft
if the user running the LAM has write permissions for this directory. Otherwise, the logs are written to $MOOGSOFT_HOME/log
. By default the log file takes the name of the LAM or integration. For example, MOO.solarwinds_lam.log.
The configuration of LAM process logs is specified in a file located at $MOOGSOFT_HOME/config/logging/integrations.log.json
.
To specify the log configuration for a particular LAM:
Make a copy of the default LAM log configuration file and rename it with the name of the LAM, for example:
cd $MOOGSOFT_HOME/config/logging cp integrations.log.json solarwinds_lam.log.json
Edit the file according to your LAM logging requirements.
Edit the
configuration_file
property in thelog_config
section of the LAM configuration file to point to the new file. For example:log_config: { configuration_file: "$MOOGSOFT_HOME/config/logging/solarwinds_lam.log.json" }
If a polling integration or LAM fails to connect to the target system using the connection details in the UI or configuration file, Moogsoft Enterprise creates an alert with critical severity and writes the details to the process log. The following example shows a log file entry for a failed Zabbix Polling integration with an invalid URL:
WARN : [target1][20190117 13:03:33.942 +0000] [CZabbixPollingTask.java:129] +|40001: An error response received from Zabbix REST server: [Invalid URL provided [http://zabbixserver1/zabbix/api_jsonrpc.php] for User Login request]|+
The following error code raises a Moogsoft Enterprise alert. The alert details are listed below:
External ID | Type | Class | Severity | Example Alert Description |
---|---|---|---|---|
40001 | Internal Integrations Error | Failed Connection Attempt | Critical | Failed Connection Attempt for target [target1] and destination [http://zabbixserver1/zabbix/api_jsonrpc.php]. This is attempt [1] out of [infinite]. |
40002 | Internal Integrations Error | Failed Connection Error | Critical | Failed Connection Error [rabbitmq-host.com: nodename nor servname provided, or not known]. This is attempt [2] out of [infinite]. |
If the integration or LAM polls successfully on the next attempt, the alert is cleared. If the integration or LAM is restarted to resolve the connection issue the alert is not cleared and must be handled manually.
Capture logs
In addition to process logs, all LAMs except the Logfile LAM allow you to capture the raw data they receive. This feature is disabled by default. To enable it, edit the LAM's configuration file and uncomment the capture_log
property in the agent section. The default path to the capture log files is $MOOGSOFT_HOME/log/data-capture/<lam_name>.log
.
An example agent section in a LAM configuration file is as follows:
agent: { name : "SolarWinds", capture_log : "$MOOGSOFT_HOME/log/data-capture/solarwinds_lam.log" }
MySQL
Log location: /var/log/mysqld.log
MySQL logging defaults to the highest level. To remove warnings from the MySQL log:
Edit
/etc/my.cnf
.Add the following line:
log_warnings = 0
Restart the MySQL service.
RabbitMQ
Log location: /var/log/rabbitmq
Refer to the RabbitMQ documentation for information on how to configure RabbitMQ.
Elasticsearch
Log location: /var/log/elasticsearch/elasticsearch.log
.
Refer to the Elasticsearch documentation for information on how to configure Elasticsearch.
Hazelcast and Kryo
Moogsoft Enterprise uses two libraries for persistence: Hazelcast and Kryo. You can configure the logging for these components in the file $MOOGSOFT_HOME/config/logging/moog_farmd.log.json
.
The logging level is set to WARN by default. Logs are written to the process log file.
Topologies
When you create, update and delete topologies and their nodes and links, Apache Tomcat logs the details in its primary log file catalina.out. An example log entry at INFO
level is as follows:
INFO : [Topologies Reporter Thread][20200225 16:48:25.105 +0000] [CReporterThread.java:142] +|Topologies server handled [200] topologies requests in the last [60] seconds.|+
Example errors at WARN
level:
“Unable to replace topology as topology to be replaced is not valid: [physical]" (WARN) "Unable to replace topology [physical] as replacing topology is not valid: [network] (WARN) "Unable to replace topology [%s] - topology [%s] has not been updated" (WARN) "Failed to get all nodes for topology [physical] as it does not exist" (WARN)
Vertex Entropy
The Graph Analyser process runs automatically as part of the Housekeeper Moolet, to calculate Vertex Entropy for your topological nodes. The Graph Analyser process logs details of its processing to the Moogfarmd log file. Example log entries:
Starts processing for a topology named "physical":
Starting graph analysis of topology [physical] ... Setting topology state from [OUTDATED] -> [PROCESSING] for [physical]
Processes nodes in the "physical" topology:
Performing analysis on 15023 nodes in topology [physical]
Example errors:
"Skipping topology [physical] as it has no nodes." (INFO) "Failed to update vertex entropy values for topology [physical]" (WARN) "Topology with name [physical] does not exist" (WARN) "Topology [physical] state is [OUTDATED], skipping pending re-analysis" (INFO)
Completes processing:
Completed graph analysis of topology [physical], time elapsed 10s
Log rotation
Moogfarmd, LAMs and integrations use a Java-based logging utility that automatically runs at startup to prevent log files from becoming unmanageably large. The utility also prevents the loss of log data when you restart Moogsoft Enterprise.
The logging utility rotates the logs when the file size reaches 500MB by default. It rotates up to 40 files per day by default. This is controlled in by two properties under RollingFile
and Policies
in $MOOGSOFT_HOME/config/logging/<component_log_file_name>.log.json
.
size
The size limit of the log file in megabytes that triggers a log rotation.
Type: Integer
Default: 500M
max
The maximum number of files that Moogsoft Enterprise can rotate.
Type: Integer
Default: 40
The default logger configuration appears in $MOOGSOFT_HOME/config/logging/<component_log_file_name>.log.json
as follows:
"Policies": { "SizeBasedTriggeringPolicy": { "size": "500M" } }, "DefaultRolloverStrategy": { "max": "40" }
Moogsoft Bridge
Moogsoft Bridge uses a store and forward architecture to push events and other messages from a local RabbitMQ cluster to the Message Bus.
Moogsoft Bridge outputs logs to:
/var/log/moogsoft/moogsoft_bridge.log
for root users$MOOGSOFT_HOME/log/moogsoft_bridge.log
for non-root users
See Moogsoft Bridge for more information.