Page tree
Skip to end of metadata
Go to start of metadata

Introduction

Moogsoft AIOps services generate log files to report on their activity and provide and to serve as an audit trail, diagnostic device, or security measure. 

For information on the Logger MooBot module, click here.

Log file locations

AIOps service log files are located as follows:

apache-tomcat

  • Location of log files:
    /usr/share/apache-tomcat/logs

  • The primary log file is catalina.out

nginx

  • Location of log files:
    /var/log/nginx
  • The primary log file is error_log

LAMs

  • Default location of log files (unless stated otherwise in the LAM information):
    /var/log/moogsoft

moogfarmd

  • Location of log file:
    /var/log/moogsoft/moogfarmd.log

mysqld

  • Location of log file:
    /var/log/mysqld.log

rabbitmq-server

  • Default location of log file:
    /var/log/rabbitmq

elasticsearch

  • Location of log files:
    /var/log/elasticsearch/elasticsearch.log
     

Changing log levels

Set the level of logging for AIOps services as explained below. Available log levels are ALL, INFO, WARN, DEBUG and NONE.

apache-tomcat

  • Various logging level properties can be set in the following file:
    /usr/share/apache-tomcat/conf/logging.properties
  • Set the log levels of the Moog AIOps servlets such as graze, moogsvr, moogpoller and toolrunner in the following file: 
    /usr/share/apache-tomcat/webapps/<servlet>/WEB-INF/web.xml 
    Set the loglevel property as follows: 

    <param-name>loglevel</param-name>
    <param-value>WARN</param-value>
  • Restart the apache-tomcat service with the following command:

    service apache-tomcat restart

nginx

  • In the following file:
    /etc/nginx/conf/nginx.conf
    Set the LogLevel property as follows:

    LogLevel debug
  • Restart the nginx service with the following command:

    service nginx restart 

LAMs

  • In the following file:
    /etc/init.d/<lamd_name> (for example, socketlamd, trapdlamd)
    Set the the LOG_LEVEL property as follows:

    LOG_LEVEL=WARN
  • Restart the LAM with the following command:

    service <lamd_name> restart

moogfarmd

  • In the following file:
    /etc/init.d/moogfarmd
    Set the LOG_LEVEL property as follows:

    LOG_LEVEL=DEBUG
  • Restart the moogfarmd service with the following command:

    service moogfarmd restart 

mysqld

  • Logging is at the highest level by default. To remove a warning from the logs add the following to /etc/my.cnf

    log_warnings = 0
  • Restart the mysqld service with the following command:

    service mysqld restart

elasticsearch

Log rotation

Log data generated by AIOps services such as moogfarmd and LAMs are written to log files. Log data for each of these files is partitioned into separate files (rotated), either if the log file size reaches 500MB or daily (whichever occurs first). The partitioned log data files are compressed into GZ (gzip format) files, which are date-stamped with their filename. Rotated log files are retained for 40 days before being purged.

Log rotation prevents the size of log files becoming unmanageable and prevents log data loss when restarting AIOps. Log rotation parameters are configurable (see below).

Log rotation in Moogsoft AIOps is handled by the unix logrotate utility.

Example rotated log files

For the moog_farmd service, in location:
/var/log/moogsoft/

moogfarmd.log
moogfarmd.log.2017-11-04-1491868800.gz
moogfarmd.log.2017-11-05-1491955200.gz
moogfarmd.log.2017-11-06-1492041600.gz

In the above file list, moogfarmd.log is the current log file. The other three .gz files are the rotated log files for the last three days, date-stamped by their filename.

Configuring log file rotation

Log rotation of *.log files in the /var/log/moogsoft directory is controlled by a cron job that runs once per hour, on the hour:

[root@ldev04 moogsoft]# crontab -l
MOOGSOFT_HOME=/usr/share/moogsoft
JAVA_HOME=/usr/java/latest
0	*	*	*	* /usr/sbin/logrotate /etc/logrotate.d/moog-logrotate
*	*	*	*	* /usr/share/moogsoft/bin/utils/moog_indexer -b 1000 -t 5
0	*	*	*	* /usr/share/moogsoft/bin/events_analyser --incremental
0	3	*	*	0 /usr/share/moogsoft/bin/events_analyser

To set the log file size limit and/or retention period, edit file /etc/logrotate.d/moog-logrotate and set the size and maxage properties appropriately.

To set the frequency at which log rotation occurs edit the logrotate entry in crontab (using command crontab -e) as per standard cron notation.