Moogsoft Docs


Moogsoft AIOps Installation and Upgrade

Yum: Incorrect Moogsoft AIOps version

If an incorrect or outdated version is offered when installing Moogsoft AIOps your Yum cache may need cleaning.

Run the following command and then re-attempt the installation:

yum clean all

Yum: HTTP Error 401 - unauthorized

If an attempt to install Moogsoft AIOps fails with an error such as the following, check your username and password credentials are correct in the configured Moogsoft Yum repository.

https://<username>:<password> [Errno 14] HTTP Error 401 - Unauthorized

Yum: Problem making SSL connection

If an attempt to install Moogsoft AIOps fails with an error such as the following:

https://<username>:<password> [Errno 14] problem making ssl connection
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: moogsoft-aiops. Please verify its path and try again

You may need to update the NSS p ackages on your server. Run the following command and then re-attempt the installation.

yum -y update nss

Yum: MySQL conflict

If an attempt to install Moogsoft AIOps fails with an error such as the following, it may be caused by a conflict with the MySQL libraries on the host.

Running rpm_check_debug
Running Transaction Test
Transaction Check Error:
  file /usr/lib64/mysql/ from install of mysql-community-libs-compat-5.7.22-2.el6.x86_64 conflicts with file from package compat-mysql51-5.1.54-1.el6.remi.x86_64
  file /usr/lib64/mysql/ from install of mysql-community-libs-compat-5.7.22-2.el6.x86_64 conflicts with file from package compat-mysql51-5.1.54-1.el6.remi.x86_64
Error Summary

Run the following bash commands to allow the product to be installed successfully:

echo "remove compat-mysql51" > /tmp/moog_yum_shell.txt
echo "install mysql-community-libs-compat-5.7.22" >> /tmp/moog_yum_shell.txt
echo "install mysql-community-client-5.7.22" >> /tmp/moog_yum_shell.txt
echo "install mysql-community-libs-5.7.22" >> /tmp/moog_yum_shell.txt
echo "install mysql-community-server-5.7.22" >> /tmp/moog_yum_shell.txt
echo "install mysql-community-common-5.7.22" >> /tmp/moog_yum_shell.txt
echo "groupinstall moogsoft" >> /tmp/moog_yum_shell.txt
echo "run" >> /tmp/moog_yum_shell.txt

cat /tmp/moog_yum_shell.txt | yum shell -y

The above error is most likely to occur on hosts on which some MySQL components are already installed. The issue is often seen when trying to install moogsoft-db on a system with an existing MySQL installation.

Nginx: Installation problems

Error: Package: moogsoft-ui-7.1.0-123.x86_64 (moogsoft-aiops) Requires: nginx >= 1.14.0

If you encounter the following error when attempting to install Moogsoft AIOps:

Requires: nginx >= 1.14.0
---> Package moogsoft-ui.x86_64 0:7.1.0-123 will be an update
--> Processing Dependency: nginx >= 1.14.0 for package: moogsoft-ui-7.1.0-123.x86_64
--> Finished Dependency Resolution
Error: Package: moogsoft-ui-7.1.0-123.x86_64 (moogsoft-aiops)
Requires: nginx >= 1.14.0

Try using --skip-broken to work around the problem, or try:

rpm -Va --nofiles --nodigest

Alternatively, you could manually install the Nginx repo with the following command and then re-attempt the Moogsoft AIOps installation.

rpm -Uvh

Single-Host Installation for Non-Root

If you cannot access the UI from your host machine, check your firewall and if you're listening on the right ports:

  1. To see if your firewall is enabled:


    This returns the status disabled if the firewall is disabled.

  2. To disable an active firewall:

    setenforce 0
  3. To check whether a port is open:

    firewall-cmd --zone=public --query-port=8443/tcp
    firewall-cmd --zone=public --query-port=8080/tcp
  4. To open a port:

    firewall-cmd --permanent --zone=public --add-port=8080/tcp
    firewall-cmd --permanent --zone=public --add-port=8443/tcp
    firewall-cmd --reload

Mobile Troubleshooting

Moogsoft AIOps includes a self-signed certificate by default. If you want to use Moogsoft AIOps for Mobile on an iPhone, you need to add a valid SSL certificate. This is because W ebSockets do not work on iOS with self-signed certificates.

If a valid root CA certificate is not added, a 'Connection Error' appears at login and Moogsoft AIOps for Mobile does not work.

Nginx: SSL configuration

To apply a valid certificate to Nginx, go to the Nginx configuration directory and edit moog-ssl.conf :

cd common/config/nginx vi moog-ssl.conf
vi moog-ssl.conf

Change the default self-signed certificate and key locations to point to the valid root certificate and key:

#ssl_certificate /etc/nginx/ssl/certificate.pem;
#ssl_certificate_key /etc/nginx/ssl/certificate.key;

ssl_certificate /etc/certificates/GeoTrust_Universal_CA.crt;
ssl_certificate_key /etc/certificates/GeoTrust_Universal_CA.key;

Restart Nginx with this command:

systemctl restart nginx

Moogsoft AIOps Processes

Required services for a functional production system

Service name



Web server that contains the servlets that provide the Moogsoft AIOps user interface.

nginx Web server that handles security, such as Moogsoft AIOps login, PHP and HTTP/SSL implementation.
LAMs, for example:

Link Access Modules used for data ingestion. Service names may differ.
At least one instance of a LAM is required for data feed.
moogfarmd Core Moogsoft AIOps system application.

Database containing Moogsoft AIOps data (database schemas etc.)


Message system for Moogsoft AIOps.


Elasticsearch service for UI search feature.

To check the status, stop, start or restart a service run one of the following:

service <service-name> status
service <service-name> stop
service <service-name> start
service <service-name> restart

Location of installation and log files

See Configure Logging .

Generic Moogsoft AIOps process not starting

No space left on the disk

  • Check the file system with the command df -m
  • Look for partitions that are full

<Service> not found
Error while loading shared libraries

  • Environmental variables may not be properly set up for your shell
  • Run the environment and check the location set for $MOOGSOFT_HOME

Moogfarmd not starting

Configuration parsing error

  • +|No config present|+ message in /var/log/moogsoft/moogfarmd.log
  • Points to a syntax error in $MOOGSOFT_HOME/config/moog_farmd.conf
  • Check the config file for punctuation mistakes


    Look for:
    • Missing commas
    • Unbalanced quotes
    • Missing {' or '}
    Use # to comment out code instead of /* and */

  • Edit moog_farmd.conf and then restart the service


Also see Message System Deployment .

RabbitMQ: Not starting - "No such user"

  • No such user message in /var/log/rabbitmq/startup_err
  • Check /etc/passwd for user rabbitmq with the following command:
    grep rabbitmq /etc/passwd
  • If no user is found, add the following to /etc/passwd :
    rabbitmq:x:491:488:RabbitMQ messaging server:/var/lib/rabbitmq:/bin/bash

RabbitMQ: Not starting - "Failed to create aux thread"

  • Failed to create aux thread message in /var/log/rabbitmq/startup_err
  • This is most likely a ulimit issue for the RabbitMQ user

  • Check ulimit settings for the RabbitMQ user by running the following command as root:

    su - rabbitmq
    bash-4.1$ ulimit -a
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 515675
    max locked memory       (kbytes, -l) 64
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 1024
    pipe size            (512 bytes, -p) 8
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 10240
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 1024
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited
  • The above example shows ulimit settings that are likely too low for RabbitMQ

  • As per instructions here it may be appropriate to increase the ulimit settings for "open files" and "max user processes" to at least 4096 for development/QA environments and 65536 for production environments

RabbitMQ: Unable to create connection

  • "Unable to create connection" message appearing in a LAM, Moogfarmd, or Apache Tomcat logs
  • This indicates that the process unable to connect to the Message Bus zone in RabbitMQ
  • Check that the RabbitMQ server is running:

    service rabbitmq-server status
  • If the service is not running start it:

    service rabbitmq-server start
  • If the service is running, check that the zone used in Moogsoft AIOps matches the vhost in RabbitMQ. List the zones (vhosts) added in RabbitMQ:

    rabbitmqctl list_vhosts
  • Check the MooMS section in /usr/share/moogsoft/config/system.conf for the zone used in Moogsoft AIOps.

  • If the zone is missing, add the zone (vhost) to RabbitMQ manually (see Message System Deployment ).
  • Restart the affected process.

LAMs fail to start from command line

If LAMs run from the command line or as a service result in the following error:

[root@moogbox2 bin]# ./socket_lam 
./socket_lam: error while loading shared libraries: cannot open shared object file: No such file or directory

it may be because /usr/java/jdk1.8.0_171/jre/lib/amd64/server has not been added to the LD_LIBRARY_PATH .

To run the LAMs via a command line, a change t he LD_LIBRARY_PATH to be as follows (t he default initd files contain this setting):

export LD_LIBRARY_PATH=$MOOGSOFT_HOME/lib:/usr/GNUstep/Local/Library/Libraries:/usr/GNUstep/System/Library/Libraries:$JAVA_HOME/jre/lib/amd64/server

Generic LAM not starting

Configuration parsing error

  • "Unable to parse configuration file" message in /var/log/moogsoft/<lamd_name>.log indicates a syntax error in the LAM configuration file.
  • Check the config file for syntax mistakes.


    Look for:
    • Missing commas
    • Unbalanced quotes

  • Compare the configuration file to a default configuration file for the same LAM. Use the following command to locate differences in the files:

    diff -y <current_lam>.conf <default_lam>.conf | less
  • Edit <current_lam>.conf to resolve any syntax errors and restart the LAM.

Connection refused error

  • "Failed to connect to [host:port]: Connection refused" error in /var/log/moogsoft/<lamd_name>.log means that the port specified in the LAM configuration file is already in use.
  • Use the following command to check that the LAM is not already started:

    ps -ef | grep <lamd_name>
  • Check that another process is not already bound to the port.
  • If required, edit the port setting for the LAM in the configuration file and restart the LAM.

Unresolvable hostname error

  • "Host [hostname] unresolvable" error in /var/log/moogsoft/<lamd_name>.log means that the LAM is unable to resolve the host, or the hostname is incorrectly set.
  • In the LAM config file, check the address property and correct any errors.
  • Check that the /etc/hosts file contains an entry for the specified hostname.

Failed to open file error

  • "Failed to open file [<path to file>] error in /var/log/moogsoft/<lamd_name>.log means that the LAM is unable to locate a file specified in the LAM configuration file.
  • Locate the missing file in $MOOGSOFT_HOME/bots/lambots or $MOOGSOFT_HOME/contrib .
  • Update the LAM configuration file with the correct file path and restart the LAM.

Generic LAM not processing

Syntax error in Presend filter

  • In the LAM configuration file, locate the presend filter file name.
  • You can do this with a JavaScript editor. Check the code to locate any syntax errors.
  • Or if you have installed Node.js you can run the following command to locate the incorrect line in the code:

    node $MOOGSOFT_HOME/bots/lambots/<path_to_filter_file>.js
  • Edit $MOOGSOFT_HOME/bots/lambots/<path_to_filter_file>.js to resolve the error and r estart the LAM.

Empty columns in alert lists

  • Empty columns in alert lists may indicate incorrect field mapping assignments.
  • Check field mappings at the bottom of the LAM configuration file.
  • Edit the configuration file to properly map the field to the column name and then restart the LAM.

Socket LAM not processing

Processing mode set incorrectly

  • The LAM may be set to Server mode rather than Client mode in the configuration file. For a description of mode types see Socket LAM .
  • Set the mode correctly in the configuration file and restart the LAM.

JSON feed not processing

  • The JSON string is incorrectly formatted. For example, event data contains nested JSON.
  • Run the LAM in debug mode and look for nested JSON.
  • Either modify the event data or edit the presend filter to match values in the nested JSON.

Logfile LAM not starting

No input file

  • "Could not stat file [-1] error: [Bad file descriptor]" error in /var/log/moogsoft/<lamd_name>.log shows that the Logfile LAM cannot locate the log file to read.
  • In the LAM configuration file check the target setting and confirm the file path to the target log file.

REST LAM not starting

Missing SSL path

  • +|No file path specified|+ message in /var/log/moogsoft/<lamd_name>.log
  • "No file path specified" error in /var/log/moogsoft/<lamd_name>.log indicates a missing SSL path in the LAM configuration file.
  • Check the value of the use_ssl property has been set correctly.
  • If using SSL, check the following properties are correctly set:

Nginx fails on startup if IPv6 not configured

Comment out the following references in two configuration files:

  1. Go to /etc/nginx/conf.d
  2. Edit out IPv6 references with a hash # :

    Configuration File Section
    moog-default.conf listen       80 default_server;
    #listen      [::]:80 default_server;
    moog-ssl.conf listen       443 ssl default_server;
    #listen      [::]:443 ssl;

User Interface (UI) Issues

Unavailable UI Login Page

  • Check that port 443 is not being blocked by the firewall on the server.
  • Check that the Nginx service is running with command:

    service nginx status
  • Check that Nginx is listening on port 443. Example expected output:

    netstat -anp|grep 443
    tcp        0      0       *                   LISTEN      42356/nginx         
    tcp        0      0 :::443                      :::*                        LISTEN      42356/nginx 

Login fails with "You could not be logged in. Please try again."

Apache-tomcat service not running

  • Check the apache-tomcat service is running:

    service apache-tomcat status

Communication problem between the UI and MySQL database

  • Check the MySQL service is running:

    service mysqld status
  • If MySQL is running on a different server, check that it is accessible from the Moogsoft AIOps web server and the required permissions have been applied.

Authentication problem between the UI and MySQL database

  • Check that the user exists in the MySQL moogdb.users table.
  • Check that the username and password used for authentication are correct.

Unable to save UI Integration

Error "Unable to save integrations data: Application was unable to save integration configuration".

Moogsoft AIOps only checks for messages in English. Check the system locale on the machine running Moogsoft AIOps. If it is set to a language other than English try the following:

  1. Switch to the Tool Runner user account:

    su moogtoolrunner
  2. Set the shell language to English for the Tool Runner user by editing the ~/.bashrc file. For example, add this line to the end of the file:

  3. Restart Apache Tomcat:

    service apache-tomcat restart
  4. Try to save the UI integration again.


See Configure Search and Indexing for more information.

ElasticSearch not running or generating errors (such as MySQL connection problems)

  • Check that the Elasticsearch service is running:

    service elasticsearch status
  • Any errors are written to /var/log/elasticsearch/elasticsearch.log

Tomcat cannot connect to Elasticsearch

  • Check /usr/share/apache-tomcat/logs/catalina.out for any errors when attempting a search from the UI.

Cron job errors

  • Check that cron job that runs the moog_indexer (created by the script to re-index against the Moogsoft AIOps database on a once-a-minute basis) exists and is not generating any warnings or errors.
  • List the configured cron jobs:

    crontab -l
  • Errors are written to /var/log/cron

  • Depending on the intervals at which Elasticsearch re-indexes against the Moogsoft AIOps database, it is possible that new alerts, Situations, threads or comments have not yet been indexed, and so will not be searchable.

  • To change the interval manually:

    crontab -ed

Elasticsearch fails to start with /tmp directory permission problems

Elasticsearch fails to start with "java.lang. UnsatisfiedLinkError: /tmp/jna--<blah>" error. For example:

[2017-08-07T14:14:31,173][WARN ][o.e.b.Natives] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna3872404023206022895.tmp: /tmp/jna--1985354563/jna3872404023206022895.tmp: failed to map segment from shared object: Operation not permitted
   at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_171]
   at java.lang.ClassLoader.loadLibrary0( ~[?:1.8.0_171]
   at java.lang.ClassLoader.loadLibrary( ~[?:1.8.0_171]
   at java.lang.Runtime.load0( ~[?:1.8.0_171]
   at java.lang.System.load( ~[?:1.8.0_171]
   at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath( ~[jna-4.2.2.jar:4.2.2 (b0)]
   at com.sun.jna.Native.loadNativeDispatchLibrary( ~[jna-4.2.2.jar:4.2.2 (b0)]
   at com.sun.jna.Native.<clinit>( ~[jna-4.2.2.jar:4.2.2 (b0)]
   at java.lang.Class.forName0(Native Method) ~[?:1.8.0_171]
   at java.lang.Class.forName( ~[?:1.8.0_171]
   at org.elasticsearch.bootstrap.Natives.<clinit>( [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Bootstrap.initializeNatives( [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Bootstrap.setup( [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Bootstrap.init( [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Elasticsearch.init( [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Elasticsearch.execute( [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.cli.SettingCommand.execute( [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.cli.Command.mainWithoutErrorHandling( [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.cli.Command.main( [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Elasticsearch.main( [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Elasticsearch.main( [elasticsearch-5.6.9.jar:5.6.9]

This is most likely due to the noexec directive in the /tmp mount. The solution is to remove the noexec directive, if it is practical to do so:

sudo mount /tmp -o remount,exec

Or set the following in /etc/sysconfig/elasticsearch :


Restart the Elasticsearch service after either of the above changes.

Processing Issues

No alerts created

License not applied

  • Ensure that the product license has been applied.

RabbitMQ not running

  • Check that the RabbitMQ server is running:

    service rabbitmq-server status

Alert Builder not started

  • Check that the "run on startup" setting for Alert Builder in the Moogfarmd configuration file is set to true.
  • Check the Moogfarmd log /var/log/moogsoft/moogfarmd.log

Alert Builder misconfiguration

  • Check the Alert Builder Moolet for syntax errors or errors in logic.

LAM misconfiguration

  • Check that the LAM is correctly parsing/mapping the data feed.
  • Check that the LAM is not performing any post-event processing that may be filtering out the events in the associated LAMbot.

No Situations Created

License not applied

  • Ensure that the product license has been applied.

Sigaliser Moolet not running

  • Check the Moogfarmd configuration file to ensure that the "run on startup" property for the Sigaliser used is set to true.

Incorrectly set "process output of" property

  • In the Moogfarmd configuration file, check that the "process output of" setting for the Sigaliser used lists the correct Moolet.
  • Check that the Moolet listed in the "process output of" property is running.

Sigaliser Moolet configuration is too restrictive or too open

If Moolet settings are too restrictive or too open they may not produce Situations. See Sigaliser Moolet for more information.

Error Messages and Status Codes

HTTP Status Codes

The following list describes the HTTP status codes used by Moogsoft AIOps:



Application Usage



Sent to the browser when a request has been processed successfully.



Sent to the browser when the request was well constructed but the server was unable to process it.


Bad Request

Sent to the browser when request parameters are invalid or missing.



Returned when the user is not authenticated.



Returned when the user is forbidden to perform a specific action.


Not Found

Sent to browser when a requested servlet path could not be found.



A conflict has occurred that can potentially be resolved by the user. For example an incorrect template name.


Internal Server Error

Sent to the browser when there is an unexpected exception or problem on the server side.


Service Unavailable

Used when the server is overloaded. This is used by Tool Runner when no execution threads are available.

Application Status Codes

The application status codes present in the response payload are defined in the following ranges. A single range may map to more than one HTTP status.

Code Range


HTTP Status


System errors.

500, 503


Validation errors.



Security errors.

401, 403


Conflict errors.


System Error Status Codes

The following system error status codes are defined:



HTTP Status


General server error.



Service unavailable.



Database unavailable.



Service busy.


Validation Error Status Codes

The following validation error status codes are defined:



HTTP Status


General validation error.  The actual parameter specific error will be defined in the additional body.



Parameter missing (not included in post/get) or no value provided (parameter included but empty value).



Parameter format error (a value that could not be converted to desired type).



Illegal value (a value that is not in the list of allowed values).



Parameter out of range (a value that is outside the range of allowed values).


Security Error Status Codes

The following security error status codes are defined:



HTTP Status


General security error.



User not authenticated.



User not permitted to perform action.



Header auth token not authenticated.


Conflict Error Status Codes

The following conflict error status codes are defined:



HTTP Status


General conflict error.



Duplicate name. For example duplicate template name, invite already sent.


Content Error Status Codes

The following content error status codes are defined:



HTTP Status


General content error.



Content not supported.



Error decoding content.



Content accepted but cached, processing not guaranteed.



Server being ahead of the UI (in terms of the Moogsoft AIOps version).



UI being ahead of the UI (in terms of the Moogsoft AIOps version).


JSON Response Body (Non-HTTP 200)

The response body will contain the detailed application error status information.  The additional information is optional for some status codes.

The format of the JSON is as follows:

	"message" 		: "User friendly message",
	"statusCode" 	: <appStatusCode>,
	"additional" 	: <jsonAdditionalInfo> 
	"message" 		: "User friendly message",
	"statusCode" 	: <appStatusCode> 

The high-level user friendly message will take the following format:

“<operation> failed due to <cause>”. For example “Create situation failed due to invalid parameters”.

JSON Addition Information

The format of the JSON additional information dependa on the HTTP status code and application status code. For certain status codes the additional information may not be present. Although the HTTP status code (400 or 500) is enough to determine the additional information structure it is best to use the application status code (as in the future more than one range could apply to a single HTTP status code).

  • HTTP 400 -> Application Status Code 2000-2999

  • HTTP 409 -> Application Status Code 4000-4999

  • HTTP 500, 503 -> Application Status Code 1000-1999

Application Status Code [1000-1999]

Optional additional information

A debug message typically containing the exception message:

{ "debugMessage" : " Connection refused" }

Application Status Code [2000-2999] or [4000-4999]

Mandatory additional information

List of the bad parameter(s) and the error information:

	"name" 		: "parameterName",
	"value" 	: <badValue>,
	"errorCode"	: <errorCode>

Ngnix Error Messages

97: Address family not supported by protocol

This an IPv6 error. Either configure IPv6 or comment it out.

To comment it out, go to /etc/nginx.conf.d and comment out the IPv6 references with a hash #:

Configuration File Section
moog-default.conf listen       80 default_server;
#listen      [::]:80 default_server;
moog-ssl.conf listen       443 ssl default_server;
#listen      [::]:443 ssl;

502 Bad Gateway (nginx/1.14.0)

Fails to load resource and server responds with a status of 502 (Bad Gateway).

Check that you have used the correct components for the version you are installing. -z MY_ZONE -bz MY_ZONE -d SERVER1:3306 -sd SERVER1:3306 -bz MY_ZONE -d SERVER1:3306 -otwxfz MY_ZONE -d SERVER1:3306