Troubleshooting

Yum: Incorrect Moogsoft AIOps version

If an incorrect or outdated version is offered when installing Moogsoft AIOps your Yum cache may need cleaning.

Run the following command and then re-attempt the installation:

yum clean all

Yum: HTTP Error 401 - unauthorized

If an attempt to install Moogsoft AIOps fails with an error such as the following, check your username and password credentials are correct in the configured Moogsoft Yum repository.

https://<username>:<password>@speedy.moogsoft.com/repo/aiops/latest/repodata/repomd.xml: [Errno 14] HTTP Error 401 - Unauthorized

Yum: Problem making SSL connection

If an attempt to install Moogsoft AIOps fails with an error such as the following:

https://<username>:<password>@speedy.moogsoft.com/repo/aiops/latest/repodata/repomd.xml: [Errno 14] problem making ssl connection
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: moogsoft-aiops. Please verify its path and try again

You may need to update the NSS packages on your server. Run the following command and then re-attempt the installation.

yum -y update nss

Yum: MySQL conflict

If an attempt to install Moogsoft AIOps fails with an error such as the following, it may be caused by a conflict with the MySQL libraries on the host.

Running rpm_check_debug
Running Transaction Test
Transaction Check Error:
  file /usr/lib64/mysql/libmysqlclient.so.16.0.0 from install of mysql-community-libs-compat-5.7.22-2.el6.x86_64 conflicts with file from package compat-mysql51-5.1.54-1.el6.remi.x86_64
  file /usr/lib64/mysql/libmysqlclient_r.so.16.0.0 from install of mysql-community-libs-compat-5.7.22-2.el6.x86_64 conflicts with file from package compat-mysql51-5.1.54-1.el6.remi.x86_64
Error Summary
-------------

Run the following bash commands to allow the product to be installed successfully:

echo "remove compat-mysql51" > /tmp/moog_yum_shell.txt
echo "install mysql-community-libs-compat-5.7.22" >> /tmp/moog_yum_shell.txt
echo "install mysql-community-client-5.7.22" >> /tmp/moog_yum_shell.txt
echo "install mysql-community-libs-5.7.22" >> /tmp/moog_yum_shell.txt
echo "install mysql-community-server-5.7.22" >> /tmp/moog_yum_shell.txt
echo "install mysql-community-common-5.7.22" >> /tmp/moog_yum_shell.txt
echo "groupinstall moogsoft" >> /tmp/moog_yum_shell.txt
echo "run" >> /tmp/moog_yum_shell.txt

cat /tmp/moog_yum_shell.txt | yum shell -y

The above error is most likely to occur on hosts on which some MySQL components are already installed. The issue is often seen when trying to install moogsoft-db on a system with an existing MySQL installation.

Nginx: Installation problems

Error: Package: moogsoft-ui-7.2.0-123.x86_64 (moogsoft-aiops) Requires: nginx >= 1.14.0

If you encounter the following error when attempting to install Moogsoft AIOps:

Requires: nginx >= 1.14.0
---> Package moogsoft-ui.x86_64 0:7.2.0-123 will be an update
--> Processing Dependency: nginx >= 1.14.0 for package: moogsoft-ui-7.2.0-123.x86_64
--> Finished Dependency Resolution
Error: Package: moogsoft-ui-7.2.0-123.x86_64 (moogsoft-aiops)
Requires: nginx >= 1.14.0

Try using --skip-broken to work around the problem, or try:

rpm -Va --nofiles --nodigest

Alternatively, you could manually install the Nginx repo with the following command and then re-attempt the Moogsoft AIOps installation.

rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-2.el7.ngx.noarch.rpm

If you cannot access the UI from your host machine, check your firewall and if you're listening on the right ports:

Moogsoft AIOps includes a self-signed certificate by default. If you want to use Moogsoft AIOps for Mobile on an iPhone, you need to add a valid SSL certificate. This is because WebSockets do not work on iOS with self-signed certificates.

If a valid root CA certificate is not added, a 'Connection Error' appears at login and Moogsoft AIOps for Mobile does not work.

Nginx: SSL configuration

To apply a valid certificate to Nginx, go to the Nginx configuration directory and edit moog-ssl.conf :

cd common/config/nginx vi moog-ssl.conf
vi moog-ssl.conf

Change the default self-signed certificate and key locations to point to the valid root certificate and key:

#ssl_certificate /etc/nginx/ssl/certificate.pem;
#ssl_certificate_key /etc/nginx/ssl/certificate.key;

ssl_certificate /etc/certificates/GeoTrust_Universal_CA.crt;
ssl_certificate_key /etc/certificates/GeoTrust_Universal_CA.key;

Restart Nginx with this command:

systemctl restart nginx

Required services for a functional production system

To check the status, stop, start or restart a service run one of the following:

service <service-name> status
service <service-name> stop
service <service-name> start
service <service-name> restart

Location of installation and log files

See Configure Logging.

Generic Moogsoft AIOps process not starting

No space left on the disk

<Service> not found Error while loading shared libraries

Moogfarmd not starting

Configuration parsing error

RabbitMQ

Also see Message System Deployment.

RabbitMQ: Not starting - "No such user"

RabbitMQ: Not starting - "Failed to create aux thread"

RabbitMQ: Unable to create connection

LAMs fail to start from command line

If LAMs run from the command line or as a service result in the following error:

[root@moogbox2 bin]# ./socket_lam 
./socket_lam: error while loading shared libraries: libjvm.so: cannot open shared object file: No such file or directory

it may be because /usr/java/jdk1.8.0_171/jre/lib/amd64/server has not been added to the LD_LIBRARY_PATH.

To run the LAMs via a command line, a change the LD_LIBRARY_PATH to be as follows (the default initd files contain this setting):

export LD_LIBRARY_PATH=$MOOGSOFT_HOME/lib:/usr/GNUstep/Local/Library/Libraries:/usr/GNUstep/System/Library/Libraries:$JAVA_HOME/jre/lib/amd64/server

Generic LAM not starting

Configuration parsing error

Connection refused error

Unresolvable hostname error

Failed to open file error

Generic LAM not processing

Syntax error in Presend filter

Empty columns in alert lists

Socket LAM not processing

Processing mode set incorrectly

JSON feed not processing

Logfile LAM not starting

No input file

REST LAM not starting

Missing SSL path

Nginx fails on startup if IPv6 not configured

Comment out the following references in two configuration files:

Unavailable UI Login Page

Login fails with "You could not be logged in. Please try again."

Apache-tomcat service not running

Communication problem between the UI and MySQL database

Authentication problem between the UI and MySQL database

Search/Elasticsearch

See Configure Search and Indexing for more information.

ElasticSearch not running or generating errors (such as MySQL connection problems)

Tomcat cannot connect to Elasticsearch

Cron job errors

Elasticsearch fails to start with /tmp directory permission problems

Elasticsearch fails to start with "java.lang. UnsatisfiedLinkError: /tmp/jna--<blah>" error. For example:

[2017-08-07T14:14:31,173][WARN ][o.e.b.Natives] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna3872404023206022895.tmp: /tmp/jna--1985354563/jna3872404023206022895.tmp: failed to map segment from shared object: Operation not permitted
   at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_171]
   at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_171]
   at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_171]
   at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_171]
   at java.lang.System.load(System.java:1086) ~[?:1.8.0_171]
   at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851) ~[jna-4.2.2.jar:4.2.2 (b0)]
   at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) ~[jna-4.2.2.jar:4.2.2 (b0)]
   at com.sun.jna.Native.<clinit>(Native.java:140) ~[jna-4.2.2.jar:4.2.2 (b0)]
   at java.lang.Class.forName0(Native Method) ~[?:1.8.0_171]
   at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_171]
   at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:203) [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) [elasticsearch-5.6.9.jar:5.6.9]
   at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) [elasticsearch-5.6.9.jar:5.6.9]

This is most likely due to the noexec directive in the /tmp mount. The solution is to remove the noexec directive, if it is practical to do so:

sudo mount /tmp -o remount,exec

Or set the following in /etc/sysconfig/elasticsearch:

ES_JAVA_OPTS="-Djna.tmpdir=/var/lib/elasticsearch/tmp"

Restart the Elasticsearch service after either of the above changes.

No alerts created

License not applied

RabbitMQ not running

Alert Builder not started

Alert Builder misconfiguration

LAM misconfiguration

No Situations Created

License not applied

Sigaliser Moolet not running

Incorrectly set "process output of" property

Sigaliser Moolet configuration is too restrictive or too open

If Moolet settings are too restrictive or too open they may not produce Situations.

HTTP Status Codes

The following list describes the HTTP status codes used by Moogsoft AIOps:

Application Status Codes

The application status codes present in the response payload are defined in the following ranges. A single range may map to more than one HTTP status.

System Error Status Codes

The following system error status codes are defined:

Validation Error Status Codes

The following validation error status codes are defined:

Security Error Status Codes

The following security error status codes are defined:

Conflict Error Status Codes

The following conflict error status codes are defined:

Content Error Status Codes

The following content error status codes are defined:

JSON Response Body (Non-HTTP 200)

The response body will contain the detailed application error status information. The additional information is optional for some status codes.

The format of the JSON is as follows:

{ 
        "message"             : "User friendly message",
        "statusCode"  : <appStatusCode>,
        "additional"  : <jsonAdditionalInfo> 
}
OR
{ 
        "message"             : "User friendly message",
        "statusCode"  : <appStatusCode> 
}

The high-level user friendly message will take the following format:

“<operation> failed due to <cause>”. For example “Create situation failed due to invalid parameters”.

JSON Addition Information

The format of the JSON additional information depends on the HTTP status code and application status code. For certain status codes the additional information may not be present. Although the HTTP status code (400 or 500) is enough to determine the additional information structure it is best to use the application status code (as in the future more than one range could apply to a single HTTP status code).

Application Status Code [1000-1999]

Optional additional information

A debug message typically containing the exception message:

{ "debugMessage" : "java.net.ConnectException: Connection refused" }

Application Status Code [2000-2999] or [4000-4999]

Mandatory additional information

List of the bad parameter(s) and the error information:

{ 
        "name"                : "parameterName",
        "value"       : <badValue>,
        "errorCode"   : <errorCode>
}

Ngnix Error Messages

97: Address family not supported by protocol

This an IPv6 error. Either configure IPv6 or comment it out.

To comment it out, go to /etc/nginx.conf.d and comment out the IPv6 references with a hash #:

502 Bad Gateway (nginx/1.14.0)

Fails to load resource and server responds with a status of 502 (Bad Gateway).

Check that you have used the correct components for the version you are installing.

moog_init_mooms.sh -z MY_ZONE
moog_init_lams.sh -bz MY_ZONE -d SERVER1:3306
moog_init_search.sh -sd SERVER1:3306
moog_init_server.sh -bz MY_ZONE -d SERVER1:3306
moog_init_ui.sh -otwxfz MY_ZONE -d SERVER1:3306