Skip to main content

Message System Troubleshooting

The Message Bus (sometimes called MooMs) is the message system for Moogsoft Onprem, implemented with RabbitMQ.

This guide outlines some common issues with the Message Bus deployment and offers alternative solutions.

Open the Management Console

You can launch the RabbitMQ management console directly from the Moogsoft Onprem UI. This provides useful statistics when debugging.

To open the console, go to Settings > Self Monitoring > Message Bus and click Launch Message Bus Console ... You can log in using the default credentials:

Username: moogsoft

Password: m00gs0ft

These are defined in system.conf. If commented out, a default 'guest' user can be used.

Examine the Log Files

Troubleshooting your Message Bus deployment often requires examining log files. The default locations of log files are as follows:

  • moog_farmd and LAMs - /usr/log/moogsoft/$SERVICE_NAME.log where $SERVICE_NAME is the process, for example socketlamd for the Socket LAM

  • Tomcat -/usr/share/apache-tomcat/logs/catalina.out

  • RabbitMQ - $RABBITMQ_HOME/var/log/rabbitmq

RabbitMQ Broker Fails to Start

If RabbitMQ broker fails to start and Security-Enhanced Linux (SELinux) is enabled, it may be related to this. SELinux and similar mechanisms such as firewalls may prevent RabbitMQ from binding to a port and starting up.

If SELinux is enabled, check that the following ports can be opened:

    • 4369 (Erlang port mapper daemon)

    • 25672 (Erlang distribution)

    • 5672, 5671 (AMQP 0-9-1 without and with TLS)

    • 15672 (if management plugin is enabled)

You may need to configure RabbitMQ to use different ports.

For more information, refer to the RabbitMQ installation documentation.

If the RabbitMQ broker fails to start it may be due to file permissions, you may see errors such as:

{error_logger,{{2015,9,1},{21,26,14}},"Failed to create cookie file '/home/moogsoft/.erlang.cookie': eacces",[]}

{error_logger,{{2015,9,1},{21,26,14}},crash_report,[[{initial_call,{auth,init,['Argument__1']}},{pid,<0.21.0>},{registered_name,[]},{error_info,{exit,{"Failed to create cookie file '/home/moogsoft/.erlang.cookie': eacces",[{auth,init_cookie,0,[{file,"auth.erl"},{line,286}]},{auth,init,1,[{file,"auth.erl"},{line,140}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,352}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[net_sup,kernel_sup,<0.10.0>]},{messages,[]},{links,[<0.19.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,975}],[]]}...

When the RabbitMQ broker is running as a service, use the following command to check that it is running:

service rabbitmq-server status
Typical Error Messages

The section below will outline examples and solutions to typical error messages with RabbitMQ.

Connection refused/ Unable to create RabbitMQ connection

If the RabbitMQ broker appears to be down or unreachable, trying to start an Moogsoft Onprem component gives warnings such as:

WARN : [main ][20150812 16:09:54.792 +0100] [CMoomsFactory.java]:707 +|Unable to create RabbitMQ connection : [amqp://localhost:5672/ZONE]|+
WARN : [main ][20150812 16:09:54.792 +0100] [CMoomsFactory.java]:256 +|Unable to create RabbitMQ connection : [java.net.ConnectException: Connection refused]|+
WARN : [main ][20150812 16:09:54.793 +0100] [CMoomsFactory.java]:707 +|Unable to create RabbitMQ connection : [amqp://localhost:5672/ZONE]|+
WARN : [main ][20150812 16:09:54.793 +0100] [CJNIMoomsWrapper.java]:253 +|Problem during mooms setup, retrying|+

The structure of the amqp url is: amqp://<hostname>:<port>/<zone>

Solution:

Check if the RabbitMQ broker is running:

service rabbitmq-server status

If it isn't running, see RabbitMQ Broker Fails to Start (above).

If it is running, check the Message Bus configuration in the 'mooms' section of system.conf.

AlreadyClosedException/broker forced connection closure

If Moogsoft Onprem components are running, the RabbitMQ broker going down or becoming unreachable gives warnings such:

WARN : [Thread-][20150812 16:15:42.295 +0100] [CLogger.java]:337 +|Problem sending message id : [4115e512-aa63-44d5-bdc9-8ed164cd75e5]
com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; protocol method: #method<connection.close>(reply-code=320, reply-text=CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', class-id=0, method-id=0)
at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:195)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:309)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:657)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:640)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:631)
at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicPublish(AutorecoveringChannel.java:168)
at com.moogsoft.mooms.CMoomsMessageSender.send(CMoomsMessageSender.java:530)
at com.moogsoft.mooms.CMoomsMessageSender.send(CMoomsMessageSender.java:448)
at com.moogsoft.mooms.CMoomsMessageSender.send(CMoomsMessageSender.java:264)
at com.moogsoft.mooms.CMoomsMessageSenderPool.send(CMoomsMessageSenderPool.java:378)
at com.moogsoft.mooms.CJNIMoomsWrapper.sendEvent(CJNIMoomsWrapper.java:288)
|+

Note

If Moogsoft Onprem is configured to connect to RabbitMQ brokers in a high availability environment, during a RabbitMQ broker fail-over, warning messages may be logged for a short time

Solution:

If the problem is not temporary, check if the RabbitMQ broker is running:

service rabbitmq-server status
Problem during mooms setup, retrying

A Moogsoft Onprem component trying to connect to a non-existent zone (vhost) in a RabbitMQ broker gives warnings such as:

DEBUG: [main ][20150812 16:24:00.764 +0100] [CMoomsFactory.java]:206 +|Setting factory zone to : [fish]|+
WARN : [main ][20150812 16:24:03.825 +0100] [CMoomsFactory.java]:707 +|Unable to create RabbitMQ connection : [amqp://localhost:5672/fish]|+
WARN : [main ][20150812 16:24:03.825 +0100] [CMoomsFactory.java]:256 +|Unable to create RabbitMQ connection : [java.io.IOException]|+
WARN : [main ][20150812 16:24:06.373 +0100] [CMoomsFactory.java]:707 +|Unable to create RabbitMQ connection : [amqp://localhost:5672/fish]|+
WARN : [main ][20150812 16:24:06.373 +0100] [CJNIMoomsWrapper.java]:253 +|Problem during mooms setup, retrying|+

Solution:

Check you have the correct zone configured in the 'mooms' section of system.conf and that the zone (vhost) has been created in the RabbitMQ broker.

The best way to check that the zone (vhost) has been created in the RabbitMQ broker is to use the RabbitMQ management console.

If the zone (vhost) has not been created, manually create it via the command line, or via the RabbitMQ Management console.

Once the zone has been created, the user defined in system.conf must be given permissions to access the new zone.

AuthenticationFailureException: ACCESS_REFUSED

A Moogsoft Onprem component trying to connect to a valid zone (vhost) in a RabbitMQ broker, but with wrong authentication details gives warnings such as:

WARN : [main ][20150812 16:20:29.760 +0100] [CMoomsFactory.java]:707 +|Unable to create RabbitMQ connection : [amqp://jimmy@localhost:5672/null]|+
WARN : [main ][20150812 16:20:29.760 +0100] [CMoomsFactory.java]:256 +|Unable to create RabbitMQ connection : [com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.]|+
WARN : [main ][20150812 16:20:29.805 +0100] [CMoomsFactory.java]:707 +|Unable to create RabbitMQ connection : [amqp://jimmy@localhost:5672/null]|+
WARN : [main ][20150812 16:20:29.805 +0100] [CJNIMoomsWrapper.java]:253 +|Problem during mooms setup, retrying|+

Solution:

Check you have the correct username and password in the 'mooms' section of system.conf and that they match those defined in the RabbitMQ broker. If they are correct in system.conf then you must correct it in the RabbitMQ broker. Do this either via the command line, or via the RabbitMQ management console.

Also see RabbitMQ Broker Fails to Start (above).

First startup of RabbitMQ broker

The first time a RabbitMQ broker is started, it creates an 'account' with the default user and password from rabbitmq.config.

If this information is subsequently edited in rabbitmq.config, and the RabbitMQ broker is restarted, the 'account' is not created, which can be confusing.

If the RabbitMQ broker has been started before, then the 'account' will need to be added manually (see Manually set up a user) rather than by defining a default user in the rabbitmq.config file.

LAMs fail to start from command line

If LAMs run from the command line or as a service result in the following error:

[root@moogbox2 bin]# ./socket_lam 
./socket_lam: error while loading shared libraries: libjvm.so: cannot open shared object file: No such file or directory

...it may be because /usr/java/jdk1.8.0_20/jre/lib/amd64/server has not been added to the LD_LIBRARY_PATH.

  • To run the LAMs via a command line, a change the LD_LIBRARY_PATH to be as follows (the default initd files have this modification made):

export LD_LIBRARY_PATH=$MOOGSOFT_HOME/lib:/usr/GNUstep/Local/Library/Libraries:/usr/GNUstep/System/Library/Libraries:$JAVA_HOME/jre/lib/amd64/server
Manually configure IP address and port

In some environments (such as SELinux) you may need to configure a RabbitMQ broker to listen on a different IP address and port.

To do this:

  1. Configure the contents of /etc/rabbitmq/rabbitmq-env.conf, for example:

    RABBITMQ_NODE_IP_ADDRESS="172.168.87.131
RABBIT_NODE_PORT="5678"

  2. Restart the rabbitmq-server service:

    service rabbitmq-server restart
Manually set up a user or zone (vhost)

To help troubleshoot an existing RabbitMQ broker, you may want to manually set up a user or zone (vhost).

  • To manually set up a new user in the RabbitMQ broker, run the following command (using your own user, password and zone):

    rabbitmqctl add_user <user> <password>
rabbitmqctl set_permissions -p <zone> <user> ".*" ".*" ".*"

    Also ensure the username, password and zone in the 'mooms' section of system.conf match those defined in the RabbitMQ broker with the above commands.

  • To manually set up a new zone in the RabbitMQ broker, run the following command (using your own user and zone):

    rabbitmqctl add_vhost <zone>
rabbitmqctl set_permissions -p <zone> <user> ".*" ".*" ".*"

    Also ensure the username and zone in the 'mooms' section of system.conf match those defined in the RabbitMQ broker with the above commands.

In this section: