Troubleshoot the UI

1

Check catalina.out for any obvious errors or warning.

Cause may be evident from any warnings or errors.

2

Check browser console or any errors or timing out requests.

Possibly a bug or more likely that the query to the database associated with the request is taking longer that 30 seconds (the default browser timeout). Investigate the root cause.

3

Check network latency between browser client machine and server using ping.

Latency of =>100ms can make login noticeably slower.

4

Check the CPU/memory usage of the server itself.

If the server, as a whole, is running close to CPU or memory limit and no other issues can be found (e.g. rogue processes or memory leaks in the Moogsoft Onprem components) then consider adding more resource to the server or distributing the Moogsoft Onprem components.

5

Check MoogSvr/Moogpoller/Graze counter logging in catalina.out

Tomcat may be processing a high number of requests or Message Bus updates.

If Moogpoller count is zero then something may be wrong with Tomcat > RabbitMQ connection. Check RabbitMQ admin UI for signs of message queue build-up.

6

Check whether Tomcat java process is showing constant high CPU/memory usage.

Tomcat may be processing the updates from an event or situation storm. Backlog should clear assuming storm subsides.

7

Has the memory of the Tomcat java processed reached a plateau?

Tomcat may have reached its java heap limit. Check the -Xmx setting in /etc/init.d/apache-tomcat.

Increase the -Xmx settings as appropriate and restart the apache-tomcat service.

8

Is the database tuned?

Check the innodb-buffer-pool-size and innodb_buffer_pool_instances settings in /etc/my.cnf as per Tuning section above. Ensure they are set appropriately and restart mysql if changes are made.

9

Check the server for any other high CPU or memory processes or that which might be impacting the database.

Something may be hogging CPU/memory on the server and starving Tomcat of resources.

The Events Analyser utility may be running or a sudden burst of Moogfarmd or Graze activity may be putting pressure on the database and affecting the UI.

10

Run DBPool Diagnostics (see previous section) several times to assess current state of Tomcat > Database connections.

Tomcat database connections may be maxed out with long running connections - this may indicate a processing deadlock - perform a kill -3 <pid> on the Tomcat java process to generate a thread dump (in catalina.out) and send it to Moogsoft Onprem Support.

Alternatively Tomcat may be very busy with lots of short but frequent connections to the database. A Graze request bombardment is another possibility (Graze does not currently have a separate DB Pool). Consider increasing the number DBPool connections for Tomcat by increasing the related properties in servlets.conf and restarting the apache-tomcat service.

11

Turn on MySQL slow query logging (see earlier section on how to do this)

Slow queries from nasty filters in the UI may be causing problems, review them for efficiency.

Alternatively slow queries from other parts of the system may be causing problems (e.g. inefficient Moobot code).

Slow queries may also be down to the sheer amount of data in the system. Consider enabling Database Split to move old data and/or using the Archiver to remove old data.

12

Is Tomcat memory constantly growing over time and a memory leak is suspected?

Note that Tomcat memory does typically increase for periods of time then is trimmed back via java garbage collection.

Take periodic heap dumps from the Tomcat java process and send them to Moogsoft support so they can analyse the growth. Use the following commands:

DUMPFILE=/tmp/tomcat-heapdump-$(date +%s).bin
sudo -u tomcat jmap -dump:format=b,file=$DUMPFILE $(ps -ef|grep java|grep tomcat|awk '{print $2}')
bzip2 $DUMPFILE

Notes: