High Availability Installation Lab

At the end of this lab, you will know how to:

  • Install and configure three virtual Moogsoft Enterpriseinstances to work together as a highly available (HA) system.

  • Configure database, message broker, search, and core processing software to provide redundancy and failover.

Prerequisites

  • You are familiar with networking concepts and enterprise IT operations.

  • You are comfortable using Linux and JavaScript Object Notation (JSON).

  • You are familiar with Moogsoft Enterprise operations and system architecture.

Requesting Lab Instances

Moogsoft customers and partners can request lab instances here. After filling out the form, you will receive an email from Moogsoft University with access information for the three virtual machines you will need to complete the lab. Keep in mind that it may take up to two days before you get your lab instances.

Note

If at any point during the lab you find yourself unable to troubleshoot errors, you may want to consider re-requesting new lab instances and starting from scratch. Some errors are unrecoverable and require a new instance.

Obtaining a Moogsoft Enterprise License

This lab requires a valid Moogsoft Enterprise license. If you are a Moogsoft partner and do not have a valid license, contact Sales Operations with your name, organization, and email address. We will provide you with a 30-day license to work on the lab.

Scenario

You have just started working for a Moogsoft partner which provides service assurance solutions for large enterprise customers. It’s your fourth day on the job. You’ve met your colleagues, signed up for benefits, and spent a day and a half learning about Moogsoft Enterprise by watching Moogsoft University videos. You are justifiably proud of your Linux and IT skills, but some of the Moogsoft proprietary technologies and open source components are new to you.

Today, you and your boss have agreed that it’s time for you to get hands-on experience with the product so that you can make the most of your onboarding time before joining a customer implementation team. She has asked you to do a practice installation of Moogsoft Enterprise on three virtual hosts and configure them as a highly available, redundant trio. Your installation won’t be as complex as at a typical distributed customer site, but if you succeed with your HA implementation you will know you are off to a good start in your new job.

Before You Begin

To complete this lab, you will draw heavily on the Moogsoft Enterprise documentation, specifically the Implementer Guide. Start by scanning the entire guide, and then read through these areas of focus:Implementer Guide

Then, review the control utility instructions from the documentation for RabbitMQ, the open source message broker which drives the Moogsoft Enterprise message bus. You may wish to scan materials about RabbitMQ clustering and queue mirroring as well.

Take note of the following points from the HA Reference Architecture diagram and the documentation:

  • Different components of Moogsoft Enterprise—for example, the user interface; the database; the combined core processing, message bus, and search facilities; and the data adapters—can be installed on different virtual or physical machines.

  • You can set up Moogsoft Enterprise to be highly available so that, depending on the component, each component is either active in a redundant group or configured to switch from passive to active if its counterpart fails. You can also set up a backup disaster recovery site, but it won’t automatically come online if the primary site fails.

  • Moogsoft recommends using a load balancer to distribute network traffic across multiple machines. A load balancer can improve responsiveness and also allows you to add machines to maintain performance as demand increases.

  • Moogsoft Enterprise uses Hazelcast to provide in-memory data persistence across core processing services.

  • Some third-party components, specifically the message broker (RabbitMQ), the search engine (Elasticsearch), and the database (Percona Server for MySQL) need three running instances to avoid "split brain." Thus, the minimum size for a fully fault tolerant Moogsoft Enterprise system is three servers.

  • How you configure high availability varies by component. Configure Moogsoft core processing and user interface for HA by editing system configuration files, and configure other components separately according to their own requirements. After components are configured and initialized, you can use control scripts to examine HA status and modify failover behavior.

  • For Moogsoft core processing components, "cluster" refers to the vertical collection of components that make up a complete Moogsoft Enterprise system, so that an HA pair comprises two clusters. Each cluster can be distributed across several hosts. The Moogsoft HA topology is further defined by groups (similar components linked to exhibit failover behavior) and instances (individual running components).

  • For the RabbitMQ message broker, "cluster" is a horizontal term analogous to group. It refers to peer nodes which share information.

About the Lab

Most of the lab steps give general instructions with specific code or settings under the "Overview" sections. If you are comfortable with Linux, try to do the steps on your own when you can, and refer to the "Step-By-Step Solution" sections when you need help or to check your work.

You will be editing system configuration files in JSON format. It's easy to make mistakes when editing JSON because a misplaced comma or curly brace can invalidate the structure of an entire file. Be careful and patient as you work on editing the system settings.

While you work on the lab, ask yourself:

  • How can I test that each step worked?

  • When something goes wrong, what should I do? Should I:

    • inspect my work for an error,

    • review log output,

    • do research in the documentation,

    • repeat the last few steps, or

    • all of the above?

  • How would I change the installation design to

    • increase reliability?

    • respond to performance demands?

Note

For troubleshooting during the lab, log files can be found in the /usr/share directory under the respective service. For example, apache-tomcat log files are located under the apache-tomcat directory.

As you gain product fluency, you’ll be able to move away from step-by-step instructions. With experience, you will gain the ability to design and install distributed systems with varying topologies and HA settings.