Set Up the Redundancy Server Role
In Moogsoft Onprem HA architecture, both RabbitMQ and Opensearch run as three-node clusters. The three-node clusters prevent issues with ambiguous data state, such as a "split-brain".
RabbitMQ is the Message Bus used by Moogsoft Onprem. Opensearch delivers the search functionality.
The three nodes are distributed across the two Core roles and the redundancy server.
HA architecture
In our distributed HA installation, the RabbitMQ and Opensearch components are installed on the Core 1, Core 2 and Redundancy servers.
Core 1: RabbitMQ Node 1, Search Node 1
Core 2: RabbitMQ Node 2, Search Node 2
Redundancy server: RabbitMQ Node 3, Search Node 3
Refer to the Distributed HA system Firewall for more information on connectivity within a fully distributed HA architecture.
Install Redundancy server
Install the Moogsoft Onprem components on the Redundancy server.
On the Redundancy server install the following Moogsoft Onprem components:
VERSION=9.1.0; yum -y install moogsoft-common-${VERSION} \ moogsoft-mooms-${VERSION} \ moogsoft-search-${VERSION} \ moogsoft-utils-${VERSION}
Edit the
~/.bashrc
file to contain the following lines:export MOOGSOFT_HOME=/usr/share/moogsoft export APPSERVER_HOME=/usr/share/apache-tomcat export JAVA_HOME=/usr/java/latest export PATH=$PATH:$MOOGSOFT_HOME/bin:$MOOGSOFT_HOME/bin/utils
Source the
.bashrc
file:source ~/.bashrc
Initialize RabbitMQ cluster node 3 on the Redundancy server and join the cluster.
On the Redundancy server initialise RabbitMQ. Use the same zone name as Core 1 and Core 2:
moog_init_mooms.sh -pz <zone>
The erlang cookies must be the same for all RabbitMQ nodes. Replace the erlang cookie on the Redundancy server with the Core 1 erlang cookie located at
/var/lib/rabbitmq/.erlang.cookie
. Make the Redundancy server cookie read-only:chmod 400 /var/lib/rabbitmq/.erlang.cookie
You may need to change the file permissions on the Redundancy server erlang cookie first to allow this file to be overwritten. For example:
chmod 406 /var/lib/rabbitmq/.erlang.cookie
Restart the
rabbitmq-server
service and join the cluster. Substitute the Core 1 server short hostname:systemctl restart rabbitmq-server rabbitmqctl stop_app rabbitmqctl join_cluster rabbit@<Core 1 server short hostname> rabbitmqctl start_app
The short hostname is the full hostname excluding the DNS domain name. For example, if the hostname is
ip-172-31-82-78.ec2.internal
, the short hostname isip-172-31-82-78
. To find out the short hostname, runrabbitmqctl cluster_status
on Core 1.Apply the HA mirrored queues policy. Use the same zone name as Core 1:
rabbitmqctl set_policy -p <zone> ha-all ".+\.HA" '{"ha-mode":"all"}'
Run
rabbitmqctl cluster_status
to verify the cluster status and queue policy. Example output is as followsCluster status of node rabbit@ldev02 Basics Cluster name rabbit@ldev02 Disk Nodes rabbit@ldev01 rabbit@ldev02 Running Nodes rabbit@ldev01 rabbit@ldev02 Versions ... [root@ldev02 rabbitmq]# rabbitmqctl -p MOOG list_policies Listing policies for vhost "MOOG" ... MOOG ha-all .+\.HA all {"ha-mode":"all"} 0
Initialize, configure and start Opensearch. Follow the Opensearch Clustering guide here: Opensearch Clustering Guide
Opensearch Encryption
You can enable password authentication on Opensearch by editing the $MOOGSOFT_HOME/config/system.conf
configuration file. You can use either an unencrypted password or an encrypted password, but you cannot use both.
You should use an encrypted password in the configuration file if you do not want users with configuration access to be able to access integrated systems.
Enable password authentication
To enable unencrypted password authentication on Opensearch, set the following properties in the system.conf
file:
"search": { ... “username” : <username>, “password” : <password>, ... }
To enable encrypted password authentication on Opensearch, set the following properties in the system.conf
file:
"search": { ... “username” : <username>, “encrypted_password” : <encrypted password> ... }
Initialize Opensearch
Opensearch already has password authentication enabled, but other users can be added. If the admin password was already changed by moog_init_search.sh while deploying Opensearch, the script will prompt for admin account details to use to create the new users. To initialize Opensearch with password authentication, run:
moog_init_search.sh -a username:password
or:
moog_init_search.sh --auth username:password
If you run moog_init_search
without the -a/--auth
parameters, you will not enable password authentication in Opensearch.
See Moog Encryptor for more information on how to encrypt passwords stored in the system.conf
file.
You can also manually add authentication to the Opensearch configuration. You should do this if you have your own local Opensearch installation. See the external documentation for Opensearch here https://opensearch.org/docs/latest/security-plugin/configuration/index/ for more information.