Opensearch Clustering Guide

This page outlines the proper procedure for clustering Opensearch. Modifications are required for each Opensearch server.

Opensearch Clustering Guide - RPM

This page outlines the proper procedure for clustering Opensearch in RPM deployments. Modifications are required for each Opensearch server.

Important

This guide assumes that OpenSearch has already been installed on all three nodes using the $MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i command

Steps

Stop OpenSearch on all three nodes:
```
service opensearch stop
```
On any node, define the hostnames of the nodes in a bash terminal (change the hostnames as needed):
```
NODE1="mynode1";
NODE2="mynode2";
NODE3="mynode3";
```
On the same node, define some required SSL certificate parameters in the same bash terminal (change the values as needed):
```
CSR_COUNTRY="UK";
CSR_STATE="SURREY";
CSR_CITY="KINGSTON";
CSR_ORGANIZATION="MOOGSOFT";
```

On the same node, run the following script in the same bash terminal to generate all the certificate files (enabling transport-layer SSL security for communication between the nodes) and OpenSearch config files:

mkdir /tmp/opensearch_certs
cd /tmp/opensearch_certs

# Create Root CA
openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -subj "/C=${CSR_COUNTRY}/ST=${CSR_STATE}/L=${CSR_CITY}/O=${CSR_ORGANIZATION}/OU=UNIT/CN=root" -out root-ca.pem -days 730

# Create Admin cert
openssl genrsa -out admin-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem
openssl req -new -key admin-key.pem -subj "/C=${CSR_COUNTRY}/ST=${CSR_STATE}/L=${CSR_CITY}/O=${CSR_ORGANIZATION}/OU=UNIT/CN=A" -out admin.csr
openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem -days 730

# Create Node certs
for NODE in "${NODE1}" "${NODE2}" "${NODE3}";
do
	openssl genrsa -out ${NODE}-key-temp.pem 2048
	openssl pkcs8 -inform PEM -outform PEM -in ${NODE}-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out ${NODE}-key.pem
	openssl req -new -key ${NODE}-key.pem -subj "/C=$CSR_COUNTRY/ST=${CSR_STATE}/L=${CSR_CITY}/O=${CSR_ORGANIZATION}/OU=UNIT/CN=${NODE}" -out ${NODE}.csr
	echo "subjectAltName=DNS:${NODE}" > ${NODE}.ext
	openssl x509 -req -in ${NODE}.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out ${NODE}.pem -days 730 -extfile ${NODE}.ext
	rm -f *-temp.pem *.csr *.ext

	YML_CONFIG="cluster.name: moog-opensearch-cluster\n"
	YML_CONFIG+="node.name: ${NODE}\n"
	YML_CONFIG+="network.host: 0.0.0.0\n"
	YML_CONFIG+="discovery.seed_hosts: [ \"${NODE1}\", \"${NODE2}\", \"${NODE3}\" ]\n"
	YML_CONFIG+="cluster.initial_master_nodes: [ \"${NODE1}\", \"${NODE2}\", \"${NODE3}\" ]\n"
	YML_CONFIG+="node.master: true\n"
	YML_CONFIG+="plugins.security.allow_default_init_securityindex: true\n"
	YML_CONFIG+="plugins.security.ssl.transport.pemcert_filepath: ${NODE}.pem\n"
	YML_CONFIG+="plugins.security.ssl.transport.pemkey_filepath: ${NODE}-key.pem\n"
	YML_CONFIG+="plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem\n"
	YML_CONFIG+="plugins.security.ssl.transport.enforce_hostname_verification: false\n"
	YML_CONFIG+="plugins.security.restapi.roles_enabled: [\"all_access\"]\n"
	YML_CONFIG+="plugins.security.authcz.admin_dn:\n"
        YML_CONFIG+="    - \"CN=A,OU=UNIT,O=${CSR_ORGANIZATION},L=${CSR_CITY},ST=${CSR_STATE},C=${CSR_COUNTRY}\"\n"
	YML_CONFIG+="plugins.security.ssl.http.enabled: true\n"
	YML_CONFIG+="plugins.security.ssl.http.pemcert_filepath: ${NODE}.pem\n"
	YML_CONFIG+="plugins.security.ssl.http.pemkey_filepath: ${NODE}-key.pem\n"
	YML_CONFIG+="plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem\n"
	YML_CONFIG+="plugins.security.nodes_dn:\n"
	YML_CONFIG+="    - \"CN=${NODE1},OU=UNIT,O=${CSR_ORGANIZATION},L=${CSR_CITY},ST=${CSR_STATE},C=${CSR_COUNTRY}\"\n"
	YML_CONFIG+="    - \"CN=${NODE2},OU=UNIT,O=${CSR_ORGANIZATION},L=${CSR_CITY},ST=${CSR_STATE},C=${CSR_COUNTRY}\"\n"
	YML_CONFIG+="    - \"CN=${NODE3},OU=UNIT,O=${CSR_ORGANIZATION},L=${CSR_CITY},ST=${CSR_STATE},C=${CSR_COUNTRY}\"\n"
	YML_CONFIG+="http.max_content_length: 500mb\n"
	mkdir ${NODE}
	mv ${NODE}.pem ${NODE}-key.pem ${NODE}/
	cp -p root-ca.pem ${NODE}/
	echo -e "${YML_CONFIG}" > ${NODE}/opensearch.yml
done

This will create the following directory structure:

-rw-------. 1 root root 1704 Aug 18 11:06 admin-key.pem
-rw-r--r--. 1 root root 1200 Aug 18 11:06 admin.pem
drwxr-xr-x. 2 root root   89 Aug 18 11:06 mynode1
drwxr-xr-x. 2 root root   89 Aug 18 11:06 mynode2
drwxr-xr-x. 2 root root   89 Aug 18 11:06 mynode3
-rw-------. 1 root root 1679 Aug 18 11:06 root-ca-key.pem
-rw-r--r--. 1 root root 1326 Aug 18 11:06 root-ca.pem
-rw-r--r--. 1 root root   41 Aug 18 11:06 root-ca.srl

Copy the corresponding files from each node-named-directory (e.g. /tmp/opensearch_certs/mynode1) to the corresponding node under directory /etc/opensearch:

scp /tmp/opensearch_certs/mynode1/* root@mynode1:/etc/opensearch/
scp /tmp/opensearch_certs/mynode2/* root@mynode2:/etc/opensearch/
scp /tmp/opensearch_certs/mynode3/* root@mynode3:/etc/opensearch/

Additionally copy the admin certificate and key to node1 to be used in later steps:

scp /tmp/opensearch_certs/admin* root@mynode1:/etc/opensearch/

Ensure the OpenSearch user credentials from node1 are set on all Moogsoft Moogsoft Enterprise servers running MoogFarmd or Apache-Tomcat in the search block in $MOOGSOFT_HOME/config/system.conf for example:
```
	"username" : "moog_opensearchuser",
	"password" : "rwy5kygVGCsXDumd",
```
On both joiner nodes (node2 and node3), empty the data directory:
```
rm -rf  /var/lib/opensearch/data/*
```

On all three nodes, fix the file permissions, and restart OpenSearch:

chmod 600 /etc/opensearch/*.pem
chown moogsoft:moogsoft /etc/opensearch/*
sudo systemctl restart opensearch

On node1, wait for OpenSearch to finish starting, then run the OpenSearch securityadmin utility:

while [ -z "$(netstat -na|grep 9200|grep LISTEN)" ]; do sleep 1; done;
bash /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
    -cd /usr/share/opensearch/config/opensearch-security/ \
    -nhnv \
    -cacert /etc/opensearch/root-ca.pem \
    -cert /etc/opensearch/admin.pem \
    -key /etc/opensearch/admin-key.pem \
    -cn moog-opensearch-cluster;

This ensures that the local moog_opensearch user created by the moog_init_search.sh -i command on node1, will be replicated to node2 and node3.

Test the cluster by running this command on all three nodes:

curl -XGET 'https://localhost:9200/_cluster/health?pretty' -u moog_opensearchuser:$($MOOGSOFT_HOME/bin/utils/moog_config_reader -k search.password)

This should report: "number_of_nodes" : 3

Important

If the response from any of the commands above is "Unauthorized", the following steps should be run on node1 to reset the OpenSearch admin user, and recreate the moog_opensearchuser account in the cluster (replace 'moog_search_P4ssword' with the one set in ${MOOGSOFT_HOME}/config/system.conf):

HASH_PASS=$(bash /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh -p admin | sed 's/\//\\\//g');
sed -i 's/hash:.*/hash: "'$HASH_PASS'"/g' /usr/share/opensearch/config/opensearch-security/internal_users.yml;
bash /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
    -cd /usr/share/opensearch/config/opensearch-security/ \
    -nhnv \
    -cacert /etc/opensearch/root-ca.pem \
    -cert /etc/opensearch/admin.pem \
    -key /etc/opensearch/admin-key.pem \
    -cn moog-opensearch-cluster;

DIsable SSL by executing the following command on all three nodes:

sed -i -e 's/plugins.security.ssl.http.enabled: true/plugins.security.ssl.http.enabled: false/g' /etc/opensearch/opensearch.yml

On all 3 nodes restart OpenSearch:

systemctl restart opensearch

Execute the following command on node1 to recreate the user:

$MOOGSOFT_HOME/bin/utils/moog_init_search.sh -a moog_opensearchuser:moog_search_P4ssword;

Then retry this step to confirm the cluster size is reported correctly from all three nodes:

curl -XGET 'http://localhost:9200/_cluster/health?pretty' -u moog_opensearchuser:$($MOOGSOFT_HOME/bin/utils/moog_config_reader -k search.password)

DIsable SSL by executing the following command on all three nodes:

sed -i -e 's/plugins.security.ssl.http.enabled: true/plugins.security.ssl.http.enabled: false/g' /etc/opensearch/opensearch.yml

On all 3 nodes restart OpenSearch:

systemctl restart opensearch

Test the cluster again by running this command on all three nodes:

curl -XGET 'http://localhost:9200/_cluster/health?pretty' -u moog_opensearchuser:$($MOOGSOFT_HOME/bin/utils/moog_config_reader -k search.password)

This should report: "number_of_nodes" : 3

Add all three Opensearch nodes to the search/nodes array in $MOOGSOFT_HOME/config/system.conf on all Moogsoft Moogsoft Enterprise servers running MoogFarmd or Apache-Tomcat:
```
"nodes" : [
  {
    "host" : "mynode1",
    "port" : 9200
  },
  {
    "host" : "mynode2",
    "port" : 9200
  },
  {
    "host" : "mynode3",
    "port" : 9200
  }
]
```
Note
If the OpenSearch cluster has been created as part of an HA (High Availability) deployment, return to that document/process at this point.
Alternatively, Moogsoft Enterprise is already installed and configured and the OpenSearch cluster is being deployed as a standalone step, follow the steps below.
Restart Apache-Tomcat and MoogFarmD on all Moogsoft Moogsoft Enterprise servers connected to the OpenSearch cluster to read in the configuration changes:
```
systemctl restart moogfarmd
systemctl restart apache-tomcat
```
Wait a couple of minutes for the processes to finish starting, then run the moog_indexer utility to ensure that existing Alerts and Situations can be searched from the UI:
```
$MOOGSOFT_HOME/bin/utils/moog_indexer -n -f
```