Moogsoft Docs

LAMs and High Availability

High Availability (HA) systems aim to minimize downtime and data loss. This is achieved with component redundancy to ensure there is no single point of failure, and ensure no loss of data or performance.

In the High Availability architecture of LAMs, we will have 2 clusters and each cluster contains an instance of the LAM. Only one LAM will be active at a time and the other will be set to passive. In case of a communication failure in the active LAM, a manual failover is initiated to the passive LAM, which will now send the data to MooMs. In the steps outlined below, 2 clusters, KINGSTON and SURBITON are used. The AWS LAM instance 1 in SURBITON is active and the AWS LAM instance 2 in KINGSTON is passive

Creating the HA configuration and switching the LAM in HA has following 2 steps:

  1. Copying and creating an instance of LAM in 2 clusters and editing their respective configuration file

  2. Manually initiating a failover of the LAM in case of communication failure in the active LAM

Copying and creating an instance of LAM in 2 clusters and editing their respective configuration file

We are taking the example of the AWS LAM, to provide HA for your LAM replace the AWS LAM with your LAM name and aws_lam.conf with the conf name of your LAM

An instance of the AWS LAM is created in each of the 2 clusters. The steps for creating the instances are as follows:

  1. Set the following properties in the system.conf file located in the config folder, if it is already set then proceed to the next step:

    • message_persistence: true This setting is in the mooms section of the system.conf file. The message_persistence configuration controls whether MooMS will persist important messages or not and when set to true will ensure that two moog_farmds in the same process group share the relevant queues or servers, this allows failover in moog_farmd to ensure that the processing of Events is continuous. The default is 'false' if not specified

  2. Create a copy of $MOOGSOFT_HOME/config/aws_lam.conf as $MOOGSOFT_HOME/config/aws_lam1.conf

  3. Create another copy as $MOOGSOFT_HOME/config/aws_lam2. conf

  4. Create a copy of /etc/init.d/awslamd as /etc/init.d/awslamd1, if the awslamd file is not in the init.d directory then it can be found in the /usr/share/moogsoft/etc/service-wrappers directory. After copying, perform the following steps in the awslamd1 file:

    • In the awslamd1 file, enter the path of the aws_lam1.conf in the CONFIG_FILE e.g. $MOOGSOFT_HOME/config/aws_lam1.conf field and enter the service name e.g. awslamd1 in the SERVICE NAME field

    • Edit the below given commands in the file with the instance and cluster properties at the end of the line of both "daemon" commands. For example, the instance name is AWS1 and cluster name is SURBITON in the below mentioned command lines

      daemon --pidfile=$PID_FILE $MOOG_WRAPPER --home=$PROCESS_HOME --process=$PROCESS_NAME --config=$CONFIG_FILE--logfile=$LOG_FILE --pidfile=$PID_FILE --loglevel=$LOG_LEVEL --service_instance AWS1 --cluster SURBITON
      
      daemon --pidfile=$PID_FILE --user=$PROCESS_OWNER $MOOG_WRAPPER --home=$PROCESS_HOME --process=$PROCESS_NAME --config=$CONFIG_FILE --logfile=$LOG_FILE --pidfile=$PID_FILE --loglevel=$LOG_LEVEL --service_instance AWS1 --cluster SURBITON
      
    • Now update the SERVICE_NAME in the /etc/init.d/awslamd1 file to be unique on that server, for example "AWS1"

  5. Create a copy of /etc/init.d/awslamd as /etc/init.d/awslamd2, if the awslamd file is not in the init.d directory then it can be found in the /usr/share/moogsoft/etc/service-wrappers directory. After copying, perform the following steps in the awslamd2 file:

    • In the awslamd2 file, enter the path of the aws_lam2.conf in the CONFIG_FILE field e.g. $MOOGSOFT_HOME/config/aws_lam2.conf and enter the service name e.g. awslamd2 in the SERVICE NAME field

    • Edit the below given commands in the file with the instance and cluster properties at the end of the line of both "daemon" commands. For example, the instance name is AWS2 and cluster name is KINGSTON in the below mentioned command lines

      daemon --pidfile=$PID_FILE $MOOG_WRAPPER --home=$PROCESS_HOME --process=$PROCESS_NAME --config=$CONFIG_FILE--logfile=$LOG_FILE --pidfile=$PID_FILE --loglevel=$LOG_LEVEL --service_instance AWS2 -- cluster KINGSTON
      
      daemon --pidfile=$PID_FILE --user=$PROCESS_OWNER $MOOG_WRAPPER --home=$PROCESS_HOME --process=$PROCESS_NAME --config=$CONFIG_FILE --logfile=$LOG_FILE --pidfile=$PID_FILE --loglevel=$LOG_LEVEL --servise_instance AWS2 --cluster KINGSTON
    • Now update the SERVICE_NAME in the /etc/init.d/awslamd2 file to be unique on that server, for example "AWS2"

  6. Under $MOOGSOFT_HOME/config directory edit the aws_lam1.conf file. If not present, then make a copy of aws_lam.conf file and rename it to aws_lam1.conf. Edit the following parameters in the file:

    • Enter the login details and the proxy details (if used) of the AWS server in the config file

    • Edit the HA section as per the example given below

      ha:
      {
        cluster: "SURBITON",          
        group: "aws_lam",           
        instance: "aws",          
        duplicate_source: false,           
        default_leader: true,          
        only_leader_active: true,         
        accept_conn_when_passive: true
      },
      
  7. Under $MOOGSOFT_HOME/config directory edit the aws_lam2.conf file. If not present, then make a copy of aws_lam.conf file and rename it to aws_lam2.conf. Edit the following parameters in the file:

    • Enter the same configuration in it as given in aws_lam1.conf

    • Edit the ha section as per the example given below

      ha:
      {
        cluster: "KINGSTON",
        group: "aws_lam",
        instance: "aws",          
        duplicate_source: false, 
        default_leader: true,
        only_leader_active: true,
        accept_conn_when_passive: true
      } ,
  8. Start both the services

    service awslamd1 start
    service awslamd2 start
    

In the ha section following fields can be configured:

  • cluster: The name of the Cluster. This supersedes anything set in system.conf (can also be overwritten by the command line)

  • group: The name of the Process Group. This defaults to the LAM process name if no value is specified (for example aws_lam)

  • instance: The name of the AWS LAM instance

  • duplicate_source: If set to true, it allows duplicate events from the same source.The default value is false

  • default_leader: A Boolean, indicating if the LAM is the Leader within its Process Group. The default value is true if not specified

  • only_leader_active: A Boolean that changes the type of process group from a Leader Only group to a Process Group where more than one process can be active

  • accept_conn_when_passive: A Boolean instructing the LAM what to do in Passive mode. If true (or not set), the LAM accepts incoming connections but discards any events received. If false, the LAM does not accept incoming connections

Manually initiating a failover of the LAM in case of communication failure of the active LAM

In the case of a communication failure the LAM has to be manually changed over to the passive LAM in another cluster by using the following command:

$MOOGSOFT_HOME/bin/ha_cntl -a KINGSTON.aws_lam

In the above case the AWS LAM process group from the SURBITON Cluster is changed over to the KINGSTON Cluster. Initially, the LAM in the SURBITON cluster was active and the LAM in KINGSTON cluster was passive. After executing the command the LAM in the KINGSTON cluster becomes active. The active LAM now publishes the events to MOOMs.