Configure Services to Restart

You can use Service Manager to control process startup for Moogsoft AIOps in non-RPM and non-root deployments.

The utility can keep processes associated with Moogsoft AIOps services alive if your system fails or restarts. For every new non-RPM based or non-root based install of Moogsoft AIOps, a cronjob entry deployed by the Moogsoft initialization script (moog_init.sh) runs the process manager script (process_keepalive.sh) every minute. This script attempts to restart the processes of any services listed in the process manager configuration file at $MOOGSOFT_HOME/config/keep_procs_alive.conf. By default, the following services are set to restart:

  • RabbitMQ

  • MySQL

  • Nginx

  • Elasticsearch

  • Apache-Tomcat

  • Moogfarmd.

You can either use the new service_manager utility to edit the configuration file, or edit the file manually (and enter a '1' for all services you want to restart and enter a '0' for those that you want left alone). If you have a custom LAM, you can add this to the configuration file with the flag to determine the restart behavior. For example:

Customlamd=1

If your system fails or restarts, the Service Manager utility automatically starts after one minute and attempts to restart the configured processes/services up to three times. If the service does not start after three attempts, the utility disables the service restart attempts in future.

Service Manager is useful for ensuring non-core processes start and run if Moogsoft AIOps fails or restarts. For example, you might want to ensure specific LAMs remain alive so no events are missed if your system reboots.

Configure Service Manager

Warning

Do not use or configure the Service Manager utility in RPM deployments or deployments where the application is running as the root user.

You can configure Service Manager to control which services and their associated processes to start up when Moogsoft AIOps reboots:

  1. Run the Service Manager:

    $MOOGSOFT_HOME/bin/utils/service_manager

    The default utility settings are shown as follows:

    ------------------------------------------------ Page [1/3] ---
    #                      Service Manager                        #
    ---------------------------------------------------------------
     Check the services that will be started and kept alive...                                                                                                                                       
    
     [x] rabbitmq                                                                                       
     [x] mysql                                                                                          
     [x] nginx                                                                                          
     [x] elasticsearch                                                                                  
     [x] apache-tomcat                                                                                  
     [x] moog_farmd                                                                                     
     [ ] ansibletower_lam                                                                               
     [ ] appdynamics_lam                                                                                
     [ ] aws_lam                                                                                        
     [ ] azure_classic_lam                                                                              
     [ ] azure_lam                                                                                      
     [ ] ca_spectrum_lam                                                                                
     [ ] datadog_client_lam                                                                             
     [ ] datadog_lam                                                                                    
     [ ] dynatrace_apm_lam                                                                              
     [ ] dynatrace_apm_plugin_lam                                                                       
     [ ] dynatrace_notification_lam                                                                     
     [ ] dynatrace_synthetic_lam                                                                        
     [ ] email_lam                                                                                      
     [ ] emc_smarts_lam                                                                                 
     [ ] extrahop_lam                                                                                   
     [ ] fluentd_lam                                                                                    
     [ ] hp_nnmi_lam                                                                                    
     [ ] hp_omi_lam                                                                                     
    
    Use arrows, page up/down and ENTER to navigate 
  2. Navigate through the list of available services using the directional arrows. There are multiple pages to scroll through.

  3. Press Space or Enter to add or remove services you want to restart or do not want to restart. An [ x ] appears next to any services you select.

  4. You can enable or disable Service Manager from restarting all services on the last page.

  5. Select 'Apply Changes' and press Enter to make the changes.

After you exit, the process_keepalive.sh script keeps selected services alive if Moogsoft AIOps fails or restarts.

Service Manager Command Line Reference

You can also configure the Service Manager using command line arguments. The utility uses the following syntax:

service_manager --service=<service_name> --command=<action>

The Service Manager resides at $MOOGSOFT_HOME/bin/utils/service_manager. You can configure the utility using the following arguments:

Argument

Input

Description

-h,--help

-

Displays the syntax and arguments available in Service Manager.

-s,--service=

service name

Name of the service you want to execute the Service Manager command on.

Apart from the core services, these include all of the LAMs. For example: email_lam, rest_lam, trapd_lam etc.

-c, --command=

enable | disable | enable_start | disable_stop

Commands the Service Manager can execute against the specified service:

  • enable - Enables the service auto start option.

  • disable - Disables the service auto start option.

  • enable_start - Starts and enables the service.

  • disable_stop - Stops and disables the service.

-a, --autostart-all=

enable | disable

Disables or enables the auto start option for all services.

You can only run a command against a single service at a time using the command line arguments. For example, if you wanted to enable the Service Manager to restart the Email LAM in the event of a failover:

service_manager --service=email_lam --command=enable

Disable Service Restart

There are two ways to disable the default service restart functionality.

Disable theprocess_keepalive.shcronjob by removing it from the cron table, the commands scheduled to run on your system. To edit the cron table:

crontab -e

Delete or comment out/usr/share/moogsoft/bin/utils/process_keepalive.sh 2>&1 from the file.

Alternatively, disable the Service Manager utility:

$MOOGSOFT_HOME/bin/utils/service_manager -a disable

Service Manager Logging

To check the Service Manager logs you can view:

/var/log/moogsoft/process/keepalive.log

This contains all logs relating to the Service Manager utility (service_manager) and to the process that attempts to keep the services alive (process_keepalive).