Page tree
Skip to end of metadata
Go to start of metadata

Overview

Amazon Web Services (AWS) is a secure cloud services platform, offering computing power, database storage, content delivery and other functionalities. The AWS LAM fetches the alarms and events from AWS CloudWatch.

Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications that run on AWS. Amazon CloudWatch is used to collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes in your AWS resources. It can monitor AWS resources such as Amazon EC2 instances, Amazon DynamoDB tables, and Amazon RDS DB instances, as well as custom metrics generated by your applications and services, and any log files your applications generate.

Process Workflow

The workflow of gathering alarms/events from AWS and publishing it to Moogsoft AIOps is as follows:

  1. AWS LAM reads the configuration from the  aws_lam.conf file.
  2. AWS LAM reads credentials and region of AWS from the config file and requests Amazon Web Services for alarms/events.
  3. The AWS LAM  parses the received alarms/events and converts it into a Map and submit it to EventFactory.
  4. The events are parsed and converted into normalized Moogsoft AIOps events.
  5. The normalized events are then published to MooMS bus.

AWS LAM Configuration

The alarms received from the AWS are processed according to the configurations in the aws_lam.conf file. The processed alarms are published to Moogsoft AIOps.

The configuration file contains a JSON object. At the first layer of the object, the LAM has a parameter called config, and the object that follows config has all the necessary information to control the LAM.

The following sections are available for configuration in the AWS LAM configuration file.

Monitor

The AWS LAM takes the alarm and event data from AWS CloudWatch. The user can configure the parameters here to establish the connection with AWS.

config:
      {
        monitor:
        {
            name                         : "AWS Monitor",

            class                        : "CAwsMonitor",

            #encrypted_access_key_id     : "";

            access_key_id                : "",

            #encrypted_secret_access_key : "";

            secret_access_key            : "",

            enable_proxy                 : false,

            proxy_host                   : "localhost",

            proxy_port                   : 8080,

            proxy_userid                 : "userid",

            #encrypted_proxy_password    : "",

            proxy_password               : "Password",
            
            filter:
            {
                    alarms:
                    {
                        alarm_name_prefix            : "test",
                        alarms_to_monitor            : []
                     },
                   #events:
	            	#{
	            	#	"us-west-2":
	            	#	{
	            	#		filter_pattern               :"",
	            	#		log_group_to_monitor		 :[]
	            	#	},
	            	#  "ap-south-1":
	            	#	{
	            	#		filter_pattern               :"",
	            	#		log_group_to_monitor		 :[]
	            	#	}
            },

            polling_interval              : 60,

            max_retries                   : 10, 

            retry_interval                : 60        
           },
  • name and class: These fields are reserved and should not be changed. The default values are AWS Monitor and C Aws Monitor respectively
  • access_key_id: The Access Key id received at the time of creating the AWS account

  • encrypted_access_key_id: The encrypted Access Key id generated using moog_encryptor

  • secret_access_key: The access key received at the time of creating the AWS account

  • encrypted_secret_access_key: The encrypted Secret Access Key generated using moog_encryptor

  • enable_proxy: Set it to true if a proxy is used for communication with AWS

  • proxy_host: The host name or the URL of the proxy

  • proxy_port: The port of the proxy

  • proxy_userid: The username of the user who has the rights to access the proxy

  • proxy_password: The password of the user whose user name is given in the proxy_userid field

  • encrypted_proxy_password: The encrypted proxy password generated using moog_encryptor
  • filter: The filter is used to filter alarms and events received from AWS.

    • alarms: The alarms are filtered by using either the alarm_name_prefix or the alarms_to_monitor field. In alarm_name_prefix, enter the prefix of the alarm name here e.g. "test", all the alarms having the text test in the starting of their name will be filtered and sent to Moogsoft AIOps GUI. In alarms_to_monitor the alarm name is given in this field e.g. "alarm1". It only sends the alarm entered here to Moogsoft AIOps GUI

      If alarms_to_monitor is left blank then all the alarms form the AWS account are forwarded to Moogsoft AIOps GUI
      At a time either the alarm_name_prefix or the alarms_to_monitor filter can be used


    • events:The events are fetched depending on the regions added in the events filter. The events will only be fetched from the region which are mentioned here. For example, we have 2 regions listed which are "us-west-2" and "ap-south-1". So, if the events filter is enabled with these 2 regions, then only the events from "us-west-2" and "ap-south-1" regions are fetched. 

      The region has 2 fields defined in it which are used for filtering within the events received from the region. The fields are filter_pattern and log_group_to_monitor. The events logged in the log group given in the log_group_to_monitor field having the pattern entered in the filter_pattern field are only forwarded to Moogsoft AIOps GUI. For example, the log group /aws/lambda/SomethingHappened has events with a word "scheduled" in it , so to filter the events having the word "scheduled" in it, "scheduled" is entered in  the filter_pattern field and  /aws/lambda/SomethingHappened is entered in the log_group_to_monitor field. 

      If filter_pattern and log_group_to_monitor is left blank then all the events form the region where it is left blank are sent to the LAM


    If alarms or events are not to be fetched, then comment the complete alarms or events section in the filter section of the config file

  • polling_interval: The polling time interval between the requests after which the event data is fetched from AWS. The polling interval is entered in seconds

    The default value is set to 60 seconds, if 0 is entered in this field then the time interval is by default set to 60 seconds

  • max_retries: The maximum number of retry attempts to reconnect with AWS in case of a connection failure

    The default value is set to 10, if 0 is entered in this field then the LAM by default takes the value 10 and will try at least 10 times to reconnect

    If all the number of retries are exhausted, then an alarm is sent to Moogsoft AIOps about the connection failure. For re-establishing the connection the LAM has to be restarted

  • retry_interval: The time interval between two successive retry attempts

    The default value is set to 60 seconds, if 0 is entered in this field then the time interval is by default set to 60 seconds

The entry in the fields proxy_port, polling_interval , max_retries and retry_interval should be an integer, therefore enter the values in these fields without quotation marks

Agent

Agent allows the user to define two parameters:

agent:
        {
                name    : "AWS Cloudwatch"
                #log    : "/var/log/moogsoft/aws_lam.log"
        },

The above example specifies: 

  • name: This is the agent name, the events sent to MooMS by the AWS LAM are identified by the agent name in the log. In this example the agent name is AWS Cloudwatch
  • log: In this instance, the AWS LAM will write its ingress contents to aws_lam.log located at /var/log/moogsoft

HA Configuration

Refer the document HA Configuration of LAM

Mapping

Variables section is not required in AWS LAM; a user can directly map the alarm/event fields of AWS with moogsoft fields displayed in Moogsoft AIOps.

mapping :
        {
            catchAll: "overflow",
            rules:
            [
                { name: "signature", rule:      "$signature" },
                { name: "source_id", rule:      "$source" },
                { name: "external_id", rule:    "$external_id" },
                { name: "manager", rule:        "AWS Cloudwatch" },
                { name: "source", rule:         "$source" },
                { name: "class", rule:          "$class" },
                { name: "agent", rule:          "$LamInstanceName" },
                { name: "agent_location", rule: "$source" },
                { name: "type", rule:           "$class" },
                { name: "severity", rule:       "$severity", conversion: "stringToInt" },
                { name: "description", rule:    "$description" },
                { name: "agent_time", rule:     "$agent_time" }
            ]
        },
        filter:
        {
            presend: "AwsLam.js"
        }

The above example specifies the mapping of the AWS alarm fields with the Moogsoft AIOps fields.

The signature field is used by the LAM to identify correlated alarms

The following images show the AWS Alarms, Events and alarms/events

AWS Alarms:


AWS Events:

Constants and Conversions

Constants and Conversions allow the user to convert formats of the received data.

 constants:
        {
            severity:
            {
                "CLEAR"             : 0,
                "INDETERMINATE"     : 1,
                "WARNING"           : 2,
                "MINOR"             : 3,
                "MAJOR"             : 4,
                "CRITICAL"          : 5,
            }
        },
        conversions:
        {
            stringToInt:
            {
                input:      "STRING",
                output:     "INTEGER"
            }
         
        },

The above example specifies:

  • Severity and sevConverter: The severity field has a conversion defined as sevConverter in the Conversions section, this looks up the value of severity defined in the severity section of constants and returns back the mapped integer corresponding to the severity  
  • stringToInt: It is used in a conversion, which forces the system to turn a string token into an integer value

Custom Info

The alarms/events are displayed in the Moogsoft AIOps, the data in the fields of the alarm or event mapped in the mapping section are shown in the respective columns of Moogsoft AIOps columns. The fields of alarms and events which are not mapped in the mapping section are displayed in the Custom Info field of the alarm. An example of Custom Info:

catchALL

The attribute that is never referenced in a rule is collected and placed as a JSON object in a variable called overflow defined here and passed as part of the event.

mapping :
        {
            catchAll: "overflow",
            rules:
            [
                { name: "signature", rule:      "$signature" },
                { name: "source_id", rule:      "$source" },
                { name: "external_id", rule:    "$external_id" },
                { name: "manager", rule:        "AWS Cloudwatch" },
                { name: "source", rule:         "$source" },
                { name: "class", rule:          "$class" },
                { name: "agent", rule:          "$LamInstanceName" },
                { name: "agent_location", rule: "$source" },
                { name: "type", rule:           "$class" },
                { name: "severity", rule:       "$severity", conversion: "stringToInt" },
                { name: "description", rule:    "$description" },
                { name: "agent_time", rule:     "$agent_time" }
            ]
        },

The AWS field first_occured is sent to AWS LAM. Since it is not mapped to a field in the aws_lam.conf file, it is placed in the overflow JSON object. The fields that are placed in the overflow variable can be viewed in the AWS LAM log file.

An example of an overflow JSON object created in the AWS LAM log file:

 "overflow":"{\"first_occurred\":1484735348818}"

Quotes

In some instances, the attribute strings are quoted. Our JSON parser ignores it, but the standard requires quoting for all strings, so Moogsoft recommends that user quote all strings.

Comments

A user can comment out lines by prefixing them with a hash. 

Starting the AWS LAM

To start the AWS LAM enter the following command:

service awslamd start

To stop the AWS LAM enter the following command:

service awslamd stop


To view the status of AWS LAM, enter the following command:

service awslamd status

Command line attributes

The aws_lam is a command line executable that can be run as a service daemon, and takes four attributes, which can be viewed by typing: 


 aws_lam --help

Option

Description

--config

Points to a pathname to find the configuration file for the LAM. This is where the entire configuration for the LAM is specified

--help

Displays all the command line options

--version

Displays the component’s version number

--log level

Specifies the level of debugging. By default, User gets everything. In common with all executables in MOOG, having it set at   that level can result in a lot of output (many messages per event message processed).

In all production implementations, it is recommended that log level is set to WARN, which only informs user of matters of importance

Optional Configurations for AWS LAM

If you don't not want to enter the Secret Access Key and the Access Key in the config file, then you can use any of the following methods to provide Secret Access Key and the Access Key to AWS LAM without entering it into config file:

Using Environment Variables

The Secret Access Key and the Access Key can be provided to the LAM using the environment variables. The Secret Access Key and the Access Key is set as an environment variable and the LAM can then utilize this environment variables to establish connection with AWS. To set the environment variables: 

  1. Open the .bashrc file using an editor e.g. vi /root/.bashrc
  2. Enter the Access Key and the Secret Access Key of your AWS account as shown in the following example

    # .bashrc
    
    # User specific aliases and functions
    
    alias rm='rm -i'
    alias cp='cp -i'
    alias mv='mv -i'
    
    # Source global definitions
    if [ -f /etc/bashrc ]; then
            . /etc/bashrc
    fi
    export AWS_ACCESS_KEY=AKIAJJYZQHUGXIT3AFE
    export AWS_SECRET_KEY=53fI7MetPl1kECQ0DM+/RhWph7yWLX1bL+gmIGV
    export MOOGSOFT_HOME=/usr/share/moogsoft
    export JAVA_HOME=/usr/java/latest
  3. Save the bashrc file.

  4. Enter the command source .bashrc.

The Access Key and the Secret Access Key are configured for the LAM and the user does not have to enter it in the config file.

Entering the keys in the awslam file

The Access Key and the Secret Access Key can also be entered in the the awslam file. To enter the keys in awslam file:

  1. Navigate to the Moogsoft bin folder located at /usr/share/moogsoft/bin or $MOOGSOFT_HOME/bin
  2. Open the awslam file using an editor e.g. vi awslam
  3. Enter the lines AWS_ACCESS_KEY and AWS_SECRET_KEY in the Set up the java environment section along with the Access Key and the Secret Access Key

    # Set up the java environment
    #
    AWS_ACCESS_KEY="AKIAJJYZQHUGXIT3AFEQ"
    AWS_SECRET_KEY="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"
    
    
    java_classpath="$APP_HOME/lib/lam_components-1.0.jar:$APP_HOME/lib/cots/httpmime-4.3.1.jar:$APP_HOME/lib/cots/httpcore-4.4.5.jar:$APP_HOME/lib/cots/httpclient-4.5.2.jar:$APP_HOME/lib/cots/commons-codec-1.10.jar:$APP_HOME/lib/cots/joda-time-2.8.2.jar:$APP_HOME/lib/cots/aws-java-sdk-1.11.158.jar:$APP_HOME/lib/cots/aws-java-sdk-cloudwatch-1.11.158.jar:$APP_HOME/lib/cots/aws-java-sdk-logs-1.11.158.jar:$APP_HOME/lib/cots/aws-java-sdk-core-1.11.158.jar:$APP_HOME/lib/cots/aws-java-sdk-ec2-1.11.158.jar:$APP_HOME/lib/cots/aws-java-sdk-s3-1.11.158.jar:$APP_HOME/lib/bot.jar:$APP_HOME/lib/mooms.jar:$APP_HOME/lib/dao.jar:$APP_HOME/lib/security.jar:$APP_HOME/lib/utilutils.jar:$APP_HOME/lib/servletutils.jar:$APP_HOME/lib/cots/commons-lang3-3.4.jar:$APP_HOME/lib/cots/commons-io-1.2.jar:$APP_HOME/lib/cots/commons-cli-1.2.jar:$APP_HOME/lib/cots/jackson-annotations-2.8.1.jar:$APP_HOME/lib/cots/jackson-core-2.8.1.jar:$APP_HOME/lib/cots/jackson-databind-2.8.1.jar:$APP_HOME/lib/cots/rhino-1.7R4.jar:$APP_HOME/lib/cots/mysql-connector-java-5.1.37.jar:$APP_HOME/lib/cots/amqp-client-3.6.5.jar:$APP_HOME/lib/cots/javax.servlet-api-3.1.0.jar:$APP_HOME/lib/cots/shiro-core-1.2.4.jar:$APP_HOME/lib/cots/snmp4j-2.5.2.jar:$APP_HOME/lib/cots/antlr4-runtime-4.5.3.jar:$APP_HOME/lib/cots/slf4j-api-1.6.4.jar:$APP_HOME/lib/cots/commons-logging-1.2.jar:$APP_HOME/lib/aws_lam-1.1.jar:$APP_HOME/lib/cots/nonDist/*"
    java_vm=$JAVA_HOME/bin/java
  4. Enter the following lines in the Run app section

    # Run app
    
    $java_vm ${JVM_OPTS[@]} -DprocName=$proc_name -DMOOGSOFT_HOME=$APP_HOME -classpath $java_classpath $java_main_class "$@" &
    #$java_vm ${JVM_OPTS[@]} -Daws.accessKeyId=$AWS_ACCESS_KEY  -Daws.secretKey=$AWS_SECRET_KEY  -DprocName=$proc_name -DMOOGSOFT_HOME=$APP_HOME -classpath $java_classpath $java_main_class "$@" &

The Access Key and the Secret Access Key are configured for the LAM and the user does not have to enter it in the config file.

Using a Credential File

The Access Key and the Secret Access Key can also be entered in a credential file, created at a default location. The LAM will automatically look at the default location for credentials file. To create the credentials file:

  1. Navigate to the directory /root/.aws. If the aws directory is not present then create the .aws directory using the mkdir command.
  2. Create a file named credentials using the commnad touch credentials 
  3. Enter the Access Key and Secret Access Key in the credentials file as shown below

    [default]
    aws_access_key_id=AKIAJJYZQHUGXIT3AFE
    aws_secret_access_key=53fI7MetPl1kECQ0DM+/RhWph7yWLX1bL+gmIGV

The Access Key and the Secret Access Key are configured for the LAM and the user does not have to enter it in the config file. 

Instance profile credentials delivered through the Amazon EC2 metadata service

Follow the link below to set up roles to access AWS resources using roles:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

Version Information

LAM Version

Tool Version

Tested?

Expected to Work

1.0

aws-java-sdk version: 1.11.61

Yes

Yes

1.1aws-java-sdk version: 1.11.61YesYes
1.2aws-java-sdk version: 1.11.61YesYes
1.3aws-java-sdk version: 1.11.158YesYes


System Information

This LAM was tested on a system with the following configurations:

CPU2 core
RAM4 GB
Operating SystemCentOS Linux release 6.7


The system must at least have the above mentioned system requirements to run the LAM.