Run Events Analyser

The Events Analyser is responsible for analysing the tokens within alerts and calculating their entropy values. The Events Analyser updates the alerts with the calculated entropy value and also updates the reference database with all the tokens and their associated entropy values.

  • Command Line Options

  • Run Events Analyser

    • Daily Run

    • Hourly Run

    • Run Events Analyser Manually

  • Multiple Streams and Partitions

  • Usage Examples

Command line options

The events_analyser command line executable accepts the following options:

Option

Input

Description

--config <arg>

String: <file path/name>

Name and path of the configuration file specific to running the Events Analyser. The default is events_analyser.conf.

Example: --config=$MOOGSOFT_HOME/etc/events_analyser.conf

-l, --loglevel <arg>

One of: WARN, INFO, DEBUG, TRACE, OFF

Specifies the amount of logging information. Defaults to WARN, which is the recommended level in all production implementations.

--incremental

-

Analyzes only new event data that was received since the last time the Events Analyser was run.

--readage <arg>

Number, followed by one of:

  • s (seconds)

  • m (minutes)

  • h (hours)

  • d (days)

  • w (weeks)

Amount of data to analyse, in seconds, minutes, hours, days or weeks.

Example: --readage 2w

--keepage <arg>

Number, followed by one of:

  • s (seconds)

  • m (minutes)

  • h (hours)

  • d (days)

  • w (weeks)

Amount of data to keep, in seconds, minutes, hours, days or weeks.

Example: --keepage 30d

--stream <arg>

String: <alert stream name>

Stream name to be given to the current analysis.

Example: --stream "PRIMARY"

--partition <arg>

String: <partition value>

Name of the partition to be analyzed. It must be a valid value of the partition_by field.

Example: --partition "SanFrancisco"

Run Events Analyser

Moogsoft recommends that you run the Events Analyser regularly as follows:

  • Daily: analyzes the last two weeks of data.

  • Hourly: in incremental mode which analyses all new event data since the last time the Events Analyser was run.

These default settings are specified in moog_init_server.sh.

You can also run the Events Analyser manually on an ad hoc basis.

Daily run

To initiate a daily run, that is, where all entropy values are calculated for the last two weeks of event data, you should specify the Events Analyser to run with the following command line options:

./events_analyser --readage 2w

In this case, the Events Analyser:

  • Uses the default configuration file $MOOGSOFT_HOME/etc/events_analyser.conf.

  • Analyzes all data received in the last two weeks, based on the timestamp_column property in the events_analyser.conf file.

  • Adds all analyzed data to the reference database for the default stream.

  • Leaves any data for other, named streams unchanged.

Hourly run

The Events Analyser utility provides the ability for incremental priming. When the Events Analyser utility is run repeatedly with the --incremental option, each subsequent run of the utility analyses the event data starting from the last analyzed event. For example, if the first run analyzes data up to event ID = 666, the next incremental run of the utility analyzes data from 667 to say 999, the third incremental run reads in data from event ID 1000, and so on.

To initiate an hourly run, that is, where all entropy values are calculated since the last analyzed event, you should specify the Events Analyser to run with the following command line options:

./events_analyser --incremental

In this case, the Events Analyser:

  • Uses the default configuration file $MOOGSOFT_HOME/etc/events_analyser.conf.

  • Analyzes all data since the last incremental run, based on the timestamp_column property in the events_analyser.conf file.

  • Adds all analyzed data to the reference database for the default stream.

  • Leaves any data for other, named, event streams unchanged.

Run Events Analyser manually

To run the Events Analyser manually, you can run it without any command line options. This command runs the Events Analyser for all new event data received in the last two weeks or since the last analysis, whichever is most recent.

./events_analyser

In this case, the Events Analyser:

  • Uses the default configuration file $MOOGSOFT_HOME/etc/events_analyser.conf.

  • Analyzes all event data received in the last two weeks or since the last time the Events Analyser was run, whichever is most recent, based on the timestamp_column property in the event_analyser.conf file.

  • Adds all analyzed data to the reference database for the default stream.

  • Leaves any data for other, named, event streams unchanged.

To run the Events Analyser to analyze event data over a longer period, you should include the --readage option. In this example, the --readage option is set to 13 weeks:

./events_analyser --readage 13w

In this case, the Events Analyser:

  • Uses the default configuration file $MOOGSOFT_HOME/etc/events_analyser.conf.

  • Analyzes all event data received in the last 13 weeks.

  • Adds all analyzed data to the reference database for the default stream.

  • Leaves any data for other, named, event streams unchanged.

Note

If you use a large value in the--readageoption, you may find that the Events Analyser fails to complete the analysis. If this occurs, rerun it using a shorter period of time.

Multiple streams and partitions

You can run the Events Analyser for specific streams or partitions. In this example, the --stream option is specified to add the analyzed data to the "SECONDARY" event stream. The --readage option restricts the data analyzed to the last eight weeks of event data.

./events_analyser --stream “SECONDARY” --readage 8w

In this case, the Events Analyser:

  • Uses the default config file $MOOGSOFT_HOME/etc/events_analyser.conf.

  • Analyzes all event data received in the last eight weeks, based on the timestamp_column property in the event_analyser.conf file.

  • Adds all analyzed data to the reference database for the “SECONDARY” event stream.

  • Leaves data for all other, named, event streams unchanged.

You can use the --partition option to limit the data that is analysed to a specified partition. In this example, the --readage option restricts the data analyzed to the last four weeks of event data:

./events_analyser --stream “SECONDARY” --partition “SanFrancisco” --readage 4w

In this case, the Events Analyser:

  • Uses the default config file $MOOGSOFT_HOME/etc/events_analyser.conf.

  • Analyzes all event data received in the last four weeks for the “SanFrancisco” partition only.

  • Adds all analyzed data to the reference database for the “SanFrancisco” partition in the “SECONDARY” event stream.

  • Leaves data for all other event streams and partitions unchanged.

Note

Moogsoft recommends that you always use the --readage option when analyzing streams or partitions to ensure that the Events Analyser processes the required amount of data. If the --readage option is not specified, the Events Analyser only analyzes new event data received in the last two weeks or since the last analysis, whichever is the most recent, regardless of whether this was for a different stream or partition.

Usage examples

There are many combinations of command line options. Some common usage scenarios include:

Command Line Options

Typical Use Case

<none>

  • Events analysis to be run incrementally.

  • Uses the default configuration for all new event data in the last two weeks or since the last analysis, whichever is most recent.

  • Updates the reference database with the new data for the default stream.

--readage 4w

  • Events analysis to be run nightly.

  • Uses the default configuration for the last four weeks of event data.

  • Updates the reference database with the new data for the default stream.

--incremental

  • An incremental events analysis to be run hourly.

  • Uses the default configuration for the all new event data since the last run.

  • Updates the reference database with the new data for the default stream.

--incremental

--keepage 2w

  • An incremental events analysis to be run hourly.

  • Uses the default configuration for all new event data received since the last run.

  • Removes all data from the reference database for the default stream that is more than two weeks old.

--stream “PRIMARY”

--partition “London” --readage 13w

  • Performs an events analysis analyzing only those events in the “London” partition.

  • The data is written to the “PRIMARY” event stream.

  • Data for all other streams remains unchanged.

  • Data for all other partitions in the “PRIMARY” stream remains unchanged.