Page tree
Skip to end of metadata
Go to start of metadata

Introduction

Archiving and removal of old Situation and Alert data is available using a Java based archiver and run from the command line. The archiver is located here:

$MOOGSOFT_HOME/bin/utils/moog_archiver 

and runs with the command line options detailed below.

The archiver interacts with the MySQL database and exports old data to files and/or removes them from the database. You can specify deletion of previously created archive files older than a certain age. Also, the archiver can connect to a remote MySQL database so that export data can be retrieved over the network. The archiver gets the database connection details from the mysql section of the $MOOGSOFT_HOME/config/system.conf file local to it.

The archiver is delivered as part of the following two Moogsoft AIOps installation packages:

  • moogsoft-db
  • moogsoft-utils

For more information about Moogsoft AIOps installation packages, see the installation guide

Archived data

CategoryDb TableDefaultStateExport to

Situations

Situation data

Alerts, Events and Snapshots (based on Situation membership of those Alerts)

Age=28 days

Age='old' (>30 days) is last_state_change column value

Closed, Dormant or Superseded


(pre-4.1.2 Alerts last_event_time column)
<table name>-<date>.csv

Alerts

Alerts, Events and Snapshots

Age=28 daysAge='old' (>30 days) is last_state_change column value

A different retention period can be set for those without any Situation membership 

<table name>-<date>.csv

Alerts with no Situation membership:
<table name>-loose<date>.csv

For example:

[root@moogbox2 moog-archives]# ls -l
total 84
-rw-rw-rw-. 1 mysql mysql 260 Apr 10 14:36 alerts-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 260 Apr 10 14:36 alerts-loose-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 77 Apr 10 14:36 comments-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 70 Apr 10 14:36 events-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 70 Apr 10 14:36 events-loose-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 97 Apr 10 14:36 invitations-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 86 Apr 10 14:36 sig_alerts-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 60 Apr 10 14:36 sig_competency_type_affected-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 75 Apr 10 14:36 sig_correlation_info-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 87 Apr 10 14:36 sig_experts_in_room-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 74 Apr 10 14:36 sig_journal-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 403 Apr 10 14:36 sigs-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 57 Apr 10 14:36 sig_services-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 87 Apr 10 14:36 sig_stats_cache-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 40 Apr 10 14:36 sig_users_impacted-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 77 Apr 10 14:36 sig_users_in_room-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 280 Apr 10 14:36 snapshots-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 280 Apr 10 14:36 snapshots-loose-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 106 Apr 10 14:36 thread_entries-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 51 Apr 10 14:36 thread_entry_feedback-20150410.143637.csv
-rw-rw-rw-. 1 mysql mysql 130 Apr 10 14:36 threads-20150410.143637.csv

Each export file contains the software version as the first line followed by lines for column names and rows of data. Values are separated by a configurable delimiter, the default being a comma. Data cells are quoted to handle any occurrences of the delimiter in a cell. A quote character in a cell is escaped with second quote character. A null value from the database will be written as \N in an archive file.

Users can specify a directory where the export files are written, otherwise the archiver will attempt to write them to /usr/local/archived

Prerequisite

Before running the archiver, do the following:

  • Make a directory for the archiver to export files to. By default the archiver will attempt to write files to the /usr/local/archived directory

    mkdir /usr/local/archived
    

    alternatively create a different directory and, when running the archiver, point to that directory for exporting files by using the -p option (see below).

Command line options

The archiver is located here:

$MOOGSOFT_HOME/bin/utils/moog_archiver 

and has the options:

moog_archiver

  Option Set 1: General Archive.
    [ --export | --remove [ --delimiter <delim> ] [ --file_age <age> ]
    [ --situation_age <age> ] [ --loose_alert_age <age> ] [ --archive_path <path> ] ]
    [ [ --situation_filter <filter_name> ] [ --alert_filter <filter_name> ] | --retain_open ]
    [ -h | --help ] [ --loglevel (INFO|WARN|ALL) ]

  Option Set 2: Archive Loose Alerts Only.
    --loose_alerts_only [ --export | --remove [ --delimiter <delim> ]
    [ --file_age <age> ] [ --loose_alert_age <age> ]
    [ --archive_path <path> ] [ --alert_filter <filter_name> ] ]
    [ -h | --help ] [ --loglevel (INFO|WARN|ALL) ]

  Option Set 3: Archive File Removal Only.
    --file_age <age> [ --archive_path <path> ]

usage:

MoogSoft moog_archiver: Utility to archives old Moog situation and alert
data.
 -a,--alert_filter <arg>        A global Alert Filter to further restrict
                                which loose alerts are archived
 -b,--update_batch_size <arg>   The maximum number of rows to insert or
                                remove at a time. Default is 1000.
 -d,--delimiter <arg>           Define delimiter. Default is ','.
 -e,--export                    Use this option if you want to export old
                                data to a file.
 -f,--file_age <arg>            Remove archive files older than this value
                                (in days). If not specified, files are not
                                removed, can be used standalone.
 -g,--loglevel <arg>            Specify (INFO|WARN|ALL) to choose the
                                amount of debug output
 -h,--help                      Print this message
 -i,--situation_filter <arg>    A global Situation Filter to further
                                restrict which Situations are archived
 -l,--loose_alert_age <arg>     Export data related to alerts (without any
                                situation membership) older than this
                                value (in days). Default is 28.
 -o,--retain_open               Use this option if you want to remove only
                                closed/dormant/superseded old data.
 -p,--archive_path <arg>        A path to an archive directory containing
                                the export files. Default is
                                /usr/local/archived.
 -r,--remove                    Use this option if you want to remove old
                                data from the database.
 -s,--situation_age <arg>       Export data related to situations
                                (including alerts with situation
                                membership) older than this value (in
                                days). Default is 28.
 -t,--loose_alerts_only         Use this option if you want to archive old
                                loose/lonely alerts only
 -y,--delay_time <arg>          The pause time (in Milliseconds) between
                                each batch remove. Default there is no
                                pause.
 -z,--id_batch_size <arg>       The maximum number of ids to batch at a
                                time. Default is 100.

Using Filters

The moog_archiver utility can be used with global Situation and Alert filters. The filters will limit the data that is eligible for archiving and then apply the specified (or default) age time constraints to that subset.
Caution:

  • The moog_archiver cannot use the retain_open option/feature if a filter is specified
     
  • Only global filters are allowed to be used, specifying a filter which does not exist for everyone will return an error message
     
  • Be wary of using filters that define a time criteria (i.e. older than a specified date) this can conflict with the specified situation/loose_alert age values (and their defaults of 28 days) i.e. a filter that lists Situations created within the last day coupled with a moog_archiver situation_age of 28 would mean that no Situations get archived
     
  • The Alert filter will only be applied to the loose Alerts, that is those Alerts which do not belong to a Situation. It will not apply to Alerts that belong to those Situations that are being archived

Loose Alerts Only Mode

You can use the moog_archiver to archive loose Alerts only. This means that only those Alerts which do not belong to a Situation will get archived. This is ideal for keeping Situations and their Alerts while pruning loose Alerts.
The --loose_alerts_only (-t) option cannot be used with --retain_open (-o) or --situation_age (-s) or --situation_filter (-i) options.
An Alert filter can be specified to filter which loose Alerts are considered for archiving (note the loose_alert_age value (default 28 days) will be applied to any filtered loose Alerts).

Archive File Maintenance

You can run the moog_archiver to only remove old archived files (i.e. no exporting or removing of data from the database will be done). You can do this by specifying just the --file_age (-f) option with the optional --archive_path (-p) option to direct the utility to a specific folder.

Examples

Command

Description

./moog_archiver -h

Print help

./moog_archiver -e

Export all data older than default age settings to default directory and do not remove that data from the database

./moog_archiver -e -d '|'

Export all data older than default age settings to default directory (using delimiter | in the export files) and do not remove that data from the database

./moog_archiver -e -f 3

Export all data older than default age settings to default directory (deleting any previous export files older than 3 days) and do not remove that data from the database

./moog_archiver -r

Remove all data older than default age settings from the database and do not export any of it

./moog_archiver -e -r -p /tmp/archive

Export data older than default age settings to directory /tmp/archive and then remove that data from the database

./moog_archiver -e -r -o

Export:

  • Closed/dormant/superseded related Situation data (and member Alerts) that have not changed in the last (default) 28 days and
  • Alerts with no Situation membership that have not changed in the last (default) 28 days

to default directory and then remove that data from the database

./moog_archiver -e -r -o -s 4

Export:

  • Closed/dormant/superseded related Situation data (and member Alerts) that have not changed in the last 4 days and
  • Alerts with no Situation membership that have not changed in the last (default) 28 days

to default directory and then remove that data from the database

./moog_archiver -e -r -o -s 7 -l 1

Export:

  • Closed/dormant/superseded related Situation data (and member Alerts) that have not changed in the last 7 days and
  • Alerts with no Situation membership that have not changed in the last day

to default directory and then remove that data from the database

./moog_archiver -r -s 0 -l 0

Remove all Situation and Alert data

./moog_archiver -r -s 0 -l 0 -i 
"My Global Situation Filter" -a "My Global Alert Filter"
Remove all Situations that match the specified global Situation filter and their Alerts. Remove all loose Alerts that match the specified global Alert filter
./moog_archiver -e -r -s 7 -i "My Global Situation Filter"
Export:
  • Situations that match the specified global Situation filter and their Alerts that have not changed in the last 7 days and
  • Alerts with no Situation membership that have not changed in the last (default) 28 days

to the default directory and then remove that data from the database

./moog_archiver -e -l 7 -a "My Global Alert Filter"
Export only:
  • All Situations that have not been changed in the past (default) 28 days and their Alerts
  • Loose Alerts that match the specified global Alert filter and which have not changed in the past 7 days

to the default directory

./moog_archiver -r -e -t
Export and remove loose Alerts only that have not changed in the last (default) 28 days, these are Alerts which do not belong to a Situation. Using -t ( --loose_alerts_only ) option means Situations and those Alerts which belong to Situations do not get archived
./moog_archiver -e -t -l 7
Export loose Alerts only that have not changed in the last 7 days , these are Alerts which do not belong to a Situation
./moog_archiver -f 7
Delete any previous exported files older than 7 days from the (default) /usr/local/archived directory. This will not export or remove data from the database
./moog_archiver -f 5 -p / tmp/archive
Delete any previous exported files older than 5 days from the specified /tmp/archive directory. This will not export or remove data from the database

After using the moog_archiver to remove data, the moog_indexer should be re-run with the following command, otherwise the search feature in the UI will be out of sync with the database:

moog_indexer -n
  • Until you are familiar with the command line options, it is recommended not to use the -r (remove) option to prevent any accidental data removal. Experiment with the age options using just the -e (export) option and examine the exported data to ensure it meets requirements
     
  • The moog_archiver can be used against a running system, however it is recommended to only use it out of hours or at quiet times to minimize the chance of data removal impacting any functions. UI users should also refresh their sessions if data removal has occurred
     
  • Using the moog_archiver to export and/or remove large amounts of data from a running system may take some time. As such, this script may be subject to performance optimizations in later releases
     
  • Exporting from a remote machine may take also longer as the data must traverse the network