Try It Yourself: Normalize Data

This lab is one of a series of Workflow Engine Labs. In this lab, you will:

  • Use the Workflow Engine to "clean up" the contents of an event field.

  • Make alert location information consistent so that you can use it to sort and filter alerts.

Examine the Data

Load data into your lab instance and decide how you want to process it.

  1. Clean up the data from the previous lab by closing any open alerts and all but one Situation. If there is no open Situation, generate one manually. (See the Mini Lab for instructions on how to do this.)

  2. Go to the Collaborate tab of your Situation’s Situation Room. Type @moog get_lab_events City into the comment box to bring in the data for this lab.

  3. Go to the Open Alerts view. In the View menu on the upper right, make the 'Agent Host' column visible, and sort by that column. The 'Agent Host' column corresponds to the Moogsoft 'agent_location' event field.

  4. Look at the list of locations. You can see that San Jose is shown in several ways: 'SANJ', 'SANJOSE', and 'sj'. The Seattle location is shown as 'SEA' or 'sea'. You decide you want to change the San Jose variants to 'San Jose' and the Seattle variants to 'Seattle' so that your operators can sort and filter by location. Right now you are not using location for event deduplication or alert clustering, but normalizing the location values would give you those options as well.

Define a Workflow

Begin setting up an event workflow that will normalize the value of location before events reach the Alert Builder.

  1. Go to Settings>Automation>Workflow Engine.

  2. Click on the Event Workflows tab and choose 'Add Workflow'.

  3. Examine the Event Workflow screen. At the top right, you can see a slider to make a workflow active or inactive. Leave it set to 'Active'.

  4. On the left is a column which you can populate with workflow actions that you choose. The first action, 'Delay', is already populated. Leave it at 0 seconds since you want to normalize the data as quickly as possible as events stream in.

  5. On the right is the workflow definition pane. You have a decision to make. You could build two separate workflows, using entry filters to select and process only the events for each city. Alternatively, you could build one workflow, matching and replacing the location field for each city in separate consecutive workflow actions.

    All the events will be processed either way so there will be little performance difference. In this case the choice is not critical, but you decide you want to see all your data edits described in your top-level workflow list so you will build two separate workflows. Name your workflow 'Normalize San Jose Location'.

  6. Fill in the description field with 'Change all 'sanjose' variants in 'agent_location' to 'San Jose''.

  7. Click on Entry Filter>Edit>Add Clause.

  8. Choose 'agent_location' in the drop-down menu and choose 'matches' for the comparison operator. The 'matches' operator looks for partial or complete text string matches and is case-insensitive.

  9. In the next text box, enter an expression for the text that you want to match. You can use regular expression syntax in this text box. Enter 'sanjose|sj|sanj' without the quotes. (The '|' operator means 'or'.)

  10. Even though you have only one data source now, if you add another data source in the future you will probably will want to set up separate workflows for data normalization. Select the clause in the graphic definition pane to make it active and then click on the 'and' operator. Fill in the text boxes to add the comparison 'manager = HWMON' to your entry filter. This ensures that your workflow only processes events from your hardware monitoring system.

  11. Click 'Apply' and then 'Done'. Your workflow is set up to accept and process only San Jose records. All other event records will bypass the workflow.

Add a Workflow Action

Add a workflow action to replace the contents of the 'agent_location' field.

  1. Go to the action column on the left, and click on 'Add Action'.

  2. In the action description pane on the right, enter 'Update Location' as the name for your action.

  3. The Function section in the action definition screen is a dropdown list of functions you can apply to incoming events. You can review how functions work in the Workflow Engine Functions Reference documentation. Choose 'replaceString'.Workflow Engine Functions Reference

  4. The Arguments section changes depending on the function you choose. You now have three text boxes to fill in. The first two boxes, 'field' and 'replace', tell the Workflow Engine where to look and what to look for: the name of the field to search, and the text you want it to search for. In this case you will repeat the logic you used in the entry filter, since you are interested in all the records with a location that is a variant of 'San Jose'. Enter 'agent_location' without the quotes in the field text box.

  5. In the expression text box, use regular expression syntax to tell the Workflow Engine what text strings to look for. Enter 'sanjose|sj|sanj'. Of course, all of the events being processed through this workflow already meet this condition because they passed the entry filter, but you do need to provide arguments to the replaceString function. Specifying the logic accurately makes the action self-documenting, and lets you make edits to the entry filter or other parts of the workflow later.

  6. The 'with' text box tells the Workflow Engine how to replace the strings it has identified. Enter 'San Jose'. This tells the Workflow Engine to replace any of the strings in the 'replace' text box with the text 'San Jose'.

  7. In the section at the bottom, leave the forwarding behavior as the default, 'Always Forward'.

  8. Click 'Save' to save your workflow.

Test Your Workflow

Test your workflow by resending the data into Moogsoft AIOps.

  1. Return to the Workbench and close all the open alerts and all but one Situation.

  2. Go back to the Situation you created earlier (or create a new one) and send the events into Moogsoft AIOps again using the @moog get_lab_events City ChatOps command.

  3. In the Open Alerts view, select the 'Agent Host' checkbox and then sort by location. You should see that all the San Jose alerts now read 'San Jose'.

Build a Second Workflow

Follow the steps you used for the San Jose workflow to build another workflow that normalizes the location value for Seattle events.

  1. Define the workflow, including an entry filter.

  2. Add an action to replace the contents of the 'agent_location' field with 'Seattle'.

  3. Test your workflow.