Skip to main content

Moogsoft Hybrid

Overview

This topic describes the deployment architecture for integrating the two Moogsoft platforms: Moogsoft Onprem (also referred to here as MO, or Onprem) and Moogsoft Cloud (also referred to here as MC or Cloud).

This architecture is designed to provide the following:

  • For an existing Moogsoft Onprem customer, it provides the ability to adopt Moogsoft Cloud without the need to immediately re-engineer event ingestion and enrichment. For example, it can allow a discrete team to benefit from metrics and event correlation capabilities, while other teams continue to use Onprem.

  • For a new Moogsoft Cloud customer, it provides a roadmap bridge between Onprem and Cloud, allowing selected Onprem functionality to be used as the Cloud roadmap is fulfilled.

Important

This architecture is designed as a short- to medium-term solution and is not intended for use as a permanent deployment solution.

For both use cases, the supported functionality used within Moogsoft Onprem is restricted to the following:

  • Event Ingestion

    • LAMs and transformational Event Workflows

    • Event export to Cloud

  • Entropy

    • Use of the Alert Analyzer, application of an entropy value to an alert

  • Enrichment

    • Use of GA WFE-based enrichers (JDBC, ServiceNow, REST)

    • Enrichment API: new customers should use the Data Catalog in Cloud for enrichment, unless the enrichment data is used for Onprem functionality (routing, maintenance window selection, and other Onprem-specific tasks)

  • Maintenance Window Management

    • GA Maintenance Window Manager, as part of the standard Alert workflow (this does not include the Related Change functionality)

  • Alert Workflows

    • Alert payload manipulation and routing

    • Alert export to Cloud using the sendToCloud action

  • Inform-based Workflows (Cloud to Onprem)

    • Selected WFE actions that support eventless triggering, including:

      • getPayloadFromInform

      • emailFromInform

      • emailUsingTemplateFromInform

      • exportViaRestFromInform, exportViaKafkaFromInform, exportViaJDBCFromInform

    • Selected workflows to synchronize Onprem and Cloud alert data

      • setCustomInfoValue

  • Selected Integrations

    • Email Endpoints and associated WFE actions

    • Export workflows

      • REST, Kafka JDBC

Important

Integrations that are not included in this list (for example, trouble ticket integrations) are not supported.

Correlation and Moogsoft Onprem Situations

Within this Hybrid architecture, correlation using Onprem Tempus and Cookbook functionality is unsupported. Use the Cloud Correlation Engine when correlating events whose final destination is Moogsoft Cloud.

Exporting Onprem Situations to Cloud Incidents is not supported. Onprem Situations can be exported to Cloud as events if required (as for a tiered Onprem → Cloud architecture).

Architectural Flow Diagram:
Architectural Flow Diagram

NOTES:

  • Event traffic can be passed directly to Cloud or via Onprem as necessary, but should not be passed to both.

  • Events should be passed to Onprem only if they need to have Onprem functionality applied (such as entropy or enrichment).

  • Metric data should not be passed to Onprem. Metrics should be passed to Cloud directly.

  • To minimize the probability of event loss, use a Retry Queue with the sendToCloud WFE action.

  • Workflows from Cloud to Onprem are triggered using webhooks from Cloud to the Onprem Graze API call sendToWorkflow.

When to export: Events or Alerts?

The Moogsoft Hybrid architecture supports exporting both events (via the Event WFE) and alerts (via an Alert WFE). In general, to prevent the complexities associated with filtering at the point of export, it is advised to export from either Events or Alerts, but not both.

  • Export Events if additional Alert functionality is not needed (entropy, enrichment, maintenance windows).

    In this use case Onprem is used solely as an event ingestion/normalization layer.

  • Export Alerts if additional Alert functionality is required.

It is a best practice to export Alerts from the Alert Workflow engine (see below).

Use Moogsoft Onprem Workflows from Moogsoft Cloud

Specific Onprem integrations and workflows have been modified to allow them to be launched from Cloud. These modified actions are “eventless,” allowing the action parameters and details to be carried in the inbound payload rather than being taken from an event or alert.

The following relevant WFE actions are eventless:

  • addToContext

    Add a key/value to the workflow context.

  • between

    Check to see if the current execution time is between specific hours on specific days.

  • copyToInformPayload

    Copy a value to the workflowContext inform payload (as during an export workflow).

  • exportViaJDBCFromInform

    Export data to a JDBC endpoint.

  • exportViaKafkaFromInform

    Export data to a Kafka endpoint.

  • exportViaRestFromInform

    Export data to a REST endpoint.

    Included for completeness only; using webhooks is the recommended method for exporting via REST from Cloud.

  • getPayloadFromInform

    Create a payload using data from the Inform and not a CEvent object.

  • logWorkflowContext

    Log the workflow context to the moog_farmd debug.

  • sendEmailFromInform

    Send an email using data from the Inform and not a CEvent object.

  • sendEmailUsingTemplateFromInform

    Send an email using a predefined template and data from the Inform and not a CEvent object.

  • sendToWorkflow

    Trigger a named workflow.

  • workflowContextSearchAndReplace

    Search and replace using data in the workflowContext.

sendToWorkflow Graze API Call

Use the Graze API call sendToWorkflow to launch an Onprem Workflow from Cloud. It has the following parameters:

Name

Description

engine_name

Name of an active Inform Workflow Engine.

workflow_name

Name of an active workflow within the specified Inform Workflow Engine.

context

Additional context to send with the message. This data will be available in the workflowContext of the action within the workflow and can be substituted using the macro substitution (example: $(workflowContext.myData)).

Example 1. Sending an email from Cloud when an incident is created

  1. Create an Email Endpoint in Onprem in the usual way, specifying necessary information (host, port, credentials).

  2. Create a Workflow in the “Cloud Inform Workflows” WFE engine called “sendEmail.”

  3. Add the sendEmailFromInform action to this workflow:

    Either hardcode, or use workflowContext substitution for the parameters

    • emailConfig - the email endpoint you defined in Step 1

    • sendTo - A CSV list of recipients

    • subject - The subject for the email; generally a workflowContext substitution

    • message - The message body; generally a workflowContext substitution

    In the example below, the emailConfig is hard-coded, and the other parameters use a combination of $(workflowContext) substitution and hard-coded text.

    sendEmail_action.png

  4. Create an outbound webhook in Cloud to call this workflow, passing the desired information in the “context" part of the body.

    {
  "workflow_name":"sendEmail",
  "context":{
    "incidentId":"$id",
    "description":"$description",
    "severity":"$severity"
  },
  "engine_name":"Cloud Inform Workflows"
}

    1. Ensure that the headers have a “content-type” : “application-json” set, and the URL is set to:

      https://<MO_UI_SERVER>/graze/v1/sendToWorkflow

    2. Ensure that the Cloud Webhook Authorization has a Graze user defined, that the Graze user has the moolet_inform privilege, and that the type is set to “Incident.”

      Add a filter to ensure that this Webhook is triggered on Incident creation only. For example, status = Open.

      enterprise-82-moc-integration-moc-setup.png

  5. When an Incident is created in Cloud, the webhook triggers the named workflow in Onprem, using the details passed in the webhook.

    Important

    Workflow parameter macro Substitution Limitations

    Macro support in substitution is supported with the exception of the $EXPAND() macro. $EXPAND() retrieves data from CEvent objects which will not exist in an Inform-based export. 

    A $SEVERITY() macro has been added to allow conversion of a numeric severity to a textual severity within the Inform-based payload.



Export Workflows

The Moogsoft Hybrid architecture supports the following Onprem-based export workflow actions:

  • exportViaRestFromInform

  • exportViaKafkaFromInform

  • exportViaJDBCViaInform

These actions have the same behavior as their non-Inform based counterparts (exportViaRest, exportViaKafka, exportViaJDBC) but are modified to be eventless. These modified actions only use a payload created using the getPayloadFromInform action—using the standard getPayload action will fail to produce a viable payload.

Creating an Inform-based Export Workflow

  1. Use the standard REST Endpoint, JDBC Endpoint or Kafka Endpoint integration to define the target of the export.

  2. Use the standard Payloads integration to define an outbound payload.

    The payload configured here should use only workflowContext substitution and the amended macro support (see below).

  3. Create a workflow for the export in the usual way with the following actions:

    • getPayloadFromInform - specify the payload name configured in step 2.

    • exportVia[Rest | Kafka | JDBC]FromInform - specify the appropriate endpoint configured in step 1.

  4. Other eventless actions (such as workflowContextSearchAndReplace and copyToInfromPayload) may be used in the workflow, if needed.

Important

Payload Macro Substitution Limitations

Like the Inform-based workflow action substitution, the payload substitution supports the full set of payload macros with the exception of $EXPAND(). $EXPAND() retrieves data from CEvent objects which do not exist in an Inform-based export.

The $SEVERITY() macro allows the conversion of a numeric severity to a textual severity within the Inform-based payload.

Configure the Moogsoft Hybrid Integration

Configure Moogsoft Onprem:

  1. Disable unsupported Onprem components (new installations only), including:

    • Cookbooks (including default): via the Cookbook select configuration setting.

    • Tempus: via the Tempus configuration setting.

  2. Delete all default alert and Situation auto-close rules.

    Delete the default alert and Situation tasks under Settings → Automation → Auto Close

  3. Delete all predefined workflows from all workflow engines (new installations)

    NOTE: When the Email Endpoints integration is installed, workflows are automatically added to the Alert, Situation, Alert Inform and Situation Inform engines. Delete these.

  4. Add a suitable Graze user (example: cloudgraze).

    Ensure that the user is a “grazer” and has the “moolet_inform” privilege.

    See Set up outbound notifications for information about authentication for outbound webhooks.

  5. Add the Email Endpoints integration if required, configuring at least one endpoint, and templates as needed.

    • Delete default email workflows from all WFE engines after installation.

  6. Add the Hybrid Integration.

    1. Configure at least one Cloud endpoint.

    2. Modify the default Alert payload as needed.

  7. Add the Retry Queues Integration.

    1. Configure at least one Retry queue.

    2. For Cloud, the retry value should be 30 seconds (the lowest granularity) with a retry count of 3. Select the “Discard Closed Items” option.

      RetryQueues_for_MOC.png

  8. Add an “Cloud Export” workflow to the Alert Workflow Engine. This exports all inline alert changes to Cloud (Alert creation, Alert deduplication). Alert Updates made by other mechanisms (example: Graze, MoogDb API) are not sent.

    This workflow can use an optional entry filter (such as to limit the scope of the alerts that will be sent to Cloud).

    {
    "first_match_only": false,
    "sequence": 3,
    "operations": [
      {
        "duration": 0,
        "reset": false,
        "type": "delay"
      },
      {
        "operation_name": "Export to Cloud",
        "function_name": "sendToCloud",
        "forwarding_behavior": "always forward",
        "function_args": {
          "retryQueue": "AlertRetry",
          "endpointName": "DEMO",
          "mapName": "AlertMap"
        },
        "type": "action"
      }
    ],
    "moolet_name": "Alert Workflows",
    "workflow_name": "Cloud Export",
    "entry_filter": "",
    "sweep_up_filter_time_limit": null,
    "active": true,
    "description": "Export an alert to Cloud",
    "sweep_up_filter": "",
    "id": 133
 }

  9. Add a "Retry Cloud Export” workflow to the Alert Inform Engine. This workflow should be a copy of the Cloud Export in the Alert Workflow Engine (defined above), using the same Cloud Endpoint, map and retry queue as the base export.

    This is the workflow that will be referenced in the RetryQueue configuration and is the workflow that will be called when a Cloud export fails and is retried.

  10. Configure an outbound webhook from Moogsoft Cloud to Moogsoft Onprem for alert synchronization. See Set up outbound notifications.

    1. Add the webhook to update the Onprem alert with Cloud alert details and closing it when the Cloud alert closes:

      1. During installation, the Moogsoft Hybrid integration adds a "Sync Cloud Alert ID" and a "Sync Cloud Alert Close” workflow to the Cloud Inform Workflows.

        You should not need to modify these workflows.

      2. In Cloud, add a new outbound webhook with a target URL of the Onprem server.

        https:<onprem_server>/graze/v1/sendToWorkflow

        1713865199.png

      3. Set the filter condition to status in [Open, Closed].

      4. Add basic auth graze credentials for the Onprem server.

      5. In section 3, add the following:

        {
  "workflow_name":"Sync Cloud Alert Id",
  "alert_id":"$tags.onprem_id",
  "context":{
      "onpremId":"$tags.onprem_id",
      "key":"cloudAlert",
      "value":"$alert_id"
    },
    "engine_name":"Cloud Inform Workflows",
}

      6. In section 4, (Map Response payload) - Leave as “None”

      7. In section 5, enable update notifications and set a target URL of the Onprem server:

        https://<onprem_server>/graze/v1/sendToWorkflow

        1713865196.png

      8. In section 6, add the following body for update operations.

        {
  "workflow_name":"Sync Cloud Alert Close",
  "alert_id":"$tags.onprem_id",
  "context":{
    "status":"$status"
  },
  "engine_name":"Cloud Inform Workflows"
}

      Note

      When creating the bodies for both the alert details and close outbound webhooks ensure that the bodies are as above. DO NOT use $ typeahead during the creation as this inserts additional quotes that will not allow the workflow to run.

Using the Moogsoft Onprem Maintenance Window Manager

One of the identified gaps that the Moogsoft Hybrid architecture is designed to fill is Maintenance or change windows.

These behave in the same way in the Moogsoft Hybrid architecture as they would in a standard Onprem architecture, but with the additional requirement to export the alert from Onprem to Cloud when an alert enters and leaves a maintenance window to ensure that the current maintenance status in Cloud is reflected accurately.

To allow the export on entry and exit of a maintenance window, two actions were added in the Add-Ons 2.3.5 release:

  • enteringMaintenanceWindow

    Returns true when an alert is detected entering a maintenance window.

  • leavingMaintenanceWindow

    Returns true when an alert is detected leaving a maintenance window.

Using the actions

Both the enteringMaintenanceWindow and leavingMaintenanceWindow actions are event delta driven—they both look at the changes to an alert to determine if a window has been entered or exited. These changes are only visible in an Alert Update message. As a result, these actions need to be used in a workflow engine that is both standalone and listening for the AlertUpdate event handler.

The default "Alert Actions Workflows” engine is both standalone and listens for the correct event handler, and should be used by default.

Important

If the Cloud Export workflow is in the Alert Workflows WFE engine, and this is processing the output of the Maintenance Window Manager Moolet, then there should be no need to configure the “Export on Entry” workflow—an event triggering a change in the Maintenance status is exported as part of the standard event processing chain.

Ensure that maintenance windows forward alerts (checkbox needs to be ticked, or API call needs to include the forward_alerts parameter set to true).

To catch both entry and exit, two workflows need to be configured:

Workflow 1: Export on Entry (see note above)

entry filter : none (every alert update is examined)

  • Action 1 : enteringMaintenanceWindow

    Forwarding behavior must be set to “Stop this workflow”

  • Action 2 : sendToCloud

    The parameters should be the same Cloud endpoint, payload map, and retry queue that are configured for the core export.

Workflow 2: Export on Exit

entry filter : none (every alert update is examined)

  • Action 1 : leavingMaintenanceWindow

    Forwarding behavior must be set to “Stop this workflow”

  • Action 2 : sendToCloud

    The parameters should be the same Cloud endpoint, payload map, and retry queue that are configured for the core export.

Example Action Definition for Export on Exit
[
  {
    "first_match_only": false,
    "sequence": 1,
    "operations": [
      {
        "duration": 0,
        "reset": false,
        "type": "delay"
      },
      {
        "operation_name": "Check maintenance status",
        "function_name": "leavingMaintenanceWindow",
        "forwarding_behavior": "stop this workflow",
        "function_args": {},
        "type": "action"
      },
      {
        "operation_name": "Export",
        "function_name": "sendToCloud",
        "forwarding_behavior": "always forward",
        "function_args": {
          "endpointName": "Demo",
          "mapName": "AlertMap"
        },
        "type": "action"
      }
    ],
    "moolet_name": "Alert Actions",
    "workflow_name": "Export Leaving Maintenance ",
    "entry_filter": "",
    "sweep_up_filter_time_limit": null,
    "active": true,
    "description": "Export an alert when it leaves a maintenance window. ",
    "sweep_up_filter": "",
    "id": 32
  }
]
In this section: