Skip to main content

Troubleshoot the Workflow Engine

The Workflow Engine is a flexible tool that can help you transform your data and manage its flow through Moogsoft Onprem data processing. Workflow Engine functions provide you programmatic control over your data, but sometimes your workflows may not behave the way you expect.

This topic contains ideas to help you debug your workflows and get the most from the Workflow Engine.

Troubleshoot from the UI

Consider the following options for troubleshooting workflows from the Moogsoft Onprem UI:

  • If you have multiple workflows enabled, but one of them is behaving unexpectedly, try temporarily disabling the other workflows to see if it works on its own. If so, reactivate the other workflows one by one, testing at each step to see if one of the other workflows is affecting it.

  • Check the forwarding behavior for your workflow actions. The forwarding behavior controls subsequent processing when the function returns false. Stop This Workflow prevents the object passing to subsequent actions and Stop All Workflows prevents the object from passing to any subsequent action or workflow.

  • Test your entry filter for the workflow. If your objects are not meeting entry filter requirements, the workflow will not process them.

  • Verify your source fields and destination fields. Make sure that the names match up exactly. If you are using complex keys, make sure that you have the path exactly right. For example: custom_info.eventDetails.services.

  • If you are using an event that specifies a Moolet, check the Moolet name under Settings > Self Monitoring > Event Processing. For example, "Default Cookbook".

Troubleshoot from the moog_farmd.log

If you have access to the log for Moogfarmd, you have a lot more troubleshooting options to identify exactly what is happening with your objects as they progress through workflows.

You can find the Moogfarmd log at /var/logs/moogsoft. See Configure Logging for more information.Configure Logging

The Workflow Engine includes the following logging functions to help you troubleshoot:

The log messages from the Worfklow Engine include the engine name along with details about the object processing in the workflow. This means that you can use the tail command to observe the activity within an engine. For example:

tail -f MOO.moog_farmd.log | grep ":Alert.Workflows"

Within the log output, you can search for specific things, including:

  • The function name you are troubleshooting.

  • Identifying data for the object you are processing, such as the event signature.

  • Identifying information about an entry or sweep up filter.

See Example Workflow Engine log for sample messages and their meanings within the log context.

Example Workflow Engine log

The following log segment includes comments to highlight the different aspects of a Workflow Engine log:

### Alert did not pass the entry filter ###
DEBUG: [3:Enrichment Workflows][20191002 16:24:55.983 -0400] [CWorkflow.java:470] +|Moolet [Enrichment Workflows] - workflow [Closed Alerts Filter]: message [{"Elements":{active_sig_list=[67, 68], agent=DATA_SOURCE, agent_location=my_agent_location, alert_id=165, class=my_class, count=3, custom_info={eventDetails={agent=TestAgent1, first_occurred=1570047828, service=SAP, name=REST LAM Post 1, team=SAP Support}}, description=DESC: Host 1 Sig 1, entropy=0.8312803355385304, event_id=2899, external_id=my_external_id, first_event_time=1570047828, int_last_event_time=1570047828, last_event_time=1570047896, last_state_change=1570047828, manager=TestMgr1, owner=2, rc_probability=null, severity=5, sig_list=[67, 68], signature=lnux100:sig1, significance=3, source=lnux100, source_id=192.168.100.101, state=2, type=TestType1}, "Topic":"alerts", "Seq":"0", "SessId":"4769192054476008521", "Pdu":"E_MooMsg", "MessageId":"c2fc745a-8572-4982-a012-69fe64b84e96", "CorrelationId":"ff62fbbb-45ff-44f8-a1b5-9341bf33e729", "Metadata":{action=Event Added To Alert, clock_time=1570047895, message_type=1, previous_data={last_event_time=1570047869, count=2}, user_id=2}, "UsedCount":"null", "AckPoint":"0"}] failed to pass entry filter [state = 9].|+

### Workflow is inactive ###
DEBUG: [3:Enrichment Workflows][20191002 16:24:55.983 -0400] [CWorkflow.java:463] +|Moolet [Enrichment Workflows] - workflow [Enrich From SNow] inactive, sending message to the next Workflow/Moolet.|+

### Active workflow begins ###
DEBUG: [3:Enrichment Workflows][20191002 16:24:55.983 -0400] [CWorkflow.java:294] +|Moolet [Enrichment Workflows] - workflow [Test External DB]: starting delay of [0] seconds for msg [{"Elements":{active_sig_list=[67, 68], agent=DATA_SOURCE, agent_location=my_agent_location, alert_id=165, class=my_class, count=3, custom_info={eventDetails={agent=TestAgent1, first_occurred=1570047828, service=SAP, name=REST LAM Post 1, team=SAP Support}}, description=DESC: Host 1 Sig 1, entropy=0.8312803355385304, event_id=2899, external_id=my_external_id, first_event_time=1570047828, int_last_event_time=1570047828, last_event_time=1570047896, last_state_change=1570047828, manager=TestMgr1, owner=2, rc_probability=null, severity=5, sig_list=[67, 68], signature=lnux100:sig1, significance=3, source=lnux100, source_id=192.168.100.101, state=2, type=TestType1}, "Topic":"alerts", "Seq":"0", "SessId":"4769192054476008521", "Pdu":"E_MooMsg", "MessageId":"c2fc745a-8572-4982-a012-69fe64b84e96", "CorrelationId":"ff62fbbb-45ff-44f8-a1b5-9341bf33e729", "Metadata":{action=Event Added To Alert, clock_time=1570047895, message_type=1, previous_data={last_event_time=1570047869, count=2}, user_id=2}, "UsedCount":"null", "AckPoint":"0"}]|+

### Name of the function that is processing ###
DEBUG: [3:Enrichment Workflows][20191002 16:24:55.984 -0400] [CWorkflowBotAction.java:196] +|Performing action [enrichOneToOne]|+

### Depending on the function, different logs here ###

###  Alert updated ###
DEBUG: [3:Enrichment Workflows][20191002 16:24:56.096 -0400] [CMooMsg.java:1086] +|Encoded size [991] json[{"_MOOTADATA_":{"action":"Alert Updated","clock_time":1570047896,"message_type":1,"previous_data":{"custom_info":{"enrichment":null,"eventDetails":{}},"last_state_change":1570047828}},"active_sig_list":[67,68],"agent":"DATA_SOURCE","agent_location":"my_agent_location","alert_id":165,"class":"my_class","count":3,"custom_info":{"eventDetails":{"agent":"TestAgent1","first_occurred":1570047828,"service":"SAP","name":"REST LAM Post 1","team":"SAP Support"},"enrichment":{"ci":{"Name":"lnux100","AssetClass":"Linux Server"}}},"description":"DESC: Host 1 Sig 1","entropy":0.8312803355385304,"external_id":"my_external_id","first_event_time":1570047828,"int_last_event_time":1570047828,"last_event_time":1570047896,"last_state_change":1570047896,"manager":"TestMgr1","owner":2,"rc_probability":null,"severity":5,"sig_list":[67,68],"signature":"lnux100:sig1","significance":3,"source":"lnux100","source_id":"192.168.100.101","state":2,"type":"TestType1"}]|+

### Action completing with an exit status of 'true' ###
DEBUG: [3:Enrichment Workflows][20191002 16:24:56.103 -0400] [CMDB-WFE.js:403] +|Enrichment Workflows::enrichOneToOne: Exiting action with a status of true|+

### Workflow Finished and sending to next Moolet ###
DEBUG: [3:Enrichment Workflows][20191002 16:24:56.104 -0400] [CPassToNextMoolet.java:63] +|Moolet [Enrichment Workflows] - Sending message to the next Moolet|+
DEBUG: [3:Enrichment Workflows][20191002 16:24:56.104 -0400] [CMsgDispatch.java:516] +|Dispatching message from [Enrichment Workflows]|+

### Name of the next Moolet for the alert ###
DEBUG: [3:Enrichment Workflows][20191002 16:24:56.104 -0400] [CMsgDispatch.java:547] +|Dispatching to [MaintenanceWindowManager]|+
In this section: