Tool Runner

In Moogsoft AIOps you can use the Tool Runner to run non-interactive command line tools that do not require root permission such as "ping" and "cat". It can not be used to execute Moogsoft AIOps client tools. The tool is a servlet running under Apache-Tomcat whose output is sent back to the web browser asynchronously.

All commands are run in a Linux shell via ssh and executed on a specified user@hostuser@host.

When a tool is run from the UI, a user/password@host can be specified but, if not provided, the tool runs on a default user@host (configured in web.xml).

Using server tools in the Moogsoft UI

From the alert view:

  • Right-click an alert to open the menu.

  • Select Tools > Server Tools to open Tool Runner.

For each alert, based on alert type, a list of configured tools is displayed with the following fields:

Field

Description

Tool

Display name of the tool.

Run from host

If left blank, the tool runs either on localhost, or the default machine (which you can configure). If specified, the tool runs on the specified host.

Username

If you specify a run host, use this field to specify the user to ssh into the run host.

Password

If a username is specified, this is the password used to log in to the run host. If left blank, the system attempts to use ssh public/private keys.

Fire & forget

If checked, the tool runs and all output is ignored (the equivalent of 'nohup &').

Tool Runner servlet configuration

Tool Runner is implemented as a servlet which can be configured using MySQL or web.xml.

Supporting configuration

Depending upon the exact environment, some support configurations may be required. Tool Runner uses ssh to log in to a system with a certain username and password to run tools/integrations. As a result, 'PasswordAuthentication' must to be set to 'yes' in /etc/ssh/sshd_config for this to work. Changes to that file require restarting the sshd service before they take effect.

MySQL

The database contains one table for the configuration of tools. The schema is as follows:

Element

Type

Description

tid

[INT AUTO_INCREMENT]

Alert tool ID.

displayname

[VARCHAR(100)]

Tool name displayed in the UI.

cmd

[VARCHAR(250)]

Tool command (with no arguments).

args

[VARCHAR(500)]

Arguments to the command.

alerttype

[TEXT]

Alert type of the alert that is used to determine the list of valid tools.

description

[TEXT]

A textual description.

When first installed, the table does not contain any tools. Tools can be added using the following SQL:

insert into moogdb.alert_tools value( 0,'Ping localhost', 'ping', '-c 5 localhost','SWBFence', 'This is an example tool');

Tool arguments

The arguments can be null if no argument is required. In addition, you can configure the argument to substitute information from the alert. For example:

        cmd = ping
        args = -c 5 $source -> -c 5 polyanna (after substitution)

You can use any alert internal field name in the argument substitution, for example $source, $source_id, $severity. The case must match the internal field name.

Escaped argument variables

When an argument is substituted into the argument string it will also be escaped using a backslash. This allows you to substitute values to contain special characters, and also provides some security against command insertion. Be careful, the combination of the escaped value and the original argument string can produce unexpected results. For example, the command echo with the following argument strings will produce slightly different results:

argsV1 = $source
argsV2 = "$source"
if $HOST = local.host
    resultV1= local.host
    resultV2 = local\.host

Only the values are escaped, not the original argument string. For further information on how to build the full argument string, see Combining multiple arguments into a single argument string.

Combining multiple arguments into a single argument string

The following command has two arguments:

printf "%s\n" "hello world"

If the 'hello' and 'world' were from $X and $Y, the configuration for the command would be as follows:

cmd = printf
args = "%s\n" $x\ $y (version 1)

Alternatively:

args = "%s\n" "$x $y" (version 2)

Due to the argument value escaping, version 1 is probably the better choice, although version 2 is probably more readable/natural.

web.xml

Here is an example web.xml (/usr/share/apache-tomcat/webapps/toolrunner/WEB-INF):

<web-app>
<servlet>
<servlet-name>toolrunner</servlet-name>
<servlet-class>com.moogsoft.toolrunner.CToolRunner</servlet-class>
<async-supported>true</async-supported>
<init-param>
<param-name>webhost</param-name>
<param-value>https://localhost</param-value>
</init-param>
<init-param>
<param-name>polluri</param-name>
<param-value>/poll</param-value>
</init-param>
<init-param>
<param-name>toolrunuri</param-name>
<param-value>/run</param-value>
</init-param>
<init-param>
<param-name>toolstopuri</param-name>
<param-value>/stop</param-value>
</init-param>
<init-param>
<param-name>sshtimeout</param-name>
<param-value>900000</param-value>
</init-param>
<init-param>
<param-name>toolrunnerhost</param-name>
<param-value>localhost</param-value>
</init-param>
<init-param>
<param-name>toolrunneruser</param-name>
<param-value>toolrunner</param-value>
</init-param>
<init-param>
<param-name>toolrunnerpassword</param-name>
<param-value>toolrunner</param-value>
</init-param>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>toolrunner</servlet-name>
<url-pattern>/poll</url-pattern>
<url-pattern>/run</url-pattern>
<url-pattern>/stop</url-pattern>
</servlet-mapping>
<listener>
<listener-class>com.moogsoft.toolrunner.CSessionListener</listener-class>
</listener>
</web-app>

The following parameters can be edited; however, these are 'global' settings and would apply to all tools:

webhost

Defines the value of the HTTP header 'Access-Control-Allow-Origin' that is used to control cross-origin resource sharing (CORS). The default value is https://localhost.

sshtimeout

If a tool running via ssh stops providing any output for a specified period of time, due to the tool hanging or a problem with the connection, sshtimeout can be used to shutdown any resources associated with the tool. This configuration value is specified in milliseconds and the default value is 900000 (1000 * 60 * 15 = 15 minutes).

toolrunnerhost / toolrunneruser / toolrunnerpassword

These three configurations relate to the running of tools 'locally'. From the UI, if only a tool name is specified then use/this configuration comes into play. There are basically two valid configurations:

  • All three configured: Any tools with no run host specified will run on the configured run host. The system will ssh to the run host using the configured username and password

  • Run host and username configured: If no password is configured, then the system will attempt to ssh to the run host using the configured username with a private key that must be located in a specific directory. For further information refer to Passwordless SSH.

SSH

You must have ssh installed on the remote machine that you want to run tools on.

Enable password authentication

In order to successfully log in to a remote machine, you must configure the remote ssh to allow password authentication. In /etc/ssh/sshd_config configure the following:

PasswordAuthentication yes

Passwordless SSH

To set up a user account that does not require a password:

  1. On the web server machine generate a secure SSH key by typing: ssh-keygen.

  2. Copy the generated key to the remote server. For each user account you plan to use to run tools you want to setup password-less logins with. Use the following command string, or, you can use scp: cat ~/.ssh/id_dsa.pub | ssh user@remotehost 'cat >> ~/.ssh/authorized_keys'. This command takes the generated SSH key from the local machine, connects to the remote host via SSH, and then uses cat to append the key file to the remote users authorized key list. As this connects with SSH to the remote machine, you will need to enter the password to use this command.

  3. Confirm that you can now login to the remote SSH server without a password: ssh user@remotehost.com.

  4. Copy the private key ( id_rsa) to a directory called keys, which you need to create in the Tomcat Apache home directory, for example, /export/apache-tomcat-7.0.29.

Additional considerations

  • At present stdout and stderr are combined together and displayed in the web UI.

  • If a tool is run that produces a large amount of output, it can take a significant amount of time for the output to be processed over to the web ui so expect a time lag.