Moogsoft Docs

Tool Runner

In Moogsoft AIOps the tool runner is used to run non-interactive command line tools that do not require root permission e.g. ping, cat. The output of running a tool will be asynchronously sent back to the web browser.

All commands are run in a Linux shell via ssh and executed on a specified user@hostuser@host.

When a tool is run from the UI, a user/password@host can be specified, but if not provided then the tool will be run on a default user@host (configured in web.xml ).

Starting the tool runner through the Moogsoft UI

From the Alert View:

  • Right-click an Alert to open the menu
  • Select Tools to open the tool runner

For each Alert (at present based on Alert type), a list of configured tools will be displayed with the following fields:

Field

Description

Tool

Display name of the tool

Run host

If left blank, the tool will run either on localhost, or, the default machine (which you can configure). If specified, the tool will run on the specified host

Username

If a run host is specified, then the username of the user that will be used to ssh into the run host needs to be specified

Password

If a username is specified, then this is the password that is used to log in to the run host. If left blank, the system will attempt to use ssh public/private keys

Fire & forget

If checked, then the tool will run and all output is ignored (basically the equivalent of 'nohup &')

Tool Runner Servlet Configuration

The tool runner is implemented as a servlet which can be configured using web.xml or MySQL .

Supporting Configuration

Depending upon the exact environment, some support configuration may be required:

  • The toolrunner uses ssh to login to a system with a certain username and password to run tools/integrations. As a result, 'PasswordAuthentication' must to be set to 'yes' in /etc/ssh/sshd_config for this to work. Changes to that file require restarting the sshd service before they take effect.

MySQL

The database contains one table for the configuration of tools. The schema is as follows:

Element

Type

Description

tid    

[INT AUTO_INCREMENT]

Alert Tool Id

displayname 

[VARCHAR(100)]

Tool Display Name (the name displayed in the UI)

cmd 

[VARCHAR(250)]

Tool Command (the actual command with no arguments)

args 

[VARCHAR(500)]

Arguments to the command

alerttype 

[TEXT]

Alert type of the alert that is used to determine the list of valid tools

description 

[TEXT]

A textual description

When first installed, the table does not contain any tools. Tools can be added using the following SQL:

insert into moogdb.alert_tools value( 0,'Ping localhost', 'ping', '-c 5 localhost','SWBFence', 'This is an example tool');

Args

The args can be null if no argument is required. In addition, the argument can be configured to substitute information from the Alert, for example:

        cmd = ping
        args = -c 5 $source -> -c 5 polyanna (after substitution)

You can use any Alert internal field name in the argument substitution e.g., $source,$source_id, $severity  (The case should match the internal field name).

Escaped arg variables

When an argument is substituted into the argument string it will also be escaped using a backslash. This allows you to substitute values to contain special characters, and also provides some security against command insertion. Be careful, the combination of the escaped value and the original argument string can produce unexpected results. For example, the command echo with the following argument strings will produce slightly different results:

        argsV1 = $source
        argsV2 = "$source"
        If $HOST = local.host
        resultV1= local.host
        resultV2 = local\.host

Only the values will be escaped, not the original argument string. For further information on how to build the full argument string, read the next section.

Combining multiple argument into a single argument string

The following command has 2 arguments:

        printf "%s\n" "hello world"

If the 'hello' and 'world' were from $X and $Y, the configuration for the command would be as follows:

        cmd = printf

        args = "%s\n" $x\ $y (version 1)

Alternatively:

        args = "%s\n" "$x $y" (version 2)

Due to the argument value escaping, version 1 is probably the better choice (although version 2 is probably more readable/natural).

web.xml

Here is an example web.xml ( /usr/share/apache-tomcat/webapps/toolrunner/WEB-INF ):

<web-app>
<servlet>
<servlet-name>toolrunner</servlet-name>
<servlet-class>com.moogsoft.toolrunner.CToolRunner</servlet-class>
<async-supported>true</async-supported>
<init-param>
<param-name>webhost</param-name>
<param-value>https://localhost</param-value>
</init-param>
<init-param>
<param-name>polluri</param-name>
<param-value>/poll</param-value>
</init-param>
<init-param>
<param-name>toolrunuri</param-name>
<param-value>/run</param-value>
</init-param>
<init-param>
<param-name>toolstopuri</param-name>
<param-value>/stop</param-value>
</init-param>
<init-param>
<param-name>sshtimeout</param-name>
<param-value>900000</param-value>
</init-param>
<init-param>
<param-name>toolrunnerhost</param-name>
<param-value>localhost</param-value>
</init-param>
<init-param>
<param-name>toolrunneruser</param-name>
<param-value>toolrunner</param-value>
</init-param>
<init-param>
<param-name>toolrunnerpassword</param-name>
<param-value>toolrunner</param-value>
</init-param>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>toolrunner</servlet-name>
<url-pattern>/poll</url-pattern>
<url-pattern>/run</url-pattern>
<url-pattern>/stop</url-pattern>
</servlet-mapping>
<listener>
<listener-class>com.moogsoft.toolrunner.CSessionListener</listener-class>
</listener>
</web-app>

The following parameters can be edited; however,  these are 'global' settings and would apply to all tools:

webhost

Defines the value of the HTTP header 'Access-Control-Allow-Origin' that is used to control cross-origin resource sharing (CORS). The default value is https://localhost .

sshtimeout

If a tool running via ssh stops providing any output for a specified period of time, due to the tool hanging or a problem with the connection, sshtimeout can be used to shutdown any resources associated with the tool. This configuration value is specified in milliseconds and the default value is 900000 (1000 * 60 * 15 = 15 minutes).

toolrunnerhost / toolrunneruser / toolrunnerpassword

These three configurations relate to the running of tools 'locally'. From the UI, if only a tool name is specified then use/this configuration comes into play. There are basically two valid configurations:

  • All three configured: Any tools with no run host specified will run on the configured run host. The system will ssh to the run host using the configured username and password
  • Run host and username configured: If no password is configured, then the system will attempt to ssh to the run host using the configured username with a private key that must be located in a specific directory. For further information refer to Password-less Ssh

Nothing configured is no longer valid. This concerned any tools with no run host specified to run on the web server; however, now if nothing is configured, the tool runner will fail to start up.

SSH

ssh must be installed on the remote machine that you want to run tools on.

Enable Password Authentication

In order to successfully log in to a remote machine, the system requires that the remote ssh is configured to allow password authentication. In /etc/ssh/sshd_config configure the following:

PasswordAuthentication yes

Passwordless Ssh

To setup a user account that does not require a password, you need to do the following:

  1. On the web server machine generate a secure SSH key by typing: ssh-keygen.
  2. Copy the generated key to the remote server. For each user account you plan to use to run tools you want to setup password-less logins with. Use the following command string, or, you can use scp: cat ~/.ssh/id_dsa.pub | ssh user@remotehost 'cat >> ~/.ssh/authorized_keys' . This command takes the generated SSH key from the local machine, connects to the remote host via SSH, and then uses cat to append the key file to the remote users authorized key list. As this connects with SSH to the remote machine, you will need to enter the password to use this command.
  3. Confirm that you can now login to the remote SSH server without a password: ssh user@remotehost.com .
  4. Copy the private key ( id_rsa ) to a directory called keys , which you need to create in the tomcat home directory, for example, / export/apache-tomcat-7.0.29 .

Miscellaneous

  • At present stdout and stderr are combined together and displayed in the web ui
  • If a tool is run that produces a large amount of output, it can take a significant amount of time for the output to be processed over to the web ui so expect a time lag
  • It is possible to run tools that require root permissions; however, this requires that a user is set up with root permissions or sudo permissions with no password. For example, in suders file => user ALL=(ALL) NOPASSWD: ALL