--- title: "Toolkit" description: "" url: "https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/toolkit-plugin/index.html" --- ## Overview This guide discusses the installation, set-up, and configuration of the Toolkit sampler. It is intended to provide different sample scripts of user scenarios and basic information to use this plug-in. The Toolkit plug-in integrates output from custom commands and scripts into Geneos. This plug-in executes these commands or scripts on a periodic basis and converts their output into a dataview. These custom commands can execute standard output application written in C/C++, Perl, TCL, Python, and scripts, such as VB, batch, and shell script. ### Prerequisites The following requirements must be met prior to the configuration and setup of the Toolkit plug-in: - Netprobe must have the appropriate access to execute the commands and scripts. - Results of the commands and scripts must be to standard output (stdout). - Results must be in the custom comma separated values (CSV) format. ### User requirements To use this plug-in, you must meet the following requirements: - Basic understanding of programming languages, such as C/C++, Perl, TCL, and shell script that can execute the custom commands. - Use of batch scripting or PowerShell scripting in Windows, and other third-party scripting languages. ## Introduction A typical dataview constructed using the Toolkit plug-in contains several headline variables followed by a detailed table. The Toolkit plug-in supports several output formatting conventions that control how script output is interpreted, including headline rows, column headers, and multi-dataview output sections. For more information, see [Toolkit output markup tags](#toolkit-output-markup-tags). By default, the Toolkit plug-in uses commas (`,`) as cell delimiters. Other commonly used delimiters include semicolons (`;`), quotation marks (`"` or `'`), and slashes (`/`). The following example shows sample Toolkit script output: ``` $ ./toolkit_test03.bash id,name,status,Value AlertDetails,this is red alert 001,agila,up,97 002,lawin,down,85 ``` In this example, `./toolkit_test03.bash` is configured in the **Sampler script** field: ![Sampler script](https://docscdn.itrsgroup.com/geneos/images/plug-ins/toolkit-ug-sampler-script.png) Given the sample script used in the example, the generic format to display in the dataview is: ``` column1,column2,column3,column4 headlineName,headlineValue row1,row1col2,row1col3,row1col4 row2,row2col2,row2col3,row2col4 ``` The dataview of this sampler configuration displays in Active Console: ![Sampler configuration](https://docscdn.itrsgroup.com/geneos/images/plug-ins/toolkit-ug-shell-dataview.png) **Warning** If the script fails by either returning no output or failing to run within the timeout period for five successive samples, then the Netprobe disables sampling for that instance of the toolkit. To re-enable the sampling, you can force a manual sample (using the Sample Now button), restart the plug-in through reconfiguration, or restart the Netprobe. ### Toolkit output markup tags Toolkit scripts can use markup tags to control how output is published to dataviews. Use `` for headline values, `` for column headers, and `` for multiple dataviews. #### \ headline values Use the `` tag to publish headline values in a Toolkit dataview. A line that begins with `` is interpreted as a headline row instead of table data. Headline values appear above the dataview table. When Toolkit processes the headline row, it removes the leading `` tag and displays only the headline name and value in the dataview. The sample dataview shown earlier includes two types of headline cells: | Variable | Description | | --- | --- | | execDuration | A headline variable that shows the script execution time. Toolkit displays it by default, but you can set `showExecutionDuration` to `false` to hide it. | | `` | Specified by either the script output (using the `` tag) or by a compute engine cell. | #### \ column headers You can optionally use the `` tag to mark a line explicitly as the column header row. A line that begins with `` is the column header row for the current dataview section. The `` tag is case-insensitive. When Toolkit processes the line, it removes the leading `` tag and displays only the column names in the dataview. For example: ``` Process,CPU,Memory nginx,12,256 java,34,1024 ``` This example produces a dataview with column headers `Process`, `CPU`, and `Memory`. If a section has no `` line, Toolkit uses the first line in that section as the column header row. If the first line starts with ``, Active Console shows the `` tag in the first column header instead of publishing a headline value: ![Active Console dataview with a -prefixed column header](https://docscdn.itrsgroup.com/geneos/images/plug-ins/toolkit-ug-output-markup-tag-include.png) If you use more than one `` line in a dataview section, Active Console returns an error: ![Multiple tags AC error](https://docscdn.itrsgroup.com/geneos/images/plug-ins/toolkit-ug-output-markup-error.png) #### \ dataview sections Use the `` tag to publish multiple dataviews from a single Toolkit sampler script. A line that begins with `` starts a new dataview section. All output after that line belongs to that dataview until the script encounters another `` line or the script ends. The text after `` on the same line becomes the dataview name. Toolkit ignores leading and trailing spaces. If you omit the dataview name, Toolkit uses the default dataview name. Each dataview section uses an `` line or the first line in that section as the column header row. Output that appears before the first `` line is published to the default dataview. For example: ``` Process,CPU,Memory nginx,12,256 java,34,1024 Disk usage Filesystem,Used,Available /dev/sda1,40G,60G /dev/sdb1,120G,80G ``` This example publishes: - The default dataview, which contains process metrics and uses an `` line for column headers. - A second dataview named `Disk usage`, which uses the first line after ` Disk usage` as column headers. **Note** - If a dataview section has no rows, Toolkit does not publish the dataview. Empty dataview sections do not consume a dataview license allocation. - If `rowLimit` is configured, the limit applies independently to each dataview section. - Do not use the same dataview name more than once in a single script execution. If Toolkit detects duplicate dataview names, it publishes the affected dataview as an error dataview. - If a later script execution no longer includes a previously published dataview, the Gateway removes that dataview. Rules, dashboards, and history views that reference the removed dataview might stop working as expected. ### Commas within view cells Since the comma is the cell delimiter, it is not possible for the toolkit's scripts to have these within the cell’s content. To include an actual comma within the cell’s content, precede the comma with a backslash `\`. Sample command script: ``` queue3, 7\,331, 45\,000,0.16, online ``` Expected output: ``` queue3 | 7,331 | 45,000 | 0.16 | online ``` ### Environment variables In addition to [Geneos variables](https://docs.itrsgroup.com/docs/geneos/7.11.0/processing/monitoring-and-alerts/geneos_rulesactionsalerts_tr/index.html), user variables can be set in the Environment Variables and Secure Environment sections for the plug-in to utilise. 1. Go to **Gateway Setup Editor > Sampler > Toolkit** plug-in. 2. Click the **Advanced** tab. ![sampler configuration](https://docscdn.itrsgroup.com/geneos/images/plug-ins/toolkit-ug-set-environment-variables.png) Secure Environment Variables are encrypted using the AES 256 encryption. This is useful for storing passwords or any sensitive data. To learn how to use these variables, see the [Appendix](#appendices) section. ### Display CSV file in the dataview Toolkit plug-in allows you to display the data in the Active Console dataview by using a command to display the file to a standard output. To display the content of a `.csv` file: 1. In the Gateway Setup Editor, open the Toolkit sampler. 2. Enter the command script in the **Sampler script** following the directory path of the file. For example: ``` /bin/cat /home/documents/toolkit.csv ``` 3. Click **Save current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/save16x16.png "icon") to apply changes. On Unix, `/bin/cat` is the common program, while on Windows, use `cmd.exe` command interpreter to execute the `type` built-in. For example: ``` C:\Windows\system32\cmd.exe /c type C:\Users\Administrator\Documents\toolkit.csv ``` ## Install and set up Ensure that you have read and can follow the system requirements prior to installation and set-up of this plug-in. If you are using this plugin with Gateway Hub, you must create a user defined data schema. For instructions, see [Create a data schema](https://docs.itrsgroup.com/docs/geneos/7.11.0/processing/gateway-hub/data-schema-ug/index.html#create-a-data-schema). This provides instructions on how to set up a basic configuration using a Toolkit plug-in: 1. To select the Toolkit plug-in, go to **Gateway > Probes > Managed entity > Samplers**. 2. In the Samplers section, select Toolkit on the **Plugin** menu. 3. In the **Sampler script** field, enter the command to execute the script. ![Sampler script](https://docscdn.itrsgroup.com/geneos/images/plug-ins/toolkit-ug-toolkit-test-01-sampler.png) **Note** The Sampler script specifies the name of the command that the plug-in runs on a periodic basis. If there are any spaces in your path, you must enclose it in double quotation marks. Otherwise, the sampler fails to execute the script. In this example, the command and script is entered in the **Sampler script** field: ``` ./toolkit_test.bash ``` 4. Click **Save current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/save16x16.png "icon") to save your changes. 5. Go to the Toolkit dataview in the Active Console to check if your configuration to monitor the path is working. ![Toolkit dataview](https://docscdn.itrsgroup.com/geneos/images/plug-ins/toolkit-ug-toolkit-test-01-dataview.png) These Toolkit sampler configurations below show the different user scenarios that you can use to execute commands. ### Use command argument To execute command using Command argument, follow these instructions: 1. In the **Sampler script** field, enter the command to execute the script. In this example, the command and script entered in the Sampler script field is: ``` /bin/cat toolkit_command_args.txt ``` 2. Click **Save current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/save16x16.png "icon") to save your changes. 3. Go to the Toolkit dataview in Active Console to check if your configuration to monitor the path is working. ![Toolkit dataview](https://docscdn.itrsgroup.com/geneos/images/plug-ins/toolkit-ug-command-args-dataview.png) ### Use PowerShell To execute command using Powershell script, follow these instructions: 1. In the **Sampler script** field, enter the command to execute the script. In this example, the command and script entered in the Sampler script field is: ``` C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -noprofile -executionpolicy bypass -file "C:\Users\Administrator\Downloads\Test Scripts\output-select-lines.ps1" ``` 2. Click **Save current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/save16x16.png "icon") to save your changes. 3. Go to the Toolkit dataview in Active Console to check if your configuration to monitor the path is working. ![Toolkit Powershell dataview](https://docscdn.itrsgroup.com/geneos/images/plug-ins/toolkit-ug-powershell-script-dataview.png) ### Use Windows BAT files To execute command using Windows bat script, follow these instruction: 1. In the **Sampler script** field, enter the command to execute the script. In this example, the command and script entered in the Sampler script field is: ``` "C:\Users\Administrator\Downloads\Test Scripts\toolkit_pebl_col273.bat" ``` 2. Click **Save current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/save16x16.png "icon") to save your changes. 3. Go to the Toolkit dataview in Active Console to check if your configuration to monitor the path is working. ![Windows BAT files](https://docscdn.itrsgroup.com/geneos/images/plug-ins/toolkit-ug-windows-bat-script.png) ## Plugin configuration **Note** If you are using this plugin with Gateway Hub, you must create a user defined data schema. For instructions, see [Create a data schema](https://docs.itrsgroup.com/docs/geneos/7.11.0/processing/gateway-hub/data-schema-ug/index.html#create-a-data-schema). The following parameters can be configured for this plug-in: ### samplerScript Name of the command that the plug-in runs on a periodic basis. If the command contains spaces, then the entire command (but not any arguments) must be wrapped in double quotes, or the sampler will fail to execute the script. For example, rather than specifying: ``` c:\monitoring\example path\example path\tkscript.bat a b c d ``` The command should be specified correctly as: ``` "c:\monitoring\example path\example path\tkscript.bat" a b c d ``` Absolute and relative paths to the command must be specified as well. **Note** The `PATH` environment variable is not used to locate the script or executable. Mandatory: Yes ### numberOfDataviews Controls how many dataview license allocations the Toolkit requests. Each allocation corresponds to one distinct Toolkit dataview that the sampler can publish. **Note** Changing the `numberOfDataviews` value changes how many license allocations the Gateway associates with the sampler. Set this value to match the number of Toolkit dataviews that you expect the script to produce. Default: `1`. If the value is missing, invalid, or not a positive integer, the Toolkit uses `1`. ### rowLimit Limits the number of rows displayed. The default rowLimit is `200`, but can be set higher or lower in the configuration. The rowLimit prevents the Netprobe from becoming overloaded if the script outputs a large amount of data. If the rowLimit is reached, the Netprobe will still read the data from the script but will not store it. This ensures any script relying on read requests will complete. **Note** - This is a limit on the number of lines read in from the script. The limit would include lines read in that became Headline Variables and Column Headers, as well as lines that became rows in the dataview. - You can override this setting with a Gateway-wide row limit that you can [configure in the Gateway Setup Editor](https://docs.itrsgroup.com/docs/geneos/7.11.0/processing/configuration/gateway_operating_environment/index.html#operatingenvironment--maximumnumberofdataviewrows) under **Operating environment** > **Maximum number of dataview rows**. ### scriptTimeout Changes the timeout value for a script. A script is allowed 60 seconds to execute. A script running longer than the timeout value will not display the desired results in the dataview. Use this setting to change the script timeout value for longer running scripts. Default: 60 (in seconds) ### terminateOnTimeOut Changes the behavior of the plug-in when a timeout occurs due to the script running longer than 60 seconds. The normal behavior is for the plug-in to kill the long running process. Setting this flag to false will change the behavior, and the plug-in will leave the process running. This behavior change can also be controlled through the use of an environment variable. Setting the environment **TOOLKIT\_TERMINATE\_ON\_TIME\_OUT** to the value of **FALSE** will have the same effect as setting the terminateOnTimeOut setting to false. The environment setting is overridden by the GSE setting. Mandatory: No Default: true (unless environment variable **TOOLKIT\_TERMINATE\_ON\_TIME\_OUT** is set to **FALSE**, then false) ### environmentVariables Defines an optional set of environment variables that will be available to the script. Mandatory: No ### environmentVariables > variable Defines a single environment variable that will be available to the script. Mandatory: No ### environmentVariables > variable > name Name of the variable being defined. Mandatory: Yes ### environmentVariables > variable > value Value of the variable being defined. Mandatory: Yes ### secureEnvironmentVariables Defines an optional set of secure environment variables that are AES 256 encrypted and will be available to the script. Environment variables in this section will override any other environment variables of the same name defined elsewhere. Mandatory: No ### secureEnvironmentVariables > variable Defines a single secure environment variable that will be available to the script. Mandatory: No ### secureEnvironmentVariables > variable > name Name of the variable being defined. Mandatory: Yes ### secureEnvironmentVariables > variable > value Value of the variable being defined. This will be AES 256 encrypted or reference a variable that is AES 256 encrypted. References to other variable types are invalid and will result in an error and the variable being ignored. Mandatory: Yes ### script Defines a script that will be sent to the Netprobe and written out as a file for execution. Mandatory: No ### script > filename Name of the script that the Netprobe will write to. This must be in or under the Netprobe's working directory. Absolute paths are not permitted in that paths must start with (./) or state the filename itself, denoting the Netprobe's relative path. The subdirectories and filename in the path will then be created if they do not exist. For example, it can be any of the following: - `.//` - `/` - `./filename` - `filename` Mandatory: Yes ### script > contents Defines the contents of the script. Mandatory: Yes ### showExecutionDuration ExecDuration Headline shows the toolkit script execution time during last sample. Mandatory: No Default: true
Setting Description
true ExecDuration Headline should be displayed
false ExecDuration Headline should not be displayed
## Appendices ### Use the secured environment variables This procedure has two steps: 1. Generate the key file and encrypted value. 2. Decrypt the encrypted value before the Toolkit executes the script. #### Generate the encrypted password **Note** This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (). 1. Generate and use a key file as follows: ``` openssl enc -aes-256-cbc -k "Now is thewinter of our discontent" -P -md sha1 >key-file ``` The key file is required for the encryption and decryption of AES 256 passwords. 2. Generate the encrypted value. ``` ./active/gateway2.linux -aes256-encrypt 12345 -key-file key-file -skip-cache ``` The `12345` isa user-specified password which is then encrypted. The Gateway produces the output in hexadecimal: ``` Encoded text: +encs+69B1E12815FA83702F0016B0E7FBD33B ``` A Gateway binary, such as `gateway2.linux_64` is needed to create the AES 256 password. The `-skip-cache` is needed only the first time you wish to change the key file. 3. In the Toolkit sampler, create a secured environment variable. 4. Add the encrypted password to the secured environment variable. You can convert the sampler view to XML. #### Decrypt the encrypted value 1. Use the sample shell script below that decodes the hexadecimal or encrypted value: ``` ./decode.sh +encs+69B1E12815FA83702F0016B0E7FBD33B ``` **decode.sh**: ``` #!/bin/bash hex=${1#*+encs+} salt=$(grep salt key-file) key=$(grep key key-file) iv=$(grep iv key-file) echo ${hex}| xxd -r -p | base64 | openssl enc -d -aes-256-cbc -S ${salt#*=} -K ${key#*=} -iv ${iv#*=} -a ``` 2. To convert an environment variable, replace the '1' in the second line with the name of the environment variable. All encrypted variables start with `+encs+`, so this removes the prefix prior to decoding. **Note** xxd is part of the vim-common package which you may have to install separately. ### Sample scripts in Windows **wgetenv3.exe** Running *wgetenv3.exe reg.txt*, for example, where the file reg.txt contains: ``` [H] logPixelsHEX hkey_current_config\software\fonts\logpixels [HEX] [H] logPixelsDEC hkey_current_config\software\fonts\logpixels [DEC] [L] sysID HKEY_LOCAL_MACHINE\HARDWARE\DESCRIPTION\System\identifier [STR] [L] sysBIOSDate HKEY_LOCAL_MACHINE\HARDWARE\DESCRIPTION\System\SystemBiosDate [STR] [L] videoBIOSDate HKEY_LOCAL_MACHINE\HARDWARE\DESCRIPTION\System\VideoBiosDate [STR] [L] videoBIOSVersion HKEY_LOCAL_MACHINE\HARDWARE\DESCRIPTION\System\VideoBiosVersion [STR] ``` generates the following input to a toolkit sampler: ``` name,value logPixelsHEX,60 logPixelsDEC,96 sysID,AT/AT COMPATIBLE sysBIOSDate,06/23/08 videoBIOSDate,03/26/09 videoBIOSVersion,HWEAASUS EN9400GT VBIOS Ver 62.94.71.00.AS01 HWEAASUS EN9400GT VBIOS Ver 62.94.71.00.AS01 HWEAASUS EN9400GT VBIOS Ver 62.94.71.00.AS01 Version 62.94.71.00.13 Version 62.94.71.00.13 Version 62.94.71.00.13 Version 62.94.71.00.13 ``` W3C XML 1.0 (Fourth Edition) Specification -