Geneos ["Geneos"]
You are currently viewing an older version of the documentation. You can find the latest documentation here.
["Geneos > Netprobe"]["User Guide"]

File Keyword Monitor (FKM)

Overview

The universal File Keyword Monitor or FKM plugin provides you a fast and simple method for monitoring multiple files. This plugin also allows you to:

  • Search specific keywords contained within the file.
  • Check if the file is updated.
  • Check if the file has remained unchanged for a given period of time.

This functionality, in combination with the Gateway rules feature, can quickly inform you of a developing problem. By monitoring a server log file for error messages, you can identify a potential risk or problem on the monitored server via the dataview format.

To view the FKM plugin configuration and settings, see File Keyword Monitor configuration.

Intended audience

This guide is intended for experienced Geneos users who want to configure the FKM plugin to monitor real-time data, and for users who have completed the recommended Geneos course training.

As a user, you should be familiar with Unix and Windows file system structure and permissions, and regular expressions or regex.

Prerequisites

The following requirements must be met prior to the configuration and setup of the FKM plugin:

Introduction

FKM also allows quick viewing of any text-based monitored files so users can see the content directly from Active Console, enabling a fast response to issues. Viewing functionality can also be restricted to specific users or user groups for sensitive files.

File Keyword Monitor has two main modes:

  • Single-trigger — shows one trigger row per key, per file. This means that if the same key appears twice or more in a file, then only the details for the latest-detected line matching that key will be shown.
  • Multiple-trigger — shows a new trigger row for each detected key. Detected keys for each file are then indexed by increasing number starting at 0. Later numbers indicate later detected keys.

Stream-based inputs such as TibRV messages can be monitored in conjunction with the appropriate plugins, as well as Windows Event Logs. The FKM plugin operates by reading multiple files that appear as a row in the published dataview. Each file can be configured with a set of fail keys and warning keys, and has a variety of scanning types.

Another feature of FKM is to dynamically ignore known problems on a single machine without reconfiguring the plugin. Ignore files provide a mechanism for configuring ignore keys for an FKM file externally to the gateway setup.

An ignore key can be used to disregard the known messages in a server log that may imply a possible error, but these messages are plain scripts or system-defined. Ignore keys should be defined before the fail or warning keys in a table, because keys in a key table are checked against a file line in order of definition

Create a sampler in GSE

Ensure that you have read and can follow the prerequisites prior to installation and setup of this plug-in.

To setup and configure the FKM plug-in, follow these steps:

  1. Open Active Console.
  2. Double-click the running Gateway to open Gateway Setup Editor.
  3. In Gateway Setup Editor, click Samplers in the Navigation panel.
  4. Click New Sampler.
  5. In the Samplers section, select FKM on the Plugin menu.
  6. Click Save current document to apply changes.

Here are the fields that you can configure on the FKM tab:

Field Description
Name

Specifies the sampler name you created in the managed entity.

Mandatory: Yes

The name of the sampler must be unique among all other samplers. The first character of your sampler name cannot be a special character.

Group

Specifies a group name.

Mandatory: No

Description

Specifies a description for your plug-in.

Mandatory: No

Sample interval

Controls the period between two samples taken by the sampler, which is measured in seconds.

The value should be set to a non-negative integer value. A value of 0 indicates that regular sampling should be disabled.

Mandatory: No

Default: 20

Unit: Seconds

Basic tab See File Keyword Monitor configuration.
Advanced tab See File Keyword Monitor configuration.
   

Monitor a source file in FKM

Now that you have created an FKM sampler, you can start configuring its basic settings to monitor a file. Creating a new sampler for each monitored log file is not required. One managed entity group can hold multiple samplers, and one sampler can monitor multiple logs files. Once completed, the newly created FKM plug-in displays in the Samplers section.

Note: The Netprobe user ID must have a read-access to read the source file.

  1. Open the Basic tab of the FKM sampler in Gateway Setup Editor.
  2. Click Add new to start a monitoring a file.
  3. Enter the path location of the file you want to monitor in the Source > filename field. For example, location/etc/log.txt.
  4. Select the file format that matches the file to be monitored on the Content type menu.
  5. Click Save current document to apply changes.
  6. Check the Active Console dataview if it displays the configuration details of the FKM plug-in.

Caution: The FKM plug-in always opens and scans a file for reading, and scans each incoming line. The FKM plug-in performs these tasks even if you choose not to define search keys. Therefore, you should not use the FKM plug-in to detect core files, as it reads core files as if they were text files. Core file detection is better served by a toolkit.

If the FKM dataview does not show any results, these are the possible errors:

  • File does not exist.
  • File is inaccessible or it has no read-access from the Netprobe. Ensure that the Netprobe has the necessary permissions to access the file and all the directories leading to the file.

When performing wildcard matches on a Unix server and there are sufficient file permissions to read and access the file, then Active Console displays OK in the status column:

On Windows, ensure that the Netprobe has the necessary permissions to both the directories and the files. If there are sufficient file permissions, then the Active Console displays OK in the status column:

For more information, see permissions in File Keyword Monitor configuration.

If there are no sufficient file permissions, then you may encounter these error messages:

  • "NOT_FOUND: Permission Denied — the Netprobe does not have permissions to traverse higher level directories. This can also occur if the wildcard pattern can match multiple directories, some of which are protected, and no matching paths leading to the file are found.
  • "NOT_FOUND: No such file or directory" — all directories are searchable, but no file matches.

Usually, the Netprobe runs as a Windows Service and this process has no access to mapped drives. Therefore, FKM can only access local files.

To view the other options that you can configure, see File Keyword Monitor configuration.

Configure FKM plugin

This sections provides other use cases and scenarios that you can configure in the FKM plug-in.

Monitor a log file for a specific error message

This functionality allows the log files to quickly identify messages that are relevant to the user. It also helps the user to interpret the output in the dataview when the log file is updated.

  1. In Gateway Setup Editor, open the FKM sampler you have created.
  2. On the Basic tab, enter the filename you want to be monitored in the Source > filename field. The value specifies the location of the file. The Netprobe working directory evaluates the relative paths.
  3. Note: If a filename contains wildcard characters, then FKM automatically checks for creation of newer files matching the wildcard pattern. When a new file is detected, FKM switches to monitor the newer file if the current file has been scanned to the end.

  4. Click Tables.
  5. Click Add new on the Tables window. When the string is found, set the Severity option depending on the business requirement.
  6. Select Fail on the Severity menu.
  7. Select data on the Key table menu.
  8. In the Tables group, click Data. This opens the Data window.
  9. Click Data > Keys. This opens the Key window.
  10. Select Match on the Set key menu. A match set key specifies the text that must appear in the log file to causing a match.
  11. Enter the specific error message you want to monitor in a file in the Set key > Search string field.
  12. Select Basic on the Rules menu. Basic matching searches the file line of the Search string using case insensitive comparison.
  13. Go back to the FKM sampler view in the Gateway Setup Editor.
  14. Click Save to apply changes.
  15. Open the Active Console dataview to view the FKM sampler you have created.
  16. If you do not have a log file that is being regularly updated, use simulation to update your monitored file. Check if it can monitor the search string keyword with the expected status:
  17. Return to FKM dataview in Active Console where the sampling is and see if it picks up the connection error message. Referring to the status column, the cell value has changed from OK to FAIL because it matches the search string connection error and the Severity ruling which was set to “Fail”.
  18. Scroll to the right of the dataview to view the triggerDetails column. This displays the line containing the matched string connection error:

Set FKM in multiple trigger mode

This setting allows you to view all lines that match the key in the dataview. To configure the FKM plug-in, follow these steps:

  1. In Gateway Setup Editor, open the FKM sampler you have created.
  2. Click the Advanced tab.
  3. Select Multiple on the Trigger mode menu.
  4. In multiple trigger mode, FKM shows a new trigger row for each detected key. Detected keys for each file are then indexed by increasing number starting at 0. Later numbers indicate later detected key.

  5. Click Save current documentto apply changes.
  6. Return to the server to update the log file.
  7. Verify that the dataview has recorded the update with a trigger row.

The advantage of setting the trigger mode to multiple is the previous record remains in the dataview, and is not replaced by a new set of data.

Display more than one trigger row per key

By default, the FKM plug-in only displays the most recently detected line that matches the key. If you want to see all the lines matching the key, you must use the multiple trigger mode.

  1. In Gateway Setup Editor, open the FKM sampler you have created.
  2. Click the Advanced tab.
  3. Select Single on the Trigger mode menu.
  4. In single trigger mode, FKM only shows one trigger row per key and per file. This means that if the same key appears twice (or more) in a file, then the dataview only displays the details for the latest detected line matching that key.

    Note: If you do not have a log file that is being regularly updated, use simulation to update your monitored file. Create a file that you can concatenate into the log file to be monitored.

  5. Click Save current document to apply the changes.
  6. Cat the file to the end of the log file.
  7. Return to the dataview to check that the addition to the log file has been detected and registered.
  8. If no, return to the server and use the cat command to correct the error in the file.
  9. Check that the existing trigger row has been replaced by the latest update to the log file.

The lastModificationTime values have changed and the previous row has been deleted.

Clear a trigger row

This allows you to clear a trigger row. This is useful if you have set a key match with a string such as “Connection Error”, but you know that the problem can be quickly fixed. To clear the trigger, you can set a key to match a second string, for example "Reconnected". Upon matching the second string, this clears the trigger row.

When FKM detects the first string connection error, it flags this in the dataview by creating a trigger row. When it detects the second string reconnected, it automatically clears the trigger row.

  1. In Gateway Setup Editor, open the FKM sampler you have created.
  2. On the Basic tab, click Tables.
  3. Click Table > Data. This opens the Data window.
  4. Click Data > Key. This opens the Key window.
  5. In the Clear key group, type “Reconnected” in the Search string field. When FKM spots “Reconnected” in the log file, this automatically clears the key from the dataview.
  6. Select Basic on the Clear key > Rules menu. There is no regular expression or regex needed on this setup.
  7. Close the screens until you return on the FKM Basic tab.
  8. Click Save current document to apply changes.
  9. Return to the dataview and verify that the trigger has been cleared.
  10. Wait until the dataview reaches the sampling time, then refresh the data. Once completed, this automatically clears a trigger when a problem has been fixed

Display a customised message in the dataview

This allows you to display a relevant message in the triggerDetails column of the FKM dataview. This is used to specify any error messages encountered.

  1. In Gateway Setup Editor, open the FKM sampler you have created
  2. On the Basic tab, click Tables.
  3. Click Table > Data. This opens the Data window.
  4. Click Data > Key. This opens the Key window. The Key button brings up the previous configuration details you have setup for Search string and Rules fields.
  5. In the Message field, enter the specific message you want to display in the dataview. Ensure the display message is clear and can be easily understood by all users.
  6. Close the screens until you return on the FKM Basic tab.
  7. Click Save current document to apply changes.
  8. If you have a running setup, check that the message has been displayed in the status column of the file row and in the triggerDetails column of the trigger row.

The Status field shows the specified error message:

The triggerDetails field shows the specified error message.

Extract a fragment of a line in a log file

This allows you to extract part of a line in a log file. This can be useful when many similar messages are sent to a log file, but most of them are of no interest. The Geneos Extractor feature allows you to search a specific range of characters, for example all numbers between 1020 and 3780, excluding all the numbers before and after the range:

Note: The prerequisite for this configuration setup is to set the trigger mode to multiple.

Set up an extractor

Note: You can create a dummy log file and files that you can use to simulate an application writing to a log file.

  1. In Gateway Setup Editor, open the FKM sampler you have created
  2. On the Basic tab, click Tables.
  3. Click Table > Data. This opens the Data window.
  4. Click Data > Key. This opens the Key window.
  5. Enter the search string value in the Search string field. The sample regular expression in the screenshot will look for digits from 0-9:
  6. Select Regexp on the Rules menu.
  7. Click Add new in the Extractors section.
  8. Enter the name of the extractor in the Name field.
  9. Enter the regular expression in the Regex field.
  10. Close the screen until you return on the FKM Basic tab.
  11. Click Save current document to apply changes.
  12. Check the FKM dataview in the Active Console to verify that the Extractor column has been created.
  13. The last column shows the Error Code that you have created from the Extractors section

Create a rule

  1. In Gateway Setup Editor, click the Rules section.
  2. Click Create a new item > New Rule.
  3. Enter the name of the rule in the Name field.
  4. Enter the rule expression in the Block field. This is where the rule codes are created. It is evaluated each time any relevant data changes.
  5. Right-click the Block field to view the most common keywords and functions.
  6. In this example, the set rule is created to test the value against the two numbers (2859 and 1182). When the program finds one of this numbers, it runs the clearTrigger action. This action automatically removes the identified values from the dataview.

  7. Once the rule is set, identify the target path.
  8. Return to the FKM dataview in Active Console.
  9. Right-click the cell of the column where the rule is to be implemented.
  10. Click Copy > Path. An example of path directory:
  11. /geneos/gateway[(@name="GATEWAY_68944")]/directory/probe[(@name="New Probe")]/managedEntity[(@name="Exercises")]/sampler[(@name="fkmtest")][(@type="")]/dataview[(@name="fkmtest")]/rows/row[(@name="/sbox/home/useralias/etc/fkm_log.txt#fail00000")]/cell[(@column="lastModificationTime")]
  12. Return to the Rule section in GSE.
  13. Paste the path of the cell in the Targets field.
  14. Click Edit to open the path editor.
  15. Click the Evaluate Path button to validate the correctness of the identified path.
  16. Note: If at least one of the elements have turned into red field, there is no match found in the validated path. Ensure the source path is accurately correct.

  17. Click Save current document to apply changes.

Execute an action

  1. In Gateway Setup Editor, click the Actions section.
  2. Click Create a new item > New Action.
  3. Enter the name of the action in the Name field to create an internal command.
  4. Select Internal command on the Options menu.
  5. In the Internal command group, select FKM:clearTrigger on the Name menu.
  6. Click Save current documentto apply changes.

At this stage, the search string, key dialog, and extractor are looking for specific numbers, and a rule to check those numbers has been configured. If it matches one of the identified numbers, then it triggers the action.

For example, if the number does not match the value set in the rule, then it displays in the Error Code row. The value 3784 appears in the Error Code row:

If the number matches any of the value set in the rule, it displays in the FKM dataview row. However, due to the action made which is clearTrigger, this automatically disappears from the dataview rowafter a few seconds.

Use wildcard when you enter filenames

This allows you to use wildcard in filenames configuration. This can be useful in a situation where you have several filenames with the same extension, for example .log, and you do not want to enter each filename individually into the sampler.

  1. In Gateway Setup Editor, open the FKM sampler you have created.
  2. On the Basic tab, scroll to the right to click Add new. The Add new button adds a new Source field in the Files section.
  3. Enter the full path name in the newly added Source filename. Use the wildcard by adding an asterisk (*) before the log extension name in the filename.
  4. Click Tables > Add new > Data.
  5. Click Data > Key. The Key button brings up the previous configuration details you have setup for Search string and Rules fields.
  6. Enter the search string value in the Search string field.
  7. Select Basic on the Rules menu.
  8. Close the screens until you return on the FKM Basic tab.
  9. Open the Advanced tab.
  10. On the Advanced tab, scroll down until you see the fields related to wildcard.
  11. Select the Wildcard monitor all matches check box.
    • If this check box is not activated, the FKM dataview only displays the latest file to be changed.
    • If this is selected, it displays all the files matching the asterisk (*) .config or the log extension name you used.

  12. Click Save current documentto apply changes.
  13. Check the FKM dataview in Active Console to view all the files that match the wildcard.

View file in FKM dataview

The View File command option allows you to remotely view the file being monitored by FKM.

There are two file viewing modes:

  • Snapshot view — this only displays the selected parts of the file at the time when the query is made
  • Continuous view — displays the same as the snapshot, but is continue to update the file view window when additional lines are added to the file.
  1. In Active Console, open the FKM sampler.
  2. Right-click the name of the file you want to view.
  3. Click View File. The View File is an example of a command that are specific to some plugins or components. Not all cells in the dataview has this functionality.
  4. The View File window opens.

The View File dialog allows users to specify how much of the file to view is relative to the end of the file. The maximum size of data from the file that can be displayed is 1 megabyte (1024 kilobytes). This is to prevent performance slowdown caused by the handling of large files that are sent over to the view.

Accept files in FKM dataview

The Accept commands let you acknowledge a file and clear its trigger.

  1. In the Active Console State Tree, click to expand the relevant Managed entity.
  2. Under the Managed entity, click to expand the relevant probe.
  3. Double-click the relevant FKM sampler. The dataview for the sampler appears.
  4. In the FKM dataview, right-click the file with the trigger and click the accept option you wish to use. For guidance, see the options below.
  5. If you are accepting multiple files, select your desired Accept and Location options in the Accept multiple files window, then click OK.
  6. Success: The trigger is cleared and the file status of the accepted files return to normal.

Accept file commands on the FKM plugin

  • Accept this File — accepts triggers of the selected file.
  • Accept files... — opens the Accept multiple files window.

When you select Accept files..., you have the following options:

FKM Accept multiple files window

Accept options:

  • this file — accepts triggers of the selected file in the locations specified.
  • all files — accepts triggers of all files listed in the dataview in the locations specified.

Location options:

  • this managedEntity (host) — accepts triggers for files under the Managed entity.
  • this sampler — accepts triggers for files under the sampler.
  • this probe — accepts triggers for files for all FKM samplers under the Netprobe.
  • this gateway (all hosts) — accepts triggers for files for all FKM samplers under the Gateway.

Change FKM dataview columns

The column setting allows you to configure the list of columns to be displayed by FKM in the dataview. You may want to add or remove any existing columns, depending on the needed information to be displayed in the dataview.

  1. In Gateway Setup Editor, open the FKM sampler you have created.
  2. Open the Advanced tab.
  3. Click Columns > Add new.
  4. On the Column menu, select the column name you want to be displayed in the dataview.
  5. Click Save current document to apply changes.

Limit the numbers of displays triggers

Use the maxConditionPerKey parameter to control the trigger rows or conditions that are recorded per file. This is used primarily in multiple trigger mode and single grouped message mode to limit the number of triggers to be created.

In this use case, set the trigger mode to multiple and use the maxConditionPerKey to limit the number of rows which by default displays more than 1,000 rows or lines of triggers.

  1. In Gateway Setup Editor, open the FKM sampler you have created.
  2. Open the Basic tab and click Tables.
  3. In the Table section, complete the necessary fields:
    • Select Fail on the Severity menu.
    • Select data on the Key table
  4. Click Tables > Data > Key. This opens the Key window.
  5. Enter the keyword in the Search string field. In this example, the search string is "test".
  6. Go back to the FKM in GSE.
  7. Open the Advanced tab.
  8. Select Multiple on the Trigger mode menu.
  9. Scroll down until you see the maxConditionPerKey parameter.
  10. Enter the limit number to be displayed in the Max conditions per key field.
  11. Note: If a new trigger arrives after this limit has been reached, the older trigger for the key will be removed and replaced with the new trigger.

  12. Click Save current document to apply changes.

Use date generation function when using dynamic files option

The dynamicFiles file source type configures FKM to match groups of files based on the configured path, pattern, and optionally the alias.

To use the date generation function when the dynamicFiles option is selected, follow these steps:

  1. In the Gateway Setup Editor, open the FKM sampler you have created.
  2. Go to Basic tab > Files > Source, and select dynamicFiles.
  3. Click Dynamic files. This opens the Dynamic files window.
  4. In the Dynamic files > Path field, enter the path that you want to monitor. For example, /opt/geneos/gateway/<today %Y>/<today %m>/<today %d>/*.txt.
  5. In the Pattern > Data > Regex field, enter the expression based on the files you want to match. In this example, enter ^.*.txt.
  6. Click Close.
  7. On the Advanced tab, tick the Use PCRE Perl Compatible Regex box.
  8. Click Save current document to apply changes.

For more information, see files > file > source > dynamicFiles in File Keyword Monitor configuration.

Create a rule to ensure the last modification time is from today

You can create a rule to compare the last modification time with the current time. With this rule, you can check if the lastModificationTime is from today.

To create a rule:

  1. In the Gateway Setup Editor, click Rules > New Rule.
  2. Create a rule in the Block field. For example:
  3. set $(timestamp) parseDate("%a %b %d %H:%M:%S %Y", value)
    if $(timestamp) < startOfDay() then
    severity warning
    else
    severity ok
    endif
    
  4. Click Save current document to apply changes.

If the last time stamp is recorded before midnight of the previous day, then the system displays a warning. Otherwise, the severity level displayed is OK.

Further reading

The File Keyword Monitor plugin can be configured to run on any Netprobe host. Configuration should consist of at least one file to be monitored, which will provide basic statistics about the file.

To know more about FKM settings, see File Keyword Monitor configuration.