Geneos
Geneos
Recommended Reads

Publish to Kafka and Nanomsg

Publishing

Introduction

The Gateway can publish data to external systems via a dynamically loaded adapter.

The 64-bit Gateway distributions include adapters for publishing via Nanomsg or Apache Kafka. These are provided as shared objects in the lib64 directory (lib64/geneos-adapter-*.so).

If you need to run the Gateway from a directory which is not the parent of lib64, please ensure that the adapter you want to use can be located either via the LD_LIBRARY_PATH environment variable or via the adapter.library.path setting. (See publishing > additionalSettings below.)

Gateway can use the Nanomsg adapter to provide a direct (peer-to-peer) link with a third party application. In a Hot Standby configuration, the application needs to connect to both the primary and secondary gateways; the protocol used allows it to detect which is active.

The Gateway can use the Kafka adapter to publish data to a Kafka cluster (minimum version 0.9), from which it can be read by any system which implements a Kafka subscriber. In a Hot Standby configuration, only the active Gateway publishes to the Kafka cluster; data consumers are isolated from any failover. The Kafka adapter supports SSL encryption and authentication; this needs to be configured both on the Kafka cluster and in the Gateway setup file. Please see publishing > additionalSettings for more details.

When an adapter is used (a gateway can be configured to use either Nanomsg or Kafka, but not both at once), the Gateway streams the following fixed set of data:

  • directory information — data about each probe, managed entity, and dataview known to the Gateway.
  • metrics — all dataview cells and headlines.
  • metadata — severity, snooze, and user assignment information.

This data is published using JSON-based formats which are described below.

Note: The publishing of Gateway data is only supported on 64-bit Linux and 64-bit Solaris x86.

Configuration

publishing

This section defines the parameters controlling publishing to external systems.

publishing > enabled

This setting allows publishing to be enabled/disabled. By default publishing is turned off, however if a publishing section exists then it is turned on. This setting allows that to be overridden, so publishing is disabled but there is configuration saved for a later date.

Mandatory: No
Default: true

publishing > adapter

The adapter section allows the adapter to be selected and holds settings specific to the selected adapter.

publishing > adapter > nanomsg

Settings for the Nanomsg adapter.

publishing > adapter > nanomsg > publishPort

This defines the port on which the Nanomsg adapter listens for connections from data subscribers.

Mandatory: No
Default: 7070

publishing > adapter > nanomsg > requestPort

This defines the port on which the Nanomsg adapter listens for requests for status information, snapshots and retransmission of directory information. If this is not specified then it will listen on the next port after the publishing > adapter > nanomsg > publishPort.

Mandatory: No
Default: (publishing > adapter > nanomsg > publishPort) + 1

publishing > adapter > nanomsg > secondaryPorts

This section is used if the secondary gateway needs to use different publishing ports to the primary, for example if both gateways are running on the same host in a testing environment. If this section is not specified then the secondary gateway will use the values specified in publishing > adapter > nanomsg > publishPort and publishing > adapter > nanomsg > requestPort.

publishing > adapter > nanomsg > secondaryPorts > publishPort

This allows the value set in publishing > adapter > nanomsg > publishPort to be overridden on the secondary gateway.

Mandatory: No
Default: publishing > adapter > nanomsg > publishPort

publishing > adapter > nanomsg > secondaryPorts > requestPort

This allows the value set in publishing > adapter > nanomsg > requestPort to be overridden on the secondary gateway.

Mandatory: No
Default: publishing > adapter > nanomsg > requestPort

publishing > adapter > kafka

Settings for the Kafka adapter.

publishing > adapter > kafka > topicPrefix

Specifies the first part of the topic names under which data is published and the first part of the topic name to which the Kafka adapter subscribes for requests, such as metrics snapshots. By default, the publishing topics used include geneos-probes, geneos-raw.table, (and eight other names beginning geneos-) and the request topic is geneos-requests. This setting allows the topics to be changed to, for example, my-test-probes, my-test-raw.table and so on.

Mandatory: No
Default: geneos-

publishing > adapter > kafka > brokerList

Specifies a comma-separated list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The adapter makes use of all servers irrespective of which servers are specified here for bootstrapping - this list only impacts the initial hosts used to discover the full set of brokers. Please refer to the description of bootstrap.servers in the Kafka documentation for more information.

Mandatory: No
Default: localhost:9092

publishing > additionalSettings

This advanced configuration option allows additional settings to be specified. These settings either control the way the adapter is loaded or are interpreted by the adapter itself.

publishing_additional_settings

Settings are entered using the same syntax as a Java properties file, each line specifies a key-value pair.

Settings with the prefix adapter. control the way the adapter is loaded. The settings available are:

  • adapter.library.path — allows you to specify the location of the shared library that implements the adapter. By default, this is loaded from the lib64 sub-directory of the Gateway's working directory; you can specify an alternate location, either as a path relative to the Gateways's working directory or as an absolute path (as in the example above.)
  • adapter.gateway.name — used with the topic prefix to construct a unique Kafka consumer group id for the Gateway. The default group id is <topicprefix><gatewayname>, e.g. geneos-DevGateway. You can override the Gateway name part with this setting, but there is generally no need to do so.

The Kafka adapter uses the librdkafka library to connect to Kafka and supports transparent pass-through of settings to the library. Any global setting (other than callbacks) defined in the librdkafka documentation can be used by prefixing their names with kafka..

In the example above, the key kafka.client.id refers to the librdkafka option documented as client.id. Additionally, default topic settings can be specified by prefixing their names with kafka.topic:. (Note the use of a colon, ':', here, since some global settings begin with the string topic..)

Note: The Kafka adapter discards incoming messages when the librdkafka publishing queue is full. When publishing is resumed, the adapter reports the number of discarded messages. The length of the queue is modified by including a kafka.queue.buffering.max.message setting here.

SSL encryption

To use SSL encryption for the connection, the SSL protocol must be enabled and the certificate used to sign the Kafka brokers' public keys must be trusted by the Kafka adapter. Use the following template, replacing the location with your credentials:

kafka.security.protocol=ssl
kafka.ssl.ca.location=<location of CA certificate PEM file>
SSL encryption with SSL authentication

To use SSL encryption with SSL authentication, the Kafka adapter must be able to present a certificate which is signed using a certificate trusted by the broker(s). Use the following template, replacing the locations with your credentials:

kafka.security.protocol=ssl
kafka.ssl.ca.location=<location of CA certificate PEM file>
kafka.ssl.certificate.location=<location of client certificate>
kafka.ssl.key.location=<location of private key for client certificate>

There are no additional settings for the Nanomsg adapter.

Mandatory: No

publishing > strategies

Publishing strategies allow you to configure greater control over the publishing of Geneos dataviews. The supported strategies are:

  • Pivoting Tables
    Some dataviews are published by default as a set of name-value pairs; 'pivoting' exchanges rows and columns to provide a more useful and SQL like row format. The templates directory of the Gateway distribution contains a gateway include file that shows how to pivot some of the more relevant Geneos dataviews.

    • publishing_hw_pivot

    A dataview that should be published in pivoted form, so that the left-hand column becomes the table header.

  • Scheduled publishing of dataview
    You may also schedule publishing of dataviews that are updated infrequently to provide data points at fixed points in time.

publishing > strategies > strategyGroup

Strategies may be grouped for organisational purposes. This has no effect on their application but is useful for grouping categories and making them easier to find in setups.

publishing > strategies > strategyGroup > name

Name of the strategy group.

Mandatory: Yes

publishing > strategies > strategy

A strategy is defined as follows:

publishing_strategy

publishing > strategies > strategy > name

Name to uniquely identify the strategy.

Mandatory: Yes

See Publishing strategy definition.

publishing > strategies > strategy > targets

Set of dataviews that the strategy applies.

Mandatory: No

See Publishing strategy definition.

publishing > strategies > strategy > targets > target

An XPath that identifies a dataview or dataviews that the strategy applies to. The target must be a dataview and this is checked when setup is validated.

Mandatory: No

See Publishing strategy definition.

publishing > strategies > strategy > pivotDataview

When selected, this pivots the target dataviews such that the rownames of the original dataview become column names and the adjacent columns become rows.

Mandatory: No

See Publishing strategy definition.

publishing > strategies > strategy > schedule

Scheduling options for matching dataviews.

Mandatory: No

See Publishing strategy definition.

publishing > publishPort

This setting has been deprecated. Please use the corresponding setting under publishing > adapter > nanomsg.

Mandatory: No
Default: 7070

publishing > requestPort

This setting has been deprecated. Please use the corresponding setting under publishing > adapter > nanomsg.

Mandatory: No
Default: (publishing > adapter > nanomsg > publishPort) + 1

publishing > secondaryPorts

These settings have been deprecated. Please use the corresponding settings under publishing > adapter > nanomsg.

publishing > secondaryPorts > publishPort

This setting has been deprecated. Please use the corresponding setting under publishing > adapter > nanomsg.

Mandatory: No
Default: publishing > adapter > nanomsg > publishPort

publishing > secondaryPorts > requestPort

This setting has been deprecated. Please use the corresponding setting under publishing > adapter > nanomsg.

Mandatory: No
Default: publishing > adapter > nanomsg > requestPort

Enable Kafka debug logging

To enable Kafka debug logging, follow these steps:

  1. Open your GSE.
  2. Click Operating environment in the Navigation tree.
  3. Select the Debug tab.
  4. Open the drop-down list by the Publishing, and tick adapter.
  5. Click Publishing in the Navigation tree.
  6. Select the Advanced tab.
  7. In Additional settings, add kafka.debug= followed by the debug categories. We recommend it is set to topic,protocol.
    • For a full set of debug categories, see the librdkafka documentation. The queue and all categories are extremely verbose.
  8. Click Save.

Standardised Formatting

Some data can vary in format from dataview to dataview or even from row to row or column to column. Standardised Formatting allows normalisation of this data to a standard format for downstream systems.

Currently dateTime is supported. The dateTime formatter can be applied as shown below: Date-times, dates and times are formatted to ISO 8601 format.

Type Format
Date-time 2012-07-27T19:12:00Z
Date-time with micro seconds 2012-07-27T19:12:00.123456Z
Date 2012-07-27

Note the adjustment to UTC. Cells containing only times are assumed to be for the current day and formatted as a date-time.

There are two types of standardised formatting: System supplied formatting and User specified formatting.

System Supplied Formatting

The Gateway provides a set of converters for cells in Geneos plugins.

These are defined in a file located in <gateway resources directory>/standardisedformats/formats.json. By default, the gateway resources directory is <gateway directory>/resources but you can modify this using a Command line options.

This data in this file is in JavaScript Object Notation (JSON).

It is recommended that this file be left unchanged. However, in unusual circumstances it may be preferable to update the formats.json file rather than generate a set of user specified formatting.

The file has a number of entries under the "formats" label at the start. Taking one as an example it breaks down as follows:

{
"formats" : [
{
"type"     : "datetime",
"format"   : "%a %b %d %H:%M:%S %Y",
"region"   : "gateway",
"plugin"   : "Gateway-gatewayData",
"cell"     : { "row" : "licenseExpiryDate", "column" : "value"}
},
...
]}
Element Description
type

Data type of the data being formatted.

Currently the only format is "datetime".
format

Specifies the expected format of the data.

See the following section for a description of the time parsing codes.

Where you need several of these to accommodate different servers that may generate dates in differing formats use the "formats" tag defined below.

formats Where a variable may be in a number of formats an array of alternative formats may be specified.
{
...
"formats",
["%a %b %d %H:%M:%S %Y", "%s"]
...
}
region

Timezone information for the formatter.

For our purposes these should be "gateway" or "netprobe" depending on the location of the sampler.

The formatter will use the timezone of the specified component. Where netprobe timezone isn't known it will fall back to the Gateway.
plugin The name of the plugin this formatter applies to. The formatter applies to all dataviews created by the plugin unless you restrict this.
dataview

Only applies to variables within the named dataview.

{
...
"dataview", "overview"
...
}
cell

Targets the formatting of a specific row / column pair.

For more flexibility, you can target a range of columns.

{
...
"cell"  : { "row" : "licenseExpiryDate", "column" : "Value"}
...
}
cols

An array or regular expressions used to target columns within a dataview.

{
...
"cols"     : ["applied|changed"]
...
}

Will target all column headings containing "applied" or "changed" as part of the column name. i.e. "appliedDate"
headlines An array or regular expressions used to target columns witin a dataview. Works as cols above and can be used in conjunction.

Note: Some "formats" entries may end with a special format called "raw". This prevents errors from being logged where none of the formats were able to process the input.

User Specified Formatting

Certain dataviews contain cells where the format of the cell cannot be determined. This may be because the data is generated by a user script, or it may be because a Geneos plugin is extracting data whose format is determied by the enviroment of the server on which the plugin is running.

The standardised formatting tab on samplers allows the user to describe the originating format of variables (cells and headlines). This allows them to be transformed into standardised format. You can also specify which cells in the sampler the format is applied to by using the dataview name and the applicability sections.

Configuration

samplers > sampler > standardisedFormatting

See Standardised Formatting.

samplers > sampler > standardisedFormatting > dataviews > dataview

Provides a list of standardised formatting variables to apply to one or more dataviews in the sampler. When the name of the dataview is set the variable definitions are restricted to the named dataview. If the name is left unset the variable definitions applies to all dataviews belonging to the sampler.

Mandatory: No
Default: Not set

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable

Variable definition specifying type and applicability of variable.

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > type

Specifies the type of the variable. Currently on dateTime is supported.

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > type > dateTime

Specifies that the variable is a date-time (includes dates and times which are assumed to be for the current day).

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > type > dateTime > formats > format

Specifies the expected format of the data. See the following section for a description of the time parsing codes. There can be several of these to accommodate that different servers may generate dates in differing formats.

stdfmt02

Mandatory: Yes

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > type > dateTime > ignoreErrorsIfFormatsFail

If set to true and the none of the formats can translate the data then the errors generated are suppressed. This can be used to specify a NULL format that overrides a system provided one with no formatting. This can be useful for example if you don't care about formatting these and the output is in an unexpected format due to locale.

Mandatory: No
Default: false

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > type > dateTime > overrideTimezone

The default timezone date is that of the dataview. i.e. if the Probe has a timezone set the date would be interpreted from that timezone. If not the timezone will match the Gateway's timezone. This setting allows you to explicitly set the timezone from which the data is being received.

Mandatory: No

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > applicability

Allows matching of formats to dataview variables.

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > applicability > headlines

Allows you to match variables to headline variables.

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > applicability > headlines > regex

A regular expression to match against the headline names.

Expression Effect
date Matches all headlines containing date.
date$ Matches all headlines ending in date$.
^date$ Matches the healdine date.

All values within the matching headlines will have the formatter applied for publishing.

Mandatory: No

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > applicability > columns > regex

A regular expression to match against the column names.

Expression Effect
date matches all columns containing date
date$ matches all columns ending in date$
^date$ matches the column date

All values within the matching columns will have the formatter applied for publishing.

Mandatory: No

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > applicability > cells

Maps to a specific row and column. These mappings are good for dataviews made from name, value pairs and take priority on lookup over column regular expressions. As these are specific the row and columns are not regular expressions.

Mandatory: No

operatingEnvironment > debug > DirectoryManager > showStandardisedFormattingErrors

Turns on error reporting for errors found processing inputs to standardised formats. If this is not enabled and errors are found then a summary of the number of errors found is logged at 10 minute intervals.

Gateway Publishing stats

Description

The Gateway package comes with a utility to help understand that amount of data that the Gateway publishes. This utility can be used to decide on the requirements of applications that consume Gateway data from Gateway publishing.

The utility is a command-line application located in resources/nanomsg_stats and called nanomsg_stats.<platform>. The utility connects to a gateway that is publishing data using nanomsg (See Publishing data to external systems). The utility consumes all the data sent by the gateway and periodically outputs statistics on the amount of data the gateway has sent on each nanomsg channel.

The user can configure the following:

  • the nanomsg port on which the gateway to connect to is publishing.
  • the file which will be used to store the statistics
  • how often to collate the statistics and write the data to the file
  • a filter to apply to the published messages before they are processed by the utility.

Once started the utility keeps running until it is killed by the user. Every n seconds (where n is specified by the user), it will output a set of statistics. One for each topic and a summary row containing information about all the messages processed.

In the example below, you can see that the program collected data every 60 seconds and ran for 3 minutes. Statistics were written to the output file every minute.

Output

The utility produces a CSV file. The first line in the file are the column headings, while subsequent lines are summary lines. Summary lines summarise the amount of data seen over a set period of time either on a single topic or on all topics. An example output is shown below

Stats Timestamp Duration URL Topic Number of Fields Number of Messages Max Message Size Avg Message Size Min Message Size Total Size of Messages
Topic 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 enriched.headlines.Probes.Probes..Self.V.G1 9 1 473 473 473 473
Topic 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 enriched.table.CPU.CPU.System.ME1.P1.G1 11 1160 515 511 505 592578
Topic 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 enriched.table.CPU.CPU.System.ME2.P2.G1 11 1200 516 511 505 613140
Topic 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 enriched.table.Hardware.Hardware.System.ME1.P1.G1 3 25 359 344 333 8612
Topic 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 enriched.table.Hardware.Hardware.System.ME2.P2.G1 3 25 359 344 333 8612
Topic 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 enriched.table.Probes.Probes..Self.V.G1 23 22 924 921 918 20262
Topic 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 raw.headlines.Probes.Probes..Self.V.G1 9 1 468 468 468 468
Topic 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 raw.table.CPU.CPU.System.ME1.P1.G1 11 1161 510 506 500 587284
Topic 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 raw.table.CPU.CPU.System.ME2.P2.G1 11 1200 511 506 500 607140
Topic 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 raw.table.Hardware.Hardware.System.ME1.P1.G1 3 25 354 339 328 8487
Topic 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 raw.table.Hardware.Hardware.System.ME2.P2.G1 3 25 354 339 328 8487
Topic 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 raw.table.Probes.Probes..Self.V.G1 23 22 919 916 913 20152
Gateway 2016-11-15 10:38:57 60 tcp://127.0.0.1:7070 4867 924 509 328 2475695
Topic 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 enriched.headlines.Probes.Probes..Self.V.G1 9 4 473 473 473 1892
Topic 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 enriched.table.CPU.CPU.System.ME1.P1.G1 11 1160 516 511 503 592641
Topic 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 enriched.table.CPU.CPU.System.ME2.P2.G1 11 1200 516 511 504 613030
Topic 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 enriched.table.Hardware.Hardware.System.ME1.P1.G1 3 13 359 345 332 4486
Topic 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 enriched.table.Hardware.Hardware.System.ME2.P2.G1 3 13 359 345 332 4486
Topic 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 enriched.table.Probes.Probes..Self.V.G1 23 24 924 921 918 22104
Topic 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 raw.headlines.Probes.Probes..Self.V.G1 9 4 468 468 468 1872
Topic 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 raw.table.CPU.CPU.System.ME1.P1.G1 11 1160 511 506 498 586841
Topic 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 raw.table.CPU.CPU.System.ME2.P2.G1 11 1200 511 506 499 607030
Topic 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 raw.table.Hardware.Hardware.System.ME1.P1.G1 3 13 354 340 327 4421
Topic 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 raw.table.Hardware.Hardware.System.ME2.P2.G1 3 13 354 340 327 4421
Topic 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 raw.table.Probes.Probes..Self.V.G1 cwb MW
Gateway 2016-11-15 10:39:57 60 tcp://127.0.0.1:7070 4828 924 511 327 2465208
Topic 2016-11-15 10:40:57 60 tcp://127.0.0.1:7070 enriched.headlines.Probes.Probes..Self.V.G1 9 3 473 473 473 1419
Topic 2016-11-15 10:40:57 60 tcp://127.0.0.1:7070 enriched.table.CPU.CPU.System.ME1.P1.G1 11 1160 516 511 502 592529
Topic 2016-11-15 10:40:57 60 tcp://127.0.0.1:7070 enriched.table.CPU.CPU.System.ME2.P2.G1 11 1200 516 511 504 613062
Topic 2016-11-15 10:40:57 60 tcp://127.0.0.1:7070 enriched.table.Hardware.Hardware.System.ME1.P1.G1 3 9 359 345 330 3108
Topic 2016-11-15 10:40:57 60 tcp://127.0.0.1:7070 enriched.table.Hardware.Hardware.System.ME2.P2.G1 3 8 359 347 330 2777
Topic 2016-11-15 10:40:57 60 tcp://127.0.0.1:7070 enriched.table.Probes.Probes..Self.V.G1 23 24 924 921 918 22104
Topic 2016-11-15 10:40:57 60 tcp://127.0.0.1:7070 raw.headlines.Probes.Probes..Self.V.G1 9 3 468 468 468 1404
Topic 2016-11-15 10:40:57 60 tcp://127.0.0.1:7070 raw.table.CPU.CPU.System.ME1.P1.G1 11 1160 511 506 497 586728
Topic 2016-11-15 10:40:57 60 tcp://127.0.0.1:7070 raw.table.CPU.CPU.System.ME2.P2.G1 11 1200 511 506 499 607062
Topic 2016-11-15 10:40:57 60 tcp://127.0.0.1:7070 raw.table.Hardware.Hardware.System.ME1.P1.G1 3 9 354 340 325 3063
Topic 2016-11-15 10:40:57 60 tcp://127.0.0.1:7070 raw.table.Hardware.Hardware.System.ME2.P2.G1 3 8 354 342 325 2737
Gateway 2016-11-15 10:40:57 60 tcp://127.0.0.1:7070 4808 924 511 325 2457977

The CSV file has the following columns;

Column Description
Stats This indicates if this row shows data from a single nanomsg topic or all the topics processed from the gateway. The value is either Topic or Gateway.
Timestamp This is the time that the statistics were gathered. It is the end of the period the stats were gathered over and not the start.
Duration This is the duration over which the stats were gathered. This value is measured in seconds
URL This is the URL used to connect to the Gateway publishing.
Topic This is the Nanomsg Topic name. (This column will be empty for Gateway rows)
Number of Fields Number of fields in the data part of the message. (This column will be empty for Gateway rows)
Number of Messages Number of messages received in the specified time period in bytes.
Max Message Size Maximum message size seen in the specified time period in bytes.
Avg Message Size Average message size over the specified time period in bytes.
Min Message Size Minimum message size seen in the specified time period in bytes.
Total Size of Messages Sum of the sizes of all messages seen in the time period. This is measured in bytes
Gateway Name of the gateway that generated data summarised in this row. (This column will be empty for Gateway rows)
Probe Name of the probe that generated data summarised in this row. (This column will be empty for Gateway rows)
Managed Entity Name of the managed entity that generated data summarised in this row. (This column will be empty for Gateway rows)
Sampler Name of the sampler that generated data summarised in this row. (This column will be empty for Gateway rows)
Sampler Type Type of the sampler that generated data summarised in this row. (This column will be empty for Gateway rows)
Dataview Name of the dataview that generated data summarised in this row. (This column will be empty for Gateway rows)
OS Type OS Type of the probe that generated data summarised in this row. (This column will be empty for Gateway rows)
Plugin Type Plugin type of sampler that generated data summarised in this row. (This column will be empty for Gateway rows)

Command Line options

The nanomsg_stats utility takes up to 5 command line parameters.

  • The first parameter is the url to connect to. This url contains the gateway host and the port on which the gateway is publishing. The argument is in the form tcp://<host>:<port>. So to connect to the default port on localhost you would use one of the following urls; tcp://locahost:7070, tcp://127.0.0.1:7070. This argument is mandatory.
  • The second parameter is a second url to connect to. This is the url that contains the gateway host and the port on which the gateway request response service is located. The argument is in the form tcp://<host>:<port>. So to connect to the default port on localhost you would use one of the following urls; tcp://locahost:7071, tcp://127.0.0.1:7071.
  • The third argument is the name of the csv file in which to store the statistics acquired. This argument is mandatory.
  • The fourth argument is the time period over which the utility gathers data before producing stats. This argument is a time in seconds. It is optional and if not supplied the system will produce stats every 60s.
  • The fifth argument is a filter that is applied to the data before it is processed. Any topic that does not start with the string provided will be ignored. So a value of "raw.table" will ignore the following topics:
    • Any topic starting with enriched.headline
    • Any topic starting with raw.headlines
    • Any topic starting with enriched.table

It should be noted that the filter occurs before the stats are generated. This means that any data that is filtered out will not contribute to the summary Gateway statistics.

Publishing Message Formats Overview

The Gateway can be configured to publish metrics, directory data and metadata in JSON format.

The Gateway supports publishing to a Kafka cluster, as a Kafka "producer client". This provides a more resilient mechanism for publishing and allows other systems to consume Geneos data by implementing a Kafka consumer client.

Support for each publishing platform and operating system is provided by an adapter implemented as a shared library. Publishing using an adapter is supported only on 64-bit Linux and Solaris x86. The format of the JSON 'payload' of the published messages is determined by the Gateway, so it is the same whichever adapter is used.

The configuration settings used to enable publishing using either adapter are described above. This remainder of this document describes the message formats used.

Message presentation

Although the format of the message payload sent by the Gateway is the same in each case, the different publishing platforms support different mechanisms for selecting and identifying messages. They also use different terminology to refer to similar concepts.

In this document we use the following terms:

  • Topic
    A string identifying the category of message, for example "raw.headlines". Messages within a topic have similar schemas (in fact, apart from metrics messages, all messages in a topic have the same schema.)
  • Key
    A string identifying a subset of a topic, for example the key "gatewayLoad.gatewayLoad..Gateway.vp.Ad-hoc GW" might appear within the topic "raw.headlines". All messages with the same key will have the same schema, but there are better ways to infer the schema than by inspecting the key.
    All messages which have a sequencing relationship to each other will have the same key: keys may be used to sub-divide topics for load balancing or to filter messages of interest for a particular application.
    Directory message topics are not subdivided in this way; they use an empty string as a key.
  • Payload
    The actual data to be consumed by a client system. The payload is a JSON object, described in detail at Message payloads and Appendix: Payload schema below.

Kafka topics, keys and messages

When the Kafka adapter is used:

  • The message topic is prefixed with a configurable string (the default is "geneos-") and used as the Kafka topic.
  • The message key is used to allocate messages to partitions within the Kafka topic and is available as the message key to Kafka clients.
  • The message payload becomes the message body.

Nanomsg topics and message payloads

The Nanomsg API simulates a limited form of topic-based subscription in that, if you subscribe to "Foo", you get all messages that begin with the string "Foo".

When the Nanomsg adapter is used, the messages published consist of a modified topic string, followed by the payload.

  • For messages within the metrics topics, the modified topic string is the message topic, followed by a period (\u002E), followed by the message key, followed by a pipe character (\u007C), so that the whole message is:

    <topic>.<key>|<payload>

  • For messages in other topics, the modified topic string is the message topic followed by a pipe character (\u007C), so that the whole message is:

    <topic>|<payload>

This message format allows Nanomsg clients to subscribe to individual topics, or to subsets of metrics topics identified by any leading substring of the key.

For example, with topic managedEntities, and payload

{
	"data": {
	"timestamp": "2015-07-01T16:17:10.301Z",
	"name": "myEntity",
	"probe": "p",
	"gateway": "gw",
	"attributes": {
	"Purpose": "Realism"
			}
		},
	"operation": "create"
}
 {
  "data": {
    "timestamp": "2015-07-01T16:17:10.301Z",
    "name": "myEntity",
    "probe": "p",
    "gateway": "gw",
    "attributes": {
      "Purpose": "Realism"
    }
  },
  "operation": "create"
}

the message sent via Nanomsg would be:

managedEntities|{"data":{"timestamp":"2015-07-01T16:17:10.301Z","name":"myEntity","probe":"p","gateway":"gw","attributes":{"Purpose":"Realism"}},"operation":"create"}

Message key format (subject to change)

In principle, the message key is an opaque string used to allocate messages to partitions in Kafka or to support a limited form of low-level message filtering (Nanomsg). Applications needing to filter messages should ignore the key and use the "target" object within the payload.

In practice, "what is the format of the message key?" is a frequently asked question. At present, it is roughly equivalent to a reversed XPath delimited by dots:

<datview>.<sampler>.<type>.<managed entity>.<probe>.<gateway>

For example (note that if a component name is not applicable, it will be omitted, but the following dot will not):

CPU.CPU..myEntiy.theProbe.Ad-hoc GW
....theProbe.Ad-hoc GW

The number of name components (and hence the number of dots) is the same for all metrics and (in the case of publishing via Kafka) metadata messages.

Message payloads

A JSON schema for the publishing message payloads is provided as Appendix: Payload schema.

The examples in this section are pretty-printed: the messages published by Gateway have no redundant whitespace.

Directory messages

Messages on the probes, managedEntities and dataviews topics provide information about the corresponding items in the Gateway directory hierarchy. They are published as these items are created and deleted and as their run-time attributes are updated.

The Nanomsg adapter also allows clients to request retransmission of all messages in this category.

Messages in this category are also retransmitted when publishing setup is reconfigured (for example to change the Kafka topic prefix) and the Nanomsg adapter is not being used.

Probes

Example messages

A message sent as a Netprobe comes up:

{
  "data": {
    "timestamp": "2015-07-01T16:18:23.263Z",
    "name": "theProbe",
    "gateway": "Ad-hoc GW",
    "osType": "Linux",
    "HostName": "linux-dev",
    "Port": "7036",
    "ConState": "Up",
    "OS": "Linux 2.6.18-371.4.1.el5",
    "Version": "GA3.1.1-150515"
  },
  "operation": "update"
}

A message describing a virtual probe:

{
  "data": {
    "timestamp": "2015-10-20T09:15:04.732Z",
    "name": "vp",
    "gateway": "Ad-hoc GW",
    "osType": "Virtual",
    "ConState": "Up"
  },
  "operation": "update"
}

Points to Note:

    • The properties whose names start with capital letters are configuration ('HostName', 'Port') or run-time ('ConState', 'OS', 'Version') parameters. Of these, only 'ConState' is applicable to virtual probes.
    • When a connection is established to a probe, several update messages will be published, one for each run-time parameter.

Managed Entities

Example message
{
  "data": {
    "timestamp": "2016-05-25T15:02:18.255Z",
    "name": "theEntity",
    "probe": "theProbe",
    "gateway": "Ad-hoc GW",
    "attributes": {
      "Team": "Middleware",
      "Purpose": "Testing"
    }
  },
  "operation": "update"
}

Note: An update message is published whenever a managed entity attribute is changed, added or deleted.

Dataviews

Example message
{
  "data": {
    "timestamp": "2016-05-27T12:51:16.009Z",
    "dataview": "CPU",
    "sampler": "CPU",
    "pluginName": "CPU",
    "type": "Default Samplers",
    "managedEntity": "basics",
    "probe": "theProbe",
    "gateway": "Ad-hoc GW",
    "topicSuffix": "CPU.CPU.Default Samplers.basics.theProbe.Ad-hoc GW",
    "availableTopics": [
      "enriched.table.CPU.CPU.Default Samplers.basics.theProbe.Ad-hoc GW",
      "enriched.headlines.CPU.CPU.Default Samplers.basics.theProbe.Ad-hoc GW",
      "raw.table.CPU.CPU.Default Samplers.basics.theProbe.Ad-hoc GW",
      "raw.headlines.CPU.CPU.Default Samplers.basics.theProbe.Ad-hoc GW"
    ]
  },
  "operation": "create"
}

Note: Update messages are only sent on the dataviews topic in response to a client request or a change in Publishing configuration.

Metrics messages

Raw and enriched forms of metrics data

Metrics data is available in two forms:

  • Raw form — includes only the rows and cells provided by the data source (normally a Netprobe, but possibly a Gateway plugin, or, in the case of Gateway sharing, an exporting Gateway.) In this case the published sample time is provided by the data source.
  • Enriched form — also includes additional rows and cells configured on the publishing Gateway and (usually) populated by rules. In this case the published sample time is the time on the publishing Gateway that the dataview was last updated (either by receiving an update or by processing a rule.)
    When new data for a row is provided by the data source and then values in that row are computed by Gateway rules, an update message will be published for each of these changes as they occur: first the change from the data source and then each change made by a rule.

Headline data

  • Raw headline messages are published on the topic raw.headlines.
  • Enriched headline messages are published on the topic enriched.headlines.

A raw message:

{
  "data": {
    "sampleTime": "2016-05-27T12:54:29.685Z",
    "target": {
      "gateway": "Ad-hoc GW",
      "probe": "theProbe",
      "managedEntity": "basics",
      "type": "Default Samplers",
      "sampler": "CPU",
      "dataview": "CPU",
      "filter": {
        "osType": "Linux",
        "pluginName": "CPU"
      }
    },
    "samplingStatus": "OK",
    "numOnlineCpus": "2",
    "loadAverage1Min": "0.00",
    "loadAverage5Min": "0.00",
    "loadAverage15Min": "0.00",
    "numPhysicalCpus": "1",
    "HyperThreadingStatus": "DISABLED",
    "numCpuCores": "2"
  },
  "operation": "create"
}

This message represents the headlines of a dataview.

The 'target' property identifies the dataview. The 'filter' property of the target provides additional information which may help a consumer of the message to determine which properties should appear in the rest of the message. For example, the Geneos CPU plugin on Windows publishes a dataview that has different headlines and column names from the CPU plugin on Linux.

The properties that follow 'target' are the headlines for the dataview. 'samplingStatus' is always present as the next property after 'target'; in some cases it is the only headine present.

An enriched message for the headlines of the same dataview:

{
  "data": {
    "sampleTime": "2016-05-27T12:54:29.877Z",
    "target": {
      "gateway": "Ad-hoc GW",
      "probe": "theProbe",
      "managedEntity": "basics",
      "type": "Default Samplers",
      "sampler": "CPU",
      "dataview": "CPU",
      "filter": {
        "osType": "Linux",
        "pluginName": "CPU"
      }
    },
    "samplingStatus": "OK",
    "numOfflineCpus": "0",
    "numOnlineCpus": "2",
    "loadAverage1Min": "0.00",
    "loadAverage5Min": "0.00",
    "loadAverage15Min": "0.00",
    "numPhysicalCpus": "1",
    "HyperThreadingStatus": "DISABLED",
    "numCpuCores": "2"
  },
  "operation": "update"
}

This message represents the headlines of the same dataview. The differences from the "raw" message are:

  • computed headlines are included: in this example there is a computed headline called "numOfflineCpus" (a meaningless number included for the sake of this example);
  • the 'sampleTime' is the time the Gateway generated the message.

Table (cell) data

  • Raw table messages are published on the topic raw.table.
  • Enriched table messages are published on the topic enriched.table.
Example messages

A raw message:

{
  "data": {
    "sampleTime": "2016-05-27T12:59:56.085Z",
    "target": {
      "gateway": "Ad-hoc GW", "probe": "theProbe", "managedEntity": "basics",
      "type": "Default Samplers", "sampler": "CPU", "dataview": "CPU",
      "filter": { "osType": "Linux", "pluginName": "CPU" }
    },
    "name": "Average_cpu",
    "row": {
      "type": "",
      "state": "",
      "clockSpeed": "",
      "percentUtilisation": "0.97 %",
      "percentUserTime": "0.53 %",
      "percentKernelTime": "0.18 %",
      "percentWaitTime": "0.25 %",
      "percentIdle": "99.03 %"
    }
  },
  "operation": "update"
}

This message represents a row from a dataview. As in the case of headline messages, the 'target' property identifies the dataview and provides additional information to help determine which properties should appear in the rest of the message.

The 'name' property identifies the row; the 'row' property contains the cell data. Because this is a "raw" message, no computed cells are shown and the 'sampleTime' property is the time the netprobe published the sample.

An enriched message for the same table row

{
  "data": {
    "sampleTime": "2016-05-27T12:59:56.283Z",
    "target": {
      "gateway": "Ad-hoc GW", "probe": "theProbe", "managedEntity": "basics",
      "type": "Default Samplers", "sampler": "CPU", "dataview": "CPU",
      "filter": { "osType": "Linux", "pluginName": "CPU" }
    },
    "name": "Average_cpu",
    "row": {
      "workDone": "3194.4816",
      "type": "",
      "state": "",
      "clockSpeed": "",
      "percentUtilisation": "0.97 %",
      "percentUserTime": "0.53 %",
      "percentKernelTime": "0.18 %",
      "percentWaitTime": "0.25 %",
      "percentIdle": "99.03 %"
    }
  },
  "operation": "update"
}

This message represents the same row from the same dataview. The differences from the "raw" message are:

  • computed columns are included: in this case the cell data includes a computed column called "workDone" (a meaningless number computed for the sake of this example);
  • the 'sampleTime' is the time the Gateway generated the message.
Sample times for imported data

Where Gateway sharing is used, the "raw form" messages for imported metrics data may include rows, columns or headlines added by the exporting Gateway. For each update, the sample time will reflect the original source of the data.

That is, when a new dataview sample is provided by a Netprobe, the exporting Gateway will first forward that sample, with sample time provided by the Netprobe and then, if a rule is triggered, will send a further update for computed data using its own time for the sample time. The raw form published by the importing Gateway will therefore include sample times from both the Netprobe and the exporting Gateway. Although all timestamps are UTC, if the operating system clocks are not synchronised, the sample times in the "raw form" published messages may be inconsistent. For example, computed updates may be published with earlier sample times than the data from which they are calculated.

Points of Note:

  • All dataview cells and headlines are published as strings. Although data may be provided as strings or numbers by plugins, some plugins may vary the data type of a column from one row to the next or from one sample to the next. If the Gateway were to attempt to use the current data type of a cell when formatting JSON messages, the implied schema might vary from one message to the next.
  • Certain cells are marked as dateTime cells by the gateway. This is done using Standardised Formatting (see Standardised Formatting). If a cell is marked as a dateTime cell then the contents of the cell will be reformatted prior to being published. The format used is ISO 8601 in the form e.g. "1970-02-01T09:00:00.123456". Note the fractional part is optional and may go to microsecond accuracy depending on the data. A dateTime can also be sent as a pure date e.g. "1970-02-01".

Metadata messages

These messages provide information about severity, snooze status and user assignment of data items at all levels of the directory hierarchy.

Severity messages

Severity messages are published on the topic metadata.severity.

Example messages

Here are two of the messages generated when the status of a dataview headline changed from "WARNING" to "OK":

{
  "data": {
    "timestamp": "2016-05-25T15:02:18.357Z",
    "target": {
      "gateway": "Ad-hoc GW",
      "probe": "vp",
      "managedEntity": "Gateway",
      "sampler": "probeData",
      "type": "",
      "dataview": "probeData",
      "headline": "samplingStatus",
      "filter": {
        "osType": "Virtual",
        "pluginName": "Gateway-probeData"
      }
    },
    "severity": "OK",
    "active": true,
    "snoozed": false,
    "snoozedParents": 2,
    "userAssigned": false,
    "value": {
      "cell": "OK"
    }
  },
  "operation": "update"
}

{
  "data": {
    "timestamp": "2016-05-25T15:02:18.358Z",
    "target": {
      "gateway": "Ad-hoc GW",
      "probe": "vp",
      "managedEntity": "Gateway",
      "filter": {
        "osType": "Virtual"
      }
    },
    "severity": "OK",
    "active": true
    "snoozed": false,
    "snoozedParents": 2,
    "userAssigned": false,
  },
  "operation": "update"
}

Here are some examples of severity messages for dataview items resulting from severity change:

 {
   "data": {
     "timestamp": "2016-07-19T14:51:15.451Z",
     "target": {
       "gateway": "Ad-hoc GW",
       "probe": "vp",
       "managedEntity": "Gateway",
       "sampler": "gateway",
       "type": "",
       "dataview": "gateway",
       "row": "databaseHost",
       "column": "value",
       "filter": {
         "osType": "Virtual",
         "pluginName": "Gateway-gatewayData"
       }
     },
     "severity": "OK",
     "active": true,
     "snoozed": false,
     "snoozedParents": 2,
     "userAssigned": false,
     "value": {
       "cell": "dbhost"
     }
   },
   "operation": "update"
 }

 {
   "data": {
     "timestamp": "2016-07-19T14:51:15.446Z",
     "target": {
       "gateway": "Ad-hoc GW",
       "probe": "vp",
       "managedEntity": "m",
       "sampler": "gw",
       "type": "",
       "dataview": "gw",
       "row": "releaseAge",
       "column": "value",
       "filter": {
         "osType": "Virtual",
         "pluginName": "Gateway-gatewayData"
       }
     },
     "severity": "CRITICAL",
     "active": true,
     "snoozed": false,
     "snoozedParents": 2,
     "userAssigned": false,
     "value": {
       "cell": "4 days",
       "number": 4
     }
   },
   "operation": "create"
 }

 {
   "data": {
     "timestamp": "2016-07-18T15:47:34.288Z",
     "target": {
       "gateway": "Ad-hoc GW",
       "probe": "vp",
       "managedEntity": "m2",
       "sampler": "gw",
       "type": "",
       "dataview": "gw",
       "row": "licenseExpiryDate",
       "column": "value",
       "filter": {
         "osType": "Virtual",
         "pluginName": "Gateway-gatewayData"
       }
     },
     "severity": "OK",
     "active": true,
     "snoozed": false,
     "snoozedParents": 2,
     "userAssigned": false,
     "value": {
       "cell": "2021-01-31T00:00:00Z",
       "dateTime": "2021-01-31T00:00:00Z"
     }
   },
   "operation": "update"
}

Points of Note:

  • A severity message is generated for each data item in the Gateway directory hierarchy whose status is changed.
  • The 'target' property identifies the data item to which the message refers. Only the path components that apply to that item are present.
  • For example, the 'type' property is present in the first example, even though it is empty, but none of the properties from 'sampler' on are present in the second example.
  • 'create' messages are never published for severity metadata, because all data items are created in active state and with undefined severity.
  • 'delete' messages are only published if the item being deleted is inactive or has a severity other than "UNDEFINED".
  • The "value" field is made up of "cell" which is the original cell value. If a number is detected then the optional "number" field is also present. If the value is identifiable is a date-time then the "dateTime" field is present. As with all date-times published this is formated to ISO 8601 format. This provides metadata to downstream systems that require type information.

Snooze messages

Snooze messages are published on the topic metadata.snooze.

Example messages

Here are the messages generated when a managed entity is snoozed and unsnoozed:

{
  "data": {
    "timestamp": "2016-05-27T14:51:10.000Z",
    "target": {
      "gateway": "Ad-hoc GW",
      "probe": "vp",
      "managedEntity": "Gateway",
      "sampler": "gatewayLoad",
      "type": "",
      "filter": {
        "osType": "Virtual"
      }
    },
    "snoozed": {
      "snoozed": true,
      "snoozedBy": "ActiveConsole1",
      "comment": "Simple example",
      "period": "Manual"
    }
  },
  "operation": "update"
}

{
  "data": {
    "timestamp": "2016-05-27T14:51:10.000Z",
    "target": {
      "gateway": "Ad-hoc GW",
      "probe": "vp",
      "managedEntity": "Gateway",
      "sampler": "gatewayLoad",
      "type": "",
      "filter": {
        "osType": "Virtual"
      }
    },
    "snoozed": {
      "snoozed": false,
      "unsnoozedBy: "ActiveConsole1"		
    }
  },
  "operation": "update"
}

Here is an example of snoozing a headline, using some more complex options:

{
  "data": {
    "timestamp": "2016-05-27T14:54:51.000Z",
    "target": {
      "gateway": "Ad-hoc GW",
      "probe": "vp",
      "managedEntity": "Gateway",
      "sampler": "gatewayLoad",
      "type": "",
      "dataview": "gatewayLoad",
      "headline": "samplingStatus",
      "filter": {
        "osType": "Virtual",
        "pluginName": "Gateway-gatewayLoad"
      }
    },
    "snoozed": {
      "snoozed": true,
      "snoozedBy": "ActiveConsole1",
      "comment": "Complex options",
      "period": "Until",
      "untilSeverity": "OK",
      "untilTime": "2016-05-27T15:54:51.000Z",
      "untilValue": "NOTE: Stats collection is disabled"
    }
  },
  "operation": "update"
}

Points of Note:

  • A snooze message is generated only for the data item which is snoozed or unsnoozed, not for its ancestors or descendants.
  • The 'target' property identifies the data item to which the message refers. Only the path components that apply to that item are present.
  • When an item is snoozed, the properties of the 'snoozed' object depend on the form of the snooze command used.
  • When an item is unsnoozed, the 'snoozed' object contains only the 'snoozed' boolean.
  • Only 'update' messages are published for snooze metadata, because, if an item is deleted and recreated, its snooze status will be preserved.

User assignment messages

User assignment messages are published on the topic metadata.userassignment.

Example messages

Here are the messages generated when a probe is assigned and unassigned:

{
  "data": {
    "timestamp": "2016-05-27T15:09:39.000Z",
    "target": {
      "gateway": "Ad-hoc GW",
      "probe": "vp",
      "filter": {
        "osType": "Virtual"
      }
    },
    "userAssignment": {
      "userAssigned": true,
        "assignedTo": "Ann Administrator",
        "assignedBy": "Joe Bloggs",
      "comment": "",
      "period": "Manual"
    }
  },
  "operation": "update"
}

{
  "data": {
    "timestamp": "2016-05-27T15:10:30.677Z",
    "target": {
      "gateway": "Ad-hoc GW",
      "probe": "vp",
      "filter": {
        "osType": "Virtual"
      }
    },
    "userAssignment": {
      "userAssigned": false,
      "unassignedBy: "John Doe"
    }
  },
  "operation": "update"
}

Here is an example of assigning a table cell, using some more complex options:

{
  "data": {
    "timestamp": "2016-05-27T15:17:15.000Z",
    "target": {
      "gateway": "Ad-hoc GW",
      "probe": "vp",
      "managedEntity": "Gateway",
      "sampler": "probeData",
      "type": "",
      "dataview": "probeData",
      "row": "theProbe",
      "column": "security",
      "filter": {
        "osType": "Virtual",
        "pluginName": "Gateway-probeData"
      }
    },
    "userAssignment": {
      "userAssigned": true,
      "assignedTo": "John Doe",
      "assignedBy": "Ann Administrator",
      "comment": "Please set up secure config",
      "period": "Until a change in value",
      "untilValue": "INSECURE"
    }
  },
  "operation": "update"
}

Points of Note:

  • The 'target' property identifies the data item to which the message refers. Only the path components that apply to that item are present.
  • When an item is assigned, the properties of the 'userAssignment' object depend on the form of the user assignment command used.
  • When an item is unassigned, the 'userAssignment' object contains only the 'userAssigned' boolean.
  • Only 'update' messages are published for user assignment metadata, because, if an item is deleted and recreated, its user assignment status will be preserved.

Features common to all types of message

Timestamps

Timestamps are formatted using the following ISO 8601 format: YYYY-MM-DDThh:mm:ss.sssZ. Note that fractional seconds are shown, to millisecond precision, and that all timestamps are in UTC. For raw metrics (see Raw and enriched forms of metrics data in Message Payloads section), the timestamp shown is the sample time reported by the sampler. Otherwise the timestamp, including the sample time shown for enriched metrics is the time on the Gateway host when the message is formatted.

Operation

The possible values for the operation property of the message payload are as follows:

  • create
    The data item described has been added to the configuration, or, in the case of a table row, re-created with a new set of columns.
  • update
    The data item has been updated in some way, including, for the headlines topics, the addition or removal of a headline.
  • snapshot
    The current state of the data item is being sent in response to a client request or a change in Publishing configuration.
  • repeat
    The current state of the data item is being sent as a result of the use of the "Schedule" publishing strategy.
  • delete
    The data item described has been removed from the configuration, or, in the case of a table row, is about to be re-created with a new set of columns.
    When a 'delete' message is sent, the properties of the 'data' part of the message reflect the last known state of the item.

Request/reply messages

In addition to the publish/subscribe mechanism used for the messages described so far, both the Nanomsg adapter and Kafka messages support a mechanism which clients can connect to make requests and receive replies.

To send a request using Nanomsg, the client opens a request/reply connection to the Gateway on the configured request port and sends one of the requests described below. The Gateway acknowledges the request using the same connection. If a client is connected to multiple Gateways, the request must be sent individually to each Gateway that is to act on it.

To send a request using Kafka, the client publishes a request to the Geneos request topic. The name of this topic is normally 'geneos-requests'; the 'geneos-' prefix is the same as the (configurable) prefix used for the publishing topics. The Gateway does not acknowledge the request. All the Gateways that publish to a given Kafka cluster using the same topic prefix will receive and act on the request.

Resend Directory request

A client can request that the Gateway re-send the directory information by sending the following request:

{"request":"resend-directory"}

The Gateway resends all the data for the directory topics. This is sent on the Nanomsg publish port or via the normal Kafka publishing topics.

In the case of a Nanomsg connection, an "ok" response (see below) will also be sent on the request port.

Snapshot Metrics request

A client can request that the Gateway provide a snapshot of selected metrics data by sending a request of the following form:

{"request":"snapshot-metrics","target":{"key":"value","key":"value"},"match":"exact"}

The JSON object provided as the value of "target" specifies the dataviews for which to provide a snapshot.

For example, to request a snapshot of all dataviews from Linux CPU samplers in managed entities with attribute "Region" set to "London" and attribute "Division" set to "FIXED INCOME", a client could send this request:

{
 "request":"snapshot-metrics",
 "target":{
           "attributes":{"Region":"London","Division":"FIXED INCOME"},
           "osType":"Linux",
           "pluginName":"CPU"
          },
 "match":"exact"
}

To request a snaphot of all dataviews from Gateway plugins, one could use this request:

{
 "request":"snapshot-metrics",
 "target":{
           "pluginName":"Gateway*"
          },
 "match":"wildcard"
}

The Gateway will send a snapshot of the metrics (raw and enriched, headlines and table data) for all dataviews which match the target specification. This is sent on the Nanomsg publish port or via the normal Kafka publishing topics.

In the case of a Nanomsg connection, an "ok" response (see below) will also be sent on the request port.

Target syntax

Keys and values are all case-sensitive. The following keys are supported:

  • gateway — Gateway name (redundant on Nanomsg transport).
  • probe — Probe name.
  • osType — Probe operating system type, e.g. "Linux", "Windows". To select virtual probes, use "Virtual".
  • managedEntity — Managed entity name.
  • attributes — Collection of name/value pairs of managed entity attributes.
  • sampler — Sampler name.
  • pluginName — Name of the plugin, e.g. "FKM".
  • dataview — Dataview name.

The osType determines the operating system which the probe is running on. This can be found in the properties of a managed entity that has a running probe:

Active Console identifies the type of operating system based on a numeric value:

  • 0 — Unknown
  • 1 — Other
  • 2 — Windows
  • 3 — Solaris
  • 4 — Linux
  • 5 — HPUX
  • 6 — AIX_OS
  • 7 — Solaris_x86

Exact vs wildcard matching

If the match key in the snapshot request has the value wildcard, then values may include the wildcard characters '*' and '?'. As wildcards, '*' matches zero or more unspecified characters and '?' matches exactly one unspecified character. '*' and '?' can be escaped (so that they match a literal asterisk or question mark) by preceding them with a backslash, '\'. Note that to encode a backslash in JSON, it needs to be doubled.

If the match key in the snapshot request has the value exact, then all characters match themselves.

Snapshot request for events

A client can request that the Gateway provide a snapshot of selected event data.

The request can be for separate snapshots of severity, snooze, and user assign events.

The snapshot responds with the severity, snooze, or userAssign state of all matching target data items (i.e. dataviews and cells), and also the state of its parent data items (i.e. sampler, managed entity, gateway).

The form and syntax required to request the snapshot is very similar to the Snapshot metrics request, with the only difference being the value of the request.

The request values are:

  • snapshot-severity
  • snapshot-snooze
  • snapshot-userassignment

For example, to make a request for severity events use:

{
 "request":"snapshot-severity",
 "target":{
          [YOUR key:value PAIRS HERE]
          },
 "match":"exact"
}

Snapshot request for all events and metrics

A client can request that the Gateway provide a snapshot of all event (severity, snooze, and userAssign) and metrics data. To make a request for this data, send a request in the following form:

{
 "request":"snapshot-all",
 "target":{
           [YOUR key:value PAIRS HERE]
          },
 "match":"exact"
}

System information request

A Nanomsg client can request system information by sending the following JSON request:

{"request":"system-info"}

The response from an active Gateway has the following form:

{
  "response": "ok",
  "data": {
    "systemInfo": {
      "gateway": {
        "name": "Ad-hoc GW",
        "osType": "Linux",
        "version": "RAfeature-160524",
        "startTime": "2016-05-27T15:22:00.074Z",
        "state": "active"
      },
      "protocol": {
        "version": "1.0"
      }
    }
  }
}

The response from an inactive Gateway is similar, but shows "state":"inactive".

Responses

When using Nanomsg, the active gateway will respond to valid requests (apart from the system information request) with the following message:

{"response":"ok"}

The inactive member of a hot standby Gateway pair will not send anything on the publishing port; it will respond as follows on the request port:

{"response":"inactive"}

If a gateway receives an unknown request, it will respond with the following message:

{"response":"error_unknown_request"}

If a gateway receives a message which is not valid JSON, it will respond with the following message:

{"response":"error_badly_formed_request"}

Appendix: Payload schema

{
  "id": "http://schema.itrsgroup.com/geneos/publishing.json",
  "$schema": "http://json-schema.org/draft-04/schema#",

  "description": "Schema for messages published by Gateway.",

  "type": "object",
  "required": [ "data", "operation" ],
  "additionalProperties": false,

  "properties": {
    "data": {
      "type": "object",
      "oneOf": [
        { "$ref": "#/definitions/directoryData" },
        { "$ref": "#/definitions/metricsData" },
        { "$ref": "#/definitions/metadataData" }
      ]
    },
    "operation": {
      "enum": [ "create", "update", "delete", "repeat", "snapshot" ]
    }
  },

  "definitions": {
    "directoryData": {
      "oneOf": [
        { "$ref": "#/definitions/probeData" },
        { "$ref": "#/definitions/managedEntityData" },
        { "$ref": "#/definitions/dataviewData" }
      ]
    },
    "probeData": {
      "description": "Schema for messages in the \"probes\" topic.",
      "properties": {
        "timestamp": {
          "type": "string", "format": "date-time",
          "description": "Time message is published."
        },
        "name": {
          "type": "string",
          "description": "Unique name for the probe."
        },
        "gateway": {
          "type": "string",
          "description": "Name of the publishing gateway."
        },
        "osType": {
          "enum": [ "Linux", "Windows", "Solaris SPARC", "Solaris X86", "AIX", "Unknown", "Virtual" ],
          "description": "Host operating system type."
        },
        "HostName": {
          "type": "string",
          "description": "Configuration parameter."
        },
        "Port": {
          "type": "string", "pattern": "^[0-9]+$",
          "description": "Configuration parameter; a port number as a string."
        },
        "ConState": {
          "type": "string",
          "description": "Status of probe connection. For example,  'Up', 'Down', 'Unreachable', 'Rejected'."
        },
        "OS": {
		  "type": "string",
          "description": "Run time parameter: The actual operating system of the probe."
        },
        "Version": {
          "type": "string",
          "description": "Run time parameter: The probe version."
        }
      },
      "required": [ "timestamp", "name", "gateway", "osType", "ConState" ],
      "additionalProperties": false
    },
    "managedEntityData": {
      "description": "Schema for messages in the \"managedEntities\" topic.",
      "properties": {
        "timestamp": {
          "type": "string", "format": "date-time",
          "description": "Time message is published."
        },
        "name": {
          "type": "string",
          "description": "unique name for the entity."
        },
        "probe": {
          "type": "string",
          "description": "probe providing metrics for the entity."
        },
        "gateway": {
          "type": "string",
          "description": "Name of the publishing gateway."
        },
        "attributes": {
          "type": "object",
          "description": "Attributes configured for the entity.",
          "additionalProperties": { "type": "string" }
        }
      },
      "required": [ "timestamp", "name", "probe", "gateway", "attributes" ],
      "additionalProperties": false
    },
    "dataviewData": {
      "description": "Schema for messages in the \"dataviews\" topic.",
      "properties": {
        "timestamp": {
          "type": "string", "format": "date-time",
          "description": "Time message is published."
        },
        "dataview": {
          "type": "string",
          "description": "Data view name, unique within sampler, type and entity."
        },
        "sampler": {
          "type": "string",
          "description": "Name of sampler providing data view."
        },
        "pluginName": {
          "type": "string",
          "description": "Name of plug-in implementing sampler."
        },
        "type": {
          "type": "string",
          "description": "Type containing sampler (may be empty string)."
        },
        "managedEntity": {
          "type": "string",
          "description": "Managed entity containing sampler."
        },
        "probe": {
          "type": "string",
          "description": "Probe hosting the sampler."
        },
        "gateway": {
          "type": "string",
          "description": "Name of the publishing gateway."
        },
        "topicSuffix": {
          "type": "string",
          "description": "Dataview path used as key for metrics messages relating to this dataview."
        },
        "availableTopics": {
          "type": "array", "minItems": 4, "maxItems": 4,
          "description": "List of Nanomsg topics for this dataview.",
          "items": {
            "type": "string"
          }
        }
      },
      "required": [ "timestamp", "dataview", "sampler", "pluginName", "type", "managedEntity", "probe", "gateway", "topicSuffix", "availableTopics" ],
      "additionalProperties": false
    },
    "metricsData": {
       "oneOf": [
        { "$ref": "#/definitions/enrichedHeadlineData" },
        { "$ref": "#/definitions/enrichedTableData" },
        { "$ref": "#/definitions/rawHeadlineData" },
        { "$ref": "#/definitions/rawTableData" }
      ]
    },
    "targetForMetrics": {
      "description": "Geneos path information identifying the dataview containing the metrics.",
      "properties": {
        "gateway": {
          "type": "string"
        },
        "probe": {
          "type": "string"
        },
        "managedEntity": {
          "type": "string"
        },
        "type": {
          "type": "string"
        },
        "sampler": {
          "type": "string"
        },
        "dataview": {
          "type": "string"
        },
        "filter": {
          "type": "object",
          "description": "Useful additional information that helps determine the set of metrics to expect.",
          "required": [ "osType", "pluginName" ],
          "properties": {
            "osType": {
              "type": "string",
              "description": "The operating system of the probe hosting the dataview."
            },
            "pluginName": {
              "type": "string",
              "description": "The name of the plug-in implementing the sampler."
            }
          }
        }
      },
      "required": [ "gateway", "probe", "managedEntity", "type", "sampler", "dataview", "filter" ],
      "additionalProperties": false
    },
    "enrichedHeadlineData": {
      "description": "Schema for messages in the \"enriched.headline\" topic.",
      "properties": {
        "sampleTime": {
          "type": "string", "format": "date-time",
          "description": "Time message is published."
        },
        "target": {
          "type": "object",
          "$ref": "#/definitions/targetForMetrics"
        },
        "samplingStatus": {
          "type": "string",
          "description": "Sampling status of the dataview. Will be followed by other headlines, including those computed by rules."
        }
      },
      "required": [ "sampleTime", "target", "samplingStatus" ],
      "additionalProperties": { "type": "string" }
    },
    "enrichedTableData": {
      "description": "Schema for messages in the \"enriched.table\" topic.",
      "properties": {
        "sampleTime": {
          "type": "string", "format": "date-time",
          "description": "Time message is published."
        },
        "target": {
          "type": "object",
          "$ref": "#/definitions/targetForMetrics"
        },
        "name": {
          "type": "string",
          "description": "Name of the row."
        },
        "row": {
          "type": "object",
          "description": "Metrics data for the row, including values computed by rules",
          "additionalProperties": { "type": "string" }
        }
      },
      "required": [ "sampleTime", "target", "name", "row" ],
      "additionalProperties": false
    },
    "rawHeadlineData": {
      "description": "Schema for messages in the \"raw.headline\" topic.",
      "properties": {
        "sampleTime": {
          "type": "string", "format": "date-time",
          "description": "Sample time reported by probe."
        },
        "target": {
          "type": "object",
          "$ref": "#/definitions/targetForMetrics"
        },
        "samplingStatus": {
          "type": "string",
          "type": "string",
          "description": "Sampling status of the dataview. Will be followed by other headlines, excluding those computed by rules."
        }
      },
      "required": [ "sampleTime", "target", "samplingStatus" ],
      "additionalProperties": { "type": "string" }
    },
    "rawTableData": {
      "description": "Schema for messages in the \"raw.table\" topic.",
      "properties": {
        "sampleTime": {
          "type": "string", "format": "date-time",
          "description": "Sample time reported by probe."
       },
        "target": {
          "type": "object",
          "$ref": "#/definitions/targetForMetrics"
        },
        "name": {
          "type": "string",
          "description": "Name of the row."
        },
        "row": {
          "type": "object",
          "description": "Metrics data for the row, excluding values computed by rules",
          "additionalProperties": { "type": "string" }
        }
      },
      "required": [ "sampleTime", "target", "name", "row" ],
      "additionalProperties": false
    },
    "metadataData": {
       "oneOf": [
        { "$ref": "#/definitions/severityData" },
        { "$ref": "#/definitions/snoozeData" },
        { "$ref": "#/definitions/userAssignmentData" }
      ]
    },
    "targetForMetadata": {
      "description": "Geneos path information identifying the item to which metadata applies.",
      "properties": {
        "gateway": {
          "type": "string"
        },
        "probe": {
          "type": "string"
        },
        "managedEntity": {
          "type": "string"
        },
        "type": {
          "type": "string"
        },
        "sampler": {
          "type": "string"
        },
        "dataview": {
          "type": "string"
        },
        "headline": {
          "type": "string"
        },
        "row": {
          "type": "string"
        },
        "column": {
          "type": "string"
        },
        "filter": {
          "type": "object",
          "description": "Allows the same criteria to be used when filtering metadata as for metrics.",
          "required": [ "osType" ],
          "properties": {
            "osType": {
              "type": "string",
              "description": "The operating system of the probe."
            },
            "pluginName": {
              "type": "string",
              "description": "The name of the plug-in implementing the sampler. Present when dataview is also present."
            }
          }
        }
      },
      "required": [ "gateway" ],
      "dependencies": {
        "managedEntity": [ "probe" ],
        "type": [ "managedEntity", "sampler" ],
        "sampler": [ "managedEntity", "type" ],
        "dataview": [ "sampler" ],
        "headline": [ "dataview" ],
        "row": [ "dataview", "column" ],
        "column": [ "dataview", "row" ],
        "filter": [ "probe" ]
      },
      "additionalProperties": false
    },
    "severityData": {
      "description": "Schema for messages in the \"metadata.severity\" topic.",
      "properties": {
        "timestamp": {
          "type": "string", "format": "date-time",
          "description": "Time message is published."
        },
        "target": {
          "type": "object",
          "$ref": "#/definitions/targetForMetadata"
        },
        "severity": {
          "enum": [ "UNDEFINED", "OK", "WARNING", "CRITICAL", "UNKNOWN" ],
          "description": "String representing the \"state/@severity\" attribute of the item."
        },
        "active": {
          "type": "boolean",
          "description": "The value of the \"state/@active\" attribute of the item."
        },
        "snoozed": {
          "type": "boolean",
          "description": "Whether the data item is currently snoozed."
        },
        "snoozedParents": {
          "type": "integer",
          "description": "The number of parent data items that are snoozed."
        },
        "userAssigned": {
          "type": "boolean",
          "description": "Whether a user is assigned to the data item."
        },
        "value": {
          "type": "object",
          "description": "Value of the dataitem.",
          "properties": {
            "cell": {
              "type": "string",
              "description": "Value of the cell as is."
            },
            "dateTime": {
              "type": "string", "format" : "date-time",
              "description": "The value of the cell as an ISO formatted date-time"
            },
            "number" : {
              "type": "number",
              "description": "The value of the cell as double."
            }
          },
          "required": ["cell"]
        }
      },
      "required": [ "timestamp", "target", "severity", "active" ],
      "additionalProperties": false
    },
    "snoozeData": {
      "description": "Schema for messages in the \"metadata.snooze\" topic.",
      "properties": {
        "timestamp": {
          "type": "string", "format": "date-time",
          "description": "Time message is published."
        },
        "target": {
          "type": "object",
          "$ref": "#/definitions/targetForMetadata"
        },
        "snoozed": {
          "oneOf": [
            {
              "description": "Schema for \"snoozed\" object for an item which is not snoozed",
              "type": "object",
              "properties": {
                "snoozed": {
                  "enum": [ false ]
                },
                "snoozedBy": {
                  "type":"string",
                  "description": "Name of the user/system that unsnoozed the item."
                },						
              },
              "required": [ "snoozed" ],
              "additionalProperties": false
            },
            {
              "description": "Schema for \"snoozed\" object for an item which is snoozed",
              "type": "object",
              "properties": {
                "snoozed": {
                  "enum": [ true ]
                },
                "snoozedBy": {
                  "type": "string",
                  "description": "Username of the user who snoozed the item."
                },
                "comment": {
                  "type": "string",
                  "description": "User entered comment."
                },
                "period": {
                  "enum": [ "Manual", "SeverityTo", "SeverityFrom", "Time", "DateTime", "ValueChanges", "SeverityToOrTime", "Until", "Unknown" ],
                  "description": "Type of snooze period."
                },
                "untilSeverity": {
                  "enum": [ "UNDEFINED", "OK", "WARNING", "CRITICAL", "UNKNOWN" ],
                  "description": "String representing the associated severity for the \"SeverityFrom\", \"SeverityTo\", \"SeverityToOrTime\" and \"Until\" period types."
                },
                "untilTime": {
                  "type": "string", "format": "date-time",
                  "description": "Time when the data item will become unsnoozed."
                },
                "untilValue": {
                  "type": "string",
                  "description": "Value of the data item when snoozed, for the \"ValueChanges\" or \"Until\" period type ."
                }
              },
              "required": [ "snoozed", "snoozedBy", "comment", "period" ],
              "additionalProperties": false
            }
          ]
        }
      },
      "required": [ "timestamp", "target", "snoozed" ],
      "additionalProperties": false
    },
    "userAssignmentData": {
      "description": "Schema for messages in the \"metadata.userassignment\" topic.",
      "properties": {
        "timestamp": {
          "type": "string", "format": "date-time",
          "description": "Time message is published."
        },
        "target": {
          "type": "object",
          "$ref": "#/definitions/targetForMetadata"
        },
        "userAssignment": {
          "oneOf": [
            {
              "description": "Schema for \"userAssignment\" object for an item which is not assigned to a user.",
              "type": "object",
              "properties": {
                "userAssigned": {
                  "enum": [ false ]
                }
                "unassignedBy": {
                  "type":"string",
                  "description": "Username of the user who unassigned the item."
                },}
              },
              "required": [ "userAssigned" ],
              "additionalProperties": false
            },
            {
              "description": "Schema for \"userAssignment\" object for an item which is assigned to a user.",
              "type": "object",
              "properties": {
                "userAssigned": {
                  "enum": [ true ]
                },
                "assignedTo": {
                  "type": "string",
                  "description": "Username of the user the item is assigned to."
                },
                "assignedBy": {
                  "type":"string",
                  "description": "Username of the user who assigned the item."
                },	
                "comment": {
                  "type": "string",
                  "description": "An explanation about the purpose of this instance."
                },
                "period": {
                  "enum": [
                    "Manual",
                    "UntilOk",
                    "Until severity changes to specified",
                    "Until severity changes from specified",
                    "Until a specific date / time",
                    "Until severity changes to specified or until a specific date / time",
                    "Until a change in value",
                    "Unknown"
                  ],
                  "description": "An explanation about the purpose of this instance."
                },
                "severity": {
                  "enum": [ "UNDEFINED", "OK", "WARNING", "CRITICAL", "UNKNOWN" ],
                  "description": "Severity to which should have (\"UntilOk\", \"Until severity changes to specified\") or not have (\"Until severity changes from specified\") for user assignment to lapse."
                },
                "untilTime": {
                  "type": "string",
                  "description": "Time at which user assignment will lapse."
                },
                "untilValue": {
                  "type": "string",
                  "description": "Value from which item should change for user assignment to lapse."
                }
              },
              "required": [ "userAssigned", "assignedTo", "comment", "period" ],
              "additionalProperties": false
            }
          ]
        }
      },
      "required": [ "timestamp", "target", "userAssignment" ],
      "additionalProperties": false
    }
  }
}