Gateway integration technical reference

Overview

This technical reference provides the configuration of the Gateway Hub top-level section, and the schema configuration in the Publishing tab of a sampler in the GSE.

For more general help regarding installation and set up of Gateway Hub, see Install.

For information on configuring Gateway to Gateway Hub publishing, see Configure.

Note: Starting in version 4.12, Gateway supports a more efficient protocol. This protocol is used when communicating with version 1.4 Gateway Hub and later. This protocol reduces the bandwidth and storage required for metrics messages by up to 50% compared to Gateway version 4.11 and earlier.

Latest versions of both components are available from the ITRS Downloads.

Anomaly detection and centralised configuration

To use anomaly detection features you must start the Gateway in centrally configured mode. To do this, use the -gateway-hub <REST URL> command line option when starting the Gateway. The URL specified here will take precedence over any value specified in the Gateway Setup Editor. For more information, see Connecting to Gateway Hub.

Gateway Hub section in GSE

Configuration

The Gateway Hub section of the GSE provides a range of fields and options:

Setting	Description	Default
Enabled	Enables/disables publishing to Gateway Hub. Gateway Hub and Publishing can be enabled at the same time. Errors generated by attempting to publish to Gateway Hub do not affect the operation of Publishing. Similarly, errors generated by attempting to publish using Publishing do not affect the operation of Gateway Hub. Note: Gateway Hub publishing is not available from shared Gateways. Instead, you must enable publishing from the source Gateways. Caution: After you have started publishing to Gateway Hub, be aware that temporarily disabling publishing may result in Gateway Hub containing references to obsolete or deleted items. For example, if a managed entity is deleted, the Gateway cannot buffer this information and the Gateway Hub is not notified of the deletion when publishing is resumed. It is recommend that once started you do not disable publishing from the Gateway to Gateway Hub.	True
Publishing Address	Specifies a comma-separated list of host/port pairs to use for establishing the connection to Gateway Hub. The adapter makes use of all servers irrespective of which servers are specified here for bootstrapping - this list only impacts the initial hosts used to discover the full set of brokers. Please refer to the description of `bootstrap.servers` in the Kafka documentation for more information. Note: The technology used to publish from Gateway to Gateway Hub is internal and may change in the future.	`hub.example.com:9092`
Additional settings	These additional settings control how the adapter is loaded, or are settings interpreted by the adapter itself. Each setting is entered on one line, and each line specifies one key-value pair. Each setting (key) starts with one of two prefixes: `adapter.` - control the way the adapter is loaded. `kafka.` - settings interpreted by the adapter. Settings with the prefix `adapter.` are: `adapter.library.path` — specifies the location of the shared library that implements the adapter. The default location is the lib64 sub-directory of the Gateway's working directory. An alternate location can be specified as a path relative to the Gateways's working directory, or as an absolute path (as in the example above). `adapter.gateway.name` — used with the topic prefix to construct a unique consumer group id for the gateway. The default group id is `<topicprefix><gatewayname>`, e.g. `geneos-DevGateway`. If necessary, you can override the gateway name part with this setting. Settings with the prefix `kafka.` used for secure publishing are : `kafka.security.protocol` — protocol used to communicate with brokers. Value is always `ssl` when used with Gateway Hub. `kafka.ssl.ca.location` — file or directory path to CA certificate for verifying the broker's key.
Secure Settings	This section allows settings that cannot be set in cleartext, such as passwords, to be encrypted in the Gateway setup file. This has two fields: Name — name of the secure setting. Value — value of the secure setting. This can either be: `stdAES` — AES 256-bit encryption. The password is entered in the Set password box. `var`— a reference to a variable that provides a password in Operating Environment. References to invalid variables, or variables that are not AES-encrypted, are treated as errors.
REST addresses	Specifies the https address that is the base URL for the REST endpoints used by Active Console. For more information about Gateway Hubconnectivity, see Server connections in Connectivity requirements. In v4.12 Gateways and higher, starting the Gateway with `-gateway-hub <URL>` on the command line overrides this setting.	`https://hub.example.com:8081`

Note: Starting the Gateway with command line options overrides these settings. See Gateway command line options.

Sampler Publishing tab

Publishing tab configuration

User-defined schemas are visible in the Publishing tab of a sampler.

Caution: Gateway Hub will reject metrics from Gateway samplers or dataviews that include the / character in their name.

Section Field Description

Dataviews

Dataview

Section	Field	Description
Dataviews	Dataview	Specifies the name of the dataview that the schema applies to. If omitted, the schema applies to the dataview that has the same name as the sampler (the default dataview).
Schema	This has two options: `data` — allows the schema to be specified in the Publishing tab of the sampler. `sampler-schema` — allows you to select from a list of schemas saved in Sampler-schemas section of Static variables.

Specifies the name of the dataview that the schema applies to. If omitted, the schema applies to the dataview that has the same name as the sampler (the default dataview).

Schema

This has two options:

data — allows the schema to be specified in the Publishing tab of the sampler.
sampler-schema — allows you to select from a list of schemas saved in Sampler-schemas section of Static variables.

Schema configuration

This screen is accessed through Data... in the Schema > Dataviews section in the Publishing tab of a sampler.

Here you indicate:

The columns or headlines in the published dataview.
The data types of the columns and headlines.
The units of measure for the columns and headlines (if applicable).

Section	Field	Description	Mandatory	Default
Headlines	Name	Name of the headline that appears in the dataview.
	Type	Data type of the headline that appears in the dataview. The data types include: `string`; `boolean`; `float32`; `float64`; `int32`; `int64`; `date`; `dateTime`. 32-bit types should be specified unless the data is of a size or precision that requires 64-bit types.
	Unit of measure	Specifies the units of measure of the data type that appears in the dataview. Only certain data types have specified units of measure. The options for units of measure are outlined in Units of measure used in schemas.
Columns	Name	Name of the column that appears in the dataview.
	Type	Data type of the column that appears in the dataview. The data types include: `string`; `boolean`; `float32`; `float64`; `int32`; `int64`; `date`; `dateTime`. 32-bit types should be specified unless the data is of a size or precision that requires 64-bit types. Note: Columns declared as `dateTime` not in the form `yyyy-mm-ddThh:mm:ss` require an entry in Standardised Formatting.
	Unit of measure	Specifies the units of measure of the data type that appears in the dataview. Only certain data types have specified units of measure. The options for units of measure are outlined in Units of measure used in schemas.
Pivot		This option should only be used for dataviews that display one value per row with values of a variety of types. For example, a Toolkit sampler with a summary dataview containing two columns - Name and Value, and three rows of varying data types - `dateTime`, `int32`, and `float32`, would be a good candidate for pivoting. If enabled, the dataview is pivoted so row names become column headings and the heading of the single data column becomes the name of a single data row.	No	False
Publish		Turns publishing of the dataview on or off.	No	True

Restrictions on column names

The Gateway Hub normalisation process uses some names that might generate an invalid internal schema for the corresponding metric series.

To avoid generating an invalid schema:

Do not attempt to create a schema for a dataview with columns or headlines that have any of the following names: sampleTime, netprobeTime, and dataviewId.
Do not attempt to create a schema for a dataview with columns or headlines which start with a_, or includes a repeated underscore __.
- For example, if a Geneos dataview contains columns or headlines named both process id and process__20_id, or 10_year_avg and a_10_year_avg, then this generates an invalid internal schema for the corresponding metric series.

Instead you should change the configuration of the sampler, or disable publishing for the dataview.

Appendix

Units of measure used in schemas

Name	Symbol
percent	%
seconds	s
milliseconds	ms
microseconds	μs
nanoseconds	ns
days	d
per second	s-1
megahertz	MHz
bytes	B
kibibytes	KiB
mebibytes	MiB
gibibytes	GiB
bytes per second	B/s
megabits	Mbit
megabits per second	Mbit/s

Sampler schema types

Below is a list of samplers and if they have an entirely built-in schema, a partially built-in schema, or are entirely user-defined.

Plugin	Type	Comments
Gateway-breachPredictor	Built-in
Gateway-clientConnectionData	Built-in
Gateway-databaseLogging	Built-in
Gateway-exportedData	Built-in
Gateway-gatewayHubData	Built-in
Gateway-gatewayLoad	Built-in
Gateway-importedData	Built-in
Gateway-includesData	Built-in
Gateway-licenceUsage	Built-in
Gateway-severityCount	Built-in
Gateway-includesData	Built-in
Gateway-licenceUsage	Built-in
Gateway-managedEntitiesData	Partial
Gateway-probeData	Built-in
Gateway-scheduledCommandData	Built-in
Gateway-scheduledCommandsHistoryData	Built-in
Gateway-severityCount	Built-in
Gateway-severityData	Built-in
Gateway-snoozeData	Built-in
Gateway-sql	User-defined
Gateway-userAssignmentData	Built-in
api	User-defined
api-streams	Built-in
bloomberg-bpipe	Built-in
citrix-apps	Built-in
citrix-processes	Built-in
citrix-sessions	Built-in
citrix-summary	Built-in
clearvision-status	Built-in
combo	User-defined
component-versions	Built-in
control-m	Built-in
cpu	Built-in
desktop-pc-monitoring	Built-in
deviceio	Built-in
disk	Built-in
e4jms-bridges	Built-in
e4jms-connections	Built-in
e4jms-durables	Built-in
e4jms-non-durables	Built-in
e4jms-queues	Built-in
e4jms-routes	Built-in
e4jms-server	Built-in
e4jms-topics	Built-in
e4jms-usersummary	Built-in
euem	Built-in
extractor	User-defined
fidessa	Built-in
fidessa-dq	User-defined
fix	Built-in
fix-analyser2	Partial	Admin data view schema provided, user must define schema for all other dataviews.
fkm	Partial
flm	Partial	User must define schema for additional data displayed based on configuration .
ftm	Built-in
gl-greffon	Built-in
gl-lostorders	User-defined
gl-orderbook	User-defined
gl-permissions	Built-in
gl-router	Built-in
gl-slc	Partial	User must define schema for additional data displayed based on configuration or SLC log file.
gl-slc-relay	Built-in
gl-sle	Built-in
gl-sle-tcp	Built-in
hardware	Built-in
ibmi-job	Built-in
ibmi-message	Built-in
ibmi-pool	Built-in
ibmi-queue	Built-in
ibmi-subsystem	Built-in
ibmi-system	Built-in
informix	Built-in
ipc	Built-in
ix-ma	User-defined
jmx-server	User-defined
jmx-threadinfo	Built-in
market-data-monitor	User-defined
message-tracker	Built-in
mibmon	User-defined
miss-x	Built-in
mq-channel	Built-in
mq-qinfo	Built-in
mq-queue	Built-in
net-ping	Built-in
network	Built-in
nyxt-papastats	Built-in
oracle	Built-in
orc	Built-in
pats-status	Built-in
pats-trading-breaches	Built-in
pats-users	Built-in
perfmon	User-defined
processes	Built-in
rest-extractor	User-defined
rmc-interface	User-defined
sets-slc	Built-in
sql-toolkit	User-defined
stateTracker	User-defined	User must define schema for user defined custom column names.
su	Built-in
sybase	Built-in
sybase-server	Built-in
tcp-links	Built-in
tib-rv	Built-in
tib-rvpublisher	Built-in
tib-rvstream	Built-in
toolkit	User-defined
top	Built-in
trading-technologies	Built-in
trapmon	Partial	User must define schema for user-defined columns in custom view.
unix-users	Built-in
veritas-cluster-server	Built-in
web-mon	Built-in
win-cluster	Built-in
win-services	Built-in
winapps	Built-in
wmi	User-defined
wts-licenses	Built-in
wts-processes	Built-in
wts-sessions	Built-in
wts-summary	Built-in
x-broadcast	Built-in
x-mcast	Built-in
x-multicast	Built-in
x-ping	Built-in
x-route	Built-in
x-services	Built-in
x-top	Built-in
x-traffic	Built-in