Hot Standby

Overview Copied

As the Gateway is responsible for consolidating all monitoring data for distribution, and this introduces a single point of failure. To alleviate the problem, two Gateways can be run as a hot standby pair, so that if one Gateway fails the other remains in operation until the fault is rectified.

A Gateway can be in one of three roles:

A single Gateway runs in Stand Alone mode. A hot standby pair consists of one primary and one secondary Gateway, both running the same version.

In hot standby mode, the primary Gateway is responsible for all Gateway operations, collecting and analysing data from Netprobes, and distributing the results to connected Active Consoles. The secondary Gateway remains in an idle state, ready to take over monitoring duties should the primary Gateway fail.

If the primary Gateway fails, the secondary Gateway connects to Netprobes and assumes the Gateway duties until the primary Gateway becomes operational again.

ref-guide104

Operation Copied

Paired operation Copied

You should run primary and secondary Gateways of the same version on different hosts. This ensures that failure of the host (or network connectivity of the host) affects the secondary Gateway and monitoring can continue unaffected.

Once hot-standby configuration is completed, you should copy the setup file containing this configuration from the primary to the secondary host. You should then apply this setup to the primary Gateway first, and then to the secondary Gateway.

The primary Gateway must be started first because the secondary Gateway attempts to connect to it to synchronise operation. This connection is maintained while both Gateways  are in operation. If the connection fails, then the secondary Gateway will become active after a brief time and take over monitoring duties from the failed primary Gateway.

While the secondary Gateway is active, you should not notice a difference in services between Gateways. In particular, it is still possible to alter the Gateway setup. When the primary Gateway begins running again, the synchronisation between Gateways will ensure that the primary Gateway receives an up-to-date copy of the setup file before it takes up monitoring operations .

Manual failback Copied

Under normal operation, restarting the primary Gateway results in the secondary Gateway releasing the Netprobes and the primary regaining control. This results in a short period of time, while the probes reconnect to the primary Gateway, during which monitoring is paused.

Alternatively, you can run your Gateways in manual failback mode. In this mode, when a secondary Gateway becomes active it does not release control until the manual failback Gateway command is issued. This allows you to restart your Gateway and transfer control at a convenient time. To use this mode, start both Gateways with the -manual-failback option on the command line.

If both Gateways are not started with the same option, then the secondary Gateway will fail to connect to the primary and the primary will run stand-alone mode until this is corrected.

To run the manual failback command, select Secondary > Failback from the right-click menu.

Scheduling failback Copied

The failback command (/Gateway:failback) can be scheduled like any other internal command. This could be useful to schedule an automatic failback outside of business hours if a secondary takes over the role of primary at some point.

A suggested XPath for the target that would ensure that the command is only run on an active secondary in a manual-failback mode is shown below:

/geneos/gateway/directory[(param("HotStandbyRole")="Secondary")][(param("HotStandbyManualFailbackActive")="true")][(rparam("HotStandbyState")="Active")

Monitor hot standby Copied

You can check the hot standby settings of the Gateway your Active Console is connected to, and configure rules using the Gateway Data plugin. For more information, see Gateway data in Gateway Plug-Ins.

A parameter, HotStandbyRole and a runtime parameter HotStandbyState are available on the directory object. These can be used to alert you when hot standby failovers occur.

The following rule is an example of how these parameters could be used.

Target:

/geneos/gateway/directory

Rule:

if param "HotStandbyRole" = "Secondary" and rparam "HotStandbyState" = "Active" then
run "myFailoverAction"
endif

Parameter HotStandbyRole and runtime parameter HotStandbyState can be described as follows:

File synchronisation Copied

Hot standby Gateways can also be configured with a set of external files to keep synchronized. These files are transferred from the active (typically primary) Gateway if the file sizes are different between Gateways, or the active Gateway detects that the file has been modified.

External files are synchronized at two points:

The frequency at which external files are checked can be controlled using the hotStandby > syncFiles > externalFilesCheckInterval setting.

Included setup files (see File merging) can also optionally be synchronized between Gateways, although these files are synchronized only when a setup change is applied, and not on file modification or size changes.

Hot Standby with secure ports Copied

To use secure ports with Gateways, configure the Gateway with a secure listen port in Using different ports for primary and secondary Gateways.

Note

It is possible to specify both secure and insecure ports with Hot Standby. However, if you do specify both secure and insecure ports, you cannot specify different ports for primary and secondary Gateways.

Using different ports for primary and secondary Gateways Copied

The Gateway setup is synchronised between primary and secondary Gateways. Therefore, configuring the settings with a different value depending upon the Gateway role must be done externally to the Gateway setup file.

This also applies to the listenPort setting for Gateway. Therefore, to use different ports for primary and secondary Gateways, you must set the listen port using the ‑port command line option. The ‑port command line option overrides the secure listen port if set, otherwise it overrides the insecure port. For example:

To start the primary Gateway:

gateway2.linux -setup hotStandbyExample.xml -port 22040

To start the secondary Gateway:

gateway2.linux -setup hotStandbyExample.xml -port 22045

The Gateway hotStandby section as displayed in the Gateway Setup Editor (GSE):

Note

You can only override one port setting using the command line option -port. Therefore, you cannot specify different ports for primary and secondary Gateways if you also need to configure both secure and insecure ports.

Configuration Copied

Basic configuration for a Hot Standby pair of Gateways consists of specifying the hostname and port for each Gateway.

Hostnames can be specified as either the machine hostname (obtained by running the hostname command), as an IP address, or as a fully qualified domain name (e.g. “somehost.somedomain.com”).

When Hot Standby Gateway setup is applied, a Gateway determines whether it is running as the primary or secondary host. It does this by comparing first the primary then the secondary hostname with the identity of the host the Gateway is running on. The first match found determines the Gateway role; if no settings match the Gateway will run as Stand Alone.

Each comparison is performed in the following order:

A secondary Gateway attempts to connect to the primary Gateway on the configured hostname and port. If the hostname is not specified as an IP address, Gateway attempts to resolve this using a forward DNS lookup.

When the primary Gateway receives a connection from another Gateway, it attempts to authenticate this Gateway by matching the incoming connection against the configured secondary Gateway. The way this is done is controlled by the hotStandby > primarySecurityCheck setting, which by default resolves the connecting IP to a hostname using a reverse DNS lookup (if required) and match this against the secondary Gateway configuration.

Basic tab Copied

These settings are found under the Basic tab.

hotStandby Copied

Holds the primary and secondary Gateway settings.

hotStandby > primary Copied

Contains settings for the primary Gateway in a hot-standby pair.

hotStandby > primary > hostname Copied

The primary Gateway host, specified as either the machine hostname or an IP address.

The secondary Gateway uses this setting to connect to the primary Gateway, and so the hostname must also resolve to an IP address using a forward DNS lookup.

hotStandby > primary > port Copied

The listen port of the primary Gateway. If the Gateway is listening on both a secure and an insecure port, this value must match the secure port the Gateway is listening on.

If the primary and secondary Gateways need to listen on different ports, one or both of the Gateways must be started with the command line -port option to override the port value set in Using different ports for primary and secondary Gateways.

hotStandby > secondary Copied

Contains settings for the secondary Gateway in a hot standby pair.

hotStandby > secondary > hostname Copied

The secondary Gateway host, specified either as the machine hostname or an IP address.

The primary Gateway checks any incoming Gateway connections against this setting, either comparing the IP address directly or by using a reverse DNS lookup to resolve the connecting host to a hostname for comparison.

hotStandby > secondary > port Copied

The listen port of the secondary Gateway. If the Gateway is listening on both a secure and an insecure port, this value must match the secure port the Gateway is listening on.

If the primary and secondary Gateways need to listen on different ports, one or both of the Gateways must be started with the command line -port option to override the port value set in Using different ports for primary and secondary Gateways.

Advanced tab Copied

These settings are found under the Advanced tab.

hotStandby > syncFiles Copied

Hot standby File synchronisation is configured using this section.

hotStandby > syncFiles > setupIncludes Copied

Boolean setting controls whether included setup files are synchronised between Gateways.

If set to true, include files are synchronised.

However, this should only be done if there is a separate copy of the include files for both the primary and secondary Gateways in the hot standby pair.

It is recommended that you locate include files in a shared location, so that they can be shared between multiple Gateways (e.g. a global users file).

Mandatory: No

Default: false

hotStandby > syncFiles > disabledSetupIncludes Copied

Boolean setting controlling whether disabled included setup files are synchronised between Gateways.

If set to true, disabled include files are synchronised.

Mandatory: No

Default: false

hotStandby > syncFiles > externalFilesCheckInterval Copied

Defines how frequently Gateway checks the external files to confirm if they need to be re-synchronised. Files are checked to investigate if the file size is different between Gateways, or if the file modification time has been updated since the last check.

The check interval is specified in seconds. After this many seconds since the previous check was made, Gateway checks the external files again. External files are also checked when a setup change is applied.

Mandatory: No

Default: 300 (5 minutes)

hotStandby > syncFiles > externalFiles Copied

A list of external files can be defined here, which the Gateway attempts to keep in sync across the hot standby pair.

Mandatory: No

hotStandby > syncFiles > externalFiles > externalFile Copied

Defines a single external file, which the Gateway attempts to keep in sync between Gateways in a hot standby pair.

hotStandby > syncFiles > externalFiles > externalFile > primaryPath Copied

Specifies the path of the external file on the primary Gateway. The Gateway searches here for the file when performing synchronisation. This location should be both readable and writeable by the Gateway.

Mandatory: Yes

hotStandby > syncFiles > externalFiles > externalFile > secondaryPath Copied

Specifies the path of the external file on the secondary Gateway. This optional setting allows you to configure external files that are present on both Gateways, but in different locations.

If a secondary path is not specified, the path is assumed to be the same as the configured primary path.

Mandatory: No

Default: path specified in hotStandby > syncFiles > externalFiles > externalFile > primaryPath

hotStandby > primarySecurityCheck Copied

This setting controls how the primary Gateway verifies that a connection is from the configured secondary Gateway when using hot standby functionality.

Mandatory: No

Default: reverseDns

Value Effect
reverseDns The IP address of the connecting secondary is resolved to a hostname using a reverse DNS lookup. This hostname is then compared against the configured secondary.
forwardDns If the configured secondary Gateway is specified as a hostname, then this is resolved to an IP address. This address is then compared with the address of the connecting secondary Gateway.
disabled Security checks are disabled. A connection from a Gateway is assumed to be from the configured secondary Gateway.

hotStandby > primaryInactiveTimeout Copied

Time in seconds a primary Gateway waits at start-up for a secondary Gateway to connect to it.

Mandatory: No

Default: 30

hotStandby > primaryActivePendingTimeout Copied

Time in seconds a primary Gateway waits for the secondary Gateway to release any Netprobes it controls, after the secondary Gateway has connected.

Mandatory: No

Default: 30

hotStandby > secondaryActivePendingTimeout Copied

Time in seconds a secondary Gateway waits to reconnect to the primary Gateway after the connection between them is lost.

Mandatory: No

Default: 30

["Geneos"] ["Geneos > Gateway"] ["Technical Reference"]

Was this topic helpful?