Gateway Hub 1.x Release Notes

 

Gateway Hub 1.8.0

Released: March 2020

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

Highlights

These are the highlights of this release:

  • Application keys.
  • Grafana Datasource plug-in.
  • Event browser.

Application keys

You can now generate Application keys that allow you to connect third party applications to Gateway Hub securely.

Application keys are designed to be used by a single application and can be centrally revoked if required.

Grafana Data source

The Grafana data source has been released as a full feature. The Geneos data source for Grafana lets you seamlessly integrate data from Gateway Hub into your Grafana dashboards.

Event browser

You can now identify alerting irregularities using the new Events page in the Web Console. The Events page allows you to visualise events generated over time, enabling noisy entities to be identified and remedied.

New features and enhancements

These are the new features and enhancements of this release:

Issue key Release description
STARK-12613 The Apache Commons Compress library has been upgraded to prevent a security vulnerability.
STARK-12594 The Jackson Databind library has been upgraded to prevent a security vulnerability.
STARK-12384 Netty has been upgraded to prevent a security vulnerability.
STARK-12346 Webhooks are now fired correctly when Kafka attempts to commit a batch.
STARK-12305 The Gateway Hub sizing tool is now available as a separate download.
STARK-12300 You can now filter the list of entity attributes and possible values by the deletedEntities parameter.
STARK-12289 Gateway Hub can now ingest much larger dataviews without having to increase Kafka's maximum message size.
STARK-12194 The UI controls for historical charts are now significantly more compact.
STARK-12187 The hubcheck script now checks that custom ports are free.
STARK-12129

Gateway Hub will now retain a maximum of 14 archived logs OR 1GB of log files, whichever is least.

Logs are rolled (archived) each day or when the file size reaches 250MB.

STARK-11937 You can now specify the retention policy for Apache Spark logs. This can help conserve disk space.
STARK-11835 The Webconsole now shows a status bar when Webhooks are active.
STARK-11806 The historical metric chart now shows data from a 24 hour time window by default.
STARK-11778 You only need to specify the SSL password once during installation.
STARK-11183 When creating chained entity filters Gateway Hub now constrains the possible values of each new filter based on the preceding filters.
   

Issues fixed

These are the issues we have fixed in this release:

Issue key Release description
STARK-12460 Gateway Hub now includes additional units of measurement.
STARK-12401 Gateway Hub now correctly handles errors when an expired oauth token is passed with an API request.
STARK-12388 Resources for Spark are now calculated correctly.
STARK-12168 Metric snapshot performance has been improved by running clean-up tasks more frequently.
STARK-12148 Significantly improved the performance of real-time updates to metrics, particularly when new metrics are added to an entity.
STARK-11971 You can now perform upgrades on installations that are using custom ports.
STARK-11879 The hubcheck script now ensures that disks are correctly shared.
STARK-11841 Upgrades using a non-default user now work correctly.
STARK-11805 The weekly view chart now calculates the time range correctly.
STARK-11692 The historical metric chart now correctly performs Y-axis rounding.
STARK-11689 Historical metric data crossing the Y-axis is now displayed correctly after panning.
STARK-11663 Ingestion Errors link is now styled correctly.
STARK-11609 The Web Console Home page no longer displays dual scrollbars in Firefox.
STARK-11069 The entity filter drop-down list is no longer obscured.
   

Known issues

These are the known issues affecting this release:

Issue key Known issue description

STARK-11411

The Archiver log might show an error the very first time the Archiver runs. The Archiver should heal and run successfully the next time it runs.

STARK-7691

Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name, for example, "test;". To resolve this issue update the dataview column name removing the semicolon.

STARK-6505

Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named "sampleTime", "netprobeTime" or "dataViewId".

STARK-4463

Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

STARK-4350

Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.
   

Gateway Hub 1.7.3

Released: 13 February 2020

Note: Gateway Hub 1.7.3 is a patch release containing a number of fixes for security and deployment issues.

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

Issues fixed

These are the issues we have fixed in this release:

Issue key Release description
STARK-12458 Gateway Hub now correctly calculates the required memory allocation for YARN during installation.
   

Known issues

These are the known issues affecting this release:

Issue key Known issue description

STARK-11411

The Archiver log might show an error the very first time the Archiver runs. The Archiver should heal and run successfully the next time it runs.

STARK-7691

Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name, for example, "test;". To resolve this issue update the dataview column name removing the semicolon.

STARK-6505

Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named "sampleTime", "netprobeTime" or "dataViewId".

STARK-4463

Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

STARK-4350

Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.
   

Gateway Hub 1.7.2

Released: 1 February 2020

Note: Gateway Hub 1.7.2 is a patch release containing a number of fixes for security and deployment issues.

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

Issues fixed

These are the issues we have fixed in this release:

Issue key Release description
STARK-12289 Gateway Hub can now ingest much larger dataviews without having to increase Kafka's maximum message size.
STARK-12283 You can now configure Kafka's maximum message size for persistent entity snapshots. Setting the maximum message size to a higher value than the default of 1 MB may be necessary when working with very large dataviews (in excess of 20,000 cells).
   

Known issues

These are the known issues affecting this release:

Issue key Known issue description

STARK-11411

The Archiver log might show an error the very first time the Archiver runs. The Archiver should heal and run successfully the next time it runs.

STARK-7691

Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name, for example, "test;". To resolve this issue update the dataview column name removing the semicolon.

STARK-6505

Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named "sampleTime", "netprobeTime" or "dataViewId".

STARK-4463

Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

STARK-4350

Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.
   

Gateway Hub 1.7.1

Released: 23 January 2020

Note: Gateway Hub 1.7.1 is a patch release containing a number of fixes for security and deployment issues.

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

Issues fixed

These are the issues we have fixed in this release:

Issue key Release description
STARK-12168 Significantly improved metric snapshot performance by running clean-up tasks more frequently.
STARK-12148 Significantly improved the performance of real-time updates to metrics, particularly when new metrics are added to an entity.
   

Known issues

These are the known issues affecting this release:

Issue key Known issue description

STARK-11411

The Archiver log might show an error the very first time the Archiver runs. The Archiver should heal and run successfully the next time it runs.

STARK-7691

Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name, for example, "test;". To resolve this issue update the dataview column name removing the semicolon.

STARK-6505

Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named "sampleTime", "netprobeTime" or "dataViewId".

STARK-4463

Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

STARK-4350

Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.
   

Gateway Hub 1.7.0

Released: 17 January 2020

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

Highlights

These are the highlights of this release:

  • You can now use Webhooks to publish Geneos events to external services like ticketing systems.

New features and enhancements

These are the new features and enhancements of this release:

Issue key Release description
STARK-11947 You can now configure Gateway Hub TLS/SSL using a PEM file. If a PEM file is not provided, Gateway Hub will default to using self-signed certificates.
STARK-11739 Improved error handling in Gateway Hub dataset generation.
STARK-11647 You can now use Webhooks to publish Geneos events to external services like ticketing systems.
STARK-11638 The hubcheck tool no longer asks for a password twice when creating ssh connections.
STARK-11203 Simplified the controls for the metric history chart.
STARK-11202 Improved date selection controls for the metric history chart.
   

Issues fixed

These are the issues we have fixed in this release:

Issue key Release description
STARK-12073 Gateway Hub now correctly updates the snapshot daemon and Kafka client when upgrading from a configuration using a non-default password for SSL.
STARK-11702 Certificates are no longer stored in the /tmp folder after configuration is completed.
STARK-11635 You can now re-run hubctl setup ssl on a failed ssl setup.
STARK-11447 The hubcheck logs are now generated in the correct location.
   

Known issues

These are the known issues affecting this release:

Issue key Known issue description
STARK-11973 Kafka depends upon Apache Log4j 1.2.17 which has known vulnerabilities.
STARK-11723 BlackDuck scans show a critical risk in Apache Log4j if used for remote logging. Our investigation shows that Gateway Hub and MapR Control System do not use Apache Log4j remote logging and so are not affected by this vulnerability.

STARK-11411

The Archiver log might show an error the very first time the Archiver runs. The Archiver should heal and run successfully the next time it runs.

STARK-7691

Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name, for example, "test;". To resolve this issue update the dataview column name removing the semicolon.

STARK-6505

Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named "sampleTime", "netprobeTime" or "dataViewId".

STARK-4463

Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

STARK-4350

Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.
   

Gateway Hub 1.6.0

Released: 12 December 2019

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

Highlights

These are the highlights of this release:

  • Web Console monitoring UI.
  • Event query API.
  • Grafana data source (Technology Preview).
  • Centralised configuration validation.

Web Console monitoring UI

The Web Console monitoring UI brings a brand new look for real-time monitoring, to supplement Active Console. The following is available in the Web Console:

  • Entities, their statuses and related metrics.
  • The ability to see the historical data for additional context and incident investigation when metric and event history is enabled in Gateway Hub.

The Web Console is available free of charge for all Geneos users. For more information, see Monitoring.

Event query API

Events such as severity, snooze and user assignment data can now be queried using an open API, providing access to the raw data for further analysis of trends and problem entities.

Centralised configuration validation

Updates to centralised configuration files can now be validated against all Gateways dependent on that configuration, in order to avoid updates causing issues remotely. By default, changes that would cause errors in any dependent Gateways are refused.

The API for centralised configuration has been finalised since the early access version, and as a result all Gateways using GA centralised configuration must be version 5.0. For more information, see Centralised gateways user guide.

Breaking changes

These are the breaking changes in this release:

Issue key Release description
STARK-9578 The API for centralised configuration has changed since Early Access. Upgrade to Gateway 5.0 to use centralised configuration with Gateway Hub 1.6 and above.
 

New features and enhancements

These are the new features and enhancements of this release:

Issue key Release description
STARK-10951 Metric and event endpoints now accept the Content type application/x-yaml.
STARK-10830

Basic authentication is refreshed more frequently, reducing the likelihood that the user will need to re-enter their credentials.

STARK-10676 Error messages now contain more detailed information.
STARK-10674 When security is disabled, Gateway Hub responds to GET requests to either of /security and /security/server/roles/mappings passing a Basic Authorization header with a 200 OK response.
STARK-10556 The Web Console now contains a User Profile page, which shows user details such as roles.
STARK-10440 The Web Console now remembers the open or closed state of the sidebar menu between sessions.
STARK-10358 New publishing filters can now be added using drop-down lists.
STARK-9578 Updated the centralised configuration API.
STARK-8703 Changes to centralised configuration resources are now validated centrally.

STARK-8391

STARK-8390

Added KPIs to the home dashboard showing:

  • The number of events raised over time.
  • The number of current entities and their severities.
STARK-7798 Improved the scrollbar style.
 

Issues fixed

These are the issues we have fixed in this release:

Issue key Release description
STARK-11157 The description for CDF metric aggregation is now correct.
STARK-10834 The Web Console no longer supplies an empty diagnostic file.
STARK-10817 Improved the readability of 401 error screens when in dark mode.
STARK-10744 Application buttons are no longer shown on error pages when the user is not authenticated.
STARK-10706 Gateway Hub now correctly handles errors produced by incorrectly configuring the Kerberos settings.
STARK-10701 Gateway Hub now correctly handles an error when attempting to change roles with an expired token.
STARK-10678 Gateway Hub now returns a 400 error when an authorisation token expires.
STARK-10616 The Web Console will now show an error when you provide an invalid security configuration.
STARK-10593 Improved the error message when a user changes a role mapping that would remove their own admin access.
STARK-10539 The security endpoint no longer returns a 400 error when security is disabled.
STARK-9515 Data with a unit of measure assigned (for example, Kilobytes) is no longer published in the canonical unit (for example, Bytes). The values are published in the form originally set by the Gateway.
STARK-9665 Long error messages are now correctly formatted in the system and ingestion error page sidebars.
STARK-6928 The hubctl setup licence command no longer fails silently when a licence file is omitted.
 

Known issues

These are the known issues affecting this release:

Issue key Known issue description

STARK-11411

The Archiver log might show an error the very first time the Archiver runs. The Archiver should heal and run successfully the next time it runs.

STARK-8239

BlackDuck scan shows a Critical risk in log4j if used for remote logging. Our investigation shows that Gateway Hub and MapR do not use log4j remote logging and so are not affected by this vulnerability.

STARK-7691

Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name e.g. "test;". To resolve this issue update the dataview column name removing the semicolon.

STARK-6505

Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named "sampleTime", "netprobeTime" or "dataViewId".

STARK-4463

Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

STARK-4350

Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.
   

Gateway Hub 1.5.4

Released: 13 November, 2019

Note: Gateway Hub 1.5.4 is a patch release containing a number of fixes for security and deployment issues. Versions 1.5.1 — 1.5.3 were only released internally.

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

New features and enhancements

These are the new features and enhancements of this release:

Issue key Release description

STARK-11306

Upgraded to Kafka 2.3.1.

STARK-11226

It is now possible to allow the installer to do a self-installation without needing to use sudo and using a root user.

STARK-11032

Upgraded to jackson-databind library 2.10.
   

Issues fixed

These are the issues we have fixed in this release:

Issue key Release description

STARK-11327

The Node Manager did not start if /tmp had noexec set. This is was fixed by the addition of a new setting in the hubctl JSON configuration: hub.install.java_io_tmpdiz. It should point to a directory that is mounted on a disk that does not have noexec set.

STARK-10046

Fixed an issue caused by a recent security update to systemd, that resulted in multiple copies of Warden and Gateway Hub services to be run and deployment to fail.

   

Known issues

These are the known issues affecting this release:

Issue key Known issue description
STARK-10849 Web Dashboard is not compatible with SSO Agent in Gateway Hub. This will be addressed in the next release.
STARK-10831 Webslinger is not compatible with SSO Agent in Gateway Hub. This will be addressed in the next release.
STARK-10046 Gateway Hub is not compatible with systemd version 219-65 due to an issue with the underlyingMapR platform. A previous version of systemd is required.
STARK-9515 Data with a unit of measure assigned (for example, Kilobytes) is currently published in the canonical unit (for example, Bytes). This means that certain metrics may have a misleading name (for example, KilobytesInQueue).

STARK-7691

Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name e.g. "test;". To resolve this issue update the dataview column name removing the semicolon.

STARK-7596

The SSL certificate install incorrectly uses the default 'hub' user when applying an SSL certificate. To specify a different install user please add the following to the hubctl installation file:

"ansible" : { 
  	"variables" : { 
		"hub_user":
 			"gwh", 
  		"hub_group":
 			"gwh", 
		} 
}

STARK-6505

Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named "sampleTime", "netprobeTime" or "dataViewId".

STARK-4463

Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

STARK-4350

Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.
   

Gateway Hub 1.5.0

Released: 8 October, 2019

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

Highlights

These are the highlights of this release:

  • Anomaly detection rules.
  • Kafka publishing filter.
  • Role-based security.

Anomaly detection rules

One of the major sources of monitoring noise is due to simple thresholds set on metrics that are inherently variable over time. Many business processes follow a fairly predictable pattern, peaking around market open and close, and dormant over the weekends.

Anomaly detection rules provide a better way of monitoring these kinds of metrics. By analysing the history of a metric stored in Gateway Hub, an anomaly detection rule can determine what is normal, alerting only when this normality is deviated from.

For example, an anomaly detection rule can be set to trigger an alert if orders fall outside a certain tolerance for a given time and day of the week, for that specific server, or for all similar servers.

Switching to anomaly detection rules can result in significantly more accurate alerting and substantially reduced events over time for metrics that follow a pattern or cycle. Some metrics are more suited to anomaly detection rules than others.

For more information, see Anomaly Detection.

Kafka publishing filter

In addition to the improved data format, normalisation and schemas published to Kafka from Gateway Hub, filtering options are now available to restrict the data that is published to an external Kafka cluster.

The publishing of metrics and events can now be individually controlled. In addition, publishing can be restricted to certain entities such as where Division = Securities and certain metrics, such as Bloomberg only.

For more information, see Publish to Kafka.

Role based security

This release brings easy-to-configure single sign on (SSO) and role based security to Gateway Hub. With this version, predefined Administrator and Operator roles are available to provide write and read access to Gateway Hub. Gateway Access control is still configured at the Gateway, although this will be centralised in a future version.

Role based security requires integration with Active Directory or LDAP.

For more information, see Configure Single Sign-On (SSO).

Breaking changes

These are the breaking changes in this release:

Issue key Release description

STARK-9970

The UI and SSO endpoints were previously accessed on unique ports (defaulting to 8888 and 9999, respectively). These are now accessed using the same port as the rest of Gateway Hub API calls, defaulting to 8080.

STARK-9969

Before you upgrade to 1.5, save your SSO configuration separately. The configuration will be lost during the upgrade.

Once 1.5 is successfully installed, SSO will need to be configured using the Security section of the Web UI.

   

New features and enhancements

These are the new features and enhancements of this release:

Issue key Release description

STARK-10538

Apache Commons BeanUtils library was updated.

STARK-10452

The user's display name is now shown in the application when SSO is enabled.

STARK-10302

Improved YARN scheduler to allow important jobs to be prioritised correctly.

STARK-10186

YARN logs are now included in diagnostics files.

STARK-10106

The licence is now checked before enabling metric and event storage.
STARK-9970 The SSO Agent and UI are now served by the Gateway Hub API, and the standalone agents have been removed. Note that the UI and SSO API calls are now done using the same port as the rest of the Gateway Hub API.
STARK-9969 The previous standalone SSO and UI agents are removed on upgrade from version 1.4.
STARK-9908 Resources are now more fairly allocated between the various components within Gateway Hub, providing better stability and performance particularly for large queries.
STARK-9560 Dashboard panels are now always displayed while the data is still loading.
STARK-9444 Publishing endpoint was updated to allow message filtering.
STARK-9398 Improved robustness of installer when attempting to request a licence key on a slow internet connection.
STARK-9236 The Web Console is now served on the same port as the REST API.
STARK-9179 Entity change events are now published as a new message type. They indicate if a new entity has been detected, removed, or the attributes of an entity have changed.
STARK-9086 Various improvements were introduced to the Dark theme of the application.
STARK-9039 Published events can now be filtered by event type, such as severity.
STARK-8859 The /admin/retention endpoints now require ISO 8601 duration formats in line with other endpoints.
STARK-8821 The Storage page in the Web Console now allows metric and event storage to be explicitly enabled and disabled.
STARK-8547 React was upgraded to version 16.8.x.
STARK-8540 Various changes were made to Spark and YARN configuration, resulting in better performance and robustness, particularly with smaller clusters.
STARK-8402 The Web Console now defaults to using the browser time zone in the absence of a configured user preference.
STARK-8401 The layout of the errors tables has been improved when viewed on larger monitors.
STARK-8388 The number of system errors in the system is now displayed as a KPI on the home dashboard.
STARK-8258 GET /versions is now available to provide the API versions supported by Gateway Hub.
STARK-8136 Now that the SSO endpoints are hosted on the same port as the rest of the Gateway Hub endpoints, the Open API definition has been unified.
STARK-8135 The Web Console now routes SSO calls via the same port as other Gateway Hub API calls.
STARK-8134 The SSO REST API endpoints are now hosted on the same port as the rest of the Gateway Hub REST API endpoints.
STARK-7742 Material UI was updated to version 3.9.3.
STARK-7512 Kafka publishing can now be filtered by sampler name.
STARK-7511 Kafka publishing can now be filtered by entity query, such as all entities where Division = Securities.
STARK-7510 Kafka publishing can now be filtered by message type, such as metrics or events.
STARK-7233 retainMetricsAndEvents: true/false has been added to the retention API, allowing metric and event history to be explicitly enabled and disabled.
STARK-6828 Administrator and Operator roles are available for assignment to LDAP users and groups.
STARK-6705 Centralised configuration resources can now be locked and unlocked via API.
   

Issues fixed

These are the issues we have fixed in this release:

Issue key Release description
STARK-10330 Metric queries now return data in the same unit of measure as the one displayed in Active Console, rather than the canonical unit of measure.
STARK-10087 Fixed an issue where retention could not be updated after previous invalid input.
STARK-9925 An error is now displayed when a metric query contains one or more invalid metric names.

STARK-9681

Fixed an issue where archiving was getting stuck in m5.2xlarge AWS instances.
STARK-9634 Errors without a message body, for example, a URL with more than 2048 characters, are now handled correctly.
STARK-9612 Configuring SSL using hubctl setup ssl no longer misallocates YARN resources.
STARK-9337 An issue with the archiving process occasionally getting stuck due to a lack of resources is now resolved through better allocation of resources on install.
STARK-9231 Kafka publishing configuration is no longer validated when publishing is disabled.
STARK-9161 Handling of invalid metric query API parameters was improved.
STARK-9138 Uploading a licence is now supported in the Web Console when the system is unlicensed.
STARK-9128 The ingestion errors loading indicator is now correctly positioned inside the table control.
STARK-9063 Fixed colour of Show related errors control in the Errors pages.
STARK-9014 Various fixes were introduced to the Web Console when used in Firefox.
STARK-8664 Repeatedly misconfiguring Kafka publishing in the Web Console no longer gives the impression that the invalid config has been saved successfully.
STARK-8535 Fixed an issue where the error search control only showed tooltip intermittently.
STARK-8510 Invalid publishing configuration no longer causes the Web Console to hang.
STARK-8506 Publishing status bar is now refreshed when configuration is updated.
STARK-8448 Attributes with special characters are now encoded correctly in the URL.
STARK-8432 The ingestion and system errors pages are now fully responsive down to mobile screens.
STARK-8387 Refreshing the Web Console while a page is loading no longer displays an incorrect error screen.
STARK-7672 Archiving is now done more frequently, instead of inside the maintenance window.
   

Known issues

These are the known issues affecting this release:

Issue key Known issue description
STARK-10849 Web Dashboard is not compatible with SSO Agent in Gateway Hub. This will be addressed in the next release.
STARK-10831 Webslinger is not compatible with SSO Agent in Gateway Hub. This will be addressed in the next release.
STARK-10046 Gateway Hub is not compatible with systemd version 219-65 due to an issue with the underlyingMapR platform. A previous version of systemd is required.
STARK-9730 The operating system language must be set to English, otherwise the build fails.
STARK-9515 Data with a unit of measure assigned (for example, Kilobytes) is currently published in the canonical unit (for example, Bytes). This means that certain metrics may have a misleading name (for example, KilobytesInQueue).

STARK-7691

Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name e.g. "test;". To resolve this issue update the dataview column name removing the semicolon.

STARK-7596

The SSL certificate install incorrectly uses the default 'hub' user when applying an SSL certificate. To specify a different install user please add the following to the hubctl installation file:

"ansible" : { 
  	"variables" : { 
		"hub_user":
 			"gwh", 
  		"hub_group":
 			"gwh", 
		} 
}

STARK-6505

Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named "sampleTime", "netprobeTime" or "dataViewId".

STARK-4463

Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

STARK-4350

Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.
   

Gateway Hub 1.4.1

Released: 14 August, 2019

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

New features and enhancements

These are the new features and enhancements of this release:

Issue key Release description

STARK-9778

Gateway Hub can now be installed to custom directories. However, please note that MapR must still be installed in /opt/mapr.

   

Known issues

These are the known issues affecting this release:

Issue key Known issue description
STARK-9730 The operating system language must be set to English, otherwise the build will fail.
STARK-9515 Data with a unit of measure assigned (for example, Kilobytes) is currently published in the canonical unit (for example, Bytes). This means that certain metrics may have a misleading name (for example, KilobytesInQueue).
STARK-9095

Occasionally a container synchronisation issue occurs during an upgrade, resulting in the following error message:

fatal: [ec2-34-247-183-246.eu-west-1.compute.amazonaws.com]: FAILED! => changed=true 
attempts: 30 
cmd: /opt/mapr/server/mrconfig info containers resync local 

If this occurs, resync the containers manually using the following command:

/opt/mapr/server/mrconfig info containers resync local

STARK-7691

Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name e.g. "test;". To resolve this issue update the dataview column name removing the semicolon.

STARK-7596

The SSL certificate install incorrectly uses the default 'hub' user when applying an SSL certificate. To specify a different install user please add the following to the hubctl installation file:

"ansible" : { 
  	"variables" : { 
		"hub_user":
 			"gwh", 
  		"hub_group":
 			"gwh", 
		} 
}

STARK-6505

Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named "sampleTime", "netprobeTime" or "dataViewId".

STARK-4463

Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

STARK-4350

Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.
   

Gateway Hub 1.4.0

Released: 31 July, 2019

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

Highlights

These are the highlights of this release:

  • Improved Gateway integration.
  • Centralised configuration locking.
  • Improved API documentation.
  • Custom install directory for Gateway Hub.

This release is also focused on improving stability and robustness of metric queries. A number of stability issues have been resolved.

Improved Gateway integration

Integration between Gateway and Gateway Hub has been optimised, resulting in almost 25% less disk required for a standard Gateway Hub with metric storage compared to the previous version. Data will also be available in the Gateway Hub significantly faster than before due to the improved processing. These improvements require Gateway version 4.12.x.

Centralised Gateway configuration locking

Centralised configuration files can now be centrally locked, preventing concurrent updates from causing issues. This feature requires Gateway 4.12.x.

For more information, see Centralised Gateways User Guide.

Improved API documentation

API documentation is now published using redoc allowing better navigation and examples.

For more information, see Gateway Hub API documentation.

Custom install directory

You can now perform new installations of Gateway Hub in custom locations, allowing Gateway Hub, the Web Console SSO, and Kafka to be installed in any location.

Note that the install location for MapR cannot be changed from /opt/mapr.

Note: Changing install directories as part of an upgrade is not supported.

New features and enhancements

These are the new features and enhancements of this release:

Issue key Release description

STARK-9518

The maintenance window section has been removed from the Storage screen. Archiving is now done several times per day rather than during a maintenance period.

STARK-9469

Apache Commons Collections library was updated.

STARK-9355

The allocation of resources is now more sensibly calculated, providing more resources to the components that need it, such as Spark and YARN. This will prevent a number of OOM issues seen previously.

STARK-9247

The raw and enriched internal Kafka topics have been replaced with a single merged topic for 4.12 Gateways and above.

   

Issues fixed

These are the issues we have fixed in this release:

Issue key Release description
STARK-9463

Fixed instances where a metric query would fail with the following exception:

java.lang.RuntimeException: org.apache.spark.SparkException:
Job aborted due to stage failure:
Exception while getting task result: scala.MatchError: None (of class scala.None$)
STARK-9337 An issue with the archiving process occasionally getting stuck due to a lack of resources is now resolved through better allocation of resources on install.
STARK-9316 Querying a metric that had previously changed schema no longer returns an empty result set.

STARK-9269

Added retry and wait steps to the install, providing more robustness when installing Gateway Hub over an unreliable connection.

   

Known issues

These are the known issues affecting this release:

Issue key Known issue description
STARK-9730 The operating system language must be set to English, otherwise the build will fail.
STARK-9515 Data with a unit of measure assigned (for example, Kilobytes) is currently published in the canonical unit (for example, Bytes). This means that certain metrics may have a misleading name (for example, KilobytesInQueue).
STARK-9095

Occasionally a container synchronisation issue occurs during an upgrade, resulting in the following error message:

fatal: [ec2-34-247-183-246.eu-west-1.compute.amazonaws.com]: FAILED! => changed=true 
attempts: 30 
cmd: /opt/mapr/server/mrconfig info containers resync local 

If this occurs, resync the containers manually using the following command:

/opt/mapr/server/mrconfig info containers resync local

STARK-7691

Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name e.g. "test;". To resolve this issue update the dataview column name removing the semicolon.

STARK-7596

The SSL certificate install incorrectly uses the default 'hub' user when applying an SSL certificate. To specify a different install user please add the following to the hubctl installation file:

"ansible" : { 
  	"variables" : { 
		"hub_user":
 			"gwh", 
  		"hub_group":
 			"gwh", 
		} 
}

STARK-6505

Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named "sampleTime", "netprobeTime" or "dataViewId".

STARK-4463

Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

STARK-4350

Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.

   

Gateway Hub 1.3.0

Released: 14 June, 2019

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

Highlights

These are the highlights of this release:

  • An open query API for retrieving metric data.
  • A new directory for storing Entity information.
  • Centralised Gateway configuration.

Metric Query API

Metric data can now be retrieved using an open REST API call:

POST /metrics/query

This endpoint can be used to retrieve metric data, grouped by time bucket with a number of built in aggregations such as median, mean, and percentiles, supporting simpler display of large amounts of monitored data. As of version 1.3 this endpoint is tuned for reporting and results can take several minutes, which is a substantial improvement over querying a relational database. Performance will be improved further in a future release.

A list of the aggregation functions is available using the following REST API call:

GET /metrics/aggregations

Entity directory

This release marks the first version of the Entity directory, providing the basis for the next generation of the state tree. The entity directory brings a number of significant advantages over the existing Geneos state tree, including:

  • Support for significantly larger monitored estates
  • Better analysis and retrieval of data across groups of similar servers

The entity store is an important part of our long term goal to simplify the data model, in particular removing Gateways and Netprobes from the XPath hierarchy and moving towards dealing with Entities and Metrics only. This will have a number of benefits, including easier configuration of Self-announcing Netprobes that rebalance across different Gateways frequently, as well as ultimately simplifying the way monitoring is configured in Geneos. Although the existing state tree will remain available for existing setups, over time we anticipate you will choose to switch to the simpler model.

In 1.4, the Entity Directory will also serve as the backbone for the new web-based monitoring UI.

For more information about the improved data model, see Introduction to Entities and Metrics.

Centralised Gateway configuration

Gateway Hub can now store and serve Gateway configuration centrally. This version allows include files to be uploaded to Gateway Hub and used by connected Gateways. This is the first step in the journey to significantly simplify administration of Gateways at scale. More features will be added in the coming release, including centralised validation and locking, as well as administration UI screens in the Web Console.

For more information about this feature, see Centralised Gateways User Guide.

New features and enhancements

These are the new features and enhancements of this release:

Issue key Release description

STARK-8477

JMX instrumentation by various components has now been secured.

STARK-8315

Processing of schema messages from the Gateway has been substantially improved.

STARK-8248

The style for snackbar error and information messages in the Web Console has been updated.

STARK-8020

Web Console has improved handling of concurrency when changing configuration.

STARK-7913

Added a confirmation modal when leaving a Web Console page with errors.

STARK-7832

Improved design for system and ingestion error pages on mobile devices for the Web Console.

STARK-7713

In the Web Console, save buttons are now disabled in the presence of validation errors.

STARK-7628

Improved memory management for various Gateway Hub components.

STARK-7625

The Archiving jobs have been combined into a single Spark job for increased performance.

STARK-7623

A ITRS-Doc-Version HTTP header is required on all PUT endpoints, preventing concurrency issues. GET endpoints now return the current document version.

STARK-7302

Password field types in the Web Console have been improved to allow better password input.

STARK-7250

The documentation link in the About page is now correctly linked to the current Web Console version.

STARK-7228

The Web Console will now preserve the collapsed/expanded status of the sidebar between sessions.

STARK-7182

The status of Kafka publishing is now displayed in the publishing screen in the Web Console.

STARK-7132

Web Console now handles an unlicensed system better.

STARK-7074

Ports are now configurable for Kafka, REST API, Web Console, SSO API, and MCS monitoring UI.

STARK-6877

DD MMM YYYY has been added to the list of supported date formats in the Web Console.

STARK-6688

MapR Enterprise Components (MEP) have been upgraded to 6.1.

STARK-6686

The MapR instance underpinning Gateway Hub has been upgraded to 6.1.

STARK-6526

System errors are no longer displayed alongside ingestion errors and have their own endpoint and Web Console screen instead.

STARK-6525

The Kafka offsets have been removed from ingestion errors and replaced with the raw message, so that older errors did not become outdated by Kafka rebalances and retention settings.

STARK-6520

The ingestion errors pages in the Web Console have been reworked for significantly better performance and robustness. Outdated links to old errors that have been resolved are now handled correctly.
   

Issues fixed

These are the issues we have fixed in this release:

Issue key Release description
STARK-8991 Dataview columns and headline cells with a full stop/period in the name no longer cause an error when queried for metrics.
STARK-8931 Dataview columns with backticks (`) in the name no longer cause the archiver to crash.
STARK-8668 Updating a metric's type no longer causes the metric to be marked as deleted.

STARK-7617

Various improvements to the way Zookeeper is configured to improve stability.

STARK-7585

The /dataview endpoint has been updated to include a path parameter, allowing filtering by partial dataview path.

STARK-7227

The Web Console sidebar will no longer get stuck when collapsed under certain circumstances.

STARK-7174

Special characters (for example, ñ) are now handled correctly in Licence page of the Web Console.

STARK-7116

When publishing is disabled, schema and snapshot requests are no longer attempted.

STARK-6726

Column name escaping has been improved, preventing crashes when querying data from certain plug-ins.
   

Known issues

These are the known issues affecting this release:

Issue key Known issue description
STARK-9095

Occasionally a container synchronisation issue occurs during an upgrade, resulting in the following error message:

fatal: [ec2-34-247-183-246.eu-west-1.compute.amazonaws.com]: FAILED! => changed=true 
attempts: 30 
cmd: /opt/mapr/server/mrconfig info containers resync local 

If this occurs, resync the containers manually using the following command:

/opt/mapr/server/mrconfig info containers resync local

STARK-7691

Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name e.g. "test;". To resolve this issue update the dataview column name removing the semicolon.

STARK-7596

The SSL certificate install incorrectly uses the default 'hub' user when applying an SSL certificate. To specify a different install user please add the following to the hubctl installation file:

"ansible" : { 
  	"variables" : { 
		"hub_user":
 			"gwh", 
  		"hub_group":
 			"gwh", 
		} 
}

STARK-6505

Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named "sampleTime", "netprobeTime" or "dataViewId".

STARK-4463

Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

STARK-4350

Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.
   

Gateway Hub 1.2.0

Released: 11 March, 2019

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

Highlights

These are the highlights of this release:

  • Publishing of normalised data with schema to an external Kafka interface.
  • Additional administration screens for the Web Console.

Kafka publishing

The Kafka Publishing available with Gateway Hub builds upon the feature previously available in the Gateway. With Gateway Hub, schemas can now be published along with the metric and event data, in order to provide easier integration with other applications, and the published data is now normalised, enabling simpler usage downstream.

For more information, see Kafka Publishing Message Formats.

Web Console

This release brings the new administration screens for the Web Console, allowing licence management and configuration of Publishing.

For more information, see Web Console Administration Pages.

Breaking changes

These are the breaking changes in this release:

Issue key Breaking change description
STARK-6346

Corrects the spelling of the "licence" keyword in the installation json file to UK English. Update the previous installation files to reflect this change.

   

New features and enhancements

These are the new features and enhancements of this release:

Issue key Release description
STARK-6860 When the Gateway Hub is not licensed, a placeholder appears in the Web Console.
STARK-6733 Dataview schemas can be requested to be published to Kafka using a REST command to Gateway Hub.
STARK-6732 Dataview snapshots can be requested to be published to Kafka using a REST command to Gateway Hub.
STARK-6619 Updates Web Console favicon.
STARK-6361 Gateway Hub can be used without saving metric and event data to disk.
STARK-6346 Standardises the use of the noun "licence" to UK English.
STARK-5983 Gateway Hub licences can be uploaded directly in the Web Console.
STARK-5980 Publishing to Kafka can be configured using a new Web Console screen.
STARK-5913 Sets the cluster name to hub.itrsgroup.com.
STARK-5613 Nodes can be added to the cluster.
STARK-4749 Dates in the Web Console, such as 20 September 2018, are now localised correctly.
STARK-4403 Allows you to navigate Web Console UI entirely by the keyboard.
STARK-4308 Upgrades Kafka to 2.0.
STARK-3270 A screen showing the current installed version of Gateway Hub is available, along with a control to download the diagnostic info if required for a support ticket.
STARK-3269 A new Web Console page for viewing and uploading Gateway Hub licences.
   

Issues fixed

These are the issues we have fixed in this release:

Issue key Release description
STARK-6431 The installation script validates that the runtime user is assigned to the correct group before the installation starts.
STARK-6423 The installation script validates that the install user is configured correctly before the installation starts.
STARK-5922 Changing the units of measures no longer prevents ingestion errors from being returned.
   

Known issues

These are the known issues affecting this release:

Issue key Known issue description
STARK-7368

While performing an upgrade from 1.1 to 1.2, there may be some errors in the upgrade log. This is due to a change in the message format and will not affect metric ingestion. The error messages can be ignored.

STARK-7118 If a snapshot is requested from Gateway with an unrecognised snapshot type, then the request is accepted by Gateway Hub but ignored by the Gateway. The error appears in the Gateway log.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
   

Gateway Hub 1.1.0

Released: 21 January, 2019

This release delivers the updated binaries for Gateway Hub in the ITRS Downloads page.

Highlights

These are the highlights of this release:

  • Ability to store historical data for all Geneos metrics and events.
  • A brand new web-based UI for Gateway Hub Administration.

Historical metric and event storage

Historical metric and event data can now be stored in Gateway Hub and retrieved using Active Console. This is significantly more reliable than database logging and is designed to handle all metrics being logged, rather than a subset.

Metric and event storage in Gateway Hub also provides the foundation for improved monitoring over the next few releases, including adaptive rules (anomaly detection), automatic baselining, predictive and causal analysis through machine learning.

For more information, see How to view metrics in Active Console.

Web Console

his release brings the first release of the new web-based UI for Geneos, the Web Console, focusing on administration of Gateway Hub. The Web Console will be expanded upon in future releases to initially complement, and eventually replace, Active Console.

For more information, see GeneosWeb Console.

Breaking changes

Note: These changes are applicable to Early Access customers only.

These are the breaking changes in this release:

Issue key Breaking change description
STARK-5233 The unit of measure endpoints have been simplified significantly.
STARK-3441 Dataview was previously written using incorrect case (e.g. "dataView") in the REST endpoints. These have been corrected.
STARK-1566

Publishing to Gateway Hub is done using a secured connection.

All publishing Gateway must be reconfigured to use SSL encryption as unsecured connections are no longer supported.

   

New features and enhancements

These are the new features and enhancements of this release:

Issue key Release description
STARK-6565 Bundled SSO Agent upgrades to version 1.3.0.
STARK-6538 Increases message throughput due to improved MapR-DB batching strategy.
STARK-6344 Gateway Hub configuration can be updated through the command line.
STARK-6326 Kafka disk location can be specified as part of the installation parameters.
STARK-6316 Better validation when adding units of measure without a locale or text.
STARK-6191 Perfmon errors are correctly converted to null.
STARK-6167 Keystore password is configurable.
STARK-6007 Error codes are updated to make them unique.
STARK-5896 The kernel settings vm.swappiness, vm.overcommit.memory, and net.ipv4.tcp_retries2 are no longer modified as part of the install and must be managed separately as a prerequisite.
STARK-5768 Removes the need to store runtime user credentials for normaliserd.
STARK-5707 Dataview errors can be filtered by error code.
STARK-5591 Ingestion errors can be filtered by partial or full dataview path.
STARK-5582 Adds filter and skip parameters to ingestion errors endpoint to allow paging.
STARK-5541 Truncation of decimal data when attempting to save as an int generates ingestion errors.
STARK-4834 Passwordless SSH is no longer required. During installation, you can interactively enter the password.
STARK-4547 The installation is done using an offline package. Gateway Hub nodes do not longer require internet access.
STARK-4545 After installation, you can enter a custom SSL certificate.
STARK-3637 Web Console is packaged with Gateway Hub.
STARK-3020 Retention policies are now specified for metric and events.
   

Issues fixed

These are the issues we have fixed in this release:

Issue key Release description
STARK-6736 Error messages with missing parameters can cause the error endpoints to fail.
STARK-6486 Zookeeper status is not reset correctly during factory reset.
STARK-6480 Logs entries are stored in UTC.
STARK-6290 Validation errors updated with unique titles.
STARK-6287 Better validation when uploading invalid licences.
STARK-6246 Duplicates unrecognisedField error message suppressed.
STARK-6237 Adds command line flag for generating faster diagnostic files.
STARK-6042 Resolves race condition causing "Unhandled Error: Failed to create table" log entries.
STARK-6039 Processing error messages no longer significantly slows down data ingestion.
STARK-6005 Systems with umask set to 077 no longer cause issues during installation.
STARK-5886 Simplifies user model to avoid permission issues in persistence.
STARK-5839 Fixes to skip and limit parameters for errors endpoint.
STARK-5783 Changes to Kafka ISR settings significantly improve resilience.
STARK-5751 Ingestion errors are no longer aggregated by dataview row name.
STARK-5283 Dataview queries return the most recent results first. If a limit is specified, the least recent results may be truncated.
STARK-5203 If a dataview had a computed row that was later removed, then the Gateway will now correctly report hasEnrichedRows as false.
STARK-5008 Changes to the casing of certain error parameters means that old error results will be cleared. These are repopulated as new errors are received and will not be an issue.
STARK-4965 Fixes an issue causing the error endpoint to return no data.
STARK-4957 Log files roll over correctly.
STARK-4517 Restarting Gateway Hub no longer causes Kafka to become corrupted under certain circumstances.
STARK-3962 Moves Kafka logs to the correct log location.
   

Known issues

These are the known issues affecting this release:

Issue key Known issue description
STARK-7239 4.10 Gateways are not able to publish to Gateway Hub 1.1. This will be resolved as a matter of priority.
STARK-6975 Immediately after an upgrade from 1.0.3 to 1.1, the dataview query API returns empty results. This issue is resolved automatically after a few minutes and normal results will be returned.
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named sampleTime, netprobeTime, or dataViewId.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
   

Other Gateway Hub releases

Gateway Hub 2.x Release Notes Released: 8 April 2020
Gateway Hub 1.x Release Notes Released: 21 January 2019

Disclaimer: The information contained in this document is for general information and guidance on our products, services, and other matters. It is only for information purposes and is not intended as advice which should be relied upon. We try to ensure that the content of this document is accurate and up-to-date, but this cannot be guaranteed. Changes may be made to our products, services, and other matters which are not noted or recorded herein. All liability for loss and damage arising from reliance on this document is excluded (except where death or personal injury arises from our negligence or loss or damage arises from any fraud on our part).