Publish to Kafka

Overview

This topic explains how to configure Gateway Hub to publish to a downstream Kafka instance as a Kafka "producer client". The messages are published in JSON format.

Publishing from Gateway Hub has advantages over publishing directly from the Gateway, because Gateway Hub outputs normalised metric and event data. Only one downstream Kafka instance is supported by a Gateway Hub cluster.

Publishing to a downstream Kafka instance is enabled using the Web Console.

For an introduction to the Web Console, see Geneos Web Console.

For an explanation of the specific fields for Publishing in the Web Console, see Publishing.

For the message formats published by Gateway Hub to Kafka, see Kafka Publishing Message Formats.

Prerequisites

Geneos components

Your Gateway Hub must be connected to a Geneos Gateway v4.10 to use publishing.

Kafka

Gateway Hub can only publish to topics that already exist on your downstream Kafka instance. Therefore, you must create the following topics, where ${prefix} is the topic prefix configured in the Publishing section of the Web Console:

  • ${prefix}metrics
  • ${prefix}events

Note: The default prefix is itrs-.

Publishing from Gateway Hub has been tested with downstream Apache Kafka instances v2.0.0 and later.

Use Kerberos to connect to Kafka

If you want to use Kerberos to connect to Kafka, you must obtain:

  • The Kerberos principal used to connect to Kafka.
  • The keytab file encoding the password for the Kerberos principal.

On every Gateway Hub node you must also update the Kerberos configuration file /etc/krb5.conf to contain the correct configuration information for your Kerberos domain. In particular, the [realms] section must contain the correct Kerberos server information.

If you are using SSL encryption together with Kerberos authentication, you must also obtain the CA certificate used to sign the Kafka broker's public keys.

Enable Kafka publishing

To enable publishing to a Kafka instance:

  1. Access your Web Console using your browser.
  2. Go to the Administration > Publishing page.
    You are presented with the following page:
  3. Click the toggle to the right of Publishing.
  4. In Bootstrap servers, enter the host:port value for your downstream Kafka instance. More rows can be added using the add button.
  5. Modify the Topic Prefix if required.
  6. Add any additional settings in Producer configuration name. Add the corresponding value in Producer configuration value.
  7. Choose the Security protocol using the drop-down list.
    • If using SASL_SSL:
      1. Upload the CA certificate using the Upload File button.
      2. In Kerberos principal, add the Kerberos principal name.
      3. In Kerberos keytab, add the keytab corresponding to the Kerberos principal.
  8. Review the settings.
    An example of a finished setup using SASL_SSL is below:
  9. Click Save Changes.
    The Confirm changes dialog appears:
  10. Click Save.

Status

If you configure Kafka publishing, an information bar is displayed at the top of this configuration screen providing the current publishing status. Important information such as the inability to connect to the configured Kafka instance is displayed here.

Note: Retrieving the status from a remote Kafka instance can take 30+ seconds depending on various factors such as the connection speed, the relative location, the processing capability, and more

The possible statuses are as follows:

  • Configuration error
  • Waiting
  • Success

Filtering

The filtering section of the configuration allows you to publish a smaller subset of the data. This can considerably reduce the storage and processing requirements of a downstream Kafka instance. You can use one or more filter predicates to filter the data by message type and entity query. A record is published if it meets any of the includes conditions. However, a record is not published if it meets any of the excludes conditions even if it also meets an includes condition.

For example, given the configuration below, all metrics and events will be published for Entities where Application = Fidessa except where Department = Fixed Income

Include/Exclude Message type Entities
Include Events Application = Fidessa
Include Metrics Application = Fidessa
Exclude All Department = 'Fixed Income'

You can specify filters on entities using the basic or advance menus:

  • The basic menu allows you to select entities by specifying an attribute and a value.
  • The advanced menu allows you to input a filter manually.