Fluentd Forward

Overview

Fluentd is an open-source application used for unifying log collection and aggregation. Fluentd supports the Fluentd Forward protocol for inbound and outbound routing of events between other Fluentd instances.

The Fluentd Forward plugin provides a collector that supports the inbound Fluentd Forward protocol messages and allows Geneos to receive Fluentd Forward unified logs for visualisation, analysis, and storage.

Compatibility with Fluentd Forward protocol specification

The protocol specifies many options and only some are currently supported. The following section provides an overview of what is currently supported and not supported:

Protocol	Supported	Not supported
Heartbeat Message	UDP heartbeats on the same port as the TCP listener	TCP heartbeats
Network Transport Protocol	Plain TCP TLS/SSL
Connection Phases	Transport — the server goes directly to this phase and ignores handshakes from the client.	Handshake — including associated message exchanges, such as authentication.
Event Modes	All MessagePack modes, such as: Message mode Forward mode PackedForward mode CompressedPackedForward mode	JSON payloads
Timestamp	EventTime format Integer format
Acks	Acks via the chunk option

Prerequisites

Geneos environment

The latest version of the Fluentd Forward plugin requires Gateway and Netprobe version 6.1.x or higher. The same version must be used for the GSE schema.

The Fluentd Forward plugin binaries are packaged with Netprobe 6.1.x and are stored in the collection_agent folder.

Alternatively, you can download separate binaries for the Fluentd Forward plugin from the ITRS Downloads.

Configure Geneos to deploy the Fluentd Forward plugin

The Fluentd Forward plugin supports Collection Agent publishing into Geneos using dynamic Managed Entities. To set up the Fluentd Forward plugin in Geneos, follow these steps:

1. Set up your plugin either by:
   • Setting up your collector in the Gateway Setup Editor by adding the following configuration in Managed Entities > Collectors. For more information, see Collectors in Dynamic Entities.
   • Adding the following configuration in collection-agent.yml file on your local machine where the binaries are stored.

   collectors:
   - type: plugin
       class-name: FluentdForwardCollector
   
       # Required. Port on which to receive TCP (+ TLS/SSL) messages and UDP heartbeats.
       port: 24224
   
       # Acceptor thread pool size (default = 1)
       acceptor-thread-pool-size: 1
   
       # Worker thread pool size (default = 1)
       worker-thread-pool-size: 1
   
       # Optional TLS configuration.
       tls-config:
       # Required when tls-config is configured.
       # Server key - required to enable TLS
       key-file: /path/to/private_key.pem
       # Required when tls-config is configured.
       # Server certificate - required to enable TLS.
       cert-file: /path/to/cert_file.pem
       # Client trust chain - only required for client authentication.
       trust-chain-file: /path/to/trust_chain.pem
   
       # Required. Name of event data key to use as the log event name.
       name-key: ident
   
       # Optional tag mapping.
       # Every batch of messages contains a tag which may be mapped into log events.
       # Values:
       #   - none (default): ignored the tag
       #   - namespace: tag value is appended to the log event namespace
       #   - name: tag value is prepended to the log event name
       #   - dimension: tag value is added as a dimension
       tag-mapping: none
   
       # Optional. Name of event data key to use to extract the log event message.
       # Default is 'message'.
       message-key: message
   
       # Optional. Event data keys to use as dimension keys.
       # The key here is used as the search key in the event data and value (if any)
       # is inserted as a dimension with key mapped as specified.
       dimensions:
       # If event data contains {host=www.myhost.com} then it is mapped to a dimension {host.name=www.myhost.com}.
       host: host.name
       pid: pid.number
   
       # Optional. Event data key and optional mappings to extract severity from the event data.  
       severity-mapping:
       # Event data key to use to extract severity value.
       key: level
       # Optional severity mapping value in case source system uses different values.
       # Supported target values are: [none, trace, debug, info, warn, error, critical]
       values:
           fatal: critical
   

2. Configure your custom mappings in the Dynamic entities > Mappings setting in the Gateway Setup Editor. For more information, see Mapping and mapping group in Dynamic Entities. Below is an example of a custom mapping for Fluentd Forward.

3. Configure your other Dynamic Entities in the Gateway. See Create Dynamic Entities in Collection Agent setup for a more detailed procedure.

Alternatively, you can use Self-announcing Netprobes to enable dynamic entities. See Self-announcing Netprobe in Collection Agent setup.

Note

To check if there are any errors in the mapping, you can set up the Dynamic Entities Health, or look at the Collection Agent log file in Collection Agent setup.

Example dataview

Below is a sample dataview using the custom mappings above.

To define how items in the Geneos tree structure are generated from the labels of a datapoint, see Geneos items in Dynamic Entities.