Fluentd Forward

Overview Copied

Fluentd is an open-source application used for unifying log collection and aggregation. Fluentd supports the Fluentd Forward protocol for inbound and outbound routing of events between other Fluentd instances.

The Fluentd forward plugin provides a collector that supports the inbound Fluentd Forward protocol messages and allows Geneos to receive Fluentd unified logs for visualisation, analysis, and storage.

Compatibility with Fluentd forward protocol specification Copied

The protocol specifies many options and only some are currently supported. The following section provides an overview of what is currently supported and not supported:

Prerequisites Copied

Geneos environment Copied

The latest version of the Fluentd plugin requires Gateway and Netprobe version 6.1.x or higher. The same version must be used for the GSE schema.

The Fluentd Forward plugin binaries are packaged with Netprobe 6.1.x and are stored in the collection_agent folder.

Alternatively, you can download separate binaries for the Fluentd plugin from the ITRS Downloads.

Configure Geneos to deploy the Fluentd plugin Copied

The Fluentd plugin supports Collection Agent publishing into Geneos using dynamic Managed Entities. To set up the Fluentd plugin in Geneos, follow these steps:

1. Set up your plugin either by:

• Setting up your collector in the Gateway Setup Editor by adding the following configuration in Managed Entities > Collectors. For more information, see Collectors in Dynamic Entities.
• Adding the following configuration in collection-agent.yml file on your local machine where the binaries are stored.

collectors:
 - type: plugin
   class-name: FluentdForwardCollector
   
   # Required. Port on which to receive TCP (+ TLS/SSL) messages and UDP heartbeats.
   port: 24224

   # Acceptor thread pool size (default = 1)
   acceptor-thread-pool-size: 1

   # Worker thread pool size (default = 1)
   worker-thread-pool-size: 1
   
   # Optional TLS configuration.
   tls-config:
     # Required when tls-config is configured.
     # Server key - required to enable TLS
     key-file: /path/to/private_key.pem
     # Required when tls-config is configured.
     # Server certificate - required to enable TLS.
     cert-file: /path/to/cert_file.pem
     # Client trust chain - only required for client authentication.
     trust-chain-file: /path/to/trust_chain.pem

   # Required. Name of event data key to use as the log event name.
   name-key: ident
   
   # Optional tag mapping.
   # Every batch of messages contains a tag which may be mapped into log events.
   # Values:
   #   - none (default): ignored the tag
   #   - namespace: tag value is appended to the log event namespace
   #   - name: tag value is prepended to the log event name
   #   - dimension: tag value is added as a dimension
   tag-mapping: none
   
   # Optional. Name of event data key to use to extract the log event message.
   # Default is 'message'.
   message-key: message
   
   # Optional. Event data keys to use as dimension keys.
   # The key here is used as the search key in the event data and value (if any)
   # is inserted as a dimension with key mapped as specified.
   dimensions:
     # If event data contains {host=www.myhost.com} then it is mapped to a dimension {host.name=www.myhost.com}.
     host: host.name
     pid: pid.number
     
   # Optional. Event data key and optional mappings to extract severity from the event data.  
   severity-mapping:
     # Event data key to use to extract severity value.
     key: level
     # Optional severity mapping value in case source system uses different values.
     # Supported target values are: [none, trace, debug, info, warn, error, critical]
     values:
       fatal: critical

2. Configure your custom mappings in the Dynamic entities > Mappings setting in the Gateway Setup Editor. For more information, see Mapping and mapping group in Dynamic Entities. Below is an example of a custom mapping for Fluentd.

3. Configure your other Dynamic Entities in the Gateway. See Create Dynamic Entities in Collection Agent setup for a more detailed procedure.

Alternatively, you can use Self-announcing Netprobes to enable dynamic entities. See Self-announcing Netprobe in Collection Agent setup.

Note

To check if there are any errors in the mapping, you can set up the Dynamic Entities Health, or look at the Collection Agent log file in Collection Agent setup.

Example dataview Copied

Below is a sample dataview using the custom mappings above.

To define how items in the Geneos tree structure are generated from the labels of a datapoint, see Geneos items in Dynamic Entities.