Fluentd Forward
Overview Copied
Fluentd is an open-source application used for unifying log collection and aggregation. Fluentd supports the Fluentd Forward protocol for inbound and outbound routing of events between other Fluentd instances.
The Fluentd forward plugin provides a collector that supports the inbound Fluentd Forward protocol messages and allows Geneos to receive Fluentd unified logs for visualisation, analysis, and storage.
Compatibility with Fluentd forward protocol specification Copied
The protocol specifies many options and only some are currently supported. The following section provides an overview of what is currently supported and not supported:
Protocol | Supported | Not supported |
---|---|---|
Heartbeat Message | UDP heartbeats on the same port as the TCP listener | TCP heartbeats |
Network Transport Protocol | Plain TCP and TLS/SSL | |
Connection Phases | Transport — the server goes directly to this phase and ignores handshakes from the client. | Handshake — including associated message exchanges, such as authentication. |
Event Modes | All MessagePack modes, such as:
|
JSON payloads |
Timestamp | EventTime format and Integer format | |
Acks | Acks via the chunk option |
Prerequisites Copied
Geneos environment Copied
The latest version of the Fluentd plugin requires Gateway and Netprobe version 6.1.x or higher. The same version must be used for the GSE schema.
The Fluentd Forward plugin binaries are packaged with Netprobe 6.1.x and are stored in the collection_agent
folder.
Alternatively, you can download separate binaries for the Fluentd plugin from the ITRS Downloads.
Configure Geneos to deploy the Fluentd plugin Copied
The Fluentd plugin supports Collection Agent publishing into Geneos using dynamic Managed Entities. To set up the Fluentd plugin in Geneos, follow these steps:
- Set up your plugin either by:
- Setting up your collector in the Gateway Setup Editor by adding the following configuration in Managed Entities > Collectors. For more information, see Collectors in Dynamic Entities.
- Adding the following configuration in
collection-agent.yml
file on your local machine where the binaries are stored.
collectors:
- type: plugin
className: FluentdForwardCollector
# Required. Port on which to receive TCP (+ TLS/SSL) messages and UDP heartbeats.
port: 24224
# Acceptor thread pool size (default = 1)
acceptorThreadPoolSize: 1
# Worker thread pool size (default = 1)
workerThreadPoolSize: 1
# Optional TLS configuration.
tlsConfig:
# Required when tlsConfig is configured.
# Server key - required to enable TLS
keyFile: /path/to/private_key.pem
# Required when tlsConfig is configured.
# Server certificate - required to enable TLS.
certFile: /path/to/cert_file.pem
# Client trust chain - only required for client authentication.
trustChainFile: /path/to/trust_chain.pem
# Required. Name of event data key to use as the log event name.
nameKey: ident
# Optional tag mapping.
# Every batch of messages contains a tag which may be mapped into log events.
# Values:
# - none (default): ignored the tag
# - namespace: tag value is appended to the log event namespace
# - name: tag value is prepended to the log event name
# - dimension: tag value is added as a dimension
tagMapping: none
# Optional. Name of event data key to use to extract the log event message.
# Default is 'message'.
messageKey: message
# Optional. Event data keys to use as dimension keys.
# The key here is used as the search key in the event data and value (if any)
# is inserted as a dimension with key mapped as specified.
dimensions:
# If event data contains {host=www.myhost.com} then it is mapped to a dimension {host.name=www.myhost.com}.
host: host.name
pid: pid.number
# Optional. Event data key and optional mappings to extract severity from the event data.
severityMapping:
# Event data key to use to extract severity value.
key: level
# Optional severity mapping value in case source system uses different values.
# Supported target values are: [none, trace, debug, info, warn, error, critical]
values:
fatal: critical
- Configure your custom mappings in the Dynamic entities > Mappings setting in the Gateway Setup Editor. For more information, see Mapping and mapping group in Dynamic Entities. Below is an example of a custom mapping for Fluentd.
3. Configure your other Dynamic Entities in the Gateway. See Create Dynamic Entities in Collection Agent setup for a more detailed procedure.
Alternatively, you can use Self-announcing Netprobes to enable dynamic entities. See Self-announcing Netprobe in Collection Agent setup.
Note
To check if there are any errors in the mapping, you can set up the Dynamic Entities Health, or look at the Collection Agent log file in Collection Agent setup.
Example dataview Copied
Below is a sample dataview using the custom mappings above.
To define how items in the Geneos tree structure are generated from the labels of a datapoint, see Geneos items in Dynamic Entities.