Geneos ["Geneos"]
["Geneos > Netprobe"]["User Guide"]

OpenTelemetry

Overview

The OpenTelemetry plugin is a Collection Agent plugin that provides data collection capability for the OpenTelemetry protocol over gRPC/protobuf.

The OpenTelemetry plugin requires an instrumented system that emits signals. The signals are the following telemetry categories that are supported by the OpenTelemetry specifications:

  • Traces — describes the flow of a single request made by a user or an application.

  • Metrics — a numeric value that measures system resources or services over a time interval.

  • Logs — records an event, including timestamps and metadata.

Prerequisites

Geneos environment

The latest version of the OpenTelemetry plugin requires Gateway and Netprobe version 6.0.x. The same version must be used for the GSE schema.

The OpenTelemetry plugin binaries are packaged with Netprobe 6.0.x and are stored in the collection_agent/plugins folder.

Alternatively, you can download separate binaries for the OpenTelemetry plugin from the ITRS Downloads.

OpenTelemetry environment

The OpenTelemetry plugin requires an instrumented system that emits signals. Please see the OpenTelemetry documentation for language-specific instrumentation details.

Protocol and Security

Protocol

The OpenTelemetry plugin supports the following requests:

  • Sequential — recommended where possible.

  • Concurrent — recommended over slow networks and/or high data volumes.

If the server responds with an OK status, then all records in the batch are ingested to persistent storage and will never require retransmission. There will be no retransmission either because they are already persisted or are considered non-critical.

The subset of gRPC error codes used are described in the table below:

gRPC code Retyrable Description
UNAVAILABLE Yes

An ingestion error has occurred and a retry is likely to succeed after a delay.

A RetryInfo record is also sent as additional details. The retry delay should be regarded as indicative of the initial delay period and clients should respect the suggested exponential backoff mechanism as per the protocol specification.

Caution: In some circumstances, gRPC emits UNAVAILABLE due to transport-related issues, such as bad TLS/SSL handshake. This is a non-retryable error but is outside the control of this plugin.

INVALID_ARGUMENT No An unknown error has occurred, and a retry of the same batch is unlikely to succeed and should not be attempted.
UNAUTHENTICATED No When an application authentication is enabled, and a client fails to authenticate.
     

Using the above mechanisms, the plugin attempts to guarantee at least once semantics. It is possible that retrying a batch will result in 1 or more individual records being ingested multiple times. You may minimize this possibility by sending records sequentially, one per batch. However, this is likely to be far less efficient from a network bandwidth and throughput perspective.

Security

This plugin supports multiple layers of security:

  • Transport layer — various forms of TLS.

  • Application layer — application keys in the form of username and password.

Specifically, the following transport layer security modes are supported:

  • Plain text

  • Untrusted TLS — server trusts any client.

  • Mutual TLS — server only trusts clients which present an appropriate certificate.

By default, when TLS is enabled, TLSv1.3 and TLSv1.2 are the only protocols allowed. It is possible to explicitly enable others, but they are generally considered less secure.

In addition to TLS, this plugin supports application layer authentication against a suitable IAM provider. In this case, aside from transport layer security, you must provide appropriate keys in the gRPC metadata, which can be used for authentication purposes.

Configure Gateway to receive OpenTelemetry data

The OpenTelemetry plugin supports Collection Agent publication into Geneos using dynamic managed entities. To set up the OpenTelemetry plugin in Geneos, follow these steps:

  1. In GSE, go to Dynamic entitiesCollectors then create an OpenTelemetry collector with the following configuration:

      
                                          

    Note: OpenTelemetry units may be translated to any available internal units, either by symbol or description. For more information, see Metric Unit Definitions.

  2. In Dynamic entitiesMapping, configure the mappings to be used. For an example mapping, see Mappings.

  3. If you will be monitoring logs, set up an FKM sampler. For more information, see File Keyword Monitor Configuration.

  4. In Dynamic entitiesMapping types, create a mapping type with the collector, mapping, and sampler you created in the previous steps.

Note: To check if there are any errors in the mapping, you can set up the Dynamic Entities Health or look at the Collection Agent log file in Collection Agent setup.

Mappings

The data mappings of the signals supported by the OpenTelemetry plugin are as follow:

  • Metrics — fully supported except for the deprecated types. All others are mapped to the internal data model in some shape.

  • Logs — fully supported.

  • Traces — partially supported. There is no internal data model representation for span yet, so golden metrics are extracted from each span.

OpenTelemetry plugin allows you to create custom mappings, so you can modify the display of information in the Geneos dataview. The mappings can be configured differently by using available dimensions or properties.

To create custom mappings for OpenTelemetry:

  1. Navigate to your OpenTelemetry mapping in Dynamic entities > Mappings.

  2. In Options, select custom.

Example mappings and dataviews

Traces
<mapping name="opentelemetry-traces"> 
	<custom> 
		<filters> 
			<filter> 
				<label>__itrs_namespace__</label> 
				<equals> 
					<value>itrsgroup.com/c2/opentelemetry-plugin/traces</value> 
				</equals> 
			</filter> 
		</filters> 
		<geneosItems> 
			<geneosItem> 
				<label>trace_id_hex</label> 
				<entity> 
					<required>true</required> 
					<useInDisplayName>true</useInDisplayName> 
				</entity> 
			</geneosItem> 
		</geneosItems> 
	</custom> 
</mapping>

Using the custom mappings above, the following dataview is generated:

Metrics
<mapping name="opentelemetry-metrics"> 
	<custom> 
		<filters> 
			<filter> 
				<label>__itrs_namespace__</label> 
				<equals> 
					<value>itrsgroup.com/c2/opentelemetry-plugin/traces</value> 
				</equals> 
			</filter> 
		</filters> 
		<geneosItems> 
			<geneosItem> 
				<label>service.name</label> 
				<entity> 
					<required>true</required> 
					<useInDisplayName>true</useInDisplayName> 
				</entity>
			 </geneosItem> 
		</geneosItems> 
	</custom> 
</mapping>

Using the custom mappings above, the following dataview is generated:

Logs
<mapping name="opentelemetry-logs"> 
	<custom> 
		<filters> 
			<filter> 
				<label>__itrs_namespace__</label> 
				<equals> 
					<value>itrsgroup.com/c2/opentelemetry-plugin/traces</value> 
				</equals> 
			</filter> 
		</filters> 
		<geneosItems> 
			<geneosItem> 
				<label>service.name</label> 
				<entity> 
					<required>true</required> 
					<useInDisplayName>true</useInDisplayName> 
				</entity>
			 </geneosItem> 
		</geneosItems> 
	</custom> 
</mapping>

Using the custom mappings above, the following dataview is generated: