OpenTelemetry

Overview Copied

The OpenTelemetry plugin is a Collection Agent plugin that provides data collection capability for the OpenTelemetry protocol over gRPC/protobuf.

The OpenTelemetry plugin requires an instrumented system that emits signals. The signals are the following telemetry categories that are supported by the OpenTelemetry specifications:

• Traces — describes the flow of a single request made by a user or an application.
• Metrics — a numeric value that measures system resources or services over a time interval.
• Logs — records an event, including timestamps and metadata.

Prerequisites Copied

Geneos environment Copied

The latest version of the OpenTelemetry plugin requires Gateway and Netprobe version 6.0.x. The same version must be used for the GSE schema.

The OpenTelemetry plugin binaries are packaged with Netprobe 6.0.x and are stored in the collection_agent/plugins folder.

Alternatively, you can download separate binaries for the OpenTelemetry plugin from the ITRS Downloads.

OpenTelemetry environment Copied

The OpenTelemetry plugin requires an instrumented system that emits signals. Please see the OpenTelemetry documentation for language-specific instrumentation details.

Protocol and Security Copied

Protocol Copied

The OpenTelemetry plugin supports the following requests:

• Sequential — recommended where possible.
• Concurrent — recommended over slow networks and/or high data volumes.

If the server responds with an OK status, then all records in the batch are ingested to persistent storage and will never require retransmission. There will be no retransmission either because they are already persisted or are considered non-critical.

The subset of gRPC error codes used are described in the table below:

Using the above mechanisms, the plugin attempts to guarantee at least once semantics. It is possible that retrying a batch will result in 1 or more individual records being ingested multiple times. You may minimize this possibility by sending records sequentially, one per batch. However, this is likely to be far less efficient from a network bandwidth and throughput perspective.

Security Copied

This plugin supports multiple layers of security:

• Transport layer — various forms of TLS.
• Application layer — application keys in the form of username and password.

Specifically, the following transport layer security modes are supported:

• Plain text
• Untrusted TLS — server trusts any client.
• Mutual TLS — server only trusts clients which present an appropriate certificate.

By default, when TLS is enabled, TLSv1.3 and TLSv1.2 are the only protocols allowed. It is possible to explicitly enable others, but they are generally considered less secure.

In addition to TLS, this plugin supports application layer authentication against a suitable IAM provider. In this case, aside from transport layer security, you must provide appropriate keys in the gRPC metadata, which can be used for authentication purposes.

Configure Gateway to receive OpenTelemetry data Copied

The OpenTelemetry plugin supports Collection Agent publication into Geneos using dynamic managed entities. To set up the OpenTelemetry plugin in Geneos, follow these steps:

1. In GSE, go to Dynamic entities > Collectors then create an OpenTelemetry collector with the following configuration:

   collectors:
   - type: plugin
       name: opentelemetry-otlp
       class-name: OpenTelemetryCollector
   
       # Optional. Default is 4317.
       port: 4317
   
       # Optional. gRPC inbound queue capacity. Default is 64.
       queue-capacity: 64
   
       # Optional. gRPC inbound queue processing threads. Default is 4.
       thread-pool-size: 4
   
       # Optional. Inbound username/password authentication against an OpenID IAM provider.
       # This mechanism is compatible with the internal IAM mechanism in the Obcerv Platform.
       authentication:
       # Mandatory. IAM provider credentials.
       iam:
           hostname: iam.domain.com
           port: 443
           client-id: ingestion
           client-secret: password
           admin-realm: master
           client-realm: obcerv
       # Optional. Username whitelist. If not specified then all users are allowed.
       allowed-users:
           - user1
           - user2
   
       # Optional. Use to enable TLS.
       # If omitted, the server accepts any connections over plain text.
       # Otherwise the server enables TLS and can either authenticate clients
       # (i.e. use mTLS) or not depending on whether the trust-chain-file is specified.
       tls:
       # Mandatory.
       cert-file: /path/to/cert.pem
       # Mandatory.
       key-file: /path/to/key.pem
       # Optional. Used only when mTLS is desired.
       trust-chain-file: /path/to/trust-chain.pem
       # Optional. List of TLS protocols to enable. Defaults to TLSv1.3 and TLSv1.2 only.
       protocols: [ TLSv1.3, TLSv1.2 ]
   
       # Optional. Defaults to true.
       # By default, metrics from each trace span are extracted and emitted.
       # This flag allows toggling of that behavior.
       emit-trace-metrics: true
   
       # EXPERIMENTAL. Optional. Defaults to false.
       # By default trace spans are not emitted, however, this toggles
       # trace span emission. This is experimental because it only enables
       # reporting to another OpenTelemetry endpoint. That is, this is
       # useful only in relay mode.
       emit-traces: false
   
       # Optional. Default is 200.
       # For the resource and span attributes mapped (according to the rules below) that are mapped into dimensions,
       # both attribute names and values are truncated to the length specified in order to pass data point validation.
       # A value of 0 switches truncation off completely.
       max-attribute-key-value-length: 200
   
       # Optional. Defines which resource attributes to map to dimensions.
       # If not specified, then the default is to map all 'semantic' attributes
       # (see 'semantic conventions' above).
       resource-attributes:
       # Optional metric resource attributes to map to dimensions. Defaults to 'semantic'.
       metrics:
           # Optional. Defaults to 'semantic', which implies default to mapping any of the
           # resource attributes defined as having special semantic meaning
           # (see 'semantic conventions' above).
           # 'all' implies default to mapping all resource attributes to dimensions.
           # 'none' implies default to mapping no resources attributes to dimensions.
           base: semantic|all|none
           # Additional attributes to include.
           include:
           - custom.attribute
           # Attributes to exclude.
           exclude:
           - another.attribute
       # Optional log resource attributes to map to dimensions. Defaults to 'semantic'.
       logs:
           base: semantic|all|none
           include:
           - custom.attribute
           exclude:
           - another.attribute
       # Optional trace resource attributes to map to dimensions. Defaults to 'semantic'.
       traces:
           base: semantic|all|none
           include:
           - custom.attribute
           exclude:
           - another.attribute
   
       # Optional. Defines which scope attributes to map to dimensions.
       # If not specified the default is to map no scope attributes.
       scope-attributes:
       # Optional scope metric attributes to map to dimensions. Defaults to 'none'.
       metrics:
           # Optional. Defaults to 'none'.
           # 'all' implies default to mapping all resource attributes to dimensions.
           # 'none' implies default to mapping no resources attributes to dimensions.
           base: all|none
           # Additional attributes to include.
           include:
           - custom.attribute
           # Attributes to exclude.
           exclude:
           - another.attribute
       # Optional scope log attributes to map to dimensions. Defaults to 'none'.
       logs:
           base: all|none
           include:
           - custom.attribute
           exclude:
           - another.attribute
       # Optional scope trace attributes to map to dimensions. Defaults to 'none'.
       traces:
           base: all|none
           include:
           - custom.attribute
           exclude:
           - another.attribute    
   
       # Optional. Custom metric unit mappings from external string to a known
       # internal data model representation.
       units:
       # Map Otel metric unit 'By' to internal unit 'bytes'
       # See below for a reference to available internal units.
       By: bytes
   
   # EXPERIMENTAL. Only suitable for relaying traces to another OpenTelemetry endpoint.
   reporters:
   - type: plugin
       class-name: OpenTelemetryReporter
       name: otel-out
   
       # Optional. Reporting target hostname. Defaults to 'localhost'
       hostname: localhost
   
       # Optional. Reporting target port. Defaults to 4317.
       port: 4317
   
       # Optional. Switch compression on/off. Defaults to true (gzip).
       use-compression: true
   
       # Optional. gRPC call deadline in milliseconds. Defaults to 10000.
       call-deadline: 10000
   
       # Optional. Defaults to false.
       # When true, overrides any TLS configuration and uses a plain text connection.
       # When false (default), and there is no 'tls' section, default is to use
       # 'insecure' TLS (i.e. we trust the server certificate presented during TLS handshake)
       # over the more recent and supported TLS protocol out of TLSv1.3 and TLSv1.2.
       use-plain-text: false
   
       # Optional. Used only when use-plain-text is false (the default).
       # When not using plain text this section is only required when mTLS is desired,
       # else the client trusts whatever public key it receives from the server during the
       # TLS handshake.
       tls:
       # Optional. Used only for mTLS.
       cert-file: /path/to/cert.pem
       # Optional. Used only for mTLS.
       key-file: /path/to/key.pem
       # Optional. Used for mTLS and trusted server TLS (i.e. contains trusted server keys).
       trust-chain-file: /path/to/trust-chain.pem
       # Optional. List of TLS protocols to enable. Defaults to TLSv1.3 and TLSv1.2 only.
       protocols: [ TLSv1.3, TLSv1.2 ]
   

   Note

   OpenTelemetry units may be translated to any available internal units, either by symbol or description. For more information, see Metric Unit Definitions.

2. In Dynamic entities > Mapping, configure the mappings to be used. For an example mapping, see Mappings.
3. If you will be monitoring logs, set up an FKM sampler. For more information, see File Keyword Monitor Configuration.
4. In Dynamic entities > Mapping types, create a mapping type with the collector, mapping, and sampler you created in the previous steps.

Note

To check if there are any errors in the mapping, you can set up the Dynamic Entities Health or look at the Collection Agent log file in Collection Agent setup.

Mappings Copied

The data mappings of the signals supported by the OpenTelemetry plugin are as follow:

• Metrics — fully supported except for the deprecated types. All others are mapped to the internal data model in some shape.
• Logs — fully supported.
• Traces — partially supported. There is no internal data model representation for span yet, so golden metrics are extracted from each span.

OpenTelemetry plugin allows you to create custom mappings, so you can modify the display of information in the Geneos dataview. The mappings can be configured differently by using available dimensions or properties.

To create custom mappings for OpenTelemetry:

1. Navigate to your OpenTelemetry mapping in Dynamic entities > Mappings.
2. In Options, select custom.

Example mappings and dataviews Copied

Traces Copied

<mapping name="opentelemetry-traces">
 <custom>
  <filters>
   <filter>
    <label>__itrs_namespace__</label>
    <equals>
     <value>itrsgroup.com/c2/opentelemetry-plugin/traces</value>
    </equals>
   </filter>
  </filters>
  <geneosItems>
   <geneosItem>
    <label>trace_id_hex</label>
    <entity>
     <required>true</required>
     <useInDisplayName>true</useInDisplayName>
    </entity>
   </geneosItem>
  </geneosItems>
 </custom>
</mapping>

Using the custom mappings above, the following dataview is generated:

Metrics Copied

<mapping name="opentelemetry-metrics">
 <custom>
  <filters>
   <filter>
    <label>__itrs_namespace__</label>
    <equals>
     <value>itrsgroup.com/c2/opentelemetry-plugin/traces</value>
    </equals>
   </filter>
  </filters>
  <geneosItems>
   <geneosItem>
    <label>service.name</label>
    <entity>
     <required>true</required>
     <useInDisplayName>true</useInDisplayName>
    </entity>
    </geneosItem>
  </geneosItems>
 </custom>
</mapping>

Using the custom mappings above, the following dataview is generated:

Logs Copied

<mapping name="opentelemetry-logs">
 <custom>
  <filters>
   <filter>
    <label>__itrs_namespace__</label>
    <equals>
     <value>itrsgroup.com/c2/opentelemetry-plugin/traces</value>
    </equals>
   </filter>
  </filters>
  <geneosItems>
   <geneosItem>
    <label>service.name</label>
    <entity>
     <required>true</required>
     <useInDisplayName>true</useInDisplayName>
    </entity>
    </geneosItem>
  </geneosItems>
 </custom>
</mapping>

Using the custom mappings above, the following dataview is generated: