Netprobe Router configuration

Use this page as a reference when you secure and configure a Netprobe Router for productions. It covers TLS and authentication first, then the YAML settings for self-monitoring and the routing pipeline.

For hands-on walkthroughs, see Introduction to Netprobe Router. If something is not working as expected, see Troubleshooting.

Security Copied

You can configure the Netprobe Router to:

• Use TLS for inbound streaming telemetry to RoutingCollector (gRPC server) and, where required, username and password authentication on that collector.
• Use TLS for outbound TCP self-monitoring traffic to a Netprobe Collection Agent reporter (TCP client).
• Use TLS for outbound connections to ITRS Analytics and, where required, username and password authentication on RoutingReporter.

Enable TLS on the Routing Collector (ingress) Copied

To require TLS from senders (Gateways, Netprobes, or other Collection Agents) that connect to the router’s gRPC ingress, add a tls block under the RoutingCollector plugin in collectors. When this block is present, the collector listens with TLS. Omit tls to accept plain-text gRPC (typical for local examples).

certFile and keyFile are the server certificate and private key that the collector presents to clients. Both are required for TLS. For more information, see Routing Collector configuration.

The following example shows a TLS-enabled collector:

collectors:
  - type: plugin
    className: RoutingCollector
    name: routing-grpc
    port: 4317
    tls:
      certFile: /path/to/server-cert.pem
      keyFile: /path/to/server-key.pem

After you enable TLS, senders must use TLS when they open the gRPC connection. If you configure mutual TLS (mTLS), senders must also supply client certificates. Plain-text clients are rejected.

Configure username and password authentication on the Routing Collector Copied

You can require username and password authentication from clients that connect to RoutingCollector. The configuration shape mirrors RoutingReporter credentials, except the collector validates credential headers instead of setting them:

• Internal ingestion service clients authenticate with the ingestion-client-username and ingestion-client-password headers.
• OpenTelemetry clients authenticate with the otel-client-username and otel-client-password headers.

RoutingReporter attaches these headers automatically from its own configuration, so clients can connect to either this collector or an ITRS Analytics instance by changing only the address.

Validation is local to this configuration. The collector does not contact IAM or another external service. Failed authentication is rejected with UNAUTHENTICATED, the same as an ITRS Analytics instance.

If you do not configure credentials for a service, clients of that service are not authenticated.

Do not put real secrets in the YAML file. Use Collection Agent environment-variable interpolation. At startup, ${env:VAR} is replaced with the value of the VAR environment variable defined for the router process. Supply those variables from your secret store at runtime (for example, Kubernetes secrets mounted as environment variables).

Use shared credentials when the same account is used for both the internal ingestion protocol and OTLP:

collectors:
  - type: plugin
    className: RoutingCollector
    name: routing-grpc
    port: 4317
    username: ${env:INGESTION_USERNAME}
    password: ${env:INGESTION_PASSWORD}

Use per-service credentials when different credentials are required:

collectors:
  - type: plugin
    className: RoutingCollector
    name: routing-grpc
    port: 4317
    internal:
      username: ${env:INTERNAL_INGESTION_USERNAME}
      password: ${env:INTERNAL_INGESTION_PASSWORD}
    otel:
      username: ${env:OTEL_INGESTION_USERNAME}
      password: ${env:OTEL_INGESTION_PASSWORD}

For setting details, see Routing Collector configuration.

Enable TLS on the TCP reporter (selfMetrics to Netprobe) Copied

The type: tcp reporter used for the standard workflow (selfMetrics toward Geneos) can use tlsConfig so the router connects to the Netprobe Collection Agent reporter over TLS instead of raw TCP. This matches a Netprobe that listens for TLS on the reporter port.

Typical fields:

• insecure — when true, skips strict server certificate verification (development only). The default is to verify the server.
• trustChainFile — Certificate Authority (CA) or certificate chain used to authenticate the Netprobe server certificate.

The following example shows TLS configuration for a TCP reporter:

reporters:
  - type: tcp
    name: netprobe-reporter
    hostname: netprobe.example.com
    port: 9137
    tlsConfig:
      insecure: false
      trustChainFile: /path/to/netprobe-ca-chain.pem

You must also configure the Netprobe and Gateway:

• Gateway setup (gateway.setup.xml) — update the Netprobe collection details to connect to Netprobe securely.

• Netprobe command line — add the public and private keys to the Netprobe command line:

  ./netprobe.linux_64 -secure -ssl-certificate /path/to/netprobe-cert.pem -ssl-certificate-key /path/to/netprobe-key.pem
  

• Gateway command line — add the Netprobe certificate chain to the Gateway command line:

  ./gateway2.linux_64 -demo -ssl-certificate-chain /path/to/netprobe-ca-chain.pem
  

Connect to ITRS Analytics by using HTTPS/TLS Copied

To secure the connection from the router to ITRS Analytics, configure each RoutingReporter to use TLS:

• Set the ITRS Analytics ingestion hostname and port on the reporter.
• Leave usePlainText as false or omit it.
• Add a tls section.
• Provide a trustChainFile when the server certificate must be validated against a known CA or internal trust chain.

The following example shows TLS configuration for a RoutingReporter:

reporters:
  - type: plugin
    className: RoutingReporter
    name: iax
    hostname: ingestion.example.com
    port: 443
    usePlainText: false
    tls:
      trustChainFile: /path/to/iax-ca-chain.pem

The supplied Unix test receivers do not provide an HTTPS endpoint, so you cannot validate this configuration by using only the local example harnesses. In production, validate TLS connectivity against a real ITRS Analytics ingestion endpoint or another compatible TLS-enabled receiver.

Recommended checks:

1. Confirm that the certificate chain is trusted by the router host or supplied through trustChainFile.
2. Confirm that the server hostname matches the certificate.
3. Confirm that every intermediary device supports HTTP/2 and gRPC.
4. Review router logs for TLS handshake and certificate validation errors.

Configure username and password authentication on the Routing Reporter Copied

You can configure shared credentials or per-service credentials on the RoutingReporter.

Use shared credentials when the same account is used for both the internal ingestion protocol and OTLP:

reporters:
  - type: plugin
    className: RoutingReporter
    name: iax
    hostname: ingestion.example.com
    port: 443
    username: ingestion-user
    password: change-me

Use per-service credentials when different credentials are required:

reporters:
  - type: plugin
    className: RoutingReporter
    name: iax
    hostname: ingestion.example.com
    port: 443
    internal:
      username: internal-user
      password: internal-password
    otel:
      username: otel-user
      password: otel-password

The supplied local test receivers do not implement authenticated HTTPS ingestion. Validate credentials against a real ITRS Analytics environment or another compatible secured endpoint.

Recommended practices:

• Use dedicated service credentials rather than personal user accounts.
• Restrict each account to the minimum required permissions.
• Store secrets in a secure secret-management system and inject them at runtime where possible.
• Combine credentials with TLS so passwords are not sent over plain-text connections.

Configuration Copied

This section describes the YAML settings for the Netprobe Router. Use it as a reference when you configure a router for production or adapt the local examples.

The router uses two separate configuration paths. They share the top-level reporters list but serve different purposes:

Configure self-monitoring when you want Geneos to monitor the router itself. Configure routing when you want the router to receive and forward customer telemetry. Both paths can be used in all deployments.

The sections below follow the self-monitoring path first, then the routing path. Routing path configuration example shows how the routing components fit together. For hands-on walkthroughs, see Run the local examples. For TLS and authentication, see Security.

Self-monitoring configuration Copied

Enable selfMetrics under monitoring to publish the router’s own self-monitoring data back into Geneos.

The monitoring.selfMetrics block switches on Collection Agent self-metrics and adds labels that can be used by Geneos mappings.

monitoring:
  healthProbe:
    enabled: false
  selfMetrics:
    enabled: true
    dimensions:
      hostname: ${env:HOSTNAME}
      app: ${env:APP}

In this configuration:

• enabled: true turns on self-metric generation for the router.
• hostname identifies the host or router instance in Geneos.
• app identifies the application or router role and is useful as a sampler-level label.

Workflow configuration Copied

Configure the top-level workflow section to control how locally generated self-monitoring data is processed and where it is sent.

Note

This is separate from routingWorkflow, which handles forwarded ingress traffic.

workflow:
  storeDirectory: .
  metrics:
    reporter: tcp
    processors:
      - type: plugin
        name: geneos-workflow
        className: GeneosProcessor
  logs:
    reporter: stdout
  events:
    reporter: stdout
  attributes:
    reporter: tcp

In this configuration:

• metrics.reporter: tcp sends self-monitoring metric output toward Geneos.
• GeneosProcessor converts metric data into a form suitable for the Netprobe and Gateway path.
• attributes.reporter: tcp sends self-monitoring attributes to the same destination so Geneos can build the expected entity and sampler structure.
• logs and events remain on stdout because they are not needed for the self-monitoring path in this example.

Reporting configuration Copied

The top-level reporters list defines all Collection Agent reporters, including both workflow sinks and routing plugins. This section documents the standard workflow reporters (logging, tcp) that carry the router’s own self-monitoring data.

Important

This self-monitoring path is independent of the routing path. The reporters here export the router’s own operational data back into Geneos; RoutingReporter plugins forward customer telemetry to ITRS Analytics or other recipients. For more information, see Routing Reporter configuration.

Logging reporter Copied

Send router diagnostics to stdout:

reporters:
  - type: logging
    name: stdout

Gateway reporter Copied

Send self-monitoring data to Geneos over TCP:

reporters:
  - type: tcp
    name: tcp
    hostname: localhost
    port: 9137

In this configuration:

• type: tcp uses the standard Collection Agent TCP reporting path.
• hostname and port must match the listening Netprobe Collection Agent reporter configuration.
• In the supplied gateway.setup.xml, the matching setting is reporterPort with value 9137.

To use TLS on the TCP reporter, see Enable TLS on the TCP reporter (selfMetrics to Netprobe).

Routing Collector configuration Copied

Configure RoutingCollector under collectors to receive inbound streaming telemetry for RoutingWorkflow. It is the only collector that supports this path. It accepts both the ITRS internal ingestion protocol and OTLP on the same port, so add only one instance per Netprobe Router.

collectors:
    # Optional, but recommended component name.
  - name: router-ingress
    # Required.
    type: plugin
    # Required.
    className: RoutingCollector
    # Required. Port on which to expose services.
    # All gRPC services are bound to the same port.
    port: 8200
    # Optional. Set an upper limit on the number of concurrent http/2 streams per connection.
    # Defaults to 2000.
    maxConcurrentCallsPerConnection: 2000
    # Optional. Set the flow control window.
    # Defaults to 64 MiB.
    flowControlWindow: 67108864
    # Client authentication, mirroring the username/password authentication performed
    # by an ITRS Analytics instance so that clients can connect to either this collector
    # or an ITRS Analytics instance, changing only the address. The configuration shape
    # mirrors the Reporter's credentials configuration, except the collector validates the
    # credential headers instead of setting them (internal ingestion service clients
    # authenticate via ingestion-client-username/ingestion-client-password headers,
    # OpenTelemetry clients via otel-client-username/otel-client-password headers;
    # the RoutingReporter attaches these automatically from its own configuration).
    # Validation is entirely local to this configuration - no IAM or other external
    # service is contacted - so values are typically injected via environment
    # variable interpolation (e.g. from Kubernetes secrets). Failures are rejected
    # with UNAUTHENTICATED, exactly as an ITRS Analytics instance would.
    # Shared credentials.
    # Can be used when the internal ingestion service and OTel service credentials
    # are the same.
    # Optional. If no credentials are configured for a service, clients of that
    # service are not authenticated (previous behaviour).
    username: user
    password: pass
    # Per service credentials.
    # Optional. Internal ingestion service credentials.
    # Only required when internal ingestion service and OTel service credentials are different.
    internal:
      username: user
      password: pass
    # Optional. OpenTelemetry service credentials.
    # Only required when internal ingestion service and OTel service credentials are different.
    otel:
      username: user
      password: pass
    # Optional. Use to enable TLS.
    # If omitted, the server accepts any connections over plain text.
    # Otherwise, the server enables TLS and can either authenticate clients
    # (i.e. use mTLS) or not depending on whether the trustChainFile is specified.
    tls:
      # Mandatory.
      certFile: /path/to/cert.pem
      # Mandatory.
      keyFile: /path/to/key.pem
      # Optional. Used only when mTLS is desired.
      trustChainFile: /path/to/trust-chain.pem
      # Optional. List of TLS protocols to enable. Defaults to TLSv1.3 and TLSv1.2 only.
      protocols: [ TLSv1.3, TLSv1.2 ]

Routing workflow configuration Copied

Configure RoutingWorkflow under the top-level routingWorkflow key (camelCase) to define routing rules and delivery behavior for inbound telemetry.

Here is an example:

routingWorkflow:

  # Circuit breaker auto-close timeout in milliseconds.
  # When a destination reporter fails, its circuit breaker opens and all
  # data destined for it is dropped until the timeout expires, at which
  # point the breaker closes and delivery resumes.
  # Optional. Defaults to 10000 (10 seconds).
  autoCloseTimeout: 10000

  # Global enrichments applied to every batch before route evaluation.
  # Optional. See the Enrichment section.
  enrich:
    - # ... enrichment configuration (see below)

  # List of route definitions. Required.
  routes:
    - # ... route configuration (see [Routes](#routes))

Routes Copied

Define each route in the routes list under routingWorkflow to specify destination reporters and, optionally, conditions that filter which data points are forwarded to them.

routingWorkflow:
  routes:
    - reporters: [ reporter-a, reporter-b ]
      scope: all
      any:
        - field: dimensions
          operator: contains
          key: service.namespace
          value: production

Important

If neither any nor all is specified, the route matches all data points unconditionally. You must specify exactly one of any or all if conditions are desired. Specifying both is a configuration error.

Batch scope Copied

Data arrives at RoutingWorkflow in batches. The scope setting controls how conditions are applied to the batch:

• all (default): Every data point in the batch is individually tested against the conditions. Only matching data points are forwarded. Non-matching data points are silently dropped from the batch for this route.
• first: Only the first data point in the batch is tested. If it matches, the entire batch is forwarded. If it does not match, the entire batch is dropped for this route. This is an optimization for cases where all data points in a batch are known to originate from the same source and share the same routing-relevant fields, for example, all from the same Geneos Gateway or the same Kubernetes namespace.

Multi-route evaluation Copied

All routes are evaluated for every incoming batch. A single batch may match multiple routes and be delivered to different sets of reporters simultaneously. Routes are evaluated independently; there is no first-match short-circuiting.

Routing conditions Copied

A condition compares one field of a data point to a value using an operator.

- field: <field_type>
  operator: <operator>
  key: <key>          # required for map-valued fields (dimensions, properties)
  value: <value>      # required for most operators

Fields Copied

Each field accesses a specific attribute of the data point. The available fields depend on the data point type being evaluated, but the configuration is unified. RoutingWorkflow automatically selects the appropriate field accessor for the data type being evaluated.

When a field is not applicable to the data type being evaluated (for example, value on a log event) the condition evaluates to true, allowing conditions to coexist across different data types without unintended filtering.

Type values Copied

Ingestion service types:

OpenTelemetry types:

Severity values Copied

Operators Copied

Key and value semantics Copied

• key is required when the field resolves to a map, such as dimensions or properties. It identifies which map entry to evaluate. For non-map fields, key is ignored.
• value is required for all operators except contains and not_contains on map fields, where omitting value tests for key existence alone.
• For the matches operator, value is a Java regular expression as accepted by java.util.regex.Pattern.

Condition examples Copied

Route all gauge metrics:

any:
  - field: type
    operator: eq
    value: gauge

Route data points with a specific resource attribute (OpenTelemetry dimension):

all:
  - field: dimensions
    operator: eq
    key: service.namespace
    value: production

Route data points whose name matches a pattern:

any:
  - field: name
    operator: matches
    value: "myapp\\..*"

Route log events with severity error or critical:

any:
  - field: severity
    operator: eq
    value: error
  - field: severity
    operator: eq
    value: critical

Route metrics where a dimension key exists regardless of value:

any:
  - field: dimensions
    operator: contains
    key: k8s.pod.name

Route only OpenTelemetry data:

any:
  - field: type
    operator: starts_with
    value: otel_

Route gauges with a value above a threshold:

all:
  - field: type
    operator: eq
    value: gauge
  - field: value
    operator: gt
    value: "100.0"

Enrichment Copied

Enrichment modifies the dimensions or properties of data points as they pass through the routing workflow. Its primary purpose is to guarantee and harmonize metadata for downstream consumers: ensuring a key is always present, normalizing key names and values from heterogeneous sources, and removing internal keys before delivery.

Enrichment is deliberately lightweight — it is not a general transformation engine. Only the dimensions and properties fields can be enriched.

You can configure enrichments in two places. Placement determines both when they apply and what can see the result:

routingWorkflow:

  # Global: applied once per incoming batch, BEFORE route evaluation.
  # Enriched values are visible to every route's conditions.
  enrich:
    - field: dimensions
      action: rename
      key: env
      value: environment

  routes:
    - reporters: [ prod-ingest ]
      all:
        # Matches on the normalised key guaranteed by the global block above.
        - field: dimensions
          operator: eq
          key: environment
          value: production

      # Per-route: applied to the data points matching THIS route, AFTER
      # condition evaluation and before delivery to this route's reporters.
      enrich:
        - field: properties
          action: delete
          key: internal.debug

The processing order for every batch is:

1. Global enrichments (in declared order) — normalize everything before routing.
2. Route condition evaluation — conditions can use globally enriched values.
3. Per-route enrichments (in declared order) — harmonize for that route’s destinations only.
4. Delivery — store-and-forward reporters persist the fully enriched batch, so replays after a restart are identical to live delivery.

Per-route enrichment is applied only to data points that match the route, and its effect is visible only to that route’s reporters. A batch matching several routes can be delivered with different enrichment per destination. Data points are never modified in place. Enrichment creates copies, so routes cannot affect each other or the collector.

Enrichment settings Copied

- field: <dimensions|properties>   # required
  action: <action>                 # required
  key: <key>                       # required - the map key to operate on
  value: <value>                   # required for set, set_if_absent, rename, prepend, append
  pattern: <regex>                 # required for sub
  replacement: <replacement>       # required for sub
  when:                            # optional conditional guard
    any: [ <conditions> ]          # exactly one of any/all
    all: [ <conditions> ]

Actions Copied

Enrichments within a block apply in declared order, so actions compose: a set followed by a sub operates on the value just set.

OpenTelemetry value typing Copied

For ingestion service data types, dimensions and properties are string maps, and all actions apply directly. OpenTelemetry attributes are typed (AnyValue):

• set and set_if_absent write string values.
• prepend, append, and sub apply only where the existing value is a string; other value types are left untouched (no-op).
• rename preserves the existing value regardless of its type.
• delete removes the key regardless of its value type.

For OpenTelemetry metrics, properties enrichment applies to the attributes of every time-series data point within the metric. This matches how properties conditions read across all data points. For logs and spans, it applies to the log record or span attributes. dimensions enrichment applies to the resource attributes for all three.

Conditional enrichment Copied

Any enrichment entry can include a when guard that uses the same condition lists as routes. The enrichment is applied only to data points that match the guard:

enrich:
  # Tag error events for downstream alerting - guard field differs from target field.
  - when:
      all:
        - field: severity
          operator: eq
          value: error
    field: dimensions
    action: set
    key: alerting
    value: "true"

  # Only stamp OpenTelemetry data.
  - when:
      any:
        - field: type
          operator: starts_with
          value: otel_
    field: dimensions
    action: set_if_absent
    key: telemetry.source
    value: opentelemetry

Guard semantics:

• Exactly one of any (logical OR) or all (logical AND) can be present. Conditions use the same form, fields, and operators as routing conditions, including fields that enrichment cannot write (type, name, severity, value, and so on).
• Guards are evaluated per data point, against its current state in the enrichment chain — a guard on a later entry can use the effects of earlier entries.
• A guard condition on a field that is not applicable to the data type evaluates to true (match-all), the same as for routing conditions.
• An entry without when (or with an empty when) applies unconditionally.

Performance considerations Copied

• Batches and data points are never mutated in place. Enrichment rebuilds only the affected data points (copy-on-write). Data points for which an enrichment is a no-op are passed through unchanged by reference.
• When no enrich blocks are configured, the workflow keeps its zero-copy pass-through and OpenTelemetry re-roll caching behavior.
• Global enrichment cost is incurred once per batch, regardless of route count. Per-route enrichment cost is incurred per enriching route.
• A route with enrichments always produces a new (filtered and enriched) batch, so the OpenTelemetry re-roll cache for catch-all routes does not apply to that route. Other routes are unaffected.

Routing Reporter configuration Copied

RoutingReporter forwards streaming telemetry from RoutingWorkflow to downstream destinations. It accepts both the ITRS internal ingestion protocol and OTLP on the same port. Configure one reporter instance per destination.

Delivery mode is set per reporter via storeAndForward:

• Best-effort (default) — send once; failed messages are dropped.
• Store-and-forward — queue messages on disk and retry until successful. Improves resilience but uses disk space; messages are dropped when storage is full.

Different destinations on the same route can use different modes.

reporters:
  - type: plugin
    className: RoutingReporter
    # Required. Name used by routing rules to identify this reporter.
    name: destination
    # Optional. Reporting target hostname. Defaults to 'localhost'
    hostname: localhost
    # Optional. Reporting target port. Defaults to 4317.
    port: 4317
    # Optional. Number of channels (connections) to use for this destination. Defaults to 8.
    # Traffic sent to this destination is balanced across all channels.
    numChannels: 8
    # Optional. Outbound gRPC client: max inbound message size (bytes). Default 8192 (8 KiB).
    maxInboundMessageSize: 8192
    # Optional. Outbound gRPC client: max inbound metadata size (bytes). Default 8192 (8 KiB).
    maxInboundMetadataSize: 8192
    # Optional. Outbound gRPC client: HTTP/2 flow-control window (bytes). Default 67108864 (64 MiB).
    # Distinct from RoutingCollector flowControlWindow, which applies to the ingress server.
    flowControlWindow: 67108864
    # Optional. Outbound gRPC keep-alive time in milliseconds. Default 60000.
    keepAliveTime: 60000
    # Optional. Outbound gRPC keep-alive timeout in milliseconds. Default 20000.
    keepAliveTimeout: 20000
    # Optional. Send keep-alive pings when there are no active RPCs. Default true.
    keepAliveWithoutCalls: true
    # Optional. Switch compression on/off. Defaults to true (gzip).
    useCompression: true
    # Optional. gRPC call deadline in milliseconds. Defaults to 10000.
    callDeadline: 10000
    # ITRS Analytics authentication shared credentials.
    # Can be used when authentication is enabled and internal ingestion service and otel
    # service credentials are the same.
    # Optional. Only required when username/password based auth is enabled on the target.
    username: user
    # Optional. Only required when username/password based auth is enabled on the target.
    password: pass
    # ITRS Analytics authentication per service credentials.
    # Optional. Internal ingestion service credentials.
    # Only required when internal ingestion service and OTel service credentials are different.
    internal:
      username: user
      password: pass
    # Optional. OpenTelemetry service credentials.
    # Only required when internal ingestion service and OTel service credentials are different.
    otel:
      username: user
      password: pass
    # TLS configuration
    # Optional. Defaults to false.
    # When false (default), and there is no 'tls' section, default is to use
    # 'insecure' TLS (i.e. we trust the server certificate presented during TLS handshake)
    # over the more recent and supported TLS protocol out of TLSv1.3 and TLSv1.2.
    # When true, overrides any TLS configuration and uses a plain text connection.
    usePlainText: false
    # Optional. Used only when usePlainText is false (the default).
    # When not using plain text this section is only required when mTLS is desired,
    # else the client trusts whatever public key it receives from the server during the
    # TLS handshake.
    tls:
      # Optional. Used only for mTLS.
      certFile: /path/to/cert.pem
      # Optional. Used only for mTLS.
      keyFile: /path/to/key.pem
      # Optional. Used for mTLS and trusted server TLS (i.e. contains trusted server keys).
      trustChainFile: /path/to/trust-chain.pem
      # Optional. List of TLS protocols to enable. Defaults to TLSv1.3 and TLSv1.2 only.
      protocols: [ TLSv1.3, TLSv1.2 ]
    # Optional. Specify store and forward based delivery for at least once delivery.
    # Note: This section is not specific to the OpenTelemetry Reporter in that it is
    #       also available for all other reporters. It is repeated here for convenience.
    storeAndForward:
      # Optional. Root directory for store and forward persisted messages.
      # Defaults to current working directory.
      directory: .
      # Optional. Store capacity. Defaults to 10000000 messages (1 message is 1 telemetry batch).
      capacity: 10000000
      # Optional. Max length of a single store file. Defaults to 1 GiB.
      maxFileLength: 1073741824
      # Optional. Number of milliseconds between explicit forced flushes.
      # -1 (default): no explicit flush (highest performance, slight risk of message loss)
      # 0: explicit flush on each write (highest resilience, lowest performance especially on slow disks)
      # N: explicit flush every N ms.
      flushOnWrite: -1
      # Optional. Switch on advisory file locking. Defaults to false.
      fileLocking: false

The following optional keys configure the outbound gRPC client channel for this RoutingReporter. They are separate from RoutingCollector settings such as flowControlWindow, which apply to the router’s ingress server.

Best-effort delivery Copied

This is the default delivery mode. When storeAndForward is not configured on a reporter, delivery is best-effort:

1. RoutingWorkflow invokes the reporter’s async delivery method on the collector’s thread.
2. If the reporter succeeds (the onSuccess callback), the circuit breaker is updated and a success metric is recorded.
3. If the reporter fails (the onFailure callback), the circuit breaker opens and a failure metric is recorded. The failed batch is not retried. While the breaker is open for autoCloseTimeout milliseconds, all subsequent batches destined for this reporter are silently dropped.
4. After the timeout expires, the breaker auto-closes and delivery resumes with the next incoming batch.

Characteristics:

• No disk I/O on the delivery path.
• No buffering. If the reporter is unavailable, data is lost.
• Minimal resource consumption.
• Self-metrics track dropped batches and data points.

When to use:

• Development, testing, and non-production environments.
• Destinations that are not expected to be reliably available at all times.
• Scenarios where the overhead of disk-based buffering is unacceptable.
• When data loss during brief reporter outages is tolerable.

Store-and-forward delivery Copied

When storeAndForward is configured on a reporter, RoutingWorkflow adds a disk-backed store between the routing logic and the actual reporter:

1. RoutingWorkflow writes each batch to an on-disk sequential store on the collector’s thread. This operation is fast (it is an append to a memory-mapped file) and decouples the collector from reporter availability.
2. A dedicated background delivery thread per data type reads batches from the store, deserializes them, and delivers them to the actual reporter.
3. On successful delivery, the store entry is committed and the read cursor advances.
4. On failed delivery, the batch is retried with a 1-second backoff. A per-store circuit breaker prevents flooding a failing reporter. When the breaker is open, one retry is attempted periodically to test for recovery. When it closes again, all pending retries are delivered.
5. Corrupt store entries, for example, from an unclean shutdown—are skipped and logged.

Seven independent stores are created per reporter, one for each data type:

Store directories are created under <storeAndForward.directory>/<RoutingWorkflow-instance-name>/<reporter-name>/, where RoutingWorkflow-instance-name is the on-disk folder name for that workflow (derived from the configured routingWorkflow instance).

Characteristics:

• Survives process restarts. Undelivered batches are replayed on startup.
• At-least-once delivery semantics.
• Callbacks are processed in strict FIFO order. Store entries are only committed after successful delivery.
• Disk consumption is proportional to throughput and reporter downtime. In high-throughput scenarios, disk space can be consumed very rapidly.
• If the store reaches capacity, the batch is dropped and an InsufficientCapacityException is raised, but the circuit breaker is not tripped, on the assumption that capacity exhaustion is a transient condition.

When to use:

• Production environments where data loss is unacceptable.
• Destinations with variable availability, for example, remote endpoints that may experience temporary outages.
• When there is sufficient disk capacity to buffer the expected throughput during the longest anticipated reporter outage.

Store-and-forward configuration Copied

Store-and-forward is configured in reporters under the storeAndForward key:

reporters:
  - type: plugin
    name: iax
    className: RoutingReporter
    hostname: ingestion-iax.com
    port: 443
    # more reporter config

    # Store and forward configuration.
    # Optional, but strongly recommended for production use.
    # Without this routed messages are not buffered or retried and are therefore silently dropped on error.
    storeAndForward:
      # Optional. Defaults to true.
      enabled: true
      # Path to use for on disk store.
      # Optional. Defaults to a relative path in the current working directory.
      directory: ./some/relative/path
      # Maximum number of batches the store can hold before dropping new writes.
      # Optional. Defaults to 10,000,000.
      capacity: 10000000
      # Maximum size in bytes of a single store file.
      # Optional. Defaults to 1,073,741,824 (1 GiB).
      maxFileLength: 1073741824
      # Control force-flush behaviour for writes.
      #   -1: disable force flush (default)
      #    0: force flush on every write
      #   >0: force flush at approximately this interval in milliseconds
      # Optional. Defaults to -1.
      flushOnWrite: -1

Note

The enabled property defaults to true when the storeAndForward block is present. Set enabled: false to explicitly disable it without removing the configuration.

Note

The maxRetries and maxAges settings from the store-and-forward configuration are not used by the store-and-forward path in RoutingWorkflow. RoutingWorkflow retries indefinitely until delivery succeeds or the store entry is committed. This differs from the standard workflow store-and-forward behavior.

Routing workflow self-metrics Copied

RoutingWorkflow emits the following selfMetrics, dimensioned by reporter (destination name) and kind (data type):

The kind dimension takes one of: metrics, logs, events, attributes, otel_metrics, otel_logs, otel_traces.

For store-and-forward reporters, the delivery task tracks its own metrics independently. For best-effort reporters, RoutingWorkflow tracks metrics directly.

Routing path configuration example Copied

collectors:
  - name: ingestion-in
    type: plugin
    className: RoutingCollector
    port: 8200
    maxConcurrentCallsPerConnection: 2000
    flowControlWindow: 8000000

reporters:
  # A best-effort destination (no store-and-forward).
  - type: plugin
    name: dev-ingest
    className: RoutingReporter

  # A production destination with store-and-forward.
  - type: plugin
    name: prod-ingest
    className: RoutingReporter
    storeAndForward:
      # Enabled with defaults.
      enabled: true

routingWorkflow:
  autoCloseTimeout: 15000

  # Normalise every batch before routing: harmonise the environment key so the
  # route below can rely on it, whatever the source called it.
  enrich:
    - field: dimensions
      action: rename
      key: env
      value: environment
    - field: dimensions
      action: set_if_absent
      key: environment
      value: unknown

  routes:
    # Route production metrics and logs to the production ingest service.
    - reporters: [ prod-ingest ]
      scope: all
      all:
        - field: dimensions
          operator: eq
          key: environment
          value: production
      # Harmonise for this destination only.
      enrich:
        - field: properties
          action: delete
          key: internal.debug

    # Send everything to the dev sink (no conditions = pass-through).
    - reporters: [ dev-ingest ]

In this example:

• Before any route is evaluated, every batch has its env dimension renamed to environment, and environment is guaranteed to exist (defaulting to unknown).
• Production ingestion service data matching environment=production is delivered with store-and-forward resilience, with the internal internal.debug property stripped for that destination only.
• Everything, including duplicates of the above, is also sent to dev-ingest in best-effort mode, with internal.debug left intact.